[{"content":"Why a Blog? I\u0026rsquo;ve been meaning to start writing for a while. The idea is simple: if I can explain something clearly, I actually understand it. This blog is my attempt to solidify what I learn by teaching it to my future self (and anyone else who stumbles here).\nWhat to Expect Here\u0026rsquo;s the kind of stuff you\u0026rsquo;ll find here:\nMachine Learning \u0026amp; Deep Learning: from theory to implementation Systems \u0026amp; Infrastructure: performance, deployment, and architecture Dev Tools \u0026amp; Workflows: things that make coding life better Learning Logs: raw notes from my journey through various topics The Philosophy I\u0026rsquo;m a big believer in learning in public. Not everything here will be polished or perfect, and that\u0026rsquo;s the point. Some posts will be deep dives, others will be quick notes. The goal is consistency over perfection.\nCode Will Be Involved Since this is a technical blog, expect code. Lots of it.\n1 2 3 4 5 6 def hello(): \u0026#34;\u0026#34;\u0026#34;The obligatory first function.\u0026#34;\u0026#34;\u0026#34; print(\u0026#34;Hello, World!\u0026#34;) print(\u0026#34;Welcome to the blog.\u0026#34;) hello() And sometimes we\u0026rsquo;ll get into the math too:\n$$ \\mathcal{L}(\\theta) = -\\frac{1}{N} \\sum_{i=1}^{N} \\left[ y_i \\log(\\hat{y}_i) + (1 - y_i) \\log(1 - \\hat{y}_i) \\right] $$ Binary cross-entropy: the loss function that haunts every ML engineer\u0026rsquo;s dreams.\nLet\u0026rsquo;s Go That\u0026rsquo;s it for the intro. The real content starts with the next post. Stay tuned.\nThanks for reading! If you have questions or feedback, feel free to open an issue on the GitHub repo.\n","permalink":"https://issabawwab.pages.dev/posts/hello-world/","summary":"The first post: why I decided to start writing about what I learn and build.","title":"Hello World: Why I Started This Blog"},{"content":"What is Micrograd? Micrograd is Andrej Karpathy\u0026rsquo;s minimalist autograd engine. It implements backpropagation over a dynamically built computation graph: the same algorithm used in larger deep learning frameworks, just reduced to scalar values.\nThe Python version is small enough to read in one sitting. I wanted to build the same idea in Rust, where the graph has to be explicit and shared nodes need to be handled carefully.\nWhy Rust? Ownership: the graph is made of nodes that point back to their parents, so shared ownership matters. Interior mutability: gradients are updated during backpropagation even when nodes are shared. Learning: implementing autograd in Rust makes the data flow harder to hand-wave. The Value Struct At the heart of the crate is ValueData, a scalar plus the metadata needed for backpropagation:\n1 2 3 4 5 6 7 8 9 use std::cell::RefCell; use std::rc::Rc; pub struct ValueData { pub data: f32, pub op: Operation, pub gradient: f32, pub parents: Vec\u0026lt;Value\u0026gt;, } The public handle is Value, which wraps Rc\u0026lt;RefCell\u0026lt;ValueData\u0026gt;\u0026gt;:\n1 2 #[derive(Debug, Clone)] pub struct Value(Rc\u0026lt;RefCell\u0026lt;ValueData\u0026gt;\u0026gt;); Rc lets multiple nodes keep references to the same parent. RefCell lets the code update gradients later, during the backward pass. That is the main Rust-specific part of this implementation.\nForward Pass Creating a leaf node stores the number and marks the operation as None:\n1 2 3 4 5 6 7 8 9 10 impl Value { pub fn new(data: f32) -\u0026gt; Value { Value(Rc::new(RefCell::new(ValueData { data, op: Operation::None, gradient: 0.0, parents: vec![], }))) } } Each operation creates a new Value and stores its parents. Addition is direct: the local derivative with respect to both inputs is 1.\n1 2 3 4 5 6 7 8 9 10 11 12 13 impl\u0026lt;\u0026#39;a, \u0026#39;b\u0026gt; Add\u0026lt;\u0026amp;\u0026#39;b Value\u0026gt; for \u0026amp;\u0026#39;a Value { type Output = Value; fn add(self, other: \u0026amp;\u0026#39;b Value) -\u0026gt; Value { let data = self.data() + other.data(); Value(Rc::new(RefCell::new(ValueData { data, op: Operation::Add, gradient: 0.0, parents: vec![self.clone(), other.clone()], }))) } } Multiplication is the same pattern, but the backward rule needs the other input\u0026rsquo;s value:\n1 2 3 4 5 6 7 8 9 10 11 12 13 impl\u0026lt;\u0026#39;a, \u0026#39;b\u0026gt; Mul\u0026lt;\u0026amp;\u0026#39;b Value\u0026gt; for \u0026amp;\u0026#39;a Value { type Output = Value; fn mul(self, rhs: \u0026amp;\u0026#39;b Value) -\u0026gt; Self::Output { let data = self.data() * rhs.data(); Value(Rc::new(RefCell::new(ValueData { data, op: Operation::Mul, gradient: 0.0, parents: vec![self.clone(), rhs.clone()], }))) } } Backward Pass The backward pass computes gradients using the chain rule:\n$$ \\frac{\\partial L}{\\partial x} = \\frac{\\partial L}{\\partial y} \\cdot \\frac{\\partial y}{\\partial x} $$The idea is simple: every node knows how it was created, so it knows how to pass its gradient to its parents.\nFor this expression:\n$$ d = (a \\\\cdot b) + (a + b) $$Here is the backward pass one step at a time:\nStep by step backward pass for d equals a times b plus a plus b A computation graph showing how gradients move backward from d to a and b. d = (a * b) + (a + b) a grad = b + 1 b grad = a + 1 * e = a * b grad = 1 + c = a + b grad = 1 + d grad = 1 * 1 * 1 * b * a * 1 * 1 1. Forward graph\nFirst build the expression graph: e = a * b, c = a + b, and d = e + c.\n2. Seed the output\nThe final value starts with d.grad = 1 because the derivative of d with respect to itself is 1.\n3. Backprop through d = e + c\nAddition passes the gradient unchanged, so e.grad and c.grad both receive 1.\n4. Backprop through e = a * b\nMultiplication sends the other input to each parent: a receives b, and b receives a.\n5. Backprop through c = a + b\nAddition sends 1 to both parents, so a receives another 1 and b receives another 1.\n6. Accumulate gradients\nThe contributions add up: a.grad = b + 1 and b.grad = a + 1.\nPrevious Next d is the final output, so its gradient starts at 1.0:\n$$ \\frac{\\partial d}{\\partial d} = 1 $$Then the graph is walked backwards. Since d = e + c, the gradient from d flows unchanged into both e and c:\n$$ \\frac{\\partial d}{\\partial e} = 1 $$$$ \\frac{\\partial d}{\\partial c} = 1 $$Now $e = a \\cdot b$, so multiplication sends the other input to each parent. a receives b, and b receives a:\n$$ \\frac{\\partial e}{\\partial a} = b $$$$ \\frac{\\partial e}{\\partial b} = a $$And c = a + b, so both parents receive 1.\nThe important detail is that gradients add up. a is used in both e and c, so its final gradient gets one contribution from the multiply node and one from the add node.\nFor each node, backward applies the local derivative and adds into each parent\u0026rsquo;s gradient:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 pub fn backward(\u0026amp;mut self) { match self.op() { Operation::Add =\u0026gt; { for parent in \u0026amp;self.0.borrow().parents { parent.0.borrow_mut().gradient += self.gradient() * 1.0; } } Operation::Mul =\u0026gt; { let data = self.0.borrow(); let p1 = \u0026amp;data.parents[0]; let p2 = \u0026amp;data.parents[1]; let p1_data = p1.data(); let p2_data = p2.data(); p1.0.borrow_mut().gradient += self.gradient() * p2_data; p2.0.borrow_mut().gradient += self.gradient() * p1_data; } Operation::None =\u0026gt; {} } } For a multiply node:\n$$ y = a \\cdot b $$So:\n$$ \\frac{\\partial y}{\\partial a} = b $$and:\n$$ \\frac{\\partial y}{\\partial b} = a $$That is why the code adds self.gradient() * p2_data to the first parent and self.gradient() * p1_data to the second parent.\nTopological Sort Before running backward, the graph needs to be ordered from the final output back through its dependencies. The crate does this with topo_sort:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 pub fn topo_sort(last: Value) -\u0026gt; Vec\u0026lt;Value\u0026gt; { let mut sorted: Vec\u0026lt;Value\u0026gt; = Vec::new(); let mut visited: Vec\u0026lt;Value\u0026gt; = vec![last.clone()]; let mut stack: Vec\u0026lt;Value\u0026gt; = vec![last.clone()]; while !stack.is_empty() { if let Some(last) = stack.last().cloned() { let mut unvisited_flag = false; for parent in \u0026amp;last.0.borrow().parents { if !visited.contains(parent) { visited.push(parent.clone()); stack.push(parent.clone()); unvisited_flag = true; } } if !unvisited_flag \u0026amp;\u0026amp; let Some(last_item) = stack.pop() { sorted.push(last_item); } } } sorted.reverse(); sorted[0].0.borrow_mut().gradient = 1.0; sorted } The output node starts with a gradient of 1.0, because:\n$$ \\frac{\\partial L}{\\partial L} = 1 $$Then back_propogate walks the sorted list and calls backward on each node:\n1 2 3 4 5 pub fn back_propogate(last: Value) -\u0026gt; Vec\u0026lt;Value\u0026gt; { let mut list = topo_sort(last); list.iter_mut().for_each(|node| node.backward()); list } The function name is currently spelled back_propogate in the crate, so the post uses the same spelling.\nExample Here is the example from main.rs:\n1 2 3 4 5 6 7 8 9 10 use microgradrs::{Value, back_propogate}; fn main() { let a = Value::new(1.0); let b = Value::new(2.0); let c = \u0026amp;a + \u0026amp;b; let d = \u0026amp;(\u0026amp;a * \u0026amp;b) + \u0026amp;c; let result = back_propogate(d); println!(\u0026#34;{:#?}\u0026#34;, result); } The expression is:\n$$ d = (a \\\\cdot b) + (a + b) $$With a = 1 and b = 2, the gradients are:\n$$ \\frac{\\partial d}{\\partial a} = b + 1 = 3 $$$$ \\frac{\\partial d}{\\partial b} = a + 1 = 2 $$What I Learned Rc\u0026lt;RefCell\u0026lt;T\u0026gt;\u0026gt; fits this small graph well because nodes need shared ownership and delayed mutation. Pointer equality matters. PartialEq uses Rc::ptr_eq, so two handles are equal only when they point to the same node. The topological order is the part that makes the backward pass work. Without it, a node can be processed before its gradient is complete. Next Steps Add more operations like subtraction, division, powers, relu, and tanh. Add tests for gradients, not just forward values. Build a tiny neural network layer on top of Value. This is based on my microgradrs project.\n","permalink":"https://issabawwab.pages.dev/posts/micrograd-from-scratch/","summary":"Implementing a small scalar autograd engine in Rust, based on Karpathy\u0026rsquo;s micrograd.","title":"Building Micrograd from Scratch in Rust"}]