Post by Puru Kathuria

Forty years of AI. Eight architectures. One equation. y = f(Wx + b) I've been building AI systems for years (first at IIIT Delhi, then at MathWorks and Google) and this equation never stops surprising me. x is your input. W is a matrix of learned weights. b shifts the result. f bends it. —-  Watch what happens as we change one thing at a time: → Linear regression: f is identity. A straight line through your data. → Logistic regression: f becomes a sigmoid. The line becomes a probability. → Neural network: nest it. a₁ = f(W₁x + b₁), then a₂ = f(W₂a₁ + b₂). Here's what most tutorials skip: the nonlinearity f is the entire point. Remove it, and a 100-layer network mathematically collapses back into a single line. → CNN: same equation, but W is forced to be tiny and reused across every patch of the image. Convolution is just matrix multiplication with a built-in belief: "an edge detector that works in one corner of the photo should work in every corner." → RNN: same equation, but now W is reused across time instead of space: hₜ = f(Wₕ·hₜ₋₁ + Wₓ·xₜ + b). The hidden state hₜ is the network's memory. It's just yesterday's output, multiplied by weights, fed back in as today's input. → Transformer: the twist that changed everything. Q, K, and V are each just Wx — three different weight matrices, same input. Then attention computes softmax(QKᵀ/√d)·V. Read that carefully. The mixing weights are no longer fixed parameters learned during training. They are computed from the data itself, fresh for every input. For the first time, W became dynamic. The model decides, at inference time, what to pay attention to. → GPT: stack that block ~100 times, scale W to a trillion parameters, and train with one loop: How wrong was the prediction? Nudge every W and b slightly downhill. Repeat a few quadrillion times. (Backprop itself is just the chain rule applied to nested f(Wx + b).) — Here's the part I wish someone had told me earlier: These architectures don't differ in the math. They differ in what they assume about the world. CNN: "patterns repeat across space" → W shared spatially RNN: "the past explains the present" → W shared temporally Transformer: "let the data decide what matters" → W computed at runtime An architecture is an assumption about the world, written in linear algebra. Once you see that, you stop memorizing architectures and start asking the only question that matters: what assumption does my problem deserve?