Neural Networks: Zero to Hero

2023-10 — 2023-11 · notes · 9 posts

I have been on parental leave over the past month and I happen to have a bit of time on my hands.

I decided to spend this time going through Andrej Karpathy's Zero-to-Hero course to learn more about neural nets and especially GPTs, which is something I have not worked with before.

Andrej is an absolute legend and this is by far the best hands on tutorial and explanation of language models I have found. I am really glad I spent the time going through it!

All of my code is here:

Lessons

notes · 2023-10 Makemore Part 1 — Bigram Model counting bigrams, sampling names, and a first loss function

notes · 2023-10 Makemore Part 2 — Multilayer Perceptron character embeddings and a hidden layer, after Bengio 2003

notes · 2023-10 Makemore Part 3 — Activations & Gradients, BatchNorm why inits matter: saturated tanh, dead neurons, and batch norm

notes · 2023-10 Makemore Part 4 — Becoming a Backprop Ninja backprop through the whole MLP by hand, checked against PyTorch

notes · 2023-11 Makemore Part 5 — Building a WaveNet a deeper, hierarchical makemore with WaveNet-style blocks

notes · 2023-11 Let's Build GPT: from scratch, in code, spelled out self-attention and a small transformer, token by token

Backprop, by hand

Side quests from Part 4 — deriving the backward passes on paper until they agreed with PyTorch.

notes · 2023-10 Backpropagating through matrix multiplication deriving dW and dx on paper until they matched PyTorch

notes · 2023-10 Backpropagating through cross entropy the softmax + NLL derivative, both branches, written in full

notes · 2023-10 Backpropagating through batch norm the batch norm backward pass, node by node