Backpropagating through matrix multiplication
I’m currently going through Karpathy’s excellent Zero to Hero course and came across a fun exercise to backpropagate through a matrix multiplication by hand.
https://www.youtube.com/watch?v=q8SA3rM6ckI
Its been a while since I have done any linear algebra so I decided to write it all out myself and really try to understand every single line. I decided to use the excalidraw tool even though it isn’t great for writing out math because it has a nice handwriting font.
It is neat that after all of this, the derived formulas are so simple. If I hadn’t actually gone through this math, it would have been very easy for me to just copy the code and call it a day.
Thanks Karpathy for encouraging us to go one level deeper and to examine things from first principles!