Backpropagating through cross entropy
Okay if you are like me, you probably struggled a bit to follow the math from exercise 2 in this video.
https://youtu.be/q8SA3rM6ckI?si=on2Bu9lDcWlOtIwQ&t=5453
The first section containing the problem setup was pretty clear, though some of the handwriting is barely legible 😅
Here is my version.
This first part of his derivation was also pretty straightforward. Note that we are skipping the mean part since we are only trying to compute the loss of a single example.
Here is my version of it. At first I was a bit confused about the usage of both i, y, and j, and thought maybe there was a typo here. But then I realized that they are indeed distinct things. What helped me link these 3 variables together was the phrase “loss with respect to the ith logit of a single example y”, where j is just local variable inside the summation expression.
Note that we only sorta applied the chain rule here. At this point, all we did was use chain rule to take care of the log part of the derivative. We don’t actually know what the derivative of the softmax function is yet, we need to further simplify.
This next part of the derivation took me quite a bit of time to grok, as “first or second year of bachelors math” was over 10 years ago for me 😅.
One of the main things that confused me a lot was the way Karpathy crossed out the negative sign, yielding this strange symbol that looks like a negative sign with an arrow pointing downwards. At first I thought maybe this was a branch of mathematics I have never seen before, but after reading through comments on the discord channel, I surmised that this is just how he crosses out the negative sign. In my work above, I prefer to just cross out the negative sign like I cross out any other term
Here is my version for the first branch where i ≠ y . The most important thing to note is that we are applying the quotient rule as opposed to the product rule + power rule as noted in his notes. Yes the quotient rule can be derived from the product rule but I am pretty sure the power rule here is a typo.
The core thing to understand here is what the following expression even means. The derivative of l_y with respect to l_i may seem a bit confusing, since l_y is not a function of l_i, they are referring to different things, right? Well not really. If y = i, then they are indeed the same thing, and so the derivative of something with respect to itself is just 1. But if i ≠ y, then they are indeed different things, so the derivative of something with respect to another unrelated thing is just 0!
In other words…
Now that we understand the above, we can use this to help us simplify this part of our quotient rule. Since we are in the i ≠ y branch, this entire expression simplifies to 0.
The second tricky thing to understand is how this line works.
In particular, what does this even mean? Again we have this strangeness where we are trying to get a derivative l_i but we don’t even see l_i in the expression! Well turns out l_i is actually in the expression if you unroll the summation into its individual terms. Since j spans all possible values of i, you can think of the sum to just be a single e to the l_i plus a bunch of other arbitrary constants. Since the derivative of a constant is 0 we can just ignore them. And the derivative of exponent is just exponent, so we are left with just the single e to the l_i.
Okay so I admit it took me a bit of time to actually understand these few lines (which Karpathy does not even write out 😅). I had to google around to find some hints and I found this following article very useful.
https://youtu.be/q8SA3rM6ckI?si=on2Bu9lDcWlOtIwQ&t=5453
Now once you understand the above, getting to the final answer for this branch is quite trivial. Well deserved smiley face indeed.
The other branch is actually pretty much the exact same thing except for one term.
Recall that now i = y
Which means that we can’t simplify this expression to 0. Instead we get a e to the l_i term.
Plugging this into our expression in the quotient rule gives us this beauty. Note that I changed the order of the terms to make it less ambiguous what is inside vs outside of the summation.
The rest of it is just basic math. But again, well deserved smiley face.
Reference: The Softmax function and its derivative — Eli Bendersky