Some Platform · 0:00 elapsed
0:04
0:52
2:31
5:08
Entropy measures the average surprise of a distribution.
Cross entropy is the cost of coding one distribution with another's code.
KL divergence is exactly that excess cost, and it is never negative.
Minimising cross entropy therefore minimises KL to the true distribution.