11.3 Continuity
If I asked you to conjure up a random function from your mind, I am almost certain that you would come up with one that is both continuous and differentiable. (unless you have weird tastes, as many mathematicians do).
However, the vast majority of functions are neither. In terms of cardinality, if you count all real functions f : ℝ→ℝ, it turns out that there are 2c of them in total, but the subset of continuous ones have cardinality c. It is hard to imagine such quantities: c and 2c are both infinite, but, well, 2c is more infinite. Yeah, I know. Set theory is weird. (Recall that c denotes the cardinality of the set of real numbers. If you would like a refresher on the topic, check out the set theory appendix Appendix C.)
Overall, as we shall see, continuity and differentiability allow us to do meaningful work with functions. For instance, the usual gradient descent-based optimization for neural networks doesn’t work if the loss function and...
