The universal approximation theorem states that a feedforward neural network $($NN$)$ with a

single hidden layer can approximate any function over some compact set, provided that it has

enough neurons on that layer. This suggests that the number of neurons is more important than

the number of layers. But in practice deep learning is obviously very successful at various

prediction tasks. Why is that? Shouldn't all deep NNs be equivalent to single layered NNs with

enough neurons? Why do we need depth when we could theoretically rewrite that neural

network with a single layer?