Problem $3\mathrm{.}8$ Consider replacing the ReLU activation function with $($i$)$ the Heaviside step function heaviside $z,($ii$)$ the hyperbolic tangent function $tanh[z],$ and $($iii$)$ the rectangular function rect $z,$ where:

Redraw a version of figure $3\mathrm{.}3$ for each of these functions. The original parameters were: $=\{{}_0,{}_1,{}_2,{}_3,{}_{10},{}_{11},{}_{20},{}_{21},{}_{30},{}_{31}\}=\{-0\mathrm{.}23,-1\mathrm{.}3,1\mathrm{.}3,0\mathrm{.}66,-0\mathrm{.}2,0\mathrm{.}4,-0\mathrm{.}9,0\mathrm{.}9,1\mathrm{.}1,-0\mathrm{.}7\}.$ Provide an informal description of the family of functions that can be created by neural networks with one input, three hidden units, and one output for each activation function.

d$)$

g$)$

b$)$

e$)$

h$)$

i$)$

i$)$

c$)$

f$)$

Figure $3\mathrm{.}3$ Computation for function in figure $3\mathrm{.}2$a$.$ a$-$c$)$ The input $x$ is passed through three linear functions, each with a different y $-$intercept ${}_{*0}$ and slope ${}_{*1}.$ d$-$f$)$ Each line is passed through the ReLU activation function, which clips negative values to zero. g$-$i$)$ The three clipped lines are then weighted $($scaled$)$ by ${}_1,{}_2,$ and ${}_3,$ respectively. j$)$ Finally, the clipped and weighted functions are