we have the following error function:

E$($w$1,$ w$2,$ v$|$X$)=$

X

t

r

t

log y

t $+(1$ r

t

$)$ log$(1$ y

t

$),$

where y

t $=$ sigmoid$($v$2$z$2+$v$1$z$1+$v$0),$ z

t

$1=$ sigmoid$($w$1,2$x

t

$2+$w$1,1$x

t

$1+$w$1,0)$ and

z

t

$2=$ LReLU$($w$2,2$x

t

$2+$ w$2,1$x

t

$1+$ w$2,0),$ the leaky rectified linear unit LReLU$($x$)$

defined as