$1$ Problem $1$: Generative adversarial networks

$(50$ points$)$

In this problem, suppose that we will implement a generative adversarial net$-$

work $($GAN$)$ that models a high$-$dimensional data distribution $p_{\text{d}\text{a}\text{t}\text{a}}(x),$ where

$xin{R}^n.$ To do so$,$ we will define a generator $G_{}$:$R^k{R}^n$; we obtain samples

from our model by first sampling a k$-$dimensional random vector $zN(0,I)$

and then returning $G_{}(z).$

We will also define a discriminator $D_{}$:$R^n(0,1)$ that judges how realistic

the generated images $G_{}(z)$ are, compared to samples from the data distribution

$x{p}_{\text{d}\text{a}\text{t}\text{a}}(x).$ Because its output is intended to be interpreted as a probability,

the last layer of the discriminattor is frequently the sigmoid function

$(x)=\frac{1}{1+e^{-x}}$

There are several common variants of the loss functions used to train a

generative adversarial network $($GAN$).$ They can all be described as a procedure

where we alternately perform a gradient descent step on $L_D($;$)$ with respect

to $$ to train the discriminator $D_{},$ and a gradient descent step on $L_G($;$)$ with

respect to $$ to train the generator $G_{}$ :

$mi{n}_{}{L}_D($;$),mi{n}_{}{L}_G($;$)$

In our lecture, we talked about the following losses, where the discriminator's

loss is given by:

$L_D($;$)=-E_{x{p}_{\text{d}\text{a}\text{t}\text{a}}(x)}[log{D}_{}(x)]-E_{zN(0,I)}[log(1-D_{}(G_{}(z)))]$

and the generator's loss is given by the minimax loss:

$L_G^{\text{m}\text{i}\text{n}\text{i}\text{m}\text{a}\text{x}}($;$)=E_{zN(0,I)}[log(1-D_{}(G_{}(z)))]$

$(25$ points$)$ the minimax loss for $L_G$ suffers from vanishing gradient

problem. In terms of the discriminator's logits$,$ the minimax loss is

$L_G^{\text{m}\text{i}\text{n}\text{i}\text{m}\text{a}\text{x}}($;$)=E_{zN(0,I)}[log(1-(h_{}(G_{}(z))))]$

Show that the derivative of $L_G^{\text{m}\text{i}\text{n}\text{i}\text{m}\text{a}\text{x}}$ with respect to $$ is approximately $0$ if

$D(G_{}(z))$~~$0,$ or equivalently, if $h_{}(G_{}(z))0.$ You may use the fact that

${}^{\text{'}}(x)=(x)(1-(x)).$ Why is this problematic for the training of the generator

when the discriminator successfully identifies a fake sample $G_{}(z)?$

$(25$ points$)$ To solve this vanishing gradient problem, we usually replace

$L_G^{\text{m}\text{i}\text{n}\text{i}\text{m}\text{a}\text{x}}$ with other loss functions such as non$-$saturating loss $L_G^{\text{n}\text{s}\text{g}\text{a}\text{n}}[1]$ and

more other forms of loss functions can be found in $[2].$ You may plot differ$-$

ent loss functions including minimax loss and non$-$saturating loss to show the

contrast. You also need to explain why non$-$saturating loss can avoid vanishing

gradient problem.

$L_G^{\text{n}\text{s}\text{g}\text{a}\text{n}}($;$)=-E_{zN(0,I)}[log{D}_{}(G_{}(z))]$