Asynchronous Parallel SGD v$/$s Sequential SGD

We are comparing the training of a model with $1$ parameter using asynchronous

vs synchronous parallel Stochastic Gradient Descent $($SGD$)$ on a machine with $4$

cores.

Assume that calculating the gradient on each core and updating the parameter vector in the shared memory takes $0\mathrm{.}2$ seconds for all cores, and the cores have the following delays: $1=0\mathrm{.}15$ s$,$ $2=0\mathrm{.}2$ s$,$ $3=0\mathrm{.}25$ s$,$ and $4=0\mathrm{.}5$ s$.$ Assume for the gradient of every data point i we have $0<$fi$,$ and all cores start the parameter update simultaneously.

Consider the parameters after running asynchronous parallel SGD for $0\mathrm{.}6$ seconds. Assume that in sequential SGD$,$ the order of execution is core $1,$ core $2,$ core $3,$ core $4.$

What is the tightest upper bound on the noise in the parameter vector, relative to what would have occurred if the same updates had been done sequentially $($as in sequential SGD$)?$

Hint: For example, if in the asynchronous parallel SGD $($HOGWILD$)$ setting, only core

$1$ and core $2$ can execute, then compare this to the parameter obtained using sequential SGD where only core $1$ and core $2$ execute, sequentially.

Your Answer:

A$.2$$,$ where is the learning rate

B$.4$$,$ where is the learning rate

C$.6$$,$ where is the learning rate

D$.8$$,$ where is the learning rate

E$.0\mathrm{.}9$s

F$.1\mathrm{.}5$s