Asynchronous Parallel SGD v$/$s Sequential SGD

Assume that calculating the gradient on each core and updating the parameter vector in the shared memory takes $0\mathrm{.}2$ seconds for all cores, and the cores have the following delays: $1=0\mathrm{.}15$ s$,$ $2=0\mathrm{.}2$ s$,$ $3=0\mathrm{.}25$ s$,$ and $4=0\mathrm{.}5$ s$.$ Assume for the gradient of every data point i we have $0<$fi$,$ and all cores start the parameter update simultaneously.

Consider the parameters after running asynchronous parallel SGD for $0\mathrm{.}6$ seconds. What is the tightest upper bound on the noise in the parameter vector, relative to what would have occurred if the same updates had been done sequentially as in sequential SGD$?$ Assume that in sequential SGD$,$ the order of execution is core $1,$ core $2,$ core $3,$ core $4.$

Hint: For example, if in the asynchronous parallel SGD $($HOGWILD$)$ setting, only core $1$ and core $2$ can execute, then compare this to the parameter obtained using sequential SGD where only core $1$ and core $2$ execute, sequentially.

A$.2$$,$ where is the learning rate

B$.4$$,$ where is the learning rate

C$.6$$,$ where is the learning rate

D$.8$$,$ where is the learning rate

E$.0\mathrm{.}9$s

F$.1\mathrm{.}5$s