Exercise $4(8$ points$)$

Training a large language model depends on various architectural choices. However, recent papers such as Kaplan et al$.(2020)$ and the "Chinchilla" paper $($Hoffman et al$,2022),$ people noticed that the performance of an LLM can be predicted quite accurately by just two quantities, i$.$e$.,(1)$ the number $N$ of model parameters, and $(2)$ the total number $D$ of tokens the model is trained on$.$

The table below contains data from the training of various LLM systems.

$\backslash$table$[[$LLM$,N-$ Parameters $($billions$),D-$ Tokens $($billions$),$Loss$],[$GPT$-2,1,21,2\mathrm{.}527663],[$GPT$-3,175,300,2\mathrm{.}001097],[$Gopher$,280,300,1\mathrm{.}994691],[$Chinchilla$,70,1400,1\mathrm{.}936333],[$PaLM$,540,780,1\mathrm{.}923154]]$

$($a$)$ Determine a power law expressing the relation between the loss, the number of parameters N and the number of tokens D used during training $($Hint: use least squares and a model of the form $($:$a{N}^{-0\mathrm{.}34}+b{D}^{-0\mathrm{.}28}+c\}.$

$($b$)$ Based on this power law, determine possible reductions in the loss for the five LLMs reported in the table under the following scenarios: $(1)$ an infinite number of parameters, $(2)$ an infinite amount of tokens.