When implementing a neural network layer from scratch, we usually implement a $$forward$$ & a $$backward$$ function for each layer. Explain what these functions do$,$ potential variables, which arguments they take, and what they return.

$<>$ 'Forward' Path

$1\_01$ Define the purpose of the 'forward' function in implementing a neural network layer from scratch. Describe the sequence of operations typically performed within this function.

$1\_02$ Enumerate the essential variables that the 'forward' function needs to save during its execution. Justify the necessity of preserving these variables for the network's proper functioning.

$1\_03$ Elaborate on the arguments typically passed to the 'forward' function when implementing a neural network layer. Provide examples of these arguments and their respective roles in the layer's computation.

$<>$ 'Backward' Path

$1\_04$ Explain the significance of the 'backward' function in neural network layer implementation. Outline the key computations involved in this function and their relevance to the network's training process.

$1\_05$ Discuss the role of the 'backward' function in computing gradients during the backpropagation process. Identify the variables required as inputs to this function and their relevance in gradient computation.

$1\_06$ Analyze the outputs returned by the 'backward' function after its execution. Explain the significance of these outputs in updating the parameters of the neural network layer through gradient descent.

$<>$ 'Forward' & 'Backward' Path

$1\_07$ Compare and contrast the implementation of 'forward' and 'backward' functions across different types of neural network layers, such as fully connected layers, convolutional layers, and recurrent layers.

$1\_08$ Evaluate the computational complexity of the 'forward' and 'backward' functions in terms of time and space requirements. Discuss strategies for optimizing these functions to enhance the efficiency of neural network training.

$1\_09$ Investigate the impact of activation functions on the implementation of 'forward' and 'backward' functions. Assess how different activation functions influence the computational flow and gradient propagation within a neural network layer.

$1\_10$ Propose modifications or enhancements to the standard 'forward' and 'backward' functions to accommodate specialized network architectures or training objectives. Justify the necessity and potential benefits of these modifications in the context of deep learning applications.

Question $1$ options: