Exercise $3$

Instructions: Implement the NER model, with the architecture discussed in the lectures. All the necessary layers are objects from the tensorflow.keras.layers library, but they are already loaded in memory, so you do not have to worry about function calls.

Please utilize help function e$.$g$.$ help$($tf$.$keras.layers.Dense$)$ for more information on a layer

tf$.$keras.Sequential: Combinator that applies layers serially $($by function composition$)-$ this is not properly a layer $($it is under tensorflow.keras only and not under tensorflow.keras.layers$).$ It is in fact a Tensorflow model object.

You can add the layers to a Sequential layer by calling the method $.$add$($layer$).$

You may skip the input shape and pass it in the first layer you instantiate, if necessary $($RNNs usually don't need to fix an input length$).$

tf$.$keras.layers.Embedding: Initializes the embedding layer. An embedding layer in tensorflow will input only positive integers.

Embedding$($input$\_$dim, output$\_$dim, mask$\_$zero $=$ False$).$

input$\_$dim is the expected range of integers for each tensor in the batch. Note that the input$\_$dim is not related to array size, but to the possible range of integers expected in the input. Usually this is the vocabulary size, but it may differ by $1,$ depending on further parameters. See below.

output$\_$dim is the number of elements in the word embedding $($some choices for a word embedding size range from $150$ to $300,$ for example$).$ Each word processed will be assigned an array of size output$\_$dim. So if one array of shape $(3,)$ is passed $($example of such an array $[100,203,204]),$ then the Embedding layer should have output shape $(3,$output$\_$dim$).$

mask$\_$zero is a boolean telling whether $0$ is a mask value or not. If mask$\_$zero $=$ True, then some considerations must be done: $1.$ The value $0$ should be reserved as the mask value, as it will be ignored in training. $2.$ You need to add $1$ in input$\_$dim, since now Tensorflow will consider that one extra $0$ value may show up in each sentence.

tf$.$keras.layers.LSTM: An LSTM layer.

LSTM$($units$,$ return$\_$sequences$)$ Builds an LSTM layer with hidden state and cell sizes equal to units. The arguments you will need: $1.$ units: It is the number of LSTM cells you will create to pass every input to$.$ In this case, set the units as the Embedding output$\_$dim. This is just a choice, in fact there is no static rule preventing one from choosing any amount of LSTM units. $2.$ return$\_$sequences: A boolean, telling whether you want to return every output value from the LSTM cells. If return$\_$sequences $=$ False, then the LSTM output shape will be $($batch$\_$size, units$).$ Otherwise, it is $($batch$\_$size, sentence$\_$length, units$),$ since there will be an output for each word in the sentence.

tf$.$keras.layers.Dense: A dense layer.

Dense$($units$,$ activation$)$: The parameters for this layer are: $1.$ units: It is the number of units chosen for this dense layer, i$.$e$.,$ it is the dimensionality of the output space. In this case, each value passed through the Dense layer must be mapped into a vector with length num$\_$of$\_$classes $($in this case, len$($tags$)).2.$ activation: This is the activation that will be performed after computing the values in the Dense layer. Since the Dense layer comes before the LogSoftmax step, you can pass the LogSoftmax function as activation function here. $**$You can find the implementation for LogSoftmax under tf$.$nn$.$ So you may call it as tf$.$nn$.$log$\_$softmax. See its documentation here.

Instructions: You will build a function that inputs the number of tags, the vocabulary size and an optional parameter to control the embedding dimension and outputs a tensorflow model as discussed in the lectures.