I am facing an issue because I am trying to concatenate a one hot encoding vector of size $5$ with the patches of the image $($I know I am using a different approach in the code$),$ I am using a pretrained vision transformer model $-$ vit$\_$base$\_$patch$16\_224.$ It is a multilabel classification problem.

Ideally, I would feed the model the image and the resolution and I would like the model to predict the most suitable labels. An image of the dataset is attached below.

And this is the model I am using; I have done all the necessary steps :

class ResizingModel$($MultilabelImageClassificationBase$)$:

def $\_$init$\_($self$,$ num$\_$classes$)$:

super$($ResizingModel$,$ self$).\_$init$\_()$

self.num$\_$classes $=$ num$\_$classes

self.vit $=$ ViTModel.from$\_$pretrained$(\text{'}$google$/$vit$-$base$-$patch$16-224-$in$21$k$\text{'})$

self.fc $=$ nn$.$Linear$($self$.$vit.config.hidden$\_$size $+5,$ num$\_$classes$)$

# print$($f$"$Input size of self.cls$\_$embedding$\_$size: $\{$self$.$vit.config.hidden$\_$size $+5\}")$

# print$($f$"$Output size of self.fc: $\{$num$\_$classes$\}")$

def forward$($self$,$ x$,$ target$\_$resolution$\_$one$\_$hot$)$:

# Process input image through Vision Transformer

vit$\_$output $=$ self.vit$($x$)$

# Extract the CLS token embedding from the Vision Transformer output from the last hidden state

#the hidden state refers to the information

cls$\_$token$\_$embedding $=$ vit$\_$output.last$\_$hidden$\_$state$[$:$,0,$ :$]$ # Use only the CLS token

# print$("$Shape of cls$\_$token$\_$embedding:", cls$\_$token$\_$embedding.shape$)$ # Shape

# print$("$Shape of target res:", target$\_$resolution$\_$one$\_$hot.shape$)$ # Shape

concatenated$\_$input $=$ torch.cat$([$cls$\_$token$\_$embedding, target$\_$resolution$\_$one$\_$hot$],$ dim$=1)$

# Apply linear layer

logits $=$ self.fc$($concatenated$\_$input$)$

return logits

I have also attached an image of the sample output, my main concern is that the accuracy is extremely low $($~$0\mathrm{.}29).$

I would appreciate it if you could help me in solve this problem.

$\backslash$table$[[$image$,$resolution,CR$,$SC$,$SNS$,$SCL$],[$images$\backslash 1\_$city.jpg$,0,1,0,0,0,0,0,1,0],[$images$\backslash 1\_$city.jpg$,0,0,0,1,0,1,1,1,1],[$images$\backslash 1\_$city.jpg$,1,0,0,0,0,0,0,1,0],[$images$\backslash 1\_$city.jpg$,0,0,1,0,0,0,0,1,0],[$images$\backslash 1\_$city.jpg$,0,0,0,0,1,1,1,1,1],[$images $\backslash \backslash 1\_$crowd.jpg$,0,0,0,1,0,1,0,1,1],[$images $\backslash \backslash 1\_$crowd.jpg$,1,0,0,0,0,1,0,0,0],[$images $\backslash \backslash 1\_$crowd.jpg$,0,0,0,0,1,1,1,1,1],[$images $\backslash \backslash 1\_$crowd.jpg$,0,1,0,0,0,1,0,1,0],[,0,0,1,0,0,1,0,1,0]]$