%% Please just do the extra credit part in matlab

%% Eigenfaces for Facial Recognition Introduction and Resources

%% (1) Load training set images from a .mat file (2 points)

clear; close all; clc; load training_set_faces %% (2) Plot one training set image (5 points)

figure(1) test_image = 200; imshow(yalefaces(:,:,test_image), 'DisplayRange', [], 'InitialMagnification', 'fit'); title("Testing loading of image from data set: Result of image " + test_image)

%% (3) Vectorize the images and combine into 2D array (8 points)

training_im_size = size(yalefaces); T = []; for i = 1:training_im_size(3) current_image = yalefaces(:,:,i); new_image = reshape(current_image, (training_im_size(1)*training_im_size(2)), 1); T = [T, new_image]; end T = double(T); % do not modify the line

%% (4) Find the mean face (5 points)

average_face_vector = mean(T,2);

%% (5) Plot the mean face (10 points) figure(2) average_face_matrix = reshape(average_face_vector, training_im_size(1), training_im_size(2)); imshow(average_face_matrix, 'DisplayRange', [], 'InitialMagnification', 'fit'); title("Mean face of initial data set");

%% (6) Subtract off mean face from training set images (5 points)

shifted_T = T - average_face_vector;

%% (7) Calculate covariance of shifted_T and obtain eigenfaces (0 points)

S = cov(shifted_T'); % do not modify this line

%% (8) Calculate the eigenfaces and associated eigevalues (5 points) [eigenvectors,eigenvalues] = eig(S);

%% (9) Sort eigenvalues and their vectors in descending order (10 points) % Detailed instructions: % !! Your code here !! Sort eigenvalues and eigenfaces from largest to smallest % (descending order) eigenvalues= diag(eigenvalues); [eigenvalue_sort,ind]=sort(eigenvalues,'descend'); eigenvector_sort= eigenvectors(:,ind);

%% (10) Plot the first 9 Eigenfaces (15 points). figure(3) for i= 1:9 eig_face = reshape(eigenvector_sort(:,i),training_im_size(1), training_im_size(2)); subplot(3,3,i); imshow(eig_face,'DisplayRange',[],'InitialMagnification','fit'); title('Eigenfaces associated by largest 9-Eigenvalues') end

%% EXTRA CREDIT %%

%% EXTRA CREDIT (up to 10 points) Find the number of eigenfaces needed to represent 95% of the total variance % Detailed instructions: % How many eigenfaces do we need to represent 95% percentage of the data?

% The number of principal components k is determined arbitrarily by setting % a threshold epsilon = 0.95 on the total variance. % % The total variance is v = lambda_1 +lambda_2 + ... + lambda_n where % n = number of eigenvalues (equal to the number pixels in each image). % Your task is to find the smallest k that satisfies: % (lambda_1 + lambda_2 + ... + lambda_k)/v > 0.95

% !! Your code here !!

%% EXTRA CREDIT (up to 5 points) Only keep eigenfaces that contain 95% of the info % Detailed instructions: % Only consider the eigenfaces that represent 95% of the data by reducing the % size of the eigenvectors/eigenfaces matrix. Only inlude the eigenvectors that % you need according to this threshold. The result will be a 2D array of size % 2016 x k where k is found in the previous section.

% !! Your code here !! Only use eigenfaces that represent 95% of the data by % reducing the size of the eigenfaces matrix.

%% EXTRA CREDIT (up to 10 points) Find weights of training images in the eigenfaces subspace % Detailed instructions: % Project the training images onto the eigenfaces subspace by taking the % dot product of each eigenface (eigenvector) with each image with the mean face % subtracted. The eigenfaces and mean-subtracted training images are vectors % with 48 x 42 = 2016 elements. The result of this dot product a scalar % (1 x 1 array) representing how much of this input image is "in the direction" % of an eigenface. Or in other words, the dot product is a measure of how similar % the input face is to the particular eigenface. % % The more similar the input image is to a particular eigenface, the larger % the contribution from this eigenface to the reconstructed image when using the % eigenfaces as a basis. This is similar to a Fourier coefficient and reconstructing % signals from Fourier coefficients and sines and cosines (the basis functions % for Fourier series). % % Store all these projections for all the training images into a 2D array. % The number of elements in each column will equal the number of eigenfaces you % are using (k). There will be 2414 columns for the 2414 training set images. % % Suggested MATLAB functions: for loop or matrix multiplication

% !! Your code here !! Project the training images onto the eigenfaces subspace.

%% EXTRA CREDIT (up to 10 points) Reconstruct one training set image from eigenface weights % Detailed instructions: % So now we have the coefficients (weights) to reconstruct all of the training % images from a linear combination of the eigenfaces with the appropriate weights, % just like a Fourier series. These coefficient vectors can be thought of as a % type of compressed representation of the input image. Instead of 2016 numbers % to describe an image (grayscale values at all the pixels), we only need % k < 2016 numbers and the eigenfaces to describe any image. % % Compare one original image from the yalefaces 3D array to it's reconstructed % image using the weights you just found and the eigenfaces. A reconstructed image % is the sum of the mean face (2016 x 1 column vector) and the k weights % (or coefficients) multiplied by their corresponding eigenface. Just like a Fourier % Series! % % Plot both the original image from the yalefaces 3D array and its reconstruction % from the weights and the eigenfaces. You will need to reshape the vectorized % reconstructed image to a 2D array before plotting. % % Suggested MATLAB functions: matrix multiplcation or for loop, reshape, % subplot, imshow % !! Your code here !! Reconstruct one of the training set images from its % k weights. Visually compare the original and reconstructed % images by plotting them side by side.

%% EXTRA CREDIT (up to 7 points) Test facial recognition using a face from the training set % Should be a perfect match! % Test the facial recognition ability of your code using one of the images % from the training set. This image will have a similarity score of 1 and % should be a perfect match to one of the images in the training % set (itself).

% !! Your code here !! Set the input image variable (your choice as to what % variable to use) to one of the training images.

% Calculate similarity score of this test input image from training set images % Detailed instructions: % Calculate the similarity score of a 'test input image' to each training set % image by taking the dot product of the mean-subtracted input image with % each eigenface/eigenvector. % % In general: First vectorize the input image. Then subtract off the mean % image vector from the input image. Then take the dot product of this mean- % subtracted test image (resized to a vector) with each eigenface. The % result will be a vector of a length equal to the number of eigenfaces % that you are using (k).

% !! Your code here !! Vectorize the test image if not already done so, % subtract off mean from test image if not already done so. Find the % mean-subtracted test image's eigenfaces coefficients.

% Plot the weights of the test input image on the eigenfaces using a stem plot. % In other words, plot the entries of the vector you just found above. % Suggested MATLAB function: stem

% !! Your code here !! Plot of eigenfaces coefficients for the test input image

% Assign a similarity score to the input image by comparing the coefficients/weights % associated with each eigenface found for the input image and the training set % images. To perform facial recognition, the similarity score is calculated between % an input face image and each of the training images. The matched face is the % one with the highest similarity, and the magnitude of the similarity score indicates % the confidence of the match (with a unit value indicating an exact match). % % Use a similarity score based in the inverse Euclidean distance defined % as % % similarity score with respect to eigenface n = ... % 1/(1 + ||w_eigenface_n - w_input_img||_2) % % w_eigenface_n is teh weight/coefficient vector of eigenface n and % w_input_img is the weight/coefficient vector of the test input image and % || A ||_2 is the L2-norm of A (also called Euclidean distance).

% Suggested MATLAB function: norm, for loop

% !! Your code here !!. Calculate the similarity score of the input images % with respect to each eigenface and store in a vector of length equal to % the number of training images (2414).

% Find the image in the training set with the highest similarity score % Detailed instructions: % The training set image that is the closest match to the input image will % will have the largest similarity score. % For an input image that is one of the images in the training set, the max % similarity score should be 1, indicating a perfect match. % % Suggested MATLAB function: max

% !! Your code here !! Find closest training set image to the input image % according to the similarity score

% Visual check of test input image facial recognition % Detailed instructions: % Display the input image and the closest training set image to visually % check the accuracy of your facial recognition code. For this test input % image that was an image from the test set, you should see the same faces. % % Suggested MATLAB functions: imshow, subplot

% !! Your code here !! Visually check the code's facial recognition ability % for your test input image that was from the training set

%% EXTRA CREDIT (up to 8 points) Test facial recognition using a non-human face % Next, test the image of a mandrill baboon (mandrill.bmp). This is a face % but not a human face. So the facial recognition should give a 'poor' % result. You will copy and paste a lot of code from the previous % section because you will be completing many of the same tasks, just on a % different image. % % First import the mandrill.bmp file into MATLAB. Convert the image to the % double data type. Resize the image to be the same size as the images in the % training set 48 x 42 px. % % Suggested MATLAB functions: imread, im2double, imresize

% !! Your code here !! Load the mandrill.bmp image. Convert to double. % Resize to training set image size.

% Calculate similarity of non-human input image to training set images % Detailed instructions: % Calculate the similarity of the non-human face input image to each training set % image by taking the dot product of the mean-subtracted input image with % each eigenface/eigenvector. % % In general: First vectorize the input image. Then subtract off the mean % image vector from the input image. Then take the dot product of this mean- % subtracted test image (resized to a vector) with each eigenface. The % result will be a vector of a length equal to the number of eigenfaces % that you are using (k).

% !! Your code here !! Subtract off mean face from input image. Find the % mean-subtracted input image's eigenfaces coefficients.

% Plot the weights of the input image on the eigenfaces using a stem plot. % In other words, plot the entries of the vector you just found above. % % Suggtested MATLAB function: stem

% !! Your code here !! Plot of eigenfaces coefficients for the test input image

% Assign a similarity score to the input image by comparing the coefficients/weights % associated with each eigenface found for the input image and the training set % images. Use a similarity score based in the inverse Euclidean distance. % % Suggested MATLAB function: norm, for loop

% !! Your code here !!. Calculate the similarity score of the input images % with respect to each eigenface and store in a vector of length equal to % the number of training images (2414).

% Find the image in the training set with the highest similarity score % Detailed instructions: % The training set image that is the closest match to the input image will % will have the largest similarity score. This is the training set image % that is the closest match to the input image. % % Suggested MATLAB function: max

% !! Your code here !! Find closest training set image to the input image % according to the similarity score

% Visual check of non-human face input image facial recognition % Detailed instructions: % Display the input image and the closest training set image to visually % check the accuracy of your facial recognition code. % % Suggested MATLAB functions: imshow or imagesc with colormap set to gray

% !! Your code here !! Visually check the code's facial recognition ability % on the non-human face.