Each image is loaded into a computer as a matrix of different intensities. All the images were converted to gray scale so that we only need to operate on one layer of image (instead of three layers for a RBG image). A vector whose direction is unchanged when multiplied by the matrix is referred to as an eigenvector of that matrix. The eigenvectors of the covariance matrix associated with a large set of faces are called eigenfaces. The eigenfaces can be thought of as a basis for the set of faces. Just as any vector in a vector space is composed of a linear combination of the basis vectors, each face in the set can be expressed as a linear combination of the eigenfaces.

To compute the eigenfaces, a portion of a given dataset is first chosen randomly to be the training set. The images in the training set are used to construct the image matrix A (Note: All images in the dataset must have the same dimensions). The training set can be chosen by selecting a given percentage of the dataset or by selecting a given number of images per person from the database. Once the images are selected, each image Ii is vectorized into a column vector Pi such that its length equals to the total number of pixels in the image. This process brings the mathematics of all the computations down to a lower-dimensionality space.

Figure 1: Sample training images

The mean face of the training set is computed and subtracted from all the images within the training set (given W images in the training set).

Figure 2: Mean face of HFH dataset

Finally, the mean subtracted training images are put into a single matrix of dimension NM x W, forming the image matrix A.

Typical PCA calculation would first retrieve the covariance matrix C. Covariance measures the relation of how much two random variables vary together such that if the covariance is positive when both dimensions increase together and negative when they are inversely proportional. The eigenfaces were then obtained by computing the eigenvectors of the covariance matrix C. This computation will yield NM unique eigenvectors.

But in the case of this project, these resulting matrix of dimension NM x NM was way too large for MATLAB to process. Furthermore, even if MATLAB had the ability to process such a large matrix it would still later be too computationally intensive. Instead of computing the covariance matrix C directly, this project utilizes a smaller matrix S with dimensions W x W that can still be able to compute the eigenfaces efficiently. This simplification stems from the fact that the rank of the covariance matrix C is limited by the number of images in the training set. Since there are at most W-1 non-trivial eigenfaces for C, there is no need to compute all of the eigenfaces for the dataset. This simplification will prove to be useful as long as NM>>W.

The smaller matrix dimensions will be computationally effective later when large databases will be sorted through since only W eigenvalues and eigenfaces will be used.

Now, using some linear algebra tricks, we could show that the eigenvalues of C and S are the same and that the top W eigenvectors of C (ui) can be obtained from the eigenvectors of S (vi).

In this manner, we can see that the eigenvectors of C can be derived from

where the computation of vi_’s were much less computationally extensive than the direct computation of ui’s.

These ui vectors will constitute the columns of the eigenfaces.

Eigenface = [u1u2u3...uW][u1u2u3...uW] size 12{ matrix { \[ u rSub { size 8{1} } {} # u rSub { size 8{2} } {} # u rSub { size 8{3} } {} # "." "." "." {} # u rSub { size 8{W} } \] {} } } {}

Even with this complexity reduction, it is still redundant to use all W eigenfaces for the reconstruction process. We could reduce the number of eigenfaces used even more by indentifying the eigenfaces that contain more content than the others. To determine this property, we bring our attentions to the eigenvalues that correspond to the individual eigenfaces. We immediately see that there are some eigenfaces that have higher eigenvalues than the others.

Figure 3: Eigenvalues of the corresponding eigenfaces of HFH dataset

After arranging the eigenvalues in descending order, the result becomes clearer. We conclude that the eigenfaces corresponding to high eigenvalues contain more content. In other words, the higher the eigenvalue, the more characteristic features of the face the particular eigenvector describes. Therefore, we simplify the reconstruction process by only using the top K eigenfaces. This completes the training process of our implementation.

Figure 4: Top 5 eigenfaces of HFH dataset

Figure 5: Last 5 eigenfaces of HFH dataset

In terms of the eigenfaces themselves, we found that the more important eigenfaces (those with higher eigenvalues) had lower spatial frequency than the less important eigenfaces (those with lower eigenvalues). This is apparent in the figure above, where the first eigenfaces look blurry and indistinct, and the later eigenfaces have sharp edges and look more like individual people. This suggests that faces can be identified based on their low-frequency components alone.

For the first test of this project’s eigenface generation algorithm, the Japanese Female Facial Expression (JAFFE) Database was used. The JAFFE database fit our ideal conditions of similar lighting conditions, solid white backgrounds, and normalization of facial features such as the nose, eyes, and lips. The database is a set of 180 images of seven facial expressions (six basic facial expressions and one neutral).

For the Rice database, we chose students of both genders and diverse ethnicities. Each subject was told to express each of six emotions: neutral, happy, sad, surprised, angry, and disgusted. We had sixteen subjects, each with six emotions, and we took two pictures per emotion, resulting in a database of 192 images. We also created a special database of two emotions: closed eyes and expression of choice, to be used for demonstration at the poster session.