Predicting race from face

We present a framework for estimating race from faces in movie data. Please refer to a detailed survey “Learning race from face: A survey”, Fu S. et. al., 2014 for a discussion on studies on recognizing race from faces. The definitions and taxonomy of race from a computer vision perspective are borrowed from this survey. The list of sources we used can be found here.

Table of contents

1. Data preparation
2. Sample size
3. Model architecture
4. Performance evaluation
   i. Confusion matrix and accuracy
   ii. ROC curves
5. What is the network learning?

Data and model architecture

We compiled face data across opensource databases which have clear definitions of labeling races and are consistent with our nomenclature. Additionally, we annotated race for IMDb identities as described in Ramakrishna et. al., 2017, ACL. Sources which required institutional EULA agreements were excluded due to the processing delays involved.

Data preparation

• All face images were resized to 100x100 and were converted to grayscale. Since a good number of images in our database were grayscale images, we converted all images to grayscale images
• Skin color has been shown to be one of the least important factors in race prediction (See survey)
• All images were aligned to achieve in-plane rotation to align landmarks of eyes and nose
• Left-right flipping of the images was performed for data augmentation
• Test images for evaluation were chosen to have no overlap of the person’s identity with the training data as well as to be variable with respect to pose, illumination, background and occlusions
• Due to lack of data from the nativeamerica/pacificislander class, we only considered the other five classes in our prediction model

• Summary of the face database

race	num_faces	num_images
eastasian	14472	28133
caucasian	267478	529741
latino/hispanic	10912	21374
asian-indian	28132	55473
african/african-american	22646	44627
nativeamerican/pacificislander	580	1069
TOTALs	344220	680417

Model architecture

A simple modified VGG-like architecture was adopted for a 5-class race classification model. The architecture is as shown below.

Performance Evaluation

• Class-wise system performance is tabulated below. Overall system performance accuracy was 82.6% for a training setup of mini-batch size 32, trained for 50 epochs.

Confusion matrix

• Note that the model is most confused between latino and caucasian faces which is consistent with previous work as well the structural similarity of the faces between these two races.
  

• Receiver operating characteristics (ROC) curves for race prediction
  

What is the network learning

We were inspired by visualizing filter activations in a CNN as described here and extended the concept to identify the input (or distorted input) that minimizes the output loss for each class. In this, we initialize the input with an average face and minimize the categorical cross entropy loss for each class and visualize the changes to the input that achieves this minimization. The resulting modified inputs are shown below.
Not surprisingly, the modified inputs resemble the per-class average faces and are quite distinct. The results were similar when the input was initialized by average face from a specific class, rather than across all images.