Facial expression Recognition using ensemble of CNNs

Table of Contents

• Benchmarking on FER2013
• Benchmarking on JAFFE
• Installation
• Download datasets
• Training on FER2013
• Evaluation results

Benchmarks

FER2013

Model	Accuracy
VGG19	70.77
EfficientNet_b2b	70.83
Googlenet	71.97
Resnet34	72.42
Inception_v3	72.72
Resnet50	72.86
Cbam_Resnet50	72.95
Bam_Resnet50	73.14
Densenet121	73.16
Resnet152	73.22
Resnet101	74.06
ResMaskingNet	74.14
ResMaskingNet + 6	76.82

JAFFE

Model	Accuracy
EfficientNet_b2b	90
Resnet18	91.42
Resnet101	92.86
Resnet50	94.28
Cbam_Resnet50	94.29
ResMaskingNet	97.1
Ensemble of above CNNs	98.2

CK+

Model	Accuracy
Resnet18	96.631
Densenet121	97.573
VGG19	98.058
Resnet101	98.544
ResNet50_pretrainedvgg	98.544
ResMaskingNet	98.87

Installation

• Install PyTorch by selecting your environment on the website and running the appropriate command.
• Clone this repository and install package prerequisites below.
• Then download the dataset by following the instructions below.

Prerequisites

• Python 3.6+
• PyTorch 1.3+
• Torchvision 0.4.0+
• requirements.txt

Datasets

• FER2013 Dataset (locate it in saved/data/fer2013 like saved/data/fer2013/train.csv)
• JAFFE Dataset (locate it in saved/data/jaffe like saved/data/jaffe/train.csv)

Training on FER2013

• To train network, you need to specify model name and other hyperparameters in config file (located at configs/*) then ensure it is loaded in main file, then run training procedure by simply running main file, for example:

python main_fer.py  # Example for fer2013_config.json file

• The best checkpoints will chosen at term of best validation accuracy, located at saved/checkpoints

• The TensorBoard training logs are located at saved/logs, to open it, use tensorboard --logdir saved/logs/

• By default, it will train alexnet model, you can switch to another model by editing configs/fer2013\_config.json file (to resnet18 or cbam\_resnet50 or my network resmasking\_dropout1.

  Follow similar process for JAFFE dataset

Evaluation

Below is an example for generating a striking confusion matrix writing things in latex.

(Read this article for more information, there will be some bugs if you blindly run the code without reading).

python ./Visualization/gen_confusion_matrix.py

Below is an example, to generate salient maps for jaffe images to find most import parts of the image used by the model for prediction.

python ./Visualization/gen_salientmap.py

Below is an example, to generate gradCAM visualization to find most import parts of the image used by the model for prediction.

python ./Visualization/gradCAM_resmasking.py

Ensemble method

I used no-weighted sum avarage ensemble method to fuse 7 different models together, to reproduce results, you need to do the following steps:

1. Download all needed trained weights and located on ./saved/checkpoints/ directory. Link to download can be found on Benchmarking section.
2. Edit file gen_results and run it to generate result offline for each model.
3. Run gen_ensemble.py file to generate accuracy for example methods.

Documentation

documentation

References