Facial Expression Recognition Using Deep Learning

University of Science and Technology of Hanoi (USTH)

December 2024

Research Overview

01

Introduction

Problem statement and objectives for emotion classification

02

Methodology

Data processing pipeline and model architectures

03

Results

Performance metrics and comparative analysis

04

Conclusion

Key findings and future directions

Problem Statement

Objective: Classify facial expressions into 7 emotion categories using Convolutional Neural Networks (CNNs) on the Cohn-Kanade (CK+) dataset.

The Seven Emotion Classes:

  • Neutral
  • Anger
  • Disgust
  • Fear
  • Happiness
  • Sadness
  • Surprise

Data Processing Pipeline

Load CK+

Import dataset images

Crop Faces

Using landmarks detection

Resize

64×64 pixels

Normalize

Scale pixel values

Split Data

70% train / 30% test

Handling Class Imbalance

The Problem

Imbalanced dataset causes model to predict majority class. Neutral expressions dominate at 64.2% of the dataset, creating significant bias.

The Solution

Compute class weights inversely proportional to class frequency using the formula:

Where wi = weight for class i, N = total samples, C = number of classes, ni = samples in class i

Computed Weights (CK+ Dataset)

Baseline Model: AAM + SVM

Active Appearance Model

Statistical model combining shape and texture analysis with PCA-based dimensionality reduction

SVM Classifier

Support Vector Machine for final classification decision

0.990

AUC Score

93.8%

Accuracy

93.4%

F1 Score

93.7%

Precision

CNN Architecture: Modified LeNet-5

Layer Structure

Total Parameters: 6,429,577

Optimizer: Adam (learning rate = 0.0005)

LeNet-5 Training Performance

Training Accuracy

97.94%

Excellent learning on training data

Test Accuracy

92.99%

Strong generalization to unseen data

Validation Trend

Follows training closely, indicating proper learning without severe overfitting

Model Performance Comparison

Results Summary

Why CNN V3 Failed

  • Too many parameters (∼1.5M)
  • CK+ dataset too small (631 training samples)
  • Severe overfitting

Why LeNet-5 Succeeded

  • Modernized architecture with ReLU activation
  • Dropout regularization (0.25, 0.5)
  • Good train/validation balance

Key Findings from Emotion Recognition

Architecture Design Matters

For small datasets like CK+ (631 training samples), proper architecture design prevents overfitting and ensures generalization.

LeNet-5 Performance

Achieved 92.99% test accuracy, comparable to AAM+SVM baseline (93.8%), demonstrating CNN effectiveness.

Overfitting Prevention

Key factors include appropriate dropout rates (0.25, 0.5), proper learning rate selection, and regularization techniques.

Exercise 5: Identity Recognition Challenge

New Objective

Classify faces by subject identity (not emotion) - a fundamentally different challenge with unique complexities.

Key Differences from Emotion Recognition

  • More classes: 123 subjects vs 7 emotions
  • Variation: Same person with different expressions
  • Similarity: Different people may look alike
  • Label extraction: From filename pattern S(+.)(+.)(+.)

Dataset Statistics

920

Total Samples

644

Training

276

Testing

123

Subjects

Average samples per subject: 5.2 - significantly limited data per class

Architecture Evolution: Emotion vs Identity

Exercise 4: Emotion Recognition

Architecture (LeNet-based):

  • Conv2D(20, 5×5) + ReLU + MaxPool + Dropout(0.25)
  • Conv2D(50, 5×5) + ReLU + MaxPool + Dropout(0.25)
  • Flatten → Dense(500) + ReLU + Dropout(0.5)
  • Dense(7) + Softmax

Task Characteristics:

  • 7 classes (emotions)
  • ∼100+ samples per class
  • Lower class imbalance
  • Same person, different expressions

Exercise 5: Identity Recognition

Architecture (Enhanced LeNet):

  • Conv2D(20, 5×5) + ReLU + MaxPool + Dropout(0.25)
  • Conv2D(50, 5×5) + ReLU + MaxPool + Dropout(0.25)
  • Conv2D(100, 3×3) + ReLU + MaxPool + Dropout(0.3) ← Extra layer!
  • Flatten → Dense(500) + ReLU + Dropout(0.5)
  • Dense(123) + Softmax

Task Characteristics:

  • 123 classes (subjects)
  • ∼5.2 samples per class
  • Higher complexity

Identity Recognition: Outstanding Results

95%

Test Accuracy

Top-1 classification performance

92%

Precision

Macro average across all classes

93%

Recall

Macro average detection rate

94%

F1-Score

Weighted average performance

Key Achievement: The model achieved 95% accuracy on identity recognition and generalizes remarkably well despite limited samples per subject (5.2 average). This demonstrates the effectiveness of the enhanced architecture with three convolutional blocks.

Conclusions & Future Directions

Emotion Recognition Success

Modified LeNet-5 achieved 92.99% test accuracy on 7-class emotion classification, matching traditional AAM+SVM baseline performance.

Identity Recognition Excellence

Successfully adapted emotion recognition CNN for identity recognition, achieving 95% test accuracy on challenging 123-class problem with limited data.

Architecture Insights

LeNet-inspired architecture with 3 convolutional blocks proved highly effective. Class weighting and regularization crucial for handling imbalanced, limited data.

Key Takeaway

Proper architecture design, regularization techniques, and class balancing strategies enable CNNs to excel even with small datasets.

Thank You!

Questions?