thesis · 2023 – 2024 · Hangzhou Dianzi University
Temporal Contrastive Learning for Interpretable Facial Emotion Recognition
Pipeline
01
Problem
Affective-computing models for facial video analysis often produce useful predictions but remain difficult to interpret, especially in sensitive health-related contexts where clinician trust requires transparent reasoning.
02
Method
01NT-Xent temporal contrastive pretraining over short-range and long-range views of 38-dimensional OpenFace features.
02Four-layer Transformer encoder with learnable positional embeddings and attention-based temporal aggregation.
03Focal loss with class weighting to handle label imbalance.
04Subject-level two-fold stratified cross-validation on 95 DAIC-WOZ sessions.
03
Results
0.82
AUROC
0.62
F1
0.90
Recall
| Model | AUROC | F1 | Recall |
|---|---|---|---|
| FFN | baseline | — | — |
| BiLSTM | baseline | — | — |
| GRU | baseline | — | — |
| TCN | baseline | — | — |
| Transformer (no pretraining) | baseline | — | — |
| Ours (contrastive + Transformer) | 0.82 | 0.62 | 0.90 |
Subject-level two-fold stratified cross-validation, 95 DAIC-WOZ sessions.
Stack
PyTorchOpenFaceTransformersContrastive learningTemporal modeling