A Controlled Empirical Comparison of ResNet18, ViT-Small and CNN–Transformer Hybrid for RAF-DB Facial Expression Recognition

Abstract

Facial expression recognition (FER) models frequently struggle with class imbalance and generalization, particularly on real-world datasets. This study provides a controlled empirical comparison under a unified training pipeline on RAF-DB. Three lightweight architectures are compared: (i) a ResNet18 CNN classifier, (ii) a pretrained Vision Transformer (ViT-Small, patch16/224) and (iii) a CNN–Transformer hybrid, that converts CNN feature maps into token sequences for transformer encoding. The same preprocessing, online augmentation, imbalance-handling strategy and evaluation protocol are applied across all models. Class-aware augmentation, weighted random sampling and class-weighted cross-entropy are used, while accuracy, macro-F1, weighted-F1 and per-class metrics are reported. On the RAF-DB test set, the ViT-Small baseline achieves the best performance with accuracy of 0.8116 and macro-F1 of 0.7353. The ResNet18 CNN obtains accuracy of 0.7386 and macro-F1 of 0.6621, while the hybrid model obtains accuracy of 0.7370 and macro-F1 of 0.6552. The results show that ViT-Small benefits from pretrained global representation learning, whereas the current hybrid configuration does not improve minority-class recognition over the CNN baseline.