A Controlled Study of CLIP-Based Body-Scene Fusion for Emotion Recognition in Context

A controlled study finds that adding a CLIP-based scene encoder to a two-stream model does not improve emotion recognition in natural images, with the baseline model achieving 34.52% mean average precision on the EMOTIC test split [1]. The research, posted on the preprint repository arXiv, investigates context-aware emotion recognition where facial cues are often insufficient because the face may be small, hidden, or neutral [2]. The model architecture uses a ResNet-18 body stream to process a crop of the target person and a CLIP ViT-B/16 scene stream to encode the full image [2]. The fused features predict 26 categorical emotion labels along with continuous valence, arousal, and dominance values [2]. arXiv, an open-access repository for electronic preprints, hosts papers across computer science and other fields without peer review [7]. The study tested whether context-debiasing techniques or rare-class training adjustments could boost performance after integrating the CLIP scene encoder [2]. Simplified versions of CCIM-style intervention, CLEF-lite context-bias subtraction, asymmetric loss tuning, and class-balanced sampling were all evaluated under the same implementation pipeline [2]. None of these variants outperformed the clean two-stream model [2]. The authors note that CLIP provides broad scene semantics, but the simplified causal, counterfactual, and rare-class changes did not automatically translate into gains [2]. Most remaining errors occur in rare and subtle emotion categories, suggesting that future work should focus on label relationships and finer-grained subject-context interactions [2]. The EMOTIC dataset used in the study is part of a broader landscape of machine-learning datasets that underpin advances in computer vision, where high-quality labeled training data is often difficult and expensive to produce [6].