- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a multimodal framework for valence-arousal (VA) regression using facial videos and blood-volume pulse (BVP) signals. The authors introduce a dataset of 23 participants with paired video-BVP recordings and participant-specific VA ratings. Their method combines visual and physiological encoders with a relation module that captures cross-modal interactions, feeding the resulting tokens into a QLoRA-adapted Qwen2.5-3B model for VA regression. Experiments against unimodal, multimodal and no-relation baselines show improvements across MAE, RMSE, PCC and CCC.
Critical strengths
-
The paper is easy to read and follow.
-
The combination of facial and physiological information is interesting.
-
The experiments provide some initial evidence for the benefit of multimodal fusion and the proposed relation branch.
Critical weaknesses
The framework mainly combines existing components (CNN & Transformer encoders; feature-level multimodal fusion; a pretrained LLM; a regression head). The relation token itself is constructed from standard operations (concatenation; absolute differences; element-wise multiplication; cosine similarity). The methodological contribution is low.
I also have a few concerns about the design and evaluation of the relation module:
-
The authors argue that the relation token captures interactions between modalities. However, the features used to construct it are obtained from two independently parameterized projection networks. It is not clear whether the resulting representations are sufficiently aligned for operations such as element-wise differences, products and cosine similarity to be meaningful. There is no explicit alignment objective and this aspect is not investigated experimentally.
-
The main ablation compares the full model with a version that completely removes the relation branch. However, removing this branch also removes additional trainable parameters and an extra input token containing information from both modalities. Therefore, the improvement does not necessarily demonstrate the benefit of explicitly modelling cross-modal relations. It could simply be due to the additional capacity or the extra modality summaries. I would expect a more detailed ablation, for example using only the pooled modality representations, removing the discrepancy features, or comparing against an additional fusion token with similar capacity.
-
More generally, I am not convinced that the use of an LLM is sufficiently justified for this task. The model performs direct numerical regression, without generating text or using language-based reasoning. The comparison with a non-LLM baseline is also quite limited, as that baseline simply pools the modality features and feeds them into an MLP. I think comparisons with stronger multimodal Transformer and/or cross-attention fusion methods are necessary. Otherwise, it is difficult to establish whether the improvements come from the pretrained LLM or simply from having a substantially more powerful fusion architecture.
The experimental evaluation is quite limited, particularly given the size of the dataset and the conclusions drawn. The dataset contains only 23 participants and the experiments use a subject-dependent split. While the authors correctly ensure that windows from the same source trial do not appear in both training and testing, the same participants can appear in both partitions. Moreover, the same stimulus videos can also be represented in both partitions through different trials. This is particularly important for physiological affect recognition, where individual differences in baseline responses can be substantial. The current setup does not establish whether the method generalises to unseen participants. A subject-independent evaluation is essential. There are also some additional issues with the experimental design:
-
Each stimulus lasts approximately 2mins and receives one VA rating, which is then assigned to every 5-second window extracted from that trial. Emotional responses can vary considerably throughout a stimulus. Therefore, it is unclear how representative the trial-level annotation is of each individual window. The paper should discuss this limitation more thoroughly and ideally provide trial-level results in addition to window-level evaluation. The overlapping windows also mean that the effective number of independent observations is much smaller than the number of windows.
-
Although the proposed method improves over the reported baselines, the final PCC values are only 0.308 for valence and 0.215 for arousal. These values suggest that the model still has rather limited ability to capture variations in the ground-truth ratings. A simple mean-prediction baseline would also help put the reported MAE improvements into perspective.
-
The authors do not use a separate validation set and select the model checkpoint based on the lowest training MAE. This does not seem right, particularly for a relatively small dataset and a large pretrained model. A proper validation partition would provide a more reliable basis for checkpoint selection and hyperparameter tuning.
Finally, I am not entirely convinced by the dataset contribution. The paper presents the combination of participant-facing video, BVP and VA annotations as an important motivation, but existing datasets such as DEAP already provide these modalities, at least for a subset of participants. The authors should better explain what makes the proposed dataset substantially different from existing resources. Details about the planned availability of the dataset should also be provided, as its practical contribution depends partly on whether other researchers can access it.
Questions for authors
Please have a look above on the detailed critical weaknesses section.