acceptodds
Withdrawn from ICLR 2027

Relation-Aware Visual–Physiological Affect Regression with an Evidence-Prefixed Large Language Model

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal affect regression aims to predict continuous emotional states by combining complementary yet imperfect evidence from observable behavior and physiological bodily responses. Facial video captures visible expressions which can be subtle or intentionally regulated, while blood‑volume pulse (BVP) measurements are susceptible to motion artifacts, sensor contact noise, and varying individual physiological baselines. We introduce a visual–physiological affect dataset of concurrently‑recorded, temporally paired facial‑video and BVP streams, with participant‑specific, continuous‑valued valence–arousal ratings collected for each stimulus presentation. We further propose an evidence‑prefixed large‑language‑model architecture for direct numerical affect regression. Visual and BVP encoders produce modality‑specific token sequences, and a continuous learned relation token summarizes their learned‑space interactions through modality summaries, absolute difference, element‑wise product, and cosine discrepancy. These evidence tokens precede a learnable valence‑arousal query in a QLoRA‑adapted Qwen model, whose final query hidden state directly predicts valence and arousal. Optimized solely via affect supervision, the relation token requires no manually annotated modality‑relation or signal‑quality labels. We perform controlled comparisons under a subject-dependent, source-trial-disjoint split and report window-level results for unimodal, non-LLM visual–BVP fusion, and no-relation variants. The relation token represents geometric discrepancy between learned representations rather than verified psychological conflict.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal affect regression, valence-arousal estimation, facial video, blood volume pulse, visual-physiological fusion, large language models, cross-modal relation modeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vryyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a multimodal framework for valence-arousal (VA) regression using facial videos and blood-volume pulse (BVP) signals. The authors introduce a dataset of 23 participants with paired video-BVP recordings and participant-specific VA ratings. Their method combines visual and physiological encoders with a relation module that captures cross-modal interactions, feeding the resulting tokens into a QLoRA-adapted Qwen2.5-3B model for VA regression. Experiments against unimodal, multimodal and no-relation baselines show improvements across MAE, RMSE, PCC and CCC.

Critical strengths
  • The paper is easy to read and follow.

  • The combination of facial and physiological information is interesting.

  • The experiments provide some initial evidence for the benefit of multimodal fusion and the proposed relation branch.

Critical weaknesses

The framework mainly combines existing components (CNN & Transformer encoders; feature-level multimodal fusion; a pretrained LLM; a regression head). The relation token itself is constructed from standard operations (concatenation; absolute differences; element-wise multiplication; cosine similarity). The methodological contribution is low.

I also have a few concerns about the design and evaluation of the relation module:

  • The authors argue that the relation token captures interactions between modalities. However, the features used to construct it are obtained from two independently parameterized projection networks. It is not clear whether the resulting representations are sufficiently aligned for operations such as element-wise differences, products and cosine similarity to be meaningful. There is no explicit alignment objective and this aspect is not investigated experimentally.

  • The main ablation compares the full model with a version that completely removes the relation branch. However, removing this branch also removes additional trainable parameters and an extra input token containing information from both modalities. Therefore, the improvement does not necessarily demonstrate the benefit of explicitly modelling cross-modal relations. It could simply be due to the additional capacity or the extra modality summaries. I would expect a more detailed ablation, for example using only the pooled modality representations, removing the discrepancy features, or comparing against an additional fusion token with similar capacity.

  • More generally, I am not convinced that the use of an LLM is sufficiently justified for this task. The model performs direct numerical regression, without generating text or using language-based reasoning. The comparison with a non-LLM baseline is also quite limited, as that baseline simply pools the modality features and feeds them into an MLP. I think comparisons with stronger multimodal Transformer and/or cross-attention fusion methods are necessary. Otherwise, it is difficult to establish whether the improvements come from the pretrained LLM or simply from having a substantially more powerful fusion architecture.

The experimental evaluation is quite limited, particularly given the size of the dataset and the conclusions drawn. The dataset contains only 23 participants and the experiments use a subject-dependent split. While the authors correctly ensure that windows from the same source trial do not appear in both training and testing, the same participants can appear in both partitions. Moreover, the same stimulus videos can also be represented in both partitions through different trials. This is particularly important for physiological affect recognition, where individual differences in baseline responses can be substantial. The current setup does not establish whether the method generalises to unseen participants. A subject-independent evaluation is essential. There are also some additional issues with the experimental design:

  • Each stimulus lasts approximately 2mins and receives one VA rating, which is then assigned to every 5-second window extracted from that trial. Emotional responses can vary considerably throughout a stimulus. Therefore, it is unclear how representative the trial-level annotation is of each individual window. The paper should discuss this limitation more thoroughly and ideally provide trial-level results in addition to window-level evaluation. The overlapping windows also mean that the effective number of independent observations is much smaller than the number of windows.

  • Although the proposed method improves over the reported baselines, the final PCC values are only 0.308 for valence and 0.215 for arousal. These values suggest that the model still has rather limited ability to capture variations in the ground-truth ratings. A simple mean-prediction baseline would also help put the reported MAE improvements into perspective.

  • The authors do not use a separate validation set and select the model checkpoint based on the lowest training MAE. This does not seem right, particularly for a relatively small dataset and a large pretrained model. A proper validation partition would provide a more reliable basis for checkpoint selection and hyperparameter tuning.

Finally, I am not entirely convinced by the dataset contribution. The paper presents the combination of participant-facing video, BVP and VA annotations as an important motivation, but existing datasets such as DEAP already provide these modalities, at least for a subset of participants. The authors should better explain what makes the proposed dataset substantially different from existing resources. Details about the planned availability of the dataset should also be provided, as its practical contribution depends partly on whether other researchers can access it.

Questions for authors

Please have a look above on the detailed critical weaknesses section.

Reviewer_85jkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an evidence-prefixed Qwen2.5-3B model for valence–arousal regression from facial video and blood-volume pulse signals, together with VP-Affect, a dataset containing 815 curated paired segments from 23 participants with participant-specific trial-level ratings. The model preserves temporal tokens for both modalities and adds a learned relation token constructed from pooled modality summaries, absolute differences, element-wise products and a final learned query provides the representation for numerical regression. Evaluation uses a subject-dependent, source-trial-disjoint split and compares unimodal models, non-LLM fusion, and a Qwen variant without the relation branch. Results show improvements across all reported metrics, with the relation branch reducing valence/arousal MAE and increasing CCC.

Critical strengths
  1. The relation branch produces a substantial improvement over a controlled multimodal comparator. Both Qwen variants retain the full visual and BVP token sequences and share initialization and training settings
  2. The experiments support the complementary predictive value of facial video and BVP. Without the relation branch, multimodal Qwen outperforms both unimodal variants across all reported metrics
Critical weaknesses
  1. The baselines leave the source of the architectural gains unresolved. The non-LLM comparator mean-pools the sensor sequences before a small MLP, whereas Qwen receives all temporal token. This comparison therefore confounds the fusion backbone with sequence preservation and model capacity.
  2. The reliability and practical significance of the gains remain uncertain. Evaluation uses a single split and seed without uncertainty estimates, and absolute concordance remains modest.
Questions for authors
  1. Did you manual segment boundaries selected without access to valence–arousal ratings, and what criteria governed segment inclusion or exclusion?
  2. Was timing error or drift between the video and BVP devices measured, and how was temporal correspondence checked?
Reviewer_vk2yOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces VP-Affect, reporting 23 participants, 22 stimuli, and 815 curated facial-video/BVP input pairs. Five-second inputs produce ten visual and ten physiological tokens. A relation token is constructed from pooled modality summaries, absolute differences, elementwise products, and cosine discrepancy. These tokens and a task instruction condition QLoRA-adapted Qwen2.5-3B-Instruct; a terminal learned query directly predicts valence and arousal. Windows inherit source-trial VA ratings. On one subject-dependent, source-trial-disjoint split, the full model improves over the evaluated unimodal, pooled-fusion, and no-relation configurations in Table 2.

Critical strengths
  1. The supervision and partition units are clearly specified. Sections 3.2 and 5.2 explain that five-second windows inherit complete source-trial ratings rather than independent time-continuous labels. Source-trial grouping precedes 50%-overlapping window generation. Under the described procedure, all derivatives of one trial remain in the same partition, reducing the risk of near-duplicate same-trial samples crossing training and testing. This makes the target evaluated in Table 2 reasonably clear. (p3 L142–153; pp6–7 L315–332)

  2. The complete relation branch shows quantified preliminary gains. Both modality sequences are retained, and the terminal query can access the complete prefix. In Table 2, adding the branch reduces valence/arousal MAE from 1.92/1.48 to 1.61/1.08 and increases CCC from 0.243/0.112 to 0.276/0.157. This supports the predictive value of the complete implementation and provides a concrete starting point for testing the relation construction. (§4.3; Table 2; p8 L378–385)

Critical weaknesses
  1. Table 2 does not establish competitiveness against representative prior fusion methods or isolate the source of the Qwen fusion gain. Section 2.1 L085–087 identifies MVP as a close visual–physiological reference, but the five rows are visual Qwen, BVP Qwen, an author-built non-LLM fusion model, no-relation Qwen, and full Qwen. No adapted published method is evaluated. Appendix B.3 mean-pools the temporally encoded sequences before MLP regression, whereas Qwen directly accesses all 10+10 tokens. Thus, the MAE changes of 2.02→1.92 and 1.75→1.48 vary both the backbone and sequence access.

    At least one justified sequence-fusion comparator is needed. For example, MVP-style cross-attention could be adapted to facial video+BVP and a two-output regression head under the same source-trial split and window metrics. If the paper’s encoders are retained to control fusion specifically, label this as a controlled adaptation rather than a full reproduction of MVP. Report the adaptations, trainable parameters, and selection rules. Published binary-classification F1 cannot be compared directly with these regression metrics.

    Furthermore, all results use seed 42 and one split (p8 L384–385). Overlapping windows share trial labels, yet between-run variation and clustered intervals are absent. Final CCC of 0.276/0.157 also needs context from simple training-only priors and target distributions. The table supports point differences among implementations, but not their stability, value beyond simple predictors, or competitiveness against representative prior methods.

  2. The affect-related meaning of cosine discrepancy is not established, and interpretive validity and predictive utility are not separately validated. Equations 2–4 independently project and pool the modalities into two 128-dimensional vectors and compute c=1−cos⁡(sv,sp)c = 1 - \cos(s_v,s_p). Section 4.2 L204–205 defines the common space only by dimensional compatibility; §4.4 L258–261 trains the relation branch solely through the final VA objective. What smaller or larger c represents in the affect task therefore remains unspecified beyond angular discrepancy.

    CLAP’s paired contrastive task grounds cosine in cross-modal semantic matching; a comparable semantic connection is not established here. Section 4.5 L273–277 appropriately excludes psychological-conflict and reliability labels, but does not positively specify the property measured or the observations that would validate it. If this is only a geometric predictive heuristic, delimit that interpretation and justify its suitability. If an affect-related relationship is intended, validate that relationship rather than substituting aggregate error reduction for interpretive evidence.

    Predictive contribution is a separate gap. Equation 5’s 513-dimensional input contains both summaries, differences, products, and c, while the ablation removes the complete projection–MLP–extra-token branch (L228–230, L258–263). Retain the complete branch and input width, fix only the cosine slot during training/testing, and retrain; also compare a similar-capacity summary-only token. The first tests explicitly supplied cosine; the second tests whether the structured interactions improve over an added summary token. Even positive predictive results would not by themselves establish affective semantics.

Questions for authors
  1. Prior methods and fair fusion: Can a sequence-fusion comparator be added on VP-Affect with the same facial-video+BVP inputs, VA regression targets, source-trial split, windowing, and metrics? Identify the modules changed from MVP or another published method. If the same sensor encoders and all 20 tokens are retained, does Qwen still improve? Scores from incompatible original tasks are unnecessary.

  2. Meaning of cosine: Specify what c≈0c \approx 0 and larger c should mean in this task and provide analysis on held-out source trials supporting that interpretation. For example, only if affective proximity is intended, examine whether within-participant cross-trial pairings with nearby versus distant source-trial VA ratings exhibit the expected c differences. This is a rating-proxy/constructed-pair diagnostic, not ground truth for psychological consistency. An analysis better suited to the actual claim is welcome; explicitly state if only a geometric predictor is intended.

  3. Predictive role of cosine and the branch: Compare the full model, a retrained model retaining the 513-dimensional input but fixing the cosine slot to a constant, and a similar-capacity token generated only from [sv;sp][s_v; s_p]. Keep other settings unchanged and report MAE/RMSE/CCC, parameters, and seed variation. Test-time masking alone does not replace retraining.

  4. Simple references and stability: Estimate global mean/median and participant-mean predictors from training source-trial labels and report MAE/RMSE; PCC for a global constant is undefined and unnecessary. Repeat full/no-relation/strongest-comparator models over several optimization seeds and give paired clustered intervals for differences, for example by resampling participants and their trials while retaining all windows within a trial. Also report the equally weighted mean of per-trial window MAEs. If predictions are averaged before scoring, label that as a different trial-level evaluation.

  5. Reproducibility: Supply realized participant/trial/curated-pair/window counts and anonymous IDs by partition, crop/exclusion rules, the exact instruction, face crop/resize dimensions, and both CNNs’ channels, kernels, and strides. Heads and widths are reported, but these missing settings still prevent reconstruction of the inputs and encoders. Specify open or controlled data/code access arrangements.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.