acceptodds
Withdrawn from ICLR 2027

SAGE: Subject-Anchored AU-Guided Expression for Pain Estimation from Video

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Pain expression is anatomically localized to a few action units (AUs) and varies strongly across individuals. We propose SAGE (Subject-Anchored AU-Guided Expression), which makes both properties explicit on top of a frozen self-supervised video transformer (MARLIN) and trains only a  2M-parameter ordinal head. AU-guided token selection routes patch tokens through the six PSPI AU regions located by facial landmarks, keeping  12% of the tokens. Subject calibration (SubjMean) centers each query on the mean of a few unlabeled clips of the same subject. On BioVid leave-one-subject-out, SAGE reaches 5-class QWK 0.399 [0.35, 0.45] (0.342 without calibration) versus 0.199 for a matched frozen baseline (PainFormer) under the same protocol. Calibration accounts for most of the gain and improves all six frozen encoders we test. It is non-parametric, and its gain rises steeply up to about five unlabeled clips per subject and only slowly beyond. Token selection improves over holistic pooling on two face backbones and outperforms both an arbitrary partition and direct AU intensities. Region attention shows that the calibrated model draws on all six AU regions, whereas the uncalibrated one collapses onto a single region. The frozen encoder transfers by linear probe to AI4Pain and zero-shot to a clinical QST pilot, exceeding PainFormer on both.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
pain estimation, facial action units, subject-specific calibration, test-time adaptation, frozen foundation models, video representation learning, ordinal regression, affective computing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_usF9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SAGE, a facial video-based pain intensity estimation framework combining anatomical action unit (AU) guidance with subject-specific calibration. The method uses a frozen MARLIN video encoder, selects spatial tokens corresponding to six pain-related AUs, and subtracts a subject-specific reference representation computed from unlabeled clips. A lightweight ordinal regression head predicts pain intensity. Experiments on BioVid using leave-one-subject-out evaluation show improvements over a matched PainFormer baseline, achieving a 5-class QWK of 0.399 compared with 0.199. Additional experiments on AI4Pain and a small QST pilot investigate cross-dataset transfer. Ablation studies indicate that subject calibration contributes most of the improvement, while AU-guided token selection provides additional benefits.

Critical strengths
  1. The paper demonstrates that simple mean-based calibration using unlabeled same-subject clips substantially improves ordinal pain estimation. The improvement is consistent across six frozen encoders, suggesting that the approach is not specific to MARLIN.
  2. The experiments investigate AU-guided selection, calibration strategies, reference-set size and composition, and backbone sensitivity. The results help identify which components contribute to performance and provide useful insights into subject variability in pain estimation.
Critical weaknesses
  1. The main components, including anatomical AU-based region selection, subject-level feature normalization, and ordinal regression, are established techniques. The contribution primarily lies in combining these components rather than introducing a fundamentally new learning method.
  2. The main results rely on BioVid, while the zero-shot evaluation uses a small, unreleased dataset of 12 subjects. Moreover, the strongest BioVid results use nearly all remaining clips from each test subject for calibration. Although smaller reference sets are evaluated, their performance is lower, raising questions about practical deployment and generalization to unseen clinical conditions.
Questions for authors
  1. The main BioVid results use approximately 100 unlabeled clips per test subject. Could the authors report the complete results using only five reference clips, including confidence intervals, to better demonstrate the practical effectiveness of the proposed calibration?
  2. How does SAGE compare with standard subject-wise feature centering applied to holistic MARLIN representations under exactly matched training and inference conditions? Can the authors further clarify the contribution of AU-guided selection beyond subject normalization?
  3. Given that QST-Pilot contains only 12 evaluated subjects and follows a pain stimulation protocol similar to BioVid, how robust are the reported transfer improvements? Can the authors provide uncertainty estimates and subject-level results?
Reviewer_gCutOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes SAGE for video-based pain estimation using a frozen face-video encoder. It uses facial landmarks to select and pool tokens from six pain-related AU regions, then calibrates each query by subtracting the mean representation of unlabeled clips from the same subject. A small ordinal prediction head is trained on the resulting features. Experiments on BioVid suggest that subject calibration contributes more than AU selection and improves several frozen encoders.

Critical strengths
  1. The calibration strategy is simple and can be applied to different frozen encoders without introducing additional trainable parameters beyond the ordinal head.
Critical weaknesses
  1. The novelty of the AU-guided component is limited and the writing needs improvement. The motivation and comparison with previous methods are not clearly presented in the abstract and introduction. This makes it difficult to assess the paper's contribution beyond existing methods.
  2. The paper contains some errors and awkward visual elements. Figures 1 and 5 contain some unusual lines. In addition, Figure 5 places FAN after MARLIN, unlike Figure 1, while Figure 1 does not explicitly show individual reference features entering fϕf_\phi. Appendix B.5 ("Optimization") is empty. These issues make it difficult to verify how the method was implemented and trained, which reduces confidence in the reliability and reproducibility of the reported experiments.
Questions for authors
  1. The inconsistent connections and unusual visual elements in Figures 1 and 5 make the intended data flow unclear. Could the authors explain how these issues arose and clarify the correct data flow?
  2. Performance still improves beyond five reference clips, and neutral-only results are reported only for the parameterized variant. Could the authors clarify why the gains are described as leveling off around five clips and whether the default SubjMean has been tested with only neutral references?
  3. Table 9 compares calibrated SAGE with uncalibrated AU/PSPI baselines, making it difficult to attribute the reported gap to the features alone. Could the authors clarify how the benefit of the features can be separated from the benefit of calibration in this comparison?
  4. The full reference set in Table 13 already has the same number of clips at each pain level, so its difference from the label-balanced set is unclear. Could the authors explain how these two sets are constructed and how their comparison supports the conclusion about label leakage?
Reviewer_VBs6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SAGE, a method for estimating pain intensity from facial videos using a frozen pretrained video encoder (MARLIN). The method combines two main components: selecting facial regions associated with pain-related action units (AUs), and subject-specific calibration using unlabeled videos of the same person. A lightweight ordinal classifier then predicts pain intensity from these features.

The authors evaluate SAGE on BioVid using leave-one-subject-out validation and report improvements over PainFormer and the uncalibrated version of SAGE. Ablation studies show that subject-specific calibration provides most of the improvement. The authors also evaluate cross-dataset transfer on AI4Pain and a small QST pilot dataset, reporting better results than PainFormer in these settings.

Critical strengths

Simple and effective subject calibration: The subject-specific calibration method is simple and does not require additional training. It improves pain estimation performance across all six pretrained encoders tested. The analysis of different reference-set sizes also provides useful insights into when calibration helps.

Good experimental analysis: The paper includes several ablation studies that help explain the contribution of each component. The authors also evaluate the method on two additional datasets, providing some evidence of its ability to generalize beyond BioVid.

Critical weaknesses

Limited novelty: The two main components, AU-based feature selection and subject-specific normalization, are already established techniques. The main contribution is combining these existing methods rather than introducing a new approach.

Limitations in experimental evaluation: The main results use nearly 100 reference clips per subject, while the improvement with fewer clips is smaller. Also, SAGE and PainFormer use different pretrained backbones, making it difficult to isolate the source of the performance improvement. The cross-dataset evaluation is also limited, particularly the small QST-Pilot dataset with only 12 subjects.

Questions for authors

Can the authors provide a comparison using the same MARLIN backbone on the QST-Pilot dataset, with and without AU-guided selection? This would help clarify how much of the improvement comes from the proposed method rather than the pretrained backbone.

In the experiments using only five reference clips per subject, was the classifier retrained with five clips, or was only the reference set reduced during inference? How sensitive are the results to the selection of reference clips?

Can the authors provide confidence intervals or statistical significance tests for the cross-dataset results, particularly on QST-Pilot? Given the small number of subjects, this would help assess the reliability of the reported improvements.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.