acceptodds
Under review as a conference paper at ICLR 2027

End-to-end Full Video-level Supervision for Facial Depression Assessment Model Training

Abstract

Depressive states are often reflected by long-term, variable-duration facial behaviours, yet existing video-based automatic depression assessment (ADA) approaches frequently model depression from short fixed-length segments (including frames) due to memory constraints. Segment-level supervision not only limits their ability to learn cross-segment varying-length facial behavioural patterns but also fails to capture depressive facial patterns at the full-video level. In this paper, we propose Dual-phase Memory-efficient Video-level Supervision (DMVS), a novel training strategy that equips existing segment-level face-video ADA models with end-to-end full video-level supervision through depression relevance-weighted gradient modulation. Specifically, the first phase constructs variable-length facial behaviour representations by combining features extracted from short, fixed-length segments from the entire face video, followed by estimating the depression relevance of each behaviour representation and segment-feature, based on which a video-level loss is computed from relevance-weighted behaviour representations. Then, the second phase uses each estimated depression relevance to weight the corresponding segment's contribution to the backbone gradients, guiding feature learning towards depression-informative behaviours by accumulating relevance-weighted gradients from all face video segments. This way, the target ADA model is updated based on the weighted video-level gradients. Experiments on AVEC 2013 and 2014 demonstrate that our DMVS strategy consistently improves ADA performance across ADA-specific and general-purpose spatio-temporal models, supporting our DMVS as a model-agnostic and plug-and-play training strategy. Our code is provided in Supplementary Material.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Automatic depression assessment, Video-level model training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mwAiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Dual-phase Memory-efficient Video-level Supervision (DMVS) for facial depression assessment, aiming to replace segment-level supervision and aggressive temporal subsampling with end-to-end video-level training. DMVS caches features from fixed-length video units, constructs adaptive-duration segment representations, and uses category-guided relevance scores to aggregate selected segments for video-level prediction. It then maps segment relevance to individual units, modulates their feature gradients, and updates the backbone through sequential activation recomputation. The approach combines video-level supervision with relevance-weighted feature learning while reducing backbone activation storage. Experiments on AVEC 2013 and AVEC 2014 report lower RMSE than in-house reproduced baselines across five backbones. Additional evaluations include component ablations, controls for gradient modulation, an ablation of adaptive segment duration, memory profiling, and an extension to apparent personality prediction on ChaLearn 2016.

Critical strengths
  1. Well-motivated and broadly applicable training strategy: DMVS addresses the mismatch between video-level depression labels and segment-level training, enabling joint optimization of segment features under full-video supervision while bounding backbone activation storage to a single FBU. Reported improvements over reproduced baselines across five ADA-specific and general-purpose backbones, using a shared hyperparameter configuration, suggest that its usefulness extends beyond a single model design. This is the main reason I view the work as potentially valuable to the ADA community.

  2. Informative component controls: With VAT on AVEC 2014 Northwind, comparisons against unit, mean, and shuffled gradient weights, together with fixed and shared learned segment durations, support the benefits of relevance-aligned gradient modulation and adaptive segment construction.

Critical weaknesses

The contribution of end-to-end encoder training is not fully isolated: Table 2 shows no consistent gain from video-level supervision alone, but it lacks a matched baseline using the same FVC head and losses on frozen segment-trained features. Adding DSC also introduces an auxiliary classification loss. Although matched controls support GMVB’s incremental benefit, the overall gains cannot be clearly attributed to end-to-end encoder adaptation versus the aggregation head and additional supervision.

Questions for authors

Equation (7) sums relevance scores over selected segments covering each FBU, so greater segment overlap can increase an FBU’s gradient weight even without higher individual segment scores. What motivates this accumulation rule, and how do the existing analyses distinguish the benefit of depression-relevance weighting from the effect of overlap-based weighting?

2 comments
AuthorsOpenReview ↗

Response to the critical weakness

The contribution of end-to-end encoder training is not fully isolated: Table 2 shows no consistent gain from video-level supervision alone, but it lacks a matched baseline using the same FVC head and losses on frozen segment-trained features. Adding DSC also introduces an auxiliary classification loss. Although matched controls support GMVB’s incremental benefit, the overall gains cannot be clearly attributed to end-to-end encoder adaptation versus the aggregation head and additional supervision.

We thank the reviewer for the comment. Based on your valuable suggestions, we have added additional results in table below. All five settings use VAT encoder with the same FBSP, DSC and VLC modules and are evaluated on AVEC 2014 Northwind. Settings A and B use "frozen segment-trained features" for the training of the aggregation head, whereas C, D and E also update the encoder (encoder adaptation). Settings A and C remove the "auxiliary classification loss" Lcls\mathcal{L}_{\mathrm{cls}}, so DSC receives gradients only from Lreg\mathcal{L}_{\mathrm{reg}} and Lord\mathcal{L}_{\mathrm{ord}}. We ran each setting five times and report the minimum RMSE and the mean RMSE with sample standard deviation (SD) over the five runs.

SettingencoderLcls\mathcal{L}_{\mathrm{cls}}MultiplierMinimum RMSEMean RMSE ±\pm SD
AFrozen–7.58587.7189±0.10397.7189\pm0.1039
BFrozen✓–7.39297.5665±0.12757.5665\pm0.1275
CAdaptative117.54087.6636±0.10157.6636\pm0.1015
DAdaptative✓117.30697.5063±0.19977.5063\pm0.1997
EAdaptative✓[u]t[\mathbf{u}]_t7.09407.1856±0.10347.1856\pm0.1034

(1) Frozen segment-trained features VS. encoder adaptation using the same aggregation head and losses (B VS. D): Encoder adaptation lowers the mean RMSE only from 7.5665 (B) to 7.5063 (D), a reduction of 0.0602, which is smaller than the SD of either setting. Encoder adaptation alone therefore brings no clear gain, in agreement with Table 2.

(2) Contribution of GMVB to the gain of encoder adaptation (B, D and E): D and E share the same aggregation head, losses and encoder adaptation, and differ only in the gradient multiplier. Replacing the unit multipliers with [u]t[\mathbf{u}]_t lowers the mean RMSE from 7.5063 (D) to 7.1856 (E), a reduction of 0.3207, which exceeds the SD of either setting. From B to E, the mean RMSE drops by 0.3809, of which encoder adaptation with unit multipliers contributes 0.0602 (B to D) and GMVB contributes 0.3207 (D to E). The gain of encoder adaptation is therefore mainly attributable to GMVB's relevance-based gradient modulation.

(3) Auxiliary classification loss in encoder adaptation (C VS. D): Adding Lcls\mathcal{L}_{\mathrm{cls}} lowers the mean RMSE from 7.6636 (C) to 7.5063 (D), a reduction of 0.1573, close to the reduction of 0.1524 from A to B with a frozen encoder. Lcls\mathcal{L}_{\mathrm{cls}} therefore brings a consistent gain regardless of encoder adaptation.

We have added these results and analyses to Appendix C of the resubmitted manuscript.

AuthorsOpenReview ↗

Response to questions for authors

Equation (7) sums relevance scores over selected segments covering each FBU, so greater segment overlap can increase an FBU’s gradient weight even without higher individual segment scores. What motivates this accumulation rule, and how do the existing analyses distinguish the benefit of depression-relevance weighting from the effect of overlap-based weighting?

We thank the reviewer for the comment. The accumulation rule in Eq. (7) is inspired by the observation that an individual facial behaviour unit (FBU) can contribute to multiple depression-informative facial behaviours occurring at different temporal scales. While each selected facial behaviour segment's relevance score reflects its estimated contribution to the video-level depression assessment, Eq. (7) aggregates the depression relevance evidence associated with each target FBU across different and multi-scale facial behavioural contexts, i.e., repeated inclusion in relevant facial behavioural contexts provides additional evidence of the FBU's importance to video-level depression assessment. This design also allows the gradient modulation to prioritise FBUs supported by multiple depression-informative behavioural contexts, thereby encouraging the backbone to learn facial behavioural features that are informative across different temporal scales.

To clarify the benefit of depression-relevance weighting from the effect of overlap, we compare GMVB with three control weighting rules, i.e., we ran each rule five times to report the minimum RMSE and the mean RMSE with SD over the five runs in the table below.

Let ctc_t count the selected segments covering FBU utu_t, and let rˉt\bar{r}_t be their mean relevance score. For ct>0c_t>0, Eq. (7) can be written as [u]t=ctrˉt[\mathbf{u}]_t=c_t\bar{r}_t. For a fair comparison, each setting uses the weight wt=min⁡(1,aqt)w_t=\min(1,a q_t), where the per-video factor aa (aova_{\mathrm{ov}}, ameana_{\mathrm{mean}} or amaxa_{\mathrm{max}}) makes ∑twt\sum_t w_t equal to the total GMVB weight ∑t[u]t\sum_t[\mathbf{u}]_t of the same video. Appendix C of the resubmitted manuscript details the computation of aa and this comparison.

(1) The overlap-only control uses the base weight qt=ctq_t=c_t, which uses the number of the selected depression-related facial behaviour segments containing the target FBU to weight the FBU without considering their depression relevance scores;

(2) The mean-relevance control uses qt=rˉtq_t=\bar{r}_t, which uses the mean depression relevance score of the selected depression-related facial behaviour segments containing the target FBU to weight the FBU without considering the number of segments;

(3) The max-relevance control uses qt=rtmax⁡q_t=r_t^{\max}, which uses the highest depression relevance score of the selected depression-related facial behaviour segments containing the target FBU to weight the FBU without considering the number of segments;

(4) GMVB accumulates the weighted sum by jointly considering the number of the FBU-related segments and their depression relevance scores.

Weighting ruleGradient weight wtw_tMinimum RMSEMean RMSE ±\pm SD
Max-relevancemin⁡(1,amaxrtmax⁡)\min(1,a_{\mathrm{max}}r_t^{\max})7.38717.4585±0.15287.4585\pm0.1528
Overlap-onlymin⁡(1,aovct)\min(1,a_{\mathrm{ov}}c_t)7.28847.3729±0.08237.3729\pm0.0823
Mean-relevancemin⁡(1,ameanrˉt)\min(1,a_{\mathrm{mean}}\bar{r}_t)7.17977.3185±0.09937.3185\pm0.0993
GMVB (sum)[u]t=ctrˉt[\mathbf{u}]_t=c_t\bar{r}_t7.09407.1856±0.10347.1856\pm0.1034

As demostrated by the table, GMVB that considers FBU counts in the selected facial behaviour segments and accumulates their relevance scores obtains the lowest minimum and mean RMSE among the four rules. In contrast, only considering counts (Overlap-only), only considering the average relevance score of of all segments containing the target FBU, or only considering the highest relevance score among all segments containing the target FBU, achieved degraded performance. The conclusions are: (i) jointly using depression-relevance and overlap to weight the FBU yielded the best results; and (ii) mean depression-relevance weighting outperformed the overlap-based weighting, while overlap-based weighting outperformed the max depression-relevance weighting.

Reviewer_75gWOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DMVS (Dual-phase Memory-efficient Video-level Supervision), a memory-efficient end-to-end training framework for facial depression assessment from long videos, which combines fixed-length Facial Behaviour Units, adaptive variable-length behaviour segmentation, depression-relevance estimation, video-level regression/ordinal/category supervision, and relevance-weighted gradient modulation to enable full-video supervision under limited GPU memory; experiments on AVEC 2013 and AVEC 2014 with five different backbone architectures demonstrate consistent performance improvements over segment-level training and existing memory-efficient training strategies. The problem is relevant, and the implementation and ablations are described in considerable detail. However, test-based checkpoint selection undermines the evidence for generalization. The incremental novelty and claims about long-range temporal modeling also require stronger validation.

Critical strengths
  1. The proposed two-stage design effectively addresses the memory bottleneck of full-video training while retaining video-level supervision and is shown to generalize across five different backbone architectures.
  2. The paper provides comprehensive ablation and sensitivity studies that separately evaluate GradCache, FBSP, DSC, GMVB, segment duration, and Top-K selection, providing reasonable evidence for the contribution of individual components.
Critical weaknesses
  1. Although the paper rigorously shows that sequential recomputation can recover the ordinary full-video gradient, the relevance-modulated gradient is generally not the gradient of the original video-level objective, and the paper does not provide a clear optimization objective or theoretical justification for this modified update.
  2. The depression relevance estimated by DSC is learned only from video-level labels without segment-level annotations, raising concerns about whether the selected segments genuinely correspond to depression-related behaviours or merely capture features correlated with the video-level label.
  3. The experimental comparison lacks sufficiently strong baselines based on standard temporal attention or multiple-instance learning with video-level GradCache, making it difficult to determine whether the gains arise from the proposed FBSP/DSC design rather than from generic temporal weighting and gradient reweighting.
  4. The evaluation is mainly conducted on the relatively small AVEC 2013 and AVEC 2014 benchmarks, so the evidence for the claimed model-agnostic and generalizable nature of DMVS remains limited beyond the evaluated backbones and datasets. In addition, Statistical support for performance gains is limited. The reported improvements over the best published RMSE are only 0.11 on AVEC 2013 and 0.01 on AVEC 2014. Most main comparisons lack multi-seed statistics and subject-level confidence intervals. Five-run results for selected Northwind ablations do not resolve uncertainty in the main benchmarks.
Questions for authors
  1. Please clarify whether the GMVB update corresponds to the gradient of an explicit surrogate objective, and if not, provide a principled optimization interpretation of the relevance-weighted gradient?
  2. Please provide additional evidence that the DSC relevance scores identify genuinely depression-informative facial behaviours rather than merely selecting segments that are statistically correlated with the video-level depression label?
  3. How does DMVS compare with a standard temporal-attention or MIL model trained with the same video-level GradCache mechanism, and can the authors demonstrate that FBSP and DSC provide benefits beyond generic temporal weighting?
8 comments
AuthorsOpenReview ↗

Response to critical weaknesses 1

1.The main methodological contribution of GMVB appears relatively incremental, as its core operation amounts to multiplying the feature gradients obtained from video-level loss by learned relevance scores, which may be viewed as a combination of existing gradient caching and relevance-based gradient reweighting.

We disagree that the main contribution of GMVB is "a combination of existing gradient caching and relevance-based gradient reweighting". No existing "relevance-based gradient reweighting" method constructs variable-length facial behaviour segments across the full video, estimating their depression relevance to the video-level label, and then selects segments with higher depression relevance within that video to train an ADA model under full-video supervision. Moreover, the mentioned multiplying operation is not the main contribution but only one computation step in implementing GMVB.

As stated in the Introduction, our approach makes two key methodological contributions:

(1) An end-to-end full-video supervision strategy that models variable-length facial behaviour segments and weights their representations using estimated depression relevance, enabling every facial segment-level features to be optimised jointly within the full-video behavioural context (line 108). Novelty: Unlike existing ADA methods that supervise fixed-length segments separately or down-sample the video, it trains on the full video with only the video-level label.

(2) A depression relevance-weighted gradient modulation strategy that maps the relevance of each variable-length segment to its contained facial behaviour units and accordingly weights their contributions for backbone optimisation under the video-level loss (line 112). These weights are derived from the segment-level depression relevance learned by FVC and guide the backbone in learning depression-informative features. The multiplication in GMVB mentioned is only one implementation step of this contribution.

AuthorsOpenReview ↗

Response to critical weaknesses 2

2.Although the paper rigorously shows that sequential recomputation can recover the ordinary full-video gradient, the relevance-modulated gradient is generally not the gradient of the original video-level objective, and the paper does not provide a clear optimization objective or theoretical justification for this modified update.

We disagree that the paper lacks a clear optimisation objective. The relevance-modulated gradient is computed from the gradient of the original video-level objective, and the modified update of GMVB aligns with the video-level objective.

GMVB updates the backbone using the modulated gradient ∑t[u]tgt(θ)\sum_t[\mathbf{u}]_t\mathbf{g}_t^{(\theta)} (Appendix I). Let θk\theta_k be the current backbone parameters at iteration kk, then the FBU feature ft(k)=ft(θk)\mathbf{f}_t^{(k)}=\mathbf{f}_t(\theta_k) and relevance score [u]t(k)[\mathbf{u}]_t^{(k)} computed in this iteration are treated as constants. We define a video-level objective L~vid(k)\tilde{\mathcal{L}}_{\mathrm{vid}}^{(k)} as:

\tilde{\mathcal{L}}_{\mathrm{vid}}^{(k)}(\theta)&=\mathrm{FVC}\bigl(\\{\tilde{\mathbf{f}}_t^{(k)}(\theta)\\}_{t=1}^{T},y\bigr),\qquad \tilde{\mathbf{f}}_t^{(k)}(\theta)=\mathbf{f}_t^{(k)}+[\mathbf{u}]_t^{(k)}\bigl(\mathbf{f}_t(\theta)-\mathbf{f}_t^{(k)}\bigr). \end{aligned}$$ At $\theta=\theta_k$, $\tilde{\mathbf{f}}_t^{(k)}(\theta_k)=\mathbf{f}_t(\theta_k)$ for every FBU $u_t$. Differentiating $\tilde{\mathbf{f}}_t^{(k)}(\theta)$ with respect to $\theta$ gives $\partial\tilde{\mathbf{f}}_t^{(k)}/\partial\theta=[\mathbf{u}]_t^{(k)}\\,\partial\mathbf{f}_t/\partial\theta$. Applying the chain rule, with all derivatives evaluated at $\theta_k$, gives $$\nabla_\theta\tilde{\mathcal{L}}_{\mathrm{vid}}^{(k)}(\theta_k)=\sum_t\left(\frac{\partial\tilde{\mathbf{f}}_t^{(k)}}{\partial\theta}\right)^{\top}\frac{\partial\tilde{\mathcal{L}}_{\mathrm{vid}}^{(k)}}{\partial\tilde{\mathbf{f}}_t^{(k)}}=\sum_t[\mathbf{u}]_t^{(k)}\left(\frac{\partial\mathbf{f}_t}{\partial\theta}\right)^{\top}\mathbf{g}_t^{\mathbf f}=\sum_t[\mathbf{u}]_t^{(k)}\mathbf{g}_t^{(\theta)}.$$ At each iteration, the update of GMVB on $\mathcal{L}_{\mathrm{vid}}$ is thus the ordinary gradient update on $\tilde{\mathcal{L}}_{\mathrm{vid}}^{(k)}$. For a small change $\delta$ in the backbone parameters, the first-order approximation of the new and original video-leve objectives are $$\begin{aligned} \mathcal{L}_{\mathrm{vid}}(\theta_k+\delta)&\approx\mathcal{L}_{\mathrm{vid}}(\theta_k)+\sum_t\langle\mathbf{g}_t^{(\theta)},\delta\rangle,\\\\ \tilde{\mathcal{L}}_{\mathrm{vid}}^{(k)}(\theta_k+\delta)&\approx\tilde{\mathcal{L}}_{\mathrm{vid}}^{(k)}(\theta_k)+\left\langle\nabla_\theta\tilde{\mathcal{L}}_{\mathrm{vid}}^{(k)}(\theta_k),\delta\right\rangle\\\\ &=\mathcal{L}_{\mathrm{vid}}(\theta_k)+\sum_t[\mathbf{u}]_t^{(k)}\\,\langle\mathbf{g}_t^{(\theta)},\delta\rangle. \end{aligned}$$ Both approximations use the same inner-product terms. These terms are summed with unit weights for the original loss change and with relevance weights in [0, 1] for the modified loss change. As the weights are non-negative, $\tilde{\mathcal{L}}_{\mathrm{vid}}^{(k)}$ keeps the gradient direction of every FBU and only reduces the magnitude for FBUs with lower relevance. The derivation above provides an explicit justification for this modified update by showing that, at each iteration, **the relevance-modulated gradient used by GMVB equals the ordinary backbone gradient of an explicitly defined objective that retains the original video-level prediction task.** We have added this analysis to Appendix I of the resubmitted manuscript.
AuthorsOpenReview ↗

Response to critical weaknesses 3 - 4

3.The depression relevance estimated by DSC is learned only from video-level labels without segment-level annotations, raising concerns about whether the selected segments genuinely correspond to depression-related behaviours or merely capture features correlated with the video-level label.

We disagree that learning “only from video-level labels without segment-level annotations” is itself a critical weakness, as practical application and existing ADA datasets barely provide segment-level annotation. As a result, our DMVS is designed for address this issue, which identifies depression-related facial behaviour segments without requiring segment-level annotation but only use video-level depression labels.

Depression relevance and correlation with the video-level label are not mutually exclusive. The video-level depression label measures depression severity, and clinical research has linked facial behaviours to changes in severity scores [2]. Correlation with the label therefore does not, by itself, undermine depression relevance. The AU analysis below provides further support by showing that the selected segments exhibit behavioural patterns consistent with clinical findings.

We compared facial action units (AUs) extracted from the 10 highest- and lowest-ranked facial behaviour segments defined their by depression relevances within each of five randomly sampled test videos (Appendix K). Effect sizes are reported as Cohen's dd. The highest-ranked segments show higher AU45 (blink) intensity (d=0.81d=0.81), greater AU04 (brow lowerer) variability (d=0.54d=0.54), and lower AU12 (lip corner puller) activity. These AU differences involve blinking, smiling and brow activity, behaviours linked to depression in clinical studies [1,2,3], providing facial behavioural support for the depression relevance of the selected segments.

[1] Altered pupil light and darkness reflex and eye-blink responses in late-life depression.

[2] Nonverbal social withdrawal in depression: Evidence from manual and automatic analyses.

[3] Agitation-increased electromyogram activity in the corrugator muscle region: a possible explanation of the “Omega sign”?

4.The experimental comparison lacks sufficiently strong baselines based on standard temporal attention or multiple-instance learning with video-level GradCache, making it difficult to determine whether the gains arise from the proposed FBSP/DSC design rather than from generic temporal weighting and gradient reweighting.

We disagree that the baselines are insufficient.

First, as a plug-and-play training strategy, DMVS lowers the RMSE of all five different models in Table 3 relative to reproduced training, e.g., VAT from 7.47 to 6.88.

Second, our ablations already included MIL with video-level GradCache and isolate the contributions of FBSP and GMVB.

(1) “Multiple-instance learning with video-level GradCache”. Row 4 of Table 2 adds DSC, a WSDDN-style MIL module, to the video-level GradCache baseline in Row 2.

(2) “FBSP/DSC design rather than generic temporal weighting”. FBSP provides additional gains beyond the MIL selection and weighting already present in Row 4. Adding FBSP while retaining DSC and GradCache (Row 4 to Row 5) lowers RMSE in all six settings.

(3) "Generic gradient reweighting". Adding GMVB to the same FVC and GradCache pipeline (Table 2, Rows 5 and 6) lowers RMSE in all six settings. With VAT on AVEC 2014 Northwind (Table 5), GMVB achieves a five-run mean RMSE of 7.1856, compared with 7.5063, 7.5638 and 7.5101 for unit, mean and shuffled weights. Shuffling preserves the weight values but disrupts their assignment to FBUs. The resulting increase in RMSE supports the importance of matching learned relevance weights to the corresponding FBUs, beyond "generic gradient reweighting".

(4) “Standard temporal attention”. The ablations in (2) and (3) already isolate the gains from FBSP and GMVB. However, we have also added an attention-pooling baseline with video-level GradCache to Appendix D to directly compare our design with conventional temporal weighting. This baseline replaces FBSP and DSC with the gated attention pooling of Ilse et al. [1] over FBU features, keeping the VAT backbone, losses and training protocol unchanged. The table below compares the five-run results on AVEC 2014 Northwind.

Video-level moduleBackbone gradient weightsRMSE (mean ±\pm SD)
Gated attention poolingUnit7.6307±0.06417.6307\pm0.0641
FVCUnit7.5063±0.19977.5063\pm0.1997
FVCRelevance (GMVB)7.1856±0.10347.1856\pm0.1034

[1] Attention-based Deep Multiple Instance Learning.

Reviewer_civgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Dual-phase Memory-efficient Video-level Supervision (DMVS) to address key challenges in long-video facial depression assessment: traditional short fixed-length segment training fails to fully leverage complete video-level supervision, whereas end-to-end training on full-length videos incurs prohibitively high GPU memory overhead. The proposed method aggregates short-term Facial Behaviour Units (FBUs) into behavioral segments spanning various time scales and computes the depression relevance score of each segment. These estimated depression-relevance scores are used to weight and modulate the feature gradients of the corresponding FBUs, thereby guiding the backbone network to focus more on behavioral segments containing depression-related features. Additionally, activation re-computation and feature-gradient caching are employed to reduce memory consumption, thereby mitigating the memory overhead of end-to-end video-level training. Experiments on the AVEC 2013 and AVEC 2014 benchmarks across multiple backbone architectures demonstrate noticeable performance improvements.

Critical strengths

1.Relevance-Weighted Gradient Modulation Mechanism. A key strength of this work lies in its novel gradient modulation strategy. It propagates depression-relevance scores estimated from variable-length behavioral segments, to the feature gradients of the corresponding Facial Behaviour Unit (FBU) backbones. This effectively bridges video-level supervision with segment-level feature learning, allowing the model to dynamically modulate the contribution of individual video segments to parameter updates based on their behavioral significance. 2. Model-Agnostic Plug-and-Play Generalizability. Instead of introducing yet another task-specific architecture, the authors design a model-agnostic and plug-and-play training paradigm applicable to a wide spectrum of existing models. The robust performance improvements across five distinct architectures validate the method's versatility and effectiveness.

Critical weaknesses

As noted in Appendix Section A, the authors adopt a checkpoint selection protocol that the training and validation sets are merged, and the best-performing checkpoint is chosen based on the official test split. Despite this honest disclosure, this test-based model selection procedure inherently risks overfitting the test data, thereby introducing evaluation bias that may compromise the reliability of reported absolute performance and comparative improvements.

Questions for authors

What is the theoretical motivation or mathematical derivation behind using the FBU-level depression relevance scores as direct multiplicative weights on the feature gradients within GMVB? Specifically, how does this gradient-scaling scheme formally align with optimizing the true video-level objective, and what happens to optimization stability if the relevance score is inaccurate or noisy?

4 comments
AuthorsOpenReview ↗

Response to the critical weaknesses

As noted in Appendix Section A, the authors adopt a checkpoint selection protocol that the training and validation sets are merged, and the best-performing checkpoint is chosen based on the official test split. Despite this honest disclosure, this test-based model selection procedure inherently risks overfitting the test data, thereby introducing evaluation bias that may compromise the reliability of reported absolute performance and comparative improvements.

We thank the reviewer for the comment. The gains of DMVS are not caused by evaluation bias, as they hold when checkpoints are selected on the development set. We compared models trained with and without DMVS using three backbones (VAT, LMT and TSSTANet) on AVEC 2013 and AVEC 2014 Northwind. We trained every model on the official training split only, selected the checkpoint with the lowest RMSE on the official development split, and evaluated this checkpoint on the official test split. Test labels are used neither for training nor for checkpoint selection, and the hyperparameters follow the settings of the main paper. We ran each setting five times and report the mean test RMSE and CCC with standard deviation (SD) over the five runs in the table below.

BackboneDMVSAVEC 2013 RMSEAVEC 2013 CCCNorthwind RMSENorthwind CCC
VATWithout8.6835±0.34308.6835\pm0.34300.6520±0.05890.6520\pm0.05898.7204±0.24968.7204\pm0.24960.6667±0.02240.6667\pm0.0224
VATWith8.0718±0.51678.0718\pm0.51670.7004±0.04660.7004\pm0.04668.1712±0.25538.1712\pm0.25530.7002±0.03390.7002\pm0.0339
LMTWithout8.8714±0.48348.8714\pm0.48340.6010±0.06570.6010\pm0.06578.6548±0.67378.6548\pm0.67370.6503±0.07540.6503\pm0.0754
LMTWith8.3066±0.52938.3066\pm0.52930.6839±0.03580.6839\pm0.03588.2565±0.37308.2565\pm0.37300.6703±0.04220.6703\pm0.0422
TSSTANetWithout9.8851±0.19099.8851\pm0.19090.5287±0.02390.5287\pm0.023910.2087±0.407310.2087\pm0.40730.4682±0.05520.4682\pm0.0552
TSSTANetWith9.1966±0.33309.1966\pm0.33300.5777±0.03710.5777\pm0.03719.7402±0.32269.7402\pm0.32260.5344±0.05860.5344\pm0.0586

DMVS lowers the mean test RMSE and raises the mean test CCC of all three backbones on both datasets. Averaged over the three backbones, DMVS reduces the RMSE by 0.62 on AVEC 2013 and by 0.47 on AVEC 2014 Northwind.

The main paper keeps the protocol of prior ADA studies for comparison with their published results. For AVEC datasets, we follow the same protocol implemented in most prior ADA studies [1,2,3,4,5] that merge the training and validation sets for model training while evaluating on the test set. In our reimplementations of [1,2], we obtained results comparable to the reported results only when we trained on the combined training and validation sets and selected checkpoints on the test set. When we instead trained on the training set, selected checkpoints on the development set, and then evaluated on the test split, the test RMSE was higher than that reported in the original papers. To support fair comparisons, we therefore used the combined splits, consistent with the public implementations of [1,2,3,4,5] (code links are provided). Under this protocol, each reproduced backbone and its DMVS counterpart are trained and selected in the same way, and DMVS lowers the RMSE of all five backbones on both AVEC 2013 and AVEC 2014 (Table 3 of the main paper). Because checkpoint selection uses test labels, the absolute results in the main paper are comparable with published results. We have stated this limitation in Appendix A and added the development-set results to Appendix M of the resubmitted manuscript.

[1] LMTformer: Facial Depression Recognition with Lightweight Multi-Scale Transformer from Videos (https://github.com/helang818/LMTformer)

[2] DepMGNN: Matrixial Graph Neural Network for Video-based Automatic Depression (https://github.com/AffectAI/MGNN)

[3] Explainable Depression Assessment from Face Videos by Weakly Supervised Learning (https://github.com/liaorongfan/ExpADA)

[4] A Deep Multiscale Spatiotemporal Network for Assessing Depression from Facial Dynamics (https://github.com/wheidima/MSN)

[5] MDN: A Deep Maximization-Differentiation Network for Spatio-Temporal Depression Detection (https://github.com/wheidima/MDN)

AuthorsOpenReview ↗

Response to questions for authors (Motivation of multiplicative weights on feature gradients)

What is the theoretical motivation or mathematical derivation behind using the FBU-level depression relevance scores as direct multiplicative weights on the feature gradients within GMVB?

We thank the reviewer for the comment.

Motivation of multiplicative weights on feature gradients: The FBUs estimated to be more relevant to depression should have more impact in model's optimisation, allowing the trained model focus more on identifying depression-related FBUs from the given face videos. Since there is no segment-level annotations available but only video-level labels for learning variable-length depressive facial behaviours from videos, we use multiple instance learning (MIL) and treat each video as a bag of variable-length facial behaviour segments that have different associations to the video-level depression states. Following the MIL ranking of [1], DSC assigns each segment a depression relevance score and selects segments with the highest scores, denoted [r]πk[\mathbf{r}]_{\pi_k}. As FBU utu_t can belong to several selected segments, its relevance score [u]t[\mathbf{u}]_t is computed by summing relevance scores of the selected segments containing it (Eq. 7). GMVB then multiplies the corresponding FBU-level feature gradient by the corresponding relevance score [u]t[\mathbf{u}]_t, enforcing the model learning focuses on such selected depression-related facial behaviour segments.

[1] Explainable Depression Assessment from Face Videos by Weakly Supervised Learning

AuthorsOpenReview ↗

Response to questions for authors (Alignment with the video-level objective)

Alignment with the video-level objective: The use of GMVB still preserves the video-level objective of predicting a depression score from the input face video. GMVB changes only the gradient for updating the backbone. To show that GMVB aligns with the video-level objective, we compare two training procedures for the same network, which consists of the backbone M\mathcal{M} with parameters θ\theta and the FVC module.

ProcedureObjectiveGradient used to update the backbone
Procedure 1Lvid\mathcal{L}_{\mathrm{vid}}∑t[u]t,gt(θ)\sum_t[\mathbf{u}]_t\\,\mathbf{g}_t^{(\theta)} (weighted by relevance, GMVB)
Procedure 2L~vid(k)\tilde{\mathcal{L}}_{\mathrm{vid}}^{(k)}∇θL~vid(k)\nabla_\theta\tilde{\mathcal{L}}_{\mathrm{vid}}^{(k)} (ordinary gradient, no weighting)

The procedure 1 is our proposed method that utilises the original objective function but modified the gradient for back-propagation, while the procedure 2 retains the ordinary gradient but changes the objective function. The following analysis shows that: (i) the two procedures produce the same network after training; and (ii) the objective L~vid(k)\tilde{\mathcal{L}}_{\mathrm{vid}}^{(k)} of the second procedure is aligned with the original objective Lvid\mathcal{L}_{\mathrm{vid}}.

Procedure 1 (this paper). The network update uses modified gradient. We write ft(θ)=M(ut;θ)\mathbf{f}_t(\theta)=\mathcal{M}(u_t;\theta) for the feature of FBU utu_t. Let FVC(⋅,y)\mathrm{FVC}(\cdot,y) denote the function that computes the video-level loss from all FBU features and the video-level label yy. The objective of Procedure 1 is

Lvid(θ)=FVC(ft(θ)t=1T,y).\mathcal{L}_{\mathrm{vid}}(\theta)=\mathrm{FVC}\bigl(\\{\mathbf{f}_t(\theta)\\}_{t=1}^{T},y\bigr).

The backbone gradient is ∇θLvid=∑tgt(θ)\nabla_\theta\mathcal{L}_{\mathrm{vid}}=\sum_t\mathbf{g}_t^{(\theta)} (Appendix I). GMVB updates the backbone using the modulated gradient ∑t[u]tgt(θ)\sum_t[\mathbf{u}]_t\mathbf{g}_t^{(\theta)}.

Procedure 2. The network update uses unmodified gradient with a new objective L~vid(k)\tilde{\mathcal{L}}_{\mathrm{vid}}^{(k)}. Let θk\theta_k be the current backbone parameters at iteration kk, then the FBU feature ft(k)=ft(θk)\mathbf{f}_t^{(k)}=\mathbf{f}_t(\theta_k) and relevance score [u]t(k)[\mathbf{u}]_t^{(k)} computed in this iteration are treated as constants. We define the objective of Procedure 2 as:

\tilde{\mathcal{L}}_{\mathrm{vid}}^{(k)}(\theta)&=\mathrm{FVC}\bigl(\\{\tilde{\mathbf{f}}_t^{(k)}(\theta)\\}_{t=1}^{T},y\bigr),\qquad \tilde{\mathbf{f}}_t^{(k)}(\theta)=\mathbf{f}_t^{(k)}+[\mathbf{u}]_t^{(k)}\bigl(\mathbf{f}_t(\theta)-\mathbf{f}_t^{(k)}\bigr). \end{aligned}$$ **(i) The two procedures produce the same network.** At $\theta=\theta_k$, $\tilde{\mathbf{f}}_t^{(k)}(\theta_k)=\mathbf{f}_t(\theta_k)$ for every FBU $u_t$. Differentiating $\tilde{\mathbf{f}}_t^{(k)}(\theta)$ with respect to $\theta$ gives $\partial\tilde{\mathbf{f}}_t^{(k)}/\partial\theta=[\mathbf{u}]_t^{(k)}\\,\partial\mathbf{f}_t/\partial\theta$. Applying the chain rule, with all derivatives evaluated at $\theta_k$, gives $$\nabla_\theta\tilde{\mathcal{L}}_{\mathrm{vid}}^{(k)}(\theta_k)=\sum_t\left(\frac{\partial\tilde{\mathbf{f}}_t^{(k)}}{\partial\theta}\right)^{\top}\frac{\partial\tilde{\mathcal{L}}_{\mathrm{vid}}^{(k)}}{\partial\tilde{\mathbf{f}}_t^{(k)}}=\sum_t[\mathbf{u}]_t^{(k)}\left(\frac{\partial\mathbf{f}_t}{\partial\theta}\right)^{\top}\mathbf{g}_t^{\mathbf f}=\sum_t[\mathbf{u}]_t^{(k)}\mathbf{g}_t^{(\theta)}.$$ **At each iteration, the update of GMVB on $\mathcal{L}_{\mathrm{vid}}$ is thus the ordinary gradient update on $\tilde{\mathcal{L}}_{\mathrm{vid}}^{(k)}$.** **(ii) $\tilde{\mathcal{L}}_{\mathrm{vid}}^{(k)}$ is aligned with the video-level objective $\mathcal{L}_{\mathrm{vid}}$.** For a small change $\delta$ in the backbone parameters, the first-order approximations are $$\begin{aligned} \mathcal{L}_{\mathrm{vid}}(\theta_k+\delta)&\approx\mathcal{L}_{\mathrm{vid}}(\theta_k)+\sum_t\langle\mathbf{g}_t^{(\theta)},\delta\rangle,\\\\ \tilde{\mathcal{L}}_{\mathrm{vid}}^{(k)}(\theta_k+\delta)&\approx\tilde{\mathcal{L}}_{\mathrm{vid}}^{(k)}(\theta_k)+\left\langle\nabla_\theta\tilde{\mathcal{L}}_{\mathrm{vid}}^{(k)}(\theta_k),\delta\right\rangle\\\\ &=\mathcal{L}_{\mathrm{vid}}(\theta_k)+\sum_t[\mathbf{u}]_t^{(k)}\\,\langle\mathbf{g}_t^{(\theta)},\delta\rangle. \end{aligned}$$ Both approximations use the same inner-product terms, which are summed with unit weights for the original loss change and relevance weights in [0, 1] for the modified loss change. **As the weights are non-negative, $\tilde{\mathcal{L}}_{\mathrm{vid}}^{(k)}$ keeps the gradient direction of every FBU and only reduces the magnitude for FBUs with lower relevance.**
AuthorsOpenReview ↗

Response to questions for authors (Optimization stability for inaccurate or noisy relevance score)

what happens to optimization stability if the relevance score is inaccurate or noisy?

We thank the reviewer for the comment.

Optimization stability for inaccurate or noisy relevance score: Inaccurate scores amplify no FBU gradient, as each weight lies in [0,1][0,1], and fully random scores reduce GMVB to mean weighting plus zero-mean noise of bounded variance. The shuffled-weight control (Table 5, Appendix C) permutes the scores within each video by a random π\pi at each iteration. For fixed gradients and scores,

=\bar u\\,\nabla_\theta\mathcal{L}_{\mathrm{vid}},$$ where $\bar u=T^{-1}\sum_t[\mathbf{u}]_t$ is the mean FBU relevance score and $T$ is the number of FBUs in the video. For $T>1$, the expected squared deviation from this expectation is at most $\frac{T}{T-1}\bar u(1-\bar u)\sum_t\\|\mathbf{g}_t^{(\theta)}-T^{-1}\nabla_\theta\mathcal{L}_{\mathrm{vid}}\\|^2$. This control gives an RMSE of $7.5101\pm0.1372$, close to mean weights ($7.5638\pm0.2431$) and unit weights ($7.5063\pm0.1997$). Shuffled scores thus lose the gain of GMVB ($7.1856\pm0.1034$) but stay at the level of unit weights. We have added this stability analysis to Appendix C and the analysis of alignment with the video-level objective to Appendix I of the resubmitted manuscript to address the reviewer’s concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.