- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Dual-phase Memory-efficient Video-level Supervision (DMVS) for facial depression assessment, aiming to replace segment-level supervision and aggressive temporal subsampling with end-to-end video-level training. DMVS caches features from fixed-length video units, constructs adaptive-duration segment representations, and uses category-guided relevance scores to aggregate selected segments for video-level prediction. It then maps segment relevance to individual units, modulates their feature gradients, and updates the backbone through sequential activation recomputation. The approach combines video-level supervision with relevance-weighted feature learning while reducing backbone activation storage. Experiments on AVEC 2013 and AVEC 2014 report lower RMSE than in-house reproduced baselines across five backbones. Additional evaluations include component ablations, controls for gradient modulation, an ablation of adaptive segment duration, memory profiling, and an extension to apparent personality prediction on ChaLearn 2016.
Critical strengths
-
Well-motivated and broadly applicable training strategy: DMVS addresses the mismatch between video-level depression labels and segment-level training, enabling joint optimization of segment features under full-video supervision while bounding backbone activation storage to a single FBU. Reported improvements over reproduced baselines across five ADA-specific and general-purpose backbones, using a shared hyperparameter configuration, suggest that its usefulness extends beyond a single model design. This is the main reason I view the work as potentially valuable to the ADA community.
-
Informative component controls: With VAT on AVEC 2014 Northwind, comparisons against unit, mean, and shuffled gradient weights, together with fixed and shared learned segment durations, support the benefits of relevance-aligned gradient modulation and adaptive segment construction.
Critical weaknesses
The contribution of end-to-end encoder training is not fully isolated: Table 2 shows no consistent gain from video-level supervision alone, but it lacks a matched baseline using the same FVC head and losses on frozen segment-trained features. Adding DSC also introduces an auxiliary classification loss. Although matched controls support GMVB’s incremental benefit, the overall gains cannot be clearly attributed to end-to-end encoder adaptation versus the aggregation head and additional supervision.
Questions for authors
Equation (7) sums relevance scores over selected segments covering each FBU, so greater segment overlap can increase an FBU’s gradient weight even without higher individual segment scores. What motivates this accumulation rule, and how do the existing analyses distinguish the benefit of depression-relevance weighting from the effect of overlap-based weighting?
2 comments
Response to questions for authors
Equation (7) sums relevance scores over selected segments covering each FBU, so greater segment overlap can increase an FBU’s gradient weight even without higher individual segment scores. What motivates this accumulation rule, and how do the existing analyses distinguish the benefit of depression-relevance weighting from the effect of overlap-based weighting?
We thank the reviewer for the comment. The accumulation rule in Eq. (7) is inspired by the observation that an individual facial behaviour unit (FBU) can contribute to multiple depression-informative facial behaviours occurring at different temporal scales. While each selected facial behaviour segment's relevance score reflects its estimated contribution to the video-level depression assessment, Eq. (7) aggregates the depression relevance evidence associated with each target FBU across different and multi-scale facial behavioural contexts, i.e., repeated inclusion in relevant facial behavioural contexts provides additional evidence of the FBU's importance to video-level depression assessment. This design also allows the gradient modulation to prioritise FBUs supported by multiple depression-informative behavioural contexts, thereby encouraging the backbone to learn facial behavioural features that are informative across different temporal scales.
To clarify the benefit of depression-relevance weighting from the effect of overlap, we compare GMVB with three control weighting rules, i.e., we ran each rule five times to report the minimum RMSE and the mean RMSE with SD over the five runs in the table below.
Let count the selected segments covering FBU , and let be their mean relevance score. For , Eq. (7) can be written as . For a fair comparison, each setting uses the weight , where the per-video factor (, or ) makes equal to the total GMVB weight of the same video. Appendix C of the resubmitted manuscript details the computation of and this comparison.
(1) The overlap-only control uses the base weight , which uses the number of the selected depression-related facial behaviour segments containing the target FBU to weight the FBU without considering their depression relevance scores;
(2) The mean-relevance control uses , which uses the mean depression relevance score of the selected depression-related facial behaviour segments containing the target FBU to weight the FBU without considering the number of segments;
(3) The max-relevance control uses , which uses the highest depression relevance score of the selected depression-related facial behaviour segments containing the target FBU to weight the FBU without considering the number of segments;
(4) GMVB accumulates the weighted sum by jointly considering the number of the FBU-related segments and their depression relevance scores.
| Weighting rule | Gradient weight | Minimum RMSE | Mean RMSE SD |
|---|---|---|---|
| Max-relevance | 7.3871 | ||
| Overlap-only | 7.2884 | ||
| Mean-relevance | 7.1797 | ||
| GMVB (sum) | 7.0940 |
As demostrated by the table, GMVB that considers FBU counts in the selected facial behaviour segments and accumulates their relevance scores obtains the lowest minimum and mean RMSE among the four rules. In contrast, only considering counts (Overlap-only), only considering the average relevance score of of all segments containing the target FBU, or only considering the highest relevance score among all segments containing the target FBU, achieved degraded performance. The conclusions are: (i) jointly using depression-relevance and overlap to weight the FBU yielded the best results; and (ii) mean depression-relevance weighting outperformed the overlap-based weighting, while overlap-based weighting outperformed the max depression-relevance weighting.
Response to the critical weakness
We thank the reviewer for the comment. Based on your valuable suggestions, we have added additional results in table below. All five settings use VAT encoder with the same FBSP, DSC and VLC modules and are evaluated on AVEC 2014 Northwind. Settings A and B use "frozen segment-trained features" for the training of the aggregation head, whereas C, D and E also update the encoder (encoder adaptation). Settings A and C remove the "auxiliary classification loss" Lcls, so DSC receives gradients only from Lreg and Lord. We ran each setting five times and report the minimum RMSE and the mean RMSE with sample standard deviation (SD) over the five runs.
(1) Frozen segment-trained features VS. encoder adaptation using the same aggregation head and losses (B VS. D): Encoder adaptation lowers the mean RMSE only from 7.5665 (B) to 7.5063 (D), a reduction of 0.0602, which is smaller than the SD of either setting. Encoder adaptation alone therefore brings no clear gain, in agreement with Table 2.
(2) Contribution of GMVB to the gain of encoder adaptation (B, D and E): D and E share the same aggregation head, losses and encoder adaptation, and differ only in the gradient multiplier. Replacing the unit multipliers with [u]t lowers the mean RMSE from 7.5063 (D) to 7.1856 (E), a reduction of 0.3207, which exceeds the SD of either setting. From B to E, the mean RMSE drops by 0.3809, of which encoder adaptation with unit multipliers contributes 0.0602 (B to D) and GMVB contributes 0.3207 (D to E). The gain of encoder adaptation is therefore mainly attributable to GMVB's relevance-based gradient modulation.
(3) Auxiliary classification loss in encoder adaptation (C VS. D): Adding Lcls lowers the mean RMSE from 7.6636 (C) to 7.5063 (D), a reduction of 0.1573, close to the reduction of 0.1524 from A to B with a frozen encoder. Lcls therefore brings a consistent gain regardless of encoder adaptation.
We have added these results and analyses to Appendix C of the resubmitted manuscript.