- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes EB-CaP for online test-time adaptation in fine-grained video facial expression recognition. The method generates class-conditioned cache entries from the current video representation and combines them with positive and negative target caches, using entropy and diversity gates to refine predictions while keeping the recognition model frozen. Experiments on three datasets report larger gains on BioVid and StressID and smaller improvements on BAH.
Critical strengths
- The paper tries to address cross-subject distribution shifts and cache cold starts. The motivation is clear and has practical relevance.
Critical weaknesses
- The comparison with the method (TTA-CaP) in Figure 1(b) is insufficient to support the paper's motivation. EB-CaP achieves lower WAR in Table 6, while this baseline is absent from the other main comparisons, leaving the practical benefits of the proposed design unclear.
- The framing of energy-based personalization appears overstated. Equation (1) yields a constant class-wise Euclidean gradient, so the sampled cache is personalized not through a subject-adaptive energy function, but through target initialization and finite-step refinement.
Questions for authors
- The validation/test separation for selecting prompts, cache capacities, and fusion weights is unclear in Appendix J, Figure 3, and Appendix K, raising concerns about test-set leakage. Were these settings selected on validation subjects separate from the final test subjects, and were any target test labels used?
- Algorithm 1 does not explain how its positive variance gain rule handles an empty cache, and Section 3.4 omits the exact cache scoring formulas. How is the first sample accepted, and how are cosine similarities transformed and aggregated into class scores?
- The temporal modeling control in Table 15 covers only MCP, leaving unclear whether the other main comparisons isolate the contribution of the adaptation strategy. Do all methods in Table 1, particularly CRG, use the same trained image and temporal encoders as EB-CaP, and if not, how were these differences controlled?
- DPE exceeds the highlighted scores for Sub-4 and Sub-8 in Table 2, while PromptAlign and ReTA exceed them for Sub-3 and Sub-6 in Table 4, respectively. Are these solely formatting errors, and do the corrections affect the subject-wise conclusions?
- The paper does not provide failure cases. Could you compare the frozen model and EB-CaP on one successful and one failed adaptation case, showing the score contributions of the sampled, positive, and negative caches?
3 comments
Next Response
"Q-1: Validation/test separation for hyperparameter selection."
"A-3": The configuration was selected on the target subjects. Reselecting it on independent validation subjects gives nearly identical results.
Our personalization setting has no labeled validation data for the target subjects. Following the protocol of previous subject-based adaptation work for FER [2, 3], the configuration analyses were therefore performed on the BioVid target subjects, and each setting was selected from these results: prompt templates (Appendix O, formerly J), cache capacities (Fig. 3) and fusion weights (Appendix P, formerly K). The same configuration was then applied unchanged to StressID and BAH. Selecting hyperparameters without labeled target data is a recognized challenge in TTA and UDA [4, 5, 6].
To assess possible leakage, we repeated the selection of all three settings on 10 held-out BioVid source subjects, disjoint from the test subjects. We retrained the source model on the remaining 67 source subjects and evaluated the selected configuration on the same 10 test subjects (Table R4).
| Configuration selected on | WAR (%) | F1 (%) |
|---|---|---|
| Test subjects (reported) | 81.0 ± 0.10 | 78.2 ± 0.18 |
| Independent validation subjects | 80.8 ± 0.07 | 78.0 ± 0.11 |
Table R4. EB-CaP on the 10 BioVid test subjects, with the configuration selected on the test subjects (as reported) or on 10 independent validation subjects. Mean ± std over test subjects.
The same prompt template and cache capacities are selected; only the fusion weights differ slightly. Performance changes by only 0.2 points (Fig. 3; Appendix P, Table 22), so our results are not driven by selection on the test subjects. We have added this analysis to Appendix E (Table 13) and clarified the protocol in Sec. 4.1. For the camera-ready version, we will repeat all experiments with configurations selected on independent validation subjects.
"Q-2: Empty-cache admission and cache scoring formulas."
"A-4": We have specified both in the revised paper.
First sample (Sec. 3.3; Algorithm 2, Appendix B). If the partition of class is empty, a sample that passes the entropy gate is admitted directly. A single-entry partition has diversity , so later samples are compared against it:
- Below capacity: a sample is admitted if it increases diversity ().
- At capacity: it replaces the stored entry with the highest entropy, but only if its own entropy is lower and the swap increases diversity (). Otherwise, it is discarded.
The same rule applies to and .
Class scores (Sec. 3.4, Eq. 3). Each cosine similarity is turned into a weight and summed over all entries of the class: , with . All entries contribute, and no single prototype is selected. The cache scores are then added to the CLIP scores, with subtracted (Eq. 4).
References
[2] Muhammad Osama Zeeshan, Masoumeh Sharafi, Benoît Savary, Alessandro Lameiras Koerich, Marco Pedersoli, and Eric Granger. Clip-autt: Test-time personalization with action unit prompting for fine-grained video emotion recognition. In European Conference on Computer Vision, pp. 680–698. Springer, 2026b.
[3] M. Sharafi et al., "Disentangled Source-Free Personalization for Facial Expression Recognition with Neutral Target Data," 2025 IEEE 19th International Conference on Automatic Face and Gesture Recognition (FG), Tampa/Clearwater, FL, USA, 2025, pp. 1-10, doi: 10.1109/FG61629.2025.11099202.
[4] Hao Zhao, Yuejiang Liu, Alexandre Alahi, and Tao Lin. On pitfalls of test-time adaptation. In International Conference on Machine Learning (ICML), 2023.
[5] Kaichao You, Ximei Wang, Mingsheng Long, and Michael Jordan. Towards accurate model selection in deep unsupervised domain adaptation. In International Conference on Machine Learning, pp. 7124–7133. PMLR, 2019.
[6] Kuniaki Saito, Donghyun Kim, Piotr Teterwak, Stan Sclaroff, Trevor Darrell, and Kate Saenko. Tune it the right way: Unsupervised validation of domain adaptation via soft neighborhood density. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9164–9173. IEEE, 2021.
Next Response
"Q-3: Same image and temporal encoders for all baselines."
"A-5": All methods in Table 1 use the same CLIP ViT-B/32 image encoder, fine-tuned on the same source subjects, with the same prompts, preprocessing and target data. The image-based methods (TPT, TDA, DPE, PromptAlign, ReTA, MCP and CRG) operate on frame embeddings averaged over each 16-frame window, while T3AL uses its own temporal modeling (Appendix C). To isolate the adaptation strategy, we equipped MCP and CRG with our trained temporal encoder (Table R5).
| Method | WAR (%) | F1 (%) |
|---|---|---|
| MCP | 75.7 | 73.0 |
| MCP + Temporal | 75.9 | 73.4 |
| MCP + Temporal + Sampled cache | 76.9 | 75.0 |
| CRG | 76.4 | 73.8 |
| CRG + Temporal | 77.0 | 74.1 |
| EB-CaP | 81.0 | 78.2 |
Table R5. Effect of temporal modeling on BioVid.
Temporal modeling alone yields only marginal gains (+0.2 and +0.6 WAR). The temporal encoder is trained on source subjects and kept fixed, so it does not adapt to the target subject. EB-CaP addresses this cross-subject shift directly by building its caches from the subject's own data. The improvement therefore comes mainly from the adaptation strategy (Appendix N, Table 20).
"Q-4: Bolding errors in Tables 2 and 4."
"A-6": These were formatting errors only: no numbers or averages changed. We have corrected the bold markings in all subject-wise tables to mark the best results among methods without source data at test time, including TTA-CaP without source data in the F1 tables. After these corrections, EB-CaP achieves the best or tied-best F1 on 5/10, 6/10 and 5/10 subjects (Tables 2–4) and the best or tied-best WAR on 10/10, 9/10 and 9/10 subjects (Tables 10–12) on BioVid, StressID and BAH, with the highest or tied-highest average on all three datasets. The original conclusions therefore hold: the gains are consistent in WAR and more concentrated in F1. We have revised Sec. 4.2 accordingly.
"Q-5: Success and failure case analysis."
"A-7": We compare a successful (Sub-4) and a failed (Sub-3) case on BioVid, adding each cache step by step to the frozen source model (Table R6). Both subjects are women aged 43.
| Subject | Source model | + Sampled cache | + Positive cache | + Negative cache (EB-CaP) |
|---|---|---|---|---|
| Sub-4 (success) | 60.0 | 78.5 | 89.2 | 95.4 |
| Sub-3 (failure) | 50.0 | 50.0 | 50.0 | 50.0 |
Table R6. WAR (%) on BioVid for a successful and a failed case (both women, aged 43), adding each cache to the frozen source model; the last column is the full EB-CaP.
For Sub-4, every cache contributes, and the sampled cache gives the largest gain (+18.5) by providing class-wise evidence before the target caches fill. For Sub-3, the source model predicts a single class for all windows, and all compared TTA methods also remain at 50.0% (Table 10). Despite the same age and sex, Sub-4 shows clear facial changes under pain, while Sub-3 shows only subtle ones, mainly eye closure (Fig. 4). Her two classes are therefore not separable in the source feature space, which cache-based adaptation cannot correct; all compared methods share this limitation. We have added this analysis to Appendix I (Table 15, Fig. 4).
Rebuttal by Authors
We thank Reviewer Yv99 for the valuable feedback and for finding the motivation clear and of practical relevance, addressing cross-subject distribution shifts and the cache cold-start problem in online video FER. We address each comment below; "W" indicates a weakness, "Q" a question, and "A" our answer.
"W-1: Comparison with TTA-CaP is insufficient."
"A-1": Following the suggestion, we have added TTA-CaP [1], with and without its source prototypes, to all main comparisons (Table R1).
Table R1. WAR (%) of TTA-CaP and EB-CaP. Mean ± std over the 10 target subjects.
The gain of TTA-CaP comes from the labeled source prototypes it stores at test time. Without them, its WAR drops by 3.4 / 2.3 / 0.4 points. EB-CaP uses no source data, labels or prototypes. It outperforms TTA-CaP without source data on all three datasets and is only 0.5 / 0.3 / 0.3 points below TTA-CaP with source data.
Table R2. Run time and memory on BioVid for a batch of 16 windows.
EB-CaP uses 32% less memory, since it stores no source prototypes. It is slower because it samples the personalized cache for each window, but only about 14 ms per window, well within real-time requirements. Removing source data also matters in practice: source data are facial videos of other people, which often cannot be kept on a deployment device (e.g., in healthcare), and source prototypes do not adapt to the target subject, whereas EB-CaP builds its entries from the subject's own video.
We have added TTA-CaP to Table 1, Table 5 and the subject-wise F1 tables (Tables 2–4), discussed the comparison in Sec. 4.2, and revised Figure 1(b) and the related work accordingly.
"W-2: Energy-based personalization framing appears overstated."
"A-2": We have revised the framing accordingly. The two parts of the sampler have distinct roles by design. Personalization comes from the initialization: each chain starts from the subject's embedding rather than from noise. The energy provides class guidance: 94.2% of chains become class-consistent within 6.7 steps, vs. 58.0% without it (Appendix F, Table 14). The energy is intentionally simple, adds no parameters and keeps sampling fast; a richer nonlinear energy is slower and less accurate (Sec. 4.3, Table 7). We have revised Sec. 3.1 and the contributions to state these roles explicitly.
To verify that the initialization personalizes the entries, we generated pain samples from each BioVid subject's neutral windows with both initializations and the same energy, step size, noise and stopping rule (Table R3).
Table R3. Euclidean distance between centroids of generated pain samples and real windows (ℓ2-normalized; lower = closer). Labels are used only to select real windows. Reference row: real pain–neutral distance of each subject. Mean ± std over BioVid subjects.
Our samples lie close to the subject's own pain windows (0.16) and far from other subjects' (0.34), while noise-initialized samples show no subject preference (0.38 vs. 0.40). Removing target initialization also reduces WAR from 81.0% to 78.2% (Tables 6 and 9). We have added this analysis to Sec. 4.3 (Table 8) and Appendix G.
References
[1] Masoumeh Sharafi, Muhammad Osama Zeeshan, Soufiane Belharbi, Alessandro Lameiras Koerich, Marco Pedersoli, and Eric Granger. Test-time adaptation via cache personalization for facial expression recognition in videos. arXiv preprint arXiv:2603.21309, 2026b.