acceptodds
Under review as a conference paper at ICLR 2027

Test-Time Adaptation with Online Personalized Energy-Based Cache for Fine-Grained Video Expression Recognition

Abstract

Facial expression recognition (FER) in videos remains challenging because models must identify subtle temporally evolving affective states that vary significantly across target individuals. Although vision-language models provide transferable visual-semantic representations, models trained on subject-independent source data often degrade under subject-specific distribution shifts at inference time. State-of-the-art test-time adaptation (TTA) methods typically optimize models during inference, increasing computational cost and latency. Cache-based approaches avoid parameter updates but typically require accumulating sufficient target samples to construct reliable class prototypes. This is difficult at the beginning of adaptation and when some classes are rarely observed. To alleviate these limitations, existing methods may store source prototypes, but these are not personalized to the current target subject. This paper introduces Energy-Based Cache Personalization (EB-CaP), a subject-based online TTA method for video FER that samples class-specific prototypes personalized to each target video on-the-fly. Unlike existing cache-based methods, \ours does not require observing and accumulating large amounts of target data or storing diverse source prototypes across subjects. Instead, it relies on a lightweight energy-based model (EBM) to sample class-wise prototypes from the current unlabeled video and populate a personalized cache online. The energy function relies only on the pretrained CLIP model, where the similarity between the visual embedding of the target video and the class text embeddings guides the energy-based sampling process. In parallel, positive and negative caches store reliable and uncertain target embeddings, respectively. An adaptive entropy gating follows the evolving confidence distribution to control cache updates, while a diversity gate prevents redundant samples from dominating the memory. Predictions are refined by combining cache-derived scores with the current CLIP scores. Experiments on 3 challenging video datasets for video FER – BioVid, StressID, and BAH– indicate that EB-CaP can outperform state-of-the-art TTA methods, while maintaining low computational and memory overhead. Our code is included in supplementary materials and will be made public.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
test time adaptation, personalization, videos, facial expression recognition, cache-based methods

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Yv99OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EB-CaP for online test-time adaptation in fine-grained video facial expression recognition. The method generates class-conditioned cache entries from the current video representation and combines them with positive and negative target caches, using entropy and diversity gates to refine predictions while keeping the recognition model frozen. Experiments on three datasets report larger gains on BioVid and StressID and smaller improvements on BAH.

Critical strengths
  1. The paper tries to address cross-subject distribution shifts and cache cold starts. The motivation is clear and has practical relevance.
Critical weaknesses
  1. The comparison with the method (TTA-CaP) in Figure 1(b) is insufficient to support the paper's motivation. EB-CaP achieves lower WAR in Table 6, while this baseline is absent from the other main comparisons, leaving the practical benefits of the proposed design unclear.
  2. The framing of energy-based personalization appears overstated. Equation (1) yields a constant class-wise Euclidean gradient, so the sampled cache is personalized not through a subject-adaptive energy function, but through target initialization and finite-step refinement.
Questions for authors
  1. The validation/test separation for selecting prompts, cache capacities, and fusion weights is unclear in Appendix J, Figure 3, and Appendix K, raising concerns about test-set leakage. Were these settings selected on validation subjects separate from the final test subjects, and were any target test labels used?
  2. Algorithm 1 does not explain how its positive variance gain rule handles an empty cache, and Section 3.4 omits the exact cache scoring formulas. How is the first sample accepted, and how are cosine similarities transformed and aggregated into class scores?
  3. The temporal modeling control in Table 15 covers only MCP, leaving unclear whether the other main comparisons isolate the contribution of the adaptation strategy. Do all methods in Table 1, particularly CRG, use the same trained image and temporal encoders as EB-CaP, and if not, how were these differences controlled?
  4. DPE exceeds the highlighted scores for Sub-4 and Sub-8 in Table 2, while PromptAlign and ReTA exceed them for Sub-3 and Sub-6 in Table 4, respectively. Are these solely formatting errors, and do the corrections affect the subject-wise conclusions?
  5. The paper does not provide failure cases. Could you compare the frozen model and EB-CaP on one successful and one failed adaptation case, showing the score contributions of the sampled, positive, and negative caches?
3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank Reviewer Yv99 for the valuable feedback and for finding the motivation clear and of practical relevance, addressing cross-subject distribution shifts and the cache cold-start problem in online video FER. We address each comment below; "W" indicates a weakness, "Q" a question, and "A" our answer.

"W-1: Comparison with TTA-CaP is insufficient."

"A-1": Following the suggestion, we have added TTA-CaP [1], with and without its source prototypes, to all main comparisons (Table R1).

MethodBioVidStressIDBAH
TTA-CaP without source data at test time78.1 ± 0.1979.2 ± 0.1768.8 ± 0.12
TTA-CaP with source data at test time81.5 ± 0.2081.5 ± 0.0969.2 ± 0.11
EB-CaP, without source data81.0 ± 0.1081.2 ± 0.1668.9 ± 0.20

Table R1. WAR (%) of TTA-CaP and EB-CaP. Mean ± std over the 10 target subjects.

The gain of TTA-CaP comes from the labeled source prototypes it stores at test time. Without them, its WAR drops by 3.4 / 2.3 / 0.4 points. EB-CaP uses no source data, labels or prototypes. It outperforms TTA-CaP without source data on all three datasets and is only 0.5 / 0.3 / 0.3 points below TTA-CaP with source data.

MethodTime / batch (ms)Memory (MB)WAR (%)
TTA-CaP (with source data)110.090481.5 ± 0.20
EB-CaP220.061281.0 ± 0.10

Table R2. Run time and memory on BioVid for a batch of 16 windows.

EB-CaP uses 32% less memory, since it stores no source prototypes. It is slower because it samples the personalized cache for each window, but only about 14 ms per window, well within real-time requirements. Removing source data also matters in practice: source data are facial videos of other people, which often cannot be kept on a deployment device (e.g., in healthcare), and source prototypes do not adapt to the target subject, whereas EB-CaP builds its entries from the subject's own video.

We have added TTA-CaP to Table 1, Table 5 and the subject-wise F1 tables (Tables 2–4), discussed the comparison in Sec. 4.2, and revised Figure 1(b) and the related work accordingly.

"W-2: Energy-based personalization framing appears overstated."

"A-2": We have revised the framing accordingly. The two parts of the sampler have distinct roles by design. Personalization comes from the initialization: each chain starts from the subject's embedding rather than from noise. The energy provides class guidance: 94.2% of chains become class-consistent within 6.7 steps, vs. 58.0% without it (Appendix F, Table 14). The energy is intentionally simple, adds no parameters and keeps sampling fast; a richer nonlinear energy is slower and less accurate (Sec. 4.3, Table 7). We have revised Sec. 3.1 and the contributions to state these roles explicitly.

To verify that the initialization personalizes the entries, we generated pain samples from each BioVid subject's neutral windows with both initializations and the same energy, step size, noise and stopping rule (Table R3).

InitializationOwn pain ↓Other subjects' pain ↑Own neutral ↑
Noise-initialized SGLD0.38 ± 0.040.40 ± 0.030.42 ± 0.05
EB-CaP (subject embedding)0.16 ± 0.030.34 ± 0.040.31 ± 0.04
Real pain windows (GT reference)N/AN/A0.35 ± 0.04

Table R3. Euclidean distance between centroids of generated pain samples and real windows (ℓ2\ell_2-normalized; lower = closer). Labels are used only to select real windows. Reference row: real pain–neutral distance of each subject. Mean ± std over BioVid subjects.

Our samples lie close to the subject's own pain windows (0.16) and far from other subjects' (0.34), while noise-initialized samples show no subject preference (0.38 vs. 0.40). Removing target initialization also reduces WAR from 81.0% to 78.2% (Tables 6 and 9). We have added this analysis to Sec. 4.3 (Table 8) and Appendix G.

References

[1] Masoumeh Sharafi, Muhammad Osama Zeeshan, Soufiane Belharbi, Alessandro Lameiras Koerich, Marco Pedersoli, and Eric Granger. Test-time adaptation via cache personalization for facial expression recognition in videos. arXiv preprint arXiv:2603.21309, 2026b.

AuthorsOpenReview ↗

Next Response

"Q-1: Validation/test separation for hyperparameter selection."

"A-3": The configuration was selected on the target subjects. Reselecting it on independent validation subjects gives nearly identical results.

Our personalization setting has no labeled validation data for the target subjects. Following the protocol of previous subject-based adaptation work for FER [2, 3], the configuration analyses were therefore performed on the BioVid target subjects, and each setting was selected from these results: prompt templates (Appendix O, formerly J), cache capacities (Fig. 3) and fusion weights (Appendix P, formerly K). The same configuration was then applied unchanged to StressID and BAH. Selecting hyperparameters without labeled target data is a recognized challenge in TTA and UDA [4, 5, 6].

To assess possible leakage, we repeated the selection of all three settings on 10 held-out BioVid source subjects, disjoint from the test subjects. We retrained the source model on the remaining 67 source subjects and evaluated the selected configuration on the same 10 test subjects (Table R4).

Configuration selected onWAR (%)F1 (%)
Test subjects (reported)81.0 ± 0.1078.2 ± 0.18
Independent validation subjects80.8 ± 0.0778.0 ± 0.11

Table R4. EB-CaP on the 10 BioVid test subjects, with the configuration selected on the test subjects (as reported) or on 10 independent validation subjects. Mean ± std over test subjects.

The same prompt template and cache capacities are selected; only the fusion weights differ slightly. Performance changes by only 0.2 points (Fig. 3; Appendix P, Table 22), so our results are not driven by selection on the test subjects. We have added this analysis to Appendix E (Table 13) and clarified the protocol in Sec. 4.1. For the camera-ready version, we will repeat all experiments with configurations selected on independent validation subjects.

"Q-2: Empty-cache admission and cache scoring formulas."

"A-4": We have specified both in the revised paper.

First sample (Sec. 3.3; Algorithm 2, Appendix B). If the partition Kc\mathcal{K}_c of class cc is empty, a sample that passes the entropy gate is admitted directly. A single-entry partition has diversity D=0D = 0, so later samples are compared against it:

  • Below capacity: a sample is admitted if it increases diversity (ΔD>0\Delta D > 0).
  • At capacity: it replaces the stored entry with the highest entropy, but only if its own entropy is lower and the swap increases diversity (ΔD>0\Delta D > 0). Otherwise, it is discarded.

The same rule applies to Kp\mathcal{K}^{p} and Kn\mathcal{K}^{n}.

Class scores (Sec. 3.4, Eq. 3). Each cosine similarity is turned into a weight and summed over all entries of the class: sK(c)=∑zj∈Kcexp⁡(−β(1−ztgt⊤zj))s^{\mathcal{K}}(c) = \sum_{\mathbf{z}_j \in \mathcal{K}_c} \exp(-\beta (1 - \mathbf{z}^{\mathrm{tgt}\top}\mathbf{z}_j)), with β=1.0\beta = 1.0. All entries contribute, and no single prototype is selected. The cache scores are then added to the CLIP scores, with sns^{n} subtracted (Eq. 4).

References

[2] Muhammad Osama Zeeshan, Masoumeh Sharafi, Benoît Savary, Alessandro Lameiras Koerich, Marco Pedersoli, and Eric Granger. Clip-autt: Test-time personalization with action unit prompting for fine-grained video emotion recognition. In European Conference on Computer Vision, pp. 680–698. Springer, 2026b.

[3] M. Sharafi et al., "Disentangled Source-Free Personalization for Facial Expression Recognition with Neutral Target Data," 2025 IEEE 19th International Conference on Automatic Face and Gesture Recognition (FG), Tampa/Clearwater, FL, USA, 2025, pp. 1-10, doi: 10.1109/FG61629.2025.11099202.

[4] Hao Zhao, Yuejiang Liu, Alexandre Alahi, and Tao Lin. On pitfalls of test-time adaptation. In International Conference on Machine Learning (ICML), 2023.

[5] Kaichao You, Ximei Wang, Mingsheng Long, and Michael Jordan. Towards accurate model selection in deep unsupervised domain adaptation. In International Conference on Machine Learning, pp. 7124–7133. PMLR, 2019.

[6] Kuniaki Saito, Donghyun Kim, Piotr Teterwak, Stan Sclaroff, Trevor Darrell, and Kate Saenko. Tune it the right way: Unsupervised validation of domain adaptation via soft neighborhood density. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9164–9173. IEEE, 2021.

AuthorsOpenReview ↗

Next Response

"Q-3: Same image and temporal encoders for all baselines."

"A-5": All methods in Table 1 use the same CLIP ViT-B/32 image encoder, fine-tuned on the same source subjects, with the same prompts, preprocessing and target data. The image-based methods (TPT, TDA, DPE, PromptAlign, ReTA, MCP and CRG) operate on frame embeddings averaged over each 16-frame window, while T3AL uses its own temporal modeling (Appendix C). To isolate the adaptation strategy, we equipped MCP and CRG with our trained temporal encoder (Table R5).

MethodWAR (%)F1 (%)
MCP75.773.0
MCP + Temporal75.973.4
MCP + Temporal + Sampled cache76.975.0
CRG76.473.8
CRG + Temporal77.074.1
EB-CaP81.078.2

Table R5. Effect of temporal modeling on BioVid.

Temporal modeling alone yields only marginal gains (+0.2 and +0.6 WAR). The temporal encoder is trained on source subjects and kept fixed, so it does not adapt to the target subject. EB-CaP addresses this cross-subject shift directly by building its caches from the subject's own data. The improvement therefore comes mainly from the adaptation strategy (Appendix N, Table 20).

"Q-4: Bolding errors in Tables 2 and 4."

"A-6": These were formatting errors only: no numbers or averages changed. We have corrected the bold markings in all subject-wise tables to mark the best results among methods without source data at test time, including TTA-CaP without source data in the F1 tables. After these corrections, EB-CaP achieves the best or tied-best F1 on 5/10, 6/10 and 5/10 subjects (Tables 2–4) and the best or tied-best WAR on 10/10, 9/10 and 9/10 subjects (Tables 10–12) on BioVid, StressID and BAH, with the highest or tied-highest average on all three datasets. The original conclusions therefore hold: the gains are consistent in WAR and more concentrated in F1. We have revised Sec. 4.2 accordingly.

"Q-5: Success and failure case analysis."

"A-7": We compare a successful (Sub-4) and a failed (Sub-3) case on BioVid, adding each cache step by step to the frozen source model (Table R6). Both subjects are women aged 43.

SubjectSource model+ Sampled cache+ Positive cache+ Negative cache (EB-CaP)
Sub-4 (success)60.078.589.295.4
Sub-3 (failure)50.050.050.050.0

Table R6. WAR (%) on BioVid for a successful and a failed case (both women, aged 43), adding each cache to the frozen source model; the last column is the full EB-CaP.

For Sub-4, every cache contributes, and the sampled cache gives the largest gain (+18.5) by providing class-wise evidence before the target caches fill. For Sub-3, the source model predicts a single class for all windows, and all compared TTA methods also remain at 50.0% (Table 10). Despite the same age and sex, Sub-4 shows clear facial changes under pain, while Sub-3 shows only subtle ones, mainly eye closure (Fig. 4). Her two classes are therefore not separable in the source feature space, which cache-based adaptation cannot correct; all compared methods share this limitation. We have added this analysis to Appendix I (Table 15, Fig. 4).

Reviewer_1d8YOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EB-CaP, an online test-time adaptation method for video facial-expression recognition. It initializes class-conditioned SGLD chains from the current target-video embedding to construct a personalized sampled cache, maintains positive and negative target caches controlled by entropy and diversity gates, and fuses cache scores with the logits of a frozen CLIP model. Experiments on BioVid, StressID, and BAH report competitive results. The main claimed contribution is to build class-wise cache evidence online from the current target feature without storing source prototypes.

Critical strengths

The paper tackles limited cache coverage at the start of adaptation to a new subject. Generating class-conditioned cache entries from the current input and combining them with target caches accumulated across videos is a promising direction. Inference requires neither model parameter updates nor retained source samples or prototypes. Table 11 ablates the gradient and noise terms separately, Table 12 examines gains at different adaptation stages, and Table 13 expands the target cohort from 10 to 40 subjects. These analyses are aligned with the motivation.

Critical weaknesses
  1. Appendix J compares 20 prompt templates on the BioVid target subjects and selects the template used by all prompt-based methods according to WAR/F1, without stating whether these subjects are separate from the final test subjects. Using the same template for every method does not ensure an independent test evaluation. The validation subjects used to select prompts, cache capacities, and fusion weights need to be identified. If final test labels were used, configuration selection must be repeated on independent validation data, followed by reevaluation on the test subjects.

  2. The interpretation of the personalized sampled cache is not established. Equation (1) gives a fixed class-directed gradient, while sampling adds isotropic noise, stops according to the same classifier, and retrieves generated embeddings against their initialization. Under these rules, inputs with identical original CLIP class scores induce identical sampled-cache score distributions. Tables 6 and 11 demonstrate useful input-conditioned stochastic refinement, but do not establish that the generated embeddings represent the current subject’s unobserved classes. The personalized-prototype interpretation requires validation against independently observed target-class representations; otherwise, the sampled-cache component should be characterized as stochastic score refinement.

  3. Section 4.1 leaves the actual tasks and evaluation units unclear. For BioVid, the selected pain levels and any merging of classes are unspecified. For StressID, the selected self-report label and resulting class definitions are unspecified. BAH has frame-level annotations, but the model produces a prediction for each 16-frame window. The paper does not explain how window predictions are aligned with annotations or whether WAR/F1 are computed over frames, windows, or videos. These definitions determine the classification tasks and reported metrics and need to be stated.

  4. Sections 3.2–3.3 and Algorithm 1 do not fully specify the cache-update rules. The positive and negative caches start empty and admit entries only when the variance gain is strictly positive, yet neither the gain for an empty class nor admission of the first sample is defined. At capacity, the paper only states that entropy is used for replacement, without specifying which entry is replaced or the order of replacement and diversity gating. These choices change the retained samples, so the update rules used in the experiments need to be specified.

  5. There are also some basic errors. Table 1 lists CRG as ICLR 2025, whereas the bibliography lists ICME 2025. Section 4.2 states that BioVid Sub-9 achieves perfect F1 across all methods, but Table 2 gives TPT an F1 of 99.0. These inconsistencies need to be checked and corrected.

Questions for authors
  1. Which subjects were used to select the prompt, cache capacities, and fusion weights, respectively, and do they overlap with the final test subjects? If final test labels were used for configuration selection, what are the test results after reselecting the configuration using independent validation data?

  2. Beyond satisfying the original classifier’s stopping criterion and improving accuracy through self-retrieval, what independent evidence shows that the generated embeddings represent the current subject’s corresponding expression classes?

  3. Which classes were used for BioVid, and were any pain levels merged? Which self-report label and class definitions were used for StressID? How are BAH's 16-frame window predictions aligned with annotations, and what is the evaluation unit for WAR/F1?

  4. How is variance gain defined for an empty class in each target cache, and how is the first sample admitted? At capacity, which entry is replaced, and in what order are replacement and diversity gating applied?

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank Reviewer 1d8Y for the thorough review and for recognizing that generating class-conditioned cache entries from the current input, combined with target caches accumulated across videos, is a promising direction for the cold-start problem, without parameter updates or retained source data. We are also glad the reviewer found the analyses of sampling, adaptation stages, and scaling well aligned with this motivation (Tables 14, 16 and 17). We address each comment below; "W" indicates a weakness, "Q" a question, and "A" our answer.

"W-1 & Q-1: Configuration selection and possible test-set leakage."

"A-1": The configuration was selected on the target subjects. Reselecting it on independent validation subjects gives nearly identical results.

Our personalization setting has no labeled validation data for the target subjects. Following the protocol of previous subject-based adaptation work for FER [1, 2], the configuration analyses were therefore performed on the BioVid target subjects, and each setting was selected from these results: prompt templates (Appendix O), cache capacities (Fig. 3) and fusion weights (Appendix P). Selecting hyperparameters without labeled target data is a recognized challenge in TTA and UDA [3, 4, 5].

To assess possible leakage, we repeated the selection of all three settings on 10 held-out BioVid source subjects, disjoint from the test subjects. We retrained the source model on the remaining 67 source subjects and evaluated the selected configuration on the same 10 test subjects (Table R1).

Configuration selected onWAR (%)F1 (%)
Test subjects (reported)81.0 ± 0.1078.2 ± 0.18
Independent validation subjects80.8 ± 0.0778.0 ± 0.11

Table R1. EB-CaP on the 10 BioVid test subjects, with the configuration selected on the test subjects (as reported) or on 10 independent validation subjects. Mean ± std over test subjects.

The same prompt template and cache capacities are selected; only the fusion weights differ slightly. Performance changes by only 0.2 points, consistent with the low sensitivity to these choices (Fig. 3; Appendix P, Table 22), so our results are not driven by selection on the test subjects. We have added this analysis to Appendix E (Table 13) and clarified the protocol in Sec. 4.1. For the camera-ready version, we will repeat all experiments for our method and the baselines on all three datasets with configurations selected on independent validation subjects.

References

[1] Muhammad Osama Zeeshan, Masoumeh Sharafi, Benoît Savary, Alessandro Lameiras Koerich, Marco Pedersoli, and Eric Granger. Clip-autt: Test-time personalization with action unit prompting for fine-grained video emotion recognition. In European Conference on Computer Vision, pp. 680–698. Springer, 2026b.

[2] M. Sharafi et al., "Disentangled Source-Free Personalization for Facial Expression Recognition with Neutral Target Data," 2025 IEEE 19th International Conference on Automatic Face and Gesture Recognition (FG), Tampa/Clearwater, FL, USA, 2025, pp. 1-10, doi: 10.1109/FG61629.2025.11099202.

[3] Hao Zhao, Yuejiang Liu, Alexandre Alahi, and Tao Lin. On pitfalls of test-time adaptation. In International Conference on Machine Learning (ICML), 2023.

[4] Kaichao You, Ximei Wang, Mingsheng Long, and Michael Jordan. Towards accurate model selection in deep unsupervised domain adaptation. In International Conference on Machine Learning, pp. 7124–7133. PMLR, 2019.

[5] Kuniaki Saito, Donghyun Kim, Piotr Teterwak, Stan Sclaroff, Trevor Darrell, and Kate Saenko. Tune it the right way: Unsupervised validation of domain adaptation via soft neighborhood density. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9164–9173. IEEE, 2021

AuthorsOpenReview ↗

Next Response

"W-2 & Q-2: Interpretation of the personalized sampled cache."

"A-2": The stopping rule only ensures class consistency; subject information comes from the initialization. Each chain starts from the subject's own embedding rather than from noise, so the entries are anchored at the current representation, not determined by the class scores alone.

Following the reviewer's suggestion, we validated the entries against independently observed target-class representations. For each BioVid subject, we generated pain samples from windows predicted as neutral and compared them with the subject's real pain windows, which are never used during sampling. As a control, we ran noise-initialized SGLD with the same energy, step size, noise and stopping rule (Table R2).

InitializationOwn pain ↓Other subjects' pain ↑Own neutral ↑
Noise-initialized SGLD0.38 ± 0.040.40 ± 0.030.42 ± 0.05
EB-CaP (subject embedding)0.16 ± 0.030.34 ± 0.040.31 ± 0.04
Real pain windows (GT reference)N/AN/A0.35 ± 0.04

Table R2. Euclidean distance between centroids of generated pain samples and real windows (ℓ2\ell_2-normalized; lower = closer). Labels are used only to select real windows. Reference row: real pain–neutral distance of each subject. Mean ± std over BioVid subjects.

Our samples lie close to the subject's unobserved pain windows (0.16) and far from other subjects' (0.34). They move away from the starting neutral windows by about the real pain–neutral separation (0.31 vs. 0.35). Noise-initialized samples show no such structure. The entries therefore represent the subject's class, not only a refinement of its scores. Noise initialization also reduces WAR from 81.0% to 78.2% (Tables 6 and 9). Because the sampled cache provides these entries before the target caches have accumulated observations, its largest gain is at the start of adaptation (+6.6 WAR on the first video; Table 16). We have added this analysis to Sec. 4.3 (Table 8) and Appendix G.

"W-3 & Q-3: Tasks and evaluation units."

"A-3": These tasks and evaluation units follow the standard protocols of prior work on these datasets [2, 6, 7, 8] and are not our own choices. We have now stated them explicitly in Sec. 4.1 (Datasets):

  • BioVid: neutral (BL1) vs. the highest pain level (PA4) [2, 6]. Intermediate levels (PA1–PA3) are not used, and no classes are merged.
  • StressID: neutral vs. stress, using the binary stress labels provided with the dataset.
  • BAH: neutral vs. ambivalence/hesitancy (A/H) [7, 8]. Using the frame-level annotations, each video is divided into 16-frame segments whose frames all share the same label, so each segment has a single, unambiguous label.

Evaluation unit. Each video is processed online as a sequence of 16-frame windows, and EB-CaP makes one prediction per window. Since each window has one ground-truth label, WAR and macro F1 are computed at the window level over all windows of each target subject, and then averaged over the 10 target subjects (Sec. 4.1). All main results use 16-frame windows; 24- and 32-frame windows on BAH are evaluated in Appendix L (Table 18).

References

[6] Muhammad Osama Zeeshan, Muhammad Haseeb Aslam, Soufiane Belharbi, Alessandro Lameiras Koerich, Marco Pedersoli, Simon Bacon, and Eric Granger. Subject-based domain adaptation for facial expression recognition. In 2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG), pp. 1–10. IEEE, 2024.

[7] M. González-González, S. Belharbi, M. O. Zeeshan, M. Sharafi, M. H Aslam, M. Pedersoli, A. L. Koerich, S. L. Bacon, and E. Granger. BAH dataset for ambivalence/hesitancy recognition in videos for digital behavioural change. In ICLR, 2026a.

[8] M. González-González, S. Belharbi, M. O. Zeeshan, M. Sharafi, M. H. Aslam, L. Sia, N. Richet, M. Pedersoli, A. L. Koerich, S. L. Bacon, and E. Granger. Multimodal ambivalence/hesitancy recognition in videos for personalized digital health interventions. In Conference on Affective Computing and Intelligent Interaction (ACII), 2026.

AuthorsOpenReview ↗

Next Response

"W-4 & Q-4: Cache-update rules."

"A-4": We have specified the full update rule in Sec. 3.3 and in a new Algorithm 2 (Appendix B). Let Kc\mathcal{K}_c be the partition of class cc, with capacity mm, and D(⋅)D(\cdot) its diversity, with D=0D = 0 for a single-entry partition. For a candidate ztgt\mathbf{z}^{\mathrm{tgt}} that passes the entropy gate:

  • (i) Empty partition: the candidate is admitted directly.
  • (ii) Below capacity (0<∣Kc∣<m0 < |\mathcal{K}_c| < m): it is admitted if ΔD>0\Delta D > 0.
  • (iii) At capacity (∣Kc∣=m|\mathcal{K}_c| = m): it is compared with the stored entry of highest entropy, z⋆\mathbf{z}^{\star}. If its entropy is lower, the swap gain ΔD=D((Kc∖{z⋆})∪{ztgt})−D(Kc)\Delta D = D((\mathcal{K}_c \setminus \lbrace \mathbf{z}^{\star} \rbrace) \cup \lbrace \mathbf{z}^{\mathrm{tgt}} \rbrace) - D(\mathcal{K}_c) is computed, and z⋆\mathbf{z}^{\star} is replaced only if ΔD>0\Delta D > 0. Otherwise, the candidate is discarded.

The order is therefore: entropy gate, then entropy-based replacement, then diversity gate. The same rule applies to both target caches.

"W-5: Basic errors."

"A-5": CRG [9] was accepted both at ICME 2025 and at the ICLR 2025 Workshop on Foundation Models in the Wild; we now cite the archival ICME 2025 version consistently in Table 1 and the bibliography. We have revised Sec. 4.2 to state that all methods achieve perfect or near-perfect F1 on BioVid Sub-9 (e.g., TPT: 99.0). We have also re-checked all tables, references and the text for consistency, corrected the bold markings in the subject-wise tables (Tables 2–4 and 10–12), and corrected the venues and years in the bibliography.

References

[9] Haotian Zhai, Xinyu Chen, Can Zhang, Tianming Sha, and Ruirui Li. Mitigating cache noise in test-time adaptation for large vision-language models. In 2025 IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6. IEEE, 2025

Reviewer_udTpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EB-CaP, a cache-based test-time adaptation (TTA) method for video facial expression recognition. For each incoming unlabeled target video, it runs class-conditional SGLD chains initialized from the current video embedding, where the energy is defined by the negative similarity between the visual embedding and the frozen CLIP class text embeddings, and stores the sampled embeddings in a personalized sampled cache. A positive cache and a negative cache retain reliable and uncertain target embeddings respectively, with updates gated by adaptive entropy and diversity. The final prediction combines the current CLIP scores with scores retrieved from the three caches, without updating model parameters at inference. On BioVid, StressID, and BAH, EB-CaP outperforms the compared optimization-based and cache-based TTA methods with lower memory overhead.

Critical strengths

1.The paper addresses the cold-start limitation of cache-based TTA with a simple and practical online prototype construction strategy. By adapting class prototypes directly from the current video without source data or parameter updates, the method enables immediate subject-specific adaptation and is well suited to online deployment.

2.The experiments are extensive and mostly well-controlled: three datasets, eight TTA baselines under one protocol, per-subject results, and complexity comparison. The temporally-augmented MCP comparison (Table 15) and the target-subject scaling study (Table 13) in the appendix address the main confounds, which strengthens the claim that the gains come from the adaptation strategy rather than from temporal modeling or subject choice.

Critical weaknesses

1.The methodological novelty of the proposed energy-based sampling is limited. The energy function is defined as Eθ(z,c)=−z⊤ecE_\theta(z,c)=-z^{\top} e_c with a fixed class text embedding ece_c, so the gradient term simply steers the target embedding toward the corresponding class embedding, with isotropic Gaussian noise added during SGLD. The resulting procedure is therefore close to noisy class-directed feature steering rather than learning a richer class-conditional energy landscape. The paper does not clearly establish that this formulation is fundamentally different from simpler prototype construction schemes.

  1. The empirical advantage of EB-CaP is not consistently strong. In Table 6, the source-prototype strategy achieves 81.5% WAR, slightly outperforming EB-CaP at 81.0%. On BAH (Table 3), the gain over CRG is also marginal, with only +0.8 WAR and +0.2 F1. In addition, the subject-wise results show noticeable variation across individuals. Without variance or significance reporting, the robustness of these relatively small gains is unclear.

  2. The prototype generation criterion is somewhat circular. Each class-conditioned SGLD chain is stopped once the generated embedding is classified as the conditioning class, and that embedding is then stored as a prototype of that class. As a result, class consistency is partly enforced by construction, which makes it unclear whether the generated embeddings capture meaningful subject-specific class structure beyond crossing the frozen CLIP decision boundary.

Questions for authors
  1. How does the sampled cache compare with simpler prototype constructions, such as directly using the class text embeddings ece_c as cache prototypes or using a fixed-ratio interpolation between the target embedding and ece_c? Given the simple linear energy function, it is unclear how much benefit is provided specifically by the SGLD sampling procedure.

  2. Are the reported results based on a single run? Please clarify the variability of EB-CaP across repeated runs, particularly given the stochastic SGLD sampling. Without variance or statistical significance information, it is difficult to assess the robustness of small differences such as 81.0% vs. 81.5% WAR in Table 6 and the gains on BAH.

  3. The SGLD chain is stopped once the generated embedding is classified as its conditioning class c, after which it is stored as a prototype for class c. To what extent does this stopping rule simply enforce class consistency by construction? In particular, how do the authors verify that the resulting embeddings represent meaningful subject-specific class prototypes rather than features that have merely crossed the frozen CLIP decision boundary?

  4. Figure 1 is difficult to read, and several details of the compared cache construction strategies are not clearly visible.

  5. The notation in Figure 2 is not fully consistent with the symbols defined in Section 3, which makes the proposed pipeline difficult to understand. Please clarify the correspondence between the figure and the main text.

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank Reviewer udTp for the careful review and for recognizing EB-CaP as a simple and practical solution to the cold-start limitation of cache-based TTA, without source data or parameter updates. We are also glad the reviewer found the experiments extensive and well controlled (Tables 17 and 20). We address each comment below; "W" indicates a weakness, "Q" a question, and "A" our answer.

"W-1 & Q-1: Energy-based sampling vs. simpler prototype constructions."

"A-1": The energy is intentionally simple: it comes from the frozen model, adds no parameters, and keeps sampling fast at test time (Sec. 3.1). The fundamental difference from simpler prototype schemes is in how the entries are constructed, not in the energy itself. Simpler schemes compute one fixed entry per class, either the class embedding ec\mathbf{e}_c or a fixed interpolation between ztgt\mathbf{z}^{\mathrm{tgt}} and ec\mathbf{e}_c, with the same displacement for every input. EB-CaP instead runs several stochastic SGLD chains per class, each starting from the subject's embedding:

  • Adaptive displacement: each chain stops as soon as its sample is predicted as class cc, so each input moves only as far as needed, preserving as much subject information as possible. A fixed interpolation moves every input by the same amount.
  • A set of entries instead of a single point: noise makes chains stop at different points, so each class is represented by several distinct entries describing the subject's local class region.

We have clarified both properties in Sec. 3.1.

(i) Simpler constructions. We replaced the sampled cache with simpler constructions, keeping all other components fixed (Table R1).

ConstructionSuccess (%)Avg. stepsDiversityWAR (%)
Text embeddings ec\mathbf{e}_cN/AN/A0.0078.6
Fixed interpolation (λ=0.5\lambda = 0.5)N/AN/A0.0079.5
Noise only (no gradient)58.011.20.3478.0
Deterministic (σ=0\sigma = 0)76.510.00.0678.8
SGLD (EB-CaP)94.26.70.5581.0

Table R1. Sampled-cache constructions on BioVid. Noise only and deterministic keep the same initialization and stopping rule. Success: chains reaching their class within 20 steps. Diversity: mean pairwise Euclidean distance between same-class entries of a window.

Single fixed entries are not enough (78.6% and 79.5%). Without guidance, only 58.0% of chains reach their class. Without noise, entries nearly collapse (0.06). SGLD combines both and gives the highest success rate, diversity and WAR. This diversity is spread around a subject-specific center: the centroid of the entries lies close to the subject's own real class windows and far from other subjects' (Sec. 4.3, Table 8).

(ii) Richer energy. We also evaluated nonlinear energies at earlier layers, where the gradient is no longer constant, tuning the step size and noise separately for each layer (Table R2).

Energy (sampling space)Time/window (ms)Add. memory (MB)WAR (%)
Nonlinear, before last temporal layer4.37076.4
Nonlinear, ViT block 11149.93,27476.7
Nonlinear, ViT block 6843.811,16877.1
Linear (EB-CaP)1.21081.0

Table R2. Linear vs. nonlinear energies on BioVid. Time per window; additional memory for a batch of 16 windows.

These energies are 3.6× to over 700× slower, need up to three orders of magnitude more memory, and reduce WAR by 3.9–4.6 points, because each step must backpropagate through the remaining layers. A richer energy therefore does not help in this setting. We have added these results to Sec. 4.3 (Table 7) and Appendix F (Table 14).

AuthorsOpenReview ↗

Next Response

"W-2 & Q-2: Empirical advantage, variance, and significance."

"A-2": The small gaps reflect a stricter setting or the default window size on BAH, and the gains are significant and consistent across subjects.

(i) Source prototypes (Table 9). The prototype-based variant stores class prototypes built from labeled source data at test time, so it is not directly comparable. EB-CaP builds its cache only from the unlabeled target video. The 0.5-point gap is the cost of removing source data, which matters when facial videos cannot be retained or shared. Without its source prototypes, TTA-CaP falls below EB-CaP on all three datasets (Table 1).

(ii) BAH. BAH is a recent in-the-wild dataset [1,2] with many more participants than BioVid (300 vs. 87) but fewer videos per participant (7 vs. 20), and ambivalence and hesitancy unfold over longer temporal contexts. We therefore evaluated larger windows for EB-CaP and CRG (Table R3).

Window sizeEB-CaPCRG
16 (default)68.9 ± 0.2068.1 ± 0.11
2469.4 ± 0.1468.5 ± 0.19
3270.6 ± 0.1669.1 ± 0.19

Table R3. WAR (%) on BAH (10 target subjects, mean ± std over 3 seeds).

EB-CaP benefits more from longer context than CRG (+1.7 vs. +1.0 points), and its advantage grows from 0.8 to 1.5 points, so the modest gain on BAH mainly reflects the short 16-frame windows (Appendix L, Table 18). For the camera-ready version, we will report all baselines on BAH with 32-frame windows.

(iii) Variance and significance. All reported results were already averaged over 3 random seeds, which change the SGLD noise and thus the sampled entries. We have now added the standard deviations to the subject-wise tables (Tables 2–4 and 10–12). They are small, showing that the results are stable despite the stochastic sampling. A paired Wilcoxon signed-rank test over subjects confirms that EB-CaP significantly outperforms CRG on all three datasets (pp = 0.008, 0.003 and 0.030 on BioVid, StressID and BAH; Sec. 4.2).

(iv) Subject variation. EB-CaP improves WAR over CRG for 7/10 subjects in BioVid (the other 3 are ties, with no losses), 9/10 in StressID and 9/10 in BAH (Tables 10–12). The variation mainly reflects how visibly each subject expresses pain (Appendix I, Fig. 4, Table 15):

  • Sub-4 shows clear facial changes: EB-CaP raises WAR from 60.0% (source model) to 95.4%, +22.4 points over CRG.
  • Sub-3 shows only subtle changes, mainly eye closure, so her classes are not separable in the source feature space, and all methods, including EB-CaP, remain at 50.0%.

The results also do not depend on the choice of target subjects: on three random target sets stratified by age and sex, with the source model retrained each time, EB-CaP obtains 79.7 ± 1.3% WAR and 77.5 ± 1.1% F1 (Appendix K).

"W-3 & Q-3: Circular prototype generation criterion."

"A-3": The stopping rule only ensures class consistency; subject information comes from the initialization. Unlike standard SGLD, which starts from random noise, each chain intentionally starts from the subject's own embedding ztgt\mathbf{z}^{\mathrm{tgt}}, so the entries keep the subject's characteristics while moving toward class cc. To test whether this yields subject-specific class structure beyond crossing the decision boundary, we generated pain samples from each BioVid subject's neutral windows with both initializations and the same energy, step size, noise and stopping rule (Table R4).

InitializationOwn pain ↓Other subjects' pain ↑Own neutral ↑
Noise-initialized SGLD0.38 ± 0.040.40 ± 0.030.42 ± 0.05
EB-CaP (subject embedding)0.16 ± 0.030.34 ± 0.040.31 ± 0.04
Real pain windows (GT reference)N/AN/A0.35 ± 0.04

Table R4. Euclidean distance between centroids of generated pain samples and real windows (ℓ2\ell_2-normalized; lower = closer). Labels are used only to select real windows. Reference row: real pain–neutral distance of each subject. Mean ± std over BioVid subjects.

Our samples lie close to the subject's own pain windows (0.16) and far from other subjects' (0.34). They move away from their starting neutral windows by about the real pain–neutral separation (0.31 vs. 0.35), so they stay within the subject's own class region. Noise-initialized samples satisfy the same stopping rule but show no subject preference. The stopping rule therefore does not determine the entries; the initialization does. This also matters for recognition: noise initialization reduces WAR from 81.0% to 78.2% (Tables 6 and 9).

We also tested sampling at earlier encoder layers, where more subject-specific detail may reside, using a nonlinear energy while keeping the entries as output embeddings. This was 3.6× to over 700× slower and reduced WAR by 3.9–4.6 points (Sec. 4.3, Table 7; see also A-1). We have added this analysis to Sec. 4.3 (Table 8) and Appendix G.

AuthorsOpenReview ↗

Next Response

"Q-4 & Q-5: Figure 1 readability and Figure 2 notation."

"A-4": We redesigned Figure 1 with stacked panels, larger fonts, a title for each strategy, and its main advantages and limitations summarized above each panel. We also updated Figure 2 to match this style and to use the notation of Section 3 (ETE_{\mathrm{T}}, EIE_{\mathrm{I}}, EVE_{\mathrm{V}}, ztgt\mathbf{z}^{\mathrm{tgt}}, ec\mathbf{e}_c, Ks\mathcal{K}^{s}, Kp\mathcal{K}^{p}, Kn\mathcal{K}^{n}, sfuses^{\mathrm{fuse}}). It now shows the entropy and diversity gates (Sec. 3.3) and the SGLD initialization from ztgt\mathbf{z}^{\mathrm{tgt}}. These changes are included in the revised manuscript.

References

[1] M. González-González, S. Belharbi, M. O. Zeeshan, M. Sharafi, M. H Aslam, M. Pedersoli, A. L. Koerich, S. L. Bacon, and E. Granger. BAH dataset for ambivalence/hesitancy recognition in videos for digital behavioural change. In ICLR, 2026a.

[2] M. González-González, S. Belharbi, M. O. Zeeshan, M. Sharafi, M. H. Aslam, L. Sia, N. Richet, M. Pedersoli, A. L. Koerich, S. L. Bacon, and E. Granger. Multimodal ambivalence/hesitancy recognition in videos for personalized digital health interventions. In Conference on Affective Computing and Intelligent Interaction (ACII), 2026.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response to all reviewers

We sincerely thank all reviewers for their thoughtful and constructive feedback, which has helped us clarify and strengthen the paper. We appreciate the recognition of EB-CaP as a simple and practical solution to the cold-start limitation of cache-based TTA without source data or parameter updates (udTp, 1d8Y), its clear motivation and practical relevance for cross-subject shifts in video FER (Yv99), and the extensive, well-controlled experiments (udTp, 1d8Y).

Reviewers mainly asked us to clarify how energy-based sampling differs from simpler prototype constructions, show that the sampled entries are subject-specific, assess the impact of hyperparameter selection, compare more fully with TTA-CaP, report variance and statistical significance, and specify the task definitions and cache-update rules more precisely. We have addressed all comments point by point below, and the revised manuscript includes the corresponding clarifications, new analyses, and redesigned figures.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Thank you again for your time and thoughtful reviews. With only two days remaining in the discussion period, we would greatly appreciate your feedback on whether our responses and the revised manuscript have addressed your concerns, and whether any points require further clarification. We would be happy to provide any additional details.

AuthorsOpenReview ↗

Summary of Rebuttal and Revisions

Dear AC, SAC, PC, and Reviewers,

Thank you for handling our submission and for the constructive reviews. We briefly summarize the reviewers' main comments, our responses, and the corresponding changes to the revised manuscript.

Strengths recognized by the reviewers

  • EB-CaP as a simple and practical solution to the cold-start limitation of cache-based TTA, without source data or parameter updates (udTp, 1d8Y);
  • The promise of generating class-conditioned cache entries from the current input, combined with target caches accumulated over time (1d8Y);
  • The clear motivation and practical relevance for cross-subject shifts in online video FER (Yv99);
  • The extensive and well-controlled experiments (udTp, 1d8Y).

Main comments and our responses

  • Energy-based sampling vs. simpler constructions (udTp, Yv99). We clarified that the sampler adapts the displacement to each input and yields several diverse entries per class. Replacing it with class text embeddings or fixed interpolation reduces WAR from 81.0% to 78.6–79.5%, and nonlinear energies at earlier layers are 3.6× to over 700× slower and less accurate (Sec. 3.1; Sec. 4.3, Table 7; Appendix F).
  • Subject specificity of sampled entries (udTp, 1d8Y, Yv99). Validated against each subject's real class windows, never used during sampling, the entries lie close to the subject's own class and far from other subjects', unlike noise-initialized samples, showing that personalization comes from initialization at the subject's embedding (Sec. 4.3, Table 8; Appendix G).
  • Configuration selection (1d8Y, Yv99). Reselecting all settings on independent validation subjects yields the same prompt and cache capacities and changes WAR by only 0.2 points (Appendix E).
  • Comparison with TTA-CaP (Yv99). EB-CaP stays within 0.5 points of TTA-CaP with source prototypes while using no source data and 32% less memory, and outperforms it when source data are removed (Tables 1–5).
  • Variance, significance and subject variation (udTp). Results are averaged over 3 seeds with small standard deviations, gains over CRG are significant on all datasets (one-sided Wilcoxon), and results hold on randomly selected target subjects stratified by age and sex (Sec. 4.2; Appendices I and K).
  • Clarity (1d8Y, Yv99, udTp). We specified the task definitions and window-level evaluation, the full cache-update rule and scoring formulas, the encoders used by all baselines, and corrected formatting errors.

Manuscript changes

  • Sec. 3.1 and 3.3: clarified the roles of initialization and energy, and the full cache-update rule (new Algorithm 2, Appendix B).
  • Sec. 4.1: stated the standard task protocols and evaluation units.
  • Sec. 4.2: added TTA-CaP to all comparisons and complexity results, standard deviations over seeds, and significance tests.
  • Sec. 4.3: added the nonlinear-energy comparison (Table 7) and subject-specificity analysis (Table 8).
  • Appendices: configuration reselection (E), sampler comparison (F), subject specificity (G), success and failure cases (I), random target subjects (K), larger temporal windows on BAH (L), and temporally augmented baselines (N).
  • Redesigned Figures 1 and 2 for readability and consistency with the notation of Sec. 3.

We thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.