acceptodds
Under review as a conference paper at ICLR 2027

ZORA: Zeroth-Order Reliability-Calibrated Adaptation with Shift-Sensitive Updates for Multimodal Test-Time Shifts

Abstract

Audio-visual continual test-time adaptation (AV-CTTA) enables pretrained multimodal models to adapt online to non-stationary target streams. However, most existing AV-CTTA methods rely on backpropagation-based updates, limiting their deployment on edge platforms with quantized backbones, inference-only engines, or strict memory and latency constraints. Zeroth-order optimization (ZOO) provides a forward-only alternative, but its direct application to AV-CTTA faces two challenges: zeroth-order queries must explore a high-dimensional multimodal parameter space without shift awareness, and entropy minimization may reinforce incorrect predictions that exhibit high confidence and low entropy under online multimodal shifts. To address these limitations, we introduce Zeroth-Order Reliability-Calibrated Adaptation (ZORA), a forward-only AV-CTTA framework that determines both where ZOO should update and which learning signals it should trust. ZORA incorporates Shift-Sensitive Layer Selection (SSLS) to restrict zeroth-order updates to shift-sensitive normalization layers in the audio and visual branches. It further employs Bias-Aware Reliability Calibration (BARC) to assign signed entropy weights based on prediction bias and confidence, while aligning target-window feature statistics with cached source-reference statistics. Experiments on Kinetics50-C and VGGSound-C demonstrate that ZORA consistently outperforms representative CTTA, multimodal adaptation, and forward-only baselines across multiple continual audio-visual shift protocols. Experiments with quantized CAV-MAE models and efficiency analyses further support resource-constrained forward-only deployment.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Audio-Visual Continual Test-Time Adaptation, Zeroth-Order Optimization, Forward-Only Adaptation, Multimodal Learning, Reliability Calibration, Distribution Shift, Edge Deployment.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_v96COpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper is about adapting audio-visual models at test time when backpropagation cannot be used, like on edge devices or quantized models. The authors mention that vanilla zeroth-order optimization (ZOO) works badly here for two reasons. It perturbs too many parameters, and entropy minimization trains on predictions that can be very confident but are wrong. Their method, ZORA, has two parts. The first part on SSLS, picks the top 3 LayerNorm layers (based on their experimental setup section) in each of the audio and video encoders. It ranks them by how well a 2-means clustering separates source features from early target features, and only those layers get updated. The second part on BARC, looks at a window of recent predictions and counts how often each class is predicted. A confident sample whose predicted class shows up a lot gets a negative entropy weight. A confident sample whose class shows up less gets a positive weight. Everything else gets zero and a loss (L_align) that matches feature statistics to the source is added on top. They test on Kinetics50-C and VGGSound-C with two streams that they built, using 5 seeds and 12 baselines. They achieve about 1.9 to 2.5points over the best baseline on the interleaved stream and about 4 to 5points on the PairTrack-Async stream. There are also fake-quantized runs (W8A8, W4A4), a second architecture, ImageNet-C and sentiment transfer, and some stress tests for BARC under class imbalance.

Critical strengths
  1. Well-motivated problem with a thorough evaluation. Forward-only AV-CTTA is relevant in practice, especially for on-edge devices.
  2. The authors provide multi-seed statistics, many protocols, a second architecture, SSLS robustness studies, and a stress test of where BARC fails (Appendix D.4).
  3. The results are quite promising for further application if they hold up.
  4. SSLS and BARC are modular components. Table 15 suggests both help even with exact gradients. This gives the work value for future TTA work beyond the specific ZO setting, if validated more rigorously.
Critical weaknesses
  1. Some of the numbers are confusing. The alignment loss uses statistics over a window of samples (Eq. 6), but it is evaluated at the two perturbed parameter points (Eq. 8). If the window features are cached, the term is the same at both points and cancels out of the finite difference. Then it does nothing, even though the ablation gives "DA" about +3.9 points. If the features are recomputed, the cost is more than 2 forward passes. Also, Fig. 5 shows accuracy flat at about 55.3 to 55.5 for all λ and γ, including 0. Based on my understanding, when λ and γ are 0, it should correspond to the ablation entry in Table 2 without the signed weights and alignment. But in Table 2, it shows that each add about 4 points.
  2. The comparison and the edge claims are not convincing. Some of the baselines could be broken in Table 11. DASP is almost the same as the source on ICS and PACS, with values on a 0.5 grid. PTA, I believe is built for prediction bias, drops below the source on PACS (35.23 and 4.83). The authors also did not mention what ZORA's hyperparameters is tuned on and if it is tuned on the same streams it is tested on. Also, the authors never defined naïve ZOO, so I am not sure what the difference is between "Entropy-only" which achieves 53.80 in Table 9, and "naive ZOO" is 44.31 in Table 2. This could suggest that their naïve ZOO is a weak baseline.
Questions for authors
  1. Alignment loss. How is L_align computed inside Eq. 8? Are window features cached or recomputed under Θ±δu? If cached, does it cancel in the finite difference? If recomputed, what is the true forward count?
  2. On Fig. 5 vs. Table 2, could you report the full model with λ=0 and with γ=0 (mean±std over 5 seeds). How is it interpreted with the +3.9 (DA) and +4.0 (SR) ablation gains?
  3. For Naïve ZOO and Table 9, could you define naïve ZOO in Table 2 (parameters, loss, δ, η, perturbation distribution)? Why does entropy-only reach 53.80 in Table 9?
  4. For the baselines, why is DASP approximately equal to source on ICS/PACS, with 0.5-granular values, while PTA and READ drop below source? What hyperparameter search did each baseline vs. ZORA receive, and on which data were ZORA's hyperparameters chosen on?
2 comments

W1. DA consistency. Cached history and recomputed current features make DA perturbation-dependent (Q1). Figure 5 is reproducible under its original seed (Q2).

W2. Baseline reliability. Matched-optimizer ablations and update diagnostics are in Q3–Q4. Efficiency results cover forward-only evaluation and simulated quantization, not physical edge deployment.

Q1. DA cache and scoring. DA uses a cached-history alignment surrogate. Historical pre-update feature sums, squared sums and counts stay fixed under (\Theta_t\pm\delta u); only current features (z_t^\pm) are recomputed. Thus (\mu_t^\pm=(S_{\mathrm{hist}}+z_t^\pm)/(n_{\mathrm{hist}}+1)), so DA need not cancel. Historical features may come from earlier parameter states and are not re-encoded; unperturbed current statistics enter the cache after adaptation. Source references and initial BARC weights stay fixed within the step. One initial, two perturbed and one post-update pass give 2 ZOO queries / 4 evaluations (Table 12). We will clarify this surrogate, gradient normalization and execution order in Eqs. (6)/(8) and Algorithm 1.

Q2. Figure 5 vs. Table 2. Table 2 uses cumulative ablations without SSLS (DA: +3.87/+4.04 pp); Figure 5 varies coefficients within full ZORA. Setting (\gamma=0) removes negative suppression but retains positive weighting.

Over five seeds (20260720–20260724), Full gives 55.64 ± 0.67% / 22.06 ± 0.45% (K50-C/VGG-C), versus 54.64 ± 0.21% / 21.66 ± 0.15% for (\lambda=0) and 54.45 ± 0.21% / 21.44 ± 0.23% for (\gamma=0). These are marginal rather than cumulative effects.

Figure 5 used seed 42, independently reproduced with identical configurations and scoring. Table X reports its VGG-C results separately: Full exceeds (\lambda=0) in 4/5 and (\gamma=0) in 5/5 additional seeds. Seed 42 is excluded from the five-seed mean ± std. The numerical gap reflects seed variation, not inconsistent implementations.

Q3. Naïve ZOO and matched-budget attribution. Original Table 2 Naïve ZOO uses raw entropy, All-Norm updates, (\delta=10^{-3},\eta=10^{-3},q=10) Rademacher directions (20 perturbation queries) and global gradient normalization. Table 9's “Entropy-only” retains SSLS+DA on non-IID streams, so its 53.80% and Table 2's 44.31% are not directly comparable. We instead ran five-seed optimizer-matched ablations with (\delta=10^{-3},\eta=10^{-2},q=1), Rademacher directions, global L2 normalization, two queries, one update and post-update scoring:

ConfigurationK50-C (%)VGG-C (%)
A. Entropy-only ZOO, All-Norm46.00 ± 0.1212.03 ± 0.15
B. RF, All-Norm46.50 ± 0.3213.23 ± 0.14
C. RF + SR, All-Norm50.50 ± 0.2317.49 ± 0.37
D. RF + SR + DA, All-Norm54.37 ± 0.2921.53 ± 0.40
E. RF + DA + SSLS54.51 ± 0.3521.08 ± 0.15
F. RF + SR + SSLS (no DA)54.64 ± 0.2121.66 ± 0.15
G. Full ZORA55.64 ± 0.6722.06 ± 0.45

All-Norm updates AV normalization parameters; DA aligns fixed Top-3 layers even without SSLS-restricted updates. Full exceeds matched entropy by 9.64/10.03 pp; SR (G−E) adds 1.13/0.98 pp and DA (G−F) 1.00/0.40 pp. We will distinguish the original unmatched Table 2 row and relabel Table 9's entropy variant.

Q4. Baseline diagnostics. DASP: We follow official method settings. Since its batch-dependent redundancy statistics cannot be computed meaningfully at B=1, we replace the batch input with a causal W=24 window (current AV pair plus up to 23 past pairs); the adaptation procedure is otherwise retained. Logged K50-C/VGG-C runs show 2,553/2,564 updates. The appendix's 0.5-pp entries came from single-seed per-condition records; corrected five-seed means (Tables 16–19) preserve aggregate results.

PTA (K50-C): With the official loss and a causal FIFO-8 extension, both partitions are populated in 76.46% of steps and 2,241 updates (76.33%) occur. PTA scores 36.24% vs. paired Source 46.01%, correcting 183 errors but introducing 470. READ updates in 99.90%/99.86% of steps; on VGG-C it corrects 17 errors but introduces 215, while dominant-class share rises 16.38%→67.30%. Both show active but potentially harmful adaptation; failures are protocol-specific.

Hyperparameter selection. ZORA's (K=3,W=24,\delta=10^{-3},\eta=10^{-2},\lambda=\gamma=0.5) were selected from 10 configurations on clip-disjoint held-out development streams and fixed before testing; no test labels were used. Baselines follow official objectives and method-specific settings, with necessary AV interfaces in Appendix C (Adam (10^{-4}) for gradient-based methods; fused-prediction losses/AV LayerNorm where applicable). Batch-dependent statistics incompatible with B=1 use causal windows (DASP W=24; PTA FIFO-8). No extra baseline tuning was performed.

AuthorsOpenReview ↗

Additional Evidence: Corrected DASP Five-Seed Per-Condition Results (Tables 16–19)

Additional Evidence: Corrected DASP Five-Seed Per-Condition Results (Tables 16–19)

The original appendix inadvertently used single-seed DASP entries for some per-condition accuracies. We have recomputed the mean across five seeds for each condition. The PACS single-seed values followed 0.5-pp increments because each condition contains 200 examples. The corrected results are reported below, with all aggregate means unchanged. Values represent accuracy (%), averaged over five seeds. A and V denote audio and visual corruptions.

Table 16. Kinetics50-C / ICS

Conditions 1–7

MethodV.VA.V.V.A.V.
DASP49.8651.1172.6650.3369.3866.5860.17

Conditions 8–14

MethodV.AVVVAV
DASP70.3266.5869.3860.9565.9568.3046.74

Conditions 15–21

MethodV.A.V.V.AV.VAvg.
DASP78.4468.7652.5263.6170.6364.8667.2163.54

Avg-V: 61.39%; Avg-A: 68.92%.

Table 17. VGGSound-C / ICS

Conditions 1–7

MethodV.VA.V.V.A.V.
DASP46.0148.6714.9146.1757.4018.6751.32

Conditions 8–14

MethodV.AVVVAV
DASP62.0922.7355.7048.8251.3221.7952.88

Conditions 15–21

MethodV.A.V.V.AV.VAvg.
DASP63.9721.3248.6756.3126.3256.0155.2344.11

Avg-V: 53.37%; Avg-A: 20.96%.

Table 18. Kinetics50-C / PairTrack-Async

Conditions 1–5

MethodA./V.A./VA./V.A./V.A./V.
DASP35.8034.3032.5041.6039.00

Conditions 6–10

MethodA/V.A/VA/VA/VA/V
DASP58.8053.2044.8037.3020.00

Conditions 11–15

MethodA./V.A./V.A./V.A/V.A/VAvg.
DASP67.4039.0047.3060.5054.5044.40

Table 19. VGGSound-C / PairTrack-Async

Conditions 1–5

MethodA./V.A./VA./V.A./V.A./V.
DASP5.707.505.707.906.70

Conditions 6–10

MethodA/V.A/VA/VA/VA/V
DASP22.6017.3016.806.303.40

Conditions 11–15

MethodA./V.A./V.A./V.A/V.A/VAvg.
DASP24.5012.9015.1019.5017.6012.63

These corrected rows will replace the DASP per-condition entries in the revised appendix. Aggregate mean ± standard deviation remains available in Table 11.


. Seed-wise Comparison of Figure 5 and Additional Ablations on VGGSound-C PACS (Accuracy, %)

Random Seed / RecordFull ZORAλ=0\lambda=0γ=0\gamma=0
s0s_0 (Figure 5, seed 42)22.5322.6722.65
s1s_1 (20260720)21.3521.4921.14
s2s_2 (20260721)22.0021.6421.38
s3s_3 (20260722)22.0921.8221.39
s4s_4 (20260723)22.3221.5521.52
s5s_5 (20260724)22.5421.8021.77
Mean ± Std (s1s_1–s5s_5)22.06 ± 0.4521.66 ± 0.1521.44 ± 0.23

Note. The independent Figure 5 seed (s0=42s_0=42) is excluded from the five-seed mean and sample standard deviation, which are computed over s1s_1–s5s_5 only.

Reviewer_J11yOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a backpropagation-free continual test-time adaptation design for handling domain shifts after deployment in audio-visual models. It employs zeroth-order optimization to achieve backpropagation-free updates with selectively updating a small number of shift-sensitive normalization layers to reduce computational costs and noisy updates. To improve the reliability of the adaptation objective, they introduce bias-aware reliability calibration with signed entropy reweighting to mitigate the limitations of conventional entropy minimization. Besides, they also integrate feature distribution alignment loss for additional regularization. The proposed method outperforms 12 evaluated baselines under the single-sample adaptation setting across interleaved corruption streams and paired-asynchronous corruption streams.

Critical strengths

Practical and effective forward-only audio-visual test-time adaptation.

The proposed method provides a practical solution for single-sample audio-visual test-time adaptation in resource-constrained conditions to handle the mixed domain shifts in inference time.

Critical weaknesses

W.1 Insufficiently justified motivation for the entropy-based objective.

The Introduction identifies unreliable entropy minimization as a key bottleneck motivating the bias-aware reliability calibration design. However, ZOO does not necessarily require entropy minimization, as other unsupervised objectives, such as pseudo-label-based or consistency-based objectives, could also be considered. Moreover, existing TTA methods (e.g., PTA, SAR, FOA, EATA-C, and E-BATS) have already established some strategies or additional learning signals to addressed the limitations of entropy minimization alone. The specific research gap motivating BARC remains unclear for me without clearly explaining why these existing approaches are insufficient for forward-only continual audio-visual adaptation.


W.2 Insufficient motivation for the single-sample adaptation setting.

The paper evaluates all methods under a single-sample streaming setting, but does not clearly motivate why this setting is necessary or central to the research problem or bottelneck. Since some existing methods rely on batch-level adaptaiton, evaluating them with batch size 1 may limit their effectiveness and potentially exaggerate the improvements attributed to SSLS and BARC targeting on current listed bottelnecks. This concern is also reflected in Appendix Figure 8, where the performance gap becomes smaller as batch size increases.

Questions for authors

Q1. Applicability to different normalization layer types.

Is ZORA specifically designed for LayerNorm layers, or can it also be applied to architectures using BatchNorm?


Q2. Token-wise layer shift sensitivity.

What motivates the token-wise used in SSLS instead of comparing sample-level representations with pooling? Could the authors clarify what might be the additional benefits?


Q3. Reliability of single-direction ZOO updates.

How does ZORA ensure reliable updates using only one randomly sampled perturbation direction per test sample?


Q4. Selection of Top-K Layers.

How was K=3K=3 chosen for SSLS?


Q5. Definition of prediction bias.

Why does Equation (3) subtract the average predicted-class frequency when calculating the bias score? If all samples in a window are predicted as the same class, wouldn't all BiB_i values become zero, even though predictions are completely concentrated on one class? How does this align with the intended purpose of detecting prediction bias?


Q6. Contribution of bias-aware signed entropy reweighting.

Table 2 provides cumulative ablations, but it remains unclear how much of the improvement comes from bias-aware signed entropy reweighting rather than distribution alignment (DA), which has already been explored in works such as FOA and E-BATS. Could the authors additionally report the performance of naïve ZOO + DA and naïve ZOO + DA + SSLS to better isolate the effectiveness of the proposed bias-aware reweighting?


Q7. Attribution of BARC's error reduction.

Figure 4 shows that BARC reduces wrong high-confidence predictions, but it is unclear how much of this reduction comes from signed reweighting (SR) versus feature distribution alignment (DA). Could the authors provide a component-level analysis to clarify their respective contributions?

5 comments
AuthorsOpenReview ↗

Response to Reviewer R2

W1. Motivation and novelty of BARC. ZOO need not minimize entropy; unreliable objectives can misdirect limited-query updates under evolving AV shifts. PTA already uses batch-wise prediction bias/confidence partitions, signed entropy and attention alignment between reliable/unreliable subsets via AGA. We do not claim these ingredients individually. BARC instead uses causal prediction-history risk and aligns current features with cached clean-source statistics in a forward-only finite-difference objective. This differs in the reference information, alignment target and update mechanism; SAR/EATA-C use distinct gradient/disagreement signals. Offline task-gradient cosine (supervised gradients only for diagnosis) is 0.696 (entropy), 0.770 (signed entropy), 0.811 (entropy+DA), 0.895 (full BARC); normalized ZOO error changes 1.000→1.016. Thus BARC improves objective alignment, not estimation accuracy.

W2. Single-sample setting and fairness. Our causal AV stream processes one new sample per step without waiting for future samples; historical observations remain available. In a separate paired K50-C diagnostic, PTA's official loss with a FIFO-8 history yields both partitions in 76.46% of steps and 2,241 updates (76.33%). This verifies active adaptation in that streaming extension, rather than the original five-seed aggregate. Appendix Fig. 8 shows ZORA still leads at B=4,8,16,32, although gaps narrow. We claim causal online processing, not measured end-to-end latency gains.

Q1. BatchNorm applicability. ZORA is not restricted to LayerNorm: SSLS scores feature shifts and BARC specifies an objective independently of the normalization type. We trained an OGM-GE-style dual-ResNet-18 on source data and adapted BN affine parameters, freezing BN running statistics. On this BN backbone, Source obtains 44.47% accuracy / 47.15% Macro-F1, random-BN selection obtains 53.23% / 52.99%, and BN-ZORA reaches 55.64±0.22% / 55.44±0.14%. This supports BN applicability.

Q2. Why token-wise LSS? Spatial/temporal corruption can affect tokens unevenly, and sample pooling may suppress localized shifts. We compared token-wise LSS against sample-level mean pooling with the same two-means scoring, Top-(K) selection, and ZOO settings. Token-wise yields 55.64/22.06% versus 54.59/21.02% on K50-C/VGG-C (+1.05/+1.04 pp).

Q3. One ZOO direction. Single-step direction accuracy is not guaranteed; SSLS restricts update space and BARC improves objective alignment (cosine 0.696→0.895). Appendix Table 13 compares 1/2/3/5 sequential within-sample ZOO updates (2/4/6/10 perturbation queries), yielding 55.57/55.57/55.63/55.53%. Gains are small relative to extra queries, supporting a practical accuracy–budget choice; this does not directly measure estimator variance.

Q4. Why K=3? The final K=3 was selected within a 10-configuration joint hyperparameter search on clip-disjoint held-out development streams, then fixed before testing. With one joint direction, changing K does not change the two ZOO queries; it changes update-space size and sensitive-layer coverage. LSS selects audio/visual layers independently. Fig. 3/Table 7 favor LSS over random/simple scoring (+1.09/~+0.92 pp), but do not establish theoretical optimality or a controlled K sweep.

Q5. Bias-score centering. Eq. (3) measures relative overrepresentation within a window. Subtracting the window-average frequency preserves the ranking used for quantile-based calibration. We agree that with complete single-class collapse, all (B_i=0), so this score alone cannot detect absolute collapse; we will state this limitation explicitly. No fully collapsed window occurred in the evaluated streams. Among highly concentrated windows (≥75% same-class predictions), accuracy remained 77.29/81.07% on K50-C/VGG-C PACS, so concentration alone is not a reliable error label.

Q6. Independent SR contribution. Requested Naïve ZOO+DA and +DA+SSLS obtain 45.46/13.02% and 46.36/13.82%, versus Naïve 44.31/11.96%. In separate five-seed optimizer-matched ablations, adding SR to RF+DA+SSLS improves 54.51/21.08%→55.64/22.06% (+1.13/+0.98 pp). Without SSLS, RF+DA→RF+SR+DA yields 53.44/20.59%→54.37/21.53% (+0.93/+0.94 pp); without DA, SR adds +4.00/+4.26 pp. Thus SR contributes beyond DA, with context-dependent gains.

Q7. Attribution of Figure 4. We measured the fraction of wrong high-confidence, low-entropy predictions (WHLE). Adding DA to RF reduces WHLE from 11.350/12.943% to 4.316/6.875% on K50-C/VGG-C. Keeping RF and DA fixed, adding SR further reduces it to 3.316/5.841%, while improving accuracy by 0.93/0.94 pp. Thus DA accounts for the larger share of WHLE reduction; SR provides an additional, separately measured benefit. We will qualify Figure 4 accordingly.

Thank you for the detailed response and additional experiments. They addressed several of my questions. I still have two main concerns which may stop me from raising the score.

  1. Motivation: I am still unclear about the methodological gap that motivates ZORA. The token-wise clustering-purity score in Eq. (1) is mathematically equivalent to the layer-selection score in ZOTTA, where the main difference seems to measure within each branch. Similarly, PTA already uses prediction bias and confidence for signed entropy weighting, including a history queue for small batches, where I could see the implementation difference of entropy weighting compared between ZORA and PTA, but I would like to understand which specific limitation of the prior methods like PTA the proposed changes want to address on the entropy handling.

  2. Conflict claim of BARC implementation: I remain concerned about the claim that BARC suppresses confidently wrong predictions under shift. Suppose a corruption causes the model to confidently predict the same incorrect class for 13 of the 24 samples in a window. These 13 samples share the highest bias score and likely wise also determine the median. Since Eq. (4) requires a score strictly above the median for a negative weight, none of the wrongly predicted classes in this case can receive the negative one. This can happen well before complete collapse, which seems conflicts with the claim and the problem that this method is trying to solve.

Thank you for the follow-up. We agree that our previous response insufficiently differentiated ZORA from ZOTTA/PTA and addressed only complete prediction collapse.

  1. Motivation and methodological distinction.

We acknowledge that the token-wise clustering-purity statistic in Eq. (1) is mathematically equivalent to ZOTTA's DRLS score. We will attribute this criterion to ZOTTA. In ZORA, it is applied separately to the audio and visual encoders to select a fixed Top-K update space in each branch while freezing the fusion module. PTA already uses prediction bias, confidence, signed entropy, and historical queues. We claim none of these ingredients alone as novel.

The specific gap we target is how to use imperfect prediction-derived risk signals when constructing a forward-only adaptation objective under heterogeneous audio-visual shifts. PTA's PDR partitions predictions by relative bias and assigns its biased subset a constant negative coefficient, −s-s, whereas BARC additionally gates on confidence and assigns a rank-dependent negative coefficient, −γQC(xt)QB−(xt)-\gamma Q_C(x_t)Q_B^-(x_t), when its negative-set condition is met. Low-confidence samples remain neutral, while selected biased predictions receive rank-dependent coefficients rather than a constant one. Under global L2 normalization of the single-direction ZOO estimate, rank scaling modulates the relative entropy/alignment contribution to the finite difference, not necessarily the update magnitude.

BARC also uses cached clean-source feature statistics as the alignment reference at SSLS-selected audio/visual layers. Unlike PTA's AGA, which aligns attention distributions of target subsets defined by the reliability partition, this source anchor does not depend on identifying a reliable target-attention subset. We acknowledge ZOTTA's prior source-statistics alignment. Our contribution is the combined design of branch-specific update selection and prediction-risk-calibrated, source-anchored objective evaluation for continual AV adaptation with one incoming pair, a fixed historical cache, and two ZOO perturbation queries.

Controlled evidence supports the role of reweighting within ZORA: with RF, DA, and SSLS fixed, adding signed reweighting improves K50-C/VGG-C PACS accuracy from 54.51/21.08% to 55.64/22.06% (+1.13/+0.98 pp). Offline task-gradient cosine similarity increases from 0.696 for entropy to 0.895 for full BARC, whereas normalized ZOO estimation error does not improve (1.000 to 1.016). These results suggest improved objective–task alignment, not improved gradient estimation. They do not isolate an advantage over PTA's weighting rule, and we will delimit our claim accordingly.

  1. Median threshold and the 13-of-24 counterexample.

We agree that this counterexample is correct. If 13 of 24 predictions belong to the same class, their bias scores are equal and maximal. Both middle order statistics equal that maximum, so τb=median⁡(Bi)=Bmax⁡\tau_b=\operatorname{median}({B_i})=B_{\max}. Since Eq. (4) requires Bi>τbB_i>\tau_b for the negative subset, no sample receives a negative entropy weight, even if the majority predictions are confidently wrong. High-confidence majority samples can instead fall into the positive subset; their eventual weights depend on the rank-tie convention. Our previous response overlooked this strict-majority boundary.

This is a genuine limitation of median partitioning. Frequency indicates risk, not correctness: the rule cannot distinguish a genuinely prevalent class from an erroneously overpredicted class. Source alignment remains active but does not guarantee recovery.

Our five-seed contiguous-burst tests on K50-C PACS support a bounded interpretation: relative to Positive-only, Full BARC changes overall accuracy by −0.30 pp at W=24W=24 and +0.38 pp at W=48W=48, with burst-accuracy changes of 0.00/+0.16 pp. Hence, we do not claim universal benefits from negative weighting. We will explicitly document the strict-majority/tie behavior of Eqs. (3)–(4), correct the categorical statements around Figure 4, and characterize BARC as empirical calibration of potentially misleading entropy feedback, not guaranteed identification or suppression of confident errors.

Reviewer_jfNeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ZORA, a forward-only framework for audio-visual continual test-time adaptation. Shift-Sensitive Layer Selection (SSLS) uses source-target feature separability to select normalization layers in each modality branch. Bias-Aware Reliability Calibration (BARC) combines signed entropy weighting based on local prediction frequency and confidence with source-anchored feature-statistics alignment. Two-point zeroth-order updates adapt the selected parameters. Experiments cover Kinetics50-C and VGGSound-C under interleaved and asynchronous paired corruption streams, with additional quantization, non-IID, architecture-transfer, and sentiment-transfer studies. The reported improvements are promising, but inconsistencies in the objective definition and incomplete specification of the online computation currently limit confidence in the main results.

Critical strengths
  1. Selecting where to update and calibrating the adaptation signal form a coherent approach to forward-only multimodal adaptation. The framework should interest researchers working on continual adaptation and resource-constrained audio-visual recognition.
  2. ZORA exceeds the strongest baseline by 2.49 and 1.86 percentage points under ICS, and by 4.96 and 4.13 points under PACS. Table 11 reports five-seed standard deviations and confidence intervals, strengthening the empirical evidence beyond a single run.
  3. Tables 4-7 study SSLS initialization and simpler selection scores, while Tables 9-10 investigate genuine class imbalance and class bursts. These analyses are useful even though some findings warrant a more cautious interpretation.
Critical weaknesses

Major Concerns

M1 : The window-alignment objective is insufficiently specified to reproduce its zeroth-order evaluation. Equation (6) uses target-window feature statistics, while Eq. (8) evaluates the objective at two perturbed parameter values. Algorithm 1 lists window-statistics computation before perturbation, but this ordering alone does not establish whether statistics are recomputed inside each query. Table 12 reports four total model evaluations per sample without specifying their inputs. Please state whether each perturbation re-evaluates the whole window or refreshes only the current feature while retaining historical features. Holding the entire alignment term fixed across both queries would make its finite-difference contribution zero; refreshing the current feature would instead define a cache-dependent objective, which is not inherently invalid but requires explicit equations. Please specify cache contents, refresh timing, fixed quantities, samples processed per query, and cache memory. The concern is incomplete specification, not a demonstrated implementation error (pp. 6, 16-17, 21; Eqs. (6)-(8), Algorithm 1, Table 12).

M2: The default BARC objective has conflicting definitions. Equation (4) assigns a negative weight of -gamma times the confidence rank and the negative-bias rank. Table 8 identifies this rule as BARC-Hard (54.61 accuracy), whereas BARC-Full uses a constant negative weight of -gamma (55.37 accuracy). The accompanying text calls BARC-Full the default. These rules generally produce different updates, leaving the objective used for Tables 1-3 and subsequent analyses unclear. Please identify the rule used for each experiment and reconcile the equation, algorithm, variant names, and implementation. This establishes an inconsistency in the manuscript, not evidence that experiments actually mixed objectives. Correct the description if this is a reporting error; rerun or revise comparisons only where the clarification reveals an experimental mismatch (pp. 6, 19; Eq. (4), Table 8).

M3: The negative-weight mechanism warrants a bounded interpretation and stronger validation in the reported stress tests. The paper explicitly acknowledges that frequency is a risk signal rather than a correctness test and discusses non-IID limitations. In Table 10, under contiguous class bursts, full BARC with W = 24 is below BARC-Positive in overall accuracy (51.51 versus 51.80) and burst accuracy (36.07 versus 36.67). Its negative-weight precision is 39.04%, so 60.96% of negatively weighted samples are correctly classified in this setting. However, these are single-seed results, and W = 48 raises overall accuracy to 52.13, above BARC-Positive. Thus, the table does not establish a statistically significant or general failure of suppression. Please repeat the positive-only/full comparisons across seeds and qualify the categorical wording around Figure 4 so that observations in its illustrated setting are not presented as a general separation of correct and incorrect confident samples (pp. 8, 19-20; Fig. 4, Tables 9-10).

Minor Comments

1.Specify the distribution and normalization of the perturbation direction in Eq. (8), as well as the percentile-rank convention for ties and singleton sets. These details affect the update scale and the frequent ties induced by a short predicted-label history (pp. 5-6).

  1. Explain the selected LayerNorm parameter counts of 27,648 in Table 12 and 18,432 in Table 15. The latter count also appears in the ZOO control, so the optimizer change alone does not explain the difference. Identify any changes in backbone, candidate-layer definition, or selected parameter tensors, and provide a reproducible count. The present evidence establishes an unexplained difference rather than which count is incorrect (pp. 21-22).

  2. Define the numerical thresholds used to identify WHLE samples in Figure 1. Also distinguish evidence that alignment improves final accuracy from evidence that it reduces estimator variance. Table 2 supports the former, while the latter would require a direct variance diagnostic or more cautious wording (pp. 2, 6, 8).

Questions for authors
  1. For each of the four evaluations in Table 12, what inputs and cached features are used? Which quantities change between the positive and negative perturbations, and is the BARC sample weight fixed as suggested by Eq. (12)?
  2. Which negative-weight rule generated the headline results, Eq. (4) or the BARC-Full row of Table 8? Can the authors provide the corresponding executable update and aligned variant results?
  3. Which split supplies source-reference samples, and do any reference clips overlap with clean counterparts of evaluation clips? Does Table 1 use the suffix-only scoring described for Table 5, and are initialization handling and prediction-before/after-update rules identical across methods? If overlap exists, do the gains persist with disjoint references?
1 comment

M1. DA Computation and Forward Cost. Eqs. (6)–(8) and Algorithm 1 do not fully specify the cached objective. For SSLS-selected layers, the FIFO cache stores per-sample feature sums, squared sums, and token counts for at most W−1W-1 historical samples, plus predictions/confidences and fixed source references. No full token features or computation graphs are retained.

We first evaluate the current AV pair at Θt\Theta_t to obtain its pre-update statistics, prediction, and fixed BARC weight It(0)I_t^{(0)}. During both perturbation queries, historical/source statistics and It(0)I_t^{(0)} are fixed, while current features are recomputed at Θt±=Θt±δu\Theta_t^\pm=\Theta_t\pm\delta u:

Lt±=It(0)H(pΘt±(xt))+λA(Ct−1,st±).L_t^\pm=I_t^{(0)}H(p_{\Theta_t^\pm}(x_t))+\lambda A(\mathcal C_{t-1},s_t^\pm).

Since st±s_t^\pm changes, DA does not cancel. After adaptation, we append the pre-update statistics/prediction obtained at Θt\Theta_t, evicting the oldest entry; the post-update prediction is used only for scoring. Historical samples are never re-encoded. The measured additional FIFO-cache footprint is 48 MB, including historical statistics and prediction/confidence records, rather than a theoretical aggregate-moment estimate. The four evaluations all process only xtx_t: pre-update, positive perturbation, negative perturbation, and post-update. Thus Table 12 reports 2 ZOO queries / 4 total evaluations, excluding one-time SSLS initialization.

M2. Default BARC Rule and Table 8. All headline experiments use Eq. (4), rank_pos_neg: I=QCQB+I=Q_CQ_B^+ for positive samples, −γQCQB−-\gamma Q_CQ_B^- for negative samples, and 00 otherwise. Table 8 mistakenly swapped the BARC-Hard/Full negative-weight descriptions and printed 55.37% for Full. The corrected K50-C PACS results are:

VariantNegative weightAcc. (%)
BARC-Hard−γ-\gamma54.61
BARC-Full (default)−γQCQB−-\gamma Q_CQ_B^-55.64

The 55.37% entry was a transcription error, not another run. Full's verified five-seed result (55.64%) matches Tables 1–3. Hard and Full share the model, data, evaluation and positive weighting; only their negative weights differ.

M3. Negative Weighting under Class Bursts. Prediction frequency/confidence indicate risk, not correctness. We compared Positive-only and Full BARC under contiguous bursts over five seeds:

Window / MethodOverall Acc. (%)Burst Acc. (%)FSR (%)NWP (%)
24 / Positive-only51.74 ± 0.4736.60 ± 2.70——
24 / Full51.44 ± 0.4736.60 ± 0.7032.58 ± 0.3338.62 ± 0.82
48 / Positive-only51.94 ± 0.3336.37 ± 0.70——
48 / Full52.32 ± 0.4836.53 ± 0.8034.67 ± 0.3931.60 ± 0.92

Relative to Positive-only, Full changes overall accuracy by −0.30/+0.38 pp at W=24/48W=24/48, and burst accuracy by 0.00/+0.16 pp. NWP of 38.62/31.60% implies that 61.38/68.40% of negatively weighted predictions are correct. We do not claim significance. Figure 4 therefore illustrates corruption-risk calibration, not universal correctness detection or general class-prior correction.

Q1. Inputs and Fixed Quantities. All four passes use the same current AV pair. The first fixes It(0)I_t^{(0)}; the ± queries recompute its prediction, entropy and features, while retaining cached history/source statistics. The fourth scores the post-update prediction. The cache is committed afterward using statistics/predictions computed at pre-update Θt\Theta_t; It(0)I_t^{(0)} is fixed across both queries (Eq. 12).

Q2. Executable Rule and Aligned Results. The default initial weight is It(0)∈{QCQB+,−γQCQB−,0}I_t^{(0)}\in\{Q_CQ_B^+,-\gamma Q_CQ_B^-,0\}. We estimate g^t=[(Lt+−Lt−)/(2δ)]ut\hat g_t=[(L_t^+-L_t^-)/(2\delta)]u_t, apply global L2 normalization, and update selected LN affine parameters. BARC-Hard changes only the negative weight to −γ-\gamma. Table 12 counts 27,648 adapted parameters for main CAV-MAE ZORA; Table 15 counts 18,432 for both BP and ZOO in a separate single-seed optimizer-transfer configuration. BP and ZOO are parameter-matched within Table 15; these are not counts for the same run.

Q3. Reference Split and Evaluation Protocol. Source references are drawn from clean source-validation clips disjoint from evaluation clips by original clip identity, including clean/corrupted counterparts. Tables 1 and 5 exclude the first 64 unlabeled target samples from scoring. ZORA uses this prefix solely for one-time SSLS layer selection, not for online parameter updates or retained target history. Table 7 reports 1,449.74 ms for feature extraction and 307.21 ms for LSS scoring (1,756.95 ms total), separately from per-sample ZOO queries. All methods use the same source checkpoint, scored suffix and post-update prediction convention.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.