- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper is about adapting audio-visual models at test time when backpropagation cannot be used, like on edge devices or quantized models. The authors mention that vanilla zeroth-order optimization (ZOO) works badly here for two reasons. It perturbs too many parameters, and entropy minimization trains on predictions that can be very confident but are wrong. Their method, ZORA, has two parts. The first part on SSLS, picks the top 3 LayerNorm layers (based on their experimental setup section) in each of the audio and video encoders. It ranks them by how well a 2-means clustering separates source features from early target features, and only those layers get updated. The second part on BARC, looks at a window of recent predictions and counts how often each class is predicted. A confident sample whose predicted class shows up a lot gets a negative entropy weight. A confident sample whose class shows up less gets a positive weight. Everything else gets zero and a loss (L_align) that matches feature statistics to the source is added on top. They test on Kinetics50-C and VGGSound-C with two streams that they built, using 5 seeds and 12 baselines. They achieve about 1.9 to 2.5points over the best baseline on the interleaved stream and about 4 to 5points on the PairTrack-Async stream. There are also fake-quantized runs (W8A8, W4A4), a second architecture, ImageNet-C and sentiment transfer, and some stress tests for BARC under class imbalance.
Critical strengths
- Well-motivated problem with a thorough evaluation. Forward-only AV-CTTA is relevant in practice, especially for on-edge devices.
- The authors provide multi-seed statistics, many protocols, a second architecture, SSLS robustness studies, and a stress test of where BARC fails (Appendix D.4).
- The results are quite promising for further application if they hold up.
- SSLS and BARC are modular components. Table 15 suggests both help even with exact gradients. This gives the work value for future TTA work beyond the specific ZO setting, if validated more rigorously.
Critical weaknesses
- Some of the numbers are confusing. The alignment loss uses statistics over a window of samples (Eq. 6), but it is evaluated at the two perturbed parameter points (Eq. 8). If the window features are cached, the term is the same at both points and cancels out of the finite difference. Then it does nothing, even though the ablation gives "DA" about +3.9 points. If the features are recomputed, the cost is more than 2 forward passes. Also, Fig. 5 shows accuracy flat at about 55.3 to 55.5 for all λ and γ, including 0. Based on my understanding, when λ and γ are 0, it should correspond to the ablation entry in Table 2 without the signed weights and alignment. But in Table 2, it shows that each add about 4 points.
- The comparison and the edge claims are not convincing. Some of the baselines could be broken in Table 11. DASP is almost the same as the source on ICS and PACS, with values on a 0.5 grid. PTA, I believe is built for prediction bias, drops below the source on PACS (35.23 and 4.83). The authors also did not mention what ZORA's hyperparameters is tuned on and if it is tuned on the same streams it is tested on. Also, the authors never defined naïve ZOO, so I am not sure what the difference is between "Entropy-only" which achieves 53.80 in Table 9, and "naive ZOO" is 44.31 in Table 2. This could suggest that their naïve ZOO is a weak baseline.
Questions for authors
- Alignment loss. How is L_align computed inside Eq. 8? Are window features cached or recomputed under Θ±δu? If cached, does it cancel in the finite difference? If recomputed, what is the true forward count?
- On Fig. 5 vs. Table 2, could you report the full model with λ=0 and with γ=0 (mean±std over 5 seeds). How is it interpreted with the +3.9 (DA) and +4.0 (SR) ablation gains?
- For Naïve ZOO and Table 9, could you define naïve ZOO in Table 2 (parameters, loss, δ, η, perturbation distribution)? Why does entropy-only reach 53.80 in Table 9?
- For the baselines, why is DASP approximately equal to source on ICS/PACS, with 0.5-granular values, while PTA and READ drop below source? What hyperparameter search did each baseline vs. ZORA receive, and on which data were ZORA's hyperparameters chosen on?
2 comments
Additional Evidence: Corrected DASP Five-Seed Per-Condition Results (Tables 16–19)
Additional Evidence: Corrected DASP Five-Seed Per-Condition Results (Tables 16–19)
The original appendix inadvertently used single-seed DASP entries for some per-condition accuracies. We have recomputed the mean across five seeds for each condition. The PACS single-seed values followed 0.5-pp increments because each condition contains 200 examples. The corrected results are reported below, with all aggregate means unchanged. Values represent accuracy (%), averaged over five seeds. A and V denote audio and visual corruptions.
Table 16. Kinetics50-C / ICS
Conditions 1–7
| Method | V. | V | A. | V. | V. | A. | V. |
|---|---|---|---|---|---|---|---|
| DASP | 49.86 | 51.11 | 72.66 | 50.33 | 69.38 | 66.58 | 60.17 |
Conditions 8–14
| Method | V. | A | V | V | V | A | V |
|---|---|---|---|---|---|---|---|
| DASP | 70.32 | 66.58 | 69.38 | 60.95 | 65.95 | 68.30 | 46.74 |
Conditions 15–21
| Method | V. | A. | V. | V. | A | V. | V | Avg. |
|---|---|---|---|---|---|---|---|---|
| DASP | 78.44 | 68.76 | 52.52 | 63.61 | 70.63 | 64.86 | 67.21 | 63.54 |
Avg-V: 61.39%; Avg-A: 68.92%.
Table 17. VGGSound-C / ICS
Conditions 1–7
| Method | V. | V | A. | V. | V. | A. | V. |
|---|---|---|---|---|---|---|---|
| DASP | 46.01 | 48.67 | 14.91 | 46.17 | 57.40 | 18.67 | 51.32 |
Conditions 8–14
| Method | V. | A | V | V | V | A | V |
|---|---|---|---|---|---|---|---|
| DASP | 62.09 | 22.73 | 55.70 | 48.82 | 51.32 | 21.79 | 52.88 |
Conditions 15–21
| Method | V. | A. | V. | V. | A | V. | V | Avg. |
|---|---|---|---|---|---|---|---|---|
| DASP | 63.97 | 21.32 | 48.67 | 56.31 | 26.32 | 56.01 | 55.23 | 44.11 |
Avg-V: 53.37%; Avg-A: 20.96%.
Table 18. Kinetics50-C / PairTrack-Async
Conditions 1–5
| Method | A./V. | A./V | A./V. | A./V. | A./V. |
|---|---|---|---|---|---|
| DASP | 35.80 | 34.30 | 32.50 | 41.60 | 39.00 |
Conditions 6–10
| Method | A/V. | A/V | A/V | A/V | A/V |
|---|---|---|---|---|---|
| DASP | 58.80 | 53.20 | 44.80 | 37.30 | 20.00 |
Conditions 11–15
| Method | A./V. | A./V. | A./V. | A/V. | A/V | Avg. |
|---|---|---|---|---|---|---|
| DASP | 67.40 | 39.00 | 47.30 | 60.50 | 54.50 | 44.40 |
Table 19. VGGSound-C / PairTrack-Async
Conditions 1–5
| Method | A./V. | A./V | A./V. | A./V. | A./V. |
|---|---|---|---|---|---|
| DASP | 5.70 | 7.50 | 5.70 | 7.90 | 6.70 |
Conditions 6–10
| Method | A/V. | A/V | A/V | A/V | A/V |
|---|---|---|---|---|---|
| DASP | 22.60 | 17.30 | 16.80 | 6.30 | 3.40 |
Conditions 11–15
| Method | A./V. | A./V. | A./V. | A/V. | A/V | Avg. |
|---|---|---|---|---|---|---|
| DASP | 24.50 | 12.90 | 15.10 | 19.50 | 17.60 | 12.63 |
These corrected rows will replace the DASP per-condition entries in the revised appendix. Aggregate mean ± standard deviation remains available in Table 11.
. Seed-wise Comparison of Figure 5 and Additional Ablations on VGGSound-C PACS (Accuracy, %)
| Random Seed / Record | Full ZORA | ||
|---|---|---|---|
| (Figure 5, seed 42) | 22.53 | 22.67 | 22.65 |
| (20260720) | 21.35 | 21.49 | 21.14 |
| (20260721) | 22.00 | 21.64 | 21.38 |
| (20260722) | 22.09 | 21.82 | 21.39 |
| (20260723) | 22.32 | 21.55 | 21.52 |
| (20260724) | 22.54 | 21.80 | 21.77 |
| Mean ± Std (–) | 22.06 ± 0.45 | 21.66 ± 0.15 | 21.44 ± 0.23 |
Note. The independent Figure 5 seed () is excluded from the five-seed mean and sample standard deviation, which are computed over – only.
W1. DA consistency. Cached history and recomputed current features make DA perturbation-dependent (Q1). Figure 5 is reproducible under its original seed (Q2).
W2. Baseline reliability. Matched-optimizer ablations and update diagnostics are in Q3–Q4. Efficiency results cover forward-only evaluation and simulated quantization, not physical edge deployment.
Q1. DA cache and scoring. DA uses a cached-history alignment surrogate. Historical pre-update feature sums, squared sums and counts stay fixed under (\Theta_t\pm\delta u); only current features (z_t^\pm) are recomputed. Thus (\mu_t^\pm=(S_{\mathrm{hist}}+z_t^\pm)/(n_{\mathrm{hist}}+1)), so DA need not cancel. Historical features may come from earlier parameter states and are not re-encoded; unperturbed current statistics enter the cache after adaptation. Source references and initial BARC weights stay fixed within the step. One initial, two perturbed and one post-update pass give 2 ZOO queries / 4 evaluations (Table 12). We will clarify this surrogate, gradient normalization and execution order in Eqs. (6)/(8) and Algorithm 1.
Q2. Figure 5 vs. Table 2. Table 2 uses cumulative ablations without SSLS (DA: +3.87/+4.04 pp); Figure 5 varies coefficients within full ZORA. Setting (\gamma=0) removes negative suppression but retains positive weighting.
Over five seeds (20260720–20260724), Full gives 55.64 ± 0.67% / 22.06 ± 0.45% (K50-C/VGG-C), versus 54.64 ± 0.21% / 21.66 ± 0.15% for (\lambda=0) and 54.45 ± 0.21% / 21.44 ± 0.23% for (\gamma=0). These are marginal rather than cumulative effects.
Figure 5 used seed 42, independently reproduced with identical configurations and scoring. Table X reports its VGG-C results separately: Full exceeds (\lambda=0) in 4/5 and (\gamma=0) in 5/5 additional seeds. Seed 42 is excluded from the five-seed mean ± std. The numerical gap reflects seed variation, not inconsistent implementations.
Q3. Naïve ZOO and matched-budget attribution. Original Table 2 Naïve ZOO uses raw entropy, All-Norm updates, (\delta=10^{-3},\eta=10^{-3},q=10) Rademacher directions (20 perturbation queries) and global gradient normalization. Table 9's “Entropy-only” retains SSLS+DA on non-IID streams, so its 53.80% and Table 2's 44.31% are not directly comparable. We instead ran five-seed optimizer-matched ablations with (\delta=10^{-3},\eta=10^{-2},q=1), Rademacher directions, global L2 normalization, two queries, one update and post-update scoring:
All-Norm updates AV normalization parameters; DA aligns fixed Top-3 layers even without SSLS-restricted updates. Full exceeds matched entropy by 9.64/10.03 pp; SR (G−E) adds 1.13/0.98 pp and DA (G−F) 1.00/0.40 pp. We will distinguish the original unmatched Table 2 row and relabel Table 9's entropy variant.
Q4. Baseline diagnostics. DASP: We follow official method settings. Since its batch-dependent redundancy statistics cannot be computed meaningfully at B=1, we replace the batch input with a causal W=24 window (current AV pair plus up to 23 past pairs); the adaptation procedure is otherwise retained. Logged K50-C/VGG-C runs show 2,553/2,564 updates. The appendix's 0.5-pp entries came from single-seed per-condition records; corrected five-seed means (Tables 16–19) preserve aggregate results.
PTA (K50-C): With the official loss and a causal FIFO-8 extension, both partitions are populated in 76.46% of steps and 2,241 updates (76.33%) occur. PTA scores 36.24% vs. paired Source 46.01%, correcting 183 errors but introducing 470. READ updates in 99.90%/99.86% of steps; on VGG-C it corrects 17 errors but introduces 215, while dominant-class share rises 16.38%→67.30%. Both show active but potentially harmful adaptation; failures are protocol-specific.
Hyperparameter selection. ZORA's (K=3,W=24,\delta=10^{-3},\eta=10^{-2},\lambda=\gamma=0.5) were selected from 10 configurations on clip-disjoint held-out development streams and fixed before testing; no test labels were used. Baselines follow official objectives and method-specific settings, with necessary AV interfaces in Appendix C (Adam (10^{-4}) for gradient-based methods; fused-prediction losses/AV LayerNorm where applicable). Batch-dependent statistics incompatible with B=1 use causal windows (DASP W=24; PTA FIFO-8). No extra baseline tuning was performed.