- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies the robustness of audio-visual question answering under environmental noise and competing speech. It proposes OP-CAD, an on-policy self-distillation method where a student generates responses from noisy audio-visual inputs, while a frozen teacher uses clean inputs and verified answers to provide token-level supervision. The main idea is to assign higher weights to tokens whose teacher predictions are sensitive to audio removal or corruption. The method also uses failure-directed data selection and a noise curriculum that gradually increases the interference level. Experiments on Daily-Omni and OmniVideoBench show improvements over SFT, GRPO, and uniform-weight OPSD under noisy conditions, with no observed drop in overall clean accuracy. The authors also provide ablations on the two weighting signals and the noise curriculum.
Critical strengths
- Consistent improvements over OPSD. OP-CAD improves noisy-condition accuracy by around 2.8–3.8 percentage points over uniform-weight OPSD across both benchmarks and interference types. This comparison is well controlled, as both methods share the same teacher, data, warm-up checkpoint, and noise curriculum. The gains hold across different question types, audio types, and video durations, and the ablation studies confirm the benefits of both sensitivity signals and the curriculum schedule.
- Simple and practical approach. The proposed token-weighting mechanism is straightforward to implement on top of standard self-distillation, as it only reweights the token-level KL loss during training. At inference time, the student model runs directly on audio-visual inputs without requiring additional model components, clean audio references, or extra teacher passes.
Critical weaknesses
-
Limited methodological novelty. The overall pipeline closely follows existing on-policy self-distillation methods, particularly OPSD, while privileged supervision from clean audio or text has already been explored in recent audio distillation works such as EchoDistill and CORD. On top of OPSD, the main additions are a token-weighting rule based on two JS divergences, along with failure-directed data screening and an SNR curriculum. Although the method achieves consistent improvements, the core methodological contribution appears relatively incremental over prior privileged distillation frameworks.
-
Insufficient justification and validation of the weighting mechanism. Section 3.4 motivates the max rule and per-response normalization, but their advantages over simpler alternatives are not directly established. The paper does not compare max with a mean or convex weighted sum, examine sensitivity to lambda, or ablate response-wise normalization. Table 4 also shows that the combined rule is not uniformly better than corruption sensitivity alone: on OmniVideoBench, Clean accuracy is 41.60% versus 43.80%, while Speech accuracy is nearly unchanged at 38.93% versus 39.00%. These observations do not negate the overall robustness gains, but leave the benefit of the particular combination rule less clear. Since the allocation signals require additional teacher evaluations, a comparison with simpler weights computed from existing predictions would help assess their added value.
Questions for authors
-
Section 3.4 explains the scale-calibration motivation for per-response normalization and the use of max to emphasize either sensitivity signal. Could the authors compare this design with a mean or convex weighted sum, ablate response-wise normalization, and report sensitivity to lambda? In particular, can response-wise normalization amplify small but nonzero sensitivity signals, and how does this affect the reliability of the allocation weights?
-
Could the authors compare OP-CAD with simpler token-weighting strategies computed from existing teacher predictions, such as privileged-teacher confidence or entropy, under the same training setup? This would help determine whether the robustness gains require the additional clean, corrupted, and visual-only teacher evaluations.
3 comments
Response to Reviewer P2fh (Part 2/3)
Q3: An ablation of normalization and an analysis of small nonzero sensitivity scores are needed.
A3: Thank you for examining the normalization behavior closely. Response normalization places the two signals on comparable scales within each response, and it can increase the relative weights assigned to signals with small absolute magnitudes. To examine this behavior, we add weight statistics for relatively low-sensitivity responses and a training control without normalization.
For each fixed response, we compute M = max_t{A_t, N_t}, where A and N are raw JS scores and t ranges over valid generated positions. Within each SNR, we rank responses by M and select the lowest fifth as the relatively low-sensitivity group. Both weighting rules are then evaluated on the same responses and token positions. The statistics are shown below.
| Rule | SNR | Responses | Mean M | Mean weight | Weight SD |
|---|---|---|---|---|---|
| Response normalization | 10 dB | 418 | 0.092 | 1.243 | 0.117 |
| Response normalization | 5 dB | 512 | 0.087 | 1.251 | 0.121 |
| Response normalization | 0 dB | 655 | 0.079 | 1.238 | 0.114 |
| Raw JS (no normalization) | 10 dB | 418 | 0.092 | 1.041 | 0.016 |
| Raw JS (no normalization) | 5 dB | 512 | 0.087 | 1.038 | 0.015 |
| Raw JS (no normalization) | 0 dB | 655 | 0.079 | 1.035 | 0.014 |
Normalization increases both the mean and variation of the weights in this group, giving relative sensitivity differences within a response greater influence on allocation. The default λ bounds the weight increment, and the final loss is still divided by the total weight.
We also train a variant that uses raw JS scores directly to construct weights, keeping λ and the other settings unchanged:
Daily-Omni (accuracy, %)
| Method | Clean | Env. 10 dB | Env. 5 dB | Env. 0 dB | Speech 10 dB | Speech 5 dB | Speech 0 dB |
|---|---|---|---|---|---|---|---|
| OP-CAD (default) | 70.09 | 69.42 | 67.08 | 64.41 | 67.08 | 63.74 | 59.82 |
| No response normalization | 69.76 | 67.34 | 64.66 | 62.24 | 63.99 | 61.24 | 57.98 |
OmniVideoBench (accuracy, %)
| Method | Clean | Env. 10 dB | Env. 5 dB | Env. 0 dB | Speech 10 dB | Speech 5 dB | Speech 0 dB |
|---|---|---|---|---|---|---|---|
| OP-CAD (default) | 41.60 | 40.10 | 39.40 | 41.20 | 39.10 | 39.50 | 38.20 |
| No response normalization | 41.30 | 38.60 | 37.50 | 39.70 | 36.30 | 37.70 | 36.50 |
The default model performs better across all noisy conditions. The weight statistics explain how normalization changes allocation, while the training comparison supports its positive effect on overall question-answering performance. Together, these analyses provide a clearer account of both the behavior and empirical value of response-wise scale calibration.
Manuscript reference: Appendix E.4 (Table 27) defines the group and reports the weight statistics; Appendix E.1.3 (Table 19) presents training without normalization. Section 3.4 explains that the weights reflect relative sensitivity.
Q4: The results should be evaluated for their dependence on the choice of λ.
A4: Thank you for suggesting an evaluation of the weighting strength. We compare λ ∈ {0, 0.25, 0.5, 1.0}, reusing the original OPSD and default OP-CAD results for λ = 0 and 0.5, respectively. The other configurations use the same training settings. Results are reported below.
Daily-Omni (accuracy, %)
| Method | Clean | Env. 10 dB | Env. 5 dB | Env. 0 dB | Speech 10 dB | Speech 5 dB | Speech 0 dB |
|---|---|---|---|---|---|---|---|
| OPSD | 69.34 | 65.83 | 63.66 | 60.99 | 64.16 | 61.57 | 56.14 |
| λ=0.25 | 69.84 | 68.84 | 66.42 | 63.83 | 66.50 | 63.24 | 59.23 |
| OP-CAD (default) | 70.09 | 69.42 | 67.08 | 64.41 | 67.08 | 63.74 | 59.82 |
| λ=1.0 | 69.76 | 68.34 | 66.25 | 63.99 | 66.08 | 63.32 | 59.06 |
OmniVideoBench (accuracy, %)
| Method | Clean | Env. 10 dB | Env. 5 dB | Env. 0 dB | Speech 10 dB | Speech 5 dB | Speech 0 dB |
|---|---|---|---|---|---|---|---|
| OPSD | 41.40 | 37.40 | 36.20 | 38.60 | 36.50 | 34.60 | 34.30 |
| λ=0.25 | 41.50 | 39.70 | 39.00 | 40.80 | 38.70 | 39.10 | 37.80 |
| OP-CAD (default) | 41.60 | 40.10 | 39.40 | 41.20 | 39.10 | 39.50 | 38.20 |
| λ=1.0 | 41.50 | 39.40 | 38.70 | 40.50 | 38.50 | 38.90 | 37.60 |
Values on either side of the default continue to outperform uniformly weighted OPSD across all noisy conditions and remain close to the default configuration. This shows that the gains persist over the tested range and do not depend on a single λ value. The default configuration performs best in these comparisons, so we retain the original setting.
Manuscript reference: Appendix E.1.5 (Table 21) reports the λ-sensitivity results.
Response to Reviewer P2fh (Part 3/3)
Q5: Confidence or entropy weights derived directly from the privileged teacher should be compared to assess the value of the additional teacher evaluations.
A5: Thank you for proposing this useful control. We add two weighting rules derived directly from the clean privileged teacher’s predictions: normalized entropy H(p)/ln|V| and confidence max p. Both use w = 1 + 0.5s and divide the weighted loss by the total weight. They reuse the existing teacher distribution without evaluating the three auxiliary teacher contexts; all other training settings and the data-pass budget remain unchanged. The results are shown below.
Daily-Omni (accuracy, %)
| Method | Clean | Env. 10 dB | Env. 5 dB | Env. 0 dB | Speech 10 dB | Speech 5 dB | Speech 0 dB |
|---|---|---|---|---|---|---|---|
| OPSD | 69.34 | 65.83 | 63.66 | 60.99 | 64.16 | 61.57 | 56.14 |
| OP-CAD (default) | 70.09 | 69.42 | 67.08 | 64.41 | 67.08 | 63.74 | 59.82 |
| Teacher entropy | 69.09 | 67.34 | 65.16 | 62.32 | 65.25 | 62.07 | 57.73 |
| Teacher confidence | 69.17 | 67.00 | 64.83 | 61.99 | 64.91 | 61.74 | 57.39 |
OmniVideoBench (accuracy, %)
| Method | Clean | Env. 10 dB | Env. 5 dB | Env. 0 dB | Speech 10 dB | Speech 5 dB | Speech 0 dB |
|---|---|---|---|---|---|---|---|
| OPSD | 41.40 | 37.40 | 36.20 | 38.60 | 36.50 | 34.60 | 34.30 |
| OP-CAD (default) | 41.60 | 40.10 | 39.40 | 41.20 | 39.10 | 39.50 | 38.20 |
| Teacher entropy | 41.30 | 38.50 | 37.70 | 39.60 | 37.30 | 37.80 | 36.60 |
| Teacher confidence | 41.10 | 38.20 | 37.40 | 39.30 | 37.00 | 37.50 | 36.30 |
Both entropy and confidence weighting outperform uniformly weighted OPSD under noise, while OP-CAD further improves accuracy across all noisy conditions. This indicates that sensitivity obtained by comparing audio conditions can guide supervision allocation more effectively than these weights derived from the target teacher alone. We appreciate your suggestion, which provides a more direct test of the value of the additional audio-condition comparisons.
Manuscript reference: Appendix E.1.6 (Table 22) gives the weighting rules and results at each SNR.
Thank you again for your thoughtful review and constructive suggestions. We hope these clarifications and additional experiments make the contribution and the rationale for our design choices clearer. We would be happy to address any remaining questions and continue the discussion to further improve the manuscript.
Response to Reviewer P2fh (Part 1/3)
Dear Reviewer P2fh,
We sincerely thank you for recognizing the robustness improvements and for your constructive comments on the contribution and weighting design. In response, we have clarified how our approach relates to prior work and added comparisons of signal fusion, response normalization, weight strength, and simple teacher-derived weights. The relevant explanations and results are included in the main paper and appendix, with revisions highlighted in blue. Our responses follow.
Q1: The method builds on OPSD and should clarify its contribution relative to approaches such as EchoDistill and CORD.
A1: Thank you for encouraging us to clarify the contribution. OP-CAD builds on OPSD’s distillation framework over student-generated trajectories. Its central addition is supervision allocation based on sensitivity to audio conditions while retaining a shared visual context. EchoDistill and CORD explore privileged supervision from clean audio and text, respectively. OP-CAD instead compares teacher predictions after audio removal and corruption against a common clean audio-visual reference, using the same student prefixes and visual context, and uses these differences to assign token-level supervision strength.
The learning target and the allocation signal serve distinct roles in this design: the target teacher receives the verified answer, whereas all three contexts used to compute allocation signals exclude it. Our additional experiments directly evaluate this mechanism. The default method outperforms shuffled weights, Mean fusion, unnormalized scores, and teacher entropy/confidence weights under noise, and retains its advantage over OPSD when training compute is approximately matched. These results support the practical value of audio-condition comparisons within the existing distillation framework. We have clarified this positioning in the method description and included the corresponding controls in the appendix.
Manuscript reference: Section 2 discusses the related methods, and Section 3.4 explains supervision allocation. Appendix E.1 (Tables 16–22) provides the additional empirical comparisons.
Q2: A comparison with Mean or convex combinations is needed, along with an explanation of why the combined signal sometimes underperforms corruption sensitivity alone.
A2: Thank you for suggesting this control and highlighting the differences in the single-signal results. We add Mean as an equally weighted convex combination, s = (Ã + Ñ)/2, and compare it with Max and the two original single-signal variants. All configurations use the same λ and other training settings. The results are shown below. Env. and Speech denote environmental noise and competing speech, respectively.
Daily-Omni (accuracy, %)
OmniVideoBench (accuracy, %)
Max outperforms Mean across all noisy conditions. When only one sensitivity signal is strong, Mean reduces the weight increment at that position, whereas Max preserves it. The additional results support this fusion choice.
The complete results also retain the single-signal advantage you noted: corruption-only weighting performs better on clean input and in some noisy OmniVideoBench conditions, while the combination is stronger in others. This reflects differences in how the signals contribute across conditions. We preserve these results and add the direct comparison with Mean to provide further empirical support for choosing Max.
Manuscript reference: Table 4 in the main paper retains the single-signal results. Appendix E.1.4 (Table 20) adds the fusion comparison at each SNR.