- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper is well organized, with clear explanations of the research motivation, knowledge construction procedure, and online decision-making process. Figure 1 effectively illustrates the framework’s central ideas and application scenarios. Figure 2 further clarifies the organization of knowledge and the corresponding interaction process, helping readers understand how historical experience is translated into explicit, inspectable conditions for the current task.
Critical strengths
This paper explores the challenges associated with cross-domain lip-reading. By considering not only visual-level variations but also the influence of linguistic patterns, the authors offer a novel perspective; the proposed SDGP method provides an effective approach to addressing the issue of unreliable linguistic dependencies. Furthermore, the study introduces a new dataset encompassing seven distinct scenarios and validates the method using both Chinese and English datasets, which enhances the robustness of the research. Experimental results demonstrate that this approach yields performance improvements across a variety of models.
Critical weaknesses
The authors argue that the model’s poor performance on cross-domain tasks stems from the acquisition of flawed linguistic habits. However, while the experimental results demonstrate a reduction in erroneous predictions, they do not confirm that these errors are rooted in linguistic habits; they could simply be the result of general error correction.
Equation (14) correctly defines the projection method, yet Algorithm S2 introduces an additional (epsilon) term in the denominator. Consequently, the gradients calculated in this manner do not consistently satisfy the constraints established in Equation (12).
The theoretical derivation is based on stochastic gradient descent (SGD), whereas the experiments actually employ the AdamW optimizer. The paper does not explicitly clarify whether these theoretical guarantees still hold when using the optimizer applied in the experiments.
Questions for authors
Could the authors explain to what extent the additional term introduced in Algorithm S2 affects the constraints in Equation (12)?
In practical applications, is this deviation small enough to be considered negligible?
Given that the theoretical analysis is based on SGD, while the experiments employ AdamW, does the proposed gradient constraint still function as intended when used in conjunction with AdamW?
Why is gradient projection necessary, rather than simply incorporating the dependency score as a regularization loss term?
4 comments
Responses to Reviewer TKA3 (2)
W3 & Q3. Optimizer correction and scope of the guarantee.
Lines 380--382 incorrectly stated that we used AdamW, which was accidentally left in the manuscript from our earlier optimizer comparisons. As stated in lines 238--239, when developing this method, we used SGD as the theoretical basis because its simple update rule makes the core idea of SDGP clear and easy to follow. After deriving the projection rule, when conducting our formal experiments, we applied it directly to three widely used optimizers: SGD, Adam, and AdamW. Since their recognition results differed very little, we selected SGD for the main paper because its update rule directly matches our theoretical derivation and allows us to test the theory cleanly. To ensure a fair comparison, all experiments in Table 3 and the subsequent ablations, including all tuned baselines, were conducted using SGD. We have corrected lines 380--382 and added the full comparison with Adam and AdamW to Appendix S5.3 of the revised manuscript.
For Adam and AdamW, we compute using the projection derived in Eq. (14) and pass it to the outer optimizer. Specifically, SGD directly applies the update , whereas Adam and AdamW use to update their first- and second-moment estimates before computing the actual parameter step, with AdamW additionally applying decoupled weight decay. On WildVSR, the WERs (%) for SGD / Adam / AdamW are 74.2 / 74.3 / 74.2 on AVEC, 75.4 / 75.5 / 75.5 on AV-RelScore, and 55.7 / 55.8 / 55.6 on Llama-AVSR. Across all three backbones, the differences from SGD are negligible.
For the first-order non-increase condition to hold under Adam or AdamW, the actual parameter change must satisfy . Among sampled Llama-AVSR steps where projection was active, this condition held for 100% of SGD updates, 85.4% of Adam updates, and 87.8% of AdamW updates. Although Adam and AdamW did not satisfy this condition at every individual step, their final recognition performance remained comparable to that of SGD. We attribute this stability in part to the smoothing effect of their running first- and second-moment estimates. Overall, the comparable WERs across optimizers further confirm the robustness of SDGP to the choice of outer optimizer.
Q4. Necessity of projection.
Table 5 in the main paper directly compares SDGP with adding the dependency score as a fixed-weight regularization term. Adding to the recognition loss yields . On WildVSR, the best tested fixed weight yielded 74.8/76.1/56.4% WER on AVEC/AV-RelScore/Llama-AVSR, compared with 74.2/75.4/55.7% for SDGP. A fixed weight applies the same coefficient in every episode. SDGP adapts the correction strength to the gradient magnitudes and their alignment, making only the change to required by the constraint.
To further isolate the effect of the projection rule, we applied Agr-Sum (Mansilla et al., ICCV 2021) to the same and . WildVSR WERs (%) for Agr-Sum/SDGP were 75.0/74.2 on AVEC, 77.4/75.4 on AV-RelScore, and 60.5/55.7 on Llama-AVSR. Agr-Sum treats the two gradients symmetrically. SDGP gives recognition priority by using to define the suppression constraint and choosing the feasible gradient closest to . These results support the need for our gradient projection.
We hope these clarifications and additional results address your concerns. We would appreciate your reconsideration of the score and welcome any further questions.
Dear Reviewer,
I hope this message finds you well. With less than two days remaining in the discussion period, we wanted to follow up and ensure that our rebuttal has satisfactorily addressed your questions and concerns. If you have any further feedback or additional points for discussion, please do let us know. We deeply appreciate your time, effort, and invaluable insights in helping us improve our work.
Best regards, The Authors
Dear Reviewer,
I hope this message finds you well. With the discussion period closing in less than ten hours, we wanted to gently follow up and see if our rebuttal has satisfactorily addressed your questions and concerns. If you have any further feedback or additional points for discussion, please do let us know. We deeply appreciate your time, effort, and invaluable insights in helping us improve our work.
Best regards, The Authors
Responses to Reviewer TKA3 (1)
Thank you for your careful review and constructive comments. Below are our point-by-point responses to your feedback. Changes in the revised manuscript are highlighted in blue.
W1. Influence of the linguistic bias and selective correction.
In the main paper, Table 4 quantifies source-biased errors and their correction by SDGP on test data, while Fig. 5 visualizes this behavior. To further isolate the influence of linguistic bias and SDGP’s effectiveness, we added a prefix-replacement experiment on LRS2 and LRS3 training data in Section S5.2 of the revised manuscript. Using training data provides more frequent occurrences of the selected dependencies, allowing a more targeted assessment of their influence.
Specifically, we evaluated trained Llama-AVSR models with and without SDGP under teacher forcing across both LRS2 → LRS3 and LRS3 → LRS2 directions. When a selected dependency favors an incorrect token on the meta-test split, we keep the visual input fixed and compare that token's probability under the triggering context against an equal-length, non-triggering context drawn from the meta-train transcripts.
With identical visual input, the original prefix raised the baseline's mean error probability by 3.72 and 3.57 percentage points across the two shifts, confirming the adverse impact of linguistic bias under cross-domain transfer. SDGP sharply suppressed this effect, narrowing the gaps to 0.77 and 0.12 percentage points, respectively.
W2 & Q1–Q2. Effect of ϵp on the projection constraint.
We have added a derivation and numerical analysis of the effect of ϵp to Appendix S5.4 of the revised manuscript. In Eq. (14), when ∥h∥ approaches zero, dividing by ∥h∥2 can amplify even a minor boundary violation into an excessively large gradient correction. To prevent this numerical instability, we add a constant term ϵp=10−12 to the denominator.
On active projection steps, this modification changes the constraint margin in Eq. (12) by
(⟨h,gϵ⟩−mk)−(⟨h,g⋆⟩−mk)=−ϵp∥h∥2+ϵpmk−⟨h,g⟩.
Across active steps, the logged maximum gaps on the English benchmark were 1.19×10−11 for AVEC+SDGP, 5.82×10−10 for AV-RelScore+SDGP, and 3.31×10−10 for Llama-AVSR+SDGP, confirming that their practical deviation is negligible.