acceptodds
Desk rejected at ICLR 2027

Rethinking Domain Generalization in Lip Reading: A Dependency Reliability Perspective

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Lip reading, which recovers spoken content from lip dynamics, has made substantial progress on standard benchmarks. However, these gains do not translate to real-world settings, where shifts in speakers, recording conditions, and linguistic distributions can sharply degrade performance. Existing domain generalization methods tackle these shifts primarily by learning domain-invariant visual representations, yet they overlook that lip-reading models inherently rely on linguistic context to compensate for visual ambiguity, thereby internalizing strong context-word dependencies. When these dependencies become unreliable across domains, such source-specific cues turn into misleading priors, causing severe generalization failures. To address this challenge, we propose Selective Dependency Gradient Projection (SDGP), a model-agnostic strategy that explicitly regulates non-transferable linguistic dependencies during optimization. Specifically, SDGP identifies dependencies exhibiting reliability mismatch across episodic domain splits, and quantifies model reliance on them via source-favored incorrect candidates. Differentiating this score yields a reliance gradient that captures the optimization direction amplifying these shortcuts. To suppress such reinforcement, SDGP projects the recognition gradient onto an adaptive half-space that imposes a first-order suppression constraint. The resulting convex subproblem admits a closed-form solution. To systematically study this mechanism, we further introduce LipWild-G7, a seven-genre Chinese benchmark designed for controlled cross-domain evaluation under joint visual and linguistic shifts. Extensive experiments on Chinese and English benchmarks show that SDGP consistently improves unseen domain recognition, highlighting dependency reliability in cross-domain lip reading.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
lip reading, visual speech recognition, domain generalization, linguistic dependency, selective gradient projection

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TKA3OpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper is well organized, with clear explanations of the research motivation, knowledge construction procedure, and online decision-making process. Figure 1 effectively illustrates the framework’s central ideas and application scenarios. Figure 2 further clarifies the organization of knowledge and the corresponding interaction process, helping readers understand how historical experience is translated into explicit, inspectable conditions for the current task.

Critical strengths

This paper explores the challenges associated with cross-domain lip-reading. By considering not only visual-level variations but also the influence of linguistic patterns, the authors offer a novel perspective; the proposed SDGP method provides an effective approach to addressing the issue of unreliable linguistic dependencies. Furthermore, the study introduces a new dataset encompassing seven distinct scenarios and validates the method using both Chinese and English datasets, which enhances the robustness of the research. Experimental results demonstrate that this approach yields performance improvements across a variety of models.

Critical weaknesses

The authors argue that the model’s poor performance on cross-domain tasks stems from the acquisition of flawed linguistic habits. However, while the experimental results demonstrate a reduction in erroneous predictions, they do not confirm that these errors are rooted in linguistic habits; they could simply be the result of general error correction.

Equation (14) correctly defines the projection method, yet Algorithm S2 introduces an additional ϵ\epsilon (epsilon) term in the denominator. Consequently, the gradients calculated in this manner do not consistently satisfy the constraints established in Equation (12).

The theoretical derivation is based on stochastic gradient descent (SGD), whereas the experiments actually employ the AdamW optimizer. The paper does not explicitly clarify whether these theoretical guarantees still hold when using the optimizer applied in the experiments.

Questions for authors

Could the authors explain to what extent the additional ϵ\epsilon term introduced in Algorithm S2 affects the constraints in Equation (12)?

In practical applications, is this deviation small enough to be considered negligible?

Given that the theoretical analysis is based on SGD, while the experiments employ AdamW, does the proposed gradient constraint still function as intended when used in conjunction with AdamW?

Why is gradient projection necessary, rather than simply incorporating the dependency score as a regularization loss term?

4 comments
AuthorsOpenReview ↗

Responses to Reviewer TKA3 (1)

Thank you for your careful review and constructive comments. Below are our point-by-point responses to your feedback. Changes in the revised manuscript are highlighted in blue.

W1. Influence of the linguistic bias and selective correction.

In the main paper, Table 4 quantifies source-biased errors and their correction by SDGP on test data, while Fig. 5 visualizes this behavior. To further isolate the influence of linguistic bias and SDGP’s effectiveness, we added a prefix-replacement experiment on LRS2 and LRS3 training data in Section S5.2 of the revised manuscript. Using training data provides more frequent occurrences of the selected dependencies, allowing a more targeted assessment of their influence.

Specifically, we evaluated trained Llama-AVSR models with and without SDGP under teacher forcing across both LRS2 → LRS3 and LRS3 → LRS2 directions. When a selected dependency favors an incorrect token on the meta-test split, we keep the visual input fixed and compare that token's probability under the triggering context against an equal-length, non-triggering context drawn from the meta-train transcripts.

Meta-train → meta-testModelOriginal Prefix (Triggering)Replacement Prefix (Neutral)
LRS2 → LRS3Llama-AVSR4.09%0.37%
Llama-AVSR + SDGP0.96%0.19%
LRS3 → LRS2Llama-AVSR3.81%0.24%
Llama-AVSR + SDGP0.37%0.25%

With identical visual input, the original prefix raised the baseline's mean error probability by 3.72 and 3.57 percentage points across the two shifts, confirming the adverse impact of linguistic bias under cross-domain transfer. SDGP sharply suppressed this effect, narrowing the gaps to 0.77 and 0.12 percentage points, respectively.

W2 & Q1–Q2. Effect of ϵp\epsilon_p on the projection constraint.

We have added a derivation and numerical analysis of the effect of ϵp\epsilon_p to Appendix S5.4 of the revised manuscript. In Eq. (14), when ∥h∥\|h\| approaches zero, dividing by ∥h∥2\|h\|^2 can amplify even a minor boundary violation into an excessively large gradient correction. To prevent this numerical instability, we add a constant term ϵp=10−12\epsilon_p=10^{-12} to the denominator.

On active projection steps, this modification changes the constraint margin in Eq. (12) by

(⟨h,gϵ⟩−mk)−(⟨h,g⋆⟩−mk)=−ϵpmk−⟨h,g⟩∥h∥2+ϵp.(\langle h,g_\epsilon\rangle-m_k) -(\langle h,g^\star\rangle-m_k) = -\epsilon_p\frac{m_k-\langle h,g\rangle}{\|h\|^2+\epsilon_p}.

Across active steps, the logged maximum gaps on the English benchmark were 1.19×10−111.19 \times 10^{-11} for AVEC+SDGP, 5.82×10−105.82 \times 10^{-10} for AV-RelScore+SDGP, and 3.31×10−103.31 \times 10^{-10} for Llama-AVSR+SDGP, confirming that their practical deviation is negligible.

AuthorsOpenReview ↗

Responses to Reviewer TKA3 (2)

W3 & Q3. Optimizer correction and scope of the guarantee.

Lines 380--382 incorrectly stated that we used AdamW, which was accidentally left in the manuscript from our earlier optimizer comparisons. As stated in lines 238--239, when developing this method, we used SGD as the theoretical basis because its simple update rule makes the core idea of SDGP clear and easy to follow. After deriving the projection rule, when conducting our formal experiments, we applied it directly to three widely used optimizers: SGD, Adam, and AdamW. Since their recognition results differed very little, we selected SGD for the main paper because its update rule directly matches our theoretical derivation and allows us to test the theory cleanly. To ensure a fair comparison, all experiments in Table 3 and the subsequent ablations, including all tuned baselines, were conducted using SGD. We have corrected lines 380--382 and added the full comparison with Adam and AdamW to Appendix S5.3 of the revised manuscript.

For Adam and AdamW, we compute g⋆g^\star using the projection derived in Eq. (14) and pass it to the outer optimizer. Specifically, SGD directly applies the update −ηg⋆-\eta g^\star, whereas Adam and AdamW use g⋆g^\star to update their first- and second-moment estimates before computing the actual parameter step, with AdamW additionally applying decoupled weight decay. On WildVSR, the WERs (%) for SGD / Adam / AdamW are 74.2 / 74.3 / 74.2 on AVEC, 75.4 / 75.5 / 75.5 on AV-RelScore, and 55.7 / 55.8 / 55.6 on Llama-AVSR. Across all three backbones, the differences from SGD are negligible.

For the first-order non-increase condition to hold under Adam or AdamW, the actual parameter change Δθ\Delta\theta must satisfy ⟨h,−Δθ⟩≥0\langle h,-\Delta\theta\rangle\geq 0. Among sampled Llama-AVSR steps where projection was active, this condition held for 100% of SGD updates, 85.4% of Adam updates, and 87.8% of AdamW updates. Although Adam and AdamW did not satisfy this condition at every individual step, their final recognition performance remained comparable to that of SGD. We attribute this stability in part to the smoothing effect of their running first- and second-moment estimates. Overall, the comparable WERs across optimizers further confirm the robustness of SDGP to the choice of outer optimizer.

Q4. Necessity of projection.

Table 5 in the main paper directly compares SDGP with adding the dependency score as a fixed-weight regularization term. Adding λMdep\lambda M_{\mathrm{dep}} to the recognition loss yields g+λhg+\lambda h. On WildVSR, the best tested fixed weight yielded 74.8/76.1/56.4% WER on AVEC/AV-RelScore/Llama-AVSR, compared with 74.2/75.4/55.7% for SDGP. A fixed weight applies the same coefficient in every episode. SDGP adapts the correction strength to the gradient magnitudes and their alignment, making only the change to gg required by the constraint.

To further isolate the effect of the projection rule, we applied Agr-Sum (Mansilla et al., ICCV 2021) to the same gg and hh. WildVSR WERs (%) for Agr-Sum/SDGP were 75.0/74.2 on AVEC, 77.4/75.4 on AV-RelScore, and 60.5/55.7 on Llama-AVSR. Agr-Sum treats the two gradients symmetrically. SDGP gives recognition priority by using hh to define the suppression constraint and choosing the feasible gradient closest to gg. These results support the need for our gradient projection.

We hope these clarifications and additional results address your concerns. We would appreciate your reconsideration of the score and welcome any further questions.

Dear Reviewer,

I hope this message finds you well. With less than two days remaining in the discussion period, we wanted to follow up and ensure that our rebuttal has satisfactorily addressed your questions and concerns. If you have any further feedback or additional points for discussion, please do let us know. We deeply appreciate your time, effort, and invaluable insights in helping us improve our work.

Best regards, The Authors

Dear Reviewer,

I hope this message finds you well. With the discussion period closing in less than ten hours, we wanted to gently follow up and see if our rebuttal has satisfactorily addressed your questions and concerns. If you have any further feedback or additional points for discussion, please do let us know. We deeply appreciate your time, effort, and invaluable insights in helping us improve our work.

Best regards, The Authors

Reviewer_4fFGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies domain generalization for visual speech recognition (VSR), arguing that a context–token association learned from source transcripts can become an unreliable linguistic prior in a new domain. It proposes Selective Dependency Gradient Projection (SDGP): episodic source-domain splits identify potentially unreliable n-gram dependencies, a reliance score measures the model’s probability on conflicting candidates, and a closed-form gradient projection is used to suppress updates that reinforce them. The paper also introduces LipWild-G7, a seven-genre Chinese benchmark with ten leave-three-genre-out protocols. Experiments cover three model families on LipWild-G7 and three on LRS2+LRS3 → WildVSR, comparing SDGP with several domain-generalization baselines. The paper reports lower target error with SDGP across the tested settings and includes controls for random dependency selection, fixed versus adaptive correction, occlusion, and hyperparameter sensitivity.

Critical strengths
  1. SDGP builds on episodic domain generalization but defines its auxiliary signal using context–token dependencies whose empirical support changes across source-domain splits. This is a potentially useful addition to visual-invariance approaches.

  2. The reported improvements span several model families and both Chinese and English evaluations. Table S8 shows lower target CER than the corresponding baseline across the reported LipWild-G7 protocol/model cells. The random-selection control is also informative: randomly selected dependencies perform worse than reliability-based selection, supporting the importance of which dependencies are targeted.

Critical weaknesses
  1. The evaluated modality and training protocols need clarification. The method formalizes the input as a lip video, but AVEC, AV-RelScore, and Llama-AVSR are audio-visual systems in their original descriptions. The paper should state whether audio is disabled for all reported results and confirm that methods use matched inputs. The English protocol also does not explain how episodic splits or the unreliable-dependency sets are formed from LRS2 and LRS3, which do not have the genre labels used for LipWild-G7. These details affect both task interpretation and reproducibility.

AVEC: Audio-Visual Efficient Conformer for Robust Speech Recognition, WACV 2023, https://openaccess.thecvf.com/content/WACV2023/html/Burchi_Audio-Visual_Efficient_Conformer_for_Robust_Speech_Recognition_WACV_2023_paper.html

AV-RelScore: Watch or Listen: Robust Audio-Visual Speech Recognition With Visual Corruption Modeling and Reliability Scoring, CVPR 2023, https://openaccess.thecvf.com/content/CVPR2023/html/Hong_Watch_or_Listen_Robust_Audio-Visual_Speech_Recognition_With_Visual_Corruption_CVPR_2023_paper.html

  1. The claimed projection behavior is not verified under the implemented optimizer, and the mechanism is not isolated from generic gradient projection. Equations 9–14 derive a first-order condition for a local SGD update, while Section 5.3 specifies AdamW. The paper should explain whether the projection is applied to the actual parameter-update direction; otherwise, the stated half-space constraint on the raw gradient does not directly guarantee the same change in the dependency score under AdamW. Further, the paper does not report whether MdepM_{\mathrm{dep}} decreases during training. A generic gradient-projection control would also help establish whether the reliability-based dependency signal adds value beyond gradient surgery. Existing work already studies gradient surgery for domain generalization (Mansilla et al. , ICCV 2021); SDGP should clarify its distinction from this line of work and from episodic MLDG, which it builds on (Li et al., AAAI 2018).

  2. Uncertainty and benchmark documentation remain incomplete. The paper reports no independent-run variability for the headline comparisons, making it difficult to assess the robustness of the smaller target CER gains. For LipWild-G7, Sections 4 and S4 describe collection from public video platforms and manual checking, but do not clearly state whether or how the dataset will be released, its licensing or permissions, or the consent/legal basis and safeguards for distributing identifiable face videos and transcripts.

Questions for authors
  1. Were audio inputs disabled for AVEC, AV-RelScore, and Llama-AVSR in all experiments? Please specify each model’s input modalities and confirm that all compared methods received matched inputs.

  2. How are the episodic train/test partitions and unreliable-dependency sets constructed for the LRS2+LRS3 → WildVSR experiments? Please also clarify whether gradients are differentiated through the virtual update θ′\theta', and how candidate probabilities are computed for CTC and hybrid architectures.

  3. Since the projection derivation assumes a local SGD update but the implementation uses AdamW, what update direction is actually projected? Can the authors report the dependency score before and after representative training updates and compare SDGP against a generic gradient-projection control?

  4. How many independent training runs were used for the headline results, and can run-to-run variability be reported? What is the planned release and governance arrangement for LipWild-G7, including the data types shared, licensing or permissions, and consent or other legal basis?

4 comments
AuthorsOpenReview ↗

Responses to Reviewer 4fFG (1)

We thank the reviewer for the careful reading and insightful comments. Below are our point-by-point responses, with all revisions highlighted in blue in the manuscript.

W1 & Q1. Input modalities.

All experiments use video only. For audio-visual methods, we disable audio inputs and feed the visual encoder output to the recognition decoder. As specified in Section 5.2, MultiVSR uses full-face video to diversify the evaluated methods; the others use lip video. Baselines and DG variants use identical inputs within each backbone.

W1 & Q2. English training protocol.

For English, LRS2 (BBC broadcasts) and LRS3 (TED/TEDx talks) serve as the two source domains. Each episode uses one for meta-training and the other for meta-testing, with both directions used during training. We construct the unreliable-dependency sets as in the Chinese experiments, using subword tokens for English.

Q2. Gradient computation and CTC/hybrid scoring.

We take one gradient step on the meta-train loss to obtain θ′\theta' from θ\theta. The episodic gradient gg sums the meta-train recognition gradient at θ\theta and the meta-test recognition gradient at θ′\theta'. We also compute the dependency-score gradient hh on the same meta-test batch at θ′\theta'. As in DomainBed’s MLDG implementation (Gulrajani & Lopez-Paz, 2021), we omit second-order derivatives of the meta-train loss.

For CTC, we force-align the reference transcript to locate the non-blank frames for each target token, and then average the candidate's probabilities over these frames. For hybrid models, we sum the CTC dependency score and the teacher-forced seq2seq dependency score.

W2 & Q3. Optimizer implementation, dependency scores, and gradient control.

Lines 380--382 incorrectly listed AdamW, a setting retained from our earlier optimizer comparisons. When developing this method, we derived SDGP under SGD because its update makes the projection rule explicit (lines 238--239). After testing SGD, Adam, and AdamW, we selected SGD for the main experiments because all three gave comparable recognition results and SGD directly matches the derivation. All results in Table 3 and the subsequent ablations, including the tuned baselines, used SGD. We corrected lines 380–382 and report the optimizer comparison in Appendix S5.3.

Eq. (14) projects the episodic recognition gradient gg to obtain g⋆g^\star. SGD directly applies Δθ=−ηg⋆\Delta\theta=-\eta g^\star. AdamW takes the same g⋆g^\star as input, updates its first- and second-moment estimates, and applies decoupled weight decay. On WildVSR, WERs (%) for SGD/Adam/AdamW were 74.2/74.3/74.2 on AVEC, 75.4/75.5/75.5 on AV-RelScore, and 55.7/55.8/55.6 on Llama-AVSR. Across all three backbones, the reported WERs differed from SGD by at most 0.1 percentage points.

For Llama-AVSR, 87.8% of sampled AdamW updates with active projection satisfied $ \langle h,-\Delta\theta_t\rangle \ge0$, indicating a first-order non-increase in MdepM_{\mathrm{dep}}. When we measured MdepM_{\mathrm{dep}} on the same episode before and after each update, the observed decrease rate was within 0.5 percentage points of 87.8%.

For Llama-AVSR+SDGP trained with SGD, ten equally spaced points along the 60-epoch training trajectory gave epoch-averaged dependency scores of 0.0140, 0.0076, 0.0032, 0.0028, 0.0023, 0.0023, 0.0020, 0.0018, 0.0016, and 0.0017, showing a sustained overall reduction.

To isolate our projection rule, we applied Agr-Sum (Mansilla et al. , ICCV 2021) to the same gg and hh. WildVSR WERs (%) for MLDG/Agr-Sum/SDGP were 75.3/75.0/74.2 (AVEC), 77.9/77.4/75.4 (AV-RelScore), and 59.8/60.5/55.7 (Llama-AVSR). MLDG optimizes episodic recognition alone, while Agr-Sum treats both gradients symmetrically and zeros conflicting components. SDGP prioritizes recognition by suppressing unreliable dependencies with the smallest required change to gg.

AuthorsOpenReview ↗

Responses to Reviewer 4fFG (2)

W3 & Q4. Run-to-run variability and benchmark release.

In our preliminary experiments, we evaluated multiple random seeds across several configurations and observed negligible variance across runs, which motivated the use of a fixed seed for subsequent experiments. To further assess SDGP’s sensitivity to random initialization, we re-evaluated Baseline, MLDG, and SDGP across five matched seeds per setting using 64 RTX 4090 GPUs. The following table reports the mean ±\pm sample standard deviation alongside the paired 95% confidence intervals (CIs) for SDGP’s error reduction relative to MLDG on the Chinese benchmark. Notably, SDGP outperforms both Baseline and MLDG across every individual seed, demonstrating that its performance gains are statistically robust and insensitive to seed choice.

MethodAVECAV-RelScoreMultiVSR
Baseline84.83 ± 0.1188.76 ± 0.0788.59 ± 0.20
+MLDG84.43 ± 0.0787.75 ± 0.1187.66 ± 0.10
+SDGP83.94 ± 0.0985.66 ± 0.1086.85 ± 0.10
Gain over MLDG (pp) [paired 95% CI]0.49 [0.36, 0.62]2.09 [1.87, 2.31]0.81 [0.75, 0.87]

We will release cropped lip videos, synchronized audio, transcripts, genre labels, and standardized dataset splits after verifying source provenance and applicable distribution permissions. Access requires a signed agreement restricting usage to non-commercial academic research and prohibiting redistribution. Furthermore, speakers and rights holders may contact the corresponding author to request data removal on privacy or copyright grounds. We have added these release protocols and safeguards in Appendix S4.5 of the revised manuscript.

We hope these detailed responses and additional evaluations adequately address your concerns. We would be grateful if you would consider reassessing your score, and we remain available for any further discussion.

Dear Reviewer,

Thank you once again for your constructive feedback. With less than two days left in the discussion period, we would love to know whether our responses have adequately resolved your concerns. We remain fully available for any further discussion or questions you may have. Thank you for your continued time and effort in reviewing our manuscript.

Best regards, The Authors

Dear Reviewer,

Thank you again for your thoughtful review. With less than ten hours remaining in the discussion period, we wanted to gently follow up and ask whether our rebuttal has satisfactorily addressed your questions and concerns. If you have any further feedback or additional points for discussion, please do let us know. We deeply appreciate your time, effort, and invaluable insights in helping us improve our work.

Best regards, The Authors

Reviewer_8zXjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper re-examines domain generalization in visual speech recognition from the perspective of linguistic dependency reliability. The authors argue that the context–token associations learned by lipreading models to compensate for visual ambiguity may become misleading priors when transferred across domains. The proposed SDGP first selects potentially unreliable dependencies based on conditional frequency ratios over source-domain internal meta-train/meta-test splits, then constructs an auxiliary objective from the probabilities of error candidates triggered by these dependencies, and finally adjusts episodic recognition gradients via a closed-form half-space projection. The paper also introduces a Chinese dataset, LipWild-G7, containing seven content categories, and validates the method on Chinese cross-category tests and on transfer tests from English LRS2/LRS3 to WildVSR. The problem is of research value, and the reported performance gains are noteworthy. However, I have substantive concerns regarding the mechanistic explanation of the auxiliary objective, the consistency between the theoretical constraints and the actual optimizer, and whether the experiments sufficiently identify the sources of the method’s gains.

Critical strengths

S1. The research motivation is clear and meaningful.

The paper points out that visual distribution invariance cannot cover all sources of cross-domain failure in lipreading. Considering that visual speech is ambiguous, investigating the role of linguistic context preferences under domain shift is a reasonable and important direction.

S2. The method is clearly structured, and the projection subproblem has a concise solution.

Dependency screening, auxiliary scores, and gradient correction together form a relatively complete training framework. Under the SGD surrogate and non-degeneracy conditions defined in the paper, the half-space projection solution in Proposition 1 and its KKT derivation are largely valid.

S3. The experiments cover multiple model architectures and two languages.

The paper evaluates CTC, hybrid, and Seq2Seq/LLM-based models. In Table 3, SDGP improves target-domain error under all reported settings, and combining it with MixStyle yields further gains, suggesting that this training strategy may have certain applicability and complementarity.

S4. The new benchmark has potential value.

LipWild-G7 provides explicit genre labels, multiple source–target combinations, and a relatively detailed data processing pipeline. If data access, identity deduplication, and split auditing are sufficiently documented, this benchmark is expected to support future cross-domain lipreading research.

Critical weaknesses

W1. The auxiliary objective cannot yet effectively identify the model’s degree of reliance on linguistic dependencies.

Eq. (8) computes the average probability that the full recognition model assigns to the selected error candidates. This probability is affected not only by linguistic context but also by visual confusion, alignment errors, model calibration, and other factors. Therefore, its gradient can be clearly interpreted as “the gradient of the probability of the selected error candidates,” but it cannot be directly interpreted as “the direction for reinforcing non-transferable linguistic shortcuts.” Similarly, reducing this score does not automatically mean that shared linguistic dependencies have been preserved. The paper does not impose explicit protection constraints on shared dependencies or related predictive behavior. The current mechanistic narrative is clearly stronger than what the actual measurement can support.

W2. There is a gap between the theoretical properties of gradient projection and the actual AdamW update.

Eqs. (9)–(14) are based on the update θ−ηg⋆\theta-\eta g^\star, so the constraint ⟨h,g⋆⟩≥0\langle h,g^\star\rangle\ge0 corresponds to first-order non-increase of the auxiliary objective under the SGD surrogate. However, the experiments use AdamW, whose actual parameter update involves momentum, per-coordinate scaling, and weight decay, and is generally not collinear with g⋆g^\star. An inner-product constraint in the raw gradient space does not guarantee that the actual update step satisfies the same condition. The authors explicitly restrict the derivation to a local SGD surrogate, but they have not verified whether this surrogate property reflects actual training behavior. The SGD update in Algorithm S2 and the experimental description also need to be unified. In addition, after adding ϵp\epsilon_p to the denominator in the pseudocode, the original projection boundary is usually no longer satisfied exactly. The theoretical conclusions should be further qualified or supported by diagnostics of the actual updates.

W3. The dependency reliability estimation does not sufficiently distinguish distribution shift from insufficient evidence.

Eq. (4) requires the meta-train joint count to reach a threshold, but does not require the meta-test context to have sufficient support. When Nte(c)=0N_{te}(c)=0, the smoothed conditional probability is determined entirely by the size of the candidate set rather than by meta-test observational evidence. Several such dependencies in Table S4 are still treated as valid instances for assessing the mismatch criterion. These contexts usually do not trigger penalties in the corresponding meta-test samples, so the problem may not directly manifest as erroneous updates; however, it weakens the interpretation of the mismatch statistics and “reliability.” Moreover, the fact that an association has different frequencies between two known source domains does not directly prove that it is non-transferable to an unknown target domain. The paper needs to discuss this extrapolation assumption and estimation uncertainty more explicitly.

W4. The existing ablations are insufficient to determine whether the gains come from the emphasized projection mechanism.

Table 5 shows that auxiliary gradient correction with a fixed coefficient already yields substantial gains, but the paper does not adequately compare alternatives such as gradient normalization, adaptive auxiliary loss weighting, and a non-increase-only constraint. In particular, the key control k=0k=0 is missing, as is a same-candidate penalty experiment without the virtual update. The random dependency control only matches the number of dependencies, but not the actual trigger frequency, candidate prediction probabilities, or gradient strength; its significant degradation is therefore insufficient to independently demonstrate the superiority of the proposed criterion. The existing results support the effectiveness of this combined strategy, but cannot clearly separate the respective contributions of dependency selection, auxiliary penalty, meta-learning, and projection.

W5. The mechanistic experiments mainly provide correlational evidence and have not yet ruled out the explanation of general recognition improvement.

The source-biased errors in Table 4 are defined by source-domain frequency, which is related to the screening logic adopted by the method. General recognition improvement may also reduce such errors, so it is necessary to also report non-source-biased errors and cases in which the use of correct dependencies is affected. Fig. 5 is based on teacher forcing and cannot directly indicate dependency behavior in free decoding; lip occlusion also simultaneously changes visual quality, prediction uncertainty, and decoding behavior, and thus cannot isolate the linguistic dependency mechanism. The current experiments are compatible with the authors’ explanation, but they cannot yet distinguish it from ordinary negative-candidate regularization or calibration improvement.

W6. The benchmark splits and the target-domain usage protocol need to be more transparent.

The paper claims that speaker identities do not overlap across genres, but the data pipeline does not sufficiently explain how identities are verified across videos and programs. Audio SHA-256 deduplication also cannot rule out near-duplicate content from re-encoding or re-editing. In addition, Table S7 shows hyperparameter sensitivity on WildVSR, and the default values are all the optima of the corresponding one-factor experiments. This by itself does not prove that the target test set was used for hyperparameter tuning, but a clear distinction must be made between post hoc sensitivity analysis and actual hyperparameter selection. Data licensing, redistribution conditions, and opt-out/removal mechanisms should also be supplemented.

Questions for authors

Q1. On mechanism identification:

What evidence do the authors have to distinguish whether the error-candidate probabilities in Eq. (8) arise from linguistic priors or from visual confusion? Can they provide diagnostics that change the context while holding visual evidence approximately constant, or other diagnostics capable of distinguishing these two explanations?

Q2. On the actual optimization constraint:

For the actual AdamW parameter updates, what proportion satisfy ⟨h,−Δθ⟩≥0\langle h,-\Delta\theta\rangle\ge0? What proportion of cases show an actual decrease in the auxiliary score after projection? Does the implementation use the full second-order meta-gradient or a first-order approximation?

Q3. On decomposition of the method’s contributions:

What are the results for the following controls: k=0k=0, normalized auxiliary gradients, a sufficiently tuned auxiliary loss weighting, and the same-candidate penalty after removing the episodic virtual update? These comparisons are crucial for judging the independent value of the projection.

Q4. On reliability estimation:

Among the selected dependencies, how many have zero or extremely low meta-test context support? How many actually trigger training penalties? After restricting meta-test support or accounting for estimation uncertainty, are the selected set and performance stable?

Q5. On selectivity and side effects:

What effect does SDGP have on non-source-biased errors and on recognition performance for samples where the selected dependencies remain correct? Is there negative transfer when the target domain happens to support the source-domain preferences?

Q6. On experimental and data transparency:

Please specify model initialization, pretraining data, freezing strategy, training budget for each method, and the hyperparameter selection procedure, and explain how speaker non-overlap across genres and exclusion of cross-video near-duplicates are verified. Can multi-random-seed results and paired uncertainty estimates be provided?

5 comments
AuthorsOpenReview ↗

Responses to Reviewer 8zXj (1)

We thank the reviewer for the careful reading and insightful comments. Below are our point-by-point responses, with all revisions highlighted in blue in the revised manuscript.

W1 & Q1. Linguistic bias and error-candidate probabilities.

Section 3.2 identifies unreliable dependencies solely from transcripts based on domain-specific frequency shifts, completely independent of visual features. Consequently, candidate targeting is explicitly driven by linguistic bias rather than visual confusion. Shared dependencies outside Rte\mathcal{R}^{te} receive no penalty, preserving them relative to suppressed pairs. On test data, Table 4 quantifies source-biased errors and their correction by SDGP, while Fig. 5 illustrates this behavior.

To isolate the contribution of linguistic context, we added a prefix-replacement experiment in Section S5.2 of the revised manuscript. We used LRS2 and LRS3 training utterances because the selected dependencies occur more frequently there. In both transfer directions, we evaluated trained Llama-AVSR models with and without SDGP under teacher forcing. Whenever a selected dependency favored a token that differed from the meta-test reference, we held the video fixed and compared the probability of that incorrect candidate under the triggering prefix and an equal-length, non-triggering replacement drawn from the meta-train transcripts.

Meta-trainMeta-testModelTriggering prefixNon-triggering prefix
LRS2LRS3Llama-AVSR4.09%0.37%
LRS2LRS3Llama-AVSR+SDGP0.96%0.19%
LRS3LRS2Llama-AVSR3.81%0.24%
LRS3LRS2Llama-AVSR+SDGP0.37%0.25%

With identical visual evidence, the triggering prefix raised the baseline’s mean probability of the source-favored incorrect candidate by 3.72 percentage points on LRS2 → LRS3 and 3.57 on LRS3 → LRS2. SDGP reduced these gaps to 0.77 and 0.12 percentage points, respectively, showing that it attenuates the effect of source-domain linguistic bias in both directions.

AuthorsOpenReview ↗

Responses to Reviewer 8zXj (2)

W2 & Q2. Optimizer updates, first-order approximation, and effect of ϵp\epsilon_p.

Lines 380--382 erroneously mentioned AdamW, a setting retained from our initial optimizer comparisons. We originally derived SDGP with SGD because its simple update makes the projection rule explicit (lines 238--239). In formal testing, SGD, Adam, and AdamW yielded comparable WERs, so we adopted SGD for the main paper to match our derivation directly. Every experiment in Table 3 and the following ablations, including tuned baselines, used SGD. We have revised lines 380--382 and included the comparison across optimizers in Appendix S5.3.

WildVSR WERs (%) for SGD/Adam/AdamW were 74.2/74.3/74.2 on AVEC, 75.4/75.5/75.5 on AV-RelScore, and 55.7/55.8/55.6 on Llama-AVSR. Across all three backbones, switching outer optimizers changed the reported WER by at most 0.1 percentage points.

We use the same first-order approximation as DomainBed’s MLDG implementation (Gulrajani & Lopez-Paz, 2021). We compute the meta-test recognition and dependency-score gradients at θ′\theta' and pass both back to θ\theta, omitting second-order derivatives of the meta-train loss.

Eq. (14) gives the projected gradient g⋆g^\star. SGD applies Δθ=−ηg⋆\Delta\theta=-\eta g^\star directly. Adam and AdamW take g⋆g^\star as input to their first- and second-moment updates, and AdamW additionally applies decoupled weight decay. We therefore checked whether each optimizer’s actual parameter change satisfied the first-order non-increase condition ⟨h,−Δθ⟩≥0\langle h,-\Delta\theta\rangle\geq 0. Among sampled Llama-AVSR steps where projection was active, this condition held for 100% of SGD, 85.4% of Adam, and 87.8% of AdamW updates. On the same AdamW steps, the fraction for which MdepM_{\mathrm{dep}} actually decreased, measured on the same episode before and after each update, was within 0.5 percentage points of 87.8%.

A detailed derivation along with a numerical analysis of ϵp\epsilon_p's impact has been added to Appendix S5.4 of the revised manuscript. In Eq. (14), dividing by ∥h∥2\|h\|^2 when ∥h∥→0\|h\|\to0 risks blowing up the gradient correction from minor boundary violations. For numerical stability, we add ϵp=10−12\epsilon_p=10^{-12} to the denominator. On active projection steps, this introduces a slight residual gap relative to the exact constraint margin in Eq. (12):

(⟨h,gϵ⟩−mk)−(⟨h,g⋆⟩−mk)=−ϵp mk−⟨h,g⟩∥h∥2+ϵp.(\langle h,g_\epsilon\rangle-m_k)-(\langle h,g^\star\rangle-m_k) =-\epsilon_p\,\frac{m_k-\langle h,g\rangle}{\|h\|^2+\epsilon_p}.

Over all active steps on the English benchmark, the maximum recorded residual gaps are negligible: 1.19×10−111.19 \times 10^{-11} (AVEC+SDGP), 5.82×10−105.82 \times 10^{-10} (AV-RelScore+SDGP), and 3.31×10−103.31 \times 10^{-10} (Llama-AVSR+SDGP), demonstrating that the perturbation has virtually no impact in practice.

W3 & Q4. Reliability estimation and meta-test support.

We require sufficient meta-train support to identify strong linguistic preferences and use meta-test frequencies for comparison. When a context is absent from meta-test, its estimated probability comes entirely from smoothing. The dependency can still be selected, but it never triggers Eq. (7) in that split.

On LRS2 → LRS3, 82 of the 86 selected dependencies triggered Eq. (7) at least once; four never appeared in the meta-test context, and nine appeared only 1--4 times, accounting for just 23 of 47,772 eligible triggers (0.05%). Similarly, on LRS3 → LRS2, 131 of the 157 selected dependencies triggered Eq. (7) at least once, while 24 had zero context occurrences and 11 appeared 1--4 times, accounting for only 18 of 28,249 eligible triggers (0.06%).

Requiring at least five meta-test context occurrences had no observable effect on recognition performance. Table 4 empirically confirms that these source-favored dependencies induce target-domain errors, which SDGP effectively corrects.

AuthorsOpenReview ↗

Responses to Reviewer 8zXj (3)

W4 & Q3. Contribution of the projection rule.

On WildVSR, Llama-AVSR achieved 56.8% WER with the non-increase-only constraint (k=0k=0), 58.0% with normalized auxiliary gradients, and 58.3% when penalizing the same candidates without the virtual update. Full SDGP achieved 55.7%. Table 5 also shows that SDGP outperformed the best tested fixed auxiliary-loss weight on all three backbones.

To isolate the projection rule, we applied Agr-Sum (Mansilla et al., ICCV 2021) and SDGP to the same recognition gradient gg and dependency gradient hh. WildVSR WERs (%) for MLDG/Agr-Sum/SDGP were 75.3/75.0/74.2 on AVEC, 77.9/77.4/75.4 on AV-RelScore, and 59.8/60.5/55.7 on Llama-AVSR.

W5 & Q5. Non-source-biased errors and correct dependencies.

We extended the WildVSR analysis in Table 4 to non-source-biased substitutions. MixStyle and SDGP corrected 2.1% and 3.6% of these baseline errors, respectively. For source-biased substitutions (n=2n=2), MixStyle corrected 105 of 1,049 baseline errors (10.0%), and SDGP corrected 232 (22.1%). SDGP’s advantage over MixStyle was therefore larger for source-biased substitutions.

We also evaluated whether SDGP degrades performance when the source-domain preference is actually valid. On the WildVSR subset where the ground-truth transcript indeed contains source-favored dependencies, SDGP achieves 49.0% WER compared to 49.2% for the baseline, confirming that it causes no performance drop on such cases.

W6 & Q6. Experimental and benchmark transparency, random seeds.

For each backbone, we followed the original implementation for initialization, pretraining data, and parameter freezing. All methods used SGD and the same training budget (Section 5.3). We selected checkpoints by the lowest error rate on the source-domain validation set. For SDGP hyperparameters, we varied one parameter at a time and selected the settings with the lowest WildVSR WER (Table S7).

During collection, we excluded duplicate uploads of the same program content across accounts. We manually checked speaker identities across videos and programs when constructing the genre splits. After segmentation, we used SHA-256 hashes of the audio to remove exact duplicate clips.

We will distribute the cropped lip videos, synchronized audio, transcripts, genre labels, and standardized splits after verifying source provenance and distribution permissions. Access requires a signed agreement that restricts usage to non-commercial academic research and prohibits redistribution. In addition, speakers and rights holders can contact the corresponding author to request removal. These release policies and safeguards are detailed in Appendix S4.5 of the revised manuscript.

During preliminary testing, running multiple random seeds across several configurations showed negligible variance across runs, which led us to fix the random seed in subsequent experiments. To rigorously examine SDGP's sensitivity to initialization, we re-ran Baseline, MLDG, and SDGP over five matched seeds per setting on 64 RTX 4090 GPUs. The table below presents the mean ±\pm sample standard deviation together with the paired 95% confidence intervals (CIs) for SDGP's error reduction over MLDG on the Chinese benchmark. SDGP achieved lower CER than both Baseline and MLDG at all five seeds on each of the three LipWild-G7 backbones, confirming that its improvements are statistically reliable and unaffected by seed selection.

MethodAVECAV-RelScoreMultiVSR
Baseline84.83 ± 0.1188.76 ± 0.0788.59 ± 0.20
+MLDG84.43 ± 0.0787.75 ± 0.1187.66 ± 0.10
+SDGP83.94 ± 0.0985.66 ± 0.1086.85 ± 0.10
Gain over MLDG (pp) [paired 95% CI]0.49 [0.36, 0.62]2.09 [1.87, 2.31]0.81 [0.75, 0.87]

We hope these clarifications and new experiments fully resolve your concerns. In light of these updates, we kindly ask you to reconsider your rating, and we remain happy to address any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.