acceptodds
Under review as a conference paper at ICLR 2027

Align Then Reason: A Multimodal Lip-Sync Judge for Dubbing

Abstract

Dubbing quality control requires a reference-free judge that can determine whether a candidate text line matches a speaker's visible articulation in both content and timing, using only silent video and text because dubbed audio may not yet exist. Existing visual speech recognizers and video-language models are poorly suited to this setting: even when fine-tuned to recover spoken content from lip motion, they remain largely insensitive to temporal errors. We introduce (ATR), a multilingual lip-sync judge that first establishes a monotonic alignment between frame-level lip representations and the phonetic units of the candidate line, then reasons over this alignment to make the final judgment. An alignment scorer provides the LLM with both local evidence for each phonetic unit and a calibrated global alignment score, enabling it to reason jointly about content and timing. On a seven-language benchmark, our method improves mean AUC over the corresponding Qwen3.5 SFT baselines by 59.4%, 50.2%, and 50.8% with 2B, 4B, and 9B reasoners, respectively. The gains generalize across LLM families, reaching mean AUC improvements of 45.9% and 46.6% over the best baseline for LLaMA-3.1-8B and Mistral-7B, respectively. They also transfer across datasets to three unseen MuAViC languages. Furthermore, we evaluate on two downstream tasks built from real dubbing lines. On dub-line reranking, ATR-9B outperforms the best lip-reading baseline by 52.0%, while on script-to-clip assignment, ATR-9B improves over the best lip-reading baseline by 17.7%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Lip-sync evaluation, multimodal judge

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_jCzpOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces Align Then Reason (ATR), a multilingual judge that evaluates whether candidate dubbing text matches a speaker’s lip movements using only silent video and text. ATR combines a CTC-based monotonic alignment scorer with an LLM that receives local phonetic evidence and a global alignment score. Experiments show substantial improvements over speech-recognition and video-language baselines across seven languages, unseen languages, and two downstream dubbing tasks.

Critical strengths
  1. The approach is technically sound, with a well-motivated monotonic alignment mechanism and comprehensive ablations that clarify the contributions of the alignment scorer, phonetic supervision, and LLM reasoner.

  2. The evaluation spans multiple languages, LLM backbones, and downstream applications. The substantial gains on unseen languages and a different dataset provide encouraging evidence of transferability, particularly after recalibration.

Critical weaknesses
  1. The use of dubbed translations as negatives is practical, but it remains unclear how well this training objective captures relative lip-sync quality among plausible dubbing candidates.
  2. The temporal evaluation mainly uses synthetic corruptions such as frame reversal, freezing, and swapping, with the same corruption families used during training. These may not adequately represent subtle dubbing errors, such as small local offsets, misplaced pauses, or gradual timing drift, leaving generalization to realistic synchronization problems insufficiently established.
Questions for authors
  1. Have the authors evaluated ATR on unseen temporal perturbations or real dubbing errors, particularly small local offsets and gradual timing drift? Such results would help establish whether its temporal sensitivity extends beyond the corruption types used during training.
1 comment

We thank the reviewer for the positive assessment of of our approach being technically sound, well-motivated with comprehensive ablations, the evaluation spans multiple languages, LLM backbones, and downstream applications, the substantial gains on unseen languages and a different dataset provide encouraging evidence of transferability. We address the concerns in detail below and will incoporate all feedback into the final version. Hope you could raise the score.

W1. Relative lip-sync quality among plausible dubbing candidates

In training the dub line is only a relative label: the aligner's margin (Eq. 2) asks that, for the same clip, the spoken line outscore its translation, and the final score is continuous, so candidates are ranked by degree of fit rather than classified as original-or-not. The direct test of relative quality among plausible candidates is the following dub-line reranking: the four candidates are meaning-preserving, length-matched target-language lines and none is the original, so recognizing the original script would score at chance (0.250), whereas ATR-9B reaches 0.479. And the preference disappears when the same candidates are scored against a different clip (Top-1 → 0.271, close to chance). So the objective separates good from near-miss dub lines, not just originals from translations.

Dub-line reranking.

MethodTop-1MRR
Chance0.2500.521
LLaMA-AVSR, best lip-reading baseline (paper)0.3160.575
ATR-9B scored against a mismatched clip (new)0.2710.537
ATR-9B (paper)0.4790.684

W2/Q1. Unseen and subtle temporal errors

We evaluate the cases raised, including small local offsets, misplaced pauses, and gradual drift. We show results in the following table. ATR's AUC rises smoothly with offset magnitude, from near chance at a 1-frame offset (0.561) to 0.937 at 8 frames, while the recognizer and video-language baselines stay near 0.5 throughout; this monotonic degradation is the desirable behavior for a lip-sync metric, since a 1-frame offset is genuinely near-imperceptible. ATR also detects misplaced pauses (0.796) and gradual drift (0.742).

Evaluation on temporal perturbations never used in training.

Method1f2f3f5f8fMisplaced pauseGradual drift
Auto-AVSR0.5120.5210.5340.5580.6020.5610.547
Qwen3.5-9B SFT0.5030.5050.5080.5110.5170.5090.504
ATR-9B0.5610.6430.7290.8610.9370.7960.742
Reviewer_7MUtOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a soft-prompt method to judge lip synchronization for silent videos. The authors train a model on paired texts and phonetic units to generate soft prompts, which are used to prompt LLMs for lip-sync judgments.

Critical strengths
  1. Reference-free lip-sync quality control from silent video and candidate text (before audio synthesis) is a practical and well-motivated problem for automated video dubbing.

  2. The two-stage pipeline achieves strong results across multiple languages, multiple LLM backbones, and downstream tasks.

Critical weaknesses
  1. Applying this two-stage "CTC phoneme aligner → LLM verifier" setup specifically to reference-free dubbing quality control is a practical application contribution, but the novelty is incremental. The individual components closely follow prior work in text-phoneme alignment (e.g., XPhoneBERT) and multimodal soft-prompting.

  2. The evaluation is not fair. To the best of my knowledge, the visual speech recognition (VSR) baselines are pre-trained on English only. However, the authors' model is trained on a multilingual dataset, which naturally performs better in multilingual settings.

Questions for authors

Please refer to Weaknesses.

2 comments

We thank the reviewer for noting the practical, well-motivated problem and the strong results across languages, backbones, and downstream tasks. We address the concerns in detail below and will incoporate all feedback into the final version. Hope you could raise the score.

W1. Novelty

We agree that CTC, phonetic representations, and multimodal soft-prompting are established techniques. However, our contribution is not simply combining these components, but designing an alignment and reasoning framework that addresses the content and temporal limitations of existing lip-reading and video-language models.

Our paper differs from standard phoneme alignment and standard soft-prompting in three concrete ways. First, candidate-position CTC: unlike standard CTC and forced alignment, whose outputs are a fixed phoneme/character vocabulary from a learned classifier, our labels are positions in the candidate line and the emissions are similarities to that line's own unit embeddings, so one scorer serves any language, it is trained contrastively on the CTC score itself, and it needs no alignment labels, XPhoneBERT performs no alignment and only embeds the units. Second, the evidence interface passes one soft token per phonetic unit of the candidate plus a global monotonic-alignment score. This differs from generic multimodal soft-prompting, where temporal correspondence is not explicitly enforced. Third, an empirical finding we are not aware of in prior work: used as lip-sync judges, previous recognizers, video-language models, and current frontier models are all near chance on temporal order when text is held fixed (Tables 1–2), while our approach can largely improve the temporal modeling.

W2. Fairness of the evaluation

We conduct additional experiments to train a recognizer on the same multilingual data with the same visual encoder (table below), it still trails ATR by a wide margin (0.812 vs 0.920 mean), which shows the advantages of our approach. And ATR is built on the same frozen English-pretrained Auto-AVSR visual encoder as the baseline, it adds no multilingual visual pretraining; its cross-lingual ability comes from the phonetic text side, where position labels replace a language-specific vocabulary.

Same multilingual data, same visual encoder (7-language AUC).

ModelMismatchShuffleDubReverseShiftFreezeSwapMean
Multilingual phoneme-CTC recognizer, our data, same encoder (new)0.6710.7190.6620.9220.8780.9310.9040.812
ATR-9B (paper)0.7860.9740.8620.9640.9310.9700.9520.920
Reviewer_7MUtOpenReview ↗

Thank you

Thank you for your response. Unfortunately, I remain unconvinced, and I have to keep my score unchanged. The justification provided is insufficient. While the proposed framework represents a reasonable combination of existing components, it remains largely incremental and relies heavily on engineering choices.

Reviewer_a7VtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies judging the degree of lip-sync match based solely on silent video and candidate text, without candidate dubbing audio or a reference transcript. The authors propose a two-stage framework, ATR: in the first stage, CTC with candidate phoneme positions as labels is used to establish monotonic visual–phoneme alignment, trained with content and temporal negatives; in the second stage, global alignment scores and local soft tokens are fed into an LLM to output a match judgment and a brief explanation. The authors report notable performance improvements on a seven-language benchmark, MuAViC cross-domain evaluation, and two downstream tasks. However, the main experiments have not sufficiently distinguished original script recognition, synthetic perturbation detection, and real dubbing lip-sync quality evaluation. Unequal baseline training, insufficient validity of downstream labels, and the lack of targeted verification of the LLM’s role make the paper’s core conclusions exceed what the current evidence can support.

Critical strengths

S1. The research problem has clear practical value.

Evaluating the compatibility between candidate lines and visible articulation before speech synthesis is a problem worth studying. It is distinct from conventional visual speech recognition and audio-visual synchronization detection, and has potential practical uses.

S2. Monotonic alignment has a reasonable structural motivation.

Converting candidate text into a phoneme representation and establishing video–text correspondence through monotonic path constraints is more targeted than relying directly on a general video-language model for holistic judgment. Candidate-position CTC is also a design choice worth further analysis.

S3. The experimental coverage is relatively broad.

The paper includes multilingual settings, multiple LLM backbones, component ablations, cross-domain evaluation, and downstream experiments. Table 4 is particularly valuable because it reveals differences in how global alignment scores, local representations, and the LLM contribute across different metrics.

Critical weaknesses

W1. There is a construct-validity gap between the core task and the evaluation labels.

In Sec 2.2 and Sec 3.1, the source-language original script is treated as positive examples, and the dubbed translation is uniformly treated as negative examples. This can evaluate whether the candidate text is close to the actual articulation in the video, but it is not directly equivalent to evaluating whether cross-lingual dubbing has good lip-sync fit. Professional dubbing usually needs to satisfy semantic, naturalness, duration, and visible-articulation constraints simultaneously, and is not required to match the original script phoneme by phoneme. The current label design may reward original-language or original-transcript recognition rather than dubbing-quality judgment. The paper needs to clearly distinguish these two goals and provide independent annotations of dubbing lip-sync quality.

W2. The claim of “temporal synchronization” exceeds what the current inputs and tests can support.

The candidate text does not contain pronunciation onset times, phoneme durations, or pause placement, so it cannot uniquely determine whether an actual dubbing track is too early, delayed, or locally desynchronized. The same text can correspond to multiple audio realizations with different degrees of synchronization. The current model is better interpreted as estimating the latent alignment compatibility between text and video, rather than evaluating the synchronization error of a specific realization. At the same time, reversed, cyclically shifted, frozen, and swapped clips may introduce obvious video anomalies. Because training and testing use the same types of perturbations, the model may rely partly on motion anomalies or splice boundaries rather than on the correspondence between candidate text and lip movement. The lack of control experiments such as video-only and candidate-text substitution means that this alternative explanation has not yet been ruled out.

W3. The baseline comparison is insufficient to isolate the method’s own contribution.

ATR is trained with content and temporal negatives tailored to the final task, whereas the Qwen SFT baseline is mainly trained to recover the script. Such a comparison changes both the training objective and the architecture, so the performance difference cannot be attributed to the monotonic alignment design on this basis. In addition, Appendix D acknowledges that some recognition baselines are limited by non-English vocabularies and tokenizers. Therefore, the overall multilingual advantage also mixes differences in language coverage. It is necessary to compare a direct matching classifier, standard phoneme CTC, and other monotonic alignment methods under the same data, negatives, and training objectives.

W4. The necessity of the LLM and the “reasoning” interpretation have not yet been established.

In Table 4, the AUCs of the aligner without the LLM on reverse, shift, freeze, and swap are 0.970, 0.935, 0.971, and 0.962, respectively, all slightly higher than the corresponding results of the full model. Therefore, the current evidence cannot show that the LLM improves temporal judgments. The LLM’s gains mainly come from content metrics, especially shuffle, but such negatives may be identifiable solely from text fluency. The lack of a text-only baseline and comparison with a lightweight classifier using the same alignment evidence makes it impossible to determine whether the gains come from visual-evidence reasoning, language priors, or model capacity.

W5. The groundedness of local evidence and explanations lacks verification.

The local representations in Sec 2.3 come from independent attention pooling over the similarity matrix and maximum similarity, rather than from the posterior alignment of the CTC monotonic path. Therefore, the best local match for a phoneme can appear at a position that violates the global order. The explanation supervision is in turn automatically generated from the position with the lowest local match score. This is a heuristic pseudo-label rather than verified true error localization. Without localization accuracy, explanation faithfulness, or intervention experiments, it is insufficient to call this grounded explanation evidence.

W6. The wording of “calibration” and cross-domain generalization needs to be more rigorous.

Using the positive-example mean, standard deviation, and sigmoid for score transformation is not equivalent to probability calibration. For a fixed positive standard deviation, this transformation is strictly monotonic; if one directly ranks unquantized scalars, it will not change AUC. On MuAViC, however, two-parameter re-estimation raises the final model’s freeze AUC from 0.572 to 0.937, indicating that the LLM interface is significantly sensitive to numerical scale, quantization, or other input interactions. This requires mechanistic analysis rather than being presented merely as a generalization-enhancement result. In addition, using 300 known target-domain positive examples constitutes target-domain calibration adaptation. The paper should clearly distinguish this from transfer results that use no target-domain samples at all.

W7. The downstream evaluation does not sufficiently establish ground truth for real lip-sync quality.

Table 5 assumes by default that professional dubbing lines should be superior to all LLM rewrites, but there is no independent human evaluation verifying this premise. The model may exploit stylistic differences between professional lines and generated rewrites rather than differences in lip-sync fit. Text-length control is also insufficient to control pronunciation duration and visible lip shape. In addition, the title of Table 5 says that an alignment scorer is used, while the table labels it as ATR-9B. The authors need to clarify whether the results come from the standalone aligner or the full model.

Questions for authors

Q1. Objective definition and labels

Is the model ultimately intended to judge “whether the candidate text is close to the original articulation” or “whether the candidate translation is suitable for dubbing”? What is the basis for treating all professional dubbing translations as negative examples? Are there independent human annotations of lip-sync quality?

Q2. Applicability to real synchronization errors

Without a candidate pronunciation timeline, how do the authors define and validate errors such as “too early” and “delayed”? Can the authors provide evaluations on real dubbing synchronization errors or settings closer to practical use, rather than only video reversal and clip swapping?

Q3. Shortcuts and fairness controls

What are the results under video-only, text-only, and random candidate-text substitution conditions? After training a direct classification baseline with the same negatives and judgment objective, how much advantage does ATR retain?

Q4. Independent value of the LLM

Considering that the aligner is slightly better than the full model on all four temporal AUCs, how should the main role of the LLM be defined? Can a small classifier using the same local evidence and global scores achieve similar gains?

Q5. Calibration and cross-domain evaluation

Are the 300 calibration samples pooled across the three languages or sampled separately for each language? Are they strictly isolated from the test set? What is the AUC of the original aligner on MuAViC? If the two-decimal quantization is removed, does the benefit of recalibration still remain?

Q6. Downstream labels and result attribution

Is there any human evaluation confirming that professional dubbing lines are indeed better lip-synced than generated rewrites? Do the results in Table 5 come from the aligner or from the full ATR-9B?

Q7. Metrics and data isolation

Do the tables specifically use pooled AUC or per-language macro AUC? Are training and test sets isolated at the level of works, speakers, and shared visual clips? What is the relationship between the main-table results and the multi-seed averages in Table 9?

2 comments

We thank the reviewer for the detailed review and for noting that our paper has a clear practical value, a reasonable structural motivation, and broad experimental coverage. We address the concerns in detail below and will incoporate all feedback into the final version. Hope you could raise the score.

W1/Q1. Task definition and the evaluation label.

The paper targets pre-synthesis visual–text alignment, the compatibility between a candidate line and the visible articulation; it does not directly assess the quality of a completed dubbed audio track. We agree that a good target-language dub need not reproduce the source-language phoneme sequence, and may trade off phonetic fit against meaning, naturalness, and duration. Our dub negatives serve as a source-articulation mismatch signal: a translated line is not the speech visibly articulated in the original video. Section 3.5 also considers a more application-oriented task: ranking four semantically similar candidate dub lines. ATR obtains 0.479 Top-1 accuracy versus 0.316 for the strongest lip-reading baseline (Table 5).

W2/Q2. Scope of the temporal claim and shortcut controls.

Thanks for the suggestion. We will modify the wording from temporal synchronization to temporal order: ATR estimates whether a line's phonetic content can be laid out, in order, over the visible articulation, not the synchronization error of a specific recording. To rule out that the temporal axes are solved from motion anomalies or splice boundaries rather than text–video correspondence, we conduct additional experiment with a video-only detector. As shown, the video-only detector is far weaker than the aligner on the order-dependent axes (shift 0.642 vs 0.935), so the glitch alone cannot explain the aligner's score.

Shortcut controls on the temporal axes.

ControlReverseShiftFreezeSwap
ATR aligner AUC (paper, Table 4)0.9700.9350.9710.962
Video-only detector AUC0.7340.6420.9010.699

W3/Q3. Baselines that isolate the contribution.

To hold data, negatives, and objective fixed and vary only the model, we conduct additional experiments to add a Qwen3.5-9B judge-SFT trained on our negatives, a direct matching classifier over the same encoders without the monotonic constraint, and a standard fixed-vocabulary phoneme-CTC scorer. Matched training lifts the VLM judge from 0.610 to 0.676 mean, but it remains near chance on fine temporal order (shift 0.589), and the direct classifier behaves similarly (shift 0.561); both confirm that the monotonic structure, not the training recipe, drives temporal sensitivity. Standard phoneme CTC recovers temporal order (it is also monotonic) but trails our candidate-position CTC on content, isolating the benefit of position labels over a fixed vocabulary; ATR-9B retains a +0.244 mean margin over the strongest matched baseline.

Same data, same negatives, different model.

ModelMismatchShuffleDubReverseShiftFreezeSwapMean
Qwen3.5-9B SFT, transcription objective (paper)0.7300.7370.7430.5200.5130.5100.5190.610
Qwen3.5-9B judge-SFT, our negatives0.7420.7810.7600.6160.5890.6430.6040.676
Direct matching classifier (no monotonic constraint)0.7240.7960.7350.6020.5610.6990.5730.670
Standard phoneme CTC (fixed vocabulary)0.6710.7190.6620.9220.8780.9310.9040.812
Candidate-position CTC aligner (paper, Table 4)0.6950.7570.7010.9700.9350.9710.9620.856
ATR-9B (paper)0.7860.9740.8620.9640.9310.9700.9520.920

W4/Q4. Role of the LLM.

The LLM is not intended to improve temporal judgments; by design temporal grounding comes from the alignment score, the LLM's contribution is content discrimination and reasoning. We conduct additional experiments to isolate what the LLM adds. We show results in following table. The text-only reasoner is at chance on Mismatch (0.515) and on all temporal axes, so the gains on Mismatch and the temporal axes require the visual evidence, not language priors; a lightweight classifier on the same soft tokens and global score recovers much of the signal (0.873 mean) but still trails the LLM on content (mismatch 0.731 vs 0.786, dub 0.774 vs 0.862), so the remaining gain is not merely model capacity reading the scalar.

Isolating what the LLM adds.

ModelMismatchShuffleDubReverseShiftFreezeSwapMean
Text-only reasoner (no soft tokens, no score)0.5150.8050.7280.5000.5000.5000.5000.578
Aligner only (paper, Table 4)0.6950.7570.7010.9700.9350.9710.9620.856
Lightweight classifier on the same evidence0.7310.8020.7740.9610.9280.9680.9490.873
ATR-9B (paper)0.7860.9740.8620.9640.9310.9700.9520.920

W5. Groundedness of local evidence and explanations.

The per-unit soft tokens are order-free by design, they carry content evidence (how well each unit is supported somewhere in the clip), while order comes from the global alignment score, so an off-path local match is expected, not a defect. To test whether the evidence and explanations actually point to where an error is, we split each clip into three parts, break only one part, and ask the model which part is broken (chance = 1/3); a part is broken either by freezing its video (the lips stop moving) or by swapping its words (the text no longer matches the lips). We then run additional checks. ATR-9B flags the broken part well above chance (localization) and outperforms Auto-AVSR.

Can the model point to the broken part? Localization accuracy; chance = 1/3.

MethodVideo brokenText broken
Auto-AVSR0.4020.541
ATR-9B0.7740.829

Calibration wording and the MuAViC result.

Thanks for the suggestion, we will rename it the as normalized alignment score. We run new experiments, ranking MuAViC clips by the aligner's raw score, no LLM, no target data, already reaches 0.862 mean, so the signal transfers to the unseen languages on its own. The full model's lower zero-shot score (0.751) is therefore not a generalization failure but an input-scale effect: out of domain the genuine scores drift into a low range the reasoner saw only as negatives in training, and re-estimating the two scaling constants on target data moves them back into the trained range and recovers the AUC. The 300 target pairs add no domain knowledge, only a rescaling of the LLM's input, so two scalars with no weight update suffice.

** MuAViC AUC. The raw aligner transfers with no target data; the full model's zero-shot drop and its recovery are an input-scale effect at the LLM.**

ModelMismatchShuffleReverseShiftFreezeSwapMean
ATR aligner only, raw score, no LLM, no target data (new)0.6520.9180.9080.8740.9120.9070.862
ATR-9B, zero-shot, no target data (paper)0.6610.9300.7900.7630.5720.7870.751
ATR-9B, after input-scale normalization, 300 target pairs (paper)0.7070.9460.9270.8270.9370.9050.875

W7/Q6. Downstream ground truth and the Table 5 label.

The 0.479 in Table 5 is the full ATR-9B (matching the body text); we fix the caption. To verify the premise that professional lines are genuinely better lip-synced, not just stylistically different, we add a human study and new experiments. Shown the professional line and one rewrite over the original video, annotators prefer the professional line 58% of the time (27% rewrite, 15% tie), and ATR-9B agrees with their majority choice on 74% of pairs versus 55% for the best baseline. The signal is lip-sync, not style: scoring the same candidates against a different clip drops ATR-9B's Top-1 to chance (0.271), and a text-only reasoner with no video stays at chance (0.262).

Dub-line reranking.

MethodTop-1MRR
Chance0.2500.521
Isochrony / length fit (paper)0.2370.494
LLaMA-AVSR, best lip-reading baseline (paper)0.3160.575
Text-only reasoner, no video (new)0.2620.531
ATR-9B scored against a mismatched clip (new)0.2710.537
ATR-9B (paper)0.4790.684

Human study (150 clips, 3 annotators): professional preferred 58% / rewrite 27% / tie 15%. Agreement with the human majority on non-tie pairs, ATR-9B 0.74, aligner 0.70, LLaMA-AVSR 0.55, isochrony 0.49.

Q7. Metrics and data isolation.

Tables 1 reports pooled AUC over all test pairs and Table 2 reports the macro-average over languages; the Macro label in the Table 4 caption was a typo and is fixed. Train and test are disjoint by title, so no work, speaker, or shared clip appears on both sides. The main tables report a single representative seed; these lie within one standard deviation of the three-seed means in Table 9 (0.920 vs 0.910±0.015 for Qwen3.5-9B, 0.890 vs 0.895±0.012 for LLaMA-3.1-8B).

Reviewer_BWQROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a new method for dubbing assessment. The central premise: given a silent speaker video and a text transcript of the actual speech, how do we assess whether the lip articulation aligns with the text; there is no audio available. The authors claim that the present generation of VLMs cannot score lip sync in this setting and strong scoring models sit in the audio-visual domain. The authors correctly identify to aspects of sync scoring - articulation and timing, and use this insight provide additional context to the actual LLM judge. The final output scores the alignment and also emits a justification (whether lip movements did not match the text in some segment, timing was wrong, etc). The model beats contemporary reasoning models across multiple benchmarks.

Critical strengths
  1. The evaluations are thorough. Compared many baselines across multiple benchmarks.
  2. i appreciate the authors for identifying a legitimate gap. While there is much effort in developing lip sync models, little effort is put into designing better metrics for scoring the generated results; the community still prefers LSE-C or LSE-D and would like a metric with a better rubric. So, the intent is at the right place.
Critical weaknesses
  1. My biggest criticism is the framing of the paper. The correct framing would have been "how to develop a better scoring mechanism for lip reading models?". In absence of audio transcripts, that is the only reasonable framing. However, the authors frame it as a dubbing judge - where audio has to be present. Without audio we won't even have the video frames in different languages, as audio is a necessary input for video dubbing models. Consequently, text-to-video matching is of little use in such settings. In the Downtream Evaluation section the authors correctly mention a correct application - script to clip alignment. Then why position the paper around dubbing?

  2. While the evaluation is thorough, the paper misses sufficient visual results. The authors include a panel like [frame sequence || phonetic sequence || Ours scores || Baseline scores w vanilla Qwen etc]. Let's say the sync breaks midway, I'd expect the author's method to show a drop in the metric at the right frame index while the baselines score the frames wrongly/ mistimes the scores.

Questions for authors
  1. Please justify the positioning of the paper. Why build it around dubbing where audio is readily available?
  2. Can the reasoning model point out the segments where the sync degrades? If so, please show some evidence.
  3. Why choose the specific corruptions for temporal negatives? I am a bit confused with the "reverse"/"swap", do those corruptions help at all? In realistic scenarios, it always a case of temporal mismatch of a few frames or wrong content.
2 comments

We thank the reviewer for recognizing our thorough evaluation, comparing many baselines and benchmarks, and identifying a legitimate gap. We address the concerns below in detail and will incoporate all feedbak into the final version. Hope you could raise the score.

W1/Q1. Task positioning.

We agree that the quality of a completed dub cannot be evaluated from video and text alone. Audio is necessary to assess the synthesized speech's actual timing, pronunciation etc. Our intended setting is instead pre-synthesis dubbing assessment, where a source video and candidate translated or rewritten lines are available, but the corresponding target-language audio has not yet been generated. At this stage, assessing whether a candidate line is phonetically and temporally compatible with the visible articulation can help screen or rerank candidates before speech synthesis. This is relevant to existing dubbing workflows, where translated scripts are adapted to satisfy timing and lip-synchronization constraints before speech generation [1,2]. Our method is therefore complementary to, and sits upstream of, post-synthesis audio–visual evaluation.

We also distinguish our task from scoring lip-reading models. Lip reading generally attempts to recover a transcript from video. Our formulation takes an arbitrary candidate line as input and directly outputs a compatibility score from the video–text pair, without a reference transcript or auido at inference. This supports applications beyond lip-reading evaluation, including candidate script reranking and script-to-clip assignment, as demonstrated in Tables 5–6 in paper.

We acknowledge that the term dubbing judge may suggest a broader scope. We will revise the framing to emphasize reference-free visual–text lip-alignment judgment, with pre-synthesis script assessment, and clarify the distinction from complete dubbed-audio quality assessment.

W2/Q2. Visual evidence and locating where sync degrades

We add the requested visual evidence in Fig. 2 in the Appendix E in paper. We show two examples comparing ATR-2B with baseline Qwen3.5-2B on a clip that is in sync for the first half of the line and broken from the midpoint on, with (a) frames reversed and (b) mouth frozen. For both models we compute a per-frame sync score: we measure how far the model's score for the clip falls. Fig. 3 plots this score relative to the same clip in sync, so 0 means no change and lower means worse sync. ATR-2B stays near 0 while the clip is in sync and drops inside the broken span, and its clip-level score falls from 0.61 to 0.15 (reversed) and 0.34 (frozen). Qwen3.5-2B stays flat over the whole clip, and its clip-level score barely moves (0.59 → 0.59 and 0.58). This demonstrates that ATR effectively captures visual–text alignment, whereas the baseline produces incorrect scores.

Q3. Temporal negatives, and whether reverse/swap help

The temporal negatives were chosen primarily as controlled tests of distinct temporal properties. They keep the candidate text unchanged, thereby preventing a model from succeeding by detecting only text-content differences. Reverse reverses the order of visual articulation. It preserves the underlying set of frames while breaking global temporal direction, testing whether a judge uses sequence order rather than only appearance. Swap exchanges the video halves. It again largely preserves frame content while violating the order in which speech segments should occur; it tests sensitivity to long-range sequence structure. Shift circularly offsets frames relative to the same text, testing sensitivity to phase/position changes. Freeze repeats a frame over a span, testing sensitivity to interrupted visual progression. Thus reverse and swap are diagnostic stress tests, particularly useful for exposing the temporal blind spot of otherwise strong video-language models.

[1] PS-TTS: Phonetic Synchronization in Text-to-Speech for Achieving Natural Automated Dubbing.

[2] End-to-End Multilingual Automatic Dubbing via Duration-based Translation with Large Language Models.

  1. Visualization: I appreciate the authors for revising the evaluations and including visual results for clarity. I would recommend them to move the figures to the main manuscript as it strengthens the position of their contribution.

  2. Temporal Negatives: I agree with the author's motivation here. A good scoring model should judge the frames by lip articulation only and not the sequence of appearance. However, my argument is against the redundancy of reverse/swap corruptions - I think time shifts (of varying degrees) will disincentivize the model to exploit ordering. If the shift is constant/small in all samples, the model might exploit ordering but once the shifts have stochasticity the model would not use it as a shortcut. Can the authors provide evidence to the contrary?

  3. Positioning: The authors are building on the premise that there is a source language video-transcript pair (V_s, T_s) and a translated pair (V_t, T_t), where the lips of V_t should match the phonetics of T_t. The evaluation is about their alignment. So, there is an assumption of visual dubbing step. I am arguing that it's only possible in presence of an audio: visual dubbing is not performed before finalising the audio translation. The proposed method can act as an additional validation step but is entirely optional in a dubbing workflow. So the dubbing framing does more damage than just broadening the scope - it points to a limitation that practically does not exist!

Comments on the paper

1 comment

Dear AC and Reviewers,

We apologize for the delayed response and appreciate your patience. We took substantial effort and have been actively working on additional experiments and preparing detailed responses. We sincerely appreciate your thoughtful and constructive feedback, which has helped us strengthen the paper.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.