acceptodds
Withdrawn from ICLR 2027

UF-VSR: A Phonetic–Lexical Decomposition for Mandarin Visual Speech Recognition

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Mandarin visual speech recognition (VSR) must infer characters from lip motion that primarily constrains syllable identity: visually similar articulations can denote different syllables, and one Pinyin syllable can denote many characters. We present UF-VSR, a two-stage system that makes this mismatch explicit. A shared visual encoder with character and toneless-Pinyin decoders first produces complementary lexical and phonetic hypotheses; a 4B language model adapted to model-generated VSR errors then refines the character hypothesis conditioned on both. The probabilistic decomposition is a modeling interpretation rather than a claim of exact marginal inference: the implemented system uses one best hypothesis from each decoder and the language model does not observe video. On CMLR, refinement reduces character error rate (CER) from 11.781 to 10.166 (13.71% relative) for our primary upstream and from 8.021 to 7.603 (5.21% relative) for a stronger upstream. Fixed-track CNVSRC-Multi.Dev CER decreases from 30.87 for a character-only model to 28.39 for the complete pipeline. Mechanism-oriented analyses show where the system succeeds and fails: reliable Pinyin is associated with larger gains, identical-Pinyin errors require context, and refinement ceases to help when either phonetic or character evidence is severely corrupted. These results support a practical Mandarin-specific interface between visual and language models while delimiting, rather than overstating, the evidence for its individual components.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
visual speech recognition

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_GvRmOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents UF-VSR, a two-stage framework for Mandarin visual speech recognition (VSR) that explicitly separates phonetic and lexical information. The motivation is that lip movements provide incomplete phonetic evidence, while multiple Chinese characters may share the same pronunciation.

In the first stage, a shared visual encoder with parallel decoders predicts both Chinese characters and toneless Pinyin sequences. In the second stage, a LoRA-adapted 4B language model refines the character hypothesis using both predictions. The language model is trained on recognition errors generated by VSR checkpoints at different training stages. The implemented system uses Top-1 hypotheses rather than probabilistic marginalization or direct access to visual features.

Experiments on CMLR and CNVSRC-Multi.Dev demonstrate improvements in character error rate (CER). On CMLR, Qwen3.5-4B refinement reduces CER from 11.781% to 10.166% for the primary VSR model, and from 8.021% to 7.603% for a stronger upstream model. The paper further analyzes how correction performance depends on Pinyin reliability, phonetic ambiguity, and the severity of upstream errors.

Critical strengths
  1. Well-motivated problem formulation for Mandarin VSR. The paper identifies two distinct sources of uncertainty: ambiguity between visual articulations and phonetic syllables, and ambiguity between syllables and Chinese characters. The proposed phonetic–lexical interface is intuitively motivated by these properties of Mandarin. Retaining an independently predicted character hypothesis alongside Pinyin is also a reasonable design choice, since Pinyin alone cannot resolve homophones through phonetic information.

  2. Meaningful empirical improvements with appropriate consideration of upstream model quality. The complete framework reduces CER on two public evaluation datasets. Importantly, the authors evaluate refinement on both their primary VSR model and a stronger upstream model on CMLR. Although the improvement becomes smaller with the stronger upstream (5.21% relative reduction versus 13.71%), it remains positive. The reported preservation rate of 98.87% for already-correct sentences also provides useful evidence that the refiner does not indiscriminately rewrite its inputs.

  3. Informative analysis of the method's operating conditions and limitations. Beyond aggregate CER, the paper investigates correction behavior under different Pinyin error rates, homophone ambiguities, and upstream error levels. These analyses reveal that refinement is more effective when the phonetic hypothesis is reliable and may become harmful when Pinyin errors are severe. I also appreciate that the authors explicitly distinguish observed correlations from causal evidence and acknowledge that the current experiments do not isolate every component's contribution.

Critical weaknesses
  1. The incremental contribution of independently predicted Pinyin is not established by controlled ablations. This is the most important limitation of the paper. The experiments do not compare the same LLM refinement pipeline with and without the predicted Pinyin input, under otherwise identical adaptation and evaluation settings, which makes it unclear how much improvement comes from the proposed phonetic–lexical interface rather than generic LLM-based error correction. A matched character-only refinement baseline would directly clarify the value of the independent phonetic branch. The authors acknowledge this limitation in Section 6, but it remains central to evaluating the paper's contribution.

  2. The practical benefit relative to the additional model cost is insufficiently characterized. The refinement stage introduces a 4B-parameter language model, but the paper does not report standardized inference latency, memory requirements, or accuracy–cost trade-offs with the baselines compared in the main results. Besides, the character/Pinyin dual-decoder improvement before LLM refinement is reported only on the insufficiently documented internal stress test, rather than on the public CNVSRC benchmark (See Table 2). Public-benchmark component results and basic inference-cost measurements would help establish the practical significance of the proposed pipeline.

Questions for authors

Please see weaknesses.

Reviewer_6vJQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

UF-VSR separates Mandarin visual speech recognition into visual prediction and language-level correction. A shared visual encoder predicts one-best character and toneless-Pinyin sequences, which a 4B language model adapted to VSR errors reads jointly to produce a transcription. The paper reports CER improvements for two character upstreams on CMLR and on fixed-resource CNVSRC-Multi.Dev, and analyzes Pinyin reliability, homophones, and heavily corrupted inputs. The probabilistic decomposition is explicitly presented as a modeling interpretation rather than exact marginal inference.

Critical strengths
  1. The interface has a clear linguistic motivation and yields consistent end-to-end improvements on public data. Retaining a character hypothesis as a lexical anchor alongside Pinyin is a reasonable modular design. With Qwen3.5 refinement, Table 3 reduces CER from 11.781 to 10.166 for the primary upstream and from 8.021 to 7.603 for the stronger character upstream. Table 2 reduces Multi.Dev CER from 30.87 to 28.39 under the same S3 resources. These results support practical utility of the complete pipeline.

  2. The paper examines meaningful failure boundaries. It distinguishes homophones requiring contextual reasoning from alternatives that Pinyin can discriminate, and reports degradation at high PER/CER together with 98.87% preservation of already-correct sentences (Table 4, Section 5). Conditional associations are not presented as causal ablations, and the Top-1 cascade is not described as uncertainty marginalization. This makes the empirical findings more credible.

Critical weaknesses
  1. The incremental value of the central interface over ordinary character correction is not established. There is no comparison of character-plus-Pinyin versus character-only refinement with the same LLM, adaptation examples, optimization budget, and character hypotheses. Consequently, the improvement may primarily reflect generic language correction rather than the proposed dual-view interface. Sections 5.1 and 6 acknowledge this limitation, but acknowledgment does not validate the distinguishing contribution. A single carefully matched character-only control would materially change the assessment.

  2. Evidence for the dual decoder itself and the stability of the smaller gains is insufficient. In Table 2, the Pinyin-decoder-only result is available only on the inadequately documented internal set; the corresponding public Multi.Dev cell is missing. Public-data improvements therefore do not separate joint training from refinement. The stronger CMLR upstream improves by only 0.418 CER percentage points, while Section 6 states that the record is insufficient to report run-to-run variance, significance tests, and all visual decoding hyperparameters. This leaves the robustness and reproducibility of that gain uncertain.

Questions for authors
  1. What CER does character-only refinement achieve with identical character hypotheses, LLM, training examples, and budget? What is the absolute gain from adding Pinyin, with a paired confidence interval?

  2. Can you provide the dual-decoder result without LLM refinement on public Multi.Dev, together with the actual training/decoding configuration, checkpoint-selection rule, and paired uncertainty estimate for the stronger-upstream result?

Reviewer_nQRmOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents a visual speech recognition model for Mandarin. The proposed solution takes into account the unique characteristics of Mandarin, where one video sequence can be mapped to multiple pinyin and each pinyin can be mapped to multiple characters. A visual encoder is used which generates characters and pinyin, which are both fed to an LLM which in turn produces the text output. The proposed solution is evaluated on a few Mandarin datasets for lipreading and compared against a few other methods.

Critical strengths

A VSR model which is tailored to Mandarin is a useful contribution, rather than blindly using VSR models trained for English.

Critical weaknesses

The paper seems unfinished. It’s hard to follow some sections and there are sentences which are rather incomplete or do not make sense. Below is a non-exhaustive list of such examples:

  • Section 4.2, Main results, This section is very short and the discussion is limited and very hard to follow.
  • Section 5. It is hard to follow this section. There are also tables and figures which are not explained properly, e.g., Fig. 5 is not explained and discussed in the text. It is not easy to understand what Table 5 shows. Its caption is not very informative.
  • Section 3, p.3, “Equation 1 is not our inference algorithm or a theoretical novelty claim.” It is not clear what the purpose of this sentence is.
  • Section 3.2, p.3, “A SANM encoder…produces H”. It is not clear what H is.
  • Section 3.3, p.4 “Each instruction triple (ˆP, ˆYvsr,Y) exposes the LLM to severe, moderate, and residual VSR errors while also including cases that should be preserved.” It is not clear what this sentence means.
  • P.5, top, “Since we do not compare against zero-shot, final-checkpoint-only, or matched-size generic correction data, results should be attributed to the adapted pipeline as a whole, not specifically to multi-checkpoint sampling”. It is not clear what this sentence means.
  • Section 4, “The available experiment record does not contain publication-grade speaker, duration, demographic, collection, annotation, or split documentation”. It is not clear what this sentence means.
  • Section 6, p.8 “Four regimes follow. When both agree and are correct, the safest action is preservation.” It is not clear what this sentence means.

There are no ablation studies in the paper, so it’s hard to judge how the different components affect the final performance.

Limited comparison with SOTA methods. The proposed method is compared only with 3 other approaches in Table 2. It is also not clear what the different settings refer to in this table.

The authors report results in an internal test. However, it is not explained how this dataset was created. In addition, the authors claim “We therefore label these numbers as a secondary stress test, make no dataset contribution claim, and do not use them to support generalization to unseen speakers. Release and complete documentation are required before this set can serve as a reproducible benchmark.” “The latter numbers are descriptive only because the set is not yet adequately documented.”, “the dataset must be documented before that result can carry broad weight.” In other words, the authors claim that the dataset cannot be trusted. Then why reporting results on this?

Several limitations in the evaluation protocol. The authors claim “This comparison indicates sensitivity to backbone quality, but both models are 4B Qwen variants; testing Llama, DeepSeek, and larger models remains necessary.”, “Generality is also limited. Qwen3 and Qwen3.5 share a family and parameter scale, so they are a within-family robustness check, not evidence across model families or scaling regimes.” In other words, the authors acknowledge the limitations of the evaluation.

Questions for authors

Please see above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.