- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents UF-VSR, a two-stage framework for Mandarin visual speech recognition (VSR) that explicitly separates phonetic and lexical information. The motivation is that lip movements provide incomplete phonetic evidence, while multiple Chinese characters may share the same pronunciation.
In the first stage, a shared visual encoder with parallel decoders predicts both Chinese characters and toneless Pinyin sequences. In the second stage, a LoRA-adapted 4B language model refines the character hypothesis using both predictions. The language model is trained on recognition errors generated by VSR checkpoints at different training stages. The implemented system uses Top-1 hypotheses rather than probabilistic marginalization or direct access to visual features.
Experiments on CMLR and CNVSRC-Multi.Dev demonstrate improvements in character error rate (CER). On CMLR, Qwen3.5-4B refinement reduces CER from 11.781% to 10.166% for the primary VSR model, and from 8.021% to 7.603% for a stronger upstream model. The paper further analyzes how correction performance depends on Pinyin reliability, phonetic ambiguity, and the severity of upstream errors.
Critical strengths
-
Well-motivated problem formulation for Mandarin VSR. The paper identifies two distinct sources of uncertainty: ambiguity between visual articulations and phonetic syllables, and ambiguity between syllables and Chinese characters. The proposed phonetic–lexical interface is intuitively motivated by these properties of Mandarin. Retaining an independently predicted character hypothesis alongside Pinyin is also a reasonable design choice, since Pinyin alone cannot resolve homophones through phonetic information.
-
Meaningful empirical improvements with appropriate consideration of upstream model quality. The complete framework reduces CER on two public evaluation datasets. Importantly, the authors evaluate refinement on both their primary VSR model and a stronger upstream model on CMLR. Although the improvement becomes smaller with the stronger upstream (5.21% relative reduction versus 13.71%), it remains positive. The reported preservation rate of 98.87% for already-correct sentences also provides useful evidence that the refiner does not indiscriminately rewrite its inputs.
-
Informative analysis of the method's operating conditions and limitations. Beyond aggregate CER, the paper investigates correction behavior under different Pinyin error rates, homophone ambiguities, and upstream error levels. These analyses reveal that refinement is more effective when the phonetic hypothesis is reliable and may become harmful when Pinyin errors are severe. I also appreciate that the authors explicitly distinguish observed correlations from causal evidence and acknowledge that the current experiments do not isolate every component's contribution.
Critical weaknesses
-
The incremental contribution of independently predicted Pinyin is not established by controlled ablations. This is the most important limitation of the paper. The experiments do not compare the same LLM refinement pipeline with and without the predicted Pinyin input, under otherwise identical adaptation and evaluation settings, which makes it unclear how much improvement comes from the proposed phonetic–lexical interface rather than generic LLM-based error correction. A matched character-only refinement baseline would directly clarify the value of the independent phonetic branch. The authors acknowledge this limitation in Section 6, but it remains central to evaluating the paper's contribution.
-
The practical benefit relative to the additional model cost is insufficiently characterized. The refinement stage introduces a 4B-parameter language model, but the paper does not report standardized inference latency, memory requirements, or accuracy–cost trade-offs with the baselines compared in the main results. Besides, the character/Pinyin dual-decoder improvement before LLM refinement is reported only on the insufficiently documented internal stress test, rather than on the public CNVSRC benchmark (See Table 2). Public-benchmark component results and basic inference-cost measurements would help establish the practical significance of the proposed pipeline.
Questions for authors
Please see weaknesses.