- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
TRANSVIS-BENCH is a benchmark for Chinese-to-English in-image machine translation, comprising 2,591 images and 19,453 source text blocks across five categories and 18 sub-scenarios. Its evaluation is anchored at the source-block level. It measures whether each block is translated, the quality of translations for matched blocks, character- and word-level legibility, and visual fidelity in terms of style, layout, and background preservation. The authors evaluate 14 end-to-end image-generation models and conduct two additional studies: scaling end-to-end SFT on two open-source backbones, and providing models with block-level OCR and English translations from Gemini-3.1-Pro.
Critical strengths
-
The evaluation is more diagnostic than a single image-level score. IIMT combines recognition, translation, text rendering, and image preservation. Evaluating each source text block and reporting these dimensions separately makes omissions and other failures easier to identify.
-
The benchmark covers varied, relevant text–visual conditions. It includes dense interfaces, long document text, and other real-world scenarios where translation and layout preservation are challenging.
Critical weaknesses
-
The reported overall scores do not match the stated formula. For example, applying Eq. (1) to the components in Table 3 gives approximately 83.32 for Qwen-Image-3-Pro, not the reported 84.43. The authors should clarify the aggregation and provide a reproducible calculation, since these totals determine the model rankings.
-
The visual judge’s agreement with humans is limited, particularly for layout. TRANSVIS-JUDGER’s mean Cohen’s is 0.470, and its layout is 0.267. Since these judgments contribute to the reported scores, the authors should show whether the model rankings are stable under resampling and how much the conclusions depend on the layout metric.
-
The judge validation set is small and its coverage is unclear. Validation uses 50 source images and 700 source–output pairs. The paper should report how these examples and human disagreements are distributed across scenarios and difficulty levels. It should also quantify how errors in matching affect downstream style and layout scores.
-
The two-stage experiment does not isolate linguistic ability as the cause of the gains. It supplies OCR and translations from Gemini-3.1-Pro, so the input differs substantially from the end-to-end condition and may contain errors. The results show that explicit text helps, but do not establish that language understanding is the primary bottleneck. Please report the auxiliary input’s error rate or test with human-corrected translations.
Questions for authors
See the weaknesses.