acceptodds
Withdrawn from ICLR 2027

TransVis: Benchmarking Real-World In-Image Machine Translation at Scale and Fine Granularity

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In-image machine translation (IIMT) aims to translate the text inside an image while seamlessly preserving the original visual layout. Existing benchmarks either stop at textual translation or assess generated images with image-level metrics; even multi-aspect protocols do not trace each source text block through discovery, translation, text rendering and visual preservation. Consequently, omissions, mistranslations and visual damage remain difficult to attribute. We therefore present TransVis-Bench, a large-scale, fine-grained benchmark for real-world Chinese-to-English IIMT. The benchmark covers major categories and sub-categories of text–visual relations, comprising images and source text blocks. We design a source-block-anchored evaluation protocol that decouples model behaviour into four dimensions: translation coverage, translation quality, rendering accuracy and visual fidelity. An evaluation of foundation image-generation models shows that a single aggregate score hides markedly different combinations of failure in text translation, text rendering and visual preservation. Controlled experiments further show that explicitly injecting linguistic information raises the translation score of an open-source backbone from to , indicating that acquiring linguistic information remains a major limitation of current end-to-end translate-and-render models. TransVis-Bench provides a comprehensive testbed for real-world IIMT and supports auditable, fine-grained analysis of generated results.

Primary area:
datasets and benchmarks
Keywords:
In-image machine translation, multimodal image generation, benchmark, text rendering, visual fidelity

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_QJAkOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

TRANSVIS-BENCH is a benchmark for Chinese-to-English in-image machine translation, comprising 2,591 images and 19,453 source text blocks across five categories and 18 sub-scenarios. Its evaluation is anchored at the source-block level. It measures whether each block is translated, the quality of translations for matched blocks, character- and word-level legibility, and visual fidelity in terms of style, layout, and background preservation. The authors evaluate 14 end-to-end image-generation models and conduct two additional studies: scaling end-to-end SFT on two open-source backbones, and providing models with block-level OCR and English translations from Gemini-3.1-Pro.

Critical strengths
  1. The evaluation is more diagnostic than a single image-level score. IIMT combines recognition, translation, text rendering, and image preservation. Evaluating each source text block and reporting these dimensions separately makes omissions and other failures easier to identify.

  2. The benchmark covers varied, relevant text–visual conditions. It includes dense interfaces, long document text, and other real-world scenarios where translation and layout preservation are challenging.

Critical weaknesses
  1. The reported overall scores do not match the stated formula. For example, applying Eq. (1) to the components in Table 3 gives approximately 83.32 for Qwen-Image-3-Pro, not the reported 84.43. The authors should clarify the aggregation and provide a reproducible calculation, since these totals determine the model rankings.

  2. The visual judge’s agreement with humans is limited, particularly for layout. TRANSVIS-JUDGER’s mean Cohen’s κ\kappa is 0.470, and its layout κ\kappa is 0.267. Since these judgments contribute to the reported scores, the authors should show whether the model rankings are stable under resampling and how much the conclusions depend on the layout metric.

  3. The judge validation set is small and its coverage is unclear. Validation uses 50 source images and 700 source–output pairs. The paper should report how these examples and human disagreements are distributed across scenarios and difficulty levels. It should also quantify how errors in matching affect downstream style and layout scores.

  4. The two-stage experiment does not isolate linguistic ability as the cause of the gains. It supplies OCR and translations from Gemini-3.1-Pro, so the input differs substantially from the end-to-end condition and may contain errors. The results show that explicit text helps, but do not establish that language understanding is the primary bottleneck. Please report the auxiliary input’s error rate or test with human-corrected translations.

Questions for authors

See the weaknesses.

Reviewer_CbgWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the lack of fine-grained evaluation for in-image machine translation. They create a new benchmark for evaluate the current model, together with an automatic, source-block-anchored evaluation protocol that scores translation coverage, translation quality, text-rendering accuracy, and visual fidelity.

Critical strengths

The paper proposes a new and useful benchmark for a real, practical task. A large benchmark with 2,591 images covering diverse categories. Good motivation, since previous benchmarks have limited data coverage or evaluation protocols.

Critical weaknesses

Data creation process: there are no human review guidelines or a description of which samples should be discarded. “Unsuitable” samples and the “collection criteria” are only mentioned in Figure 2, without a clear definition. No details on the creation of the 518 synthetic samples: where the source text and images come from, where the text is rendered/overlaid on the image, and what text is used. The validation set contains only 50 images, which is small. Is the evaluation on this validation set reliable? Human agreement is reported only for the four visual judges, not for the translation and rendering scores. In the final score, where do the weights 0.15, 0.25, 0.15, and 0.10 come from? There is no explanation for these weights.

Questions for authors

See the weaknesses above

Reviewer_KhdCOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that existing IIMT benchmarks are insufficient because they do not trace each source text block through discovery, translation, text rendering, and visual preservation, which makes it difficult to attribute errors. Therefore, it introduces TransVis-Bench, a benchmark for Chinese-to-English IIMT covering 5 major categories and 18 sub-categories. Its source-block-anchored evaluation protocol assesses model performance across four dimensions separately, including translation coverage, translation quality, rendering accuracy, and visual fidelity.

Critical strengths

The paper emphasises that translation coverage, translation quality, rendering accuracy, and visual fidelity are distinct aspects of IIMT, and a single aggregate score cannot accurately reflect every aspect. It is a meaningful attempt to build a benchmark to assess model performance across various aspects.

Critical weaknesses
  1. The "source-block anchor" is not clearly specified. The paper emphasises its 19,453 block-level annotations, but it is unclear what those annotations include and how they are used in the evaluation protocol. According to Appendix D, it is likely the scores are produced by the judges based on the source and translated image, and the annotations are never used. The match judge is asked to identify blocks in each image pair, and if it overlooks a source block, the evaluation will miss exactly the omissions it is intended to find.

  2. The visual judge validation in Section 3.3 is insufficient and inconsistent.

    1. It uses only 50 images, which is a small portion compared to the 2,591 total images.
    2. It only compares labels of "Yes / No / NA", instead of the 0-100 scale.
    3. Annotators' majority vote forms the human reference, but the human–human agreement/κ\kappa is not reported.
    4. Table 2 can demonstrate that the performance of the TransVis-Judger is close to the GPT-5.5 teacher but not human annotators. Around 75% agreement means there is around a 1/4 chance that the TransVis-Judger disagrees with humans. The 0.380 and 0.267 κ\kappa for style and layout are also not as strong as the paper claims.

    Besides, it does not validate translation or rendering.

Questions for authors
  1. Please explain in more detail how the source-block anchor is guaranteed and how the annotations are used in the evaluation protocol.

  2. Please address the concerns about the judge validation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.