- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces VISTA-Bench, a benchmark for evaluating multilingual image translation. Unlike traditional evaluation methods that mainly compare translations with reference text, VISTA-Bench uses image-specific rubrics to check whether models preserve important information, semantic relationships, and acceptable translations. I
Critical strengths
-
The image-specific rubrics are a useful contribution because they define what information must be preserved while allowing different but equally valid translations.
-
The authors combine model-assisted annotation with human verification, including corrections to text recognition, reading order, and semantic grouping, which helps improve the quality of the benchmark.
-
The authors examine both dataset-size sensitivity and agreement between automatic judges and human annotators, providing useful initial evidence for the reliability of their evaluation protocol.
Critical weaknesses
-
The benchmark relies heavily on LLM-generated reference translations and rubrics, and some languages were reviewed without fluent language specialists. This raises concerns about annotation consistency, particularly for lower-resource languages.
-
The human evaluation of the automatic judges covers only four candidate models, and the reported agreement is based on model-level averages rather than individual translation decisions. This makes it difficult to assess how reliable the judge is on challenging examples.
-
The scoring system uses manually assigned importance weights and sets a criterion's score to zero whenever a forbidden error is detected. The paper does not sufficiently examine how sensitive the results are to these scoring choices or possible judge errors.
-
The text-only baselines receive human-organized source text, whereas multimodal models must recognize and organize the text themselves. This makes direct comparisons difficult to interpret and does not clearly isolate the benefit of visual context.
-
The dataset-size analysis considers only four models. Although their rankings remain stable, this is not enough to establish that the smaller benchmark will reliably preserve rankings among models with more similar performance.
Questions for authors
-
How do the authors ensure the quality of reference translations and rubrics for languages without fluent human reviewers? Could they provide inter-annotator agreement statistics or additional validation by native speakers?
-
Given the very high correlations between translation, visual understanding, and knowledge scores, can the authors demonstrate that these dimensions measure genuinely different capabilities?
-
Can the authors evaluate the automatic judges at the individual-criterion level across more languages and models, rather than relying mainly on agreement between four model-level averages?
Dear Reviewer qyPt,
As the rebuttal deadline is approaching, please review the authors’ response and participate in the discussion with them at your earliest convenience.
Best regards,
The AC