acceptodds
Under review as a conference paper at ICLR 2027

VISTA-Bench: Benchmarking Multilingual Image Translation with Image-Specific Rubrics

13 views

Abstract

Image translation is a fundamental capability of multimodal models for multilingual applications, requiring visual understanding and meaning preservation across languages. However, existing benchmarks have limited language coverage and often lack explicit image-specific evaluation criteria, making it difficult to comprehensively assess this capability. To systematically evaluate this capability, we introduce VISTA-Bench, covering 22 languages and 10 domains, and develop an image-specific rubric evaluation protocol. The benchmark combines sampling for language and scenario coverage with model-assisted, human-verified annotations that group related text into coherent semantic units and provide multilingual reference translations. The rubrics specify essential content, semantic relations, and acceptable translation variants, yielding separate output-based scores for translation quality and the preservation of visual and knowledge-dependent information. We conduct extensive evaluations of 16 mainstream models, including 12 multimodal models and four text-input models, and provide systematic analyses across languages, domains, and evaluation dimensions.

Primary area:
datasets and benchmarks
Keywords:
Image translation, multimodal machine translation, translation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qyPtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VISTA-Bench, a benchmark for evaluating multilingual image translation. Unlike traditional evaluation methods that mainly compare translations with reference text, VISTA-Bench uses image-specific rubrics to check whether models preserve important information, semantic relationships, and acceptable translations. I

Critical strengths
  1. The image-specific rubrics are a useful contribution because they define what information must be preserved while allowing different but equally valid translations.

  2. The authors combine model-assisted annotation with human verification, including corrections to text recognition, reading order, and semantic grouping, which helps improve the quality of the benchmark.

  3. The authors examine both dataset-size sensitivity and agreement between automatic judges and human annotators, providing useful initial evidence for the reliability of their evaluation protocol.

Critical weaknesses
  1. The benchmark relies heavily on LLM-generated reference translations and rubrics, and some languages were reviewed without fluent language specialists. This raises concerns about annotation consistency, particularly for lower-resource languages.

  2. The human evaluation of the automatic judges covers only four candidate models, and the reported agreement is based on model-level averages rather than individual translation decisions. This makes it difficult to assess how reliable the judge is on challenging examples.

  3. The scoring system uses manually assigned importance weights and sets a criterion's score to zero whenever a forbidden error is detected. The paper does not sufficiently examine how sensitive the results are to these scoring choices or possible judge errors.

  4. The text-only baselines receive human-organized source text, whereas multimodal models must recognize and organize the text themselves. This makes direct comparisons difficult to interpret and does not clearly isolate the benefit of visual context.

  5. The dataset-size analysis considers only four models. Although their rankings remain stable, this is not enough to establish that the smaller benchmark will reliably preserve rankings among models with more similar performance.

Questions for authors
  1. How do the authors ensure the quality of reference translations and rubrics for languages without fluent human reviewers? Could they provide inter-annotator agreement statistics or additional validation by native speakers?

  2. Given the very high correlations between translation, visual understanding, and knowledge scores, can the authors demonstrate that these dimensions measure genuinely different capabilities?

  3. Can the authors evaluate the automatic judges at the individual-criterion level across more languages and models, rather than relying mainly on agreement between four model-level averages?

1 comment

Dear Reviewer qyPt,

As the rebuttal deadline is approaching, please review the authors’ response and participate in the discussion with them at your earliest convenience.

Best regards,

The AC

Reviewer_bJDDOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces VISTA-Bench, an image translation benchmark consisting of 2.2K rich annotated images, covering 22 languages and 10 topic domains. In it, every image receives its own specified rubrics, in terms of required meanings, acceptable paraphrases, and forbidden errors. The authors evaluate 12 multimodal and 4 text-only models. They report all the specified baseline performances across all involved languages and domains.

Critical strengths

I take a special liking to VISTA-Bench’s flexible evaluative scheme and its traceable scoring method.

  • The rubric allows room for valid variations. Each rubric criterion includes what alternative translation can be considered acceptable. I really appreicate the flexibility allowance in design, so that correct but differently worded translations are at least not outright penalized.
  • The ‘independent dimension scores’ are reported separately, rather than being agglomorated by the judge model. The judge only returns per-item decisions, so each score can be traced conveniently back to the respective requirement of an image (as in Section 3.4).
Critical weaknesses

Though I appreciate the completeness of this work, I find one particular reliability issue regarding how VISTA-Bench has been annotated.

We Do Not Know the Full Scope on How Human Annotators Validate the Translations of the Image-Specific Rubrics.

The authors present VISTA-Bench as built on model-assisted, human-verified annotations. However, upon my inspections (Section 3.3), I find that the authors have only involved one LLM (GPT-Sol-5.6) to draft and generate all parallel translations into references and rubrics. In addition, I find that in Appendix A, there is no documentation regarding any agreement between the human annotators. The authors also concede that some languages had no fluent reviewer, such as Burmese.

Per VISTA-Bench’s construction, each rubric is written once in English and reused for every target language. Thus, I fear that many key findings, such as those derived out of Figure 4, are built on the least verified components of the entire evaluative pipeline. The benchmarking results would be unreliable if the textual rubrics rendered in a target language have been mistranslated in the first place. Also, without reliable human verification, the language-level claims such as Burmese being the hardest (Section 4.4) could reflect the annotations’ quality, rather than an image-translation model’s ability.

My suggestion: The authors should run a double annotation by native speakers on a downsampled portion to cover at least the languages without fluent reviewers, and report the revision rates for the down-sample’s references and rubrics.

Questions for authors

Please find my suggestions in the Weakness section above. Other than that, I have one more question to follow up.

Q1. Sometimes, it must take a native to precisely comprehend the types of images being involved in this benchmark. Have the authors encountered any example that is culturely unique?

1 comment

Dear Reviewer bJDD,

As the rebuttal deadline is approaching, please review the authors’ response and participate in the discussion with them at your earliest convenience.

Best regards,

The AC

Reviewer_919ROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

VISTA-Bench evaluates image-to-text translation using 2,228 images spanning 22 languages, ten domains, and over 100 scenarios. Its contributions are multilingual coverage, human-reviewed grouping of source text, and image-specific rubrics specifying required meanings, acceptable variants, and forbidden errors. A judge model produces criterion-level decisions, which are aggregated into translation-quality, visual, and knowledge-related scores. Experiments cover twelve image-input models and four text-input baselines, with additional analyses of language/domain performance, subset selection, and agreement with human judgments.

Critical strengths

The combination of 22 source and target languages, varied real-world scenarios, and semantic-block annotations addresses important aspects of image translation. The benchmark substantially extends target-language coverage. This makes it a useful resource for evaluating capabilities beyond a single aggregate score.

Critical weaknesses
  • The added value of the evaluation complexity is insufficiently demonstrated. The paper does not establish why its elaborate image-specific rubric procedure is preferable to COMET or a simpler model judge using shared translation-quality criteria. Decisions about criterion grouping, importance weights, and forbidden-error penalties determine scores, yet their effects are not evaluated. The additional V and K dimensions overlap conceptually with translation fidelity, and their strong aggregate correlations with T provide limited evidence of distinct diagnostic value. A controlled comparison against simpler evaluators would make the contribution substantially easier to assess.

  • Multilingual annotation and evaluation reliability remain unclear. Appendix A notes that some languages were reviewed without native or proficient speakers, relying instead on related languages, dictionaries, and AI tools. Although the authors conducted two verification rounds, they do not specify which languages were affected or report annotation agreement. This raises concerns about the reliability of the annotations, particularly since they are used to generate the reference answers and evaluation rubrics.

  • The actual evaluation coverage is unclear. The paper acknowledges that fewer than the nominal 46,788 image–target pairs were scored, with coverage varying across dimensions, but does not report the final counts.

Questions for authors
  1. What measurable benefit does the proposed evaluator provide over COMET and a simpler judge model with shared translation-quality criteria? What useful errors do V and K reveal beyond T?

  2. From reviewing the example in B.2, there seems to be considerable overlap between T, V and K. How are requirements divided into criteria and assigned to T, V, or K where these dimensions overlap? What motivates the 3/2/1 weights and the rule that any forbidden error zeroes an entire criterion?

  3. Which judge model, prompt, and inference settings produced the main tables? Section 4.7 evaluates Qwen3.6 and Qwen3.8 Max against human judgments, but it is unclear which judge and configuration were used for the main tables. Does providing the image to the judge improve reliability beyond verified source text and annotations? Please reconcile Appendix B.4’s template disclaimer with the reported experiments.

  4. What conclusion are the four text-input baselines intended to support? They receive human-organized source text, use different base models from the VLM models, and have a different aggregation scheme (why?). Can the authors provide a comparison using the same model, examples, and aggregation with image versus text inputs?

2 comments

Dear Reviewer 919R,

As the rebuttal deadline is approaching, please review the authors’ response and participate in the discussion with them at your earliest convenience.

Best regards,

The AC

Thank you for the additional analyses. Given the remaining discussion time, I have two clarification requests:

  1. Annotation reliability: The revised Appendix A describes annotator backgrounds but does not resolve the original statement that some languages lacked directly proficient reviewers. Which languages lacked proficient reviewers, and how were their references and rubrics verified? For the 93% agreement and 20% inspection, please specify what was assessed and the counts relative to the totals.
  2. Human evaluation (W1/Q1): Which target languages were used, and did the evaluators also have proficiency in those languages? The authors also define human ties, but how were numerical score differences converted into win/loss/tie predictions for COMET and the rubric evaluator, and how were ties counted in the reported agreement?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.