acceptodds
Under review as a conference paper at ICLR 2027

Less Is an Act: Probing Selective Judgment in Vision-Language Models through Image Translation

Abstract

In-image machine translation (IIMT) has advanced rapidly. Existing systems and benchmarks are largely concerned with two things: the visual quality of the rendered output and the fidelity of the translation itself. However, we argue these objectives sit closer to a design-and-application problem than to a core question about model capability: when a user simply asks to “translate this image,” does the model understand what should be translated and what left untouched? Brand names, ingredient codes, and text physically part of an object must be preserved for translation to serve the user. We find that state-of-the-art vision-language models (VLMs), given no guidance, systematically mistranslate what should be left untouched, most often by over-translating: rewriting brand names and other text that competent translators would preserve. We introduce selective translation, a new task that shifts the focus from translation quality to selective judgment and restraint: deciding, for each piece of text, whether to translate or preserve it. We take a new approach to in-image translation, building SPOT-Bench, a benchmark that evaluates what a model chooses to translate rather than how well it translates, with constructional ground truth, derived from how each image is made rather than annotated by taste, across advertisements, news, and captioned video. Evauating current proprietary and open VLMs, we find a large and consistent gap between what models know the rules to be and what they do: they fail not from missing knowledge but from failing to apply it when acting on the image. Our analysis shows that making the translate/preserve decision explicit substantially narrows this gap, pointing toward more selective, context-aware VLMs.

Primary area:
datasets and benchmarks
Keywords:
In-Image Machine Translation, Selective Translation, Vision-Language Models Abstention

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_LEHTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SPOT-Bench, a benchmark designed to evaluate the selective judgment of vision-language models (VLMs) in image translation tasks. A practical image translation requires models to translate localized text (e.g., headlines, captions) while preserving identity text (e.g., brand names, logos, product marks). The paper constructs four diagnostic probes—KNOW (rule knowledge), DECISION (span classification), ACT (translation execution), and PRESERVE (identity text retention)—and evaluates both open and proprietary VLMs. Empirical findings indicate that while models perform well on rule recognition, they frequently fail during execution due to severe over-translation.

Critical strengths
  • Practical and Meaningful Failure Mode: Highlights a critical limitation ignored by traditional metrics like BLEU or COMET, specifically the unwanted translation of proprietary brand names and identifiers in image editing/translation.

  • Informative Diagnostic Analyses: Features informative contrasts, including guided vs. unguided setups, over- vs. under-translation error splits, and redundant re-translation checks on target-language inputs.

  • Controlled Annotation Pipeline: Labels are derived during stimulus synthesis and verified by human checks, reducing reliance on subjective human annotator preferences.

Critical weaknesses
  • Flawed Causal Reasoning for "Knowing vs. Doing": The claim that models "know the rule but fail to act" conflates fundamentally different task setups. In the unguided ACT condition, the model is never explicitly given the translate/preserve rule, so low preservation may simply reflect an unspecified task rather than a failure to apply a known rule. The paper lacks a crucial control: appending a single natural-language sentence stating the rule to the unguided prompt (without per-span labels and without the structured output of Bind-then-Act). Adding this control is necessary to separate "not told" from "told but not applied", showing how much of the gains in Tables 6 and 7 stem purely from instruction clarity.

  • Weak Support for Scaling Claims: The paper's claims that "model scaling does not close the gap" and "proprietary systems exhibit identical patterns" lack solid empirical backing. The tables lack sufficient same-family model scaling comparisons; the only available pair (Gemma-3 12B vs. 27B) actually narrows the gap substantially (from 24.2 down to 19.6), directly contradicting the paper's stated conclusion.

  • Evaluation Confounded by OCR Recall and Lack of Consistency Verification: Un-detected text spans are automatically counted as translation failures, confounding the "selective judgment" metric with base OCR detection performance rather than purely assessing model decision-making.

Questions for authors
  • Natural-Language Rule Control: Can you add a control condition that simply appends a one-sentence natural-language rule statement to the unguided prompt (without per-span labels or structured Bind-then-Act formatting) to isolate the effect of instruction clarity on the ACT task? Additionally, how does a setup perform when feeding the model's own DECISION predictions back into the ACT prompt?

  • Scaling Arguments Clarification: Given that Gemma-3 shows noticeable gap reduction with scale, can you provide more same-family scaling comparisons to solidly support your claims regarding model size?

  • Multi-OCR Consistency Evaluation: Can you incorporate evaluations across multiple different OCR engines to verify the consistency of your metrics and eliminate potential bias or interference from a single OCR tool?

4 comments
AuthorsOpenReview ↗

Response to Reviewer LEHT (Part 1/3)

We are grateful for your thoughtful review and for recognizing the failure mode as practical and meaningful. The controls you suggested are now in §5.3 and App. E; changes in the revised manuscript are marked in red.

Terminology and settings. The ACT metric, called PRESERVE in the submission, is now called adherence: whether the model's ACT output follows its own DECISION label on the spans it detected itself (§3.1). All main claims use the unguided setting (image only, no rules, no gold); guided ACT adds the gold labels as an upper bound, and Bind-then-Act uses no gold, letting the model list its own spans and decisions before acting, with the rules in the prompt.

W1 / Q1. - Not told, or told but not applied

Your concern was that the gap may reflect what the model was not told rather than a failure to act. The controls you suggested point to the latter: told the rule, models still leave many spans inconsistent with their own labels, whereas binding their own decisions to the spans, with no rule, closes most of the gap.

What we measure. We agree that "not told" and "told but not applied" must be separated. The told setting is well studied, with the regions to translate given in in-image MT (e.g., "AnyTrans") and terminology constraints given in text MT (e.g., "Findings of the WMT 2023 Shared Task on Machine Translation with Terminologies"); we now discuss these works in §2. SPOT-Bench targets the complementary, autonomous case: whether a model decides on its own when to keep text, as a professional translator keeps a brand name without being told to. Unguided DECISION is itself uninstructed (App. C), yet models label most spans correctly; the question is whether they act on it.

The controls. We ran your control and your second suggestion on the full benchmark. The one-sentence rule is appended to the instruction and adds external information. Structure only implements your second suggestion as a two-pass cascade: the model first performs DECISION, and its own output, the list of spans with their labels, is given back as input to ACT, so that each action is bound to the span it decided on. It is Bind-then-Act without the rules, and it adds nothing the model did not already produce.

Decision / Adherence under each control (%)

Model       Unguided         One-sentence rule       Structure only        Bind-then-Act     
InternVL3.5-8B75.4 / 51.686.0 / 59.674.0 / 73.485.3 / 78.2
Qwen3-VL-8B79.1 / 54.586.3 / 66.280.3 / 74.984.4 / 75.6
Qwen3.8-27B84.6 / 59.287.4 / 63.485.6 / 80.488.1 / 82.2
Gemma-3-12B77.0 / 49.086.3 / 63.479.9 / 77.081.3 / 77.1
Gemma-3-27B81.7 / 52.882.3 / 61.782.1 / 81.387.5 / 86.8
Pixtral-12B73.5 / 48.977.1 / 53.277.4 / 76.179.6 / 77.8
Mean78.6 / 52.784.2 / 61.379.9 / 77.284.4 / 79.6
  • Stating the rule helps, but much of the inconsistency remains. One appended sentence ("Keep brand names, product names, text printed on the product, ingredient codes, and text that belongs to the scene (signs, logos) in their original form; translate everything else.") improves both decision accuracy and adherence, yet 38.7% of detected spans remain inconsistent with the model's own labels. Structure only lowers this to 22.8% while decision accuracy stays close to the baseline (79.9% vs. 78.6%), and its adherence exceeds that of the rule in every model. Access to the rule alone is thus not enough for consistent execution; carrying the model's own decisions into ACT is what reduces the remaining inconsistency. These results are now in §5.3, Table 8, and Table 19 (App. E).
  • Bind-then-Act. We offer it as a diagnostic and a direction rather than as a method. About 90% of its gain comes from this structure, which uses only the model's own decisions, so the direction it points to does not require guiding the model. Rephrasing the instruction as "localize" or "make this image readable for X speakers" changes mean adherence by at most 0.7 points (Table 19).
AuthorsOpenReview ↗

Response to Reviewer LEHT (Part 2/3)

W2 / Q2. - Does scale close the gap?

We agree that a single pair cannot support a scaling claim, and we have revised our statement that the gap "persists, and in some cases widens, at 27B", since within Gemma-3 the gap in Table 10 in fact narrows (24.2 → 19.6). We therefore added same-family series on the full benchmark (Table 11, App. D.1).

Same-family scaling (Decision and Adherence in %; BLEU and COMET on gold-TRANSLATE spans)

Family    Size      Decision   Adherence      BLEU       COMET    
Qwen3-VL2B32.131.69.863.8
Qwen3-VL4B71.351.611.969.3
Qwen3-VL8B79.154.514.372.2
Qwen3-VL32B80.470.115.573.0
InternVL3.51B47.69.81.647.8
InternVL3.52B36.635.110.764.2
InternVL3.54B64.347.013.568.7
InternVL3.58B75.451.615.169.7
InternVL3.514B73.950.514.670.4
InternVL3.538B74.957.717.071.9
Gemma-34B58.151.210.665.5
Gemma-312B77.049.014.470.3
Gemma-327B81.752.813.870.8

Scale helps, but it does not close the gap. In Qwen3-VL and InternVL3.5, adherence rises clearly with size; in Gemma-3, decisions and translation quality improve but adherence does not. Even at the largest size, every family still leaves nearly 30% or more of spans inconsistent with the model's own label, and adherence stays below decision accuracy at every size. In some families, larger models therefore seem to reduce the gap partially, but a substantial gap remains at every size. §5.2, App. D.1, and the conclusion now state this, and we replaced the claim that proprietary systems show the same pattern with what Table 3 supports: they preserve more in rendered images than open models but still lose about a quarter of identity-bearing spans (§5.1).

AuthorsOpenReview ↗

Response to Reviewer LEHT (Part 3/3)

W3 / Q3. - Detection and OCR

Your concern was that detection errors might drive the results. First, our Fig. 2 caused a misunderstanding here: DECISION and ACT use no external OCR. The VLM reads the image and lists the text regions itself, so an undetected span is the model's own miss. The "OCR Detection" box in Fig. 2 has been corrected. In the submission, external OCR appears only when reading rendered images (Table 3) and as text input in the vision ablation (Table 5).

Detection is also reliable on our images: every model detects more than 90% of gold spans in every medium (Table 14, App. D.4).

Detection rate of gold spans (%)

Model      Ads             News           Video      
InternVL3.5-8B96.894.290.4
Qwen3-VL-8B97.197.093.2
Qwen3.8-27B97.496.394.3
Gemma-3-12B95.695.394.0
Gemma-3-27B96.594.195.6
Pixtral-12B94.491.490.8
Mean96.394.793.1

Adherence counts only spans the model itself detected, so these misses do not enter it. DECISION accuracy counts an undetected gold span as a failure, by design, so that a model cannot appear accurate by skipping hard text; at these detection rates, the effect is small.

To remove model-specific detection entirely, as you suggested, we also added a cascade setting: an external OCR engine draws boxes around the text regions it finds, and every model receives the same boxed image (Table 15, App. D.4).

Cascade with external OCR (Decision / Adherence, %)

Model  Own detection   PP-OCRv5 (PaddleOCR)     EasyOCR            Surya       
InternVL3.5-8B75.4 / 51.670.7 / 50.568.5 / 47.761.5 / 46.0
Qwen3-VL-8B79.1 / 54.577.3 / 59.271.2 / 54.575.2 / 39.8
Qwen3.8-27B84.6 / 59.289.9 / 62.588.4 / 55.084.6 / 50.5
Gemma-3-12B77.0 / 49.076.2 / 50.080.4 / 44.773.5 / 47.5
Gemma-3-27B81.7 / 52.888.5 / 49.483.3 / 47.378.7 / 48.4
Pixtral-12B73.5 / 48.975.8 / 50.569.4 / 48.968.0 / 48.6
Mean78.6 / 52.779.7 / 53.776.9 / 49.773.6 / 46.8

Individual models shift with the engine, since each engine boxes different regions, but the finding holds under every engine: mean adherence stays close to that with the models' own detection, and every model still leaves more than a third of spans inconsistent with its own label. The gap therefore does not stem from a particular detector.

 

The controls you proposed made the central claim sharper, and we hope the revision addresses your concerns.

Dear Reviewer LEHT,

Thank you again for your thoughtful review and the controls you proposed. In our rebuttal, we addressed each of your points as follows:

  • Not told vs. told but not applied (W1/Q1): with your one-sentence rule appended, 38.7% of detected spans remain inconsistent with the model's own labels; feeding the model's own DECISION output back into ACT, with no rule, lowers this to 22.8% (Tables 8, 19).
  • Scaling (W2/Q2): same-family series for Qwen3-VL, InternVL3.5, and Gemma-3 show that larger models reduce the gap only partially, and a substantial gap remains at every size (Table 11). Our scaling and proprietary-model claims are revised accordingly.
  • Detection and OCR (W3/Q3): DECISION and ACT use no external OCR (Fig. 2 corrected), every model detects more than 90% of gold spans, and a cascade with three external OCR engines leads to the same finding (Tables 14, 15).

Thank you for reading our work so closely and for asking the questions that mattered most. If any doubt remains, we would welcome the chance to talk it through before the discussion closes.

With warm thanks and best wishes, The Authors

Reviewer_ynWkOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the task of selective translation, an extended image translation task that tests if VLMs can withhold from translating unnecessarily text regions (over-translation). For this task, the authors construct SPOT-Bench of 5K annotated images across six languages. The authors probe each VLM in four distinct aspects, focusing on seeing if the model knows what not to translate, and if it actually performs the withholding action (PRESERVE). The authors find that most of the evaluated VLMs fail at the PRESERVE stage, and the translation outputs are dominated by the over-translation phenomenon.

Critical strengths

I find this paper presents a real problem in a clear way. Most notably,

  • The Research Question (RQ) is well scoped and clearly motivated. I find it follows reasonable logics to read through, all thanks to the authors’ explanations. I’m able to understand why the over-translation is a valuable problem to study for.
Critical weaknesses

I find the paper holds potential merits. But I have a serious doubt on the core error mode that the authors aim to address.

Critical Design Weakness: The Authors Overlook ‘Transliteration’ and Treat It Simply as Over-translation.

Transliteration, or phonetic translation, is a legitimate procedure in localization. To give an anecdotal reference, international brand names are commonly transliterated when it comes to the Chinese market, e.g. Coca-Cola → 可口可乐 (ke3 kou3 ke3 le4), or Estée Lauder → 雅诗兰黛 (ya3 shi1 lan2 dai4).

Therefore, I find the authors’ binary TRANSLATE-or-PRESERVE scheme ambiguous when the human annotators choose to apply transliteration as the proper translation. In addition, Figure 6 in the appendix overwhelmingly counts the transliterated brand name in Hangul simply as an error. As a result, I am afraid that the reported over-translation rate (i.e. Table 5), which is supposed to measure the occurrence of unnecessary translations, is inflated due to the overlook of transliteration.

Ref: https://columbia.demo.elsevierpure.com/en/publications/creating-local-brands-in-multilingual-international-markets-3

Questions for authors

Given the data sources and the example showcases in this paper, I would guess the authors are likely based in South Korea (given that Chosun Ilbo and Dong-A Ilbo are two mainstream newspapers in South Korea). Feel free to correct me if I’m wrong.

Questions.

  1. Regarding the aforementioned weakness, is it possible to report these three Decision outcomes, i.e., preserved or left untouched as-is, transliterated, and semantically translated?
  2. The authors have not yet given any details about what languages the three professional human annotators can understand. What are their native languages? It may need to take a native in the target language to fully understand all the subtle conventions when it comes to what not to translate.

Misc.

I believe the authors may still find the classic translation theories established by Monk Xuanzang 玄奘 much relatable, who played a significant role translating Buddhist mantras from Sankrit into Chinese back in the Tang dynasty. He standardized the theory of Five Untranslatables 五种不翻 - literally what to preseve and transliterate - as a protocol of text-based localization. See: https://maxryynanen.net/wp-content/uploads/2021/11/JCLA-44.1_Spring-2021-Complete-Issue-1.pdf#page=120

With all being said, I'm very interested in following up with the authors in the rebuttal period.

5 comments
AuthorsOpenReview ↗

Response to Reviewer ynWk

Thank you for the thoughtful and engaging review. We were glad you found the problem interesting, and your point on transliteration will make the benchmark more faithful to practice.

A note on terminology. The ACT metric was called PRESERVE in the submission; we have renamed it Adherence, because it measures whether the model's translation follows its own DECISION label (kept if PRESERVE, changed if TRANSLATE) on the spans the model itself detected (§3.1).

W. - Transliteration

You are right that the submission treated every transliteration as over-translation. Transliteration is not a convention in every language, but where it is established practice, as in many Asian markets and especially in Chinese (e.g., Coca-Cola → 可口可乐, Estée Lauder → 雅诗兰黛), it should not be penalized. We now cite "Creating Local Brands in Multilingual International Markets" in §2 and have re-scored all outputs, accepting a transliteration whenever the brand has an official local form in the target market (checked automatically against official target-market sources and verified by hand). Text printed on the product itself (logos, packaging) still counts as correct only if unchanged, since the image depicts the physical object. Fig. 7 (Fig. 6 in the submission) is unchanged, as the practice varies by brand: Mardi Mercredi, the brand shown there, keeps its original name in its official Korean and Chinese advertising.

For Table 6, which you cited, transliteration can inflate the over-translation rate only slightly. A DECISION label alone cannot show whether TRANSLATE meant transliteration, but even if every transliterated output in Q1 came from such a label, the affected share of brand spans would be at most the Transliterated column, under 6% for every model.

 

Q1. - Kept, transliterated, or translated

Yes. For brand and product names in ad copy, each ACT output is now classified as kept, transliterated, or semantically translated (Table 17, App. D.6).

ACT outputs for brand and product names in ad copy (%)

Model          Kept               Transliterated     Semantically translated 
InternVL3.5-8B53.82.543.7
Qwen3-VL-8B55.84.739.5
Qwen3.8-27B59.43.137.5
Gemma-3-12B50.55.743.8
Gemma-3-27B56.24.539.3
Pixtral-12B51.83.544.7
Mean54.64.041.4

Transliteration is rare compared with semantic translation, so accepting official transliterations raises adherence only slightly (Table 18).

Adherence under strict and lenient scoring (%)

Model         Strict         Official transliteration accepted
InternVL3.5-8B51.654.0
Qwen3-VL-8B54.555.7
Qwen3.8-27B59.262.4
Gemma-3-12B49.051.3
Gemma-3-27B52.857.2
Pixtral-12B48.950.5
Mean52.755.2

Among the evaluated brand and product names in ad copy, semantic translation is much more frequent than transliteration. Our central observation of substantial decision–action inconsistency remains under the more permissive scoring.

 

Q2. - Annotators

We agree that conventions of what not to translate are best judged by speakers of the target language. Double-blind review limits how much we can say about the annotators here, but we can share the following. Five of our six languages were checked by native speakers or annotators with university-level proficiency, and the sixth relies on sources with official translations. Language judgment also enters our labels less than it may seem. Gold TRANSLATE/PRESERVE labels follow from how each image is constructed (which layer a text belongs to), not from an annotator's preference, and whether a transliteration is accepted depends on whether the local form is official, which is checked against the brand's own target-market sources rather than judged by an annotator. We will describe the annotators fully in the final version.

We also appreciated the pointer to Xuanzang's Five Untranslatables (五种不翻), an early precedent for this decision, which we now cite in §2 through the source you pointed to.

 

We enjoyed this exchange and would be glad to continue the discussion.

Reviewer_ynWkOpenReview ↗

One more question regarding the effect of transliteration...

I appreciate the authors' response to my concerns. I would like to lock in on one last issue stemming from my original Weakness/Q1 point. I feel there is still one missing piece left unaddressed despite the additional results.

Q1, missing piece. Transliteration can be applied beyond just brand or product names. Even when restricted to ads, which is a 40% split of all images according to Figure 3, I wouldn't call an average shift of ~4.0% rare or neglegible anymore.

By my observation, under lenient scoring (i.e. accepting the official transliterations), Gemma-3-27B will overtake Qwen3-VL-8B in Table 18. The models' gains from strict to lenient range from 1.2% to 4.4% in the second table of the response, so strict scoring (without transliteration considered) penalizes some tested models more than others.

Can the authors help confirm the transliteration's percantage among the entire benchmark (i.e. all 5k of them), rather than just within the ad subset, if possible?

AuthorsOpenReview ↗

Response to Reviewer ynWk: Transliteration across the full benchmark

Thank you for your follow-up, and for continuing the discussion. Your question made us go back through every image in the benchmark, and we are grateful for what it showed us.

Scope of our earlier tables. Table 17 is restricted to brand and product names in ad copy, but Table 18 already reports every model on all 5,000 images, with legitimate official transliterations accepted. Over the full benchmark, this raises mean adherence from 52.7% to 55.2%.

Transliteration across the full benchmark. You are right that transliteration applies well beyond brand and product names: names of people, places, and organizations, titles of works, and loanwords are often rendered by sound. Following your suggestion, we marked every such expression in the gold annotations of all images and counted them both word by word and as spans that consist of one. Since it arises only between different scripts, we count directions into or out of a non-Latin script (e.g., Chinese, Korean).

Transliteration-relevant expressions by gold label (5,000 images: 2,000 ads, 2,000 news, 1,000 video; %)

Image typeWordsSpans
CountTRANSLATEPRESERVECountTRANSLATEPRESERVE
Ads2,48827.073.01,07414.585.5
News9,35698.02.01635.594.5
Video34016.583.51870.599.5
All12,18481.218.81,42411.788.3

Such expressions are more common than our earlier wording implied. Counted word by word, most of them (81.2%) sit inside TRANSLATE text, mainly news bodies, as single words in translated sentences. Counted as spans, where the region is itself the expression, they are concentrated in PRESERVE regions (88.3%), mostly scene or product text where only the original form is correct. News mastheads are excluded.

ACT outputs on such spans with gold PRESERVE labels (%; K: kept, T: transliterated, S: semantically translated; All: T over all spans)

Model Ads (K / T / S)  News (K / T / S)  Video (K / T / S) All (T)
InternVL3.5-8B52.7 / 3.7 / 43.624.7 / 2.6 / 72.770.7 / 3.6 / 25.73.6
Qwen3-VL-8B54.3 / 6.4 / 39.324.7 / 5.2 / 70.173.3 / 2.8 / 23.95.7
Qwen3.8-27B57.8 / 5.1 / 37.128.6 / 1.3 / 70.178.3 / 4.5 / 17.24.5
Gemma-3-12B49.7 / 6.9 / 43.423.4 / 2.6 / 74.074.4 / 5.7 / 19.96.2
Gemma-3-27B54.3 / 6.8 / 38.924.7 / 2.6 / 72.774.3 / 2.0 / 23.75.6
Pixtral-12B49.4 / 5.9 / 44.727.3 / 1.3 / 71.470.7 / 5.1 / 24.25.2
Mean53.0 / 5.8 / 41.225.6 / 2.6 / 71.873.6 / 4.0 / 22.45.1

Across the full benchmark, 5.1% of these outputs are transliterations.

What our metric evaluates. Our labels are defined per region: the task asks whether a region should be translated, not how each word in it is rendered. A news body is a single paragraph, while text on a product or in a scene is split into short separate units, so a region holding a brand name is usually the brand name itself.

  • Inside a TRANSLATE region, such as a news body or a caption, how well a name is transliterated is a matter of translation quality, a worthwhile topic evaluated by MT and IIMT benchmarks; our metric checks only that the region is translated.
  • In a PRESERVE region of the depicted product or scene, such as a brand on a product or a sign in a news photo, only the original form is correct.
  • For brand mentions in ad copy, an official local form is legitimate localization, and our lenient scoring accepts it (Table 18).

Where the ranking change comes from. Your observation on Gemma-3-27B and Qwen3-VL-8B is correct. Lenient scoring changes nothing for a model that already kept a name in its original form; it raises only models that wrote an official transliteration, which strict scoring counted as an error. For Nike, a model that wrote the official Chinese form "耐克" now gains, while one that kept "Nike" does not. Where a model transliterates therefore matters more than how often. The two transliterate at nearly the same overall rate (5.7% vs. 5.6%), but Gemma-3-27B does so more in ads (6.8% vs. 6.4%), where official forms are credited, and Qwen3-VL-8B more in news and video, where they are not. Adherence is also averaged per image, so spans in images with few regions weigh more. Together, these help explain the range of 1.2 to 4.4 points. The ranking changes only for this adjacent pair, 1.7 points apart under strict scoring, and adherence stays well below decision accuracy for every model.

On "rare". This shows that transliteration can change how models compare, and we will treat it with more care. We therefore no longer describe it as rare. Many such expressions must not be changed at all, but where transliteration is legitimate, identifying it and scoring it separately is a complement to our metric that your question helped us build. We have added this analysis to App. D.6.

Thank you again for engaging so closely with our work.

Reviewer_at9zOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces selective translation. When asked to translate an image, a VLM should translate localizable text (headlines, copy, captions) but preserve identity-bearing text (brand names, ingredient codes, in-scene signage). It builds SPOT-Bench: 5,000 ad, news and video images in six languages, whose translate/preserve labels follow from how each image is constructed and are validated by professional translators. Separate probes test knowing the rules, deciding per span, and acting while translating. Across open and proprietary VLMs, models decide fairly well but over-translate when acting. The authors attribute this to a knowing-doing gap, which a guided inventory or a Bind-then-Act prompt largely closes.

Critical strengths
  1. A real, under-measured problem. Existing IIMT benchmarks assume the translation target is given. The paper shows convincingly that deciding what to translate matters for factual integrity, using brand names, INCI codes, diegetic signs and text already in the target language. Over-translation is observed consistently across open and proprietary models.
  2. A scalable route to non-subjective labels. Labels derived from image construction, validated against professional translators and spread over three media with different reasons for preservation, could make SPOT-Bench a useful resource even if the analysis claims are revised.
  3. A substantial benchmark effort. The coverage of three media types and six languages, together with independent validation by three professional translators on 600 images (Appendix B), provides a potentially reusable resource beyond isolated failure examples.
Critical weaknesses

The benchmark addresses a meaningful evaluation problem, but the current evidence does not sufficiently isolate the claimed binding failure. My concerns are as follows.

  1. The interventions do not isolate decision structure. Section 5.3 describes the guided inventory as non-privileged, whereas Appendix C states that it contains gold source strings and translate/preserve labels. This provides correct recognition and decision information, not merely the model’s own predictions. Bind-then-Act also introduces explicit rules and changes the instruction from translation to localization (Appendices C and E; Figure 7). The gains therefore cannot be attributed to decision structure alone.

  2. The reported gaps do not directly measure correct decisions being violated. Table 2 compares DECISION accuracy over both labels with ACT preservation accuracy on gold-preserve spans. Their difference is not a paired measure of decision-to-output inconsistency. Table 9’s Pres@Dec and Act-P strengthen the evidence for a cross-task discrepancy, but do not uniquely establish binding failure rather than differences in recognition or task interpretation.

  3. The KNOW-rules protocol is inconsistently documented. Appendix A asks whether a localization rule is correct. For A1, “Brand names must stay in their original form,” agreement should be YES, yet Table 1 uses P (NO) as the gold answer. This may only be a documentation error, but until clarified, it leaves uncertainty about the evidence that models endorse the benchmark’s rules.

  4. Translation quality under the intervention is unclear. Table 7 reports decision and preservation gains for Bind-then-Act but does not evaluate translation quality on gold-TRANSLATE spans. Without matched translation-quality or omission results, it is unclear whether improved preservation comes at the expense of required translations.

  5. Region matching for repeated text is underspecified. The benchmark includes identical strings requiring different actions based on location (Section 3.3), yet Section 4 describes alignment only by normalized string matching. Without spatial disambiguation, region-level scores may be unreliable in these cases.

Questions for authors

The following clarifications and evidence could change my assessment.

Q1. Under matched task wording and rule information, how does Bind-then-Act compare with a rules-only baseline? How does Guided ACT perform when supplied with the model’s predicted inventory rather than the gold inventory?

Q2. On the same gold-preserve spans, how often does ACT change text that was correctly recognized and classified as PRESERVE in DECISION? Please report paired results and clarify how recognition errors are handled.

Q3. Was KNOW-rules asking whether the rule was correct or whether the text should be translated? Please provide the actual prompt and answer mapping, and clarify whether resolving the inconsistency changes the reported scores.

4 comments
AuthorsOpenReview ↗

Response to Reviewer at9z (Part 1/3)

We appreciate your detailed review and your recognition of SPOT-Bench as a real, under-measured problem. We begin with W2, since it clarifies a term used throughout; changes in the revised manuscript are marked in red.

W2 / Q2. - Paired evidence of the binding failure

Terminology. We apologize for the confusion caused by our terminology, which underlies several of your points. We used "PRESERVE" both as a span label and as the name of the ACT metric. As a metric, "preserve" meant that the model preserves its own decision at ACT: it keeps a span it labeled PRESERVE and changes a span it labeled TRANSLATE. Using the same word as the label, together with captions that wrongly described the metric as gold-preserve accuracy, hid this meaning. We have therefore renamed the metric adherence: for each span the model itself detected at DECISION, whether its ACT output follows its own label. Undetected spans are excluded, and detected spans missing from the output count as omitted. "PRESERVE" now denotes a label only (§3.1, Table 2, Fig. 2).

Paired result. You are right, however, that the Δ column of Table 2 and the Pres@Dec–Act-P gap (now Table 10) subtract separately computed rates and do not show decisions being violated. We removed the Δ column and computed the paired result you asked for (Table 4, §5.1).

ACT outcomes on gold-PRESERVE spans the model labeled PRESERVE at DECISION (%)

Model     n (of 9,425)              Kept               Translated             Omitted        
InternVL3.5-8B6,68441.353.65.1
Qwen3-VL-8B6,82548.549.61.9
Qwen3.8-27B7,50661.536.91.6
Gemma-3-12B7,37750.746.92.4
Gemma-3-27B7,14859.738.81.5
Pixtral-12B6,36542.551.06.5
Mean–50.746.13.2

To answer Q2 directly: ACT translates or omits 49.3% of these spans on average (Translated + Omitted), and even the best model, Qwen3.8-27B, does so for 38.5%. The model recognized each of these spans and judged that it should be kept, yet about half the time it did not keep it.

Detection and labeling errors are excluded by construction: the table counts only gold-PRESERVE spans that the model itself detected and labeled PRESERVE. Spans missing from the ACT output, including empty or malformed output, count as omitted, not translated.

Your remaining concern, task interpretation, is answered in W1: with the same "translate" instruction, binding the model's own decisions to the spans raises adherence substantially while its decisions barely change.

 

W3 / Q3. - KNOW-rules prompt and answer mapping

We are glad you caught this. The probe asked, in effect, whether each kind of text should be translated, and the scores are unchanged; the inconsistency came from our presentation in App. A, which was not sufficiently clear. Table 9 summarizes each rule in its gold form (e.g., "Brand names must stay in their original form"), but every statement shown to the models claimed that a kind of text should be translated (e.g., "When localizing a product advertisement, brand names should be translated."). The model was asked whether it agrees and answered YES or NO with a short explanation; only the YES/NO answer was scored. The expected answer is thus NO for the six PRESERVE rules and YES for the three TRANSLATE rules, as in Table 1. The revised App. A now states this and shows the exact prompt, all nine statements, and example raw outputs (Figure 4).

AuthorsOpenReview ↗

Response to Reviewer at9z (Part 2/3)

W1 / Q1. - What drives the Bind-then-Act gain

Settings. All main claims rest on the unguided setting. The settings compared below are:

  • Unguided: the image only, with no gold information and no rules.
  • Guided (Oracle): adds the gold inventory, i.e., the list of text spans with their gold translate/preserve labels (App. C). It serves only as an upper bound.
  • Bind-then-Act: no gold. The model lists its own spans and decisions and then acts on them, with the rules in the prompt.
  • Rules only: the rules, without the per-span structure.
  • Structure only: the structure, without the rules. A two-pass cascade in which the model's own DECISION output is given back as input to ACT.

The "non-privileged" sentence in §5.3 meant that the model produces the same kind of inventory itself at DECISION; placed after the guided result, it read as a claim about the guided setting, so we have rewritten it. Since Bind-then-Act also added the rules and the word "localize", we re-ran all variants on the full benchmark with the same instruction, "Translate this image into X".

Components of Bind-then-Act (Decision / Adherence, %)

Model       Baseline             Rules only          Structure only        Bind-then-Act     
InternVL3.5-8B75.4 / 51.686.0 / 62.074.0 / 73.485.3 / 78.2
Qwen3-VL-8B79.1 / 54.587.4 / 68.180.3 / 74.984.4 / 75.6
Qwen3.8-27B84.6 / 59.286.9 / 74.085.6 / 80.488.1 / 82.2
Gemma-3-12B77.0 / 49.086.9 / 74.879.9 / 77.081.3 / 77.1
Gemma-3-27B81.7 / 52.887.7 / 75.482.1 / 81.387.5 / 86.8
Pixtral-12B73.5 / 48.982.8 / 69.277.4 / 76.179.6 / 77.8
Mean78.6 / 52.786.3 / 70.679.9 / 77.284.4 / 79.6

Your concern was that the rules, not the structure, might explain the gain. With the wording fixed, the rules alone raise adherence less than the structure alone in every model, and they mainly improve decisions. The structure alone nearly closes the gap between decision and adherence, whereas the rules alone leave much of it. Table 8, §5.3, and App. E now report this and attribute the gain mainly, not solely, to the structure.

 

Own vs. gold inventory (Decision / Adherence, %)

ModelOwn inventory (no rules)Gold inventory (Oracle)
InternVL3.5-8B74.0 / 73.493.3 / 92.0
Qwen3-VL-8B80.3 / 74.995.4 / 93.3
Qwen3.8-27B85.6 / 80.497.5 / 97.3
Gemma-3-12B79.9 / 77.093.8 / 90.8
Gemma-3-27B82.1 / 81.396.5 / 95.5
Pixtral-12B77.4 / 76.195.8 / 93.7
Mean79.9 / 77.295.4 / 93.8

This answers the second part of Q1. With its own predicted inventory instead of the gold one, and no rules, the model already raises adherence well above the baseline while its decisions barely change. The gain thus does not depend on correct information being supplied; the remaining gap to the gold inventory reflects the model's own errors. Table 7 now reports the gold inventory on the full benchmark.

 

Instruction wording (Decision / Adherence, %)

Model      Translate              Localize           Make readable     
InternVL3.5-8B75.4 / 51.676.7 / 51.570.0 / 51.1
Qwen3-VL-8B79.1 / 54.572.5 / 50.972.8 / 49.6
Qwen3.8-27B84.6 / 59.285.2 / 64.881.4 / 63.7
Gemma-3-12B77.0 / 49.079.6 / 47.082.8 / 54.4
Gemma-3-27B81.7 / 52.884.3 / 54.483.3 / 51.3
Pixtral-12B73.5 / 48.970.5 / 48.273.5 / 50.5
Mean78.6 / 52.778.1 / 52.877.3 / 53.4

Replacing "translate" with "localize" or "make readable" changes mean adherence by at most 0.7 points, with no consistent direction, so the gain does not come from the word "localize" (Table 19, App. E).

AuthorsOpenReview ↗

Response to Reviewer at9z (Part 3/3)

W4. - Translation quality on TRANSLATE spans

This concern also stems from our terminology (W2). The "Pres." column of our Bind-then-Act table (now Table 8, adherence) was not computed on gold-preserve spans only. It covers every span the model detected, and a span the model labeled TRANSLATE counts only if its output actually changes it. A model that gained by simply leaving more text untranslated would therefore lose adherence on its TRANSLATE spans, and its Decision accuracy, which counts gold-TRANSLATE spans labeled PRESERVE as errors, would drop; both rise under Bind-then-Act.

Adherence shows that TRANSLATE spans are changed, however, not how well they are translated. As you suggested, we therefore evaluated translation quality on the same gold-TRANSLATE spans under each setting (Table 16, App. D.5).

Translation quality on gold-TRANSLATE spans (COMET / BLEU / chrF++)

Model        Unguided             Bind-then-Act               Oracle         
InternVL3.5-8B69.7 / 15.1 / 24.368.9 / 16.1 / 24.476.4 / 19.5 / 31.8
Qwen3-VL-8B72.2 / 14.3 / 24.172.9 / 16.4 / 25.978.1 / 19.4 / 30.9
Qwen3.8-27B73.6 / 16.5 / 26.574.1 / 18.7 / 29.479.3 / 22.8 / 36.4
Gemma-3-12B70.3 / 14.4 / 23.968.5 / 13.0 / 23.179.2 / 20.5 / 33.2
Gemma-3-27B70.8 / 13.8 / 24.370.6 / 15.2 / 25.680.2 / 21.6 / 35.9
Pixtral-12B57.9 / 9.5 / 14.456.5 / 9.3 / 14.177.2 / 17.7 / 30.7
Mean69.1 / 13.9 / 22.968.6 / 14.8 / 23.878.4 / 20.3 / 33.2

Translation quality stays essentially unchanged under Bind-then-Act: mean COMET moves from 69.1 to 68.6, with per-model changes within 1.8 points, and mean BLEU and chrF++ rise slightly. Improved adherence therefore does not come at the expense of the required translations.

 

W5. - Matching outputs to gold spans

This is a fair point: string matching alone cannot tell which output belongs to which span when the same text appears more than once. Spans are matched by region, not by string; the sentence in §4, which was not sufficiently clear, describes only the text comparison that follows. Every gold span is annotated with its region in the image (position and role, e.g., the overlaid caption vs. the shop sign). For each region it reads, the model reports the source text, its output, and a brief description of where the region appears, and each reported region is matched to the gold span at that position. Only then is the text compared after normalization: a PRESERVE span passes if its output is unchanged. The rendered images of Table 3 are checked against the same gold regions.

Repeated strings are also rare: they occur in 122 of the 5,000 images (2.4%; 78 of 2,000 ads, 23 of 2,000 news pages, 21 of 1,000 captioned-video frames), so even if all of them were scored wrongly, image-level scores would change by at most 2.4 points. §4 and App. C now describe the region matching explicitly.

 

Your review shaped much of the revision: the clarified metric, the paired analysis (Table 4), and the component ablations (Table 8) now form the core evidence of the paper. We hope these changes address your concerns and would be glad to discuss any of them further.

Dear Reviewer at9z,

Thank you again for your detailed review and constructive suggestions. In our rebuttal, we addressed each of your points as follows:

  • Paired evidence (W2/Q2): on gold-PRESERVE spans that the model itself recognized and labeled PRESERVE, ACT still translates or omits 49.3% on average (Table 4). The ACT metric is renamed adherence to make this paired definition explicit.
  • Decision structure (W1/Q1): under the same "translate" instruction, structure alone raises adherence more than rules alone in every model, using only the model's own inventory. Rewording the instruction changes mean adherence by at most 0.7 points.
  • Translation quality (W4): COMET, BLEU, and chrF++ on gold-TRANSLATE spans stay essentially unchanged under Bind-then-Act (Table 16).
  • KNOW-rules and region matching (W3, W5): the prompt, answer mapping, and region-based matching are now described explicitly; the reported scores are unchanged.

We know this is a busy season, and we are grateful for the time you have already given to our work. If anything still feels unresolved, we would be truly glad to continue the conversation before the discussion period closes.

With warm thanks, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We are glad that all three reviewers recognized the problem SPOT-Bench addresses: deciding what not to translate, a real problem that existing benchmarks leave unmeasured. The reviewers found that the paper "shows convincingly" that this decision matters for factual integrity (at9z), that the research question is "well scoped and clearly motivated" (ynWk), and that it exposes a failure "ignored by traditional metrics like BLEU or COMET" (LEHT).

We especially appreciate that the reviewers recognized the effort behind the two choices at the core of our design. Labels follow from how each image is constructed, and we validated them with professional translators across three media and six languages; we are grateful that this work was seen as a scalable, controlled route to non-subjective labels and as a resource that can be reused beyond isolated failure examples. Because knowing, deciding, and acting are probed separately, the diagnostics can explain the failure rather than only measure it, revealing over-translation consistently across open and proprietary models. We are also glad that all three reviewers found the paper clear and easy to follow.

We sincerely thank all three reviewers for their time and for engaging so closely with our work. Their questions led to new experiments that make our central claim sharper, as detailed in the individual responses; every change in the revised manuscript is marked in red.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.