- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces SPOT-Bench, a benchmark designed to evaluate the selective judgment of vision-language models (VLMs) in image translation tasks. A practical image translation requires models to translate localized text (e.g., headlines, captions) while preserving identity text (e.g., brand names, logos, product marks). The paper constructs four diagnostic probes—KNOW (rule knowledge), DECISION (span classification), ACT (translation execution), and PRESERVE (identity text retention)—and evaluates both open and proprietary VLMs. Empirical findings indicate that while models perform well on rule recognition, they frequently fail during execution due to severe over-translation.
Critical strengths
-
Practical and Meaningful Failure Mode: Highlights a critical limitation ignored by traditional metrics like BLEU or COMET, specifically the unwanted translation of proprietary brand names and identifiers in image editing/translation.
-
Informative Diagnostic Analyses: Features informative contrasts, including guided vs. unguided setups, over- vs. under-translation error splits, and redundant re-translation checks on target-language inputs.
-
Controlled Annotation Pipeline: Labels are derived during stimulus synthesis and verified by human checks, reducing reliance on subjective human annotator preferences.
Critical weaknesses
-
Flawed Causal Reasoning for "Knowing vs. Doing": The claim that models "know the rule but fail to act" conflates fundamentally different task setups. In the unguided ACT condition, the model is never explicitly given the translate/preserve rule, so low preservation may simply reflect an unspecified task rather than a failure to apply a known rule. The paper lacks a crucial control: appending a single natural-language sentence stating the rule to the unguided prompt (without per-span labels and without the structured output of Bind-then-Act). Adding this control is necessary to separate "not told" from "told but not applied", showing how much of the gains in Tables 6 and 7 stem purely from instruction clarity.
-
Weak Support for Scaling Claims: The paper's claims that "model scaling does not close the gap" and "proprietary systems exhibit identical patterns" lack solid empirical backing. The tables lack sufficient same-family model scaling comparisons; the only available pair (Gemma-3 12B vs. 27B) actually narrows the gap substantially (from 24.2 down to 19.6), directly contradicting the paper's stated conclusion.
-
Evaluation Confounded by OCR Recall and Lack of Consistency Verification: Un-detected text spans are automatically counted as translation failures, confounding the "selective judgment" metric with base OCR detection performance rather than purely assessing model decision-making.
Questions for authors
-
Natural-Language Rule Control: Can you add a control condition that simply appends a one-sentence natural-language rule statement to the unguided prompt (without per-span labels or structured Bind-then-Act formatting) to isolate the effect of instruction clarity on the ACT task? Additionally, how does a setup perform when feeding the model's own DECISION predictions back into the ACT prompt?
-
Scaling Arguments Clarification: Given that Gemma-3 shows noticeable gap reduction with scale, can you provide more same-family scaling comparisons to solidly support your claims regarding model size?
-
Multi-OCR Consistency Evaluation: Can you incorporate evaluations across multiple different OCR engines to verify the consistency of your metrics and eliminate potential bias or interference from a single OCR tool?
4 comments
Response to Reviewer LEHT (Part 2/3)
W2 / Q2. - Does scale close the gap?
We agree that a single pair cannot support a scaling claim, and we have revised our statement that the gap "persists, and in some cases widens, at 27B", since within Gemma-3 the gap in Table 10 in fact narrows (24.2 → 19.6). We therefore added same-family series on the full benchmark (Table 11, App. D.1).
Same-family scaling (Decision and Adherence in %; BLEU and COMET on gold-TRANSLATE spans)
| Family | Size | Decision | Adherence | BLEU | COMET |
|---|---|---|---|---|---|
| Qwen3-VL | 2B | 32.1 | 31.6 | 9.8 | 63.8 |
| Qwen3-VL | 4B | 71.3 | 51.6 | 11.9 | 69.3 |
| Qwen3-VL | 8B | 79.1 | 54.5 | 14.3 | 72.2 |
| Qwen3-VL | 32B | 80.4 | 70.1 | 15.5 | 73.0 |
| InternVL3.5 | 1B | 47.6 | 9.8 | 1.6 | 47.8 |
| InternVL3.5 | 2B | 36.6 | 35.1 | 10.7 | 64.2 |
| InternVL3.5 | 4B | 64.3 | 47.0 | 13.5 | 68.7 |
| InternVL3.5 | 8B | 75.4 | 51.6 | 15.1 | 69.7 |
| InternVL3.5 | 14B | 73.9 | 50.5 | 14.6 | 70.4 |
| InternVL3.5 | 38B | 74.9 | 57.7 | 17.0 | 71.9 |
| Gemma-3 | 4B | 58.1 | 51.2 | 10.6 | 65.5 |
| Gemma-3 | 12B | 77.0 | 49.0 | 14.4 | 70.3 |
| Gemma-3 | 27B | 81.7 | 52.8 | 13.8 | 70.8 |
Scale helps, but it does not close the gap. In Qwen3-VL and InternVL3.5, adherence rises clearly with size; in Gemma-3, decisions and translation quality improve but adherence does not. Even at the largest size, every family still leaves nearly 30% or more of spans inconsistent with the model's own label, and adherence stays below decision accuracy at every size. In some families, larger models therefore seem to reduce the gap partially, but a substantial gap remains at every size. §5.2, App. D.1, and the conclusion now state this, and we replaced the claim that proprietary systems show the same pattern with what Table 3 supports: they preserve more in rendered images than open models but still lose about a quarter of identity-bearing spans (§5.1).
Response to Reviewer LEHT (Part 3/3)
W3 / Q3. - Detection and OCR
Your concern was that detection errors might drive the results. First, our Fig. 2 caused a misunderstanding here: DECISION and ACT use no external OCR. The VLM reads the image and lists the text regions itself, so an undetected span is the model's own miss. The "OCR Detection" box in Fig. 2 has been corrected. In the submission, external OCR appears only when reading rendered images (Table 3) and as text input in the vision ablation (Table 5).
Detection is also reliable on our images: every model detects more than 90% of gold spans in every medium (Table 14, App. D.4).
Detection rate of gold spans (%)
| Model | Ads | News | Video |
|---|---|---|---|
| InternVL3.5-8B | 96.8 | 94.2 | 90.4 |
| Qwen3-VL-8B | 97.1 | 97.0 | 93.2 |
| Qwen3.8-27B | 97.4 | 96.3 | 94.3 |
| Gemma-3-12B | 95.6 | 95.3 | 94.0 |
| Gemma-3-27B | 96.5 | 94.1 | 95.6 |
| Pixtral-12B | 94.4 | 91.4 | 90.8 |
| Mean | 96.3 | 94.7 | 93.1 |
Adherence counts only spans the model itself detected, so these misses do not enter it. DECISION accuracy counts an undetected gold span as a failure, by design, so that a model cannot appear accurate by skipping hard text; at these detection rates, the effect is small.
To remove model-specific detection entirely, as you suggested, we also added a cascade setting: an external OCR engine draws boxes around the text regions it finds, and every model receives the same boxed image (Table 15, App. D.4).
Cascade with external OCR (Decision / Adherence, %)
| Model | Own detection | PP-OCRv5 (PaddleOCR) | EasyOCR | Surya |
|---|---|---|---|---|
| InternVL3.5-8B | 75.4 / 51.6 | 70.7 / 50.5 | 68.5 / 47.7 | 61.5 / 46.0 |
| Qwen3-VL-8B | 79.1 / 54.5 | 77.3 / 59.2 | 71.2 / 54.5 | 75.2 / 39.8 |
| Qwen3.8-27B | 84.6 / 59.2 | 89.9 / 62.5 | 88.4 / 55.0 | 84.6 / 50.5 |
| Gemma-3-12B | 77.0 / 49.0 | 76.2 / 50.0 | 80.4 / 44.7 | 73.5 / 47.5 |
| Gemma-3-27B | 81.7 / 52.8 | 88.5 / 49.4 | 83.3 / 47.3 | 78.7 / 48.4 |
| Pixtral-12B | 73.5 / 48.9 | 75.8 / 50.5 | 69.4 / 48.9 | 68.0 / 48.6 |
| Mean | 78.6 / 52.7 | 79.7 / 53.7 | 76.9 / 49.7 | 73.6 / 46.8 |
Individual models shift with the engine, since each engine boxes different regions, but the finding holds under every engine: mean adherence stays close to that with the models' own detection, and every model still leaves more than a third of spans inconsistent with its own label. The gap therefore does not stem from a particular detector.
The controls you proposed made the central claim sharper, and we hope the revision addresses your concerns.
Dear Reviewer LEHT,
Thank you again for your thoughtful review and the controls you proposed. In our rebuttal, we addressed each of your points as follows:
- Not told vs. told but not applied (W1/Q1): with your one-sentence rule appended, 38.7% of detected spans remain inconsistent with the model's own labels; feeding the model's own DECISION output back into ACT, with no rule, lowers this to 22.8% (Tables 8, 19).
- Scaling (W2/Q2): same-family series for Qwen3-VL, InternVL3.5, and Gemma-3 show that larger models reduce the gap only partially, and a substantial gap remains at every size (Table 11). Our scaling and proprietary-model claims are revised accordingly.
- Detection and OCR (W3/Q3): DECISION and ACT use no external OCR (Fig. 2 corrected), every model detects more than 90% of gold spans, and a cascade with three external OCR engines leads to the same finding (Tables 14, 15).
Thank you for reading our work so closely and for asking the questions that mattered most. If any doubt remains, we would welcome the chance to talk it through before the discussion closes.
With warm thanks and best wishes, The Authors
Response to Reviewer LEHT (Part 1/3)
We are grateful for your thoughtful review and for recognizing the failure mode as practical and meaningful. The controls you suggested are now in §5.3 and App. E; changes in the revised manuscript are marked in red.
Terminology and settings. The ACT metric, called PRESERVE in the submission, is now called adherence: whether the model's ACT output follows its own DECISION label on the spans it detected itself (§3.1). All main claims use the unguided setting (image only, no rules, no gold); guided ACT adds the gold labels as an upper bound, and Bind-then-Act uses no gold, letting the model list its own spans and decisions before acting, with the rules in the prompt.
W1 / Q1. - Not told, or told but not applied
Your concern was that the gap may reflect what the model was not told rather than a failure to act. The controls you suggested point to the latter: told the rule, models still leave many spans inconsistent with their own labels, whereas binding their own decisions to the spans, with no rule, closes most of the gap.
What we measure. We agree that "not told" and "told but not applied" must be separated. The told setting is well studied, with the regions to translate given in in-image MT (e.g., "AnyTrans") and terminology constraints given in text MT (e.g., "Findings of the WMT 2023 Shared Task on Machine Translation with Terminologies"); we now discuss these works in §2. SPOT-Bench targets the complementary, autonomous case: whether a model decides on its own when to keep text, as a professional translator keeps a brand name without being told to. Unguided DECISION is itself uninstructed (App. C), yet models label most spans correctly; the question is whether they act on it.
The controls. We ran your control and your second suggestion on the full benchmark. The one-sentence rule is appended to the instruction and adds external information. Structure only implements your second suggestion as a two-pass cascade: the model first performs DECISION, and its own output, the list of spans with their labels, is given back as input to ACT, so that each action is bound to the span it decided on. It is Bind-then-Act without the rules, and it adds nothing the model did not already produce.
Decision / Adherence under each control (%)