acceptodds
Under review as a conference paper at ICLR 2027

ConServe: Context-Preserving and Compute-Conserving Text Image Machine Translation

Abstract

Text Image Machine Translation (TIMT) requires models to recognize embedded text and use visual context to resolve translation ambiguities. However, existing approaches leave unclear which visual evidence supports translation decisions and how much context is necessary. We introduce Translation Context Lens (TCL), a causal diagnostic framework based on controlled visual interventions. TCL reveals that visual evidence for translation is distributed yet redundant, with different regions providing overlapping support. This finding motivates ConServe, a framework for preserving translation quality under visual context compression. Spatially balanced token sampling distributes the background budget across regions and selects representative tokens locally, preserving all visual tokens in target text regions. Privileged Context On-Policy Self-Distillation (PC-OPSD) addresses shifts in translation preferences caused by compression. The teacher uses full visual context, correct interpretations, and explanations of incorrect alternatives to supervise predictions on prefixes generated by the student. We also introduce SC-TIMT, a multilingual benchmark for screen translation that depends on visual context. Experiments show that our 9B model outperforms larger multimodal models on SC-TIMT and remains competitive on MCiT with substantially fewer visual tokens. These results suggest that efficient TIMT depends on preserving distributed contextual evidence rather than retaining the entire visual input.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Text Image Machine Translation, Visual Context-Aware Translation, Multimodal Large Language Models, Causal Visual Intervention, Visual Token Compression, On-Policy Self-Distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_quAVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses Text Image Machine Translation (TIMT) which is the problem of translating text in images while incorporating visual cues, rather than just examining the text alone.

Current approaches use multimodal models which consider visual information in the final decision, but sometimes it is not clear which and how visual cues influence the final translation.

To understand how visual cues inform the translation, the paper provides an analysis Translation Context Lens (TCL), which removes regions from the image to find which regions affect the translations. The paper finds that visual information is spread across the image background somewhat evenly with some redundancy.


Based on these findings, the paper presents ConServe, a new method for TIMT. The method uses a subset of visual context regions sampled from the background of the full image. Training has two stages: 1) SFT using the full image 2) Student model trained with a frozen teacher that has seen the full image. The Student only sees a compressed representation of the image from SQ-FPS, which samples regions according to visual diversity and a given budget. At inference time, the student model outputs the final translation learned from the compressed SQ-FPS representation.

To better evaluate TIMT methods on using visual context, the paper introduces a new benchmark ScreenContext (SC-TIMT), which focuses on cases where the correct translation highly depends on visual context. It also comes with an agent: Error-Aware Translation Agent for evaluation. Results are also reported on existing benchmark MCiT, along with an ablation study for the training components SQ-FPS and PC-OPSD. The results also find that ConServe outperforms baselines under the same compression conditions and benefits from reduced latency even with a low background retention ratio of r=0.3

Critical strengths

The results are mainly positive and show improvement over the baselines given compressed visual inputs.

The proposed method ConServe works well with a relatively low background retention ratio r=0.3 and performs similarly even

It is interesting to have work focused on what kinds of image contents inform multimodal model decisions, in this case for TIMT.

Critical weaknesses

While the findings of Section 2 motivate the SQ-FPS portion of the method in Section 3, I feel like they could be more fully connected. For instance, SQ-FPS contains a visual diversity score, whereas the analysis in Section 2 is limited to region blocks if I understand correctly. It would be nice to connect and better motivate visual diversity score with the findings in Section 2, which is that the visual context is distributed yet redundant across the background regions.

While Section 2 provides motivation for the method, the evaluation in Section 4 uses Contextual Consistency Rate (CCR) to quantify how well the translations align with the visual information. How could these be connected or incorporated with Section 2, which uses separate metrics to measure how regions affect the final translation outputs relevance with the visual information. Overall, there seems to be a bit of a disconnect between the metrics used for the analysis, method, and the final evaluation, and this could be either better explained/connected by the paper.

The new benchmark SC-TIMT, which is highlighted in the introduction as one of the main contributions of the paper for its focus on the importance of visual context, is moved the appendix and not described much in the main paper.

Discussion of related work is moved to the appendix, and it could benefit the paper to include some of these discussions in the main text. For instance, why are some of the mentioned current approached (e.g. InImageTrans, MT3, CPR-Trans, GLoTran) not included as comparison methods in Section 4?

The paper points out that previous work is lacking in that they do not identify which visual cues are important and affect the final translation. While the paper includes visualizations of the region-based analysis TCL in Section 2 and the appendix, which then motivates the inclusion of SQ-FPS in the method presented in Section 3, there are no visuals results showing the final regions sampled by SQ-FPS. If the method allows this, it would be an important result and feature of this method to include in the paper, especially since this is featured as a limitation of existing work, or is the main goal better compression, reducing unnecessary computation and faster inference?

Questions for authors

For TCL, how are the background regions decided? How are the sizes/shapes/contexts of the regions important?

 If I understand correctly, the image regions are removed by masking? If image editing operations were used to dynamically remove or perturb background regions, or even change regions with complex shapes, how would this affect the results?

How does ConServe compare to just the SFT model? Is this included in the ablation study in Table 3? I found the ablations to be quickly brushed over and hard to interpret. According to Figure 4(a), the translation quality is maintained from background retention ratio r>=0.3r >= 0.3 , especially on the MCiT benchmark. However, there are some performance gains from including more of the image on the newly introduced benchmark SC-TIMT. Could you provide more of an explanation why there are more performance gains specifically for SC-TIMT? And some insights into what kind of examples cause this result and the failure cases of ConServe?

The EATA scores seem quite high, with most scores >= 4 and no scores seemingly below 3. Why is this, and is this a limitation or feature of the agent?

Small notes: Figure 3 text on the plots is very small. How can TIMT help text-only translation and vice versa?

6 comments
AuthorsOpenReview ↗

Response to Reviewer quAV — Part 1/5

Thank you very much for your careful review and overall positive assessment of our work. In response to your comments and questions, we have added further experimental analyses and made corresponding revisions to the paper’s organization and experimental discussion. We resond below.


Q1: How does TCL’s analysis motivate the visual diversity score in SQ-FPS, and can the visual regions actually selected by SQ-FPS be illustrated?

A1: TCL reveals two characteristics of translation-relevant visual context: visual evidence is distributed across multiple background regions, and different regions provide functionally overlapping information. These two findings motivate the spatial quota and visual diversity components of SQ-FPS, respectively. Spatial quotas maintain visual coverage across regions, while visual diversity measures differences among visual features within each region and helps adjust the token budget allocated to each region. This allows regions with more diverse visual information to receive a larger sampling budget, reducing the risk of losing distinct visual cues during compression. To further connect visual diversity with TCL’s regional analysis, we examined the correlation between visual diversity and regional contributions. The mean Spearman correlations of visual diversity with the strengths of positive independent support (S>0S>0) and positive necessity contributions (C>0C>0) are +0.255 and +0.210, respectively. After controlling for the number of background tokens, the correlations remain +0.244 and +0.169, respectively. These results indicate a positive association between visual diversity and translation-relevant regional contributions, but visual diversity is not equivalent to semantic importance. We therefore use it as an auxiliary cue for budget allocation rather than as a direct predictor of the translation importance of visual regions. The ablation results further support this design: removing diversity-based quota allocation reduces the Agent Scores on SC-TIMT and MCiT from 4.32 to 4.17 and from 4.71 to 4.50, respectively. In addition, following the reviewer’s suggestion, we have added visualizations of SQ-FPS compression and the corresponding translations to the appendix, showing the visual information actually retained and its effects on translation. The relevant definitions and analyses are provided in Eq. (8) of Sec. 4.2, Table 3 of Sec. 5, and Appendix E.3 of the revised manuscript. Appendix F presents the SQ-FPS compression visualizations and corresponding translations.

Q2: How are TCL’s analysis of regional effects on translation decisions and CCR’s evaluation of contextual consistency in the final translations connected?

A2: We agree that the original manuscript did not sufficiently explain the relationship between TCL and CCR. They examine the influence of visual context on translation from different perspectives. TCL constructs competing context-consistent and context-inconsistent interpretations and applies controlled interventions to the visual background to analyze how different regions affect the model’s translation preferences, thereby revealing the distributed and redundant structure of translation-relevant visual evidence. CCR evaluates whether the final translation preserves the intended meaning in the context of the full image. For each translation item, the evaluator makes 10 binary judgments, each taking the full image, source text, expert gold translation, and candidate translation as input. The majority vote determines the item-level binary label (0 or 1), while a 5–5 tie is resolved through human adjudication. CCR is then calculated as the proportion of items judged to be contextually consistent. The evaluation-metric description in Sec. 5 of the revised manuscript and Eq. (13) in Appendix D.2 provide this definition and distinguish TCL’s diagnosis of decision sensitivity from CCR’s evaluation of contextual consistency in the final translation. TCL’s analysis motivates the design of SQ-FPS, while CCR assesses whether the final translations produced from compressed visual inputs preserve the intended contextual meaning.

AuthorsOpenReview ↗

Response to Reviewer quAV — Part 2/5

Q3: Are the construction and presentation of SC-TIMT sufficient, and why are some existing TIMT methods not included in the comparison experiments?

A3: We have moved Related Work into Sec. 2 of the main text. Sec. 5 briefly introduces SC-TIMT, including its 1,100 real-world screen images, four source languages, and focus on context-sensitive evaluation. Due to the page limit, more detailed information about image sources, sample selection, privacy review, Chinese gold translations provided by five translation experts, and the separation between training and test data is presented in Appendix A and Table 4. This provides an overview of the benchmark’s role in the main text while retaining verifiable details of its construction in the appendix.

Regarding comparisons with existing TIMT methods, we have added experiments with InImageTrans [1] and CPR-Trans [2]. For InImageTrans, we used its official open-source implementation and resolved reproduction issues through communication with the authors. We completed evaluations on SC-TIMT and MCiT, and the results have been added to Tables 1 and 2 in Sec. 5 of the revised main text. For CPR-Trans, since no official open-source implementation is available, we independently implemented its multi-stage pipeline for scene understanding, text perception, and translation reasoning based on the paper’s description, and evaluated it on both benchmarks. The results show that ConServe outperforms InImageTrans on both benchmarks and surpasses CPR-Trans in Agent Score. In particular, compared with CPR-Trans, ConServe reduces the average number of tokens by 13.9% on SC-TIMT and 31.6% on MCiT, and reduces inference latency by 78.2% and 77.5%, respectively, while maintaining competitive translation quality. These results further demonstrate its advantage in balancing translation performance and inference efficiency. The detailed results are as follows.

Additional results on SC-TIMT

MethodSizeENKOJAVIAgent ↑BLEU ↑METEOR ↑Tokens / Image ↓Latency (s) ↓
InImageTrans9B3.032.872.443.112.8843.4738.963,2386.22
CPR-Trans9B4.524.094.104.074.2152.7069.364,61132.24
ConServe9B4.564.034.214.094.3253.8373.493,9697.04

Additional results on MCiT

MethodSizePaperNewsNovelTitleSignIntroductionCoverLeafletAgent ↑BLEU ↑METEOR ↑Tokens / Image ↓Latency (s) ↓
InImageTrans9B3.903.672.314.574.143.442.863.763.6139.8458.521,3475.12
CPR-Trans9B4.924.664.434.674.804.744.524.374.6546.6163.112,84426.82
ConServe9B4.904.594.064.974.924.804.594.794.7144.7262.621,9466.03

Neither MT3 [3] nor GLoTran [4] provides a complete official implementation or the corresponding training data, and both rely on specialized training pipelines. Specifically, MT3 uses multi-task reinforcement learning, with task-level rewards jointly optimizing text recognition, contextual reasoning, and translation. GLoTran relies on a specially constructed GLoD dataset and fine-tuning with global–local visual inputs. Due to the lack of open-source code and corresponding training resources, independently reconstructing the training pipelines based only on the papers’ descriptions would make it difficult to ensure consistency with the original implementations and training settings. Therefore, to ensure reliable experimental comparisons, we have not included reproduction results that cannot be sufficiently verified in the formal baseline comparisons.

AuthorsOpenReview ↗

Response to Reviewer quAV — Part 3/5

Q4: How are background regions defined in TCL, and how do region granularity, shape, and intervention choices affect the analysis?

A4: In TCL, the background is defined relative to the current translation target: the target text’s bounding box with a margin is retained, while the rest of the image is treated as the visual context that can be intervened on, including surrounding text, image content, and interface elements. Specifically, we construct regular rectangular regions on the base visual grid and map them back to the original image for pixel-level interventions. The partition is determined before effect estimation and is independent of model importance scores. In the following tables, “substitutable regions” are regions satisfying Ci,j≤0C_{i,j}\leq 0 and Si,j>0S_{i,j}>0. Ri(k)R_i^{(k)} denotes the joint removal effect of the Top-kk regions ranked by individual necessity, normalized by the effect of removing the entire background. The mean and median are computed over the diagnostic samples, and the Ri(k)<0.5R_i^{(k)}<0.5 row reports the proportion of samples satisfying this condition. Here, 2×22\times2, 4×44\times4, and 8×88\times8 indicate the number of base-grid cells contained in each intervention region, not the spatial quota grid used by SQ-FPS.

To analyze the effect of region size, we compared the three regular rectangular partition granularities above. As the regions become coarser, the mean number of substitutable regions per image decreases from 55.41 to 32.41, while the proportions of samples containing multiple substitutable regions remain at 99.46%, 98.97%, and 95.62%, respectively. The mean Ri(20)R_i^{(20)} values are 0.2033, 0.2996, and 0.3859, respectively; the corresponding proportions of samples satisfying Ri(20)<0.5R_i^{(20)}<0.5 are 90.32%, 80.85%, and 77.74%. Coarser regions contain more visual information and therefore produce larger joint removal effects, but the overall pattern of distributed and substitutable evidence remains stable. TCL measures regional contributions using necessity Ci,jC_{i,j} and independent support Si,jS_{i,j}, which respectively quantify the effect of removing a region from the full background and restoring it alone when the other background regions are masked.

To examine the effect of intervention choices, we added experiments using mean-color filling, strong blur, and strong noise. Under all four operations, the proportion of samples containing multiple substitutable regions is above 97%. The mean Ri(20)R_i^{(20)} is 0.2996 for the original Mask, compared with 0.2584, 0.2871, and 0.2547 for mean-color filling, strong blur, and strong noise, respectively. The corresponding proportions of samples with Ri(20)<0.5R_i^{(20)}<0.5 are 83.26%, 81.08%, and 84.29%, close to 80.85% for the original Mask. Thus, the observed distributed visual contributions are not an artifact of the original Mask operation alone. Tables 6–9 in Appendix E.2 of the revised manuscript report the complete statistics for different partition granularities and intervention choices. Complex-shaped regions have not yet been included in this robustness analysis, and we do not generalize the conclusions to arbitrary region shapes.

TCL analysis under different region granularities

Statistic2×22\times24×44\times48×88\times8
Proportion of samples with multiple substitutable regions99.46%98.97%95.62%
Substitutable regions per image (mean / median)55.41 / 43.5045.74 / 39.0032.41 / 28.00

Top-k joint removal effects under different region granularities

GranularityStatisticTop-1Top-5Top-10Top-20
2×22\times2Mean Ri(k)R_i^{(k)}0.14550.18500.21020.2033
2×22\times2Median Ri(k)R_i^{(k)}0.09690.12800.15200.1111
2×22\times2Proportion with Ri(k)<0.5R_i^{(k)}<0.596.77%93.55%90.32%90.32%
4×44\times4Mean Ri(k)R_i^{(k)}0.16640.21750.25730.2996
4×44\times4Median Ri(k)R_i^{(k)}0.08910.12800.13920.1801
4×44\times4Proportion with Ri(k)<0.5R_i^{(k)}<0.593.62%90.43%86.28%80.85%
8×88\times8Mean Ri(k)R_i^{(k)}0.18340.25720.35320.3859
8×88\times8Median Ri(k)R_i^{(k)}0.12500.16480.20250.2439
8×88\times8Proportion with Ri(k)<0.5R_i^{(k)}<0.593.55%89.10%87.42%77.74%

Substitutable-region statistics under different background interventions

StatisticOriginal MaskMean fillStrong blurStrong noise
Proportion of samples with multiple substitutable regions98.97%99.01%98.32%97.43%
Substitutable regions per image (mean / median)45.74 / 39.0045.88 / 39.0043.09 / 37.0044.66 / 38.00
Reviewer_VJcbOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ConServe, a framework for efficient context-sensitive Text Image Machine Translation (TIMT). The authors first introduce Translation Context Lens (TCL), which analyzes the contribution of visual background regions through controlled masking and restoration interventions. Based on the observation that translation-relevant visual evidence is spatially distributed and potentially redundant, they propose Spatial-Quota Farthest Point Sampling (SQ-FPS) to preserve target-text tokens while compressing background visual information. The framework further incorporates Privileged Context On-Policy Self-Distillation (PC-OPSD), where a teacher with access to full visual context and additional disambiguation supervision guides a student operating on compressed representations. The authors also introduce SC-TIMT, a multilingual context-sensitive screen translation benchmark, and EATA, an error-aware translation evaluator. The proposed 9B model demonstrates improved translation quality over its base model and competitive results against larger MLLMs, while reducing visual token overhead.

Critical strengths
  • Well-motivated problem formulation. The paper addresses an important efficiency challenge in TIMT. Preserving target-text tokens while selectively compressing background context is well aligned with the requirements of the task.

  • Interesting diagnostic perspective. TCL investigates regional necessity and independent support through visual interventions, providing a more informative analysis than conventional attention visualization alone.

  • Practical compression mechanism. SQ-FPS is training-free, computationally straightforward, and explicitly balances spatial coverage and visual diversity. The compression experiments and ablations provide encouraging evidence of its effectiveness.

Critical weaknesses

W1. The contributions of different training components to the final performance are not sufficiently disentangled.

ConServe combines Multi-region SFT, SQ-FPS compression, and PC-OPSD with additional privileged semantic supervision. Although the paper provides several component ablations, it remains unclear how much of the final improvement comes from task-specific fine-tuning, additional semantic supervision, or compression-aware distillation.

In particular, Table 3 shows that removing the teacher's full visual context decreases the SC-TIMT Agent Score from 4.32 to 4.26, whereas removing privileged context decreases it to 4.08. This suggests that additional semantic supervision may be an important source of the improvement.

More controlled comparisons with SFT-only models, conventional knowledge distillation, and additional SFT using the same privileged annotations are needed to establish the specific contribution of PC-OPSD.

W2. The actual improvement in translation quality attributable to visual context is not directly quantified.

The paper investigates the influence of visual background through TCL and evaluates translation quality under different background token retention ratios. However, these experiments do not directly quantify the benefit of visual context over text-only translation.

Specifically, TCL measures changes in translation preferences between full-image and background-masked conditions, rather than reporting aggregate translation-quality metrics under these conditions. The compression experiments compare partial and full background retention, but do not include OCR-only or text-only translation baselines.

Consequently, it remains unclear how much translation quality is gained from visual context in the first place and how much of that gain is preserved by ConServe.

W3. The evidence supporting the distributed and redundant nature of visual context requires stronger justification.

TCL provides an interesting analysis of regional contributions, but the interpretation of functional redundancy may be stronger than the evidence supports.

The Top-k important regions are selected according to their individual removal effects, which do not necessarily identify the subset with the largest joint contribution when regional interactions are nonlinear.

Furthermore, the evidence for regional substitutability relies on regions having non-positive necessity and positive independent support, without sufficiently characterizing effect magnitudes or robustness to intervention choices.

The analysis is also performed on a selected subset of context-sensitive examples using a single backbone model. Additional robustness and interaction analyses would strengthen the central motivation for spatially distributed context preservation.

W4. The completeness and fairness of comparisons with existing TIMT approaches need further clarification.

The main translation-performance comparisons primarily involve general-purpose MLLMs, while recent specialized TIMT approaches discussed in the related work are not included in the main quantitative comparisons.

Moreover, ConServe benefits from task-specific SFT, OCR-based target localization, and explicitly marked target regions. It is not sufficiently clear whether all baselines receive equivalent target-region annotations, translation prompts, and structured output requirements.

Therefore, stronger matched comparisons would help establish the performance advantages attributable to the proposed framework rather than differences in training or task specification.

Questions for authors

Q1. What drives the performance gains of PC-OPSD?

Table 3 shows that privileged semantic context contributes more than the teacher's full visual context. How much of the improvement comes from additional semantic supervision rather than on-policy distillation? A comparison with standard SFT using the same privileged annotations would help clarify this distinction.

Q2. How much additional value does visual information provide beyond textual context?

The TCL analysis demonstrates changes in translation preferences under background masking, but the actual improvement in translation quality remains unclear. It would be helpful to include a full-image OCR-text-only baseline, using all recognized text without visual input, to quantify the additional benefit of visual information.

Q3. How reliable is TCL's conclusion about distributed and redundant visual evidence?

Since Top-k regions are selected based on individual removal effects, potential interactions between regions may affect the interpretation. Could the authors clarify how such interactions influence their conclusions and provide further analysis of the magnitude of regional effects?

Q4. Are the baseline comparisons conducted under consistent settings?

Please clarify whether the evaluated MLLMs use the same target-region annotations, translation prompts, and output formats. It would also be useful to discuss how ConServe compares with recent specialized TIMT approaches beyond general-purpose MLLMs.

6 comments
AuthorsOpenReview ↗

Response to Reviewer VJcb — Part 1/4

We thank the reviewer for the careful review and constructive feedback. To address the your concerns regarding training component contributions, visual context benefits, TCL reliability, and baseline fairness, we have added experiments and analyses and further clarified our method design. The detailed responses are provided below. In the revised manuscript, newly added experimental results in tables are highlighted in yellow, while revisions to the main text are marked in blue for ease of reference.


Q1: How do the different training components, particularly PC-OPSD, contribute to the final performance?

A1: Regarding the comparisons involving SFT-only and conventional knowledge distillation, the revised main text explicitly names the corresponding variants SFT-only and Off-policy KD in Table 3. The former retains Multi-region SFT and SQ-FPS at inference but does not undergo PC-OPSD training; the latter retains the same teacher inputs and distillation objective, while using fixed prefixes from the gold translations in the training set instead of student-generated prefixes. Their Agent Scores on SC-TIMT are 4.01 and 4.14, respectively, both lower than the full model's 4.32. w/o Full Context removes only the additional low-resolution Global Image provided to the teacher, while retaining the uncompressed framed image and privileged translation context, and obtains a score of 4.26; w/o Privileged Context retains the full visual input but removes only the positive and negative privileged contexts, obtaining a score of 4.08. These settings and their controlled variables are described in Appendix C, “Ablation Study Details.”

For the comparison involving additional SFT using the same privileged annotations, the main issue is the inconsistency between the input conditions during training and inference. If the same privileged annotations were used in SFT, the training examples would need to include the full visual background, and the prompts would need to provide positive and negative privileged semantic information. However, at inference time, the student model can only receive the SQ-FPS-compressed visual representation and cannot access this privileged information. Therefore, this training strategy would make it difficult to maintain consistency between the training and inference conditions. The resulting performance differences would also be affected by both the additional supervision and the change in input conditions, making it difficult to fairly evaluate the contribution of privileged semantic supervision itself.

Q2: What additional benefit does visual context provide over text-only translation?

A2: We added a text-only baseline, OCR+LLM, based on Qwen3.5-9B. PaddleOCR is first used to recognize the text in the full image, and only the recognized text is then provided to Qwen3.5-9B for translation, without providing any image pixels. The results are as follows:

Comparison between the text-only baseline and ConServe

BenchmarkMethodAgent Score ↑SacreBLEU ↑METEOR ↑
SC-TIMTOCR+LLM3.4737.2862.72
SC-TIMTConServe4.3253.8373.49
MCiTOCR+LLM3.6841.0453.79
MCiTConServe4.7144.7262.62

The Agent Score in the table is EATA's overall score for translation quality, while SacreBLEU and METEOR are reference-based metrics. ConServe outperforms the text-only baseline on all metrics across both benchmarks, supporting the contribution of visual information to TIMT translation quality. Further analysis shows that text-only methods not only lose the spatial layout of the image, hierarchical relationships among text blocks, and associations between text and surrounding visual elements, but also rely heavily on OCR accuracy: once text is missed or incorrectly recognized, the errors are propagated to the subsequent translation. In contrast, after using OCR to localize text regions, ConServe continues to use the image's visual representation and compresses background tokens through SQ-FPS, thereby retaining the ability to use visual context for semantic disambiguation. The Agent Score of this baseline and the corresponding analysis are reported in Table 3 of the revised main text; the PaddleOCR-to-Qwen3.5-9B pipeline and the two reference-based metrics are provided in Appendix C, “Text-only baseline.”

AuthorsOpenReview ↗

Response to Reviewer VJcb — Part 2/4

Q3: How robust is TCL's finding that visual evidence is distributed and redundant?

A3: Regarding regional interactions, we agree that ranking regions by their individual contributions does not necessarily identify the combination of regions with the largest joint effect. To further examine this issue, we added an exhaustive Top-2 region analysis, computing the joint removal effects of all two-region combinations in the diagnostic samples and comparing them with those of the Top-2 regions selected based on individual necessity scores. Given the computational cost of exhaustively evaluating higher-order combinations, we focus on Top-2. The following table reports the joint removal effects and their normalized values under the two selection strategies, where Ji(2)J_i(2) denotes the change in translation preference caused by jointly removing two regions, and Ri(2)=Ji(2)/CiallR_i^{(2)}=J_i(2)/C_i^{\mathrm{all}} denotes the normalized effect relative to removing the entire background.

Top-2 region interaction analysis

StatisticIndividual-score Top-2Exhaustive best pair
Mean Ji(2)J_i(2)0.90361.0577
Mean Ri(2)R_i^{(2)}0.18820.2014
Median Ri(2)R_i^{(2)}0.15810.1765
Proportion of samples with Ri(2)<0.5R_i^{(2)} < 0.592.55%91.32%

The results show that exhaustive selection of the best pair increases the mean normalized effect from 0.1882 to 0.2014, indicating that individual region ranking can indeed miss some joint contributions. However, even when the two regions with the largest joint effect are selected, 91.32% of the samples still satisfy Ri(2)<0.5R_i^{(2)} < 0.5. This indicates that, even when two-region interactions are considered, a small number of local regions is generally insufficient to explain the effect of the full background, further supporting TCL's finding that visual evidence is distributed. These results are included in Table 5 of Appendix E.2.

In addition, to examine whether functional overlap among regions depends on the choice of background intervention, we further added an analysis of regional substitutability under different intervention conditions. Besides the original Mask operation, we applied mean-color filling, strong blur, and strong noise to the background, and compared the distribution of substitutable regions under each condition. As before, substitutable regions are identified by the criteria Ci,j≤0C_{i,j}\leq 0 and Si,j>0S_{i,j} > 0. The results are shown below.

Substitutable-region statistics under different background interventions

StatisticOriginal MaskMean fillStrong blurStrong noise
Proportion of samples with multiple substitutable regions98.97%99.01%98.32%97.43%
Substitutable regions per image (mean / median)45.74 / 39.0045.88 / 39.0043.09 / 37.0044.66 / 38.00

As shown above, the proportion of samples containing multiple substitutable regions exceeds 97% under all intervention conditions, and the number of substitutable regions per image is also similar across conditions. This indicates that functional overlap among regions is not caused by a specific Mask operation, but can also be observed under different background interventions. These results are reported in Table 8 of Appendix E.2.

Furthermore, we compared the Top-k joint removal effects under different background interventions to examine whether the distributed nature of visual evidence is affected by the choice of intervention. The following table summarizes the mean and median of Ri(k)=Ji(k)/CiallR_i^{(k)}=J_i(k)/C_i^{\mathrm{all}}, as well as the proportion of samples with Ri(k)<0.5R_i^{(k)} < 0.5, under each intervention. Top-1, Top-5, Top-10, and Top-20 denote the top kk regions selected according to their individual necessity scores.

Top-k joint removal effects under different interventions

InterventionStatisticTop-1Top-5Top-10Top-20
Original MaskMean Ri(k)R_i^{(k)}0.16640.21750.25730.2996
Original MaskMedian Ri(k)R_i^{(k)}0.08910.12800.13920.1801
Original MaskProportion with Ri(k)<0.5R_i^{(k)} < 0.593.62%90.43%86.28%80.85%
Mean fillMean Ri(k)R_i^{(k)}0.15820.22310.24480.2584
Mean fillMedian Ri(k)R_i^{(k)}0.08340.13320.12240.1545
Mean fillProportion with Ri(k)<0.5R_i^{(k)} < 0.594.23%90.28%87.06%83.26%
Strong blurMean Ri(k)R_i^{(k)}0.16270.20180.26010.2871
Strong blurMedian Ri(k)R_i^{(k)}0.08570.11930.14130.1693
Strong blurProportion with Ri(k)<0.5R_i^{(k)} < 0.593.67%91.57%86.12%81.08%
Strong noiseMean Ri(k)R_i^{(k)}0.15930.23010.22970.2547
Strong noiseMedian Ri(k)R_i^{(k)}0.08600.14220.12010.1349
Strong noiseProportion with Ri(k)<0.5R_i^{(k)} < 0.594.01%90.16%87.88%84.29%
AuthorsOpenReview ↗

Response to Reviewer VJcb — Part 3/4

Across different intervention conditions, the Top-k normalized joint removal effect remains below 0.5 for most samples, and the results are generally similar across conditions. This further indicates that, although the choice of intervention affects the magnitude of individual effects, the main finding that visual evidence is distributed across multiple background regions remains stable. The complete statistics are provided in Table 9 of Appendix E.2.

Finally, regarding the concerns about the use of a single backbone model and the selection of examples, the main text uses Qwen3.5-9B as the primary model for TCL analysis to remain consistent with ConServe's base model. We also conducted supplementary analyses using Qwen2.5-VL and Qwen3-VL, representing different generations of Qwen vision-language models, and observed the same common trend: visual evidence is distributed across multiple background regions, with functional overlap among different regions. In addition, the TCL diagnostic set is constructed by selecting context-sensitive examples from the candidate pool for which background information demonstrably affects translation decisions. This selection allows us to examine the spatial distribution of visual evidence and the functional overlap among regions in such cases, rather than estimate the average dependence on background context across all TIMT samples. We have clarified the cross-model observations, the scope of the diagnostic set, and the applicability boundaries of the conclusions at the beginning of Appendix E.2.

Q4: Are the baseline comparisons fair and comprehensive, including comparisons with specialized TIMT methods?

A4: Regarding consistency in baseline evaluation, all general-purpose MLLM baselines use the same target-region annotations, translation prompts, and output formats, and are evaluated on the same test sets to ensure comparability. For specialized TIMT methods, we follow the input-processing and inference pipelines adopted in their respective papers and evaluate them using unified evaluation metrics.

For recent specialized TIMT methods, we added experiments with InImageTrans [1] and CPR-Trans [2]. For InImageTrans, we used its official open-source implementation and resolved issues encountered during reproduction through communication with the authors, completing evaluations on SC-TIMT and MCiT. The corresponding results have been added to Tables 1 and 2 of the revised manuscript. For CPR-Trans, because no official open-source implementation is available, we independently implemented its multi-stage pipeline for scene understanding, text perception, and translation reasoning based on the paper's description, and evaluated it on both benchmarks. The detailed results are provided below.

Results on SC-TIMT

MethodSizeENKOJAVIAgent ↑BLEU ↑METEOR ↑Tokens / Image ↓Latency (s) ↓
InImageTrans9B3.032.872.443.112.8843.4738.963,2386.22
CPR-Trans9B4.524.094.104.074.2152.7069.364,61132.24
ConServe9B4.564.034.214.094.3253.8373.493,9697.04

Results on MCiT

MethodSizePaperNewsNovelTitleSignIntroductionCoverLeafletAgent ↑BLEU ↑METEOR ↑Tokens / Image ↓Latency (s) ↓
InImageTrans9B3.903.672.314.574.143.442.863.763.6139.8458.521,3475.12
CPR-Trans9B4.924.664.434.674.804.744.524.374.6546.6163.112,84426.82
ConServe9B4.904.594.064.974.924.804.594.794.7144.7262.621,9466.03
Reviewer_kU8DOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ConServe, a framework for reducing visual-token costs in text image machine translation while preserving context needed for disambiguation. TCL analyzes contextual contributions through background masking and restoration. Motivated by this analysis, SQ-FPS retains target-text tokens and samples background tokens across spatial regions, while PC-OPSD distills a teacher with full visual input and privileged disambiguation information into a compressed-input student. Evaluation covers the proposed multilingual SC-TIMT benchmark and the existing MCiT benchmark.

Critical strengths
  1. Practical problem and coherent design. The framework jointly addresses text perception, contextual disambiguation, and inference efficiency, with components tailored to the task.
  2. Demonstrated efficiency gains. Figure 4 reports total-token reductions of 47.1% and 34.5% on SC-TIMT and MCiT, respectively, at a background retention ratio of 0.3, alongside lower inference latency.
  3. Comprehensive experiments and ablation studies are conducted. The authors compare against multiple visual compression baselines, validate the contribution of each sub‑module in SQ‑FPS and PC‑OPSD, and analyse performance‑efficiency trade‑offs under different compression ratios.
Critical weaknesses
  1. Dataset coverage is limited. SC‑TIMT only supports foreign‑language‑to‑Chinese translation. Both PC‑OPSD training data and SC‑TIMT are dominated by mobile screen screenshots, lacking sufficient coverage of general TIMT scenarios such as documents and posters.
  2. Table 1 and the evaluation settings of Figure 4 require clarification. At r = 0.3, they report 4.32 and 4.410, respectively. Please explain whether the same model version, test samples, and score averaging method were used.
  3. The analysis of where useful context appears is conducted on only one model and a selected set of examples.
Questions for authors
  1. Why does Table 1 report 4.32 on SC-TIMT while Figure 4 reports 4.410? Were these results obtained under different settings? (“Unless otherwise specified, inference uses r = 0.3.”)
  2. Under the adopted inference setting r=0.3, overall performance remains strong. Could the authors characterise which categories of samples suffer severe performance degradation under heavy compression? Could you provide analysis on concrete failure cases?
4 comments
AuthorsOpenReview ↗

Response to Reviewer kU8D — Part 1/2

We thank the reviewer for the careful assessment and constructive comments. In response, we have added further experimental analyses, updated the experimental results, and clarified the scope of our work. Our specific responses are provided below. In the revised manuscript, newly added experimental results in tables are highlighted in yellow, while revisions to the main text are marked in blue for ease of reference.


Q1: Does SC-TIMT's language and scenario coverage, together with the PC-OPSD training data, sufficiently support the claimed scope of the work?

A1: SC-TIMT is not intended to cover all language directions and TIMT scenarios. Rather, it is designed specifically as a benchmark for evaluating context-sensitive translation that depends on visual context. It contains real-world screen images from more than 23 application settings, covering social media, e-commerce, travel and transportation, news, video entertainment, and other mobile and web interfaces. It supports translation from English, Japanese, Korean, and Vietnamese into Chinese.

Regarding the training data, PC-OPSD is built from independently collected visual-context-dependent translation data, comprising 1,365 positive--negative pairs from 1,069 images. We acknowledge that our experiments currently evaluate only these four source languages translated into Chinese; therefore, the results do not establish performance for other target languages. SC-TIMT is primarily intended for evaluating context-sensitive screen translation, whereas MCiT complements it by covering non-screen image types, including documents, scenes, and posters. The revised manuscript clarifies the language directions and image-type coverage in Appendix A, "Datasets and Data Construction," and Table 4. Broader validation across additional languages and TIMT scenarios remains an important direction for future work.

Q2: Why do Table 1 and Figure 4 report different SC-TIMT Agent Scores at r=0.3r=0.3, and were the results computed using the same model, test set, and averaging procedure?

A2: Upon rechecking, we confirmed that the value of 4.32 in Table 1 was computed on the full SC-TIMT test set, whereas the value of 4.410 in the original Fig. 4 was an interim result based on only a subset of the test samples. The discrepancy therefore arose from different test-set coverage, rather than from differences in the model version or inference settings. This was an oversight when preparing the figures, and we apologize for it. We have recomputed the results on the full test set and updated Fig. 4(a) in the main text, ensuring a consistent model version, inference setup, and per-sample averaging procedure for Agent Scores. At (r=0.3), the unrounded mean is 4.3249. Fig. 4(a) reports this value to three decimal places (4.325), while Table 1 reports it to two decimal places (4.32).

Q3: Is the TCL analysis sufficiently robust across different models and sample selections to support its conclusions about visual-context distribution?

A3: TCL's analysis is not limited to Qwen3.5-9B. We also conducted supplementary analyses using Qwen2.5-VL and Qwen3-VL, representing different generations of Qwen vision-language models, and observed the same qualitative pattern: visual evidence is distributed across multiple background regions, with functional overlap among regions. We report the Qwen3.5-9B results in the main text to keep the analysis consistent with ConServe's base model.

Regarding sample coverage, TCL's candidate samples were drawn from 5,000 real-world mobile screenshots and 1,450 images from MCiT. The diagnostic set was then constructed by screening for context sensitivity; thus, the analysis is not restricted to mobile-screen scenarios. This screening is necessary because, for samples whose translations do not depend on visual context, removing the entire background typically does not substantially change the model's translation preference, making meaningful regional contributions difficult to observe. We therefore focus on samples in which background information demonstrably affects translation decisions, allowing us to examine the spatial distribution of visual evidence and the functional overlap among regions. The revised manuscript further clarifies the scope of the diagnostic set and the robustness of the analysis in Appendix E.2, "Robustness of the TCL Analysis."

AuthorsOpenReview ↗

Response to Reviewer kU8D — Part 2/2

Q4: Which types of samples are most vulnerable to severe translation-quality degradation under heavy visual compression, and can concrete failure cases illustrate this?

A4: We further analyzed translation outputs at different compression ratios and found that errors under aggressive compression mainly occur in samples that require multiple spatially dispersed visual cues, fine-grained interface information, or relationships between text and icons or objects to resolve ambiguity. When the background retention ratio is too low, some complementary visual cues may not be retained together. The model may consequently lose essential contextual information, leading to incorrect word-sense disambiguation or misinterpretation of semantic relations.

In contrast, samples that rely on a single local cue, contain substantial redundant context, or have sufficient discriminative information in the target foreground itself are generally more robust to compression. We have added failure cases and SQ-FPS compression visualizations to Appendix F, "Visual Analysis of SQ-FPS Compression." By comparing the full images, the visual content retained after compression, and the corresponding translations, these examples illustrate how the loss of visual information affects translation.

We thank you again for the careful review and constructive feedback, which have helped us strengthen the experimental analysis and clarify the scope and limitations of our work.

AuthorsOpenReview ↗

Response to Reviewer kU8D

Dear Reviewer kU8D,

I hope this message finds you well. As the discussion period is nearing its end with less than two days remaining, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

Best regards,

The authors of Paper 25596 — ConServe

The authors have provided detailed responses and additional experimental analyses that adequately address my concerns. In particular, they have clarified the scope of language and scenario coverage, corrected the inconsistency in the reported Agent Scores, strengthened the robustness analysis of TCL across different models and sample selections, and added concrete failure cases to better illustrate the effects of visual compression.

Overall, I appreciate the authors' efforts in addressing the feedback and improving the manuscript. Their responses have sufficiently resolved my concerns, and I will therefore maintain my original score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General comments

We thank the ACs and all reviewers for their invaluable time and comments. We have uploaded a revised PDF, in which newly added experimental results in tables are highlighted in yellow and textual revisions throughout the manuscript are marked in blue. The main changes are:

  • TCL analysis and SQ-FPS visualizations (Appendices E.2 and F). We add visualizations of SQ-FPS compression and the corresponding translations to show the retained visual information and its effects on translation. We further strengthen the TCL analysis through experiments with different region granularities and background intervention methods, validating the robustness of the findings.

  • Ablation study refinements (Table 3 and Appendix C). We clarify the configurations and performance differences of the SFT-only, Off-policy KD, w/o Full Context, and w/o Privileged Context ablations. We also introduce an OCR+LLM text-only baseline to demonstrate the benefits of visual context over text-only translation.

  • Additional experiments (Secs. 2 and 5; Appendices A and D). We provide additional details on SC-TIMT and its data construction, clarify the relationship between TCL and CCR, and add comparison experiments with InImageTrans and CPR-Trans.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.