- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses Text Image Machine Translation (TIMT) which is the problem of translating text in images while incorporating visual cues, rather than just examining the text alone.
Current approaches use multimodal models which consider visual information in the final decision, but sometimes it is not clear which and how visual cues influence the final translation.
To understand how visual cues inform the translation, the paper provides an analysis Translation Context Lens (TCL), which removes regions from the image to find which regions affect the translations. The paper finds that visual information is spread across the image background somewhat evenly with some redundancy.
Based on these findings, the paper presents ConServe, a new method for TIMT. The method uses a subset of visual context regions sampled from the background of the full image. Training has two stages: 1) SFT using the full image 2) Student model trained with a frozen teacher that has seen the full image. The Student only sees a compressed representation of the image from SQ-FPS, which samples regions according to visual diversity and a given budget. At inference time, the student model outputs the final translation learned from the compressed SQ-FPS representation.
To better evaluate TIMT methods on using visual context, the paper introduces a new benchmark ScreenContext (SC-TIMT), which focuses on cases where the correct translation highly depends on visual context. It also comes with an agent: Error-Aware Translation Agent for evaluation. Results are also reported on existing benchmark MCiT, along with an ablation study for the training components SQ-FPS and PC-OPSD. The results also find that ConServe outperforms baselines under the same compression conditions and benefits from reduced latency even with a low background retention ratio of r=0.3
Critical strengths
The results are mainly positive and show improvement over the baselines given compressed visual inputs.
The proposed method ConServe works well with a relatively low background retention ratio r=0.3 and performs similarly even
It is interesting to have work focused on what kinds of image contents inform multimodal model decisions, in this case for TIMT.
Critical weaknesses
While the findings of Section 2 motivate the SQ-FPS portion of the method in Section 3, I feel like they could be more fully connected. For instance, SQ-FPS contains a visual diversity score, whereas the analysis in Section 2 is limited to region blocks if I understand correctly. It would be nice to connect and better motivate visual diversity score with the findings in Section 2, which is that the visual context is distributed yet redundant across the background regions.
While Section 2 provides motivation for the method, the evaluation in Section 4 uses Contextual Consistency Rate (CCR) to quantify how well the translations align with the visual information. How could these be connected or incorporated with Section 2, which uses separate metrics to measure how regions affect the final translation outputs relevance with the visual information. Overall, there seems to be a bit of a disconnect between the metrics used for the analysis, method, and the final evaluation, and this could be either better explained/connected by the paper.
The new benchmark SC-TIMT, which is highlighted in the introduction as one of the main contributions of the paper for its focus on the importance of visual context, is moved the appendix and not described much in the main paper.
Discussion of related work is moved to the appendix, and it could benefit the paper to include some of these discussions in the main text. For instance, why are some of the mentioned current approached (e.g. InImageTrans, MT3, CPR-Trans, GLoTran) not included as comparison methods in Section 4?
The paper points out that previous work is lacking in that they do not identify which visual cues are important and affect the final translation. While the paper includes visualizations of the region-based analysis TCL in Section 2 and the appendix, which then motivates the inclusion of SQ-FPS in the method presented in Section 3, there are no visuals results showing the final regions sampled by SQ-FPS. If the method allows this, it would be an important result and feature of this method to include in the paper, especially since this is featured as a limitation of existing work, or is the main goal better compression, reducing unnecessary computation and faster inference?
Questions for authors
For TCL, how are the background regions decided? How are the sizes/shapes/contexts of the regions important? If I understand correctly, the image regions are removed by masking? If image editing operations were used to dynamically remove or perturb background regions, or even change regions with complex shapes, how would this affect the results?
How does ConServe compare to just the SFT model? Is this included in the ablation study in Table 3? I found the ablations to be quickly brushed over and hard to interpret. According to Figure 4(a), the translation quality is maintained from background retention ratio , especially on the MCiT benchmark. However, there are some performance gains from including more of the image on the newly introduced benchmark SC-TIMT. Could you provide more of an explanation why there are more performance gains specifically for SC-TIMT? And some insights into what kind of examples cause this result and the failure cases of ConServe?
The EATA scores seem quite high, with most scores >= 4 and no scores seemingly below 3. Why is this, and is this a limitation or feature of the agent?
Small notes: Figure 3 text on the plots is very small. How can TIMT help text-only translation and vice versa?
6 comments
Response to Reviewer quAV — Part 2/5
Q3: Are the construction and presentation of SC-TIMT sufficient, and why are some existing TIMT methods not included in the comparison experiments?
A3: We have moved Related Work into Sec. 2 of the main text. Sec. 5 briefly introduces SC-TIMT, including its 1,100 real-world screen images, four source languages, and focus on context-sensitive evaluation. Due to the page limit, more detailed information about image sources, sample selection, privacy review, Chinese gold translations provided by five translation experts, and the separation between training and test data is presented in Appendix A and Table 4. This provides an overview of the benchmark’s role in the main text while retaining verifiable details of its construction in the appendix.
Regarding comparisons with existing TIMT methods, we have added experiments with InImageTrans [1] and CPR-Trans [2]. For InImageTrans, we used its official open-source implementation and resolved reproduction issues through communication with the authors. We completed evaluations on SC-TIMT and MCiT, and the results have been added to Tables 1 and 2 in Sec. 5 of the revised main text. For CPR-Trans, since no official open-source implementation is available, we independently implemented its multi-stage pipeline for scene understanding, text perception, and translation reasoning based on the paper’s description, and evaluated it on both benchmarks. The results show that ConServe outperforms InImageTrans on both benchmarks and surpasses CPR-Trans in Agent Score. In particular, compared with CPR-Trans, ConServe reduces the average number of tokens by 13.9% on SC-TIMT and 31.6% on MCiT, and reduces inference latency by 78.2% and 77.5%, respectively, while maintaining competitive translation quality. These results further demonstrate its advantage in balancing translation performance and inference efficiency. The detailed results are as follows.
Additional results on SC-TIMT
| Method | Size | EN | KO | JA | VI | Agent ↑ | BLEU ↑ | METEOR ↑ | Tokens / Image ↓ | Latency (s) ↓ |
|---|---|---|---|---|---|---|---|---|---|---|
| InImageTrans | 9B | 3.03 | 2.87 | 2.44 | 3.11 | 2.88 | 43.47 | 38.96 | 3,238 | 6.22 |
| CPR-Trans | 9B | 4.52 | 4.09 | 4.10 | 4.07 | 4.21 | 52.70 | 69.36 | 4,611 | 32.24 |
| ConServe | 9B | 4.56 | 4.03 | 4.21 | 4.09 | 4.32 | 53.83 | 73.49 | 3,969 | 7.04 |
Additional results on MCiT
| Method | Size | Paper | News | Novel | Title | Sign | Introduction | Cover | Leaflet | Agent ↑ | BLEU ↑ | METEOR ↑ | Tokens / Image ↓ | Latency (s) ↓ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| InImageTrans | 9B | 3.90 | 3.67 | 2.31 | 4.57 | 4.14 | 3.44 | 2.86 | 3.76 | 3.61 | 39.84 | 58.52 | 1,347 | 5.12 |
| CPR-Trans | 9B | 4.92 | 4.66 | 4.43 | 4.67 | 4.80 | 4.74 | 4.52 | 4.37 | 4.65 | 46.61 | 63.11 | 2,844 | 26.82 |
| ConServe | 9B | 4.90 | 4.59 | 4.06 | 4.97 | 4.92 | 4.80 | 4.59 | 4.79 | 4.71 | 44.72 | 62.62 | 1,946 | 6.03 |
Neither MT3 [3] nor GLoTran [4] provides a complete official implementation or the corresponding training data, and both rely on specialized training pipelines. Specifically, MT3 uses multi-task reinforcement learning, with task-level rewards jointly optimizing text recognition, contextual reasoning, and translation. GLoTran relies on a specially constructed GLoD dataset and fine-tuning with global–local visual inputs. Due to the lack of open-source code and corresponding training resources, independently reconstructing the training pipelines based only on the papers’ descriptions would make it difficult to ensure consistency with the original implementations and training settings. Therefore, to ensure reliable experimental comparisons, we have not included reproduction results that cannot be sufficiently verified in the formal baseline comparisons.
Response to Reviewer quAV — Part 3/5
Q4: How are background regions defined in TCL, and how do region granularity, shape, and intervention choices affect the analysis?
A4: In TCL, the background is defined relative to the current translation target: the target text’s bounding box with a margin is retained, while the rest of the image is treated as the visual context that can be intervened on, including surrounding text, image content, and interface elements. Specifically, we construct regular rectangular regions on the base visual grid and map them back to the original image for pixel-level interventions. The partition is determined before effect estimation and is independent of model importance scores. In the following tables, “substitutable regions” are regions satisfying and . denotes the joint removal effect of the Top- regions ranked by individual necessity, normalized by the effect of removing the entire background. The mean and median are computed over the diagnostic samples, and the row reports the proportion of samples satisfying this condition. Here, , , and indicate the number of base-grid cells contained in each intervention region, not the spatial quota grid used by SQ-FPS.
To analyze the effect of region size, we compared the three regular rectangular partition granularities above. As the regions become coarser, the mean number of substitutable regions per image decreases from 55.41 to 32.41, while the proportions of samples containing multiple substitutable regions remain at 99.46%, 98.97%, and 95.62%, respectively. The mean values are 0.2033, 0.2996, and 0.3859, respectively; the corresponding proportions of samples satisfying are 90.32%, 80.85%, and 77.74%. Coarser regions contain more visual information and therefore produce larger joint removal effects, but the overall pattern of distributed and substitutable evidence remains stable. TCL measures regional contributions using necessity and independent support , which respectively quantify the effect of removing a region from the full background and restoring it alone when the other background regions are masked.
To examine the effect of intervention choices, we added experiments using mean-color filling, strong blur, and strong noise. Under all four operations, the proportion of samples containing multiple substitutable regions is above 97%. The mean is 0.2996 for the original Mask, compared with 0.2584, 0.2871, and 0.2547 for mean-color filling, strong blur, and strong noise, respectively. The corresponding proportions of samples with are 83.26%, 81.08%, and 84.29%, close to 80.85% for the original Mask. Thus, the observed distributed visual contributions are not an artifact of the original Mask operation alone. Tables 6–9 in Appendix E.2 of the revised manuscript report the complete statistics for different partition granularities and intervention choices. Complex-shaped regions have not yet been included in this robustness analysis, and we do not generalize the conclusions to arbitrary region shapes.
TCL analysis under different region granularities
| Statistic | |||
|---|---|---|---|
| Proportion of samples with multiple substitutable regions | 99.46% | 98.97% | 95.62% |
| Substitutable regions per image (mean / median) | 55.41 / 43.50 | 45.74 / 39.00 | 32.41 / 28.00 |
Top-k joint removal effects under different region granularities
| Granularity | Statistic | Top-1 | Top-5 | Top-10 | Top-20 |
|---|---|---|---|---|---|
| Mean | 0.1455 | 0.1850 | 0.2102 | 0.2033 | |
| Median | 0.0969 | 0.1280 | 0.1520 | 0.1111 | |
| Proportion with | 96.77% | 93.55% | 90.32% | 90.32% | |
| Mean | 0.1664 | 0.2175 | 0.2573 | 0.2996 | |
| Median | 0.0891 | 0.1280 | 0.1392 | 0.1801 | |
| Proportion with | 93.62% | 90.43% | 86.28% | 80.85% | |
| Mean | 0.1834 | 0.2572 | 0.3532 | 0.3859 | |
| Median | 0.1250 | 0.1648 | 0.2025 | 0.2439 | |
| Proportion with | 93.55% | 89.10% | 87.42% | 77.74% |
Substitutable-region statistics under different background interventions
| Statistic | Original Mask | Mean fill | Strong blur | Strong noise |
|---|---|---|---|---|
| Proportion of samples with multiple substitutable regions | 98.97% | 99.01% | 98.32% | 97.43% |
| Substitutable regions per image (mean / median) | 45.74 / 39.00 | 45.88 / 39.00 | 43.09 / 37.00 | 44.66 / 38.00 |
Response to Reviewer quAV — Part 1/5
Thank you very much for your careful review and overall positive assessment of our work. In response to your comments and questions, we have added further experimental analyses and made corresponding revisions to the paper’s organization and experimental discussion. We resond below.
Q1: How does TCL’s analysis motivate the visual diversity score in SQ-FPS, and can the visual regions actually selected by SQ-FPS be illustrated?
A1: TCL reveals two characteristics of translation-relevant visual context: visual evidence is distributed across multiple background regions, and different regions provide functionally overlapping information. These two findings motivate the spatial quota and visual diversity components of SQ-FPS, respectively. Spatial quotas maintain visual coverage across regions, while visual diversity measures differences among visual features within each region and helps adjust the token budget allocated to each region. This allows regions with more diverse visual information to receive a larger sampling budget, reducing the risk of losing distinct visual cues during compression. To further connect visual diversity with TCL’s regional analysis, we examined the correlation between visual diversity and regional contributions. The mean Spearman correlations of visual diversity with the strengths of positive independent support (S>0) and positive necessity contributions (C>0) are +0.255 and +0.210, respectively. After controlling for the number of background tokens, the correlations remain +0.244 and +0.169, respectively. These results indicate a positive association between visual diversity and translation-relevant regional contributions, but visual diversity is not equivalent to semantic importance. We therefore use it as an auxiliary cue for budget allocation rather than as a direct predictor of the translation importance of visual regions. The ablation results further support this design: removing diversity-based quota allocation reduces the Agent Scores on SC-TIMT and MCiT from 4.32 to 4.17 and from 4.71 to 4.50, respectively. In addition, following the reviewer’s suggestion, we have added visualizations of SQ-FPS compression and the corresponding translations to the appendix, showing the visual information actually retained and its effects on translation. The relevant definitions and analyses are provided in Eq. (8) of Sec. 4.2, Table 3 of Sec. 5, and Appendix E.3 of the revised manuscript. Appendix F presents the SQ-FPS compression visualizations and corresponding translations.
Q2: How are TCL’s analysis of regional effects on translation decisions and CCR’s evaluation of contextual consistency in the final translations connected?
A2: We agree that the original manuscript did not sufficiently explain the relationship between TCL and CCR. They examine the influence of visual context on translation from different perspectives. TCL constructs competing context-consistent and context-inconsistent interpretations and applies controlled interventions to the visual background to analyze how different regions affect the model’s translation preferences, thereby revealing the distributed and redundant structure of translation-relevant visual evidence. CCR evaluates whether the final translation preserves the intended meaning in the context of the full image. For each translation item, the evaluator makes 10 binary judgments, each taking the full image, source text, expert gold translation, and candidate translation as input. The majority vote determines the item-level binary label (0 or 1), while a 5–5 tie is resolved through human adjudication. CCR is then calculated as the proportion of items judged to be contextually consistent. The evaluation-metric description in Sec. 5 of the revised manuscript and Eq. (13) in Appendix D.2 provide this definition and distinguish TCL’s diagnosis of decision sensitivity from CCR’s evaluation of contextual consistency in the final translation. TCL’s analysis motivates the design of SQ-FPS, while CCR assesses whether the final translations produced from compressed visual inputs preserve the intended contextual meaning.