- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces TextReVision, a bilingual benchmark with 1,200 image text-editing samples across 16 scenarios. It provides sample-specific rubrics and a Gate-and-Craft scoring rule intended to prioritize task completion over visual quality. Evaluations of 18 models reveal substantial weaknesses in semantic reasoning, while a human study assesses the scoring rule and automatic judges.
Critical strengths
- The benchmark covers practical editing requests beyond text replacement, including translation, form filling, and multi-reference composition. Its 6,426 sample-specific criteria make failures more interpretable than a single similarity score.
- The evaluation separates several aspects of editing quality and compares multiple judges against expert annotations. The resulting capability profiles provide useful evidence that strong text rendering does not imply reliable execution of complex instructions.
Critical weaknesses
- The Gate does not consistently enforce essential task completion. For artistic insertion, Table 6 makes Font Accuracy the Gate and assigns Text Accuracy a 0.35 Craft weight. Under Eq. (1), incorrect requested text with otherwise perfect ratings can therefore score 82.5/100. The independent dimension definitions do not specify a rule excluding this combination, leaving the scoring framework vulnerable to the failure it aims to prevent.
- The validation does not clearly separate calibration from evaluation. The same 32 prompts and four generating systems are used to select β and report aggregation agreement. Thus, the headline 0.935 agreement is not a held-out estimate, and the pooled results provide limited evidence about reliability across individual scenarios.
Questions for authors
- Can incorrect text still receive full Gate credit in typography- or layout-gated tasks? If not, where is this dependency defined, and how does it interact with independent dimension scoring?
- Does the aggregation advantage persist when β is selected on separate prompts? Can held-out results distinguish reliability across scenarios and generating systems?
5 comments
Response to Reviewer WiQ6 (Part2)
Q1: The Robustness of Gate-and-Craft Design.
We are very grateful for your feedback on the potential issue of Gate-and-Craft and your suggestion to examine how Multi-Gate formulations affect model rankings. We address these concerns from three complementary perspectives.
A1.1: Design Motivation and Failure-Case Frequency.
Our Gate-and-Craft design emphasizes the primary capability of each editing scenario for more targeted evaluation. Importantly, TextReVision already assesses fundamental text insertion, deletion, and replacement in Basic Text Editing, where GPT-Image-2 substantially outperforms BAGEL (97.7/98.7 vs. 49.8/70.6), revealing clear differences in basic text editing accuracy. Thus, for advanced scenarios such as Artistic Insertion, we prioritize scenario-specific capabilities, such as font accuracy.
To directly address your concern, we therefore conducted a systematic analysis covering all 16 scenarios, comprising 1,200 editing tasks evaluated across 21 model configurations (25,200 outputs). Among these, 16,894 valid outputs were associated with tasks where Text Accuracy (TA) was assessed as a Craft dimension rather than the Gate. Only 15 cases (0.089%) exhibited this specific failure pattern (TA = 0, with all other dimensions receiving perfect scores). These results indicate that this specific extreme failure pattern is exceptionally rare in pratice, further supporting the stability of our evaluation framework.
A1.2: Single-Gate vs. Multi-Gate: Consistent Overall Model Rankings.
Following your suggestion, we investigate whether incorporating multiple Gate dimensions affects model rankings across the full benchmark (25,200 outputs). We extend Single-Gate by combining Text Accuracy (TA) with the original scenario-specific Gate (G), affecting 16,894 outputs across 12 scenarios where TA serves as a Craft dimension.
We evaluate three strategies: Minimum (min(TA, G)), Product (TA × G), and Geometric Mean (√(TA × G)). All variants retain the original model outputs and dimension scores, removing TA from Craft and renormalizing the remaining weights. We assess ranking consistency between Single-Gate and each Multi-Gate variants using Spearman’s ρ (95% CI) and Kendall’s τb, both overall and across individual scenarios.
Table1: Ranking Consistency Between Single-Gate and Multi-Gate Variants.
| Multi-Gate Strategy | Overall Spearman’s ρ↑[95%CI] | Overall Kendall’s τb↑ | Artistic Insertion Spearman’s ρ↑ | Geometric Transformation Spearman’s ρ↑ |
|---|---|---|---|---|
| Minimum | 0.9909[0.9870,0.9974] | 0.9619 | 0.9831 | 0.9987 |
| Product | 0.9909[0.9844,0.9961] | 0.9619 | 0.9857 | 0.9987 |
| Geometric Mean | 0.9909[0.9883,0.9974] | 0.9619 | 0.9857 | 1.0000 |
The results demonstrate robust overall model rankings, with Spearman's ρ = 0.9909 across all three strategies. Notably, ranking consistency remains high in the reviewer-highlighted scenarios: Artistic Insertion (ρ = 0.9831–0.9857) and Geometric Transformation (ρ = 0.9987–1.0000). These findings further support the robustness of our original Single-Gate formulation.
A1.3: Agreement with Human Judgments.
Ranking robustness alone does not guarantee agreement with human judgments. To further validate this, we expanded our human evaluation to 300 distinct prompts proportionally sampled across 16 scenarios and two languages, yielding 1,200 outputs from four representative models. We compare Single-Gate and three Multi-Gate formulations against independent human ratings using CCC, MAE, and RMSE. Human annotators were blinded to model identities and automatic scores.
Table2: Agreement with Human Judgments Across Gate Formulations.
| Evaluation Strategy | CCC ↑ [95% CI] | MAE ↓ | RMSE ↓ | Spearman ρ ↑ [95% CI] | ΔCCC vs Single-Gate [95% CI] |
|---|---|---|---|---|---|
| Single-Gate (Ours) | 0.9567 [0.9450, 0.9659] | 0.0479 | 0.1114 | 0.9361 [0.9205, 0.9511] | — |
| Multi-Gate (Minimum) | 0.9562 [0.9467, 0.9645] | 0.0504 | 0.1127 | 0.9277 [0.9104, 0.9450] | −0.0005 [−0.0073, +0.0077] |
| Multi-Gate (Product) | 0.9535 [0.9436, 0.9619] | 0.0567 | 0.1188 | 0.9330 [0.9153, 0.9499] | −0.0032 [−0.0115, +0.0060] |
| Multi-Gate (Geometric Mean) | 0.9538 [0.9434, 0.9627] | 0.0543 | 0.1148 | 0.9369 [0.9215, 0.9520] | −0.0029 [−0.0097, +0.0057] |
Note: The reported human agreement is calculated against holistic overall scores assigned by humans to each output.
The results show consistently strong agreement with human judgments across Gate formulations, in both absolute scoring (CCC = 0.9535–0.9567) and ranking consistency (Spearman’s ρ = 0.9277–0.9369). Differences between Single-Gate and Multi-Gate are small, with all ΔCCC 95% confidence intervals including zero. Together with the rarity of the specific extreme failure pattern (0.089%), these findings demonstrate the empirical robustness of Single-Gate, while Multi-Gate offers a complementary approach to enforcing multiple essential constraints.
Response to Reviewer WiQ6 (Part3)
Additional Summary of Our Response to Q1
Finally, we would like to briefly summarize our response to Q1. We acknowledge the limitations of Single-Gate in enforcing multiple essential constraints, while our additional analyses demonstrate its empirical robustness in model rankings and agreement with human judgments. We would also like to draw your attention to three key contributions of TextReVision: (1) Practical Relevance: covering diverse real-world image text editing needs; (2) Fine-Grained Evaluation: providing extensively human-curated, sample-specific rubrics that support interpretable assessment and flexible aggregation strategies; and (3) Actionable Diagnosis: identifying capability gaps, such as FLUX's weaknesses in Chinese text rendering and target localization, and widespread limitations in reasoning-intensive editing, to guide future improvements. We will incorporate the Single-Gate and Multi-Gate sensitivity analyses into the revised manuscript, further clarifying their strengths and limitations. We hope these findings address your concerns and reinforce the broader value of TextReVision. We sincerely thank you once again for your valuable suggestions and willingness to engage in further discussion.
Response to Reviewer WiQ6 (Part1)
First, we truly thank for your thoughtful review and constructive feedback. We particularly appreciate your recognition of TextReVision's practical task coverage, fine-grained and interpretable evaluation criteria, and the diagnostic insights revealed by our benchmark.
Your positive assessment and score are deeply encouraging and important for us, as it helps us recognize the value and significance of our work and motivates us to further improve it.
Meanwhile ,we carefully considered your concerns regarding the robustness of Gate-and-Craft design (Q1) and the held-out validation of aggregation reliability (Q2). To address these concerns, we conducted additional failure-case analyses, explored Multi-Gate formulations, and expanded our human validation with further experiments on held-out prompts, as detailed below.