acceptodds
Under review as a conference paper at ICLR 2027

TextReVision: Benchmarking Image Text Editing towards Intent Fidelity with Gate-and-Craft Evaluation

Abstract

Reliable image text editing requires satisfying textual, spatial, stylistic, and semantic constraints while preserving non-target content. Existing benchmarks inadequately cover complex editing intents, while text recognition and image similarity alone cannot establish whether an instruction has been fulfilled. We present TextReVision, a bilingual, scenario-driven benchmark comprising 1,200 human- curated samples across 16 scenarios and five capability families: basic editing, layout control, style design, semantic reasoning, and multi-reference composition. Sample-specific rubrics provide 6,426 verifiable criteria across eight evaluation dimensions. Furthermore, we propose Gate-and-Craft scoring, in which essential task completion bounds the attainable score and scales the contribution of execution quality. The scoring rule achieves a concordance correlation coefficient of 0.935 against human reference scores. Across 18 models and 21 configurations, the leading scenario-balanced scores reach 89.5/89.3 on Chinese/English samples, yet semantic reasoning remains a major weakness. SenseNova-U1.5-8B-MOT’s official thinking mode raises its reasoning scores from 34.7/31.8 to 71.7/70.1, revealing that reasoning over user intent and translating it into actionable edits are critical. TextReVision advances evaluation toward verifiable, intent-grounded image text editing.

Primary area:
datasets and benchmarks
Keywords:
Text Editing, Text Rendering, Diffusion Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WiQ6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces TextReVision, a bilingual benchmark with 1,200 image text-editing samples across 16 scenarios. It provides sample-specific rubrics and a Gate-and-Craft scoring rule intended to prioritize task completion over visual quality. Evaluations of 18 models reveal substantial weaknesses in semantic reasoning, while a human study assesses the scoring rule and automatic judges.

Critical strengths
  1. The benchmark covers practical editing requests beyond text replacement, including translation, form filling, and multi-reference composition. Its 6,426 sample-specific criteria make failures more interpretable than a single similarity score.
  2. The evaluation separates several aspects of editing quality and compares multiple judges against expert annotations. The resulting capability profiles provide useful evidence that strong text rendering does not imply reliable execution of complex instructions.
Critical weaknesses
  1. The Gate does not consistently enforce essential task completion. For artistic insertion, Table 6 makes Font Accuracy the Gate and assigns Text Accuracy a 0.35 Craft weight. Under Eq. (1), incorrect requested text with otherwise perfect ratings can therefore score 82.5/100. The independent dimension definitions do not specify a rule excluding this combination, leaving the scoring framework vulnerable to the failure it aims to prevent.
  2. The validation does not clearly separate calibration from evaluation. The same 32 prompts and four generating systems are used to select β and report aggregation agreement. Thus, the headline 0.935 agreement is not a held-out estimate, and the pooled results provide limited evidence about reliability across individual scenarios.
Questions for authors
  1. Can incorrect text still receive full Gate credit in typography- or layout-gated tasks? If not, where is this dependency defined, and how does it interact with independent dimension scoring?
  2. Does the aggregation advantage persist when β is selected on separate prompts? Can held-out results distinguish reliability across scenarios and generating systems?
5 comments
AuthorsOpenReview ↗

Response to Reviewer WiQ6 (Part1)

First, we truly thank for your thoughtful review and constructive feedback. We particularly appreciate your recognition of TextReVision's practical task coverage, fine-grained and interpretable evaluation criteria, and the diagnostic insights revealed by our benchmark.

Your positive assessment and score are deeply encouraging and important for us, as it helps us recognize the value and significance of our work and motivates us to further improve it.

Meanwhile ,we carefully considered your concerns regarding the robustness of Gate-and-Craft design (Q1) and the held-out validation of aggregation reliability (Q2). To address these concerns, we conducted additional failure-case analyses, explored Multi-Gate formulations, and expanded our human validation with further experiments on held-out prompts, as detailed below.

AuthorsOpenReview ↗

Response to Reviewer WiQ6 (Part2)

Q1: The Robustness of Gate-and-Craft Design.

We are very grateful for your feedback on the potential issue of Gate-and-Craft and your suggestion to examine how Multi-Gate formulations affect model rankings. We address these concerns from three complementary perspectives.


A1.1: Design Motivation and Failure-Case Frequency.


Our Gate-and-Craft design emphasizes the primary capability of each editing scenario for more targeted evaluation. Importantly, TextReVision already assesses fundamental text insertion, deletion, and replacement in Basic Text Editing, where GPT-Image-2 substantially outperforms BAGEL (97.7/98.7 vs. 49.8/70.6), revealing clear differences in basic text editing accuracy. Thus, for advanced scenarios such as Artistic Insertion, we prioritize scenario-specific capabilities, such as font accuracy.

To directly address your concern, we therefore conducted a systematic analysis covering all 16 scenarios, comprising 1,200 editing tasks evaluated across 21 model configurations (25,200 outputs). Among these, 16,894 valid outputs were associated with tasks where Text Accuracy (TA) was assessed as a Craft dimension rather than the Gate. Only 15 cases (0.089%) exhibited this specific failure pattern (TA = 0, with all other dimensions receiving perfect scores). These results indicate that this specific extreme failure pattern is exceptionally rare in pratice, further supporting the stability of our evaluation framework.


A1.2: Single-Gate vs. Multi-Gate: Consistent Overall Model Rankings.


Following your suggestion, we investigate whether incorporating multiple Gate dimensions affects model rankings across the full benchmark (25,200 outputs). We extend Single-Gate by combining Text Accuracy (TA) with the original scenario-specific Gate (G), affecting 16,894 outputs across 12 scenarios where TA serves as a Craft dimension.

We evaluate three strategies: Minimum (min(TA, G)), Product (TA × G), and Geometric Mean (√(TA × G)). All variants retain the original model outputs and dimension scores, removing TA from Craft and renormalizing the remaining weights. We assess ranking consistency between Single-Gate and each Multi-Gate variants using Spearman’s ρ (95% CI) and Kendall’s τb, both overall and across individual scenarios.

Table1: Ranking Consistency Between Single-Gate and Multi-Gate Variants.

Multi-Gate StrategyOverall Spearman’s ρ↑[95%CI]Overall Kendall’s τb↑Artistic Insertion Spearman’s ρ↑Geometric Transformation Spearman’s ρ↑
Minimum0.9909[0.9870,0.9974]0.96190.98310.9987
Product0.9909[0.9844,0.9961]0.96190.98570.9987
Geometric Mean0.9909[0.9883,0.9974]0.96190.98571.0000

The results demonstrate robust overall model rankings, with Spearman's ρ = 0.9909 across all three strategies. Notably, ranking consistency remains high in the reviewer-highlighted scenarios: Artistic Insertion (ρ = 0.9831–0.9857) and Geometric Transformation (ρ = 0.9987–1.0000). These findings further support the robustness of our original Single-Gate formulation.


A1.3: Agreement with Human Judgments.


Ranking robustness alone does not guarantee agreement with human judgments. To further validate this, we expanded our human evaluation to 300 distinct prompts proportionally sampled across 16 scenarios and two languages, yielding 1,200 outputs from four representative models. We compare Single-Gate and three Multi-Gate formulations against independent human ratings using CCC, MAE, and RMSE. Human annotators were blinded to model identities and automatic scores.

Table2: Agreement with Human Judgments Across Gate Formulations.

Evaluation StrategyCCC ↑ [95% CI]MAE ↓RMSE ↓Spearman ρ ↑ [95% CI]ΔCCC vs Single-Gate [95% CI]
Single-Gate (Ours)0.9567 [0.9450, 0.9659]0.04790.11140.9361 [0.9205, 0.9511]—
Multi-Gate (Minimum)0.9562 [0.9467, 0.9645]0.05040.11270.9277 [0.9104, 0.9450]−0.0005 [−0.0073, +0.0077]
Multi-Gate (Product)0.9535 [0.9436, 0.9619]0.05670.11880.9330 [0.9153, 0.9499]−0.0032 [−0.0115, +0.0060]
Multi-Gate (Geometric Mean)0.9538 [0.9434, 0.9627]0.05430.11480.9369 [0.9215, 0.9520]−0.0029 [−0.0097, +0.0057]

Note: The reported human agreement is calculated against holistic overall scores assigned by humans to each output.

The results show consistently strong agreement with human judgments across Gate formulations, in both absolute scoring (CCC = 0.9535–0.9567) and ranking consistency (Spearman’s ρ = 0.9277–0.9369). Differences between Single-Gate and Multi-Gate are small, with all ΔCCC 95% confidence intervals including zero. Together with the rarity of the specific extreme failure pattern (0.089%), these findings demonstrate the empirical robustness of Single-Gate, while Multi-Gate offers a complementary approach to enforcing multiple essential constraints.

AuthorsOpenReview ↗

Response to Reviewer WiQ6 (Part3)


Additional Summary of Our Response to Q1


Finally, we would like to briefly summarize our response to Q1. We acknowledge the limitations of Single-Gate in enforcing multiple essential constraints, while our additional analyses demonstrate its empirical robustness in model rankings and agreement with human judgments. We would also like to draw your attention to three key contributions of TextReVision: (1) Practical Relevance: covering diverse real-world image text editing needs; (2) Fine-Grained Evaluation: providing extensively human-curated, sample-specific rubrics that support interpretable assessment and flexible aggregation strategies; and (3) Actionable Diagnosis: identifying capability gaps, such as FLUX's weaknesses in Chinese text rendering and target localization, and widespread limitations in reasoning-intensive editing, to guide future improvements. We will incorporate the Single-Gate and Multi-Gate sensitivity analyses into the revised manuscript, further clarifying their strengths and limitations. We hope these findings address your concerns and reinforce the broader value of TextReVision. We sincerely thank you once again for your valuable suggestions and willingness to engage in further discussion.

Reviewer_HT7eOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

TEXTREVISION is a bilingual benchmark for image text editing with 1,200 human-curated examples across 16 scenarios. It introduces sample-specific rubrics and a Gate-and-Craft scoring method that separates task completion from visual quality. The authors evaluate 18 models in 21 configurations and find that semantic reasoning, layout control, and multi-reference editing remain challenging. The scoring method shows strong agreement with human ratings on a sample of outputs.

Critical strengths
  1. The benchmark covers realistic editing tasks in both Chinese and English, going beyond simple text replacement.
  2. Gate-and-Craft scoring treats task completion as essential and is checked against human judgments, making the evaluation useful for diagnosing model failures.
Critical weaknesses
  1. Human validation covers only 128 outputs. This is useful initial evidence, but limited for a benchmark with 18 models and 16 scenarios.
  2. The automatic evaluator depends on a vision-language model and reference edits. Small text defects and open-ended design choices may still be judged inconsistently.
Questions for authors
  1. How sensitive are model rankings to the choice of automatic judge and the reference edit provided to the judge?
  2. How were the 128 human-evaluated outputs selected, and how representative are they of the full benchmark?
8 comments
AuthorsOpenReview ↗

Response to Reviewer HT7e (Part1)

First, we sincerely thank you for your careful review and thoughtful feedback. We especially appreciate your recognition of our TextReVision's coverage of realistic bilingual editing tasks and the diagnostic value of our Gate-and-Craft evaluation.

Your valuable concerns regarding the scale and representativeness of human validation (Q1), the sensitivity of model rankings to automatic judges and reference edits (Q2), and the reliability of evaluating small text defects and open-ended editing tasks (Q3) have motivated us to investigate these issues more thoroughly. Accordingly, we have expanded our human study, conducted additional sensitivity analyses, and further clarified our evaluation design.

We truly hope that our explanations and additional experiments below address your concerns.

AuthorsOpenReview ↗

Response to Reviewer HT7e (Part2)


Q1: The Scale and Representativeness of Human Validation.


We thank you for the constructive comments on the scale and rigor of our human validation, which help us further clarify the scope and significance of TextReVision. To address your concern, we conduct a more comprehensive human evaluation covering all 16 scenarios, both two languages, and four representative generating models from diverse model families to ensure broad coverage. Specifically, the evaluation comprises 300 distinct editing prompts and 1,200 outputs, with prompts selected through stratified random sampling across scenario-language groups. Based on the extended human validation, we draw the following three key conclusions.


A1.1: Strong Agreement with Human Judgments on a Substantially Expanded Evaluation Set.


Compared with the original 128-output validation, we expanded human evaluation to 1,200 outputs from 300 distinct prompts across all 16 scenarios and both languages. The expanded validation set was randomly sampled from each scenario in proportion to its original distribution, ensuring representative coverage of the benchmark. Notably, despite the substantially broader task coverage, Gemini-3.7-Flash achieves an improved human-agreement CCC of 0.9319, compared with 0.905 in the original 128-output version. The two independent human annotators also achieve a high inter-annotator CCC of 0.9799, further confirming the reliability and consistency of our human annotations. These results provide stronger empirical evidence for the reliability and generalizability of our evaluation framework across diverse editing tasks.

Table1: Human–Judge Agreement Across Validation Sets.

Judge ComparisonCCC↑MAE↓RMSE↓
1200-output validation
Human1–Human20.97990.02240.0754
Gemini-3.7–Mean(Human1,Human2)0.93190.07480.1376
Gemini-3.7–Human10.92740.07830.1423
Gemini-3.7–Human20.92680.07380.1430
128-output validation
Human1–Human20.9690.0440.092
Gemini-3.7–Mean(Human1,Human2)0.9050.1000.161

Note: The reported agreement is calculated against Gate-and-Craft overall scores aggregated from dimension-level ratings assigned by humans to each output.


A1.2: Consistent Evaluation Reliability Across Diverse Scenarios and Languages.


To address your concern about limited task coverage, we further examine human agreement across five capability categories and both languages, rather than relying solely on aggregated results. As shown below, Gemini-3.7-Flash maintains consistently high agreement with mean human ratings across all five capabilities (CCC = 0.9081–0.9416), including challenging Reasoning (0.9242) and Reference-guided Editing (0.9350). Similarly, agreement remains highly consistent between English (0.9311) and Chinese (0.9324). These results provide stronger evidence that our evaluation framework generalizes across diverse editing capabilities and languages, rather than being driven by a limited subset of tasks.

Table2: Human–Judge Agreement Across Capabilities and Languages.

CategoryGemini-3.7–Human1CCC↑Gemini-3.7–Human2CCC↑Gemini-3.7–Mean(Human1,Human2)CCC↑
Capability
Basic0.93910.93930.9416
Layout0.90190.90210.9081
Style0.90780.90960.9109
Reasoning0.91830.91760.9242
Reference-guided0.92890.93860.9350
Language
English0.93050.92090.9311
Chinese0.92430.93230.9324

Note: The reported agreement is calculated against Gate-and-Craft overall scores aggregated from dimension-level ratings assigned by humans to each output.


A1.3: Robust Agreement Across Different Generative Model Families.


Consistent with the above experiments and further assess evaluation reliability across different generating systems, we further compare the rankings of the four evaluated generating systems to assess the reliability of relative model comparisons. Notably, Gemini-3.7-Flash and both independent human annotators yield identical model rankings: GPT-Image-2 > SenseNova (Thinking) > SenseNova > FLUX.2, achieving Spearman's ρ = 1.000 and Kendall's τb = 1.000. These findings further demonstrate strong alignment between TextReVision's automatic evaluation and human judgments in relative model comparisons.


Additional Summary of Our Response to Q1


Finally, we would like to briefly summarize our response to Q1. Our expanded human study substantially strengthens the empirical validation of TextReVision, providing more representative coverage and demonstrating consistent agreement across scenarios, languages, and generating systems. We'll incorporate these additional validation into the revised manuscript to further strengthen the reliability of our work. We hope these findings address your concerns and reinforce the value of TextReVision as a fine-grained and interpretable benchmark. We sincerely thank you again.

AuthorsOpenReview ↗

Response to Reviewer HT7e (Part3)

Q2: The Sensitivity Across Different Automatic Judges and Reference Edits.

We thank you for raising this important question. To examine the reliability of our automatic evaluation, we conduct additional analyses of judge selection and reference availability on the same 1,200-output human validation set.


A2.1: Advanced VLM Judges Achieve Strong Agreement with Human Judgments.


When designing our evaluation framework, we recognized that VLM judges vary in their evaluation capabilities, making the selection of a reliable judge essential for TextReVision. To further examine sensitivity to judge selection, we compare three automatic judges against independent human ratings on our expanded 1,200-output validation set, reporting ranking consistency (Spearman’s ρ) and score agreement (CCC).

Table 3: Agreement Between Automatic Judges and Human Ratings.

Automatic JudgeSpearman ρ ↑CCC ↑
Gemini-3.7-Flash0.93610.9567
GPT-6-Astra0.95350.9682
Qwen3.8-27B0.77770.6746

Note: The reported human agreement is calculated against holistic overall scores assigned by humans to each output.

The results show that our selected Gemini-3.7-Flash and GPT-6-Astra achieve comparably strong agreement with human judgments, while Qwen3.8-27B shows larger discrepancies. These results support the reliability of our selected judge, Gemini-3.7-Flash, while highlighting the importance of judge selection.


A2.2: Human-Verified Reference Edits Improve Evaluation Reliability.


During benchmark development, we observed that VLM judges often struggle to identify correct outcomes in complex editing tasks when relying solely on potentially ambiguous textual rubrics. This motivated us to invest substantial effort in constructing high-quality, human-verified reference edits that provide concrete visual guidance for evaluation.

To further validate their importance, we compare evaluations with and without reference edits on identical generated outputs, using setting-specific rubrics while keeping the judges and other procedures unchanged. Providing references substantially improves agreement with human judgments for Gemini-3.7-Flash and GPT-6-Astra, with smaller gains for Qwen3.8-27B (We speculate may be limited by its weaker evaluation capabilities). These results further support our reference-guided evaluation design and the importance of carefully curated reference edits.

Table 4: Model Ranking Sensitivity to Reference Edit.

Automatic JudgeValid Distinct PromptsValid Distinct OutputsWith Ref. (ρ / τ) ↑Without Ref. (ρ / τ) ↑Δρ / Δτ
Gemini-3.7-Flash267 / 3001,068 / 1,2000.9358 / 0.91140.7466 / 0.7089-0.1892 / -0.2025
GPT-6-Astra273 / 3001,092 / 1,2000.9503 / 0.92260.8213 / 0.7924-0.1290 / -0.1302
Qwen3.8-27B187 / 300748 / 1,2000.8051 / 0.77200.7985 / 0.7660-0.0066 / -0.0060

Note: Δ = Without Ref. − With Ref. Spearman’s ρ and Kendall’s τ are averaged across valid prompts, comparing automatic rankings against human rankings. Prompts with undefined correlations due to fully tied scores are excluded, and each judge is evaluated on the same valid prompts under both reference settings. Valid sample counts may differ across judges.

Reviewer_KkXfOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a bilingual image text editing benchmark, TextReVision, consisting of 1200 sampless across 16 scenarios and 5 capability families. For each data point in the benchmark, authors provide specific rubrics covering eight evaluation dimensions. To conduct a reliable and practical evaluation, this paper further introduces a new evaluation method: Gate-and-Craft. Specifically, this evaluation method has a designated task-completion criterion that controls the contribution of other quality dimensions. The comprehensive evaluation on 18 models across 21 configurations show that there are substantial weaknesses in semantic reasoning and model-dependent benefits from thinking modes.

Critical strengths
  1. This paper is easy to follow. The motivation and the method are introduced clearly.
  2. The benchmark is meaningful and practically motivated. The benchmark extends beyond basic text replacement to challenging tasks such as knowledge-dependent editing and multi-reference composition.
  3. The annotation looks reliable and interpretable.
Critical weaknesses
  1. From my perspective, the main contribution including two parts: the datasets, and the evaluation metrics. Although the datasets are meaningful, there are some issues in the evaluation method: Gate-and-Craft. The most severe issue is that this method only use one dimension as "gate" and all others are craft. However, sometimes, one edit may contain multiple important aspects. For example, for artistic insertion, Table 6 assigns Text Accuracy a Craft weight of 35%, therefore, a hypothetical output with completely incorrect text but fully satisfied remaining criteria would receive 82.5/100. This raises a substantive concern about whether the scoring rule consistently measures intent fidelity. One way to address my concern is that, use a subset to show that assigning more dimensions as gate will not significantly change the final ranking. Otherwise, this design has a severe issue.
  2. Human validation has limited task coverage, which only covers 128 outputs originate from only 32 distinct prompts across 16 scenarios and two languages. This means that for each combination of "scenario x language", only one prompt are manually evaluated. Although the prompt-grouped bootstrap analysis is appropriate, it does not establish reliability across the broader task distribution.

I'd like to join the discussion during rebuttal if authors can provide enough additional experiments to rebuttal my reviews.

Questions for authors
  1. How frequently do outputs that violate an essential non-Gate requirement nevertheless receive high scores? Would jointly gating multiple necessary criteria better match human judgments?
7 comments
AuthorsOpenReview ↗

Response to Reviewer KkXf (Part1)

First, we sincerely thank you for your thoughtful feedback and encouraging recognition of the clarity of our work, the significance and practical value of our benchmark, and the quality of our annotations. Your positive assessment means a great deal to us and motivates us to further improve our work.

Meanwhile, we understand that your concerns primarily center on two aspects: the robustness of our Gate-and-Craft design (Q1) and the limited task coverage and representativeness of the original human validation (Q2). We address both concerns below with additional analyses and empirical evidence.

We truly hope that our explanations and additional experiments below address your concerns.

AuthorsOpenReview ↗

Response to Reviewer KkXf (Part2)

Q1: The Robustness of our Gate-and-Craft Design.

We are very grateful for your feedback on the potential issue of Gate-and-Craft and your suggestion to examine how Multi-Gate formulations affect model rankings. We address these concerns from three complementary perspectives.


A1.1: Design Motivation and Failure-Case Frequency.


Our Gate-and-Craft design emphasizes the primary capability of each editing scenario for more targeted evaluation. Importantly, TextReVision already assesses fundamental text insertion, deletion, and replacement in Basic Text Editing, where GPT-Image-2 substantially outperforms BAGEL (97.7/98.7 vs. 49.8/70.6), revealing clear differences in basic text editing accuracy. Thus, for advanced scenarios such as Artistic Insertion, we prioritize scenario-specific capabilities, such as font accuracy.

To directly address your concern, we therefore conducted a systematic analysis covering all 16 scenarios, comprising 1,200 editing tasks evaluated across 21 model configurations(25,200 outputs). Among these, 16,894 valid outputs were associated with tasks where Text Accuracy was assessed as a Craft dimension rather than the Gate. Only 15 cases (0.089%) exhibited this specific failure pattern (TA = 0, with all other dimensions receiving perfect scores). These results indicate that this specific extreme failure pattern is exceptionally rare in practice, further supporting the stability of our evaluation framework.


A1.2: Single-Gate vs. Multi-Gate: Consistent Overall Model Rankings.


Following your suggestion, we investigate whether incorporating multiple Gate dimensions affects model rankings across the full benchmark (25,200 outputs). We extend Single-Gate by combining Text Accuracy(TA) with the original scenario-specific Gate (G), affecting 16,894 outputs across 12 scenarios where TA serves as a Craft dimension.

We evaluate three strategies: Minimum (min(TA, G)), Product (TA × G), and Geometric Mean (√(TA × G)). All variants retain the original model outputs and dimension scores, removing TA from Craft and renormalizing the remaining weights. We assess ranking consistency between Single-Gate and each Multi-Gate variants using Spearman’s ρ (95% CI) and Kendall’s τb, both overall and across individual scenarios.

Table1: Ranking Consistency Between Single-Gate and Multi-Gate Variants.

Multi-Gate StrategyOverall Spearman’s ρ ↑ [95% CI]Overall Kendall’s τb ↑Artistic Insertion Spearman’s ρ ↑Geometric Transformation Spearman’s ρ ↑
Minimum0.9909 [0.9870, 0.9974]0.96190.98310.9987
Product0.9909 [0.9844, 0.9961]0.96190.98570.9987
Geometric Mean0.9909 [0.9883, 0.9974]0.96190.98571.0000

The results demonstrate robust overall model rankings, with Spearman's ρ = 0.9909 across all three strategies. Notably, ranking consistency remains high in the reviewer-highlighted scenarios: Artistic Insertion (ρ = 0.9831–0.9857) and Geometric Transformation (ρ = 0.9987–1.0000). These findings further support the robustness of our original Single-Gate formulation.


A1.3: Agreement with Human Judgments.


Ranking robustness alone does not guarantee agreement with human judgments. To further validate this, we expanded our human evaluation to 300 distinct prompts proportionally sampled across 16 scenarios and two languages, yielding 1,200 outputs from four representative models. We compare Single-Gate and three Multi-Gate formulations against independent human ratings using CCC, MAE, and RMSE. Human annotators were blinded to model identities and automatic scores.

Table2: Agreement with Human Judgments Across Gate Formulations.

Evaluation StrategyCCC ↑ [95% CI]MAE ↓RMSE ↓Spearman ρ ↑ [95% CI]ΔCCC vs Single-Gate [95% CI]
Single-Gate (Ours)0.9567 [0.9450, 0.9659]0.04790.11140.9361 [0.9205, 0.9511]—
Multi-Gate (Minimum)0.9562 [0.9467, 0.9645]0.05040.11270.9277 [0.9104, 0.9450]−0.0005 [−0.0073, +0.0077]
Multi-Gate (Product)0.9535 [0.9436, 0.9619]0.05670.11880.9330 [0.9153, 0.9499]−0.0032 [−0.0115, +0.0060]
Multi-Gate (Geometric Mean)0.9538 [0.9434, 0.9627]0.05430.11480.9369 [0.9215, 0.9520]−0.0029 [−0.0097, +0.0057]

Note: The reported human agreement is calculated against holistic overall scores assigned by humans to each output.

The results show consistently strong agreement with human judgments across Gate formulations, in both absolute scoring and ranking consistency. Differences between Single-Gate and Multi-Gate are small, with all ΔCCC 95% confidence intervals including zero. Together with the rarity of the specific extreme failure pattern (0.089%), these findings demonstrate the empirical robustness of Single-Gate, while Multi-Gate offers a complementary approach to enforcing multiple essential constraints.

AuthorsOpenReview ↗

Response to Reviewer KkXf (Part3)


Additional Summary of Our Response to Q1


Finally, we would like to briefly summarize our response to Q1. We acknowledge the limitations of Single-Gate in enforcing multiple essential constraints, while our additional analyses demonstrate its empirical robustness in model rankings and agreement with human judgments. We would also like to draw your attention to three key contributions of TextReVision: (1) Practical Relevance: covering diverse real-world image text editing needs; (2) Fine-Grained Evaluation: providing extensively human-curated, sample-specific rubrics that support interpretable assessment and flexible aggregation strategies; and (3) Actionable Diagnosis: identifying capability gaps, such as FLUX's weaknesses in Chinese text rendering and target localization, and widespread limitations in reasoning-intensive editing, to guide future improvements. We will incorporate the Single-Gate and Multi-Gate sensitivity analyses into the revised manuscript, further clarifying their strengths and limitations. We hope these findings address your concerns and reinforce the broader value of TextReVision. We sincerely thank you once again for your valuable suggestions and willingness to engage in further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.