- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3(+1)
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
STRIDE is a training-free, multi-agent refinement loop that wraps a frozen open-weight text-to-image generator (ERNIE-Image, Infographic-V2, or Qwen-Image-2512) for infographic generation. A rubric verifier (Qwen3.5-397B-A17B) scores each candidate against atomic yes/no constraints, which on BizGenEval and IGenBench are taken directly from the released evaluation checklists; a perceptor (Kimi-K3) returns OCR text boxes and flags garbled text. A lead editor (Kimi-K3) then routes each round either to a composer, which appends failure-derived constraints (optionally web-retrieved) to the prompt and resamples the generator, or to a retoucher, which masks one OCR box and pastes a re-rendered text/LaTeX string. Four candidates are proposed per round, and a (1+λ) rule replaces the incumbent only when the verifier score strictly improves. With best-of-6 initialization and up to three rounds, the paper reports BizGenEval hard scores of 51.1 (ERNIE, from 28.0) and 69.5 (Infographic-V2, from 41.1), IGenBench Q-ACC/I-ACC of 0.84/0.38 (from 0.72/0.17), and 0.92 WiScore on WISE. Supporting evidence includes component ablations (Tables 3, 9), an evolution-strategy comparison (Tables 4, 17), oracle runs that use the benchmark evaluators as verifier/perceptor (Tables 11, 12), and a 100-pair human study with the same atomic questions (Table 5).
Critical strengths
-
The incumbent-retention mechanism is isolated cleanly and matters. Table 4 / Table 17 compare forced replacement with champion retention at matched candidate counts: (1,1) 39.4 vs (1+1) 45.6, and (1,4) 45.4 vs (1+4) 51.1. This is a well-controlled ablation and shows that rejecting verifier-non-improving updates is a large part of what makes iterative verifier-guided refinement stable. It is useful to anyone building such loops.
-
The gains are large and survive human rubric evaluation. On the 100 paired prompt–generator cases in Table 5, human-scored BizGenEval hard scores rise by +20.3 (ERNIE) and +29.8 (Infographic-V2), and IGenBench Q-ACC by +0.16 and +0.12, with human–evaluator correlations of 0.65–0.85. This rules out the worry that the improvement exists only in the eyes of the internal verifier. The appendix also documents prompts, verifier inflation (Table 15) and routing statistics (Table 10) in useful detail.
Critical weaknesses
1. The paper's central methodological claim — that scale-aware routing between a global composer and a local retoucher drives the improvement — is not supported by its own ablations.
- The main-text argument (l. 449–451) compares "retoucher-only" (50.2) with the full system (51.1). According to the ✓/– columns of Table 3 and Table 9, the 50.2 row is lead + retoucher without the perceptor, while 51.1 includes the perceptor. Holding the perceptor fixed, routing adds only +0.7 (lead + retoucher + perceptor = 50.4, Table 3 row 6 / Table 9 row 14).
- Without the perceptor, adding the composer hurts: lead + retoucher 50.2 vs lead + composer + retoucher 48.9 (Table 3 rows 4 and 5).
- At matched iteration budgets, retoucher-only is better than the full system after one round (46.9/47.0 vs 46.4) and after two rounds (48.5 vs 48.1); the full system leads only after round three (Table 9). On the easy track after three rounds, the full system is worse (73.7 vs 75.1 for both retoucher-only rows).
- By capability (three rounds, full vs lead + retoucher + perceptor), the full system gains on attribute (+6.8) but loses on text (−3.8) and knowledge (−1.2), and gains only +0.8 on layout, even though Table 10 shows the lead editor sends 93.7% of layout decisions to the composer (ERNIE). The routing distribution varies by dimension, but this variation does not translate into gains on the dimensions where the composer is used.
- No result in the paper reports repeated runs, seeds, or confidence intervals, so a 0.7-point difference cannot carry the novelty claim. What the evidence supports is "strong local text repair + multiple proposals + verifier selection + incumbent retention", not error-scale routing.
2. The comparisons with baselines and proprietary models are not made under the same information or compute conditions, so the headline claims cannot be read at face value.
- Privileged access to the evaluation questions. On BizGenEval and IGenBench the rubric is taken directly from the evaluation checklists (l. 199–201; Appendix A). The composer prompt receives the failed hard questions and the benchmark's capability label (
domain=… dimension=…, Appendix J.4, l. 1805–1812), and the retoucher is instructed to answer the failed checklist questions correctly and write those answers into the image (Appendix J.5, l. 1885–1901). For IGenBench, 1,529 of the 5,259 questions are expert-informed rather than prompt-derived (Appendix B), so this channel adds specification beyond the prompt. None of the T2I, proprietary, or agentic baselines receive this information. Appendix I (l. 1645–1646) itself notes that monotonic admission concerns the internal verifier score and does not guarantee improvement under independent evaluation. - Gains track evaluator alignment. The Qwen verifier inflates its own best-of-6 picks by +9.1 points relative to Gemini (Table 15), and replacing it with the official evaluator lifts BizGenEval from 51.1 to 64.4 (Table 17). This is consistent with a substantial part of the improvement coming from fitting the scoring model, and it means the oracle numbers should not be presented as "headroom" of the method.
- Weak baselines and missing controls. Most agentic baselines score below the frozen generator they wrap (BizGenEval ERNIE: frozen 28.0 vs PE 24.2, GenEvolve 22.9, PromptEnhancer-7B/32B 12.0/14.3; Infographic-V2: frozen 41.1 vs PromptEnhancer 9.8/14.0; IGenBench Infographic-V2: frozen 0.72 vs Gen-Searcher 0.60, PromptEnhancer-32B 0.52), which suggests they are mis-configured or ill-suited to these generators. The most informative baselines are missing: best-of-N with the same Qwen verifier and the same number of generator calls (up to 6 + 3×4 = 18), and Kimi-K3 prompt rewriting (with search) + best-of-N. The planners of the baselines are 3B–32B models, whereas STRIDE uses Kimi-K3 and a 397B verifier; the Params column nevertheless lists STRIDE as "8B" (Table 1/2) while listing PE as "8B-3B". No wall-clock, token, or API-call cost is reported anywhere.
- Selective comparison with proprietary systems. The abstract states that STRIDE outperforms Nano-Banana-2.0 and Seedream-5.0-Pro on BizGenEval and matches GPT-Image-2.0 on IGenBench. In the same tables, Nano-Banana-Pro is clearly stronger (BizGenEval 76.7 vs 69.5, knowledge 82.6 vs 54.0; IGenBench 0.90/0.49 vs 0.84/0.38), and the GPT-Image-2.0 match holds only for I-ACC (Q-ACC 0.88 vs 0.84).
- WISE does not test the claimed mechanism. In Table 9, lead + composer with a single round already raises WISE from 0.58 to 0.89; the full system reaches 0.92. Almost all of the WISE gain therefore comes from Kimi-K3 making the implicit entity explicit in the prompt. Appendix A's "prompt parser" is in fact a prompt enhancer that may rewrite the prompt entirely and is told not to reveal search-engine use (l. 725–755). WISE evidence therefore speaks to LLM prompt enhancement, not to scale-aware evolutionary refinement.
Questions for authors
Clarifications that could change my assessment
- Please report mean ± std (or CIs) over at least 3 independent runs for Table 3 rows 4, 6 and 9 and for the BizGenEval/IGenBench main results. Is the +0.7 difference between lead + retoucher + perceptor (50.4) and the full system (51.1) statistically reliable?
- How does the retoucher obtain block indices in the "no perceptor" configurations (Table 3 rows 4–5)? The method section says it first queries the perceptor for OCR blocks (l. 256–260).
- Can you re-run BizGenEval and IGenBench with rubrics generated by your own prompt parser from the prompt only (as on WISE), with no access to the released checklists, the failed evaluation questions, or the
domain/dimensionlabels? How much of the gain remains? - Please add (a) best-of-N with the same Qwen verifier and the same number of generator calls as STRIDE, and (b) Kimi-K3 prompt rewriting (+ search) followed by best-of-N, on all three benchmarks, together with per-prompt wall-clock time, number of LLM/VLM calls and tokens.
- What is
{rules_prior}in the lead-editor prompt (Appendix J.3)? Were these priors derived from runs on BizGenEval/IGenBench test items? - Human study: how many annotators, were they blind to condition (pasted text may be visually identifiable), what is the inter-annotator agreement, and what are the paired CIs of the reported gains with 25 prompts per cell?
- Why do most agentic baselines underperform the frozen generator (Table 1/2)? Were their prompts or settings adapted to ERNIE-Image and Infographic-V2?
Suggestions (not required for my assessment)
- Report a visual-quality or artifact check for retoucher outputs (e.g., a holistic human preference test, or detection of pasted patches), since the rubric-based study cannot reveal such artifacts.
- Revise the abstract to mention Nano-Banana-Pro and to state that the GPT-Image-2.0 match is on I-ACC only, and report the full agent stack in the Params column.
- Explain how the verifier (Appendix F.1, chosen on all 400 BizGenEval items), the perceptor (20 BizGenEval images), and the thresholds c=0.7, r=0.8 were selected, ideally on held-out data.
14 comments
Response to Reviewer eTcN (2/4)
Q3. Can you re-run BizGenEval and IGenBench with rubrics generated by your own prompt parser from the prompt only (as on WISE), with no access to the released checklists, the failed evaluation questions, or the domain/dimension labels? How much of the gain remains?
Prompt-only setting. Yes. We reran both benchmarks using our Kimi-K3 prompt parser (paper Appendix A) to construct rubrics from the original prompt alone. No released checklist, failed official evaluation question, or domain/dimension label enters the refinement loop. Feedback comes exclusively from verification against the generated rubric. The official evaluators score only the final images.
How much gain remains. As shown in Table E3, prompt-only Stride retains 89.2% and 92.6% of the BizGenEval Hard gains for ERNIE-Image and Infographic-V2, respectively. On IGenBench, it retains 64.3% / 50.0% of the Q-ACC / I-ACC gains for ERNIE-Image and 75.0% / 76.2% for Infographic-V2. These results show that substantial improvements persist without access to the released evaluation questions.
Table E3. Gain retained with prompt-only rubrics. Retained = (Prompt-only - Frozen) / (Checklist - Frozen).
| Benchmark | Generator | Frozen | Checklist Stride | Prompt-only Stride | Retained |
|---|---|---|---|---|---|
| BizGenEval (Hard) | ERNIE-Image | 28.0 | 51.1 | 48.6 | 89.2% |
| BizGenEval (Hard) | Infographic-V2 | 41.1 | 69.5 | 67.4 | 92.6% |
| IGenBench (Q-ACC / I-ACC) | ERNIE-Image | 0.41 / 0.01 | 0.55 / 0.03 | 0.50 / 0.02 | 64.3% / 50.0% |
| IGenBench (Q-ACC / I-ACC) | Infographic-V2 | 0.72 / 0.17 | 0.84 / 0.38 | 0.81 / 0.33 | 75.0% / 76.2% |
Retained percentages are calculated from the reported scores. IGenBench values follow the Q-ACC / I-ACC order.
Q4. Please add (a) best-of-N with the same Qwen verifier and the same number of generator calls as STRIDE, and (b) Kimi-K3 prompt rewriting (+ search) followed by best-of-N, on all three benchmarks, together with per-prompt wall-clock time, number of LLM/VLM calls and tokens.
Generation-budget controls. We added both requested controls on all three benchmarks with ERNIE-Image. Both use the same Qwen verifier and benchmark-specific rubric as Stride. We set for BizGenEval and for IGenBench and WISE, rounding up Stride’s average T2I calls of 11.2, 12.6 and 12.9. The controls therefore receive slightly larger generation budgets. Table E4 reports per-prompt wall-clock time, LLM/VLM calls and tokens.
Results. As shown in Table E4, Compared with Best-of-, Stride improves BizGenEval Hard by 14.9 points, IGenBench Q-ACC by 0.11, and WISE WiScore by 0.25. Against Kimi-K3 rewriting with search followed by Best-of-, the gains are 18.0, 0.10 and 0.06, respectively. These improvements persist at approximately matched generation budgets, with additional LLM/VLM, token and latency overhead reported in the table.
Table E4. Generator-budget comparisons on all three benchmarks with ERNIE-Image. Scores are BizGenEval Hard, IGenBench Q-ACC, and WISE WiScore.
| Benchmark | Method | Score | T2I calls | LLM/VLM calls | Tokens | Time (min) |
|---|---|---|---|---|---|---|
| BizGenEval | Best-of-12 | 36.2 | 12 | 24 | 29,240 | 2.05 |
| BizGenEval | Kimi-K3 rewrite + search + Best-of-12 | 33.1 | 12 | 25 | 34,548 | 5.22 |
| BizGenEval | Stride | 51.1 | 11.2 | 27.7 | 118,983 | 16.65 |
| IGenBench | Best-of- | 0.44 | 13 | 26 | 32,479 | 2.67 |
| IGenBench | Kimi-K3 rewrite + search + Best-of- | 0.45 | 13 | 27 | 37,665 | 6.06 |
| IGenBench | Stride | 0.55 | 12.6 | 29.3 | 134,428 | 18.21 |
| WISE | Best-of- | 0.67 | 13 | 26 | 20,773 | 1.43 |
| WISE | Kimi-K3 rewrite + search + Best-of- | 0.86 | 13 | 27 | 22,496 | 4.20 |
| WISE | Stride (, ) | 0.92 | 12.9 | 31.8 | 48,461 | 16.08 |
All resources are averaged per prompt. Generation budgets are approximately matched by rounding up Stride’s average T2I calls. WISE uses one initial sample and up to three refinement rounds.
Q5. What is {rules_prior} in the lead-editor prompt (Appendix J.3)? Were these priors derived from runs on BizGenEval/IGenBench test items?
{rules_prior} supplies two fixed collapse heuristics to the lead editor (Appendix J.3). They suggest global resampling when the image has either near-total constraint failure or extensive garbled text. The same heuristics are used for every item and benchmark, and the lead editor may override them based on the current evidence:
-
IF easy=0 AND hard=0 → likely structure_collapsed → redraw -
IF garbled_rate>0.70 AND acc<=0.10 → likely text_collapsed → redraw
These heuristics were specified from general experience before the benchmark experiments and were not derived from or tuned on BizGenEval or IGenBench test runs.
Response to Reviewer eTcN (3/4)
Q6. Human study: how many annotators, were they blind to condition (pasted text may be visually identifiable), what is the inter-annotator agreement, and what are the paired CIs of the reported gains with 25 prompts per cell?
Three annotators scored all 200 images. They were blind to condition: images were shuffled and condition labels hidden. Each annotator answered the same atomic rubric questions used by the benchmark (the 10 hard questions for BizGenEval; all released questions for IGenBench). The human score for each image is the arithmetic mean of the three annotators’ scores.
Table E5 reports mean pairwise agreement on the original binary rubric judgments, ranging from 89% to 93%. The 95% CIs reports the 25 prompts within each cell, retaining both images and all three annotators’ scores for each sampled prompt. Human denotes Stride minus the frozen generator. All four paired intervals exclude zero, supporting positive human-rated improvements with both generators on both benchmarks.
Table E5. Inter-annotator agreement and prompt-paired 95% CIs (25 prompts per cell).
| Benchmark | Generator | Pairwise agreement | Human | 95% CI |
|---|---|---|---|---|
| BizGenEval (Hard) | ERNIE-Image | 0.92 | +20.3 | [16.2, 24.5] |
| BizGenEval (Hard) | Infographic-V2 | 0.93 | +29.8 | [25.9, 34.0] |
| IGenBench (Q-ACC) | ERNIE-Image | 0.89 | +0.16 | [0.09, 0.20] |
| IGenBench (Q-ACC) | Infographic-V2 | 0.92 | +0.12 | [0.07, 0.18] |
Q7. Why do most agentic baselines underperform the frozen generator (Table 1/2)? Were their prompts or settings adapted to ERNIE-Image and Infographic-V2?
Baseline settings. We used the released prompts and default settings of each baseline, without generator-specific adaptation to ERNIE-Image or Infographic-V2.
Why the baselines underperform here. The original infographic prompts already specify exact text, numerical values and layout requirements. The baselines introduce prompt rewriting or additional planning, which can alter these specifications and reduce constraint satisfaction. Our controlled comparison shows that rewriting can indeed hurt performance: with the same ERNIE-Image generator, Qwen verifier and 12-sample budget, Kimi-K3 rewriting with search scores 33.1 on BizGenEval Hard, compared with 36.2 for direct Best-of-12 (Table E4). This result is consistent with specification drift as an explanation for the declines, although it does not establish the failure cause of each individual baseline.
Limited correction of text errors. Text constitutes a substantial part of infographic content, including titles, labels and numerical annotations. The prompt-rewriting and resampling baselines evaluated here lack a dedicated text-repair step. They rely on the generator to render the text again, which can reproduce existing errors and introduce new ones in previously correct regions. Additional generation therefore consumes compute without reliably resolving the text errors that limit constraint-satisfaction scores.
Q8. Report a visual-quality or artifact check for retoucher outputs (e.g., a holistic human preference test, or detection of pasted patches), since the rubric-based study cannot reveal such artifacts.
Visual-quality check. We compared 25 final outputs whose last admitted update was a retoucher edit with their immediate pre-edit parents (50 images). Three annotators judged overall visual preference and disruptive patch artifacts on randomized A/B pairs with method labels hidden. Results use case-level majority judgments. Unclear includes no clear preference or no majority. Artifact rates use 25 images per condition.
As shown in Table E6, edited outputs were preferred in 72% of cases, versus 12% for parents, with 16% unclear. Disruptive patch artifacts were observed in 4% of images in each condition. These results support a visual-quality benefit from accepted retoucher edits, with a low observed incidence of disruptive patch artifacts.
Table E6. Visual preference and patch artifacts (25 paired cases).
| Measure | Percentage |
|---|---|
| Preference: edited / parent / unclear | 72% / 12% / 16% |
| Disruptive patch artifacts: edited / parent | 4% / 4% |
Q9. Revise the abstract to mention Nano-Banana-Pro and to state that the GPT-Image-2.0 match is on I-ACC only, and report the full agent stack in the Params column.
We will revise the abstract to mention Nano-Banana-Pro and clarify that STRIDE matches GPT-Image-2.0 on I-ACC only. We will also update the Params column to report the full agent stack.
Response to Reviewer eTcN (1/4)
We thank the reviewer for the close reading of our ablations and evaluation protocols, and for recognizing the evidence from incumbent retention and human evaluation. Below, we address the incremental contribution of scale-aware refinement, the information and compute conditions of the comparisons, and the requested clarifications and checks.
Q1. Please report mean ± std (or CIs) over at least 3 independent runs for Table 3 rows 4, 6 and 9 and for the BizGenEval/IGenBench main results. Is the +0.7 difference between lead + retoucher + perceptor (50.4) and the full system (51.1) statistically reliable?Three-run results.
Table E1reports the scores averaged over three independent runs and their 95% CIs forpaper Table 3 rows 4, 6 and 9, and for the BizGenEval and IGenBench main results with both generators. The three configurations retain the ordering 50.2, 50.4 and 51.1, with Full Stride achieving the highest average Hard score.Table E1. Results under three independent runs.Table 3 row 4: Lead + RetoucherTable 3 row 6: Lead + Retoucher + PerceptorTable 3 row 9 / main: Stride (ERNIE-Image), BizGenEvalInterpreting the 0.7-point gap. Full Stride (51.1) and retoucher only with grounding (50.4) differ by 0.7 points. Taken alone, this gap is smaller than the approximately 3.7-point half-width of the individual-score intervals. However, it forms part of a consistent pattern across all router variants (Please see
Table E2). Adding the composer and the scale-aware router to the retoucher raises the score from 50.2 to 51.1 (+0.9). Removing the grounding from that full configuration lowers it from 51.1 to 48.9 (−2.2), placing it below retoucher only. Keeping the grounding while fixing execution to the retoucher gives 50.4, which sits between the two. The rule-based, random and fixed routers (46.5 to 47.7) all sit further below. These shifts consistently support the value of scale-aware routing when paired with grounding. We therefore interpret the +0.7 as a modest positive average increment within this broader pattern, and do not claim statistical significance for that increment alone.Table E2. Ablation of routing and history controls (BizGenEval Hard, ERNIE-Image).Table 3 row 3)row 4)row 6)row 5)row 9)Q2. How does the retoucher obtain block indices in the "no perceptor" configurations (Table 3 rows 4–5)? The method section says it first queries the perceptor for OCR blocks (l. 256–260).The block indices come from the retoucher’s branch-local OCR queries. Each branch independently queries the perceptor module for indexed OCR blocks of the current champion, then binds a selected block index to a corrected string. In
paper Table 3 rows 4–5, “no perceptor” disables only the controller-facing grounding Et: the lead editor receives no OCR view, but the retoucher’s OCR queries remain active. A retoucher round therefore makes N=4 independent OCR queries, one per branch, in both configurations. The method section describes these branch-local queries, which are distinct from the grounding supplied to the lead editor.