acceptodds
Under review as a conference paper at ICLR 2027

Can Image Generation Benefit Visual Reasoning?

Abstract

If language models can "think in words", can multimodal models "think in images"? Recent advances in visual generation raise this possibility, but whether generated images can reliably support multimodal reasoning remains largely untested. We conduct a systematic study of visual generation for multimodal reasoning, spanning both modular VLM-Generator pipelines and unified models across diverse benchmarks. We find that, in common settings, incorporating visual generation does not improve over text-only reasoning, with gains confined to limited conditions. We find two main limitations. First, current models struggle to use generated images as effective intermediate reasoning steps: visual information that appears useful does not always translate into better downstream reasoning. Second, test-time scaling mainly benefits text reasoning, while scaling visual generation brings little improvement. Overall, current VLMs remain much stronger at reasoning in text than reasoning with generated images as intermediate evidence.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Visual Geneartion, Multimodal Reasoning, Image Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QU62OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies “Reasoning with Imagination” through a two-stage pipeline: a VLM first proposes an image-generation hint, then answers using the original image, the hint, and the generated image. It evaluates both external-generator pipelines and the unified model BAGEL. Across nine benchmarks and ten task settings, generation augmentation generally reduces accuracy, with notable gains on RoboSpatial-Compatibility and on MathVista with BAGEL. Further analyses examine oracle hints and images, text- and vision-side test-time scaling. The contribution is primarily a cross-model empirical evaluation rather than a new training method. The results establish limitations of the tested pipelines, but the explanations of failure mechanisms and scaling behavior require stronger evidence.

Critical strengths
  1. The negative results have broad coverage and retain important counterexamples. The tested pairings of four VLM configurations and four generators, together with BAGEL, show broadly similar trends; the conclusion does not depend on a single weak generator. For example, GLM with Qwen-Image-Edit loses 31.3 percentage points on VSTAR. At the same time, Table 2 explicitly reports BAGEL gains of 8.1 points on MathVista and 25.7 points on RoboSpatial-Compatibility. This supports the qualified claim that most current configurations do not benefit, rather than the stronger claim that generation is intrinsically unhelpful.
  2. The diagnostics go beyond the main benchmark tables. Table 3a includes controlled prompts and ground-truth helper images, while Table 3b examines CoT. This work also includes rule-based scoring and a 100-instance human audit. It adopts the better-performing multi-turn implementation and explicitly leaves the caching gap unexplained, rather than attributing it to an inherent defect in caching. These additions mitigate concerns that evaluation was never checked or that only one implementation was considered.
Critical weaknesses
  1. Demonstrating degradation does not identify its principal cause. The final context contains the original image, hint, and generated image (Section 2.1), but there is no control that preserves the hint and number of conversational turns while removing only the generated image, nor a repeated-original-image or genuine-crop control. Consequently, the effects of misleading hints, additional context, and generation distortion remain entangled. Table 3a provides useful interventions but does not fully separate these factors. Section 5.3 annotates only 10–20 failures per benchmark. The 66%, 28%, 6% breakdown should therefore be treated as descriptive sample statistics. Similarly, low invocation rates alone do not establish that models recognize generation utility, and a correct answer alongside an imprecise image does not establish that the helper image was unused (Figure 4 and Sections 5.1–5.2).
  2. The scaling evidence is narrow, and a key numerical discrepancy needs correction. Section 4.2 compares CoT with one or two generation calls, without reporting actual token counts, latency, or compute budgets, or clarifying how the second call is combined with the first. This supports a conclusion about the tested configurations, not a general law that “Text scales; vision does not.” More concretely, Qwen3-VL 8B with FLUX.1 changes ERQA accuracy from 44.3 to 42.8, a decrease of 1.5 points (Tables 2 and 3b), whereas Figure 3a reports a decrease of 5.8 points. This does not reverse the direction of degradation, but it directly affects the reported magnitude and its comparison with text-side scaling.
  3. The oracle experiments and small differences are not sufficiently reproducible. Appendices A/B provide standard decoding settings and the main prompts, but not a task-by-task mapping of oracle experiments to data sources, sample counts, the actual hard-coded prompts, or the construction of ground-truth helper images. Neither the main results nor the intervention results include paired confidence intervals or variation across seeds. The controlled-prompt gains in Table 3a are only 0.7 and 2.2 points, so point estimates alone do not establish stable improvements.
Questions for authors
  1. Could you compare hint-only, repeated-original-image/genuine-crop, and generated-image conditions on the same instances with matched hints and conversational turns, separately reporting newly introduced and corrected errors? This would test the conversational-context or randomness explanations proposed in Section 5.2.
4 comments

We thank the reviewer for the careful assessment and constructive feedback. We appreciate the recognition of the practical relevance and breadth of our empirical study. Below, we address the concerns with additional experiments and clarifications, using W# for weaknesses and Q# for explicit questions.

TL;DR: Matched controls identify additional harm from generated helpers on VSTAR; the 66%/28%/6% breakdown describes the annotated failure sample.

W1/Q1: Demonstrating degradation does not identify its principal cause. / Could you compare hint-only, repeated-original-image/genuine-crop, and generated-image conditions on the same instances with matched hints and conversational turns, separately reporting newly introduced and corrected errors? This would test the conversational-context or randomness explanations proposed in Section 5.2.

Thank you for highlighting this distinction. We added matched controls using Qwen3-VL-8B + FLUX.1-Kontext, keeping the hint, conversational turns, and final-answer instructions fixed. We use greedy answer decoding to avoid answer-sampling variability.

BenchmarkNo helperRepeated originalGenerated helper
VSTAR60.2158.6449.74
RoboSpatial-Compatibility59.0552.3860.95
ChartQAPro39.8639.7939.79

On VSTAR, generated helpers reduce accuracy by 10.47 pp relative to no helper and 8.90 pp relative to repeating the original image; both paired 95% confidence intervals exclude zero. Thus, neither the planning/conversation step nor merely adding another image fully explains the loss.

Faithful crop/zoom controls. We apply non-generative cropping and resizing only when explicitly requested by the hint, comparing the same eligible instances.

BenchmarkEligible instancesNo helperFaithful crop/zoomGenerated helper
VSTAR4369.7772.0951.16
RoboSpatial-Compatibility2166.6747.6252.38
ChartQAPro250.0050.0050.00

On VSTAR, faithful cropping outperforms generation by 20.93 pp (95% CI: [4.65, 37.21]). Both conditions follow the same hint and provide a helper image, identifying an additional loss from generation rather than faithful execution of the requested crop/zoom operation.

Error transitions. Each row compares generated helpers with the listed control on the same instances. Introduced errors are correct-to-incorrect transitions; corrected errors are incorrect-to-correct transitions. No helper is the hint-only condition, and crop/zoom rows include only instances with explicit crop/zoom requests.

BenchmarkControlNIntroduced errorsCorrected errors
VSTARNo helper191288
VSTARRepeated original191269
VSTARFaithful crop/zoom43123
RoboSpatial-CompatibilityNo helper10568
RoboSpatial-CompatibilityRepeated original105312
RoboSpatial-CompatibilityFaithful crop/zoom2112
ChartQAProNo helper1,4252827
ChartQAProRepeated original1,4252222
ChartQAProFaithful crop/zoom200

Scope of the qualitative analysis. We agree that the 66%/28%/6% breakdown describes the annotated failure sample, not population-level causal proportions. We will clarify this and revise statements suggesting that low invocation rates establish awareness of generation utility, or that a correct answer with an imprecise helper establishes that the helper was unused. We will update the PDF to reflect these clarifications.

Together, the matched controls demonstrate additional harm from generated helpers on VSTAR beyond the shared pipeline, while the qualitative annotations remain descriptive evidence.

W2: The scaling evidence is narrow, and a key numerical discrepancy needs correction.

TL;DR: Extending scaling to 10 calls shows task-dependent effects: ChartQAPro and MathVista stay below baseline, while ERQA improves.

Thank you for highlighting the scope of the scaling evidence and the numerical inconsistency. We extended the scaling experiment with Qwen3-VL-8B + FLUX.1-Kontext to 1, 2, 3, 4, 5, 8, and 10 generation calls.

Experimental setup. One hint is generated and kept fixed. Each generation call edits the previous output, starting from the original image. At each checkpoint, the VLM answers using the original image(s) and only the latest helper. We reran all conditions, including the baseline and 1/2-call settings, under this explicitly specified protocol.

Resource budget. Each instance uses one hint-generation call (up to 256 tokens), n image-generation calls to reach round n, and one final-answer call (up to 1,024 tokens) per evaluated checkpoint. Each FLUX call uses 24 denoising steps and takes approximately 47–49 seconds on an L40, or about 96 seconds for two calls, excluding planning and answering.

Generation callsChartQAProERQAMathVista
0: baseline39.1642.5067.70
137.6142.5065.90
237.8944.0066.60
337.8243.2565.00
438.1143.7566.30
538.3942.7566.40
837.8244.0066.90
1037.4045.5066.30

Conclusion. We narrow our scaling conclusion: failure rates do not increase consistently with additional generation rounds. Instead, ChartQAPro and MathVista remain below their no-generation baselines across all evaluated rounds, while ERQA shows improvement. We will update the manuscript to reflect these task-dependent scaling outcomes.

Numerical correction. For Qwen3-VL-8B + FLUX.1-Kontext without thinking, the original ERQA result is 44.3% → 42.8%, a change of −1.5 pp. The −5.8 pp value in Figure 3a and Appendix C.6 was a reporting error. The updated PDF will report −1.5 pp consistently in both locations.

W3: The oracle experiments and small differences are not sufficiently reproducible.

TL;DR: Table 3a separates fixed-prompt generation from direct reference-image input; small gains are point estimates, complemented by paired confidence intervals from our new matched controls.

Thank you for the suggestion. Table 3a evaluates Qwen3-VL-8B with Nano-Banana under two targeted interventions:

InterventionTasksHelper construction
Controlled promptJigsaw, SpatialReplace the model-generated hint with a task-specific fixed instruction, retaining the generation step.
Oracle visualJigsaw, Maze, Multi-hopSupply reference helper images directly, bypassing generation. Reference information is benchmark-provided or constructed for the task, as described in Appendix A.

The benchmark sources cited for these controlled tasks are SAT, ROVER, Zebra-CoT, EMMA, and RealUnify, with Uni-MMMU additionally cited for reference resources. We will clarify the distinction between reference-image construction and fixed-prompt generation in the appendix.

The +0.7 and +2.2 pp controlled-prompt gains are reported as point estimates suggesting potential headroom. Our additional matched-control experiments provide statistical support for the degradation finding: on VSTAR, adding the generated helper reduces accuracy by 10.47 pp, with a paired 95% CI of [−16.75, −4.71], while holding the hint and conversational protocol fixed.

Reviewer_QU62OpenReview ↗

Response to authors' rebuttal

Thanks for the author’s response. These additional experiments have addressed some of my questions. Overall, however, this version of the manuscript still requires further polishing: the key experiments need to be refined and presented in the main text. I will keep my original score.

Reviewer_BjBUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a framework, Reasoning with Imagination, which investigates reasoning with generated images in two settings: modular and unified.

In the modular setting, a VLM plans a hint, an image generator generates an image based on the hint, and the VLM then answers the question with the help of the generated image.

In the unified setting, the above pipeline is implemented within a single unified model.

They find that existing image generation methods and unified models fail to generate precise, task-relevant visual evidence for downstream reasoning.

Critical strengths
  1. This paper systematically investigates the influence of generated images on multimodal reasoning, finding that existing image generation methods fail to provide beneficial evidence.
Critical weaknesses
  1. The proposed Reasoning with Imagination framework has been explored in prior works. On the one hand, some works use draft images to assist reasoning (e.g., Zebra-CoT [1]). On the other hand, similar frameworks (planning → generation → answering) and the proposed unified setting have also been employed and analyzed in works on unified models (e.g., RealUnify [2]).

  2. The experiments in the unified setting are limited, with only BAGEL being evaluated.

References:

[1] Zebra-CoT: A Dataset for Interleaved Vision Language Reasoning. ICLR 2026.

[2] RealUnify: Do Unified Models Truly Benefit from Unification? A Comprehensive Benchmark. CVPR 2026.

Questions for authors

I have concerns about the example in Figure 1(a). It seems that the pipeline is “Question + Image → Hint → Generated Image → Answer”. I think the pipeline lacks a necessary prompt to tell the model which image it should refer to when answering the question. Otherwise, the model may be confused about which image to base its answer on, naturally leading to failure.

4 comments

We thank the reviewer for the careful assessment and constructive feedback. We appreciate the recognition of the practical relevance and breadth of our empirical study. Below, we address the concerns with additional experiments and clarifications, using W# for weaknesses and Q# for explicit questions.

TL;DR: Our contribution lies in empirically diagnosing when current image-generation models harm visual reasoning, supported by new matched controls, rather than in proposing the planning → generation → answering pipeline.

W1: The proposed Reasoning with Imagination framework has been explored in prior works. On the one hand, some works use draft images to assist reasoning (e.g., Zebra-CoT [1]). On the other hand, similar frameworks (planning → generation → answering) and the proposed unified setting have also been employed and analyzed in works on unified models (e.g., RealUnify [2]).

Thank you for highlighting these connections. Our contribution is the systematic empirical finding that current image-generation models do not reliably improve visual reasoning and often reduce accuracy across the modular and unified configurations we evaluate. Our matched controls further show that generated helpers can introduce additional harm beyond the shared hint and conversational protocol. The planning → generation → answering pipeline provides the evaluation framework for this diagnosis; it is not itself our novelty claim.

To strengthen this empirical diagnosis, we added matched controls using Qwen3-VL-8B + FLUX.1-Kontext that hold the hint and conversation fixed.

BenchmarkNo helperRepeated originalGenerated helper
VSTAR60.2158.6449.74
RoboSpatial-Compatibility59.0552.3860.95
ChartQAPro39.8639.7939.79
BenchmarkEligible instancesNo helperFaithful crop/zoomGenerated helper
VSTAR4369.7772.0951.16
RoboSpatial-Compatibility2166.6747.6252.38
ChartQAPro250.0050.0050.00

On VSTAR, generated helpers reduce accuracy by 10.47 pp relative to no helper and by 8.90 pp relative to repeating the original image. On the same crop/zoom-requested instances, faithful cropping outperforms generation by 20.93 pp. These results identify an additional performance loss associated with generated helpers, beyond the shared pipeline.

Zebra-CoT develops interleaved reasoning data and trained models, while RealUnify evaluates understanding–generation synergy in unified models. Our study complements them through broad evaluation across modular and unified configurations, identifying where generation helps or hurts. Our new matched controls further show that generated helpers can introduce additional harm beyond the hint/conversation protocol itself.

We will make this distinction explicit in the revised paper.

W2: The experiments in the unified setting are limited, with only BAGEL being evaluated.

TL;DR: We added Janus-Pro-7B on 5,906 paired instances across ten settings; generation reduces accuracy in eight of ten.

Thank you for encouraging broader evaluation of unified models. We expanded the evaluation to Janus-Pro-7B, covering 5,906 paired instances across ten task settings.

BenchmarkWithout generationWith generationΔ (pp)
VSTAR47.1242.93−4.19
ChartQAPro16.149.75−6.39
MathVista37.4028.00−9.40
ERQA34.5020.25−14.25
VisuLogic18.3010.70−7.60
VisualPuzzles16.1017.38+1.28
Rover10.7612.79+2.03
SAT46.6744.67−2.00
RoboSpatial-Compatibility34.2924.76−9.52
RoboSpatial-Configuration71.5460.16−11.38

Generation reduces accuracy in eight of ten settings, extending the observed degradation beyond BAGEL while retaining positive cases.

Q1: I have concerns about the example in Figure 1(a). It seems that the pipeline is “Question + Image → Hint → Generated Image → Answer”. I think the pipeline lacks a necessary prompt to tell the model which image it should refer to when answering the question. Otherwise, the model may be confused about which image to base its answer on, naturally leading to failure.

TL;DR: The answer prompt already identifies both original and helper images; we will update Figure 1(a) to show this explicitly.

Thank you for pointing out this ambiguity. Figure 1(a) omits the final-answer instruction and does not clearly show that the original image(s) remain available alongside the generated helper. The actual prompt explicitly states:

“You will see (A) the ORIGINAL input image(s) and (B) an OPTIONAL GENERATED helper image. Answer using evidence from all provided image(s).”

Thus, the model is instructed to use both sources rather than answer from the generated image alone. We will update Figure 1(a) to show the original-image input to the answering stage and include this instruction.

Thank you for the detailed response. My concerns still remain. Although the paper focuses on empirical findings, it evaluates only two unified models (one in the original submission and one added during rebuttal), which is insufficient to draw general conclusions about unified multimodal models. Besides, the paper still needs improvements in presentation. Overall I keep my rating.

Reviewer_onWyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether generated helper images improve visual question answering with existing models. Its pipeline first asks a VLM to propose a short generation instruction, generates a helper image, and then answers using both the original and generated images. It evaluates modular VLM–generator combinations and the unified BAGEL model across several task families. Generation usually decreases accuracy, with notable exceptions such as RoboSpatial-Compatibility and BAGEL on MathVista. Oracle/controlled interventions, thinking prompts, selective invocation, and limited test-time scaling experiments investigate possible explanations. The contribution is an empirical evaluation and diagnosis of generation-augmented reasoning, rather than a new training algorithm. The “text-only” baseline still receives the original images; only its intermediate reasoning is restricted to text.

Critical strengths
  1. A practically relevant negative result with breadth across model combinations. The degradation is not confined to a single generator or backbone. For example, Gemini with Nano-Banana drops from 88.0 to 71.7 on VSTAR, while BAGEL drops from 67.0 to 47.6 (Table 1, p. 4). These observations caution against assuming that a plausible generated image constitutes useful reasoning evidence. The negative nature of the result, and the absence of a new algorithm, are not weaknesses in themselves.
  2. Useful positive boundary cases and diagnostic interventions. The paper reports rather than obscures substantial gains on RoboSpatial-Compatibility, including BAGEL's 46.7 to 72.4 improvement (Table 2, p. 5). On the maze task, accuracy is 44.7 without generation, 40.0 with standard augmentation, and 49.8 with an oracle helper image (Table 3a, p. 6). These findings help distinguish the potential value of accurate intermediate visual evidence from the limitations of the evaluated pipelines. The judge checks in Appendix C.2 are also a useful evaluation safeguard.
Critical weaknesses
  1. The observed pipeline-level degradation is better established than its causal explanation or robustness to the prompting protocol. The principal comparison changes both the conversation structure/planning step and the availability of a generated image. A matched two-turn condition retaining the same hint but omitting the helper image would help distinguish hint-induced bias from the image's incremental effect. For examples whose requested operation is cropping or zooming, a faithful non-generative transformation would further distinguish information selection from generative distortion. Appendix B's principal hint prompts request a helper image but do not explicitly require preservation of task-relevant facts. Testing such a preservation instruction would establish whether the negative trend is robust to a straightforward protocol change. The oracle and selective-invocation analyses are valuable, but do not replace these controls. This concern limits the explanatory claims; it does not invalidate the reported end-to-end comparison.
  2. The scaling conclusion needs a narrower interpretation and more consistent reporting. Section 4.2 compares one versus two generation calls and enabling textual chain-of-thought on three tasks. Without a broader budget sweep or matched resource measurements, this supports a local result for the tested settings rather than a general conclusion that visual reasoning does not scale. There is also an unresolved numerical discrepancy: Appendix C.6 labels the FLUX.1 ERQA augmentation effect as −5.8 points, whereas Tables 2 and 3b give 44.3 to 42.8, i.e. −1.5 points; Figure 3a also displays −5.8 for the one-generation ERQA condition. The intended configuration and corrected values should be made explicit because they affect a central diagnostic figure.
Questions for authors
  1. On representative tasks, how does the same generated hint followed by a second answer turn without a helper image perform? For crop/zoom requests, how does a faithful non-generative transformation compare? Would the main negative trend survive a prompt explicitly preserving the evidence required to answer the original question?
  2. How are the second hint and second helper image produced and incorporated in the two-generation condition: independently, or conditioned on the first result? Please report the corresponding token/call budgets and at least a practical resource measure, or explicitly restrict the scaling claim to the evaluated one-to-two-call comparison.
  3. Which model/configuration produces the ERQA −5.8 value in Figure 3a and Appendix C.6, and how does it relate to the −1.5 value in Tables 2 and 3b? Please reconcile these results.
5 comments
AuthorsOpenReview ↗

W1/Q1 comment

We thank the reviewer for the careful assessment and constructive feedback. We appreciate the recognition of the practical relevance and breadth of our empirical study. Below, we address the concerns with additional experiments and clarifications, using W# for weaknesses and Q# for explicit questions.

TL;DR: Matched controls show additional harm from generated helpers on VSTAR, including with evidence-preserving prompts and relative to faithful crop/zoom.

W1/Q1: The observed pipeline-level degradation is better established than its causal explanation or robustness to the prompting protocol. / On representative tasks, how does the same generated hint followed by a second answer turn without a helper image perform? For crop/zoom requests, how does a faithful non-generative transformation compare? Would the main negative trend survive a prompt explicitly preserving the evidence required to answer the original question?

Thank you for this suggestion. We agree that the original end-to-end comparison does not isolate the cause of degradation. We therefore added matched controls on VSTAR, RoboSpatial-Compatibility, and ChartQAPro, holding the hint, conversational turns, and final-answer instructions fixed.

BenchmarkNo helperRepeated originalGenerated helper
VSTAR60.2158.6449.74
RoboSpatial-Compatibility59.0552.3860.95
ChartQAPro39.8639.7939.79

These controls isolate the generated helper’s contribution from the pipeline. Compared with no helper, the generated-image condition retains the same hint and conversational history but reduces VSTAR accuracy by 10.47 pp. Compared with repeating the original image, which also adds an image in the answer turn, it reduces accuracy by 8.90 pp. Both paired 95% confidence intervals exclude zero. Thus, neither the planning/conversation step nor merely adding another image fully explains the degradation: the generated helper introduces an additional performance loss.

We acknowledge that small observed differences may reflect sampling variability. Relative to no helper, the changes on RoboSpatial-Compatibility (+1.90 pp; paired 95% CI: [−5.71, 8.57]) and ChartQAPro (−0.07 pp; paired 95% CI: [−1.12, 0.98]) are not statistically distinguishable from zero. These two comparisons therefore do not establish a net benefit or harm; VSTAR provides clear evidence of additional loss from generated helpers.

We apply non-generative cropping and resizing only when the hint explicitly requests it. Each row compares the same eligible instances.

BenchmarkEligible instancesNo helperFaithful crop/zoomGenerated helper
VSTAR4369.7772.0951.16
RoboSpatial-Compatibility2166.6747.6252.38
ChartQAPro250.0050.0050.00

For an explicit crop/zoom request, Qwen3-VL-8B uses the fixed hint and first original image to predict a bounding box, without ground-truth answers. We then crop the original pixels and resize the crop with bicubic interpolation to a longest side of 1,024 pixels.

For the same 43 VSTAR instances with explicit crop/zoom requests, faithful cropping outperforms generation by 20.93 pp (95% CI: [4.65, 37.21]). Both conditions follow the same hint and provide a helper image; the difference is how that image is produced. This isolates an additional loss from using generation rather than faithful cropping, beyond the requested information-selection operation itself.

We explicitly instruct the hint planner to preserve task-relevant objects, text, numbers, and spatial relationships.

BenchmarkPreservation hint onlyPreservation hint + generated helperChange
VSTAR62.8352.88−9.95
RoboSpatial-Compatibility53.3355.24+1.90
ChartQAPro37.3338.60+1.26

Even with an explicit evidence-preservation instruction, adding the generated helper reduces VSTAR accuracy by 9.95 pp relative to the same hint without a helper (95% CI: [−15.18, −4.19]). Thus, the loss cannot be explained solely by the original prompt omitting this requirement: generated helpers still introduce additional harm under this prompt modification.

Together, these controls demonstrate additional harm from generated helpers, beyond the pipeline itself, even with evidence-preserving prompting.

W2: The scaling conclusion needs a narrower interpretation and more consistent reporting

TL;DR: Extending scaling to 10 calls shows task-dependent effects: ChartQAPro and MathVista stay below baseline, while ERQA improves.

Thank you for highlighting the limited budget range and unclear multi-call protocol. We conducted an extended scaling experiment with Qwen3-VL-8B + FLUX.1-Kontext, evaluating 1, 2, 3, 4, 5, 8, and 10 generation calls.

Experimental setup. We keep one hint fixed per instance. Each call edits the previous output, starting from the original image. The VLM answers using the original image(s) and only the latest helper, with decoding settings fixed across checkpoints. We reran all conditions, including 0/1/2 calls; their results may differ from the previously reported values.

Generation callsChartQAProERQAMathVista
0: baseline39.1642.5067.70
137.6142.5065.90
237.8944.0066.60
337.8243.2565.00
438.1143.7566.30
538.3942.7566.40
837.8244.0066.90
1037.4045.5066.30

Conclusion. Increasing generation calls yields task-dependent returns: ChartQAPro and MathVista remain below baseline, while ERQA improves. We accordingly narrow the scaling claim to the absence of consistent gains across the evaluated tasks, rather than universally increasing failure.

We will narrow the conclusion in the revised PDF to task-dependent scaling outcomes, without claiming that additional generation rounds consistently increase failure rates.

Q2: How are the second hint and second helper image produced and incorporated in the two-generation condition: independently, or conditioned on the first result? Please report the corresponding token/call budgets and at least a practical resource measure, or explicitly restrict the scaling claim to the evaluated one-to-two-call comparison.

TL;DR: One fixed hint guides sequential edits, and answering uses the originals plus only the latest helper; two generation calls take about 96 seconds on an L40.

We will update the scaling figure and its caption in the revised PDF to make this sequential editing protocol explicit.

We use one fixed hint per instance, without generating a second hint. The second call edits the first generated image using that same hint. The final answer uses the original image(s) plus only the second helper image; earlier helpers are not accumulated.

For the two-generation condition, the budget is one hint-generation VLM call, two image-generation calls, and one final-answer VLM call. The hint and answer limits are 256 and 1024 tokens, respectively. Each FLUX call uses 24 denoising steps and takes approximately 47–49 seconds on an L40, totaling about 96 seconds for two generations, excluding planning and answering.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.