- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies “Reasoning with Imagination” through a two-stage pipeline: a VLM first proposes an image-generation hint, then answers using the original image, the hint, and the generated image. It evaluates both external-generator pipelines and the unified model BAGEL. Across nine benchmarks and ten task settings, generation augmentation generally reduces accuracy, with notable gains on RoboSpatial-Compatibility and on MathVista with BAGEL. Further analyses examine oracle hints and images, text- and vision-side test-time scaling. The contribution is primarily a cross-model empirical evaluation rather than a new training method. The results establish limitations of the tested pipelines, but the explanations of failure mechanisms and scaling behavior require stronger evidence.
Critical strengths
- The negative results have broad coverage and retain important counterexamples. The tested pairings of four VLM configurations and four generators, together with BAGEL, show broadly similar trends; the conclusion does not depend on a single weak generator. For example, GLM with Qwen-Image-Edit loses 31.3 percentage points on VSTAR. At the same time, Table 2 explicitly reports BAGEL gains of 8.1 points on MathVista and 25.7 points on RoboSpatial-Compatibility. This supports the qualified claim that most current configurations do not benefit, rather than the stronger claim that generation is intrinsically unhelpful.
- The diagnostics go beyond the main benchmark tables. Table 3a includes controlled prompts and ground-truth helper images, while Table 3b examines CoT. This work also includes rule-based scoring and a 100-instance human audit. It adopts the better-performing multi-turn implementation and explicitly leaves the caching gap unexplained, rather than attributing it to an inherent defect in caching. These additions mitigate concerns that evaluation was never checked or that only one implementation was considered.
Critical weaknesses
- Demonstrating degradation does not identify its principal cause. The final context contains the original image, hint, and generated image (Section 2.1), but there is no control that preserves the hint and number of conversational turns while removing only the generated image, nor a repeated-original-image or genuine-crop control. Consequently, the effects of misleading hints, additional context, and generation distortion remain entangled. Table 3a provides useful interventions but does not fully separate these factors. Section 5.3 annotates only 10–20 failures per benchmark. The 66%, 28%, 6% breakdown should therefore be treated as descriptive sample statistics. Similarly, low invocation rates alone do not establish that models recognize generation utility, and a correct answer alongside an imprecise image does not establish that the helper image was unused (Figure 4 and Sections 5.1–5.2).
- The scaling evidence is narrow, and a key numerical discrepancy needs correction. Section 4.2 compares CoT with one or two generation calls, without reporting actual token counts, latency, or compute budgets, or clarifying how the second call is combined with the first. This supports a conclusion about the tested configurations, not a general law that “Text scales; vision does not.” More concretely, Qwen3-VL 8B with FLUX.1 changes ERQA accuracy from 44.3 to 42.8, a decrease of 1.5 points (Tables 2 and 3b), whereas Figure 3a reports a decrease of 5.8 points. This does not reverse the direction of degradation, but it directly affects the reported magnitude and its comparison with text-side scaling.
- The oracle experiments and small differences are not sufficiently reproducible. Appendices A/B provide standard decoding settings and the main prompts, but not a task-by-task mapping of oracle experiments to data sources, sample counts, the actual hard-coded prompts, or the construction of ground-truth helper images. Neither the main results nor the intervention results include paired confidence intervals or variation across seeds. The controlled-prompt gains in Table 3a are only 0.7 and 2.2 points, so point estimates alone do not establish stable improvements.
Questions for authors
- Could you compare hint-only, repeated-original-image/genuine-crop, and generated-image conditions on the same instances with matched hints and conversational turns, separately reporting newly introduced and corrected errors? This would test the conversational-context or randomness explanations proposed in Section 5.2.
4 comments
W2
W2: The scaling evidence is narrow, and a key numerical discrepancy needs correction.
TL;DR: Extending scaling to 10 calls shows task-dependent effects: ChartQAPro and MathVista stay below baseline, while ERQA improves.
Thank you for highlighting the scope of the scaling evidence and the numerical inconsistency. We extended the scaling experiment with Qwen3-VL-8B + FLUX.1-Kontext to 1, 2, 3, 4, 5, 8, and 10 generation calls.
Experimental setup. One hint is generated and kept fixed. Each generation call edits the previous output, starting from the original image. At each checkpoint, the VLM answers using the original image(s) and only the latest helper. We reran all conditions, including the baseline and 1/2-call settings, under this explicitly specified protocol.
Resource budget. Each instance uses one hint-generation call (up to 256 tokens), n image-generation calls to reach round n, and one final-answer call (up to 1,024 tokens) per evaluated checkpoint. Each FLUX call uses 24 denoising steps and takes approximately 47–49 seconds on an L40, or about 96 seconds for two calls, excluding planning and answering.
| Generation calls | ChartQAPro | ERQA | MathVista |
|---|---|---|---|
| 0: baseline | 39.16 | 42.50 | 67.70 |
| 1 | 37.61 | 42.50 | 65.90 |
| 2 | 37.89 | 44.00 | 66.60 |
| 3 | 37.82 | 43.25 | 65.00 |
| 4 | 38.11 | 43.75 | 66.30 |
| 5 | 38.39 | 42.75 | 66.40 |
| 8 | 37.82 | 44.00 | 66.90 |
| 10 | 37.40 | 45.50 | 66.30 |
Conclusion. We narrow our scaling conclusion: failure rates do not increase consistently with additional generation rounds. Instead, ChartQAPro and MathVista remain below their no-generation baselines across all evaluated rounds, while ERQA shows improvement. We will update the manuscript to reflect these task-dependent scaling outcomes.
Numerical correction. For Qwen3-VL-8B + FLUX.1-Kontext without thinking, the original ERQA result is 44.3% → 42.8%, a change of −1.5 pp. The −5.8 pp value in Figure 3a and Appendix C.6 was a reporting error. The updated PDF will report −1.5 pp consistently in both locations.
W3
W3: The oracle experiments and small differences are not sufficiently reproducible.
TL;DR: Table 3a separates fixed-prompt generation from direct reference-image input; small gains are point estimates, complemented by paired confidence intervals from our new matched controls.
Thank you for the suggestion. Table 3a evaluates Qwen3-VL-8B with Nano-Banana under two targeted interventions:
| Intervention | Tasks | Helper construction |
|---|---|---|
| Controlled prompt | Jigsaw, Spatial | Replace the model-generated hint with a task-specific fixed instruction, retaining the generation step. |
| Oracle visual | Jigsaw, Maze, Multi-hop | Supply reference helper images directly, bypassing generation. Reference information is benchmark-provided or constructed for the task, as described in Appendix A. |
The benchmark sources cited for these controlled tasks are SAT, ROVER, Zebra-CoT, EMMA, and RealUnify, with Uni-MMMU additionally cited for reference resources. We will clarify the distinction between reference-image construction and fixed-prompt generation in the appendix.
The +0.7 and +2.2 pp controlled-prompt gains are reported as point estimates suggesting potential headroom. Our additional matched-control experiments provide statistical support for the degradation finding: on VSTAR, adding the generated helper reduces accuracy by 10.47 pp, with a paired 95% CI of [−16.75, −4.71], while holding the hint and conversational protocol fixed.
Response to authors' rebuttal
Thanks for the author’s response. These additional experiments have addressed some of my questions. Overall, however, this version of the manuscript still requires further polishing: the key experiments need to be refined and presented in the main text. I will keep my original score.
W1/Q1
We thank the reviewer for the careful assessment and constructive feedback. We appreciate the recognition of the practical relevance and breadth of our empirical study. Below, we address the concerns with additional experiments and clarifications, using W# for weaknesses and Q# for explicit questions.
TL;DR: Matched controls identify additional harm from generated helpers on VSTAR; the 66%/28%/6% breakdown describes the annotated failure sample.
W1/Q1: Demonstrating degradation does not identify its principal cause. / Could you compare hint-only, repeated-original-image/genuine-crop, and generated-image conditions on the same instances with matched hints and conversational turns, separately reporting newly introduced and corrected errors? This would test the conversational-context or randomness explanations proposed in Section 5.2.
Thank you for highlighting this distinction. We added matched controls using Qwen3-VL-8B + FLUX.1-Kontext, keeping the hint, conversational turns, and final-answer instructions fixed. We use greedy answer decoding to avoid answer-sampling variability.
On VSTAR, generated helpers reduce accuracy by 10.47 pp relative to no helper and 8.90 pp relative to repeating the original image; both paired 95% confidence intervals exclude zero. Thus, neither the planning/conversation step nor merely adding another image fully explains the loss.
Faithful crop/zoom controls. We apply non-generative cropping and resizing only when explicitly requested by the hint, comparing the same eligible instances.
On VSTAR, faithful cropping outperforms generation by 20.93 pp (95% CI: [4.65, 37.21]). Both conditions follow the same hint and provide a helper image, identifying an additional loss from generation rather than faithful execution of the requested crop/zoom operation.
Error transitions. Each row compares generated helpers with the listed control on the same instances. Introduced errors are correct-to-incorrect transitions; corrected errors are incorrect-to-correct transitions. No helper is the hint-only condition, and crop/zoom rows include only instances with explicit crop/zoom requests.
Scope of the qualitative analysis. We agree that the 66%/28%/6% breakdown describes the annotated failure sample, not population-level causal proportions. We will clarify this and revise statements suggesting that low invocation rates establish awareness of generation utility, or that a correct answer with an imprecise helper establishes that the helper was unused. We will update the PDF to reflect these clarifications.
Together, the matched controls demonstrate additional harm from generated helpers on VSTAR beyond the shared pipeline, while the qualitative annotations remain descriptive evidence.