- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how LVLMs can solve visual reasoning tasks requiring precise pixel-level calculations through code execution, while leveraging historical execution feedback to refine problem-solving strategies. The authors first introduce the "Code-with-Image" reasoning paradigm, enabling models to treat images as computational data, autonomously design algorithms, and generate and execute Python code to derive final answers. Subsequently, they construct CwI-Bench, a dataset comprising 6,300 samples across four categories of procedurally generated visual computation tasks. Building on this, the authors propose "Reflection-Driven Refinement," a method that iteratively optimizes a reusable "Skill Library" by reflecting on historical execution trajectories—all without updating model parameters. Experimental results demonstrate that the Code-with-Image approach outperforms traditional CoT reasoning across multiple LVLM backbones, with Reflection-Driven Refinement further enhancing task performance. The authors also validate the framework's effectiveness and transferability through ablation studies and cross-model skill transfer experiments.
Critical strengths
- Proposed a visual reasoning benchmark featuring verifiable answers. The authors constructed CwI-Bench, which encompasses 30 visual tasks requiring numerical computation. The data is automatically generated via programs and includes corresponding correct answers and verification mechanisms; this reduces the need for manual annotation and facilitates the evaluation of model performance across different tasks.
- Conducted comprehensive experiments; the proposed method achieved strong results. Experiments were performed using nine LVLM backbones, with comparisons made against methods such as TextGrad, GEPA, and SkillOpt. The results demonstrate that the proposed Reflection framework effectively improves model accuracy. Furthermore, cross-model experiments indicate that the learned skills are reusable across different models.
Critical weaknesses
-
The necessity of "Code-with-Image" and its specific role within the Reflection process lack sufficient justification. While the paper highlights the advantages of "Code-with-Image" over CoT reasoning, the tasks in the benchmark—which primarily involve pixel measurements and numerical calculations—are inherently better suited for code-based tools. Consequently, the experimental results are insufficient to demonstrate the unique value of the "Code-with-Image" reasoning paradigm itself. Furthermore, the Reflection component primarily optimizes a naturallanguage "Skill Library" by analyzing execution history—a strategy that could equally be applied to CoT reasoning. Although E-Refl incorporates additional code execution, the authors fail to adequately articulate the specific contribution of these code operations to the final performance gains. Therefore, it is recommended that the authors include a comparative experiment involving "CoT + Reflection" to further validate, under identical configurations, the unique advantages of combining "Code-withImage" with Reflection.
-
The distinction from existing related work is insufficiently clear, and the methodological innovation appears limited. There has already been significant research into feedback-based visual program optimization. For instance: [1] De-fine: Decomposing and Refining Visual Programs with AutoFeedback; [2] VDebugger: Harnessing Execution Feedback for Debugging Visual Programs; both investigate the use of execution feedback to identify and correct errors in visual programs. Additionally, [3] SkillOpt: Executive Strategy for Self-Evolving Agent Skills; explores the continuous optimization of reusable natural-language skills using historical execution records and feedback. In comparison, the current paper also improves problem-solving strategies by analyzing historical execution results, with the primary distinction being the introduction of three specific Reflection mechanisms. However, these mechanisms bear similarities to existing feedback-driven program revision and skill optimization methods, and the authors have not sufficiently highlighted their unique technical innovations. It is recommended that the authors clarify the differences between their work and the aforementioned studies, specifically detailing the new capabilities offered by the proposed Reflection mechanisms compared to existing approaches.
Questions for authors
- Why must "Reflection-Driven Refinement" be based on the "Code-withImage" paradigm? While current experiments demonstrate the effectiveness of code execution and reflection respectively, a comparison involving CoT combined with reflection is missing. Could the authors apply the same form of reflection to non-code CoT reasoning strategies under identical configurations? This would help disentangle the gains derived from code execution itself from the additional benefits provided by reflection.
- What is the specific contribution of code-specific execution feedback to performance improvement? Could the authors further distinguish between the following three settings: (1) performing strategic reflection using only the predicted result and the ground truth; (2) additionally providing historical programs and their intermediate execution records; and (3) further allowing for the proactive execution of diagnostic code and the testing of repair solutions? Such an experiment would help determine whether performance gains stem primarily from general error feedback, inspectable execution traces, or proactive code-level diagnostic capabilities.
- Could the authors highlight the paper's key methodological innovations by comparing them with existing visual program optimization methods?
3 comments
Thanks for the detailed response and the additional experiments. I appreciate the authors' efforts to address my concerns. The new experiments provide further evidence for the effectiveness of the proposed approach, and I agree that coding can be advantageous for certain visual reasoning tasks. However, I still have some reservations about the overall contribution and its generalization to real-world images. I will therefore maintain my original score.
Follow-up Response to Reviewer myLH
Follow-up Response to Reviewer myLH
Thank you for the follow-up and for recognizing the additional evidence. We would like to clarify the two remaining concerns.
1. Overall contribution
Benchmark contribution
CwI-Bench provides a testbed for visual computation and the refinement of reusable solution procedures. It comprises 30 tasks with verifiable answers and disjoint training, validation, and test splits. The benchmark evaluates whether models can perform precise computations from pixels and whether a procedure refined on training instances improves performance on unseen instances. Its contribution is therefore both the task suite and a setting for studying generalization after reflection.
Method contribution
We introduce execution-based reflection that actively gathers diagnostic evidence and consolidates it into a reusable task-level skill. The reflector can run new diagnostic code on training images, test repairs, and combine evidence across rounds. This is particularly suited to our tasks: an incorrect image measurement or intermediate computation can cause failure without its cause being clear from the final answer or saved execution trace. Active execution lets the reflector check these steps and refine the procedure, which is then applied to new instances without updating model weights. Tables 2–3 of the revised manuscript examine the roles of code execution and different forms of reflection feedback.
2. Generalization to real-world images
The real-image generalization experiments were already reported in our responses to Reviewers mJRY and pmYA and included in the revised manuscript. For convenience, we reproduce the relevant results below.
Yes, our approach generalizes to real-world images, and its skills hold under distribution shift. CwI-Bench itself is built on real images: 23 of its 30 tasks build every instance on a natural photograph. To test whether the approach generalizes further, to new real-world images, tasks and domains, we ran new experiments on six external benchmarks, using their subtasks that require computation, and report the results in Table R1 (Section 5.2) of the revised paper. These are core vision problems with direct practical use:
- MMIU geometry: homography and spatial-transformation estimation between two photographs, the basis of image stitching and registration.
- HPatches point transfer: finding the same physical points in another view of a planar scene, the standard HPatches homography-estimation test with the official ground-truth homographies, and the correspondence problem behind visual localization and AR.
- SpatialScore Camera: camera intrinsics, extrinsics and homography across up to four views, as needed for 3D reconstruction.
- TIR-Bench: tool-integrated visual reasoning, e.g., jigsaws, rotated images, word search and instrument reading.
- CHOICE image quality: judging cloud interference, noise and quality of remote-sensing images, a routine screening step for satellite data.
- AVI-Math geometry: measuring vehicle distances, sizes, flight height and camera pitch in drone images from the camera parameters.
Notably, CHOICE and AVI-Math are remote-sensing benchmarks, with overhead imagery far from our benchmark's photographs. Skills refined from at most 16 labeled items per task raise accuracy over bare Code-with-Image (CwI) on every benchmark and model, except MMIU with 27B, where bare CwI is already at 94.4%:
| Benchmark | 27B CoT | 27B CwI | 27B CwI + skill | 9B CoT | 9B CwI | 9B CwI + skill |
|---|---|---|---|---|---|---|
| MMIU geometry (ICLR'25) | 48.9 | 94.4 | 94.4 | 33.9 | 53.3 | 90.0 |
| HPatches point transfer (CVPR'17) | 0.0 | 72.6 | 90.4 | 0.0 | 18.9 | 25.2 |
| SpatialScore Camera (CVPR'26) | 48.5 | 72.9 | 76.1 | 42.8 | 52.8 | 58.5 |
| TIR-Bench (ECCV'26) | 34.0 | 47.6 | 49.9 | 29.5 | 38.1 | 42.3 |
| CHOICE image quality (NeurIPS'25 D&B) | 67.7 | 71.6 | 83.4 | 70.8 | 67.3 | 78.3 |
| AVI-Math geometry (ISPRS'25) | 25.0 | 36.8 | 49.5 | 20.4 | 21.5 | 30.4 |
- Large gains from very few labels: the skill lifts 9B on MMIU from 53.3% to 90.0%, close to 27B's 94.4%, and 27B on HPatches from 72.6% to 90.4%.
- Precise computation that text reasoning cannot reach: on HPatches, CoT gets no item right, while the median point error of 27B falls from 144 pixels (CoT) to 1.6 (bare CwI) and 0.9 (skill).
This validates that our approach generalizes beyond CwI-Bench, to real-world images, to tasks defined by others, and to new domains such as remote sensing, from a handful of labels.
(Table R1 in Section 5.2, with benchmarks and protocol in Appendix G.8)
We hope these clarifications and results address your remaining concerns. We would appreciate any further clarification of the aspects that remain concerning.
Response to Reviewer myLH
We thank the reviewer for the constructive questions, which led to new experiments that clarify the role of our method. All changes in the revised paper are marked in blue.
W1 / Q1. CoT + Reflection
Yes, the same reflection can also be applied to CoT. We ran this as a new experiment, applying the same reflection to CoT under identical configurations, and added CoT + Reflection to Table 2 of the revised paper. It adds only 0.9% to 2.0% to CoT, and CoT + Reflection stays below even bare Code-with-Image (CwI) on every model. The gain needs both code execution and reflection. Reflection can only refine a procedure that the solver can actually carry out: a chain of thought cannot measure a homography or count pixels exactly, so better instructions barely help it, while with code every refined step becomes an exact computation.
(Table 2, protocol in Appendix G.4)
Q2. Which feedback drives the gain
We thank the reviewer for this insightful suggestion. As suggested, we ran the three settings and report them in Table 3 of the revised paper: (1) reflection on the predicted result and the ground truth only, (2) additionally giving the past programs and their execution records, and (3) further running diagnostic code and testing repairs. Settings (2) and (3) are our O-Refl and O-Refl + E-Refl. Each added layer of evidence helps, and the two richer layers each add a gain with a 95% CI above zero.
All settings share one protocol and differ only in what the reflector sees. On 9B the largest step comes from diagnostic execution (+8.7%): running new code on the failing images reveals causes that the recorded attempt does not contain. On 27B, programs, execution records and diagnostic runs together add +14.5% beyond result-only feedback. (Table 3)
W2 / Q3. Difference from De-fine, VDebugger and SkillOpt
Compared with De-fine and VDebugger, our approach is cheaper at deployment and its gains carry over to every new question: they debug the program of each question at inference time, while we learn the algorithm for a whole class of problems once.
Compared with SkillOpt, our reflector gets better evidence for each skill edit, and this makes it more accurate: SkillOpt can only read what a recorded attempt already contains, while our reflector runs new code on the failing images to obtain evidence the attempt never produced.
At matched budget, ours beats SkillOpt on every LVLM, by 3.9% to 9.7% in test accuracy with 95% CIs above zero, and SkillOpt given 1.72× our budget still stays below us on 27B (63.0% vs 64.6%) (Table 2, Table R3).
(Positioning added to Related Work)