acceptodds
Desk rejected at ICLR 2027

Reflection-Driven Code-with-Image Reasoning

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Many visual tasks ask for quantitative properties of an image, such as displacement, rotation angle, or color cast. Solving these tasks requires computation over image pixels rather than visual inspection alone. We study Code-with-Image, a problem-solving paradigm in which a large vision-language model designs an algorithm, implements it as code, and executes it in a general-purpose interpreter to compute the answer. Each attempt leaves behind code and execution records that provide evidence for diagnosing failures, creating an opportunity for models to improve their solution strategies through reflection. To evaluate both computational visual reasoning and improvement through reflection, we introduce Code-with-Image Bench (CwI-Bench), a benchmark of 30 procedurally generated tasks with verifiable answers. Its disjoint training, validation, and test instances separate strategy refinement and selection from evaluation on unseen instances. We propose Reflection-Driven Refinement, a framework that develops and refines reusable solution strategies through reflection on prior attempts without updating model weights. Without reflection, Code-with-Image outperforms chain-of-thought reasoning across all nine backbones. Our framework further raises test accuracy from 41.3% to 70.1% on GPT-5.6-luna and from 32.6% to 64.6% on Qwen3.5-27B. The strategies also transfer across model scales and families.

Primary area:
datasets and benchmarks
Keywords:
Code-with-Image, vision-language models, reflection, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_myLHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how LVLMs can solve visual reasoning tasks requiring precise pixel-level calculations through code execution, while leveraging historical execution feedback to refine problem-solving strategies. The authors first introduce the "Code-with-Image" reasoning paradigm, enabling models to treat images as computational data, autonomously design algorithms, and generate and execute Python code to derive final answers. Subsequently, they construct CwI-Bench, a dataset comprising 6,300 samples across four categories of procedurally generated visual computation tasks. Building on this, the authors propose "Reflection-Driven Refinement," a method that iteratively optimizes a reusable "Skill Library" by reflecting on historical execution trajectories—all without updating model parameters. Experimental results demonstrate that the Code-with-Image approach outperforms traditional CoT reasoning across multiple LVLM backbones, with Reflection-Driven Refinement further enhancing task performance. The authors also validate the framework's effectiveness and transferability through ablation studies and cross-model skill transfer experiments.

Critical strengths
  1. Proposed a visual reasoning benchmark featuring verifiable answers. The authors constructed CwI-Bench, which encompasses 30 visual tasks requiring numerical computation. The data is automatically generated via programs and includes corresponding correct answers and verification mechanisms; this reduces the need for manual annotation and facilitates the evaluation of model performance across different tasks.
  2. Conducted comprehensive experiments; the proposed method achieved strong results. Experiments were performed using nine LVLM backbones, with comparisons made against methods such as TextGrad, GEPA, and SkillOpt. The results demonstrate that the proposed Reflection framework effectively improves model accuracy. Furthermore, cross-model experiments indicate that the learned skills are reusable across different models.
Critical weaknesses
  1. The necessity of "Code-with-Image" and its specific role within the Reflection process lack sufficient justification. While the paper highlights the advantages of "Code-with-Image" over CoT reasoning, the tasks in the benchmark—which primarily involve pixel measurements and numerical calculations—are inherently better suited for code-based tools. Consequently, the experimental results are insufficient to demonstrate the unique value of the "Code-with-Image" reasoning paradigm itself. Furthermore, the Reflection component primarily optimizes a naturallanguage "Skill Library" by analyzing execution history—a strategy that could equally be applied to CoT reasoning. Although E-Refl incorporates additional code execution, the authors fail to adequately articulate the specific contribution of these code operations to the final performance gains. Therefore, it is recommended that the authors include a comparative experiment involving "CoT + Reflection" to further validate, under identical configurations, the unique advantages of combining "Code-withImage" with Reflection.

  2. The distinction from existing related work is insufficiently clear, and the methodological innovation appears limited. There has already been significant research into feedback-based visual program optimization. For instance: [1] De-fine: Decomposing and Refining Visual Programs with AutoFeedback; [2] VDebugger: Harnessing Execution Feedback for Debugging Visual Programs; both investigate the use of execution feedback to identify and correct errors in visual programs. Additionally, [3] SkillOpt: Executive Strategy for Self-Evolving Agent Skills; explores the continuous optimization of reusable natural-language skills using historical execution records and feedback. In comparison, the current paper also improves problem-solving strategies by analyzing historical execution results, with the primary distinction being the introduction of three specific Reflection mechanisms. However, these mechanisms bear similarities to existing feedback-driven program revision and skill optimization methods, and the authors have not sufficiently highlighted their unique technical innovations. It is recommended that the authors clarify the differences between their work and the aforementioned studies, specifically detailing the new capabilities offered by the proposed Reflection mechanisms compared to existing approaches.

Questions for authors
  1. Why must "Reflection-Driven Refinement" be based on the "Code-withImage" paradigm? While current experiments demonstrate the effectiveness of code execution and reflection respectively, a comparison involving CoT combined with reflection is missing. Could the authors apply the same form of reflection to non-code CoT reasoning strategies under identical configurations? This would help disentangle the gains derived from code execution itself from the additional benefits provided by reflection.
  2. What is the specific contribution of code-specific execution feedback to performance improvement? Could the authors further distinguish between the following three settings: (1) performing strategic reflection using only the predicted result and the ground truth; (2) additionally providing historical programs and their intermediate execution records; and (3) further allowing for the proactive execution of diagnostic code and the testing of repair solutions? Such an experiment would help determine whether performance gains stem primarily from general error feedback, inspectable execution traces, or proactive code-level diagnostic capabilities.
  3. Could the authors highlight the paper's key methodological innovations by comparing them with existing visual program optimization methods?
3 comments
AuthorsOpenReview ↗

Response to Reviewer myLH

We thank the reviewer for the constructive questions, which led to new experiments that clarify the role of our method. All changes in the revised paper are marked in blue.

W1 / Q1. CoT + Reflection

Yes, the same reflection can also be applied to CoT. We ran this as a new experiment, applying the same reflection to CoT under identical configurations, and added CoT + Reflection to Table 2 of the revised paper. It adds only 0.9% to 2.0% to CoT, and CoT + Reflection stays below even bare Code-with-Image (CwI) on every model. The gain needs both code execution and reflection. Reflection can only refine a procedure that the solver can actually carry out: a chain of thought cannot measure a homography or count pixels exactly, so better instructions barely help it, while with code every refined step becomes an exact computation.

Test accuracyQwen3.5-9Bgemma-4-26BQwen3.5-27BGPT-5.6-luna
CoT7.210.88.612.9
CoT + Reflection8.611.710.613.9
CwI11.129.632.641.3
  • The CoT loop matches our O-Refl arm (seeds, rounds, probe, validation gate).
  • Bare CwI also beats CoT on the computational subtasks of all six external real-image benchmarks with 27B (Table R1, Section 5.2).

(Table 2, protocol in Appendix G.4)

Q2. Which feedback drives the gain

We thank the reviewer for this insightful suggestion. As suggested, we ran the three settings and report them in Table 3 of the revised paper: (1) reflection on the predicted result and the ground truth only, (2) additionally giving the past programs and their execution records, and (3) further running diagnostic code and testing repairs. Settings (2) and (3) are our O-Refl and O-Refl + E-Refl. Each added layer of evidence helps, and the two richer layers each add a gain with a 95% CI above zero.

Reflector input (each row adds to the row above)Qwen3.5-9BQwen3.5-27BGain from this row (9B / 27B)
None (bare CwI)11.132.6–
Setting (1): predicted result and ground truth14.641.4+3.5 / +8.8
Setting (2): + past programs and execution records (O-Refl)21.949.1+7.3 / +7.7
Setting (3): + diagnostic code and repair tests (O-Refl + E-Refl)30.655.9+8.7 / +6.8

All settings share one protocol and differ only in what the reflector sees. On 9B the largest step comes from diagnostic execution (+8.7%): running new code on the failing images reveals causes that the recorded attempt does not contain. On 27B, programs, execution records and diagnostic runs together add +14.5% beyond result-only feedback. (Table 3)

W2 / Q3. Difference from De-fine, VDebugger and SkillOpt

Compared with De-fine and VDebugger, our approach is cheaper at deployment and its gains carry over to every new question: they debug the program of each question at inference time, while we learn the algorithm for a whole class of problems once.

  • De-fine, VDebugger: debugging each case. They take the program generated for one question, which composes pretrained vision modules (e.g., find, VQA), execute it, and use the execution feedback to fix that program before answering. Every new question must be debugged again, with extra model calls at inference time.
  • Ours: the algorithm for the task. Reflection runs offline on failures across many training instances and writes the visual algorithm for this kind of problem (e.g., how to register two images or estimate a grating period) as a skill. The skill is learned once per task, and at test time the solver applies it to unseen questions with no debugging and no extra calls.

Compared with SkillOpt, our reflector gets better evidence for each skill edit, and this makes it more accurate: SkillOpt can only read what a recorded attempt already contains, while our reflector runs new code on the failing images to obtain evidence the attempt never produced.

  • SkillOpt: reading the record. Designed mainly for text and agent tasks, where the trajectory records the reasoning, it proposes an edit from recorded trajectories and verifier feedback.
  • Ours: running new diagnostics. In visual computation the cause of a failure often lies in the pixels and is not in the trajectory. E-Refl writes and runs diagnostic code on the failing images and checks a repair on those images before it enters the skill. C-Refl gathers its own evidence for each revision: it groups the remaining failures by type, re-implements the procedure of past successful attempts on training cases to check that it reproduces the gold answers, and repairs any step that does not before rewriting the skill (Table 3).

At matched budget, ours beats SkillOpt on every LVLM, by 3.9% to 9.7% in test accuracy with 95% CIs above zero, and SkillOpt given 1.72× our budget still stays below us on 27B (63.0% vs 64.6%) (Table 2, Table R3).

(Positioning added to Related Work)

Thanks for the detailed response and the additional experiments. I appreciate the authors' efforts to address my concerns. The new experiments provide further evidence for the effectiveness of the proposed approach, and I agree that coding can be advantageous for certain visual reasoning tasks. However, I still have some reservations about the overall contribution and its generalization to real-world images. I will therefore maintain my original score.

AuthorsOpenReview ↗

Follow-up Response to Reviewer myLH

Follow-up Response to Reviewer myLH

Thank you for the follow-up and for recognizing the additional evidence. We would like to clarify the two remaining concerns.

1. Overall contribution

Benchmark contribution

CwI-Bench provides a testbed for visual computation and the refinement of reusable solution procedures. It comprises 30 tasks with verifiable answers and disjoint training, validation, and test splits. The benchmark evaluates whether models can perform precise computations from pixels and whether a procedure refined on training instances improves performance on unseen instances. Its contribution is therefore both the task suite and a setting for studying generalization after reflection.

Method contribution

We introduce execution-based reflection that actively gathers diagnostic evidence and consolidates it into a reusable task-level skill. The reflector can run new diagnostic code on training images, test repairs, and combine evidence across rounds. This is particularly suited to our tasks: an incorrect image measurement or intermediate computation can cause failure without its cause being clear from the final answer or saved execution trace. Active execution lets the reflector check these steps and refine the procedure, which is then applied to new instances without updating model weights. Tables 2–3 of the revised manuscript examine the roles of code execution and different forms of reflection feedback.

2. Generalization to real-world images

The real-image generalization experiments were already reported in our responses to Reviewers mJRY and pmYA and included in the revised manuscript. For convenience, we reproduce the relevant results below.

Yes, our approach generalizes to real-world images, and its skills hold under distribution shift. CwI-Bench itself is built on real images: 23 of its 30 tasks build every instance on a natural photograph. To test whether the approach generalizes further, to new real-world images, tasks and domains, we ran new experiments on six external benchmarks, using their subtasks that require computation, and report the results in Table R1 (Section 5.2) of the revised paper. These are core vision problems with direct practical use:

  • MMIU geometry: homography and spatial-transformation estimation between two photographs, the basis of image stitching and registration.
  • HPatches point transfer: finding the same physical points in another view of a planar scene, the standard HPatches homography-estimation test with the official ground-truth homographies, and the correspondence problem behind visual localization and AR.
  • SpatialScore Camera: camera intrinsics, extrinsics and homography across up to four views, as needed for 3D reconstruction.
  • TIR-Bench: tool-integrated visual reasoning, e.g., jigsaws, rotated images, word search and instrument reading.
  • CHOICE image quality: judging cloud interference, noise and quality of remote-sensing images, a routine screening step for satellite data.
  • AVI-Math geometry: measuring vehicle distances, sizes, flight height and camera pitch in drone images from the camera parameters.

Notably, CHOICE and AVI-Math are remote-sensing benchmarks, with overhead imagery far from our benchmark's photographs. Skills refined from at most 16 labeled items per task raise accuracy over bare Code-with-Image (CwI) on every benchmark and model, except MMIU with 27B, where bare CwI is already at 94.4%:

Benchmark27B CoT27B CwI27B CwI + skill9B CoT9B CwI9B CwI + skill
MMIU geometry (ICLR'25)48.994.494.433.953.390.0
HPatches point transfer (CVPR'17)0.072.690.40.018.925.2
SpatialScore Camera (CVPR'26)48.572.976.142.852.858.5
TIR-Bench (ECCV'26)34.047.649.929.538.142.3
CHOICE image quality (NeurIPS'25 D&B)67.771.683.470.867.378.3
AVI-Math geometry (ISPRS'25)25.036.849.520.421.530.4
  • Large gains from very few labels: the skill lifts 9B on MMIU from 53.3% to 90.0%, close to 27B's 94.4%, and 27B on HPatches from 72.6% to 90.4%.
  • Precise computation that text reasoning cannot reach: on HPatches, CoT gets no item right, while the median point error of 27B falls from 144 pixels (CoT) to 1.6 (bare CwI) and 0.9 (skill).

This validates that our approach generalizes beyond CwI-Bench, to real-world images, to tasks defined by others, and to new domains such as remote sensing, from a handful of labels.

(Table R1 in Section 5.2, with benchmarks and protocol in Appendix G.8)

We hope these clarifications and results address your remaining concerns. We would appreciate any further clarification of the aspects that remain concerning.

Reviewer_pmYAOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces the "Code-with-Image" (CwI) task and the CwI-Bench benchmark. It investigates how vision-language models (VLMs) can leverage code execution to perform precise image-based computations and utilize historical execution trajectories to reflect upon and refine their strategies. The proposed "Reflection-Driven Refinement" framework achieves optimization without requiring parameter updates, employing fault diagnosis, strategy review, and strategy reconstruction. Experiments demonstrate that both code execution and multi-round reflection significantly enhance performance, outperforming existing related methods.

Critical strengths
  1. CwI-Bench focuses on a model's ability to construct and execute reliable computational processes based on pixel data. It covers a diverse range of visual computation challenges and facilitates research into strategy improvement through program generation, answer verification, and distinct data partitioning.
  2. Multiple experiments indicate that the reflection mechanism consistently improves model performance, surpassing other skill-optimization approaches. Ablation studies, budget-based comparisons, and case analyses further validate the practical contributions of each component.
  3. Cross-model experiments reveal that skill transfer effectiveness depends on the specific model used for execution; superior skill-generation capability does not guarantee better transfer, suggesting that strategy evaluation must be considered in the context of actual execution proficiency.
Critical weaknesses
  1. The core mechanism overlaps with existing works such as PyVision, GEPA, and SkillOpt. The primary contributions lie in the introduction of the benchmark, execution-level diagnostics, and the integration of cross-round strategy reconstruction. The paper lacks precise positioning relative to these works and direct, like-for-like comparisons, leaving the boundaries of its originality somewhat ambiguous.
  2. The method yields an improvement of only 3.4–6.2 percentage points over strong baselines, with the performance gap narrowing under higher computational budgets. Furthermore, the study lacks results from independent runs and an analysis of uncertainty. It also fails to fully account for the costs associated with token generation, code execution, and image processing, leaving the claim of cost-efficiency unverified.
  3. The experiments primarily validate strategy generalization within a fixed task distribution; they do not yet demonstrate the ability to adapt to novel visual computation tasks. The underlying causes of performance variations across different models remain unclear, and the paper lacks a thorough discussion of failure cases and the limits of the model's capabilities.
Questions for authors
  1. There are slight inconsistencies in the descriptions of the acceptance and selection process for C-Refl candidate skills across Figure 2, Algorithm 3, and Appendix G.4. Please clarify whether the acceptance or rejection of candidates in each round of C-Refl is determined using a training probe or a validation score. Additionally, specify whether feedback from the validation set is incorporated into the subsequent reflection history. This is crucial for understanding the precise role of the validation set and ensuring a fair comparison against existing baselines for skill optimization.
  2. Given that PyVision has demonstrated the capability to perform numerical computations directly based on pixels, and that methods such as SkillOpt utilize execution trajectories to optimize external skills, please further articulate the unique contributions of this work regarding its computational paradigm and strategy optimization mechanism. Specifically, beyond differences in application scenarios, what are the fundamental mechanistic distinctions between "executable reflection" and existing trajectory-driven skill optimization methods?
5 comments
AuthorsOpenReview ↗

Response to Reviewer pmYA (1/3)

We thank the reviewer for the detailed and thoughtful assessment. The questions on positioning, uncertainty and cost led to several new experiments that strengthen the paper. All changes in the revised paper are marked in blue.

W1 / Q2 (a). PyVision

We ran a new experiment with PyVision's official prompt on the same backbones and test items. In PyVision, code produces views for the model to read. In Code-with-Image (CwI), the program computes the answer itself, and on the same backbone bare CwI scores higher.

Same test items, 5 roundsQwen3.5-9BQwen3.5-27B
PyVision official prompt, on our tool-call interface8.524.1
Bare CwI11.132.6
PyVision − CwI [95% CI]−2.6 [−3.9, −1.3]−8.5 [−10.3, −6.7]

The gap is largest on 27B (−8.5%), and it comes from what the code is used for. PyVision's programs mostly prepare views of the image, so the answer is still read off by the model, while our tasks ask for exact quantities such as a homography, a grating period or a count, which only a computation can deliver. PyVision is a solver paradigm rather than a strategy optimizer, so it is complementary to our reflection.

W1 / Q2 (b). GEPA and SkillOpt

The fundamental mechanistic difference is where the evidence for a skill edit comes from: trajectory-driven optimizers (GEPA, SkillOpt, TextGrad) can only read what a recorded attempt already contains, while our reflector runs new code on the failing images to obtain evidence the attempt never produced.

  • GEPA, SkillOpt, TextGrad: reading the record. Designed mainly for text and agent tasks, where the trajectory records the reasoning, they propose an edit from recorded trajectories and scores and judge it only by re-running the solver.
  • Ours: running new diagnostics. In visual computation the cause of a failure often lies in the pixels and is not in the trajectory. E-Refl writes and runs diagnostic code on the failing images (e.g., re-computing a suspect intermediate or measuring an image property) and checks a repair on those images before it enters the skill. C-Refl gathers its own evidence for each revision: it groups the remaining failures by type, re-implements the procedure of past successful attempts on training cases to check that it reproduces the gold answers, and repairs any step that does not before rewriting the skill (Table 3).

All methods share the solver, instances, seeds, skill injection and validation-based delivery, at matched budgets (0.99× to 1.15×), and ours beats all three on every LVLM (Table 2). Table 3 isolates this evidence: executable reflection adds 3.6% to 8.7% in test accuracy on top of observational reflection.

AuthorsOpenReview ↗

Response to Reviewer pmYA (2/3)

W2 (a). Margins, independent runs, uncertainty

We now report 95% confidence intervals for every method's gain over bare CwI (below), and new repeated test evaluations of our delivered skills (Appendix G.4 of the revised paper). The gains are statistically robust: every method's gain over bare CwI has a 95% CI above zero, ours is the largest on every LVLM, and all 14 Ours − baseline differences also have 95% CIs above zero.

Gain over bare CwI [95% CI]Qwen3.5-9Bgemma-26BQwen3.5-27Bluna
TextGrad+16.3 [+14.8, +17.8]+18.8 [+17.1, +20.6]+23.7 [+22.0, +25.5]+19.6 [+17.9, +21.2]
GEPA+23.5 [+22.2, +24.9]+20.8 [+19.3, +22.3]+24.5 [+22.8, +26.2]+22.6 [+21.0, +24.2]
SkillOpt+19.8 [+18.3, +21.4]+20.3 [+18.7, +22.0]+28.1 [+26.4, +29.8]+19.1 [+17.3, +20.8]
Ours+26.9 [+25.4, +28.3]+24.2 [+22.5, +26.0]+32.0 [+30.3, +33.7]+28.8 [+27.1, +30.4]
  • Every method, ours included, runs the same seeds per task and delivers the skill with the best validation accuracy.
  • Independent runs: reflection runs three independent seeds per task on 9B, gemma-26B and 27B, with the validation gain of each seed in Table 12 and Table 13 (Appendix G.6, G.7).
  • CIs: paired, task-stratified bootstrap over the 3,000 test items.

(Gains as in Table 2)

Independent test re-evaluations of the delivered skills (Appendix G.4):

OursPaperRe-run 1Re-run 2Mean ± std
Qwen3.5-27B64.664.364.164.3 ± 0.3
GPT-5.6-luna70.170.371.770.7 ± 0.9

Every re-run stays above every baseline.

W2 (b). Budget and cost

We now measure the full optimization cost of every method, including our pipeline at reduced budgets, and report it in Table R3 of the revised paper. At matched budget, ours is the most accurate method. We count every model call and every output token of the whole optimization, including the code execution and image processing inside each solve.

Budget is the cap on model calls that each method had for skill optimization, as a multiple of ours. The other columns report what each method actually used per task and seed. No method used its full budget, since reflection and solving often stop early.

Qwen3.5-27BBudget (cap)Actual model callsOutput tokensTest acc.Gain over bare CwI
Bare CwI–––32.6–
TextGrad1.06×1.78K1.81M56.3+23.7
GEPA1.12×1.18K1.00M57.1+24.5
SkillOpt1.15×2.17K1.62M60.7+28.1
SkillOpt1.72×3.24K2.43M63.0+30.4
Ours0.80×1.38K1.16M62.8+30.2
Ours1.00×1.76K1.48M64.6+32.0
GPT-5.6-lunaBudget (cap)Actual model callsOutput tokensTest acc.Gain over bare CwI
Bare CwI–––41.3–
TextGrad1.07×2.44K0.60M60.9+19.6
GEPA1.00×1.29K0.17M63.9+22.6
SkillOpt0.99×2.14K0.31M60.4+19.1
Ours0.60×1.21K0.21M64.6+23.3
Ours0.80×1.61K0.29M68.4+27.1
Ours1.00×2.05K0.37M70.1+28.8
  • With 0.80× of the budget, ours already beats every baseline given about the full budget (27B: 62.8 vs ≤ 60.7, luna: 68.4 vs ≤ 63.9).
  • More budget does not close the gap. SkillOpt given 1.72× (2.43M output tokens) reaches 63.0, still below our 64.6. GEPA stops earlier on its own once its validation score stops improving, so a larger budget does not raise its accuracy.
  • On the overhead of each mechanism: O-Refl and E-Refl form the main optimization loop, and C-Refl runs at most once per task after it. On the open LVLMs, C-Refl uses 19% to 28% of our budget and adds 6.1 to 8.7 test points (Table 3, and Table 12 in Appendix G.6).
  • Reflection runs once per task, offline, and using the skill adds no model calls.
  • Our lower-budget rows are the same pipeline with early stopping.
  • The lower token use of GPT-5.6-luna reflects a stronger model, which solves and reflects with more compact outputs.

(Table R3, Appendix G.4)

AuthorsOpenReview ↗

Response to Reviewer pmYA (3/3)

W3. Novel tasks, cross-model differences, failure cases

Novel tasks. Skills are task-specific by design, but a new task needs only a few labels. We ran new experiments on the computational subtasks of six external real-image benchmarks, with their own images and task definitions, and report the results in Table R1 (Section 5.2) of the revised paper. Skills refined from at most 16 labeled items per task raise accuracy over bare CwI everywhere except MMIU with 27B, where bare CwI is already at 94.4%.

Benchmark27B CoT27B CwI27B CwI + skill9B CoT9B CwI9B CwI + skill
MMIU geometry (ICLR'25)48.994.494.433.953.390.0
HPatches point transfer (CVPR'17)0.072.690.40.018.925.2
SpatialScore Camera (CVPR'26)48.572.976.142.852.858.5
TIR-Bench (ECCV'26)34.047.649.929.538.142.3
CHOICE image quality (NeurIPS'25 D&B)67.771.683.470.867.378.3
AVI-Math geometry (ISPRS'25)25.036.849.520.421.530.4

CHOICE and AVI-Math come from remote sensing, a domain far from our benchmark, and 16 labels per task were still enough for reflection to write skills that help there.

(Table R1 in Section 5.2, with benchmarks and protocol in Appendix G.8)

Out of distribution. We ran a new experiment on 20 CwI-Bench tasks regenerated with photographs from a new source (Objects365) and parameters outside the training range, and report it in Table R6 (Appendix G.11) of the revised paper. Applied without further reflection, the original 27B skills add +22.8% over bare CwI, as much as the +22.2% on the original test.

Cross-model differences trace to the reflector and to the solver's ability to execute a written procedure.

FactorSettingTest accuracy
Reflector (Fig. 3b)9B solver; self / 27B / gemma-26B / luna reflector21.9 / 31.4 / 25.8 / 21.1
Executing a procedure (Table 4)9B with the 27B skill / 27B itself47.4 / 64.6

The two factors interact: a reflector writes skills at its own level of execution, so luna's skills are hard for 9B to carry out, while 27B's suit it best. A skill written once by a stronger model thus lifts a weaker solver, raising 9B to 47.4%, well above the 38.0% with its own skill.

Failure cases. We added a new failure analysis in Appendix G.10 of the revised paper. Seven tasks stay below 30% on at least three models, two of them (unshuffle 2D, rotated jigsaw) on all four. Reflection gains most when it can contrast failures with some correct attempts, so it has little to build on where the bare solver never succeeds. The table groups the 117 model–task pairs with complete records by how well the bare solver does on the 32 training probe items before any reflection:

Bare solver before reflection (probe accuracy)Model–task pairsSkill deliveredMean test gain
Solves none (0%)1540%+3.5
Up to 10%2584%+26.2
10% to 30%3497%+39.9
Above 30%4393%+26.6

Once the bare solver solves even one probe item, 84% to 97% of pairs receive a skill. On unshuffle 2D, for example, 27B's programs cut the image into equal parts and ignore the badge strip along the left and top edges, so every cut falls inside a tile and reflection has almost no correct attempt to learn from. (Table R5, Appendix G.10)

Q1. C-Refl acceptance and the role of validation

We thank the reviewer for the careful reading. The question is whether C-Refl accepts a candidate by the training probe or by validation, and whether validation feedback enters the later reflection history. Whether a C-Refl candidate is accepted is decided on validation, and in all three mechanisms validation decides which skill is delivered. The reflector never sees the validation examples, and the test set is used only for the final evaluation. We have made this explicit in Algorithm 3 and Appendix G.4 of the revised paper.

  • C-Refl: each round, a candidate that improves on the training probe goes on to validation, which decides whether it replaces the current skill.
  • The reflection history records accept or reject outcomes, never validation images, trajectories or answers.

The baselines also select on validation (e.g., SkillOpt accepts each skill update only through its held-out validation gate), and every method delivers its best-validation skill.

Reviewer_mJRYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Code-with-Image, a paradigm in which vision-language models generate and execute code to solve quantitative visual tasks. The authors construct CwI-Bench, consisting of 30 procedurally generated tasks, and propose Reflection-Driven Refinement with three reflection mechanisms to improve reusable solution strategies without updating model parameters. Experiments across multiple models demonstrate substantial improvements over direct reasoning and existing skill optimization methods.

Critical strengths
  1. Interesting problem formulation and benchmark design. CwI-Bench emphasizes computational visual reasoning with verifiable answers, providing a useful evaluation setting for code-based visual problem solving.
  2. Strong experimental results. The proposed framework achieves substantial improvements across different model families. Comparisons with existing skill optimization methods, component ablations, and cross-model skill transfer experiments support its effectiveness.
Critical weaknesses
  1. Limited evaluation of real-world generalization. The benchmark consists entirely of procedurally generated tasks. Although the test instances are held out, they follow the same task definitions and generation procedures. It remains unclear whether the learned strategies generalize to more diverse real-world visual problems.
  2. Insufficient computational efficiency analysis. The framework requires multiple reflection rounds, code executions, and validation evaluations. Although optimization budgets are matched by model inference calls, token consumption, execution overhead, and overall computational costs are not thoroughly evaluated.
Questions for authors
  1. How well do the refined skills generalize to real-world images or tasks with different generation distributions?
  2. Could the authors provide a more detailed computational cost comparison, including token consumption, execution time, and the overhead introduced by different reflection mechanisms?
  3. How sensitive is the framework to the number of labeled training instances available for reflection? Can it maintain its effectiveness under more limited supervision?
2 comments
AuthorsOpenReview ↗

Response to Reviewer mJRY (1/2)

We thank the reviewer for the careful reading, for recognizing the benchmark design and the breadth of the experiments, and for the thoughtful questions on generalization and cost, which helped us strengthen the paper. All changes in the revised paper are marked in blue.

W1 / Q1. Generalization to real-world images and other distributions

Yes, our approach generalizes to real-world images, and its skills hold under distribution shift. CwI-Bench itself is built on real images: 23 of its 30 tasks build every instance on a natural photograph. To test whether the approach generalizes further, to new real-world images, tasks and domains, we ran new experiments on six external benchmarks, using their subtasks that require computation, and report the results in Table R1 (Section 5.2) of the revised paper. These are core vision problems with direct practical use:

  • MMIU geometry: homography and spatial-transformation estimation between two photographs, the basis of image stitching and registration.
  • HPatches point transfer: finding the same physical points in another view of a planar scene, the standard HPatches homography-estimation test with the official ground-truth homographies, and the correspondence problem behind visual localization and AR.
  • SpatialScore Camera: camera intrinsics, extrinsics and homography across up to four views, as needed for 3D reconstruction.
  • TIR-Bench: tool-integrated visual reasoning, e.g., jigsaws, rotated images, word search and instrument reading.
  • CHOICE image quality: judging cloud interference, noise and quality of remote-sensing images, a routine screening step for satellite data.
  • AVI-Math geometry: measuring vehicle distances, sizes, flight height and camera pitch in drone images from the camera parameters.

Notably, CHOICE and AVI-Math are remote-sensing benchmarks, with overhead imagery far from our benchmark's photographs. Skills refined from at most 16 labeled items per task raise accuracy over bare Code-with-Image (CwI) on every benchmark and model, except MMIU with 27B, where bare CwI is already at 94.4%:

Benchmark27B CoT27B CwI27B CwI + skill9B CoT9B CwI9B CwI + skill
MMIU geometry (ICLR'25)48.994.494.433.953.390.0
HPatches point transfer (CVPR'17)0.072.690.40.018.925.2
SpatialScore Camera (CVPR'26)48.572.976.142.852.858.5
TIR-Bench (ECCV'26)34.047.649.929.538.142.3
CHOICE image quality (NeurIPS'25 D&B)67.771.683.470.867.378.3
AVI-Math geometry (ISPRS'25)25.036.849.520.421.530.4
  • Large gains from very few labels: the skill lifts 9B on MMIU from 53.3% to 90.0%, close to 27B's 94.4%, and 27B on HPatches from 72.6% to 90.4%.
  • Precise computation that text reasoning cannot reach: on HPatches, CoT gets no item right, while the median point error of 27B falls from 144 pixels (CoT) to 1.6 (bare CwI) and 0.9 (skill).

This validates that our approach generalizes beyond CwI-Bench, to real-world images, to tasks defined by others, and to new domains such as remote sensing, from a handful of labels.

(Table R1 in Section 5.2, with benchmarks and protocol in Appendix G.8)

Distribution shift. We ran a new experiment on a much harder, shifted test and report it in Table R6 of the revised paper. We regenerated 20 CwI-Bench tasks with photographs from a different dataset (Objects365 instead of COCO) and moved at least one generation parameter of every task outside the training range, e.g., 7×7 jigsaw grids, larger translations for alignment, fainter gratings and 13 to 16 colors for color counting, with 100 new instances per task. The original Qwen3.5-27B skills were applied unchanged, without further reflection, and keep their full gain:

Test items (same 20 tasks)CoTCwICwI + skill+ skill − CwI [95% CI]
Original test10.435.958.0+22.2 [+20.1, +24.3]
Objects365 photographs, new parameters3.017.840.7+22.8 [+20.9, +24.8]

The shift is hard: bare CwI drops from 35.9% to 17.8%. Yet the skills still add +22.8% (+22.2% on the original), so with them 27B is more accurate on the shifted test than bare CwI is on the original (40.7% vs 35.9%). This shows the approach is robust: reflection writes down how to compute the answer (e.g., how to register two images), not what the training answers look like, so the skills hold on photographs and parameter values they never saw. (Table R6, Appendix G.11)

AuthorsOpenReview ↗

Response to Reviewer mJRY (2/2)

W2 / Q2. Computational cost

We now measure the full optimization cost of every method, including our pipeline at reduced budgets, and report it in Table R3 of the revised paper. At matched budget, ours is the most accurate method. We count every model call and every output token of the whole optimization, for the solver and the reflector alike. Wall-clock time depends on the hardware and on API latency, so we report hardware-independent counts, and the code execution and image processing of every solve are part of the counted solves. Once delivered, the skill adds no model calls at deployment.

Budget is the cap on model calls that each method had for skill optimization, as a multiple of ours. The other columns report what each method actually used per task and seed. No method used its full budget, since reflection and solving often stop early.

Qwen3.5-27BBudget (cap)Actual model callsOutput tokensTest acc.Gain over bare CwI
Bare CwI–––32.6–
TextGrad1.06×1.78K1.81M56.3+23.7
GEPA1.12×1.18K1.00M57.1+24.5
SkillOpt1.15×2.17K1.62M60.7+28.1
SkillOpt1.72×3.24K2.43M63.0+30.4
Ours0.80×1.38K1.16M62.8+30.2
Ours1.00×1.76K1.48M64.6+32.0
GPT-5.6-lunaBudget (cap)Actual model callsOutput tokensTest acc.Gain over bare CwI
Bare CwI–––41.3–
TextGrad1.07×2.44K0.60M60.9+19.6
GEPA1.00×1.29K0.17M63.9+22.6
SkillOpt0.99×2.14K0.31M60.4+19.1
Ours0.60×1.21K0.21M64.6+23.3
Ours0.80×1.61K0.29M68.4+27.1
Ours1.00×2.05K0.37M70.1+28.8
  • With 0.80× of the budget, ours already beats every baseline given about the full budget (27B: 62.8 vs ≤ 60.7, luna: 68.4 vs ≤ 63.9).
  • More budget does not close the gap. SkillOpt given 1.72× (2.43M output tokens) reaches 63.0, still below our 64.6. GEPA stops earlier on its own once its validation score stops improving, so a larger budget does not raise its accuracy.
  • Reflection runs once per task, offline, and using the skill adds no model calls.
  • On the overhead of each mechanism: O-Refl and E-Refl form the main optimization loop, and C-Refl runs at most once per task after it. On the open LVLMs, C-Refl uses 19% to 28% of our budget and adds 6.1 to 8.7 test points (Table 3, and Table 12 in Appendix G.6).
  • Our lower-budget rows are the same pipeline with early stopping.
  • The lower token use of GPT-5.6-luna reflects a stronger model, which solves and reflects with more compact outputs.

(Table R3, Appendix G.4)

Q3. Sensitivity to the number of labeled instances

We ran new experiments in which reflection gets only 8 or 16 labeled instances per task, and report them in Table R2 (Section 5.2) of the revised paper. The loop remains effective with few labels: with 16 labeled instances per task, O-Refl + E-Refl reach 54.0% on 27B, against 55.9% with all 110 and 32.6% without reflection.

O-Refl + E-Refl, labeled instances per task9B27Bluna
0 (bare CwI)11.132.641.3
823.249.860.4
1624.954.062.3
110 (paper setting: 60 train + 50 val)30.655.964.6
  • With 8 or 16, the instances come from the training split (N/4 per difficulty tier) and serve as training cases, probe and selection set. The test set is unchanged.
  • On CwI-Bench, every gain over bare CwI has a 95% CI above zero.

Even 8 labeled instances recover most of the gain, and 16 recover about 90% of it on 27B and luna. A new task therefore needs only a handful of labeled examples, which is exactly how the skills for the six external benchmarks in W1 were refined.

(Table R2 in Section 5.2, Appendix G.9)

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to All Reviewers

Dear Reviewers,

Thank you for your constructive feedback. We have uploaded a revised manuscript to OpenReview, with changes highlighted in blue. The main updates are summarized below; all references are to the revised manuscript.

  • Code execution and novelty (pmYA, myLH). We added a CoT + Reflection control and a feedback ablation to examine the contributions of code execution and execution feedback (Tables 2–3). We also clarified how our reusable skill refinement differs from prior approaches (Section 2), with a PyVision comparison in our rebuttal.
  • Generalization and label efficiency (mJRY, pmYA). We evaluated generalization beyond the original benchmark and sensitivity to limited supervision, using tasks from six external benchmarks, distribution shifts, and only 8 or 16 labeled instances per task (Tables R1, R2 and R6).
  • Reliability and cost (mJRY, pmYA). We added confidence intervals, model-call/output-token costs, and reduced-budget results to clarify uncertainty and the performance–cost trade-off (Table R3), alongside repeated test evaluations.
  • Selection protocol and limitations (pmYA). We clarified that validation determines C-Refl acceptance and skill delivery, while reflection history records accept/reject outcomes without exposing validation examples (Algorithm 3). We also discuss limitations in cross-model skill transfer (Table 4) and remaining failure cases (Appendix G.10).

Please see our point-by-point responses and the revised manuscript for details. We welcome any further questions.

Dear reviewers,

Please be aware that little time remains for the author-reviewer discussion. As the deadline is approaching, we kindly request that you refrain from asking for additional experiments. Please summarize your final position concisely.

Sincerely, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.