acceptodds
Withdrawn from ICLR 2027

When Visual Programs Fail Silently: An Execution-Guided Repair Agent for Spatial Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual program synthesis makes spatial reasoning inspectable, but an incorrect perception output can pass through a well-formed program without raising an exception. We present PitCrew, an execution-guided repair framework that turns failed perception calls and inconsistent intermediate results into actionable program revisions. A question-conditioned Scene Captioner provides supplementary visual context for synthesis. The Program Executor records tool calls, and a deterministic Consistency Checker organizes answer-validity failures and disagreements with the scene context into feedback for a Repair Agent. The Repair Agent uses the failed program, execution trace, and scene context to localize the issue and revise perception queries or program logic within a bounded budget. Mask-aware depth and size operators support both initial and repaired executions. PitCrew achieves 49.8% on Omni3D-Bench and 89.4% on CLEVR, improving on the original VADAR configuration by 9.4 and 35.8 percentage points without task-specific training. Matched-backbone controls and component ablations support the utility of the combined system.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
visual program synthesis, spatial reasoning, compositional visual reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_h6qfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PitCrew, an execution-guided repair framework for visual programs. It combines question-conditioned scene descriptions, execution traces, validity checks, and uses an LLM repair agent to revise perception queries or program logic within a given budget. It also includes mask-aware depth aggregation and calibrated backprojection for spatial measurement. Experiments show that it shows performance gains over its foundation VADAR. Failure analyses showcase undetected errors and dependence on fallible visual evidence.

Critical strengths
  • The paper distinguishes successful program execution from correct visual reasoning, addressing a meaningful failure mode.
  • The repair mechanism uses the failed code and execution trace to revise the program, making the connection between observed failure and corrective action understandable.
Critical weaknesses
  • In Alg. 1, an admissible answer is accepted before checking execution-scene discrepancies. Therefore, a plausible but incorrect result can escape repair unless the generated program explicitly exposes a failure.
  • The current ablation only shows that removing the repair mechanism, including additional inference, results in performance degradation, which is acknowledged. However, there is no equal-budget comparison against simple baselines like regeneration, self-debugging, or repair with no diagnostic report. It remains unclear whether the proposed method is simply benefiting from additional attempts or genuine repair.
  • Some evaluated counting routes return the captioner’s count directly, allowing accuracy to improve without recovering missing or removing spurious detections. From the current experiment, it remains unclear how much improvement reflects genuine perception repair or merely from using another model's estimate.
  • From Appendix E, PitCrew has higher latency compared with VADAR (19.4s vs. 5s). It might be worth reporting complete latency and inference costs for all baselines, including those use equal-budget retry or direct answering, to better clarify and validate whether PitCrew offers a competitive accuracy-cost trade-off in the main experiment.
Questions for authors
  • See weakness.
Reviewer_duFyOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper extends VADAR, a visual agentic pipeline for single-image spatial reasoning, with three additions: a) Scene-Captioner, which produces a structured text context; b) checking and repairing loop 3) spatial-perception API. On the Omni3D-Bench and CLEVR benchmarks, the proposed framework PitCrew improves results over VADAR: 49.8% vs. 41 and 91.3 vs. 69.3%, respectively. But there is no (or only a slight) improvement over the direct GPT-5 mini, which serves as the framework's backbone, raising concerns because training and inference costs are much higher. The claimed repair mechanism fires in fewer than 10% of cases, and the promised captioning of silent mistakes only fires when the program's output is "unknown". Together with the lack of novelty, these controversial claims raise serious concerns about the method.

Critical strengths
  1. The real and well-motivated gap: the observation that exception-based retry cannot catch a type-correct wrong-perception return (Figure 1) is correct and important.
  2. Thorough ablations, including detailed "Leave-one-component-out", "Backbone and configuration controls", "Scene-context", and "Component sensitivity".
  3. The deterministic checker, fixed repair budget, and cached tool results are a well-judged design that keeps the loop cheap and reproducible
Critical weaknesses
  1. In the introduction, you compare numbers with different backbones: Omni3D-Bench from 40.4% to 49.8% and CLEVR from 53.6% to 89.4%. You basically compare apples to oranges. You had numbers for the GPT-5 backbone.
  2. Datasets. Seems like CLEVR is a 2017 synthetic dataset that current VLMs essentially saturate (Claude at 95, InternVL at 94, Qwen3-VL-32B at 92 in Table 1). And I don't understand why you use only a subset. VADAR used 1155 samples from CLEVR. Please add some of the SoTA datasets to the work, including SpatialClaw, pySpatial, and S-Agent. At least all of them should be mentioned in the literature review.
  3. Your main table lacks the Cost, Time, and Spent API tokens columns. PitCrew is almost on par with GPT-5-mini on more complex tasks from Omni3D, even though your method uses the same GPT-5 as the backbone. Obviously, it should spend more tokens and time, and I don't see the improvement.
  4. The reason for the above-mentioned concern may be an unclear methodology. The “repair” part of the paper, which is the title and the framing, touches 48 of 500 questions and fixes 22 of them, which is a very small number. The rest of the gain over VADAR comes from a better prompt (scene description from a second VLM) and better geometry, neither of which has anything to do with catching failures. That raises huge questions about methodology itself.
  5. It seems the authors overclaimed about catching silent failures; repair in PitCrew is triggered only when a program crashes or returns “unknown”; any program that returns a plausible but wrong answer, which is exactly what the title calls a silent failure, is never checked or repaired. This behavior is much narrower than the one in the title and abstract.
Questions for authors

I would like to see the authors add new SoTA datasets to the analysis, clearly demonstrate the working ideas behind the proposed method, and add columns for cost, time, and API tokens. Also, I hope the authors can improve the paper's style, as it's really hard to follow. The main text is nine pages of abstract nouns (“execution evidence”, “observable proxies”, “answer routes”, “operational labels”), with interesting mechanics and details pushed to the appendix.

Reviewer_yJG5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a method for repairing visual programs by creating admissibility/consistency criteria for intermediate steps, aided by a scene captioner. The results of the admissibility criteria are fed to a repair loop to create a revised program that overcomes the originally flagged inconsistency. The authors show significant improvement over the basic VADAR configuration on two benchmarks without additional training.

Critical strengths
  • The motivation is clear and addresses an important problem in the context of visual programs.
  • The components of the method are well-justified. As shown in the ablation study (Table 2), each component adds significant improvements, and they work well together to further improve the overall score.
  • The empirical results demonstrate significant gains over the base VADAR model, showing that the framework is even competitive with proprietary frontier VLMs.
Critical weaknesses
  • The method is overly reliant on the query—it could keep searching for an item that does not exist simply because it is mentioned in the query, instead of actually verifying the visual logic. Even worse, it could look for synonyms that are not exactly right. For example, if the query requests the width of a microwave but one does not exist in the image, the system might look for an oven instead; if an oven exists, this will lead to a false response.
  • As the authors note, this method adds computational overhead. While this is not a problem in itself, a baseline that matches this computational overhead is missing, such as majority voting across 4 independent programs from VADAR (1 initial generation + 3 repair loops) or a best-of-N setup with a VLM verifier.
  • The related work focuses on text-based code debugging but fails to cover approaches that target visual program repair. For example, it misses work like Code as Policies (Liang et al., 2023) that uses exception-based execution feedback, ProgPrompt (Singh et al., 2023) that integrates repairs as assertions in the programs, as well as methods like ViUniT (Panagopoulou et al., 2025) that dynamically generate visual unit tests to repair visual programs.
Questions for authors
  • How does the method compare to VADAR if the computational requirements are matched? (See weaknesses for example baselines)
  • How often does the terminal fallback occur? Even though line 431 states that its call rate is not separately logged, could this be manually recovered by looking at the programs of the 48 questions that entered the repair loop for Omni3D (Appendix D.1)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.