- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes PitCrew, an execution-guided repair framework for visual programs. It combines question-conditioned scene descriptions, execution traces, validity checks, and uses an LLM repair agent to revise perception queries or program logic within a given budget. It also includes mask-aware depth aggregation and calibrated backprojection for spatial measurement. Experiments show that it shows performance gains over its foundation VADAR. Failure analyses showcase undetected errors and dependence on fallible visual evidence.
Critical strengths
- The paper distinguishes successful program execution from correct visual reasoning, addressing a meaningful failure mode.
- The repair mechanism uses the failed code and execution trace to revise the program, making the connection between observed failure and corrective action understandable.
Critical weaknesses
- In Alg. 1, an admissible answer is accepted before checking execution-scene discrepancies. Therefore, a plausible but incorrect result can escape repair unless the generated program explicitly exposes a failure.
- The current ablation only shows that removing the repair mechanism, including additional inference, results in performance degradation, which is acknowledged. However, there is no equal-budget comparison against simple baselines like regeneration, self-debugging, or repair with no diagnostic report. It remains unclear whether the proposed method is simply benefiting from additional attempts or genuine repair.
- Some evaluated counting routes return the captioner’s count directly, allowing accuracy to improve without recovering missing or removing spurious detections. From the current experiment, it remains unclear how much improvement reflects genuine perception repair or merely from using another model's estimate.
- From Appendix E, PitCrew has higher latency compared with VADAR (19.4s vs. 5s). It might be worth reporting complete latency and inference costs for all baselines, including those use equal-budget retry or direct answering, to better clarify and validate whether PitCrew offers a competitive accuracy-cost trade-off in the main experiment.
Questions for authors
- See weakness.