- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Trace2Prog, which compiles the historical execution experience of agents into reusable Python programs. The accuracy and quality of each program are verified in a small number of sandbox environments, and a small amount of labeled data is used to test whether it can outperform the agent, which determines whether it is retained for reuse. At test time, a lightweight VLM router selects, for each task, either an appropriate reusable program or fallback agent to act. The method ultimately surpasses the original agent on controlled synthetic tasks and shows preliminary gains on real-world benchmarks.
Critical strengths
- S1. A meaningful research question with practical value. The paper addresses the problem that visual agents must repeatedly reason over and execute code for recurring tasks, and proposes a training-free experience-reuse framework that improves performance. By cutting redundant overhead, the approach has genuine practical value for real-world settings.
- S2. An in-depth exploration of the reliability of experience reuse. The paper goes beyond how to compile historical experience into reusable programs, examining how to assess program quality, decide whether a program is worth substituting for the original agent, and determine when deployment is appropriate. In particular, measuring the practical benefit of experience reuse through Fix, Hurt, and Net offers an interesting and valuable perspective on reliable agent experience reuse.
- S3. Clear problem definition and well-organized presentation. The paper is well structured and easy to follow. The three-stage Compile–Calibrate–Deploy framework is logically coherent and readable.
Critical weaknesses
- W1. The method achieves gains in controlled environments but only limited gains on the full TIR-Bench and PuzzleVQA. Moreover, the experiments concentrate on visual puzzles and other highly rule-based tasks, with no validation on more open-ended tasks such as mathematical reasoning or complex tool use, which limits the method's practical value and generalization in real-world.
- W2. The main expeirments compare mainly against baselines such as Per-query Synthesis and Success-only Compilation, lacking systematic comparison with recent agent experence-reuse methods (e.g., MMSkills, FlowEvo, CRAFT). The current experiments are insuficient to demonstrate a clear advantage of Trace2Prog over these recent approaches.
- W3. The method requires labeled samples from the target environment to judge whether a program is worth reusing, and the types of target visual variations in the controlled experiments already appear during calibration. The effectiveness of Trace2Prog under entirely novel visual conditions is not sufficently demonstrated.
[1] MMSkills: Towards Multimodal Skills for General Visual Agents
[2] FlowEvo: Self-Evolving Agents through the Co-Evolution of Workflows and Executable Skills
[3] CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets
Questions for authors
-
Q1. The method achieves very large gains in controlled experiments but only limited gains on the full benchmarks. Is this mainly because the reusable programs have low task coverage? How would coverage and actual gains change on mathematcal reasoning, code execution, or more complex agent tasks? Could this limit the method's real-world applicability?
-
Q2. The paper mainly relies on strong API models for program compilation and agent reasoning. Does the framework remain applicable to smaller open-source models (e.g., 4B/8B)? How would models of differing capability affect program quality and the final gains?