- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The work examines how the result of a coding agent’s first attempt might influence the decision to repair the existing solution or solve the task again, possibly using a different agent. Empirical studies train different continuation policies on exploratory tasks from a subset from BigCodeBench, and then evaluate on unseen tasks. The result is no practical effect of continuation policy on hidden test pass rate, compared to a fixed (model regeneration) policy.
Critical strengths
- The study separates policy learning from evaluation by ensuring that the continuation policy never observed hidden test outcomes at decision time, and also preregisters the continuation policy, baseline, and tasks.
Critical weaknesses
- There’s an asymmetry in the budget allocation between the baseline regeneration (B-solo) and review (B-expert) that confounds the experimental design. B-Solo was given a budget that included the cost of the first-pass generation, while B-expert was restricted to only the continuation budget. B-expert suffered execution errors caused by hitting this lower budget ceiling. As noted in the paper, this asymmetry prevents isolating whether review performed poorly due to intrinsic difficulty of reviewing code or due to token constraints.
- As the authors conclude, a prompt-plus-program ablation that omits execution feedback is needed to isolate whether information from the first coding attempt in the form of a visible-check feedback helps the continuation policy choose the next best action. The current design of the post-generation policy (P2-post) mixes executed check outcomes with other types of info into a single feature representation.
Questions for authors
- Pg. 2: where is the P1 Table? It would be helpful to display at least a stylized version.
- The central comparison of continuation policy (P2-Post) vs. regeneration strategy (B-solo) yields no practical difference. What can we conclude about the generalizable benefit of the continuation policy?