- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose CODEMAPS, a framework that enables coding agents to maintain an explicit progress tracker and update it based on running explicit test cases generated. The authors argue that this is particularly useful for long-horizon tasks where the agents need to scope out a lot of work and may diverge at some point in time. The authors provide results on a benchmark ProgramBench where they outperform other baselines such as MSA-full, Claude Code, Spec-First. The authors also perform ablations to tease apart contributions of different components in their system and also find that the performance on generated test cases is highly positively correlated with the performance on hidden tests.
Critical strengths
- The core problem of providing a persistent checklist and grounding in long-agent horizon tasks is quite relevant and interesting. In that regard the work takes a meaningful step in that direction and provides impressive gains across different model configurations and wide variety of task difficulties.
Critical weaknesses
-
Compute fairness: While the authors mention that all models receive six hours and 1000 turns, CodeMaps additionally uses four hunters. It is unclear whether subagent turns would count toward the same limit. A dollar cost analysis here could also have been useful . The paper does not report around tokens, API cost, reference executions, or actual runtime utilization.
-
I am not fully convinced with the explicit progress signal helping argument since the correlations are computed between six checkpoint averages across tasks, separately for each model and not between progress and correctness within individual runs. It could happen that in many cases a particular program’s visible score improves while hidden score worsens; we need more instance level metrics as well. Additionally, I find that the paper does not provide enough qualitative examples to more clearly understand what types of tests are being generated and get an intuition of how that ends up helping the model answer correctly. Those would be useful to get better intuition.
Questions for authors
- Could you report the per-task agreement between changes in corpus and hidden-test scores using the signal actually available during execution.
- How frequently does the corpus score increase while hidden-test performance decreases within a run? Does choosing checkpoints by corpus score outperform simply choosing the latest checkpoint
- the 1,000-turn budget includes all hunter calls? Please report aggregate tokens, cost, reference queries, and runtime for each method, and clarify the baseline subagent budgets
- Could you please add more qualitative examples around different trajectories and motivate how progress signals help.