acceptodds
Under review as a conference paper at ICLR 2027

Manufacturing a Progress Signal: Self-Evaluating Coding Agents for Long-Horizon Program Synthesis from Scratch

Abstract

Coding agents are increasingly undertaking long-horizon software development tasks that require hours of work. Program synthesis from scratch, as studied in ProgramBench, is particularly challenging when agents must infer behavior from usage documentation and an executable reference. Existing agents may lack concrete repair targets, lose track of explored and untested behaviors, and struggle to obtain trustworthy and timely progress feedback. We introduce , a framework that enables coding agents to manufacture and update an explicit progress signal from executable tests grounded in reference behavior. The framework preserves tests in a persistent corpus to provide reproducible repair targets and regression checks, while an obligation ledger guides further probing toward untested documented behaviors. A new test selection strategy controls redundancy and corpus growth through behavioral classes, separate quotas for divergent and regression tests, and explicit admission and corpus limits. When results are averaged equally across the four model configurations, achieves a mean hidden-test pass rate of 44.49%, compared with 35.58% for MSA-full, 32.63% for Claude Code, and 39.17% for SpecFirst. These correspond to relative improvements of 25.02%, 36.35%, and 13.58%, respectively. Ablation studies support the contributions of corpus selection and the obligation ledger. Retrospective evaluations of candidate snapshots further present positive correlations between corpus and hidden-test pass rates, each averaged across tasks. These results indicate that the manufactured signal tracks improvements in average held-out performance during program synthesis. The code is available at https://anonymous.4open.science/r/codemaps-4F5B/https://anonymous.4open.science/r/codemaps-4F5B/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
coding agent, program synthesis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_m3zAOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose CODEMAPS, a framework that enables coding agents to maintain an explicit progress tracker and update it based on running explicit test cases generated. The authors argue that this is particularly useful for long-horizon tasks where the agents need to scope out a lot of work and may diverge at some point in time. The authors provide results on a benchmark ProgramBench where they outperform other baselines such as MSA-full, Claude Code, Spec-First. The authors also perform ablations to tease apart contributions of different components in their system and also find that the performance on generated test cases is highly positively correlated with the performance on hidden tests.

Critical strengths
  • The core problem of providing a persistent checklist and grounding in long-agent horizon tasks is quite relevant and interesting. In that regard the work takes a meaningful step in that direction and provides impressive gains across different model configurations and wide variety of task difficulties.
Critical weaknesses
  • Compute fairness: While the authors mention that all models receive six hours and 1000 turns, CodeMaps additionally uses four hunters. It is unclear whether subagent turns would count toward the same limit. A dollar cost analysis here could also have been useful . The paper does not report around tokens, API cost, reference executions, or actual runtime utilization.

  • I am not fully convinced with the explicit progress signal helping argument since the correlations are computed between six checkpoint averages across tasks, separately for each model and not between progress and correctness within individual runs. It could happen that in many cases a particular program’s visible score improves while hidden score worsens; we need more instance level metrics as well. Additionally, I find that the paper does not provide enough qualitative examples to more clearly understand what types of tests are being generated and get an intuition of how that ends up helping the model answer correctly. Those would be useful to get better intuition.

Questions for authors
  • Could you report the per-task agreement between changes in corpus and hidden-test scores using the signal actually available during execution.
  • How frequently does the corpus score increase while hidden-test performance decreases within a run? Does choosing checkpoints by corpus score outperform simply choosing the latest checkpoint
  • the 1,000-turn budget includes all hunter calls? Please report aggregate tokens, cost, reference queries, and runtime for each method, and clarify the baseline subagent budgets
  • Could you please add more qualitative examples around different trajectories and motivate how progress signals help.
Reviewer_6UoiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents CodeMaps, a framework that enables coding agents with process signal and tests derived from reference software. Although coding agents demonstrated significant performance improvements, they still struggle when developing software entirely from scratch. The authors propose CodeMaps to address three limitations: 1) insufficient signal during software development, 2) difficulty of maintaining exploration with diverse tests, and 3) a lack of reliable internal feedback signals during software development.

CodeMaps consists of obligation ledger, divergence hunter agents, along with algorithm to select tests from growing test corpus. The obligation ledger searches document for untested functionality, while divergence hunter agents generates new tests based on untested functionality from the obligation set. Test cases are then grouped in behavioral classes then solves the optimization problem to select the test to maximize the class weight as defined in equation (4).

Empirical results on ProgramBench demonstrate that CodeMaps consistently outperforms existing baselines, such as MSA-full, Claude Code, SpecFirst on ProgramBench under diverse models.

Critical strengths
  1. This paper addresses important and timely problem in long-horizon software development with agents. As software related tasks get more complex, obtaining reliable signal is crucial part of the development process. CodeMap provides a practical and straightforward approach by introducing additional subagents with selection algorithm.

  2. The paper presents a consistent empirical improvement, with different baselines, models, and ablation studies. Furthermore, this method can easily be applicable to different coding agents without requiring additional model training.

Critical weaknesses
  1. The methodological novelty is limited. The paper combines already established techniques such as differential testing, test coverage, and test selection. Although the integration is practically useful, the paper proposed a limited algorithmic novelty.

  2. The problem setting is limited. This method requires a reference program, which is unlikely available in the practical scenario. This assumption limits the generalizability of the proposed framework.

Questions for authors
  1. CodeMap introduces additional components. Could the authors specify a detailed comparison of computational costs (e.g. API calls, execution overhead, etc.)

  2. The experiment is conducted with fixed 300 tests. This seems quite important as maintaining more tests may improve the coverage while increasing test computation costs. Is there any comparison or analysis regarding the size of the test corpus?

  3. Is there additional evaluation on different benchmarks? The paper evaluates on one benchmark (ProgramBench) which seems limited. Although, SWE-Bench targets different type of software development agentic task, such evaluation would help demonstrate the generalizability.

  4. The proposed framework relies on the reference program. Is this setting practical?

  5. Could the authors provide a more analysis on the reliability of the proposed signals in Section 3.4? Is the correlation calculated with the tests used during the development process or is the pass rate calculated with the fixed test set for all six checkpoints? As I understand it, the evaluation is conducted with same test sets for all checkpoints, but this does not necessarily reflect the reliability of the progress signals during the actual development process.

  6. The comparison with existing methods is limited. Prior works such as SWT-Bench, CURE, UTRL addresses automatic test generation. How does CodeMaps compare with previous methods?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.