acceptodds
Under review as a conference paper at ICLR 2027

Self-Improved Intent Alignment via Post Hoc Grounding for Repository Generation

15 views

Abstract

Benefiting from increasingly capable large language models (LLMs), coding agents are able to tackle increasingly complex programming tasks. However, existing agents still face significant limitations in end-to-end repository generation. In particular, fine-grained implementation intents described in large-scale task specifications often lack sufficient salience, especially over long-horizon generation trajectories, leading to discrepancies between the generated repositories and the intended functionality. In this paper, we propose SIRA, a runtime augmentation framework that mitigates specification drift by re-grounding coding agents in task intents at file-creation checkpoints. SIRA combines self-improved key intent extraction with retrieval-driven post hoc grounding. It derives transferable extraction guidance from discrepancies observed between task specifications and their generated repositories in prior repository-generation tasks and uses this guidance to construct a compact index of behaviorally critical intents for the current task. As each new file is created, SIRA retrieves the file-relevant intents and injects them into the agent's context, enabling local inconsistencies to be identified and corrected before they propagate. We evaluate SIRA on two representative repository-generation benchmarks, NL2Repo and RepoGenesis. Experimental results show that SIRA consistently achieves the best performance on the primary metrics, surpassing the strongest baseline by an average of 5.28 points in Score on NL2Repo and 9.63 points in Pass@1 on RepoGenesis.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Coding Agents, Repository Generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5NxNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

In long-horizon repository generation tasks, fine-grained implementation intents (such as precise values, boundary conditions, and output schemas) are easily forgotten during multi-file generation or overwritten by default programming conventions, leading to discrepancies between the generated code repository and the natural language task specification. This paper leverages violation experiences observed from prior tasks in the same domain to prioritize key intents. Whenever the agent successfully creates a new file, it constructs a query based on the structural features of the file, hybrid-retrieves relevant files and intents, and injects them into the environment observations to guide the agent in performing local self-correction before errors propagate. Evaluations were conducted on two representative repository-generation benchmarks, NL2Repo-Bench and RepoGenesis. Experimental results show that SIRA achieves the best primary performance metrics across all settings with limited additional API costs.

Critical strengths
  1. Through a detailed motivation study, the paper points out that a large portion of repository generation failures stem from "wrong implementation" or "adopted conflicting conventions" rather than complete omission. SIRA grasps this essence by injecting targeted intents at file creation checkpoints, achieving low-cost and efficient local self-correction without interfering with the agent's early generation or requiring an external validator.
  2. Rather than simply feeding the entire specification into a retriever, SIRA induces and extracts guidance by offline analyzing violations in historical tasks within the same domain. Ablation experiments and domain analysis show that this experience can be effectively transferred across tasks, significantly improving the quality of key intent extraction.
Critical weaknesses
  1. SIRA's post-hoc validation mechanism is strictly triggered by the "successful creation of a new file." However, in many practical development scenarios, major logic corrections, interface adjustments, or constraint violations may occur during repeated modifications, refactoring of existing files, or pure configuration file updates. If the agent does not trigger a new file creation, or if the violation of key intents is hidden in subsequent continuous modifications to existing files, SIRA cannot intervene. Although the paper counts the proportion of modifying related files, it lacks a systematic resolution mechanism for deviations caused by "non-file-creation actions."
  2. SIRA's self-improvement relies heavily on the collection and analysis of historical tasks in the same domain. For completely emerging long-horizon generation tasks that lack historical data or are difficult to clearly demarcate into domains, this experience induction module may fail or degrade to the version without experience. The paper's heavy reliance on domain partitioning limits its plug-and-play applicability in general, open-ended software engineering scenarios to some extent.
  3. There is a lack of comparison and analysis with related repository generation works, including repository generation methods in JavaBench and RealBench.
Questions for authors
  1. In the retrieval phase, the file-side query only extracts the public interfaces of the file while ignoring the specific implementation details inside the function bodies. If the agent introduces complex logic that violates the intent inside a function body while keeping its signature unchanged, can the current retrieval and grounding mechanism still effectively match and push the correct intent to the agent? Have you considered combining function body summaries or code change diffs for more precise retrieval?
  2. Current evaluations focus primarily on the Python language and two specific benchmarks (NL2Repo and RepoGenesis). Considering that different programming languages (such as statically typed languages like Java or C++) have stricter compile-time checks and different code organization paradigms, do SIRA's file checkpoint trigger strategy and query construction methods need to be refactored for strongly typed languages? When extending it to other languages in the future, what do you anticipate the biggest challenges to be?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 5NxN

Thank you for your thoughtful comments and questions. We address them individually below.

W1. Grounding Beyond File Creation

Your concern is valid. Violations of key intents can indeed arise during subsequent modifications to existing files, and SIRA does not inject intents during these modifications. However, we do not extend grounding to subsequent edits for two reasons.

First, in end-to-end repository generation, most delivered code is already established when the corresponding file is first created. We analyzed OpenHands with GPT-5 trajectories from 22 evaluation tasks per benchmark. The Files column reports the number of implementation and configuration files in the repositories. Initial Content Retention (ICR) measures the proportion of final nonempty lines retained from the initial file version, while Later Additions and Revisions (LAR) measures the proportion added or rewritten afterward. Both metrics are averaged equally across repositories.

BenchmarkFilesICR (%)LAR (%)
NL2Repo-Bench33095.084.92
RepoGenesis21793.666.34

Second, injecting intents during subsequent edits could lead to repeated revision and inspection, causing the agent to focus excessively on local optimization. Actually, during method development, we also experimented with post hoc grounding after all file editing operations, including creation and modification, but observed no clear performance improvement. As a result, we restrict grounding to file creation.

Revision: We will clarify the trigger rationale and add the analysis to the appendix.

W2. Dependence on Domain Experience

We understand your concern about the applicability of SIRA’s intent extraction guidance. However, even without historical experience, SIRA exceeds OpenHands by 9.66 and 5.54 percentage points on RepoGenesis and NL2Repo-Bench, respectively, with GPT-5 (Table 2). Historical experience complements post hoc grounding by helping prioritize requirements prone to implementation errors and further improves performance. Moreover, in open-ended software engineering scenarios without historical experience, manually authored expert guidance could serve as an alternative.

W3. Comparison with Repository Generation Methods

Table 1 reports comparisons against CodeS alongside six coding agents. CodeS is a dedicated repository-generation method that constructs repository and file sketches before completing implementations, as detailed in Appendix E.2.

The works you highlight provide additional methodological perspectives. JavaBench studies synthesis within supplied Java skeletons, while RealBench studies generation guided by supplied UML designs. Both evaluate implementation under predefined structures. Adapting them to our setting would introduce an additional design stage whose quality would also affect the results. CodeS already incorporates this stage and therefore provides a more direct comparison for generation from task specifications. SIRA complements such generation pipelines by improving coding agents for end-to-end repository generation through experience-guided intent extraction and retrieval-driven post hoc grounding.

Revision: We will cite both works and expand the discussion of their methodological differences and relationship to SIRA.

Q1. Retrieval of Intents Relevant to Function Bodies

At file creation checkpoints, SIRA can retrieve intents relevant to the function bodies you describe through their associated implementation responsibilities. SIRA uses file paths, imports, declarations, and decorators to identify a file’s responsibilities and match them against intent ownership anchors. For example, in the math-verify case in Appendix D, the private helper _extract_latex_segments incorrectly treats boxed_match_priority = -1 as an ordinary priority, although the specification requires this value to disable boxed extraction. Although neither the helper nor its body appears in the query, the relevant behavioral intent is retrieved and delivered through its anchor to the public class LatexExtractionConfig.

Regarding your suggestion for more precise retrieval, function body summaries and diffs are reasonable extensions, but we expect limited benefits. Summaries describe what the code does and may omit what it should do but does not. SIRA therefore emphasizes matching implementation responsibilities with critical intents. Beyond refining retrieval through function body summaries, directly identifying intent violations is a promising direction for our future work. Diffs are more relevant to subsequent edits, but face the limitations discussed in our response to W1. We therefore expect that incorporating them would offer limited performance gains.

AuthorsOpenReview ↗

Response to Reviewer 5NxN

Q2. Extension to Other Programming Languages

Thank you for your constructive question. SIRA has so far been evaluated only on Python tasks from two benchmarks. Its core mechanisms extract key intent, retrieve intents relevant to code responsibilities, and correct deviations during generation. These mechanisms do not depend on Python or dynamic typing and are applicable across languages.

For query construction, extending SIRA primarily requires replacing the language frontend without redesigning the retrieval framework. Current Python queries describe file responsibilities using file paths, imports, declarations, and decorators (§2.2 and Appendix D). Java packages, classes, interfaces, inheritance relationships, and annotations, as well as C++ namespaces, classes, overloaded signatures, and template parameters, could be mapped to a unified responsibility representation.

The checkpoint policy requires reassessment for the target language’s code organization and generation workflow. When interfaces or skeletons precede implementations, or C++ declarations and implementations reside in separate files, initial file creation may provide insufficient responsibility information. We therefore need to evaluate triggering checks when a meaningful interface or implementation first emerges and constructing queries from related declaration and implementation files, while bounding the number of checks. These refinements could also benefit Python tasks.

We expect the main challenge to be determining when sufficient information becomes available and which task intents are relevant as implementations emerge across files and generation stages, while preserving timely correction and low overhead.

Revision: We will add an appendix section discussing extensions to other programming languages.

Area_Chair_G9DoOpenReview ↗

Reminder from AC

Dear Reviewers,

As the ICLR discussion period is closing, please take a moment to read the authors’ responses carefully and let them know which of your concerns have been addressed and which questions remain open. If anything needs clarification or would benefit from further discussion, please raise it and engage with the authors and fellow reviewers while there’s still time.

Thank you for your time, your thoughtful reviews, and your service to the ICLR community.

Best Regards, AC

Reviewer_77wVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SIRA, a runtime add-on for coding agents that build a whole repository from a natural language specification. Before generation, SIRA extracts a compact index of key intents from the specification, guided by domain-level directives that an LLM derives from failures in previously generated repositories; during generation, each newly created file triggers retrieval of related intents, which are appended to the agent's observation so the agent can check and revise the file. On subsets of NL2Repo-Bench and RepoGenesis with GPT-5 and DeepSeek-V4-Pro, SIRA built on OpenHands improves the primary metrics over OpenHands by 8.5 to 11.8 points and outperforms the other evaluated agents.

Critical strengths

(S1) Informative motivation study. The manual analysis of failed benchmark tests separates failures the specification does not determine from failures that violate it, and further splits violations into omissions, adopted conventions, and wrong implementations (App. A). This analysis is useful on its own and directly motivates the design.

(S2) Practical design with clear ablations. SIRA does not modify the base agent and adds about 13% cost on average; it also transfers to OpenCode as a plugin with similar gains (App. I). The ablations separate what to inject from when to inject it, and the deferred review variants are given a larger iteration budget, which makes the comparison favor them rather than SIRA.

Critical weaknesses

(W1) Limited novelty. Inducing natural language guidance from previously completed tasks and reusing it on new tasks is what ExpeL [1] and Agent Workflow Memory [2] do, and neither is cited. The guidance is built once offline and never updated, so the term "self-improved" overstates what the method does. The grounding trigger, which adds context right after a file is created, is also already supported by commercial agents, for example through PostToolUse hooks in Claude Code [3]; SIRA mainly differs in what it injects. What remains new is mostly the design of the intent index.

(W2) Cold start and generality of the guidance. The experience guidance needs completed tasks from the same domain, domain labels (which the authors assigned by hand for NL2Repo-Bench, App. E.4), and a manually written description of each domain. Each domain's guidance is induced only from that domain's analysis tasks, and some domains have very few: the Machine Learning guidance has 22 directives induced from five tasks and is evaluated on two, yet this domain shows the largest gain from experience (+9.88, Fig. 4). Since the analysis and evaluation tasks are drawn from the same benchmark, some directives look like benchmark conventions rather than general knowledge, e.g., which status code to return for validation errors, or using only the framework's own start command. Whether guidance induced from same-domain tasks outside the benchmark still helps is not tested, and there is no curve showing how the gain changes with the number of analysis tasks. It is also unclear which agent and backbone produced the analysis repositories, and whether the same guidance was used for the DeepSeek-V4-Pro runs.

(W3) Baselines. No Claude model is evaluated, even though Claude Sonnet 4.5 with Claude Code is the best system reported in the NL2Repo-Bench paper (40.2%) [4]. Claude Code is only run with DeepSeek-V4-Pro through a protocol bridge, and GPT-5 is left out for API reasons. However, GPT-5 can also be run in Claude Code through an Anthropic-compatible gateway such as LiteLLM, which is essentially the same kind of bridge the authors built for DeepSeek-V4-Pro. It is therefore unclear whether the gains hold for the strongest systems on these benchmarks.

(W4) Metrics. On RepoGenesis, the paper reports "Pass@1" as the pass rate of hidden API tests, whereas the RepoGenesis paper defines Pass@1 at the repository level, counting a repository as solved only if it passes all tests [5]. Because the abstract and main results use the same name, readers may take these numbers (e.g., 67.43% for OpenHands with GPT-5) as the official metric, even though they are not comparable with published results such as the 23.67% best result in the original paper. The number of fully passing repositories, which both original papers report [4, 5], is also missing.

[1] A. Zhao, D. Huang, Q. Xu, M. Lin, Y.-J. Liu, and G. Huang. ExpeL: LLM Agents Are Experiential Learners. AAAI 2024.

[2] Z. Z. Wang et al. Agent Workflow Memory. ICML 2025.

[3] Anthropic. Claude Code Hooks Reference. https://docs.claude.com/en/docs/claude-code/hooks

[4] J. Ding et al. NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents. ICML 2026.

[5] Z. Peng et al. RepoGenesis: Benchmarking End-to-End Microservice Generation from Readme to Repository. ACL 2026.

Questions for authors
  • Q1. How does SIRA's cross-task guidance differ from ExpeL and Agent Workflow Memory in how it is induced and used, and can the guidance be updated as new tasks are completed?
  • Q2. How does the gain change with the number of analysis tasks per domain, and does guidance induced from same-domain tasks outside the benchmark still help? Which agent and backbone produced the analysis repositories, and was the same guidance used for both backbones?
  • Q3. How do SIRA and the baselines perform with Claude models, and with GPT-5 in Claude Code through a gateway? Can SIRA be attached to Claude Code through hooks, as was done for OpenCode?
  • Q4. Can the authors report Pass@1 as defined by RepoGenesis (the fraction of repositories passing all tests), the number of fully passing repositories on NL2Repo-Bench, and results of the baselines and the w/o experience variant on the full NL2Repo-Bench?
  • Q5. How much of the NL2Repo-Bench gain comes from packaging fixes such as dependency declarations, as opposed to correcting the behavioral deviations (adopted conventions and wrong implementations) that motivate the method?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 77wV

Thank you for your detailed comments on novelty, experience guidance, baseline comparisons, and evaluation metrics.

W1/Q1. Novelty and Experience Updating

A1. Thank you for highlighting these related works. ExpeL derives insights from successful and failed trajectories and retrieves successful examples, while Agent Workflow Memory abstracts reusable workflows from successful trajectories. SIRA instead consolidates discrepancies between specifications and delivered repositories across tasks into domain-specific guidance for prioritizing critical intents in new specifications, whereas ExpeL and Agent Workflow Memory reuse trajectory-derived insights, examples, or workflows to guide task execution.

As defined in Section 2.1, self-improved refers to improving intent extraction using guidance automatically derived from prior implementation discrepancies. SIRA thereby converts historical specification violations into selective attention to requirements in subsequent tasks. Regarding your question about updating, newly completed tasks can undergo the same analysis, and their findings can be merged with existing findings to regenerate the domain guidance.

In terms of novelty, SIRA introduces a novel specification-grounding mechanism for long-horizon repository generation that complements planning and correction through execution feedback. Hooks provide a convenient implementation interface. However, the methodological contribution lies in a unified runtime augmentation framework that determines which requirements to resurface, associates them with newly created files, and uses them to guide local correction before deviations propagate. This coordinated design extends beyond the intent index alone and operates without modifying the underlying agent or requiring an external verifier.

Revision: We will cite ExpeL and Agent Workflow Memory and explain these distinctions in the method design section, clarify how guidance can be refreshed, and make SIRA’s contribution more explicit.

W2/Q2. Experience Dependence and Generalization

A2. Thank you for your comments and questions on these issues. We address your concerns individually.

  1. Repository generation benchmarks contain relatively few tasks, and Appendix E.4 (pp. 23–24, Table 7) explains our task selection and partitioning. A single implementation error can affect many tests, introducing variability into task-level evaluation results. The two tasks with particularly large gains happen to belong to the ML domain.

  2. SIRA is explicitly designed to avoid tailoring its guidance to benchmark-specific conventions through constraints on both induction inputs and prompts. Experience induction uses only task specifications and generated repositories, excluding benchmark tests and test results. The prompt explicitly requires generalization beyond individual records and prohibits copying task names, APIs, frameworks, or concrete values. These constraints encourage transferable extraction guidance.

  3. Your question about the number of analysis tasks requires accounting for randomness in experience analysis, guidance generation, and agent execution. With few tasks per domain, subset composition also changes the error types covered. Reliable estimation therefore requires repeated subset sampling, guidance generation, and evaluation at each sample size. Simply reducing the task set to a smaller fixed subset may not support robust conclusions. We have not conducted this systematic analysis, although our ablations support the contribution of experience guidance under the existing analysis sets.

  4. To address your concern about generalization, we have conducted supplementary experiments evaluating the transfer of guidance between Web tasks in RepoGenesis and NL2Repo-Bench using GPT-5, retaining the original generation settings. Each target uses guidance induced exclusively from the other benchmark's analysis tasks. The results below reflect the generalization of SIRA's experience guidance across benchmarks within related domains, with positive gains in both transfer directions.

    Evaluation targetGuidance sourceGainRetained gain
    RepoGenesisNo experience0.00N/A
    RepoGenesisRepoGenesis+2.09100.0%
    RepoGenesisNL2Repo-Bench Web+1.1555.0%
    NL2Repo-Bench WebNo experience0.00N/A
    NL2Repo-Bench WebNL2Repo-Bench Web+6.26100.0%
    NL2Repo-Bench WebRepoGenesis+4.4270.6%
  5. OpenHands generated the analysis repositories separately with each backbone. SIRA derives separate guidance for each backbone, which then uses its own guidance for subsequent repository generation. GPT-5 and DeepSeek-V4-Pro therefore do not share the same guidance.

Revision: We will add the supplementary generalization experiments and their analysis to the appendix and clarify that each backbone uses separate guidance derived from analysis repositories.

AuthorsOpenReview ↗

Response to Reviewer 77wV

W3/Q3. Stronger Baselines and Claude Code Integration

A3. Following your suggestion, we conduct supplementary experiments using Claude Sonnet 4.5 across agents and GPT-5 with Claude Code under the same task split and budget.

RepoGenesis

ModelAgentAve. TPR (%)DSR (%)AC (%)Cost ($)Fully passing (%)
GPT-5Claude Code66.91±0.9166.91\pm0.9193.94±2.6293.94\pm2.6295.08±0.6695.08\pm0.660.4934.85±2.6234.85\pm2.62
Claude Sonnet 4.5Mini SWE Agent65.18±0.7665.18\pm0.7687.88±2.6287.88\pm2.6296.21±0.6696.21\pm0.660.6533.33±2.6233.33\pm2.62
Claude Sonnet 4.5OpenCode61.32±0.6861.32\pm0.6890.91±0.0090.91\pm0.0094.83±2.2394.83\pm2.230.4734.85±2.6234.85\pm2.62
Claude Sonnet 4.5OpenHands66.63±0.9466.63\pm0.9495.45±0.0095.45\pm0.0096.21±0.0096.21\pm0.000.6039.39±2.6239.39\pm2.62
Claude Sonnet 4.5Claude Code67.84±0.8367.84\pm0.8393.94±2.6293.94\pm2.6296.11±1.4996.11\pm1.490.6440.91±4.5540.91\pm4.55
Claude Sonnet 4.5Copilot60.91±1.3260.91\pm1.3289.39±2.6289.39\pm2.6296.97±0.6696.97\pm0.660.5136.36±0.0036.36\pm0.00
Claude Sonnet 4.5SIRA79.06±0.8779.06\pm0.8798.48±2.6298.48\pm2.6296.59±0.0096.59\pm0.000.6745.45±4.5545.45\pm4.55

NL2Repo-Bench

Fully passing reports the mean number of repositories passing all tests per run across the 22 evaluation tasks.

ModelAgentAve. Score (%)Cost ($)Fully passing
GPT-5Claude Code36.48±0.4836.48\pm0.480.980.33±0.580.33\pm0.58
Claude Sonnet 4.5Mini SWE Agent35.76±0.3635.76\pm0.362.060.67±0.580.67\pm0.58
Claude Sonnet 4.5OpenCode38.48±0.5738.48\pm0.570.930.00±0.000.00\pm0.00
Claude Sonnet 4.5OpenHands37.19±0.4937.19\pm0.491.280.00±0.000.00\pm0.00
Claude Sonnet 4.5Claude Code39.57±0.4639.57\pm0.461.220.67±0.580.67\pm0.58
Claude Sonnet 4.5Copilot40.08±0.3840.08\pm0.381.010.33±0.580.33\pm0.58
Claude Sonnet 4.5SIRA46.51±0.4146.51\pm0.411.510.67±0.580.67\pm0.58

These results indicate that SIRA outperforms the strongest baseline on each benchmark by 11.22 percentage points in RepoGenesis Ave. TPR and 6.43 percentage points in NL2Repo-Bench Ave. Score, while maintaining its performance advantage across different model backbones.

SIRA can be adapted through PostToolUse hooks that detect successful file creation and return retrieved intents to the agent. The existing OpenCode integration already improves the two metrics by 14.27 and 8.17 points with GPT-5, supporting applicability beyond OpenHands.

Revision: We will add the specific results and analysis of the supplementary experiments to the paper.

W4/Q4. Metric Definitions and Evaluation Coverage

A4. Thank you for pointing out the issue with our metric naming. We will rename our RepoGenesis metric Ave. TPR throughout the paper. For NN repositories, with pip_i passed tests among tit_i required tests, the distinction is

\mathrm{Pass@1}=\frac{100}{N}\sum_{i=1}^{N}\mathbf{1}[p_i=t_i].$$ We report the additional metrics below for the 22 evaluation tasks in each benchmark. NL2Repo-Bench Fully passing counts repositories passing all tests per run. As in Table 1, values are means and sample standard deviations over three independent runs. | Model | Agent | RepoGenesis Pass@1 (%) | NL2Repo-Bench Fully passing | |---|---|---|---| | GPT-5 | Mini SWE Agent | $34.85\pm2.62$ | $0.00\pm0.00$ | | GPT-5 | OpenCode | $34.85\pm2.62$ | $0.00\pm0.00$ | | GPT-5 | OpenHands | $36.36\pm0.00$ | $0.33\pm0.58$ | | GPT-5 | Codex | $31.82\pm4.55$ | $0.00\pm0.00$ | | GPT-5 | Copilot | $34.85\pm2.62$ | $0.00\pm0.00$ | | GPT-5 | CodeS | $12.12\pm2.62$ | $0.00\pm0.00$ | | GPT-5 | SIRA | $42.42\pm2.62$ | $0.67\pm0.58$ | | DeepSeek-V4-Pro | Mini SWE Agent | $31.82\pm0.00$ | $0.67\pm0.58$ | | DeepSeek-V4-Pro | OpenCode | $31.82\pm0.00$ | $0.33\pm0.58$ | | DeepSeek-V4-Pro | OpenHands | $37.88\pm2.62$ | $0.33\pm0.58$ | | DeepSeek-V4-Pro | Claude Code | $39.39\pm2.62$ | $0.67\pm0.58$ | | DeepSeek-V4-Pro | Copilot | $34.85\pm2.62$ | $0.00\pm0.00$ | | DeepSeek-V4-Pro | CodeS | $15.15\pm2.62$ | $0.00\pm0.00$ | | DeepSeek-V4-Pro | SIRA | $43.94\pm2.62$ | $0.67\pm0.58$ | SIRA achieves the highest RepoGenesis Pass@1 with both backbones. Fully passing repositories remain rare on NL2Repo-Bench across all methods, indicating substantial room for improvement in complete repository correctness. Revision: We will correct the metric name throughout the paper and report the results for repositories passing all tests on both benchmarks.
AuthorsOpenReview ↗

Response to Reviewer 77wV

Q5. Sources of Performance Gains

A5. To address your concern, Table 2 provides a relevant control through the w/o task-specific content variant. It retains the file creation checkpoints, provides generic review reminders, and preserves the instruction to defer installation, deployment, and execution until repository generation is complete.

On NL2Repo-Bench, generic reminders improve the average test pass rate from 31.74% to 33.65%, whereas SIRA reaches 41.84%, yielding gains of 1.91 and 10.10 percentage points, respectively. Complementary evidence comes from RepoGenesis, where SIRA exceeds generic reminders by only 3.03 points in deployment success but by 10.57 points in API test pass rate. These results suggest that SIRA’s gains extend beyond packaging and deployment fixes to correcting behavioral deviations from the specification.

More broadly, our approach focuses on aligning generated code with the intents expressed in the specification, rather than specifically optimizing for higher scores on benchmark tests. To this end, we also provide a concrete example (Figure 5) to illustrate how our approach improves specification alignment.

Reviewer_jg99OpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces SIRA, a "runtime augmentation framework that mitigates specification drift by re-grounding coding agents in task intents at file-creation checkpoints". The idea is to first extract a compact set of "key intents" from the spec, and then, whenever the agent creates a new file, retrieve the relevant intents and remind the agent so it can check and revise the file before mistakes propagate. Experiments on two repo-generation benchmarks show consistent gains over the base agent and several other coding agents at a similar cost.

Critical strengths
  • I really like the motivation analysis. Showing that many failures come from the agent drifting away from what the spec says (e.g., falling back to conventions, or writing self-tests that share the same misreading) is convincing and makes the problem very concrete.
  • The method is simple and neat. Grounding at file creation feels like a sensible checkpoint, and it's nice that it doesn't need a separate verifier and can be attached to other agents.
  • The ablations are cleanly designed around "what" to ground and "when" to ground, and the takeaway that end-of-run review is both worse and more expensive is practically useful.
Critical weaknesses

(Many of these are more suggestions for the paper's generalizability and systematic re: agent actions but imo not required.)

  • Agent decisions and drift. In general I'd love some more understanding on how the agent decisions may trigger drifts. For example, it would help to understand how the order of the agent's actions interacts with grounding. As I understand it, the reminder fires only when a file is first created, and intents that were already surfaced get filtered out. So the order in which the agent writes files kind of decides which intents show up where, and something surfaced for an early file may never come back for a later file where it actually matters. Some analysis on whether order matters (or whether re-triggering on bigger edits helps) would be helpful.

  • Generalizability. The paper could use some more discussion on how this generalizes to more realistic long-horizon settings where the spec isn't fixed upfront. In practice users often add or change requirements midway, but here the intents are extracted once before generation, and the experience part relies on having a pool of previously completed tasks in the same domain. It would be nice if the authors could at least discuss how SIRA would handle evolving specs or a brand new domain, even if a full evaluation is out of scope.

  • Examples of cross-task experience. The cross-task experience part could use more concrete examples in the main paper; right now it's hard to picture what a learned "directive" looks like.

  • Writing. The intro is a bit dense: the motivation study, two large figures, and three challenges all come before the method. I feel the analysis could work better as its own Section 2, with the intro keeping just one example and the key takeaway.

Questions for authors

Could you please reflect a bit on the "Agent decisions and drift" and "Generalizability" points in the weakness section?

2 comments
AuthorsOpenReview ↗

Response to Reviewer jg99

Thank you for your positive assessment and constructive suggestions on agent decisions, generalizability, and presentation.

W1. Agent decisions and drift

A1. We agree with your observation that action order affects grounding. We clarify that each intent can be injected multiple times, up to Nmax=4N_{max}=4 (Appendix D, Table 5). Typically, only broad intents, such as installation and packaging configuration, require repeated injection, while implementation-specific intents can resurface at appropriate points within this limit.

Regarding grounding triggers, we restrict intent injection to file creation as a design trade-off between alignment effectiveness and intervention overhead. First, this choice is motivated by our observation that most final content is already present when a file is first created. We analyzed differences between initial and final file content using OpenHands with GPT-5 trajectories from 22 evaluation tasks on each benchmark. The Files column reports the number of implementation and configuration files in the repositories. Initial Content Retention (ICR) measures the proportion of final nonempty lines retained from the initial file version. Later Additions and Revisions (LAR) measures the proportion added or rewritten afterward. Both metrics are averaged equally across repositories.

BenchmarkFilesICR (%)LAR (%)
NL2Repo-Bench33095.084.92
RepoGenesis21793.666.34

Second, injecting intents during edits could lead to repeated revision and inspection. Given the limited expected benefit of extending injection to subsequent edits, we restrict injection to file creation.

Revision: We will clarify these design choices in the method section and discuss this issue in a new Weaknesses section. We can also provide the analysis and statistical details of differences between initial and final file content in the appendix.

W2. Generalizability

A2. We agree with you that evolving specifications and new domains require further consideration. For evolving specifications, SIRA could incrementally update the intent index and trigger targeted grounding for existing files affected by changed requirements, while using the updated index for subsequent file creation. For a new domain without historical tasks, SIRA can extract intents directly from the specification and retain runtime grounding, as in our variant without experience. Table 2 in Section 3.3 shows that this variant also performs well. Experts could also manually develop intent extraction guidelines tailored to the specification style and task characteristics to extend SIRA to new domains.

Revision: We will discuss SIRA’s generalizability in a new Weaknesses section and outline approaches for handling evolving specifications and new domains in the appendix.

W3/W4/Q1. Examples, presentation, and limitations

A3. We already provide the cross-task experience generated by SIRA using GPT-5 in Appendix G, but acknowledge its omission from the main text. We will follow your suggestions on the introduction’s organization and the Weaknesses section.

Revision: We will include representative directives and explain their effects on intent extraction in the main text, move the detailed motivation analysis to a separate section, and add a Weaknesses section discussing action decisions, evolving specifications, and reliance on domain experience. These revisions will clarify the existing content without substantially changing the paper.

Area_Chair_G9DoOpenReview ↗

Reminder from AC

Dear Reviewers,

As the ICLR discussion period is closing, please take a moment to read the authors’ responses carefully and let them know which of your concerns have been addressed and which questions remain open. If anything needs clarification or would benefit from further discussion, please raise it and engage with the authors and fellow reviewers while there’s still time.

Thank you for your time, your thoughtful reviews, and your service to the ICLR community.

Best Regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.