- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
In long-horizon repository generation tasks, fine-grained implementation intents (such as precise values, boundary conditions, and output schemas) are easily forgotten during multi-file generation or overwritten by default programming conventions, leading to discrepancies between the generated code repository and the natural language task specification. This paper leverages violation experiences observed from prior tasks in the same domain to prioritize key intents. Whenever the agent successfully creates a new file, it constructs a query based on the structural features of the file, hybrid-retrieves relevant files and intents, and injects them into the environment observations to guide the agent in performing local self-correction before errors propagate. Evaluations were conducted on two representative repository-generation benchmarks, NL2Repo-Bench and RepoGenesis. Experimental results show that SIRA achieves the best primary performance metrics across all settings with limited additional API costs.
Critical strengths
- Through a detailed motivation study, the paper points out that a large portion of repository generation failures stem from "wrong implementation" or "adopted conflicting conventions" rather than complete omission. SIRA grasps this essence by injecting targeted intents at file creation checkpoints, achieving low-cost and efficient local self-correction without interfering with the agent's early generation or requiring an external validator.
- Rather than simply feeding the entire specification into a retriever, SIRA induces and extracts guidance by offline analyzing violations in historical tasks within the same domain. Ablation experiments and domain analysis show that this experience can be effectively transferred across tasks, significantly improving the quality of key intent extraction.
Critical weaknesses
- SIRA's post-hoc validation mechanism is strictly triggered by the "successful creation of a new file." However, in many practical development scenarios, major logic corrections, interface adjustments, or constraint violations may occur during repeated modifications, refactoring of existing files, or pure configuration file updates. If the agent does not trigger a new file creation, or if the violation of key intents is hidden in subsequent continuous modifications to existing files, SIRA cannot intervene. Although the paper counts the proportion of modifying related files, it lacks a systematic resolution mechanism for deviations caused by "non-file-creation actions."
- SIRA's self-improvement relies heavily on the collection and analysis of historical tasks in the same domain. For completely emerging long-horizon generation tasks that lack historical data or are difficult to clearly demarcate into domains, this experience induction module may fail or degrade to the version without experience. The paper's heavy reliance on domain partitioning limits its plug-and-play applicability in general, open-ended software engineering scenarios to some extent.
- There is a lack of comparison and analysis with related repository generation works, including repository generation methods in JavaBench and RealBench.
Questions for authors
- In the retrieval phase, the file-side query only extracts the public interfaces of the file while ignoring the specific implementation details inside the function bodies. If the agent introduces complex logic that violates the intent inside a function body while keeping its signature unchanged, can the current retrieval and grounding mechanism still effectively match and push the correct intent to the agent? Have you considered combining function body summaries or code change diffs for more precise retrieval?
- Current evaluations focus primarily on the Python language and two specific benchmarks (NL2Repo and RepoGenesis). Considering that different programming languages (such as statically typed languages like Java or C++) have stricter compile-time checks and different code organization paradigms, do SIRA's file checkpoint trigger strategy and query construction methods need to be refactored for strongly typed languages? When extending it to other languages in the future, what do you anticipate the biggest challenges to be?
3 comments
Response to Reviewer 5NxN
Q2. Extension to Other Programming Languages
Thank you for your constructive question. SIRA has so far been evaluated only on Python tasks from two benchmarks. Its core mechanisms extract key intent, retrieve intents relevant to code responsibilities, and correct deviations during generation. These mechanisms do not depend on Python or dynamic typing and are applicable across languages.
For query construction, extending SIRA primarily requires replacing the language frontend without redesigning the retrieval framework. Current Python queries describe file responsibilities using file paths, imports, declarations, and decorators (§2.2 and Appendix D). Java packages, classes, interfaces, inheritance relationships, and annotations, as well as C++ namespaces, classes, overloaded signatures, and template parameters, could be mapped to a unified responsibility representation.
The checkpoint policy requires reassessment for the target language’s code organization and generation workflow. When interfaces or skeletons precede implementations, or C++ declarations and implementations reside in separate files, initial file creation may provide insufficient responsibility information. We therefore need to evaluate triggering checks when a meaningful interface or implementation first emerges and constructing queries from related declaration and implementation files, while bounding the number of checks. These refinements could also benefit Python tasks.
We expect the main challenge to be determining when sufficient information becomes available and which task intents are relevant as implementations emerge across files and generation stages, while preserving timely correction and low overhead.
Revision: We will add an appendix section discussing extensions to other programming languages.
Reminder from AC
Dear Reviewers,
As the ICLR discussion period is closing, please take a moment to read the authors’ responses carefully and let them know which of your concerns have been addressed and which questions remain open. If anything needs clarification or would benefit from further discussion, please raise it and engage with the authors and fellow reviewers while there’s still time.
Thank you for your time, your thoughtful reviews, and your service to the ICLR community.
Best Regards, AC
Response to Reviewer 5NxN
Thank you for your thoughtful comments and questions. We address them individually below.
W1. Grounding Beyond File Creation
Your concern is valid. Violations of key intents can indeed arise during subsequent modifications to existing files, and SIRA does not inject intents during these modifications. However, we do not extend grounding to subsequent edits for two reasons.
First, in end-to-end repository generation, most delivered code is already established when the corresponding file is first created. We analyzed OpenHands with GPT-5 trajectories from 22 evaluation tasks per benchmark. The Files column reports the number of implementation and configuration files in the repositories. Initial Content Retention (ICR) measures the proportion of final nonempty lines retained from the initial file version, while Later Additions and Revisions (LAR) measures the proportion added or rewritten afterward. Both metrics are averaged equally across repositories.
Second, injecting intents during subsequent edits could lead to repeated revision and inspection, causing the agent to focus excessively on local optimization. Actually, during method development, we also experimented with post hoc grounding after all file editing operations, including creation and modification, but observed no clear performance improvement. As a result, we restrict grounding to file creation.
Revision: We will clarify the trigger rationale and add the analysis to the appendix.
W2. Dependence on Domain Experience
We understand your concern about the applicability of SIRA’s intent extraction guidance. However, even without historical experience, SIRA exceeds OpenHands by 9.66 and 5.54 percentage points on RepoGenesis and NL2Repo-Bench, respectively, with GPT-5 (Table 2). Historical experience complements post hoc grounding by helping prioritize requirements prone to implementation errors and further improves performance. Moreover, in open-ended software engineering scenarios without historical experience, manually authored expert guidance could serve as an alternative.
W3. Comparison with Repository Generation Methods
Table 1 reports comparisons against CodeS alongside six coding agents. CodeS is a dedicated repository-generation method that constructs repository and file sketches before completing implementations, as detailed in Appendix E.2.
The works you highlight provide additional methodological perspectives. JavaBench studies synthesis within supplied Java skeletons, while RealBench studies generation guided by supplied UML designs. Both evaluate implementation under predefined structures. Adapting them to our setting would introduce an additional design stage whose quality would also affect the results. CodeS already incorporates this stage and therefore provides a more direct comparison for generation from task specifications. SIRA complements such generation pipelines by improving coding agents for end-to-end repository generation through experience-guided intent extraction and retrieval-driven post hoc grounding.
Revision: We will cite both works and expand the discussion of their methodological differences and relationship to SIRA.
Q1. Retrieval of Intents Relevant to Function Bodies
At file creation checkpoints, SIRA can retrieve intents relevant to the function bodies you describe through their associated implementation responsibilities. SIRA uses file paths, imports, declarations, and decorators to identify a file’s responsibilities and match them against intent ownership anchors. For example, in the math-verify case in Appendix D, the private helper _extract_latex_segments incorrectly treats boxed_match_priority = -1 as an ordinary priority, although the specification requires this value to disable boxed extraction. Although neither the helper nor its body appears in the query, the relevant behavioral intent is retrieved and delivered through its anchor to the public class LatexExtractionConfig.
Regarding your suggestion for more precise retrieval, function body summaries and diffs are reasonable extensions, but we expect limited benefits. Summaries describe what the code does and may omit what it should do but does not. SIRA therefore emphasizes matching implementation responsibilities with critical intents. Beyond refining retrieval through function body summaries, directly identifying intent violations is a promising direction for our future work. Diffs are more relevant to subsequent edits, but face the limitations discussed in our response to W1. We therefore expect that incorporating them would offer limited performance gains.