acceptodds
Under review as a conference paper at ICLR 2027

SpecAlign: Test-Driven Specification Refinement for Coding Agents

Abstract

Software specifications, such as issue descriptions, are usually informal and often vague. Nevertheless, users insist that code written or generated based on a specification must align with it. This may require clarifying the specification itself. Prior work attempted to improve code+spec alignment by automatically rewriting the specification in a standardized way. Unfortunately, doing so just shifts the problem: now it is burdensome for users to tell whether the standardized spec aligns with the original. Instead, we propose SPECALIGN, which takes a different approach: it pinpoints parts of the spec to improve and surgically rewrites only those parts. That makes it is easier to compare the original and rewritten spec. To align a spec with generated code, SPECALIGN generates both code and tests, and then optimizes how well they agree. We demonstrate that this leads to better spec+code alignment, and ultimately, better issue resolution rates on SWE-Bench Pro and SWE-rebench. Even without a human, just pinpointing the gap already yields 2.4-3.6% improvement while only touching 35% of words. To understand the remaining gap, we introduce a simulated user into the SPECALIGN loop. Thanks to SPECALIGN pinpointing the gap, a simulated human can get 13.8-34.4% improvement. We quantify how often it requires information missing from the spec and available only in the ground truth. Overall, we hope that this paper contributes to better aligning AI-generated code with human intentions.

Primary area:
datasets and benchmarks
Keywords:
Specification alignment, Software engineering agents, Reproduction test

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_uV1FOpenReview ↗
Rating
2
Confidence
3(−1)
Informativeness
2(−1)
Presentation
2(−1)
Soundness
2
Read review
Paper summary

SpecAlign refines issue specifications using disagreements between independently generated code and tests. Experiments show small automated gains and larger improvements with a simulated developer given access to gold patches.

Critical strengths

The task is relevant in practice, and localized modifications provide a good user interface for developer input. The evaluation includes two model types and different programming languages, as well as helpful ablation studies and analyses of specification modifications.

Critical weaknesses
  • Gains from automation are marginal: merely 10 to 12 more resolved tasks in 731. SpecAlign can create up to three patches, but comparisons to repeated sampling or code fixing with comparable budgets are not available.
  • The simulated developer has access to gold patches, but leakage analysis does not take into account the information derived from the gold test suite. Thus, "None" does not exclude any gains from the privileged information.
  • Oracle feedback diminishes code-test alignment while increasing task resolution. Word counts offer insufficient proof of reduced effort from the developer.

Thanks the authors for their reply and explanations. I tend to maintain my score.

Questions for authors
  1. Do the gains hold against repeated sampling with test-based selection and code repair under equal budgets? Please report confidence intervals.
  2. How do leakage conclusions change when gold-test-derived information is included?
  3. Does localized refinement outperform direct rewriting with the same oracle information and edit budget?
  4. Why does oracle feedback reduce agreement while improving resolution?
2 comments

Response [1/2]

We thank the reviewer for the constructive feedback and for recognizing the importance of specification refinement. We conducted additional experiments and analyses to address the reviewer's concerns:

  • Additional baselines: Evaluate repeated sampling with test-based and LLM-based selection, along with iterative code repair.
  • Expanded leakage analysis: Include solution leakage from both gold code and gold test patches.
  • New no-leakage result: Among 341 instances with no detected leakage across any existing specification version, final resolution improves from 52.5% to 74.2% (+21.7 percentage points).
  • Localized refinement ablation: Compare simulated HITL with and without SpecAlign's highlighting on 300 instances.

1. Repeated Sampling and Computational Comparison

To address the reviewer's concern, we evaluated three independent coding-agent runs, repeated sampling with patch selection, and iterative code repair. Across the three independent runs, the mean resolution rate was 58.19% (95% CI: 57.33%–59.04%), indicating limited variation in baseline performance.

ApproachResolvedResolution (%)
Independent run 1425/73158.1
Independent run 2428/73158.5
Independent run 3423/73157.9
Repeated sampling + test-based selection430/73158.8
Repeated sampling + LLM-based selection431/73159.0
Iterative code repair427/73158.4
Automated SpecAlign435/73159.5
SpecAlign + simulated HITL (final-patch selection)571/73178.1
HITL, no detected leakage at s₂257/36770.0
HITL, no detected leakage at s₃336/43178.0
HITL, no detected leakage in any existing version (final)253/34174.2

The no-leakage HITL results use different subsets. Simulated HITL uses privileged oracle information and is not directly comparable to the automated baselines.

Iterative code repair uses the original specification, generated code and tests, and execution logs to repair code for up to three attempts without changing the specification. While the automated baselines achieve similar resolution rates, they cannot directly correct missing requirements. SpecAlign instead identifies potential specification gaps for targeted clarification before regenerating code and tests. The HITL results illustrate this opportunity, although they do not isolate gap identification from oracle-provided information.

Regarding computational cost, SpecAlign requires approximately 2.4 coding-agent iterations on average, compared to three runs for repeated sampling. However, this does not guarantee equal monetary costs, since SpecAlign also requires test generation, execution, and specification refinement.

2. Leakage Analysis Including Gold-Test Information

We extended our three-LLM leakage analysis to include both gold code and gold test patches. The proportion of instances classified as having no detected leakage decreases from 72.5% to 67.6% at s₂ and from 94.0% to 89.0% at s₃. Gold-test-only leakage accounts for 7.2% and 3.7%, respectively.

No-leakage groupnInitial s₁ (%)After refinement (%)Change (pp)
None at s₂, evaluated at s₂36752.670.0+17.4
None at s₃, evaluated at s₃43151.578.0+26.5
None in every existing version, evaluated at final34152.574.2+21.7

Even with a stricter leakage judge, the HITL experiments demonstrate substantial gains, suggesting that they are indeed enabled by pinpointing specification gaps (which is the main objective of our paper). We will include the updated Table 2, presented in our response to Reviewer yeW8 (Q2), in the revised paper.

3. Localized Refinement vs. Direct Rewriting

We conducted an additional experiment on 300 instances. Both settings used the same execution logs and oracle information. Without highlighting, the simulated developer was not shown SpecAlign's identified gaps and could edit any part of the specification.

MetricWith HighlightsWithout Highlights
Test-based selection187/300 (62.3%)191/300 (63.7%)
Final-patch selection230/300 (76.7%)242/300 (80.7%)
Detected solution leakage at s₂96/300 (32.0%)120/300 (40.0%)

Detected leakage includes minor, moderate, and severe leakage from either gold code or gold test patches.

Unrestricted editing achieves higher final-patch resolution (80.7% vs. 76.7%) but also increases detected leakage (40% vs. 32%). Thus, localized refinement reduces detected leakage while highlighting specific gaps for developer review. This experiment combines highlighting and editing restrictions and does not directly compare against rewriting the original issue under an identical edit budget.

Response [2/2]

4. Why Does Oracle Feedback Reduce Agreement While Improving Resolution?

One possible explanation is that generated code and tests can agree on the same incorrect interpretation of an incomplete specification. Such agreement does not guarantee correctness. HITL provides additional information that may correct these assumptions, reducing code-test agreement while improving actual issue resolution. Therefore, lower agreement does not necessarily indicate lower correctness, and code-test agreement alone is an imperfect measure of success. We will clarify this interpretation in the revised paper.

Reviewer_yeW8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SpecAlign, which refines a repository issue description when independently generated code and reproduction tests disagree. An LLM rewrites only the parts of the issue it thinks cause the disagreement, and the loop repeats up to three times. The paper also adds a simulated developer, an LLM with access to the gold code and test patches, who further edits the rewritten parts.

Critical strengths

1/ Using code-test disagreement to point a developer to specific parts of an issue is a reasonable idea for repository-level tasks, and the paper evaluates it across four languages on SWE-Bench Pro.

Critical weaknesses

1/ The contribution of the automatic workflow is unclear. Localized spec refinement driven by test failures is already done in VeriSpecGen [1]. ClarifyGPT [2] already detects ambiguous requirements from inconsistent generated code and then refines the requirement. Selecting patches with generated reproduction tests is standard in SWE agents such as Agentless [3]. The paper doesn't explain what is new beyond applying these to issue descriptions. The gain is also small: 58.1% to 59.5% with Claude and 45.6% to 47.2% with GPT (the 2.4-3.6% in the abstract is relative). When the code from the latest version of the issue is used instead (final-patch selection), the gain almost disappears (59.1% and 45.7%). So the gain seems to come from picking among three attempts, not from the refined issue. Disagreement can also come from a code bug or a bad test, but the method always resolves it by editing the issue. This can add wrong assumptions.

2/ The HITL gains may come from leaking the answer rather than from SpecAlign. The simulated developer is an LLM that sees the gold code and gold tests, and SWE-Bench Pro evaluates with tests from the gold test patch. So its edits can tell the coding agent what the hidden tests check. The paper reports gains even on instances labeled as no leakage, but the leakage check ignores information taken from the gold tests, and the labeling models only see the gold code. The paper even presents an edit that copies a constant from the golden patch ("must be '2.14' — this is a literal constant in the golden patch") as an example of a clarification (Fig. 6). There is also no baseline where the same LLM edits the original issue directly, without SpecAlign's highlights. Without it, we can't tell whether the gain comes from SpecAlign or from the gold patches.

[1] Intent-aligned Formal Specification Synthesis via Traceable Refinement. Ye et al.

[2] ClarifyGPT: Empowering LLM-based Code Generation with Intention Clarification. Mu et al.

[3] Agentless: Demystifying LLM-based Software Engineering Agents. Xia et al.

Questions for authors

1/ Can authors explain the methodological difference from [1,2,3] beyond the setting?

2/ Can authors report HITL results where the simulated developer edits the original issue without SpecAlign's highlights? This would show whether the improvement comes from SpecAlign or just from the gold patches. Can the leakage analysis also cover information from the gold test patch?

3 comments

Response [1/2]

We thank the reviewer for the constructive feedback. We conducted additional experiments and analyses to address the reviewer's concerns:

  • Additional baselines: Evaluate repeated sampling with test-based and LLM-based selection, along with iterative code repair.
  • Expanded leakage analysis: Include solution leakage from both gold code and gold test patches.
  • New no-leakage result: Among 341 instances with no detected leakage across any existing specification version, final resolution improves from 52.5% to 74.2% (+21.7 percentage points).
  • Localized refinement ablation: Compare simulated HITL with and without SpecAlign's highlighting on 300 instances.

1. Novelty and Automatic SpecAlign

The main goal of SpecAlign is to identify specification gaps and enable focused developer clarification, rather than simply select better code patches.

VeriSpecGen focuses on generating and refining formal specifications in Lean. In contrast, SpecAlign works directly with natural-language issue descriptions and uses disagreements between independently generated code and tests to identify potential specification gaps.

ClarifyGPT detects ambiguous requirements by comparing generated programs and requesting user clarification, mainly for function-level code generation. SpecAlign instead addresses repository-level issues, where function signatures and implementation details may be missing or ambiguous, and uses code-test disagreement to identify specific regions for refinement.

Agentless uses generated tests to validate and select code patches for repository-level issues. While SpecAlign also uses test-based selection, its main contribution is identifying specification gaps that may require clarification, rather than only selecting better patches.

Automatic Gains and Patch Selection: We agree that SpecAlign's automatic improvements are modest. To investigate whether they come mainly from patch selection, we evaluated repeated sampling and iterative code repair.

ApproachResolvedResolution (%)
Independent run 1425/73158.1
Independent run 2428/73158.5
Independent run 3423/73157.9
Repeated sampling + test-based selection430/73158.8
Repeated sampling + LLM-based selection431/73159.0
Iterative code repair427/73158.4
Automated SpecAlign435/73159.5
SpecAlign + simulated HITL (final-patch selection)571/73178.1
HITL, no detected leakage at s₂257/36770.0
HITL, no detected leakage at s₃336/43178.0
HITL, no detected leakage in any existing version (final)253/34174.2

The no-leakage HITL results use different subsets. Simulated HITL uses privileged oracle information and is not directly comparable to the automated baselines.

Although the automated baselines achieve similar resolution rates, they keep the original specification fixed and cannot directly address missing or ambiguous requirements. SpecAlign instead identifies potential gaps for targeted clarification before regenerating code and tests. The HITL results illustrate the value of this approach, although they do not isolate gap identification from oracle-provided information.

2. HITL and Solution Leakage

Leakage Analysis Including Gold-Test Information: We extended our three-LLM leakage analysis to include both gold code and gold test patches, with a stricter evaluation of instances showing no detected leakage across any version.

Leakage groupnGroup share (%)Initial s₁ (%)After refinement (%)Change (pp)
s₂: None36767.652.670.0+17.4
s₂: Minor295.372.469.0-3.4
s₂: Moderate8615.848.882.6+33.7
s₂: Severe6111.249.288.5+39.3
s₃: None43189.051.578.0+26.5
s₃: Minor173.564.747.1-17.6
s₃: Moderate285.850.078.6+28.6
s₃: Severe81.750.087.5+37.5
Final: None in every existing version341—52.574.2+21.7

Leakage analysis covers 543 instances at s₂ and 484 at s₃. The final row includes 341 instances with no detected leakage across all versions. Changes are in percentage points.

Including gold-test information reduces the proportion classified as having no detected leakage from 72.5% to 67.6% at s₂ and from 94.0% to 89.0% at s₃. Gold-test-only leakage accounts for 7.2% and 3.7%, respectively. Importantly, the stricter 341-instance group still improves by 21.7 percentage points, suggesting that detected solution leakage alone cannot explain the HITL gains. However, LLM-based judgments cannot rule out all oracle influence.

Response [2/2]

HITL Without SpecAlign's Highlighting: We also conducted an experiment on 300 instances. Both settings used the same execution logs and oracle information. In the unrestricted setting, the simulated developer was not shown the highlighted specification gaps and could edit any part of the specification.

MetricWith HighlightsWithout Highlights
Test-based selection187/300 (62.3%)191/300 (63.7%)
Final-patch selection230/300 (76.7%)242/300 (80.7%)
Detected solution leakage at s₂96/300 (32.0%)120/300 (40.0%)

Detected solution leakage includes minor, moderate, and severe leakage from either gold code or gold test patches.

Without highlighting, final-patch resolution improves from 76.7% to 80.7%, but detected solution leakage also increases from 32% to 40%. This shows a tradeoff: unrestricted editing achieves higher resolution, while localized refinement reduces detected leakage and enables focused clarification. However, this experiment combines the effects of highlighting and editing restrictions and does not directly compare against rewriting the original issue under an identical edit budget.

Thank you for the response and the new experiments. My main concerns remain. Automated SpecAlign is only 4 instances above repeated sampling with LLM selectio, so the automatic gain is not clearly separated from patch selection. Also HITL without highlights resolves more instances, which can suggest HITL gains come from the oracle information or leakage rather than from SpecAlign. I will maintain my score.

Reviewer_6hQxOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

SpecAlign is a proposed framework for solving SWE-bench -type tasks which (1) generates a code patch, (2) independently generates test cases, (3) runs the test cases on the patch, (4) if the test cases fail, use this feedback to selectively update the spec (e.g. issue) with new text placed in <new> ... </new> tags. Optionally, a simulated human in the loop can be added to this framework to give feedback on and refine the spec further.

This framework is implemented with Opus 4.6 or GPT 5.4 using the mini-SWE-agent harness. It is evaluated on SWE-bench PRO and 200 instances from SWE-rebench that were created after Opus 4.6's cutoff date in order to be contamination free. The approach costs improves performance by 1-2%, from 58.1% to 59.5% on SWE-bench PRO and from 51.0% to 53.0% on the selected SWE-rebench instances.

The HITL approach further improves performance, although this improvement can be attributed at least in part due to solution leakage, which the authors study in section 4.4.

The authors analyze the size of spec edits made by this framework, analyze the type of feedback provided by a simulated HITL, and conduct ablations of different components in their system including the inclusion of regression tests and inclusion of the gold test and code patches to the HITL.

Critical strengths

The proposed method is fairly principled and reasonable way to improve performance of LLM's on SWE-bench style tasks. The inclusion of the contamination free benchmark, new tasks from SWE-rebench, is a great inclusion for the paper.

Critical weaknesses

First, the approach is not very novel: generating tests and code in parallel and using their agreement has been done before (CodeT, Chen et al 2023), AgentCoder (Huang et al, 2023) and others. Using tests and code disagreement to refine the specification may not have been done before, but it is a pretty small change to existing workflows. I would think of it more as one small part of an agentic engineering set up, and less of a scientifically interesting, principled approach that is worth writing in an ICLR paper.

Second, performance improvement is extremely modest (1-2% improvement). A proper baseline of "instead of spending extra effort on the harness, spend longer creating the patch" is not included as a baseline. As Opus 5.5 pointed out to me, "The gain may come from selection, not spec refinement. There’s no best-of-k baseline at matched cost".

Questions for authors

Would it be possible to see a baseline where the same total spending is used for mini-SWE-agent and SpecAlign? Right now, if I understand correctly, SpecAlign gets around 3x the compute that the baseline gets. A particularly informative baseline suggested in collaboration with Opus 5.5 would be to run mini-SWE-agent up to 3 times on the original spec s₁, as independent samples.

Other suggestions:

  • Provide an ablation which shows HITL without gold patch of both test and code patch?
  • I suggest improvements are reported in absolute percentage points instead of relative percentage points. Relative is confusing.
2 comments

We thank the reviewer for the constructive feedback. We conducted additional experiments and analyses to address the reviewer's concerns:

  • Additional baselines: Evaluate three independent coding-agent runs, repeated sampling with test-based and LLM-based selection, and iterative code repair.
  • Expanded leakage analysis: Include solution leakage from both gold code and gold test patches.
  • New no-leakage result: Among 341 instances with no detected leakage across any existing specification version, final resolution improves from 52.5% to 74.2% (+21.7 percentage points).

1. Novelty

The main goal of SpecAlign is to identify specification gaps and enable focused developer clarification, rather than simply improve automated issue resolution. We agree that CodeT and AgentCoder also use generated code and tests, but they mainly focus on single-function code generation benchmarks such as HumanEval and MBPP, where function signatures are typically provided. In contrast, SpecAlign addresses repository-level issues, where function signatures and other implementation details may be missing or ambiguous. While CodeT uses tests to select code candidates and AgentCoder uses test feedback to repair code, SpecAlign uses code-test disagreement to identify potential specification gaps and highlight specific regions for refinement.

SpecAlign's contribution is not simply combining code and test generation, but using their disagreement to identify specification gaps and create opportunities for targeted developer intervention.

2. Additional Baselines and Computational Comparison

We thank the reviewer for this suggestion. We conducted three independent mini-SWE-agent runs on the original specification without refinement. The runs resolved 425, 428, and 423 instances, respectively. We then evaluated test-based and LLM-based patch selection, along with an iterative code-repair baseline suggested by Reviewer uV1F.

ApproachResolvedResolution (%)
Independent run 1425/73158.1
Independent run 2428/73158.5
Independent run 3423/73157.9
Repeated sampling + test-based selection430/73158.8
Repeated sampling + LLM-based selection431/73159.0
Iterative code repair427/73158.4
Automated SpecAlign435/73159.5
SpecAlign + simulated HITL (final-patch selection)571/73178.1
HITL, no detected leakage at s₂257/36770.0
HITL, no detected leakage at s₃336/43178.0
HITL, no detected leakage in any existing version (final)253/34174.2

The no-leakage HITL results are evaluated on different subsets (n=367 at s₂, n=431 at s₃, and n=341 at the final stage). Leakage analysis includes both gold code and gold test patches. Simulated HITL uses privileged oracle information and is not directly comparable to automated baselines.

For iterative code repair, the agent receives the original specification, generated code and test patches, and execution logs. It repairs code for up to three attempts while keeping the original specification fixed.

These findings highlight an important distinction between the approaches. Repeated sampling and iterative code repair keep the original specification fixed and therefore cannot directly correct missing or ambiguous requirements. In contrast, SpecAlign uses code-test disagreement to identify potential specification gaps and highlight regions for refinement. This creates an opportunity for developers to provide missing information or clarify requirements before independently regenerating code and tests. Thus, SpecAlign's main value is not simply improving automated resolution, but enabling focused correction of specification gaps that coding agents may not reliably resolve on their own.

Regarding computational cost, SpecAlign requires approximately 2.4 coding-agent iterations on average, compared to three complete runs for repeated sampling. However, matching the number of coding-agent iterations does not guarantee equal monetary costs, since SpecAlign also includes test generation, execution, and specification refinement. We will clarify this distinction and include the additional experiments in the revised paper.

3. Additional Suggestions

HITL Without Gold Patches: Removing both gold code and gold test patches removes the oracle information available to the simulated developer. This makes the setting similar to our fully automated workflow, although a real developer could still provide useful clarification based on their own knowledge. We will clarify this limitation.

Absolute Improvements: We agree and will report improvements in absolute percentage points rather than relative percentages.

Thank you for your response.

I do not think the HITL results should be bolded, as they include gold patches in the simulated workflow, even when no leakage is detected.

I understand that perhaps the best-of-3 is not compute matched (do you have compute/API measures for the different methods?) but a 0.5–0.7% improvement over them is quite small.

Comments on the paper

1 comment

Summary of Additional Experiments and Clarifications

We thank the Area Chair and all reviewers for their constructive feedback. We conducted additional experiments and analyses to address the shared concerns regarding novelty, computational comparisons, and the validity of simulated human-in-the-loop (HITL) results.

SpecAlign's central contribution is identifying potential specification gaps through code-test disagreement and enabling targeted developer clarification. Our simulated HITL experiments provide evidence that targeted specification refinement can substantially improve issue resolution when the original specification is incomplete.

Our main additional findings are:

  1. Additional baselines: We evaluated repeated sampling with test-based and LLM-based selection, along with iterative code repair. These achieve 58.4–59.0% resolution, only marginally improving upon the original 58.1% baseline, compared to 59.5% for automated SpecAlign. This suggests that additional code generation, selection, or repair alone provides limited gains when the original specification remains unchanged. SpecAlign instead identifies potential specification gaps and enables targeted clarification.

  2. Expanded leakage analysis: We extended our analysis to include both gold code and gold test patches. Among 341 instances with no detected leakage across any existing specification version, final resolution improves from 52.5% to 74.2% (+21.7 percentage points). These substantial gains persist under stricter leakage criteria, supporting the potential value of identifying specification gaps and refining them through targeted clarification, which is the central objective of SpecAlign.

  3. Localized refinement ablation: On 300 instances, unrestricted editing achieves higher final-patch resolution (80.7% vs. 76.7%) but also increases detected solution leakage (40% vs. 32%). This highlights the tradeoff between unrestricted rewriting and targeted refinement.

Together, these findings support specification-gap identification as a complementary direction to conventional code generation and patch selection, highlighting the value of targeted clarification when the original specification is incomplete.

We will incorporate the additional baselines, updated leakage analysis (Table 2), highlighting ablation, and associated limitations into the revised paper. Detailed results and discussions are provided in our individual responses to Reviewers 6hQx, yeW8, and uV1F.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.