- Rating
- 2
- Confidence
- 3(−1)
- Informativeness
- 2(−1)
- Presentation
- 2(−1)
- Soundness
- 2
Read review
Paper summary
SpecAlign refines issue specifications using disagreements between independently generated code and tests. Experiments show small automated gains and larger improvements with a simulated developer given access to gold patches.
Critical strengths
The task is relevant in practice, and localized modifications provide a good user interface for developer input. The evaluation includes two model types and different programming languages, as well as helpful ablation studies and analyses of specification modifications.
Critical weaknesses
- Gains from automation are marginal: merely 10 to 12 more resolved tasks in 731. SpecAlign can create up to three patches, but comparisons to repeated sampling or code fixing with comparable budgets are not available.
- The simulated developer has access to gold patches, but leakage analysis does not take into account the information derived from the gold test suite. Thus, "None" does not exclude any gains from the privileged information.
- Oracle feedback diminishes code-test alignment while increasing task resolution. Word counts offer insufficient proof of reduced effort from the developer.
Thanks the authors for their reply and explanations. I tend to maintain my score.
Questions for authors
- Do the gains hold against repeated sampling with test-based selection and code repair under equal budgets? Please report confidence intervals.
- How do leakage conclusions change when gold-test-derived information is included?
- Does localized refinement outperform direct rewriting with the same oracle information and edit budget?
- Why does oracle feedback reduce agreement while improving resolution?
2 comments
Response [2/2]
4. Why Does Oracle Feedback Reduce Agreement While Improving Resolution?
One possible explanation is that generated code and tests can agree on the same incorrect interpretation of an incomplete specification. Such agreement does not guarantee correctness. HITL provides additional information that may correct these assumptions, reducing code-test agreement while improving actual issue resolution. Therefore, lower agreement does not necessarily indicate lower correctness, and code-test agreement alone is an imperfect measure of success. We will clarify this interpretation in the revised paper.
Response [1/2]
We thank the reviewer for the constructive feedback and for recognizing the importance of specification refinement. We conducted additional experiments and analyses to address the reviewer's concerns:
1. Repeated Sampling and Computational Comparison
To address the reviewer's concern, we evaluated three independent coding-agent runs, repeated sampling with patch selection, and iterative code repair. Across the three independent runs, the mean resolution rate was 58.19% (95% CI: 57.33%–59.04%), indicating limited variation in baseline performance.
The no-leakage HITL results use different subsets. Simulated HITL uses privileged oracle information and is not directly comparable to the automated baselines.
Iterative code repair uses the original specification, generated code and tests, and execution logs to repair code for up to three attempts without changing the specification. While the automated baselines achieve similar resolution rates, they cannot directly correct missing requirements. SpecAlign instead identifies potential specification gaps for targeted clarification before regenerating code and tests. The HITL results illustrate this opportunity, although they do not isolate gap identification from oracle-provided information.
Regarding computational cost, SpecAlign requires approximately 2.4 coding-agent iterations on average, compared to three runs for repeated sampling. However, this does not guarantee equal monetary costs, since SpecAlign also requires test generation, execution, and specification refinement.
2. Leakage Analysis Including Gold-Test Information
We extended our three-LLM leakage analysis to include both gold code and gold test patches. The proportion of instances classified as having no detected leakage decreases from 72.5% to 67.6% at s₂ and from 94.0% to 89.0% at s₃. Gold-test-only leakage accounts for 7.2% and 3.7%, respectively.
Even with a stricter leakage judge, the HITL experiments demonstrate substantial gains, suggesting that they are indeed enabled by pinpointing specification gaps (which is the main objective of our paper). We will include the updated Table 2, presented in our response to Reviewer yeW8 (Q2), in the revised paper.
3. Localized Refinement vs. Direct Rewriting
We conducted an additional experiment on 300 instances. Both settings used the same execution logs and oracle information. Without highlighting, the simulated developer was not shown SpecAlign's identified gaps and could edit any part of the specification.
Detected leakage includes minor, moderate, and severe leakage from either gold code or gold test patches.
Unrestricted editing achieves higher final-patch resolution (80.7% vs. 76.7%) but also increases detected leakage (40% vs. 32%). Thus, localized refinement reduces detected leakage while highlighting specific gaps for developer review. This experiment combines highlighting and editing restrictions and does not directly compare against rewriting the original issue under an identical edit budget.