acceptodds
Under review as a conference paper at ICLR 2027

Semantic Contracts for Traceable NL-to-Code Generation

Abstract

Large language models can generate code from natural-language problem statements, yet remain unreliable on constraint-intensive tasks. A failure may mean the model misread the task, or that it implemented a reasonable reading incorrectly, and the two call for different remedies. Requirement-alignment methods improve task understanding before generation; program-repair methods use post-generation feedback to fix implementation errors. The two are rarely connected by an explicit, stable semantic target, so repair has no fixed set of task obligations against which to compare successive programs. We introduce a semantic-contract-based dual-loop framework that keeps the semantic target and the program as separate states, each updated by its own loop. The Semantic Alignment Loop (SAL) induces a structured specification and revises it field by field, admitting a revision only when a coverage–faithfulness–precision diagnostic and a Spec Alignment Score (SAS) show non-regression and grounded progress; the specification current at termination is frozen as a semantic contract. The Implementation Repair Loop (IRL) then selects an initial program under that contract and repairs it iteratively from public-test feedback and conservatively compiled property clauses. Because the contract is fixed before repair begins, semantic revisions and implementation repairs are recorded as separate state transitions. On 1,055 LiveCodeBench release_v6 problems, Full Dual-Loop produces 401 complete-suite passes, a Final Pass Rate of 0.3801: 9.19 percentage points above Direct NL-to-Code and 3.98 points above the strongest shared-protocol external baseline. Final correctness improves while the evidence for task obligations and implementation transitions remains inspectable.

Primary area:
foundation or frontier models, including LLMs
Keywords:
code generation, large language models, program repair, requirement alignment, specifications, execution feedback, traceability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Wa5jOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a two-stage dual-loop pipeline for NL-to-code generation, with semantic alignment and implementation repair as the focus. The paper showcases the effectiveness of the pipeline via cases from livecodebench, humaneval+, and further ablations and trace inspections.

Critical strengths
  • The paper illustrates the proposed design principle clearly, with clear ablation experiments showcasing the effectiveness.
Critical weaknesses
  • The paper has limited novelty. Although the paper introduces a large number of notations and symbols for the main pipeline, it does seem that the core pipeline is no different from a simple harness that compares that whether the code implements the specification truthfully. Additionally, the paper's own audit undercuts the traceability claim. In table 7, the agreement with human calibration is only 1/12.
  • The paper does not have a sound implementation. LiveCodeBench contains competitive programming programs, where the bottleneck is less on the specification side and more on the complexity and capability of the model. Additionally, the paper does not separate the gain from the harness and the gain from more compute or the effectively best of N.
Questions for authors
  • Specification-side labels agree with human review in only 1/12 audited cases. What does this imply for the 579 spec-side labels in Fig. 2?
  • How many of the 401 solved problems had at least one property-clause violation during repair? Did FpF_p (Table 6) ever change a selection or repair outcome?
  • For every row of Table 1, how many program candidates, LLM calls and tokens are used per problem, and at what sampling temperature? Do the Self-Refine-style and Reflexion-style baselines generate one or three candidates per round, and do they use the same public-test selector?
Reviewer_8n4nOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a two-stage code generation workflow that first refines and freezes a structured task specification, then uses it to guide code generation and repair with test and property feedback. It reports improved correctness on LiveCodeBench while recording specification changes and code repairs for inspection.

Critical strengths

The workflow clearly separates task interpretation from code repair, making intermediate decisions easier to inspect.

The full system improves over the evaluated baselines on all 1055 LiveCodeBench problems, with private tests held out during generation and repair.

The paper clearly acknowledges the limitations of LLM-generated contracts, judge scores, and trace labels.

Critical weaknesses

The workflow is clear, and the full system shows encouraging improvements over the evaluated baselines. I have a few questions and suggestions that could help strengthen the paper:

  1. The distinction from existing workflows could be clearer. LPW already uses a verified plan to guide implementation and repair. It would be helpful to isolate the additional benefits of structured contracts, the acceptance gate, and property feedback.

  2. The motivation for freezing the contract would benefit from a controlled comparison. The reopening experiment uses previously failed cases and is not compute-matched. Comparing frozen and revisable contracts from the same starting states under the same budget would strengthen this design choice.

  3. A larger ablation would make the findings more convincing. On the 50-problem subset, Full solves three more tasks than Decomposition. Larger paired experiments and repeated runs would help establish how consistently combining the loops improves performance.

  4. More information about computational cost would help interpret the results. Matching program-candidate counts does not account for specification generation, judging, and revision calls. Reporting actual token usage and comparisons under equal budgets would make the trade-offs clearer.

  5. The trace-label results deserve further discussion. In Table 7, strict agreement is 1/12 for spec-side labels and 9/24 when solved cases are excluded. Since most unresolved cases receive spec-side labels, a larger independent audit would help establish their practical usefulness.

  6. Independent validation would strengthen confidence in SAS. Since the same backbone generates and judges specifications, it would be useful to check how often accepted revisions improve task understanding according to independent assessment.

  7. The role of property feedback could be evaluated more directly. Table 6 provides useful clause and violation statistics. An ablation without property feedback would help show whether these checks improve final correctness beyond ordinary test feedback.

Overall, this is a promising approach with positive empirical results. Additional controlled experiments would help clarify which components contribute most to the improvement.

Questions for authors

See Weaknesses

Reviewer_guSZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a dual-loop NL-to-code framework that keeps the task interpretation separate from the program. Semantic Alignment Loop (SAL) induces a six-field structured specification. The same backbone LLM scores it (coverage, faithfulness, precision, and an aggregate Spec Alignment Score, SAS), and field-level revisions are accepted only when safety and improvement predicates hold. The final specification is frozen, and a small set of executable property clauses is compiled from it. Finally, Implementation Repair Loop (IRL) generates up to two contract-conditioned candidates, selects one with public tests, and runs up to three repair rounds (three candidates each) using public-test and property-clause feedback. Because the contract stays fixed during repair, semantic and implementation transitions are recorded separately in a trace.

On all 1,055 LiveCodeBench release_v6 problems with Qwen2.5-Coder-7B-Instruct, the full method reaches 0.3801 Final Pass. Baselines reach 0.2882 (direct generation), 0.3365 (Reflexion-style), and 0.3403 (adapted SpecFix).

Critical strengths
  1. Leakage-free evaluation on a full, constraint-heavy benchmark. The method is evaluated on all 1,055 LCB release_v6 problems, and private tests are used only after a single final program is selected. Two recent external methods (SpecFix, LPW) are adapted to the same protocol. The gain over the strongest adapted baseline (+3.98 points, 42 problems) therefore does not come from selection on the evaluation partition.

  2. A well-motivated, testable design question. The paper separates two failure sources: misreading the task, and implementing a reasonable reading incorrectly. It also keeps the semantic target fixed during repair. Both ideas are sensible. The authors are also honest about the limits such as LLM-induced contract, self-judged SAS, single runs.

Critical weaknesses
  1. The amount of compute is not matched. Full uses up to 11 programs and it makes roughly 7 LLM calls per program in the SAL sweep. Direct is a single sample without feedback. For the Self-Refine/Reflexion-style baselines, the paper does not clearly mention whether they get the same number of candidates.

  2. The results suggest that much of the gain may come from sampling and selection rather than from the contract. 288 of the 401 solved problems involve no SAL or IRL state change in Figure 2.

  3. In Table 3(a), Decomposition with three candidates and no loop reaches 0.66, against 0.72 for Full, which is a 3-problem difference. And SAL-only and IRL-only both score below Decomposition, which suggests this ablation with n=50 is within sampling noise.

  4. So I would suggest two experiments for supporting the main claim. 1) a compute-budget-matched comparison of a frozen contract against a reopenable one; 2) SAL-only and IRL-only ablations on the full release.

  5. The dominant trace label is unreliable. 579 of the 654 unsolved traces carry a specification-side label in Figure 2. In the human audit result from Table 7, only 1 of 12 spec-side labels agrees with the reviewer, and 10 disagree. Since traceability is in the title and the contribution list, I consider this a core soundness problem.

Questions for authors
  1. For each method in Table 1, please report the number of initial candiates, candidates per repair round, and average LLM calls and tokens per problem.
  2. What does a compute-budget-matched sampling baseline achieve?
  3. Can you report SAL-only and IRL-only on all 1,055 problems, plus a compute-budget-matched comparison of a frozen vs. a reopenable contract?
  4. Given that only 1 of 12 spec-side labels agrees with the audit, how should the 579 spec-side residuals in Figure 2 be read? Can the audit be repeated with blinded, independent annotators and reported inter-annotator agreement?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.