- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes a two-stage dual-loop pipeline for NL-to-code generation, with semantic alignment and implementation repair as the focus. The paper showcases the effectiveness of the pipeline via cases from livecodebench, humaneval+, and further ablations and trace inspections.
Critical strengths
- The paper illustrates the proposed design principle clearly, with clear ablation experiments showcasing the effectiveness.
Critical weaknesses
- The paper has limited novelty. Although the paper introduces a large number of notations and symbols for the main pipeline, it does seem that the core pipeline is no different from a simple harness that compares that whether the code implements the specification truthfully. Additionally, the paper's own audit undercuts the traceability claim. In table 7, the agreement with human calibration is only 1/12.
- The paper does not have a sound implementation. LiveCodeBench contains competitive programming programs, where the bottleneck is less on the specification side and more on the complexity and capability of the model. Additionally, the paper does not separate the gain from the harness and the gain from more compute or the effectively best of N.
Questions for authors
- Specification-side labels agree with human review in only 1/12 audited cases. What does this imply for the 579 spec-side labels in Fig. 2?
- How many of the 401 solved problems had at least one property-clause violation during repair? Did (Table 6) ever change a selection or repair outcome?
- For every row of Table 1, how many program candidates, LLM calls and tokens are used per problem, and at what sampling temperature? Do the Self-Refine-style and Reflexion-style baselines generate one or three candidates per round, and do they use the same public-test selector?