- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how requirement formats affect LLM code generation. It compares 13 formats across five models and four programming benchmarks, then checks whether the differences persist after manual correction of rewriting errors. Based on these observations, REFORM selects a format using model-specific preferences within algorithmic categories. On held-out OJBench, it improves average Pass@1 over both original requirements and the best fixed format. The paper also evaluates combinations with planning methods and a coding agent.
Critical strengths
-
The research question has direct practical value. Developers need guidance on how to express requirements clearly for code-generation models.
-
The comparison is systematic. Studying multiple formats, models, and problem categories provides more insight than proposing a single prompt template.
-
Manual verification addresses an important confound: rewriting can alter the task’s meaning. OJBench is excluded from recommendation construction. The best fixed format is selected on OJBench itself, providing a strong reference. The workflow combinations are also useful.
Critical weaknesses
-
Format changes are mixed with additional explanation. In Figure 1, the DR version explicitly explains index shifts and reachable positions. These observations help solve the problem. Semantic equivalence alone does not separate formatting benefits from those of making useful deductions explicit.
-
The additional gain from dynamic selection is modest. On 232 OJBench problems, REFORM exceeds the best fixed format by 1.90 points on average, with a 0.43-point decline for one model.
-
The algorithmic novelty is limited, and cost and applicability need further evaluation. REFORM mainly classifies problems and selects formats through a lookup table. Classification and rewriting add model-call costs. The evaluation focuses on standalone algorithmic problems and does not establish applicability to requirement-change tasks in real repositories.
Questions for authors
How would the DR format compare with unstructured prose containing the same explicit explanations? This could help distinguish the effect of organization from that of added reasoning.
2 comments
R2: The additional gain from dynamic selection is modest.
Thanks for your constructive suggestion! We add other datasets to evaluate the effectiveness and generalizability of REFORM: We add experiments on LeetCodeDataset. Table 3 shows that REFORM achieves 7.48% improvement in Pass@1 compared to using the original requirements (OR), and 5.04% improvement over the best fixed single format (Best fixed). This further demonstrates the effectiveness and generalizability of our REFORM.
Table 3: REFORM results (Pass@1). OR denotes the original requirement. Best fixed is the highest-scoring single format for each model; REFORM selects a format per problem.
| Model | OJBench (OR) | OJBench (Best fixed) | OJBench (REFORM) | LeetCodeDataset (OR) | LeetCodeDataset (Best fixed) | LeetCodeDataset (REFORM) |
|---|---|---|---|---|---|---|
| Claude-Sonnet-4.6 | 14.22 | 21.12 | 23.28 () | 71.17 | 73.62 | 76.07 () |
| DeepSeek-V4-Pro | 22.84 | 31.47 | 34.48 () | 63.19 | 63.80 | 69.33 () |
| Gemini-3-Flash | 25.43 | 33.62 | 36.21 () | 58.90 | 63.80 | 67.48 () |
| GLM-5 | 21.55 | 23.28 | 25.43 () | 58.90 | 58.28 | 64.42 () |
| Qwen3.5-Plus | 30.17 | 31.03 | 30.60 () | 55.83 | 60.74 | 68.10 () |
| Average | 22.84 | 28.10 | 30.00 () | 61.60 | 64.05 | 69.08 () |
R3: The evaluation focuses on standalone algorithmic problems.
Thank you for your suggestion. We will extend our research to repository-level code generation tasks in future work.
R1: Format changes are mixed with additional explanation.
Thank you for this profound suggestion! This paper investigates transforming original requirements into appropriate requirement formats via requirements engineering, aiming to resolve the uncertainty developers face regarding which requirement format to choose. The contents across different formats are strictly derived from original requirements without introducing additional information. Keeping the underlying resources and information constant, we evaluate model code generation performance across formats. To address your concern, we have added two in-depth analyses:
1. Qualitative Analysis: We analyze different requirement formats and their corresponding code. Specifically, on the CodeContests dataset with DeepSeek-V4-Pro in Table 2, comparing original requirements, human-verified Dimensional Requirements (DR) causes 15.76% of samples to transition from incorrect correct and 7.88% from correct incorrect. We analyze examples and find that 1.1 Incorrect → Correct: Original requirements often entangle task goals, backgrounds, and constraints. DR explicitly separates these dimensions, helping models comprehend critical conditions. For instance, in a CodeContests sequence-construction problem requiring that “node b must not lie on the simple path between node a and node c,” the original requirement separated the objective and core constraints, leading the model to miss the constraint. Conversely, DR reorganized them into clear dimensions (e.g., purpose and invariants) without extra reasoning, enabling correct code generation. 1.2 Correct → Incorrect: In some cases where DR led to failures, the explicit breakdown caused the model to overly focus on peripheral boundary constraints rather than the primary goal.
2. Controlled Natural Language Experiment: We transform DR back into a continuous natural language description (DR_des) while using human verification to preserve complete information. Results show that DR_des achieves 37.82% Pass@1, whereas DR achieves 46.30%. This confirms that performance variance across requirement formats is primarily driven by the degree of explicitness in requirement dimensions.