acceptodds
Under review as a conference paper at ICLR 2027

How Requirement Formats Affect Large Language Models for Code Generation: Empirical Study

Abstract

Large language models (LLMs) generate code from requirements expressed as prose, scenarios, tests, or specifications, yet the effect of these formats remains unclear. We compare 13 requirement formats across four programming benchmarks and five LLMs, then manually verify transformed requirements to examine whether rewriting errors explain the observed differences. Across model–benchmark pairs, the highest and lowest Pass@1 scores under automatic transformation differ by 6.34%–32.72%. Manual correction raises mean Pass@1 by 1.44%, but substantial differences among formats remain: semi-structured formats lead on average, while preferred formats vary by model and algorithmic task. We use these findings to develop ReForm, which estimates model-specific format preferences from execution outcomes grouped by algorithmic subcategory and selects a format for each new requirement. On held-out OJBench, ReForm achieves 30.00% mean Pass@1 across five models, compared with 22.84% for original requirements and 28.10% for the best single format measured on OJBench. For Claude-Sonnet-4.6, the selected formats also improve Pass@1 when combined with Self-Planning, SCoT, and OpenCode. These results show that requirement-format choice matters beyond transformation fidelity and that format selection transfers to new problems and downstream coding workflows. Our source code and details are publicly available at https://figshare.com/s/b8c47472426b87eb7776}{https://figshare.com/s/b8c47472426b87eb7776.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Requirement Analysis, Code Generation, Large Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_zcCdOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how requirement formats affect LLM code generation. It compares 13 formats across five models and four programming benchmarks, then checks whether the differences persist after manual correction of rewriting errors. Based on these observations, REFORM selects a format using model-specific preferences within algorithmic categories. On held-out OJBench, it improves average Pass@1 over both original requirements and the best fixed format. The paper also evaluates combinations with planning methods and a coding agent.

Critical strengths
  1. The research question has direct practical value. Developers need guidance on how to express requirements clearly for code-generation models.

  2. The comparison is systematic. Studying multiple formats, models, and problem categories provides more insight than proposing a single prompt template.

  3. Manual verification addresses an important confound: rewriting can alter the task’s meaning. OJBench is excluded from recommendation construction. The best fixed format is selected on OJBench itself, providing a strong reference. The workflow combinations are also useful.

Critical weaknesses
  1. Format changes are mixed with additional explanation. In Figure 1, the DR version explicitly explains index shifts and reachable positions. These observations help solve the problem. Semantic equivalence alone does not separate formatting benefits from those of making useful deductions explicit.

  2. The additional gain from dynamic selection is modest. On 232 OJBench problems, REFORM exceeds the best fixed format by 1.90 points on average, with a 0.43-point decline for one model.

  3. The algorithmic novelty is limited, and cost and applicability need further evaluation. REFORM mainly classifies problems and selects formats through a lookup table. Classification and rewriting add model-call costs. The evaluation focuses on standalone algorithmic problems and does not establish applicability to requirement-change tasks in real repositories.

Questions for authors

How would the DR format compare with unstructured prose containing the same explicit explanations? This could help distinguish the effect of organization from that of added reasoning.

2 comments

R1: Format changes are mixed with additional explanation.

Thank you for this profound suggestion! This paper investigates transforming original requirements into appropriate requirement formats via requirements engineering, aiming to resolve the uncertainty developers face regarding which requirement format to choose. The contents across different formats are strictly derived from original requirements without introducing additional information. Keeping the underlying resources and information constant, we evaluate model code generation performance across formats. To address your concern, we have added two in-depth analyses:

1. Qualitative Analysis: We analyze different requirement formats and their corresponding code. Specifically, on the CodeContests dataset with DeepSeek-V4-Pro in Table 2, comparing original requirements, human-verified Dimensional Requirements (DR) causes 15.76% of samples to transition from incorrect correct and 7.88% from correct incorrect. We analyze examples and find that 1.1 Incorrect →\rightarrow Correct: Original requirements often entangle task goals, backgrounds, and constraints. DR explicitly separates these dimensions, helping models comprehend critical conditions. For instance, in a CodeContests sequence-construction problem requiring that “node bb must not lie on the simple path between node aa and node cc,” the original requirement separated the objective and core constraints, leading the model to miss the constraint. Conversely, DR reorganized them into clear dimensions (e.g., purpose and invariants) without extra reasoning, enabling correct code generation. 1.2 Correct →\rightarrow Incorrect: In some cases where DR led to failures, the explicit breakdown caused the model to overly focus on peripheral boundary constraints rather than the primary goal.

2. Controlled Natural Language Experiment: We transform DR back into a continuous natural language description (DR_des) while using human verification to preserve complete information. Results show that DR_des achieves 37.82% Pass@1, whereas DR achieves 46.30%. This confirms that performance variance across requirement formats is primarily driven by the degree of explicitness in requirement dimensions.

R2: The additional gain from dynamic selection is modest.

Thanks for your constructive suggestion! We add other datasets to evaluate the effectiveness and generalizability of REFORM: We add experiments on LeetCodeDataset. Table 3 shows that REFORM achieves 7.48% improvement in Pass@1 compared to using the original requirements (OR), and 5.04% improvement over the best fixed single format (Best fixed). This further demonstrates the effectiveness and generalizability of our REFORM.

Table 3: REFORM results (Pass@1). OR denotes the original requirement. Best fixed is the highest-scoring single format for each model; REFORM selects a format per problem.

ModelOJBench (OR)OJBench (Best fixed)OJBench (REFORM)LeetCodeDataset (OR)LeetCodeDataset (Best fixed)LeetCodeDataset (REFORM)
Claude-Sonnet-4.614.2221.1223.28 (ΔOR=+9.06\Delta_{\text{OR}} = \mathbf{+9.06})71.1773.6276.07 (ΔOR=+4.90\Delta_{\text{OR}} = \mathbf{+4.90})
DeepSeek-V4-Pro22.8431.4734.48 (ΔOR=+11.64\Delta_{\text{OR}} = \mathbf{+11.64})63.1963.8069.33 (ΔOR=+6.14\Delta_{\text{OR}} = \mathbf{+6.14})
Gemini-3-Flash25.4333.6236.21 (ΔOR=+10.78\Delta_{\text{OR}} = \mathbf{+10.78})58.9063.8067.48 (ΔOR=+8.58\Delta_{\text{OR}} = \mathbf{+8.58})
GLM-521.5523.2825.43 (ΔOR=+3.88\Delta_{\text{OR}} = \mathbf{+3.88})58.9058.2864.42 (ΔOR=+5.52\Delta_{\text{OR}} = \mathbf{+5.52})
Qwen3.5-Plus30.1731.0330.60 (ΔOR=+0.43\Delta_{\text{OR}} = \mathbf{+0.43})55.8360.7468.10 (ΔOR=+12.27\Delta_{\text{OR}} = \mathbf{+12.27})
Average22.8428.1030.00 (ΔOR=+7.16\Delta_{\text{OR}} = \mathbf{+7.16})61.6064.0569.08 (ΔOR=+7.48\Delta_{\text{OR}} = \mathbf{+7.48})

R3: The evaluation focuses on standalone algorithmic problems.

Thank you for your suggestion. We will extend our research to repository-level code generation tasks in future work.

Reviewer_4qBUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how requirement formats affect LLM-based code generation, comparing 13 formats across five models and four programming benchmarks. Manual semantic verification is used to examine whether transformation errors explain the performance differences. Based on model- and task-dependent format preferences, the authors propose ReForm, which selects a format using execution outcomes grouped by algorithmic category. Evaluation on held-out OJBench and combinations with reasoning strategies and a coding agent show improvements over original requirements.

Critical strengths
  1. The empirical comparison is reasonably broad. The study covers multiple requirement formats, models, and benchmarks under a shared code-generation and execution protocol. Manual verification of transformed requirements is a useful addition: it addresses whether the observed differences simply arise from errors introduced during rewriting.

  2. The paper evaluates whether format preferences can be used on new problems. OJBench is excluded from the recommendation knowledge base, and ReForm improves mean Pass@1 from 22.84% with original requirements to 30.00%. The experiments with Self-Planning, SCoT, and OpenCode also provide some evidence that requirement reformulation can complement downstream workflows.

Critical weaknesses
  1. The significance of the research question and resulting insights is not sufficiently established. The paper documents many performance differences, but the main conclusions remain that presentation matters, more structure is not always better, and preferences vary across models and tasks. It is unclear what substantially new understanding this adds beyond the prompt sensitivity discussed in the related work. The practical lessons in Section 5—clarify important concepts and choose a suitable representation—are sensible but fairly general. Moreover, the templates change several factors together, including organization, explanations, examples, and hints. Manual semantic verification addresses correctness, but does not isolate which of these factors drives the gains. The study therefore provides a detailed set of observations without deriving a sufficiently clear, generalizable principle.

  2. The practical value of the proposed selection mechanism remains limited. ReForm improves mean Pass@1 by 1.90 percentage points over the best fixed format reported on OJBench, with a decrease for one model. This is encouraging, particularly because the fixed-format reference is selected on the test benchmark, but uncertainty estimates and calibration-cost analysis are missing. The paper does not establish when maintaining model-specific preferences over 31 categories is worthwhile compared with a simple, well-chosen reformulation template. All evaluations concern self-contained algorithmic problems; the OpenCode experiment uses the same setting and only one model. Consequently, the connection between these findings and consequential requirements-engineering or software-development problems remains weak.

Questions for authors

What specific new principle should readers take away beyond the observation that prompt presentation affects code generation? Which experiment most directly supports that principle?

Can you separate the effect of requirement organization from added explanations or hints, for example through content-matched comparisons or template-component ablations?

How robust is the improvement over fixed-format reformulation, and what calibration and inference costs are required? A comparison with a fixed format selected only on the calibration set would also help establish the practical tradeoff.

3 comments

RQ1: What specific new principle should readers take away? Which experiment most directly supports that principle?

Response: Thanks for your suggestion! Rather than prescribing static, manual formatting guidelines, our core takeaway is the principle of model- and task-aware requirement adaptation: code generation performance is heavily governed by the alignment between the algorithmic subcategory of the requirement and the model's specific format preferences. The experiment most directly supporting this principle is our evaluation of REFORM (Section 4 and Table 3), which demonstrates that dynamically selecting a format per problem significantly outperforms static baselines (e.g., original requirements and the best fixed single format) across held-out benchmarks. In addition, we add two in-depth experiments to provide more insights:

1. Qualitative Analysis: We analyze different requirement formats and their corresponding code. Specifically, on the CodeContests dataset with DeepSeek-V4-Pro in Table 2, comparing original requirements, human-verified Dimensional Requirements (DR) causes 15.76% of samples to transition from incorrect correct and 7.88% from correct incorrect. We analyze examples and find that 1.1 Incorrect →\rightarrow Correct: Original requirements often entangle task goals, backgrounds, and constraints. DR explicitly separates these dimensions, helping models comprehend critical conditions. For instance, in a CodeContests sequence-construction problem requiring that “node bb must not lie on the simple path between node aa and node cc,” the original requirement separated the objective and core constraints, leading the model to miss the constraint. Conversely, DR reorganized them into clear dimensions (e.g., purpose and invariants) without extra reasoning, enabling correct code generation. 1.2 Correct →\rightarrow Incorrect: In some cases where DR led to failures, the explicit breakdown caused the model to overly focus on peripheral boundary constraints rather than the primary goal.
2. Controlled Natural Language Experiment: We transform DR back into a continuous natural language description (DR_des) while using human verification to preserve complete information. Results show that DR_des achieves 37.82% Pass@1, whereas DR achieves 46.30%. This confirms that performance variance across requirement formats is primarily driven by the degree of explicitness in requirement dimensions.

RQ2: How robust is the improvement over fixed-format reformulation, and what inference costs are required?

Thank you for your constructive suggestions! We address your points as follows: 1. Generalization on LeetCodeDataset: To further validate the effectiveness and robustness of REFORM, we conduct additional evaluations on LeetCodeDataset, a function-level code generation dataset collected from LeetCode. Each problem provides original requirements, function signatures, difficulty labels (Easy/Medium/Hard), and test cases. As shown in Table 3, REFORM achieves a mean Pass@1 of 69.08%, outperforming the original requirements (OR, 61.60%) by +7.48% and the best fixed single format (64.05%) by +5.03%, demonstrating strong generalizability. 2. Selection Stability and Robustness: To evaluate the stability of REFORM, we repeat the REFORM selection and execution pipeline 3 times on OJBench. The proportion of samples assigned to the exact same algorithmic subcategory across all three runs reached 96.17% (with a confidence interval exceeding 95%), demonstrating that our automated framework is highly stable and reliable against classification variance. 3. Inference Costs: The inference costs of REFORM are larger (approximately 3.28 times) than the costs of fixed-format and OR, since REFORM requires requirements classification and requirement format transition.

Table 3: REFORM results (Pass@1). OR denotes the original requirement. Best fixed is the highest-scoring single format for each model; REFORM selects a format per problem.

ModelOJBench (OR)OJBench (Best fixed)OJBench (REFORM)LeetCodeDataset (OR)LeetCodeDataset (Best fixed)LeetCodeDataset (REFORM)
Claude-Sonnet-4.614.2221.1223.28 (ΔOR=+9.06\Delta_{\text{OR}} = \mathbf{+9.06})71.1773.6276.07 (ΔOR=+4.90\Delta_{\text{OR}} = \mathbf{+4.90})
DeepSeek-V4-Pro22.8431.4734.48 (ΔOR=+11.64\Delta_{\text{OR}} = \mathbf{+11.64})63.1963.8069.33 (ΔOR=+6.14\Delta_{\text{OR}} = \mathbf{+6.14})
Gemini-3-Flash25.4333.6236.21 (ΔOR=+10.78\Delta_{\text{OR}} = \mathbf{+10.78})58.9063.8067.48 (ΔOR=+8.58\Delta_{\text{OR}} = \mathbf{+8.58})
GLM-521.5523.2825.43 (ΔOR=+3.88\Delta_{\text{OR}} = \mathbf{+3.88})58.9058.2864.42 (ΔOR=+5.52\Delta_{\text{OR}} = \mathbf{+5.52})
Qwen3.5-Plus30.1731.0330.60 (ΔOR=+0.43\Delta_{\text{OR}} = \mathbf{+0.43})55.8360.7468.10 (ΔOR=+12.27\Delta_{\text{OR}} = \mathbf{+12.27})
Average22.8428.1030.00 (ΔOR=+7.16\Delta_{\text{OR}} = \mathbf{+7.16})61.6064.0569.08 (ΔOR=+7.48\Delta_{\text{OR}} = \mathbf{+7.48})

Thank you for your detailed response. While I appreciate the authors' clarifications, I still have concerns regarding the limited significance of the contribution. Therefore, I will maintain my original score.

Reviewer_Lp2BOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper compares 13 requirement representations across five LLMs and four programming benchmarks. It manually corrects transformed requirements to investigate whether transformation errors explain performance differences. Based on model and algorithmic category preferences, it proposes REFORM, which selects a requirement template for each new problem. On 232 OJBench problems, REFORM achieves 30.00% average Pass@1, compared with 22.84% for original requirements and 28.10% for the best fixed format. Further experiments combine REFORM with planning methods and a coding agent.

Critical strengths

Broad comparison under a shared evaluation procedure. Comparing 13 representations across five models and four benchmarks provides useful evidence that different reformulation pipelines produce different code-generation outcomes. Keeping the evaluation tests fixed makes these differences directly observable.

Evaluation beyond the calibration problems. Testing REFORM on a separate benchmark is a meaningful attempt to establish transfer. The manual inspection and workflow experiments also address relevant concerns about transformation quality and practical integration, although their reporting and controls need improvement.

Critical weaknesses

The experiments do not isolate requirement format from added reasoning and information. The DR template explicitly requests hints, invariants, example explanations, and error-prone features. These can introduce useful deductions beyond reorganizing the original statement. Figure 1 illustrates this concern: the transformed requirement adds an explanation of the positions an element can eventually reach, which directly supports the successful solution. Such deductions can preserve task semantics while reducing the reasoning needed to solve it, so semantic verification alone does not eliminate this confound. Original requirements also bypass the transformation call, giving the reformulated conditions additional model computation. Without a comparable neutral-rewriting baseline and controls for added hints and content, the results do not establish how much of the improvement comes from format itself. The manual verification procedure is also insufficiently documented to assess the consistency and scope of human edits.

The added value of adaptive format selection is insufficiently established. REFORM selects among 13 formats within 31 categories using empirical pass rates, but the manuscript does not report category sample sizes, selection stability, or uncertainty. Its average improvement over the best fixed format on OJBench is 1.90 percentage points. Reconstructing the counts from Table 3, this corresponds to net differences of five, seven, six, five, and minus one solved problems across the five models. These results are potentially useful, but paired confidence intervals or tests are needed to assess their reliability. Comparisons with a format selected globally from calibration data and with shuffled category labels would also clarify whether the algorithmic categorization contributes beyond generally strong rewriting templates.

Questions for authors

see weakness above

2 comments

R1: Experiments do not isolate requirement format from reasoning and information.

Thank you for this profound suggestion! This paper investigates transforming original requirements into appropriate requirement formats via requirements engineering, aiming to resolve the uncertainty developers face regarding which requirement format to choose. The contents across different formats are strictly derived from original requirements without introducing additional information. Keeping the underlying resources and information constant, we evaluate model code generation performance across formats. To address your concern, we have added two in-depth analyses: 1. Qualitative Analysis: We analyze different requirement formats and their corresponding code. Specifically, on the CodeContests dataset with DeepSeek-V4-Pro in Table 2, comparing original requirements, human-verified Dimensional Requirements (DR) causes 15.76% of samples to transition from incorrect correct and 7.88% from correct incorrect. We analyze examples and find that 1.1 Incorrect →\rightarrow Correct: Original requirements often entangle task goals, backgrounds, and constraints. DR explicitly separates these dimensions, helping models comprehend critical conditions. For instance, in a CodeContests sequence-construction problem requiring that “node bb must not lie on the simple path between node aa and node cc,” the original requirement separated the objective and core constraints, leading the model to miss the constraint. Conversely, DR reorganized them into clear dimensions (e.g., purpose and invariants) without extra reasoning, enabling correct code generation. 1.2 Correct →\rightarrow Incorrect: In some cases where DR led to failures, the explicit breakdown caused the model to overly focus on peripheral boundary constraints rather than the primary goal.
2. Controlled Natural Language Experiment: We transform DR back into a continuous natural language description (DR_des) while using human verification to preserve complete information. Results show that DR_des achieves 37.82% Pass@1, whereas DR achieves 46.30%. This confirms that performance variance across requirement formats is primarily driven by the degree of explicitness in requirement dimensions.

R2: The manual verification procedure is also insufficiently documented to assess the consistency and scope of human edits.

Thank you for your valuable suggestion! We invited three Ph.D. students, each with over three years of programming experience, to conduct the requirement verification. Two validators independently inspected the correctness of the transformed requirements, while the third resolved any disagreements. Validators compared and corrected the transformed requirements against the original ones across task objectives, expected behaviors, inputs, outputs, constraints, examples, edge cases, and template structures to ensure strict fidelity. We will include these detailed procedures in the revised manuscript.

R3: The added value of adaptive format selection is insufficiently established.

Thank you for your review! We address your points as follows. 1. Evaluate the Stability of REFORM: We repeated the REFORM experiments 3 times on OJBench. The proportion of samples assigned to the exact same category across all three runs reached 96.17% (with a confidence interval exceeding 95%), demonstrating high stability and reliability in requirement format selection. 2. Evaluation on LeetCodeDataset: To further validate the generalizability of REFORM, we add experiments on LeetCodeDataset, a function-level code generation dataset collected from LeetCode. Each problem in this dataset provides requirements, function signatures, difficulty labels (Easy/Medium/Hard), and unit test cases. As shown in Table 3, REFORM achieves a +7.48% Pass@1 improvement over original requirements (OR) and +5.03% over the best fixed single format, further demonstrating the effectiveness and generalizability of REFORM.

Table 3: REFORM results (Pass@1). OR denotes the original requirement. Best fixed is the highest-scoring single format for each model; REFORM selects a format per problem.

ModelOJBench (OR)OJBench (Best fixed)OJBench (REFORM)LeetCodeDataset (OR)LeetCodeDataset (Best fixed)LeetCodeDataset (REFORM)
Claude-Sonnet-4.614.2221.1223.28 (ΔOR=+9.06\Delta_{\text{OR}} = \mathbf{+9.06})71.1773.6276.07 (ΔOR=+4.90\Delta_{\text{OR}} = \mathbf{+4.90})
DeepSeek-V4-Pro22.8431.4734.48 (ΔOR=+11.64\Delta_{\text{OR}} = \mathbf{+11.64})63.1963.8069.33 (ΔOR=+6.14\Delta_{\text{OR}} = \mathbf{+6.14})
Gemini-3-Flash25.4333.6236.21 (ΔOR=+10.78\Delta_{\text{OR}} = \mathbf{+10.78})58.9063.8067.48 (ΔOR=+8.58\Delta_{\text{OR}} = \mathbf{+8.58})
GLM-521.5523.2825.43 (ΔOR=+3.88\Delta_{\text{OR}} = \mathbf{+3.88})58.9058.2864.42 (ΔOR=+5.52\Delta_{\text{OR}} = \mathbf{+5.52})
Qwen3.5-Plus30.1731.0330.60 (ΔOR=+0.43\Delta_{\text{OR}} = \mathbf{+0.43})55.8360.7468.10 (ΔOR=+12.27\Delta_{\text{OR}} = \mathbf{+12.27})
Average22.8428.1030.00 (ΔOR=+7.16\Delta_{\text{OR}} = \mathbf{+7.16})61.6064.0569.08 (ΔOR=+7.48\Delta_{\text{OR}} = \mathbf{+7.48})
Reviewer_SfgbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper conducts an empirical study on the impact of requirements formatting in LLM code generation. Using 13 requirement formats and 5 LLMs (2 open, 3 closed), they evaluated the pass@1 scores for each benchmark, model, and requirement formats and found that formatting could impact pass@1 anywhere from 6.34% to 32.72%. From this, they developed ReForm, a requirement-format aggregation and selection process that selects the best format for the given problem based on aggregated format data on similar problems. ReForm shows that requirement formatting has noticeable performance gain on the OJBench dataset when using the original pass@1 metric, reasoning strategies, and coding agents.

Critical strengths

The main strength of the paper is that it is a broad empirical study on the impact of requirements formatting across 13 different formats, 5 models, and 5 benchmarks. It finds that requirement formatting does impact the pass@1 scores for all 5 models and benchmarks and uses manual verification to validate the results. It then proposes ReForm, a method that aggregates and selects the best performing format for different categories of problems. ReForm's experimental results on the holdout dataset, OJBench, does show that lessons learned from the format aggregation are transferable and also can be applied to reasoning strategies and coding agents.

Critical weaknesses

There are a few points that need to be addressed in the experimental design and generalizability of ReForm.

  1. The paper only evaluates ReForm on one heldout dataset, OJBench. Although the results do show that requirement formatting is still impactful, how do the other datasets compare to OJBench? Also, what was the motivation for the selection of OJBench?

  2. It is mentioned in the methodology that there are 17 top-level categories and 31 classification labels, how were these categories and labels determined? Were they derived from previous work or were they extracted by LLMs?

  3. The empirical lessons are interesting, but they are grounded in the pass@1 scores, which does not shed light into what actually affects the performance based on the formatting differences between the 13 different formats. The paper mentions that DR is recommended across categories, but there is no study into what part of DR affects it's performance (and how each format compares to each other). Also, how is DR's distinct fields different from current prompt engineering strategies and techniques?

Questions for authors

refer to weaknesses

2 comments

R1: Add other datasets to evaluate REFORM and the motivation for dataset selection.

Thanks for your constructive suggestion! We address your points as follows: 1. Add datasets to evaluate REFORM: To further validate the generalizability of REFORM, we add experiments on LeetCodeDataset. Table 3 shows that REFORM achieves 7.48% improvements in Pass@1 compared to using the original requirements (OR), and 5.04% improvements over the best fixed single format (Best fixed). This further demonstrates the effectiveness and generalizability of our REFORM. 2. Motivation for Dataset Selection: OJBench and LeetCodeDataset share the same function-level and competition-level task complexity as our RQ1 benchmarks, making them ideal for testing generalizability. Moreover, their high data quality and large variance in difficulty distributions provide sufficient room for performance gains via requirement formatting.

Table 3: REFORM results (Pass@1). OR denotes the original requirement. Best fixed is the highest-scoring single format for each model; REFORM selects a format per problem.

ModelOJBench (OR)OJBench (Best fixed)OJBench (REFORM)LeetCodeDataset (OR)LeetCodeDataset (Best fixed)LeetCodeDataset (REFORM)
Claude-Sonnet-4.614.2221.1223.28 (ΔOR=+9.06\Delta_{\text{OR}} = \mathbf{+9.06})71.1773.6276.07 (ΔOR=+4.90\Delta_{\text{OR}} = \mathbf{+4.90})
DeepSeek-V4-Pro22.8431.4734.48 (ΔOR=+11.64\Delta_{\text{OR}} = \mathbf{+11.64})63.1963.8069.33 (ΔOR=+6.14\Delta_{\text{OR}} = \mathbf{+6.14})
Gemini-3-Flash25.4333.6236.21 (ΔOR=+10.78\Delta_{\text{OR}} = \mathbf{+10.78})58.9063.8067.48 (ΔOR=+8.58\Delta_{\text{OR}} = \mathbf{+8.58})
GLM-521.5523.2825.43 (ΔOR=+3.88\Delta_{\text{OR}} = \mathbf{+3.88})58.9058.2864.42 (ΔOR=+5.52\Delta_{\text{OR}} = \mathbf{+5.52})
Qwen3.5-Plus30.1731.0330.60 (ΔOR=+0.43\Delta_{\text{OR}} = \mathbf{+0.43})55.8360.7468.10 (ΔOR=+12.27\Delta_{\text{OR}} = \mathbf{+12.27})
Average22.8428.1030.00 (ΔOR=+7.16\Delta_{\text{OR}} = \mathbf{+7.16})61.6064.0569.08 (ΔOR=+7.48\Delta_{\text{OR}} = \mathbf{+7.48})

R2: How were the 17 top-level categories and 31 classification labels determined?

Thank you for your valuable feedback! The categories and subcategories are determined by prior work and mainstream programming platforms. Specifically, the 17 top-level categories are established based on existing literature [1], which formalizes the standard algorithmic tags used in competitive programming. We then refined the subcategories into 31 classification labels by incorporating the problem taxonomy of LeetCode, a mainstream programming platform that provides finer-grained classifications for these algorithms. For instance, the platform further sub-categorizes "Trees" into "Binary Trees," "Segment Trees," and "Tries."

[1] Mahbub, M. S., et al. (2024). Predicting the Algorithm Tags and Difficulty for Competitive Programming Problems. arXiv preprint arXiv:2310.05791.

R3: What affects LLMs' performance based on the formatting differences, especially for DR.

Thank you for your insightful suggestion! We add two in-depth analyses to address this: 1. Qualitative Analysis: We analyze different requirement formats and their corresponding code. Specifically, on the CodeContests dataset with DeepSeek-V4-Pro in Table 2, comparing original requirements, human-verified Dimensional Requirements (DR) causes 15.76% of samples to transition from incorrect →\rightarrow correct and 7.88% from correct →\rightarrow incorrect. We analyze examples and find that 1.1 Incorrect →\rightarrow Correct: Original requirements often entangle task goals, backgrounds, and constraints. DR explicitly separates these dimensions, helping models comprehend critical conditions. For instance, in a CodeContests sequence-construction problem requiring that “node bb must not lie on the simple path between node aa and node cc,” the original requirement separated the objective and core constraints, leading the model to miss the constraint. Conversely, DR reorganized them into clear dimensions (e.g., purpose and invariants) without extra reasoning, enabling correct code generation. 1.2 Correct →\rightarrow Incorrect: In some cases where DR led to failures, the explicit breakdown caused the model to overly focus on peripheral boundary constraints rather than the primary goal.
2. Controlled Natural Language Experiment: We transform DR back into a continuous natural language description (DR_des) while using human verification to preserve complete information. Results show that DR_des achieves 37.82% Pass@1, whereas DR achieves 46.30%. This confirms that performance variance across requirement formats is primarily driven by the degree of explicitness in requirement dimensions.

R4. How are DR's distinct fields different from current prompt engineering?

Thanks for the review. Broadly speaking, any input fed into an LLM can be considered a prompt. However, our work focuses on a distinct dimension: how user requirements should be expressed in software development tasks. By applying requirements engineering principles, we investigate which format suits LLMs and resolving the uncertainty developers face regarding how to format their requirements. Unlike generic prompt engineering (which focuses on heuristic modifiers, persona adoption, or formatting output templates), our work bridges software requirements engineering and LLM code generation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.