acceptodds
Under review as a conference paper at ICLR 2027

SACP-Coder: State-based Adaptive Computation for Progressive Code Generation

Abstract

Code generation performance has been considerably improved by recent agentic systems, but such gains are often accompanied by increased token consumption. To better adjust computation, existing adaptive methods either select a solving method before generation or repeatedly revise candidate code using test feedback under a predefined strategy. However, these methods tend to use the same solving strategy as a problem’s solving state evolves, overlooking changes in computational needs and potentially wasting tokens on unnecessary computation. To better balance performance and cost, we propose SACP-Coder, a state-adaptive framework that dynamically adjusts computation to match evolving solving needs. We design state-adaptive mechanisms, including retrieval augmentation, problem reformulation, and dual-level correction, and progressively adapt their use to the solver state characterized by candidate solutions and public-test execution outcomes. In particular, problem reformulation facilitates solving challenging problems by deriving a simpler problem with the same core algorithmic structure and transferring its solution pattern back to the original problem. Experiments on four code generation benchmarks across three LLMs demonstrate a favorable performance–cost trade-off. SACP-Coder reduces average token consumption by over 76% relative to comparably accurate baselines on each LLM backbone.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Code Generation, Adaptive Test-Time Computation, Inference Efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jr7WOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SACP-Coder, an inference-time framework that aims to balance code-generation accuracy and token consumption. It defines a solver state using the current candidate program, its pass ratio on public samples, and execution feedback. After an initial direct-generation attempt, the framework uses predefined rules to retain the solution, retrieve reference solutions, reformulate the problem into a simpler structurally related problem, or perform code-level correction and replanning. Experiments on four programming benchmarks and three LLM backbones report competitive accuracy with substantially lower token consumption than comparably accurate multi-agent baselines.

Critical strengths
  1. A practically relevant accuracy–cost trade-off. Reducing the inference cost of agentic code generation is an important problem. The method does achieve less token consumption and relative the same performance comparing multi-agent methods. The results suggest that the proposed workflow could provide practical value.
  2. Evaluation beyond a single default configuration. The paper evaluates multiple benchmarks and backbones, includes cumulative component ablations, and studies accuracy–token trade-offs as the iteration limit changes. The comparison with uniformly applying reformulation and correction is particularly useful for assessing the benefit of selective computation.
Critical weaknesses
  1. The methodological novelty and the distinction from existing feedback-driven workflows are not sufficiently established. Although the method is formulated as a sequential decision process over solver states, the initial action is determined by a threshold on public-sample pass ratio, and subsequent failures largely follow a predefined escalation sequence. The distinction between 'rewrite' and 'replan' is also based on predefined checks. These choices may be useful, but the paper does not sufficiently establish what is gained beyond combining familiar mechanisms in a heuristic workflow.
  2. The evidence does not sufficiently isolate or validate the key routing and retrieval choices. The cumulative ablations and scaling curves provide useful evidence, but leave important questions unresolved. First, the routing signal can be very coarse: the supplied CodeContests artifact records only one public sample I/O pair for 108 of 165 problems. For those single-pair problems, the pass ratio is necessarily either zero or one, so the intermediate 'augment' branch cannot be selected. Second, Table 3 evaluates adding AUGMENT, but does not isolate the contribution of label-based retrieval and code-length-based reference selection relative to simpler selection rules. Finally, in Table 4, the token-nearest comparisons still involve more token consumption than baselines.
Questions for authors
  1. Clarification of novelty: What precisely is held fixed in “fixed-strategy iteration”? How does SACP-Coder differ from methods that already revise reasoning or plans using execution feedback? Please clarify the specific contribution of the state-conditioned policy beyond the individual retrieval, reformulation, and correction mechanisms.
  2. Reliability of the routing signal: What are the distributions of public sample counts and initial routing decisions across all four benchmarks?
  3. Retrieval protocol and contribution: How is the case bank initialized and updated, and which tests establish that a stored solution is successful?
6 comments
AuthorsOpenReview ↗

Response to Reviewer jr7W (Part 1/3)

We appreciate your constructive review and specific questions. We answer them below and have revised the manuscript to address the points you raised. The corresponding additions are highlighted in blue.

W1&Q1: Novelty

W1: Novelty and distinction.

Compared with combining familiar mechanisms in a heuristic workflow, the novelty and distinction of SACP-Coder lie in two aspects:

  • State-conditioned progressive workflow. Instead of applying the same correction strategy after each failure, SACP-Coder uses the solver state to select among different actions and progressively escalates when needed, reducing unnecessary token consumption.

  • The unique REFORMULATE mechanism. Unlike familiar mechanisms that generate related solved examples or only a code-generation plan, REFORMULATE derives a simpler problem while preserving the target problem’s core algorithmic structure, identifies the key solution pattern of the resulting simpler problem, and transfers that pattern back to the original target.

To further qualify the distinction between SACP-Coder and existing workflows, we added controlled comparisons of action selection while holding the interventions and update budget fixed:

  • Equal-quota blind randomly assigns AUGMENT to the same number of Direct failures as the original SACP-Coder configuration, and assigns REFORMULATE to the remaining failures. Results are averaged over 1,000 random assignments per benchmark.
  • Uniform AUGMENT → REFORMULATE first applies AUGMENT to every Direct failure and escalates to REFORMULATE if it fails.
  • Uniform REFORMULATE applies REFORMULATE to every Direct failure.

Together, these comparisons show that state-conditioned selection achieves the highest micro-average accuracy among the three controls, while using fewer tokens than uniform REFORMULATE.

DatasetOurs Acc. / Tok. (K)Equal-quota blindUniform AUGMENT → REFORMULATEUniform REFORMULATE
CodeContests61.82 / 8.6960.41 / 8.7258.79 / 6.5060.61 / 9.00
KodCode85.33 / 1.9485.33 / 1.9479.33 / 1.8285.33 / 1.94
APPS83.33 / 3.5081.57 / 3.4678.00 / 2.8683.33 / 3.77
xCodeEval73.58 / 5.8872.38 / 5.6667.92 / 4.4874.53 / 6.22
Micro average75.83 / 5.0374.74 / 4.9970.93 / 3.9475.66 / 5.26

Q1: Further clarification of the policy.

Fixed-strategy iteration repeatedly applies the same type of intervention while solving a failed code. SACP-Coder uses the evolving solver state to select among multiple actions and organizes them progressively, allowing action escalation when needed.

AuthorsOpenReview ↗

Response to Reviewer jr7W (Part 2/3)

W2: Evidence for action selection and retrieval choices

1. Coarse signal from limited public samples

The number of public samples is determined by the benchmark data. We had considered mitigating the resulting coarse signal by generating additional candidate programs, but did not include this extension in the original paper because enriching that signal was outside our main focus. Specifically, for single-sample problems, we generate additional candidate programs at temperature 0.8: one for 2-Candidate Voting and three for 4-Candidate Voting, in addition to the original Direct code. Their average pass rate is used for state-conditioned action mapping. We also test One Random Sample, which retains one randomly selected original public sample, and 50% of Samples, which randomly retains half of the original samples. Results for One Random Sample and 50% of Samples are averaged over 10 seeds; for the latter, half is rounded up. As the results show, reducing samples lowers accuracy, while candidate voting improves accuracy at additional estimated token cost. We have added these results to the appendix.

DatasetOriginal2-Candidate Voting4-Candidate VotingOne Random Sample50% of Samples
CodeContests61.82 / 8.6963.64 / 9.6163.03 / 9.8959.58 / 8.1260.18 / 8.44
KodCode85.33 / 1.9485.33 / 2.7886.00 / 3.2585.33 / 1.9485.33 / 1.94
APPS83.33 / 3.5083.33 / 3.6083.33 / 3.6778.27 / 2.8280.80 / 3.13
xCodeEval73.58 / 5.8873.58 / 5.9473.58 / 5.9972.17 / 5.1572.26 / 5.33
Micro Avg.75.83 / 5.0376.36 / 5.5576.36 / 5.7973.59 / 4.5574.45 / 4.76

2. Retrieval and reference selection

On the same problems selected for AUGMENT, we compare the original retrieval pool with random and BM25 pools, and code-length-based selection with random, difficulty-based, and all-example selection. As shown in the table in our response to Reviewer Wstf (Q3), problem-type-based retrieval outperforms the alternative pools. Code-length-based selection matches random and difficulty-based selection in accuracy while allowing better control over the lengths of selected examples without requiring problem difficulty labels. Using all examples lowers accuracy and increases token consumption. AUGMENT serves as a lower-cost first attempt within the progressive policy. It is not, by itself, our primary novelty. We use code-length-based selection because it is simple and requires no difficulty labels.

3. Token-nearest comparisons in Table 4

The original Table 4 reports only SACP-Coder (R=1R=1), which uses more tokens than the simpler RoutingGen workflow but also achieves substantially higher accuracy. To address the token-cost concern, we additionally report SACP-Coder (R=0R=0) and Direct + AUGMENT as lower-computation settings. R=0R=0 substantially improves accuracy at a moderately higher token cost, while Direct + AUGMENT improves accuracy and reduces token consumption on both backbones. These results show that SACP-Coder retains an accuracy–token advantage across different backbone models.

MethodQwen3-Coder-30BGLM-4.5-Air
Acc. (%) ↑Tok. (K) ↓Acc. (%) ↑Tok. (K) ↓
RoutingGen40.284.8246.414.14
Ours (Direct + AUGMENT)41.16 (+0.88 pp)3.43 (−28.83%)48.51 (+2.10 pp)3.01 (−27.32%)
Ours (R=0R=0)44.83 (+4.55 pp)5.61 (+16.50%)53.42 (+7.01 pp)4.87 (+17.62%)
Ours (R=1R=1)47.99 (+7.71 pp)8.80 (+82.7%)54.99 (+8.58 pp)7.42 (+79.31%)
AuthorsOpenReview ↗

Response to Reviewer jr7W (Part 3/3)

Q2: Reliability of the routing signal

We report the distributions of public-sample counts and initial RETAIN, AUGMENT, and REFORMULATE decisions across all four benchmarks in the table below. Problems without public samples return Direct and are counted as RETAIN here. The concern about the coarse signal from limited public samples and our mitigation experiments are addressed in our response to W2.

Benchmark0 samples1 sample2 samples≥3 samplesRETAINAUGMENTREFORMULATE
CodeContest0.00%65.45%13.33%21.21%46.67%6.06%47.27%
KodCode0.00%100.00%0.00%0.00%84.67%0.00%15.33%
APPS8.67%10.67%31.33%49.33%64.67%12.00%23.33%
xCodeEval0.00%9.43%44.34%46.23%67.92%10.38%21.70%
Overall2.28%49.74%20.32%27.67%65.32%6.83%27.85%

Q3: Retrieval bank initialization and updates

The retrieval bank is initially empty and grows online as problems are processed. After each problem is completed, its generated solution is added to the bank only if it passes the hidden tests. Retrieval for the current problem uses only successful cases from previously processed problems. Neither the current nor future problems’ reference solutions are available to it.

Reviewer_wJ8ZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes SACP-Coder, a framework that adapts inference-time computation to the evolving state of code generation. It uses public-test execution feedback to select among retrieval augmentation, problem reformulation, and code correction. The reformulation mechanism derives a simpler, structurally related problem to guide solution generation. Experiments across four coding benchmarks and three LLM backbones show comparable or higher accuracy than existing methods while substantially reducing token consumption relative to similarly accurate baselines.

Critical strengths

The paper addresses the practical problem of reducing unnecessary computation in code generation. Using public-test feedback to adjust the solving strategy provides a straightforward approach to allocating inference-time computation. The evaluation covers four benchmarks and three LLM backbones, showing substantial token savings while maintaining comparable or higher accuracy than similarly accurate baselines. The comparisons of allocation strategies and component ablations provide additional empirical support for the proposed approach.

Critical weaknesses

The proposed state-adaptive policy relies on a public-test pass-ratio threshold and predefined action transitions. Conditional reasoning after failed code generation [1] and execution-feedback-based correction [2] have already been studied. Related-example generation also has precedent in analogical prompting [3], although it differs from the proposed reformulation mechanism. While the combination is practical, the methodological contribution beyond these existing approaches remains unclear.

The experimental evidence does not fully establish the contribution of the proposed mechanisms. Although the ablations and comparisons with analogical prompting support their empirical effectiveness, they do not establish whether the generated simpler problems preserve the claimed algorithmic structure or whether this property explains the gains. The construction of the retrieval case bank, particularly its separation from evaluation instances and associated preparation cost, also requires clarification.

[1] What Makes Large Language Models Reason in (Multi-Turn) Code Generation? ICLR 2025.

[2] Teaching Large Language Models to Self-Debug. ICLR 2024.

[3] Large Language Models as Analogical Reasoners. ICLR 2024.

Questions for authors

Please address the remaining concerns raised in the weaknesses above.

5 comments
AuthorsOpenReview ↗

Response to Reviewer wJ8Z

Thank you for your review and constructive suggestions. We respond to your concerns below and have updated the manuscript accordingly. Please see the corresponding additions highlighted in blue.

W1: Contribution and novelty

As stated in our General Response, SACP-Coder is not a simple aggregation of [1]–[3]. Its state-conditioned progressive workflow and REFORMULATE differ from the corresponding parts of these methods in concrete ways, as follows.

  • CoT-retry: CoT-retry invokes additional reasoning after an initial code attempt fails. SACP-Coder progressively escalates across different actions as the solving state changes.
  • Self-Debug: Self-Debug repeatedly revises generated code through the same type of action. SACP-Coder instead selects among distinct actions according to the solver state, so different failure states can lead to different actions.
  • Analogical prompting: Analogical prompting self-generates solved examples of separate analogous problems as in-context guidance. REFORMULATE instead simplifies the target problem itself while preserving its core algorithmic structure, then transfers the simpler problem’s solution pattern back to the original target.

The following additional comparisons show that SACP-Coder achieves the highest micro-average accuracy among these methods while using fewer tokens than Self-Debug. Our controlled action-selection comparisons in the response to Reviewer jr7W (W1) further isolate the benefit of selecting actions according to solver state.

Method (Acc. / Tok. (K) )CodeContestsKodCodeAPPSxCodeEvalMicro Avg.
Ours (R=1)61.82 / 8.6985.33 / 1.9483.33 / 3.5073.58 / 5.8875.83 / 5.03
Ours (R=2)68.48 / 14.2985.33 / 2.0483.33 / 6.1173.58 / 10.5277.76 / 8.22
Ours (R=3)68.48 / 19.2085.33 / 2.1684.67 / 10.0273.58 / 18.3078.11 / 12.14
Self-Debug (1 turn)55.15 / 8.5180.67 / 5.7076.67 / 5.8870.75 / 5.7970.40 / 6.58
Self-Debug (2 turns)61.21 / 15.7981.33 / 7.2081.33 / 8.4072.64 / 8.8473.91 / 10.30
Self-Debug (3 turns)64.85 / 23.1983.33 / 7.8782.00 / 10.4372.64 / 10.4975.66 / 13.45
CoT-retry (1 attempt)41.82 / 1.0775.33 / 0.7968.00 / 0.8357.55 / 0.8160.42 / 0.88
CoT-retry (2 attempts)48.48 / 2.4380.67 / 0.9972.67 / 1.2857.55 / 0.8164.97 / 1.45
CoT-retry (3 attempts)52.12 / 4.6682.67 / 1.1277.33 / 1.8757.55 / 0.8167.78 / 2.28
Analogical prompting47.27 / 4.2774.67 / 2.4832.00 / 3.0666.98 / 3.3754.12 / 3.31

W2: Structural-Preservation Evidence and Retrieval Bank Construction

1. Effect of the Structural-Preservation Constraint.

To assess structural preservation directly, we manually compared 40 cases actually routed to REFORMULATE with their generated simpler problems. We found that 90% retained this structure. The remaining cases lost this structure because the original problems were oversimplified. We further evaluate the no-structure variant on the same challenging problems as in Figure 4(b), whose initial Direct public-test pass ratio is zero. This variant removes the structural-preservation instructions from the REFORMULATE prompt while keeping the other settings fixed. Its accuracy is lower at every tested iteration budget, supporting the contribution of this constraint on challenging problems.

Iteration (Acc. / Tok. (K))SACP-CoderNo-structure
R=028.10 / 6.9224.79 / 6.97
R=149.59 / 14.8744.63 / 17.99
R=257.85 / 25.5149.59 / 27.41

2. Case-bank construction and preparation cost.

The case bank starts empty and grows online. After each problem, a generated solution enters the bank only if it passes the hidden tests. A test instance can retrieve successful solutions only from previously processed problems, never from itself or future instances. The model calls used to build and query the case bank, including problem-type labeling, are included in the reported token consumption.

Dear Reviewer wJ8Z,

Our revision addresses what distinguishes SACP-Coder from prior work and adds evidence for the proposed reformulation mechanism. We have also clarified how the case bank is built.

We would appreciate your feedback and hope you will reconsider your assessment in light of these additions.

Thank you for your consideration.

Dear Reviewer wJ8Z,

We respectfully ask that you consider our responses in your final evaluation of the paper.

Thank you for your time and consideration.

Reviewer_WstfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SACP-Coder, an adaptive code generation framework that dynamically adjusts its repair strategy based on the current state of the solver. It aims to address two limitations of existing approaches in competitive programming: prediction errors caused by difficulty assignment before generation, and the inability of fixed modification strategies to adapt to different solving states. The paper conducts relatively comprehensive experiments with 3 backbone models on 4 benchmarks, comparing SACP-Coder with multiple representative recent baselines. The results demonstrate the effectiveness of SACP-Coder and show that it can improve solving performance while reducing token consumption. In addition, multiple ablation and analytical experiments further validate the contributions of its individual components.

Critical strengths

1.The paper addresses two limitations of existing approaches in competitive programming: static difficulty assignment before generation and the inability of iterative fixing strategies to adapt their modification strategies. The proposed state-adaptive solving mechanism provides a potential approach to addressing these limitations.

2.The paper also has some degree of novelty in its perspective on iterative problem solving. While existing methods primarily focus on making better use of available information during debugging, this work uses the current solving state to guide the selection of modification strategies. This provides an interesting perspective on how iterative code-generation systems could adapt their behavior during problem solving.

Critical weaknesses

First, In Equation 7, the action is determined based on the pass rate on the public tests. The paper assumes that when the pass rate falls between (\rho) and 1, the Augment action should be selected. However, it does not necessarily indicate that the current solution is suitable for incremental augmentation. A partially correct solution may still contain fundamental algorithmic errors that require substantial reformulation rather than improvement. Moreover, although the paper introduces additional execution feedback (fai_{s^*}(t)), it is unclear whether and how this information is incorporated into the action-selection process.

Second, the state-transition threshold ((\rho)) in Equation 6 is defined based on the median pass rate on the validation set and is fixed to 0.5. This design may be sensitive to the composition and difficulty of the public tests. Competitive programming problems often have only a small number of public tests, and their pass rates can be highly skewed: difficult problems may have pass rates close to zero, while simple problems may have pass rates close to one. Although the paper provides experimental validation to support the reasonableness of this value, further validation would be helpful, such as dynamically determining the threshold for each benchmark.

Finally, the action space changes during subsequent iterations: Augment is removed and Reformulate and Correct are merged into a single Correct action. This results in a mismatch and reduction of the available actions compared with the original action space, but the rationale and potential impact of this design choice are not sufficiently explained.

Presentation Issues: There are some presentation and notation issues that should be clarified. For example, the second sentence in Section 3.1 contains a grammatical issue in the description of (T_s = {(x_i, y_i)}) as “its public samples tests.” In addition, the notation for the problem is inconsistent: Section 3.2 uses (s) to denote the problem, while Section 3.1 describes (s) as the “current candidate solution for (s),” which makes the notation ambiguous.

Questions for authors

1.What information from (fai_{s^*}(t)) is actually used when selecting the next action? Is pass rate alone sufficient to distinguish between cases that require incremental correction and cases that require a complete algorithmic reformulation?

2.Why is Augment removed in later iterations, and why are Reformulate and Correct merged into a single Correct action? What is the effect of this design on performance?

3.What is the prediction accuracy of problem-type label, and how does potential label noise affect the Augment mechanism?

4.Why are successful problem-solution pairs grouped according to solution length, with one example randomly selected from each group? Have the authors compared this strategy against alternatives such as difficulty-based selection or using all relevant examples?

  1. Is there an ablation study showing that the retrieved successful examples themselves improve performance, rather than the improvement coming primarily from the state-adaptive action selection?
5 comments
AuthorsOpenReview ↗

Response to Reviewer Wstf (Part 1/3)

Thank you for your constructive suggestions. We address your points below and have revised the manuscript accordingly. Please see the corresponding additions highlighted in blue.

W1 & Q1: Partial pass rate and execution feedback

1. Partial pass rate and suitability for AUGMENT

  • SACP-Coder uses the public-test pass rate as a low-cost probabilistic signal, not a definitive diagnosis of the error type. We agree that partial pass ratio ρ≤r<1\rho\le r < 1 does not guarantee that the current solution is free of fundamental algorithmic errors. Pass rate alone is also insufficient to reliably distinguish cases suitable for incremental augmentation from those requiring substantial reformulation.

  • For cases with a pass ratio ρ≤r<1\rho\le r < 1, AUGMENT is a lower-cost first attempt. If it still fails any public test, SACP-Coder escalates to REFORMULATE when the budget permits. A fundamentally flawed solution is therefore not locked into AUGMENT.

  • To validate the rationale and effectiveness of the AUGMENT design, we compare AUGMENT and REFORMULATE on the same partially passing Direct solutions and report paired correctness and average token consumption. AUGMENT solves some cases that REFORMULATE misses and uses fewer tokens on average. REFORMULATE solves some cases that AUGMENT misses, allowing escalation after an unsuccessful AUGMENT attempt. These results support AUGMENT as a lower-cost first attempt before REFORMULATE.

AUGMENT ↓ / REFORMULATE →CorrectIncorrectAUGMENT overall
Correct25.64%15.38%41.03% / 5.99K
Incorrect12.82%46.15%—
REFORMULATE overall38.46% / 7.01K——

2. Use of execution feedback ϕ\phi

The initial action selection uses only the public-test pass rate. The detailed information in ϕ\phi does not select the initial action. After entering REFORMULATE, failed public-test inputs, expected outputs, and actual outputs or errors in ϕ\phi are provided to the subsequent repair and replanning prompts to guide corrected code or a revised transfer plan.

W2: Benchmark-Specific Threshold Validation

Thank you for acknowledging the experimental evidence in Figure 5 supporting the reasonableness of the default threshold. Following your suggestion, we constructed a separate validation set of 100 problems for each benchmark, with no overlap with its test set, to select a benchmark-specific threshold ρ\rho (Validation). We then fix the selected thresholds and evaluate them on the original test sets. With the paper’s default configuration (R=1R=1), thresholds were selected by maximizing validation accuracy, then minimizing token consumption, and finally choosing the smaller threshold in case of a tie. Arrows compare the default ρ=0.5\rho=0.5 with the selected thresholds. Benchmark-specific selection reduces average token consumption by 3.18%, with a 0.52 pp decrease in accuracy.

BenchmarkSelected ρ (Validation)Test Acc. (%)Test Tok. (K)
CodeContests1/1261.82 → 60.618.69 → 8.39
KodCode0.585.33 → 85.331.94 → 1.94
APPS0.2583.33 → 83.333.50 → 3.29
xCodeEval1/1473.58 → 72.645.88 → 5.80
Micro Avg.—75.83 → 75.315.03 → 4.87
AuthorsOpenReview ↗

Response to Reviewer Wstf (Part 2/3)

W3 & Q2: Action space during iterations

1. Removal of AUGMENT in later iterations.

After State-Conditioned Action Mapping, we consider two cases to explain why AUGMENT is removed in later iterations:

  • If the initial AUGMENT attempt fails the public tests, this motivates a stronger intervention through REFORMULATE rather than repeated AUGMENT.
  • If REFORMULATE fails, its plan, failed code, and execution feedback provide a basis for CORRECT to revise the implementation or transfer plan. Returning to AUGMENT would not use this accumulated failure feedback.

To test the effect of this choice, we allow AUGMENT in later iterations when the subsequent pass rate is at least ρ\rho but below one. If AUGMENT was used before, we select a different example combination from the same frozen candidate pool. All other settings and the update budget remain fixed, with each AUGMENT or CORRECT consuming one update. This variant leaves accuracy unchanged and reduces average token consumption by about 1.6%, indicating a small effect under the paper’s default budget.

StrategyMicro Avg. Acc. (%) ↑Micro Avg. Tok. (K) ↓
SACP-Coder (Default)75.835.03
Retaining AUGMENT in later iterations75.834.95

2. Relationship between REFORMULATE and CORRECT.

Initial REFORMULATE constructs a simpler problem and a transfer plan. After a failed attempt, CORRECT uses the existing plan, failed code, and execution feedback either to repair the implementation through REWRITE or to revise the transfer plan through REPLAN. Both forms revise an existing attempt, so we group them under CORRECT. This grouping is terminological: treating REWRITE and REPLAN as separate top-level actions would not change execution or performance if their selection rules, prompts, and update accounting remained the same.

AuthorsOpenReview ↗

Response to Reviewer Wstf (Part 3/3)

Q3 & Q4: AUGMENT with Candidate Filtering

1. Problem-type labels and candidate retrieval.

We conduct a label-coverage evaluation on CodeContests and xCodeEval, which provide native problem-type labels. As the table shows, the generated labels cover over 80% of the native labels for both queries and examples, while also providing finer-grained problem-type labels. In a manual audit of complete generated problem-type profiles for 40 problems (10 per benchmark), 95% were judged accurate. The errors mainly stemmed from overly coarse problem analysis. To test whether label-based retrieval remains useful despite possible label noise, we replace it with either a randomly formed candidate pool or a BM25-ranked pool, while keeping the original code-length filtering. Both alternatives reduce accuracy on the same AUGMENT cases.

Evaluation targetNative-label hit rate (%)
CodeContests query90.18
CodeContests example91.30
xCodeEval query83.02
xCodeEval example85.48
MethodAccuracy (%)Avg. Tok. (K)
AUGMENT41.035.99
Random pool + original selection35.905.53
BM25 pool + original selection33.336.07
Original pool + random selection41.036.26
Original pool + difficulty-based selection41.035.90
Original pool + all examples33.3310.78

All variants in the second table are evaluated on the same problems initially assigned to AUGMENT under the paper’s default configuration.

2. Reason for code-length filtering.

Grouping retrieved candidates by solution length and sampling one from each group helps control the lengths of selected references and their input-token cost. It requires neither native difficulty labels nor extra model calls to generate them. This simple rule suits AUGMENT’s role as a lower-cost first attempt before REFORMULATE. We compare it with random selection, difficulty-based selection, and using all retrieved candidates on the same fixed pool and cases. Random and difficulty-based selection match its accuracy, while using all candidates lowers accuracy and increases token consumption. Random selection offers no explicit length control, and difficulty-based selection depends on native or generated difficulty labels.

Q5: Contribution of retrieved successful examples.

Table 3 in the original paper shows that adding AUGMENT to Direct improves accuracy on three benchmarks, with KodCode unchanged. The paired comparison in our W1 & Q1 response further shows that AUGMENT solves some cases REFORMULATE misses. To separate the effect of references from state-conditioned action selection, the comparisons in our Q3 & Q4 response change the candidate pool or example selection. Random and BM25 pools both reduce accuracy. These results show that the retrieved references and their selection contribute beyond the action decision alone.

W4. Presentation

Thank you for pointing out these presentation issues. We corrected “public samples tests” to “public sample tests” and clarified the notation in Section 3.1: ss denotes the programming problem, while fs(t)f_s^{(t)} denotes its current candidate program at step tt. These symbols are used consistently throughout the method section.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their careful reading and constructive feedback. We have strengthened the main text with evidence for our core contributions and added further analyses to the appendix. New manuscript content is highlighted in blue. To preserve the numbering of the original tables, newly added tables are numbered Table R1, Table R2, and so on.

We reaffirm SACP-Coder's novelty and clarify AUGMENT’s role in the progressive workflow for code generation. The reviewer-specific responses provide corresponding controlled comparisons and detailed experimental results.

  1. Restating SACP-Coder's Novelty.

    • State-conditioned progressive workflow. SACP-Coder organizes AUGMENT, REFORMULATE, and CORRECT into a progressive workflow, using the evolving solver state to select the applicable action and escalate when needed. This state-conditioned progression allocates computation to reduce unnecessary token use and goes beyond a simple combination of the individual actions.
    • The unique REFORMULATE mechanism. REFORMULATE derives a simpler problem while preserving the target problem’s core algorithmic structure, identifies the key solution pattern of the simpler problem, and transfers that pattern back to the original target. This target-problem transformation differs from supplying analogous but different examples or only a code-generation plan.
  2. Clarifying the Role of AUGMENT.

    • Position and candidate retrieval. Within the progressive workflow, AUGMENT serves as a lower-cost first attempt. If AUGMENT remains unsuccessful, the workflow escalates to REFORMULATE when the action-update budget permits. It uses problem-type labels to retrieve related solved cases from the case bank into a candidate pool.
    • Candidate filtering. It groups retrieved cases by the code lengths of their solutions and randomly selects one case from each group. This simple rule helps control the lengths of selected references and their input-token cost without requiring difficulty labels or model calls to estimate difficulty.
    • Evidence. Comparisons with alternative candidate pools and reference-selection rules, together with the paired comparison against REFORMULATE on the same cases, support AUGMENT’s role as a lower-cost first attempt.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.