- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SACP-Coder, an inference-time framework that aims to balance code-generation accuracy and token consumption. It defines a solver state using the current candidate program, its pass ratio on public samples, and execution feedback. After an initial direct-generation attempt, the framework uses predefined rules to retain the solution, retrieve reference solutions, reformulate the problem into a simpler structurally related problem, or perform code-level correction and replanning. Experiments on four programming benchmarks and three LLM backbones report competitive accuracy with substantially lower token consumption than comparably accurate multi-agent baselines.
Critical strengths
- A practically relevant accuracy–cost trade-off. Reducing the inference cost of agentic code generation is an important problem. The method does achieve less token consumption and relative the same performance comparing multi-agent methods. The results suggest that the proposed workflow could provide practical value.
- Evaluation beyond a single default configuration. The paper evaluates multiple benchmarks and backbones, includes cumulative component ablations, and studies accuracy–token trade-offs as the iteration limit changes. The comparison with uniformly applying reformulation and correction is particularly useful for assessing the benefit of selective computation.
Critical weaknesses
- The methodological novelty and the distinction from existing feedback-driven workflows are not sufficiently established. Although the method is formulated as a sequential decision process over solver states, the initial action is determined by a threshold on public-sample pass ratio, and subsequent failures largely follow a predefined escalation sequence. The distinction between 'rewrite' and 'replan' is also based on predefined checks. These choices may be useful, but the paper does not sufficiently establish what is gained beyond combining familiar mechanisms in a heuristic workflow.
- The evidence does not sufficiently isolate or validate the key routing and retrieval choices. The cumulative ablations and scaling curves provide useful evidence, but leave important questions unresolved. First, the routing signal can be very coarse: the supplied CodeContests artifact records only one public sample I/O pair for 108 of 165 problems. For those single-pair problems, the pass ratio is necessarily either zero or one, so the intermediate 'augment' branch cannot be selected. Second, Table 3 evaluates adding AUGMENT, but does not isolate the contribution of label-based retrieval and code-length-based reference selection relative to simpler selection rules. Finally, in Table 4, the token-nearest comparisons still involve more token consumption than baselines.
Questions for authors
- Clarification of novelty: What precisely is held fixed in “fixed-strategy iteration”? How does SACP-Coder differ from methods that already revise reasoning or plans using execution feedback? Please clarify the specific contribution of the state-conditioned policy beyond the individual retrieval, reformulation, and correction mechanisms.
- Reliability of the routing signal: What are the distributions of public sample counts and initial routing decisions across all four benchmarks?
- Retrieval protocol and contribution: How is the case bank initialized and updated, and which tests establish that a stored solution is successful?
6 comments
Response to Reviewer jr7W (Part 2/3)
W2: Evidence for action selection and retrieval choices
1. Coarse signal from limited public samples
The number of public samples is determined by the benchmark data. We had considered mitigating the resulting coarse signal by generating additional candidate programs, but did not include this extension in the original paper because enriching that signal was outside our main focus. Specifically, for single-sample problems, we generate additional candidate programs at temperature 0.8: one for 2-Candidate Voting and three for 4-Candidate Voting, in addition to the original Direct code. Their average pass rate is used for state-conditioned action mapping. We also test One Random Sample, which retains one randomly selected original public sample, and 50% of Samples, which randomly retains half of the original samples. Results for One Random Sample and 50% of Samples are averaged over 10 seeds; for the latter, half is rounded up. As the results show, reducing samples lowers accuracy, while candidate voting improves accuracy at additional estimated token cost. We have added these results to the appendix.
| Dataset | Original | 2-Candidate Voting | 4-Candidate Voting | One Random Sample | 50% of Samples |
|---|---|---|---|---|---|
| CodeContests | 61.82 / 8.69 | 63.64 / 9.61 | 63.03 / 9.89 | 59.58 / 8.12 | 60.18 / 8.44 |
| KodCode | 85.33 / 1.94 | 85.33 / 2.78 | 86.00 / 3.25 | 85.33 / 1.94 | 85.33 / 1.94 |
| APPS | 83.33 / 3.50 | 83.33 / 3.60 | 83.33 / 3.67 | 78.27 / 2.82 | 80.80 / 3.13 |
| xCodeEval | 73.58 / 5.88 | 73.58 / 5.94 | 73.58 / 5.99 | 72.17 / 5.15 | 72.26 / 5.33 |
| Micro Avg. | 75.83 / 5.03 | 76.36 / 5.55 | 76.36 / 5.79 | 73.59 / 4.55 | 74.45 / 4.76 |
2. Retrieval and reference selection
On the same problems selected for AUGMENT, we compare the original retrieval pool with random and BM25 pools, and code-length-based selection with random, difficulty-based, and all-example selection. As shown in the table in our response to Reviewer Wstf (Q3), problem-type-based retrieval outperforms the alternative pools. Code-length-based selection matches random and difficulty-based selection in accuracy while allowing better control over the lengths of selected examples without requiring problem difficulty labels. Using all examples lowers accuracy and increases token consumption. AUGMENT serves as a lower-cost first attempt within the progressive policy. It is not, by itself, our primary novelty. We use code-length-based selection because it is simple and requires no difficulty labels.
3. Token-nearest comparisons in Table 4
The original Table 4 reports only SACP-Coder (), which uses more tokens than the simpler RoutingGen workflow but also achieves substantially higher accuracy. To address the token-cost concern, we additionally report SACP-Coder () and Direct + AUGMENT as lower-computation settings. substantially improves accuracy at a moderately higher token cost, while Direct + AUGMENT improves accuracy and reduces token consumption on both backbones. These results show that SACP-Coder retains an accuracy–token advantage across different backbone models.
| Method | Qwen3-Coder-30B | GLM-4.5-Air | ||
|---|---|---|---|---|
| Acc. (%) ↑ | Tok. (K) ↓ | Acc. (%) ↑ | Tok. (K) ↓ | |
| RoutingGen | 40.28 | 4.82 | 46.41 | 4.14 |
| Ours (Direct + AUGMENT) | 41.16 (+0.88 pp) | 3.43 (−28.83%) | 48.51 (+2.10 pp) | 3.01 (−27.32%) |
| Ours () | 44.83 (+4.55 pp) | 5.61 (+16.50%) | 53.42 (+7.01 pp) | 4.87 (+17.62%) |
| Ours () | 47.99 (+7.71 pp) | 8.80 (+82.7%) | 54.99 (+8.58 pp) | 7.42 (+79.31%) |
Response to Reviewer jr7W (Part 3/3)
Q2: Reliability of the routing signal
We report the distributions of public-sample counts and initial RETAIN, AUGMENT, and REFORMULATE decisions across all four benchmarks in the table below. Problems without public samples return Direct and are counted as RETAIN here. The concern about the coarse signal from limited public samples and our mitigation experiments are addressed in our response to W2.
| Benchmark | 0 samples | 1 sample | 2 samples | ≥3 samples | RETAIN | AUGMENT | REFORMULATE |
|---|---|---|---|---|---|---|---|
| CodeContest | 0.00% | 65.45% | 13.33% | 21.21% | 46.67% | 6.06% | 47.27% |
| KodCode | 0.00% | 100.00% | 0.00% | 0.00% | 84.67% | 0.00% | 15.33% |
| APPS | 8.67% | 10.67% | 31.33% | 49.33% | 64.67% | 12.00% | 23.33% |
| xCodeEval | 0.00% | 9.43% | 44.34% | 46.23% | 67.92% | 10.38% | 21.70% |
| Overall | 2.28% | 49.74% | 20.32% | 27.67% | 65.32% | 6.83% | 27.85% |
Q3: Retrieval bank initialization and updates
The retrieval bank is initially empty and grows online as problems are processed. After each problem is completed, its generated solution is added to the bank only if it passes the hidden tests. Retrieval for the current problem uses only successful cases from previously processed problems. Neither the current nor future problems’ reference solutions are available to it.
Response to Reviewer jr7W (Part 1/3)
We appreciate your constructive review and specific questions. We answer them below and have revised the manuscript to address the points you raised. The corresponding additions are highlighted in blue.
W1&Q1: Novelty
W1: Novelty and distinction.
Compared with combining familiar mechanisms in a heuristic workflow, the novelty and distinction of SACP-Coder lie in two aspects:
State-conditioned progressive workflow. Instead of applying the same correction strategy after each failure, SACP-Coder uses the solver state to select among different actions and progressively escalates when needed, reducing unnecessary token consumption.
The unique REFORMULATE mechanism. Unlike familiar mechanisms that generate related solved examples or only a code-generation plan, REFORMULATE derives a simpler problem while preserving the target problem’s core algorithmic structure, identifies the key solution pattern of the resulting simpler problem, and transfers that pattern back to the original target.
To further qualify the distinction between SACP-Coder and existing workflows, we added controlled comparisons of action selection while holding the interventions and update budget fixed:
Together, these comparisons show that state-conditioned selection achieves the highest micro-average accuracy among the three controls, while using fewer tokens than uniform REFORMULATE.
Q1: Further clarification of the policy.
Fixed-strategy iteration repeatedly applies the same type of intervention while solving a failed code. SACP-Coder uses the evolving solver state to select among multiple actions and organizes them progressively, allowing action escalation when needed.