acceptodds
Under review as a conference paper at ICLR 2027

Improving Test-Time Scaling for Software Engineering Agents

Abstract

Test-time scaling (TTS) offers a promising path for solving complex software engineering tasks, yet its effectiveness depends on generating correct solutions within a fixed rollout budget. We introduce EntroPO, an entropy-enhanced preference optimization framework for multi-turn, tool-using agents. We formulate agent alignment as an entropy-regularized MDP and derive the EntroPO-DPO and EntroPO-KTO objectives. Theoretically, we analyze why standard DPO/M-DPO objectives collapse and show that EntroPO promotes both accuracy and diversity among correct solutions. Empirically, we validate EntroPO on SWEBench, LiveCodeBench, and TerminalBench, where it achieves state-of-the-art results among open-weight models. At the time of leaderboard submission, our 30B model ranked 1st on SWEBench-Lite and 4th on SWEBench-Verified among open-weight submissions, trailing only models over 10× larger on SWEBench-Verified. These results demonstrate the effectiveness of EntroPO for improving software engineering performance with test-time compute scaling. To support further research, we release our code and data through an anonymous link.

Primary area:
foundation or frontier models, including LLMs
Keywords:
software engineering agents, preference optimization, test-time scaling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6muTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

To mitigate diversity collapse in standard preference optimization methods, such as DPO, and further improve test time scaling on coding tasks. This paper proposes EntroPO, a preference optimization framework for multi-turn coding agents. Specifically, the authors formulate preference optimization as an entropy-regularized MDP and derive two objectives, EntroPO-DPO and EntroPO-KTO, for pairwise and pointwise feedback, respectively. Theoretically, they show that EntroPO encourages greater diversity among correct trajectories and can improve accuracy over standard DPO under certain conditions. The authors also conducted extensive experiments that evaluate EntroPO across multiple model families and four benchmarks, including SWE-bench Verified, SWE-bench Lite, LiveCodeBench, and TerminalBench. The results demonstrate improvements in both single-rollout performance and test-time scaling, with competitive results on the SWE-bench leaderboards.

Critical strengths
  1. The proposed method mitigates the important problem of diversity collapse in preference-based training of multi-turn coding agents, which is relevant to improving test-time scaling.
  2. The extensive experiments support for the effectiveness and generalizability of the proposed method.
Critical weaknesses
  1. Although EntroPO achieves higher pass@16 than the baselines, this does not provide direct evidence for improved solution diversity. The gains in test time scaling could also be partly attributed to improved single-rollout accuracy, even if the diversity of generated solutions is still limited. That said, I consider this is a relatively minor concern, and would not significantly affect my overall recommendation for acceptance.
Questions for authors
  1. Could the authors provide more analysis on the relevance between the improved performance and the diversity of agent trajectory?
4 comments

We thank Reviewer 6muT for the constructive and insightful comments. Please see our response to each of your questions below.

Although EntroPO achieves higher pass@16 than the baselines, this does not provide direct evidence for improved solution diversity. Could the authors provide more analysis on the relevance between the improved performance and the diversity of agent trajectory?

Response: Thanks for the question. We agree that pass@16 alone does not show improved diversity, and that part of the gain can come from higher single-rollout accuracy. We have added direct measurements of diversity among correct solutions to the revised paper (Appendix D.8, summarized in Section 4.3), and we relate them to the gain from test-time scaling below.

Diversity measures (SWE-bench Verified, Qwen3-Coder, 16 rollouts per task):

  • Correct-set entropy. Following [1,2], we measure the length-normalized predictive entropy on the model's self-generated tokens, computed on the rollouts that pass the tests. It is the empirical counterpart of the correct-set entropy H+H^+ in Theorem 3.6.

  • DA@16. Following [3], we normalize and deduplicate the patches from the correct rollouts, and an LLM judge (gpt 6.1 terra with codex harness) decides whether each pair of patches implements the same approach. DA@16 (distinct algorithms) is the number of distinct approaches among the correct patches from 16 rollouts, averaged over tasks.

Relation between diversity and performance:

ModelSingle rollout (%)TTS, N=16 (%)Gain from scalingCorrect-set entropy (nats/token)DA@16
Base model37.742.5+4.80.16011.33
M-DPO47.852.8+5.00.14871.35
EntroPO-DPO49.857.7+7.90.19232.29

Two observations separate the effect of accuracy from the effect of diversity:

  1. M-DPO improves single-rollout accuracy by 10.1 points over the base model, but it does not increase diversity among correct solutions (lower entropy, nearly identical DA@16), and its gain from scaling is the same as the base model's (+5.0 vs. +4.8). In this comparison, higher single-rollout accuracy alone does not make test-time scaling more effective.
  2. EntroPO-DPO improves single-rollout accuracy by a further 2.0 points, which accounts for part of its advantage, as you note. Its correct-set entropy is also 29.3% higher than M-DPO's, it produces about 1.7x as many distinct correct approaches, and its gain from scaling rises to +7.9 points.

The larger gain from test-time scaling therefore appears together with higher diversity among correct solutions, not with higher single-rollout accuracy alone. We also track entropy during training: M-DPO's entropy falls below the base model, while EntroPO-DPO's rises from the first checkpoint (Figure 6).

We have added these results and the discussion to Appendix D.8 (Table 12, Figure 6) in the revised paper.

References

[1] Uncertainty Estimation in Autoregressive Structured Prediction. ICLR 2021.

[2] Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints. ICLR 2024.

[3] How Diversely Can Language Models Solve Problems? Exploring the Algorithmic Diversity of Model-Generated Code. Findings of EMNLP 2025.

I appreciate that the authors provided additional measurements of diversity among correct solutions and further analysis of the relationship between diversity and test-time scaling performance. These results have addressed my concern.

AuthorsOpenReview ↗

Thanks for your feedback

Thank you for taking the time to read our response and the revised paper. We are glad that the additional diversity measurements and the analysis of their relationship with test-time scaling have addressed your concern. We appreciate your constructive feedback, which helped us improve the paper.

We hope you have a great weekend!

AuthorsOpenReview ↗

Thank you, and a final check

Dear Reviewer 6muT,

Thank you again for your reply. We are glad that the additional diversity measurements addressed your concern. As the discussion period is coming to an end, we would like to check whether you have any remaining questions. We are more than happy to address them and revise the paper accordingly.

If there are none, we would be grateful if you would consider reflecting this in your final score.

Best regards,

The authors

Reviewer_gkzzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes EntroPO, an offline preference-optimization framework for multi-turn, tool-using software engineering agents. It adds entropy regularization to a reference-constrained objective, derives DPO- and KTO-style losses with separate coefficients for learned-policy and reference-policy log probabilities, and analyzes the optimal trajectory distribution under deterministic transitions. A binary-reward analysis studies accuracy and conditional entropy among correct trajectories. Experiments on SWE-bench Verified/Lite, LiveCodeBench, and TerminalBench compare preference-training methods, repeated-sampling budgets, and candidate selectors. The reported results show improved task success, including when selecting one final patch from multiple rollouts.

Critical strengths
  1. Consistent empirical gains on a useful application. The evaluation spans SWE-bench Verified/Lite, LiveCodeBench, and TerminalBench, with several model families and common evaluation settings within comparisons. The paper reports variation across three independently trained models. On SWE-bench Verified with Qwen3-Coder and 16 candidate rollouts, the selected-patch resolve rate improves from 52.8 to 57.7 for M-DPO versus EntroPO-DPO and from 53.8 to 59.4 for M-KTO versus EntroPO-KTO (Table 1). These results provide useful evidence for practitioners even if the DPO objective is closely related to prior work.

  2. A practical offline recipe with informative analysis. The modification is lightweight and requires no additional online training rollouts. Comparisons across selectors, raw-candidate accuracy/coverage, and tool-call/token usage help characterize its practical benefits. The theoretical distinction between correctness probability and entropy within the correct set is also useful; Theorem 3.6 explicitly states the conditions for its ideal-optimum accuracy result.

Overall, I lean toward weak acceptance. The breadth of the empirical improvements and the practical training recipe provide value. My reservations concern the precise contribution relative to SPL and the empirical support for the diversity mechanism. These appear addressable through clearer positioning and appropriately bounded claims; they do not negate the reported task-success gains.

Critical weaknesses
  1. The relationship to trajectory-level SPL needs a more precise account. Let ℓθ(τ)=∑hlog⁡πθ(ah∣sh)\ell_\theta(\tau)=\sum_h\log\pi_\theta(a_h\mid s_h), with Δℓ\Delta\ell denoting the preferred-minus-rejected difference. Equation (6) becomes −log⁡σ(ζΔℓθ−γΔℓref)-\log\sigma(\zeta\Delta\ell_\theta-\gamma\Delta\ell_{\rm ref}). Under the paper's deterministic, full-history setting and identical masking/aggregation of model-generated actions, this appears identical to SPL Equation (5), with α=ζ\alpha=\zeta and β=γ\beta=\gamma, applied to complete trajectories. Environment observations remain conditioning context rather than prediction targets. Existing multi-turn preference optimization also already addresses tool-feedback masking and the deterministic-transition reduction.

Consequently, the related-work statement that extending these objectives to multi-turn settings remains unclear needs further explanation. The paper should distinguish the contributions of the DPO objective, the KTO extension, the correctness/diversity analysis, and the SWE evaluation. SPL is not directly compared in the reported experiments, but an additional run would be redundant if the objectives and implementations are equivalent. The key issue is accurate attribution and a clear account of the incremental contribution.

  1. The empirical evidence for diversity among correct solutions is less direct than the evidence for improved task success. Table 9 shows useful gains in both mean@16 and pass@16, but neither directly measures the distribution within the correct set. For independent rollouts on a fixed task, pass@N is 1−(1−p)N1-(1-p)^N, where pp is per-rollout success probability. Thus, improved candidate coverage alone does not establish greater correct-solution diversity. The qualitative example is illustrative but does not establish this property across tasks. Theorem 3.6 provides a conditional ideal-optimum analysis; the empirical findings should be distinguished from that theoretical result. This limits the strength of the mechanism interpretation, rather than undermining the observed performance improvement.

References:

Questions for authors
  1. With identical trajectory conditioning, action-token masking, and aggregation, is EntroPO-DPO equivalent to SPL on complete trajectory pairs? Please clarify this relationship and identify the incremental contributions of the KTO extension, the theoretical analysis, and the SWE experiments. If there is a substantive difference, please specify it. Establishing equivalence would resolve the need for a separately labeled but identical baseline.

  2. Is there evidence in the existing sampled trajectories for increased diversity among correct solutions, beyond improvements in per-rollout accuracy and candidate coverage? A modest analysis of successful patches or strategies could help, while acknowledging its limitations as a proxy for trajectory entropy. Alternatively, a clearer separation of the theoretical diversity result from the empirical success-rate findings would address this concern without requiring a new experimental campaign.

4 comments

We thank Reviewer gkzz for the constructive and insightful comments. Please see our response to each of your questions below.

The relationship to trajectory-level SPL needs a more precise account.

Response: Thank you for your comment. Under deterministic transitions, with identical trajectory conditioning, action-token masking and aggregation, the EntroPO-DPO loss (Eq. 6) is identical to SPL Eq. (5) applied to complete trajectory pairs, with α = ζ and β = γ. EntroPO-DPO can therefore be read as trajectory-level SPL, and as you note, a separately labeled SPL baseline would be redundant.

We have made the following changes in the revised paper:

  • Related work (Section 2). We removed the statement that extending these objectives to multi-turn settings remains unclear. The text now identifies SPL as one of the closest prior works and states the equivalence above.
  • After Theorem 3.2. We added a remark that Eq. (6) coincides with SPL in the single-turn case and with trajectory-level SPL under deterministic transitions.

However, to the best of our knowledge, we are the first to systematically study correctness and diversity jointly in multi-turn, tool-using agents. This is more than an empirical extension of SPL:

  1. Derivation for the multi-turn setting. SPL is formulated for single-turn prompt-response pairs. We derive the loss from the multi-turn entropy-regularized objective (Eq. 2). Lemma 3.1 holds for general transitions, both deterministic and stochastic, and isolates a transition-value residual Δ∗(π)\Delta^*(\pi). The trajectory-level form is exact when this residual vanishes, as under the deterministic transitions of our SWE setting.
  2. KTO extension. SPL is formulated for pairwise preferences. EntroPO-KTO (Appendix B) uses pointwise correct/incorrect labels and needs no preference pairs. It is also the stronger variant on SWE-bench Verified with Qwen3-Coder.
  3. Correctness and diversity analysis. SPL's theory characterizes the optimal policy and its match to the preference distribution, and its accuracy gains are shown empirically. Theorem 3.6 analyzes the binary-reward setting of verifiable tasks: the EntroPO optimum has conditional entropy over correct solutions at least as high as the DPO optimum, and, under an explicit sufficient condition, accuracy at least as high.
  4. Complex software engineering with state-of-the-art results. We evaluate tool-using agents on four benchmarks with three model families and test-time scaling. The revision also adds direct measurements of entropy and of strategy-level diversity among correct solutions.

The empirical evidence for diversity among correct solutions is less direct than the evidence for improved task success.

Response: Thanks for the question. The existing sampled trajectories show increased diversity among correct solutions. We agree that mean@16 and pass@16 do not measure the distribution within the correct set, so we have added three analyses to the revised paper (Appendix D.8, summarized in Section 4.3): entropy at inference, entropy during training, and strategy-level diversity of correct solutions. Analyses (1) and (3) use the existing 16 rollouts per task on SWE-bench Verified and are computed within the correct set.

Entropy metric. Following [1,2], we measure the length-normalized predictive entropy on the model's self-generated tokens. We sample rollouts from the policy, compute the predictive entropy over the tokens the agent generates (prompts and tool observations are excluded), and normalize by the number of generated tokens. This is also how policy entropy is tracked in online RL [3]. We report it on all 16 rollouts per task (full set) and on the rollouts that pass the tests (correct set). The correct-set value is the empirical counterpart of the correct-set entropy H+H^+ in Theorem 3.6.

(1) Entropy at inference. SWE-bench Verified, Qwen3-Coder, 16 rollouts per task (nats/token):

ModelFull setCorrect set
Base model0.16340.1601
M-DPO0.15430.1487
EntroPO-DPO0.18850.1923

M-DPO lowers entropy below the base model (-5.6% on the full set, -7.1% on the correct set), showing that standard preference optimization sharpens the policy. EntroPO-DPO raises it by 22.2% (full set) and 29.3% (correct set) over M-DPO. The increase is largest on the correct set, which is the quantity Theorem 3.6 concerns.

(2) Entropy during training. We follow [2] to generate fresh rollouts from the training checkpoints of both methods on 50 held-out SWE-smith tasks (not used in training), with 16 rollouts per task. Step 0 is the shared base model, and step 196 is the end of training.

Training stepM-DPO (full)EntroPO-DPO (full)M-DPO (correct)EntroPO-DPO (correct)
00.15890.15890.15150.1515
500.15250.15920.15350.1598
1000.14960.16240.14760.1653
1500.14650.16840.14540.1675
1960.14620.16900.14500.1677

The two methods move in opposite directions from the same starting point. M-DPO ends 8.0% (full set) and 4.3% (correct set) below the base model, while EntroPO-DPO ends 6.4% and 10.7% above it. The gap widens at every checkpoint (Figure 6 in the revised paper).

(3) Strategy-level diversity of correct solutions. Higher token-level entropy does not by itself show that correct solutions differ in substance, so we also measure diversity at the level of the solution approach, following [4]. For each SWE-bench Verified task, we take the patches from the correct rollouts, normalize them (comments and whitespace), and remove duplicates. An LLM judge (gpt 6.1 terra with codex harness) then decides whether each pair of patches implements the same approach, using the prompt of [4]. DA@16 (distinct algorithms [4]) is the number of distinct approaches among the correct patches from 16 rollouts, averaged over tasks.

ModelDA@16
Base model1.33
M-DPO1.35
EntroPO-DPO2.29

EntroPO-DPO produces 2.29 distinct correct approaches per task, compared with 1.35 for M-DPO (~1.7x). This is not a by-product of having more correct rollouts. M-DPO raises per-rollout accuracy over the base model from 37.7% to 47.8% (Table 1) but leaves DA@16 almost unchanged (1.33 to 1.35), whereas EntroPO-DPO raises accuracy by a further 2.0 points and DA@16 to 2.29. The additional entropy therefore corresponds to different ways of fixing the same issue.

Together, these results show that EntroPO increases policy entropy during training, that the increase carries over to inference and is largest on correct rollouts, and that it yields more distinct correct solutions per task, which is what test-time scaling benefits from.

We have added these results and the discussion to Appendix D.8 (Table 12, Figure 6) in the revised paper.

References

[1] Uncertainty Estimation in Autoregressive Structured Prediction. ICLR 2021.

[2] Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints. ICLR 2024.

[3] The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models. arXiv 2025.

[4] How Diversely Can Language Models Solve Problems? Exploring the Algorithmic Diversity of Model-Generated Code. Findings of EMNLP 2025.

AuthorsOpenReview ↗

Follow-up before the discussion period ends

Dear Reviewer gkzz,

Thank you again for your careful review. As the discussion period is coming to an end, we would like to kindly check whether our response and the revised paper have addressed your points

If anything remains unclear, we are more than happy to provide further clarification or results and to revise the paper accordingly. If your concerns have been resolved, we would be grateful if you would reconsider your score.

Thank you for your time.

Best regards,

The authors

Dear Reviewer gkzz,

Thank you for your valuable review. The discussion closes in 12 hours. We are still hoping to hear back on our rebuttal. We are happy to address any additional question if you have.

Authors

Reviewer_CvkkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EntroPO, which is a new preference learning technique that introduces entropy regularization during training to improve test-time scaling capabilities of coding agents. Specifically, EntroPO introduces additional entropy regularization to the multi-turn preference learning objective, so that the policy can balance task correctness and the diversity of the generated traces during training and test-time scaling. Evaluation across multiple models and benchmarks shows that EntroPO improves the pass@1 and test-time scaling performance successfully, compared to DPO/KTO without entropy regularization.

Critical strengths
  • This paper explores an interesting and important direction.
Critical weaknesses
  • While the paper focuses on controlling entropy during preference learning, the evaluation does not examine in detail how entropy changes during training or how rollout diversity improves during training and inference, beyond reporting final benchmark pass rates.
  • The comparison with online RL (i.e., GRPO) does not seem entirely fair, as entropy regularization has also become common practice in GRPO training. It would be more informative to also compare EntroPO with GRPO using entropy control.
  • The experiments use only the R2E scaffold as the agent harness, which limits the comprehensiveness of the evaluation.
Questions for authors
  • How does the entropy of generated rollouts differ between policies trained with and without entropy regularization?
  • The paper title seems to be too general and does not demonstrate the concrete idea in this paper.
5 comments

We thank Reviewer Cvkk for the constructive and insightful comments. Please see our response to each of your questions below.

While the paper focuses on controlling entropy during preference learning, the evaluation does not examine in detail how entropy changes during training or how rollout diversity improves during training and inference, beyond reporting final benchmark pass rates. How does the entropy of generated rollouts differ between policies trained with and without entropy regularization?

Response: Thanks for raising this point. We have added three analyses to the revised paper (Appendix D.8, summarized in Section 4.3): entropy at inference, entropy during training, and strategy-level diversity of correct solutions.

Entropy metric. Following [1,2], we measure the length-normalized predictive entropy on the model's self-generated tokens. We sample rollouts from the policy, compute the predictive entropy over the tokens the agent generates (prompts and tool observations are excluded), and normalize by the number of generated tokens. This is also how policy entropy is tracked in online RL [3]. We report it on all 16 rollouts per task (full set) and on the rollouts that pass the tests (correct set). The correct-set value is the empirical counterpart of the correct-set entropy H+H^+ in Theorem 3.6.

(1) Entropy at inference. SWE-bench Verified, Qwen3-Coder, 16 rollouts per task (nats/token):

ModelFull setCorrect set
Base model0.16340.1601
M-DPO0.15430.1487
EntroPO-DPO0.18850.1923

M-DPO lowers entropy below the base model (-5.6% on the full set, -7.1% on the correct set), showing that standard preference optimization sharpens the policy. EntroPO-DPO raises it by 22.2% (full set) and 29.3% (correct set) over M-DPO. The increase is largest on the correct set, which is what Theorem 3.6 concerns.

(2) Entropy during training. We follow [2] to generate fresh rollouts from the training checkpoints of both methods on 50 held-out SWE-smith tasks (not used in training), with 16 rollouts per task. Step 0 is the shared base model, and step 196 is the end of training.

Training stepM-DPO (full)EntroPO-DPO (full)M-DPO (correct)EntroPO-DPO (correct)
00.15890.15890.15150.1515
500.15250.15920.15350.1598
1000.14960.16240.14760.1653
1500.14650.16840.14540.1675
1960.14620.16900.14500.1677

The two methods move in opposite directions from the same starting point. M-DPO ends 8.0% (full set) and 4.3% (correct set) below the base model, while EntroPO-DPO ends 6.4% and 10.7% above it. The gap widens at every checkpoint (Figure 6 in the revised paper).

(3) Strategy-level diversity of correct solutions. Higher token-level entropy does not by itself show that correct solutions differ in substance, so we also measure diversity at the level of the solution approach, following [4]. For each SWE-bench Verified task, we take the patches from the correct rollouts, normalize them (comments and whitespace), and remove duplicates. An LLM judge (gpt 6.1 terra with codex harness) then decides whether each pair of patches implements the same approach, using the prompt of [4]. DA@16 (distinct algorithms [4]) is the number of distinct approaches among the correct patches from 16 rollouts, averaged over tasks.

ModelDA@16
Base model1.33
M-DPO1.35
EntroPO-DPO2.29

EntroPO-DPO produces 2.29 distinct correct approaches per task, compared with 1.35 for M-DPO (~1.7x). The additional entropy therefore corresponds to different ways of fixing the same issue.

Together, these results show that EntroPO increases policy entropy during training, that the increase carries over to inference and is largest on correct rollouts, and that it yields more distinct correct solutions per task, which is what test-time scaling benefits from.

We have added these results and the discussion to Appendix D.8 (Table 12, Figure 6) in the revised paper.

The comparison with online RL (i.e., GRPO) does not seem entirely fair, as entropy regularization has also become common practice in GRPO training. It would be more informative to also compare EntroPO with GRPO using entropy control.

Response: Thanks for suggesting this comparison, and we run this comparison. Starting from our GRPO baseline (Table 4: Qwen3-Coder-30B-A3B-Instruct, R2E scaffold, R2E-Gym-subset training tasks), we added an actor-entropy bonus, sweeping its coefficient over {0.005, 0.01, 0.03, 0.05} and kept all other training and evaluation configs unchanged. The results are reported below and in Table 6 of the revised paper.

MethodActor-entropy coefficientPass@1 (%)Raw pass@16 (%)
GRPO (original recipe)–46.471.8
GRPO + entropy bonus0.00546.971.9
GRPO + entropy bonus0.0146.572.0
GRPO + entropy bonus0.0346.271.7
GRPO + entropy bonus0.0546.071.4
EntroPO-KTO–51.679.2

The results show that entropy control does not close the gap. Small coefficients have little effect, and larger ones slightly reduce the performance. EntroPO-KTO remains at least 4.7 points higher in pass@1 and 7.2 points higher in pass@16, using about 1/32 of the compute of a GRPO run.

This is consistent with prior findings that entropy regularization in online RL is sensitive to its coefficient [3,5]: weak regularization has limited impact, while stronger regularization does not necessarily improve performance. One possible explanation is that, in online RL, the bonus acts through the policy's own rollouts, and more diverse rollouts are not necessarily more informative ones. On hard SWE tasks, the extra diversity can produce more failed trajectories rather than new successful ones. EntroPO instead trains on a fixed, preference-labeled dataset whose examples and labels remain available regardless of changes in the policy's sampling behavior. This avoids the dependence on online rollout quality.

Finally, EntroPO and GRPO are complementary at the training-pipeline level rather than competing. EntroPO is an offline stage on fixed trajectories, so it can serve as a warm start before online GRPO. We leave this to future work.

We have added these results and the discussion to Appendix D.6.1 in our revised paper.

The experiments use only the R2E scaffold as the agent harness, which limits the comprehensiveness of the evaluation.

Response: Thanks for the comment. We would like to clarify that R2E is not the only agent scaffold in our evaluation. We use the R2E scaffold only for the SWE-bench tasks. For LiveCodeBench, we use the Cobalt multi-turn code-generation scaffold, and for TerminalBench, we use the Endless Terminals terminal-agent scaffold (Appendix D.2). EntroPO achieves the strongest overall results under all three scaffolds (Table 1).

To further address this concern on SWE-bench, we added an evaluation with a second scaffold, mini-SWE-agent, on SWE-bench Verified with Qwen3-Coder-30B. Both M-DPO and EntroPO-DPO are trained on trajectories collected with mini-SWE-agent, using the same data-construction pipeline and training hyperparameters as in the R2E setting:

MethodPass@1 (%)TTS, N=16 (%)
Base model36.440.8
M-DPO46.051.6
EntroPO-DPO48.855.4

The trend is consistent with the R2E results. EntroPO-DPO outperforms M-DPO in both settings, and its advantage again grows with test-time scaling.

We have clarified the scaffolds in Section 4.1 and added the mini-SWE-agent results to Table 10 in the revised paper.

The paper title seems to be too general and does not demonstrate the concrete idea in this paper.

Response: Thanks for the suggestion. We have changed the title in the revised paper to better reflect the key idea: "Improving Test-Time Scaling for Software Engineering Agents via Multi-Turn Entropy-Regularized Preference Optimization". It names the method (multi-turn entropy-regularized preference optimization) together with its goal (test-time scaling for software engineering agents). Thanks again for your valuable comments.

References

[1] Uncertainty Estimation in Autoregressive Structured Prediction. ICLR 2021.

[2] Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints. ICLR 2024.

[3] The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models. arXiv 2025.

[4] How Diversely Can Language Models Solve Problems? Exploring the Algorithmic Diversity of Model-Generated Code. Findings of EMNLP 2025.

[5] Differential Smoothing Mitigates Sharpening and Improves LLM Reasoning. ICML 2026.

Dear Reviewer Cvkk,

Thank you again for your time and constructive comments. We would like to kindly follow up on our response and check whether it has addressed your concerns.

To summarize, we have addressed each of your points in the revised paper:

  • Entropy and rollout diversity. We added measurements of entropy at inference, entropy during training, and strategy-level diversity among correct solutions (Appendix D.8, Table 12, Figure 6). EntroPO-DPO raises the correct-set entropy by 29.3% over M-DPO and produces about 1.7x as many distinct correct approaches per task (DA@16 of 2.29 vs. 1.35).
  • GRPO with entropy control. We added GRPO with an entropy bonus under four coefficients (Appendix D.6.1, Table 6). All settings stay within 0.5 points of the original GRPO, and EntroPO-KTO remains 4.7 points higher in pass@1 and 7.2 points higher in pass@16.
  • Agent scaffold. We clarified that our evaluation uses three scaffolds across benchmarks, and we added mini-SWE-agent as a second scaffold on SWE-bench Verified (Table 10). EntroPO-DPO again outperforms M-DPO (48.8 vs. 46.0 in pass@1, 55.4 vs. 51.6 with test-time scaling).
  • Title. Thanks a lot for this feedback and we revised the title to name the method, as you suggested.

We would appreciate it if you could let us know whether any concerns remain. If our response has addressed them, we would be grateful if you would consider raising your score. We are happy to provide further clarification or results during the discussion period.

Thank you again for your valuable feedback.

Best regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.