- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
To mitigate diversity collapse in standard preference optimization methods, such as DPO, and further improve test time scaling on coding tasks. This paper proposes EntroPO, a preference optimization framework for multi-turn coding agents. Specifically, the authors formulate preference optimization as an entropy-regularized MDP and derive two objectives, EntroPO-DPO and EntroPO-KTO, for pairwise and pointwise feedback, respectively. Theoretically, they show that EntroPO encourages greater diversity among correct trajectories and can improve accuracy over standard DPO under certain conditions. The authors also conducted extensive experiments that evaluate EntroPO across multiple model families and four benchmarks, including SWE-bench Verified, SWE-bench Lite, LiveCodeBench, and TerminalBench. The results demonstrate improvements in both single-rollout performance and test-time scaling, with competitive results on the SWE-bench leaderboards.
Critical strengths
- The proposed method mitigates the important problem of diversity collapse in preference-based training of multi-turn coding agents, which is relevant to improving test-time scaling.
- The extensive experiments support for the effectiveness and generalizability of the proposed method.
Critical weaknesses
- Although EntroPO achieves higher pass@16 than the baselines, this does not provide direct evidence for improved solution diversity. The gains in test time scaling could also be partly attributed to improved single-rollout accuracy, even if the diversity of generated solutions is still limited. That said, I consider this is a relatively minor concern, and would not significantly affect my overall recommendation for acceptance.
Questions for authors
- Could the authors provide more analysis on the relevance between the improved performance and the diversity of agent trajectory?
4 comments
I appreciate that the authors provided additional measurements of diversity among correct solutions and further analysis of the relationship between diversity and test-time scaling performance. These results have addressed my concern.
Thanks for your feedback
Thank you for taking the time to read our response and the revised paper. We are glad that the additional diversity measurements and the analysis of their relationship with test-time scaling have addressed your concern. We appreciate your constructive feedback, which helped us improve the paper.
We hope you have a great weekend!
Thank you, and a final check
Dear Reviewer 6muT,
Thank you again for your reply. We are glad that the additional diversity measurements addressed your concern. As the discussion period is coming to an end, we would like to check whether you have any remaining questions. We are more than happy to address them and revise the paper accordingly.
If there are none, we would be grateful if you would consider reflecting this in your final score.
Best regards,
The authors
We thank Reviewer 6muT for the constructive and insightful comments. Please see our response to each of your questions below.
Response: Thanks for the question. We agree that pass@16 alone does not show improved diversity, and that part of the gain can come from higher single-rollout accuracy. We have added direct measurements of diversity among correct solutions to the revised paper (Appendix D.8, summarized in Section 4.3), and we relate them to the gain from test-time scaling below.
Diversity measures (SWE-bench Verified, Qwen3-Coder, 16 rollouts per task):
Correct-set entropy. Following [1,2], we measure the length-normalized predictive entropy on the model's self-generated tokens, computed on the rollouts that pass the tests. It is the empirical counterpart of the correct-set entropy H+ in Theorem 3.6.
DA@16. Following [3], we normalize and deduplicate the patches from the correct rollouts, and an LLM judge (gpt 6.1 terra with codex harness) decides whether each pair of patches implements the same approach. DA@16 (distinct algorithms) is the number of distinct approaches among the correct patches from 16 rollouts, averaged over tasks.
Relation between diversity and performance:
Two observations separate the effect of accuracy from the effect of diversity:
The larger gain from test-time scaling therefore appears together with higher diversity among correct solutions, not with higher single-rollout accuracy alone. We also track entropy during training: M-DPO's entropy falls below the base model, while EntroPO-DPO's rises from the first checkpoint (Figure 6).
We have added these results and the discussion to Appendix D.8 (Table 12, Figure 6) in the revised paper.
References
[1] Uncertainty Estimation in Autoregressive Structured Prediction. ICLR 2021.
[2] Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints. ICLR 2024.
[3] How Diversely Can Language Models Solve Problems? Exploring the Algorithmic Diversity of Model-Generated Code. Findings of EMNLP 2025.