- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the cross-paradigm generalization capabilities of large language models on combinatorial optimization tasks. The authors introduce Reasoning-to-Code Optimization (R2CO), a training setting that requires the model to first design a solution strategy and then implement it as an executable Python program. Building on this, the authors propose Reflective RL, which uses historical successful or failed reasoning trajectories and verifier feedback to generate new rollouts. Through context relabeling and GRPO, the model is trained to generate effective solution strategies without relying on historical feedback.
The experiments primarily use 43 combinatorial optimization task families from NLCO for training and evaluate the model on optimization modeling, heuristic discovery, direct text reasoning, and additional reasoning benchmarks. The results show that R2CO training provides cross-paradigm generalization benefits, while Reflective RL further improves performance on these tasks.
Critical strengths
-
This paper presents a training approach for combinatorial optimization that leverages historical reasoning experiences and verifier feedback to improve the model's reasoning. The experiments demonstrate that this approach not only improves performance on the source tasks but also transfers these gains to different optimization paradigms, offering meaningful research value.
-
The paper evaluates cross-paradigm transfer across multiple target benchmarks and evaluation frameworks, with comparisons against R2CO-SFT, Generic Code-RL, and R2CO-RL. The authors further investigate the sources of performance improvements through reasoning–code cross-evaluation, reflection mode ablations, and experiments with different base models. These experiments provide substantial support for the paper's main observations.
Critical weaknesses
- Limited methodological novelty and insufficient differentiation from existing reflection-based training methods.
Although the method achieves performance improvements, its unique algorithmic contribution remains unclear compared with existing approaches such as Strategy-Guided Exploration (SGE), feedback-guided training, and self-distillation. A direct comparison with closely related methods such as SGE under matched training budgets would strengthen the paper.
- Insufficient explanation of the underlying cross-paradigm generalization phenomenon.
Although the authors provide training dynamics analysis and reasoning–code cross-evaluation, a systematic investigation of how reasoning strategies evolve during training is still lacking. For example, does the model progressively learn new perspectives on problem analysis, algorithm selection, or search strategies?
Furthermore, the authors emphasize that reflecting only on reasoning rather than referring to previous code facilitates the learning of transferable strategies, but a direct ablation study is missing to validate the necessity of this design.
- Reliance on a single source training dataset limits the generality of the conclusions.
The proposed training approach has potentially broad applications. Although the authors evaluate the model on multiple external benchmarks, the source post-training data comes exclusively from NLCO. It remains unclear whether similar cross-paradigm transfer benefits would emerge when training on other combinatorial optimization datasets. Additional training experiments using alternative source datasets would help validate the generality of the findings.
Questions for authors
Q1. What is the core algorithmic novelty of Reflective RL compared with existing feedback-guided exploration and self-distillation methods? Could the authors provide direct comparisons with methods such as SGE?
Q2. Can the cross-paradigm transfer findings of R2CO generalize to other source training datasets?
Q3. Could the authors further analyze how the model's reasoning strategies evolve during training and investigate how these changes relate to cross-paradigm transfer gains?
9 comments
Reply to Reviewer Ub4J (2/3)
[W2.1, Q3] Strategy evolution during training and its relation to transfer
Thank you for suggesting this analysis. We analyzed responses to the same held-out instances across five training checkpoints to examine how the model's reasoning strategies evolve and how these relate to cross-paradigm gains. We find that the number of successful strategies initially increases and then declines, while successful responses become more frequent. Transfer can continue to improve even as sampled strategy coverage narrows.
1. How do reasoning strategies evolve during training?
To examine how reasoning strategies evolve during training, we investigate whether the model becomes better at problem analysis, such as correctly identifying the objective and all key constraints, and how its solution strategies change across checkpoints.
We evaluate the base model (step 0) and the Reflective RL checkpoints at steps 30, 60, 90, and 120, sampling 32 responses per instance on the same 129 held-out NLCO instances. For each instance, we consider the responses sampled from all five checkpoints. Following the strategy counting method proposed by Lee et al. [4], we use an LLM (i.e., DeepSeek-V4.1-Flash) to group these responses by solution strategy based on their reasoning traces and generated programs. Responses using the same strategy are assigned to the same group, regardless of which checkpoint produced them. We then count the distinct strategies represented in the 32 responses sampled at each checkpoint. During the initial grouping step, we also ask the judge to check whether each reasoning trace correctly identifies the objective and all key constraints. The full evaluation procedure is provided in Appendix E.14.
A response is successful if its generated program returns a feasible solution with a relative optimality gap of at most . For each checkpoint, let denote the number of distinct strategies in the 32 responses sampled for instance , and the number of these strategies with at least one successful response. We report Strategy Count (SC) and Successful Strategy Count (SSC) as
\qquad \mathrm{SSC}=\frac{1}{129}\sum_{i=1}^{129}K_i^{\mathrm{succ}}.$$ The successful-response count is also averaged over the 129 instances. The correct problem-analysis rate is the percentage of responses whose reasoning correctly identifies the objective and all key constraints. | Checkpoint | Correct problem analysis (%) | SC | SSC | Successful responses (of 32) | | :--- | ---: | ---: | ---: | ---: | | Step 0 | 92.93 | 5.496 | 1.054 | 5.364 | | Step 30 | 92.20 | 5.636 | 1.442 | 13.442 | | Step 60 | 92.51 | 5.574 | 1.356 | 13.791 | | Step 90 | 92.85 | 5.178 | 1.288 | 15.411 | | Step 120 | 92.37 | 4.651 | 1.240 | 15.938 | The table shows that the base model already correctly identifies the objective and all key constraints in most responses, and this rate changes little during training. SSC initially increases, but after step 30 both SC and SSC decline, while successful responses continue to increase. **These results suggest that training improves the model’s ability to produce successful solutions beyond correctly identifying the problem requirements. During later training, successful responses become more frequent, while the number of distinct strategies observed under the same sampling budget decreases.** Thus, higher solution accuracy does not necessarily imply broader sampled strategy coverage. This is consistent with prior findings that RL can improve solution performance while concentrating generation on fewer solution strategies [4–6]. We have added this analysis to Appendix E.14. **2. How do these changes relate to cross-paradigm transfer gains?** We compare the strategy statistics above with transfer performance at the corresponding checkpoints (Appendix E.2, Figure 3). From initialization to step 120, performance improves across all three transfer paradigms: optimization modeling, heuristic discovery, and direct text reasoning. These gains accompany more frequent successful responses on the held-out NLCO instances evaluated under R2CO, without a sustained increase in the number of successful strategies. From step 30 to step 120, both the total number of sampled strategies and the number of successful strategies decrease, while successful responses continue to increase. Over the same period, transfer performance improves overall across all three paradigms. These results show that transfer can continue to improve as successful R2CO responses become more frequent, even as sampled strategy coverage narrows.Reply to Reviewer Ub4J (3/3)
[W2.2] Effect of access to previous code during reflection
We omit previous code from the reflection prompt to encourage the model to reconsider the solution approach instead of editing its earlier program. To test whether this choice contributes to the transfer gains, we compare Reflective RL with a matched variant that also receives the previous program during reflection.
Specifically, both reflection configurations use the same NLCO training split, verifier feedback and reward, and a 120-update schedule with 32 instances per update and eight rollouts per instance. The variant with previous code receives the previous program during both positive and negative reflection. During reflection, the prompt asks the model to inspect the implementation when revising its strategy. The results are summarized below.
| Configuration | R2CO AFR / Acc. (%) | Modeling Acc. (%) | EoH RI (%) | ReEvo RI (%) | Direct Text AFR / Acc. (%) |
|---|---|---|---|---|---|
| Qwen3-4B-Instruct (base) | 25.4 / 19.2 | 60.09 | 0.00 | 0.00 | 65.9 / 34.8 |
| R2CO-RL | 64.4 / 51.4 | 73.66 | 2.07 | 3.24 | 69.2 / 36.1 |
| Reflective RL with previous code | 66.5 / 52.5 | 75.07 | 2.39 | 2.65 | 69.2 / 37.0 |
| Reflective RL | 69.3 / 54.5 | 76.83 | 4.99 | 4.22 | 71.2 / 37.8 |
Compared to incorporating previous code, our setting achieves higher optimization-modeling accuracy and direct-text accuracy. It also yields stronger heuristic discovery results in both EoH (4.99% vs. 2.39% RI) and ReEvo (4.22% vs. 2.65% RI). These results support our design choice across all three evaluated transfer paradigms.
The variant with previous code improves over the variant without reflection in modeling, EoH, and direct text accuracy, although its ReEvo RI is lower (2.65% vs. 3.24%). Thus, reflection with previous code remains beneficial on several target evaluations, while Reflective RL without previous code yields stronger results throughout the reported transfer evaluations. The ablation details and results have now been added to Appendix E.15 (Table 26) of the revised manuscript.
[W3, Q2] Transfer from a second source dataset
Thank you for suggesting this experiment. We have evaluated R2CO training on another dataset (i.e., GraphArena), and we found that the transfer gains also appear when we train on another dataset, but the performance gain is smaller than that obtained by training on NLCO.
Specifically, we initialized the same Qwen3-4B-Instruct base model and trained it with Reflective RL for 120 updates on 1,920 GraphArena instances from six tasks at two difficulty levels. We then evaluated all three transfer paradigms, with direct text reasoning tested on held-out NLCO instances, and the results are summarized below.
| Training source | Modeling Acc. (%) | EoH RI (%) | ReEvo RI (%) | Direct Text AFR / Acc. (%) |
|---|---|---|---|---|
| Qwen3-4B-Instruct (base) | 60.09 | 0.00 | 0.00 | 65.9 / 34.8 |
| GraphArena | 71.20 | 1.76 | 1.09 | 69.92 / 36.74 |
| NLCO | 76.83 | 4.99 | 4.22 | 71.2 / 37.8 |
We still see that R2CO training works for GraphArena, showing the generalizability of our method. However, the gains with GraphArena are smaller than those with NLCO. NLCO covers 43 tasks, including routing, scheduling, packing, assignment, and graph optimization, whereas the GraphArena training set used here contains six graph tasks. NLCO's natural-language problem descriptions therefore expose the model to a broader range of problem structures, objectives, and constraints. This broader coverage may help explain the stronger transfer results and is why we chose NLCO as our main training source. The experimental setup and results are provided in Appendix E.16, Table 27, of the revised manuscript.
References
[1] Szot, A., Kirchhof, M., Attia, O., and Toshev, A. 2026. Expanding LLM Agent Boundaries with Strategy-Guided Exploration. arXiv preprint arXiv:2603.02045.
[2] Hübotter, J., et al. 2026. Reinforcement Learning via Self-Distillation. Proceedings of the 43rd International Conference on Machine Learning, PMLR 306, pp. 48677–48721.
[3] Shenfeld, I., Damani, M., Hübotter, J., and Agrawal, P. 2026. Self-Distillation Enables Continual Learning. arXiv preprint arXiv:2601.19897.
[4] Lee, S., Kim, M., Kim, J., Kim, D., Rhee, S., and Jung, K. 2026. Are We Measuring Strategy or Phrasing? The Gap Between Surface- and Approach-Level Diversity in LLM Math Reasoning. Proceedings of EMNLP 2026, pp. 52966–52996.
[5] Cheng, Z., Huang, Y., Zhu, H., and Sojoudi, S. 2026. How RL Reshapes LLM Reasoning: Transferability, Coverage, and Scaling Laws. arXiv preprint arXiv:2610.04158.
[6] Massoudi, S., Apaza, G., Habibi, M., and Fuge, M. 2026. Beyond Inference-Time Search: Reinforcement Learning Synthesizes Reusable Solvers. arXiv preprint arXiv:2605.18374.
Reply to Reviewer Ub4J (1/3)
Dear Reviewer Ub4J,
Thank you for recognizing the value of cross-paradigm transfer and the evidence from our controlled comparisons. We address your concerns by clarifying the contribution relative to prior methods and examining strategy evolution, access to previous code, and transfer from a second training source.
[W1, Q1] Methodological contribution and comparison with SGE
1. Contribution of R2CO and Reflective RL
Thank you for asking us to clarify the novelty. Rather than proposing a completely new algorithm for feedback-guided exploration, we study a novel research question: whether learning to develop optimization algorithms can improve an LLM across different solving paradigms, and reflection-based learning is our way to enhance performance. Specifically, we introduce R2CO to investigate this new question. We find that training in this source setting improves optimization modeling, heuristic discovery, and direct text reasoning, without further learning on these target tasks.
To further strengthen this transfer, we introduce Reflective RL, motivated by the limitation that outcome rewards evaluate solutions but provide little guidance on how to improve the underlying reasoning. Our method builds on the general idea of strategy reflection in SGE [1], as acknowledged in Section 3.3, but introduces several designs tailored to R2CO. First, we separate reflection-guided reasoning from program generation: previous experiences guide the refinement of solution strategies, while fresh code is generated without the reflection context. Second, context relabeling allows these refined strategies to be learned under the original task prompt. Third, optimization-specific feedback, including constraint violations and optimality gaps, enables reflection on feasible but suboptimal solutions, beyond execution failures. These designs aim to improve the learning and reliable application of optimization strategies rather than simply increasing strategy diversity.
Empirically, Reflective RL improves all three transfer paradigms over standard R2CO-RL. For example, optimization modeling accuracy increases from 73.66% to 76.83%, while relative improvements in heuristic discovery rise from 2.07% to 4.99% on EoH and 3.24% to 4.22% on ReEvo (Tables 2–3 and Appendix E.3).
Regarding self-distillation, SDPO [2] and SDFT [3] train a policy to match self-teacher predictions conditioned on feedback or expert demonstrations. Reflective RL instead uses previous reasoning and feedback to generate a new attempt, then updates the policy using its verified reward. The update uses the original problem prompt, so the trained model does not need reflection context at inference time.
2. Direct comparison with SGE under matched training settings
Following your suggestion, we added a direct comparison with SGE [1]. We adapted SGE to single-turn optimization by treating the reasoning block as the strategy and the generated program as the action, with sampling temperatures of 1.2 and 0.7, respectively. For this adaptation, we used the same R2CO reward function and negative and positive reflection probabilities, (p−,p+)=(0.15,0.10), as Reflective RL. We trained SGE for 120 optimization steps on the same NLCO training data as Reflective RL. The results are summarized below.
Under these matched settings, Reflective RL achieves higher R2CO feasibility and accuracy and stronger transfer across all three paradigms, showing the effectiveness of our implementation.