- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies exploration contraction in long-chain reasoning and proposes LOrE, a framework that trains an orchestrator to allocate computation across different solution directions. The method combines synthetic midtraining with multi-turn RL. Experiments on mathematical and scientific reasoning benchmarks show improved exploration behavior and test-time scaling.
Critical strengths
-
Interesting problem analysis. The paper provides evidence of exploration contraction from both semantic-level annotations and token-level statistics (Figures 1–2). This helps explain why simply extending reasoning does not always lead to better exploration.
-
Strong empirical results. LOrE improves performance across several challenging benchmarks. The comparisons in Table 5 help establish the benefit of orchestration training over standard CoT RL, while Appendix K provides useful cross-judge validation.
Critical weaknesses
-
Limited methodological novelty. Learning to orchestrate parallel computation has been studied in APR, Kimi PARL, and ParaManager. LOrE focuses more specifically on diverse solution directions, but its architecture and two-stage training recipe overlap substantially with prior work. The distinction is not yet convincingly established experimentally.
-
Insufficient control of computation and component contributions. LOrE uses substantially more test-time tokens than some baselines (Appendix G). Although the paper includes useful comparisons, it remains unclear how much of the improvement comes from better exploration rather than additional computation. More targeted ablations of the synthetic midtraining and RL components would also help.
Questions for authors
-
How does LOrE compare with existing RL-trained orchestrators, particularly PARL-style approaches, under matched computation budgets?
-
Can the authors isolate the contributions of direction selection, synthesized orchestration reasoning, and the exploration bonus?
6 comments
Response to Reviewer YatD (2/3)
Comparisons with APR, Kimi PARL, and ParaManager (cont.)
Why do open-ended tasks require different orchestration behaviors?
Existing parallel-agent approaches primarily consider problems with relatively clear decompositions into independently executable subtasks. For example, PARL considers only browsing and wide-search tasks (BrowseComp, WideSearch), often decomposable by entity, source, or sub-question. APR considers Countdown, a constrained arithmetic game whose search tree can be partitioned into well-defined branches. ParaManager meanwhile is trained on short-answer competition math (DAPO-Math), using external tools (Python, web search) and predefined specialist roles that provide structured delegation strategies. In these settings, the central challenges involve invoking subagents, delegating appropriate subtasks, and combining their outputs into useful knowledge.
In contrast, LOrE targets research-level problems (TCSBench, ArxivMath) and challenging Olympiad proofs (IMO ProofBench) that generally lack obvious upfront decompositions. Solution paths are typically unknown beforehand, plausible approaches are numerous, and mathematical challenges may only emerge after tens of thousands of tokens of reasoning. The challenge is therefore not merely distributing computation, but discovering, evaluating, and continually revising promising directions of exploration.
These differences motivate LOrE's methodological contributions. First, our synthetic data pipeline constructs demonstrations of effective exploration by selecting diverse, promising directions and synthesizing orchestration reasoning focused on exploration rather than direct problem-solving or superficial delegation behavior. Second, midtraining on synthetic data is used to establish these behaviors, which many strong open source LLMs do not exhibit and are difficult to discover through outcome-based RL alone (see our ablations in Section 5). Finally, multi-turn RL with an exploration bonus improves the orchestrator's ability to adapt its exploration strategy based on previous findings while continuing to discover useful directions. Together, these components train LOrE not merely to coordinate parallel execution, but to sustain productive exploration over long reasoning horizons. Such contributions are not present in the aforementioned prior work.
Compute-matched comparisons
We also conducted extensive compute-matched experiments, with full results presented in our response to Reviewer YGDf (comment #1/3) and Appendix G of our revised PDF submission. We find that LOrE-RL-9B outperforms both its midtrained-only counterpart and competing reasoning harnesses across nearly all tested inference compute budgets (up to ~1.5M tokens), including parallel-sequential approaches (RSA) and sequential compaction (RC). Furthermore, reducing the number of parallel exploration directions while compensating with greater orchestration depth degrades performance even at matched compute. Together, these findings provide evidence that LOrE's improvements arise not merely from using more test-time computation, but from using that computation more effectively to sustain diverse exploration.
Response to Reviewer YatD (3/3)
Isolating the contributions of direction selection, synthesized orchestration reasoning, and the exploration bonus
We have conducted additional experiments isolating the contributions of synthesized orchestration reasoning and the exploration bonus.
Exploration bonus
We conducted ablations of our RL exploration bonus, with detailed results presented in our response to Reviewer YGDf (comment #3/3). We find that removing the bonus degrades performance and substantially reduces the number of directions proposed per turn, while retaining it throughout training encourages excessive breadth at the expense of exploration depth, also reducing performance. These findings demonstrate that the exploration bonus plays an important role in regulating the breadth–depth tradeoff, with our default schedule achieving the best empirical balance.
Synthesized orchestration reasoning
We interpret this to mean ablating the (synthesized) orchestration reasoning we use in the midtraining stage of training. We therefore conduct an experiment where we removed the orchestration reasoning traces from the midtraining data while leaving the execution and final answer/proof traces untouched. We complemented this experiment with analogous experiments where we removed either the execution trace or the final answer/proof traces. We evaluated the resulting midtrained models without further RL.
| Training data removed | TCSBench | ArxivMath | IMO ProofBench |
|---|---|---|---|
| Final answer/proof | 21.5% | 25.0% | 45.6% |
| Orchestration | 19.3% | 18.8% | 41.6% |
| Execution | 18.0% | 25.0% | 39.1% |
| Full midtraining data (no removal) | 27.3% | 29.9% | 52.8% |
| Data removed | TCSBench | ArxivMath | IMO ProofBench |
|---|---|---|---|
| Final answer/proof | −5.8 pp | −4.9 pp | −7.2 pp |
| Orchestration | −8.0 pp | −11.1 pp | −11.2 pp |
| Execution | −9.3 pp | −4.9 pp | −13.7 pp |
From the table above, we see that removing any of the three components reduces the performance of the resulting model. Removing orchestration reasoning is the most damaging, especially on ArxivMath.
Direction selection
We interpret this as referring to the data curation step used during synthetic midtraining data generation (Section 4.1), which serves to filter out redundant directions and prioritize those likely to yield useful information. In earlier experiments with a preliminary version of our data pipeline that did not include this filtering step, we observed that the resulting orchestrator frequently proposed highly redundant exploration directions. This observation motivated the inclusion of explicit direction selection in our final pipeline.
However, because these earlier experiments also differed in other aspects of data generation, we do not interpret this as a controlled ablation or claim that the improvements are attributable to direction selection alone. A rigorous study of this would require regenerating the synthetic exploration trajectories while holding the remaining pipeline fixed. This process would be very costly and time consuming. We therefore leave it for future work.
Response to Reviewer YatD (1/3)
Thank you for your review. We have addressed your concerns by (1) comparing PARL-style RL training with LOrE, (2) providing detailed discussion of our contributions differ substantially from previous work, including PARL, APR and ParaManager, (3) conducting compute-controlled experiments comparing LOrE against baselines, and (4) conducting additional experiments isolating the contributions of synthesized orchestration reasoning and the RL exploration bonus.
Comparisons with APR, Kimi PARL, and ParaManager
We agree that the orchestrator-executor architecture is an important component of LOrE, but emphasize that the architecture alone is insufficient to enable effective exploration on difficult problems. Our central contribution is learning effective orchestration behavior and then amplifying these behaviors in order to sustain diverse, productive exploration on challenging open-ended problems, with the orchestrator-executor architecture itself being only one contribution of our work. To clarify this distinction, we first present a direct experimental comparison with PARL-style training, and then discuss the differences in learned behaviors and methodology.
Direct experimental comparison with PARL-style training
We trained Qwen3.5-9B using a PARL-style RL setup on the same training problems and training compute budget as LOrE, replacing our training procedure with PARL-style multi-turn RL.
Setup. The orchestrator receives a 128k-token budget and may spawn up to 20 subagents, each with a 90k-token reasoning budget. This permits up to 1.92M tokens per trajectory, exceeding the typical ~1M tokens consumed per trajectory in LOrE. We train for 200 steps, use the same RL dataset and set both PARL auxiliary reward coefficients to 0.1.
As can be seen above, PARL-style training substantially underperforms LOrE-RL-9B. While it improves over standard CoT on TCSBench and IMO ProofBench, it performs worse on ArxivMath. This suggests that adopting an existing parallel-agent training procedure does not on its own reproduce the improvements achieved by LOrE on open-ended reasoning problems.
To understand these results, we examined the PARL trajectories and identified three recurring failure modes:
These are precisely the behavioral limitations identified in Section 3 that motivate LOrE's midtraining. Rather than simply teaching models to invoke and coordinate subagents, LOrE explicitly instills and amplifies exploration-oriented behaviors: proposing diverse, nonredundant directions; meta-reasoning about exploration rather than directly solving problems; and sustaining adaptive exploration deep into reasoning trajectories. Our ablations (see other rebuttal responses) further demonstrate the importance of these behaviors, including the contribution of orchestration reasoning during midtraining and the performance degradation caused by reducing exploration diversity.