acceptodds
Under review as a conference paper at ICLR 2027

Scaling Test-Time Compute by Learning to Orchestrate Exploration

Abstract

Large language models (LLMs) are increasingly used to solve hard reasoning problems with large search spaces, where success can require sustained exploration over substantial test-time compute. We study how today's LLMs use this compute and find that chain-of-thought (CoT) can be contractive: as reasoning progresses, models devote increasingly less computation to novel directions and instead concentrate on approaches discovered earlier. Existing reasoning harnesses broaden exploration, but do not eliminate this contraction. We therefore study a harness in which an orchestrator explicitly decides which directions to explore and delegates them to parallel execution models. Current LLMs still struggle in this role, exhibiting redundant delegation, premature termination, and substantial problem-solving within the orchestrator itself. We hypothesize that this reflects a mismatch between standard reasoning training and the requirements of orchestration: models are trained to solve problems within their own reasoning trajectories, but receive little supervision for deciding where additional computation should be allocated. To address this, we introduce LoRE (Learning to Orchestrate Exploration), which first midtrains a model on synthetic exploration trajectories to establish orchestration priors, then applies multi-turn RL within the orchestrator-executor harness to optimize downstream task success. Applied to a 9B-parameter model, LoRE sustains broader exploration, scales effectively to millions of generated test-time tokens, and performs well on research-level benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, reinforcement learning, reasoning, harness, orchestrator

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_YatDOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies exploration contraction in long-chain reasoning and proposes LOrE, a framework that trains an orchestrator to allocate computation across different solution directions. The method combines synthetic midtraining with multi-turn RL. Experiments on mathematical and scientific reasoning benchmarks show improved exploration behavior and test-time scaling.

Critical strengths
  1. Interesting problem analysis. The paper provides evidence of exploration contraction from both semantic-level annotations and token-level statistics (Figures 1–2). This helps explain why simply extending reasoning does not always lead to better exploration.

  2. Strong empirical results. LOrE improves performance across several challenging benchmarks. The comparisons in Table 5 help establish the benefit of orchestration training over standard CoT RL, while Appendix K provides useful cross-judge validation.

Critical weaknesses
  1. Limited methodological novelty. Learning to orchestrate parallel computation has been studied in APR, Kimi PARL, and ParaManager. LOrE focuses more specifically on diverse solution directions, but its architecture and two-stage training recipe overlap substantially with prior work. The distinction is not yet convincingly established experimentally.

  2. Insufficient control of computation and component contributions. LOrE uses substantially more test-time tokens than some baselines (Appendix G). Although the paper includes useful comparisons, it remains unclear how much of the improvement comes from better exploration rather than additional computation. More targeted ablations of the synthetic midtraining and RL components would also help.

Questions for authors
  1. How does LOrE compare with existing RL-trained orchestrators, particularly PARL-style approaches, under matched computation budgets?

  2. Can the authors isolate the contributions of direction selection, synthesized orchestration reasoning, and the exploration bonus?

6 comments
AuthorsOpenReview ↗

Response to Reviewer YatD (1/3)

Thank you for your review. We have addressed your concerns by (1) comparing PARL-style RL training with LOrE, (2) providing detailed discussion of our contributions differ substantially from previous work, including PARL, APR and ParaManager, (3) conducting compute-controlled experiments comparing LOrE against baselines, and (4) conducting additional experiments isolating the contributions of synthesized orchestration reasoning and the RL exploration bonus.


Comparisons with APR, Kimi PARL, and ParaManager

We agree that the orchestrator-executor architecture is an important component of LOrE, but emphasize that the architecture alone is insufficient to enable effective exploration on difficult problems. Our central contribution is learning effective orchestration behavior and then amplifying these behaviors in order to sustain diverse, productive exploration on challenging open-ended problems, with the orchestrator-executor architecture itself being only one contribution of our work. To clarify this distinction, we first present a direct experimental comparison with PARL-style training, and then discuss the differences in learned behaviors and methodology.


Direct experimental comparison with PARL-style training

We trained Qwen3.5-9B using a PARL-style RL setup on the same training problems and training compute budget as LOrE, replacing our training procedure with PARL-style multi-turn RL.

Setup. The orchestrator receives a 128k-token budget and may spawn up to 20 subagents, each with a 90k-token reasoning budget. This permits up to 1.92M tokens per trajectory, exceeding the typical ~1M tokens consumed per trajectory in LOrE. We train for 200 steps, use the same RL dataset and set both PARL auxiliary reward coefficients to 0.1.

ModelTCSBenchArxivMathIMO ProofBench
Qwen3.5-9B PARL19.0%12.5%29.4%
Qwen3.5-9B (CoT)11.1%20.8%25.8%
LOrE-RL-9B34.2%40.6%62.2%

As can be seen above, PARL-style training substantially underperforms LOrE-RL-9B. While it improves over standard CoT on TCSBench and IMO ProofBench, it performs worse on ArxivMath. This suggests that adopting an existing parallel-agent training procedure does not on its own reproduce the improvements achieved by LOrE on open-ended reasoning problems.

To understand these results, we examined the PARL trajectories and identified three recurring failure modes:

  • Direct problem-solving instead of orchestration. The orchestrator spends substantial portions of its CoT solving the problem itself, often delegating variants of subtasks it has already attempted in its own CoT rather than new directions.
  • Redundant delegation. Subagents frequently receive overlapping directions that differ mainly in wording or framing, limiting exploration diversity.
  • Premature termination. Despite a budget of 20 subagents, the orchestrator invokes only 8.3 on average, typically performing one or two rounds of delegation before producing a final answer. This occurs despite PARL’s use of a parallelism bonus that incentivizes the spawning of subagents.

These are precisely the behavioral limitations identified in Section 3 that motivate LOrE's midtraining. Rather than simply teaching models to invoke and coordinate subagents, LOrE explicitly instills and amplifies exploration-oriented behaviors: proposing diverse, nonredundant directions; meta-reasoning about exploration rather than directly solving problems; and sustaining adaptive exploration deep into reasoning trajectories. Our ablations (see other rebuttal responses) further demonstrate the importance of these behaviors, including the contribution of orchestration reasoning during midtraining and the performance degradation caused by reducing exploration diversity.

AuthorsOpenReview ↗

Response to Reviewer YatD (2/3)

Comparisons with APR, Kimi PARL, and ParaManager (cont.)


Why do open-ended tasks require different orchestration behaviors?

Existing parallel-agent approaches primarily consider problems with relatively clear decompositions into independently executable subtasks. For example, PARL considers only browsing and wide-search tasks (BrowseComp, WideSearch), often decomposable by entity, source, or sub-question. APR considers Countdown, a constrained arithmetic game whose search tree can be partitioned into well-defined branches. ParaManager meanwhile is trained on short-answer competition math (DAPO-Math), using external tools (Python, web search) and predefined specialist roles that provide structured delegation strategies. In these settings, the central challenges involve invoking subagents, delegating appropriate subtasks, and combining their outputs into useful knowledge.

In contrast, LOrE targets research-level problems (TCSBench, ArxivMath) and challenging Olympiad proofs (IMO ProofBench) that generally lack obvious upfront decompositions. Solution paths are typically unknown beforehand, plausible approaches are numerous, and mathematical challenges may only emerge after tens of thousands of tokens of reasoning. The challenge is therefore not merely distributing computation, but discovering, evaluating, and continually revising promising directions of exploration.

These differences motivate LOrE's methodological contributions. First, our synthetic data pipeline constructs demonstrations of effective exploration by selecting diverse, promising directions and synthesizing orchestration reasoning focused on exploration rather than direct problem-solving or superficial delegation behavior. Second, midtraining on synthetic data is used to establish these behaviors, which many strong open source LLMs do not exhibit and are difficult to discover through outcome-based RL alone (see our ablations in Section 5). Finally, multi-turn RL with an exploration bonus improves the orchestrator's ability to adapt its exploration strategy based on previous findings while continuing to discover useful directions. Together, these components train LOrE not merely to coordinate parallel execution, but to sustain productive exploration over long reasoning horizons. Such contributions are not present in the aforementioned prior work.


Compute-matched comparisons

We also conducted extensive compute-matched experiments, with full results presented in our response to Reviewer YGDf (comment #1/3) and Appendix G of our revised PDF submission. We find that LOrE-RL-9B outperforms both its midtrained-only counterpart and competing reasoning harnesses across nearly all tested inference compute budgets (up to ~1.5M tokens), including parallel-sequential approaches (RSA) and sequential compaction (RC). Furthermore, reducing the number of parallel exploration directions while compensating with greater orchestration depth degrades performance even at matched compute. Together, these findings provide evidence that LOrE's improvements arise not merely from using more test-time computation, but from using that computation more effectively to sustain diverse exploration.

AuthorsOpenReview ↗

Response to Reviewer YatD (3/3)

Isolating the contributions of direction selection, synthesized orchestration reasoning, and the exploration bonus

We have conducted additional experiments isolating the contributions of synthesized orchestration reasoning and the exploration bonus.


Exploration bonus

We conducted ablations of our RL exploration bonus, with detailed results presented in our response to Reviewer YGDf (comment #3/3). We find that removing the bonus degrades performance and substantially reduces the number of directions proposed per turn, while retaining it throughout training encourages excessive breadth at the expense of exploration depth, also reducing performance. These findings demonstrate that the exploration bonus plays an important role in regulating the breadth–depth tradeoff, with our default schedule achieving the best empirical balance.


Synthesized orchestration reasoning

We interpret this to mean ablating the (synthesized) orchestration reasoning we use in the midtraining stage of training. We therefore conduct an experiment where we removed the orchestration reasoning traces from the midtraining data while leaving the execution and final answer/proof traces untouched. We complemented this experiment with analogous experiments where we removed either the execution trace or the final answer/proof traces. We evaluated the resulting midtrained models without further RL.

Training data removedTCSBenchArxivMathIMO ProofBench
Final answer/proof21.5%25.0%45.6%
Orchestration19.3%18.8%41.6%
Execution18.0%25.0%39.1%
Full midtraining data (no removal)27.3%29.9%52.8%
Data removedTCSBenchArxivMathIMO ProofBench
Final answer/proof−5.8 pp−4.9 pp−7.2 pp
Orchestration−8.0 pp−11.1 pp−11.2 pp
Execution−9.3 pp−4.9 pp−13.7 pp

From the table above, we see that removing any of the three components reduces the performance of the resulting model. Removing orchestration reasoning is the most damaging, especially on ArxivMath.


Direction selection

We interpret this as referring to the data curation step used during synthetic midtraining data generation (Section 4.1), which serves to filter out redundant directions and prioritize those likely to yield useful information. In earlier experiments with a preliminary version of our data pipeline that did not include this filtering step, we observed that the resulting orchestrator frequently proposed highly redundant exploration directions. This observation motivated the inclusion of explicit direction selection in our final pipeline.

However, because these earlier experiments also differed in other aspects of data generation, we do not interpret this as a controlled ablation or claim that the improvements are attributable to direction selection alone. A rigorous study of this would require regenerating the synthetic exploration trajectories while holding the remaining pipeline fixed. This process would be very costly and time consuming. We therefore leave it for future work.

Reviewer_NRjhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies exploration contraction: as reasoning progresses, models increasingly develop or verify existing approaches rather than explore new ones. It proposes LOrE, an orchestrator–executor framework trained through synthetic midtraining and multi-turn RL. The orchestrator learns to propose exploration directions and build on the executors’ findings. Experiments show broader exploration and improved performance on challenging mathematical and scientific reasoning tasks.

Critical strengths
  1. The idea of learning a reasoning harness to guide exploration is interesting. Explicitly training a model to choose exploration directions and allocate computation provides a useful approach to scaling test-time reasoning.

  2. The method is motivated by careful observations and analysis. The study of exploration contraction and orchestration failures provides a clear rationale for the design, while the experiments demonstrate improvements in task performance.

Critical weaknesses
  1. The relationship between exploration diversity and task performance is not fully discussed. The results support the value of broader exploration, but novel directions are not necessarily useful. A clearer discussion of when to explore further and when to develop existing approaches would strengthen the paper’s central argument.

  2. The contribution and presentation could be clearer. The paper combines a harness, synthetic data, and multi-turn RL, but could better highlight its central innovation relative to existing orchestration and parallel reasoning methods. Bringing together the definitions of exploration behaviors and evaluation metrics would also make the results easier to interpret.

Questions for authors

As stated in weaknesses, could the authors clarify whether the existing analysis suggests that the gains come mainly from exploring more distinct approaches or from making better progress along existing ones? This would help explain what makes the learned exploration effective.

3 comments
AuthorsOpenReview ↗

Response to Reviewer NRjh (1/2)

Thank you for your review. We have addressed your concerns by (1) conducting additional experiments examining the relationship between exploration diversity and task performance, including the tradeoff between exploring new directions and developing existing ones, and (2) clarifying LOrE's central contribution.


The relationship between exploration diversity and task performance

Please see our detailed response to Reviewer YGDf (comment # 2/3), where we present several additional experiments investigating the role of exploration diversity. We summarize the key findings here.

First, reducing the diversity of exploration directions degrades performance even when the execution budget is held fixed. Specifically, we replace a subset of the orchestrator's proposed directions with duplicates of other directions from the same turn, preserving the number of executor calls. Performance consistently declines, particularly on research-level benchmarks (TCSBench and ArxivMath). This suggests that LOrE's gains arise not merely from allocating more computation, but from using that computation to explore distinct approaches.

Second, novel exploration directions continue to contribute key insights throughout the reasoning process. An analysis of successful TCSBench trajectories shows that directions classified as New directions contain solution-critical ideas even at later orchestration turns. Moreover, key insights continue to emerge late in these trajectories, supporting the value of sustained exploration rather than prematurely committing to a single approach.

Importantly, while our results demonstrate the importance of maintaining exploration diversity, effective exploration also requires balancing breadth and depth: the orchestrator must continually decide whether to investigate new directions or further develop existing ones. Our exploration-bonus ablations illustrate this tradeoff (see our response to Reviewer YGDf, comment #3/3). Removing the bonus leads to fewer directions per turn and longer trajectories, while maintaining it throughout training encourages greater breadth but shorter trajectories. Both configurations underperform our default schedule, suggesting that sustaining diverse exploration while allocating sufficient compute to develop promising directions is important for effective problem-solving.

AuthorsOpenReview ↗

Response to Reviewer NRjh (2/2)

The contribution and presentation could be made clearer.

Our central contribution is to formulate and address exploration orchestration as a distinct learning problem, motivated by the observation that existing CoT-based reasoning methods exhibit exploration contraction (Section 3). Unlike conventional parallel reasoning approaches, where orchestration primarily involves distributing computation across relatively well-defined subtasks, LOrE requires the orchestrator to discover, evaluate, and continually revise promising directions of exploration under uncertainty. See our response to Reviewer YatD (comments #1/3 and #2/3) for a detailed comparison of our work and prior agent orchestration work.

The three components of LOrE serve complementary roles toward this objective: (1) the orchestrator-executor harness separates exploration planning from execution, (2) synthetic midtraining instills exploration behaviors that current LLMs struggle to exhibit, and (3) multi-turn RL improves these behaviors through downstream outcome-based optimization. We will make this unified motivation more explicit in the introduction and clarify how our approach differs from existing parallel orchestration methods.

We also agree that the presentation of exploration behaviors and their evaluation could be better integrated. In our camera-ready paper, we will consolidate the discussion of the behavioral limitations identified in Section 3 (such as redundant delegation, premature termination, and direct problem-solving by the orchestrator) and explicitly connect these to the metrics used in our evaluation, alongside downstream task performance. This will clarify both which exploration behaviors LOrE aims to improve and how our experiments evaluate them.

With the discussion deadline fast approaching, we would like to kindly follow up on our rebuttal and additional experimental results. We would greatly appreciate it if you could take a look at our response to your feedback.

Thank you again for taking the time to help us improve our paper.

Reviewer_YGDfOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates exploration contraction in long-chain-of-thought reasoning, where models increasingly concentrate on previously discovered approaches rather than exploring new directions. To address this limitation, the authors propose LOrE, an orchestrator–executor framework that explicitly allocates test-time computation across multiple exploration directions. The method combines synthetic midtraining to establish orchestration behaviors with multi-turn reinforcement learning to optimize downstream performance. Experiments on research-level mathematics, theoretical computer science, and science benchmarks demonstrate improvements over standard reasoning and alternative inference harnesses. Additional analyses examine exploration diversity, orchestration behavior, and performance scaling to over one million generated tokens.

Critical strengths
  1. A well-motivated investigation of exploration contraction.

The paper provides empirical evidence that reasoning models progressively allocate less computation to novel directions (Figures 1–2). The analysis of existing reasoning harnesses and orchestration failures further motivates studying explicit exploration allocation rather than simply extending reasoning trajectories.

  1. Meaningful performance improvements with relevant training baselines.

The proposed method achieves substantial improvements across several challenging benchmarks (Table 3). Importantly, Table 5 compares against standard RL, teacher-distilled reasoning, and compaction-based training, providing evidence that the improvements cannot be explained solely by stronger conventional reasoning training.

Critical weaknesses
  1. The causal role of exploration diversity remains insufficiently established.

The paper identifies exploration contraction as a limitation of test-time scaling and attributes LOrE’s improvements partly to sustained exploration. However, Table 4 shows that RL substantially increases the number of explored directions (18.6 to 34.4 per problem), while the breadth ablation in Section 5 also changes the available computation. Consequently, the experiments do not clearly establish whether greater exploration diversity itself improves performance beyond simply allocating more computation.

  1. The contributions of individual RL mechanisms are not sufficiently isolated.

Section 5 introduces an exploration bonus and a context-buffer mechanism alongside multi-turn RL. Although Table 5 provides useful comparisons against alternative training strategies, it does not isolate these components. Since both mechanisms directly affect exploration behavior and training efficiency, their individual contributions remain unclear, making it difficult to determine which design choices are essential to the reported improvements.

Questions for authors
  1. Can the authors provide a compute-matched comparison that isolates the benefit of exploration diversity from the increased number of generated directions and tokens?
  2. How does performance change when the exploration bonus and context-buffer mechanism are individually removed from multi-turn RL? Are both necessary for achieving the reported gains?
5 comments
AuthorsOpenReview ↗

Response to Reviewer YGDf (1/3)

Thank you for your review. We have addressed your concerns by (1) running compute-matched comparisons of LOrE against a variety of baselines and ablation setups, (2) running additional experiments that explicitly study the role of exploration diversity in LOrE, and (3) running ablations that study the impact of the exploration bonus and context-buffer mechanisms in our RL setup.


Compute matched experiments

We provide the experimental details and results of our compute-matched experiments in our updated PDF submission, in the new Appendix G. Our main finding is that LOrE-RL-9B outperforms our baseline methods and its midtrained-only counterpart at almost all compute budgets we tested. We also find that limiting the number of parallel directions explored while maintaining matching compute reduces performance. More specifically:

  • [Appendix G.1] We plot compute-matched curves comparing LOrE-RL-9B against a variety of baseline methods, including the parallel-sequential harness RSA at various settings (including a parallel-sampling only variant) and the sequential-compaction approach RC, using both the base Qwen3.5-9B model and with an RL-trained variant as the LLM. We find that LOrE-RL-9B significantly outperforms all baselines at more or less all compute budgets up to ~1.5M tokens.

  • [Appendix G.2] We investigate the effect of limiting the number of parallel exploration directions (K) in LOrE-RL-9B, as in Section 5, while increasing the orchestration depth (T) to match the total inference compute of our default (K=5) setting. We find that reducing (K) generally degrades performance, even when compensated for by greater orchestration depth. This suggests that exploring diverse directions in parallel provides benefits that cannot be fully recovered by allocating the same compute to deeper sequential exploration.

  • [Appendix G.3] We plot compute-matched curves comparing LOrE-RL-9B against the midtrained only LOrE-MidT-9B. Again, we find that our final RL model outperformed the midtrained model at all compute budgets.

Together, these results demonstrate that LOrE's performance gains cannot be explained by increased test-time computation alone. Rather, LOrE uses its compute more effectively, with parallel exploration contributing to performance beyond what can be recovered by allocating equivalent compute to sequential exploration.

AuthorsOpenReview ↗

Response to Reviewer YGDf (2/3)

The role of exploration diversity

To investigate the importance of exploration diversity in LOrE, we conduct two complementary experiments. First, we show that artificially reducing the diversity of exploration directions degrades performance, even when the number of executor calls (and thus compute budget) is held fixed. Second, we show that novel exploration directions continue to contribute key ideas to successful solutions, including late into the reasoning trajectory. Together, these findings support the importance of sustained, diverse exploration for solving challenging problems.


Experiment 1: Reducing exploration diversity at fixed compute

We evaluate LOrE-RL-9B while artificially reducing the diversity of directions executed in parallel. At turn tt, let KtK_t denote the number of directions proposed by the orchestrator, and let DD denote the number of unique directions retained. We retain the first DD directions in their original generation order and replace the remaining Kt−DK_t-D directions with duplicates randomly sampled from among those retained. All directions are then executed as usual, with up to T=12T=12 orchestration turns and a maximum of Kt=5K_t=5 directions per turn.

Importantly, although some directions are duplicated, executor sampling remains stochastic, so identical directions can still produce different execution trajectories. This experiment therefore isolates the effect of high-level direction diversity while preserving total compute as well as the number of executor calls and allowing some diversity within execution. In contrast, our complementary experiments that directly reduce KK (Appendix G.2 and Section 5) reduce both direction diversity and the number of independent execution attempts.

Directions retainedTCSBench score / tokensArxivMath score / tokensIMO ProofBench score / tokens
D=129.0% / 1.334M32.6% / 1.456M57.8% / 1.025M
D=231.8% / 1.354M36.8% / 1.418M61.9% / 1.008M
Original34.2% / 1.339M40.6% / 1.457M62.2% / 1.006M

As shown above, replacing unique directions with duplicates consistently degrades performance, with the largest effects on the more open-ended research-level benchmarks TCSBench and ArxivMath. This provides direct evidence that exploring distinct high-level directions is more beneficial than allocating the same execution budget to repeated attempts at fewer directions, particularly on open-ended problems.


Experiment 2: Do novel directions contribute key insights?

We analyze successful LOrE-RL-9B trajectories on TCSBench (score ≥5/7\geq 5/7, up to T=8T=8 turns), using GPT-5.6-Sol (medium) to annotate each exploration direction along two dimensions:

  1. Direction type: Classify each direction as New direction, Continuation, or Verification, following the protocol in Section 3.1 (Appendix D).
  2. Key contribution: Using the problem's answer rubric as a guide, identify directions containing one or more key moves, ideas, or strategies that directly contribute to the correct solution.

For increasing turn cutoffs, we measure (1) the proportion of key moves associated with New directions, and (2) the proportion of trajectories containing at least one key move discovered after the cutoff.

Turn cutoffNew direction share of key movesRollouts with any key move at or after cutoff
Turn 2 onward31.9%80.7%
Turn 3 onward31.6%63.6%
Turn 4 onward35.9%52.8%
Turn 5 onward26.2%50.0%
Turn 6 onward17.4%33.3%
Turn 7 onward7.1%20.0%

Two observations:

  1. Novel exploration directions continue to contribute key insights even at later turns. A substantial fraction of key moves are associated with New directions, even as exploration progresses to approximately T=6T=6. This indicates that important ideas do not arise solely from developing directions identified early in the trajectory; continued exploration of new and diverse approaches contributes directly to successful solutions.
  2. Important discoveries continue to occur late in the trajectory. A meaningful fraction of successful trajectories contain key moves discovered only after several rounds of exploration. This supports the importance of LOrE's iterative orchestration, where findings from previous rounds inform further exploration rather than prematurely terminating the search.

Together, these results provide evidence that exploration diversity is an important contributor to LOrE's performance gains, beyond simply increasing test-time compute.

AuthorsOpenReview ↗

Response to Reviewer YGDf (3/3)

The contributions of individual RL mechanisms

We conduct two ablation studies examining the contributions of our RL mechanisms. We find that the exploration bonus plays an important role in balancing exploration breadth and depth (which translates into significant impacts on performance), while the context buffer primarily improves training efficiency rather than downstream performance.


Exploration bonus

We consider two ablations: (1) removing the exploration bonus entirely, and (2) retaining it throughout all 200 training steps, rather than disabling it after 100 steps as in our default setup.

TCSBenchArxivMathIMO ProofBench
no bonus31.8%33.3%55.9%
always bonus32.1%34.1%60.3%
default34.2%40.6%62.2%

As can be seen, removing the bonus reduces performance and substantially changes the orchestrator's exploration behavior: the average number of proposed directions per turn falls from approximately 4.5 to fewer than 2, while the average trajectory length increases from 7.6 to 10.3 turns. Conversely, retaining the bonus throughout training also reduces performance, albeit less severely. In this case, the orchestrator proposes 6.2 directions per turn on average (up from 4.5), many of which are discarded because only the first five are executed, while the average trajectory length decreases from 7.6 to 6.3 turns.

These results suggest that the exploration bonus regulates the breadth–depth tradeoff: removing it encourages narrower, deeper exploration, while applying it throughout training encourages excessive breadth at the expense of depth. Our default schedule achieves the best empirical balance.


Context Buffer

Our context buffer is primarily an efficiency mechanism, rather than a component intended to directly improve performance. By reusing previously generated exploration histories as starting contexts for new rollouts, the buffer allows us to avoid repeatedly regenerating early exploration turns, which reduces the cost of training at greater orchestration depths (e.g., T=8T=8 in our main experiments). To evaluate its contribution, we disable the buffer and reduce the maximum training depth from T=8T=8 to T=5T=5, which we estimate from previous experiments to approximately match the rollout compute of our default buffered setup. This comparison isolates the practical benefit of buffering under a fixed training compute budget: without the buffer, reaching the same training depth would require substantially more rollout computation.

TCSBenchArxivMathIMO ProofBench
no buffer34.5%37.5%62.2%
w. buffer34.2%40.6%62.2%

Evaluated at the usual extrapolated depth of T=12T=12, the no-buffer model achieves comparable performance despite training only to T=5T=5. This suggests that the buffer primarily enables more compute-efficient training at greater depths, rather than directly improving downstream accuracy. It also indicates that training at T=8T=8 may not be necessary for strong performance on our benchmarks, and that LOrE generalizes to orchestration depths beyond those encountered during training more effectively than we initially anticipated.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.