acceptodds
Under review as a conference paper at ICLR 2027

ReasonFork: Routing Small and Large Models by Future Consequence

Abstract

Collaboration between small and large language models can make reasoning more efficient by letting a small draft model handle routine steps and consulting a larger teacher selectively. The central challenge is to identify reasoning forks where replacing a draft action with a teacher action can improve task success within a limited step budget while controlling teacher costs. Existing routers based on draft confidence or process scores evaluate the draft's proposed action, but do not explicitly estimate how replacing it with a teacher action would affect subsequent task completion. To address this limitation, we introduce ReasonFork, a lightweight router that learns when to intervene by comparing the future consequences of draft and teacher actions. During training, the draft and teacher each propose an action from the same decision state. An offline evaluator compares the two successors under the same remaining step budget, and the difference in task outcomes provides supervision for the router. At inference, ReasonFork estimates the benefit of replacement using only information available before the teacher call and weighs it against the cost of calling the teacher. At each intervention, the teacher provides one action, after which the draft resumes the task. Experiments across five benchmarks spanning symbolic reasoning and interactive tasks demonstrate the effectiveness of ReasonFork. On reasoning tasks, the reported ReasonFork configurations match or exceed the accuracy of Always Teacher with approximately 72% lower teacher cost, and achieve absolute accuracy gains of 5.4–6.4% over strong baselines at comparable teacher cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM routing, stepwise routing, logical reasoning, cost-efficient reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_HCh6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ReasonFork studies stepwise routing between a small draft model and a larger teacher. At each decision state, the draft and teacher propose actions; an offline evaluator compares their successors under the same remaining horizon and labels the teacher replacement as a repair, tie, or harm. A router then predicts the expected value of replacing the draft action from pre-call features and requests the teacher when this value exceeds the estimated cost. The paper also gives a local utility rule and a reachability result for monotone, unit-cost proof systems. Experiments on RuleTaker, ProofWriter, ProverQA, ALFWorld, and WebShop show strong accuracy/token trade-offs on proof tasks, while results on interactive tasks are more mixed.

Critical strengths
  1. The paper distinguishes draft uncertainty from the benefit of replacing a draft action. Comparing the two successors from the same state gives the router a concrete supervision target, while online routing avoids searching over future action sequences.
  2. The proof-task evaluation goes beyond a single operating point, with budget curves, comparisons to stepwise and query-level routers, and an independent RuleTaker evaluation. This makes the reported proof-task gains more convincing than a fixed-budget comparison alone.
Critical weaknesses
  1. The evidence for signed repair/harm supervision is less clear than the evidence for learning replacement value in general. On the independent RuleTaker split, ReasonFork improves over draft-safety by 1.18 pAUC points on Qwen and 1.59 on Phi, with intervals excluding zero; its differences from repair-only are +0.26 [−0.17, 0.73] and −0.25 [−0.74, 0.21] (Table 8). The experiments therefore do not establish that explicitly modeling harm improves over a repair-only target.
  2. The efficiency conclusion depends on the cost metric. On ALFWorld, ReasonFork uses 11.213M total tokens and completes 29 episodes, compared with 7.511M total tokens and 35 successes for Always Teacher; total-token cost per success is 0.387M versus 0.215M (Table 18). On WebShop, ReasonFork reports lower wall time than Always Teacher (9.35 s versus 25.08 s), but higher total token use (4.436M versus 3.133M; Table 19). Teacher-token savings therefore do not imply lower total inference cost across tasks, and the cost of generating paired offline labels is not reflected in these serving-time comparisons.
Questions for authors
  1. What result in the submitted experiments establishes the benefit of signed repair/harm supervision over repair-only supervision? In Table 8, the intervals for ReasonFork minus repair-only include zero for both Qwen and Phi. Is there another prespecified comparison that resolves this, or should the claim be narrowed?
  2. Which efficiency claim is intended to be primary: fewer teacher tokens, fewer total tokens, lower wall time, or lower monetary serving cost? Please clarify the task-specific results in Tables 18–19 and whether the cost of generating paired offline labels is included or assumed to be amortized.
4 comments
AuthorsOpenReview ↗

Official Response to Reviewer HCh6 [1/3]

Thank you for your detailed feedback and valuable suggestions. We appreciate your positive assessment of our routing formulation and the thorough evaluation on proof tasks. Below, we respond to your comments and questions point by point.

W1: Evidence for the additional benefit of modeling harm

Thank you for this valuable comment. On WebShop, signed scoring improves reward pAUC over repair-only by 6.01 points [4.28, 7.81] (Table 4 and Appendix C.4). This comparison holds routing inputs, predictor capacity, training data and the allocation grid fixed. On the independent RuleTaker split, the confidence intervals for this comparison include zero. To help interpret this pattern, we examined whether the harm term changes which interventions are selected.

For this post hoc diagnostic, we used the original out-of-fold (OOF) predictions from five folds and kept the prediction heads fixed. Only the routing score changes. Both scores select from the same bank of paired states, using feasibility labels on RuleTaker and recorded continuation rewards on WebShop. The table below compares selections at a fixed 30% quota, matching the number of selected states rather than teacher token costs. Appendix C.3 reports this selection diagnostic, and Appendix C.4 gives the controlled comparison under teacher token budgets.

OOF settingStates selected per scoreShared selections (% of each quota)Without harm REPAIR / TIE / HARMWith harm REPAIR / TIE / HARM
RuleTaker, Qwen44794.18%68 / 379 / 069 / 378 / 0
RuleTaker, Phi42494.81%45 / 379 / 044 / 380 / 0
WebShop1,24965.73%573 / 657 / 19637 / 607 / 5

On RuleTaker, the selected sets overlap by more than 94%. Neither score selects any states labeled HARM by the feasibility proxy at this quota, and the REPAIR count changes by only one in each model family. The harm term therefore has little effect on selection in this diagnostic.

On WebShop, signed scoring selects 64 more REPAIR states and 14 fewer HARM states at the same quota. The diagnostic suggests that the harm term adds value by refining intervention selection where repair probability alone leaves harmful replacements among the preferred candidates. This effect is more evident in the WebShop selections and is consistent with the reward-pAUC improvement reported above. We describe the controlled comparison and additional scoring ablations in Q1 below.

AuthorsOpenReview ↗

Official Response to Reviewer HCh6 [2/3]

W2: Teacher allocation, total usage and timing.

Thank you for raising this practical concern. Our primary efficiency measure is teacher prompt and completion tokens, which quantifies the large-model assistance used during task execution. The approximately 72% reduction in the Abstract refers to the five proof configurations. To show how this saving relates to overall execution, we also report total model tokens and recorded episode times alongside task performance.

ALFWorld. Table 25 separates teacher usage from total model usage:

PolicyTeacher tokens (M)Total tokens (M)Successful episodesTokens per success (M)
ReasonFork3.70611.213290.387
Confidence5.44314.028170.825
Always Teacher7.5117.511350.215

Under the same 25-call cap, ReasonFork completes 29 episodes versus 17 for Confidence while using fewer teacher and total tokens. Always Teacher operates without this cap and completes 35 episodes with fewer total tokens per success. ReasonFork’s total includes 7.507M draft tokens in addition to its 3.706M teacher tokens. This explains how reducing teacher usage can still result in a higher total token count when draft generation is included.

WebShop. ReasonFork uses 0.267M teacher tokens, compared with 3.133M for Always Teacher. Their total usage is 4.436M and 3.133M tokens, respectively, while recorded episode times are 9.35 and 25.08 seconds (Table 26). Exact success is 14.8% and 24.4%, so the timing results describe different operating points rather than a comparison at matched task quality.

Proof tasks. At the Table 1 operating points, the breakdown similarly shows lower teacher usage alongside 17.8%–33.0% more total unweighted tokens than the recorded Always Teacher references (Table 8). Since the draft and teacher differ in size, summing their tokens describes model usage but does not directly determine FLOPs or monetary cost.

Thus, teacher tokens measure the assistance allocated to the larger model, while total tokens and timing describe additional aspects of execution. We have made this distinction explicit in Sections 4.1–4.3 and presented each measure together with the corresponding task performance. Offline preparation is addressed separately in Q2.

AuthorsOpenReview ↗

Official Response to Reviewer HCh6 [3/3]

Q1: Which submitted comparison supports signed supervision?

The controlled WebShop comparison in Table 4 and Appendix C.4 supports the additional reward benefit of signed supervision. The scoring ablations on WebShop and ALFWorld further examine how the harm term affects task outcomes and teacher usage when the prediction heads remain unchanged.

  • Controlled WebShop comparison. The study compares repair-only scoring pRp_{R} with signed scoring pR−2pHp_{R}- 2p_{H}. Public inputs, predictor capacity, training data and the allocation grid are held fixed to focus the comparison on the supervision target. It evaluates 19,000 trajectories on 500 shared tasks over the fixed 10%–60% teacher-token range. Signed scoring improves reward pAUC by 6.01 points [4.28, 7.81], with the 95% confidence interval obtained from 10,000 paired task-bootstrap resamples. Success pAUC also meets the predefined noninferiority criterion, with a margin of one point.

  • WebShop scoring ablation. Here, the prediction heads are kept fixed, and the harm term is removed only when computing the routing score. Teacher-action share increases from 6.39% to 10.23%, while reward decreases from 46.35% to 45.77% (Appendix E.3). At these operating points, including harm achieves higher reward with a smaller share of teacher actions.

  • ALFWorld scoring ablation. We also perform a separate scoring comparison on ALFWorld, keeping public inputs, fitted heads and the scheduler fixed. Under the same 25-call cap, signed scoring completes 28 episodes versus 23 for repair-only, while using 50.61% versus 59.85% of Always Teacher tokens (Appendix E.4).

Thus, the controlled WebShop comparison and the scoring ablations together support the practical value of accounting for harm. In the decision-making process, considering both the possibility of improving the outcome and the risk of worsening it helps the router make more effective use of teacher assistance.

Q2: Primary efficiency claim and offline preparation.

Our primary efficiency claim concerns reduced teacher-token usage during serving. The serving results exclude offline preparation, which is reported separately. For a fixed training bank, this preparation is performed once, and the resulting router can be reused for subsequent requests without repeating the paired evaluations.

The recorded prompt and completion tokens used to prepare supervision are:

SettingRecorded LM tokensIncluded preparation
WebShop32,064,703600 training and 120 development tasks, including paired proposals and draft continuations
ALFWorld23,343,888All 200 collected episodes, including discarded episodes

The WebShop total consists of 27,240,031 draft tokens and 4,824,672 teacher tokens. The 164 retained ALFWorld episodes account for 19,104,124 tokens, which are already included in the full collection total. Each training bank is counted once.

For a fixed bank reused across NN deployment requests, the recorded preparation tokens contribute Tprep/NT_{\mathrm{prep}}/N per request, in addition to the serving tokens. The reported serving results do not include this preparation term or assume a particular deployment volume. As the same bank is reused for more requests, its preparation cost is spread over those requests.

We have reported the preparation token counts and their coverage in Appendix B.6, separately from serving usage. This allows readers to assess the model usage required to prepare and use the router for their intended deployment volume.

Thank you again for your thoughtful review and insightful suggestions. We sincerely appreciate the time you have devoted to reviewing our work. We hope that our responses have satisfactorily addressed the remaining concerns.

Thank you for the clarifications. My concerns have been addressed. And I will maintain my rating (3: Weak acceptance).

Reviewer_q5DVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReasonFork, a routing method for collaboration between small and large language models. Instead of relying on the small model's confidence, ReasonFork predicts whether replacing its current action with a large model's action would improve downstream task outcomes. The router is trained using offline paired comparisons and makes cost-aware intervention decisions at inference time. Experiments on five benchmarks show promising improvements in task performance while reducing teacher model usage.

Critical strengths
  1. The paper identifies a meaningful limitation of confidence-based routing and proposes a reasonable alternative based on downstream outcomes.

2.ReasonFork achieves substantial accuracy gains while reducing teacher costs, particularly on symbolic reasoning tasks.

  1. Comprehensive evaluation: The paper includes comparisons with strong baselines, ablation studies, and transfer experiments to support its effectiveness.
Critical weaknesses
  1. Limited novelty. The core idea resembles existing value-based routing and advantage estimation methods. The main contribution appears to be the supervision target rather than a new routing algorithm.

  2. Dependence on task-specific supervision. The method relies on symbolic evaluators, planners, or rollouts to obtain training labels, which may be difficult or expensive for general LLM tasks.

  3. Limited generalizability. The strongest results come from structured reasoning tasks with verifiable actions. The gains are less significant on interactive tasks, raising questions about applicability to more complex real-world agents.

Questions for authors
  • Do you have any deep insights about what kind of features do the router learn? Why does it work?
  • Is the method specific to certain tasks?
4 comments
AuthorsOpenReview ↗

Official Response to Reviewer q5DV [1/4]

Thank you for your constructive feedback and valuable suggestions. We appreciate your recognition of ReasonFork’s motivation and the strong empirical support for its effectiveness. Our responses are provided below.

W1: Concerns about methodological novelty.

Thank you for your thoughtful and constructive feedback. ReasonFork predicts whether replacing a draft action with a teacher action would help the draft finish the task within the remaining step budget. Our contribution is to characterize these intervention points and learn their replacement value, so the router can decide whether a teacher call is worth its cost. The distinctions from related work are as follows:

  • Decision granularity and routing signal. RouteLLM [1] learns to select a model for an entire request, while STEER [2] uses small-model confidence to route individual steps. However, low confidence does not guarantee a better teacher action, and a confidently chosen action may still leave too few steps to finish. ReasonFork therefore routes according to the predicted downstream benefit of replacing the draft action.

  • Learning replacement value. TRIM [3] uses process scores in threshold, reinforcement learning and POMDP policies. AlphaRouter [4] optimizes token-level routing using counterfactual advantages from collaborative inference trees. ReasonFork directly learns the value of replacing one draft action from paired draft and teacher successors evaluated from the same state under the same remaining horizon. The estimate supports local routing without additional sequential policy optimization.

  • Characterizing consequential interventions. Beyond the supervision target, Theorem 1 characterizes when one legal action allows completion within the remaining step budget while another does not, under the stated proof-system assumptions. This explains why legality alone is insufficient for routing. Proposition 1 translates predicted replacement benefit and calling cost into a conditional local utility rule.

The experiments examine the practical value of this design. With routing inputs and predictor capacity fixed, replacement scoring improves over draft-risk routing by 4.13 success AUC points on ALFWorld and 3.06 reward pAUC points on WebShop (Table 4). Under shared supervision and routing inputs on RuleTaker, PPO shows no statistically significant pAUC improvement over the local gate (Table 2(b) and Appendix C.2).

[1] Ong et al. RouteLLM: Learning to Route LLMs from Preference Data. ICLR 2025.

[2] Lee et al. Confidence-Guided Stepwise Model Routing for Cost-Efficient Reasoning. AAAI 2026.

[3] Kapoor et al. TRIM: Hybrid Inference via Targeted Stepwise Routing in Multi-Step Reasoning Tasks. ICLR 2026.

[4] Liao et al. AlphaRouter: Token-level Routing Between SLM and LLM with Reinforcement Learning and Tree Search. ICML 2026.

AuthorsOpenReview ↗

Official Response to Reviewer q5DV [2/4]

W2: Obtaining task-specific supervision and its cost.

Thank you for raising this practical concern. ReasonFork uses a common action-replacement framework, with each task implementing the following three components:

  • Action interface. The draft and teacher use the same task interface. The teacher replaces one draft action, after which the draft resumes.

  • Observable routing inputs. The router uses only pre-call information, including public observations, interaction history, the draft proposal and the remaining step budget.

  • Paired outcome feedback. Offline training compares draft and teacher successors from the same state under the same evaluation criterion and remaining horizon to learn replacement value.

For broader reasoning tasks with checkable outcomes, paired feedback could be obtained by evaluating draft continuations after the two candidate actions. At inference, the trained router estimates replacement value from observable inputs without repeating these paired evaluations.

Offline preparation cost. The following table reports the prompt and completion tokens used to prepare offline supervision (Appendix B.6). This preparation is needed only once, and the trained router can be reused for subsequent requests.

SettingModel tokensIncluded preparation
RuleTaker, Qwen1,462,732Paired action proposals
RuleTaker, Phi1,356,215Paired action proposals
ProverQA1,095,410Paired action proposals
ALFWorld23,343,888All 200 collected episodes
WebShop32,064,703600 training and 120 development tasks

Reducing supervision requirements. We also examined the effects of using fewer labeled examples, fewer sampled continuations and approximate search on Qwen RuleTaker:

  • Training data. Using 75 labeled groups yields 96.56% accuracy, compared with 97.00% using 300 groups, at similar teacher usage: 32.79% and 32.61% of Always Teacher tokens, respectively (Appendix D.9).

  • Rollout budget. One sampled continuation per branch uses 2.226M continuation tokens and achieves 94.43 success pAUC, compared with 17.807M tokens and 93.56 pAUC for eight samples, over the common teacher-token range (Appendix D.9).

  • Approximate search. Beam-1 uses 33.54% of exact search expansions, with 99.06% label agreement. The resulting router achieves 96.33% accuracy, compared with 97.00% using exact labels (Appendix D.9).

AuthorsOpenReview ↗

Official Response to Reviewer q5DV [3/4]

W3: Concerns about generalizability beyond structured reasoning.

Thank you for raising this important point. ReasonFork predicts whether replacing a draft action would improve downstream outcomes enough to justify the teacher cost. The same decision arises in agent tasks where current actions affect subsequent observations, available choices and eventual task completion.

  • ALFWorld. The agent must complete multi-step tasks from partial observations, using interaction history to guide subsequent actions. The router therefore needs to assess whether a teacher replacement can help the draft finish within the remaining step budget, despite incomplete information about the environment state.

  • WebShop. The agent must satisfy a shopping instruction through successive searches and clicks. Each action changes the visible page and available choices, so its value depends on how it affects subsequent interactions. The router predicts this downstream benefit without access to future pages, hidden product metadata or a target-support graph.

Experimental evidence. At the same 25-call cap on ALFWorld, ReasonFork completes 29 episodes versus 19 for TRIM-Thr, using 20.8% fewer teacher tokens (Table 3). On WebShop, the request-level baseline is interpolated to ReasonFork’s 6.39% teacher-action share, allowing comparison of request-level and stepwise allocation at the same action budget (Appendix E.3).

EvaluationImprovement in points
ALFWorld success over TRIM-Thr+7.46
WebShop reward over matched request-level allocation+6.21
WebShop exact success over matched request-level allocation+3.64

The ALFWorld router is trained on Phi trajectories and frozen before evaluation with Qwen on all 134 unseen episodes. The gains are therefore obtained without refitting for the target model pair, providing evidence of cross-model reuse.

ReasonFork uses observable inputs and paired outcome feedback, making its routing framework adaptable across tasks. This framework also has broad potential for expansion into other areas, such as intelligent agents for web page operations and tool invocation, code repair, and embodied intelligent planning. In code repair, a teacher could replace a proposed edit before the draft resumes, with test results providing training feedback. Furthermore, the framework could also extend from ALFWorld to embodied agents using visual observations and interaction history for high-level planning. Our goal is to help small-model agents handle more complex tasks while keeping teacher costs manageable.

AuthorsOpenReview ↗

Official Response to Reviewer q5DV [4/4]

Q1: What do the router features capture, and why do they help?

Thank you for this thoughtful question. The router learns which draft choices a teacher is likely to improve, using information about the goal and the steps left to finish. On RuleTaker, the features help assess whether the draft action advances the proof and whether enough steps remain to finish (Section 3.3 and Table 7).

A valid action may use up a step without helping prove the target. When few steps remain, such a detour can prevent completion. During offline training, draft and teacher actions are compared from the same state under the same remaining budget. The router learns how its features relate to these paired outcomes, including when replacement helps, offers no benefit or causes harm. The router learns when the teacher can help, not just when the draft might fail.

In the feature ablation, removing all graph features lowers Phi RuleTaker accuracy from 94.67% to 88.67% at similar teacher usage (Table 23). The intervention results further show how routing affects task outcomes. On Qwen RuleTaker, ReasonFork produces 83 repairs and one harm, compared with 50 repairs and 12 harms for TRIM-Thr at similar teacher usage, with repairs and harms measured relative to Always Draft (Figure 5).

Q2: Is ReasonFork specific to certain tasks?

Thank you for your valuable comments. ReasonFork uses the same routing principle across tasks, with its inputs and feedback adapted to each setting. Applying it to a new task involves defining the actions available to both models and how their outcomes are compared. The router then estimates whether a teacher replacement would improve the outcome enough to justify its cost. We discuss reuse at two levels:

  • Reuse of the framework. The available actions, routing inputs and outcome feedback are specified for each task. For example, WebShop retains the same replacement decision while using a router trained on page observations and paired outcome feedback.

  • Reuse of a trained router. We also test transfer with both source weights and thresholds frozen. The Qwen-trained RuleTaker router achieves 93.67% accuracy on Phi, using 27.02% of Always Teacher tokens. Transferring the RuleTaker router to ProofWriter QDep3 achieves 94.59% accuracy at 29.75% of Always Teacher tokens (Table 6 and Appendix F). ALFWorld provides another example of cross-model reuse: a router trained on Phi trajectories is applied to Qwen without retraining.

The experiments support a common routing framework across tasks and provide evidence of transfer across models and related proof tasks.

We appreciate your positive assessment of our motivation and evaluation, as well as your constructive suggestions for improving the paper. Thank you again for your time and thoughtful feedback. We hope that our responses have satisfactorily addressed your concerns.

Reviewer_jCpPOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ReasonFork, a step-level collaboration framework between a small draft model and a larger teacher under limited action horizons and teacher-call budgets. Rather than relying on draft uncertainty, ReasonFork predicts whether replacing a draft action with a teacher action improves downstream task success. During training, an offline evaluator compares successor outcomes from draft and teacher actions to generate repair, tie, or harm labels. A lightweight router then estimates the expected replacement benefit against teacher cost using pre-call features. At inference, the teacher provides at most one action per intervention before the draft resumes, avoiding online successor search. Experiments across five reasoning and interactive benchmarks demonstrate competitive accuracy with substantially reduced teacher usage. The main contribution lies in replacement-benefit supervision and systematic cost–success evaluation, although offline labeling requirements, feature dependence, and sequential deployment reliability remain concerns.

Critical strengths
  1. A well-defined routing rule and deployable inference process. The paper derives a cost-sensitive routing rule from expected replacement utility, separating expensive offline evaluation from lightweight online inference. Each teacher intervention provides one action before returning control to the draft. Although local optimality does not guarantee trajectory-level optimality, the framework clearly establishes its assumptions and deployment procedure.

  2. Strong proof-task performance and cost efficiency. Across five proof configurations, ReasonFork achieves high accuracy using only 21.91–32.61% of teacher tokens, often matching or outperforming Always Teacher. Paired budget-curve comparisons against TRIM-Agg further demonstrate statistically supported improvements beyond a single budget setting.

  3. Meaningful evaluation beyond symbolic reasoning. Experiments on ALFWorld and WebShop extend the framework to partially observed interactive environments. In ALFWorld, ReasonFork achieves 21.64% success under a 25-call cap, outperforming competing routers. Controlled ablations further support the value of signed replacement supervision over simpler risk-based or repair-only targets.

  4. Validation of realized outcomes and robustness. The paper evaluates actual intervention consequences through paired rollouts, examines frozen transfer across settings, and investigates sensitivity to planning horizons, graph perturbations, and approximate supervision. These analyses strengthen the empirical evidence beyond standard routing comparisons.

Critical weaknesses
  1. The strongest supervision relies on privileged offline outcome evaluation. For symbolic proof tasks, training labels compare reachability or outcomes for both successors from a verified state, assisted by graph structure and legal-action semantics (Section 3; Appendix A). The partial-observation experiments demonstrate that exact graphs are not always necessary, but they use task-specific observations, structured action interfaces, and offline continuation feedback. In unrestricted natural-language reasoning, obtaining comparable counterfactual labels may require costly, noisy rollouts or a learned judge. This dependency limits how generally the method can be transferred without additional evidence and should be explicit in the contribution claim.

  2. A local utility rule is not a solution to the full sequential allocation problem. Proposition 1 establishes an optimal single-step conditional decision when the immediate signed outcome estimate and call cost are given. It does not optimize a finite-horizon policy with changing remaining budget, future opportunities for teacher calls, or the state distribution induced by earlier interventions. The deployed scheduler handles budgets heuristically, while the supervision is fitted from offline state distributions. Stronger comparisons to sequential policies trained on the same paired labels, features, and compute budget are needed to show that the particular local value formulation is sufficient.

  3. Teacher-token savings are not equivalent to lower total inference cost in every environment. The favorable 72% figure in the Abstract summarizes proof-task teacher-token usage relative to Always Teacher. In ALFWorld at 25 calls, Table 3 reports 11.213M total tokens for ReasonFork versus 7.511M for Always Teacher, although ReasonFork improves over several cheaper-quality routing policies at the same call cap. This does not invalidate the proof results; it shows that draft continuations and environment-specific trajectory lengths can reverse a total-cost comparison. Accuracy, teacher tokens, total tokens, successful-task-normalized cost, and latency should be kept separate throughout.

  4. Proxy feasibility does not always predict the real benefit of an intervention. Table 5 is a helpful control: teacher-first completion improves 16.53–18.98 points at the original horizon H, but only 2.66–3.61 points at 2H. Further, some proxy-TIE states yield real repairs or harms once the draft continues, and only approximately 65–72% of proxy-REPAIR states realize a repair. These findings illustrate horizon sensitivity and misspecification of the offline target. An analysis of when this mismatch misroutes calls—especially after prior teacher interventions—would materially clarify robustness.

Questions for authors
  1. If paired labels must be estimated by finite sampled continuations or imperfect judges, how does performance vary with label error, rollout budget, and calibration data size? What is the complete offline label-generation cost for the reported settings?

  2. Could a sequential budget-aware router trained with the same paired outcomes, feature vectors, training states, and offline compute outperform the one-step decision rule? If so, what is ReasonFork’s practical advantage?

  3. Can the authors explicitly report total draft + teacher + router computation, rather than only teacher-token fraction, for all proof and interactive tasks? In particular, how should readers interpret the ALFWorld Always Teacher total-token advantage in Table 3?

4 comments
AuthorsOpenReview ↗

Official Response to Reviewer jCpP [1/4]

We thank Reviewer jCpP for the careful and constructive review. We appreciate your recognition of ReasonFork’s practical contribution to efficient model collaboration and our comprehensive empirical evaluation. Below are our responses to your comments and questions.

W1: Offline supervision and applicability beyond structured reasoning.

Thank you for your valuable feedback. ReasonFork learns when to replace a draft action with a teacher action by comparing their outcomes from the same state during offline training. At inference, the router uses observable inputs to estimate the benefit of replacement and decide whether teacher intervention is worthwhile. We clarify the action interface, routing inputs and training feedback below:

  • Action replacement. The draft and teacher use the same task interface. The teacher replaces one draft action, after which the draft resumes.

  • Observable routing inputs. The router uses only pre-call information, including public observations, interaction history, the draft proposal and the remaining action budget.

  • Paired outcome feedback. Offline evaluation compares the outcomes following the two actions under the same evaluation criterion and remaining horizon, providing repair, tie or harm labels.

ReasonFork can be adapted to other tasks by defining the action interface and routing inputs and obtaining suitable offline outcome feedback. We demonstrate this adaptation beyond structured reasoning in ALFWorld and WebShop and also evaluate router reuse without refitting:

  • Task adaptation. ALFWorld uses planner feedback during offline training, while WebShop uses returns from paired continuations (Appendices E.1 and C.4). With inputs and predictor capacity held fixed, replacement scoring improves over draft-risk scoring by 4.13 success AUC points on ALFWorld and 3.06 reward pAUC points on WebShop (Table 4).

  • Parameter reuse. With the source router’s weights and thresholds fixed, ReasonFork achieves 93.67% accuracy on Phi at 27.02% of Always Teacher tokens and 94.59% accuracy on ProofWriter QDep3 at 29.75% of Always Teacher tokens (Table 6).

For broader natural language reasoning with checkable outcomes, offline supervision could come from comparing draft continuations after draft and teacher actions from the same prefix. Reliable feedback requires accurate evaluation, while generating and evaluating both continuations incurs offline costs. Q1 discusses supervision reliability and offline preparation costs in detail.

We have clarified these offline supervision requirements and the scope of our evaluation in Section 3.3 and Appendix B.2.

W2 and Q2: Sequential allocation and the practical advantage of the local rule.

Thank you for this valuable comment. Proposition 1 gives a conditional rule for deciding whether to intervene at the current step. To examine whether optimizing decisions over a full trajectory provides additional benefit, we compared ReasonFork’s local gate with a myopic controller and PPO across teacher token budgets.

On RuleTaker, PPO and the myopic controller used the same paired supervision, training states and routing features, including budget and cost information. They also shared the same actor architecture and initialization from the paired predictor. PPO was trained to maximize final verified success minus normalized teacher token cost. We trained the models on the training set and evaluated them on the existing confirmation set.

The table reports PPO minus each comparator over their common teacher token ranges.

ComparisonSuccess pAUC difference (PPO minus comparator, with 95% CI)Common teacher token range (tokens per task)
PPO vs. local gate−0.26 [−0.95, +0.44]142.56–521.07
PPO vs. myopic controller−0.50 [−2.38, +0.47]142.56–345.46

We also compared PPO and the myopic controller using the same number of optimization updates per fit.

Updates per fit (both controllers)Success pAUC difference (PPO minus myopic, with 95% CI)Common teacher token range (tokens per task)
64−0.49 [−2.35, +0.50]141.86–345.46
192−0.96 [−1.65, +0.81]142.56–299.29

According to RuleTaker's comparison results, PPO did not show a statistically significant improvement compared to the other two comparative models (Section 4.2, Table 2(b), and Appendix C.2).

ReasonFork’s practical advantage is that the replacement value estimator can be deployed directly as a lightweight gate, without additional sequential policy training. At each step, the gate compares the predicted benefit of replacement with the teacher cost and decides whether to intervene.

AuthorsOpenReview ↗

Official Response to Reviewer jCpP [2/4]

W3: Teacher token savings and total inference cost.

Thank you for this valuable comment. Since the larger teacher model is more computationally expensive to run than the draft, ReasonFork aims to achieve strong task performance with limited teacher assistance by learning when intervention is worthwhile. Total token usage also includes draft inference, so we report teacher and total token usage separately below.

  • Teacher resource usage. We use teacher prompt and completion tokens to quantify demand on this resource. Across the five proof configurations, ReasonFork achieves 95.4% average accuracy with approximately 72% fewer teacher tokens than Always Teacher.

  • Overall token usage. Table 25 reports both teacher and total tokens on the same 134 ALFWorld episodes. ReasonFork and Confidence share a 25-call cap, while Always Teacher uses the teacher at every step.

PolicyTeacher tokens (% of AT)Total tokens (M)Successful episodesTotal tokens per success (M)
ReasonFork49.3411.213290.387
Confidence72.4714.028170.825
Always Teacher100.007.511350.215

ReasonFork first obtains a draft proposal and selectively requests a teacher replacement, so its total includes both draft and teacher inference: 7.507M and 3.706M tokens, respectively. Always Teacher directly generates each action and uses 7.511M tokens in total. Under the same 25-call cap, ReasonFork completes 29 episodes versus Confidence’s 17, using 31.9% fewer teacher tokens and 20.1% fewer total tokens. Its total tokens per successful episode are also less than half those of Confidence (0.387M versus 0.825M).

In response to Q3, we further report model-specific token usage and measured routing times to provide a more complete account of the resource usage and overhead of collaborative inference.

W4: Feasibility labels and actual outcomes after earlier teacher calls.

Thank you for raising this point. We examined whether differences between feasibility labels and actual outcomes affect routing after earlier teacher interventions. The analysis covered all 1,215 states visited by the deployed router in the Qwen RuleTaker setting, including 448 states reached after an earlier teacher intervention.

At each recorded state, we compared two branches: one started with the stored draft action and the other with the stored teacher action. Both then continued with the same frozen draft model using greedy decoding and no further teacher calls. The router, thresholds and original call decisions remained fixed.

For the 448 states following an earlier intervention, completion outcomes at the original deadline H are:

States evaluatedBeneficialUnchanged outcomeHarmful
All 448 states4039711
192 states selected for a call351561

After earlier teacher interventions, the router selects 35 of the 40 beneficial replacements and only one of the 11 harmful replacements. Among the 156 selected calls with unchanged completion, both branches succeeded in 113 cases and failed in 43.

Across all 1,215 states at H, we identified 18 missed beneficial replacements, six harmful calls and 302 selected calls with unchanged completion. All missed beneficial replacements and harmful calls occurred in states labeled TIE by the feasibility proxy. TIE indicates equal feasibility, but the draft may follow a successful path after only one of the two actions.

We further examined the effect of the completion deadline by extending H to 2H, with both deadlines measured from the task start. Of the 40 cases beneficial at H, 38 succeed in both branches at 2H. With the original call decisions held fixed, the same 192 selected calls yield four beneficial, 186 unchanged and two harmful outcomes at 2H. These transitions indicate that many replacements help the draft finish within the original step budget, even when it can succeed without replacement under the longer budget. Appendix D.10 reports the outcome counts at both deadlines and the transitions described above.

AuthorsOpenReview ↗

Official Response to Reviewer jCpP [3/4]

Q1: Label errors, rollout budget, calibration data size and offline labeling cost.

Thank you for the suggestions. We evaluated sensitivity to label errors, rollout budgets and data size, and report offline model usage below.

Label errors and data size. We used the same features and classifier as in the original RuleTaker experiments and evaluated all conditions on the same 300 confirmation theories. All conditions used a teacher usage target of 30% of Always Teacher (AT). We randomly changed 5%, 10% or 20% of the labels to another class. The results show that ReasonFork retains high accuracy across the tested noise levels and training set sizes. Results for 5%, 10% and 20% label errors and for 75 and 150 training groups are averaged over three random seeds. The clean baseline with 300 training groups reports the original result.

Training conditionAccuracy (%)Teacher tokens (% of AT)
Clean labels, 300 groups97.0032.61
5% label errors96.0034.25
10% label errors95.5631.59
20% label errors95.1127.64
Clean labels, 75 groups96.5632.79
Clean labels, 150 groups95.8933.26

Rollout budget. We used 1, 2, 4 or 8 sampled draft continuations per branch to compare completion rates at H after the teacher and draft actions, keeping the paired actions and other sampling settings fixed. We sampled continuations for 714 pairs with different actions, while the remaining 773 pairs had identical actions and were assigned TIE without additional sampling. Agreement is measured against labels from eight samples per branch, so the reference row has 100% agreement by definition. The results indicate that one sample per branch already yields high pAUC, with no observed gain from larger sampling budgets in this control.

Samples per branchLabel agreement on different-action pairs (%)Success pAUC (0–100)Continuation tokens (M)
178.2994.432.226
286.8394.384.445
492.5893.998.908
8100.0093.5617.807

Calibration data size. We fixed a predictor trained on 240 theory groups and used 15, 30 or 60 groups from a separate calibration set to select the threshold. The teacher usage target remained 30% of AT.

Calibration groupsAccuracy (%)Teacher tokens (% of AT)
1596.5635.99
3096.6736.59
6097.0037.57

Offline preparation cost. ReasonFork prepares paired supervision offline and reuses the trained predictor for subsequent requests within a fixed task and model setting. Online routing does not repeat the paired outcome evaluations, so the preparation cost can be amortized over continued use. The table reports the model tokens used to prepare each supervision bank. ProofWriter uses a fixed structural score and requires no predictor trained on paired labels.

SettingModel tokensIncluded preparation
RuleTaker, Qwen1,462,732Paired action proposals
RuleTaker, Phi1,356,215Paired action proposals
ProverQA1,095,410Paired action proposals
ALFWorld23,343,888All 200 collected episodes
WebShop32,064,703600 training and 120 development tasks

For the proof tasks, the measured preparation steps took 2.24, 1.58 and 5.56 CPU seconds on RuleTaker/Qwen, RuleTaker/Phi and ProverQA, respectively, plus 3.10 CPU seconds of shared initialization. On WebShop, model selection, fitting and deployment preparation across the comparison models took 208.32 CPU seconds.

The supplementary RuleTaker experiment also shows that a smaller supervision budget can be effective. Using one rather than eight continuations per branch reduces continuation tokens by 87.5%, while achieving success pAUC of 94.43 versus 93.56 over the same teacher-token range. Appendix D.9 reports this experiment, and Appendix B.6 provides the offline model-token accounting.

AuthorsOpenReview ↗

Official Response to Reviewer jCpP [4/4]

Q2: Sequential allocation and the practical advantage of the local rule.

We have addressed this question together with W2. Please see the combined response above.

Q3: Total computation for the draft, teacher and router.

Thank you for the suggestion. We report token usage for both models and execution times for the measured routing components. The following table summarizes the proof tasks at the operating points in Table 1, with Always Teacher (AT) as the reference.

SettingDraft tokensTeacher tokensTotal tokensAT total tokens
RuleTaker, Qwen608,502197,730806,232606,345
RuleTaker, Phi582,107186,980769,087595,751
ProofWriter, Qwen234,47068,211302,681235,815
ProofWriter, Phi218,52449,945268,469227,965
ProverQA, Qwen1,174,411297,5571,471,9681,179,386

The total includes the additional draft inference. Because draft and teacher tokens have different computational costs, a higher combined token count does not by itself imply a higher inference cost.

The measured routing components are lightweight, with median CPU runtimes ranging from 0.291 to 5.747 ms. Table 10 in Appendix B.6 reports these timings and their measurement scope. Additional measurements give a mean elapsed time of 1.88 ms per ProofWriter step and a mean CPU time of 4.18 ms per ALFWorld decision, including graph construction, prediction and budget scheduling.

On ALFWorld, ReasonFork uses 49.34% of AT’s teacher tokens. Its 11.213M total comprises 7.507M draft tokens and 3.706M teacher tokens, since each routing decision starts with draft inference. AT directly generates each action and uses 7.511M tokens without draft inference. Under the same 25-call cap, ReasonFork completes 29 episodes compared with Confidence’s 17, using 31.9% fewer teacher tokens and 20.1% fewer total tokens (Table 25 and W3). For WebShop, Table 26 reports total token usage, including scorer usage, and recorded wall time.

In closing, we thank the reviewer again for the careful review and for recognizing the clarity of our framework and the breadth of our evaluation. Your thoughtful questions and suggestions have helped us improve the paper. We hope that our response has satisfactorily addressed your concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the Area Chair and Reviewers

Dear Area Chair,

We would like to express our sincere gratitude for your efforts in coordinating the review process. We are also grateful for the thoughtful and constructive feedback from all three reviewers.

ReasonFork is a lightweight routing method for stepwise collaboration between small and large language models. Our key contribution is learning to identify reasoning forks where teacher replacement can help the draft finish within the remaining step budget, combining offline training on paired outcomes from the same state with online routing that weighs predicted replacement value against teacher cost. Across the five proof configurations, ReasonFork achieves 95.4% average accuracy with an average teacher token reduction of approximately 72% relative to Always Teacher.

The reviewers highlighted several strengths of our work: (1) a well-defined routing objective based on the downstream consequences of teacher replacement, (2) lightweight online deployment using information available before the teacher call without searching future branches, (3) competitive accuracy on proof tasks with fewer teacher tokens, and (4) evaluation across task types, including interactive benchmarks and transfer experiments.

Below, we summarize the changes made in the revised paper:

  • Reviewers q5DV and HCh6 requested clarification of replacement value learning. We clarified its relationship to value estimation and distinguished paired supervision from harm modeling in Sections 3.2 and 4.4. Table 4 reports WebShop’s 6.01-point reward pAUC gain over repair only. Appendix C.4 gives the comparison details.

  • Reviewers jCpP and q5DV asked for more detail on supervision availability and reliability. We clarified the offline feedback sources and online inputs in Section 3, and added label error and data size experiments in Section 4.5. Appendix D.9 provides the experimental settings and results.

  • Reviewer jCpP requested a comparison with sequential allocation. We added a comparison between PPO, a myopic controller and the deployed local gate under shared supervision in Section 4.2 (Table 2(b)). PPO showed no statistically resolved pAUC gain over either comparator. Appendix C.2 provides the training protocol and resource accounting.

  • Reviewers q5DV and jCpP suggested further analysis of router features and intervention outcomes. We clarified feature construction in Section 3.3 and added actual continuation and routing-error analyses in Appendix D.10 for the 1,215 deployed RuleTaker states, including 448 following earlier teacher interventions. Appendices D.10 and D.11 provide the continuation analysis and feature ablation.

  • Reviewers jCpP and HCh6 requested a more detailed cost breakdown. We clarified the teacher token metric in Section 4.1 and expanded the total token analysis in Section 4.3 and Appendix B.6. Appendix B.6 reports model usage, recorded offline preparation costs and routing component timings.

  • Reviewers jCpP and q5DV asked us to clarify applicability beyond structured proof tasks. We clarified task requirements in Section 3.3 and Appendix B.2. Sections 4.3 and 4.5 present the interactive evaluations and frozen transfer results, with implementation and transfer details in Appendices E and F.

The revised content is highlighted in blue.

We sincerely thank all reviewers again for their valuable suggestions, which have helped strengthen our paper. We are also deeply grateful for your oversight and coordination throughout the review process. We would be happy to discuss any remaining questions and provide further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.