- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
ReasonFork studies stepwise routing between a small draft model and a larger teacher. At each decision state, the draft and teacher propose actions; an offline evaluator compares their successors under the same remaining horizon and labels the teacher replacement as a repair, tie, or harm. A router then predicts the expected value of replacing the draft action from pre-call features and requests the teacher when this value exceeds the estimated cost. The paper also gives a local utility rule and a reachability result for monotone, unit-cost proof systems. Experiments on RuleTaker, ProofWriter, ProverQA, ALFWorld, and WebShop show strong accuracy/token trade-offs on proof tasks, while results on interactive tasks are more mixed.
Critical strengths
- The paper distinguishes draft uncertainty from the benefit of replacing a draft action. Comparing the two successors from the same state gives the router a concrete supervision target, while online routing avoids searching over future action sequences.
- The proof-task evaluation goes beyond a single operating point, with budget curves, comparisons to stepwise and query-level routers, and an independent RuleTaker evaluation. This makes the reported proof-task gains more convincing than a fixed-budget comparison alone.
Critical weaknesses
- The evidence for signed repair/harm supervision is less clear than the evidence for learning replacement value in general. On the independent RuleTaker split, ReasonFork improves over draft-safety by 1.18 pAUC points on Qwen and 1.59 on Phi, with intervals excluding zero; its differences from repair-only are +0.26 [−0.17, 0.73] and −0.25 [−0.74, 0.21] (Table 8). The experiments therefore do not establish that explicitly modeling harm improves over a repair-only target.
- The efficiency conclusion depends on the cost metric. On ALFWorld, ReasonFork uses 11.213M total tokens and completes 29 episodes, compared with 7.511M total tokens and 35 successes for Always Teacher; total-token cost per success is 0.387M versus 0.215M (Table 18). On WebShop, ReasonFork reports lower wall time than Always Teacher (9.35 s versus 25.08 s), but higher total token use (4.436M versus 3.133M; Table 19). Teacher-token savings therefore do not imply lower total inference cost across tasks, and the cost of generating paired offline labels is not reflected in these serving-time comparisons.
Questions for authors
- What result in the submitted experiments establishes the benefit of signed repair/harm supervision over repair-only supervision? In Table 8, the intervals for ReasonFork minus repair-only include zero for both Qwen and Phi. Is there another prespecified comparison that resolves this, or should the claim be narrowed?
- Which efficiency claim is intended to be primary: fewer teacher tokens, fewer total tokens, lower wall time, or lower monetary serving cost? Please clarify the task-specific results in Tables 18–19 and whether the cost of generating paired offline labels is included or assumed to be amortized.
4 comments
Official Response to Reviewer HCh6 [2/3]
W2: Teacher allocation, total usage and timing.
Thank you for raising this practical concern. Our primary efficiency measure is teacher prompt and completion tokens, which quantifies the large-model assistance used during task execution. The approximately 72% reduction in the Abstract refers to the five proof configurations. To show how this saving relates to overall execution, we also report total model tokens and recorded episode times alongside task performance.
ALFWorld. Table 25 separates teacher usage from total model usage:
| Policy | Teacher tokens (M) | Total tokens (M) | Successful episodes | Tokens per success (M) |
|---|---|---|---|---|
| ReasonFork | 3.706 | 11.213 | 29 | 0.387 |
| Confidence | 5.443 | 14.028 | 17 | 0.825 |
| Always Teacher | 7.511 | 7.511 | 35 | 0.215 |
Under the same 25-call cap, ReasonFork completes 29 episodes versus 17 for Confidence while using fewer teacher and total tokens. Always Teacher operates without this cap and completes 35 episodes with fewer total tokens per success. ReasonFork’s total includes 7.507M draft tokens in addition to its 3.706M teacher tokens. This explains how reducing teacher usage can still result in a higher total token count when draft generation is included.
WebShop. ReasonFork uses 0.267M teacher tokens, compared with 3.133M for Always Teacher. Their total usage is 4.436M and 3.133M tokens, respectively, while recorded episode times are 9.35 and 25.08 seconds (Table 26). Exact success is 14.8% and 24.4%, so the timing results describe different operating points rather than a comparison at matched task quality.
Proof tasks. At the Table 1 operating points, the breakdown similarly shows lower teacher usage alongside 17.8%–33.0% more total unweighted tokens than the recorded Always Teacher references (Table 8). Since the draft and teacher differ in size, summing their tokens describes model usage but does not directly determine FLOPs or monetary cost.
Thus, teacher tokens measure the assistance allocated to the larger model, while total tokens and timing describe additional aspects of execution. We have made this distinction explicit in Sections 4.1–4.3 and presented each measure together with the corresponding task performance. Offline preparation is addressed separately in Q2.
Official Response to Reviewer HCh6 [3/3]
Q1: Which submitted comparison supports signed supervision?
The controlled WebShop comparison in Table 4 and Appendix C.4 supports the additional reward benefit of signed supervision. The scoring ablations on WebShop and ALFWorld further examine how the harm term affects task outcomes and teacher usage when the prediction heads remain unchanged.
-
Controlled WebShop comparison. The study compares repair-only scoring with signed scoring . Public inputs, predictor capacity, training data and the allocation grid are held fixed to focus the comparison on the supervision target. It evaluates 19,000 trajectories on 500 shared tasks over the fixed 10%–60% teacher-token range. Signed scoring improves reward pAUC by 6.01 points [4.28, 7.81], with the 95% confidence interval obtained from 10,000 paired task-bootstrap resamples. Success pAUC also meets the predefined noninferiority criterion, with a margin of one point.
-
WebShop scoring ablation. Here, the prediction heads are kept fixed, and the harm term is removed only when computing the routing score. Teacher-action share increases from 6.39% to 10.23%, while reward decreases from 46.35% to 45.77% (Appendix E.3). At these operating points, including harm achieves higher reward with a smaller share of teacher actions.
-
ALFWorld scoring ablation. We also perform a separate scoring comparison on ALFWorld, keeping public inputs, fitted heads and the scheduler fixed. Under the same 25-call cap, signed scoring completes 28 episodes versus 23 for repair-only, while using 50.61% versus 59.85% of Always Teacher tokens (Appendix E.4).
Thus, the controlled WebShop comparison and the scoring ablations together support the practical value of accounting for harm. In the decision-making process, considering both the possibility of improving the outcome and the risk of worsening it helps the router make more effective use of teacher assistance.
Q2: Primary efficiency claim and offline preparation.
Our primary efficiency claim concerns reduced teacher-token usage during serving. The serving results exclude offline preparation, which is reported separately. For a fixed training bank, this preparation is performed once, and the resulting router can be reused for subsequent requests without repeating the paired evaluations.
The recorded prompt and completion tokens used to prepare supervision are:
| Setting | Recorded LM tokens | Included preparation |
|---|---|---|
| WebShop | 32,064,703 | 600 training and 120 development tasks, including paired proposals and draft continuations |
| ALFWorld | 23,343,888 | All 200 collected episodes, including discarded episodes |
The WebShop total consists of 27,240,031 draft tokens and 4,824,672 teacher tokens. The 164 retained ALFWorld episodes account for 19,104,124 tokens, which are already included in the full collection total. Each training bank is counted once.
For a fixed bank reused across deployment requests, the recorded preparation tokens contribute per request, in addition to the serving tokens. The reported serving results do not include this preparation term or assume a particular deployment volume. As the same bank is reused for more requests, its preparation cost is spread over those requests.
We have reported the preparation token counts and their coverage in Appendix B.6, separately from serving usage. This allows readers to assess the model usage required to prepare and use the router for their intended deployment volume.
Thank you again for your thoughtful review and insightful suggestions. We sincerely appreciate the time you have devoted to reviewing our work. We hope that our responses have satisfactorily addressed the remaining concerns.
Thank you for the clarifications. My concerns have been addressed. And I will maintain my rating (3: Weak acceptance).
Official Response to Reviewer HCh6 [1/3]
Thank you for your detailed feedback and valuable suggestions. We appreciate your positive assessment of our routing formulation and the thorough evaluation on proof tasks. Below, we respond to your comments and questions point by point.
Thank you for this valuable comment. On WebShop, signed scoring improves reward pAUC over repair-only by 6.01 points [4.28, 7.81] (Table 4 and Appendix C.4). This comparison holds routing inputs, predictor capacity, training data and the allocation grid fixed. On the independent RuleTaker split, the confidence intervals for this comparison include zero. To help interpret this pattern, we examined whether the harm term changes which interventions are selected.
For this post hoc diagnostic, we used the original out-of-fold (OOF) predictions from five folds and kept the prediction heads fixed. Only the routing score changes. Both scores select from the same bank of paired states, using feasibility labels on RuleTaker and recorded continuation rewards on WebShop. The table below compares selections at a fixed 30% quota, matching the number of selected states rather than teacher token costs. Appendix C.3 reports this selection diagnostic, and Appendix C.4 gives the controlled comparison under teacher token budgets.
On RuleTaker, the selected sets overlap by more than 94%. Neither score selects any states labeled HARM by the feasibility proxy at this quota, and the REPAIR count changes by only one in each model family. The harm term therefore has little effect on selection in this diagnostic.
On WebShop, signed scoring selects 64 more REPAIR states and 14 fewer HARM states at the same quota. The diagnostic suggests that the harm term adds value by refining intervention selection where repair probability alone leaves harmful replacements among the preferred candidates. This effect is more evident in the WebShop selections and is consistent with the reward-pAUC improvement reported above. We describe the controlled comparison and additional scoring ablations in Q1 below.