- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
DistilRecipe constructs reusable recipes containing applicability conditions, procedures, and checks, validated through a frozen student’s execution. A larger-model router selects a recipe for each problem, and recipe revisions must preserve success on previously supported problems. Recipes improve inference-time performance and are distilled into students that require no recipes at deployment. Experiments on GSM8K and MATH-500 show gains over matched recipe-free teacher–student distillation under SFT and OPD, although self-distillation results are mixed.
Critical strengths
- Execution-based recipe construction. Student execution, adherence checks, and replay of previously supported problems provide a concrete mechanism for maintaining reusable guidance.
- Useful distillation results. Matched teacher–student comparisons show consistent gains under both SFT and OPD without requiring recipes at deployment.
Critical weaknesses
- Recipe selection is insufficiently evaluated. Lexical retrieval may miss suitable recipes when the required procedure is not apparent from the problem’s wording. The paper does not distinguish coverage, retrieval, and routing failures or quantify their performance impact.
- The incremental contribution remains unclear. Comparisons with reference-conditioned OPSD and experiential-context OPCD are missing. Ablations of recipe fields, adherence verification, and historical replay are also absent, leaving unclear whether the proposed complexity is necessary.
Questions for authors
- How do oracle or best-of-bank recipe selection results compare with the current router, for both prompting and distillation?
- How does recipe conditioning compare with reference-solution-conditioned OPSD and experiential-context-conditioned OPCD under comparable conditions?
- What are the contributions of applicability, procedures, and checks? Do adherence verification and historical replay improve over answer-correctness filtering alone?
3 comments
Response to Reviewer WwUv (2/2)
W2 / Q3. Ablation of recipe fields: applicability, procedure and checks
The three fields act at different points of the pipeline (§3.2, App. A). Applicability states which problems a procedure fits; it is read by the router, whose profiles contain applicability and procedure but not checks (App. A.2), and by the student, who is told to use a recipe only when its applicability matches (App. B.3). The procedure is the ordered list of operations the student executes and the monitor audits (App. B.4). The checks are post-conditions applied at the end of an execution, such as reconstructing a total (App. E.3) or confirming that the answer covers the whole domain (App. E.4).
We removed each field from the routed recipe at inference and, separately, removed applicability from routing only while still showing the full recipe (Qwen 2B recipient, 2,048 tokens; change from the full recipe in parentheses):
| recipe shown to the student | MATH-500 | GSM8K |
|---|---|---|
| Full recipe | 69.00 | 77.79 |
| Without procedure | 60.40 (−8.6) | 70.62 (−7.2) |
| Without applicability | 63.60 (−5.4) | 72.56 (−5.2) |
| Applicability removed from routing only | 62.60 (−6.4) | 71.92 (−5.9) |
| Without checks | 66.80 (−2.2) | 75.87 (−1.9) |
| No recipe | 59.60 | 71.57 |
Each field matters in the way its role predicts:
- Procedure. Without it, the recipe loses its entire value: accuracy falls to within one point of using no recipe.
- Applicability. Without it, the student still has the procedure but applies it where it does not fit, and loses more than half of the gain. Removing it from routing alone costs at least as much, so it is also the signal the router relies on.
- Checks. They add a smaller but consistent gain, slightly larger on MATH-500, where the final form of the answer is part of its correctness.
W2 / Q3. Ablation of adherence verification and historical replay
We rebuilt the MATH bank with the same author, authoring problems and router, changing only the acceptance rule ( MATH-500, 2,048 tokens):
| Acceptance rule | Recipes | 0.8B | 2B | Mean gain |
|---|---|---|---|---|
| No recipe | – | 38.00 | 59.60 | – |
| No verification (author's recipes as written) | 178 | 42.20 | 64.40 | +4.5 |
| Monitor judgment only (no student execution) | 73 | 42.40 | 66.00 | +5.4 |
| Final-answer correctness only (no monitor) | 80 | 43.40 | 65.40 | +5.6 |
| Full e·α·h (ours) | 68 | 45.60 | 69.00 | +8.5 |
Adherence verification clearly improves over answer-correctness filtering: it adds 2.9 points on the mean (2.2 for the 0.8B recipient, 3.6 for the 2B), with a smaller bank. Answer-only filtering is barely better than no verification at all, and the construction logs explain why: 26.7% of the executions that reached the correct answer did not follow the recipe, so an answer-only rule accepts them as evidence for a procedure the student never used. Judging applicability without execution is no better, because a recipe can read as applicable and still be one the student cannot carry out. Only the combination identifies procedures the student can execute.
Historical replay protects the evidence the bank is built on. A revision is committed only if it still solves every problem its predecessor solved, so the support recorded for a recipe stays true after every edit (Prop. 1). Without replay, a revision that fixes its triggering problem can silently break earlier ones. The bank would keep recipes whose recorded support no longer holds. This happens often in practice: during MATH construction, 496 replays on inherited supports failed, each one a previously solved problem that a revision passing its own triggering problem would have broken.
We thank the reviewer again for the constructive feedback, which has strengthened the paper. We hope our response addresses the concerns, and we welcome any further questions.
References
[1] Zhao, S., Xie, Z., Liu, M., Huang, J., Pang, G., Chen, F., and Grover, A. Self-distilled reasoner: On-policy self-distillation for large language models. In ICML, 2026.
[2] Ye, T., Dong, L., Wu, X., Huang, S., and Wei, F. On-policy context distillation for language models. arXiv:2602.12275, 2026.
Follow-up before the discussion deadline
Dear Reviewer WwUv,
We truly appreciate the time and effort you've already dedicated to reviewing our submission. We prepared our rebuttal with great care in hopes of directly addressing the valuable points raised in your review, including an oracle analysis of recipe selection, comparisons with OPSD and OPCD, ablations of the recipe fields, and adherence verification.
If these results address your concerns, we would be grateful if you would consider raising your score. We would also welcome any further thoughts or clarifications you might wish to share.
Thank you again for your contribution to the review process.
Best regards,
The Authors
Response to Reviewer WwUv (1/2)
We thank the reviewer for recognizing execution-based recipe construction and the matched distillation gains under both SFT and OPD, without recipes at deployment. We address the concerns below.
W1 / Q1. Recipe selection: separating coverage, retrieval and routing errors
We ran every shortlisted recipe, then every recipe in the bank, on every test problem (Qwen-2B, 2,048 tokens). A recipe counts as applicable only if the student, given it, follows it to the correct answer.
The gaps between columns attribute the router's errors to three causes (points; share of errors):
Lexical retrieval therefore accounts for only 14–18% of errors, while coverage accounts for about two thirds, which is why gains grow with the bank (Figures 6–7). Routing and retrieval together leave 8.4 points on GSM8K and 10.8 on MATH-500 that are already in the bank and recoverable by a better selector without authoring new recipes.
W2 / Q2. Comparison with OPSD and OPCD
All three methods use the same on-policy objective and differ only in the teacher's context (§3.5): the reference solution (OPSD [1]), one experiential context shared by all problems (OPCD [2]), or the recipe routed from the question (ours). We use the same 9B teacher, 2B student, problems and schedule, with three seeds:
Distillrecipe gives a consistent gain: +1.8 to +4.2 points over OPSD and OPCD.