acceptodds
Under review as a conference paper at ICLR 2027

Instruction Retrieval at Inference Time for Small Language Models

Abstract

The facts a language model stores are tied to its parameter count, so small models that fit on edge devices fail on expert problems, which need specialized knowledge and follow multi-step procedures. Fine-tuning for a specific domain or task writes the knowledge into the parameters but must be repeated for every model and domain, and a retrieved passage leaves the model to find the relevant fact and apply it on its own. We introduce instruction retrieval, which distills a teacher model's expertise into a corpus of instructions tailored so that a small model can follow. For each cluster of a domain's problems, the teacher writes one instruction with the background knowledge the cluster depends on, a procedure for that kind of problem, and the common mistakes made on it. This reusable corpus needs only to be built once per domain and requires no run-time teacher access. At inference, a frozen small model retrieves the instructions nearest its question and follows them, with no fine-tuning. Across medicine, law, and mathematics benchmarks, we demonstrate the corpus improves over zero-shot on every task and over few-shot prompting, self-consistency, and other retrieved text on medicine and law. On MedQA the corpus raises mean accuracy by 10.6 points, where retrieved textbook passages raise it by 3.9 and few-shot examples from the same teacher lower it. An error analysis shows that only the background knowledge fixes the questions a small model always gets wrong, and that the procedure and common mistakes fix only the questions where it wavers between options. Our results show that automatically retrieved inference-time procedural guidance and domain knowledge can yield substantial gains for small models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
small language models, inference-time intervention, instruction retrieval, retrieval-augmented reasoning, model efficiency, structured reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Vmk3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a methods for SLMs to store instructions to retrieve facts that do not fit in memory. A kind of run time compression. Distillation into corpus, not into weights. After clustering. So text, not embeddings.

Critical strengths

a new method for making SLM is quite useful

Critical weaknesses

Idea of generating a domain specific corpus is not new. The analysis may be interesting. Results are better on some domains, not all. This can be due to benchmark quality.

GPT-5 wrote the instruction corpus. No study of the impact of other models.

No supplementary material provided. Abstract does not contain measurements.

Questions for authors

Why would is be better to create a corpus (text) than weights (embeddings)? Store on disk, not memory? Textual swapping? RAG? Please explain more clearly.

Interesting idea, not ground breaking. Better suited at specialized conference? weak reject.

3 comments

We thank the reviewer for finding the method "quite useful" for small language models. We will provide an updated paper with the clarifications and results requested.

“Idea of generating a domain specific corpus is not new. The analysis may be interesting.”

The new idea is how a teacher model builds a corpus that a small model can follow, i.e., the methodology. The analysis shows why a teacher-written corpus outperforms an existing domain corpus. GPT-5, as judge, marks the retrieved textbook passages as pointing toward a wrong option on 22% of MedQA questions, against 9% for the instructions, and where the text lacks the deciding fact, the passages turn a third of the questions a small model had right into wrong answers, against a fifth for the instructions (Section 6).

“Results are better on some domains, not all. This can be due to benchmark quality."

Instruction retrieval improves mean accuracy over zero-shot on all three benchmarks (10.6 points on MedQA, 3.3 on MMLU Law, 2.4 on MathQA), and no small model loses accuracy on any benchmark. Measured against the room left to gain, the corpus closes 35% of the gap to GPT-5 on MedQA and 41% on MathQA, where two students with the corpus match GPT-5 (Section 5). The size of the gain differs because each benchmark tests a different skill (Section 6). MedQA tests recall of medical facts, which the instructions supply, while MathQA tests arithmetic procedures the small models already perform well on, and every student gets only about 6% of MathQA questions wrong on every attempt (Section 6).

“GPT-5 wrote the instruction corpus. No study of the impact of other models.”

GPT-5’s strength alone does not produce the gain. GPT-5 also writes the few-shot examples and the RICP principles, and both fall below instruction retrieval on all three tasks (Table 1). To address this further, we rebuilt the corpus for each benchmark with gpt-oss-120b, an open-weight teacher, keeping the clusters, teacher prompt, student errors and retrieval fixed. On MedQA, the gpt-oss-120b corpus raises the mean accuracy of the five students by 7.1 points, against 10.6 with the GPT-5 corpus, and every student’s interval is above zero.

Mean gain over zero-shotGPT-5 corpusgpt-oss-120b corpus
MedQA+10.6+7.1
MMLU Law+3.3+1.4
MathQA+2.4+2.0

Similar to distillation and other methods, a stronger teacher yields better results, but the method does not depend on any one teacher. Please let us know if there are any other experiments that would help clarify this. The full model level breakdown will be in the revised appendix.

“No supplementary material provided.”

As stated in the paper the code and instruction corpora will be released on publication. All prompt, settings and additional results are listed in the appendix.

“Abstract does not contain measurements.”

The submitted abstract already reports the headline result, "On MedQA the corpus raises mean accuracy by 10.6 points, where retrieved textbook passages raise it by 3.9 and few-shot examples from the same teacher lower it", and Table 1 gives the few-shot drop (1.6 points). Please let me know if there are specific changes you want us to include.

“Why would it be better to create a corpus (text) than weights (embeddings)?”

Section 2 explains why the corpus is stored as text rather than in the small model's weights. Fine-tuning specializes the student, spending its limited capacity on one target ability at a measured cost in generality (Fu et al., 2023), and must be repeated for every new student and domain, whereas instruction retrieval keeps the student frozen and general. A small model also has little room for new facts, because the facts a model stores are tied to its parameter count (Section 1). The same corpus serves any small model without training, and Ministral 3 3B, whose errors were never used to build the corpus, gains 10.6 points on MedQA (Table 1). Embeddings are used only to find the nearest instructions, and what the student reads is text. The corpus is stored on disk (2.1M tokens for MedQA), and only the five retrieved instructions enter the prompt. Unlike retrieval augmentation, which retrieves general passages, instruction retrieval retrieves text written for a small model to follow, and retrieved textbook passages raise MedQA accuracy by only 3.9 points.

"...Better suited at specialized conference?"

The question the paper asks, what text a frozen small model needs at inference to solve problems it fails, is general to language models, and the paper answers it on medicine, law and mathematics with five students from three families. Related work the paper cites appears at ICLR, including Self-RAG (ICLR 2024) and Retrieval-of-Thought (ICLR 2026), and related methods such as ReasoningBank and ACE appeared at ICLR 2026.

We hope the new teacher experiment and these clarifications address your concerns and welcome further questions during the discussion.

Reviewer_Vmk3OpenReview ↗

Some concerns addressed

Thank you for the reply, and for the extra experiments. The ideas of the paper remain interesting, and follow up on related work such as Self-RAG and Retrieval-of-Thought. I remain unconvinced about how substantive the idea is, as another reviewer phrased it as limited conceptual novelty. The concern on the benchmark side was phrased as narrow evaluation surface. In summary, nice result, but unconvinced that this is the right venue.

My score reflects this.

Thank you for the follow-up and for engaging with the new results. We would like to understand your remaining concerns well enough to address them, so we respond to each and ask one question about each.

On the evaluation surface: the method is tested on five students from three model families, on three benchmarks that test different skills (medical knowledge, legal reasoning and arithmetic). Since the first response, we have added a second teacher (the open-weight gpt-oss-120b), a second, locally run encoder, and paired tests against the strongest baseline on every task. We are now running an open-ended version of MathQA and a transfer test of the MedQA corpus on MedMCQA, and we will report both in this discussion. If there is a specific task, model size or setting that would change your view, please tell us, and we will run it if we can before the discussion closes.

On conceptual novelty: we read the question as whether the gain comes from anything beyond retrieving teacher text, which existing methods already do. We tested this directly in our reply to another reviewer. When both methods retrieve the same training questions, written up by the same teacher, instructions beat the teacher's worked solutions on MedQA by 3.2 points, and RICP's teacher-written insights do not help at all. What the teacher writes, and for whom, therefore matters and we provide a new method for doing this. Methods such as Self-RAG and Retrieval-of-Thought help a model that can already solve the problem. Our setting is a frozen small model that fails the problem and cannot write the guidance itself. If there is prior work you see as already doing this, or an experiment that would separate our method from it, we would welcome the pointer.

Reviewer_fZ2eOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper proposes instruction retrieval for frozen small language models. A benchmark's training questions are clustered (about three per cluster), and GPT-5 writes one instruction per cluster with background knowledge (K), a procedure (P), and common mistakes (M, derived from student errors on the training set). At inference, the five instructions whose source questions are nearest to the test question are prepended to the prompt. On MedQA, MMLU Professional Law, and MathQA with five 2–4B students, mean accuracy over zero-shot improves by 10.6, 3.3, and 2.4 points, exceeding fixed few-shot, RICP, and, on MedQA, textbook passage retrieval (+3.9). An error analysis and a K/P/M ablation suggest that stated knowledge fixes questions the model always gets wrong, while procedures and mistakes help where it is inconsistent.

Critical strengths
  1. Simple, training-free, and consistent on MedQA. The gain holds for all five students with intervals well above zero (Table 4), including Ministral 3 3B (+10.6), which did not contribute to corpus construction.
  2. Careful controls and analysis. The random-instruction control, decontamination, question-withheld check, K/P/M ablation, and full per-student results make the findings easy to scrutinize.
Critical weaknesses
  1. The source of the gain is not isolated. Compared with textbook passages, the instructions differ in teacher strength (GPT-5 scores 94.7% on MedQA), benchmark-specific tailoring (they are written from the same benchmark's training questions and answers), and format (K/P/M). The key missing control is retrieved few-shot: since retrieval already matches the test question to training questions, the natural baseline is the five nearest training questions with gold answers and GPT-5 rationales. The fixed few-shot baseline used in the paper is weak by comparison. Table 2 reinforces the concern: cluster size 1 (one instruction per training question) is at least as good as the chosen size 3 in every row, and size 27 is much worse. This suggests the gain comes mainly from near-neighbor training items rather than from type-level guidance, which is the paper's central framing.
  2. The reusability and on-device claims are not supported. Each corpus is built and tested on the same benchmark, with no cross-dataset transfer to back the "once per domain" claim. Query embedding uses OpenAI's text-embedding-3-large at inference, which conflicts with the edge-device motivation. M is derived from four of the five evaluated students, and the construction and added-context costs are not reported.
Questions for authors
  1. Could you report a retrieved few-shot baseline (five nearest training questions with gold answers and GPT-5 rationales, same encoder and k)? If instruction retrieval remains clearly ahead, I would raise my score.
  2. Could you add an unstructured GPT-5 summary per cluster, to separate the effect of the teacher and tailoring from the K/P/M format?
  3. Does a corpus built on one benchmark transfer to another in the same domain (e.g., MedQA corpus on MedMCQA)? How does the gain vary with test-to-train question similarity?
  4. How do results change with a small local encoder instead of text-embedding-3-large?
  5. Clarification: could you report paired tests against the strongest baseline, especially on MMLU Law, and explain why RICP falls below zero-shot on MedQA and MMLU Law?
5 comments

Thank you for your thoughtful engagement with the paper!

“The source of the gain is not isolated. Compared with textbook passages, the instructions differ in teacher strength (GPT-5 scores 94.7% on MedQA), benchmark-specific tailoring (they are written from the same benchmark's training questions and answers), and format (K/P/M). The key missing control is retrieved few-shot: since retrieval already matches the test question to training questions, the natural baseline is the five nearest training questions with gold answers and GPT-5 rationales. The fixed few-shot baseline used in the paper is weak by comparison. Table 2 reinforces the concern: cluster size 1 (one instruction per training question) is at least as good as the chosen size 3 in every row, and size 27 is much worse. This suggests the gain comes mainly from near-neighbor training items rather than from type-level guidance, which is the paper's central framing.”

We ran the retrieved few-shot baseline on MedQA, along with three follow-up comparisons, and will add all of them to the revised paper which we upload shortly. For each test question, the baseline retrieves the five nearest training questions with their correct answers and GPT-5 worked solutions, using the paper's encoder. Because it is tailored to each test question, it is a much stronger baseline than fixed few-shot.

First, to separate what the teacher writes from which training questions are retrieved, we wrote one instruction per training question. Both methods then retrieve the same number of examples (k=5), written up by the same teacher, and differ only in the text.

Gain over zero-shotQwen 3.5 2BQwen 3.5 4BGemma 4 E2BGemma 4 E4BMinistral 3 3BMean [95% CI]
Worked solutions (10,178 texts)+13.4+3.3+10.0+7.1+10.4+8.8
Instructions, 1 training question each (10,178)+18.1+5.3+14.1+9.0+13.3+12.0
Lead of instructions+4.8+2.0+4.2+1.9+2.9+3.2 [+1.8, +4.5]

Instructions lead by 3.2 points on average and for every student. Second, we compared the paper's exact setting, 3 training questions average cluster size, with the same tailored retrieval baseline.

Gain over zero-shotQwen 3.5 2BQwen 3.5 4BGemma 4 E2BGemma 4 E4BMinistral 3 3BMean [95% CI]
Instructions, 3 training questions each (3,393)+15.9+5.3+12.2+9.1+10.6+10.6
Lead over worked solutions+2.5+2.0+2.2+2.0+0.2+1.8 [+0.5, +3.0]

Instructions are still ahead for every student, by 1.8 points on average, using a third as many documents. Third, we tested broader instructions, written from eight training questions each. The corpus shrinks to 1,272 instructions, 12.5% of the 10,178 worked solutions. These instructions gain 8.3 points on average, within the margin of error of the worked solutions of the exact nearest training questions (−0.5 [−1.8, +0.8]).

Finally, we split the test questions into quarters by their similarity to the nearest training question and compared the paper's setting with the worked solutions in each quarter.

Gain over zero-shot, mean of five studentsWorked solutionsInstructions (paper)Lead
Least similar quarter+4.0+5.8+1.8
Second quarter+6.5+9.4+2.9
Third quarter+11.3+13.4+2.1
Most similar quarter+13.5+13.8+0.3

Both gain more when a close training question exists, as any retrieval method should. Instructions stay ahead in every quarter. The two are close only in the most similar quarter, where a worked solution to a near-identical question is available. The gain also does not come from copying a neighbor's answer. Only 11% of test questions share their correct answer with their nearest training question, and on the rest, one instruction per training question still leads worked solutions by 3.0 points.

These results show that retrieving the right documents matters, and that writing about them as instructions rather than worked solutions adds a consistent gain and lets a much smaller corpus match the worked solutions which separates the content of the instructions from the teachers strength along. We will add the retrieved baseline and the full cluster-size sweep to the main results because as you said they are the stronger baseline to compare against. We will also make sure to clarify our description of an instruction as guidance for a small group of similar training questions rather than a broad problem type. We hope this addresses the core concern, and we are glad to run anything else that would help.

"Each corpus is built and tested on the same benchmark, with no cross-dataset transfer to back the 'once per domain' claim." / "Does a corpus built on one benchmark transfer to another in the same domain?"

We agree that "domain" is too broad and will narrow it. We meant the area of problems a corpus is built from, which here is a benchmark's training split, not all of medicine or mathematics. Within that scope, we tested whether a corpus built once can be reused by students it was not built for. One corpus serves five students from three model families with no training. Ministral 3 3B, whose errors were not used to build the corpus, gains 10.6 points on MedQA, the same as the five-student mean. A corpus therefore carries over to new students, and adding a student only requires retrieval. Transfer to a different benchmark is the open question, and we will not claim it. However, we are now running the MedQA corpus on MedMCQA and will report the results in this thread once complete.

"Query embedding uses OpenAI's text-embedding-3-large at inference, which conflicts with the edge-device motivation." / "How do results change with a small local encoder?"

We agree that an API encoder conflicts with the on-device motivation. We reran retrieval with Qwen3-Embedding-4B, an open-weight model that runs locally, and the revised paper uses it.

Mean gain over zero-shottext-embedding-3-largeQwen3-Embedding-4B
MedQA+10.6+11.0
MMLU Law+3.3+3.6
MathQA+2.4+2.5

The local encoder performs as well as text-embedding-3-large on all three tasks. The method therefore does not depend on a proprietary encoder. Because the corpus is built and clustered once, offline, only the incoming question needs to be embedded on the device.

"M is derived from four of the five evaluated students"

Ministral 3 3B, whose errors were not used, gains 10.6 points on MedQA, the same as the five-student mean. The corpus is therefore not tailored to the students that built it. The teacher also already writes common mistakes for clusters where no student erred. To remove student errors entirely, we are now building a corpus whose common mistakes are written without them.

"the construction and added-context costs are not reported"

We agree these costs belong in the paper. Building a corpus takes one teacher call per cluster, once. With GPT-5 at the batch price, the corpus cost 50 US dollars on MedQA, 5 on MMLU Law and 114 on MathQA. A cheaper open-weight teacher also works: a corpus written by gpt-oss-120b still improves every student on MedQA (+7.1 points, about two thirds of GPT-5's gain) and comes close to GPT-5 on MathQA (+2.0 against +2.4). At inference, five instructions add about 1,400 words to each question and about 0.1 seconds of student time on MedQA (0.4 to 0.5 seconds). Running the student costs about 0.01 to 0.02 cents per question, against 0.14 to 0.23 cents for GPT-5 answering directly.

"Could you report paired tests against the strongest baseline, especially on MMLU Law?"

We ran paired tests against the strongest baseline on each task. Below are paired differences in points, with 95% bootstrap intervals over test questions (10,000 resamples, shared across the five students).

TaskComparisonQwen 3.5 2BQwen 3.5 4BGemma 4 E2BGemma 4 E4BMinistral 3 3BMean of five
MedQAInstructions − passages+9.1 [+6.0, +12.2]+4.7 [+2.4, +7.0]+7.4 [+4.6, +10.1]+6.7 [+4.3, +9.0]+5.8 [+3.1, +8.6]+6.7 [+5.1, +8.3]
MMLU LawInstructions − self-consistency+0.7 [−4.3, +5.7]+2.2 [−2.4, +6.9]+4.6 [+0.0, +9.3]+3.0 [−1.3, +7.2]+1.7 [−3.1, +6.5]+2.4 [+0.0, +4.8]
MMLU LawInstructions − zero-shot+0.0 [−5.2, +5.0]+5.9 [+1.3, +10.4]+6.5 [+2.0, +10.9]+1.7 [−2.8, +6.1]+2.6 [−2.2, +7.4]+3.3 [+0.9, +5.8]
MathQAInstructions − self-consistency−3.4 [−4.8, −1.9]−0.1 [−0.9, +0.7]+0.9 [−0.1, +1.9]+0.3 [−0.5, +1.1]−2.2 [−3.3, −0.9]−0.9 [−1.4, −0.3]
MedQAInstructions + self-consistency − self-consistency+13.5 [+10.4, +16.5]+4.8 [+2.6, +7.0]+12.3 [+9.6, +15.0]+6.8 [+4.5, +9.0]+11.4 [+8.6, +14.2]+9.7 [+8.2, +11.3]
MMLU LawInstructions + self-consistency − self-consistency+5.9 [+1.1, +10.7]+3.0 [−1.3, +7.0]+3.3 [−1.1, +7.8]+5.0 [+0.7, +9.4]+1.7 [−3.0, +6.5]+3.8 [+1.5, +6.1]
MathQAInstructions + self-consistency − self-consistency+2.6 [+1.3, +3.9]+0.8 [+0.0, +1.5]+2.4 [+1.5, +3.4]+1.4 [+0.7, +2.2]+1.5 [+0.4, +2.5]+1.7 [+1.2, +2.2]

Instructions beat the strongest baseline on MedQA and match or beat it on MMLU Law. On MedQA they lead textbook passages by 6.7 points, and every student's interval is above zero. On MMLU Law the strongest baseline is self-consistency, which votes over three answers. With a single answer, instructions lead it by 2.4 points and every student is ahead, though with only 540 test questions the interval on the mean starts at zero. Given the same three answers, instructions clearly beat self-consistency on Law (+3.8 [+1.5, +6.1]), and against zero-shot the Law gain of 3.3 points is clearly above zero.

On MathQA, self-consistency is ahead by 0.9 points. Much of that gap comes from answers cut off at the length limit, which account for 381 of the 390 unreadable answers. Voting over three samples recovers most of them.

When both methods use the same three samples, instructions plus self-consistency beat self-consistency alone on every task and for every student. On MedQA, instructions beat every baseline with a single answer. On every task, they add to self-consistency rather than replace it.

"explain why RICP falls below zero-shot on MedQA and MMLU Law"

RICP's drop below zero-shot is small and mostly within noise. The mean falls 0.4 points on MedQA and 1.1 on MMLU Law, and only Ministral 3 3B on MedQA has an interval below zero. RICP's text is general advice about why a student erred, with the same ten principles above every question, rather than the knowledge a question relies on. Text not matched to the question tends to lower these students' accuracy slightly, as five random instructions do (2.0 points on both tasks). This is the gap instructions are written to fill: they state the knowledge a group of similar questions turns on.

Reviewer_8LXZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes instruction retrieval: a teacher model (GPT-5) clusters the training questions of a domain, writes one instruction per cluster containing background knowledge (K), application steps (P), and discriminating features/common mistakes (M), and a frozen small model retrieves the five nearest instructions at inference. Across MedQA, MMLU Law, and MathQA with five small students from three families, the corpus improves over zero-shot on every task (+10.6 on MedQA) and beats few-shot, self-consistency, RICP, and textbook-passage retrieval on medicine and law.

Critical strengths
  1. Clean, well-controlled experimental design. Random-instruction and few-shot-from-same-teacher controls isolate the gain to retrieving the right instruction rather than "any extra text," and the with-/without-question sanity check rules out direct answer leakage.
  2. Useful diagnostic decomposition. The K/P/M ablation mapped against the always-wrong vs. rarely-right error taxonomy is the most actionable finding — K rescues factual failures, M rescues discriminative ones — and the GPT-5 judge analysis (9% vs. 22% misleading rate) gives a concrete mechanism for why instructions beat passages.
Critical weaknesses
  1. Limited conceptual novelty. The core move — distill a teacher's rationale into a retrievable corpus and prepend it at inference — is a natural combination of RAG, self-distillation, and prior "insight library" baselines (RICP, ToTAL, Hypotheses-to-Theories), and the paper's main contribution is an empirical comparison rather than a new mechanism.
  2. Narrow evaluation surface. All three benchmarks are multiple-choice, the teacher is a single proprietary model (GPT-5), and the retriever is an off-the-shelf embedder with fixed k=5, so it is unclear whether the gain survives with open-weight teachers, learned rerankers, or open-ended generation tasks.
Questions for authors

Does the K/P/M decomposition replicate on open-ended or generative tasks where there are no answer options to discriminate, and does the gain shrink as the student gets larger (e.g., 7B–14B) rather than staying frozen at 2–4B?

4 comments

Thank you for recognizing the controlled design and the error analysis. In response, we add an open-weight teacher, a second embedder, and an open-ended version of MathQA.

"The core move — distill a teacher's rationale into a retrievable corpus"

To clarify, the corpus contains instructions rather than rationales. Each instruction is written for q type of problem and states the background knowledge it needs, the steps to solve it, and the mistakes students commonly make. Teacher-written worked solutions i.e. rationales are a baseline in the paper. Given as five fixed few-shot examples, GPT-5's worked solutions slightly lower MedQA accuracy, while instruction retrieval raises it by 10.6 points. The student is also never trained, so the teacher's knowledge lives in the corpus and not in the student's weights.

"a natural combination of RAG, self-distillation, and prior 'insight library' baselines"

Two of the closest prior methods are baselines in the paper, and neither reproduces the gain. On MedQA, retrieved textbook passages raise accuracy by 3.9 points and RICP's retrieved insights leave it unchanged, against 10.6 for instruction retrieval. Buffer of Thoughts and ToTAL build their libraries from a large model's own reasoning. These methods make a model that can already solve the problem more efficient. Our small students fail these problems and cannot write the guidance themselves. The contribution is a method for a teacher to write guidance for a frozen small model that lacks the knowledge, together with an analysis of which part of that guidance fixes which kind of error.

“an empirical comparison rather than a new mechanism"

The paper contributes a method for building instruction corpora, defined formally in Section 3, and an explanation of why the method works (Section 6). We separate the questions a student sometimes gets right from the questions it gets wrong on every attempt. More sampling or a thinking mode rarely fixes the second kind, because the student lacks the needed fact. The background knowledge in the instructions fixes about a fifth of them on MedQA. The gain therefore comes from what the instructions contain, since RICP also has a teacher write and a student retrieve and does not reproduce it.

"the teacher is a single proprietary model (GPT-5)"

We rebuilt each corpus with gpt-oss-120b, keeping the clusters, the teacher prompt, the student errors, and the retrieval fixed. The open-weight corpus recovers about two thirds of the MedQA gain, with every student's interval above zero, and raises the mean on MMLU Law and MathQA by smaller amounts.

Mean gain over zero-shotGPT-5 corpusgpt-oss-120b corpus
MedQA+10.6+7.1
MMLU Law+3.3+1.4
MathQA+2.4+2.0

Therefore, the gain survives an open-weight teacher, though it is smaller as a stronger teacher writes more useful instructions. But the method does not require a proprietary model. The revised paper reports these results per student with intervals.

"an off-the-shelf embedder with fixed k=5"

The number of retrieved instructions is tuned on the development set. To check the embedder, we repeated retrieval with Qwen3-Embedding-4B, a small open-weight model.

Mean gain over zero-shottext-embedding-3-largeQwen3-Embedding-4B
Instructions, MedQA+10.6+11.0
Instructions, MMLU Law+3.3+3.6
Instructions, MathQA+2.4+2.5

The instruction gains are nearly identical across the two embedders. Instruction gains therefore do not depend on the embedder, while passage gains do. The revised paper reports both embedders. We kept the retriever fixed across instructions and passages so the comparison isolates the retrieved text, and a reranker would apply to both.

"All three benchmarks are multiple-choice"

We are running MathQA as an open-ended task and will report back shortly with the results.

"does the gain shrink as the student gets larger (e.g., 7B–14B)"

The gain depends on how many questions remain for the instructions to recover, mainly the questions a student gets wrong on every attempt. That number tends to fall as students grow, but size does not set it. Across our students, the share of questions answered wrong on every attempt depends more on the task than on model size. On MedQA, where larger students do miss fewer questions, the larger student in each family gains less. A 7B–14B student would therefore gain less on tasks it already answers well, and would still gain where many questions lie beyond its knowledge.

We hope the new teacher experiment, small embedder, and these clarifications address your concerns, and welcome any further questions during the discussion.

"Does the K/P/M decomposition replicate on open-ended or generative tasks where there are no answer options to discriminate?"

We ran MathQA with the options removed and scored the student's final number by exact match. The gain holds without answer options.

SettingMultiple choiceOpen ended
Zero-shot accuracy84.472.6
Accuracy with instructions86.676.6
Gain from instructions+2.2 [+1.6, +2.8]+4.0 [+3.3, +4.6]

Without the options, zero-shot accuracy falls from 84.4 to 72.6, and the gain from instructions grows from 2.2 to 4.0 points, with every student's interval above zero. The gain therefore does not depend on the multiple-choice format. It is the same under a looser 1% tolerance on the number (+4.0 [+3.3, +4.7]). The intervals are paired 95% bootstrap intervals over test questions.

We also tested the decomposition on open-ended MathQA by giving the student each part of the instruction alone. The decomposition holds.

Open-ended MathQA, mean of fiveGain over zero-shot
Full instruction+4.0 [+3.3, +4.6]
Background knowledge alone+2.8 [+2.1, +3.4]
Procedure alone+1.9 [+1.3, +2.6]
Common mistakes alone+1.8 [+1.2, +2.5]

Each part helps on its own, and the full instruction gives the largest gain for every student.

We hope this addresses your core concerns. On top of the original 5 students from 3 model families and 3 benchmarks, we have added an open-weight teacher, a locally run embedder, and an open-ended task. Please let us know if there are any other concerns we may address.

I thank the authors for the thorough and honest rebuttal. The new experiments — the open-weight teacher, the second embedder, and the open-ended MathQA — directly address my concerns, and the paper's logic is internally consistent throughout. The K/P/M decomposition holding on open-ended inputs, and the finding that instruction gains are embedder-independent while passage gains are not, are genuinely useful contributions.

However, my score is unchanged. My remaining concern is not soundness but significance. The contribution is an empirical refinement of a known recipe (retrieve teacher-written text and prepend it), and its value is demonstrated only on three multiple-choice-style benchmarks (MedQA, MMLU Law, MathQA) with small frozen students. These are exactly the settings where a missing fact is the bottleneck and a prepended fact helps most. Whether the gain transfers to open-ended real-world tasks, larger production models, or domains where the bottleneck is reasoning or planning rather than a missing fact remains untested. A +10.6 gain on MedQA is encouraging, but I am not convinced it predicts strong performance outside this narrow, fact-deficient regime. The paper is solid and carefully controlled, but its scope and novelty do not, in my judgment, clear the bar. My score is unchanged.

We are glad the new experiments settle soundness. We answer the significance concern directly, because the paper's evidence speaks to each part of it.

"These are exactly the settings where a missing fact is the bottleneck"

The paper measures the bottleneck on each task, and on MathQA it is not a missing fact. We separate the questions a student gets wrong on every attempt, where it lacks something it needs, from the questions it gets wrong only sometimes, where it has what it needs and misapplies it. On MathQA about 6% of questions are wrong on every attempt for every student, and instructions still raise accuracy. We also show that they raise it even in the open-ended setting, where the answer options are removed. Similarly, on MMLU Law, the common mistakes fix more of the sometimes-wrong questions than the background knowledge does, because a student that knows the rule still picks the option that differs from it in one condition, which is the finding your review called useful.

"an empirical refinement of a known recipe (retrieve teacher-written text and prepend it)"

That recipe does not by itself produce the result, because it also describes RICP, and RICP leaves MedQA accuracy unchanged for these students. GPT-5 writes both the RICP insights and the instructions for the same students, both are retrieved for each question, and on MedQA, of the two, only the instructions help, by 10.6 points. The paper contributes a methodology for this particular setting, and we differentiate it from the prior work both in the paper itself and in our responses here. Moreover, the published papers that we cite (Self-RAG, KARD) and that your review names (RICP) would also fail this definition of novelty, since they combine known parts for a particular setting and improve on them, as is common in research. For example, Self-RAG (ICLR 2024) adds learned self-critique and on-demand retrieval to retrieval augmentation and KARD (NeurIPS 2023) adds retrieval to reasoning distillation. Each was a contribution for what it showed in its setting, as is the case here.

"its value is demonstrated only on three multiple-choice-style benchmarks"

The methodology is not specific to multiple-choice questions. We selected three different areas that are commonly used in similar literature, for a clear comparison. Moreover, the result holds without answer options, which is what your review asked us to test. We removed the options from MathQA and the gain grew from 2.2 to 4.0 points, with each part of the instruction still helping on its own. The evaluation now covers medicine, law and mathematics, five students from three model families, two teachers, and, for MathQA, both answer formats. Held-out benchmarks are also how the closest prior work establishes its methods. RICP, which you cite, reports seven reasoning benchmarks with two API models, and KARD reports three benchmarks, all multiple choice or yes/no, with fine-tuned students.

"larger production models"

Small frozen models are the setting the paper is written for. A model of a few billion parameters runs on a laptop or a phone, and instruction retrieval keeps it frozen and general instead of fine-tuning it for every domain. The corpus is built once, after which a small student answers a MedQA question for about a tenth of what GPT-5 costs to answer it directly. On MedQA the paper shows where the gain goes for a larger model, since the larger student in each family gets fewer questions wrong on every attempt and gains less.

"not convinced it predicts strong performance outside this narrow, fact-deficient regime"

Lacking the fact is the ordinary condition of a small model on an expert problem, so we do not see the regime as narrow. Our students answer up to a quarter of MedQA questions and up to a third of MMLU Law questions wrong on every attempt, and more sampling or a thinking mode fixes few of them. The paper also does not use the MedQA gain to predict other domains. We report each task separately and explain the differences, with MedQA gaining most because a stated fact is what a board-exam question needs and law and mathematics gaining from instructions that also correct misapplied rules and procedures. The error analysis gives a practitioner the choice of remedy, since a question a student gets wrong on every attempt needs stated knowledge, and a question it gets wrong only sometimes responds to the procedure, the common mistakes, or a second sample.

Your review asked for an open-weight teacher, a second embedder and a task without answer options, and your comment says each is addressed. We would like to understand what further evaluation you have in mind for a method aimed at small frozen models. Is it a particular open-ended task, a particular student size, or something else? If it can be run on these students before the discussion closes, we will run it and report the results.

Comments on the paper

1 comment

Dear Area Chair,

Thank you for your time in handling our submission.

We write to summarize how we addressed the main concerns raised in the reviews. Our paper shows how to distill a large teacher model's expertise into a corpus of instructions that a frozen small language model retrieves at inference, so that a small model can answer expert questions with no fine-tuning. Reviewers primarily raised two concerns, novelty and the breadth of the evaluation.

Novelty

Reviewer 8LXZ describes the method as "an empirical refinement of a known recipe (retrieve teacher-written text and prepend it)". Prior methods do have a teacher write text for a student, and the paper cites them and tests the closest ones as baselines. RICP and few-shot worked solutions written by the same teacher both gain less than the instructions on all three tasks. Our contribution is a methodology for distilling a teacher's knowledge into a corpus suited to small models, together with an error analysis of why the corpus works, which 8LXZ calls "the most actionable finding".

The strongest test of what the teacher writes is the baseline Reviewer fZ2e asked for, the teacher's worked solutions to the five training questions nearest the test question. If the content of the instructions matters, instructions should beat worked solutions written by the same teacher about the same questions, and on MedQA they do. When the two corpora are the same size, instructions lead by 3.2 points and every student is ahead. In the paper's setting the worked-solution corpus is three times larger, and the instructions still lead, while an instruction corpus one eighth the size of the baseline matches it. The instructions also stay ahead whether or not a test question has a close neighbor among the training questions, so the lead comes from the content of the instructions rather than from a near-identical training question. fZ2e writes that this control "resolve[s] my soundness concern".

Breadth of the evaluation

8LXZ's review notes that "All three benchmarks are multiple-choice, the teacher is a single proprietary model (GPT-5), and the retriever is an off-the-shelf embedder". The evaluation now covers 5 students from 3 model families on medicine, law and mathematics, with a second, open-weight teacher and a second, locally run embedder. Nothing in the method requires answer options at inference, and we tested this directly during the rebuttal period. With the options removed from MathQA, the gain grows from 2.2 to 4.0 points. The locally run embedder also answers fZ2e's concern about on-device use, and the corpus carries over to students it was not built from, with Ministral 3 3B, whose errors were not used, gaining 10.6 points on MedQA. 8LXZ writes that these experiments "directly address my concerns" and that the paper is "solid and carefully controlled", and fZ2e finds the paper "sound and carefully executed".

We would be grateful if you could take the revised PDF and our responses into account. We would also like to take this opportunity to thank the reviewers for their careful reading and concrete suggestions. The additional experiments, clarified metrics, and revised tone mapping all stem directly from the reviewers' constructive feedback, substantially improving the clarity and rigor of our paper.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.