- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces a methods for SLMs to store instructions to retrieve facts that do not fit in memory. A kind of run time compression. Distillation into corpus, not into weights. After clustering. So text, not embeddings.
Critical strengths
a new method for making SLM is quite useful
Critical weaknesses
Idea of generating a domain specific corpus is not new. The analysis may be interesting. Results are better on some domains, not all. This can be due to benchmark quality.
GPT-5 wrote the instruction corpus. No study of the impact of other models.
No supplementary material provided. Abstract does not contain measurements.
Questions for authors
Why would is be better to create a corpus (text) than weights (embeddings)? Store on disk, not memory? Textual swapping? RAG? Please explain more clearly.
Interesting idea, not ground breaking. Better suited at specialized conference? weak reject.
3 comments
Some concerns addressed
Thank you for the reply, and for the extra experiments. The ideas of the paper remain interesting, and follow up on related work such as Self-RAG and Retrieval-of-Thought. I remain unconvinced about how substantive the idea is, as another reviewer phrased it as limited conceptual novelty. The concern on the benchmark side was phrased as narrow evaluation surface. In summary, nice result, but unconvinced that this is the right venue.
My score reflects this.
Thank you for the follow-up and for engaging with the new results. We would like to understand your remaining concerns well enough to address them, so we respond to each and ask one question about each.
On the evaluation surface: the method is tested on five students from three model families, on three benchmarks that test different skills (medical knowledge, legal reasoning and arithmetic). Since the first response, we have added a second teacher (the open-weight gpt-oss-120b), a second, locally run encoder, and paired tests against the strongest baseline on every task. We are now running an open-ended version of MathQA and a transfer test of the MedQA corpus on MedMCQA, and we will report both in this discussion. If there is a specific task, model size or setting that would change your view, please tell us, and we will run it if we can before the discussion closes.
On conceptual novelty: we read the question as whether the gain comes from anything beyond retrieving teacher text, which existing methods already do. We tested this directly in our reply to another reviewer. When both methods retrieve the same training questions, written up by the same teacher, instructions beat the teacher's worked solutions on MedQA by 3.2 points, and RICP's teacher-written insights do not help at all. What the teacher writes, and for whom, therefore matters and we provide a new method for doing this. Methods such as Self-RAG and Retrieval-of-Thought help a model that can already solve the problem. Our setting is a frozen small model that fails the problem and cannot write the guidance itself. If there is prior work you see as already doing this, or an experiment that would separate our method from it, we would welcome the pointer.
We thank the reviewer for finding the method "quite useful" for small language models. We will provide an updated paper with the clarifications and results requested.
The new idea is how a teacher model builds a corpus that a small model can follow, i.e., the methodology. The analysis shows why a teacher-written corpus outperforms an existing domain corpus. GPT-5, as judge, marks the retrieved textbook passages as pointing toward a wrong option on 22% of MedQA questions, against 9% for the instructions, and where the text lacks the deciding fact, the passages turn a third of the questions a small model had right into wrong answers, against a fifth for the instructions (Section 6).
Instruction retrieval improves mean accuracy over zero-shot on all three benchmarks (10.6 points on MedQA, 3.3 on MMLU Law, 2.4 on MathQA), and no small model loses accuracy on any benchmark. Measured against the room left to gain, the corpus closes 35% of the gap to GPT-5 on MedQA and 41% on MathQA, where two students with the corpus match GPT-5 (Section 5). The size of the gain differs because each benchmark tests a different skill (Section 6). MedQA tests recall of medical facts, which the instructions supply, while MathQA tests arithmetic procedures the small models already perform well on, and every student gets only about 6% of MathQA questions wrong on every attempt (Section 6).
GPT-5’s strength alone does not produce the gain. GPT-5 also writes the few-shot examples and the RICP principles, and both fall below instruction retrieval on all three tasks (Table 1). To address this further, we rebuilt the corpus for each benchmark with gpt-oss-120b, an open-weight teacher, keeping the clusters, teacher prompt, student errors and retrieval fixed. On MedQA, the gpt-oss-120b corpus raises the mean accuracy of the five students by 7.1 points, against 10.6 with the GPT-5 corpus, and every student’s interval is above zero.
Similar to distillation and other methods, a stronger teacher yields better results, but the method does not depend on any one teacher. Please let us know if there are any other experiments that would help clarify this. The full model level breakdown will be in the revised appendix.
As stated in the paper the code and instruction corpora will be released on publication. All prompt, settings and additional results are listed in the appendix.
The submitted abstract already reports the headline result, "On MedQA the corpus raises mean accuracy by 10.6 points, where retrieved textbook passages raise it by 3.9 and few-shot examples from the same teacher lower it", and Table 1 gives the few-shot drop (1.6 points). Please let me know if there are specific changes you want us to include.
Section 2 explains why the corpus is stored as text rather than in the small model's weights. Fine-tuning specializes the student, spending its limited capacity on one target ability at a measured cost in generality (Fu et al., 2023), and must be repeated for every new student and domain, whereas instruction retrieval keeps the student frozen and general. A small model also has little room for new facts, because the facts a model stores are tied to its parameter count (Section 1). The same corpus serves any small model without training, and Ministral 3 3B, whose errors were never used to build the corpus, gains 10.6 points on MedQA (Table 1). Embeddings are used only to find the nearest instructions, and what the student reads is text. The corpus is stored on disk (2.1M tokens for MedQA), and only the five retrieved instructions enter the prompt. Unlike retrieval augmentation, which retrieves general passages, instruction retrieval retrieves text written for a small model to follow, and retrieved textbook passages raise MedQA accuracy by only 3.9 points.
The question the paper asks, what text a frozen small model needs at inference to solve problems it fails, is general to language models, and the paper answers it on medicine, law and mathematics with five students from three families. Related work the paper cites appears at ICLR, including Self-RAG (ICLR 2024) and Retrieval-of-Thought (ICLR 2026), and related methods such as ReasoningBank and ACE appeared at ICLR 2026.
We hope the new teacher experiment and these clarifications address your concerns and welcome further questions during the discussion.