acceptodds
Under review as a conference paper at ICLR 2027

Universal Textual Teaching for LLMs

Abstract

Knowledge distillation (KD) transfers knowledge from stronger *Teacher* models to weaker *Student* models, but most methods require training the *Student* parameters, thereby binding the distilled knowledge to a specific architecture and checkpoint. This implicit representation is difficult to interpret or reuse across models and limits KD for API-only or costly-to-train models. This paper studies knowledge transfer for large language models (LLMs). We introduce **Universal Textual Teaching** (UTT), a parameter-update-free framework that distills observed *Teacher*-*Student* knowledge gaps into a textual, interpretable, and reusable natural-language artifact called *Primer*. Specifically, UTT first identifies representative gap cases through paired evaluations, and iteratively updates the *Primer* via multi-role interactions: the *Student* attempts each task, the *Prompter* turns evaluation feedback into a teaching instruction, the *Teacher* provides a targeted demonstration, and the *Synthesizer* consolidates validated lessons. Empirically, on the challenging math (Omni-MATH-2) and code generation (KernelBench) tasks, extensive results confirm the effectiveness of the method: UTT remarkably raises the *Student*’s accuracy from 9.4% to 48.6% and accuracy from 9% to 35% on KernelBench, while increasing mathematical reasoning accuracy from 27.6% to 51.7%. UTT also performs better than representative prompt engineering and parameter-based KD methods. Of note, UTT is shown to be generalizable across different *Teachers* and *Students*: a *Primer* synthesized for one *Teacher*-*Student* pair can generalize to other *Students* that do not participate in the synthesis.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Knowledge Distillation, Prompt Engineering, Large Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_w4VKOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses a key challenge of model-update-based knowledge distillation, which becomes infeasible for heterogeneous model architectures. It proposes a text-based knowledge distillation method that synthesizes knowledge as a primer through cyclic multi-agent interaction (Teacher to student to evaluator to prompter, then back to the teacher, and so on). The method was evaluated on two datasets: math and coding (common testing benchmarks). The core idea is to close the knowledge gap by synthesizing a textbook from LLM interactions and using it as a prompt for students. It sounds interesting but seems more like a modified version of in-context learning, where the context comes from multi-role LLMs. Due to the trivial nature of the approach, the lack of literature grounding, and the limited domain experiments, the proposed approach is unsuitable for acceptance at the target venue.

Critical strengths

The paper is easy to follow, with simple English and a consistent narrative. The overview illustrations (Figures 1 & 2) help readers understand the approach even without reading the method section. The reported results show performance improvements over approaches in both the coding and mathematics domains. The manuscript reports experiments on cross-model knowledge transfer, transferring knowledge from one student to another while maintaining accuracy across domains. The appendices provide prompts and dataset details to clarify the implementation. I enjoyed reading the paper; I appreciate its writing quality, structure, illustrations, and presentation of results.

Critical weaknesses

The comparison with State-of-The-Art approaches is loosely related, as most approaches are based on standard KD methods (parameter updates) with the same architecture; it would be helpful to see comparative results with parameter-update-free methods. The core drawback of the proposed approach is its heavy reliance on the synthesizer, and the method nowhere addresses this issue. ction-tuned Because inLLMs can be sycophanticLLMs, relying on a single verifier and synthesizer makes the system fragile; one ill-designed synthesis can derail the entire process, and the proposed method offers no way to recover. The approach ignores the obvious limitations of LLM-only interactions and lacks grounding in the literature. The lack of theoretical analysis and grounding in the literature makes it difficult to evaluate the approach's novelty and practicality, especially given the limited domain experiments.

Questions for authors

Key questions to answer to establish the novelty of the approach:

  1. What is Fast? It's not defined anywhere in the abstract, and it reports performance numbers.
  2. How does the approach differ from SOT in-context learning methods (given that learning from LLM interactions is inherently flawed)?
  3. What should the size of the primer be, and how should it be retrieved at inference time?
  4. If the approach is truly UNIVERSAL, then why don't we create one-primer-for-all and let it be used with small models?
  5. A direct comparison with a closely related approach is necessary to establish the approach's superiority, but it is missing in its current state.
  6. Claiming UNIVERSALITY by testing only on two domains is misleading; are the authors trying to claim UNIVERSAL is just for the sake of terminology?
  7. Some domains don't follow a standard structure for answering and reasoning, such as creative writing, ARC-AGI, and virtual Games. It would be great to see how the approach works on such domains.
  8. Failure cases should also be reported where the student learns nothing or its performance worsens with the primer. This behavior is observable in small models when over-tuning or forcing context following.
  9. Code for reproducibility would help evaluate the proposed approach and establish its practicality.
Reviewer_4aDqOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

First, I would like to thank the authors for their response. Because of a heavy review workload, I inadvertently included some incorrect review comments; I apologize for this error.

This paper proposes a knowledge distillation framework, Universal Textual Teaching (UTT), to transfer capabilities from a teacher LLM to student LLMs by natural-language artifact called Primer through dynamically synthesized textual instruction. UTT consists of three main steps: Knowledge-gap construction, Primer synthesis, and Evaluation. UTT first identifies representative knowledge-gap cases through the paired evaluation of Teacher and Student outputs. Next, it builds a Primer through interactions among Student, Prompter, Teacher, and Synthesizer; this context synthesis comes from multi-role LLMs. Experiments show substantial improvements in Student performance, and the resulting Primers can also be applied to other Student models without retraining.

Critical strengths

S1. The parameter-free design is a genuine practical advantage: UTT can transfer knowledge between any API-accessible models without requiring access to model weights, gradients, or training infrastructure. This democratizes knowledge transfer and makes it applicable to proprietary models where fine-tuning is impossible.

S2. The multi-role interaction framework allowing each component to be improved independently. The synthesis-based approach avoids the common pitfall of directly copying teacher outputs, which often degrades student performance due to distribution mismatch.

Critical weaknesses

Overall, this approach relies on synthetic data to enhance a student model. I am curious about how the accuracy of this synthetic data is guaranteed. If one relies solely on an LLM to make such judgments, essentially having the model generate its own data and then act as its own judge, how can the reliability or authority of these judgments be ensured? The paper proposes a method for distilling knowledge to improve a student model's performance on specific tasks. However, if the student model lacks this knowledge to begin with, and given that its parameters remain frozen, it is somewhat counterintuitive to ask where this knowledge is actually being derived from. Furthermore, for what types of student models is this approach effective, and how does its performance correlate with the size of the student model?

Questions for authors

see the weaknesses

1 comment

I appreciate the work done by the SPC and the response provided by the authors. Having reviewed the relevant details, I am maintaining my rating.

Reviewer_SJgMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces Universal Textual Teaching (UTT), a method for transferring knowledge from stronger Teacher models to weaker Student models through natural-language Primers without updating model parameters. It identifies knowledge gaps by comparing Teacher and Student performance, then builds a Primer through interactions among four roles: Student, Prompter, Teacher, and Synthesizer, with a validation step to control updates. Experiments on KernelBench and Omni-MATH-2 show substantial improvements in Student performance, and the resulting Primers can also be applied to other Student models without retraining. UTT further achieves better results than several prompt optimization and parameter-based knowledge distillation baselines.

Critical strengths
  1. UTT enables knowledge transfer through natural-language Primers without requiring Student model training or parameter updates, making it applicable to API-only LLMs. Unlike conventional knowledge distillation methods, the resulting Primers can be inspected, edited, and reused across different models. This approach reduces the complexity of model adaptation and offers a practical way to transfer explicit knowledge between LLMs.
  2. The paper reports substantial performance gains on mathematical reasoning and GPU kernel generation tasks across different Teacher–Student configurations. The generated Primers also improve the performance of Student models that were not involved in their synthesis, indicating potential for cross-model knowledge transfer. Furthermore, the improvements on the held-out distillation-test subset in Figure 3 provide initial evidence that the learned guidance generalizes beyond the synthesis examples.
Critical weaknesses
  1. According to Eq. (5) and Appendix A.3, the Prompter receives the task and evaluation feedback without explicit access to the Student's incorrect solution or reasoning trace. Unless the feedback contains detailed diagnostic information, the Prompter may generate general task-level guidance rather than identify the actual causes of Student failures. In addition, Eq. (6) filters Teacher demonstrations based on task correctness, which does not guarantee that the reasoning principles extracted into the Primer are valid. The reliability of the distilled knowledge requires further examination.
  2. Although Figure 3 reports improvements on held-out subsets, the main comparisons in Tables 1–3 use the full task pool, including problems used during Primer synthesis. It is therefore unclear whether UTT maintains its advantage over competing methods when evaluated only on unseen problems. Appendix C.1 shows that validation gating improves performance on certain subsets, but Eq. (9) accepts Primer updates based only on the current batch, without checking their effects on previously solved problems. The retention accuracy drops reported in Figure 3 suggest that this mechanism does not reliably preserve the Student's existing capabilities.
Questions for authors
  1. What exactly does the evaluation feedback provided to the Prompter contain, and does it include the Student's incorrect solution or reasoning trace? If not, how can the Prompter identify Student-specific failure causes rather than provide general task-level guidance?
  2. Table 1 includes problems used during Primer synthesis, whatif comparing UTT and baseline performance on the same strictly held-out evaluation set?
  3. How much of UTT's improvement comes from Student-specific knowledge-gap diagnosis rather than general Teacher-generated guidance? A controlled comparison using the same teaching examples and optimization budget is needed.
  4. Since Eq. (9) evaluates Primer updates only on the current batch, how does the method prevent performance degradation on previously solved problems?
  5. How is the correctness of the resulting Primer content independently verified?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.