- Rating
- 1(−1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses a key challenge of model-update-based knowledge distillation, which becomes infeasible for heterogeneous model architectures. It proposes a text-based knowledge distillation method that synthesizes knowledge as a primer through cyclic multi-agent interaction (Teacher to student to evaluator to prompter, then back to the teacher, and so on). The method was evaluated on two datasets: math and coding (common testing benchmarks). The core idea is to close the knowledge gap by synthesizing a textbook from LLM interactions and using it as a prompt for students. It sounds interesting but seems more like a modified version of in-context learning, where the context comes from multi-role LLMs. Due to the trivial nature of the approach, the lack of literature grounding, and the limited domain experiments, the proposed approach is unsuitable for acceptance at the target venue.
Critical strengths
The paper is easy to follow, with simple English and a consistent narrative. The overview illustrations (Figures 1 & 2) help readers understand the approach even without reading the method section. The reported results show performance improvements over approaches in both the coding and mathematics domains. The manuscript reports experiments on cross-model knowledge transfer, transferring knowledge from one student to another while maintaining accuracy across domains. The appendices provide prompts and dataset details to clarify the implementation. I enjoyed reading the paper; I appreciate its writing quality, structure, illustrations, and presentation of results.
Critical weaknesses
The comparison with State-of-The-Art approaches is loosely related, as most approaches are based on standard KD methods (parameter updates) with the same architecture; it would be helpful to see comparative results with parameter-update-free methods. The core drawback of the proposed approach is its heavy reliance on the synthesizer, and the method nowhere addresses this issue. ction-tuned Because inLLMs can be sycophanticLLMs, relying on a single verifier and synthesizer makes the system fragile; one ill-designed synthesis can derail the entire process, and the proposed method offers no way to recover. The approach ignores the obvious limitations of LLM-only interactions and lacks grounding in the literature. The lack of theoretical analysis and grounding in the literature makes it difficult to evaluate the approach's novelty and practicality, especially given the limited domain experiments.
Questions for authors
Key questions to answer to establish the novelty of the approach:
- What is Fast? It's not defined anywhere in the abstract, and it reports performance numbers.
- How does the approach differ from SOT in-context learning methods (given that learning from LLM interactions is inherently flawed)?
- What should the size of the primer be, and how should it be retrieved at inference time?
- If the approach is truly UNIVERSAL, then why don't we create one-primer-for-all and let it be used with small models?
- A direct comparison with a closely related approach is necessary to establish the approach's superiority, but it is missing in its current state.
- Claiming UNIVERSALITY by testing only on two domains is misleading; are the authors trying to claim UNIVERSAL is just for the sake of terminology?
- Some domains don't follow a standard structure for answering and reasoning, such as creative writing, ARC-AGI, and virtual Games. It would be great to see how the approach works on such domains.
- Failure cases should also be reported where the student learns nothing or its performance worsens with the primer. This behavior is observable in small models when over-tuning or forcing context following.
- Code for reproducibility would help evaluate the proposed approach and establish its practicality.
I appreciate the work done by the SPC and the response provided by the authors. Having reviewed the relevant details, I am maintaining my rating.