- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper frames online continual distillation as a post-deployment learning problem, introducing SAGE: a closed-loop framework that identifies competence-boundary violations, clusters emerging deficiencies into stable knowledge regions, and injects targeted teacher supervision via efficient low-rank adaptation.
Critical strengths
- The paper identifies and solves the boundary saturation for distilling SLMs in streaming environments.
- The proposed SAGE integrates a label-free Trigger, streaming clustering Buffer, and a two-stage low-rank adaptation Debugger to handle streaming data without retraining the whole model.
- Extensive experiments demonstrate that SAGE improves performance and enables cumulative gains across multiple models.
Critical weaknesses
- The performance relies heavily on the quality of the teacher signal, which means noisy teacher feedback can degrade downstream adaptation. The three benchmarks used in the article is limited and lacks evaluation for complex reasoning tasks.
- While real-time triggers and buffering add minimal latency, the Debugger's cluster optimization remains computationally expensive
- The presentation of the experimental results feels unstructured and lacks a cohesive layout.
Questions for authors
- SAGE combines standard anomaly detection, off-the-shelf clustering, and LoRA tuning. What is the distinct breakthrough over a naive pipeline coupling these off-the-shelf modules, and why is this specific design uniquely necessary for LLMs?
- How is SAGE generalized to the complex tasks? In a realistic post-deployment streaming scenario where user inputs arrive as unstructured raw text, how can SAGE guarantee task atomicity without an expensive, pre-processed teacher pipeline?
4 comments
Response: Complex-Task Generalization, and Task Atomicity**
W1 and Q2.1: Can SAGE generalize to more complex reasoning tasks?
Response to W1 and Q2.1. We provide three complementary results to examine SAGE's generalization to complex reasoning tasks, covering experience selection, progressive capability acquisition, and structured adaptation:
-
Cross-domain experience selection (Appendix C, p. 16). Trigger serves as the gatekeeper for knowledge acquisition, determining which incoming experiences enter the Buffer for subsequent adaptation. On EntailmentBank, WorldTree, and ARC, it achieves approximately 1.0 AUC using TriviaQA-calibrated parameters (Figure 9), supporting its ability to identify emerging capability gaps across domains.
-
Progressive reasoning adaptation (Figure 3, p. 8). Curriculum-based distillation enables SAGE to acquire increasingly complex reasoning capabilities. Level-2 reasoning EM improves from nearly 0% to approximately 81%, while previously learned Level-1 performance remains above 99%, demonstrating capability expansion with knowledge retention.
-
Adaptation to complex tasks (Table A). Since complex reasoning may involve entangled supervision, we use CoT decomposition to construct atomic learning experiences (Appendix E, p. 24). We further evaluate SAGE and baselines under matched supervision, as shown in Table A. SAGE (GT) with CoT achieves 95.70% and 87.14% accuracy on PubMedQA and LexGLUE, outperforming CA-LoRA with identical CoT supervision by 7.41 and 9.05 percentage points, respectively.
Table A. CoT Supervision and SAGE Adaptation (LLaMA2-7B).
| Method | Training Data | PubMedQA Acc | PubMedQA F1 | LexGLUE Acc | LexGLUE F1 |
|---|---|---|---|---|---|
| LoRA | Original | 66.60 | 64.13 | 35.29 | 35.29 |
| LoRA | CoT | 72.91 | 72.06 | 42.84 | 41.97 |
| CA-LoRA | Original | 84.80 | 84.70 | 72.21 | 75.50 |
| CA-LoRA | CoT | 88.29 | 87.67 | 78.09 | 78.92 |
| LwF | Original | 58.20 | 56.45 | 39.80 | 38.50 |
| LwF | CoT | 65.54 | 64.42 | 46.31 | 45.47 |
| SAGE (GPT) | Original | 78.03 | 78.11 | 75.46 | 74.96 |
| SAGE (GT) | Original | 87.51 | 87.30 | 84.21 | 84.08 |
| SAGE (GPT) | CoT | 84.73 | 83.67 | 77.22 | 77.87 |
| SAGE (GT) | CoT | 95.70 | 95.37 | 87.14 | 87.63 |
Together, these results support SAGE's potential to extend to more complex tasks through transferable experience selection, progressive learning, and structured knowledge adaptation.
Q2.2: How does SAGE ensure task atomicity from unstructured streaming inputs without costly preprocessing?
Response to Q2.2. We clarify that CoT decomposition is a one-time, task-specific preparation step, rather than a pipeline executed for every incoming query. Specifically, we first identify task-relevant examples through keyword matching, then apply LLM-assisted CoT decomposition to construct atomic reasoning steps (Appendix E, p. 24, L1250–1259). We evaluate its cost and quality in Table B below. This approach achieves 97.50% and 94.70% atomicity on PubMedQA and LexGLUE, demonstrating its applicability across both domains.
Table B. Cost and Quality of Task-Specific CoT Decomposition.
| Dataset | Source Samples | Training Samples | Preprocessing Time (s) | LLM Time (min) | Atomicity (%) |
|---|---|---|---|---|---|
| PubMedQA | 211,000 | 780 | 24 | 45 | 97.50 |
| LexGLUE | 45,000 | 644 | 8 | 38 | 94.70 |
Also, decomposition makes complex tasks more manageable for SLMs by providing fine-grained supervision. For less-structured tasks, curriculum-based incremental adaptation, as demonstrated on GSM8K (Figure 3, p. 8), offers a complementary strategy for progressively introducing more challenging reasoning tasks.
Gentle Reminder: Request for Feedback on Our Rebuttal
Thank you again for your thoughtful review. We have posted a detailed response addressing each of your concerns, together with the additional experiments and clarifications you requested. As the discussion period closes in 1 days, we would greatly appreciate your feedback on whether our responses resolve your questions. We are happy to answer any remaining concerns promptly.
If our revisions have addressed your main points, we would be grateful if you could consider updating your assessment.
Thank you for your time and effort.
Gentle Reminder
With only 9 hours remaining in the discussion period, we would be glad to answer any further questions you may have. If our responses have resolved your main concerns, we would appreciate your consideration of an updated assessment. Thank you for your time and constructive feedback.
Response: Framework Novelty
We thank the reviewer for the SAGE's framework-level novelty, teacher dependence, and generalization to complex tasks. And we provide additional CoT and preprocessing results.
Q1: What distinguishes SAGE from a naive combination of existing modules, and why is its design necessary for LLMs?
Response to Q1. We agree that anomaly detection, clustering, and LoRA are established techniques. However, SAGE's contribution lies in coordinating these components to transform deployment-time experiences into persistent model capabilities, rather than simply combining existing algorithms (Sec. 1, p. 2, L070–089). The key distinctions are twofold:
Framework-level novelty. SAGE establishes a coordinated Detect–Organize–Inject process: Trigger identifies what to learn through teacher–student discrepancies, Buffer determines when to adapt through cluster stability, and Debugger determines how to update through cluster-specific LoRA optimization (Sec. 3, pp. 4–7).
Necessity for LLMs. Conventional offline distillation leaves deployed SLMs unable to adapt when evolving inputs exceed their learned capabilities. Addressing this boundary saturation requires three capabilities: detecting emerging deficiencies, distinguishing persistent knowledge gaps from transient noise, and incorporating targeted knowledge without retraining the entire model (Sec. 1, p. 2, L054–089). SAGE realizes these requirements through its three coordinated modules(Trigger, Buffer and Debugger), enabling SLMs to learn from deployment-time interactions and progressively acquire domain-specific capabilities under distribution shifts. Table 3 (p. 8) further supports this design: Debugger improves EM from 80.64% to 94.65% over fixed LoRA with the same 315 samples.
W1: How does teacher signal quality affect SAGE's adaptation performance?
Response to W1. We acknowledge that SAGE's adaptation performance depends on teacher quality. We use ground-truth supervision to simulate an ideal teacher and estimate the performance upper bound (Sec. 4.1, p. 7, L351–356). Table 5 (p. 17) shows that GSM8K EM decreases from 99.80% with GT to 88.65% with GPT-5 and 73.06% with DeepSeek. This gap partly reflects limitations in current teacher capabilities, which may narrow as LLMs improve, while robustness to noisy supervision remains future work.