- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Recursive Continual Post-Training (ReCaP), a framework that combines the benefits of distillation and expert expansion in continual learning (CL). Specifically, it learns and stores a LoRA expert per-task, then performs multi-teacher distillation to distill the experts' knowledge into a shared base model. Extensive experiments show that ReCaP outperforms SOTA CL models on multiple benchmarks and backbones.
Critical strengths
-
The proposed method in the paper is well-motivated, combining the advantages of two widely used strategies to resolve CL issues.
-
The paper conducts thorough experiments on multiple backbones and benchmarks. Most results support the superiority of ReCaP. The influences of design choices in the model are well discussed.
Critical weaknesses
-
Although the inference cost remains , the training of ReCaP requires storing experts and flow-matching models for tasks, and the distillation takes extra time. What is the computational overhead introduced in this process? Specifically, how does the stored memory grow with tasks? What is the overall time consumption in training and inference, compared to other models (e.g., MoE models)?
-
In Table 3, ReCaP does not outperform previous models in LLM 3-stage stream. It could be helpful if the authors can provide more insights to explain this phenomenon.
Questions for authors
Please see the weakness.
Response to Reviewer GCAN: Training and Inference Cost, Text-Only Stream
We thank the reviewer for finding ReCaP well-motivated in combining the strengths of distillation and expert expansion, its experiments thorough across multiple backbones and benchmarks with most results supporting its superiority, and its design choices well discussed.
W1: Although the inference cost remains […], the training of ReCaP requires storing experts and flow-matching models for tasks, and the distillation takes extra time. What is the computational overhead introduced in this process? Specifically, how does the stored memory grow with tasks? What is the overall time consumption in training and inference, compared to other models (e.g., MoE models)?
A1: Training storage grows by under 6% of the base per task, training time by 21% over expert training on CoIN, and inference is cheaper than expert pools.
Inference (one A100, without any optimization). ReCaP deploys one merged checkpoint with the base architecture:
Expert pools keep their experts and routers in memory; ReCaP's serving cost is the base model's and does not grow with the number of tasks.
Storage. All experts share one frozen base, so for T tasks the training archive is Marchive=Mbase+T(Mexpert+Mflow):
The archive grows linearly with T, which the revision states as a limitation, but each task adds under 6% of the base and the archive is never deployed. Expert pools instead keep their experts at deployment.
Training time (CoIN, reported runs):
Expert training is shared with expert-pool methods, which also train one adapter per task; ReCaP's addition is paid once, while its inference savings apply to every request. On the text-only stream, each flow takes about 30 min on one A100 and the three consolidation stages take 52, 73 and 75 min on two A100s.
W2: In Table 3, ReCaP does not outperform previous models in LLM 3-stage stream. It could be helpful if the authors can provide more insights to explain this phenomenon.
**A2:The text-only stream is short and small (three tasks, 14,098 training samples); EWC is best on it and ReCaP is second, and the gap is the last task.
ReCaP is level with EWC on science and ahead on tool; the gap is medical, and it is answer length rather than medical knowledge. Medical is the only long-form task (median reference 88 words), scored by token F1, while science and tool, learned first, have short answers (a letter, a JSON call). ReCaP's medical answers stay short (median 62 words), so they lose recall; at matched length its F1 equals the expert's (51.5 vs 52.7). Consolidating medical alone from the base model with the same recipe reaches the expert (48.2 vs 48.4). Training the medical stage longer (3,000 instead of 1,000 steps) raises medical to 44.3 while keeping tool use, and ReCaP's skill average (61.6) then exceeds EWC's (61.2).
On the longer vision-language streams the ranking reverses, so EWC's advantage is specific to this short text-only stream.
We hope the measured storage, training and inference costs, and the analysis of the text-only stream, address the reviewer's concerns, and would appreciate the reviewer considering an increased score.