- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper identifies a tension in the continual training of small models. Training on off-policy data can often induce catastrophic forgetting and poorer generalization. On the other hand, smaller models can be poorer in-context learner which renders self-distillation-style approaches weak. This paper aims to bridge the gap between the benefits of on-policy data and high quality supervision by retaining off-policy supervision while rewriting it via an aligner model and performing rejection sampling to preserve correctness. They identify desiderate for the aligner model: a model whose generations induce the lowest loss, while simultaneously achieving higher performance. Practically, they find that slightly larger variants of the same model family tend to achieve the best tradeoff and that the largest aligner is not necessarily the best. They demonstrate comparable retention to existing methods for leveraging on-policy data, while demonstrating greater acquisition across 3 continual learning settings. They demonstrate that these gains primarily arise from rewriting full demonstrations in a closer distribution to the student, rather than merely leveraging the stronger capabilities of the aligner via ablations.
Critical strengths
- Problem Formulation: This work tackles a problem of practical and conceptual significance. Cases where a model’s context-learning ability are too weak to provide adequate supervision represent a key limitation for self-distillation-based approaches (which are an otherwise promising approach for continual learning) and this paper highlights and takes steps toward addressing this limitation.
- Simple and Well-Motivated Approach: The approach presented by the paper is simple and presented in a relatively general way and appears to be broadly applicable. Moreover, the authors justify their choices via analysis of the weaknesses of existing approaches. Additionally, ablations generally provide justification for the design choices. Thus, I feel that the approach is likely to provide benefits across multiple settings and be generally applicable.
- Well-Studied Ablations and Comparisons: The authors compare against a comprehensive range of benchmarks and provide analysis of their relative advantages compared to both fine-tuning based approaches, as well as existing distillation based approaches. The results they present support their conceptual claims and motivation for their results. They also demonstrate cases where their approach does not improve over the baselines (when the base model retains sufficient capability to produce supervision via context).
Critical weaknesses
-
Novelty: This paper may be less novel relative to existing work on generating synthetic data, including studies on understanding the impact of different synthetic data generators, the benefits of scaling generators, etc. While this may not necessarily be a reason for rejection, I feel that the article can be improved if the authors were to comment on this connection to prior work including commenting on (a) their novelty relative to this field (either in approach or problem formulation) and (b) the extent to which their results confirm or reject existing findings in this setting. I believe that without clarification of this point, the contribution of this paper is hard to evaluate that the authors can clarify further.
-
Limited Number of Settings and Examination of Task Difficulty: The authors use a set of three settings based off of the Self-Distillation Enables Continual Learning paper. While this is an appropriate choice to make comparisons between their approach and the baseline, I believe that examining a larger gradient of task types/difficulties could make their claims stronger. They present a claim on requiring just a slightly larger model of the same family for performing alignment. However, I feel that making this claim thoroughly would require examining potentially more challenging tasks or maybe understanding how the minimal size of the aligner model varies across tasks. Intuitively, there must be some scaling of the task difficulty/novelty/etc with the model size necessary to achieve a task right? If my assertion here is incorrect, would be helpful for the authors to clarify.
-
SDFT+Rejection Sampling: Apologies if I have missed this in the paper, but is/could SDFT also be performed with rejection sampling in order to partially mitigate the impact of noisy supervision when using the same student/teacher? I understand that this is not the exact setting examined by the original SDFT paper and also it is not technically a baseline. However, I think that it would be of conceptual interest
-
Sequence of Continual Learning: As far as I can understand, the experiments in this paper use a single continual learning stage. However, one other setting that SDFT examines is a sequence of continued training stages. I wonder whether SelfSynth would encounter difficulty by drifting away from the aligner in this setting? I wonder if the authors could comment on whether this is a concern/potentially run some experiment establishing that this is not an issue.
Questions for authors
Please respond to the weaknesses I raised