- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies the effectiveness of lightweight adaptation and compares it with base model training for Multi-task Mixture of Experts Settings. The paper tests multiple combinations of using adaptation during base MMoE model training, including, along with base model training, reusing the adaptation learned from previous checkpoints and not using an adapter at all. The paper also observes that the ranking among the adaptation methods reverses after separately adapting on the frozen MMoE after base model training. The paper also provides a method to predict whether an adaptation would be valuable compared to deferred adaptation.
Critical strengths
The idea of predicting headroom for adaptation and deciding between continual base model training and choosing to adapt is interesting. The paper provides comprehensive empirical analysis on multiple datasets and the chosen training setting.
Critical weaknesses
- The confounding effect of optimization, including a higher frequency of parameter updates and the choice of learning rate and optimizer, is not well studied. This affects the validity of the conclusions made in the paper. Selecting checkpoints based on validation loss is understandable, but we need to make sure that the base model has sufficiently converged and that we can no longer obtain a lower validation loss. Please provide evidence for this. The frequency of parameter updates mentioned in the paper is too short.
- This affects the conclusion regarding the availability of headroom as well as the dynamic performance change between different adaptation methods and rank reversal.
- The paper has significant clarity issues, including excessive use of unnecessary jargon and not defining terminology and notations before use. A list of examples is provided below, but such issues are present throughout the paper. For a good paper, readers should not be required to guess the definition of terms and phrases used in almost every paragraph.
- There are no citations for the statement in the first paragraph of the introduction. Please provide relevant citations. Also, please define or elaborate on these terms and phrases: "shared model", "accessible improvement", "retain accessible improvement near a validation-selected checkpoint", "earlier operator" (what is an operator?).
- Please define or elaborate on these terms/phrases:
- Lines 50- 53: "recovery resources", "accessible adaptation", "divergence from reuse" (reuse of what?), "recovery" (recovery of what?), "coordinates", "feedback into the base", "inherited information".
- Lines 84-90: "task-visible", "the task-visible additive expert boundary", "prespecified bounded intervention", "each readout" (readout of what?), "fresh current-state fitting", "earlier operator", "intervention family, budget, and information access."
- Lines 91-98: "tested cohorts", lines 93-95 ("It persists near high-quality ..."), "New-candidate",
- Lines 98-105: "Native checkpoint", "deferred, crossed, and shadow controls associate a tested coupled schedule’s deficit predominantly with its base trajectory."
- Lines 105-113: "state-relative adaptation value", "its construct validity", "opportunity from compatibility", "transport, and controlled trajectory comparisons", "selection, acquisition cost, calibration", "secondary-quality", "environments", "cross-domain measurements", " controlled trajectories"
- Equations 1 and 2: many notations are not properly defined: .
Thus, the paper needs significant revision before it can be accepted to a high-impact venue such as ICLR.
Questions for authors
Please refer to the weakness section.