- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper shows that merging many LoRA adapters leads to significant performance degradation on many expert contract tasks (32 tasks to be exact). To solve this, they make use of multi-teacher on-policy distillation. They show that this can retain performance across the 32 tasks.
Critical strengths
-
The paper is well thought through - it seems well put together and sound for the most part.
-
Scaling up the number of experts is good. These many contract tasks can be of utility to the community as long as the authors open source them after acceptance - the data to train them as well as the expert adapters themselves. And showing that performance is retained with 32 experts is nice as well.
Critical weaknesses
-
I can't help but think this work is slightly incremental. Both MOPD and CollectionLoRA (50 experts) follow the same approach but the former is focused on non-PEFT language setting while the latter is focused on vision + PEFT. So this papers core contribution is something similar in the language + PEFT setting. While this paper seems generally sound to me, I'm unsure if the contribution is significant enough to warrant acceptance. I am curious to hear what the authors have to say about this and am open to discussion. Is routing their primary novelty - on the surface it seems like engineering? Or is it the 32 experts in language?
-
The lack of task categorization is slightly hurtful to the papers claims. 32 tasks seems like an arbitrary number and its unclear how well these tasks transfer from one to another. Perhaps a useful addition to this work would be some more detail on this but not a big problem.
-
Smaller models are a problem. I'd like to understand if merging continues to cause significant loss of ability at the 12B+ scale, on the tasks the authors train and evaluate. I cannot deduct points for this because of compute constraints but would urge the authors to include such results, if feasible.
-
How much of the 0% failure after merging comes from lack of ability to follow output format instructions? It seems unclear in the paper and perhaps the author could simply add some more details in the appendix.
-
The paper motivates its problem statement by bringing up inherited LoRA libraries but train their own for the most part across most important experiments. I would tone down the inherited LoRA libraries claim.
My primary concern still is the lack of novelty or insights in this work. Degradation with merging is known. Multi-teach on/off policy distillation has been attempted, with success. I'm not entirely sure I've learned something very new or fascinating but I might have missed something. Curious to hear the authors‘ thoughts.
Questions for authors
I've included the relevant questions within the weakness section.