- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper focus on continual learning with adapter libraries where a frozen model is accompanied with several adapters, and during inference one adapter has to be chosen for an input question. In one setting, they keep the number of adapters fixed, and the adapters are trained with full replay buffer. When the training data for the router decreases, the router accuracy drops. Next, when the replay buffer for adapter training decreases, the router accuracy drops even further. In these cases, a model that selects an adapter with maximum confidence has stable accuracy. In the second setting, they keep the training data fixed, and increase the number of adapters. The accuracy of model that predicts by the most confident adapter, drops. And the accuracy of the router based approach remains stable. They suggest a fix called as Reference Anchored Slack (RAS). In this case, they add 250 reference examples to each adapter training which teaches the model to abstain on the same examples for the different adapters. They show that adding such examples, improve the accuracy of the most confident adapter model.
Critical strengths
- Paper shows that adding RAS training improve the accuracy of the most confident adapter model compared to an approach that doesn't include these examples. The advantage holds when the number of adapters is increased from 15 to 75
Critical weaknesses
-
The paper is very hard to read. In the introduction itself, for example lines 50-76 "A fitted router", "cross-task meta-training data", "router-label starvation", "surface-separable intents", "max-over-K arbitration amplifies the upper tail of imperfectly comparable wrong-expert score", "substrate", "thinner wrong-expert score tails" Many new terms are coined but not explained.
-
The paper assumes that the reference data is fixed. If a new task arrives, and its data is similar to the reference data, then the RAS training will abstain on such examples. And if the new task data is very dissimilar to the reference data, then will RAS training help?
-
Table 3 results have a confounding factor of the addition of 250 examples for training the RAS model. The plain model doesn't contain these training examples.
-
There are insufficient details of the datasets and no examples