- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper considers the problem of training a two-model cascade to handle an online data stream. The authors are specifically interested in the case where the task is changing over time.
This paper proposes the following idea. We have a small student model, and a large teacher model. On a stream of input queries, you first apply the student. The student produces a vector describing its internal state, which is fed to a regression model that is used to estimate the reward of moving to the teacher (including an uncertainty estimate). I this exceeds zero, we go ahead and ask the teacher for the result. If the student and the teacher differ, the reward is then positive. If the student and teacher are the same, the reward is negative. There is zero reward if the decision is not to call the teacher. This reward is then used to update the regression model and the process continues. When the teacher is called, the (query, student answer, teacher answer) triple is stored, and once enough are stored the student is updated by self-distillation using the teacher’s answer as an in-context demonstration.
On the negative side, the main body of the paper includes little in the way of detail on how the student is trained. The central algorithm is not complete. There is no learning for the student in the actual algorithm and the reader must figure how this will work from the material in Section 3.2, Figure 2, and some statements all the way on page 30 of the paper! The paper gives no details on how re-used rollouts are used when computing the Kl objective, though it may be there somewhere. The paper needs to be self-contained and these details would ideally be in the central algorithm. Statements like those on line 1608 of the paper are not “experiment configurations”, they are algorithmic details.
One of the biggest contributions claimed by the paper is that routing happens in “a non-stationary system, with both the router and the student constantly adapting” so the router is trying to learn to manage a student that is always adapting to the changing data. This is also tested experimentally. However, it seems that there is actually nothing algorithmically that’s meant to address this, so it’s not really a contribution. The parameters of the router seem to be accumulated forever, and the only mechanism to adapt to the “non-stationary system” is averaging of these parameters.
In fact, using standard LinUCB in this way with a shifting student may be a particular problem. Once the system has throughly explored a region of the space because it has seen a query many times, it decides that it is done there and the router will never choose to re-explore. This can be a problem specifically in a non-stationary system because the student may perform poorly in that part of the space after previously performing well there. The reward design may make this worse. IF the system chooses NO-T-CALL it cannot change any of the system parameters, so once it has a NO-T-CALL in a region it will not go back there and has no obvious mechanism to discover that the student has stopped doing a good job there.
Another issue is that the contribution of the paper is somewhat minimal. One big component here is the router. But the router is based on the standard LinUCB algorithm with a somewhat novel exploration weighting. Then there is the self-distillation. But this is also standard.
Finally, I will point out that that the paper simply assumes that self-distillation makes sense. The simplest thing to do would be to use supervised fine-tuning (SFT). Or, if one was worried that this would yank the model around too much, use SFT plus a KL-divergence regularizer. One obvious problem with self-distillation is that if the student simply can’t handle the query, then it will never learn (maybe this is what is happening with the Chemistry result?). Our group has played with small models where the model is given data that are far from anything they’ve ever seen before, and the model just can’t handle the input. Even with the teacher’s output, the model is likely to produce nothing of value before it has been trained. In such a situation, it can never learn. The paper could easily ablate this decision, and it should!
Critical strengths
The strongest aspect of the paper is the testbed and the problem definition: the authors have done a lot of work carefully designing a set of testbeds that have various changes in the task composition, introducing new tasks mid-stream. Prior work has not sufficiently considered the problem of non-stationary tasks, and this is a nice contribution.
The experiments are quite good. The two main components of the system are carefully ablated: are you self-distilling the student model, and which router will you use (LinUCB vs. MLP vs. confidence threshold)??
I really like that a key design aspect is the paper is to build a system that does not need access to the internals of the teacher model. Especially if the teacher is a closed-weight commercial frontier model this is absolutely necessary.
Critical weaknesses
Though I've listed several weaknesses above, I think that only one of them may support rejection: the paper sets up an interesting problem (routing in “a non-stationary system, with both the router and the student constantly adapting”) but then it's not clear that there is anything algorithmic to support this. In fact, a thought experiment (as described) would seem to indicate that (as I wrote) if "the system chooses NO-T-CALL it cannot change any of the system parameters, so once it has a NO-T-CALL in a region it will not go back there and has no obvious mechanism to discover that the student has stopped doing a good job there."
Questions for authors
I would like to read what the author have to say about the lack of specific algorithmic innovations that are meant to support use in a non-stationary system.
3 comments
Gentle reminder about the posted author response
As the discussion period approaches its end, we wanted to see if you had a chance to consider our response. In particular, we explained how our router adapts to the changing student and data, and addressed the failure case in your thought experiment. We hope our response addresses the concerns you raised, and would be grateful if you would consider updating your assessment.
Thanks to the authors for responding to my comments. I've read them carefully. My review of the paper was already positive and I'll take these comments into account during the discussion.
Author Response
Addressing the critical weakness:
A contextual bandit router adapts naturally to a changing student and data.
It updates its routing rule continuously and uses the context embedding to distinguish query types, including new ones. As the student improves on a query type, teacher calls on that type more often receive negative rewards, shifting later queries of that type toward no teacher call (Lines 202-204).
Queries from a new task are unfamiliar to the router, so its uncertainty is high and it explores with teacher calls. If the student cannot yet do the task, these calls raise system accuracy and train the student; if it already can, the negative rewards quickly steer the router away from teacher calls on that task. This makes the contextual bandit handle changing data.
An MLP router would also update its routing rule continuously, but empirically we find it performs worse, with teacher call rate fluctuates more sharply across a stream (Line 459-470). We think it could be that the gradient updates to a neural classifier tend to overfit to recent local patterns.
You are right that the router receives no feedback on NO-T-CALL rounds. However, we did not observe this degradation in our experiments: the student retained performance on learned tasks even after long absences (S2 in Sec 6.2 and Appendix C.2).
We agree this risk is conceivable with a suboptimal student learning algorithm (catastrophic forgetting may happen). However, the student learning process should be designed to prevent it in the first place, rather than relying on the router to detect and correct it. Many continual-learning techniques mitigate forgetting and should be used for student learning (we use self-distillation and standard replay).
In a hypothetical extreme case where forgetting may occur after the router has considered the task as well-learned, the forgetting may also affect the student’s hidden state (thus the context vector), which increases the router’s uncertainty and prompt renewed exploration. Additionally, this hypothetical case can be addressed by keeping a small fixed rate of forced teacher calls to maintain exploration and recalibrate. With an effective student learning method this is unnecessary, so we leave this scenario to future work.
Addressing other points raised by the reviewer
We will reorganize to add the student update step to Algorithm 1 and move other method details into Section 3, and make the main sections self-contained.
We do not claim a new bandit or distillation algorithm. Our contributions are the problem formulation, the overall framework (a student that keeps learning from a black-box teacher, with a router that adapts online), and the evaluation suite with distribution shifts. Showing that we can combine LinUCB and self-distillation in this new setting is a novel application of these techniques, and is part of our contribution.
We will add experiments with the SFT + KL regularization alternative in the camera-ready if accepted. Swapping the student learning method leaves the router and the evaluation suite unchanged, so this comparison does not affect our conclusions, which focus on router comparison and our evaluation suite.