- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes TACIT-Switch, a method for deciding when an LLM agent should permanently switch from a cheaper model to a stronger model during a multi-step task.
The method uses paired Cheap and Strong rollouts and teacher-annotated intervention times as training supervision. It combines a model of Strong's success probability with a learned threshold over cumulative trajectory risk, using a mixture-cure survival model and interval censoring.
Experiments on simulated tasks, ALFWorld, and DABench show improved success-cost trade-offs compared with task-level, step-level, and fixed-prefix routing baselines.
Critical strengths
-
Useful and practical problem. Deciding when to switch from a cheaper to a stronger model is an important problem for multi-step agents. The idea of learning an adaptive, permanent handoff policy is interesting, especially since it avoids expensive model-based decisions during deployment.
-
Promising experimental results. The method shows meaningful improvements over existing routing baselines in both simulations and real-agent benchmarks. The larger ALFWorld results are particularly encouraging, with TACIT-Switch achieving 58.0% success compared to 42.0% for the SWE-Router-style baseline at lower reported cost. The paper also includes reasonable experimental controls, ablations, and a larger-scale evaluation.
Critical weaknesses
-
It is unclear how much of the improvement comes from the proposed statistical model. The paper shows that cumulative trajectory risk and task features improve routing, but does not establish why the mixture-cure survival model is necessary. A simpler classifier trained with the same features and teacher annotations might perform similarly. The baselines also differ in important ways. TACIT-Switch uses richer trajectory information than some baselines and continues from the current state, while SWE-Router restarts Strong. These differences make it difficult to attribute the improvements to the proposed learning method rather than the additional information or handoff strategy.
-
The supervision does not directly establish whether switching is beneficial. The model uses Strong's success from the initial state and teacher-estimated intervention times to learn switching decisions. However, Strong succeeding independently does not necessarily mean it can recover from Cheap's intermediate state. Similarly, Strong might fail independently but succeed after Cheap has made useful progress. The teacher annotations are not validated through actual handoffs on real-agent tasks. While the simulations provide some evidence, it remains unclear how accurately the learned score captures the benefit of switching at a particular point in a real trajectory.
Overall, I find the problem useful and the results promising. However, the paper does not yet convincingly establish that its particular statistical formulation is better than simpler adaptive switching approaches, or that the supervision reliably identifies useful intervention points.
Questions for authors
-
Why is the survival-analysis formulation necessary? Can the authors compare against a simple switching classifier or learned cumulative-risk threshold using the same features, training data, and teacher annotations? This would help establish whether the statistical formulation provides additional value.
-
How reliable are the teacher-annotated intervention times? Have the authors tested whether Strong actually succeeds when resumed from these checkpoints on real-agent tasks? It would be useful to compare actual success when switching at the annotated checkpoint versus continuing Cheap or switching at nearby checkpoints.
-
Are the baseline comparisons isolating the right factors? In particular, could the authors compare against a fixed-prefix policy that continues from the current state instead of restarting Strong, and a step-level policy using the same trajectory information? This would help separate the benefits of adaptive timing, richer features, and preserving progress.
2 comments
Response to Reviewer Q9eF (2/2): timing models, checkpoint rescues, and baselines (Q1–Q3)
Q1. Is survival analysis necessary?
Our contribution is how we obtain and learn from the labels, not a particular survival model. An offline LLM teacher reviews paired Cheap and Strong trajectories and marks a range of plausible handoff steps. For example, it may be confident that a handoff is warranted around steps 6-8 without being able to justify step 7 over either neighbor. We fit a statistical model to the whole window as an interval-censored observation. This avoids the time and compute expense of generating counterfactual training labels by running Strong separately from every candidate checkpoint. To our knowledge, this is the first use of LLM-teacher handoff windows as interval-censored supervision for learning permanent handoff between LLM agents.
Can a different timing model learn from the same windows? Yes. We chose mixture-cure for interpretability. To test an alternative, we kept the component predicting Strong’s success from the initial state fixed and replaced the timing component with a discrete-time logistic hazard fitted to the full teacher intervals. Both models used the same 200 training tasks, 89 teacher windows and paired outcomes. We selected each handoff threshold on Dev300 under the same cost budget and froze it before testing.
| Controller | Solved / 274 | Mean weighted call cost |
|---|---|---|
| TACIT mixture-cure refit | 148 | 188.38 |
| Interval-censored logistic hazard | 151 | 199.56 |
On these 274 previously evaluated ALFWorld tasks, the logistic hazard had a similar observed success rate to the TACIT refit (55.1% vs. 54.0%), at a higher mean cost. As you suggest, the timing module can be replaced without changing how the teacher labels are collected or reannotating the trajectories.
Q2. Can Strong rescue teacher-marked states?
Yes. We ran Strong from four saved checkpoints on each Cheap trajectory. Of 75 ALFWorld tasks where Cheap failed and Strong succeeded from the start, 34 had all four checkpoints available. We compare handoff times on those same 34 tasks. Continuing with Cheap solved 0/34 by selection. Mean weighted call cost includes both the Cheap prefix and the Strong continuation.
| Offset from teacher recommendation | Solved / 34 | Mean cost |
|---|---|---|
| 6 steps earlier | 20 (58.82%) | 202.83 |
| 3 steps earlier | 23 (67.65%) | 189.13 |
| At recommendation | 21 (61.76%) | 191.32 |
| 3 steps later | 18 (52.94%) | 191.32 |
Strong rescued 21/34 tasks at the teacher’s recommendation. Handing off three steps earlier rescued two more at a slightly lower mean cost; moving six steps earlier offered no further benefit. On these trajectories, the teacher often identified a state from which Strong could recover, but not necessarily the best step to switch.
Q3. What do the baseline controls isolate?
Fixed prefix: restart or continue? We kept the SWE-style router’s four-action prefix and its learned scoring head. First we changed only what happened when the router called Strong: it resumed from Cheap’s state instead of restarting. We then allowed the continuation variant to reselect its gate on Dev. Results below are from the original ALFWorld 9B-to-27B scale-up evaluation.
| Policy | Solved / 274 | Mean cost |
|---|---|---|
| SWE-style restart, original gate | 115 | 189.85 |
| SWE-style continuation, same gate | 106 | 181.37 |
| SWE-style continuation, gate reselected on Dev | 147 | 194.83 |
| TACIT permanent handoff | 159 | 182.79 |
Keeping progress alone did not improve the SWE-style router at its original gate. Retuning the gate made continuation more competitive, but TACIT still solved 12 more tasks at lower mean cost. The separate fixed-16 comparison in W1 also retains progress in both arms.
One-action deferral with trajectory information. We gave a step-level controller the same 12 task descriptors and five underlying trajectory diagnostics rather than just action uncertainty. At each step, it could delegate one action to Strong and then return control to Cheap. It solved 87/274 tasks at mean cost 143.37, charging every Cheap proposal and Strong call. This controller was cheaper but solved fewer tasks than TACIT. Because its Strong-action scoring procedure also differed, we do not attribute the entire gap to permanent handoff.
Response to Reviewer Q9eF (1/2): handoff timing and teacher supervision (W1, W2)
W1. What accounts for the gains?
Does the gain persist when both policies keep Cheap’s progress? Yes, relative to a fixed handoff on the original ALFWorld 9B-to-27B evaluation. Under the Dev budget, the best fixed delay was 16 Cheap actions. Strong continued from the current state under both policies. TACIT solved 159/274 tasks versus 137/274 for fixed handoff at similar mean weighted call cost (182.79 vs. 181.09). That is 22 additional successes, or 8.03 percentage points (paired 95% bootstrap interval: 3.28 to 12.77). Preserving progress alone therefore does not explain this difference.
Where should a handoff label come from? A Cheap trajectory shows what happened while Cheap remained in control, not what would have happened had Strong taken over. A direct way to estimate the best handoff time under a given budget is to save the state after each Cheap action, run Strong several times from each state, and compare success and cost across handoff times and against continuing with Cheap. Repeating these counterfactual rollouts at every step of every training trajectory is expensive in both time and compute.
What is new in our supervision? Using an LLM to review a past trajectory is not itself new: for example, Zhang et al. (2025) use execution logs to identify the step responsible for a failure. Our offline teacher instead reviews existing paired Cheap and Strong runs to mark when a handoff might help. If several adjacent steps are plausible, asking for one exact step would turn the teacher’s uncertainty into an arbitrary label. We retain the whole window as an interval-censored observation. To our knowledge, this is the first use of teacher-marked windows as interval-censored supervision for learning permanent handoff between LLM agents.
Is mixture-cure required? No. We chose it because it handles censoring and separates Strong’s chance of success from the start from the timing of handoff conditional on that success. A practitioner could instead use a nonparametric model or another machine-learning method that learns from interval-censored labels, without changing how the trajectories are annotated.
We fitted a discrete-time logistic hazard model to the same training tasks and teacher windows. Its observed success rate was similar to that of the TACIT refit, at a higher mean cost (Q1). We therefore distinguish the evidence for choosing when to switch from evidence for any particular model of timing. Q3 separates the choice to continue or restart from the choice to hand off permanently or for just one action.
W2. Does supervision identify useful interventions?
Can Strong rescue a task at the teacher’s recommended step? We tested four handoff times on the same 34 ALFWorld tasks where Cheap failed, Strong succeeded from the start, and all four checkpoints were available. Strong rescued 20/34 tasks six steps before the recommendation, 23/34 three steps before, 21/34 at the recommendation, and 18/34 three steps after. Three steps early had the highest observed rescue rate on this subset. Q2 reports the corresponding costs.
Should we calibrate the fitted policy’s timing? At Reviewer F6yD’s suggestion, we compared TACIT’s actual handoff step with the teacher’s recommendation on 117 annotated ALFWorld test tasks. TACIT handed off 2.31 steps later on average. This suggests testing whether an earlier handoff can compensate for that lag. The 34-task comparison above supports exploring that direction, but it set the checkpoints directly rather than shifting TACIT’s decisions. In the revision, we will make timing calibration an explicit step: select the training-window offset and handoff threshold on Dev under a fixed cost budget, then freeze both before testing. On the full 300-task Dev set, this procedure selected no shift.
Initial success is neither a guarantee nor a prerequisite for rescue. Although Strong succeeded from the start on all 34 tasks, it failed on 13 when it took over at the recommendation. Conversely, on a separate set of 65 tasks where both models had failed from the start, Strong rescued 12 after 16 Cheap actions. We therefore use Strong’s initial outcome as a training signal, not as a label of whether an intermediate state is recoverable.
Why use teacher windows? Testing every candidate step on every training trajectory would require a separate Strong continuation from each checkpoint. The teacher instead marks a plausible window on existing paired runs. The actual handoffs above show that many recommended checkpoints were recoverable; they do not establish that the learned score measures the causal gain from switching.