acceptodds
Under review as a conference paper at ICLR 2027

AgentBaton: Efficient Model Collaboration in Agentic Tasks

Abstract

Solving long-horizon agentic tasks requires language agents to interact with their environments over multiple steps, and reliable performance often comes from the expensive capable models available through cloud providers. Meanwhile, alternative efficient models with inferior performance but a much lower cost are emerging, including lower-tier cloud models and smaller open-weights models deployable on local hardware. This raises a natural question that remains less explored than traditional query-level routing: can capable and efficient models collaborate within a single long-horizon agentic task to substantially reduce cost while preserving task accuracy? In this paper, we systematically investigate key design choices for such collaboration through model handoffs, where one model takes over from another, focusing on handoff direction and context transfer. We find that efficient-to-capable handoffs generally underperform capable-only execution with limited or even no cost reduction, whereas capable-to-efficient handoffs at suitable stages of task progress largely preserve or even improve accuracy while substantially reducing cost. These findings suggest that a one-time, simple, well-timed handoff strategy is good enough for cost-effective agent collaboration, motivating AgentBaton, a lightweight detector that learns when to hand off from recurring patterns in task progress. Across 11 model pairs and two representative benchmarks, AgentBaton reduces API costs by 65.9% on average while largely preserving task accuracy, with accuracy gains of up to 8.8 percentage points over capable-only execution. Experiments across various reasoning-effort settings further show that AgentBaton extends the empirical accuracy–cost Pareto frontier.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Agent, Efficiency

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_byo4OpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studied "handoff" strategies between a capable model and an efficient model. It started by understanding how handoff works (what to hand off, and when to handoff). Then it labelled training data and trained a light handoff detector (AgentBaton) which pushes the accuracy-cost frontier for agentic tasks.

Critical strengths
  1. I applaud the overall narrative with empirical evidence backing up and motivating each step. It is a pleasure to read. Well executed!
  2. Section 4 (where AgentBaton training data is curated) is my favorite. And I learned something new about when an effective handoff opportunity arises.
  3. Empirical evidence (compared to query routing and the cost effective frontier) is strong.
Critical weaknesses

None.

Questions for authors

Clarification questions.

  1. The initial investigation around logfile vs. context transfer is quite nice. Have you considered explicit prompt the strong model to produce a handoff file (/handoff)? My hypothesis is capable model will produce a clear direction for efficient model to pick up and may improve accuracy.
  2. About training data curation: does the labels agree across different models / where human would switch to handoff? Human study of N(10) will be sufficient.

Just one minor confusion: why isn't "handoff" is in the title? That is all this paper about. "Model collaboration" had me mistaken the theme for multi-agent collaboration (where agents chat back and forth).

3 comments

Thank you for the time and care devoted to reading our paper and for the encouraging feedback. We appreciate your recognition of the empirical motivation behind our design choices, the insights from our training-data curation, and the comparisons with query routing and the cost–accuracy frontier.

Have you considered prompting the strong model to produce a handoff file (/handoff)?

Thank you for the suggestion. We tested this at the original AgentBaton handoff point: GLM or Kimi uses the full context to generate a /handoff file summarizing progress, evidence, unresolved issues, and next steps. Qwen retains the full history and replayed workspace, with an additional prompt inviting it to read /handoff.

Source → QwenBenchmarkOriginal Acc (%)+ /handoff Acc (%)Original Cost ($/task)+ /handoff Cost ($/task)
GLMTB2.180.980.90.270.33
GLMDeepSWE58.459.30.490.60
KimiTB2.186.584.30.570.71
KimiDeepSWE72.670.81.281.58

The above results show that adding /handoff does not consistently improve accuracy and increases cost in all four settings.

Do the labels agree across different models and with where humans would switch to handoff?

Although our annotation aims to identify the optimal handoff point, practical use does not require exact turn-level precision: a suitable handoff can occur after the observed task-progress patterns emerge. We also conducted a human-check case study on 40 samples, finding that human inspection agreed with the appropriateness of the annotated handoff point in 34 cases (85%).

Why isn’t “handoff” in the title?

Thank you for this question. Our initial idea was that this simple yet effective “single handoff” could itself be regarded as a form of model collaboration. However, we agree that, since this paper does not analyze many other forms of collaboration, including “handoff” in the title would better highlight its core theme.

My questions are answered. Thanks for your reply, additional experiments, and for conducting this research. Best of luck.

Thank you for your kind words! We would also like to thank you again for your recognition of our work and the valuable questions you raised, which have helped us improve our paper. Finally, we truly appreciate your time and effort in reviewing our paper and participating in the discussion. Have a nice day!

Reviewer_TuepOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AgentBaton learns when to hand an ongoing agent task from a capable model to a cheaper one. The authors compare handoff directions and two ways to transfer execution history. They then train a small detector on trajectories annotated after execution. Tests on DeepSWE and Terminal-Bench cover 11 model pairs. API spending falls substantially, but the accuracy trade-off depends on the model pair and benchmark.

Critical strengths
  1. Choosing a model before execution misses changes in task difficulty. AgentBaton can hand off after the difficult work is done, using progress observed during execution. Restricting the method to one transfer also keeps the coordination problem manageable.

  2. The paid-receiver results make the cost claim more convincing. For example, DeepSWE Opus-to-Luna improves from 67.3% to 76.1% while API cost falls from $9.91 to $2.92. The study covers two benchmarks and 11 model pairs, so the evidence is broader than a single favorable pairing.

  3. Handoff direction and transferred context are tested rather than left as design assumptions. The reasoning-effort sweep provides another way to trade accuracy for cost. Retrospective progress checkpoints are explicitly distinguished from the online detector, which avoids presenting an oracle diagnostic as a deployable policy.

Critical weaknesses
  1. Tables 3 and 6 disagree on configurations that appear to match. The GLM baseline changes from 61.1% to 69.0%, and GLM-to-Luna changes from 66.4% to 72.6%. These are not rounding differences. Which runs produced each table? The claimed Pareto improvement is difficult to assess until the settings are reconciled.

  2. The accuracy claim needs a narrower scope. Only 12 of 22 settings improve accuracy or lose less than 5% relatively. DeepSWE GPT-Sol-to-Qwen, for instance, falls from 68.1% to 60.2%. Paired uncertainty estimates and repeated runs are needed before these configurations can be recommended for a specified accuracy-loss budget.

  3. The comparisons leave the detector's added value uncertain. The fixed-turn baseline uses one threshold and sometimes costs less than AgentBaton, while the query router almost always selects the capable model. A threshold sweep at matched cost would be a stronger comparison. So would a simple readiness rule based on implementation progress and test feedback.

  4. API savings do not establish a reduction in total operating cost. Local inference is priced at zero, while receiver compute, detector calls, and amortized training and annotation remain unpriced. Timing the detector on an H100 does not resolve latency on the proposed local hardware. A break-even calculation would make the deployment claim assessable.

Questions for authors
  1. What accounts for the GLM and GPT-Sol differences between Tables 3 and 6? Please specify the task sets, run seeds, model settings, context limits, and baseline reuse, and provide paired outcomes with uncertainty estimates.

  2. How does AgentBaton compare with validation-tuned fixed-turn or cost thresholds at the same API budget, and with a simple online readiness heuristic? Please describe MTRouter's training and calibration. How are no-clear-handoff trajectories used to train the detector?

  3. What are the cumulative detector latency and receiver inference usage per task? These would let readers estimate a break-even point when local compute has a nonzero price.

3 comments

Thank you for the careful evaluation and detailed suggestions. We appreciate your recognition of progress-aware handoffs, the evidence for cost savings with paid receivers, and our empirical analysis of handoff direction and context transfer.

GLM and GPT-Sol differences between Tables 3 and 6, and uncertainty estimates

Thank you for identifying this inconsistency and raising concerns about run-to-run variability. We used an agent to execute the experiments. When conducting the reasoning-effort sweep for Table 6, the agent inadvertently reran the GPT-5.6 Sol and GLM-5.3 max-effort configurations instead of reusing the corresponding Table 3 results. This was an unintended execution error, not a deliberate decision to use different runs across the two tables. All other settings—including task sets, model configurations, sampling parameters, context limits, and task-execution parameters—were identical and are documented in the paper. We apologize for overlooking this duplication and not clarifying the provenance of the reported results.

To address your concerns about uncertainty and variability in agentic task execution, we conducted two additional repetitions of the Table 3 experiments, giving three runs in total. Below, we report mean accuracy ± standard deviation across these runs. The standard deviation describes the observed variation and uses divisor n=3. Each run includes all 113 DeepSWE tasks and 89 Terminal-Bench tasks.

Capable modelExecutionDeepSWE v1.1Terminal Bench 2.1
GPT-5.6 SolCapable-only70.80 ± 2.1785.39 ± 1.83
→ Qwen55.46 ± 3.4181.27 ± 0.53
→ Luna67.85 ± 2.2182.40 ± 1.40
Claude Opus 5Capable-only68.44 ± 1.1080.90 ± 2.75
→ Qwen57.23 ± 3.4186.14 ± 1.40
→ Luna73.75 ± 2.7486.89 ± 0.53
GLM-5.3Capable-only67.55 ± 4.8181.65 ± 3.71
→ Qwen63.13 ± 3.3477.90 ± 2.31
→ Luna73.16 ± 4.8182.02 ± 2.43
Kimi K3Capable-only69.91 ± 4.4085.77 ± 1.06
→ Qwen69.91 ± 2.1785.39 ± 1.59
→ Luna73.45 ± 2.8984.64 ± 2.65
Gemini 3.8 FlashCapable-only68.73 ± 2.9285.39 ± 0.92
→ Qwen59.59 ± 2.7486.14 ± 1.40
→ Luna71.39 ± 1.5085.77 ± 1.40
GPT-5.6 LunaCapable-only62.54 ± 2.2179.03 ± 2.80
→ Qwen55.46 ± 2.5477.15 ± 0.53

The repeated experiments broadly support our main finding: well-timed handoffs can substantially reduce cost, while the accuracy trade-off depends on the model pair and benchmark. Based on the three-run means, 17 of 22 handoff settings improve accuracy or incur no more than a 5% relative accuracy loss, compared with 12 of 22 in the original single-run table. All 11 Terminal-Bench settings meet this criterion in the three-run means. On DeepSWE, all five handoffs to Luna meet it, whereas several handoffs to Qwen continue to show substantial losses—for example, GPT-5.6 Sol→Qwen decreases from 70.80% to 55.46%. We will therefore revise our claims to acknowledge these differences rather than imply uniform accuracy preservation.

The repetitions also make the variation between individual executions explicit. For example, GLM’s capable-only DeepSWE accuracy was 61.1% in the original Table 3, compared with a three-run mean of 67.55% and a standard deviation of 4.81 percentage points. These observations reinforce the value of reporting repeated-run results rather than relying on a single execution.

We will update Table 3 with the repeated-run results and clarify that Table 6 uses separate executions of the matching max-effort configurations. Due to time and budget constraints, we could not repeat the full reasoning-effort sweep during the rebuttal period. We will explicitly acknowledge this limitation as well.

The accuracy claim needs a narrower scope

Thank you for pointing this out. We agree that our accuracy-preservation claim is a bit too broad. We will revise it to state that AgentBaton substantially reduces API cost, while the accuracy trade-off depends on the capable–efficient model pair and benchmark. We will explicitly acknowledge settings with substantial accuracy drops, rather than imply that accuracy is preserved across all configurations.

The comparisons leave the detector’s added value uncertain

Thank you for the suggestion. We added a fixed-turn baseline with thresholds chosen to approximately match AgentBaton’s average API cost for each source model and benchmark. Both methods hand off to Qwen, with results below.

SourceBenchmarkFixed turnFixed-turn Acc (%)Fixed-turn Cost ($/task)AgentBaton Acc (%)AgentBaton Cost ($/task)
GLM5.3TB2.13083.150.2780.900.27
GLM5.3DeepSWE3151.330.4858.410.49
KimiK3TB2.12784.270.5786.520.57
KimiK3DeepSWE3370.801.2872.571.28

At comparable cost, AgentBaton achieves higher accuracy in 3 of 4 settings, supporting the benefit of adaptively selecting handoff points.

Using the same training data as AgentBaton, we identify a simple heuristic as an extra baseline: hand off to Qwen at the first detected file edit.

SourceBenchmarkHeuristic Acc (%)Heuristic Cost ($/task)AgentBaton Acc (%)AgentBaton Cost ($/task)
GLM5.3TB2.173.030.073980.900.27
GLM5.3DeepSWE42.480.135158.410.49
KimiK3TB2.184.270.243286.520.57
KimiK3DeepSWE47.790.325472.571.28

The heuristic reduces cost further but lowers accuracy by significantly percentage points relative to AgentBaton, showing the advantage on performance of AgentBaton.

MTRouter’s training and calibration; how are no-clear-handoff trajectories used to train the detector?

We trained MTRouter on 1,500 trajectories from 500 SWE tasks, with three trajectories per task: one successful capable-model trajectory, one efficient-model trajectory, and one trajectory that randomly selects between the two models with equal probability at each turn. We used the official repository’s default single-head Q-function configuration, comprising a frozen Qwen3-Embedding-0.6B encoder and a value head with two 256-unit hidden layers, with a 10% validation split and early stopping.

For trajectories with no clear handoff point, we train AgentBaton to predict NOT_READY at every turn.

Total operating cost, cumulative detector latency, and receiver inference usage per task

We treated local inference cost as zero assuming already-owned hardware, where the additional electricity expense is expected to be small; we acknowledge that this is an approximation. Training is a one-time development cost that users adopting our pretrained detector do not incur, so we exclude it from user-side operating costs. As a practical alternative, we repriced Qwen3.8-27B using Darkbloom’s published API available on OpenRouter, which reflects service overhead and provider margins rather than electricity alone. Based on recorded usage and cache hits, estimated savings remain 54.0%/63.5% for Kimi→Qwen and 57.7%/71.8% for GLM→Qwen on TerminalBench/DeepSWE, respectively. Thus, savings remain possible with a paid receiver.

To complement this receiver-cost analysis, we measured cumulative detector latency across 1,212 trajectories on a single H100 GPU with concurrency 1. The total detector overhead per task averaged 9.15 seconds, with a median of 4.11 seconds and a 95th percentile of 41.09 seconds.

We also measured full online agent-loop latency on 10 tasks per benchmark, with three repetitions. The table below reports mean end-to-end agent-loop time per task.

ConfigurationDeepSWE: mean seconds/taskTerminal-Bench: mean seconds/task
GLM-only944.7107.3
GLM → Qwen714.5121.2
GLM → Luna966.3103.7
Kimi-only1190.2182.6
Kimi → Qwen823.9134.6
Kimi → Luna1094.8131.1

These results show that handoff can reduce overall latency even when it increases the number of model calls. For example, for Kimi → Qwen on DeepSWE, mean total model calls increase from 52.5 to 81.0 (+54.3%), while mean end-to-end agent-loop time decreases from 1190.2 to 823.9 seconds (−30.8%) compared with Kimi-only execution. Kimi → Qwen also reduces mean end-to-end agent-loop time on Terminal-Bench by 26.3%.

Dear Reviewer Tuep,

Thank you again for your time and thoughtful participation in the review and discussion process. We would appreciate hearing whether our responses and revisions have adequately addressed your concerns and questions. If anything remains unresolved, we would be happy to discuss it further.

If you feel that your concerns have been addressed, we would be grateful if you could share your final assessment of our paper or consider updating your original rating/review as you deem appropriate.

Thank you again for your time and consideration.

Reviewer_gwx3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates key design choices for agentic collaboration through model handoffs for long-horizon tasks, where one model takes over from another, focusing on handoff direction and context transfer. The authors study the collaboration potential of capable and efficient models within a single long-horizon agentic task to substantially reduce cost while preserving task accuracy. They also introduce AgentBaton, a lightweight detector that learns when to hand off from recurring patterns in task progress as a means to discover a one-time, simple, well-timed handoff strategy for cost-effective agent collaboration. They show that AgentBaton reduces API costs by 65.9% on average while largely preserving task accuracy, with accuracy gains of up to 8.8 percentage points over capable-only execution.

Critical strengths
  1. Interesting analysis of agentic model-to-model handoffs - The analysis the authors provide on the model-to-model handoffs is interesting, and although in concept is simple, it is decently novel.
  2. Lightweight AgentBaton detector - The proposed AgentBaton detector is a very small model and does not add too much computational overhead.
  3. Good results in terms of cost efficiency - The authors show significant cost reduction by performing handoffs at the points their proposed detector recommends
Critical weaknesses
  1. Limited generalizability - The authors have focused only on SWE tasks which narrows the analysis to a very limited domain. Inclusion of more varied datasets like GAIA[1] and AssistantBench[2] would make the analysis and the proposed component more general.
  2. Limited variety of open weight models - The authors show results/analysis only on Qwen3.8-27B on the open-weight side. More model families like LLaMA and Gemma would add to the analysis
  3. Initial Analysis of handoffs (Table 1) unclear - bold convention is a bit difficult to understand in table 1. Direct comparison with single model baselines would be appreciated as it would give a clear picture of the difference between a single model baseline and a handoff case.
  4. Justification of choice of model for AgentBaton missing - AgentBaton uses Qwen 3.5 2B as a backbone. However this choice is not justified as there are other lightweight models which are available.
  5. Lack of qualitative examples - There are no qualitative examples in the paper. The authors should have included a sample from the benchmark and how the framework navigates the handoff. This would give a great amount of insight into the framework.
  6. Analysis for temporal latency missing - The model efficiency is only in terms of cost. However it is necessary to analyze the model's temporal cost change and effect of handoff on the same.

[1] Mialon, G., Fourrier, C., Wolf, T., LeCun, Y., and Scialom, T. Gaia: a benchmark for general ai assistants. In The Twelfth International Conference on Learning Representations, 2023. [2] Yoran, O., Amouyal, S. J., Malaviya, C., Bogin, B., Press, O., and Berant, J. Assistantbench: Can web agents solve realistic and time-consuming tasks?, 2024.

Questions for authors

Clarification

  1. Does ‘cloud model’ mean ‘proprietary/closed weight model’? (some open-weight models also have cloud versions)

Suggestion

  1. Would multiple handoffs (back-and-forth) between ‘efficient’ and ‘capable’ models be more beneficial in this setting in terms of performance and cost?
  2. Does length of the trajectory make any difference in performance? It would be interesting to see an analysis of number of steps/time vs performance at different handoff points.
5 comments

Thank you for the time and effort devoted to reviewing our work and for the constructive feedback. We appreciate your recognition of our analysis of model-to-model handoffs, the lightweight design of AgentBaton, and the resulting cost savings.

Limited generalizability

Thank you for raising this limitation. In our additional GAIA experiments, the existing AgentBaton detector did not consistently identify suitable handoff points as it did on coding tasks. One possible explanation is that its coding-based training captures transitions such as moving from bug localization to implementation, whereas information gathering and reasoning often remain intertwined in the GAIA tasks we examined. We therefore acknowledge that transfer beyond coding is limited and will narrow the paper’s scope and claims accordingly. But we still believe the findings remain valuable for coding agents, an important application domain.

Limited variety of open weight models

Thank you for the suggestion. We additionally evaluated Gemma 4 31B as an open-weight alternative, with results below. Although Gemma performs substantially worse standalone than Qwen 3.8 27B, AgentBaton handoffs consistently improve over standalone Gemma, without recovering capable-only performance. These results highlight the importance of receiver capability; when alternatives with similar cost are available, we recommend choosing a stronger efficient model.

Terminal-Bench 2.1

Capable modelCapable-onlyAgentBaton → Gemma 4 31BAgentBaton → Qwen 3.8 27B
GLM86.569.780.9
Kimi86.577.586.5
GPT-Sol87.664.082.0
Efficient-only (no handoff)—42.769.7

DeepSWE

Capable modelCapable-onlyAgentBaton → Gemma 4 31BAgentBaton → Qwen 3.8 27B
GLM61.121.258.4
Kimi73.547.872.6
GPT-Sol68.127.460.2
Efficient-only (no handoff)—0.929.2

Initial analysis of handoffs (Table 1) unclear

Thank you for the suggestion. We will add single-model baselines to Table 1 for direct comparison and clarify the boldface convention in the caption more clearly. To clarify, the single model baselines are as follows:

ModelDeepSWE v1.1Terminal-Bench 2.1
Qwen 3.8 27B29.269.7
GLM-5.361.186.5
Kimi K373.586.5
GPT-5.6 Sol68.187.6

Justification of choice of model for AgentBaton missing

Thank you for the suggestion. We targeted an approximately 2B-parameter detector to keep the cost and latency of per-turn handoff decisions low. We selected Qwen 3.5 2B for its compact size and competitive reasoning capabilities. Other possible alternatives like Gemma 4 E2B perform somewhat worse than Qwen 3.5 2B and support a shorter native context window (128K vs. 262K). We will clarify this rationale in the revised paper.

Lack of qualitative examples

Thank you for the suggestion. We will add the following qualitative example to the revised paper.

Example: DeepSWE — testem-per-launcher-reports.

The task is to add separate test reports for each launcher. Opus 5 implements report-file templating and launcher integration. AgentBaton predicts NOT_READY for turns 1–12 and READY at turn 13, after these substantive changes. Qwen then inherits the workspace and full history and completes the task.

  • Turns 1–10: Exploration and baseline checks. Opus inspects the repository, relevant implementations, and tests, then runs the existing tests (64 passed).
  • Turns 11–12: Foundation implementation. Opus implements report-file templating and launcher-name sanitization, integrates these utilities into the launcher, and performs basic checks.
  • Turn 13: Handoff after implementation progress. Opus inspects the configuration setter. With the preceding implementation and observations in its history, AgentBaton predicts READY and transfers control to Qwen to complete the remaining work.
ExecutionVerificationAPI cost
Qwen 3.8 27B aloneFailed$0.00
Claude Opus 5 alonePassed$7.02
Opus → Qwen with AgentBatonPassed$1.13

This example illustrates a successful handoff after the capable model establishes the implementation foundation.

Analysis for temporal latency missing

Thank you for highlighting temporal cost. We have now measured end-to-end agent-loop wall-clock time, rather than detector latency alone. We randomly selected 10 tasks from each of DeepSWE and Terminal-Bench. We ran each task three times under the following configurations: GLM-only, Kimi-only, and the four GLM/Kimi-to-Qwen/Luna AgentBaton configurations.

Each AgentBaton execution starts a fresh capable-model run, performs online handoff detection, and continues with the efficient model in the same task environment using the full available history. Timing includes model requests, tool execution, online detector calls, context conversion, and receiver continuation.

ConfigurationDeepSWE: mean seconds/taskTerminal-Bench: mean seconds/task
GLM-only944.7107.3
GLM → Qwen714.5121.2
GLM → Luna966.3103.7
Kimi-only1190.2182.6
Kimi → Qwen823.9134.6
Kimi → Luna1094.8131.1

The results show that handoff can in general keep or reduce overall latency even when it increases the number of model calls. For example, for Kimi → Qwen on DeepSWE, mean total model calls increase from 52.5 to 81.0 (+54.3%), while mean agent-loop time decreases from 1190.2 to 823.9 seconds (−30.8%). Kimi → Qwen also reduces Terminal-Bench mean time by 26.3%.

We also separately measure the overhead of detector calls, input preparation, and context transfer. Across the four handoff configurations, this overhead averages only 3.3–4.0 seconds per DeepSWE task and 0.7–1.3 seconds per Terminal-Bench task in our online runs, indicating that these operations add little latency.

Does ‘cloud model’ mean ‘proprietary/closed weight model’?

Thank you for pointing this out. By “cloud model,” we mean a model accessed through a remotely hosted service, which can be either proprietary or open-weight.

Would multiple handoffs (back-and-forth) between ‘efficient’ and ‘capable’ models be more beneficial in terms of performance and cost?

Thank you for the suggestion. We already compare against MTRouter, a per-turn routing baseline, in Table 4. AgentBaton matches or exceeds its accuracy in all four evaluated settings while reducing API cost by 46.7–86.6%. In preliminary experiments, we also used GPT-5.6 Sol as an oracle per-turn controller to choose between capable and efficient models. Across several model pairs, this approach yielded lower accuracy and higher API cost than capable-only execution. Although more than 95% of individual routing decisions appeared reasonable upon inspection, this did not translate into better end-to-end accuracy or lower cost. A possible explanation is that switching from the efficient model back to the capable model introduces additional overhead, as shown in the paper, which may offset the benefits of capable-to-efficient handoffs.

Does length of the trajectory make any difference in performance?

Thank you for the suggestion. In addition to our existing handoff-point experiments in Figure 2, we grouped trajectories into short, medium, and long thirds by capable-only model turns within each benchmark × capable model, keeping ties together. We then pooled results across GLM, Kimi, Opus, Gemini, and GPT-Sol:

BenchmarkTrajectory lengthSizeCapable-onlyAgentBaton → QwenAgentBaton → Luna
TB2.1Short15590.396.193.5
TB2.1Medium14984.682.677.9
TB2.1Long14183.774.576.6
DeepSWEShort19565.165.675.4
DeepSWEMedium18666.760.872.6
DeepSWELong18469.662.567.9

Across the four benchmark–receiver settings, AgentBaton preserves or even increases accuracy better on short trajectories, with larger losses relative to capable-only execution on long trajectories.

I appreciate the authors' efforts towards their rebuttal. I am satisfied by the detailed response from the authors and new results providing much required additional insights.

I also respect the authors for acknowledging the limitation in terms of generality and I incline to agree with them, that despite the narrow scope the paper still holds value in terms of contributions.

I am leaning towards raising my score. I do have a couple of suggestions for the authors

  1. Please mention the exact scope and focus of the paper (even if limited) in the abstract/introduction, clearly
  2. I would appreciate if the authors included the additional analysis they have conducted for this rebuttal (more open-weight models, single model baselines, qualitative examples, temporal latency analysis, trajectory length analysis), in their paper. It can be in the main paper or the appendix depending on the importance of the particular analysis and space constraints.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.