acceptodds
Under review as a conference paper at ICLR 2027

Backbeat: Language Agents Keep Working While Tools Run

Abstract

Tool-using language agents commonly suspend reasoning until a call returns. Launching calls in parallel removes serialization but still waits for the slowest final result, even when stable partial output already determines the next action. We introduce Backbeat, a model-agnostic execution interface that represents tool calls as typed futures with launch, observe, resolve, and cancel operations. Tools declare which fields are stable before completion; the agent can reason over these observations, finish when its decision predicate is satisfied, and cancel irrelevant tails. We evaluate Backbeat on PendingBench: 144 fresh tasks across six tool families, three model families, and 1,728 controlled trajectories, followed by complete agent sessions, generated concurrent pytest suites, and unmodified public-package tests. Against parallel-final execution, Backbeat reduces mean virtual evidence-ready latency from 35.21 to 21.56 seconds and raises deadline-correct completion from 25.7% to 93.8%, while ordinary accuracy remains 95.1% versus 95.4%. Even an optimistic adaptive final-only replay remains 11.69 seconds slower and 53.5 deadline points lower. The mechanism is visible in the schedule: 104/144 tasks place the deadline inside a stable tail averaging 13.65 seconds. In complete agent sessions at duration scale 1.0, Backbeat saves 14.47 seconds [11.05, 18.53] and wins all 24 paired runs. Through generated suites executed by pytest 9.1.1, it saves 2.24 session seconds [0.08, 4.38] and 5.46 seconds of executed suite service, with every task answer preserved. On 12 upstream SciPy and scikit-learn files, stable target-test reports precede full-suite completion in all 12 pairs, saving 0.50 tool seconds [0.16, 1.06] at zero API cost. The 1,980-trajectory campaign costs $2.56. Pending calls are not dead time: they are a useful test-time reasoning substrate.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
language agents, tool use, asynchronous execution, futures, partial results, latency, agent runtimes

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_whHQOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Backbeat, an execution interface for tool-using agents. Tool calls are treated as futures with launch, observe, resolve and cancel operations. Tools declare which output fields become stable before the call finishes, so the agent can make a decision early and cancel the remaining tool work. The authors evaluate it on PendingBench, a synthetic benchmark of 144 four-option decision tasks, plus a small set of live CLI sessions, generated pytest suites and 12 public SciPy/scikit-learn test files. They report that deadline-correct completion improves from 25.7% to 93.8% over parallel execution, with similar accuracy.

Critical strengths

The idea of using stable partial tool outputs to make earlier decisions is simple and easy to understand.

Critical weaknesses
  1. The main results mostly come from the task design. PendingBench is fully synthetic, and in 104/144 tasks the deadline is placed between the time when the key fields appear and the time when the tools finish. Also, the main experiment is single-turn and the latency is computed from the schedule, so the agent does not really "keep working while tools run".
  2. The gains in real settings are small or even negative. In live CLI sessions at the default time scale, Backbeat is slower than parallel execution (15.93s vs. 14.16s). Generated pytest saves only 2.24s, and the real SciPy/scikit-learn tests save only 0.5s.
  3. The novelty is limited and the sample size is very small. Streaming partial results and cancelling unneeded work are well-known ideas, and most real tools cannot guarantee stable fields. Each live experiment only uses 12 tasks, the gain at scale 0.50 is not significant, and there are no real agent tasks like the popular SWE-Bench/Terminal-Bench style benchmarks.
Questions for authors
  1. Can you show a full example of a PendingBench task, including the exact prompt and the event stream that the model sees under each interface?
  2. How are the deadlines chosen? How do the results change if the deadlines are not placed between the stable time and the final time?
  3. Can you evaluate Backbeat on real agent tasks (e.g., SWE-bench or Terminal-Bench), where tools are not designed for this setting?
  4. Which real tools can provide the “stable field” guarantee in practice?
Reviewer_apC3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether language agents must wait for final tool outputs when calls are still pending but their partial results already suffice for decision-making. To address this, the authors propose BACKBEAT, which represents tool calls as observable futures: tools can expose declared-stable fields before resolution, allowing agents to answer once decision readiness is reached and cancel remaining computation irrelevant to the decision.

The authors conduct controlled comparisons on their synthetic PENDINGBENCH benchmark, live CLI sessions, generated pytest suites, and public-package tests. In the controlled evaluation, BACKBEAT reduces virtual evidence-ready latency from 35.21 to 21.56 seconds and substantially improves deadline-correct completion while largely preserving ordinary accuracy. The live experiments further show that its benefit depends on whether tool tails are sufficiently long to offset the additional observation and orchestration overhead. Overall, the experiments support the effectiveness and efficiency of BACKBEAT and validate the proposed mechanism of using stable partial observations to enable earlier decision-making.

Critical strengths
  1. The research question is interesting and meaningful. Tool calls are an important source of latency in agentic systems, making it worthwhile to investigate how stable partial outputs can be used before final results become available. Avoiding irrelevant final tails may not only accelerate task completion but also reduce the context processed by subsequent model calls.
  2. The experiments clearly analyze the distinct sources of improvement. The paper separately reports task completion time and executed tool service, effectively distinguishing the effects of using partial results early from those of canceling subsequent work.
  3. The method is simple and easy to follow. BACKBEAT describes the tool-call lifecycle using only LAUNCH, OBSERVE, RESOLVE, and CANCEL, and the workflow is intuitive.
Critical weaknesses
  1. BACKBEAT relies heavily on predefined stable fields and evidence-sufficiency conditions, limiting its applicability to realistic agent workflows. Real-world tasks may involve tens or hundreds of interdependent tool calls whose outputs and relevance evolve with the execution state, making fixed decision predicates and cancellation conditions difficult to define. Manual configuration or additional LLM judgments introduce costs and errors: premature stopping may remove information needed for later planning, while conservative stopping may eliminate the latency benefit. The paper does not evaluate these risks or validate the method on long-horizon, multi-turn tasks.
  2. The reported timing gains are difficult to interpret because both the synthetic and live evaluations have insufficiently controlled time factors. PENDINGBENCH does not fully specify the joint generation of stable time, final time, cross-check, and deadline, even though the main results directly depend on these constructed windows. Meanwhile, live CLI and pytest measurements may vary with machine load, network conditions, API queuing, and provider latency; this is particularly concerning because BACKBEAT is 1.77 seconds slower in the default setting and saves only 2.24 seconds on pytest. The authors should disclose the full benchmark-generation procedure, randomize execution order, repeat runs, and separately report tool, model, polling, and orchestration time.

Minor Weakness: The paper has an imbalanced and unclear presentation: Figures 2 and 3 occupy excessive space, while page 7 contains substantial blank space, leaving the effective main text relatively short. Moreover, benchmark construction, experimental settings, and evaluation protocols are mixed within the PENDINGBENCH section; task formulation, data generation, stable fields, and label construction should be separated from model selection, runners, batching, and duration-scaling details.

Questions for authors

See Weaknesses.

Reviewer_Z79kOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

In this work, the authors explore different strategies for tool-calling as a source of latency in agentic workloads. They identify the agent waiting for multiple tool-calls to finish either in seuqential or parallel settings contributes significantly to this latency. To address this, they propose backbeat, which represents tool-calls as typed futures where a tool adapter can declare partial outputs as stable and can be reasoned over. If the objective is met, then unresolved tool-calls are cancelled. The authors evaluate on PendingBench and live CLI session across different models. They report gains in deadline-correct completion with roughly unchanged accuracy compared to parallel-final tool-calling.

Critical strengths
  • The motivation is clear and relevant, as tool latency is a bottleneck in agentic workloads.
  • The proposed technique is simple and model agnostic.
  • The papers evaluates across different model familities.
Critical weaknesses
  • The mathematical notation introduced in section 2.1 is not re-used anywhere else and it is not clear how the quantities defined drive evaluation.
  • The stability criterion is not defined, as the authors say that "the tool-adapter names fields that become semantically stable" but don't explain why and how this was determined.
  • In table 4, it is shown that backbeat has worse mean session length and only a small improvement in median, with higher costs than either blocking and parallel-final.
  • Terms are used before their definition. For example 'probe' line 37, 'opportunity' (line 174), 'tool adapter' (line 122).
  • There are insufficient details provided regarding the implementation details, how the data is processed of bbtool and the how the typed futures are implemented.
Questions for authors
  • In line 19, what are ' six tool families' (line 19) and 'six equal families' (line 152) which are mentioned only once in the entire paper?
  • What is a probe as defined in this paper - is it a parallel tool call or a parallel experiment?
  • Could you explain how is the tool adapter implemented? How are its criterion selected, and how does it evaluate them in the outputs?
  • How are the typed futures implemented?
  • How is a duration-scale computed? What does a scale 1.0 mean in wall-clock terms and what does it mean to compress it?
  • Why are the results pooled across models and why is the per-model table in the appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.