- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Backbeat, an execution interface for tool-using agents. Tool calls are treated as futures with launch, observe, resolve and cancel operations. Tools declare which output fields become stable before the call finishes, so the agent can make a decision early and cancel the remaining tool work. The authors evaluate it on PendingBench, a synthetic benchmark of 144 four-option decision tasks, plus a small set of live CLI sessions, generated pytest suites and 12 public SciPy/scikit-learn test files. They report that deadline-correct completion improves from 25.7% to 93.8% over parallel execution, with similar accuracy.
Critical strengths
The idea of using stable partial tool outputs to make earlier decisions is simple and easy to understand.
Critical weaknesses
- The main results mostly come from the task design. PendingBench is fully synthetic, and in 104/144 tasks the deadline is placed between the time when the key fields appear and the time when the tools finish. Also, the main experiment is single-turn and the latency is computed from the schedule, so the agent does not really "keep working while tools run".
- The gains in real settings are small or even negative. In live CLI sessions at the default time scale, Backbeat is slower than parallel execution (15.93s vs. 14.16s). Generated pytest saves only 2.24s, and the real SciPy/scikit-learn tests save only 0.5s.
- The novelty is limited and the sample size is very small. Streaming partial results and cancelling unneeded work are well-known ideas, and most real tools cannot guarantee stable fields. Each live experiment only uses 12 tasks, the gain at scale 0.50 is not significant, and there are no real agent tasks like the popular SWE-Bench/Terminal-Bench style benchmarks.
Questions for authors
- Can you show a full example of a PendingBench task, including the exact prompt and the event stream that the model sees under each interface?
- How are the deadlines chosen? How do the results change if the deadlines are not placed between the stable time and the final time?
- Can you evaluate Backbeat on real agent tasks (e.g., SWE-bench or Terminal-Bench), where tools are not designed for this setting?
- Which real tools can provide the “stable field” guarantee in practice?