- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
RUBATO moves trajectory compression off the critical path of agentic RAG. While compression runs in the background, the agent uses the latest committed compressed prefix plus the subsequent raw steps. A prefix-based commit rule handles late results, while cancellation, per-request isolation, pacing, and load-adaptive admission manage contention. The framework requires no additional training or changes to the compressors. Across four compressors, six QA datasets, and three agent backbones, the paper reports 1.69–2.32× lower mean single-request latency and 2.17–2.71× lower median job completion time under concurrency, with non-inferior average answer quality according to an LLM judge.
Critical strengths
- A consequential serving bottleneck with substantial measured gains. The paper measures both single-request latency and concurrent completion time, helping separate savings from removing blocking compression from reductions in queueing. Gains across compressors and backbones support the practical significance of compression placement.
- Careful treatment of asynchronous execution. The context and commit rules address late or out-of-order results, while the orchestration rules address wasted work and contention. These details make the proposal more useful than simply running compression in a background thread.
Critical weaknesses
- The incremental novelty over a simple asynchronous baseline is unclear. Run-ahead execution, version checks, and task scheduling are established systems techniques. Although the paper includes component ablations, it does not clearly show how much the full framework gains over a reasonably implemented baseline with asynchronous compression, prefix versioning, and basic cancellation. This matters to my assessment of its methodological contribution.
- The limits of the gains are insufficiently characterized. Background compression still consumes compute, and carrying a longer raw suffix increases model input cost; the paper itself identifies cases where this outweighs the benefit. The quality claim also relies on one LLM judge and QA workloads. These results establish benefits in the tested setting, but leave total resource cost and broader applicability uncertain.
Questions for authors
- Under identical triggers and hardware, how does RUBATO compare with a minimal asynchronous baseline using prefix version checks and basic cancellation? Which additional components account for the largest gains?
- How do total compression compute, agent prefill cost, and GPU time per request change relative to synchronous compression?
We thank the reviewer for the constructive feedback and the precise questions.
Q1. Comparison with a minimal asynchronous baseline, and which components matter most.
Rubato is 5.43× faster than SC in median JCT, against 3.42× for a minimal asynchronous baseline with basic cancellation. Moving compression off the critical path gives the largest gain, and pacing and adaptation add the most among the remaining components.
Q2. Total compression compute, prefill cost, and GPU time per request.
Rubato leaves total GPU time per request unchanged at 0.99× of SC, while cutting JCT to 0.60×. Rubato therefore changes only how compression and agent inference are orchestrated, without reducing the inference that the task requires.
W1, gain over a simple asynchronous baseline. Q1 above gives this comparison directly. Against a baseline with asynchronous compression, prefix versioning, and basic cancellation, Rubato raises the speedup over SC from 3.42× to 5.43×. We will add this experiment to the appendix.
W2, single judge and QA workloads. Quality holds question by question, not only on average. Rubato and synchronous compression reach the same judgement on 87% of questions (response to TCJc, W2). The quality result does not depend on the LLM judge. Against gold answers, Rubato averages 0.371 exact match and 0.499 F1, against 0.360 and 0.488 for synchronous compression. On AppWorld, task success is decided by the benchmark's own programmatic evaluator. Beyond QA, on the AppWorld coding agent Rubato is 3.62× faster than synchronous compression in mean JCT and raises task success from 33% to 50% (response to TCJc, Q1).