acceptodds
Under review as a conference paper at ICLR 2027

RUBATO: A Run-Ahead Asynchronous Compression Framework for Agentic RAG

Abstract

Agentic retrieval-augmented generation interleaves reasoning and retrieval, causing the model input to grow throughout a request. Prompt-level context compression limits this growth, yet existing agent pipelines place each compression task between two agent steps, blocking subsequent reasoning or retrieval. In our measurements, compression under synchronous execution accounts for 43%–54% of single-request service time. Longer service times increase worker occupancy and queueing under concurrent serving. We present Rubato, a run-ahead asynchronous framework that removes trajectory compression from the request's critical path. A context-construction rule (CCR) combines the latest committed compressed context with subsequent uncompressed steps, allowing the agent to advance while compression runs in the background. A prefix-monotone criterion (PMC) accepts a result only for an active request and only when it extends the prefix covered by the committed context, preserving consistent model inputs despite late or out-of-order completions. Across four compression methods, six question-answering datasets, and three agent backbones, Rubato reduces dataset-averaged mean single-request latency by – and median job completion time under concurrent serving by – while preserving average answer quality. These results support treating compression placement as a serving decision separate from the compression method. Our code are openly accessible to support future research.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM agents, LLM serving, context compression, asynchronous execution, retrieval-augmented generation, inference systems

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xmr7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RUBATO moves trajectory compression off the critical path of agentic RAG. While compression runs in the background, the agent uses the latest committed compressed prefix plus the subsequent raw steps. A prefix-based commit rule handles late results, while cancellation, per-request isolation, pacing, and load-adaptive admission manage contention. The framework requires no additional training or changes to the compressors. Across four compressors, six QA datasets, and three agent backbones, the paper reports 1.69–2.32× lower mean single-request latency and 2.17–2.71× lower median job completion time under concurrency, with non-inferior average answer quality according to an LLM judge.

Critical strengths
  1. A consequential serving bottleneck with substantial measured gains. The paper measures both single-request latency and concurrent completion time, helping separate savings from removing blocking compression from reductions in queueing. Gains across compressors and backbones support the practical significance of compression placement.
  2. Careful treatment of asynchronous execution. The context and commit rules address late or out-of-order results, while the orchestration rules address wasted work and contention. These details make the proposal more useful than simply running compression in a background thread.
Critical weaknesses
  1. The incremental novelty over a simple asynchronous baseline is unclear. Run-ahead execution, version checks, and task scheduling are established systems techniques. Although the paper includes component ablations, it does not clearly show how much the full framework gains over a reasonably implemented baseline with asynchronous compression, prefix versioning, and basic cancellation. This matters to my assessment of its methodological contribution.
  2. The limits of the gains are insufficiently characterized. Background compression still consumes compute, and carrying a longer raw suffix increases model input cost; the paper itself identifies cases where this outweighs the benefit. The quality claim also relies on one LLM judge and QA workloads. These results establish benefits in the tested setting, but leave total resource cost and broader applicability uncertain.
Questions for authors
  1. Under identical triggers and hardware, how does RUBATO compare with a minimal asynchronous baseline using prefix version checks and basic cancellation? Which additional components account for the largest gains?
  2. How do total compression compute, agent prefill cost, and GPU time per request change relative to synchronous compression?
1 comment

We thank the reviewer for the constructive feedback and the precise questions.

Q1. Comparison with a minimal asynchronous baseline, and which components matter most.

Rubato is 5.43× faster than SC in median JCT, against 3.42× for a minimal asynchronous baseline with basic cancellation. Moving compression off the critical path gives the largest gain, and pacing and adaptation add the most among the remaining components.

  1. Under identical triggers and hardware, we start from asynchronous compression with the prefix version check and one shared FIFO compression lane for all requests, and add Rubato's components one at a time. We report FoldAgent at high load (ρ=1.5, 2Wiki, 60 questions), where requests contend for compression.
Configurationp50 JCT (s)Speedup over SCQueue (s)Judge
SC324.71.00×195.30.55
Minimal async (version check, shared lane)110.82.93×39.10.55
+ Cancellation94.93.42×33.60.60
+ Per-request isolation80.64.03×26.60.55
+ Pacing and adaptation (Rubato)59.85.43×16.90.58
  1. Each component shortens queueing, which falls from 39.1 s to 16.9 s. Cancellation drops compressions whose prefix is already stale, isolation keeps one request's compression backlog from delaying another request's commit, and pacing launches a compression only once enough new context has accumulated.
  2. With ACON, requests barely queue at this load, at 2.1 s in the minimal baseline. Its gain comes from moving compression off the critical path and from pacing and adaptation, and Rubato reaches a median JCT of 41.4 s against 415.0 s for SC.

Q2. Total compression compute, prefill cost, and GPU time per request.

Rubato leaves total GPU time per request unchanged at 0.99× of SC, while cutting JCT to 0.60×. Rubato therefore changes only how compression and agent inference are orchestrated, without reducing the inference that the task requires.

Quantity per request, mean at ρ=0.6Rubato / SC
Compression GPU time0.76×
Agent generation GPU time1.44×
Compression plus generation GPU time0.99×
Prompt tokens (prefill volume)2.24×
JCT0.60×
  1. Compression GPU time falls to 0.76× because cancellation discards compressions that would be stale on arrival.
  2. Prompt tokens rise to 2.24× because the raw suffix is carried until a compression commits, which shows up as 1.44× longer generation.
  3. The two offset each other, so total GPU time stays at 0.99×.

W1, gain over a simple asynchronous baseline. Q1 above gives this comparison directly. Against a baseline with asynchronous compression, prefix versioning, and basic cancellation, Rubato raises the speedup over SC from 3.42× to 5.43×. We will add this experiment to the appendix.

W2, single judge and QA workloads. Quality holds question by question, not only on average. Rubato and synchronous compression reach the same judgement on 87% of questions (response to TCJc, W2). The quality result does not depend on the LLM judge. Against gold answers, Rubato averages 0.371 exact match and 0.499 F1, against 0.360 and 0.488 for synchronous compression. On AppWorld, task success is decided by the benchmark's own programmatic evaluator. Beyond QA, on the AppWorld coding agent Rubato is 3.62× faster than synchronous compression in mean JCT and raises task success from 33% to 50% (response to TCJc, Q1).

Reviewer_3tVpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Rubato removes prompt-level trajectory compression from the agentic RAG. Instead of waiting for compression between steps, the agent proceeds using the latest committed compressed context plus the raw steps generated since. Background compression results are committed only if the request is still active and the result covers a longer prefix than the current one, so late or out-of-order results can't corrupt the context. Four orchestration rules (cancellation, per-request FIFO isolation, threshold-based pacing, load-adaptive thresholds) control contention from background tasks. Across 4 compressors, 6 QA datasets, and 3 ~30B backbones, Rubato reports 1.69–2.32× lower single-request latency, 2.17–2.71× lower median concurrent JCT, and non-inferior LLM-judge quality.

Critical strengths
  1. It speeds up existing compressors without changing them. Rubato works with four structurally different compression methods on three backbones, with no retraining and no changes to prompts, models, or inference engine. A practitioner could adopt it today, which makes the contribution practical rather than tied to one method. The consistency of the gains across this grid, including honestly reported failure cases, makes the result credible.

  2. It reframes compression placement as a serving decision, and explains when that decision pays off. The paper shows that when compression runs matters separately from how it compresses. Its simple latency model predicts which methods benefit most and when asynchrony stops helping, and the experiments match those predictions. This gives practitioners a clear rule for when to use the approach, not just a benchmark win.

Critical weaknesses
  1. The evaluation doesn't test the setting where compression matters. Compression exists to keep long-horizon agents within their context budget, but these QA trajectories are short enough that running without compression is often competitive or better. In this regime, Rubato's speedup may come simply from compressing less and carrying more raw context, not from smarter scheduling. The paper never measures context length, so it can't rule this out. It also leaves open whether Rubato still works when the context limit actually binds, which is the case that justifies the framework.

  2. The concurrent gains may come from the experimental setup rather than the method. Synchronous compression is penalised twice: it holds a worker slot while compressing, and it is run at loads far beyond saturation, where small service-time differences turn into very large queueing differences. Rubato, meanwhile, runs more total work on the GPU in parallel. Without baselines that match concurrency and compression frequency, the headline 2–3× concurrent improvement can't be attributed to asynchrony itself.

Questions for authors
  1. Is Rubato faster because it compresses asynchronously, or because it compresses less? Pacing and adaptation reduce how often compression runs, and the raw suffix keeps more uncompressed context. A synchronous baseline that compresses equally often would tell us how much of the gain comes from asynchrony itself. If most of it does not, the central claim weakens considerably.

  2. Does the approach hold up when context length is the real constraint? Rubato lets uncompressed context grow while compression catches up, and under heavy load it delays compression further. How large do model inputs get compared to synchronous compression, and what happens when they approach the context window? This determines whether Rubato is usable in the long-horizon settings that motivate compression in the first place.

  3. Is the concurrent comparison fair to synchronous compression? Synchronous compression is evaluated at loads well past saturation, with fewer effective parallel calls than Rubato. Would the advantage survive if both placements ran at matched concurrency and the comparison were made at a realistic latency target? The answer determines whether the headline 2–3× concurrent improvement reflects the method or the setup.

1 comment

We thank the reviewer for the insightful questions.

Q1. Faster because asynchronous, or because it compresses less?

Rubato is faster because compression leaves the critical path, not because it compresses less. At the same compression frequency, it is still 1.21× faster on average with the same quality.

  1. We built a frequency-matched synchronous baseline. For every question, it runs exactly as many compressions as Rubato committed on that question, triggered at the first eligible points. It has hindsight no deployable system has. Rubato is 1.21× faster on average, and its judge score differs from the baseline by 0.02.
JCT speedup over frequency-matched SC2WikiHotpotQAMuSiQue
FoldAgent1.55×0.94×1.35×
ACON1.16×1.28×1.20×
MemAgent1.11×1.42×1.34×
ReSum1.00×0.97×1.19×
  1. The remaining gap is the compression time taken off the critical path. Per question, the matched baseline spends 34.4 s blocked on compression, all of which Rubato removes. Rubato pays back 12.1 s in longer generation, because its inputs carry the raw suffix until a compression commits. Queueing differs by only 1.0 s. The net gap is 22.6 s.

Q2. Does it hold when context length is the real constraint?

Yes, and the gain is larger there. When context is the binding constraint, the speedup on MuSiQue rises from 1.61× to 2.43×, at equal or higher judge scores. Rubato's speedup therefore does not come from slack in the context budget.

  1. We imposed a hard 4,000-token input limit on MuSiQue, which has the longest trajectories, truncating any input above it from the oldest raw steps. A tight limit forces synchronous compression to run more often, and every run blocks the agent. Rubato runs the same compressions off the critical path.
CompressorSC JCT, limit (s)Rubato JCT, limit (s)Speedup, limitSpeedup, no limitJudge, limit, SC → Rubato
FoldAgent672.7289.92.32×1.54×0.375 → 0.475
ACON480.5239.32.01×1.76×0.425 → 0.450
MemAgent664.2212.23.13×1.59×0.325 → 0.400
ReSum584.6257.42.27×1.56×0.450 → 0.450
Average2.43×1.61×0.394 → 0.444
  1. Rubato does not exploit slack when the limit is loose. At the same compression frequency as the matched baseline in Q1, its mean input is only 1.24× longer, because the uncompressed suffix covers only the steps taken while a compression runs. Its largest input is 8,195 tokens, one quarter of the 32,768-token window and below the matched baseline's largest input of 10,298 tokens. Under heavy load (ρ=1.5), where compression lags the most, Rubato's largest input is 7,112 tokens, about one fifth of the window.

Q3. Is the concurrent comparison fair to synchronous compression?

Yes. The gain does not come from overload or from extra parallelism. Rubato is 1.91× faster below saturation, 2.03× faster at matched concurrency, and sustains 3.02× the arrival rate of SC at a realistic latency target.

  1. Below saturation, the whole gain is service time. We split the paper's concurrent experiments at an effective utilization of 0.8. Below this point, queueing accounts for 2% of SC's JCT and none of Rubato's, and the JCT speedup of 1.91× matches the service-time speedup of 1.86×. Above it, the service-time speedup stays at 1.87×, and the JCT speedup rises to 2.47× only because queueing grows to 34% of SC's JCT against 15% of Rubato's.
CompressorBelow sat., JCTBelow sat., serviceAbove sat., JCTAbove sat., serviceAbove sat., queue share SC → Rubato
FoldAgent1.86×1.83×2.25×1.70×33% → 14%
ACON2.20×2.14×2.84×2.07×39% → 16%
MemAgent1.76×1.75×2.34×1.85×32% → 17%
ReSum1.89×1.81×2.41×1.83×31% → 13%
Average1.91×1.86×2.47×1.87×34% → 15%
  1. At matched concurrency, Rubato is 2.03× faster. We placed agent calls and compression calls under one shared limit of 8 concurrent calls, equal to the number of workers SC uses, at the paper's ρ=0.6 arrival rates. Rubato also does not use more GPU time, at 0.99× of SC per request (response to xmr7, Q2).
Matched concurrencySpeedup over SC
FoldAgent, 2Wiki1.80×
FoldAgent, MuSiQue1.97×
ACON, 2Wiki2.32×
ACON, MuSiQue2.03×
Average2.03×
  1. At a latency target that keeps median JCT within SC's closed-loop mean service time, Rubato sustains 3.02× the arrival rate of SC.
CompressorFoldAgentACONMemAgentReSumAverage
Sustained arrival rate, Rubato / SC3.53×3.64×2.63×2.40×3.02×

W1 and W2. The results above answer both points, and we will add them to the appendix. Rubato does not change the compressor, so it keeps what compression offers, such as a more concise context, and removes only its latency.

Reviewer_TCJcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduced one framework to accelerate agentic RAG through async context compression. Previous methods pause the model generation, tool execution while compressing the interaction history. The propose Rubato runs compression in the background. The major technical contribution is a scheduling framework that decomposes compression from other RAG operations with four orchestration rules to control resource allocation. Across amultiple QA datasets and three models, the proposed methods achieved large speedup in single request latency and median completion time under concurrent serving.

Critical strengths
  1. The paper identified important problems and performance bottleneck, which is common in the agentic serving especially for long context compression.

  2. A comprehensive framework with four specific designed orchestration rulels to reuse exsiting compression models without training. the cost is low and it can maintain the comparable performance with sync settings.

  3. Clearly analyze the trade-off between less compression waiting time and more compression overhead. It proposes the future directions for further optimization.

Critical weaknesses
  1. the experiment scale is limited and it only covers single agent with question answering tasks. For commonly used agents like coding agent with long horizon, it is not sure about the performance gains from async compression.

  2. context consistency does not guarantee semantic preservation. PMC prevents stale summaries from overwriting newer ones, but it does not ensure that summaries retain all critical evidence. comparable average performance does not mean no significant degradation on individual taks.

Questions for authors
  1. How to extend the proposed framework to more general agent settings or multi-agent serving scenarios? Have you compared with latest agent serving system?
1 comment

We thank the reviewer for the careful reading and the suggestions on broader agent settings.

Q1. Extension to general agents and multi-agent serving, and comparison with agent serving systems.

General agents. Rubato extends to general agents without change, and on a coding agent it is 3.62× faster than synchronous compression in mean JCT, with a higher task success rate.

  1. Rubato relies only on three properties of the agent loop, none of which is specific to question answering. The agent appends steps to a trajectory, a compressor maps a prefix of that trajectory to a shorter context, and the next model call can proceed on the latest committed prefix plus the raw steps after it. Coding agents satisfy all three, so extending Rubato to them requires only registering the compressor and taking each step boundary as the commit point.
  2. We ran this on AppWorld, where the agent writes Python against app APIs and reads the execution output. We served AppWorld's official ReAct code agent with FoldAgent (Qwen3.6-27B, test-normal split, 24 tasks, ρ=0.6, up to 25 steps per task). As in the paper, every step is a compression unit.
AppWorld, FoldAgentSCRubatoSpeedup
Mean JCT (s)6191713.62×
p50 JCT (s)6361574.06×
Task success33%50%
  1. The gain is larger than on QA because code-agent steps are long. Synchronous compression blocks each task for 399 s on average, two thirds of its service time, and Rubato removes all of it from the critical path.

Multi-agent serving. Rubato extends directly to multi-agent serving, where it is 2.14× faster than SC in mean JCT with a higher judge score. Each agent keeps its own context, so each agent maps to one Rubato request. An orchestrator splits each question into up to three sub-questions per round for up to three rounds, and sub-agents answer them in parallel, each with its own FoldAgent compressor. Four tasks are in flight at ρ=0.6 (2Wiki, 24 tasks).

SCRubatoSpeedup
Mean JCT (s)207.297.02.14×
p50 JCT (s)197.276.72.57×
Judge0.540.58

Every orchestrator round waits for its slowest sub-agent, so synchronous compression on any one sub-agent delays the whole task. Rubato removes compression from the critical path of every sub-agent at once.

Agent serving systems. Rubato is complementary to agent serving systems, and it remains 1.5–2.4× faster under Autellix's scheduling policy.

  1. Autellix schedules calls using program-level information, Parrot exposes semantic dependencies between calls to the engine, and InferCept manages engine state across tool interruptions. Each takes the dependency structure of the program as given. Rubato changes that structure, since compression no longer precedes the next step.
  2. We implemented Autellix's program-level least-attained-service policy (PLAS) in the engine, with compression calls charged to their program, and served SC and Rubato under it (FoldAgent, ρ=0.6, 40 questions).
DatasetSC + PLAS, mean JCT (s)Rubato + PLAS, mean JCT (s)Speedup
2Wiki189.277.72.44×
MuSiQue282.2190.71.48×
  1. CoMem and AgentInfer also overlap compression with agent execution. CoMem achieves this with a separately trained memory model and AgentInfer with a co-designed inference system. Rubato reaches the same overlap with each published compressor unchanged and without modifying the inference engine, which is what allows it to apply across four structurally different compressors.

W1, coding agents. Rubato's gain carries over to coding agents. On the AppWorld coding agent (Q1 above), Rubato is 3.62× faster than synchronous compression in mean JCT and raises task success from 33% to 50%. Under a hard 4,000-token context limit (response to 3tVp, Q2), Rubato is 2.43× faster on average, with equal or higher judge scores. We will add both experiments to the appendix.

W2, semantic preservation per task. Rubato introduces no systematic per-question degradation. The commit rule guarantees consistency, and what each summary retains is determined by the unchanged compressor, a distinction we will state explicitly. Compared question by question across all compressors and datasets, Rubato and synchronous compression reach the same judgement on 87% of questions, and the remaining disagreements go in both directions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.