acceptodds
Under review as a conference paper at ICLR 2027

TraSURE: Exploiting Cross-trajectory Recurrence to Accelerate Information-seeking RL Rollouts over Shared Corpora

Abstract

Training agents with reinforcement learning (RL) to answer questions over shared corpora, such as enterprise knowledge bases, incurs substantial rollout cost. We identify cross-trajectory recurrence in this setting: trajectories for the same or different queries retrieve the same external items under different contexts, causing repeated prefill that group-based sampling further amplifies. However, this recurrence remains unexplored for rollout acceleration. We present TraSURE, which exploits this recurrence to accelerate RL rollouts over shared corpora. (1) For prefill, we use position-independent caching (PIC) to share recurring-item KV states across trajectories and avoid redundant prefills. (2) Building on this caching layer, we design demand-aware rollout scheduling to coordinate cache admission and retention using current and future item demand. (3) Beyond prefill, we use a next-outcome posterior to select historical continuations as drafts for posterior-guided speculative decoding. Across six models and four information-seeking workloads over shared corpora, TraSURE delivers up to speedup over standard rollouts, with all task scores within 1 pp of full-causal counterparts.

Primary area:
reinforcement learning
Keywords:
agentic reinforcement learning, information-seeking agents, rollout acceleration, cross-trajectory recurrence, KV-cache reuse, position-independent caching, speculative decoding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JgtnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies RL training for agents that retrieve information from a shared corpus. Different trajectories often read the same documents, but ordinary prefix caching cannot reuse their KV states when the preceding contexts differ. TraSURE encodes retrieved items independently and trains the policy with the same attention rule, allowing those states to be shared across trajectories. It also schedules requests around document reuse and uses historical continuations for speculative decoding. Experiments on text and multimodal tasks report up to 6.01× rollout acceleration and 2.16× full-RL acceleration, with final task scores within one percentage point of independently trained full-causal policies.

Critical strengths
  1. Clear motivation: Shared-corpus rollouts offer substantial document reuse beyond prefix caching. Using RL to adapt the policy to independently encoded documents is a useful part of the design.
  2. Substantial gains: The paper reports improvements in both rollout and full-RL time against a radix-caching baseline, with ablations supporting the additional components.
Critical weaknesses

Narrow applicable setting: Because cached KV states are invalidated at every policy update, reuse is confined to the trajectories of a single RL step, and the paper demonstrates this confined setting only on four fixed benchmark corpora. Whether the speedup survives on larger corpora, where items recur less often within a step, is not established.

Questions for authors
  1. How does the speedup vary with corpus size and item-access skew? A smaller benchmark, such as a restricted corpus, would clarify whether the gains hold when items recur less often.
3 comments
AuthorsOpenReview ↗

Response to Reviewer Jgtn (1/2)

Thank you very much for your review, and in particular for the point about recurrence. Both of our answers below are about it.

Weakness 1 (W1): Narrow applicable setting: Because cached KV states are invalidated at every policy update, reuse is confined to the trajectories of a single RL step, and the paper demonstrates this confined setting only on four fixed benchmark corpora. Whether the speedup survives on larger corpora, where items recur less often within a step, is not established.

Answer: You are right that reuse is confined to one RL step. This holds for any on-policy method: once the policy is updated, KV computed under θv\theta_v no longer matches it, and that includes the radix cache of vanilla slime. So what matters is how much reuse there is inside one step, and it is large. Trajectories of the same prompt, and often of different prompts, retrieve the same items. Vanilla slime recomputes an item for every trajectory that meets it under a different prefix. PICache computes it once per step unless memory pressure evicts it, which alone gives 1.74×–2.97× (Table 2). Scheduling then makes sure each item is computed once in the new step and read by the requests that follow, adding 1.16×–1.47× from current demand and a further 1.01×–1.19× from lookahead (Table 2).

As for larger corpora, the speedup depends on how many trajectories in a step draw on the same items, more than on the size of the corpus. We measured this inside single RL steps, with only the prompts each step samples (table below, replay traces of Table 1). Most repeated item accesses come from trajectories of the same prompt. This part exists for any corpus, because the G=16G=16 trajectories of a prompt answer the same question. The rest comes from the other prompts in the same step and follows the access skew of the workload (Q1).

WorkloadSame-prompt share of repeated item accesses within a step
MultiHop-RAGQwen3-4B 51.3%, Qwen3-8B 57.8%, Qwen3-14B 52.1%, Llama-3.1-8B 63.6%
MHDocVQAQwen3-Omni 59.4%, MiniCPM-o 4.5 68.9%
AVHaystacksQwen3-Omni 64.2%, MiniCPM-o 4.5 76.5%
CiteVQAQwen3-Omni 73.7%, MiniCPM-o 4.5 79.3%
AuthorsOpenReview ↗

Response to Reviewer Jgtn (2/2)

Question 1 (Q1): How does the speedup vary with corpus size and item-access skew? A smaller benchmark, such as a restricted corpus, would clarify whether the gains hold when items recur less often.

Answer: For the first part of your question, access skew, the four workloads in the paper already differ widely (Fig. 10):

WorkloadShare of rollout accesses on the top 10% of itemsReplay rollout speedup (Table 1)
MultiHop-RAG54%3.49×–5.82×
MHDocVQA45%2.74×–4.08×
AVHaystacks27%3.09×–4.41×
CiteVQA23%3.16×–3.66×

Skew helps. It is a property of shared-corpus workloads: when many questions draw on the same few items, reuse also happens across prompts, and the most skewed workload, MultiHop-RAG, has the highest speedup, up to 5.82×. However, the gain does not rest on skew alone. The least skewed workload, CiteVQA, still reaches 3.16×–3.66×, because most of its repeated accesses come from trajectories of the same prompt (74%–79%, table in W1), and that reuse is there whatever the skew.

For the second part, corpus size, we have not varied the corpus itself. Your restricted-corpus test is a way to see whether the gains hold when items recur less often. We tested the extreme version of that: each PICache entry is tagged with the prompt that created it, and a request may reuse an entry only if it carries the same tag. Every access from another prompt is recomputed, as if no two questions shared any item. This removes everything that depends on questions sharing items, which is the part a larger corpus would reduce.

WorkloadModelRollout, all sharing (Table 1)Rollout, same-prompt onlyFull RL, all sharing (Table 1)Full RL, same-prompt only
MultiHop-RAGQwen3-4B5.08×2.29×1.98×1.53×
MultiHop-RAGQwen3-8B4.82×2.45×1.79×1.49×
MultiHop-RAGQwen3-14B5.82×2.23×1.91×1.46×
MultiHop-RAGLlama-3.1-8B3.49×2.46×1.87×1.63×
AVHaystacksQwen3-Omni4.41×2.45×1.47×1.32×
AVHaystacksMiniCPM-o 4.53.09×2.47×1.50×1.41×
CiteVQAQwen3-Omni3.66×2.60×1.41×1.33×
CiteVQAMiniCPM-o 4.53.16×2.63×1.79×1.67×
MHDocVQAQwen3-Omni4.08×2.19×1.63×1.39×
MHDocVQAMiniCPM-o 4.52.74×2.07×1.48×1.36×

Replay on the Table 1 traces; the full-RL columns use the unchanged policy-update time of Table 1. With no reuse across prompts at all, rollout speedup stays at 2.07×–2.63× and full-RL speedup at 1.32×–1.67× on all ten configurations. What remains is the reuse among trajectories of the same question, which the table in W1 shows to be the larger part of the repetition. We will add this experiment to Appendix B.

AuthorsOpenReview ↗

Follow-up on Our Response

Thank you very much for your review. Your question about recurrence was a very helpful one for us, and we hope the same-prompt-only experiment answers whether the gains hold when items recur less often. Could you let us know if any of your concerns remain? We would be glad to continue the discussion.

Reviewer_jGVeOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The TraSURE framework is proposed in this paper to accelerate the rollout phase of agentic RL training. A key observation is that different rollouts keep retrieving the same external items but at different positions, so the prefix caching cannot reuse them. Each item is cached as a position-independent KV unit and shared across trajectories, together with demand-aware scheduling and posterior-guided speculative decoding. Then, the policy is trained under the same attention semantics as the cache uses and adapts to the cached items. Experiments were run on six models and four workloads and accelerated the rollout by 2.7-6x, and the end-to-end RL training by 1.4-2.2x, with all task scores within 1 pp of full-causal training.

Critical strengths

I think the speed measurements are designed very carefully and the ablations are well performed to back it up. Controlled replay fixes the same trajectories across all systems, and the cache and scheduling only use causal history, so the speedup cannot come from shorter or easier trajectories. The full-RL numbers are still reported after the policy-update cost dilutes the speedup, and the component-cumulative ablation can help attribute the gain to each mechanism.

It also demonstrated that the result is checked across the range where it could fail. Six models and four workloads including two multimodal setups were evaluated, and the access skew spans from 54% down to 23%. The consistency ablation shows that the mismatch between rollout and training attention collapses the quality, and the idea survives it. The skew-dependence is stated rather than hidden, which is worth crediting.

Critical weaknesses

I feel the across-seed variance are missing from the quality comparison. The within-1-pp claim is supported by a single training run per configuration, and the error bars in Fig 8(b) are not defined by the caption or the text. My concern is that the seed-level stability of the headline number is unmeasured, in a regime where RL post-training is known to be noisy across seeds. A few seeds on 2-3 configurations would settle this.

The speed comparisons are against vanilla slime+SGLang only, and a selective-recompute baseline like CacheBlend is missing, which should compare the training-adaptation route with the recomputation route. Without this comparison, the paper cannot tell whether training-time adaptation is better than recomputation or simply the one the authors built.

Questions for authors
  1. Fig. 8(b) appears to show error bars on the PIC–full-causal score gaps, but neither the caption nor the text states what they represent (across-seed variance? checkpoint variation? evaluation sampling variance?). If they are not across-seed variance, do the ±1 pp gaps persist across training seeds?

  2. Could you characterize how a selective-recompute approach (CacheBlend-style) would interact with cross-trajectory sharing, and report even a partial quality-speed comparison?

  3. Does deploying a policy trained under PIC semantics require a PIC-aware serving stack at inference time? And how much quality is lost if the policy is served with standard full-causal attention?

  4. Is the code intended to be released? The system components carry a lot of implementation detail, and reproducibility without code is a real concern for a systems paper.

5 comments
AuthorsOpenReview ↗

Response to Reviewer jGVe (1/3)

Thank you for your questions, and especially for Q3. It tested something we had taken for granted: that a policy trained under the PIC mask has to be served with it.

Weakness 1 (W1): I feel the across-seed variance are missing from the quality comparison. The within-1-pp claim is supported by a single training run per configuration, and the error bars in Fig 8(b) are not defined by the caption or the text. My concern is that the seed-level stability of the headline number is unmeasured, in a regime where RL post-training is known to be noisy across seeds. A few seeds on 2-3 configurations would settle this.

Question 1 (Q1): Fig. 8(b) appears to show error bars on the PIC–full-causal score gaps, but neither the caption nor the text states what they represent (across-seed variance? checkpoint variation? evaluation sampling variance?). If they are not across-seed variance, do the ±1 pp gaps persist across training seeds?

Answer: The error bars in Fig. 8(b) are ±1 std across 5 training seeds, and every point in Fig. 8 is a 5-seed mean. For each seed, the PIC and full-causal policies are trained separately from the same checkpoint, with the same prompts sampled at every step and the same evaluation sampling seed, and the gap is taken per seed; the mean gap of every configuration stays within 1 pp across seeds.

Protocol for the evaluation results. Live results are averaged over 5 training seeds. Replay results run fixed recorded trajectories, so the only variation is timing noise; each is the mean of 2 repetitions, which differ by less than 1% relative.

ResultQuantityProtocolRuns
Fig. 8(a)Learning curves, Qwen3-4B on MultiHop-RAGLive5 seeds; band is ±1 std
Fig. 8(b)Final-score gap, PIC − full causal, ten configurationsLive5 seeds; bar is ±1 std
Table 1, Live columnsRollout and full-RL speedupLiveSame 5 training runs as Fig. 8
Table 1, Replay columnsRollout and full-RL speedupReplayMean of 2 repetitions
Table 2Progressive component ablationReplayMean of 2 repetitions
Table 4 (Appendix)Absolute rollout timeReplayMean of 2 repetitions

Revision. We will define the bars and bands in the Fig. 8 caption, state the seed and repetition counts in Section 5.1 and Appendix B.3, and mark each result table and figure as live or replay.

AuthorsOpenReview ↗

Response to Reviewer jGVe (2/3)

Weakness 2 (W2): The speed comparisons are against vanilla slime+SGLang only, and a selective-recompute baseline like CacheBlend is missing, which should compare the training-adaptation route with the recomputation route. Without this comparison, the paper cannot tell whether training-time adaptation is better than recomputation or simply the one the authors built.

Question 2 (Q2): Could you characterize how a selective-recompute approach (CacheBlend-style) would interact with cross-trajectory sharing, and report even a partial quality-speed comparison?

Answer: CacheBlend was built for serving a fixed model, but it can also be used here: inside an RL rollout it supplies the item KV just as PICache does. Seen this way the two routes are close. Both reuse a cached item across trajectories, and the main difference is that CacheBlend recomputes about 15% of each item's tokens, chosen dynamically for every request, while we train the policy under the PIC mask and recompute nothing.

On speed, that 15% is the part that cannot be shared across trajectories. It depends on each trajectory's own preceding context, so it is paid once per trajectory, G times per group, and again whenever the item reappears under a different prefix. To put a number on it, we ran a CacheBlend-style baseline inside our rollout system under replay on all ten configurations of Table 1: full-causal policy, PICache for the item KV, and 15% of each item's tokens, the ratio CacheBlend reports as typically sufficient, selected per request by KV deviation and recomputed with full attention. Because it reuses items the same way we do, the rest of TraSURE carries over to it unchanged: scheduling still batches requests around shared items, and speculative decoding still drafts from sibling continuations. The baseline is therefore TraSURE with only the item handling replaced.

WorkloadModelTraSURE (Table 1)CacheBlend-styleTraSURE over CacheBlend-style
MultiHop-RAGQwen3-4B5.08×3.26×1.56×
MultiHop-RAGQwen3-8B4.82×3.18×1.52×
MultiHop-RAGQwen3-14B5.82×3.46×1.68×
MultiHop-RAGLlama-3.1-8B3.49×2.64×1.32×
AVHaystacksQwen3-Omni4.41×3.13×1.41×
AVHaystacksMiniCPM-o 4.53.09×2.49×1.24×
CiteVQAQwen3-Omni3.66×2.81×1.30×
CiteVQAMiniCPM-o 4.53.16×2.54×1.24×
MHDocVQAQwen3-Omni4.08×2.93×1.39×
MHDocVQAMiniCPM-o 4.52.74×2.30×1.19×

Replay rollout speedup over vanilla slime, G=16. The TraSURE column is the replay number of Table 1, and the CacheBlend-style column was measured under the same protocol.

The CacheBlend-style route is itself 2.30×–3.46× faster than vanilla slime, so selective recompute does carry over to RL rollouts; the per-request recomputation accounts for the remaining 1.19×–1.68×. The gap is larger than 15% because the 15% is a share of the item prefill that vanilla slime pays. TraSURE has removed almost all of that prefill, so the part that comes back is small next to the vanilla rollout but large next to TraSURE's own.

Training is where the two routes differ more. Because the recomputed tokens are chosen dynamically, per request and per layer, they are not a fixed mask that the training forward can reproduce, and adapting the training kernel to them is a substantial change that we did not make, so we have no end-to-end quality number for this route. Without that adaptation there is a mismatch between rollout and optimization: in the few training steps we ran with the stock kernel, training collapsed and the score dropped sharply, as the no-training-mask curve in Fig. 8(a) does.

CacheBlend recomputes because in serving the weights are fixed: the interactions an item loses when it is encoded on its own can only be restored by computing them again for every request. RL is already updating the weights, so we go one step further and let the policy learn to work with independently cached items directly, and nothing has to be recomputed. The PIC mask is static and is applied unchanged in slime, and the same run that teaches the agent to seek information teaches it this as well, in the same number of steps (Fig. 8(a)).

Revision. We will add this comparison to Appendix B and rewrite the CacheBlend sentence in Section 4.1 so that it reads as a difference in setting, fixed-model serving versus a policy under training, rather than as a claim of superiority.

AuthorsOpenReview ↗

Response to Reviewer jGVe (3/3)

Question 3 (Q3): Does deploying a policy trained under PIC semantics require a PIC-aware serving stack at inference time? And how much quality is lost if the policy is served with standard full-causal attention?

Answer: No. We serve the policy with the PIC mask by default, since that is what it was trained with, and we had not tested it without the mask before your question.

To measure it, we took the final PIC checkpoint of every training seed in each configuration of Fig. 8(b) and evaluated it again under standard full-causal attention, with everything else unchanged. The table reports the same quantity as Fig. 8(b), the final-score gap to the full-causal policy in pp, mean ± std over the 5 seeds; the first column is Fig. 8(b), the second is the re-evaluation.

WorkloadModelPIC policy, PIC mask − full causal (pp)PIC policy, standard attention − full causal (pp)
MultiHop-RAGQwen3-4B+0.27 ± 0.21+0.19 ± 0.31
MultiHop-RAGQwen3-8B+0.13 ± 0.25+0.20 ± 0.27
MultiHop-RAGQwen3-14B+0.15 ± 0.30−0.06 ± 0.49
MultiHop-RAGLlama-3.1-8B+0.10 ± 0.36−0.18 ± 0.45
AVHaystacksQwen3-Omni−0.10 ± 0.44+0.17 ± 0.30
AVHaystacksMiniCPM-o 4.5−0.99 ± 0.33−0.87 ± 0.38
CiteVQAQwen3-Omni+0.26 ± 0.52+0.11 ± 0.36
CiteVQAMiniCPM-o 4.5−0.78 ± 0.33−0.49 ± 0.26
MHDocVQAQwen3-Omni−0.13 ± 0.34−0.08 ± 0.54
MHDocVQAMiniCPM-o 4.5−0.30 ± 0.41+0.13 ± 0.17

We had expected a small drop under standard attention, since the policy never generated under full-causal attention during RL, and the table shows none. The opposite switch, from full-causal attention to the PIC mask, does cost quality: at step 0 in Fig. 8(a), the base model scores about 41 under the PIC mask against about 49 under full-causal attention. So switching the mask is not symmetric, and we think pretraining is the reason. The base model was pretrained entirely under full-causal attention and has never seen the PIC mask, so it has to learn the mask through RL. RL under the PIC mask, in turn, adds an ability without removing the pretrained one: the policy learns to use items that are encoded on their own, but full-causal attention is still what its pretraining was built on, so it loses nothing when the items see their full context again. The two modes appear to coexist in the model.

Revision. We will add this table to Appendix B, and state in Section 5.2 that a PIC-trained policy can be served with standard attention.

Question 4 (Q4): Is the code intended to be released? The system components carry a lot of implementation detail, and reproducibility without code is a real concern for a systems paper.

Answer: Yes. We commit to releasing the full source code with the camera-ready version, under the Apache 2.0 license. TraSURE is built on SGLang and slime, and the release will cover both sides: on the SGLang side, the PIC attention kernel with RoPE rebasing and the PICache integration into the radix tree, together with the scheduler and the draft-tree builder; on the slime side, the matching PIC mask used during policy optimization. What we release will be the exact version used for the experiments in the paper, together with the replay tooling and the training and evaluation configurations. We will state this commitment in the Reproducibility Statement.

Reviewer_s2SxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

Papaer is addressing a major inefficiency of RL which is the prefill of retrieved documents over and over due to the preceeding different tokens. If we can find a KV-cache of these documents without the preceeding tokens, we can accelarate the LLM post-training significantly. The paper proposes a method to solve this inefficiency. The main idea is making document tokens depend only on themselves independent of the previous tokens. This impacts the model and it requires post-training.

Critical strengths
  • The problem of improving the efficiency of post-training of LLMs is very important and the methodology/algorithm is well-motivated for this problem.
  • The experimental results are very promising and significantly speeding up RL training is definitely an important/beneficial for the community.
  • The evaluation methodology is solid as it measures final RL run-time directly as the final system metric. This is also partially thanks to another strength which is considering every part like KV-cache, scheduling etc. The paper is considering full system in a good way without independently studying a single part.
  • PICache is sensible and an effective contribution which is novel.
Critical weaknesses
  • The paper can be written much more cleanly with significantly less notation. See, equation 4. This is simply attention mask and as far as I understand, it is simply saying M_{uv} = 0 if v\leq u and u is not a document token or u,v belongs to the same document. There are lots of logical notations and formalism to say something as simple as this. This is just a single example.
  • I am puzzled with the speculative decoding section. It is using N-gram as a starting point and also comparing with N-gram. N-gram is a really bad speculator. MTP and also other existing learned speculators would be significantly better than N-gram. I think authors should at least compare against such effective speculators. I also find the speculative decoding entirely orthogonal to the paper as the majority of the paper is about improving prefill and this part is about decoding.
  • The overall method is only available within a single online policy update as KV-cache is only valid for the current model parameters. Is this relevant beyond information retrieval training loads? Most post-training workload include multiple tasks and this much complexity only for a single post-training task is not entirely justified.
  • I would appreciate more details on how SGLang is set-up.
Questions for authors
  • Can't there be much simpler solutions like simply re-ordering the documents with a global order and also always putting documents tokens in the beginning so you don't have any mismatch. I am not saying this is a baseline or authors should make any comparison but I am curious why did authors chose a bit more of a complicated setup? Did simpler solutions fail in early experiments?
  • Another very simple alternative is just using document summaries. Once can simply summarize documents and use them to decrease prefill size. I am a bit puzzled with the directly going to a very complicated methodology.
4 comments
AuthorsOpenReview ↗

Response to Reviewer s2Sx (1/3)

Thank you for your review, and especially for W2. Running the learned speculator brought out something about RL that the paper had left unsaid: the policy never stops changing, and that decides which drafters hold up.

Weakness 1 (W1): The paper can be written much more cleanly with significantly less notation. See, equation 4. This is simply attention mask and as far as I understand, it is simply saying M_{uv} = 0 if v\leq u and u is not a document token or u,v belongs to the same document. There are lots of logical notations and formalism to say something as simple as this. This is just a single example.

Answer: Your reading of Eq. (4) is correct, and your one sentence is the clearer way to say it. We will use it in place of Eq. (4), simplify the remaining equations in Section 4 in the same way, and keep notation throughout the paper to what the method actually needs.

Weakness 2 (W2): I am puzzled with the speculative decoding section. It is using N-gram as a starting point and also comparing with N-gram. N-gram is a really bad speculator. MTP and also other existing learned speculators would be significantly better than N-gram. I think authors should at least compare against such effective speculators. I also find the speculative decoding entirely orthogonal to the paper as the majority of the paper is about improving prefill and this part is about decoding.

Answer: A learned speculator such as an EAGLE or MTP head is trained on the outputs of one fixed model. That works in serving, where the weights never change. In on-policy RL the policy's weights change at every step, so the head is out of date after the first update; ReSpec (MLSys 2026) measures this for a fixed EAGLE-3 head and repairs it only by retraining the head online. N-gram drafting needs no training. It drafts from the policy's own recent outputs, so it follows the policy as it changes, and posterior guidance decides which of them to draft from.

We compared against the public EAGLE-3 heads for the two Table 3 models, on the same recorded traces as Table 3:

ModelDrafterAccepted / verificationSpeedup vs. no SD
Qwen3-4BN-gram2.80 ± 0.141.14×
Qwen3-4BPosterior (ours)3.90 ± 0.341.35×
Qwen3-4BEAGLE-33.15 ± 0.491.19×
Llama-3.1-8BN-gram1.44 ± 0.081.10×
Llama-3.1-8BPosterior (ours)1.92 ± 0.191.23×
Llama-3.1-8BEAGLE-31.34 ± 0.511.04×

On average over the run, EAGLE-3 sits between N-gram and posterior drafting on Qwen3-4B and below both on Llama-3.1-8B. The average hides a decline. The head matches the released checkpoint and is kept fixed, so at the start of training it is the best drafter, accepting 4.03 tokens per verification on Qwen3-4B and 2.38 on Llama-3.1-8B. By the end it has fallen to 2.84 and 1.23, level with or below N-gram. N-gram and posterior drafting show no trend, since the rollout itself refreshes their pools. Retraining the head online, as ReSpec does, adds training cost and still leaves it one update behind. We will add EAGLE-3 to Table 3.

On whether speculative decoding belongs in the paper: the paper is about one thing, the repetition among trajectories. Different trajectories may retrieve the same item, often with different timing in their rollouts, and after the same item they often produce similar continuations. The first shows up as repeated prefill, the second as repeated decode. We start with prefill because that is where the time goes at first: decode is only 11%–29% of vanilla rollout time (Table 4). Once PIC and scheduling remove the repeated prefill, decode becomes the largest remaining cost, 49%–58% for the four text models (from Tables 2 and 4, decode time held fixed). It was natural to try the same repetition on this new bottleneck as well: posterior-guided speculative decoding drafts from those similar continuations and adds 1.12×–1.41× (Table 2).

AuthorsOpenReview ↗

Response to Reviewer s2Sx (2/3)

Weakness 3 (W3): The overall method is only available within a single online policy update as KV-cache is only valid for the current model parameters. Is this relevant beyond information retrieval training loads? Most post-training workload include multiple tasks and this much complexity only for a single post-training task is not entirely justified.

Answer: The one-step window is a property of on-policy RL, not of our method: KV computed under θv\theta_v, including the baseline's radix cache, is invalid after the update, and reusing it would make the rollouts off-policy. Inside a step the reuse is large. Trajectories of the same prompt, and often of different prompts, retrieve the same items; vanilla slime recomputes an item for every trajectory that meets it under a different prefix, while PICache computes it once unless it is evicted, which alone gives 1.74×–2.97× (Table 2). Across steps every item in use is computed again, and scheduling keeps that close to one materialization per item per step: without it, concurrent requests for the same item each prefill it, 15%–41% of item-prefill tokens in Fig. 5. The two scheduling rows of Table 2 together add 1.26×–1.66× on top of PIC.

We focus on information-seeking RL over a shared corpus (abstract, Section 1). Many post-training workloads do mix several tasks. Teaching an agent to search, though, is commonly done as a stage of its own: Search-R1, R1-Searcher, and ReSearch, for example, each train on the retrieval task alone. That stage is what our method is built for.

Weakness 4 (W4): I would appreciate more details on how SGLang is set-up.

Answer: Both systems run the same stack: slime v0.3.0 with the SGLang release it pins, v0.5.12.post1. TraSURE adds its PIC, scheduling, and drafting changes as patches on that release; vanilla slime runs the unmodified release with the same settings.

SettingValue (baseline and TraSURE)
Tensor parallelism per rollout engine2
mem-fraction-static0.85
max-running-requests16
KV cacheRadix cache (baseline); PICache for items on top of the radix cache (TraSURE)
Chunked prefilldisabled
Page size1
Scheduling policyFCFS (baseline); demand-aware admission (TraSURE)

Chunked prefill is off in both systems: it mainly helps latency, which serving systems care about, and does not help throughput, which is what an RL rollout needs. mem-fraction-static is the share of GPU memory reserved for the model weights and the KV memory pool. PICache stores items in that same pool, so there is no separate cache size to set. We will add this table to Appendix B.2.

AuthorsOpenReview ↗

Response to Reviewer s2Sx (3/3)

Question 1 (Q1): Can't there be much simpler solutions like simply re-ordering the documents with a global order and also always putting documents tokens in the beginning so you don't have any mismatch. I am not saying this is a baseline or authors should make any comparison but I am curious why did authors chose a bit more of a complicated setup? Did simpler solutions fail in early experiments?

Answer: Both are natural ideas to start from. Each runs into a problem that is specific to agentic rollouts.

Documents first. An agent decides what to retrieve turn by turn, so before generation starts nobody knows which items a trajectory will need. Putting them first means supplying all candidates up front. The policy then sees the evidence before deciding to seek it, which removes the decision-making ability we are training, and every trajectory carries all candidates instead of the few it actually retrieves. We tried this early on and the engine ran out of KV memory.

Global order. Items arrive one at a time as the policy retrieves them, so there is nothing to sort in advance, and sorting later would move a new item in front of text the policy has already written. Sorting also helps only when two trajectories retrieve exactly the same items, because a prefix cache stops matching at the first difference: if one trajectory retrieves A and C and another retrieves B and C, C comes after A in one and after B in the other, and is still computed twice.

PIC removes this dependence directly: an item is encoded on its own and reused wherever it appears, which alone gives 1.74×–2.97× (Table 2).

Question 2 (Q2): Another very simple alternative is just using document summaries. Once can simply summarize documents and use them to decrease prefill size. I am a bit puzzled with the directly going to a very complicated methodology.

Answer: Summaries help when an item is a short passage about one thing. The items here are different: a news article, a document page, or a video clip is long and dense, and different questions draw on it, each caring about a different part. One summary cannot keep all of those parts, and it cannot know which ones a later question will need. A smarter summarizer that adapts to each question could fix that, but it only moves the complexity from one place to another, since it then has to read the whole item for every question. Even where summaries are used, they come in every length, from a few lines to several pages, and many documents in a company knowledge base are already condensed from something longer. A summary is still an item: many trajectories retrieve it, in different contexts, and each one prefills it again. So summarizing makes each item shorter but leaves the repetition in place, and the repetition is what we address and remove.

AuthorsOpenReview ↗

Follow-up on Our Response

Thank you again for your detailed review. Your comments have already improved the paper. We would really value your thoughts on whether our responses have addressed your concerns, in particular the comparison with a learned speculator (EAGLE-3) and the SGLang setup. If anything remains unclear, we would be very happy to discuss it further.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Code release. We are committed to releasing the full source code with the camera-ready version under the Apache 2.0 license. This covers both the SGLang-side changes (PICache, the attention kernel, scheduling, and speculative decoding) and the slime-side changes, together with the replay tooling and all experiment configurations. We will state this in the Reproducibility Statement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.