- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies RL training for agents that retrieve information from a shared corpus. Different trajectories often read the same documents, but ordinary prefix caching cannot reuse their KV states when the preceding contexts differ. TraSURE encodes retrieved items independently and trains the policy with the same attention rule, allowing those states to be shared across trajectories. It also schedules requests around document reuse and uses historical continuations for speculative decoding. Experiments on text and multimodal tasks report up to 6.01× rollout acceleration and 2.16× full-RL acceleration, with final task scores within one percentage point of independently trained full-causal policies.
Critical strengths
- Clear motivation: Shared-corpus rollouts offer substantial document reuse beyond prefix caching. Using RL to adapt the policy to independently encoded documents is a useful part of the design.
- Substantial gains: The paper reports improvements in both rollout and full-RL time against a radix-caching baseline, with ablations supporting the additional components.
Critical weaknesses
Narrow applicable setting: Because cached KV states are invalidated at every policy update, reuse is confined to the trajectories of a single RL step, and the paper demonstrates this confined setting only on four fixed benchmark corpora. Whether the speedup survives on larger corpora, where items recur less often within a step, is not established.
Questions for authors
- How does the speedup vary with corpus size and item-access skew? A smaller benchmark, such as a restricted corpus, would clarify whether the gains hold when items recur less often.
3 comments
Response to Reviewer Jgtn (2/2)
Question 1 (Q1): How does the speedup vary with corpus size and item-access skew? A smaller benchmark, such as a restricted corpus, would clarify whether the gains hold when items recur less often.
Answer: For the first part of your question, access skew, the four workloads in the paper already differ widely (Fig. 10):
| Workload | Share of rollout accesses on the top 10% of items | Replay rollout speedup (Table 1) |
|---|---|---|
| MultiHop-RAG | 54% | 3.49×–5.82× |
| MHDocVQA | 45% | 2.74×–4.08× |
| AVHaystacks | 27% | 3.09×–4.41× |
| CiteVQA | 23% | 3.16×–3.66× |
Skew helps. It is a property of shared-corpus workloads: when many questions draw on the same few items, reuse also happens across prompts, and the most skewed workload, MultiHop-RAG, has the highest speedup, up to 5.82×. However, the gain does not rest on skew alone. The least skewed workload, CiteVQA, still reaches 3.16×–3.66×, because most of its repeated accesses come from trajectories of the same prompt (74%–79%, table in W1), and that reuse is there whatever the skew.
For the second part, corpus size, we have not varied the corpus itself. Your restricted-corpus test is a way to see whether the gains hold when items recur less often. We tested the extreme version of that: each PICache entry is tagged with the prompt that created it, and a request may reuse an entry only if it carries the same tag. Every access from another prompt is recomputed, as if no two questions shared any item. This removes everything that depends on questions sharing items, which is the part a larger corpus would reduce.
| Workload | Model | Rollout, all sharing (Table 1) | Rollout, same-prompt only | Full RL, all sharing (Table 1) | Full RL, same-prompt only |
|---|---|---|---|---|---|
| MultiHop-RAG | Qwen3-4B | 5.08× | 2.29× | 1.98× | 1.53× |
| MultiHop-RAG | Qwen3-8B | 4.82× | 2.45× | 1.79× | 1.49× |
| MultiHop-RAG | Qwen3-14B | 5.82× | 2.23× | 1.91× | 1.46× |
| MultiHop-RAG | Llama-3.1-8B | 3.49× | 2.46× | 1.87× | 1.63× |
| AVHaystacks | Qwen3-Omni | 4.41× | 2.45× | 1.47× | 1.32× |
| AVHaystacks | MiniCPM-o 4.5 | 3.09× | 2.47× | 1.50× | 1.41× |
| CiteVQA | Qwen3-Omni | 3.66× | 2.60× | 1.41× | 1.33× |
| CiteVQA | MiniCPM-o 4.5 | 3.16× | 2.63× | 1.79× | 1.67× |
| MHDocVQA | Qwen3-Omni | 4.08× | 2.19× | 1.63× | 1.39× |
| MHDocVQA | MiniCPM-o 4.5 | 2.74× | 2.07× | 1.48× | 1.36× |
Replay on the Table 1 traces; the full-RL columns use the unchanged policy-update time of Table 1. With no reuse across prompts at all, rollout speedup stays at 2.07×–2.63× and full-RL speedup at 1.32×–1.67× on all ten configurations. What remains is the reuse among trajectories of the same question, which the table in W1 shows to be the larger part of the repetition. We will add this experiment to Appendix B.
Follow-up on Our Response
Thank you very much for your review. Your question about recurrence was a very helpful one for us, and we hope the same-prompt-only experiment answers whether the gains hold when items recur less often. Could you let us know if any of your concerns remain? We would be glad to continue the discussion.
Response to Reviewer Jgtn (1/2)
Thank you very much for your review, and in particular for the point about recurrence. Both of our answers below are about it.
Answer: You are right that reuse is confined to one RL step. This holds for any on-policy method: once the policy is updated, KV computed under θv no longer matches it, and that includes the radix cache of vanilla slime. So what matters is how much reuse there is inside one step, and it is large. Trajectories of the same prompt, and often of different prompts, retrieve the same items. Vanilla slime recomputes an item for every trajectory that meets it under a different prefix. PICache computes it once per step unless memory pressure evicts it, which alone gives 1.74×–2.97× (Table 2). Scheduling then makes sure each item is computed once in the new step and read by the requests that follow, adding 1.16×–1.47× from current demand and a further 1.01×–1.19× from lookahead (Table 2).
As for larger corpora, the speedup depends on how many trajectories in a step draw on the same items, more than on the size of the corpus. We measured this inside single RL steps, with only the prompts each step samples (table below, replay traces of Table 1). Most repeated item accesses come from trajectories of the same prompt. This part exists for any corpus, because the G=16 trajectories of a prompt answer the same question. The rest comes from the other prompts in the same step and follows the access skew of the workload (Q1).