- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies resource-state interference in agentic LLM GRPO. Sibling rollouts of the same prompt share quotas, queues, or mutable tool state, so one rollout’s action can change its siblings’ rewards, affecting its own group-relative baseline.
- Theory. The authors decompose the expected update into own-return term and spillover term. They separate two objectives: independent-deployment performance and shared-service welfare. They show that the expected GRPO, Dr. GRPO, and RLOO updates can have the opposite sign from either objective.
- Method. They propose receipt-conditioned credit control, which logs each tool call’s resource exposure as a receipt. It has two variants, RCC and PRSC.
- Experiments. With a shared quota of two retrieval calls per group of 8, training lowers NDCG@3 relative to isolation by 10.74--35.63 points. PRSC recovers most of this gap while using about 38% fewer retrieval executions than isolation.
Critical strengths
- Valid conceptual point. Group-relative estimators assume that sibling rewards depend only on each sibling’s own actions. Shared mutable state breaks this assumption. The paper states this clearly and connects it to the related literature.
- PRSC is simple and principled. PRSC is based on the self-normalized counterfactual estimation. When every altered trajectory is replayed (p=1), it reduces exactly to the full credit.
Critical weaknesses
W1. The problem setting seems artificial, and the paper does not properly justify its relevance to realistic agentic LLM RL settings.
- In the main experiment, authors cap the whole group of eight rollouts at two retrieval calls. The retrieval backend is a local BM25 index, which has no natural quota.
- In standard agentic RL pipelines, rollouts can maintain independent execution states and be assigned separate tool-call budgets. A group-level quota smaller than the group size is an additional constraint the authors created, one that practitioners would rarely encounter in practice, especially in this BM25 retrieval setting.
- Moreover, one natural solution of this problem setting is that giving each rollout its own quota by reducing the group size. (For example, in Appendix Table 8, if the per-group quota is 4, then one can reduce the group size from 8 to 4) Although a smaller group may yield a noisier group-relative baseline, the paper does not compare PRSC against this simple alternative.
- The motivating citation (Mercor & SkyRL, 2026) does not support the practicality of problem setting. The report describes judge API rate limits at cluster-level concurrency ("800+ concurrent rollouts") and MCP disconnects from sharing one process across agent loops. Both were solved with retry/backoff and per-rollout isolation. It does not describe any sibling-level shared quota and no distortion of group-relative credit.
- The paper does not evaluate or discuss settings where tool execution isolation is truly infeasible or expensive, such as GPU-heavy operations.
W2. The main experiments are not true agentic RL; they are single-turn query generation.
- In the search experiments, the policy is prompted to produce exactly one search query and not to answer the question. The training reward is NDCG of that generated query's retrieval results, and only query tokens are trained.
- The answer step does not affect training at all.
- Therefore, the task in the paper is single-turn query rewriting with a retrieval-quality reward. There is no multi-turn interaction, and the trained tokens are generated before any tool observations.
- This makes the description in Section 4.1 ("The agent generates a query, retrieves evidence …, and answers from the retrieved passages") and the "agent RL" framing in the title misleading.
- The multistep tasks (inventory, cache) are small toy environments evaluated with Qwen3-8B only. They do not involve training in a multi-turn agentic loop.
- Whether interference matters or compounds in genuine multi-turn agent training remains untested.
Questions for authors
Q1. Realistic settings. Can the authors show a realistic agentic RL setting where per-rollout isolation is infeasible or very costly and interference arises naturally rather than by construction?
Q2. Multi-turn agentic training. Can the authors evaluate a genuine multi-turn agent, for example Search-R1-style interleaved reasoning and retrieval with an answer-based reward and the full trajectory trained? Does interference persist or compound there?
Q3. The ‘Shared’ condition confounds interference with length-based reward shaping. Under 'Shared' condition, valid queries are served in ascending order of length. Such condition systematically pushes the policy toward shorter completions. It is well-known that there exists performance--length trade-off in GRPO training, so part of the improvement of the replay-based methods may come from removing this shaping rather than from correcting interference. Please report online training under the shared quota with randomized, length-independent dispatch. Under this setting, how much of the Shared--ISO gap remains, and how much of PRSC's improvement?
[Minor Suggestions/Questions]
S1. In the Introduction, "Mercor and SkyRL describe … (Mercor & SkyRL, 2026)" should use \citet{}.
S2. Emphasizing the key term at its first definition, e.g., "We call this dependence \textit{resource--state interference}", could be a good option.
S3. Do not simply list studies in the "Related Work" section. Instead, describe the main difference between previous research and your work.
S4. Please add 'README.md' file in the supplementary materials. It is difficult to understand hundreds of files without it.
S5. What is "12-dimensional score vectors" in App. A.5? It seems that is not explained.
S6. Table 1 reports a Qwen3-4B raw-gradient cosine of 0.037 from a "later Shared-only rerun" (App. J). The supplementary's own reporting configuration (configs/protocol/gradient/reporting.yaml, seeds 6911–6913) and its table-assembly report (supplements/frozen_table/assemble_table.py) give 0.755 under the same frozen-256 protocol. Please specify what differs in the rerun (adapter/action seeds, reward definition, conditions), provide its artifacts, and explain why it replaced the original value.
S7. The AI use statement lists only language polishing, literature search and reference checking, and table preparation. I did not inspect the entire supplementary code, but several passages suggest that generative AI agents were used for code development, experiment orchestration, and analysis. For example, (1) generated analysis reports that refer to the person requesting an experiment as "用户" (2) author/auditor separation scheme with directory-based agent identities (3) reviewer-gap-closure audit tools. Please make sure to clearly state the use of AI.
S8. The supplementary includes reviewer-gap-closure tooling that takes a file named 意见.md (review comments) as input and maps its items ("Weakness 1–8", "Question 1–7", "Missing Related Work 1–6") to closure statuses. If these comments were generated by an AI system, that should also appear in the AI use statement. Since the statement mentions neither, I cannot tell which is the case. Please clarify.