acceptodds
Withdrawn from ICLR 2027

Shared Resources Distort Credit Assignment in Group-Relative Agent Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

Group-relative reinforcement learning trains language-model agents by comparing rewards across rollouts of the same prompt. We identify resource-state interference, in which shared quotas, queues, and mutable tool state make a rollout's reward depend on other rollouts' actions. A rollout can thus influence its own evaluation baseline by altering sibling rewards. Our theoretical analysis distinguishes independent-deployment performance from shared-service welfare and provides finite-group constructions in which expected group-relative updates oppose the gradient of either objective. To address this problem, we develop receipt-conditioned credit control, using resource receipts to identify affected rewards and selectively replay trajectories from reference states. This enables either censoring or propensity-corrected credit estimation. Resource receipts and fixed-action paired replay isolate execution effects on outcomes, credit, and parameter gradients. Search audits on Qwen3 and Llama-3.1 show that shared execution distorts raw parameter gradients. In matched training with Qwen3-8B and Llama-3.1-8B, shared execution lowers retrieval NDCG@3 by 10.74 and 35.63 points, respectively, relative to isolation. Partial replay improves raw-gradient alignment with paired isolated execution. In separate matched training experiments, it improves retrieval performance over shared execution while using approximately 38% fewer training retrieval executions than isolation on both models.

Primary area:
reinforcement learning
Keywords:
Large language models, agentic reinforcement learning, group-relative policy optimization, credit assignment, resource-state interference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_QKohOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies resource-state interference in agentic LLM GRPO. Sibling rollouts of the same prompt share quotas, queues, or mutable tool state, so one rollout’s action can change its siblings’ rewards, affecting its own group-relative baseline.

  • Theory. The authors decompose the expected update into own-return term and spillover term. They separate two objectives: independent-deployment performance and shared-service welfare. They show that the expected GRPO, Dr. GRPO, and RLOO updates can have the opposite sign from either objective.
  • Method. They propose receipt-conditioned credit control, which logs each tool call’s resource exposure as a receipt. It has two variants, RCC and PRSC.
  • Experiments. With a shared quota of two retrieval calls per group of 8, training lowers NDCG@3 relative to isolation by 10.74--35.63 points. PRSC recovers most of this gap while using about 38% fewer retrieval executions than isolation.
Critical strengths
  1. Valid conceptual point. Group-relative estimators assume that sibling rewards depend only on each sibling’s own actions. Shared mutable state breaks this assumption. The paper states this clearly and connects it to the related literature.
  2. PRSC is simple and principled. PRSC is based on the self-normalized counterfactual estimation. When every altered trajectory is replayed (p=1), it reduces exactly to the full credit.
Critical weaknesses

W1. The problem setting seems artificial, and the paper does not properly justify its relevance to realistic agentic LLM RL settings.

  • In the main experiment, authors cap the whole group of eight rollouts at two retrieval calls. The retrieval backend is a local BM25 index, which has no natural quota.
  • In standard agentic RL pipelines, rollouts can maintain independent execution states and be assigned separate tool-call budgets. A group-level quota smaller than the group size is an additional constraint the authors created, one that practitioners would rarely encounter in practice, especially in this BM25 retrieval setting.
  • Moreover, one natural solution of this problem setting is that giving each rollout its own quota by reducing the group size. (For example, in Appendix Table 8, if the per-group quota is 4, then one can reduce the group size from 8 to 4) Although a smaller group may yield a noisier group-relative baseline, the paper does not compare PRSC against this simple alternative.
  • The motivating citation (Mercor & SkyRL, 2026) does not support the practicality of problem setting. The report describes judge API rate limits at cluster-level concurrency ("800+ concurrent rollouts") and MCP disconnects from sharing one process across agent loops. Both were solved with retry/backoff and per-rollout isolation. It does not describe any sibling-level shared quota and no distortion of group-relative credit.
  • The paper does not evaluate or discuss settings where tool execution isolation is truly infeasible or expensive, such as GPU-heavy operations.

W2. The main experiments are not true agentic RL; they are single-turn query generation.

  • In the search experiments, the policy is prompted to produce exactly one search query and not to answer the question. The training reward is NDCG of that generated query's retrieval results, and only query tokens are trained.
  • The answer step does not affect training at all.
  • Therefore, the task in the paper is single-turn query rewriting with a retrieval-quality reward. There is no multi-turn interaction, and the trained tokens are generated before any tool observations.
  • This makes the description in Section 4.1 ("The agent generates a query, retrieves evidence …, and answers from the retrieved passages") and the "agent RL" framing in the title misleading.
  • The multistep tasks (inventory, cache) are small toy environments evaluated with Qwen3-8B only. They do not involve training in a multi-turn agentic loop.
  • Whether interference matters or compounds in genuine multi-turn agent training remains untested.
Questions for authors

Q1. Realistic settings. Can the authors show a realistic agentic RL setting where per-rollout isolation is infeasible or very costly and interference arises naturally rather than by construction?

Q2. Multi-turn agentic training. Can the authors evaluate a genuine multi-turn agent, for example Search-R1-style interleaved reasoning and retrieval with an answer-based reward and the full trajectory trained? Does interference persist or compound there?

Q3. The ‘Shared’ condition confounds interference with length-based reward shaping. Under 'Shared' condition, valid queries are served in ascending order of length. Such condition systematically pushes the policy toward shorter completions. It is well-known that there exists performance--length trade-off in GRPO training, so part of the improvement of the replay-based methods may come from removing this shaping rather than from correcting interference. Please report online training under the shared quota with randomized, length-independent dispatch. Under this setting, how much of the Shared--ISO gap remains, and how much of PRSC's improvement?

[Minor Suggestions/Questions]

S1. In the Introduction, "Mercor and SkyRL describe … (Mercor & SkyRL, 2026)" should use \citet{}.

S2. Emphasizing the key term at its first definition, e.g., "We call this dependence \textit{resource--state interference}", could be a good option.

S3. Do not simply list studies in the "Related Work" section. Instead, describe the main difference between previous research and your work.

S4. Please add 'README.md' file in the supplementary materials. It is difficult to understand hundreds of files without it.

S5. What is "12-dimensional score vectors" in App. A.5? It seems that is not explained.

S6. Table 1 reports a Qwen3-4B raw-gradient cosine of 0.037 from a "later Shared-only rerun" (App. J). The supplementary's own reporting configuration (configs/protocol/gradient/reporting.yaml, seeds 6911–6913) and its table-assembly report (supplements/frozen_table/assemble_table.py) give 0.755 under the same frozen-256 protocol. Please specify what differs in the rerun (adapter/action seeds, reward definition, conditions), provide its artifacts, and explain why it replaced the original value.

S7. The AI use statement lists only language polishing, literature search and reference checking, and table preparation. I did not inspect the entire supplementary code, but several passages suggest that generative AI agents were used for code development, experiment orchestration, and analysis. For example, (1) generated analysis reports that refer to the person requesting an experiment as "用户" (2) author/auditor separation scheme with directory-based agent identities (3) reviewer-gap-closure audit tools. Please make sure to clearly state the use of AI.

S8. The supplementary includes reviewer-gap-closure tooling that takes a file named 意见.md (review comments) as input and maps its items ("Weakness 1–8", "Question 1–7", "Missing Related Work 1–6") to closure statuses. If these comments were generated by an AI system, that should also appear in the AI use statement. Since the statement mentions neither, I cannot tell which is the case. Please clarify.

Reviewer_VK6cOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies resource-state interference in group-relative reinforcement learning for agents, where shared resources induce dependencies among rollouts’ rewards. The paper provides theoretical analyses of gradient misalignment and proposes receipt-conditioned credit control, including resource-conditioned censoring (RCC) and partial reference-state credit (PRSC). Experiments on Qwen3 and Llama3.1 models demonstrate gradient distortion and learning degradation under shared execution, while PRSC improves gradient alignment and recovers much of the performance.

Critical strengths
  1. The paper formalizes and provides a clear theoretical formulation of resource-state interference in group-relative agent RL.
  2. The experiments cover multiple models and different interference mechanisms.
Critical weaknesses
  1. The paper provides insufficient justification of the practical significance of the studied problem. The main search experiments impose a restrictive quota of two retrieval calls for eight rollouts, making resource contention inevitable. In practice, this limitation in the search tool or api is unlikely and should be considered as an infrastructure  issue that could be addressed before training or through resource isolation or failure handling. The paper does not establish how prevalent such interference is in realistic agentic RL, nor does it adequately compare against practical systems-level mitigations.
  2. The paper only imposes a restrictive quota of two retrieval calls for eight rollouts in a rollout group, but the resource limitation problem can also happen in the training batch level, which has many rollout groups. Also, the Resource-state interference problem studied in the paper can corrupt rollout rewards regardless of the policy optimization algorithm, potentially affecting PPO and other policy-gradient methods. The paper should also study these cases
Questions for authors

Please see weaknesses

Reviewer_m2m4OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how shared resources can make sibling rollouts' rewards interdependent and distort credit assignment in group-relative agent reinforcement learning. It distinguishes independent-deployment performance from shared-service and analyzes how group-relative updates can become misaligned with these objectives. It proposes receipt-conditioned credit control, using resource receipts to identify affected trajectories for censoring or selective reference-state replay.

Critical strengths
  1. The paper identifies a practically relevant source of credit distortion in agent training. Shared tool resources can couple rollouts that are treated as comparable samples for group-relative learning. The distinction between independent-deployment performance and shared-service helps clarify which objective a training update should serve.

  2. The paired-replay audits provide concrete evidence of the proposed mechanism. Holding policy actions fixed while varying resource exposure can trace changes from execution outcomes to credit and raw parameter gradients. Capacity-matched and trajectory-private controls further support the diagnosis.

Critical weaknesses
  1. The theoretical and algorithmic advances appear limited. The decomposition in Section 3.2 gives

    ∇θJjoint=gown+gspill,\nabla_\theta J_{\mathrm{joint}} = g_{\mathrm{own}} + g_{\mathrm{spill}},

    and

    = g_{\mathrm{own}} - \frac{1}{G-1}g_{\mathrm{spill}}.$$ I find these identities correct under the stated assumptions. However, their central implication is that a baseline can no longer be treated as an expectation-preserving control variate when it depends on the sampled action through its effect on sibling rewards. The theoretical analysis is not sufficiently compelling to me. My concern about algorithmic novelty is similar. PRSC identifies resource-affected trajectories, randomly replays them from a reference state, and uses inverse-propensity weights with self-normalized group moments to estimate reference-state credit. This is a sensible correction strategy, but I do not find its distinct algorithmic contribution sufficiently established beyond combining exposure detection, selective replay, and propensity weighting.
  2. The online learning evidence has limited task coverage, and the attribution of performance degradation remains insufficiently resolved. The main experiments testing whether the method improves learned policies concentrate on HotpotQA, whereas the other environments primarily provide mechanism audits. Moreover, the main shared-quota setting uses G=8G=8 rollouts with only q=2q=2 search executions per group, so many rollouts are denied access to the tool.

    This makes it difficult to determine how much of the learned-policy degradation is specifically attributable to biased group-relative credit, rather than reduced access to useful observations and rewards during training.

Questions for authors

see weaknesses above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.