acceptodds
Under review as a conference paper at ICLR 2027

SHARING ENVIRONMENT RANDOMNESS: WHEN REWARD VARIANCE FALLS BUT GRADIENT RISK RISES

Abstract

Group-based reinforcement learning is widely used to train language model agents by comparing rewards across sampled trajectories. Sharing environmental randomness can make these comparisons less noisy, but it also reduces independent environment coverage under a fixed rollout budget. We construct counterexamples with identical reward distributions but opposite sharing effects on gradient risk, showing that reward statistics alone cannot determine when sharing is beneficial. In this paper, we introduce Environment-Sharing Risk Analysis (ESRA), a framework for evaluating environment sharing through its effect on gradient estimation risk. We derive an exact finite-group risk criterion for raw leave-one-out gradients at a frozen policy, quantifying the trade-off between within-environment fluctuations and differences in mean gradients across environments. We test the criterion with independent gradient audits and controlled sharing interventions. Using Qwen3.5-35B-A3B-Base, we evaluate ESRA on HotpotQA and ALFWorld, covering multi-hop question answering and interactive decision-making. At matched total training cost, environment sharing alone improves native GRPO by 7.10 percentage points on average. Holding the raw-LOO estimation and calibration rules fixed, shared collection improves the two-task mean by 7.00 percentage points over independent collection.

Primary area:
reinforcement learning
Keywords:
Length-Normalized Policy Optimization, Long Chain-of-Thought (CoT) Reasoning, Adaptive Token-level Credit, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_MeALOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at sharing part of the environment randomness within a rollout group in GRPO-style training of LLM agents. It argues that sharing lowers reward variance but does not necessarily lower gradient error, because it also reduces environment coverage. The authors derive a criterion for when sharing helps, based on an existing U-statistic analysis of GRPO, and check its predictions against measured gradient errors. They also report training gains on HotpotQA and ALFWorld.

Critical strengths

The paper makes a clear point: lower reward variance does not imply a better gradient estimate. The analysis seems correct, and the experiments are set up carefully.

Critical weaknesses
  1. Limited novelty. The idea that shared random inputs help only when the induced covariance is favorable is well known from the common random numbers literature. The main result mostly applies an existing GRPO analysis to this setting. I did not learn much new from it.
  2. The theory and the training results are not well connected. The criterion is never used to decide whether to share in training. Also, the randomness being shared is largely introduced by the authors' own environment setup, so it is unclear how much of the gain would carry over to standard settings.
Questions for authors
  1. How does the baseline perform in the standard environment setup, without the added randomness?
  2. Does using the criterion to decide whether to share actually help in training, compared with always or never sharing?
3 comments

1. Novelty (Weakness 1). We agree that favorable covariance is an established mechanism of common random numbers (CRN). However, favorable reward covariance does not guarantee more accurate policy gradients. ESRA further studies cases that reward-level assessments cannot resolve, aiming to determine whether environmental sharing improves or harms learning updates. Our methodological contribution is to turn this ambiguity under reward-only reasoning into an estimable risk comparison between shared and independent collection. Building on the established U-statistic decomposition, we provide an estimable criterion for sharing risk at finite group sizes and use it to guide collection during training.

Favorable reward covariance does not guarantee lower gradient error. Proposition 1 in the main text provides an explicit counterexample. The two cases have identical joint laws of shared inputs and rewards. Rows indicate whether the higher-reward action changes across environments. Arrows: independent → shared (G=2; N: trajectories).

CaseCentered-reward varianceGradient MSE
Unchanged0.025 → 0.020 (−20%)0.10/N → 0.08/N (−20%)
Reversed0.025 → 0.020 (−20%)0.10/N → 0.16/N (+60%)

ESRA's exact frozen-policy raw-LOO criterion uses reward–score dependence and variation in mean gradients across environments to guide collection and can improve gradient estimation. Its sign agreement with independent gradient audits is 97/120, versus 64/120 for reward-only and 80/120 without β (Tables 1 and 11). At N=64, increasing group size on HotpotQA turns sharing from beneficial to harmful despite greater centered-reward variance reductions; the full criterion captures this boundary, while removing β misses it (Figure 2).

2. Standard environment (Question 1). Using Qwen3.5-35B-A3B-Base, we compare GRPO-base trained in standard and randomized environments on common standard (STD) tests.

REF-H/REF-A denote the manuscript's randomized HotpotQA/ALFWorld environments, respectively; environmental inputs can be sampled independently or shared. STD removes added randomization but retains native randomness: up to seven ranked HotpotQA passages and native ALFWorld interfaces. Runs use native GRPO without value estimation or λ calibration, matched initialization/data and five paired seeds. We average four attempts per item, then 7,405 questions or 134 unseen games; scores are %, mean±SD.

Both rows use GRPO-base and the same standard tests. The randomized-environment training row uses no environmental input sharing.

Table R1. Standard-environment evaluation.

Training environmentHotpotQAALFWorld
Standard54.67±1.7768.66±2.48
Randomized54.62±1.9768.25±2.91

The standard-environment baseline averages 61.66% across tasks. Training in randomized environments without environmental input sharing changes this by −0.23 pp (95% CI [−0.76,0.24]). Fixed partial sharing gains +6.71 pp [5.53,7.87] over standard-environment training on the same STD tests (95% CI), showing that sharing gains persist without added test-time randomization.

3. Criterion-selected training (Question 2). Using the same Qwen3.5-35B-A3B-Base model, we test whether criterion-selected training improves on the manuscript's GRPO (no environmental input sharing) and GRPO+B (always partial sharing). All three arms train independently from the same initializer, changing only sharing while fixing native GRPO advantages, updates and hyperparameters and omitting full GRPO+ESRA's value estimation and λ calibration.

Table R2 evaluates all arms in the manuscript's randomized environments, reusing Table R1's baseline checkpoints. Diagnostic and validation costs count toward the common 35B training-resource cap.

Table R2. Fixed and criterion-selected environmental input sharing, evaluated in the manuscript's randomized environments (%, mean±SD).

Training designHotpotQAALFWorld
GRPO-base (standard environment training)54.6±2.068.9±2.8
GRPO-base (no environmental input sharing)54.0±2.068.0±3.1
GRPO+B (always sharing a subset of environmental inputs)61.8±2.874.4±1.9
Our method (criterion-guided environmental input sharing)63.2±3.078.0±5.1

The three randomized-environment arms differ only in environmental input sharing. At updates 0/160/320 (G=8, N=32/domain), the selector diagnoses its current policy by domain using data disjoint from actor/evaluation data. It shares partially only if the input-block-bootstrap 95% upper bound on the full shared-minus-independent raw-LOO risk difference is negative; otherwise, it collects independently. Sharing is selected in 19/30 decisions.

Table R2 shows that criterion-guided sharing outperforms both fixed strategies in the two-task mean. It improves on always partial sharing and never sharing by 2.51 and 9.61 percentage points, respectively.

Thank you for the clarification. The rebuttal has addressed my main concern regarding novelty, and I am raising my score accordingly.

Thank you for your time and constructive feedback. We are glad that our responses have addressed your concerns, and we appreciate your thoughtful reassessment.

Reviewer_TBb5OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates when sharing environmental randomness can improve policy gradient estimation in group reinforcement learning using language model agents. The authors point out that while sharing random environmental inputs can reduce reward fluctuations within a group, it also reduces the coverage of independent environmental instances under a fixed sampling budget; therefore, a decrease in reward variance does not necessarily imply more reliable gradient estimates. The paper proposes Environment-Sharing Risk Analysis (ESRA): First, using counterexamples with identical reward distributions but opposite gradient risk directions, it proves that reward statistics alone cannot determine whether environment sharing is beneficial; then, under assumptions such as a frozen policy, matching marginal distributions, and intra-group conditional independence, the paper derives the exact finite-group risk of the raw-LOO gradient estimator and the necessary and sufficient conditions for environment sharing to be beneficial. The criteria distinguish between inter-environment mean gradient heterogeneity, intra-environment gradient variability, and finite-group correction terms. Overall, this paper explains the benefits and risks of environment sharing at the gradient level and establishes an empirical link between this diagnosis and training performance.

Critical strengths
  1. Starting from gradient risk rather than reward variance, authors propose a theoretical criterion with well-defined conditions for applicability. The paper first constructs counterexamples with identical reward distributions but opposite shared effects, and then provides an exact finite-group decomposition of the raw-LOO gradient risk. This result explains why sharing randomness can both improve intra-group comparisons and lead to higher gradient risk due to reduced environmental coverage.
  2. The authors link theoretical predictions to actual training results through independent gradient audits and controlled training interventions. The authors use independent data to estimate risk terms and audit the predictions using newly sampled full gradient batches. The full criterion yields a significantly higher number of confirmed symbol matches compared to the reward-only control. Additionally, Table 3 does not only show conditions where sharing is beneficial but also reports conditions where reward variance decreases while gradient risk and GRPO training performance deteriorate. Table 6 further distinguishes the performance contributions of sharing alone and the addition of value calibration.
Critical weaknesses
  1. Insufficient reporting of sampling-validation outcomes and realized computational budgets. The paper provides detailed specifications for verifying marginal preservation and deterministic replay in Appendix C.1, including explicit statistical acceptance criteria. However, the actual outcomes of these sampler-validation tests are not summarized in the provided manuscript. While Appendix C.3 reports numerical consistency checks for the objective adapters, these do not directly verify the sampling properties required by the theoretical analysis. Similarly, Appendix E.3 defines a comprehensive accounting protocol for matched-total-cost comparisons, and Appendix E.1 specifies the substantial additional sampling required by the calibrated estimator. Nevertheless, the manuscript does not report the realized per-arm or per-seed GPU-hour costs, actor-update counts, calibration statistics, or paired budget discrepancies described in the accounting protocol.
  2. Since the empirical validation relies on two explicitly designed environmental randomization protocols, the external generalizability of the conclusions remains limited. In HotpotQA, the shared information consists of tail-passage selection from retrieval results, while in ALFWorld, it consists of object instance ID mappings. Although these two protocols feature explicit random variable partitioning and mechanisms for preserving marginal distributions, they do not directly represent more general random tool responses, non-stationary environments, or noise in complex interactive environments. Although the paper reports transfer evaluations on WebShop, WebArena, τ-bench, and BFCL, Appendix C clarifies that these evaluations use existing training checkpoints rather than re-executing and validating environment-shared training in these new environments. Table 4 provides frozen gradient audits for other Qwen3.5 models, but the complete training results for matching costs remain focused on the core model.
Questions for authors
  1. Can authors provide the actual sampling inspection and resource accounting results used to verify the main experimental conclusions? Appendix C.1 specifies in detail the acceptance criteria for environmental sampling, replay, and margin distribution preservation, and Appendix E.3 explicitly requires the recording of actual costs. We ask the authors to clarify the actual pass/fail status of these tests and to provide a summary of resource consumption per seed for the main GRPO, GRPO+B, GRPO+ESRA, and raw-LOO independent/shared control groups, including GPU-hour equivalents, actor updates, calibration overhead, and budget variances.

  2. Does the author intend for ESRA to be understood as an offline risk explanation tool, or as a practical decision-making mechanism for selecting sharing strategies before or during training?

    Section 5.2 and Appendix D.1 present independent diagnostic, risk prediction, and auditing procedures, but these require a large number of additional trajectories and full gradient calculations. We ask the authors to clarify whether the reported risk predictions are actually used in the selection of sharing strategies during training, or whether they are primarily used for post-hoc validation under predefined experimental conditions; if they are intended to support actual selection, how is this diagnostic overhead accounted for in the budget for the corresponding training decisions? Can the authors provide results from actual sampling checks and resource accounting used to validate the main experimental conclusions?

4 comments

Response to Reviewer TBb5 — Part 1/2

Reviewer questions are summarized in the headings.

Q1. Did sampler checks pass, and what did each seed cost?

We report sampler pass/fail results and full training costs for all five methods.

REF-H/REF-A: 18/22 fixture classes, independent/partial/full sharing, G=4/8/16; 95% Bonferroni family coverage (C.1). C.3 separately checks objective adapters.

CheckAcceptance ruleResult
Output validity, replay, member/schedule isolation, physical semantics, wrapper removal, random streams and sharing structureExact agreement for each applicable check0 mismatches across all seven check families
Marginal and small-fixture joint probabilitiesEach multiplicity-adjusted Clopper–Pearson interval lies entirely within target probability ±0.016,735 passed; 0 failed, unresolved or incomplete; maximum deviation 0.003067

Qwen3.5-35B-A3B-Base; five paired seeds; G=8, N=64; 8×A100 80GB. GPU-hours = Σ(GPUs × occupied hours) to the scored checkpoint, including heads/calibration. Costs: A100 GPU-hour equivalents including non-GPU services; scores: mean ± seed SD.

Method and samplingMean GPU-hoursMean value-head GPU-hoursMean calibration GPU-hours, including failuresMean total costMean actor updatesCompleted / accepted refreshesHotpotQA (%)ALFWorld (%)
GRPO (independent)299.20.00.0302.5497.6—54.0±2.068.0±3.1
GRPO+B (partial sharing)299.10.00.0302.5503.4—61.8±2.874.4±1.9
GRPO+ESRA (partial sharing)298.293.6167.5302.652.827 / 2165.4±1.781.8±1.5
Calibrated raw-LOO (independent)298.186.8167.1302.569.629 / 1758.9±1.877.0±2.1
Calibrated raw-LOO (partial sharing)298.390.3175.0302.647.428 / 1966.7±1.483.2±1.7

All 25 checkpoints meet the 302.8 cap; max gap across 20 pairs: 0.5 (0.16%). Final costs: 302.9–303.4; overshoot ≤0.2%. Checks use unrounded costs; per-seed details: second reply.

Each refresh uses 72 actor-sized candidate/reference batches, charging collection, fitting, failures and retries. Accepted weights serve one fresh update; otherwise use Monte Carlo.

Q2. Does ESRA guide sharing, and are diagnostic costs justified?

ESRA is a risk-diagnosis framework that predicts the effect of environmental sharing on gradient risk before independent validation, providing evidence for environment design and training optimization.

Original training used preset sharing. A separate warm-start comparison uses the same 35B model, native GRPO without value calibration and five paired seeds. We predict once before continuation: choose partial sharing only if the risk-difference 95% interval is wholly negative; otherwise use independent sampling, then hold the choice fixed.

The 302.8 per-run cap includes all prediction/check costs, including full-gradient computation; fixed baselines use that budget for training.

StrategyHotpotQA (%)ALFWorld (%)Mean costMean updates
Always independent60.56±1.2974.18±1.07302.6516.4
Always partial sharing63.76±2.8372.76±5.75302.6513.0
Prediction-guided sharing63.74±2.2875.22±2.53302.5496.4

Q1's optional extensions spend 83.9%–87.7% of total cost on value-head/full-calibration GPU work. Here, one prediction/check step before continuation, without value calibration, uses 4.97% of checkpoint cost (3.99%–5.36% per seed) and 5.23% of normal actor GPU-hours. At matched cost, HotpotQA gains 3.18 pp over always-independent sampling (95% interval [1.04, 5.33]); other paired intervals cross zero. Maximum paired cost gap: 0.3 (0.107%).

W2. Does sharing generalize beyond controlled environments?

From controlled to stochastic interaction. We use τ²-bench v0.1.3 to test sharing when later feedback remains stochastic and identical openings can produce different trajectories.

Setup. With Qwen3.5-35B-A3B-Base and a fixed Qwen3.5-9B user (temperature=1), we audit gradients and continue training on Telecom, comparing independent GRPO with GRPO+B sharing openings/initial states followed by independent interactions (five paired seeds; 103 updates per arm in total), and use Retail only for final transfer evaluation. After sharing the initial state, each rollout uses an independent random stream, with user replies and tool choices sampled at temperature 1 from its own history.

Dataset / metricIndependent GRPOESRA sampling (GRPO+B)Difference [95% interval]
Telecom gradient risk (×10⁻³)12.5510.79-1.76 [-2.64, -0.84]
Telecom success (%)41.49 ± 1.5347.54 ± 3.47+6.05 [1.93, 9.96]
Retail transfer success (%)38.25 ± 1.2044.56 ± 3.45+6.32 [1.89, 10.09]

Sharing lowers gradient risk by 14.00%, with four improving seeds and one declining seed in each of Telecom and Retail and greater variation across seeds.

Response to Reviewer TBb5 — Part 2/2: Per-seed resources

Reviewer:

Can authors provide the actual sampling inspection and resource accounting results used to verify the main experimental conclusions?

Appendix C.1 specifies in detail the acceptance criteria for environmental sampling, replay, and margin distribution preservation, and Appendix E.3 explicitly requires the recording of actual costs. We ask the authors to clarify the actual pass/fail status of these tests and to provide a summary of resource consumption per seed for the main GRPO, GRPO+B, GRPO+ESRA, and raw-LOO independent/shared control groups, including GPU-hour equivalents, actor updates, calibration overhead, and budget variances.

Response:

These are the 25 main-training runs summarized in our first response (Qwen3.5-35B-A3B-Base; five paired seeds). Costs are A100 GPU-hour equivalents, including non-GPU services. The common cap is 302.8; all checks use unrounded values.

Per-seed resources. GPU-hours, costs and updates stop at the scored checkpoint. Value-head collection/fitting and calibration, including failures, are included in the totals. Unused budget = cap − checkpoint cost; final overshoot = job-end cost − cap. Resource values are rounded to one decimal.

Method and samplingSeedGPU-hoursTotal costActor updatesValue-head GPU-hoursCalibration GPU-hoursUnused budgetFinal overshoot
GRPO (independent)0299.2302.75260.00.00.10.4
GRPO (independent)1298.9302.24830.00.00.70.0
GRPO (independent)2299.4302.54670.00.00.30.3
GRPO (independent)3299.1302.55100.00.00.40.2
GRPO (independent)4299.2302.55020.00.00.30.3
GRPO+B (partial sharing)0298.9302.45320.00.00.50.1
GRPO+B (partial sharing)1299.1302.44910.00.00.40.2
GRPO+B (partial sharing)2299.1302.44940.00.00.50.2
GRPO+B (partial sharing)3299.1302.45000.00.00.40.2
GRPO+B (partial sharing)4299.5302.85000.00.00.10.5
GRPO+ESRA (partial sharing)0298.3302.760103.1152.10.20.4
GRPO+ESRA (partial sharing)1298.2302.74299.5169.40.20.4
GRPO+ESRA (partial sharing)2298.8302.45483.5173.50.50.3
GRPO+ESRA (partial sharing)3298.2302.76788.6166.20.10.5
GRPO+ESRA (partial sharing)4297.7302.44193.1176.40.50.1
Calibrated raw-LOO (independent)0297.8302.38782.1162.80.50.0
Calibrated raw-LOO (independent)1297.8302.68184.3164.00.30.3
Calibrated raw-LOO (independent)2298.0302.45884.8175.60.50.1
Calibrated raw-LOO (independent)3298.4302.68392.8153.50.30.3
Calibrated raw-LOO (independent)4298.3302.53989.7179.30.30.3
Calibrated raw-LOO (partial sharing)0298.2302.46189.9164.90.50.1
Calibrated raw-LOO (partial sharing)1298.0302.33197.3177.90.50.1
Calibrated raw-LOO (partial sharing)2298.7302.71593.7190.50.20.4
Calibrated raw-LOO (partial sharing)3298.0302.68090.9157.00.20.3
Calibrated raw-LOO (partial sharing)4298.4302.85079.7184.50.00.5

Calibration refreshes. Each cell is completed / accepted refreshes at job end.

Method and samplingSeed 0Seed 1Seed 2Seed 3Seed 4
GRPO+ESRA (partial sharing)5 / 55 / 45 / 26 / 56 / 5
Calibrated raw-LOO (independent)6 / 36 / 56 / 15 / 56 / 3
Calibrated raw-LOO (partial sharing)5 / 36 / 36 / 45 / 46 / 5

Paired budget differences. Checkpoint cost differences use unrounded records; a displayed 0.0 may be nonzero before rounding.

Comparison (left − right)Seed 0Seed 1Seed 2Seed 3Seed 4
GRPO+B (partial sharing) − GRPO (independent)-0.30.2-0.20.00.3
GRPO+ESRA (partial sharing) − GRPO (independent)-0.10.5-0.20.3-0.2
GRPO+ESRA (partial sharing) − GRPO+B (partial sharing)0.30.20.00.3-0.4
Calibrated raw-LOO (partial sharing) − Calibrated raw-LOO (independent)0.0-0.20.30.00.3

All 25 scored checkpoints remain within budget. The maximum paired cost difference is 0.5 (0.16% of cap); maximum job-end overshoot is 0.2%.

Thank you for the detailed responses and additional experimental results. The sampler validation, per-seed resource accounting, and new experiments on prediction-guided sharing and stochastic interactions have adequately addressed my main concerns. These additions strengthen the practical relevance and credibility of the paper. I appreciate the authors' efforts in addressing my questions. Given these clarifications, I am inclined to raise my score. I will also consider the concerns raised by other reviewers during the discussion phase before making my final decision.

Thank you for taking the time to review the new experiments. We’re glad they addressed your main concerns, and we hope you’ll take these results into account in your final assessment.

Reviewer_vBhPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to handle environmental randomness in reinforcement learning. It provides a thorough theoretical analysis and proposes a strategy that balances the benefits of sharing environmental randomness and preserving independent randomness across rollouts. The problem is meaningful, and the experiments demonstrate the effectiveness of the proposed approach.

Critical strengths
  1. Important problem. Environmental randomness is an important issue in practical RL applications, especially in interactive environments.

  2. Simple and generalizable. The proposed strategy is relatively simple and can potentially be applied to different RL frameworks.

  3. Thorough theoretical analysis. The paper provides a comprehensive theoretical analysis of the trade-off between reducing reward variance and maintaining gradient estimation quality.

Critical weaknesses
  1. Limited methodological novelty. The proposed strategy is relatively straightforward: sharing part of the environmental randomness while preserving independent randomness across rollouts. Similar ideas have been explored in earlier work on common random numbers and related rollout strategies. The main contribution appears to be the theoretical characterization rather than a fundamentally new method.

  2. Limited applicability to real-world environments. In practical RL scenarios, environmental randomness may be difficult or even impossible to control or reproduce. For example, real-world events, external API responses, and interactions with other agents may not be replayable. This raises concerns about the applicability of the proposed method beyond controlled environments.

Questions for authors
  1. How does the proposed sharing strategy differ from earlier approaches? Since sharing environmental randomness while retaining independent rollout randomness is a relatively natural idea, could the authors clarify the methodological novelty beyond the theoretical analysis?

  2. How practical is environment sharing in real-world RL? The experiments rely on environments where specific sources of randomness can be explicitly controlled. How would the method work in real-world scenarios where environmental events cannot be reliably reproduced or synchronized across rollouts?

1 comment

1. Methodological contribution

We take the changes in within-group return comparisons caused by sharing environmental randomness as a starting point. We then systematically investigate when sharing improves group-relative gradient estimation and examine its practical effects through controlled agent training. Prior work learns input-dependent baselines or reuses evaluation scenarios; we change within-group environmental sampling with a fixed estimator.

Identical reward distributions and the same reduction in centered-reward variance can accompany opposite changes in gradient MSE:

Analytical examples: frozen policy, raw-LOO, G=2, fixed N rollouts. Rows: higher-reward action across environments. Arrows: independent → shared.

CaseCentered-reward varianceGradient MSE
Unchanged0.025 → 0.020 (-20%)0.10/N → 0.08/N (-20%)
Reversed0.025 → 0.020 (-20%)0.10/N → 0.16/N (+60%)

The difference lies in how rewards relate to policy scores. We therefore evaluate sharing through the full gradient estimator, including the cost of reduced independent environment coverage. Our finite-group raw-LOO criterion quantifies this trade-off at a frozen policy; independent gradient batches audit its predictions.

These findings inform three concrete designs:

  1. Interaction-level sharing. We preserve trajectory marginals and independent exploration; e.g., matching retrieval requests share content selection with independent ordering.

  2. Gradient-risk diagnosis. Independent-sample gradient inner products estimate variation in mean gradients across environments without squaring noisy estimates; separate batches audit the risk change.

  3. Optional value calibration. Independent gradient references fit return/value advantage weights; fresh data confirm estimated risk reduction before use on new actor data.

Controlled training: mean HotpotQA/ALFWorld gains are 7.10 percentage points for GRPO and 7.00 for matched raw-LOO; value calibration adds 5.50 over sharing-only GRPO.

2. Applicability with unreplayable environmental feedback

Extending the evaluation from controlled environments to stochastic interactions. To evaluate performance with ongoing stochastic feedback and assess our method's effectiveness when some environmental feedback cannot be controlled, we added experiments in τ2\tau^2-bench, using v0.1.3. We chose Telecom because, under our stochastic user configuration, user replies and tool choices vary with the dialogue and device state, while agent–user interactions jointly shape how the environment evolves. This captures several sources of uncertainty in service interactions and allows the same opening to lead to different subsequent trajectories. Using 35B and 9B agent models, we conducted gradient audits and policy continuation in Telecom, followed by transfer evaluation in Retail.

Agents: Qwen3.5-35B-A3B-Base and Qwen3.5-9B-Base; user: fixed Qwen3.5-9B (temperature=1). For unreplayable later feedback, ESRA shares one sampled user opening and state snapshot per group (backend and device states, histories and memory, tool queues and scheduling state). Later replies and tool choices are sampled independently from each rollout's history. Training requires state copying and rollout isolation; evaluation uses independent openings and feedback.

Frozen raw-LOO audit: 16 tasks. GRPO vs sharing-only GRPO+B: 5 paired seeds, same policy/optimizer states, no value calibration, 103 updates/arm/model.

MSE: mean task-conditional risk (×10−3\times10^{-3}); success: %; success gains: pp, from unrounded values.

ModelMetricIndependentESRA samplingDifference
35BMSE12.5510.79-1.76 (-2.64, -0.84)
35BTelecom41.49 ± 1.5347.54 ± 3.47+6.05 (1.93, 9.96)
35BRetail38.25 ± 1.2044.56 ± 3.45+6.32 (1.89, 10.09)
9BMSE15.5613.82-1.74 (-2.61, -0.80)
9BTelecom35.18 ± 1.5940.75 ± 4.04+5.57 (1.49, 9.30)
9BRetail33.16 ± 1.5838.55 ± 4.50+5.39 (1.05, 9.12)

Success: five-seed mean ±\pm SD; 114 tasks ×4\times4 independent interactions/seed/arm/domain. Retail: final evaluation only. 95% intervals: 10,000 paired bootstrap draws over tasks/batch pairs (MSE) or seeds/tasks (success). Two-sided seed sign-flip p=0.125p=0.125 throughout.

Results. With ongoing stochastic feedback, both models show lower gradient risk and higher mean Telecom and Retail transfer success, with greater seed variation. Raw-LOO MSE falls 14.00%/11.18% (35B/9B).

Scope and limitations. These results show that ESRA can still offer benefits in some interactive settings with ongoing stochastic feedback. Broader experiments and deeper analysis are needed for environments with uncontrollable initial conditions and feedback and no useful shareable component, as well as for more complex feedback dependencies and distribution shifts. Limited coverage of these settings is a limitation of the current work and a direction for future research.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.