- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper looks at sharing part of the environment randomness within a rollout group in GRPO-style training of LLM agents. It argues that sharing lowers reward variance but does not necessarily lower gradient error, because it also reduces environment coverage. The authors derive a criterion for when sharing helps, based on an existing U-statistic analysis of GRPO, and check its predictions against measured gradient errors. They also report training gains on HotpotQA and ALFWorld.
Critical strengths
The paper makes a clear point: lower reward variance does not imply a better gradient estimate. The analysis seems correct, and the experiments are set up carefully.
Critical weaknesses
- Limited novelty. The idea that shared random inputs help only when the induced covariance is favorable is well known from the common random numbers literature. The main result mostly applies an existing GRPO analysis to this setting. I did not learn much new from it.
- The theory and the training results are not well connected. The criterion is never used to decide whether to share in training. Also, the randomness being shared is largely introduced by the authors' own environment setup, so it is unclear how much of the gain would carry over to standard settings.
Questions for authors
- How does the baseline perform in the standard environment setup, without the added randomness?
- Does using the criterion to decide whether to share actually help in training, compared with always or never sharing?
3 comments
Thank you for the clarification. The rebuttal has addressed my main concern regarding novelty, and I am raising my score accordingly.
Thank you for your time and constructive feedback. We are glad that our responses have addressed your concerns, and we appreciate your thoughtful reassessment.
1. Novelty (Weakness 1). We agree that favorable covariance is an established mechanism of common random numbers (CRN). However, favorable reward covariance does not guarantee more accurate policy gradients. ESRA further studies cases that reward-level assessments cannot resolve, aiming to determine whether environmental sharing improves or harms learning updates. Our methodological contribution is to turn this ambiguity under reward-only reasoning into an estimable risk comparison between shared and independent collection. Building on the established U-statistic decomposition, we provide an estimable criterion for sharing risk at finite group sizes and use it to guide collection during training.
Favorable reward covariance does not guarantee lower gradient error. Proposition 1 in the main text provides an explicit counterexample. The two cases have identical joint laws of shared inputs and rewards. Rows indicate whether the higher-reward action changes across environments. Arrows: independent → shared (G=2; N: trajectories).
ESRA's exact frozen-policy raw-LOO criterion uses reward–score dependence and variation in mean gradients across environments to guide collection and can improve gradient estimation. Its sign agreement with independent gradient audits is 97/120, versus 64/120 for reward-only and 80/120 without β (Tables 1 and 11). At N=64, increasing group size on HotpotQA turns sharing from beneficial to harmful despite greater centered-reward variance reductions; the full criterion captures this boundary, while removing β misses it (Figure 2).
2. Standard environment (Question 1). Using Qwen3.5-35B-A3B-Base, we compare GRPO-base trained in standard and randomized environments on common standard (STD) tests.
REF-H/REF-A denote the manuscript's randomized HotpotQA/ALFWorld environments, respectively; environmental inputs can be sampled independently or shared. STD removes added randomization but retains native randomness: up to seven ranked HotpotQA passages and native ALFWorld interfaces. Runs use native GRPO without value estimation or λ calibration, matched initialization/data and five paired seeds. We average four attempts per item, then 7,405 questions or 134 unseen games; scores are %, mean±SD.
Both rows use GRPO-base and the same standard tests. The randomized-environment training row uses no environmental input sharing.
Table R1. Standard-environment evaluation.
The standard-environment baseline averages 61.66% across tasks. Training in randomized environments without environmental input sharing changes this by −0.23 pp (95% CI [−0.76,0.24]). Fixed partial sharing gains +6.71 pp [5.53,7.87] over standard-environment training on the same STD tests (95% CI), showing that sharing gains persist without added test-time randomization.
3. Criterion-selected training (Question 2). Using the same Qwen3.5-35B-A3B-Base model, we test whether criterion-selected training improves on the manuscript's GRPO (no environmental input sharing) and GRPO+B (always partial sharing). All three arms train independently from the same initializer, changing only sharing while fixing native GRPO advantages, updates and hyperparameters and omitting full GRPO+ESRA's value estimation and λ calibration.
Table R2 evaluates all arms in the manuscript's randomized environments, reusing Table R1's baseline checkpoints. Diagnostic and validation costs count toward the common 35B training-resource cap.
Table R2. Fixed and criterion-selected environmental input sharing, evaluated in the manuscript's randomized environments (%, mean±SD).
The three randomized-environment arms differ only in environmental input sharing. At updates 0/160/320 (G=8, N=32/domain), the selector diagnoses its current policy by domain using data disjoint from actor/evaluation data. It shares partially only if the input-block-bootstrap 95% upper bound on the full shared-minus-independent raw-LOO risk difference is negative; otherwise, it collects independently. Sharing is selected in 19/30 decisions.
Table R2 shows that criterion-guided sharing outperforms both fixed strategies in the two-task mean. It improves on always partial sharing and never sharing by 2.51 and 9.61 percentage points, respectively.