- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors discover and define credit–update misalignment and propose a new method called EMPO. EMPO groups same-context, same-action responses and shares one length-normalized ratio inside each group while keeping every advantage. The authors also plug into GRPO and GiGPO and evaluate the effectiveness of the method empirically.
Critical strengths
This work is well-written and easy to follow. In particular, the diagnosis is easy to grasp, and the authors conduct relatively comprehensive empirical studies to demonstrate the effectiveness of the proposed method.
Critical weaknesses
- The theory is not as well-established as the framing suggests. Eq. 4 is just the covariance identity, and the variance result assumes that advantage and score are independent within a cell, so the true covariance is zero. In addition, it is not discussed that Eq. 20 implies bias whenever that covariance carries real signal.
- It is not clear where the gain comes from. For example, dropping length normalization collapses results well below plain GRPO. Moreover, a sequence-level ratio baseline might be needed for comparisons.
Questions for authors
- It seems that the mechanism may not explain why WebShop gains when WebShop coverage is low.
- There are no timing numbers to support the claim of small extra computation.
4 comments
Response to Reviewer 3rSY (2/3)
Q2. Performance attribution: sequence-ratio normalization versus effect sharing
We added matched comparisons against GSPO, advantage pooling, and random sharing to isolate the contributions of sequence-ratio normalization and execution-aware sharing.
Under GRPO credit, independent ratios correspond to GSPO, while effect sharing corresponds to GSPO + EMPO. Under GiGPO credit, we use the same GiGPO sequence-ratio objective and compare results with and without EMPO.
Table R2. Matched objective comparisons (SR %, three seeds, mean ± sample std).
| Objective | ALFWorld / GRPO | ALFWorld / GiGPO | WebShop / GRPO | WebShop / GiGPO |
|---|---|---|---|---|
| Effect Sharing (+ EMPO) | 48.05 ± 2.76 | 79.69 ± 1.45 | 57.55 ± 0.45 | 67.97 ± 1.56 |
| Independent Ratios | 39.45 ± 13.81 | 71.09 ± 4.32 | 52.86 ± 7.83 | 64.06 ± 3.41 |
| Advantage Pooling | 42.97 ± 14.36 | 76.56 ± 3.18 | 52.60 ± 6.83 | 61.20 ± 7.26 |
| Random Sharing | 21.09 ± 1.10 | 62.50 ± 2.05 | 50.78 ± 8.45 | 59.11 ± 9.64 |
Effect sharing achieves the highest mean success rate in all four settings.
Specifically:
- GSPO + EMPO achieves 48.05 ± 2.76% SR on ALFWorld, improving over GSPO by +8.60 percentage points.
- GSPO + EMPO achieves 57.55 ± 0.45% SR on WebShop, improving over GSPO by +4.69 points.
- GiGPO + sequence-ratio objective + EMPO achieves 79.69 ± 1.45% SR on ALFWorld, improving over the independent-ratio counterpart by +8.60 points.
- On WebShop, the GiGPO-based setting achieves 67.97 ± 1.56% SR, improving over the independent-ratio counterpart by +3.91 points.
Relative to advantage pooling, effect sharing improves mean SR by +5.08, +3.13, +4.95, and +6.77 points across the four settings.
Relative to random sharing, the improvements are +26.96, +17.19, +6.77, and +8.86 points, respectively.
Effect sharing also exhibits lower cross-seed standard deviations than independent ratios and advantage pooling in all four settings.
Length normalization alone does not explain the observed improvements. With length-normalized sequence ratios held fixed, EMPO further improves mean success rates under both GSPO and GiGPO credit.
The advantage-pooling and random-sharing comparisons further support the contributions of the shared-ratio surrogate and execution-aware grouping.
For WebShop, we additionally measured local sharing coverage: 5.32%–6.71% of processed response occurrences participate in local sharing. This quantifies the intervention's scope, while the controlled comparisons establish the corresponding differences in overall task success rates.
Response to Reviewer 3rSY (3/3)
Q3. Measured computational overhead
We provide direct timing measurements covering actor forward passes, loss computation, and backward passes, with 20 paired repetitions per setting across two hardware-repeat blocks.
Table R3. Measured actor-update computational overhead.
| Setting | Base time (s) | EMPO time (s) | Relative overhead |
|---|---|---|---|
| ALFWorld / GRPO | 1.0732 ± 0.0207 | 1.0878 ± 0.0074 | 1.39% |
| ALFWorld / GiGPO | 1.0939 ± 0.0123 | 1.1132 ± 0.0067 | 1.77% |
| WebShop / GRPO | 1.6048 ± 0.0034 | 1.6320 ± 0.0114 | 1.70% |
| WebShop / GiGPO | 1.5435 ± 0.0132 | 1.5721 ± 0.0152 | 1.86% |
EMPO introduces only 1.39%–1.86% measured actor-update overhead in the profiled settings, without requiring additional training rollouts or auxiliary critics.
These measurements cover actor forward computation, loss computation, and backward propagation. CPU cell construction, environment execution, old/reference probability computation, and checkpointing are outside the timing scope.
We therefore report these measurements specifically as actor-update overhead.
Taken together, the additional experiments strengthen the paper in three respects:
- Mechanism validation: Real environment continuation experiments verify execution invariance across textual realizations under fixed conditions.
- Performance attribution: Controlled GSPO, advantage-pooling, and random-sharing comparisons demonstrate EMPO's incremental gains over existing sequence-ratio objectives.
- Computational efficiency: Direct timing measurements quantify the additional cost of EMPO during actor updates.
We appreciate the reviewer's suggestions and will incorporate these results into the revision to strengthen the theoretical analysis and empirical evaluation of execution-conditioned policy optimization.
Dear Reviewer 3rSY,
We sincerely appreciate your thoughtful review and constructive suggestions.
We have provided additional experimental evidence addressing your concerns, including fixed-condition continuation experiments, controlled comparisons against GSPO and advantage pooling, and measured computational overhead.
We would greatly appreciate it if you could revisit our responses when convenient. We remain happy to discuss any outstanding questions or provide further clarification.
Thank you again for your time and consideration.
Response to Reviewer 3rSY (1/3)
We thank the reviewer for recognizing the motivation and empirical investigation. To address the concerns about theoretical justification, performance attribution, and computational overhead, we provide real environment continuation experiments, controlled objective comparisons, and measured actor-update costs.
Q1. Theoretical mechanism and covariance analysis
EMPO is motivated by the many-to-one mapping from textual responses to executable actions in long-horizon agent RL. Different textual realizations can induce the same executable effect, while conventional policy optimization applies learning signals through their respective response-specific probability ratios.
The covariance decomposition characterizes how this credit–update pairing influences policy updates.
For an idealized fixed cell containing n iid credit–score pairs with population covariance C, at the behavior policy,
E(Gshare)−E(Gind)=−(1−1/n)C
This identity identifies the update component modified by execution-aware sharing. Under the corresponding conditional-independence assumptions, sharing can reduce gradient-estimator variance. When covariance contains genuine conditional signal, sharing can also change the expected update direction.
To examine the conditional basis of execution equivalence, we conducted real ALFWorld prefix-replay experiments.
Table R1. Fixed-condition continuation experiment.
The intervention fixes the full environment state, policy context, and executed action while varying the textual realization.
All 16 cells produce identical next environment states and next policy inputs across textual realizations.
This directly verifies execution invariance under fixed conditions and provides a concrete conditional basis for organizing policy updates through execution equivalence.
Building on this structure, EMPO introduces an effect-conditioned policy surrogate and analyzes its implications for policy updates. Its empirical benefits are examined through the controlled comparisons below.