acceptodds
Under review as a conference paper at ICLR 2027

One Act, Many Masks: Effect-Marginal Policy Optimization for Long-Horizon Agents

Abstract

Recent advances in multi-turn reinforcement learning (RL) have substantially improved the performance of LLM agents on complex, long-horizon interactive tasks. However, despite progress in fine-grained credit assignment, policy optimization still applies these learning signals through response-specific probability ratios. Different textual responses can induce the same executable effect, yet credit differences arising from subsequent outcomes remain tied to their individual textual realizations. We identify this mismatch as . Our analysis reveals that credit-text pairing directly shapes policy-update directions, even when the policy context and executable effect are held fixed. Motivated by this finding, we propose ffect-arginal olicy ptimization (EMPO), which aligns policy optimization with execution-level equivalence through an effect-level surrogate that couples updates across alternative textual realizations. EMPO preserves the original credit signals, integrates with both trajectory-level and step-level credit estimators, and requires no additional critics or rollouts. Experiments on ALFWorld, WebShop, and Sokoban show that EMPO outperforms strong baselines in both task performance and learning efficiency.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Large Language Model, Long-horizon Agent

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3rSYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors discover and define credit–update misalignment and propose a new method called EMPO. EMPO groups same-context, same-action responses and shares one length-normalized ratio inside each group while keeping every advantage. The authors also plug into GRPO and GiGPO and evaluate the effectiveness of the method empirically.

Critical strengths

This work is well-written and easy to follow. In particular, the diagnosis is easy to grasp, and the authors conduct relatively comprehensive empirical studies to demonstrate the effectiveness of the proposed method.

Critical weaknesses
  1. The theory is not as well-established as the framing suggests. Eq. 4 is just the covariance identity, and the variance result assumes that advantage and score are independent within a cell, so the true covariance is zero. In addition, it is not discussed that Eq. 20 implies bias whenever that covariance carries real signal.
  2. It is not clear where the gain comes from. For example, dropping length normalization collapses results well below plain GRPO. Moreover, a sequence-level ratio baseline might be needed for comparisons.
Questions for authors
  1. It seems that the mechanism may not explain why WebShop gains when WebShop coverage is low.
  2. There are no timing numbers to support the claim of small extra computation.
4 comments
AuthorsOpenReview ↗

Response to Reviewer 3rSY (1/3)

We thank the reviewer for recognizing the motivation and empirical investigation. To address the concerns about theoretical justification, performance attribution, and computational overhead, we provide real environment continuation experiments, controlled objective comparisons, and measured actor-update costs.

Q1. Theoretical mechanism and covariance analysis

EMPO is motivated by the many-to-one mapping from textual responses to executable actions in long-horizon agent RL. Different textual realizations can induce the same executable effect, while conventional policy optimization applies learning signals through their respective response-specific probability ratios.

The covariance decomposition characterizes how this credit–update pairing influences policy updates.

For an idealized fixed cell containing n iid credit–score pairs with population covariance C, at the behavior policy,

E(Gshare)−E(Gind)=−(1−1/n)CE(G^{share}) - E(G^{ind}) = -(1 - 1/n)C

This identity identifies the update component modified by execution-aware sharing. Under the corresponding conditional-independence assumptions, sharing can reduce gradient-estimator variance. When covariance contains genuine conditional signal, sharing can also change the expected update direction.

To examine the conditional basis of execution equivalence, we conducted real ALFWorld prefix-replay experiments.

Table R1. Fixed-condition continuation experiment.

MetricResult
Effect cells16
Textual realizations51
Continuation samples1,235
Continuation failures0
Cells with identical next states and policy inputs16/16 (100%)

The intervention fixes the full environment state, policy context, and executed action while varying the textual realization.

All 16 cells produce identical next environment states and next policy inputs across textual realizations.

This directly verifies execution invariance under fixed conditions and provides a concrete conditional basis for organizing policy updates through execution equivalence.

Building on this structure, EMPO introduces an effect-conditioned policy surrogate and analyzes its implications for policy updates. Its empirical benefits are examined through the controlled comparisons below.

AuthorsOpenReview ↗

Response to Reviewer 3rSY (2/3)

Q2. Performance attribution: sequence-ratio normalization versus effect sharing

We added matched comparisons against GSPO, advantage pooling, and random sharing to isolate the contributions of sequence-ratio normalization and execution-aware sharing.

Under GRPO credit, independent ratios correspond to GSPO, while effect sharing corresponds to GSPO + EMPO. Under GiGPO credit, we use the same GiGPO sequence-ratio objective and compare results with and without EMPO.

Table R2. Matched objective comparisons (SR %, three seeds, mean ± sample std).

ObjectiveALFWorld / GRPOALFWorld / GiGPOWebShop / GRPOWebShop / GiGPO
Effect Sharing (+ EMPO)48.05 ± 2.7679.69 ± 1.4557.55 ± 0.4567.97 ± 1.56
Independent Ratios39.45 ± 13.8171.09 ± 4.3252.86 ± 7.8364.06 ± 3.41
Advantage Pooling42.97 ± 14.3676.56 ± 3.1852.60 ± 6.8361.20 ± 7.26
Random Sharing21.09 ± 1.1062.50 ± 2.0550.78 ± 8.4559.11 ± 9.64

Effect sharing achieves the highest mean success rate in all four settings.

Specifically:

  • GSPO + EMPO achieves 48.05 ± 2.76% SR on ALFWorld, improving over GSPO by +8.60 percentage points.
  • GSPO + EMPO achieves 57.55 ± 0.45% SR on WebShop, improving over GSPO by +4.69 points.
  • GiGPO + sequence-ratio objective + EMPO achieves 79.69 ± 1.45% SR on ALFWorld, improving over the independent-ratio counterpart by +8.60 points.
  • On WebShop, the GiGPO-based setting achieves 67.97 ± 1.56% SR, improving over the independent-ratio counterpart by +3.91 points.

Relative to advantage pooling, effect sharing improves mean SR by +5.08, +3.13, +4.95, and +6.77 points across the four settings.

Relative to random sharing, the improvements are +26.96, +17.19, +6.77, and +8.86 points, respectively.

Effect sharing also exhibits lower cross-seed standard deviations than independent ratios and advantage pooling in all four settings.

Length normalization alone does not explain the observed improvements. With length-normalized sequence ratios held fixed, EMPO further improves mean success rates under both GSPO and GiGPO credit.

The advantage-pooling and random-sharing comparisons further support the contributions of the shared-ratio surrogate and execution-aware grouping.

For WebShop, we additionally measured local sharing coverage: 5.32%–6.71% of processed response occurrences participate in local sharing. This quantifies the intervention's scope, while the controlled comparisons establish the corresponding differences in overall task success rates.

AuthorsOpenReview ↗

Response to Reviewer 3rSY (3/3)

Q3. Measured computational overhead

We provide direct timing measurements covering actor forward passes, loss computation, and backward passes, with 20 paired repetitions per setting across two hardware-repeat blocks.

Table R3. Measured actor-update computational overhead.

SettingBase time (s)EMPO time (s)Relative overhead
ALFWorld / GRPO1.0732 ± 0.02071.0878 ± 0.00741.39%
ALFWorld / GiGPO1.0939 ± 0.01231.1132 ± 0.00671.77%
WebShop / GRPO1.6048 ± 0.00341.6320 ± 0.01141.70%
WebShop / GiGPO1.5435 ± 0.01321.5721 ± 0.01521.86%

EMPO introduces only 1.39%–1.86% measured actor-update overhead in the profiled settings, without requiring additional training rollouts or auxiliary critics.

These measurements cover actor forward computation, loss computation, and backward propagation. CPU cell construction, environment execution, old/reference probability computation, and checkpointing are outside the timing scope.

We therefore report these measurements specifically as actor-update overhead.

Taken together, the additional experiments strengthen the paper in three respects:

  1. Mechanism validation: Real environment continuation experiments verify execution invariance across textual realizations under fixed conditions.
  2. Performance attribution: Controlled GSPO, advantage-pooling, and random-sharing comparisons demonstrate EMPO's incremental gains over existing sequence-ratio objectives.
  3. Computational efficiency: Direct timing measurements quantify the additional cost of EMPO during actor updates.

We appreciate the reviewer's suggestions and will incorporate these results into the revision to strengthen the theoretical analysis and empirical evaluation of execution-conditioned policy optimization.

Dear Reviewer 3rSY,

We sincerely appreciate your thoughtful review and constructive suggestions.

We have provided additional experimental evidence addressing your concerns, including fixed-condition continuation experiments, controlled comparisons against GSPO and advantage pooling, and measured computational overhead.

We would greatly appreciate it if you could revisit our responses when convenient. We remain happy to discuss any outstanding questions or provide further clarification.

Thank you again for your time and consideration.

Reviewer_BbLMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies LLM agents whose different textual responses (different reasoning or wording) parse to the same executable action. It argues that response-wise PPO/GRPO updates pair each response's credit with its own realization-specific gradient. It formalizes this through the decomposition GKind=AˉKsˉK+Cov^K(A^,s),G^{\text{ind}}_K = \bar A_K\bar s_K + \widehat{\mathrm{Cov}}_K(\hat A, s), and calls the covariance term "credit–update misalignment."

EMPO groups responses into "effect cells" (matched context key h(c)h(c) and identical parsed action). Within a cell, it replaces each response's token ratios with one shared ratio: the cell average of the length-normalized sequence ratio, qi=exp⁡(1Ti∑tlog⁡ri,t)q_i=\exp(\frac{1}{T_i}\sum_t\log r_{i,t}). Each response keeps its own advantage A^i\hat A_i, and the shared ratio enters the standard clipped surrogate.

Experiments apply EMPO on top of GRPO and GiGPO:

  • ALFWorld and WebShop with Qwen2.5-1.5B and 7B;
  • Sokoban with Qwen2.5-VL-3B.

The paper reports gains in success rate and in time-to-60% success, plus ablations and gradient-cosine and dispersion diagnostics.

Critical strengths
  1. A real and practically relevant observation. In agent RL, the map from text to executed action is many-to-one. For a fixed context and action, credit differences across textual realizations are noise whenever the next context does not depend on the reasoning text. Grouping by executed effect is a sensible, cheap way to exploit this, and it plugs into both GRPO and GiGPO without extra rollouts or critics.
  2. Honest supporting analysis. Appendix C.3 and Table 6 explicitly compare ratio sharing with advantage aggregation and admit the two coincide in the unclipped case. Section 5.5 and Figure 4 report how often cells actually form. Both make the method's scope checkable.
Critical weaknesses

W1. At first order, EMPO changes the credit, not the update; and the update it does change weakens the trust region.

It is credit re-estimation. At θold\theta_{\text{old}}, the shared-ratio gradient is AˉKsˉK\bar A_K\bar s_K. This is exactly what you get by assigning every response in the cell its cell-mean advantage AˉK\bar A_K (the paper's own C.3). That is a Rao-Blackwellized / Monte-Carlo estimate of the action value at (c,e)(c,e), i.e. credit re-estimation, of the same kind as GiGPO's anchor-state grouping and ECPO's canonical-action aggregation (both cited). So the claim that misalignment is "distinct from the quality of the underlying credit estimator" does not hold.

The theory is a textbook identity. Eq. 4 is AB‾=AˉBˉ+Cov\overline{AB}=\bar A\bar B+\mathrm{Cov}. The variance result (C.2) assumes A^⊥s∣z\hat A\perp s\mid z. The paper never states when cell-averaging is actually justified, namely when the next context is independent of the reasoning text (e.g. the history stores only actions). Under that condition the averaging is simply correct variance reduction.

Clipping no longer bounds individual responses. Clipping applies to the cell mean qˉK\bar q_K, not to any individual qjq_j. One realization's ratio can rise while another's falls, leaving qˉK\bar q_K inside [1−ϵ,1+ϵ][1-\epsilon,1+\epsilon] with no clipping. Per-response ratio deviations are therefore no longer controlled, which defeats the purpose of PPO's trust region.

The paper's own diagnostics are consistent with this:

  • In Figure 7, KL to the reference grows several times faster for EMPO-GiG on WebShop.
  • Entropy rises late in training (Figs. 6–7).

The paper presents both as beneficial "continued adaptation," without any analysis of the per-response ratio spread inside cells.

Length normalization changes the objective. The shared direction uses length-normalized scores ∇1Tilog⁡π\nabla\frac{1}{T_i}\log\pi, not the effect score in Eq. 6. The resulting update is no longer the gradient of any effect-level objective. Under token-weighted aggregation, Eq. 18 additionally weights the cell's direction by TiA^iT_i\hat A_i.

W2. The ablations point to a different source of the gains, and key baselines are missing.

  • "w/o Ratio Sharing" is essentially GSPO-style ratios. It keeps the length-normalized sequence ratio without sharing, i.e. a GSPO-style sequence ratio, and scores 79.7 on ALFWorld vs GRPO's 69.5. Most of the ALFWorld gain from GRPO to EMPO-G (69.5 → 83.3) is therefore already explained by the ratio form. Yet GSPO, which the paper cites, is not a baseline.
  • "w/o Length Normalization" falls below GRPO (WebShop SR 27.3 vs 62.8). This mostly shows that unnormalized sequence ratios are unstable, which is already known, rather than validating effect sharing.
  • Advantage averaging nearly matches EMPO. Table 6 shows the simple advantage-averaging variant within ~1–3 points of EMPO in three of four settings.
  • The decisive comparisons are missing:
    • GSPO;
    • GSPO + cell-mean advantages;
    • ECPO.
  • The context key is unclear. If h(c)h(c) is GiGPO's anchor state, then EMPO-G is essentially GRPO plus GiGPO-style step-level grouping, which explains its gains without the "misalignment" story.

The statistics are also weak:

  • 3 seeds against very large baseline variance (GRPO 1.5B ALFWorld 69.5 ± 11.1; 7B 71.9 ± 9.0).
  • Several 7B per-task deltas are negative.
  • Sokoban reports no deviation.
  • The checkpoint-selection rule across the every-5-iteration evaluations is not stated.
  • PPO, RLOO and the prompting rows appear identical to the numbers in Feng et al. (2025) and seem not to have been rerun.
  • WebShop gains are large even though only ~2–25% of responses fall into eligible cells. That leaves much of the improvement unexplained by the mechanism.
Questions for authors
  1. (Decisive.) Please add GSPO and "GSPO + cell-mean advantage" baselines under identical settings. How much of EMPO's gain remains over the better of the two?
  2. What exactly is h(c)h(c) for each benchmark, and for EMPO-G in particular: the full history, GiGPO's anchor observation, or something else? Does the next-step context include the previous response's reasoning text? If it does, realizations sharing an effect do not share consequences, and averaging is biased.
  3. Please report the within-cell spread of individual ratios qjq_j (e.g. max ∣qj−1∣|q_j-1|) and the clip fraction, compared with the base method. Does EMPO's larger KL come from unclipped individual ratio drift?
  4. How are checkpoints selected for Tables 1 and 2? Were PPO and RLOO rerun or taken from Feng et al. (2025)?
  5. Given 2–25% cell coverage on WebShop, what is the gain when restricting evaluation to tasks where cells actually form, or when EMPO is applied only to the covered responses?
6 comments
AuthorsOpenReview ↗

Response to Reviewer BbLM (1/4)

We thank the reviewer for the detailed technical analysis. To address the questions concerning EMPO's incremental contribution, its relationship to credit pooling, shared-ratio clipping, and WebShop coverage, we provide controlled objective comparisons, fixed-condition continuation experiments, and individual-ratio diagnostics.

Q1. Does EMPO provide additional gains over GSPO and GSPO + cell-mean advantages?

We added four matched objective comparisons: independent ratios, effect sharing, advantage pooling, and random sharing.

Under GRPO credit, independent ratios correspond to GSPO, effect sharing corresponds to GSPO + EMPO, and advantage pooling corresponds to GSPO + cell-mean advantages. Under GiGPO credit, the variants use the same sequence-ratio objective with GiGPO credit.

Table R1. Matched objective comparisons (SR %, three seeds, mean ± sample std).

ObjectiveALFWorld / GRPOALFWorld / GiGPOWebShop / GRPOWebShop / GiGPO
Effect Sharing (+ EMPO)48.05 ± 2.7679.69 ± 1.4557.55 ± 0.4567.97 ± 1.56
Independent Ratios39.45 ± 13.8171.09 ± 4.3252.86 ± 7.8364.06 ± 3.41
Advantage Pooling42.97 ± 14.3676.56 ± 3.1852.60 ± 6.8361.20 ± 7.26
Random Sharing21.09 ± 1.1062.50 ± 2.0550.78 ± 8.4559.11 ± 9.64

Effect sharing achieves the highest mean success rate across all four settings.

Compared with GSPO, GSPO + EMPO improves mean SR by +8.60 points on ALFWorld and +4.69 points on WebShop.

With GiGPO credit, adding EMPO improves mean SR by +8.60 points on ALFWorld and +3.91 points on WebShop.

Relative to advantage pooling, the improvements are +5.08, +3.13, +4.95, and +6.77 points. Relative to random sharing, they are +26.96, +17.19, +6.77, and +8.86 points, respectively.

Effect sharing also exhibits lower cross-seed standard deviations than independent ratios and advantage pooling in all four settings.

These controlled comparisons demonstrate additional mean SR gains from EMPO under the same sequence-ratio objective. The advantage-pooling comparison supports the empirical relevance of the nonlinear shared-ratio surrogate, while the random-sharing comparison supports the importance of execution-aware grouping.

AuthorsOpenReview ↗

Response to Reviewer BbLM (2/4)

Q2. Relationship to credit re-estimation and theoretical contribution

Ratio sharing has a precise mathematical relationship to advantage pooling, while their nonlinear policy optimization objectives exhibit different update behavior.

For fixed cells, response-balanced reduction, and the unclipped policy-gradient term, define

qˉK=1∣K∣∑j∈Kqj,AˉK=1∣K∣∑i∈KA^i.\bar{q}_K = \frac{1}{|K|}\sum_{j\in K}q_j, \qquad \bar{A}_K = \frac{1}{|K|}\sum_{i\in K}\hat{A}_i.

Then,

1∣K∣∑i∈KA^iqˉK=AˉK1∣K∣∑j∈Kqj.\frac{1}{|K|}\sum_{i\in K}\hat{A}_i\bar{q}_K = \bar{A}_K\frac{1}{|K|}\sum_{j\in K}q_j.

Thus, under these conditions, ratio sharing and cell-mean advantage pooling are exactly equivalent in their unclipped objective terms.

Under token-weighted reduction, the equivalent coefficient on an independent ratio becomes

A^j′=∑i∈KTiA^i∣K∣Tj.\hat{A}'_j = \frac{\sum_{i\in K}T_i\hat{A}_i}{|K|T_j}.

With PPO-style clipping, EMPO clips the shared cell ratio, whereas advantage pooling clips individual response ratios. These operations induce different nonlinear surrogate objectives.

The new experiments demonstrate the empirical relevance of this distinction: effect sharing achieves higher mean SR than advantage pooling in all four controlled settings.

For an idealized fixed cell containing n iid credit–score pairs with population covariance C, at the behavior policy,

E[Gshare]−E[Gind]=−(1−1n)C.\mathbb{E}[G^{\mathrm{share}}] - \mathbb{E}[G^{\mathrm{ind}}] = -\left(1-\frac{1}{n}\right)C.

This identity characterizes the update component modified through credit–score pairing. Under conditional independence, sharing can reduce gradient-estimator variance; when covariance contains genuine conditional signal, it can also change the expected update direction.

We additionally conducted real ALFWorld prefix-replay experiments.

Table R2. Fixed-condition continuation experiment.

MetricResult
Effect cells16
Textual realizations51
Continuation samples1,235
Continuation failures0
Cells with identical next states and policy inputs16/16 (100%)

The intervention fixes the full environment state, policy context, and executed action while varying the textual realization. All 16 cells produce identical next environment states and next policy inputs, directly verifying execution invariance under fixed conditions.

This provides a concrete conditional basis for sharing updates across execution-equivalent textual realizations.

EMPO exploits this structure through an effect-conditioned policy surrogate, whose optimization behavior and downstream performance are examined in our controlled experiments.

AuthorsOpenReview ↗

Response to Reviewer BbLM (3/4)

Q3. Cell-level clipping and individual-ratio behavior

EMPO applies ratio sharing and clipping at the cell level. We additionally measured individual normalized ratios in WebShop/GRPO-credit runs over training steps 1–100 to characterize the resulting update behavior.

Table R3. Individual-ratio diagnostics.

StatisticIndependent RatiosEffect Sharing
Mean within-cell Std(q_i)0.000680–0.0009340.000814–0.001064
Largest within-cell ratio range0.02179–0.024730.02865–0.04922
p99 of abs(log(q_i))0.0215–0.02290.0216–0.0259
Maximum abs(log(q_i))0.0964–0.16910.1576–0.3242
Active clipped-branch fraction31.71%–33.11%32.49%–33.23%

The intervals indicate the ranges of per-run statistics across seeds 0/2/3.

Among shared response occurrences whose cell-mean ratio lies within the clipping interval [0.9997, 1.0004], individual ratios fall outside that interval in 9.35%, 6.86%, and 6.24% of cases across the three seeds.

These measurements characterize the within-cell dispersion and tail behavior of individual ratios under cell-level clipping. Shared clipping permits individual-ratio deviations, which can be examined alongside the KL diagnostics reported in the paper.

We will include these additional diagnostics in the revision.

Reviewer_BDToOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EMPO addresses long-horizon agent RL by grouping textual responses with the same context–effect key and sharing their length-normalized likelihood ratios while retaining the original advantages. It integrates with GRPO and GiGPO without additional rollouts or auxiliary models. Tables 1–3 show improved success rates and fewer training iterations across three environments. Tables 4 and 6 and Figures 3–8 examine components and update mechanisms. However, direct evidence that execution equivalence justifies decoupling credit from response-specific gradients remains missing.

Critical strengths
  1. The paired comparisons are reasonably controlled. Section 5.1 matches initialization, rollouts, and budgets. In Table 1, 1.5B ALFWorld SR increases from 69.5 to 83.3, supporting practical utility.

  2. The mechanism analysis is inspectable. Appendix C.3 explains the relationship to advantage averaging, Table 6 provides a direct comparison, and Figure 3 examines realized one-step updates.

Critical weaknesses
  1. The removed covariance is not shown to be harmful. Variance reduction requires conditional independence; otherwise, the expected update changes (Appendix C.2). Figures 3 and 8 cannot distinguish noise from useful credit.

  2. The distinct contribution is not isolated. Explicit GSPO and carefully matched random-grouping controls are missing. In Table 6, GRPO WebShop SR is 70.2 versus 70.3, without reported variance.

Questions for authors
  1. How are h(c)h(c) and canonical effects constructed in each environment? Do they match the task, full history, step index, and observation? How are cells handled across mini-batches (Section 4.3, Algorithm 1)?

  2. Can independent continuations from a fixed full context and action establish that within-cell covariance primarily reflects return noise rather than useful credit (Appendix C.2, Figure 8)?

  3. Can you add budget-matched GSPO, carefully matched random grouping, and advantage aggregation with consistent token weighting, report multi-seed differences, and clarify checkpoint selection (Tables 4 and 6, Appendix D.4)?

4 comments
AuthorsOpenReview ↗

# Response to Reviewer BDTo (1/2)

We thank the reviewer for the constructive comments. To address the questions concerning execution equivalence, structured sharing, and performance attribution, we provide fixed-condition continuation experiments and controlled objective comparisons.

Q1. Context keys, executable effects, and minibatch handling

EMPO constructs each effect cell using a task-group identifier, a context key, and a canonical executed action. Specifically, ALFWorld matches the current anchor observation and normalized executable command; WebShop uses a hash of the complete constructed prompt and the canonical search/click action; Visual Sokoban uses a hash of the environment state and the executed direction.

This design groups textual realizations according to their decision context and executable effect. In ALFWorld, generated reasoning is not retained in the next policy input, so differences in textual realization do not directly alter subsequent decision contexts.

Sharing operates within actor microbatches. When an effect cell contains at least two eligible responses, they use a shared length-normalized sequence ratio. Other responses retain independent normalized ratios.

Q2. Independent continuations under fixed conditions

We conducted real ALFWorld prefix-replay experiments to examine whether execution-equivalent textual realizations induce identical subsequent decision conditions. The intervention fixes the full environment state, policy context, and executed action while varying the textual realization.

Table R1. ALFWorld fixed-condition continuation experiment.

MetricResult
Preselected effect cells16
Textual realizations51
Continuation samples1,235
Continuation failures0
Cells with identical next states and policy inputs16/16 (100%)
Independent continuations per nonterminal condition unit16

All 16 cells produced identical next environment states and next policy inputs across textual realizations.

This directly verifies a central conditional premise of EMPO: when the full decision condition and executed action are fixed, differences in reasoning text that is neither executed nor retained do not change the subsequent decision problem.

Execution equivalence therefore provides a concrete conditional basis for sharing updates across textual realizations. These real environment interactions also support our analysis of realization-dependent components of credit–update covariance.

AuthorsOpenReview ↗

Response to Reviewer BDTo (2/2)

Q3. Controlled comparisons with GSPO, advantage pooling, and random grouping

We added four matched objective comparisons: independent ratios, effect sharing, advantage pooling, and random sharing.

Under GRPO credit, independent ratios correspond to GSPO, while effect sharing corresponds to GSPO + EMPO. Under GiGPO credit, all variants use the same GiGPO credit estimator and sequence-ratio objective, differing in their ratio-sharing or advantage-pooling mechanisms.

The comparisons use length-normalized sequence ratios, response-balanced loss reduction, the same clipping interval, and matched training budgets.

Table R2. Matched objective comparisons (SR %, three seeds, mean ± sample std).

ObjectiveALFWorld / GRPOALFWorld / GiGPOWebShop / GRPOWebShop / GiGPO
Effect Sharing (+ EMPO)48.05 ± 2.7679.69 ± 1.4557.55 ± 0.4567.97 ± 1.56
Independent Ratios39.45 ± 13.8171.09 ± 4.3252.86 ± 7.8364.06 ± 3.41
Advantage Pooling42.97 ± 14.3676.56 ± 3.1852.60 ± 6.8361.20 ± 7.26
Random Sharing21.09 ± 1.1062.50 ± 2.0550.78 ± 8.4559.11 ± 9.64

Effect sharing achieves the highest mean success rate in all four settings.

Relative to independent ratios, the mean SR improvements are +8.60, +8.60, +4.69, and +3.91 percentage points, respectively.

Relative to advantage pooling, the improvements are +5.08, +3.13, +4.95, and +6.77 points. Relative to random sharing, the improvements are +26.96, +17.19, +6.77, and +8.86 points.

Effect sharing also exhibits lower cross-seed standard deviations than independent ratios and advantage pooling in all four settings.

These experiments establish three findings:

  1. Incremental gains over GSPO: Adding EMPO improves mean SR by +8.60 points on ALFWorld and +4.69 points on WebShop.
  2. Structured versus random sharing: Effect sharing consistently outperforms random sharing, demonstrating the importance of execution-aware grouping.
  3. Sharing versus credit pooling: Effect sharing achieves higher mean SR than advantage pooling under both credit estimators and across both environments.

Together with the fixed-condition continuation experiments, these results provide direct evidence for the conditional basis of execution-aware sharing and its incremental value in long-horizon agent RL.

We appreciate the reviewer's suggestions and will incorporate these results into the revision.

Dear Reviewer BDTo,

We sincerely appreciate your constructive feedback and the time you have devoted to reviewing our work.

We have posted additional responses addressing your main concerns, including fixed-condition continuation experiments, matched GSPO and random-sharing controls, and clarifications of effect-cell construction.

We would be grateful if you could take a look at the new evidence when convenient. We would also welcome any further questions or suggestions and would be happy to clarify any remaining concerns.

Thank you again for your time and consideration.

Reviewer_BDToOpenReview ↗

Reply to the authors

Thanks to the authors for the additional experiments, which address some of my concerns. However, I still find the justification for removing the within-effect covariance insufficient. I will therefore maintain my score of 2.

Comments on the paper

1 comment
AuthorsOpenReview ↗

We appreciate the time and dedication of the Area Chair and all reviewers

Dear Area Chair,

We sincerely appreciate your time and valuable guidance throughout the discussion, as well as the reviewers' careful and constructive assessments. We would like to summarize the additional empirical evidence and technical clarifications provided in our rebuttal.

We believe that our rebuttal substantively addresses the main concerns through controlled objective comparisons, direct verification of execution equivalence, and further analysis of policy-update behavior:

  1. [To Reviewer BDTo] We conducted real ALFWorld prefix-replay experiments on 16 preselected effect cells containing 51 textual realizations, completing 1,235 continuation samples. With the full environment state, policy context, and executed action fixed, all 16/16 cells produced identical next environment states and next policy inputs, providing direct evidence of execution invariance under fixed conditions. We also clarified the context/effect keys across three environments and the microbatch-local sharing mechanism. In matched three-seed experiments, GSPO + EMPO improves SR over GSPO by +8.60 points on ALFWorld and +4.69 points on WebShop. Compared with random sharing, effect sharing improves SR by +26.96/+17.19 points on ALFWorld and +6.77/+8.86 points on WebShop under GRPO/GiGPO credit, supporting the importance of execution-aware grouping in shared-ratio optimization.

  2. [To Reviewer BbLM] We added matched GSPO, GSPO + cell-mean advantage, and random-sharing controls. Across ALFWorld and WebShop with GRPO/GiGPO credit, effect sharing achieves the highest mean SR in all four settings. Relative to independent ratios, the improvements are +8.60, +8.60, +4.69, and +3.91 points; relative to advantage pooling, +5.08, +3.13, +4.95, and +6.77 points, with lower cross-seed standard deviations in all four settings. Theoretically, we clarify the exact equivalence between unclipped ratio sharing and cell-mean advantage pooling under response-balanced reduction, and their distinct nonlinear surrogates after clipping. Individual-ratio diagnostics show that 6.24%–9.35% of individual ratios fall outside the clipping interval even when their shared cell mean remains inside. Together with the random-sharing comparisons, these findings highlight the role of execution-aware grouping and its interaction with nonlinear clipping. Regarding the difference between the original Table 1 and the new results, the two experiments use different loss-reduction and ratio-fallback configurations; the new experiments are designed for controlled objective attribution under a unified training recipe.

  3. [To Reviewer 3rSY] We further clarify the theoretical interpretation of the covariance identity and the conditional assumptions underlying variance reduction. The 16-cell, 1,235-continuation experiment directly verifies execution invariance under fixed conditions. Matched comparisons against GSPO, advantage pooling, and random sharing further distinguish the contributions of sequence-ratio normalization and effect sharing. For computational efficiency, 20 paired timing repetitions per setting show only 1.39%–1.86% measured actor-update overhead across ALFWorld/WebShop with GRPO/GiGPO, without additional training rollouts or auxiliary critics, providing direct evidence of EMPO's lightweight implementation.

We will incorporate these results and clarifications into the revision, further strengthening the discussion of execution equivalence, policy-update mechanisms, and related work. We welcome any additional technical questions during the remaining discussion phase and respectfully hope that the Area Chair and reviewers will consider the complete rebuttal and additional evidence.

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.