acceptodds
Under review as a conference paper at ICLR 2027

Same Groups, Different Values: Decomposing Step-Level Credit Assignment for LLM Agents

Abstract

Step-level group-relative estimators for language-model agents are often characterized by the states they place in the same group. We show that this description is incomplete: the rollout partition determines where statistics are shared, whereas the value functional determines what signal is aggregated. We introduce a component-resolved audit that varies these choices separately on matched rollouts, and complement it with a support-aware variance decomposition for measuring history-associated value variation after the task and current observation are fixed. The framework combines an unbalanced random-effects estimator with task-level resampling, explicit accounting of variance-bearing groups, and replay-based validation of the collected trajectories. Across replay-verified policy snapshots, two text environments, and an independently emitted matched-row batch, the audit reveals that apparently different grouping methods can remain closely aligned when they retain the same base operator, while added value machinery can dominate their differences. It also shows why sparse terminal feedback can leave most formally eligible groups uninformative even when a denser progress signal varies within them. The academic value is a reusable attribution protocol: it separates algorithmic structure, measurable support, inferential units, and data fidelity before a snapshot difference is assigned a mechanism.

Primary area:
reinforcement learning
Keywords:
step-level credit assignment, group-relative policy optimization, language-model agents, reinforcement learning, estimator attribution, variance components, effective support, trajectory replay

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2gsDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Existing LLM-agent RL methods claim to improve credit assignment through different state or history grouping strategies. This paper tries to propose an auditing framework to decouple how samples are grouped and how values/advantages are computed after grouping. After disentangling them, the authors find that GiGPO and G2PO use essentially the same base grouping, while HGPO has different grouping but still produces advantage estimates that are very close to those of GiGPO.

Critical strengths
  1. The motivation is clear and I agree with auditing the step-level credit assignment through a clean decomposition can be beneficial

  2. The controlled comparison is very clean. It makes the observed differences in the advantage vectors much easier to attribute to that component itself.

Critical weaknesses
  1. My main concern is the significance of the contribution. The paper carefully decomposes and attributes step-level advantage estimators, but the analysis does not lead to meaningful guidance for training and new credit-assignment methods. So far in my opinion, the practical and scientific implications of the decomposition remain limited.

  2. The technical novelty of the auditing framework seems to also be limited. The ICC analysis seems to be standard, and does not have a lot of practical significance given sparse rewards.

  3. The paper writing seems very weird to me. The introduction is unusually brief and does not sufficiently motivate the scientific stakes of the proposed decomposition. And the writing gives me the impression not of a paper, but a blog documenting some thoughts...

Questions for authors

My main question is on the practical value of the paper and it will significantly influence my assessment on the paper:

Is the framework proposed primarily a tool for attributing differences in step-level advantage estimates to different estimator components? If so, what actionable guidance does this attribution provide for agent training or credit-assignment design? For example, does the framework help determine when a particular estimator should be preferred, which components are beneficial under different training regimes, or how to design a better estimator? Currently the paper seems to mainly focus on identifying where estimator differences come from.

5 comments
AuthorsOpenReview ↗

Official Response to Reviewer 2gsD (Part 1/4)

Thank you for recognizing the motivation and controlled comparison. We address your questions below.

About the Practical Value of the Framework

Is the framework proposed primarily a tool for attributing differences in step-level advantage estimates to different estimator components? If so, what actionable guidance does this attribution provide for agent training or credit-assignment design?

The framework identifies which intervention can test a proposed explanation. History refinement can have a small effect because the flat level survives; additional G2PO value machinery can change signals substantially without changing the flat partition; and subdividing a constant-return anchor cannot create return-relative contrast. These findings guide component selection and experimental design.

The history footprint is quantified beyond cosine: on the verified 1.5B snapshot, normalized mean absolute GiGPO/HGPO difference is 0.072 and strict sign reversals are 0.09% (§4.4; Appendix J). This measures the intervention's size; E5 tests its complete-operator learning outcome.

The support constraint is specific. If every row in an anchor has Ri=cR_i=c, then zG(Ri)=(c−c)/(sG(R)+ϵ)=0z_G(R_i)=(c-c)/(s_G(R)+\epsilon)=0, and each history refinement retains the same zero centered signal. Other components, including an episode-level term, may still contribute to the update.

This leads to a practical sequence: count return-varying anchors, check surviving refinements and value constructions on matched rows, then test learning with independent runs. E4/E6 implement the prediction and regularization checks described in Parts 3–4/4. The eight-pair training extension (E5) gives HGPO-minus-GiGPO final success of +15.039 percentage points at update 20 (95% paired-t interval [5.216, 24.863]; two-sided exact sign-flip p=0.016) on 128 held-out ALFWorld tasks. This comparison evaluates the complete operators, including their normalization and episode-term differences, under the stated 20-update budget. The full protocol and seed table appear in the response to ovnA, Part 3/5.

Thus the added training result addresses downstream usefulness under a fixed budget, while the audit explains which component comparisons are needed to interpret it. ICC remains a standard supporting tool.

AuthorsOpenReview ↗

Official Response to Reviewer 2gsD (Part 2/4)

About the Technical Contribution and Presentation

The ICC analysis seems to be standard, and does not have a lot of practical significance given sparse rewards.

The introduction is unusually brief and does not sufficiently motivate the scientific stakes of the proposed decomposition.

We use standard ICC (§2). Proposition 1 specifies the unbalanced moment identity and its assumptions; its role is to measure within-anchor history association with explicit support and inferential units:

E[MSW]=σW2,E[MSB]=σW2+nˉ0σB2.\mathbb{E}[MS_W]=\sigma_W^2,\qquad \mathbb{E}[MS_B]=\sigma_W^2+\bar n_0\sigma_B^2.

The moment identity applies to the untruncated between component under fixed non-informative label sizes and common independent effects (Proposition 1; Appendix C). Whole-task resampling provides uncertainty intervals for the ICC ratio while preserving within-task trajectory dependence.

Sparse reward makes support accounting consequential: only 90/1,964 preferred anchors carry return variation. The nominal-to-informative count ratio is about 21.82. This ratio quantifies the difference between nominally eligible and return-varying anchor counts.

The contribution is the controlled attribution and support protocol. Its empirical content is the crossed comparison: hierarchy can change while the signal stays close, whereas the same flat partition can accompany a large change when value construction changes (Table 1). The independent producer and Yang fallback checks (§4.6; Appendix B) test this attribution beyond a single emitter. E4–E6 connect the diagnostics to prediction, regularization, and complete-operator training.

The scientific question can be stated directly: when a structured estimator differs from a flat one, is the cause the partition, the value construction, or the available return support? Matched controls address the first two; support analysis addresses the third. The design recommendations and E4–E6 results connect this decomposition to predictive accuracy and policy learning.

AuthorsOpenReview ↗

Official Response to Reviewer 2gsD (Part 3/4)

About the History Support and Predictive Usefulness (E6)

For example, does the framework help determine when a particular estimator should be preferred, which components are beneficial under different training regimes, or how to design a better estimator?

E6 tests the design choice suggested by finite support: shrink history predictions toward the flat anchor estimate, and select the strength using other tasks.

Experiment settings.

Using the same collections and whole-trajectory holdouts as E4, the raw-history predictor requires at least two training rows from two independent trajectories; otherwise it uses exactly the flat anchor mean. Shrinkage combines history and flat predictions with distinct-trajectory weight n(h): p(kappa)=[n(h)p(history)+kappa p(flat)]/[n(h)+kappa]. The grid is {0,1,2,4,8,16,32,infinity}. For each target task, kappa is selected from pooled loss on all other tasks, with ties favoring larger kappa. Anchor means may still use other trajectories of the target task. We report row-weighted MSE with 2,000 paired whole-task bootstrap draws (seed 25576), repeating task-excluded selection in every draw.

Results.

CollectionTasks/rowsSupported rowsFlat MSERaw-history MSETuned MSETuned-flat [95% task CI]
ALFWorld 1.5B100/33508202060.4130.3020.262-0.152 [-0.293, -0.045]
Supplementary 7B239/51768258051.3421.1011.031-0.311 [-0.419, -0.217]
ScienceWorld/no-op removed80/579725930.100.0780.075-0.026 [-0.034, -0.018]
CollectionSupported flat MSESupported tuned MSEFallback MSE (all methods)
ALFWorld 1.5B0.4460.1950.363
Supplementary 7B1.130.5051.554
ScienceWorld/no-op removed0.0970.040.103

Shrinkage reduces MSE relative to both the flat and raw-history predictors. Selected kappa values are 2 for all 100 ALFWorld 1.5B tasks, 1 for all 239 supplementary 7B tasks, and 1 for all 80 ScienceWorld tasks; no task selects another grid value. Selection sets overlap because each excludes only its target task. Fallback predictions are identical across methods.

This shows how the support diagnosis guides a specific, testable regularization change. It evaluates auxiliary RTG prediction; the separate E5 comparison evaluates native-operator policy learning.

Differences, intervals and derived metrics are computed before rounding.

Reviewer_cgivOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper audits step-level credit assignment for LLM agents by separating how rollout steps are grouped from how their advantage signals are computed. On matched rollout records, it compares GiGPO, faithful HGPO, a grouping-only G2PO control, and full G2PO. It also measures history-associated return variation within fixed task/observation groups, examines how sparse rewards limit informative groups, and checks trajectory replay and statistical sensitivity. The experiments use Qwen2.5-Instruct snapshots in ALFWorld and ScienceWorld. The work concerns fixed-data diagnostics; it does not compare training outcomes or establish a ranking of credit-assignment quality.

Critical strengths
  1. The matched-rollout controls separate grouping changes from changes to value construction. This is a useful way to avoid attributing all differences between credit-assignment methods to their grouping rules.

  2. The paper transparently reports implementation corrections, replay failures, and withdrawn analyses. Explicitly counting groups with return variation is also useful: many nominal groups may provide little information under sparse terminal rewards.

Critical weaknesses
  1. The scientific takeaway and guidance for credit assignment remain limited. The paper shows that the methods produce similar or different advantage vectors, but does not establish which method gives better credit assignments or why it should be preferred. Vector similarity or disagreement alone does not establish estimator quality or downstream usefulness. I recognize that a measurement paper need not introduce an optimizer or rank methods; however, the paper still needs to demonstrate what substantial, non-obvious knowledge follows from these measurements. In particular, identical grouping and value computations yielding identical signals is not itself a strong finding. The connection from the remaining diagnostics to a useful methodological conclusion is insufficiently established for me.

  2. The empirical support is narrow and difficult to assess independently. Only one model family is studied. Removing no-op steps reduces the ScienceWorld ICC from 0.154 to 0.026; the authors appropriately withdraw the magnitude comparison, but this leaves weaker cross-environment support for the history-associated variance findings. The two benchmarks therefore do not validate every central result in the same way. In addition, the exact task splits, task/variation coverage, and reason for the 800 versus 1,320 ALFWorld trajectories are not sufficiently clear. Matching trajectory counts does not necessarily match task composition. With no accessible analysis code or raw rollout records, the central numerical claims cannot currently be independently recomputed. Together, these limitations leave me unconvinced that the retained findings provide a sufficiently substantial and general contribution. My recommendation is clear rejection.

Questions for authors
  1. What is the main non-obvious scientific conclusion beyond attributing differences between computed advantage vectors to their components? Which prior interpretation does the evidence overturn, and what concrete choice about credit assignment should change as a result? Is there evidence that connects the measured differences to signal quality or usefulness, rather than only to numerical disagreement?

  2. Please specify the exact benchmark splits or collection partitions, unique task/variation counts, task-type coverage, and selection rules for each analysis. Were complete standard splits used or selected subsets? Why are there 800 trajectories for 1.5B and 1,320 for 7B, and were the task sets and collection rules identical? How are task-composition differences separated from policy differences?

  3. After no-op removal, which ScienceWorld conclusions remain supported? Please report the remaining task/group support and clarify which findings are demonstrated in both environments, versus only in ALFWorld. How should the remaining ICC of 0.026 be interpreted given the large change from 0.154?

  4. Could a minimal anonymous rollout sample and analysis implementation be provided to check the main operator comparison and variance calculation? Please also clarify the scope of the one-family evidence: which conclusions are implementation identities, and which depend on the sampled Qwen policies?

8 comments
AuthorsOpenReview ↗

Official Response to Reviewer cgiv (Part 1/7)

Thank you for the careful review. We address your questions below.

About the Scientific Conclusion and Signal Quality

What is the main non-obvious scientific conclusion beyond attributing differences between computed advantage vectors to their components? Which prior interpretation does the evidence overturn, and what concrete choice about credit assignment should change as a result? Is there evidence that connects the measured differences to signal quality or usefulness, rather than only to numerical disagreement?

The identity is the control that makes the crossed comparison interpretable: additional hierarchy can leave signals close, while an unchanged flat partition can accompany large changes when value construction changes. The interpretation challenged is that flat/history/graph labels alone explain the credit signal. This identifies the source of computed-signal differences across the existing methods (§2; Appendix E).

On matched rows, GiGPO/HGPO cosines are 0.965/0.944/0.937; GiGPO/full-G2PO cosines are 0.187/0.272/0.183; and the grouping-only control is 1 in all collections (Table 1). Table 5 locates full-G2PO differences on rows carrying most of the reference advantage energy. These observations identify value-construction ablations that a grouping-only comparison would miss.

The support result adds a practical constraint: subdividing a constant-return anchor cannot create return-relative contrast. It changes the evaluation order: establish available contrast, check which refinements contribute, and then test learning effects.

We distinguish signal disagreement from quality. For score vectors si=∇log⁡π(ai∣hi)s_i=\nabla\log\pi(a_i\mid h_i) and Kij=si⊤sjK_{ij}=s_i^\top s_j, gradient cosine is

A⊤KA′(A⊤KA)(A′⊤KA′).\frac{A^\top K A'}{\sqrt{(A^\top K A)(A'^\top K A')}}.

Advantage cosine measures fixed-row signal alignment; gradient cosine additionally depends on the score geometry shown above (Appendix L). We report the former as a component diagnostic and assess task success directly in E5.

We added separate usefulness checks: E4/E6 evaluate held-out prediction and support-dependent regularization; E5 evaluates complete-operator policy learning. The eight-pair training extension (E5) gives HGPO-minus-GiGPO final success of +15.039 percentage points at update 20 (95% paired-t interval [5.216, 24.863]; two-sided exact sign-flip p=0.016) on 128 held-out ALFWorld tasks. This comparison evaluates the complete operators, including their normalization and episode-term differences, under the stated 20-update budget. The full seed table and protocol appear in the response to ovnA, Part 3/5. This supplies a downstream test while preserving the distinction between a complete-operator effect and a pure grouping mechanism.

AuthorsOpenReview ↗

Official Response to Reviewer cgiv (Part 2/7)

About the Collection Composition and ScienceWorld Support

Please specify the exact benchmark splits or collection partitions, unique task/variation counts, task-type coverage, and selection rules for each analysis.

After no-op removal, which ScienceWorld conclusions remain supported? Please report the remaining task/group support and clarify which findings are demonstrated in both environments, versus only in ALFWorld. How should the remaining ICC of 0.026 be interpreted given the large change from 0.154?

Collection composition. The submitted ALFWorld snapshots contain 800 trajectories at 1.5B and 1,320 at 7B. The supplementary records used for the added analyses have the task and episode counts below. These counts describe the supplementary collections; the original snapshot comparisons retain their reported trajectory totals.

Supplementary collectionTask instances/variationsEpisodesRealized episodes per taskCoverage
ALFWorld 1.5B100800100×8Six task types
ALFWorld 7B block A120960120×8Six types
ALFWorld 7B block B120360120×3Six types
ALFWorld 7B union2391,320119×8, 119×3, one×11Six types
ScienceWorld8064080×827 types

The supplementary 7B records contain 960+360 episodes with one overlapping task. The inspected collectors select a shuffled training-pool prefix for ALFWorld and use shuffled task/variation selection with per-type caps for ScienceWorld. Task-composition sensitivity is evaluated on the exact common-task subset below.

Task-composition sensitivity (E2). Use the exact task-ID intersection, equal numbers of complete trajectories per task capped at eight, deterministic ID order and no outcome selection:

E2 common-task subsetTasksEpisodesSuccessPreferred anchorsReturn-varying anchorsICC
1.5B161034.85%272270.217
7B161032.91%260360.304

The intersection covers five of the six task types, excluding heating. Paired whole-task resampling gives ICC difference +0.086, descriptive 95% interval [−0.152,+0.378], over 936 defined draws; 64/1,000 draws have a degenerate return channel. Success difference is −1.94pp [−7.42,+3.23] over all draws (seed 25572). This provides a descriptive comparison with task identity and trajectory count matched on the common subset. Policy-dependent state visitation and remaining horizons are retained.

ScienceWorld robustness. Appendix I/Table 6 reports ICC 0.154→0.026 after no-op removal. We use the filtered estimate to characterize history-associated variation in the dense-return setting. Supplementary retained-row counts are:

ScienceWorld rowsTasksEpisodesRowsPreferred anchorsReturn-varying anchorsInformative fractionNonzero-return rows
All8064014,46757641872.57%30.97%
No-op removed806407,86349634168.75%34.75%

All 80 tasks and 640 episodes remain represented after removing 6,604 rows containing the case-insensitive observation text “no known action matches”. Stored history labels and RTG are retained: filtering neither recomputes trajectories nor erases historical invalid actions. The nonzero-return percentage is calculated using the retained-row denominator.

The filtered ICC of 0.026 estimates the pooled between-history variance share under the working model. Its change after filtering identifies the contribution of stereotyped no-op observations. Appendix L also reports the retained operator-alignment and bias-cancellation diagnostics.

Retained conclusionALFWorldScienceWorld after filtering
Partition/value-construction distinctionsDemonstrated on matched rowsOperator/alignment checks after no-op filtering
Variance-bearing support can be countedDemonstrated341/496 preferred anchors remain informative
Zero-on-failure return constraintApplies under the stated terminal-only assumptionDense-reward setting

The submitted sparse-reward policy/stability comparisons concern ALFWorld. The added E5 experiment separately evaluates the complete GiGPO/HGPO operators on ALFWorld. Differences and intervals are computed before rounding.

AuthorsOpenReview ↗

Official Response to Reviewer cgiv (Part 3/7)

About the Independent Checking and Model-family Scope

Could a minimal anonymous rollout sample and analysis implementation be provided to check the main operator comparison and variance calculation? Please also clarify the scope of the one-family evidence: which conclusions are implementation identities, and which depend on the sampled Qwen policies?

E3 contains 2,100 anonymous rows from eight complete task groups and a standalone standard-library analyzer: 1,184 ALFWorld and 916 ScienceWorld rows. Independent integer-node and tuple-key representations give identical flat partitions and controlled step signals. Faithful HGPO agrees with the project reference within 8.89×10⁻¹⁶, and pooled ICC within 4.86×10⁻¹⁷.

The sample selects two supported and two unsupported tasks per environment in hash order. Its ICCs of 0 and 0.038 are demonstration calculations. The analyzer independently checks the controlled flat step term, faithful HGPO, and ICC on the supplied records.

The grouping/control identity and zero-return constraint follow from definitions and reward assumptions. Numerical cosines, fragmentation, support fractions, ICC, and prediction errors depend on the sampled policies and tasks. The present model-family evidence is from Qwen (§6).

The submission also includes a separately emitted matched-row stream and a formula-audited Yang progress-fallback implementation (§4.6; Appendix B, Table 4). The latter inherits the same grouping while activating roughly one third of eligible anchors with a silent terminal-return channel. These checks extend producer and operator coverage.

Table 2 reports exact observation replay for all successful primary trajectories (26/26 and 118/118) and stratified failure samples (240/240 and 260/260), with Wilson 95% contamination upper bounds of 1.60% and 1.50% in the failure stratum.

Parts 4–5/7 supply a smaller self-contained example: concatenate the data and implementation blocks and run the file. It retains all trajectories of one short task per environment, selected within the existing support-stratified sample. No external download is required. The larger 2,100-row artifact is described here; the embedded records provide the directly inspectable example.

Execution output.

{"HGPO_cosine": 0.9985445915293143, "SSB": 1.9653398076171875, "SSW": 1.0170945312499995, "df_B": 4, "df_W": 6, "environment": "ALFWorld_1.5B", "episodes": 8, "max_control_error": 0.0, "n0": 1.8125, "partition_mismatches": 0, "preferred": 2, "rho": 0.5115821541213132, "rows": 29, "varied": 2}
{"HGPO_cosine": 0.9328772488393698, "SSB": 1.577411693163115, "SSW": 3.130295599612907, "df_B": 50, "df_W": 130, "environment": "ScienceWorld_7B", "episodes": 8, "max_control_error": 0.0, "n0": 2.8263182281530908, "partition_mismatches": 0, "preferred": 6, "rho": 0.09889553644929712, "rows": 217, "varied": 3}

Both flat controls have zero discrepancy and zero partition mismatches. Against the project references, maximum hierarchical-step errors are 0 for ALFWorld and 8.890e-16 for ScienceWorld; ICC errors are zero. These selected examples verify the calculation; the main estimates use the full datasets. In particular, the short ALFWorld example contains eight successful episodes, so its ICC of 0.512 should be read as an example-specific result.

Reviewer_ovnAOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes and compares different step-level credit assignment strategies in existing RL methods for LLM agents. The analysis mainly focuses on advantage estimation, the role of historical trajectories, and the challenges introduced by sparse rewards.

Critical strengths
  1. Important problem. Step-level credit assignment is an important problem in multi-step RL, especially for long-horizon agent tasks.

  2. Thorough analysis. From an analytical and comparative perspective, the paper provides reasonably comprehensive mathematical derivations and numerical analyses of existing methods.

Critical weaknesses
  1. Limited practical guidance. The analysis does not seem to provide clear insights or actionable recommendations for designing better credit assignment methods. In my view, an important goal of theoretical and analytical research is to guide future method development and practical improvements.

  2. Lack of training validation. The paper focuses entirely on analyzing existing estimators and their computed advantages, without conducting actual RL training experiments to validate whether the observed differences translate into meaningful performance improvements.

Questions for authors

Based on the findings of this paper, what concrete insights or recommendations can be drawn to guide the development of better step-level credit assignment methods? In particular, how should practitioners use these findings to improve future RL training?

6 comments
AuthorsOpenReview ↗

Official Response to Reviewer ovnA (Part 1/5)

Thank you for the careful reading and constructive questions. We address your questions below.

About the Practical Guidance

Based on the findings of this paper, what concrete insights or recommendations can be drawn to guide the development of better step-level credit assignment methods? In particular, how should practitioners use these findings to improve future RL training?

Our findings support three decisions.

  1. Check variance-bearing support before refining groups. When failed episodes receive zero reward throughout, every all-failure anchor has constant return, as do its history refinements (§4.2). This directs practitioners to exploration, outcome coverage, and the choice of value channel before increasing partition depth.
  2. Measure effective refinement rather than nominal depth. Faithful HGPO retains the flat current-observation level, skips singleton refinements, and renormalizes surviving contributions (§3.3; Appendix G). Surviving refinement coverage and weights show whether an increase in maximum history depth actually changes the estimator.
  3. Match the ablation to the mechanism claimed. To evaluate grouping, hold the value target, aggregation/normalization, collection, and update settings fixed. To evaluate value construction, hold grouping fixed. The grouping-only/full-G2PO contrast supplies the control that attributes signal differences to the value construction.

The support result concerns the step-relative return channel. If every row in an anchor has Ri=cR_i=c, then zG(Ri)=(c−c)/(sG(R)+ϵ)=0z_G(R_i)=(c-c)/(s_G(R)+\epsilon)=0, and every history refinement has the same zero centered signal. Other components, such as an episode-level term, may still contribute to the policy update.

The added experiments test the practical implications. E4/E6 evaluate whether a supported history channel improves held-out RTG prediction and whether shrinkage improves on raw history. E5 supplies the matched learning comparison described in Part 3/5. These tests let us assess predictive usefulness and policy success separately, rather than infer either from vector disagreement.

AuthorsOpenReview ↗

Official Response to Reviewer ovnA (Part 2/5)

About the Held-out Return-Prediction Check

The paper focuses entirely on analyzing existing estimators and their computed advantages, without conducting actual RL training experiments to validate whether the observed differences translate into meaningful performance improvements.

The submission analyzes fixed rollout data (§6). We supplement it with a held-out prediction check (E4) and a separate native-operator training comparison (E5, Part 3/5).

Experiment settings.

Each fold withholds a complete trajectory. Flat prediction uses the mean stored discounted return-to-go (RTG) for the same task/observation from other trajectories, with at least two training rows. History additionally uses hist_hash_k1 and requires two training rows from at least two independent trajectories; otherwise it uses exactly the flat prediction. Both methods score the same eligible rows at gamma=0.95, without tuning. We report row-weighted MSE with 1,000 paired whole-task bootstrap draws, seed 25573.

Results.

CollectionTasksTest rowsFlat MSEHistory MSEHistory-flat MSE / 95% task interval
ALFWorld 1.5B100335080.4130.302-0.111 [-0.24, -0.02]
Supplementary 7B239517681.3421.101-0.242 [-0.338, -0.162]
ScienceWorld/no-op removed8057970.100.078-0.022 [-0.029, -0.015]

Eligible-row coverage is 85.45%, 82.92%, and 73.73%, respectively; history-supported shares are 60.30%, 49.85%, and 44.73%. MSE is reported separately for each environment in squared return units.

History improves held-out prediction of the stored return target in all three collections. E6 uses the same raw-history predictions and evaluates support-dependent shrinkage. This evaluates the auxiliary RTG predictor; E5 reports the native-operator policy-learning result.

Differences, intervals and derived metrics are computed before rounding.

AuthorsOpenReview ↗

Official Response to Reviewer ovnA (Part 3/5)

About the Matched Training Results (E5)

The paper focuses entirely on analyzing existing estimators and their computed advantages, without conducting actual RL training experiments to validate whether the observed differences translate into meaningful performance improvements.

Experiment settings.

The separate 25572 training extension uses Qwen2.5-1.5B-Instruct revision 989aa7980e4cf806f80c7fef2b1adb7bc71aa306 and a common native HGPO trainer for both native GiGPO and native HGPO. Advantages are constructed before padding/balancing. GiGPO uses episode+flat terms with sample standard deviation; HGPO uses the depth-1/2/3 step hierarchy, population standard deviation, history_length=2, alpha=1, base_group=False. The comparison evaluates the complete native operators, including their hierarchy, normalization, and episode terms.

The train pool has 3,553 eligible ALFWorld tasks; the held-out set is the first 128 of 140 sorted valid_seen candidates, frozen by registry. Per update: 16 task dispatches, 8 rollouts each, at most 50 actions, context history 2, prompt/response limits 2048/512, temperature 0.4. Raw success reward is 10; gamma=0.95 RTG is followed by the current invalid penalty 0.1 in the native step/token channels. KL is loss-only, coefficient 0.01. AdamW uses lr=1e-6, betas=(0.9,0.999), weight_decay=0.01, constant schedule/no warmup; PPO minibatch 256 step rows, microbatch 1, one PPO epoch, clip 0.2, grad clip 1, token-mean reduction, TP=2. Both arms disable extra planner/progress queries.

There are eight new paired seeds 27201-27208, environment seed 27200, a fixed 20-update budget and evaluations at 0/5/10/15/20 on 128 tasks. Primary effect is HGPO-minus-GiGPO final success in percentage points. Inference uses eight seed pairs, a 95% paired-t interval and the prespecified two-sided exact sign-flip reference over 256 assignments. The evaluation seed is (run_seed*1000003+checkpoint*503+turn*17) mod2147483647. Qualification seed 27200 is excluded.

Results.

SeedGiGPO successesHGPO successesHGPO-GiGPO pp
2720121/12850/128+22.656
2720211/12838/128+21.094
2720317/12815/128-1.562
2720416/12822/128+4.688
272059/12854/128+35.156
2720613/12837/128+18.75
2720726/12842/128+12.50
2720820/12829/128+7.031
Primary summaryGiGPO mean %HGPO mean %Paired gain ppSample SD95% paired-t CITwo-sided exact p
Update2012.98828.02715.03911.75[5.216, 24.863]0.016

HGPO reaches mean success of 28.027%, compared with 12.988% for GiGPO, a paired gain of 15.039 percentage points. All eight seed pairs are reported.

E5 connects the complete-operator comparison directly to final task success after 20 updates on the ALFWorld 1.5B held-out set. The measured gain reflects the joint contribution of hierarchy, normalization, and episode terms in the tested configurations.

Seed 27200 is excluded from inference. The admission protocol includes a two-step check of updates, checkpoint saving, and model/optimizer/extra-state reload.

Differences, intervals and derived metrics are computed before rounding.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.