acceptodds
Under review as a conference paper at ICLR 2027

DSA: Progress Inside the Group Standardizer in Sparse-Reward Agent Learning

Abstract

Sparse terminal rewards can silence a group-relative learner even when the states inside a comparison group differ in their proximity to the goal: equal returns are mapped to equal, zero-centered coefficients. We introduce Dense Signal Augmentation (DSA), an advantage operator that combines action-before state progress with return before applying the inherited group standardizer. This placement preserves the original grouping and optimizer, recovers the baseline exactly when augmentation is disabled, and exposes progress variation without defining a second normalization scale. A conditional return identity explains the temporal alignment, while a score-function analysis identifies why current-batch scheduling, sampled grouping, and nonstationary proxies prevent an invariance or unbiased-credit guarantee. Predeclared matched training experiments exhibit earlier held-out attainment on average, although the exact paired evidence remains statistically inconclusive. Matched operator controls further test signal placement and temporal alignment, while history-aware and cross-setting experiments delimit deployment and breadth. The principal contribution is therefore a controlled and falsifiable interface between state progress and group-relative credit assignment, supported by training interventions rather than snapshot diagnostics.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, sparse rewards, group-relative policy optimization, credit assignment, LLM agents, reward shaping, advantage estimation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_jPVyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

DSA (Dense Signal Augmentation) modifies group-relative RL by adding a progress signal to returns before group normalization, with a success-dependent weighting schedule. The paper provides theoretical analyses of the operator and evaluates it through four paired training runs, operator controls, and comparisons with ProGPO variants.

Critical strengths

The paper explicitly acknowledges the inconclusive primary test and distinguishes evidence for task improvement, mechanism, distinctiveness, and deployability. The predeclared evaluation protocol is a positive aspect of the work, although it does not compensate for the limited empirical findings.

Critical weaknesses
  1. The primary paired test is inconclusive (p = 0.125, four run pairs), and DSA does not demonstrate superiority over matched-budget ProGPO variants. The consistently positive results rely on a non-deployable oracle planner, while the learned proxy remains exploratory. Thus, the practical benefit of the proposed operator is not convincingly established.

  2. The core idea is relatively simple, but extensive custom terminology and overly detailed explanations make the paper difficult to follow. Redundant elements, such as the repeated "measured" status in Table 2, add complexity without improving clarity. The presentation obscures the main contribution and makes the empirical evidence harder to assess.

Questions for authors
  1. With only four run pairs, the exact one-sided sign-flip test has a minimum attainable p-value of 0.0625. How was this evaluation designed to provide sufficient statistical power?

  2. Given the comparable gains of ProGPO variants, what evidence supports a distinct advantage of DSA? Could you provide uncertainty estimates for these comparisons?

  3. What are the quantitative fidelity and task-performance results for the deployable history-aware proxy? Do they support the practical effectiveness of DSA?

5 comments
AuthorsOpenReview ↗

Official Response to Reviewer jPVy (Part 1/3)

About Statistical Power and Comparative Utility

With only four run pairs, the exact one-sided sign-flip test has a minimum attainable p-value of 0.0625. How was this evaluation designed to provide sufficient statistical power?

Given the comparable gains of ProGPO variants, what evidence supports a distinct advantage of DSA? Could you provide uncertainty estimates for these comparisons?

DSA adds action-before progress to the inherited return, then standardizes once within the inherited group. The disabled identity and fixed comparison rule make the intervention explicit, while the analysis characterizes return–progress coupling and temporal alignment.

The original four-pair study has one-sided exact sign-flip resolution 0.0625. The separate E1 extension uses 32 independent pairs and retains the 20/128 attainment endpoint: the mean gain over GiGPO is 9.844 updates [5.537, 14.151], with exact p=4.375e-05. Final success also improves by 4.932 percentage points [3.723, 6.140] as a secondary endpoint.

For the closest-method comparisons, the submitted Appendix C.2 reports gains of 3.750 ± 12.500 updates against Yang (mean ± sample SD; exact p=0.375) and 6.250 ± 12.500 against the paper-derived Fan–Liu implementation (p=0.250). E2 additionally compares controlled operators: mean early gains are 7.500/12.500/11.250 over dead-only/RMS-matched separate/return-stratified controls. The E2 dead-only interval is [-2.771, 17.771]. These comparisons report the observed effect and paired uncertainty under each protocol.

The contribution is the combine-before-standardization operator, its exact disabled-baseline identity, and the controlled comparisons of scale, activation and signal placement.

AuthorsOpenReview ↗

Official Response to Reviewer jPVy (Part 2/3)

About History-Aware Signal Fidelity and Applicability

What are the quantitative fidelity and task-performance results for the deployable history-aware proxy? Do they support the practical effectiveness of DSA?

E4 history-contrast NRMSE is 0.707 [0.620, 0.787], centered R² is 0.500 and rank concordance is 0.691 [0.657, 0.728]. Contrast minus global NRMSE is -0.172 [-0.235, -0.103]. E5 calibration lowers contrast NRMSE to 0.665 while preserving rank. E6 task-balanced fitting further lowers calibrated NRMSE to 0.517; its rank difference from unweighted fitting is 0.051 [-0.012, 0.118].

D1 reports contrast cosine 0.750 [0.695, 0.807] and relative standardized error 0.500 [0.386, 0.610]. Its relation to reconstruction error is explained by the equal-energy condition in the D1 response.

These scores improve on the constant predictor's NRMSE=1 and tie rank=0.5. Reconstruction error, ordering and standardized direction describe complementary aspects of progress-signal quality.

For task performance, the original Table 7 learned-history proxy reaches the threshold at a median 67.5 updates versus 75.0 for GiGPO, with final success 56.64% versus 52.73%. Table 7 reports task-training outcomes for the original predictor; E4–E6 report reconstruction, calibration and objective-alignment results for the specified history-based predictors. Fitting uses planner labels, while frozen inference requires only causal history.

AuthorsOpenReview ↗

Official Response to Reviewer jPVy (Part 3/3)

About Presentation and Terminology

The core idea is relatively simple, but extensive custom terminology and overly detailed explanations make the paper difficult to follow.

The central definition is:

“Within each inherited comparison group, add progress measured before the action to return, then standardize once.”

The simplicity provides an attribution advantage: progress placement can change while the inherited grouping rule and optimizer stay fixed, and disabling augmentation supplies an exact elementwise control on the same collected rows. This makes the interface directly testable without introducing a new comparison rule.

The replies use this definition and the direct scientific questions: whether training improves, whether this operator helps beyond the stated controls, and whether a usable progress signal exists. Algebra identifies when coefficients change; held-out training evaluates utility; predictor specification defines signal availability.

We will simplify the presentation around this definition, remove repeated status labels from tables, and shorten the terminology and explanations around the main comparisons.

Reviewer_AzhNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Dense Signal Augmentation (DSA), which replaces the scalar standardized within a GiGPO step group by R+λg(s)R+\lambda g(s), where g(s)g(s) measures progress in the state before the action. Group membership and the optimizer remain fixed, and λ=0\lambda=0 exactly recovers the baseline. The paper derives a retrospective potential-shaping identity while explicitly explaining why it does not establish policy invariance or unbiased action credit after normalization and current-batch scheduling.

The primary experiment comprises four matched ALFWorld/Qwen2.5-1.5B run pairs. DSA reaches the predeclared 20/128 held-out success threshold 17.5 updates earlier on average, with substantial between-pair dispersion and an inconclusive exact paired test. Additional experiments compare normalization and temporal alignment, progress proxies, nearby methods, model scale, and environments. The authors appropriately present these results as exploratory evidence for a narrowly defined operator.

Critical strengths
  • The intervention is unusually explicit. Sections 3.1 and 5.1 specify the group key, action-before indexing, single normalization, and exact disabled identity. Equation 3 also correctly retains the effect of finite ϵ\epsilon, rather than claiming that the magnitude of positive λ\lambda cancels exactly.

  • The theoretical discussion avoids an important misconception. Equation 5 explains why a pre-action state signal does not automatically provide action-quality information. Sections 3.1–3.2 then distinguish the telescoping identity from the additional effects of sampled groups, outcome-dependent scheduling, and PPO reuse. These qualifications substantially improve the interpretation of the method.

  • The evaluation includes relevant interventions and reports unfavorable uncertainty. The shuffle, next-state, separate-normalization, and dead-group controls address meaningful alternatives. Tables 4 and 10 use paired inference, while Appendix C.6 explicitly prevents the smaller unpaired sensitivity pp-value from becoming the headline conclusion. Separate reporting of the dense-reward boundary and sparse-task replication is also appropriate.

Critical weaknesses
  1. The demonstrated advance remains limited by uncertainty about utility. The primary mean gain is 17.5 updates with a paired standard deviation of 19.4 updates over four pairs, and the later-attainment and final-performance endpoints are similarly uncertain (Tables 4 and 10). With four pairs, even the smallest attainable one-sided sign-flip pp-value is 1/161/16; the planned sample cannot satisfy a conventional 0.05 decision threshold. This does not demonstrate absence of benefit or invalidate the descriptive results. It does mean that, because the theory deliberately supplies no improvement guarantee, the paper currently leaves its main practical question unresolved. Report the individual paired endpoints and uncertainty on the effect, and prioritize a prospectively specified extension with enough pairs to distinguish a practically relevant benefit from run variation. The stochastic evaluator and first-crossing endpoint also warrant sensitivity reporting that preserves the original primary endpoint.

  2. The evidence does not yet establish the value of the distinctive joint-normalization component. Section 3.1 identifies live-group return–progress coupling as the difference beyond the restricted constant-return overlap with the Yang comparator. Yet dead-group-only activation has a median matched difference of zero updates relative to full DSA (Table 2), and comparisons with both nearby methods remain uncertain (Appendix C.2). Furthermore, the separate-normalization control changes coefficient magnitude as well as coupling: in a constant-return group and with negligible ϵ\epsilon, joint DSA is approximately zG(g)z_G(g), whereas the separate control is λzG(g)\lambda z_G(g). Consequently, its slower training cannot isolate the benefit of shared normalization from augmentation strength. The existing comparisons are useful operator comparisons, but a scale-matched control or focused evidence about the live-group contribution would better support the proposed distinction. No superiority claim is required; the missing evidence concerns why this particular modification is useful.

  3. The learned-proxy and robustness results are not reproducible from the submission. Sections 3.3 and A.3 establish why within-key information is necessary, but Appendix C.5 does not specify the history encoder, prediction target, supervision source and quantity, training objective, optimization, or whether the predictor is frozen during policy training. It also refers to a fixed corruption grid without specifying the corruption process or its levels. Figure 3(b) and Table 7 cannot replace these details. This matters because the primary oracle signal requires simulator planning information, making the learned signal the only presented route toward practical applicability. Supply the actual proxy specification, data-role separation, corruption settings, and paired task outcomes with uncertainty. The statement that artifacts will be released after publication does not currently permit these checks.

  4. The actual reward construction needs reconciliation with the terminal-return argument. Section 4 correctly establishes its all-failure result under terminal-only reward, while Table 5 lists an invalid-action penalty of 0.1. The cited GiGPO implementation description, Appendix E.1, assigns ALFWorld a success reward of 10 and failure reward of 0, and applies a reward penalty of −0.1 to invalid actions. If those penalties enter DSA's inherited RR, failed trajectories can have different returns. This would not invalidate Equation 3 for genuinely constant-return groups, but it changes how directly the stated dead-zone mechanism applies to the experiments. Specify the success-reward magnitude, penalty placement, and complete return construction, and report how frequently the actual training groups satisfy the constant-return condition. Reward scale also matters for interpreting the fixed λ0=0.5\lambda_0=0.5.

Questions for authors
  1. What are the four individual paired primary endpoints, and how are evaluation sampling seeds coupled across arms and checkpoints? Do the observed gains remain directionally similar under a predeclared neighboring-threshold or sustained-attainment sensitivity analysis?

  2. Does joint DSA improve over dead-group-only activation on any reported endpoint with meaningful precision? Can the authors provide an augmentation-scale-matched normalization control, or otherwise separate coefficient magnitude from the proposed live-group coupling?

  3. What precisely is the history-aware predictor, including its supervision and update schedule? Please also provide the corruption grid and clarify whether the invalid-action penalty contributes to the returns standardized by DSA.

7 comments
AuthorsOpenReview ↗

Official Response to Reviewer AzhN (Part 1/6)

About Statistical Precision and Evaluation Coupling (E1)

The original four-pair study has one-sided exact sign-flip resolution 1/161/16. The E1 extension adds a separate 32-pair cohort while retaining the original attainment endpoint. Using the original paired SD of 19.4 updates and a reference gain of 10, the paired-t planning calculation gives approximately 0.886 power at n=32. The reported E1 inference uses exact sign-flip tests and paired-t intervals.

Data and model. ALFWorld, 3,553 training tasks; evaluate the first 128 of the sorted 140-task valid_seen registry. Qwen2.5-1.5B-Instruct revision 989aa7980e4cf806f80c7fef2b1adb7bc71aa306. This registry and configuration define the supplemental cohort.

Training. 16 prompts ×8 rollouts, at most 50 actions, prompt/response limits 2048/512, gamma .95, sample standard deviation, epsilon 10−610^{-6}, singleton coefficient zero. DSA uses pre-action progress and λu=.5max⁡(0,1−p^u/.5)\lambda_u=.5\max(0,1-\hat p_u/.5) from the completed current batch. GiGPO disables augmentation. LR 10−610^{-6}, PPO minibatch 256, TP 2, KL loss .01 (low_var_kl), no KL reward. Four RTX 3090 GPUs, microbatch 1, activation offload and vLLM memory utilization 0.30 define the execution profile.

Pairing and evaluator. Seeds 19005–19036 define 32 GiGPO/DSA pairs with the same initial checkpoint. Both arms train for 150 updates; we evaluate 128 binary held-out outcomes every 5 updates at temperature .4. For run seed s, checkpoint u and turn t, use (1000003s+503u+17t) mod 2147483647(1000003s+503u+17t)\bmod2147483647 for sampling, shared across arms within a checkpoint. Checkpoints use different streams. Both arms share the initial evaluation within each pair. The independent unit is a paired training run; checkpoint repeats are reproducibility checks. Checkpoints are saved every 25 updates, with a final reload evaluation.

Endpoints and inference. Primary: first 20/128 updates↓. Secondary: first 39/128 and final success%. Sensitivities: first 16/128, first 24/128 and first of two consecutive evaluations attaining 20/128. An endpoint of 151 is assigned only to runs that complete without reaching the threshold; infrastructure-incomplete runs remain missing. We report all 32 pairs, mean/sample SD, approximate paired-t intervals and exact one-sided mean-gain sign-flip tests over 2322^{32} assignments. No optional stopping, seed selection, original-cohort pooling or held-out tuning. The protocol was specified for this rebuttal-stage extension.

“Exact” refers to enumeration of the sign-flip reference distribution. Inferential validity requires independent pairs and sign exchangeability under the null; paired-t intervals additionally use an approximately normal paired-difference model. Run-level inference is conditional on the fixed 128-task registry and describes variation across paired training/evaluation seeds.

Mean gains at first 16/128, first 24/128 and sustained 20/128 are 10.625, 10.625 and 8.906 updates. The sustained interval is [3.938, 13.875]. The consistent direction extends the attainment result across neighboring thresholds and the two-checkpoint criterion.

AuthorsOpenReview ↗

Official Response to Reviewer AzhN (Part 2/6)

About Individual Paired Endpoints: Seeds 19005–19020 (E1)

SeedGiGPO 20 ↓DSA 20 ↓Gain ↑GiGPO 39 ↓DSA 39 ↓GiGPO final /128 ↑DSA final /128 ↑
19005856520120856367
1900690100-101301307478
1900710090101451356874
19008909001251157081
19009706551151057480
1901010510501401406970
1901111095151451258388
1901290100-101201306775
1901310590151351207382
19014806515115955662
19015857510120957888
190168090-101101307480
190179580151301106666
190189070201151107275
1901911511501451457084
190208070101101007989

Attainment is measured in updates. A final count k/128k/128 corresponds to 100k/128100k/128 percent; one success is exactly 100/128 percentage points (approximately 0.781). Arm medians and paired mean gains are different statistics.

AuthorsOpenReview ↗

Official Response to Reviewer AzhN (Part 3/6)

About Individual Paired Endpoints: Seeds 19021–19036 (E1)

SeedGiGPO 20 ↓DSA 20 ↓Gain ↑GiGPO 39 ↓DSA 39 ↓GiGPO final /128 ↑DSA final /128 ↑
19021755025115857885
1902210565401451057277
190237590-151101256769
19024805030115857580
19025959051251356669
190269070201201006170
19027856520120955775
19028909001101357481
1902970655110856671
190309585101151255957
190319080101251257692
190329580151351155962
19033908551201206975
190349075151201207479
1903510090101401256771
1903610085151451105258

Attainment is measured in updates. A final count k/128k/128 corresponds to 100k/128100k/128 percent; one success is exactly 100/128 percentage points (approximately 0.781). Arm medians and paired mean gains are different statistics.

Across all 32 pairs, primary arm medians are 90.000/82.500 updates and final-success medians are 54.297%/58.594% for GiGPO/DSA. The 4.297-point difference between arm medians is distinct from the paired mean gain of 4.932 pp.

Reviewer_1AomOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

DSA combines pre-action state progress with returns before group standardization to provide learning signals when sparse returns lack variation. The paper presents analytical properties, implementation checks, and matched training experiments.

Critical strengths

The operator is simple and exactly recovers the baseline when augmentation is disabled.

Matched controls and explicit theoretical limitations make the intervention transparent and inspectable.

Critical weaknesses

The incremental contribution over existing progress-based methods remains limited. Normalization controls also change coefficient scale, leaving the benefit of signal placement insufficiently isolated.

Only four matched training pairs support the primary comparison, with an inconclusive paired test (p=0.125p=0.125). The results suggest improvement but do not establish a reliable task effect.

DSA’s novelty appears concentrated in moving progress inside the group standardizer; its constant-return behavior can coincide with a prior progress-based rule (§3.1). Near-parity with dead-group-only activation and controls that also change coefficient scale leave the added value of joint normalization unclear (Table 2).

Questions for authors

What measurable benefit does joint normalization provide beyond dead-group-only activation?

Do the gains persist when operator controls are matched for effective advantage scale?

Are the primary gains reproducible with more independent paired runs?

What benefit does joint normalization provide over the closest progress-based alternatives after controlling for advantage scale and activation conditions?

7 comments
AuthorsOpenReview ↗

Official Response to Reviewer 1Aom (Part 1/5)

About Advantage Scale and Signal Placement

Do the gains persist when operator controls are matched for effective advantage scale?

For G=(uid,normalized anchor observation)G=(\mathrm{uid},\text{normalized anchor observation}), define zG(x)=(x−xˉG)/(σx,G+ϵ)z_G(x)=(x-\bar x_G)/(\sigma_{x,G}+\epsilon). DSA computes J=zG(R+λgbefore)J=z_G(R+\lambda g_{\rm before}) once, using one lambda from the completed current training batch. At lambda zero it is exactly the inherited baseline on the same rows.

If R is constant, Ji=λ(gi−gˉG)/(∣λ∣σg,G+ϵ)J_i=\lambda(g_i-\bar g_G)/(|\lambda|\sigma_{g,G}+\epsilon). A varying progress signal can create contrast; a deterministic observation-only predictor that is constant within G cannot. Finite epsilon prevents exact cancellation of positive lambda's magnitude.

For the same collected group with lambda fixed, an additive progress offset cancels algebraically: zG(R+λ(g+cG))=zG(R+λg)z_G(R+\lambda(g+c_G))=z_G(R+\lambda g) for cGc_G constant within GG. The operator uses within-key progress contrasts and is invariant to a groupwise progress offset. This is the constant-signal property already checked in Table 1.

In live groups, the numerator combines centered return and progress and the denominator includes Var⁡(R)+2λCov⁡(R,g)+λ2Var⁡(g)\operatorname{Var}(R)+2\lambda\operatorname{Cov}(R,g)+\lambda^2\operatorname{Var}(g) (Eq. 4). Separate normalization changes relative channel weights as well as scale. E2 tests this operator difference with complete-coefficient RMS matching.

An exact denominator-only comparison has a useful limit. Let J=vG/dJJ=v_G/d_J and C=vG/dCC=v_G/d_C with the same numerator and positive scalar denominators. Whenever their masked norms are nonzero,

C RMSG(J)RMSG(C)=J.C\,\frac{\mathrm{RMS}_G(J)}{\mathrm{RMS}_G(C)}=J.

Both-zero cases remain zero. Therefore a denominator-only vector difference disappears after exact step-RMS matching. The E2 separate-normalization control tests the direction and episode-balance differences after matching the complete coefficient, including the episode term. The matched quantity is coefficient RMS; gradients and PPO clipping remain properties of the subsequent optimization step.

AuthorsOpenReview ↗

Official Response to Reviewer 1Aom (Part 2/5)

About Scale and Activation Controls (E2)

E2 uses the E1 model, data, 150-update budget, evaluator, optimizer and fixed hyperparameters. Four paired blocks use seeds 19005–19008, sharing their E1 joint arms under the same configuration. The controls isolate scale and activation without held-out tuning.

For inherited group GG, write J=zG(R+λg)J=z_G(R+\lambda g) and S=zG(R)+λzG(g)S=z_G(R)+\lambda z_G(g). Add the inherited episode-level coefficient to obtain each complete coefficient AA. The complete separate coefficient is rescaled by the detached factor

cG=∥AGJ∥valid response tokens∥AGS∥valid response tokens.c_G=\frac{\|A^{J}_G\|_{\mathrm{valid\ response\ tokens}}}{\|A^{S}_G\|_{\mathrm{valid\ response\ tokens}}}.

The norm includes response lengths and masks. Both-zero norms give zero; a zero control with nonzero joint target is treated as undefined, without dropping the case or substituting a fallback coefficient. The rescaling is applied once and matches complete-coefficient RMS within each inherited group.

Fixed arms. (1) Joint DSA. (2) Dead-only: use JJ only when the actual inherited post-penalty RR is exactly constant in GG, otherwise use baseline. (3) Complete-advantage RMS-matched separate. (4) Return-stratified progress: T=zG(R)+λzH(g)T=z_G(R)+\lambda z_H(g) for H=(G,exact R)H=(G,\text{exact }R), with singleton/constant strata zero, then the same complete-coefficient RMS match to joint. Exact tensor equality, rather than rounded log values, defines constant returns.

Endpoints and reporting. The endpoints are first 20/128 attainment, updates ↓, and final success ↑. We report paired control-minus-joint early-attainment gains, joint-minus-control final points, sample SD, approximate 95% paired-t intervals and exact one-sided sign-flip p values.

E2 reports four-pair descriptive operator comparisons with paired intervals and exact tests; the one-sided sign-flip test has resolution 1/16=.06251/16=.0625.

AuthorsOpenReview ↗

Official Response to Reviewer 1Aom (Part 3/5)

About Matched Operator Outcomes (E2)

Joint outcomes reuse the same four E1 joint arms in paired seed order. Positive early gain is control minus joint updates; positive final gain is joint minus control percentage points.

MethodMedian first 20/128 ↓Mean control−joint ↑Sample SDApprox. 95% paired CIExact pMedian final % ↑Pairs
Joint DSA90.000referenceN/AN/AN/A59.3754
Dead-only97.5007.5006.455[-2.771, 17.771]0.12557.4224
RMS-matched separate107.50012.5006.455[2.229, 22.771]1/16 (≈0.063)56.2504
Return-stratified RMS100.00011.2502.500[7.272, 15.228]1/16 (≈0.063)56.2504

Joint attainment is [65,100,90,90], with final counts [67,78,74,81]. Dead-only attainment is [65,115,95,100], final [64,74,73,75]; RMS-matched separate is [70,110,105,110], final [63,70,74,76]; return-stratified RMS is [80,110,100,100], final [68,74,70,82]. All counts have denominator 128.

ComparatorMean gain, ppSample SD, ppApprox. 95% paired-t CI, ppExact p
Dead-only2.7341.626[0.147, 5.322]1/16 (≈0.063)
RMS-matched separate3.3202.581[-0.787, 7.428]0.125
Return-stratified RMS1.1722.255[-2.417, 4.761]0.250

Mean early gains are 7.500/12.500/11.250 updates over dead-only/RMS-matched separate/return-stratified RMS controls. Final-success gains in the same order are 2.734/3.320/1.172 percentage points. All point estimates favor joint DSA at both endpoints. The tables report the paired uncertainty for each comparison. Intervals are pointwise, and the six comparator/endpoint tests are descriptive and unadjusted for multiplicity.

The RMS match applies to complete coefficients, including the inherited episode term. It controls coefficient magnitude while allowing direction and episode balance to differ; gradient and PPO-clipping behavior are separate quantities.

Reviewer_6CMcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

In intra-group relative advantage learning, if samples within the same group yield identical returns, the normalized learning coefficients become zero, even if these samples differ in their proximity to the target. DSA addresses this by combining the progress (from the state prior to an action) with the return, and then performing an additional round of intra-group normalization. It retains the original grouping and optimizer structure of GiGPO, while the weight assigned to the "progress" component decreases as the success rate of the current batch increases.

Critical strengths
  • By keeping the grouping and optimizer fixed and modifying only the input to the normalization step, the experimental intervention remains clear and well-defined.
  • The authors carefully distinguish between "generating non-zero coefficients" and "providing valid credit for actions," avoiding the common pitfall of conflating theoretical identities (derived from potential functions) with the actual unbiasedness of the algorithm.
Critical weaknesses
  • The fact that a state is near the target does not necessarily imply that the subsequent action deserves reinforcement; therefore, resolving the "zero-coefficient" issue requires empirical evidence—specifically, reliable training gains—to demonstrate its effectiveness.
  • The "progress" metric used in the main results is derived from the negative remaining plan length provided by an actual planner—a type of information that may not be available for general tasks (e.g., in coding tasks, defining such state-based progress is often difficult).
Questions for authors
  1. When an independent training objective is added, does DSA consistently outperform GiGPO, the two variants of ProGPO, and a version that applies normalization only within groups sharing the same return?
  2. How are the "progress proxies" (based on historical information) encoded and trained, and how are the progress labels obtained? Does this process still require supervision from a planner? Please provide the full configuration, the scale of the data used, and the results for each individual experimental run.
8 comments
AuthorsOpenReview ↗

Official Response to Reviewer 6CMc (Part 1/7)

About Action Credit and the Training Objective

When an independent training objective is added, does DSA consistently outperform GiGPO, the two variants of ProGPO, and a version that applies normalization only within groups sharing the same return?

DSA changes a coefficient in the inherited PPO objective; it does not add an independent auxiliary loss. Its step term is J=zG(R+λgbefore)J=z_G(R+\lambda g_{\rm before}), replacing zG(R)z_G(R) while retaining the inherited episode term, grouping and optimizer.

Equation 5 characterizes the pre-action signal under its stated conditions, and the retrospective potential identity explains its temporal alignment. E1 evaluates the utility of the actual sampled, standardized PPO update, including current-batch scheduling and sampled group statistics.

E1 shows average benefit over GiGPO under the supplemental protocol: 9.844 updates earlier attainment [5.537, 14.151] over 32 pairs, with 24 improvements, four ties and four reversals. Final success improves by 4.932 percentage points [3.723, 6.140]. These results address both learning speed and fixed-budget performance under the inherited PPO objective.

For the closest progress-based methods, Appendix C.2 of the submission reports paired early-attainment gains of 3.750 ± 12.500 updates against the Yang comparator (mean ± sample SD; exact p=0.375) and 6.250 ± 12.500 against the paper-derived Fan–Liu implementation (p=0.250). These matched-budget comparisons use four paired runs. E2 separately tests scale- and activation-matched operators; the exact-return-stratum response below distinguishes the two possible interpretations of that control.

AuthorsOpenReview ↗

Official Response to Reviewer 6CMc (Part 2/7)

About the History-Aware Progress Predictor (E4)

How are the "progress proxies" (based on historical information) encoded and trained, and how are the progress labels obtained? Does this process still require supervision from a planner?

E4 is a new supervised proxy follow-up using a causal history representation. It tests prediction of the within-group progress contrasts used by DSA. The predictor is fitted separately and frozen during evaluation; policy training retains its original objective.

Data and supervision. We used the archived GiGPO iteration-0 ALFWorld trajectories: 39,215 rows, 800 trajectories and 100 tasks. We joined existing action-before planner labels by (trajectory,step), deduplicating identical replay records and rejecting conflicting labels. The available label coverage is 36,815 rows from 94 tasks; six tasks have no labels and are reported as uncovered. Progress is negative remaining planner-plan length. Planner supervision is required; frozen prediction alone is planner-free.

Split and encoder. We assigned each entire task to train/validation/test by the first 16 hexadecimal digits of SHA256(task_id), modulo 100: <60/<80/otherwise. The matched archive has 53/25/16 tasks and 20,978/9,483/6,354 labelled rows. We built histories on complete raw trajectories before masking missing labels. The inputs are normalized observations through the current state and actions strictly before the current action. No task ID, step index, return, current action or future observation is a predictor input. Observation-only sees exactly the current normalized observation. All methods are evaluated on the same oracle-varying groups, including prediction failures.

Fixed fitting. Word 1–2-gram hashing, 32,768 nonnegative hash features, lowercase, then train-only TF–IDF (smoothed IDF, sublinear TF, L2 norm). Ridge alpha=1, LSQR, tolerance 1e-6, maximum 1,000 iterations; no hyperparameter search (scikit-learn 1.5.1, SciPy 1.13.1). Global-target models use an intercept; contrast-target models do not. For each train group (task,normalized observation), choose its first labelled row ordered by (trajectory,step) as reference, and fit differences of history features to differences of planner progress. Inference is a frozen function of each causal history, without test labels. We added a within-train-group label-permutation control, using a fixed permutation stream. Validation is reported, never used for selection.

Exact E4 losses. For the stated features, the global fit minimizes

Lglobal=∑i∈train(gi−b−w⊤xi)2+∥w∥22.L_{\rm global}=\sum_{i\in\rm train}(g_i-b-w^\top x_i)^2+\|w\|_2^2.

The contrast fit minimizes

Lcontrast=∑G∑i≠rG[(gi−grG)−w⊤(xi−xrG)]2+∥w∥22.L_{\rm contrast}=\sum_G\sum_{i\ne r_G}[(g_i-g_{r_G})-w^\top(x_i-x_{r_G})]^2+\|w\|_2^2.

There is no 1/n loss scaling. Constant-label multirow groups remain in training; singleton groups supply no difference. Hasher, IDF and fitted weights are frozen afterward; there is no policy joint training.

Primary endpoint. In each oracle-varying group, divide squared error in centered predicted progress by centered oracle energy. Average over groups within task, then over tasks, and take the square root (NRMSE ↓). Thus a constant predictor has NRMSE=1; centered R²=1−NRMSE². Rank concordance ↑ compares non-tied oracle pairs, giving predicted ties half credit, with the same group-then-task averaging. We used 2,000 paired percentile bootstrap resamples of whole tasks. Intervals are pointwise and conditional on the frozen fit and split. The analysis uses the previously inspected archive.

History-contrast NRMSE is 0.707 [0.620, 0.787], centered R² is 0.500 and rank concordance is 0.691 [0.657, 0.728]. Contrast minus global NRMSE is -0.172 [-0.235, -0.103], supporting more accurate within-key progress prediction with contrast-target fitting.

Frozen predictorNRMSE ↓Task-bootstrap 95% CICentered R² ↑Rank concordance ↑
Constant1.000[1.000, 1.000]0.0000.500
Observation-only, global target1.000[1.000, 1.000]0.0000.500
History, global target0.879[0.801, 0.966]0.2280.607
History, contrast target0.707[0.620, 0.787]0.5000.691
History contrast, permuted labels1.470[1.395, 1.547]-1.1620.453
AuthorsOpenReview ↗

Official Response to Reviewer 6CMc (Part 3/7)

About Individual Proxy-Task Outcomes (E4)

The table reports all 16 test tasks for one frozen fit per candidate. T01–T16 follow the fixed task-ID sort order; eligibility and group counts are identical across methods. The statistical unit here is a test task, distinct from a policy-training run.

Test taskOracle-varying groupsHistory-global NRMSE ↓History-contrast NRMSE ↓Contrast rank ↑
T0150.8840.7400.657
T02130.8830.3390.668
T03160.8130.9200.746
T0440.8190.6310.661
T05160.8150.5960.827
T0691.1380.8760.826
T0790.8650.6720.663
T08140.7040.3930.573
T09120.9540.7880.708
T10180.6690.6220.755
T11120.9050.6320.645
T12180.9020.6790.682
T1391.2491.0110.562
T14130.6410.6330.739
T15130.8790.8390.659
T1640.7260.6050.689

Fifteen of sixteen tasks have contrast error below the constant reference, fifteen improve over global-target fitting, and all sixteen improve over shuffled labels. Counts use unrounded values. All candidates use the same oracle-defined evaluation groups, so the task support is identical.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.