- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Reinforcement Learning with Decomposed Subtasks (RLDS), which replaces a single trajectory-level advantage with credit assigned across named competencies and trajectory positions. The policy grades each rollout along a fixed subtask taxonomy, distributes the environment reward among those subtasks, and computes group-relative advantages. Gaussian kernels centered on self-identified pivotal steps then distribute these advantages across tokens. The method operates within a reflection-and-retry training framework. Experiments cover FrozenLake, HotpotQA, ScienceWorld, and DeepResearch, with positive point estimates on the first and third tasks and little improvement on the other two.
Critical strengths
The paper proposes a concrete approach to a meaningful credit-assignment problem. Distinguishing which competency failed from where it failed is a reasonable extension of trajectory-level training. The method is explicitly formulated, and the authors clearly acknowledge which parts of the reflection-and-retry framework are inherited. I also appreciate the reporting of unsuccessful settings, uncertainty estimates, and computational overhead. These make the results easier to assess and show that the authors are examining the method’s limitations.
Critical weaknesses
-
The experiments do not isolate the value of the main methodological contribution. Reflection, retries, distillation, and pivotal credit assignment already appear in the prior approaches discussed by the paper, including ERL (https://arxiv.org/abs/2602.13949) and R^3L (https://arxiv.org/abs/2601.03715). RLDS adds competency-based reward allocation and subtask-specific temporal weighting, but these are evaluated together with a change in advantage normalization. The reflect-retry baseline is useful, yet it does not establish which of these changes accounts for the improvement. Separate tests of decomposition, pivot weighting, and normalization are especially important because these mechanisms constitute the paper’s distinct contribution. Comparisons with the full R^3L method and relevant token-level credit-assignment methods would also strengthen the case beyond scalar GRPO.
-
The empirical evidence is too uncertain to support the broader conclusions. The reported gains over scalar GRPO are +6.3 points on ScienceWorld, with a 95% confidence interval of [−0.3, +12.8], and +5.9 on FrozenLake, with [−0.7, +12.5]. Both intervals include zero, while the other two benchmarks show little improvement. This does not establish that the method is ineffective, but it leaves its advantage unresolved. The computational tradeoff also matters: FrozenLake training steps cost about 2.4 times as much, and the paper explicitly leaves comparisons under matched compute budgets to future work. ScienceWorld’s lower step time is encouraging, but the combined evidence remains insufficient for the conclusion that subtask decomposition is generally the appropriate training primitive for these tasks.
Questions for authors
-
Evidence for the methodological contribution: What happens when subtask decomposition is used without pivot weighting, and when temporal weighting is used without competency-specific advantages? Can these comparisons keep normalization, reflection, and auxiliary losses fixed?
-
Clarification of the main comparison: What are the final-checkpoint gains and uncertainty intervals relative to the reflect-retry baseline, rather than scalar GRPO? This comparison is particularly relevant to assessing the incremental contribution of SDAE.
-
Evidence for practical significance: Does the FrozenLake improvement remain when methods receive the same wall-clock budget? For ScienceWorld, does the reduced step time translate into a reliable advantage in reward reached per unit of computation?
Reviewer's comment:
Response:
To isolate the contribution of token-level pivot masking (Stage 3) from reward decomposition (Stages 1–2), we ran an ablation on FrozenLake in which the decomposed reward is broadcast uniformly across the rollout, with no spatial credit routing. Decomposition alone accounts for the majority of the lift over base reflection (+0.031, ~69% of the total gain), while spatial credit contributes a smaller additional increment (+0.014, ~31%). Spatial credit's marginal contribution is positive in point estimate across seeds but is not statistically separable from seed variance at n=3.
Spatial Credit Ablation
Reviewer's comment:
Response:
| | ScienceWorld | 51.6 | 47.4 | +4.2 |
−2.2,+10.6|
RLDS is above the base reflection approach on both tasks, but both intervals include zero. We therefore do not claim a statistically significant incremental gain from SDAE at n=3. The direction is consistent across both tasks. The point estimates are +3.4 points on FrozenLake and +4.2 on ScienceWorld, compared with +5.9 and +6.3 over scalar GRPO. So a substantial share of the gain over GRPO remains once reflection, retries and auxiliary supervision are held fixed, and part of the headline lift is attributable to the reflect-retry machinery, as you suspected.
Reviewer's comment:
Response:
We report the full compute accounting for all three methods below. Reward-versus-wall-clock learning curves will be included in a future revision, and we do not claim compute-matched results for the cases below that need them. For ScienceWorld, the table supports a direct comparison against scalar GRPO.
ScienceWorld
All ScienceWorld runs used the same 8×H100 setup and ran for 400 training steps. Step time is therefore a direct proxy for compute. An RLDS step takes 262.2 s versus 294.4 s for RLVR, even though RLDS adds self-grading (25.3 s) and reflection (8.4 s) on top of generation. The saving comes from fewer rollouts per step (442 vs. 640) and much shorter generations (2.2K vs. 4.0K tokens per episode). Over 400 steps, RLDS takes about 29.1 h of wall-clock (233 GPU-hours) against about 32.7 h (262 GPU-hours) for RLVR, roughly 11% less. RLDS also reaches a higher final reward (51.6 vs. 45.4). So on ScienceWorld, RLDS reaches higher reward than scalar GRPO at lower total compute. Equivalently, in RLDS's 29.1 h of wall-clock, RLVR completes about 356 steps, and its validation curve is flat over its last 30 steps (its peak, 45.4, matches its final value). The gap therefore does not close under a matched wall-clock budget.
FrozenLake
An RLDS step costs about 2.4x an RLVR step (119.0 s vs. 49.8 s) and about 1.2x a reflect-retry step (119.0 s vs. 98.7 s). The improvement has not been shown to survive a matched wall-clock budget, and we do not claim it does.