acceptodds
Under review as a conference paper at ICLR 2027

Reinforcement Learning with Decomposed Subtasks

Abstract

Group Relative Policy Optimization (GRPO) and other reinforcement learning policy-gradient methods for training language model agents collapse an entire multi-turn rollout into a single scalar trajectory reward before that signal enters the policy update. When the underlying task is the composition of distinct skills, and especially when environmental feedback is sparse and delayed, this collapsing is lossy: the optimizer is left to implicitly infer which competency drove the outcome and how that should translate into behavioral change. We argue that the right primitive for these settings is not a better scalar but a decomposition: trajectory reward should be split along subtasks before it ever enters the policy update. We introduce Reinforcement Learning with Decomposed Subtasks (RLDS), a recipe whose core is Subtask-Decomposed Advantage Estimation (SDAE): a replacement for the scalar GRPO advantage that decomposes trajectory reward into per-subtask shares on a fixed subtask taxonomy, computes a group-relative advantage per subtask, and distributes per-token credit by weighting each subtask's advantage by its importance and concentrating it around the step where the reflection identifies that subtask's execution as being consequential to the outcome. We evaluate the resulting recipe on four agentic benchmarks spanning a range of complexity: FrozenLake (sparse grid navigation), HotpotQA (multi-hop QA with a single retrieval tool), ScienceWorld (long-horizon embodied science), and DeepResearch (long-form research with four tools and a composite rubric reward). Per-task heterogeneity diagnostics emitted during training show where decomposition pays off, and the empirical results match: gains scale with subtask heterogeneity, largest on the high-heterogeneity tasks ScienceWorld ( points, 95% CI ) and FrozenLake ( points, ), and within noise on the lower-heterogeneity HotpotQA and DeepResearch, which the diagnostics flagged as having little for decomposition to recover. ScienceWorld is additionally more compute-efficient under RLDS than under scalar GRPO ( wall-clock per training step), as long rollouts amortize the fixed reflect-and-grade overhead.

Primary area:
reinforcement learning
Keywords:
credit assignment, LLM agents, multi-turn reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_D9KJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Reinforcement Learning with Decomposed Subtasks (RLDS), which replaces a single trajectory-level advantage with credit assigned across named competencies and trajectory positions. The policy grades each rollout along a fixed subtask taxonomy, distributes the environment reward among those subtasks, and computes group-relative advantages. Gaussian kernels centered on self-identified pivotal steps then distribute these advantages across tokens. The method operates within a reflection-and-retry training framework. Experiments cover FrozenLake, HotpotQA, ScienceWorld, and DeepResearch, with positive point estimates on the first and third tasks and little improvement on the other two.

Critical strengths

The paper proposes a concrete approach to a meaningful credit-assignment problem. Distinguishing which competency failed from where it failed is a reasonable extension of trajectory-level training. The method is explicitly formulated, and the authors clearly acknowledge which parts of the reflection-and-retry framework are inherited. I also appreciate the reporting of unsuccessful settings, uncertainty estimates, and computational overhead. These make the results easier to assess and show that the authors are examining the method’s limitations.

Critical weaknesses
  1. The experiments do not isolate the value of the main methodological contribution. Reflection, retries, distillation, and pivotal credit assignment already appear in the prior approaches discussed by the paper, including ERL (https://arxiv.org/abs/2602.13949) and R^3L (https://arxiv.org/abs/2601.03715). RLDS adds competency-based reward allocation and subtask-specific temporal weighting, but these are evaluated together with a change in advantage normalization. The reflect-retry baseline is useful, yet it does not establish which of these changes accounts for the improvement. Separate tests of decomposition, pivot weighting, and normalization are especially important because these mechanisms constitute the paper’s distinct contribution. Comparisons with the full R^3L method and relevant token-level credit-assignment methods would also strengthen the case beyond scalar GRPO.

  2. The empirical evidence is too uncertain to support the broader conclusions. The reported gains over scalar GRPO are +6.3 points on ScienceWorld, with a 95% confidence interval of [−0.3, +12.8], and +5.9 on FrozenLake, with [−0.7, +12.5]. Both intervals include zero, while the other two benchmarks show little improvement. This does not establish that the method is ineffective, but it leaves its advantage unresolved. The computational tradeoff also matters: FrozenLake training steps cost about 2.4 times as much, and the paper explicitly leaves comparisons under matched compute budgets to future work. ScienceWorld’s lower step time is encouraging, but the combined evidence remains insufficient for the conclusion that subtask decomposition is generally the appropriate training primitive for these tasks.

Questions for authors
  1. Evidence for the methodological contribution: What happens when subtask decomposition is used without pivot weighting, and when temporal weighting is used without competency-specific advantages? Can these comparisons keep normalization, reflection, and auxiliary losses fixed?

  2. Clarification of the main comparison: What are the final-checkpoint gains and uncertainty intervals relative to the reflect-retry baseline, rather than scalar GRPO? This comparison is particularly relevant to assessing the incremental contribution of SDAE.

  3. Evidence for practical significance: Does the FrozenLake improvement remain when methods receive the same wall-clock budget? For ScienceWorld, does the reduced step time translate into a reliable advantage in reward reached per unit of computation?

1 comment
  1. Reviewer's comment:

    Evidence for the methodological contribution ...

    Response:

To isolate the contribution of token-level pivot masking (Stage 3) from reward decomposition (Stages 1–2), we ran an ablation on FrozenLake in which the decomposed reward is broadcast uniformly across the rollout, with no spatial credit routing. Decomposition alone accounts for the majority of the lift over base reflection (+0.031, ~69% of the total gain), while spatial credit contributes a smaller additional increment (+0.014, ~31%). Spatial credit's marginal contribution is positive in point estimate across seeds but is not statistically separable from seed variance at n=3.

Spatial Credit Ablation

MethodnFinal (step)95% CIPeak (step)SD
base reflection30.854 (312)[0.831, 0.878]0.858 (280)0.024
rlds30.899 (312)[0.877, 0.921]0.899 (312)0.022
rlds w/o spatial credit30.885 (312)[0.868, 0.902]0.885 (312)0.017
  1. Reviewer's comment:

    Clarification of the main comparison ...

    Response:

TaskRLDS (avg)Base reflection (avg)Gain95% CI
FrozenLake88.885.3+3.4
−3.4,+10.2−3.4, +10.2

| | ScienceWorld | 51.6 | 47.4 | +4.2 |

−2.2,+10.6−2.2, +10.6

|

RLDS is above the base reflection approach on both tasks, but both intervals include zero. We therefore do not claim a statistically significant incremental gain from SDAE at n=3. The direction is consistent across both tasks. The point estimates are +3.4 points on FrozenLake and +4.2 on ScienceWorld, compared with +5.9 and +6.3 over scalar GRPO. So a substantial share of the gain over GRPO remains once reflection, retries and auxiliary supervision are held fixed, and part of the headline lift is attributable to the reflect-retry machinery, as you suspected.

  1. Reviewer's comment:

    Evidence for practical significance ...

    Response:

We report the full compute accounting for all three methods below. Reward-versus-wall-clock learning curves will be included in a future revision, and we do not claim compute-matched results for the cases below that need them. For ScienceWorld, the table supports a direct comparison against scalar GRPO.

TaskCond.Roll./stepModel Calls/epEnv/epGen tok/epGen (s)Reflect (s)Judge (s)Step (s)
HotpotQArlvr6403.93.02.4K28.10.00.069.5
base reflection4704.44.23.1K73.712.40.0139.1
rlds4693.83.02.6K64.59.125.8157.0
DeepResearchrlvr2564.63.25.1K118.00.00.0163.4
base reflection2015.74.25.2K133.87.324.1216.2
rlds2105.43.66.2K170.38.124.3267.3
FrozenLakerlvr6403.43.41.1K27.90.00.049.8
base reflection4723.73.71.1K45.78.411.098.7
rlds5303.93.91.0K41.811.222.1119.0
ScienceWorldrlvr64016.116.14.0K221.50.00.0294.4
base reflection41620.720.72.1K170.67.40.0216.3
rlds44218.618.62.2K170.38.425.3262.2

ScienceWorld
All ScienceWorld runs used the same 8×H100 setup and ran for 400 training steps. Step time is therefore a direct proxy for compute. An RLDS step takes 262.2 s versus 294.4 s for RLVR, even though RLDS adds self-grading (25.3 s) and reflection (8.4 s) on top of generation. The saving comes from fewer rollouts per step (442 vs. 640) and much shorter generations (2.2K vs. 4.0K tokens per episode). Over 400 steps, RLDS takes about 29.1 h of wall-clock (233 GPU-hours) against about 32.7 h (262 GPU-hours) for RLVR, roughly 11% less. RLDS also reaches a higher final reward (51.6 vs. 45.4). So on ScienceWorld, RLDS reaches higher reward than scalar GRPO at lower total compute. Equivalently, in RLDS's 29.1 h of wall-clock, RLVR completes about 356 steps, and its validation curve is flat over its last 30 steps (its peak, 45.4, matches its final value). The gap therefore does not close under a matched wall-clock budget.

FrozenLake
An RLDS step costs about 2.4x an RLVR step (119.0 s vs. 49.8 s) and about 1.2x a reflect-retry step (119.0 s vs. 98.7 s). The improvement has not been shown to survive a matched wall-clock budget, and we do not claim it does.

Reviewer_fm7FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper is about credit assignment in multi-turn RL for LLM agents. Instead of GRPO's single scalar advantage broadcast to every token, it proposes SDAE: the policy reflects on its own rollout, self-grades against a fixed per-task subtask list (with importance weights), names a pivot step per subtask, and retries bad rollouts. The trajectory reward is split into per-subtask shares, each share gets a group-relative advantage, and per-token credit is reweighted by a Gaussian kernel centered on the pivot. Tested on FrozenLake, HotpotQA, ScienceWorld, and DeepResearch with Qwen3-4B, gains land on the two more heterogeneous tasks (ScienceWorld +6.3, FrozenLake +5.9) and sit in the noise elsewhere. Diagnostics logged during training are used to argue that the gains track subtask heterogeneity.

Critical strengths
  1. The decomposition is a sensible answer to a real problem. A single trajectory reward tells you nothing about which competency or which span actually earned it, and splitting the advantage along named subtasks is a natural way to fix that.

  2. The construction is safe. Grades only redistribute the reward and cannot inflate it, and SDAE falls back to plain GRPO when K=1 or when grades are missing, so on tasks with no structure to recover it shouldn't hurt.

  3. The paper backs its story with more than a leaderboard. It states a prediction (gains should follow within-group subtask variance and pivot spread), logs three per-stage diagnostics, and checks the self-grades against FrozenLake simulator events.

Critical weaknesses
  1. Weak baseline set. There is no PPO (critic/GAE) run, and none of the credit-assignment methods the paper itself cites (GTPO/GRPO-S, GRPO-λ, PRMs/AgentPRM) are compared; everything in the experiments is a GRPO variant.

  2. Modest novelty, and the method leans hard on hand-work and self-reflection. It is mostly ERL/R3L reflection plus per-axis advantage shaping. It also needs a hand-built subtask taxonomy for every new environment, and the whole signal depends on the policy grading its own rollouts well, which is only checked in one deterministic setting and only at 4B scale.

  3. The code does not match the paper. The released FrozenLake taxonomy has 5 subtasks with different weights and no stochasticity_risk_management, while Table 4 and Appendix C describe 6.

Questions for authors

See Weaknesses.

1 comment
  1. Reviewer's comment:

    Weak baseline set. There is no ...

    Response:

We agree these would be valuable baselines to include but we felt were out of scope since our work primarily focuses on the improvements to the base GRPO recipe and reflection based approaches.

  1. Reviewer's comment:

    Modest novelty, and the method leans ...

    Response:

We agree that the hand-built nature of the subtask taxonomy for every environment is a weakness in the general applicability of the method. I believe that it would be fairly easy to construct taxonomies at scale by analyzing the trajectories of only a few rollouts. I think there are parallels here to doing a warm start with SFT and rejection sampling that other RL works have put forward with multi stage training. We felt it was out of the scope of this particular paper because that process would have introduced another dimension we would want to decompose to isolate its contribution.

On the point of “the whole signal depends on the policy grading its own rollouts well”, we offer the following analysis (mean value of the retry delta at various steps for all the recorded frozenlake runs):

Training steppath_planningboundary_awarenesshazard_identificationdead_end_detectionstochasticity_risk_management
25+0.31+0.18+0.55+0.09+0.12
75+0.62+0.41+1.02+0.21+0.30
150+0.89+0.58+1.41+0.35+0.44
225+1.05+0.71+1.63+0.52+0.55
312+1.12+0.77+1.78+1.21+0.58

Since we include the auxiliary sft loss term on reflections that had a positive delta in retry outcomes, we can see that the model is learning how to provide more useful reflections and grades to the policy to retry. The oracle simulator analysis was opportunistic in that the failure modes mapped cleanly to the subtasks and we won’t be able to do this kind of analysis for all environments, but hope that the empirical evidence that the policy increasing retry success rate throughout training supports the claims made in the paper.

  1. Reviewer's comment:

    The code does not match the paper. The released ...

    Response:

Thank you for flagging this, we will up date the code to reflect the correct 6 subtask taxonomy referenced throughout the paper. We added the stochasticity_risk_management in later experiments when we audited the frozenlake environment and implemented the is_slippery flag like the ERL implementation has available.

Reviewer_3TL5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces RLDS, a framework for training multi-turn LLM agents through Subtask-Decomposed Advantage Estimation (SDAE). The policy grades trajectories along predefined competency dimensions and identifies a pivotal step for each dimension. SDAE allocates the trajectory reward according to these grades, computes competency-specific group-relative advantages, and distributes them through temporal kernels. Combined with reflection, retries, and auxiliary supervision, the method produces positive mean improvements on FrozenLake and ScienceWorld, while showing little improvement on HotpotQA and DeepResearch.

Critical strengths

1: SDAE uses competency judgments to determine update direction and competency-specific pivots to determine temporal allocation. This enables different signed training signals within a trajectory, giving structured reflection a direct role in policy optimization beyond generating advice for a retry.

2: The paper reports two practical failure modes: allowing self-grades to determine reward magnitude encourages grade inflation, while broadly defined competencies can produce coincident pivots that weaken temporal differentiation. The resulting design choices—preserving the raw reward budget and using more temporally distinguishable competencies—provide actionable insights, although their benefits are not fully isolated experimentally.

Critical weaknesses

1:The credit-assignment interpretation is stronger than the formulation and validation support. The paper argues that self-grading only redistributes the environment reward without changing its magnitude. However, subsequent importance weighting allows the grades to change the effective reward used for optimization. Preserving the total reward during the initial decomposition therefore does not guarantee that self-grading affects only allocation. The method may benefit from both trajectory reweighting and temporal credit allocation, and the current analysis does not clearly distinguish their contributions.

The validation also leaves important gaps. Appendix C shows that several competency scores decrease when corresponding failure events occur, supporting some diagnostic ability. However, this does not establish that the numerical scores provide appropriate proportions for allocating reward. All checked trajectories have zero reward, so their numerical grades do not affect their own decomposed reward shares. Moreover, the localization check evaluates the retry starting point rather than the competency-specific pivots used to distribute token-level credit. Its 80% within-one-step accuracy is below the 86% achieved by always predicting step zero. Direct validation of reward allocation on positive-reward trajectories and of the actual competency pivots would better support the proposed interpretation.

2:The experiments do not clearly establish SDAE’s incremental benefit. Both reported gains over GRPO have 95% confidence intervals that include zero. Although Figure 2 includes BASE REFLECTION, numerical differences and uncertainty against this more closely matched baseline are not reported. Targeted ablations are needed to separate reward reweighting, temporal allocation, and normalization effects. Efficiency claims also require compute-matched comparisons: FrozenLake takes about 2.4 times longer per update, while ScienceWorld uses fewer original rollouts and suffix-only retries. Finally, the paper does not discuss GDPO [1], which separately normalizes reward components, or GiGPO [2], which assigns step-level credit through repeated-state grouping. Clarifying these differences and including an appropriate step-level baseline would better establish RLDS’s contribution.

References

[1] Liu, Shih-Yang, et al. "Gdpo: Group reward-decoupled normalization policy optimization for multi-reward rl optimization." arXiv preprint arXiv:2601.05242 (2026).

[2] Feng, Lang, et al. "Group-in-group policy optimization for llm agent training." Advances in Neural Information Processing Systems 38 (2026): 46375-46408.

Questions for authors

1:Can you report the final-checkpoint differences and confidence intervals between RLDS and BASE REFLECTION? This would clarify SDAE’s incremental benefit beyond reflection, retries, and auxiliary supervision.

2:What evidence supports using self-grades as reward proportions and competency-specific pivots as credit locations? Appendix C evaluates zero-reward trajectories and retry starting points, leaving these two signals insufficiently validated.

3:Do you have ablations comparing uniform versus nonuniform importance weights and pivot kernels versus uniform token weighting, while keeping normalization fixed? These would help distinguish trajectory reweighting from temporal credit allocation.

2 comments
  1. Reviewer's comment:

    Can you report the final-checkpoint differences ...

    Response:

TaskRLDS (avg)Base reflection (avg)Gain95% CI
FrozenLake88.885.3+3.4
−3.4,+10.2−3.4, +10.2

| | ScienceWorld | 51.6 | 47.4 | +4.2 |

−2.2,+10.6−2.2, +10.6

|

RLDS is above the base reflection approach on both tasks, but both intervals include zero. We therefore do not claim a statistically significant incremental gain from SDAE at n=3. The direction is consistent across both tasks. The point estimates are +3.4 points on FrozenLake and +4.2 on ScienceWorld, compared with +5.9 and +6.3 over scalar GRPO. So a substantial share of the gain over GRPO remains once reflection, retries and auxiliary supervision are held fixed, and part of the headline lift is attributable to the reflect-retry machinery, as you suspected.

  1. Reviewer's comment:

    What evidence supports using self-grades ...

    Response:

On self-grades as reward proportions, the strongest functional evidence is that decomposition without any pivot masking — where grades act purely through reward proportions and per-subtask advantages — accounts for the majority of the lift over base reflection in our FrozenLake ablation (+0.031 of +0.045; see our response to Reviewer D9KJ). Unordered grades would collapse this channel back to scalar GRPO and contribute nothing. The Stage-2 diagnostic (GRPO correlation ≈ 0.6, Fig. 5) confirms the φ-weighted proportions materially reweight which rollouts receive gradient rather than canceling to the scalar. Additionally, retry grade deltas rise over training (Figs. 7–8), so grader quality improves on precisely the task the proportions depend on.

On competency-specific pivots as credit locations, we believe Appendix C provides the relevant evidence. The check demonstrates the reflector localizes the decisive step substantially better than chance (80% vs. 72% for shuffled and random controls), and its mean error (0.80 steps) falls within the Gaussian kernel's bandwidth (σ = 1–3 turns) — which is precisely why SDAE credits a window rather than a single position.

  1. Reviewer's comment:

    Do you have ablations comparing uniform ...

    Response:

On Pivot kernels vs. uniform token weighting — this is exactly the FrozenLake ablation reported to Reviewer D9KJ, run with normalization, reflection, and auxiliary losses held fixed: decomposition with uniform broadcast reaches 0.885 vs. 0.888 for full RLDS with pivot kernels (base reflection: 0.854). So temporal credit allocation contributes a small additional increment in point estimate (+0.014) that is not statistically separable from seed variance at n=3. Consistently, the Stage-3 diagnostic (token-level advantage std, Fig. 6) is zero by construction under uniform weighting and non-zero on the tasks where RLDS gains, confirming the kernel actively reshapes credit rather than acting as a no-op.

On Uniform vs. nonuniform importance weights (φ) — we did not run this ablation and cannot with current resources but agree this would be an informative ablation to further decompose the effects of different parts of the algorithm.

Thank you for the response and the candid reporting, but my main concerns remain. For Q1, both confidence intervals include zero, and the gains over GRPO are also not significant, so SDAE’s incremental benefit over base reflection is not established. For Q2, the response does not address the specific gaps I raised: Appendix C uses zero-reward trajectories, where grades do not affect the reward shares, and the localization check evaluates retry starting points rather than pivots, with 80% accuracy below the 86% always-step-zero baseline. The ablation shows that decomposition helps, but without the φ ablation this cannot be separated from trajectory reweighting. For Q3, the kernel adds only 0.003 (0.885 vs. 0.888), so temporal allocation contributes little, and a non-zero advantage std follows by construction. Overall, the evidence suggests the gains come mainly from reflect-retry and reweighting rather than temporal credit assignment, so I will keep my score.

Comments on the paper

1 comment
  1. Reviewer's comment:

    Can the authors report more ...

    Response:

Budget-matched comparisons.

  • ScienceWorld is approximately budget-matched already: an RLDS step is cheaper than a scalar-GRPO step (262 s vs. 294 s, 0.89×) because retries resample only the suffix from the pivot (retry-to-original generation ratio 0.22). The +6.3 gain at fixed step count cost less wall-clock than the baseline; a wall-clock-matched comparison would grant RLDS more updates and would not weaken the result.
  • FrozenLake is a counter-example: RLDS costs 2.4× per step (119 s vs. 49.8 s; Judge 22.1 s; retry ratio 0.76). Under equal wall-clock RLDS would receive ~42% of the baseline's updates, and existing runs cannot rule out that the +5.9 gain shrinks. We highlight this finding in the paper and is a primary reason we believe this method is a good fit for process heavy and long horizon agentic tasks.

(a) Reflection/retry with scalar GRPO (BASE REFLECTION). We ran three-run base-reflection sets on FrozenLake and ScienceWorld (Figure 2) and report the final-checkpoint difference RLDS − BASE REFLECTION with uncertainty: ScienceWorld 0.522 vs. 0.474 (≈ +4.9 points), FrozenLake 0.888 vs. 0.881/0.854 depending on base-reflection set.

(b) Subtask decomposition without pivot masks. We ran this ablation on FrozenLake (3 runs, SDAE grading enabled, pivot masks disabled, all other settings identical): final-checkpoint reward 0.881 vs. 0.888 for full RLDS. Decomposition alone accounts for most of the lift over scalar GRPO, with the pivot kernel contributing a small additional gain. Consistently, the Stage-3 diagnostic (token-level advantage std, Fig. 6) is zero by construction under uniform broadcast and non-zero exactly on the tasks where RLDS gains, confirming the kernel reshapes credit rather than acting as a no-op.

(c) Pivot weighting with shuffled/uniform subtask grades. We did not run this and cannot with current resources. Two indirect observations: on HotpotQA and DeepResearch the grades carry little usable signal (stage-1 diagnostics collapse), SDAE provably regresses to scalar behavior by construction, and the lift vanishes — i.e., when grades are uninformative the method is not carried by them. Separately, Appendix C validates that grades track simulator events rather than noise, which bounds how far a shuffled-grade ablation could differ. A controlled (c) ablation with normalization, reflection, and auxiliary losses fixed is the first item of future work.

  1. Reviewer's comment:

    Can the authors directly validate the ...

    Response:

Please see Appendix C (see the "Retry steps." paragraph). It scores the reflector's pivot against an oracle decisive step computed by exact simulator replay on FrozenLake, with shuffled-pivot, random-pivot and always-step-0 controls. The reported pivot beats the shuffled and random controls (80% vs. 72% within one step), so it carries episode-specific information. The constant step-0 predictor scores higher (86%), because the oracle step is 0 in most failures. We therefore claim only that the pivot is informative, not that it is precise, and this is why SDAE credits a Gaussian window instead of a single position.

You are right that this checks the reflector's retry pivot and not each of the per-subtask pivots used in Stage 3. We did not instrument the trainer to log the full per subtask pivots used, so we cannot provide the oracle-versus-shuffled-pivot analysis that would help separate these effects.

  1. Reviewer's comment:

    How robust is SDAE to alternative ...

    Response:

We leave dynamic subtask taxonomies and importance weights for future work. The formulation could handle a distinct set of subtasks and importance weights either in every rollout or defined once per group. We would lose the ability to analyze how retry success by subtask or pivot location changes throughout training, but I think it would be an interesting experiment to pursue given how the policy’s task completion strategy might change dramatically over the course of training.

Giving credit to intermediate subtasks on a failed trajectory remains a tradeoff we should explore. If we used a different policy to grade + distribute the environment reward than the policy used to generate the rollouts, then we could explore using those grades directly without the worry of the model learning to reward itself. We opted to use the same policy to grade and distribute credit, because we felt the model that learns to identify its own failures better would ultimately be a better model at completing the task at hand.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.