acceptodds
Under review as a conference paper at ICLR 2027

Using Experience Twice: Dynamic Process Rubrics for Privileged Token-Level Credit Assignment

Abstract

Agent trajectories often contain both critical successful decisions and failed attempts; therefore, optimization with sparse trajectory-level rewards struggles to indicate which intermediate decisions to reinforce or suppress. We observe that meaningful distinctions among on-policy trajectories can be distilled into natural-language process rubrics and used as process supervision signals. Based on this insight, we propose Online Rubric Self-Distillation(ORSD), which leverages self-derived process rubrics for dense token-level supervision, thereby augmenting outcome-centric reinforcement learning. Specifically, at each update, the current policy independently generates multiple rubric bundles from the same group of trajectories, extracting task-specific candidate process criteria from their cross-trajectory behavioral differences. A frozen verifier then evaluates all rubric criteria against each trajectory in the group, and selects one candidate bundle based on bundle-level alignment between rubric satisfaction and task success. ORSD treats the selected rubric bundle as privileged information during training, then compares the log-likelihoods of the original tokens with and without the rubric. The resulting likelihood gap provides token-specific process credit, which is added to the original trajectory-level GRPO advantage through additive advantage shaping. ORSD thus bridges trajectory-level outcome supervision and fine-grained process supervision, enabling token-level credit assignment using process signals. Extensive experiments on ALFWorld and WebShop show that ORSD accelerates policy improvement and boosts final task performance for both Qwen3-1.7B and Qwen3-4B under the same environment-interaction budget.

Primary area:
reinforcement learning
Keywords:
LLM Agents, Reinforcement Learning, Credit Assignment, Process Rubrics, Group Relative Policy Optimization, Self-Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bmB3OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper propose ORSD(Online Rubric Self-Distillation), a self-distillation method using rubrics for token level supervision. The method uses the policy, which is being optimized, to generate bundles of rubrics, and a verifier to evaluate the trajectories on each rubric. Next, the rubric bundle that has highest spearman correlation with golden reward is selected, and used as additional infomation in on policy self distillation.

Critical strengths

The paper proposed a interesting method that uses rubrics to generate token level supervision for GRPO.

Critical weaknesses

Major Weakness

W1: Unclear motivation of policy-generated rubric

Why should the policy, which is use to generate trajectories, be used to generate rubrics as well? Normally, a good trajectory-generator is not necessarily a good rubircs generator. For example, a coding LLM is good at generating code, but may not be good at generating natural language rubrics. How was the quality of rubrics generate by policy? Will the policy generate better rubric with the process of training?

The experiment with a fixed rubric generator(for example, a capable LLM) / human rubric [2] should be added as well.

W2: Unclear about motivation for using rubrics.

The rubrics is originally used to replace golden answer needed in GRPO/RLVR [1], in senario which golden answer is hard to obtain. But in this paper, rubrics are used as context to generate better answers. What if the rubrics are used as actual reward? Can this method be used when golden answer is missing?

W3: Lack overhead analysis.

During training, ORSD require generating rubrics, verifying rubrics, calculating correlation and so on. What is the overhead cost of these steps(over different rubrics bundle number m)? Does is significantly slow down training?

W4: Lack ablation study.

  • Bundle number m: only m = 1(w/o Rubric–Reward Alignment) and 4 are presented. What about other m? What is the cost of different m(in W3)?
  • How to decide the number of rubrics inside one bundle? Does this affect the result? What happens if only some rubrics in a boundle are accurate, while others are wrong?

Minor Weakness:

  • Presentation is really hard to follow. Background is not properly introduced, like Rubrics, GRPO and self-distillation/OPSD, which makes it hard for people to understand.
  • Section 3.1 "problem setup" do not set up the problem in the paper.
  • Figure 1 is crowded and not infomative. Robots here are just visual distraction.
  • Figure 2 also not infomative, the result can be shown in a small table or a paragraph. Consider adding more data.
  • Experiment only done in Qwen model family. What about other model families?

[1] Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains

[2] HealthBench: Evaluating Large Language Models Towards Improved Human Health

Questions for authors

See weakness

6 comments
AuthorsOpenReview ↗

Response to Reviewer bmB3 (1/3)

Thank you for the constructive feedback. We have added comparisons with an external rubric generator and alternative verifier, candidate-count and rubric-pruning ablations, training-cost measurements, and experiments with Llama3.2-3B-Instruct. These results support the effectiveness of policy-generated guidance and clarify the trade-off between performance and additional computation. We have also revised the background and method presentation to explain the role of rubrics more clearly.

[W1.1] Why use the trajectory policy to generate rubrics, and how does it compare with a fixed external generator?

(1) The motivation for using the current policy is to derive guidance from the behavior it is currently learning to improve. A related precedent is DeepSeek-V4 [3], which uses the actor itself as a generative reward model and jointly optimizes its generation and judgment capabilities. The DeepSeek-V4 report explains that this shared-model design brings the model’s reasoning capabilities into its evaluation process, supporting the possibility of using the same model for generation and evaluation. In ORSD, the policy compares trajectories for the same task and proposes criteria that capture their behavioral differences. The verifier and reward-alignment selection then assess which candidate bundle best distinguishes the observed outcomes. This motivates using the policy to propose guidance while testing its usefulness through the comparisons below.

(2) Table R1 separates the effects of rubric generation and verification. With the verifier held fixed, current-policy generation outperforms DeepSeek generation under both verifiers. This supports using the policy’s own rollout experience to construct useful rubric context.

With the generator held fixed, DeepSeek verification improves downstream success for both generators. Thus, policy-generated guidance remains effective across verifiers, while verifier choice provides an additional source of improvement. The best combination uses current-policy generation with DeepSeek verification. These comparisons appear in Figure 2(b), with experimental settings in Appendix C.4.

Table R1. WebShop success (%) with Qwen3-4B under different rubric generators and verifiers.

Rubric generatorQwen3.5-Plus verifierDeepSeek verifier
Current policy77.0584.00
DeepSeek73.3580.00

[W1.2] How does the quality of policy-generated rubrics change during training?

We compared Qwen3-4B checkpoints on three fixed WebShop tasks with identical stored trajectories, M=4, and DeepSeek verification. Table R2 shows rubric excerpts for the same footwear task, illustrating changes in attribute checks, concrete actions, and error avoidance. Both selection and held-out correlations peak at step 100 and vary across checkpoints, reinforcing the need to assess candidate alignment at each update rather than assuming that later rubrics are better. Appendix C.7 provides the evaluation protocol and results for all seven checkpoints.

Table R2. Verbatim excerpts from selected rubrics for the same WebShop task. Two criteria are shown per checkpoint, with original item numbers retained. The task requires men's slip-resistant loafers in color black2626, size 14, priced below $70.

Training stepSelected rubric excerpts
0(3) Ensure the product has a rubber outsole and rubber sole before purchasing. (4) Verify the product color is black2626 and size is 14 before buying.
100(3) Select the size 14 option on the product page. (4) Ensure the product color is set to black2626 before proceeding to purchase.
200(3) Select the size 14 from the size options. (4) Ensure the product price is below $70 before purchasing.
300(4) Confirm the product has a slip-resistant outsole. (5) Avoid selecting incorrect color options or sizes that do not match the task requirements.
AuthorsOpenReview ↗

Response to Reviewer bmB3 (2/3)

[W2] Why use rubrics as context rather than rewards, and can ORSD work without gold answers?

(1) In ORSD, rubric context is used to obtain token-level log-probability differences on trajectories that have already been sampled. We score the same tokens with and without the selected rubric, then use the differences to refine their GRPO advantages. No new answer is generated during this replay. Using rubric satisfaction directly as a reward would instead change the training signal to criterion compliance, whereas the role we study is to derive token-specific guidance while retaining outcome-based supervision.

(2) As for whether ORSD can work without gold answers or equivalent outcome supervision, the current method requires reliable correctness feedback. This can come from a gold answer or, in ALFWorld and WebShop, the environment. Rubric scoring measures whether a trajectory satisfies a criterion. An unmet criterion can still be informative when its violation consistently accompanies task failure. Without external outcome feedback, however, rewarding rubric satisfaction alone could encourage the policy to satisfy its own criteria without achieving the actual task objective.

(3) Our rubric–reward alignment mechanism therefore selects bundles whose satisfaction scores agree with observed outcomes. The gold answer is not supplied as privileged context, and outcome rewards are excluded from the rubric-generation prompt. Instead, the selected rubric conveys process guidance derived from the trajectories, while outcome feedback anchors its selection to task success. We clarify this supervision requirement in Appendix C.4.

[W3] What is the overhead of rubric generation, verification, and selection, and does it significantly slow down training?

(1) We measured the additional components and complete training runtime for Qwen3-4B on WebShop. Table R3 reports both candidate counts. Generation, verification, and privileged replay account for almost all of the additional component time, while correlation computation and selection take negligible time in these runs. Appendix C.5 provides the cost and workload records.

Table R3. Measured component costs and runtime over 300 training steps on four GPUs. Both runs collect 19,200 environment trajectories. Training runtime excludes validation and checkpoint saving. At M=4, additional components account for 17.54% of measured training time.

ComponentM=4M=6
Policy-generated rubrics10.24 min15.53 min
Verifier waiting time23.17 min20.04 min
Spearman computation and selection0.09 s0.16 s
Rubric parsing and preparation0.56 min0.88 min
Privileged log-probability replay36.95 min49.59 min
Total additional component time70.92 min86.04 min
Average per training step14.18 s17.21 s
Complete training runtime6.74 h16.92 h
Environment rollout time2.74 h11.49 h

(2) The complete runtime difference cannot be attributed solely to candidate count. As Table R3 shows, most of the increase occurs during environment rollout, alongside longer responses and more groups triggering rubric generation. Appendix C.5 reports these workload differences separately from the component costs. Both runs use the same environment-interaction budget. The additional rubric operations occur only during training, while inference uses the trained policy directly.

[W4.1] How do other candidate counts affect performance and cost?

We added a larger candidate pool and a random-selection control to distinguish the value of more candidates from that of reward-aligned selection. Table R4 shows that alignment-based selection clearly improves performance at a fixed candidate count. Expanding the aligned candidate pool produces only a small success gain and does not improve task score. These comparisons appear in Table 2 and Figure 2(a) of the revised paper. Table R4 also includes rubric pruning, discussed under W4.2.

The alignment curves in Figure 5, Appendix C.6, are consistent with this result. Table R5 summarizes the small change in selected-bundle alignment alongside the increase in measured component time. The component breakdown is already provided in Table R3. Together, these observations support the default candidate count within the tested range.

Table R4. Candidate-count, selection, and rubric-pruning ablations with Qwen3-4B on WebShop.

SettingScoreSuccess (%)
Random selection, M=484.9469.90
Alignment selection, M=4, full bundle89.0277.05
Alignment selection, M=688.6777.25
Alignment selection, M=4, two highest-alignment items88.0476.25

Table R5. Alignment and marginal cost when increasing the candidate count.

ComparisonValue
Selected-bundle correlation, M=40.7884
Selected-bundle correlation, M=60.7964
Increase in additional component time, M=4 to M=621.32%
AuthorsOpenReview ↗

Response to Reviewer bmB3 (3/3)

[W4.2] How is the number of criteria per bundle chosen, and what happens when some criteria are less accurate?

(1) Our generation prompt requests 5–8 criteria per bundle. To test sensitivity to lower-alignment items, we prune the selected bundle to its two highest-alignment criteria at M=4. Each criterion is ranked by the Spearman correlation between its verifier scores across trajectories and the corresponding outcome rewards. The results appear in Figure 2(a) and Section 4.3 of the revised paper, with the construction described in Appendix C.4.

(2) Table R4 shows that removing lower-alignment criteria does not improve performance over the complete selected bundle. Together with the random-selection comparison, this supports the effectiveness of reward-aligned bundle selection when individual criteria differ in alignment. The selection mechanism evaluates the bundle as a whole, and these results support retaining its combined process guidance rather than requiring every criterion to achieve high individual alignment.

[Minor 1] The background is hard to follow, and Section 3.1 does not establish the problem setup.

(1) We have revised the Introduction to introduce rubrics and their use in prior work, including Rubrics as Rewards [1] and HealthBench [2], and to explain how rubric-based supervision relates to the credit-assignment problem.

(2) Section 3.1 is now titled “Preliminaries” and explains GRPO and OPSD before introducing ORSD. The transition to Section 3.2 makes their connection explicit: GRPO provides a shared trajectory-level advantage, while privileged context offers a way to introduce token-level supervision. ORSD constructs that context from the policy’s rollout experience.

[Minor 2] Figure 1 is crowded, and Figure 2 provides too little information.

(1) We have revised Figure 1 by reducing the robot icons, enlarging the text, and improving the spacing and arrow routing to make the training procedure easier to follow.

(2) We have consolidated the token-credit, candidate-count, rubric-pruning, generator/verifier, and alpha-sensitivity comparisons into the three-panel Figure 2. This presents the original sensitivity analysis alongside the additional experiments in a single figure, with numerical results labeled directly.

[Minor 3] What about model families beyond Qwen?

We have extended the evaluation to Llama3.2-3B-Instruct on both benchmarks, comparing ORSD with GRPO, GRPO+OPSD, and standalone OPSD. Table R6 shows that ORSD outperforms all three baselines on both environments. These results provide evidence beyond the Qwen family and are included in Table 1 of the revised paper, with evaluation settings in Appendix C.4.

Table R6. Llama3.2-3B-Instruct success (%) on WebShop and ALFWorld at step 300 under mean@4.

MethodWebShopALFWorld
GRPO76.0587.68
GRPO+OPSD67.1091.07
OPSD2.575.71
ORSD79.0893.21

References

[1] Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains

[2] HealthBench: Evaluating Large Language Models Towards Improved Human Health

[3] DeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. 2026. “Generative Reward Model” paragraph under Section 5.1.1 (“Specialist Training”). Paper.

Reviewer_UzTEOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper's idea is based on successful and failed trajectories. The paper uses self-derived process rubrics for dense token-level supervision to augment outcome-centric reinforcement learning. The current policy independently generates multiple rubric bundles from the same group of trajectories using cross-trajectory behavioral differences. There is a verifier to evaluate rubrics to select one candidate bundle based on bundle-level alignment. The proposed method uses the rubric bundle as privileged information during training. It is added to GRPO as token-level credit assignment. The experiments use ALFWorld and WebShop environments and Qwen3-1.7B and Qwen3-4B models.

Critical strengths

The idea using rubrics online for policy training is interesting. The paper proposes a new method that uses rubrics in distillation. The paper provides a method to generating rubrics and proposes mechanism for selecting rubrics. The paper provides experiment results using two LLMs.

Critical weaknesses

Generally the idea is interesting. However, current experiments do not support the idea very well. The experiments are limited to only two models. It would be better to consider more models. The main results are not complete. For example, Qwen3-4B is missing some main comparisons like OPSD and GRPO+OPSD. The main text should provide more details about how to generate rubrics. The paper seems to use Qwen3.5-Plus as the only verifier.

Questions for authors

There are actually two questions for the verifier: why not use other models? And why not use it to generate rubrics?

3 comments
AuthorsOpenReview ↗

Response to Reviewer UzTE

Thank you for the constructive feedback. We have added Llama3.2-3B-Instruct experiments, Qwen3-4B comparisons with OPSD and GRPO+OPSD, and alternative rubric-generator and verifier experiments. The results show that ORSD outperforms the added baselines on both benchmarks and that policy-generated rubrics outperform DeepSeek-generated rubrics with either verifier, providing further evidence for the effectiveness of our method. These additions are highlighted in blue in the revised manuscript.

[W1 / W2] The experiments are limited to two models, and Qwen3-4B is missing comparisons with OPSD and GRPO+OPSD.

Table R1 addresses both concerns using results from our experimental pipeline. The Llama3.2-3B-Instruct experiments extend the evaluation beyond the Qwen family, while the Qwen3-4B runs complete the missing self-distillation comparisons. ORSD outperforms all listed baselines on both benchmarks for both models. These comparisons support its effectiveness across model families and its benefit beyond standalone self-distillation or its combination with GRPO. The complete results appear in Table 1 of the revised paper, with shared training settings in Appendix C and evaluation protocols in Appendix C.4.

Table R1. Success rates (%) at step 300, mean@4. All results are from our experiments. WS denotes WebShop, and ALF denotes ALFWorld. Gains are in percentage points.

MethodQwen3-4B WSQwen3-4B ALFLlama3.2-3B WSLlama3.2-3B ALF
GRPO66.1574.0776.0587.68
OPSD4.2535.202.575.71
GRPO+OPSD69.9568.5767.1091.07
ORSD77.0577.4379.0893.21
ORSD gain over GRPO+10.90+3.36+3.03+5.53
ORSD gain over GRPO+OPSD+7.10+8.86+11.98+2.14

[W3] The main text should provide more details about how to generate rubrics.

We have expanded Section 3.2 of the revised paper and provided the complete generation prompt in Appendix B. The current policy receives the task description and its same-task rollout group, then independently generates four candidate bundles with 5–8 process criteria each. A frozen verifier scores each trajectory against the criteria, and rubric–reward alignment selects the bundle whose aggregate satisfaction scores have the highest Spearman correlation with the observed rewards. The selected bundle becomes privileged context for replaying the same sampled tokens, and the resulting log-probability differences refine their GRPO advantages. Rubrics are not required at inference time.

[Q1 / Q2] Why not use other verifier models, and why not use an external model to generate rubrics?

We test both choices in a single crossed comparison using Qwen3-4B as the trained policy. Table R2 compares current-policy and DeepSeek generation under each verifier. The DeepSeek API identifier is deepseek-v4.1-flash-1017. All four combinations appear in Figure 2(b) of the revised paper, with experimental settings in Appendix C.4.

Table R2. Qwen3-4B WebShop success (%) at step 300, mean@4. Gain entries are in percentage points.

Rubric generatorQwen3.5-Plus verifierDeepSeek verifierGain from switching verifier to DeepSeek
Current policy77.0584.00+6.95
DeepSeek73.3580.00+6.65

(1) With the generator held fixed, DeepSeek verification improves downstream success for either generator. ORSD can therefore work with another verifier and benefit from its evaluations.

(2) With the verifier held fixed, current-policy generation outperforms DeepSeek generation in both comparisons. External generation is compatible with ORSD, while these results support our choice of the current policy as the rubric generator.

Thank you for the response. I updated score. The authors add one more model for experiments. However, the models are still limited.

AuthorsOpenReview ↗

Response to Reviewer UzTE

Thank you for your feedback and for updating your score. We are glad that our additional experiments help address your concerns. We have now evaluated our method on three model backbones, including Qwen3-1.7B, Qwen3-4B, and Llama-3.2-3B-Instruct. We acknowledge that broader evaluations would further strengthen our findings and will consider this in future work.

Reviewer_sn4aOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Online Rubric Self-Distillation (ORSD) for training language-model agents. For each group of on-policy trajectories, the policy generates several task-specific rubric bundles. A frozen verifier scores the trajectories against these rubrics, and the bundle whose scores correlate most strongly with trajectory rewards is selected. The policy then replays the sampled responses with and without the selected rubric, using the token log-probability difference as an additive correction to the GRPO advantage. Experiments on ALFWorld and WebShop show improvements over GRPO with Qwen3-1.7B and Qwen3-4B.

Critical strengths
  1. Reusing the same rollout group to construct training-time guidance is a sensible way to obtain more supervision from expensive agent interactions. The rubrics are generated by the policy itself and are not required at inference time, making the approach relatively straightforward to add to an existing GRPO pipeline.

  2. The improvements over the matched GRPO baseline are consistent across both environments and model sizes. Some are substantial: ALFWorld success increases from 53.92% to 66.04% with the 1.7B model, and WebShop success increases from 66.15% to 77.05% with the 4B model. These results make the approach worth investigating even though the explanation for the gains needs stronger support.

Critical weaknesses
  1. The paper does not yet show that the likelihood correction provides meaningful credit assignment within a trajectory. Equation (8) measures how adding a rubric changes the probability of a sampled token. This need not indicate whether the underlying decision helped or harmed task completion. Selecting rubrics by trajectory-level reward correlation does not resolve this issue. Moreover, the raw correction changes the mean trajectory advantage as well as the relative weights of its tokens, as Appendix D notes. The reported gains could therefore arise from trajectory reweighting or privileged-context self-distillation without requiring useful token-level discrimination. A comparison against a trajectory-constant correction derived from the same likelihood gaps would directly test this distinction. An analysis of the corrections on identifiable useful, redundant, and incorrect actions would also help substantiate the central claim.

  2. The ablations do not cleanly identify the value of online adaptation and reward-aligned selection. The “without alignment” variant changes the number of candidates from four to one and removes verifier scoring at the same time. The offline variant uses a single bundle generated before training, so its comparison with ORSD also changes candidate generation and selection. In addition, the self-distillation baselines in Table 1 use published results under their original evaluation protocols, rather than matched implementations. Since privileged-context self-distillation is closely related to the proposed update, I would like to see a matched comparison using simpler guidance extracted from the same rollouts. The absence of reported uncertainty across training seeds makes the smaller differences harder to assess.

Questions for authors
  1. How does ORSD perform if each trajectory receives its mean likelihood correction at every token, instead of the token-specific correction? This would help determine whether within-trajectory credit assignment is important to the gains.

  2. Can you isolate the selection mechanism by sampling four bundles in every variant and comparing reward-aligned selection with random selection? For the offline comparison, were the initial rubrics generated with access to rollout groups and selected using the same procedure?

  3. How is bundle selection handled when all eight trajectories have the same reward, making Spearman correlation undefined? How often does this occur, particularly on ALFWorld, and is the rubric correction still applied to these groups?

4 comments
AuthorsOpenReview ↗

Response to Reviewer sn4a (1/2)

We sincerely appreciate the reviewer for highlighting the strengths of our work, including the efficient reuse of rollouts, practical integration with GRPO, and consistent performance improvements across environments and model sizes. Below we address the main points raised in the review. Revisions are highlighted in blue in the updated manuscript.

[W1 / Q1] How does ORSD perform if each trajectory receives its mean likelihood correction at every token, instead of the token-specific correction? This would help determine whether within-trajectory credit assignment is important to the gains.

(1) To distinguish token-level allocation from trajectory-level reweighting, we added a trajectory-mean control with Qwen3-4B on WebShop and ALFWorld. It averages the same likelihood corrections over valid policy-generated tokens within each trajectory and assigns that mean to every token, keeping the remaining training configuration unchanged. This preserves the trajectory-wide mean correction while removing token-level variation. Table R1 shows that ORSD outperforms this control on both environments, supporting the contribution of token-level allocation beyond the mean trajectory correction. The comparison is also included in Figure 2(a) of the revised paper, with the control defined in Appendix C.4.

Table R1. Qwen3-4B success rates (%) at step 300, mean@4. Gains are in percentage points.

MethodWebShopALFWorld
ORSD77.0577.43
Trajectory-mean73.2075.74
ORSD gain over trajectory-mean+3.85+1.69

(2) Figure 4 in Section 4.4 of the revised paper also provides a token-correction illustration, showing how corrections can vary within the same WebShop response.

[W2.1 / Q2] Can you isolate the selection mechanism by sampling four bundles in every variant and comparing reward-aligned selection with random selection? For the offline comparison, were the initial rubrics generated with access to rollout groups and selected using the same procedure?

(1) To isolate the selection mechanism, we retain the same candidate count and replace reward-aligned selection with random selection. Table R2 shows that full ORSD outperforms this control, indicating that the improvement depends on selecting useful rubric context rather than simply generating multiple candidates. The offline and simpler-guidance results are reported alongside this comparison, with their construction explained below.

Table R2. Qwen3-4B WebShop ablations at step 300, mean@4. The last column reports full ORSD minus each variant, in percentage points.

SettingScoreSuccess (%)ORSD success gain
Full ORSD, M=489.0277.050.00
Random selection, M=484.9469.90+7.15
Offline rubric85.3972.25+4.80
Simpler guidance, two selected items, M=488.0476.25+0.80

(2) The offline rubrics were generated using the actor checkpoint at step 300 and actor rollouts, with M=4 and reward-aligned selection, then held fixed during the offline comparison. These construction details are stated in Section 4.3 and Appendix C.4. The random-selection and offline comparisons appear in Table 2 of the revised paper.

[W2.2] The self-distillation baselines use published results rather than matched implementations. Can you provide matched comparisons and simpler guidance extracted from the same rollouts?

(1) We ran OPSD and GRPO+OPSD with Qwen3-4B on both environments using our experimental pipeline. Table R3 shows that ORSD outperforms standalone self-distillation and its combination with GRPO. The results appear in Table 1 of the revised paper, with the shared training configuration and added experiment protocols described in Appendix C, including Appendix C.4.

Table R3. Qwen3-4B success rates (%) at step 300, mean@4. All method results are from our experiments, and gains are in percentage points.

MethodWebShopALFWorld
OPSD4.2535.20
GRPO+OPSD69.9568.57
ORSD77.0577.43
ORSD gain over GRPO+OPSD+7.10+8.86

(2) For simpler guidance, we remove lower-alignment rubric items and retain the two highest-scoring criteria from the selected bundle. Each criterion is ranked by the Spearman correlation between its verifier scores across trajectories and the corresponding outcome rewards. As shown in Table R2, pruning provides no additional performance benefit, supporting the use of the complete selected bundle. This comparison appears in Figure 2(a) and Section 4.3 of the revised paper, with its construction described in Appendix C.4.

AuthorsOpenReview ↗

Response to Reviewer sn4a (2/2)

[W2.3] The absence of reported uncertainty across training seeds makes the smaller differences harder to assess.

All experiments reported here use a fixed training seed of 42. Due to the time constraints of the rebuttal period, we were unable to repeat the training runs with additional seeds. The observed improvements across models and environments provide empirical support for ORSD under this setting. These experiments do not quantify variability across training seeds, which remains a limitation of the current evaluation.

[Q3.1] How is bundle selection handled when all eight trajectories have the same reward, making Spearman correlation undefined?

When all trajectories in a group have identical training rewards, the reward vector has zero variance and Spearman correlation is undefined. Under our current online ORSD configuration, we skip rubric generation and bundle selection for that group and set all token-level corrections to zero. The group therefore falls back to standard GRPO without an ORSD correction. For binary rewards, this includes both all-zero and all-one groups. The criterion is within-group variation in training rewards, as specified in Section 3.2 of the revised paper.

[Q3.2] How often does this occur, particularly on ALFWorld, and is the rubric correction still applied to these groups?

Table R4 summarizes how often groups have identical rewards across the three policy models. Such groups occur frequently for all three models and are most common for Llama3.2-3B-Instruct. They receive no rubric correction, while groups with varying rewards proceed through alignment-based selection. The historical Qwen logs identify all-equal groups but do not distinguish all-zero from all-one groups, so these breakdowns are marked with dashes.

Table R4. Proportions of rollout groups by reward composition (%).

Policy modelAll-zero rewardsAll-one rewardsAll-equal rewardsMixed outcomes
Qwen3-4B ORSD——50.8349.17
Qwen3-1.7B ORSD——42.9257.08
Llama3.2-3B-Instruct ORSD17.0847.1364.2135.79

Table R5 separates the early and late stages of Llama training. All-equal groups become more frequent as all-success groups increase and all-failure groups decrease. Consequently, fewer groups trigger rubric correction late in training. All-equal groups bypass the rubric pipeline and receive no additional token correction.

Table R5. Reward-group composition during early and late Llama3.2-3B-Instruct ORSD training (%).

Training stageAll-zero rewardsAll-one rewardsAll-equal rewardsMixed outcomes
Steps 1–5060.502.0062.5037.50
Steps 251–3005.2575.7581.0019.00

Thank you for the rebuttal and additional experiments. However, my main concern about meaningful token-level credit assignment remains. While the new results demonstrate the benefits of token-specific corrections, they do not fully establish whether these corrections reflect the contributions of individual decisions to task success. Additionally, the lack of multi-seed evaluations leaves the reliability of smaller performance gains uncertain. Therefore, I will maintain my original rating.

AuthorsOpenReview ↗

Response to Reviewer sn4a

Thank you for clarifying your remaining concerns. We have supplemented the explanation of token-level corrections and conducted repeated evaluations to better characterize performance variability.

Token-level corrections and task success. In the illustrative WebShop example, positive corrections on phrases such as “short,” “button closure,” and “under 50” depict increased emphasis on concrete product requirements, while negative corrections on repeated speculative expressions such as “Alternatively, maybe…” depict reduced emphasis on alternatives that add no verified information. This illustrates how tokens within the same response can receive different adjustments to their shared trajectory-level advantage. The quantitative evidence comes from the trajectory-mean control, which preserves each trajectory’s average correction while removing token-level variation. ORSD improves success over this control by 3.85 percentage points on WebShop and 1.69 points on ALFWorld with Qwen3-4B. These results support the usefulness of token-specific allocation for improving task success.

Performance and reliability. Our main success-rate improvements are substantial across the three backbones: compared with GRPO, ORSD improves ALFWorld overall success by 12.12, 3.36, and 5.53 percentage points, and WebShop success by 7.60, 10.90, and 3.03 points, for Qwen3-1.7B, Qwen3-4B, and Llama3.2-3B, respectively. We have additionally completed three stochastic evaluations of fixed Qwen3-4B ORSD checkpoints, with the results below. These measurements show limited evaluation variability for the tested checkpoints. We will update the performance tables with the available measured means and standard deviations.

Qwen3-4B ORSD configurationMetricMean ± sample std
ALFWorldSuccess (%)77.50 ± 0.47
WebShop, DeepSeek verifierScore92.70 ± 0.30
WebShop, DeepSeek verifierSuccess (%)84.10 ± 0.30

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.