acceptodds
Withdrawn from ICLR 2027

Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning with verifiable rewards (RLVR) often relies on sparse outcome rewards, providing coarse supervision for long-horizon agents. On-policy self-distillation (OPSD) complements this signal with dense privileged feedback. However, we identify Decision–Timestamp Mismatch: privileged guidance may be misaligned with the student's functional decision because the corresponding decision can occur at a different timestep, while the student's decision itself may span multiple timesteps rather than being tied to a single timestamp. Thus, timestamp-local supervision can misalign both the context and the temporal scope of credit. To address this mismatch, we introduce AlignOPSD, following the principle of aligning supervision before assigning credit. Decision-Aligned Supervision Rectification re-scores the same student-sampled response in functionally matched contexts across sibling rollouts to calibrate local teacher evidence. Semi-Markov Hierarchical Credit Assignment then derives variable-duration decision spans from correspondence changes and uses rectified evidence to allocate outcome-grounded credit across spans and their constituent turns. We evaluate AlignOPSD with Qwen2.5-3B and Qwen2.5-7B on ALFWorld, WebShop, and Search-QA against representative baselines. AlignOPSD outperforms both GRPO and StepOPSD across all eight backbone–aggregate-metric comparisons, improving on GRPO by 5.5–8.7 % and ranking first in six. Additional analyzes examine the two alignment stages and hyperparameter sensitivity between tasks.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Agentic RL, Agent, OPD

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_xByLOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies a decision-timestamp mismatch in privileged on-policy self-distillation for long-horizon agents. The same functional decision can occur at different turns across sibling rollouts, and one decision can span several turns within a rollout. The proposed ALIGNOPSD method first retrieves functionally matched teacher contexts using similarity between student and privileged thinking traces, then re-scores the same student response in those contexts. It next segments trajectories using changes in the cross- rollout correspondence profile and redistributes the trajectory-level GRPO advantage across spans and turns. Experiments use Qwen2.5-3B and Qwen2.5-7B on ALFWorld, Search-QA, and WebShop. The method improves over GRPO and StepOPSD on all eight reported aggregate comparisons and ranks first in six. WebShop ablations and additional diagnostics examine rectification, allocation granularity, hyperparameter sensitivity, temporal displacement, correspondence coverage, and learned span lengths.

Critical strengths
  1. The problem formulation is useful and the two-stage design is coherent. Separating cross-rollout context alignment from within-trajectory temporal credit exposes an assumption that step-local distillation methods usually leave implicit. Re-scoring the same sampled response under matched privileged contexts is also cleaner than imitating retrieved responses because the optimized behavior remains on-policy. For related-work positioning, the authors could discuss Ning Yang et al., Token-Importance Guided Direct Preference Optimization (ICLR 2026; https://openreview.net/forum?id=cMEnMVvMw9). Its semantic token-importance weighting offers a conceptual comparison for distributing optimization emphasis within a response, whereas ALIGNOPSD matches decisions across rollouts and allocates trajectory credit across temporal spans. The different granularity and optimization setting should be made explicit.
  2. The empirical scope and diagnostic effort are stronger than a single-benchmark demonstration. The paper covers three distinct agent settings, two model scales, competitive RLVR and self-distillation baselines, targeted ablations, and mechanism-level measurements. The consistent gains over GRPO and StepOPSD make the central idea worth taking seriously if the reliability and configuration issues below are resolved.
Critical weaknesses
  1. The main empirical claim lacks uncertainty and sufficient run-level information. Table 1 and the WebShop ablation report point estimates without the number of independent training seeds, across-seed variance, confidence intervals, or evaluation sample counts. Figure 3 explicitly says its bands are temporal variation rather than across-seed uncertainty. Several rank claims depend on differences below one percentage point, so the evidence does not establish that the reported ordering is robust to training and stochastic evaluation noise. The checkpoint-selection rule and whether hyperparameters were selected on the same validation histories also need to be reported precisely.
  2. The key correspondence surrogate is not directly validated, and the released configurations do not match the manuscript. The correspondence analysis shows that selected pairs have high embedding similarity, but that increase partly follows from selecting by the same score and does not measure semantic precision. Off-turn usage also measures frequency, not correctness. The paper gives no labeled match precision/recall and no boundary-level comparison for the learned spans. This matters because teacher- forcing a target response under another rollout history can penalize state-specific incompatibility rather than decision quality. In addition, the released CONFIGS.md uses profile-dependent similarity thresholds, top-K values, credit temperatures, action filters, and GRPO normalization; it also enables an SDAR auxiliary term for ALFWorld 3B. These settings conflict with Tables 6-8 and the statement that no auxiliary distillation loss is used, leaving it unclear which method produced the reported results.
Questions for authors
  1. How many independent training seeds and evaluation samples underlie each Table 1 and Table 2 entry? Please provide per-seed results and uncertainty intervals, and state the exact checkpoint-selection and hyperparameter-selection protocol. Would the six first-place rankings remain under paired bootstrap or across-seed comparisons?
  2. Please provide the exact resolved configuration and checkpoint hash for every reported run. In particular, explain why the released CONFIGS.md uses task/model-specific thresholds, top-K values, credit temperatures, action-filter settings, and an SDAR auxiliary term for ALFWorld 3B, while the paper reports shared settings and no auxiliary distillation loss.
  3. Can you evaluate the semantic correctness of retrieved cross-rollout matches on an independently labeled sample, stratified by task and turn displacement? Please also report how often the target response is valid in the matched source state and provide a boundary-level span metric rather than only aggregate mean lengths under forced 2-8 turn constraints.
  4. What is the added training cost of generating privileged thinking traces and teacher-forcing each target response under up to K source histories? Please report wall-clock time, GPU-hours, peak memory, and an equal-compute baseline or learning curve against compute.
Reviewer_WqBQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AlignOPSD to address the mismatch between functional decisions and timestamps in agent self-distillation. The method uses thinking-trace similarity to re-score student responses under matched contexts from other trajectories of the same task. It then constructs variable-duration spans for hierarchical credit assignment. Experiments with Qwen2.5-3B/7B on ALFWorld, WebShop, and Search-QA show improvements over both GRPO and StepOPSD, with further empirical support from ablation studies and training diagnostics.

Critical strengths
  1. The combination of supervision rectification and hierarchical credit assignment has a clear motivation. The method first aligns supervisory evidence, then selects credit units and re-scores the same student response. The design is concrete, incorporating fallback to the original context, stop-gradient weights, and preservation of the token-weighted mean advantage. This provides a useful direction for exploiting privileged feedback in agent training.
  2. The experiments provide initial evidence across tasks and model scales. Table 1 covers three environments, two backbones, and several reinforcement learning and self-distillation baselines, showing improvements over GRPO and StepOPSD across all eight aggregate comparisons. Appendix F.1 states that data, rollout budgets, and optimization updates are matched across methods. In the WebShop 7B ablation, removing supervision rectification reduces Score/Acc from 87.9/78.9 to 84.2/71.9 (Table 2), supporting the claim that this component makes a meaningful contribution.
Critical weaknesses
  1. The motivation for cross-history rectification is only partially supported. Temporal displacement across independent rollouts does not directly establish a limitation of OPSD scoring under the student’s own history. Although the matching and confidence controls are sensible, it remains unclear whether cross-history rescoring provides more accurate decision-level supervision, rather than reflecting differences between the underlying contexts.

  2. The turn-level and random-span ablations are useful, but do not establish that the gains depend on identifying functional decision boundaries. The random spans do not preserve the adaptive spans’ length distribution, while Appendix D.3 compares aggregate lengths rather than boundary correspondence. A comparison with GEAR would also help establish the contribution of correspondence-driven segmentation.

  3. Improvements over GRPO are encouraging, but the reported gains over stronger baselines are often small—only 0.1–0.8 percentage points across the four aggregate metrics for the 7B model. Without variability estimates and measured training-cost comparisons, the robustness and practical significance of these improvements remain uncertain.

Questions for authors
  1. Can you provide direct evidence that cross-history rectification reflects the quality of the current decision more accurately than OPSD scoring under the student’s own history?
  2. Does adaptive segmentation retain an advantage when random spans follow the same length distribution as the adaptive spans?
  3. How much improvement does correspondence-driven segmentation provide over GEAR?
  4. Are the small improvements over strong baselines stable across different training seeds?
  5. How much additional wall-clock training time, GPU-hours, and peak GPU memory does the method require relative to the strongest baseline?
Reviewer_98RNOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes ALIGNOPSD, a method for improving credit assignment in long-horizon agent RL. The authors identify a decision–timestamp mismatch problem, where similar decisions may occur at different turns across rollouts, and a single decision may span multiple turns. To address this, ALIGNOPSD matches decisions across sibling rollouts using thinking embeddings, re-scores student responses under matched privileged contexts, and redistributes trajectory-level advantages through hierarchical credit assignment. Experiments on ALFWorld, WebShop, and Search-QA show consistent improvements over GRPO and existing self-distillation methods.

Critical strengths
  • The paper identifies an interesting problem in long-horizon agent RL, where functional decisions do not necessarily align with individual timestamps. I think this is a reasonable motivation, and the proposed cross-rollout alignment provides an interesting perspective on credit assignment beyond individual turns.
  • The experiments cover three agent benchmarks, two model sizes, and a relatively large number of baselines. The consistent improvements over GRPO and StepOPSD provide some evidence that the proposed method is effective, which is the main reason I consider this work potentially useful.
Critical weaknesses
  • I am not fully convinced that matching similar decisions across different rollout histories provides more reliable teacher supervision. Even when thinking and actions are similar, the underlying environment states and constraints may be different. It is unclear why evaluating a student response under another rollout's privileged context should be better than using its original context. The current analysis mainly shows embedding similarity and temporal displacement, but does not directly validate the correctness of these matches. Since this is the core motivation of ALIGNOPSD, I think stronger evidence is needed to support the proposed method.
  • The empirical improvements are relatively small compared with the complexity of the method. In Table 1, ALIGNOPSD improves over StepOPSD by only 0.7–0.9 percentage points on the four aggregate metrics with Qwen2.5-7B. However, the paper does not report multi-seed results or standard deviations. Moreover, ALIGNOPSD introduces additional teacher evaluations and embedding computations, but the comparison only matches optimization steps rather than total compute. The source and construction of the privileged SkillBank are also not sufficiently explained, making it difficult to assess the fairness and reproducibility of the experiments. Overall, the current results do not fully justify the additional complexity of the proposed method.
Questions for authors

Here are my main questions and concerns. I would be happy to change my opinion if the major concerns are resolved.

  • Can the authors provide stronger evidence that cross-rollout matching improves the quality of teacher supervision? Thinking similarity does not necessarily imply functional equivalence under different environment states. I would like to see a direct evaluation of matching accuracy. An ablation comparing the proposed method with random cross-rollout matching or action-only matching, while keeping the number of teacher evaluations the same, would also be helpful. This is important to verify whether the improvements actually come from the proposed alignment mechanism.
  • Can the authors report multi-seed results and compute-matched comparisons? Since the improvements over StepOPSD on the 7B model are all below one percentage point, reporting means and standard deviations across multiple independent runs is important. Also, how much additional training compute does ALIGNOPSD require compared with StepOPSD? I think a comparison under matched GPU-hour budgets would help determine whether the proposed method is practically useful.
  • How is the privileged SkillBank constructed for each benchmark? Are the skills generated from expert demonstrations, successful trajectories, or external knowledge? Does the construction involve any information from evaluation tasks? I think the authors should clarify the source of these skills, provide concrete examples, and explain which baselines use the same privileged information. This is important for understanding the contribution of the proposed method and the reproducibility of the experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.