- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper identifies a decision-timestamp mismatch in privileged on-policy self-distillation for long-horizon agents. The same functional decision can occur at different turns across sibling rollouts, and one decision can span several turns within a rollout. The proposed ALIGNOPSD method first retrieves functionally matched teacher contexts using similarity between student and privileged thinking traces, then re-scores the same student response in those contexts. It next segments trajectories using changes in the cross- rollout correspondence profile and redistributes the trajectory-level GRPO advantage across spans and turns. Experiments use Qwen2.5-3B and Qwen2.5-7B on ALFWorld, Search-QA, and WebShop. The method improves over GRPO and StepOPSD on all eight reported aggregate comparisons and ranks first in six. WebShop ablations and additional diagnostics examine rectification, allocation granularity, hyperparameter sensitivity, temporal displacement, correspondence coverage, and learned span lengths.
Critical strengths
- The problem formulation is useful and the two-stage design is coherent. Separating cross-rollout context alignment from within-trajectory temporal credit exposes an assumption that step-local distillation methods usually leave implicit. Re-scoring the same sampled response under matched privileged contexts is also cleaner than imitating retrieved responses because the optimized behavior remains on-policy. For related-work positioning, the authors could discuss Ning Yang et al., Token-Importance Guided Direct Preference Optimization (ICLR 2026; https://openreview.net/forum?id=cMEnMVvMw9). Its semantic token-importance weighting offers a conceptual comparison for distributing optimization emphasis within a response, whereas ALIGNOPSD matches decisions across rollouts and allocates trajectory credit across temporal spans. The different granularity and optimization setting should be made explicit.
- The empirical scope and diagnostic effort are stronger than a single-benchmark demonstration. The paper covers three distinct agent settings, two model scales, competitive RLVR and self-distillation baselines, targeted ablations, and mechanism-level measurements. The consistent gains over GRPO and StepOPSD make the central idea worth taking seriously if the reliability and configuration issues below are resolved.
Critical weaknesses
- The main empirical claim lacks uncertainty and sufficient run-level information. Table 1 and the WebShop ablation report point estimates without the number of independent training seeds, across-seed variance, confidence intervals, or evaluation sample counts. Figure 3 explicitly says its bands are temporal variation rather than across-seed uncertainty. Several rank claims depend on differences below one percentage point, so the evidence does not establish that the reported ordering is robust to training and stochastic evaluation noise. The checkpoint-selection rule and whether hyperparameters were selected on the same validation histories also need to be reported precisely.
- The key correspondence surrogate is not directly validated, and the released configurations do not match the manuscript. The correspondence analysis shows that selected pairs have high embedding similarity, but that increase partly follows from selecting by the same score and does not measure semantic precision. Off-turn usage also measures frequency, not correctness. The paper gives no labeled match precision/recall and no boundary-level comparison for the learned spans. This matters because teacher- forcing a target response under another rollout history can penalize state-specific incompatibility rather than decision quality. In addition, the released CONFIGS.md uses profile-dependent similarity thresholds, top-K values, credit temperatures, action filters, and GRPO normalization; it also enables an SDAR auxiliary term for ALFWorld 3B. These settings conflict with Tables 6-8 and the statement that no auxiliary distillation loss is used, leaving it unclear which method produced the reported results.
Questions for authors
- How many independent training seeds and evaluation samples underlie each Table 1 and Table 2 entry? Please provide per-seed results and uncertainty intervals, and state the exact checkpoint-selection and hyperparameter-selection protocol. Would the six first-place rankings remain under paired bootstrap or across-seed comparisons?
- Please provide the exact resolved configuration and checkpoint hash for every reported run. In particular, explain why the released CONFIGS.md uses task/model-specific thresholds, top-K values, credit temperatures, action-filter settings, and an SDAR auxiliary term for ALFWorld 3B, while the paper reports shared settings and no auxiliary distillation loss.
- Can you evaluate the semantic correctness of retrieved cross-rollout matches on an independently labeled sample, stratified by task and turn displacement? Please also report how often the target response is valid in the matched source state and provide a boundary-level span metric rather than only aggregate mean lengths under forced 2-8 turn constraints.
- What is the added training cost of generating privileged thinking traces and teacher-forcing each target response under up to K source histories? Please report wall-clock time, GPU-hours, peak memory, and an equal-compute baseline or learning curve against compute.