- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes WAM-OPD to adapt pretrained World Action Models through on-policy distillation without additional environment interaction during distillation. Its core method, PWTR, reweights histories in a fixed trajectory pool using denoising-path compatibility scores. The results show promising target-task improvements with good skill retention.
Critical strengths
The paper addresses a practically important problem. The distinction between on-policy denoising states and off-policy environment histories is well motivated, and the simulation and real-robot results are encouraging.
Critical weaknesses
- Insufficient validation of PWTR as a history-distribution correction mechanism.
The ideal importance ratio in Eq. (8) depends on executed-action probabilities, whereas PWTR uses denoising-path compatibility scores from Eq. (3). These quantities are not necessarily equivalent, and temporal subsampling introduces further approximation. The paper lacks direct evidence that PWTR weights meaningfully approximate the evolving on-policy history distribution.
- Insufficient evidence for the independent contribution of history-specific weighting.
In Table 2, PWTR improves mixed-pool performance from 67.25% to 69.75% over uniform weighting. However, M-Shuffle does not preserve task- and trajectory-source-level weight statistics, making it difficult to isolate the benefit of history-specific weighting. Moreover, skill retention is not reported for these ablations.
Questions for authors
- Can the authors validate PWTR weights against approximate history importance ratios or online OPD update directions?
- Can the authors provide group-statistics-preserving weighting ablations and report both target-task performance and skill retention?
2 comments
thank you
Dear Reviewer tRHw,
Following constructive discussions during the rebuttal period, we wanted to briefly follow up to let you know that we have uploaded the revised manuscript, incorporating all the diagnostics and ablations prompted by your feedback alongside a broader framing of our work:
- Per-decision path-score vs. log-density difference & online-gradient diagnostics (W1, Q1): Detailed in Appendix A.5 (Tables 4 and 6) and referenced in Section 4.3, together with the decision-point subsampling analysis () in Appendix D.8 (Table 14).
- Group-preserving shuffle (
G-Shuffle) & skill retention (W2, Q2): Added in Appendix A.5 (Table 5) (kept separate from Table 2 for layout clarity and cross-referenced in Section 4.3), confirming PWTR's gains in both target adaptation (5.25 percentage points over G-Shuffle and 2.50 percentage points over unit weights) and 6-task skill retention (2.25 percentage points over G-Shuffle and 2.42 percentage points over unit weights) under matched group-level weight statistics. - Broader framing across flow-based policies: In light of reviewer discussions on the generality of our action-level OPD formulation (validated on both Fast-WAM and LingBot-VLA in Section 4.5), we have updated the title to "RoboOPD: On-Policy Distillation for Flow-Based Policies in Robotic Manipulation" and unified the presentation around flow-based policies throughout the manuscript.
We sincerely appreciate your insightful questions, which substantially strengthened the empirical validation of PWTR. Please let us know if you have any further questions on the revised manuscript or the new results, and we would be happy to discuss further.
Response to Reviewer [tRHw]
We thank the reviewer for the suggestions on direct validation. In response, we have added history-ratio and online-gradient diagnostics, a group-preserving shuffle control, and skill retention evaluations.
W1, Q1: History Ratios and Online Gradient Alignment
We have conducted both requested validations. PWTR weights correlate positively with a separately estimated history-ratio proxy. Among the three weighting schemes, PWTR also achieves the highest mean alignment with online OPD gradients.
1. Fresh student samples at stored histories. At each stored history from the fixed pool, the current student starts from fresh noise and generates an action-denoising path. Conditional action sampling therefore follows the current student at every update, with the teacher supplying velocity targets at query states along each path. As the student changes, the histories it would encounter in the environment may follow a different distribution from those in the fixed pool. PWTR approximately corrects this mismatch by reweighting the distillation loss at each stored history. Together, these steps approximate online OPD without collecting fresh environment rollouts during distillation.
2. History-ratio diagnostic. At updates 24, 72, and 96, we sampled fresh action chunks from the student checkpoint, initial student, and teacher at each stored decision. We fitted Gaussian densities to the executed-action coordinates and accumulated the resulting estimated log-density differences along each history, yielding a separate proxy for the history log-ratio in Eq. (8).
Within-group Spearman correlations between PWTR weights and the history-level proxy are 0.42, 0.39, and 0.36. Per-decision correlations between Eq. (3) scores and estimated log-density differences are 0.27, 0.62, and 0.63. Together, these comparisons assess the surrogate at both the decision and history levels.
3. Online-gradient alignment. On paired replay and online batches, mean cosine similarity with the online OPD gradient is 0.101 for PWTR, compared with 0.084 for unit weighting and 0.080 for G-Shuffle (defined below).
4. Decision-point subsampling. Using 8, 16, or all decision points yields target SRs of 69.00%, 69.75%, and 69.75%, respectively. Using all points removes prefix-sum subsampling and achieves the same SR as the default 16 points while requiring approximately 2.4× as many queries.
Manuscript revision. We will add the ratio and gradient analyses with their protocols to Sec. 4.3 and Appendix A, and the subsampling comparison to Appendix D.
W2, Q2: Group-Preserving Shuffle and Skill Retention
We have added the requested group-preserving shuffle control and retention evaluation. PWTR exceeds G-Shuffle by 5.25 points in target SR and 2.25 points in retention. These gains support the value of matching weights to the histories for which they were computed, beyond preserving group statistics.
1. Group-preserving shuffle protocol. The original M-Shuffle permutes weights among decision points at the same sampled position. G-Shuffle permutes weights within groups defined by task × collection condition × rollout source × normalized prefix depth (five bins). This preserves each group's weight distribution while changing the assignment of weights to histories.
2. Target performance and skill retention. We evaluated all three weighting schemes on the two target tasks and six retention tasks:
Retention SR is the equally weighted mean success rate on Switch, Mug, Laptop, Pillbottle, Dual Bottles, and Handover, all excluded from adaptation.
PWTR also exceeds unit weighting by 2.50 points in target SR and 2.42 points in retention. These comparisons support the contribution of history-specific weighting to both adaptation and skill retention.
Manuscript revision. We will add G-Shuffle and retention results to Table 2 and document the grouping and shuffling protocol.