- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose PTR, a way of reweighting the per-example action loss when fine-tuning a flow-matching VLA. The idea is to use the future observations that are already stored with the demonstrations. A small scorer gets the policy context and the action-head features for a chunk and has to pick out that chunk’s own future frame from a list of futures belonging to other samples. How well it does, relative to guessing, is turned into a weight between 0.25 and 4 on the action loss. The weight is detached and normalized within the batch. The same scorer is also trained as an auxiliary contrastive objective, together with a rank loss, and the full system includes a memory module and a controller that adjusts temperatures and hard negatives. There is some analysis relating the expected score to the information in the candidate list and bounding how much the weights can shift supervision. Experiments are on Being-H0.5 with LIBERO, RoboCasa and twelve real-robot tasks on three different robots, under pooled training, training on corrupted demonstrations, and clean single-benchmark training. The main claims are a 13.8-point improvement on the pooled real-robot tasks and much smaller drops than SFT when the training data is corrupted.
Critical strengths
- The ablation in Table 2 is what convinced me. The authors train a version that keeps the full scorer, including its gradients into the shared representation, but sets every action weight to one. PTR still beats this version on RoboCasa in all three task categories, and the result holds over three seeds. So the gain is not just from the extra contrastive loss; the weights themselves are doing something. The corruption study points the same way. The scorer is never shown which trajectories were corrupted, but the corrupted chunks end up with clearly lower weights than the clean ones.
- The real-robot evaluation is good. Twelve tasks on three quite different robots, and in pooled training all twelve improve, with gains on each platform rather than one robot carrying the result. Pooled training over mixed embodiments is the setting the method is meant for, so this matters to me.
Critical weaknesses
-
The real-robot comparison does not tell me what the weighting contributes. The 50.0% to 63.8% result compares plain SFT with the full method, but the full method also includes the memory module and the auxiliary losses. In simulation the authors do break these apart (Table 2), and there the weights are responsible for only part of the improvement, and on LIBERO a small part. No such breakdown exists for the robots. There is no SFT+Memory run and no run with the weights fixed to one, so some or most of the 13.8 points could be coming from memory or from the auxiliary losses rather than from the reweighting. Also, one policy per method and 20 trials per task, so a 10 to 15 point gap on a task is 2 or 3 trials. That’s thin for the headline number in the abstract. A real-robot run with the weights fixed to 1 is needed to show how much of it is the weighting.
-
I am not convinced that the score reflects the action rather than the scene. The scorer is given the full context along with the action-head features, and in manipulation data the correct future frame can usually be picked out from the context alone: it shows the same table, the same objects and the same camera as the current observation. The authors acknowledge this in Sec. 3.2 and again in Appendix A.1, where they say that candidates sharing a context or camera can provide identification cues without isolating the action. The rank loss is meant to push the scorer toward the action features, but the only evidence is Table 4c, where removing it costs a little. That shows the rank loss helps, not how much of the score actually comes from the action. The direct test is simple: compute the weights with a scorer that sees only the context. If that version performs about as well as PTR, the method is reweighting by how recognizable the scene is, and the “learning from consequences” story would not hold up.
Questions for authors
- What do you get if the weights come from a scorer that only reads the context, with no action-head features? This is the one that would move my score, either way.
- Can you add SFT+Memory and “Full scorer, w = 1” on the real robots, even on a subset of the tasks? I realize hardware runs are expensive, but without them I can’t tell what the weighting contributes to the 13.8 points.
- In Table 4a, dropping the clip, the controller or the memory each puts clean RoboCasa below SFT. How sensitive is the method to beta, the clip range, lambda_id, lambda_rank and the controller thresholds? I’m trying to get a sense of how much tuning a new dataset would need.
- For trajectory truncation, the kept prefix still has valid action-future pairs (Appendix C.4), and chunks with no future keep a weight of one by construction. So what is lowering the weights of the truncated trajectories? Per-type tagged vs untagged weight statistics would help here, and a plot of raw weight against episode phase on clean data would show more generally which chunks get down-weighted.
- Everything is on Being-H0.5, but the conclusion says the mechanism applies to any BC objective with a per-example loss. Have you tried a second base policy? If not, I’d soften that claim. This won’t change my score either way.
- Related work should mention weighted BC from noisy demonstrations, e.g. Sasaki & Yamashina (ICLR 2021) and DWBC (Xu et al., ICML 2022). And since Proposition 1 is essentially the standard contrastive / density-ratio result from Oord et al. and Poole et al., which the authors note themselves, I would call it supporting analysis rather than a main contribution.
Summary of Rebuttal
Dear AC and Reviewers,
As the discussion period nears its end, we would like to provide a brief summary of our responses and the revised manuscript.
All three reviews are positive, highlighting the reward-free, likelihood-free formulation, carefully scoped analysis, and broad validation across simulation and twelve real-robot tasks. In our rebuttal and revision, we addressed the questions raised by each reviewer:
As robot post-training increasingly relies on imperfect demonstrations without rewards or tractable action likelihoods, we believe turning already-recorded future observations into a practical training signal can be useful to the community. We are grateful to the reviewers for their constructive feedback in helping us improve this paper, and we sincerely hope to have the opportunity to share and discuss this work with the ICLR community.