acceptodds
Desk rejected at ICLR 2027

Learning from Consequences: Posterior-Transition Reweighting for Behavior Cloning

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Robot demonstrations contain more than action labels: they also record the observations that follow each action. We investigate whether these future observations can guide how action supervision is allocated during vision-language-action (VLA) post-training. Posterior-Transition Reweighting (PTR) identifies the observed future of each demonstrated action chunk among alternatives drawn from the training data. The same identification model provides an auxiliary representation-learning objective and a bounded, stop-gradient weight on the original action loss. This separates two uses of future information: learning features and weighting supervision. PTR requires neither reward labels nor an evaluable action likelihood and applies directly to flow-matching regression. Our analysis shows that the expected score equals the identification information carried by the candidate list minus the scorer's error, and that the bounded weights redistribute supervision without discarding any example. Experiments on LIBERO, RoboCasa, and twelve real-robot tasks examine heterogeneous pooled training and controlled corruption of demonstrations. In pooled training, PTR raises real-robot success from 50.0% to 63.8% over standard fine-tuning, with gains on all twelve tasks. Controls in pooled simulation training show that reweighting contributes beyond memory and auxiliary representation learning. These results support future observations as a practical source of additional supervision for VLA adaptation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Model, Robotics

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_onaQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose PTR, a way of reweighting the per-example action loss when fine-tuning a flow-matching VLA. The idea is to use the future observations that are already stored with the demonstrations. A small scorer gets the policy context and the action-head features for a chunk and has to pick out that chunk’s own future frame from a list of futures belonging to other samples. How well it does, relative to guessing, is turned into a weight between 0.25 and 4 on the action loss. The weight is detached and normalized within the batch. The same scorer is also trained as an auxiliary contrastive objective, together with a rank loss, and the full system includes a memory module and a controller that adjusts temperatures and hard negatives. There is some analysis relating the expected score to the information in the candidate list and bounding how much the weights can shift supervision. Experiments are on Being-H0.5 with LIBERO, RoboCasa and twelve real-robot tasks on three different robots, under pooled training, training on corrupted demonstrations, and clean single-benchmark training. The main claims are a 13.8-point improvement on the pooled real-robot tasks and much smaller drops than SFT when the training data is corrupted.

Critical strengths
  1. The ablation in Table 2 is what convinced me. The authors train a version that keeps the full scorer, including its gradients into the shared representation, but sets every action weight to one. PTR still beats this version on RoboCasa in all three task categories, and the result holds over three seeds. So the gain is not just from the extra contrastive loss; the weights themselves are doing something. The corruption study points the same way. The scorer is never shown which trajectories were corrupted, but the corrupted chunks end up with clearly lower weights than the clean ones.
  2. The real-robot evaluation is good. Twelve tasks on three quite different robots, and in pooled training all twelve improve, with gains on each platform rather than one robot carrying the result. Pooled training over mixed embodiments is the setting the method is meant for, so this matters to me.
Critical weaknesses
  1. The real-robot comparison does not tell me what the weighting contributes. The 50.0% to 63.8% result compares plain SFT with the full method, but the full method also includes the memory module and the auxiliary losses. In simulation the authors do break these apart (Table 2), and there the weights are responsible for only part of the improvement, and on LIBERO a small part. No such breakdown exists for the robots. There is no SFT+Memory run and no run with the weights fixed to one, so some or most of the 13.8 points could be coming from memory or from the auxiliary losses rather than from the reweighting. Also, one policy per method and 20 trials per task, so a 10 to 15 point gap on a task is 2 or 3 trials. That’s thin for the headline number in the abstract. A real-robot run with the weights fixed to 1 is needed to show how much of it is the weighting.

  2. I am not convinced that the score reflects the action rather than the scene. The scorer is given the full context along with the action-head features, and in manipulation data the correct future frame can usually be picked out from the context alone: it shows the same table, the same objects and the same camera as the current observation. The authors acknowledge this in Sec. 3.2 and again in Appendix A.1, where they say that candidates sharing a context or camera can provide identification cues without isolating the action. The rank loss is meant to push the scorer toward the action features, but the only evidence is Table 4c, where removing it costs a little. That shows the rank loss helps, not how much of the score actually comes from the action. The direct test is simple: compute the weights with a scorer that sees only the context. If that version performs about as well as PTR, the method is reweighting by how recognizable the scene is, and the “learning from consequences” story would not hold up.

Questions for authors
  1. What do you get if the weights come from a scorer that only reads the context, with no action-head features? This is the one that would move my score, either way.
  2. Can you add SFT+Memory and “Full scorer, w = 1” on the real robots, even on a subset of the tasks? I realize hardware runs are expensive, but without them I can’t tell what the weighting contributes to the 13.8 points.
  3. In Table 4a, dropping the clip, the controller or the memory each puts clean RoboCasa below SFT. How sensitive is the method to beta, the clip range, lambda_id, lambda_rank and the controller thresholds? I’m trying to get a sense of how much tuning a new dataset would need.
  4. For trajectory truncation, the kept prefix still has valid action-future pairs (Appendix C.4), and chunks with no future keep a weight of one by construction. So what is lowering the weights of the truncated trajectories? Per-type tagged vs untagged weight statistics would help here, and a plot of raw weight against episode phase on clean data would show more generally which chunks get down-weighted.
  5. Everything is on Being-H0.5, but the conclusion says the mechanism applies to any BC objective with a per-example loss. Have you tried a second base policy? If not, I’d soften that claim. This won’t change my score either way.
  6. Related work should mention weighted BC from noisy demonstrations, e.g. Sasaki & Yamashina (ICLR 2021) and DWBC (Xu et al., ICML 2022). And since Proposition 1 is essentially the standard contrastive / density-ratio result from Oord et al. and Poole et al., which the authors note themselves, I would call it supporting analysis rather than a main contribution.
Reviewer_mBVrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Posterior-Transition Reweighting (PTR), a supervised post-training method that uses recorded future observations to weight action supervision in flow-matching vision-language-action policies. A jointly trained scorer identifies the recorded future among alternative observations using policy context and action-head features. Its identification score becomes a bounded, stop-gradient coefficient on the original action loss. Auxiliary identification and ranking losses train shared representations, and a recurrent memory path supplies temporal context.

The analysis characterizes the identification score at a frozen training state and derives properties of the normalized supervision coefficients and induced regression objective. Evaluation covers LIBERO, RoboCasa, and twelve real-robot tasks across three platforms. PTR improves performance particularly in pooled and corrupted-data settings. A pooled-simulation control that retains memory and auxiliary losses but fixes action weights to 1 provides evidence that learned weighting contributes beyond those components.

Critical strengths
  1. The paper directly tests whether weighting adds value beyond auxiliary representation learning. The "Full scorer, w=1w=1" control retains memory, EMA targets, candidate construction, and identification/ranking gradients while disabling learned action weighting. Enabling weighting improves pooled RoboCasa success from 52.0±0.7%52.0 \pm 0.7\% to 54.9±0.6%54.9 \pm 0.6\%, and LIBERO from 96.6±0.3%96.6 \pm 0.3\% to 97.2±0.3%97.2 \pm 0.3\% (Table 2). This is the most consequential evidence for the central contribution, especially on RoboCasa. It isolates the weighting intervention across complete training runs, although the learned representations can subsequently diverge.

  2. The method combines a carefully scoped formulation with broad practical validation. PTR requires neither rewards nor action-likelihood evaluation and preserves the original per-example action target. The theory distinguishes identification evidence from task value and does not claim a general policy-improvement guarantee. Experiments span two simulation benchmarks and three robot platforms. In pooled hardware evaluation, successes increase from 120/240120/240 to 153/240153/240, with positive observed differences on all twelve tasks. The reported training overhead is 11.8%11.8\% relative to SFT, and deployment retains only the memory path, matching the inference cost of SFT+Memory. Hardware evidence is nevertheless limited to one trained policy per method in each regime.

Critical weaknesses
  1. I see that the source of the larger robustness and hardware gains remains incompletely isolated. The strongest matched control, “Full scorer, w=1w=1,” is reported only for pooled simulation. Consequently, the improvements in corrupted data and real robot cannot be attributed specifically to learned weighting, but rather to the combined memory, auxiliary learning, and weighting system. The additional curation, Hard-PTR, and component comparisons are useful but do not replace this control.

The weighting signal also admits explanations beyond those related to demonstration reliability. Because eie_i comes from the action head evaluated at Xi=τiAi+(1−τi)ϵiX_i=\tau_i A_i+(1-\tau_i)\epsilon_i, scores may depend on flow time and noise as well as action-future consistency. A systematic preference for larger τi\tau_i is plausible but not demonstrated. Appendix A.3 correctly explains that such dependence alone does not establish a change in the regression target; the remaining concern is the empirical interpretation.

Trajectory truncation makes this attribution question particularly relevant. The paper acknowledges that retained-prefix transitions remain valid, yet reports improved robustness to truncation. The Combined-corruption raw-weight averages do not establish which corruption types drive the association, or how much normalized action-loss mass each receives. These limitations weaken the mechanistic explanation without invalidating the reported performance gains.

  1. Additionally, several essential implementation choices are explicitly left unresolved. Appendices A.3 and B.5 do not identify whether reported runs normalize action weights per device or globally. These choices produce different coefficients:

Γiglobal=Bgwi∑r=1Bgwr\Gamma_i^{\mathrm{global}} = \frac{B_g w_i}{\sum_{r=1}^{B_g} w_r}

Γilocal=nwi∑r∈Ggwr\Gamma_i^{\mathrm{local}} = \frac{n w_i}{\sum_{r \in \mathcal{G}_g} w_r}

Here, BgB_g is the global batch size, nn is the per-device batch size, and Gg\mathcal{G}_g denotes the examples on device gg. Cross-GPU candidate gathering does not determine which normalization was used. The appendix also leaves unspecified the exact ρ\rho schedule, controller-event frequency, future-offset probabilities, and operational definition and retrieval procedure for hard negatives.

These choices directly affect the identification problem and effective supervision coefficients. Their omission is therefore more consequential than a minor documentation issue. The authors should provide the actual configuration used for each reported experimental block.

Questions for authors
  1. Which normalization scope was used in the reported runs? Please also specify the exact weighting schedule, controller update frequency, future-offset sampling probabilities, and hard-negative selection procedure. Were these settings shared across regimes?

  2. Are existing "Full scorer, w=1w=1" or SFT+Memory results available for corrupted-data or hardware settings? If not, please clarify that the gains in those settings establish the benefit of the complete PTR system rather than isolate the contribution of learned weighting. I am not requesting a new hardware study during rebuttal.

  3. Do existing logs show how TiT_i, raw wiw_i, or normalized BαiB\alpha_i vary with τi\tau_i? A stratification by flow time would help distinguish demonstration-dependent weighting from a change in the allocation of training effort across noise levels. This question concerns the empirical mechanism, not a claimed violation of Appendix A.3.

  4. If available from existing runs, please provide tagged-versus-untagged normalized weight summaries separately for ANI, TT, and LN. In particular, what evidence links the truncation robustness gain to learned weighting, given that valid retained transitions are preserved, and missing-future examples receive raw wi=1w_i=1, rather than necessarily uniform normalized coefficients?

Reviewer_ZYHYOpenReview ↗
Rating
3
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

In this work, the authors present PTR, a way to reweight BC datasets by the predictability of the future states. The hypothesis is that observations that are more predictable are more intelligible, or at least has more intention behind it, and therefore each transition that is more predictable gets higher weight next epoch. The authors create an algorithm for re-weighting around it, and validate it in some simulated and real robot experiments.

Critical strengths

The high level idea of PTR is interesting, where predictability of a demonstration also matters in how much we weigh that trajectory or transition. The way the prediction is done is via contrastive classification is also neat, although it brings up some question about sampling hard negatives which has always been a hard problem in any kind of contrastive learning. Finally, the experiments seem thorough on a diverse set of tasks and datasets.

Critical weaknesses

Primarily, the introduction is not clear on whether the predictable outcomes are upweighted or downweighted, I can see arguments for both since surprising transitions may teach us more but maybe we don't want to copy those actions as well. The primary weakness of this paper is mix in a very complex VLA training setup when the idea itself is much simpler and can be tested with very standard BC algorithms instead. Also weak is not comparing the method with other dataset reweighting, for example a method that density-normalizes the dataset before training by features instead of using transition predictability.

Questions for authors
  1. What happens if we apply PTR to a standard training model, like diffusion policy?
  2. How were the reweighting scheme coefficients decided? What happens if a simpler weight/schedule is used?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

Dear AC and Reviewers,

As the discussion period nears its end, we would like to provide a brief summary of our responses and the revised manuscript.

All three reviews are positive, highlighting the reward-free, likelihood-free formulation, carefully scoped analysis, and broad validation across simulation and twelve real-robot tasks. In our rebuttal and revision, we addressed the questions raised by each reviewer:

  • Reviewer mBVr: We specified all run-level implementation details, added matched Full scorer (w=1) controls in corrupted simulation and on physical robots, and provided flow-time τi\tau_i and per-corruption weight analyses. Reviewer mBVr confirmed that all points were answered and maintained a positive evaluation.
  • Reviewer onaQ: We added the context-only scorer control, real-robot controls (SFT+Memory and w=1) on Adam-U and FR3, hyperparameter sensitivity checks, and phase analyses. Reviewer onaQ confirmed that main concerns were resolved and noted that the Diffusion Policy and π0.5\pi_{0.5} results support generality; we also updated the paper per their suggestion to clarify the full PTR system vs. supervised fine-tuning (SFT).
  • Reviewer ZYHY: We evaluated PTR on standalone Diffusion Policy and π0.5\pi_{0.5} without Being-H0.5 memory (including a controller-free fixed-scale recipe), compared against offline Feature-Density reweighting, and clarified the weighting direction upfront in Section 1.

As robot post-training increasingly relies on imperfect demonstrations without rewards or tractable action likelihoods, we believe turning already-recorded future observations into a practical training signal can be useful to the community. We are grateful to the reviewers for their constructive feedback in helping us improve this paper, and we sincerely hope to have the opportunity to share and discuss this work with the ICLR community.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.