acceptodds
Withdrawn from ICLR 2027

Rethinking Where Robot Actions Begin: Learning an Expert-Inversion Prior

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Flow-matching robot policies have demonstrated strong performance, yet typically sample starting points from a fixed Gaussian prior without explicitly leveraging source information from expert demonstrations. We find that expert-inverted source distributions can deviate from this prior even with numerically accurate inversion, and that source–observation correspondence affects generated actions. Our theoretical analysis establishes that, under ideal invertibility, matching the expert conditional source distribution is equivalent to matching the expert conditional action distribution. These findings motivate Expert-Inversion Prior Flow, a two-stage framework in which a conditional prior flow learns to transport Gaussian noise to observation-matched sources, while a frozen action flow decodes these sources into action sequences. Systematic experiments demonstrate improved agreement with expert actions over unconditional source modeling, supporting the value of observation-conditioned expert-source learning for inference-time guidance.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Learning, Flow Matching, Action Inversion

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_dR89OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This manuscript proposes Expert-Inversion Prior (EIP) to improve generative policy by learning a conditioned noise prior. The authors invert expert actions into the initial noise space and train either a conditional Gaussian or Flow prior. The paper further provides theoretical analysis based on KL-divergence preservation and evaluates the proposed approach across simulation and real-world tasks, demonstrating substantial improvements in task success rates.

Critical strengths
  • The manuscript investigates an interesting and practically relevant problem: improving pretrained flow-based policy without modifying the original action generator or requiring additional demonstrations.

  • The empirical results are encouraging, showing substantial improvements across multiple benchmarks, architectures, and real-world tasks.

Critical weaknesses
  • Limited novelty and incomplete related work. The central idea of learning a conditional initial noise prior for a frozen generative model has been explored in previous works, such as NoiseAR[1] and Golden Noise[2]. Moreover, FRS/DSBC[3] has already investigated learning noise-space policies from inverted expert demonstrations, while BridgePolicy[4] explores observation-informed initialization for robotic policies. Although some related robotic approaches are cited, this work does not sufficiently distinguish its methodological contributions from these prior studies, and several closely related works are missing.

  • The necessity of prior learning is not convincingly demonstrated. The deterministic MLP baseline already achieves most of the performance improvements, while EIP-Flow provides only modest additional gains over EIP-Gaussian. Furthermore, the retrieval experiments show that directly selecting expert-inverted starting points can substantially improve success rates without training an additional generative prior. These results raise questions about whether learning a complex conditional noise distribution is necessary.

  • Limited evaluation of generalization and robustness. Optimizing an distributional prior may improves the perfomance on some specific tasks, however, it can also harms the generalization and robustness. The authors may conduct some addition experiments on generalization and robustness.

[1] Li Z, Liu X, Zhang X, et al. Noisear: Autoregressing initial noise prior for diffusion models[J]. arXiv preprint arXiv:2506.01337, 2025.

[2] Zhou Z, Shao S, Bai L, et al. Golden noise for diffusion models: A learning framework[C]//2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025: 17688-17697.

[3] Tang A, Chen W, Wagenmaker A, et al. Improving Robotic Generalist Policies via Flow Reversal Steering[J]. arXiv preprint arXiv:2606.13675, 2026.

[4] Liu Z, Pan M, Wang Z, et al. Sample from What You See: Visuomotor Policy Learning via Diffusion Bridge with Observation-Embedded Stochastic Differential Equation[J]. arXiv preprint arXiv:2512.07212, 2025.

Questions for authors

See Critical Weaknesses

Reviewer_NncqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the "Expert-Inversion Prior" (EIP) to enhance the performance of fixed-flow-matching robot policies. By performing an offline inversion of expert demonstrations to derive starting-point targets, the method learns a Gaussian distribution (or a flow-based prior) conditioned on observations. During inference, this prior replaces the standard Gaussian initialization. The paper provides a distributional interpretation and reports performance improvements across 12 simulated tasks and 4 real-world tasks.

Critical strengths
  1. Well-motivated and straightforward.

The paper distinguishes between numerical inversion errors and discrepancies in the expert-policy distribution, and explains why initial values ​​must be dependent on observed data.

  1. Significant performance improvement over Gaussian initialization.

Results from various benchmarks and real-world tasks confirm that existing simulations can be enhanced - even when using a fixed policy, by employing learned initial values.

Critical weaknesses
  1. Practical benefit on stronger pretrained policies remains untested.

Closed-loop improvements are demonstrated only on task-specific FM-UNet and FM-DiT policies. Although π0 is included in inversion diagnostics, the paper does not evaluate whether EIP improves its task success. Does EIP remain useful with a stronger pretrained action generator, or are the reported gains primarily specific to the less capable baselines? A matched π0 evaluation would materially affect my assessment.

Questions for authors
  1. Does EIP improve closed-loop task success on the π0 checkpoint already used for inversion diagnostics? If this was not evaluated, were there specific obstacles to applying EIP?

  2. How does EIP compare with further training the base policy on the same demonstrations under a comparable training budget?

  3. Are the improvements over the deterministic MLP consistent across prior-training seeds?

Reviewer_dePcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Expert-Inversion Prior (EIP). EIP inverts expert demonstrations through a frozen flow-matching policy to recover their starting noises. It then trains an observation-conditioned prior over those noises (a diagonal Gaussian or a small conditional flow), which replaces Gaussian sampling at inference. A KL-invariance argument motivates the approach. Across 12 simulated tasks and 4 real-robot tasks, EIP improves average success over the base policies, and it also allows fewer decoding steps.

Critical strengths
  • Simple idea with improvements across many tasks: The generator stays frozen, and no new data is needed, so the idea is simple to implement, but it shows substantial improvements across many benchmarks. This makes EIP easy to add on top of existing policies, which I find valuable.
  • Careful treatment of inversion numerics. The solver-convergence studies show that the gap between inverted expert noise and Gaussian noise persists as inversion error drops to around 1e-6. The effect is therefore not an artifact of inaccurate inversion.
Critical weaknesses
  • Limited baselines: EIP is compared only with the base policy and the authors’ own deterministic MLP. DSBC (most relevant with the same setup), DSRL, LAFM, and WarmPrior are missing. There is also no control that spends the extra training budget on the base policy instead.
  • Reporting gaps: The real-robot setup (platform, number of demos, backbone, success criteria) is not described. Uncertainty estimates are not provided for results.
Questions for authors
  1. How does EIP compare with DSBC and DSRL using the same frozen generator?
  2. Could you report Table 2 over multiple seeds with confidence intervals?
Reviewer_PTuJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper considers flow-matching (FM) policies that transform a Gaussian distribution into a sequence of actions conditioned on an observation. Given the expert demonstrations (which are (observation, expert's action sequence) pairs), the paper provides the following: (1) the inversion map of expert data onto the source space differs from the Gaussian (Section 3.2), and (2) the KL divergence between the Gaussian and the pull-back measure of expert data onto the source space by the pretrained policy is the same as the KL between the model distribution and the expert data distribution (Section 3.3). Based on these, the paper proposes to learn an observation-conditioned source distribution to minimise the KL in the source space, namely the Expert-Inversion Prior (EIP), which doesn't require fine-tuning of the pretrained flow model. The experiments compare EIP variants and the base FM policy on robot manipulation tasks.

Critical strengths

None

Critical weaknesses

1. The main theory is already well established in the generative modelling literature

The finding in Section 3.2 essentially means that the policy's pretrained output distribution doesn't match the expert dataset, which is not surprising given that your setup assumes this. The KL equivalence in Section 3.3 seems standard, and a very similar statement can be found in [1] for normalising flows (see their Eq. 23). A similar finding is given in [2], while it considers the case where no data is available but the target distribution is given as a reward-tilted distribution, which is, in a sense, more general. Section 4 shows that the source distribution that minimises the KL depends on the observation, which, by itself, is not surprising as well, while I agree that this motivates the observation-conditional source distribution learning. Overall, Sections 3 & 4 add little to the contribution, and I'd rather treat them as background, rather than as theoretical contributions.

2. A trivial baseline is missing in the experiments

I believe the nearest-neighbor retrieval variant in Section 4 should be included in the main experiments (Section 5). I think the authors rely on two assumptions: (A) the observations in the expert dataset are similar to the observations from the test environment, (B) the mapping from observation to good (or optimal or expert) starting points is smooth enough. (A) is because there's no discussion or analysis of the robustness to out-of-distribution observations, and (B) is because, if not, the learned starting point distributions cannot generalise. I believe, under these two assumptions, the nearest-neighbor retrieval would perform well in the experiments and should be considered as a main baseline. With (A), I expect the retrieval to perform comparably, so an OOD experiment should also be included to justify the learning component.

I'm open to increasing the score depending on clarification of the theoretical contributions and justification of the learning components.

Some suggestions about writing; it is not a critical point and doesn't affect my assessment

While the writing is in general easy to read, it needs to be improved in many aspects, especially on technical terms. Here I provide several points I found sloppy:

  • line 212. π0\pi_0 is not defined.
  • line 214. "the same two tasks" should be clarified.
  • line 257. qEq_E, pEp_E, pθp_{\theta} are not defined in the main section.
  • line 265. pϕ(⋅∣c)=(Fc)qϕ(⋅∣c)p_{\phi}(\cdot \mid c) = (F_c)q_{\phi}(\cdot \mid c) ⟶\longrightarrow pϕ(⋅∣c)=(Fc)♯qϕ(⋅∣c)p_{\phi}(\cdot \mid c) = (F_c)_{\sharp}q_{\phi}(\cdot \mid c)
  • line 326. I~ci\tilde{I}_{c_i} is not defined.

References
[1] Papamakarios, George, et al. "Normalizing flows for probabilistic modeling and inference." JMLR 2021
[2] Venkatraman, Siddarth, et al. "Outsourced diffusion sampling: Efficient posterior inference in latent spaces of generative models." ICML 2025

Questions for authors
  1. Related to the motivation of this work, why do we want to learn a starting point distribution rather than just using nearest-neighbor retrieval? I want a high-level explanation rather than just empirical results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.