acceptodds
Under review as a conference paper at ICLR 2027

RoMAN-Flow: Taming Autoregressive Normalizing Flows for Offline Reinforcement Learning in Robotic Manipulation

Abstract

Offline reinforcement learning improves robotic policies using previously collected data without further environment interaction. Yet prevalent diffusion and flow-matching policies provide flexible continuous action distributions but do not offer tractable exact likelihoods, limiting their use in likelihood-based offline RL post-training. AR-NFs offer both expressive action modeling and exact likelihood evaluation, but their sequential sampling incurs substantial sampling overhead during policy optimization and deployment. We present RoMAN-Flow (Robotic Manipulation with Autoregressive Normalizing Flows), an offline reinforcement learning framework that makes AR-NF policies practical for robotic manipulation by addressing this sampling bottleneck in both stages. During policy optimization, RoMAN-Flow employs a sampling-free, advantage-weighted likelihood objective that assigns higher likelihood to high-advantage actions from the offline dataset without sampling from the autoregressive policy. For efficient deployment, it distills the optimized autoregressive policy into a one-step action generator, enabling low-latency action prediction. Experiments across simulated manipulation benchmarks and real-world robotic platforms demonstrate that RoMAN-Flow achieves competitive policy performance while substantially reducing inference latency.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robotic Manipulation, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XUuwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents RoMAN-Flow, an offline reinforcement learning framework using autoregressive normalizing flows to model continuous robotic actions with exact likelihoods. Its NF-IQL procedure applies advantage-weighted likelihood optimization to dataset action chunks, avoiding expensive autoregressive sampling during policy updates. The optimized policy is then distilled into a one-step actor by matching intermediate flow states and final actions, enabling parallel action generation. Experiments across MetaWorld, LIBERO, RoboMimic, and real-world manipulation report improved performance over imitation learning. Notably, distillation achieves an 8.55× inference speedup on LIBERO-Long.

Critical strengths
  • Introducing AR-NF architectures for robotic manipulation, which have not been widely used previously.
  • The proposed algorithm focuses on both task performance and efficiency.
  • Reported inference efficiency is promising for real-time execution of robot policies.
Critical weaknesses
  • The methodological novelty appears primarily integrative. The architecture adapts SimFlow, the policy update instantiates IQL, and the deployment method adapts an existing one-step inverse distillation.
  • Many design choices (Please refer to Q.1., Q.2., Q.3.) are possible within the proposed algorithm, but ablation studies investigating on different choices are lacking
  • Writing could have been better. It was initially hard to follow the motivations and details of the approach since everything is introduced directly in the Method section. Between the Related Work and Method sections, it would be nice if a preliminary section existed to connect prior work to the base algorithm RoMAN-Flow is building on.
Questions for authors
  • Q.1. How would RoMAN-Flow perform when attached to backbones other than SimFlow, e.g., TarFlow, STARFlow, FARMER, and BiFlow? I am especially curious about FARMER and BiFlow, since they further improve efficiency through one-step inversion.
  • Q.2. Why should we perturb ata_t with noise during data-induced distillation? Does it empirically lead to better performance, or is it based on some theoretical insight? How does the performance change with different noise variance values?
  • Q.3. Why do we need both data-induced and prior-sampled distillations? Adding on Table 8, please provide an ablation study on one more task environment that compares three methods: (1) only using the data-induced distillation loss, (2) only using the prior-sampled distillation loss, and (3) using both.
2 comments

Thank you for the clarification.

However, I have remaining concerns, which are largely related to substantial reorganization of the writing.

  • Why is tracking the exact likelihood important? The main motivation of the research starts from the limitation that flow and diffusion policies cannot exactly model the likelihood. However, the work lacks clarity on why modeling the exact likelihood is crucial.

  • Lack of Clear Analysis on the AR-NF family. In my perspective, the proposed method lacks novelty since the approach is to combine various pre-existing techniques. However, the authors study a new class of policy architecture called AR-NF, and I value this aspect highly. In this case, I think the authors should have presented a substantial amount of controlled ablation and analysis to reveal when and why AR-NF performs well.

Although some of the related content was prepared during the rebuttal, I think the paper requires substantial reorganization of the presentation based on this content. Therefore, I remain tilted towards weak rejection.

Thank you for the follow-up.

I maintain my score for the following reason:

  • The Introduction part of the latest submission makes it clear to me that the authors investigate AR-NF because there is a lack of prior work that applies AR-NF. This is itself good, but in this case, the authors should have substantially investigated the full possible variants of AR-NF categories that have been applied to other fields, e.g., LLMs. The authors' rebuttal addresses some of it due to my questions (Section 5.5), but I think the paper in its current form still lacks in this aspect. After extensive analysis, I hope the authors find some key observations that can lead to good methodological novelty, which I think is also currently lacking.

Thank you, and I hope my review further improves the quality of your paper.

Reviewer_FCyJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RoMAN-Flow uses autoregressive normalizing flows for offline policy post-training with an IQL-style advantage-weighted likelihood objective. Actor updates evaluate the likelihood of dataset actions without autoregressive sampling. The resulting policy is then distilled into a one-step generator to reduce inference latency. Experiments cover RoboMimic, MetaWorld, LIBERO, and four real-robot tasks.

Critical strengths
  1. The motivation is clear. AR-NFs can evaluate the likelihood of dataset actions without autoregressive sampling, making them suitable for advantage-weighted policy updates. One-step distillation then addresses the sampling cost at deployment.

  2. The implementation and evaluation are substantial. The paper implements the full pipeline from imitation learning through offline post-training to distillation, with both simulation and real-robot experiments. Post-training produces clear improvements on MetaWorld, LIBERO-Long, and the real-robot tasks.

Critical weaknesses
  1. The benefit of the AR-NF architecture itself remains unclear. The prefix critic, Q-value averaging, and IQL-style policy update build on existing designs. There is a Gaussian policy comparison, but its capacity is not specified. In Table 6, AR-NF success rates remain at 77%–78% as capacity increases from roughly 34M to 466M parameters, reaching 85% only at around 686M, so capacity differences matter. Comparisons with other generative policies that support likelihood evaluation under the same offline objective are missing. The speed experiment also compares only IQL and TD3+BC with the same AR-NF, rather than establishing a computational advantage over other generative policies.

  2. Some tasks are close to saturation, and RL baselines on LIBERO are limited. Success rates are already high on RoboMimic Lift and Can, as well as LIBERO Spatial, Object, and Goal, leaving limited room to distinguish methods. The LIBERO comparisons focus mainly on imitation learning and supervised fine-tuning, with limited coverage of RL post-training methods.

  3. Distillation does not always incur only a “limited” performance loss. The mean success rates for imitation learning, the post-trained teacher, and the distilled student are 80% → 85% → 80% on Square, 93.5% → 95.3% → 93.7% on LIBERO, and 57.3% → 81.5% → 69.5% on the real robot. About half of the real-robot improvement from post-training is lost after distillation. Retention is better on MetaWorld and LIBERO-Long, but reporting student success as a fraction of teacher success can obscure how much of the post-training gain is lost elsewhere.

Questions for authors
  1. Methods such as GenPO and GenPO++ use reversible constructions to support likelihood-ratio evaluation for flow policies, sharing the use of invertible mappings with this paper. Could such a construction be adapted to advantage-weighted likelihood updates in an offline IQL setting, with comparisons of performance and training cost?

  2. What are the Gaussian policy’s architecture and parameter count, and is its capacity comparable to that of the AR-NF? Could the LIBERO comparison include RL post-training baselines?

  3. Section 3.4 encodes perturbed actions and reconstructs clean actions, whereas Algorithm 1 does not include the perturbation. Is this denoising objective used in practice? Please specify the perturbation scale and loss weights λs,λa\lambda_s,\lambda_a.

Reviewer_EFr1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RoMAN-Flow, an offline RL method for robots that trains the policy over continuous action chunks conditioned on a SmolVLM encoder and does transformer-based autoregressive normalizing flow. The method improves over regular imitation learning on MetaWorld and Libero.

Critical strengths

Autoregressive normalizing flows compute likelihoods quickly in parallel but generate actions slowly. The IQL actor update only needs likelihoods of dataset actions and never generates actions. The method is effective as it avoids the slow step so the formulation is a strong point.

Critical weaknesses

The critical weakness of this paper is limited method novelty. The method seems to be a combination of existing components instead of novel ideas. This is consequential because novelty is valuable in a research paper.

Questions for authors

How were hyperparameters selected for the method? Is that a fair comparison to baselines?

On real robot experiments, how was the setting initialized?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.