acceptodds
Under review as a conference paper at ICLR 2027

Coherent Off-Policy Improvement of Large Behavior Models with Learned Rewards

15 views

Abstract

Distilling expert demonstration data into large generative models using behavioral cloning is a scalable approach to learning capable policies for robotic control, particularly for dexterous manipulation. Reinforcement learning (RL) can be used as a means to finetune these policies further using additional experience. An open question is whether RL is more sample-efficient than collecting more expert demonstrations. Prior work has finetuned large pretrained policies in a scalable fashion by applying RL to a smaller residual policy that corrects the pretrained model. However, for typical sparse reward tasks, RL algorithms can struggle to optimize the behavior in a sample-efficient manner. We explore inverse reinforcement learning, where a dense reward function is learned from expert demonstrations, potentially reducing the challenge of RL finetuning. We specifically consider coherent imitation learning, an IRL method that facilitates improvement of the BC policy by using a specific learned reward formulation with theoretical guarantees for imitation learning. We show that our IRL method maintains or improves the performance of pi-0.5 on all six sparse manipulation tasks and achieves a >= 90% success rate on five out of six complex manipulation tasks, outperforming RL-based baselines using sparse rewards. We also demonstrate real-world learning on a robot manipulation task, finetuning pi-0.5 to success with less than 30 minutes of environment interaction time. By ensuring our initial pretrained finetuning policy is optimal for our initial reward and critic, our method circumvents the initial drop commonly seen in RL finetuning and enables faster improvement.

Primary area:
reinforcement learning
Keywords:
inverse reinforcem learning, vision-language-action models, finetuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7JSiOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose an alternative to sparse-reward RL post-training of VLAs, using inverse RL to recover a dense reward from expert demonstrations. The authors claim that reinforcement learning often suffers from initial performance drops and the unlearning of already learned behaviors. They propose optimizing an averaged action, in which the actions predicted by the VLA are averaged with the action predictions of a small corrective policy. In addition the corrective policy is initialized from a BC checkpoint, thereby mitigating initial performance drops. The CSIL recipe is followed, in which the log-policy ratio is used as the reward. The authors incorporate recent improvements from the literature, including normalization and regularization techniques, into CSIL and call the improved version CSIL++. Experiments show that CSIL++ works better than CSIL on an image-based task. In addition, CSIL works better than the selected baselines for VLA fine-tuning across six different tasks.

Critical strengths
  • The idea of using CSIL for VLA post-training is clear and well explained.
  • Overall, the experiments cover a proper number of tasks and are extensive. The results look promising and show that CSIL++ works well for fine-tuning VLAs.
  • The experiments on demonstration efficiency and the comparison with BC using additional data are really nice and show how RL performs better.
  • The paper clearly presents how averaged and residual actions differ in terms of their reward, which is quite interesting.
Critical weaknesses
  • “The authors claim that reinforcement learning often suffers from initial performance drops and the unlearning of already learned behaviors.” How do you know that the performance drop is caused by the unlearning of initial behaviors? In DSRL-style RL, only a small policy is trained while the VLA is kept frozen, so there is no unlearning. Moreover, such a performance drop can be mitigated by a gated start, warm-up steps, or bounded residuals. Please clearly discuss and demonstrate the reason for such a performance drop, analyzing the critic as well, rather than only how the policy differs from the BC starting point.
  • CSIL++ seems to be an updated version of CSIL with better normalization and regularization. It is clear that employing such recent findings from the literature will improve prior methods. Therefore, the novelty of this component is questionable, and it is hard to accept it as a key contribution of the paper.
  • The improvement over CSIL appears limited, since CSIL++ is compared with CSIL on only two tasks, and on one of them both methods perform equally well. It would be great if this comparison could be extended to two more tasks.
  • In Table 1, the lowest upper quartile is bolded, which is not common practice. Please bold the average values instead. In addition, ResFit performs better than CSIL++ on Coffee and Mug Cleanup and equally well on Hammer Cleanup. Please bold these numbers properly, otherwise, the table is misleading.
  • The strong performance of ResFit in Table 1 is not properly discussed. Please clearly show how ResFit’s performance is unstable and compare it with that of CSIL++.
  • Please compare the method with DSRL with a gated start. In addition, a comparison with filtered SFT would be nice, since success/failure detection is already available.
  • CSIL++ uses its own encoder, but this choice is not ablated. Although it is not critical, an ablation would be valuable to see what happens when VLA features are used instead.
Questions for authors

Questions are listed in Weaknesses part, but here are them again:

  • Please clearly discuss and demonstrate the reason for a performance drop in RL post-training, analyzing the critic as well, rather than only how the policy differs from the BC starting point.
  • Please extend the comparison between CSIL++ and CSIL and include at least two more tasks.
  • Please bold the numbers correctly on Table 1, it is misleading in the current shape.
  • The strong performance of ResFit in Table 1 should be properly discussed, and clearly show how ResFit’s performance is unstable and compare it with that of CSIL++.
  • Please compare the method with DSRL with a gated start. In addition, a comparison with filtered SFT would be nice, since success/failure detection is already available.
  • Can you ablate the choice of using a separate encoder for CSIL++?
2 comments

We thank the reviewer for their extensive review!

Since the reviewer has not clarified their comments in the past seven days, we provide a partial rebuttal here for the aspects we can address at this stage.

How do you know that the performance drop is caused by the unlearning of initial behaviors?

We recommend the reviewer read the original CSIL paper by Watson et al. 2023 to appreciate this phenomenon. If the BC performance is strong but performance drops after a few RL updates, the actor’s objective must cause the policy to unlearn the initial strong behaviour. This is due to function approximation errors in the critic. CSIL fixes this with a dense coherent reward, critic pretraining and KL regularization against the BC policy in the actor loss. We would appreciate it if the reviewer could clarify the source of their confusion with this phenomena.

In DSRL-style RL, only a small policy is trained while the VLA is kept frozen, so there is no unlearning.

The DSRL paper’s results show initial performance drops across environments so we would disagree that the method does not unlearn. Our experimental results using their code reproduce this unlearning as well.

Therefore, the novelty of this component is questionable, and it is hard to accept it as a key contribution of the paper.

The development of CSIL++ is crucial to this project, after observing that CSIL does not scale to harder tasks such as Square and beyond. The original CSIL paper only benchmarks on image-based Lift and Can in robomimic. Pi-0.5 already solves Lift and Can with BC, so for IRL finetuning to be useful and meaningful we need a method that scales to harder tasks that pi-0.5 struggles to solve with BC. We believe the capability that CSIL++ unlocks makes it valuable as a central contribution, as the paper would not be publishable without it due to poor experimental results.

The improvement over CSIL appears limited, since CSIL++ is compared with CSIL on only two tasks, and on one of them both methods perform equally well. It would be great if this comparison could be extended to two more tasks.

We have new results showing the better performance of CSIL++ compared to CSIL across all six mimicgen tasks we consider.

In Table 1, the lowest upper quartile is bolded, which is not common practice.

Fixed. However we would argue this is a perfectly valid method selection criteria, as methods with high performance variance can have competitive medians even if several seeds fail to solve the task, such as ResFit, so using the upper quartile captures this lack of consistency.

Please compare the method with DSRL.

We have added this to the appendix in Section C.3.

CSIL++ uses its own encoder, but this choice is not ablated.

The pi-0.5 encoder is much larger than our CNN. Averaging over 1000 inference calls, we measure on our hardware

Steps/s: Resnet encoder: ~1800 VLA embeddings: ~45

FLOPs per step: Resnet encoder: 0.365 GFLOP VLA embeddings: 2560 GFLOP

Therefore using the pi-0.5 encoder would slow our experiments down by an impractical degree, and this is with freezing the representations for the RL policy.

Reviewer_7JSiOpenReview ↗

Response to Rebuttal

Thanks to the authors for the response and additional experiments. The extended CSIL vs. CSIL++ comparison, the corrected bolding, the ResFit discussion, and the DSRL comparison are appreciated. However, they also narrow the empirical claim. With correct bolding, ResFit has the lowest regret on Coffee and Mug Cleanup and is within 0.01 on Hammer Cleanup. DSRL matches CSIL++ in final success rate. The advantage of CSIL++ is therefore mostly on sample efficiency and avoiding the initial drop, not on the final performance and text should reflect this. Therefore this makes my comment on performance drop (W1) more important and it remains untested. By gated start, I mean keeping the learned policy’s contribution off, or annealed in, until the critic has been trained on base-policy data. If DSRL or ResFit with such a start or XQC+OD with KL term no longer drops initially, then the main claim of the paper is considerably weakened. For filtered SFT, your reading was correct and it is a common approach for VLA post-training, would be great to see a comparison with it. Regarding the encoder (W7), I accept the cost argument, but “better and faster” is unsupported without an ablation, so please soften it. Given these, although my main concerns are far from being addressed, I will keep my score.

Reviewer_quZUOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to efficiently improve Large Behavior Models (LBMs) with online Reinforcement Learning. Based on CSIL, the authors propose a few modifications on it and make it CSIL++. On 6 RoboSuite tasks, the proposed method CSIL++ demonstrate performance advantage over its original version CSIL and other relevant online improvement methods. The authors also evaluate the effectiveness of CSIL++ in real-robot experiments.

Critical strengths
  1. The paper studies how to improve Large Behavior Models (LBMs) with online Reinforcement Learning, which is a very important and timely topic.

  2. The authors conduct real-robot experiments to evaluate its proposed CSIL++'s effectiveness.

Critical weaknesses
  1. I am confused about what are the authors' contributions. It seems that CSIL is an existing algorithm, and the authors add some modifications to it and name it CSIL++. These modifications do not appear sufficient novelty. I am not sure if the authors consider "average" formulation as part of its contribution

  2. The experiments appear strange to me. CSIL should be the most important baseline, but the comparison between CSIL and CSIL++ only happens in 2 tasks and the main experiment results do not have CSIL. What is the consideration here ?

  3. The real-robot setup only has a sketch instead of a real picture. Why not providing the real setup picture ?

Questions for authors

See the weaknesses I list.

4 comments

Thank you for your feedback!

Firstly we wish to highlight that the review's summary of our submission is not correct. We are performing inverse reinforcement learning (IRL), not reinforcement learning, for improving LBMs. This is important because we are the first to do this and it is our primary contribution, as stated in our introduction. Inverse RL is attractive as it allows us to use the existing demonstrations to provide a dense reward, instead of the sparse rewards usually used in manipulation tasks.

I am confused about what are the authors' contributions.

As already stated, please refer to the three listed contributions in our introduction section. CSIL++ is our secondary contribution, and was necessary to scale CSIL to more challenging tasks that LBMs need finetuning for. Our main contribution is showing that IRL can be used for LBM finetuning and has qualities over standard RL.

The experiments appear strange to me. CSIL should be the most important baseline, but the comparison between CSIL and CSIL++ only happens in 2 tasks and the main experiment results do not have CSIL

Since our primary contribution is applying IRL to LBM finetuning, our main experimental question is showing performance over standard RL finetuning with sparse rewards. Therefore our main baselines are our method RL ‘backend’ with sparse rewards (XQC+OD) and prior works (ResFit, PLD, Policy Decorator). CSIL vs CSIL++ is arguably an implementation detail, albeit an important one. Since we show in Figure 5 that CSIL cannot solve the vision-based square task, we did not see the experimental value in evaluating CSIL on harder tasks. However, we have now run these experiments and can show that CSIL++ outperforms CSIL on these harder tasks as well, apart from Nut Assembly where RL is challenging for both. See the table reporting cumulative regret below, where lower is better

MethodSquareCoffeeMug CleanupThreadingNut AssemblyHammer Cleanup
CSIL++0.72 [0.68, 0.82]0.46 [0.37, 0.55]2.16 [2.10, 2.19]1.23 [0.93, 1.60]4.97 [4.91, 4.97]0.13 [0.02, 0.18]
CSIL2.39 [2.38, 2.46]1.54 [1.48, 1.57]4.35 [4.13, 4.38]2.67 [2.25, 3.63]4.95 [4.94, 4.96]2.38 [2.35, 4.20]

We still believe that LBM finetuning with CSIL is not experimentally interesting due to its established performance limitations, but we can run those experiments if this would convince the reviewer to advocate for acceptance. Please let us know!

The real-robot setup only has a sketch instead of a real picture. Why not providing the real setup picture ?

We believe the vector graphic in Figure 7 gives the paper a cleaner, clearer and more professional presentation of our experimental. setup. Moreover, in robotics research, photos of the lab environment can break double-blind anonymity if the lab space is recognized by a reviewer. If you wish to see the real-world experiments, we have uploaded videos to an anonymous website https://anonymous-submission-abcde.github.io/coherent-lbm-improvement/.

We hope this rebuttal addresses your concerns. Please let us know if you have further questions, in particular regarding our primary contribution of using inverse reinforcement learning for LBM finetuning.

Thanks for your response ! The authors claimed that their most important contribution is showing that IRL can be used for LBM fine-tuning and has qualities over standard RL. It is already demonstrated by multiple literature that in RL settings, IRL has advantages over standard RL, and in LBM fine-tuning settings, having a dense reward function improves sample efficiency. It is well expected that IRL, as one way of getting a dense reward function, can help LBM fine-tuning. I wouldn't say it is not a contribution, but it is surely not a major one that is up to the standards of ICLR. I choose to keep my score.

Thank you for engaging with our rebuttal.

We respectfully disagree that our result is “well expected,” and would like to clarify why:

1. IRL does not straightforwardly work for LBM finetuning. If dense IRL rewards straightforwardly helped LBM finetuning, existing IRL methods would succeed off the shelf. Our experiments show they do not. CSIL, a state-of-the-art IRL method, fails on vision-based Square (Figure 5) and incurs a much higher cumulative regret than CSIL++ on five of six tasks (see the table in our initial response). Making deep off-policy IRL work in this setting required the modifications we introduce in this submission. That crucial gap between “should help in principle” and “actually works in practice” is the contribution of our submission.

2. Request for references of claims. Regarding the claim "It is already demonstrated by multiple literature that... in LBM fine-tuning settings, having a dense reward function improves sample efficiency." We are not aware of prior work applying IRL to LBM/VLA finetuning, and we believe we are the first to do so. If the reviewer could point us to the literature they have in mind, we would gladly cite and discuss it, and position our contribution accordingly. We are also unfamiliar with LBM finetuning works that use a handcrafted dense reward, as doing this from vision is an engineering challenge, and would appreciate references.

3. Consistency with ICLR standards. Regarding the claim "it is surely not a major one that is up to the standards of ICLR", recent ICLR papers on this exact problem have been accepted for comparable contributions. For example, PLD (Xiao et al., ICLR 2026), which we compare against directly, combines established components (residual off-policy sparse RL and SFT distillation) to improve generalist policies. Our work addresses the same problem with a different and, we argue, complementary insight: demonstrations can supply dense rewards via IRL, removing the reliance on sparse task rewards. Our method outperforms the RL component of PLD in our experiments. Moreover, both PLD and our method are validated with real-robot experiments.

We also note that the original review rates soundness as having “minor or fixable concerns,” and we believe we have addressed the concrete concerns raised in the review (missing CSIL comparisons, real-robot video evidence). We would be grateful if the reviewer could reconsider their assessment in light of this.

Thanks for further replies. My response is as follows:

  1. "LBM does not straightforwardly work for LBM fine-tuning". If CSIL can directly work very well on LBM fine-tuning settings, that means your paper has no novelty but a technical report. My acknowledgement that your paper has some contribution is based on the fact that you provide some modifications on CSIL to make it work better. However, these modifications are simple ML/RL tricks instead of algorithm-level innovations.

  2. "Request for references of claims". I am not sure if the authors understand my points. What I say is that (in RL settings, IRL has advantages over standard RL) and (in LBM fine-tuning settings, having a dense reward function can help) are demonstrated by multiple literature. For example, VLA-RL and RoboMeter all show the evidence of having a dense reward function helps.

  3. "Consistency with ICLR standards". It is not my position or the authors' position to comment on a paper accepted by previous ICLR conferences here, and comparison with previously accepted papers and claiming that why it got accepted does not make any sense. My point here is that the authors' paper do not provide sufficient algorithm-level contributions or valuable insights.

Reviewer_7wnFOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes using coherent soft imitation learning which is an inverse RL method that uses a reward of the scaled log-ratio between a BC policy and a uniform prior and using that to finetune a frozen large behavior model pi 0.5 model online. The method tests on Robomimic and MimicGen and compares against ResFit, Policy Decorator, the RL part of PLD, and XQC.

Critical strengths

The paper has a central claim around preventing the initial drop of reinforcement learning. This is backed up by experimental evidence and the performance is more stable than other methods it compared to. This is important because training a large model needs to be stable. If methods dip large in performance, it may not be as practical in the real world.

Critical weaknesses

The main weakness is limited novelty. CSIL is prior work and most CSIL++ components come from XQC/BRO-style methods, so the contribution is mainly the combination and the averaging scheme. This is important because the paper also provides limited ablations toward understanding of different components.

Questions for authors

What is the success rate of the averaged policy before RL on each task?

How many seeds were run in the real-world experiment, and what is the wall-clock training time for the RL methods?

1 comment

Thank you for the review of our submission!

The main weakness is limited novelty. CSIL is prior work and most CSIL++ components come from XQC/BRO-style methods

CSIL++ is the secondary contribution of this paper, as stated in our introduction. The first is showing that inverse RL is a viable finetuning strategy for VLA and can provide a dense reward for learning tasks without the need for reward design and complex scene understanding, e.g. pose estimation. We would ask the reviewer to comment on the novelty of our primary contribution, as to the best of our knowledge no paper has demonstrated the utility of CSIL and inverse RL in this setting before.

the paper also limited ablations toward understanding of different components.

Which ablations would the review like to see? Our XQC+OD baseline ablates the coherent reward, as it is the XQC RL components and BC training maximizing the sparse reward, like prior works have proposed.

What is the success rate of the averaged policy before RL on each task?

The performance at an environment steps value of 0 represents the BC performance.

How many seeds were run in the real-world experiment

The real world experiment represents one RL seed with 25 evaluation seeds, as its aim is to validate the more extensive simulation results in a real-world setting.

what is the wall-clock training time for the RL methods?

In simulation, on an RTX A5000 it's around 5 hours for 500k steps and on a RTX5080 it's around 2.5 hours for 500k steps. For the real world experiments it was around 50 minutes. If you would like us to rerun our experiments with wall-clock duration monitoring per step, please let us know.

We hope this rebuttal addresses your questions and concerns, please let us know if you have any further issues.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We wish to thank the reviewers for their feedback.

This is a summary rebuttal post summarizing the feedback, our rebuttal and subsequent changes.

The main common criticism from reviewers quZU, 7wnF and 7JSi is novelty. However this criticism cited our second contributions (CSIL++) not our primary contribution (VLA finetuning with inverse RL). Inverse RL uses demonstrations to provide a dense reward for RL finetuning, which is easier to learn from compared to sparse rewards, but IRL methods are typically challenging to use due to many methods using adversarial optimization, e.g. GAIL, DAC. We show that CSIL++ is stable and scalable enough to outperform RL with sparse rewards in this setting. We would appreciate it if the reviewers reassess their novelty assessment in light of our primary contribution.

Reviewer 7JSi also requested a comparison to DSRL and changes to the paper’s presentation and discussion which we have done. However, several of the reviewer’s suggestions were unclear to us and they have not yet clarified their comments upon our request last week.

In summary, we have added

  • Benchmark against diffusion steering RL on MimicGen
  • Extended comparison between CSIL and CSIL++ on the mimicgen tasks.
  • Additional baseline of VLA with CSIL finetuning.
  • Change table bolding
  • Expand ResFit discussion

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.