acceptodds
Under review as a conference paper at ICLR 2027

CRL-VLA: Continual Vision–Language–Action Learning

Abstract

Lifelong learning is critical for embodied agents in open-world environments, where reinforcement learning fine-tuning has emerged as an important paradigm to enable Vision-Language-Action (VLA) models to master dexterous manipulation through environmental interaction. Thus, Continual Reinforcement Learning (CRL) is a promising pathway for deploying VLAs in lifelong robotic scenarios, yet balancing stability (retaining old skills) and plasticity (learning new ones) remains a formidable challenge. We introduce CRL-VLA, a framework for continual post-training of VLA models with theoretical performance-change bounds. The bounds relate old- and new-task return changes to goal-conditioned advantage magnitude and policy divergence. Motivated by this sensitivity analysis, the revised CRL-VLA formulation combines a frozen retention action critic, a trainable adaptation action critic, and policy regularization for retention and adaptation. Evaluations in simulation and on physical robots show that CRL-VLA supports continual adaptation across diverse VLA RL post-training paradigms and robot embodiments while maintaining robust task performance.

Primary area:
applications to robotics, autonomy, planning
Keywords:
VLA, Reinforcement learning, Continual Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bDddOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies continual reinforcement learning for VLA models. It combines a frozen retention critic and a trainable adaptation critic with replay and policy regularization to learn new tasks while reducing forgetting. Experiments cover LIBERO, ManiSkill, and physical robots.

Critical strengths
  1. The evaluation is reasonably comprehensive. It includes online and offline learning, different models, and multiple robot platforms.
  2. Some settings show meaningful improvements. The method performs well under visual disturbances, and average success in the five-task physical experiment improves from 34% to 58%.
Critical weaknesses
  1. Combining a frozen historical critic and a trainable critic with replay and policy regularization follows a familiar retention strategy in continual learning. The paper does not clearly establish a substantial methodological advance.
  2. In Table 2, success improves only from 81.2% for sequential learning to 81.6%, while forgetting becomes worse. Although the five-task physical experiment improves average success, the earliest task still falls to 0%. It remains unclear when the method better preserves previous capabilities.
  3. The method combines value retention, replay, and policy regularization, with different implementations across backends. Without matched single-critic comparisons, it is difficult to determine whether the gains come from the dual-critic design.
Questions for authors
  1. How do you explain the worse forgetting and transfer metrics in Table 2? Under which conditions does the method clearly outperform standard sequential RL?
  2. How does performance change in each backend when the frozen retention critic is removed, with replay, policy regularization, and training budgets held fixed?
1 comment
AuthorsOpenReview ↗

Response to Reviewer bDdd: Retention Across Settings and Single-Critic Comparison

Thank you for recognizing the breadth of evaluation and the gains under visual disturbances and in the five-task robot study. We address when retention helps and what happens without the frozen critic, including all additional benchmark comparisons.

Q1: Table 2 and conditions for improvement. On the original five-task Spatial protocol, CRL-VLA and SL achieve 81.6% and 81.2% final success, respectively. NBT is 4.2 versus 0.3, and FWT is 0.13 versus 3.9. Thus, similar final success coexists with worse retention and forward-transfer metrics in this setting. SL already exhibits little forgetting here.

After review, we added CRL-VLA evaluations on Long. The tables include every reference baseline in the expanded Table 2; baseline values are reproduced from the cited sources. Values are in %, with AVG measuring final success, NBT retention (lower is better), and FWT forward transfer.

LIBERO-Long

MethodAVG ↑NBT ↓FWT ↑
SL89.8±0.9−2.4±1.00.5±0.1
EWC86.6±0.30.8±1.33.0±1.3
ER88.8±0.8−0.2±1.7−1.1±0.6
DER87.6±0.40.7±0.80.7±0.2
DWE88.4±0.50.0±0.00.0±0.0
SLCA86.9±0.6−1.3±1.0−0.2±0.3
RETAIN86.2±0.91.6±1.01.0±1.2
MTL (Oracle)90.5±0.8——
CRL-VLA (new)94.0±3.2−2.3±7.4−1.5±3.8

On Long, mean AVG exceeds SL by 4.2 points and ER by 5.2 points, with NBT close to SL (−2.3 versus −2.4).

Visual perturbations provide direct evidence for historical retention in the evaluated setting. In our additional three-task Spatial study, replay-only ER has FAR 0.611 and mean forgetting 0.22. Historical KL alone yields 0.889 and 0.05; frozen-teacher Q retention alone yields 0.868 and 0.04. Both FAR improvements over ER occur in all three seeds. Full achieves the highest mean FAR, 0.917±0.07.

The earliest task reaching 0%. This result is from the five-task offline robot experiment using WAV. T1 full-task success falls from 50% at checkpoint 4 to 0% at checkpoint 5 (Table 12), while grasp success at that final checkpoint remains 100% (Table 13). Failures therefore occur after grasping, during task completion.

Final T2–T5 success is 40%, 100%, 90%, and 60% for CRL-VLA, versus 0%, 50%, 50%, and 40% for the baseline. Overall final success increases from 34% to 58%. We will present this aggregate improvement alongside T1's trajectory. Each checkpoint–task pair has ten evaluation trials.

Q2: Removing the frozen critic. We implemented your suggested single-critic setting as C in OpenVLA-OFT/RIPT: one critic learns from historical and current data, retaining replay and historical KL but removing the frozen retention teacher. This also addresses AjcJ's component-analysis request and kEhZ's request to distinguish value retention from policy constraints.

Variants share the perturbed three-task Spatial sequence, training budget, and replay of the previous stage's latest 30 successful trajectories. Each seen task receives 16 evaluations after each stage. Results are mean±SD over three seeds.

VariantRetention configurationFAR ↑
ERReplay only0.611±0.18
CSingle critic on old/new data + historical KL0.667±0.11
BFrozen-teacher V distillation + historical KL0.861±0.11
DFrozen-teacher Q distillation, no historical KL0.868±0.06
AHistorical KL, no value distillation0.889±0.08
FullFrozen-teacher Q distillation + historical KL0.917±0.07

Full's mean FAR is 25.0 points above C. C is also below A and D in all three seeds. These results support a stable historical reference in the evaluated configurations. We will report each variant's actual loss coefficients to make the comparison conditions explicit.

The new single-critic experiment covers RIPT. WAV uses a frozen value expert for historical supervision; Expo-FT combines frozen and trainable Q branches for candidate evaluation. We will distinguish these implementations and tie each quantitative conclusion to its evaluated backend.

Reviewer_AjcJOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents CRL-VLA, a dual-critic framework for continual post-training of VLA models using Reinforcement Learning, aiming for continually improve the robot on new tasks while avoiding degrading performance on previously learned tasks. The authors frame each learning task as a goal-conditioned Markov Decision Process, and define two initial quantities: KL-Divergence between new and old action distributions, where states are visited by the old policy, and KL-Divergence between new and old action distributions, where states are visited by the new policy. The authors proceed by leveraging these divergence quantities to establish two bounds: an upper bound on old task performance, and an upper bound on new task performance. As both bound quantities are proportional to the respective divergence and advantage magnitude, the authors propose policy regularization to reduce divergence on old tasks and preserve historical Q-values, since advantage magnitude is computed from Q-values. However, the authors point out that preserving Q-values does not directly minimize advantage magnitude, and so it is not theoretically guaranteed to reduce forgetting. From a methodological point of view, the authors propose to use a standard actor loss to improve the policy on the new tasks, alongside two Q-value-related losses to regulate the learning of Q-values, and two KL-based losses to regulate policy behavioural changes. Specifically, they propose LretL_{ret} to preserve previously learned Q-values, LadaptL_{adapt} to learn new Q-values, RoldR_{old} to discourage excessive behavioural change on previous tasks, and RnewR_{new} to discourage excessive behavioural change when learning new tasks. Finally, the authors evaluate CRL-VLA in both simulation and real-world settings, considering multiple types of scenarios, including single-task continual learning, learning over longer sequences, and tasks with visual disturbances.

Critical strengths

The two performance bounds described in the paper provide a useful perspective to solve the stability–plasticity trade-off and motivate the proposed approach.

Critical weaknesses

W1) The core part of the paper around which the methodology is based is the two performance bounds. However, they are built upon established RL theory, as also described in Appendix B.2. Therefore, the technical novelty is unclear.

W2) The paper exhibits an uneven level of clarity and presentation. The first part of the paper, up to Preliminaries (included) the paper is relatively easier to follow. The Methodology section instead would benefit from more detailed explanations and intuition behind the proposed formulations. For instance, the divercence quantities and the performance bounds quantities are introduced with limited intuitive interpretation and no clear naming. Providing intuitive explanations alongside the mathematical definitions would significantly improve the reader's understanding. Moreover, the Experiments section introduces a large number of baselines without any motivation of their selection.

W3) Performance improvement looks variable across experiments. For instance, in Five-task LIBERTO-Spatial results (Table 2), there is really marginal improvement with respect to the SL baseline. Also, in the same table, standard deviation is reported, but it is not specified the number of runs nor described in the text or the caption.

W4) The Ablation and Mechanism Analysis Section (5.6) is very limited and immediately defers the reader to Appendix D. Given that the proposed method combines several loss components, the ablation analysis of such loss components should be included in the main corpus of the paper. Moreover, even in the Appendix, the individual loss components are not systematically ablated. Instead, only the modulation of their strength coefficients is analyzed. It would be better to gradually introduce the losses one by one to assess their individual contributions.

Questions for authors
  1. Can the authors clarify the novel contribution of the two performance bounds?
  2. How many runs were used to compute the standard deviation in Table 2?
  3. Could the authors clarify the individual contributions of each loss component?
3 comments
AuthorsOpenReview ↗

Response to Reviewer AjcJ: Clarifications and Complete Additional Experiments

Thank you for recognizing the bounds' useful perspective and requesting clearer intuition and component evidence. We provide the complete additional results below.

W1/Q1: Theory. Theorem 1 uses established performance-difference and occupancy analysis to express old- and new-task sensitivity under their respective goal/state distributions. Its role is to motivate separate historical-retention and current-adaptation objectives. The advantage magnitude belongs to a fixed reference policy; Q distillation preserves historical predictions rather than directly minimizing that bound term. We will clarify this distinction and the backend-specific mechanisms.

W2: Intuition and baseline selection. We will introduce the objectives before their equations:

  • Historical KL constrains behavioral changes on old-task states.
  • Value distillation keeps trainable predictions close to a frozen historical teacher.
  • Adaptation value learning fits current-task returns.
  • The actor objective improves current-task behavior.

RIPT uses RLOO advantages and PPO clipping, with αnew=0\alpha_{\mathrm{new}}=0; WAV uses historical supervision, and Expo-FT uses candidate scoring. We will specify active objectives per backend. SL measures sequential learning; ER isolates replay; EWC/DER and related methods compare retention strategies; MTL provides joint-data training; Stellar-VLA and Info-VLA represent continual VLA approaches. We will also clarify metric names and remove internal version labels.

W3/Q2: Runs and variation across settings. CRL-VLA results in Table 2 use three training seeds. The cited Hu et al. (2026) baseline results use three seeds and mean±SE. We will label run counts and uncertainty definitions separately for each result group. The component experiment below uses three seeds and mean±SD.

Spatial's final success is similar for CRL-VLA and SL (81.6% versus 81.2%). To examine performance beyond that setting, we added CRL-VLA evaluations on Long after review. The complete comparison follows; reference baseline values are reproduced from the sources cited in the expanded Table 2. Values are in %.

LIBERO-Long

MethodAVG ↑NBT ↓FWT ↑
SL89.8±0.9−2.4±1.00.5±0.1
EWC86.6±0.30.8±1.33.0±1.3
ER88.8±0.8−0.2±1.7−1.1±0.6
DER87.6±0.40.7±0.80.7±0.2
DWE88.4±0.50.0±0.00.0±0.0
SLCA86.9±0.6−1.3±1.0−0.2±0.3
RETAIN86.2±0.91.6±1.01.0±1.2
MTL (Oracle)90.5±0.8——
CRL-VLA (new)94.0±3.2−2.3±7.4−1.5±3.8

On Long, mean AVG improves over SL by 4.2 points, with similar NBT.

W4/Q3: Individual component contributions. This experiment also addresses kEhZ's CLEAR-related comparison and bDdd's single-critic question. We use OpenVLA-OFT/RIPT with the three-task LIBERO-Spatial sequence and stage-wise visual perturbations. Variants share task order, training budget, and replay of the previous stage's latest 30 successful trajectories. Each seen task receives 16 evaluations after each stage.

VariantRetention configurationFAR ↑
ERReplay only0.611±0.18
CSingle critic on old/new data + historical KL0.667±0.11
BFrozen-teacher V distillation + historical KL0.861±0.11
DFrozen-teacher Q distillation, no historical KL0.868±0.06
AHistorical KL, no value distillation0.889±0.08
FullFrozen-teacher Q distillation + historical KL0.917±0.07

Adding historical KL alone (A) improves mean FAR over ER by 27.8 points; adding frozen-teacher Q retention alone (D) improves it by 25.7 points. Both directions hold in all three seeds. Their mean forgetting decreases from ER's 0.22 to 0.05 and 0.04, respectively.

C is below A and D in all three seeds, supporting the use of a stable historical reference. Full achieves the highest mean FAR, 2.8 points above A. These comparisons describe mean performance.

In Expo-FT, Q ranks candidate actions at a given state, while V cannot distinguish them. Full versus B supplies the corresponding empirical comparison in RIPT.

We will include the component table and its interpretation in the main text, and report each variant's actual loss coefficients, per-seed results, and success matrices in the appendix.

Reviewer_AjcJOpenReview ↗

Rebuttal reply (1)

Thank you for the additional results and clarifications. I may be misunderstanding parts of your response. Thus, I would appreciate clarification on the following.

(1) For the theory, do you claim that Q retention is theoretically connected to the bound, or is the Expo-FT candidate-ranking explanation a method-specific motivation? You acknowledge that Q distillation does not directly control the advantage term. What, then, is the nontrivial conclusion of the bound beyond established performance differences and occupancy analyses, and is there a testable prediction supporting Q retention?

I also have a few remaining questions:

(2) Could you clarify the intuition and naming of the divergence and bound quantities, and explain the motivation for selecting the baselines? Your response says you will add these explanations, but I could not find them in the attached paper.

(3) Could you clarify how the uncertainty values in Tab. 2 are reported for each result group, including which are standard deviations and which are standard errors?

(4) For the new component study, could you clarify which loss components are isolated, whether the comparison conditions and coefficients are matched, and how broadly the findings apply across the different backends? The new table and the promised per-seed results are not in the attached paper. Am I missing something?

Thanks again!

AuthorsOpenReview ↗

Clarifications on Theory, Baselines, Statistical Metrics, and Ablation Results

Dear Reviewer AjcJ,

We appreciate the opportunity to clarify our theoretical framework and experimental rigor. Please accept our apologies for the confusion regarding the manuscript version; while our previous response outlined our planned revisions, these updates were not yet reflected in the PDF available for download.

(1) Theoretical Insights and Q-Retention

Theorem 1 as a Foundation for CRL-VLA: Theorem 1 formally characterizes the performance sensitivity of the continual learning objective. It applies the goal-conditioned Performance Difference Lemma (PDL) and state-occupancy bounds to reveal that old-task performance drop ∣Jold(πnew)−Jold(πold)∣|J_{\text{old}}(\pi_{\text{new}}) - J_{\text{old}}(\pi_{\text{old}})| is governed by the product of policy divergence DoldD_{\text{old}} and the advantage magnitude Mold=sup⁡s,a∣Agπold(s,a)∣M_{\text{old}} = \sup_{s, a} |A_g^{\pi_{\text{old}}}(s, a)|.

  • Theoretical Insight: This bound demonstrates that stability is not solely a function of policy regularization. It provides the theoretical justification for our dual-critic framework: while policy regularization controls behavioral divergence, historical value supervision (QQ-retention) is necessary to stabilize the shared representation against value-landscape degradation, ensuring that the advantage function remains accurate across task transitions.

Motivation for Q-Retention: We adopt QQ-retention because Q(s,a,g)Q(s, a, g) provides a rich, action-conditioned supervision signal naturally aligned with the backbone's objective of evaluating action quality. Unlike state-value functions (VV), QQ-functions are intrinsically suited to guide action ranking, which is central to the efficacy of CRL-VLA.

Testable Prediction: Our framework predicts that models relying solely on policy regularization will fail to maintain action optimality when the value landscape shifts. We validated this through our ablation: removing the value retention critic leads to a performance drop, confirming that QQ-retention is critical for stabilizing the shared representation backbone.

(2) Intuition and Baseline Selection

Physical Intuition:

Behavioral drift. DoldD_{\text{old}} and DnewD_{\text{new}} quantify the behavioral drift between policies, weighted by task-specific goal distributions and state-occupancy measures. For old tasks,

Dold2=2Eg∼pold,,s∼dgπold[DKL(πnew∥πold)].D_{\text{old}}^2 = 2 \mathbb{E}_{g \sim p_{\text{old}},\\, s \sim d_g^{\pi_{\text{old}}}} \left[ D_{\text{KL}} \left( \pi_{\text{new}} \Vert \pi_{\text{old}} \right) \right].

Advantage magnitude. MoldM_{\text{old}} measures the sensitivity of task performance to action selection errors in historical tasks:

Mold=Eg∼pold[sup⁡s,a∣Agπold(s,a)∣2].M_{\text{old}} = \sqrt{ \mathbb{E}_{g \sim p_{\text{old}}} \left[ \sup_{s,a} \left| A_g^{\pi_{\text{old}}}(s,a) \right|^2 \right] }.

Horizon factors. The factor 1(1−γ)2\frac{1}{(1-\gamma)^2} for old tasks reflects the compounded occupancy shift error when evaluating historical tasks under a reference occupancy measure, compared to the factor 11−γ\frac{1}{1-\gamma} for new tasks. Baseline Selection:

Our baselines include: (i) Continual RL (SL, ER, EWC, DER) to establish paradigms; (ii) Rigorous Benchmarks (Hu et al., 2026) to provide head-to-head comparisons under identical protocols (LoRA, backbone, environments); and (iii) Continual VLA Methods (Stellar-VLA, Info-VLA) to demonstrate relative performance in imitation learning.

(3) Uncertainty Metrics in Table 2

All results in Table 2 represent the Mean ± Standard Deviation (SD) across N=3N=3 independent random seeds. We apologize for the ambiguity and will explicitly standardize these labels in the final revision to ensure statistical clarity.

(4) Component Ablation and Results

Rigorous Ablation Setup: We ablated components on the LIBERO-Spatial sequence using consistent training budgets, replay buffers, and matching hyperparameter configurations (β=0.01,λKL=10−6,η=1.0\beta=0.01, \lambda_{\text{KL}}=10^{-6}, \eta=1.0) across all variants.

Ablation Results (FAR ↑):

VariantConfigurationMean ± SD
ER (Replay only)Baseline Replay0.611 ± 0.167
A (Policy KL only)Policy Regularization0.889 ± 0.083
B (V-Value + KL)State-Value Retention0.861 ± 0.111
C (Single Critic)Adaptive Critic0.667 ± 0.111
D (Q-Retention only)Action-Value Retention0.868 ± 0.063
CRL-VLA (Full)Dual-Critic + KL0.917 ± 0.056
  • Evidence for Necessity: The performance gap between our Full method and the adaptive single-critic (Variant C) demonstrates the vital role of the frozen retention critic in preventing representation interference.
  • Robustness: QQ-retention provides a robust signal for action-conditioned alignment, which is the cornerstone of our VLA optimization process.
Reviewer_kEhZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies continual RL post-training of VLAs, presenting goal-conditioned performance-change bounds and a framework combining historical value retention, current-task value learning, and policy regularization. Backend-specific implementations use online Q distillation, offline value supervision, or dual-critic action selection, evaluated in simulation and on real robots. The empirical gains are valuable, but the theory's standalone contribution and explanatory connection to the method are limited, while methodological gains beyond CLEAR remain insufficiently established by matched comparisons and ablations.

Critical strengths

The problem is practically relevant, with substantial gains in some settings: FAR increases from ER's 0.60 to 0.98 on perturbed LIBERO-Spatial, and final full-task success improves from 34% to 58% in the five-task offline robot study.

Critical weaknesses
  1. Theorem 1 largely follows classical performance-difference and occupancy bounds, with no obvious derivation error. However, the nontrivial insight beyond goal conditioning is unclear. Its advantage magnitude belongs to a fixed reference policy and is not directly controlled by sampled Q distillation; empirical KL distributions also need not match the theorem. Although correctly framed as motivation rather than a guarantee, the analysis offers limited additional insight into the stability–plasticity trade-off and does not clearly justify Q retention over V cloning.

  2. CLEAR (Rolnick et al., 2019) already combines historical V cloning with policy KL. The closest online variant mainly adds action-conditioned Q, a stage-frozen teacher, and VLA integration.The novelty and significance of these modifications need to be more clearly articulated and justified. The current comparisons and ablations are insufficient to isolate the contributions of these differences or establish their added value over existing policy/value-retention methods.

Questions for authors
  1. What nontrivial conclusion does Theorem 1 add beyond established performance-difference and occupancy analyses? With the reference policy fixed, how does Q distillation on historical samples affect the performance sensitivity characterized by the theorem? Can you provide an explicit theoretical connection or a testable empirical prediction supporting this Q-retention mechanism beyond general motivation?
  2. CLEAR comparison and ablations.
  3. For the flow-based π0.5 experiments in Section 5.1, how are the PPO likelihood ratio in Eq. (8) and historical policy KL computed?
2 comments
AuthorsOpenReview ↗

Response to Reviewer kEhZ: Theory, Component Comparisons, and Implementation

Thank you for recognizing the practical relevance and empirical gains. We provide new experiments and implementation details below.

W1/Q1: Role of the theory. Theorem 1 uses established performance-difference and occupancy tools to separate old- and new-task sensitivity under their respective goal/state distributions. It motivates separate retention and adaptation objectives. As you note, its advantage magnitude belongs to a fixed reference policy; sampled Q distillation does not directly minimize that quantity.

The connection is backend-specific. In Expo-FT, retained Q-values directly influence candidate selection: Q(s,a,g)Q(s,a,g) distinguishes actions at the same state, whereas V(s,g)V(s,g) cannot rank them. This is the structural reason for action-conditioned retention. In RIPT, actor advantages come from RLOO, while value supervision is auxiliary. In one replay-only run, historical-buffer KL reached 31.98 and first-task success fell from 1.00 to 0.06. This illustrates co-occurring drift and forgetting, rather than quantitatively validating the bound.

W2/Q2: CLEAR and component contributions. CLEAR is the cited Rolnick et al. (2019) method. B below implements CLEAR-style V retention plus policy KL within our training framework; it is not a reproduction of native CLEAR.

We use OpenVLA-OFT/RIPT, the three-task LIBERO-Spatial sequence in App. E.2, and stage-wise visual perturbations. All variants share task order, training budget, and replay of the previous stage's latest 30 successful trajectories. Each seen task receives 16 evaluations after each stage. Results are mean±SD over three seeds.

VariantRetention configurationFAR ↑
ERReplay only0.611±0.18
CSingle critic on old/new data + historical KL0.667±0.11
BFrozen-teacher V distillation + historical KL0.861±0.11
DFrozen-teacher Q distillation, no historical KL0.868±0.06
AHistorical KL, no value distillation0.889±0.08
FullFrozen-teacher Q distillation + historical KL0.917±0.07

A and D improve mean FAR over ER by 27.8 and 25.7 percentage points, respectively, with consistent directions across all three seeds. C is below A and D in all three seeds, supporting a stable historical reference. Full has the highest mean FAR. Full versus B (0.917 versus 0.861) compares Q/V retention; candidate ranking remains the main rationale. We will add this table to the main text and report each variant's actual loss coefficients separately.

Additional benchmarks. After review, we added CRL-VLA evaluations Long. All reference baselines from the expanded Table 2 are included; their values are reproduced from the cited sources. Values are in %, with AVG measuring final success, NBT retention (lower is better), and FWT forward transfer.

LIBERO-Long

MethodAVG ↑NBT ↓FWT ↑
SL89.8±0.9−2.4±1.00.5±0.1
EWC86.6±0.30.8±1.33.0±1.3
ER88.8±0.8−0.2±1.7−1.1±0.6
DER87.6±0.40.7±0.80.7±0.2
DWE88.4±0.50.0±0.00.0±0.0
SLCA86.9±0.6−1.3±1.0−0.2±0.3
RETAIN86.2±0.91.6±1.01.0±1.2
MTL (Oracle)90.5±0.8——
CRL-VLA (new)94.0±3.2−2.3±7.4−1.5±3.8

On Long, mean AVG exceeds SL by 4.2 points, with similar NBT (−2.3 versus −2.4). On Object, relative to ER, AVG increases by 1.86 points and NBT decreases by 4.17 points. These extend benchmark coverage; the component study addresses mechanism contributions.

Q3: PPO ratio and historical KL. LIBERO uses OpenVLA-OFT/RIPT's Laplace action distribution. The ratio is the exponential of current minus rollout-policy log-probabilities. Eq. 11 uses DKL(π−∥πθ)D_{\mathrm{KL}}(\pi^-\|\pi_\theta), summed over action dimensions and averaged over chunk positions.

For ManiSkill's flow-based π0.5\pi_{0.5}, PPO follows πRL: ρ=exp⁡(ℓθ−ℓb)\rho=\exp(\ell_\theta-\ell_b), evaluated on the same sampled denoising variables under current and rollout policies. These are tractable stochastic-denoising likelihoods, not the final-action marginal likelihood. Historical regularization is separate: we sample an action from the frozen reference flow, evaluate both velocity heads at a fixed denoising step, form Laplace approximations, and apply the analytic KL used by RIPT. This is a Laplace-based historical surrogate. We will document both computations in the appendix.c

I appreciate the authors' responses and the additional results. My main concern remains the limited theory and methodological novelty. In addition, the use of on-policy PPO may pose challenges for real-world deployment, as it relies on on-policy sampling of trajectories and typically requires a large number of interaction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.