acceptodds
Under review as a conference paper at ICLR 2027

Bellman-Corrected Dualization for Safe Vision-Language-Action Models

Abstract

Vision-language-action (VLA) models offer a promising framework for generalist robot policies, but their deployment in long-horizon embodied environments faces critical safety challenges. Standard post-training approaches struggle to maintain a stable reward–safety trade-off under policy-induced distribution shifts, as evolving policies alter trajectory occupancy and invalidate static constrained alignment. We propose Dualized-SafeVLA, a Bellman-corrected dualization framework that constructs a local one-shot dual under frozen occupancy, producing a low-dimensional dual, a closed-form target policy, and trust-region-safe updates. This approach systematically aligns VLA with safety constraints while maintaining task performance under evolving distributions. On safety-critical long-horizon mobile manipulation benchmarks, Dualized-SafeVLA improves task success by 9.04 points and reduces cumulative safety cost by 79.9% compared to a clean-environment SafeVLA baseline under matched-step evaluation, and further achieves 6.03 points SR improvement with 93.3% cost reduction under a 2.1M-step budget, demonstrating a practical and robust path toward safer VLA post-training.

Primary area:
reinforcement learning
Keywords:
Vision-Language-Action Models, Safe Reinforcement Learning, Constrained Markov Decision Processes, Bellman-Corrected Dualization, Safety Alignment, Robot Learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_f12tOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes Dualized-SafeVLA, a safety-alignment framework for vision-language-action models that addresses changes in state visitation as robot policies evolve. Its main contribution is a Bellman-aware dual formulation and a practical local update procedure that freezes the current occupancy distribution, solves a low-dimensional safety dual, and distills the resulting closed-form target policy under a trust-region constraint. The paper provides theoretical derivations and a bound connecting local constraint satisfaction to actual safety, alongside evaluations on Safety-CHORES, component ablations, and analyses of difficult tasks and solver convergence. It reports a 9.04-percentage-point success-rate improvement and a 79.9% reduction in cumulative safety cost over SafeVLA at approximately matched training steps, although differences in training settings and checkpoint selection limit causal attribution of these gains.

Critical strengths
  • The paper addresses a concrete and consequential problem: improving task performance while controlling explicit object and robot safety costs during VLA post-training. Accounting for changes in the states visited by an evolving policy is relevant to maintaining safety in closed-loop robotic execution.
  • The method is accompanied by theoretical analysis connecting the constrained formulation to a Bellman-aware dual, an explicit convex local dual, and a closed-form target policy. The trust-region analysis also states conditions under which local constraint satisfaction implies satisfaction of the original safety constraints.
Critical weaknesses

Summary of major concerns. I find it difficult to identify the precise contribution of the proposed method, primarily because of its presentation. The problem being addressed is not explained clearly enough, the figures are difficult to interpret, and the notation and newly introduced concepts lack sufficient definition and context. These issues obscure the distinction between the proposed approach and existing methods. I strongly recommend a thorough rewrite that clearly states the unresolved problem, explains the key technical ideas, and establishes what the method contributes beyond prior work.

  • The distinction from prior constrained VLA training needs sharper justification. The introduction motivates the method by arguing that a globally fixed dual variable is generally insufficient under policy-induced occupancy shift. However, SafeVLA already updates its Lagrange multiplier iteratively. This argument therefore explains a limitation of directly transferring static one-shot alignment, rather than a limitation of SafeVLA itself. Although the paper identifies Bellman consistency and local dualization as its contributions, it should more clearly distinguish these elements from existing adaptive primal–dual updates and explain what additional capability or efficiency they provide.
  • The central formulation and its practical implications are insufficiently explained in the main text. Several presentation choices make the contribution difficult to follow:
    • Figure 1(a) illustrates a change in occupancy but does not clearly demonstrate why the proposed fixed-dual transfer fails. The figure also uses occupancy notation ρ\rho before its formal introduction in Section 4.2. A compact example, such as the counterexample in Appendix E, would make the issue more concrete.
    • The discussion around lines 154–160 would benefit from a clearer connection between occupancy shift and specific robotic safety failures (especially the concept of "policy-induced occupancy shift"). Table 2 provides relevant ablations, but the operational definitions of “remove Bellman correction” and “freeze occupancy” need clarification to establish what those comparisons isolate.
    • Section 4.1 repeats much of the preceding problem formulation: Eqs. (6), (7), (8), and (9) restate Eqs. (1), (2), (4), and (5), respectively. That space could instead explain Bellman flow consistency and why it is needed.
    • Around lines 208–210, the paper announces an occupancy reformulation and a dual over safety multipliers and state-flow potentials, but presents only the resulting policy in Eq. (10). Because this formulation is central to the claimed contribution, the main text should include the occupancy constraints and explicit dual optimization problem currently deferred to Appendices D and F.

Minor Comments

  • Please use parenthetical citations where required by the sentence structure; several citations currently appear as detached author–year phrases. (\citep instead of \cite)
  • I could not locate a dedicated AI-use disclosure section in the manuscript. Please ensure that the submission includes the statement required by the ICLR 2027 author policy.
  • The caption of Table 3 contains an unresolved reference, “Table ??”. Please correct it.
Questions for authors
  • Algorithms 1 and 2 substantially overlap. Please consolidate them or clearly explain their distinct purposes.
  • SafeVLA already updates its multiplier and uses reward and cost advantage estimates. What specific operation in your implemented update constitutes the additional Bellman correction, and how does it differ from SafeVLA beyond solving the frozen-occupancy dual before policy distillation? In particular, what exactly is changed in the “remove Bellman correction” ablation?
  • I understand that the proposed procedure retains outer occupancy-refresh iterations while replacing repeated policy–multiplier interleaving within each frozen-occupancy region with a dual solve followed by policy distillation. What are the actual numbers of dual iterations and policy optimization steps per region, and what computational savings does this achieve relative to SafeVLA under comparable conditions?
Reviewer_BZW5OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Dualized-SafeVLA, which applies local frozen-occupancy dualization and weighted imitation distillation to language-conditioned CMDPs for safe VLA post-training. While the motivation for addressing policy-dependent occupancy shifts is reasonable, the paper has serious issues with its technical quality and research integrity.

Critical strengths

While applying two-stage local dualization and weighted distillation to VLA safety alignment is a reasonable direction, the paper is not ready for publication. It suffers from severe mathematical inconsistencies and consists of synthetic/placeholder figures.

Critical weaknesses
  1. The paper presents policy-dependent occupancy shift as a major challenge, but this is already well known in reinforcement learning and has been studied in safe RL methods.

  2. The comparison with static LLM alignment is also not convincing. The authors should explain what is actually missing from existing methods and what their approach adds beyond prior work.

  3. Section 3 and Section 4.1 repeat the same text and equations,

  4. Algorithms 1 and 2 on pages 5–6 seem to be duplicates. Doesn't make sense to me

  5. Theorem statements for the proofs in Appendices G and J are missing.

  6. Figure 1 seems AI-generated and is difficult to read (unreadable labels in the boxes and distorted math notation/texts)

  7. Figure 3 also seems AI-generated and doesn't look like a real simulator output.

More issues

  1. (lines 585- 586) -"the May 4, 2026 baseline evaluations and the April 27, 2026 Final evaluation"-- why these dates? Seems like an AI response?
  2. (lines 591- 592) - "The comparison referred to as “matched-step” in the main text uses baseline@1808366 and Final@1705608" -- why these numbers? The same are used in Table 4.
  3. Ref missing Table 3. ("..separately in Table ??.")

Violation of ICLR 2027 AI disclosure policy: The paper is missing the mandatory AI-use statement required by ICLR 2027 (https://iclr.cc/Conferences/2027/AuthorGuidelines).

Questions for authors

N/A

Reviewer_644pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Dualized-SafeVLA. The authors formulate a global dual using occupancy measures and Bellman flow constraints, then freeze the current state occupancy to construct a local constrained problem with reward and cost advantages. Each outer iteration solves a low-dimensional convex dual, constructs an exponentially reweighted target policy, and fits a parametric policy through weighted imitation with trust-region control. The paper also derives an occupancy-surrogate error bound and shows improvements in success and cumulative safety cost over SafeVLA on Safety-CHORES.

Critical strengths
  1. Equation (8) models task reward and object/robot safety costs separately, and Eqs. (14)-(16) give a local dual and closed-form target policy under frozen occupancy. This structure makes the effect of both safety costs on action weights inspectable and provides a concrete route to implementing constrained VLA updates.
  2. The authors considered an important research topic, analyzed and revealed the limitations of the algorithms from previous work. This is a good starting point and valuable for subsequent algorithm design and model training.
  3. Tables 4-5 in Appendix C report means, paired success outcomes, task-level cost dispersion, bootstrap intervals, and checkpoint-selection limitations. The authors acknowledge different training histories, an intersection of 198 records for the approximate-step comparison, and a non-significant success-rate difference for the best-under-budget comparison. These disclosures help readers distinguish descriptive gains from controlled algorithmic effects and assess the scope of the available evidence.
Critical weaknesses
  1. FOCOPS §3.1, equations (4)-(8) already derives an exponential reward/cost-advantage target under the old state distribution and projects it into a parametric policy, while CVPO §3.2-3.3, equations (8)-(12) separates convex dual optimization from weighted supervised fitting, closely matching the central construction here. Replacing Q with A in equation (16) cancels state baselines during normalization and leaves the target unchanged. The VLA implementation and safety margin δ may add value, which requires comparison within the same training pipeline. Furthermore, equation (8) uses fixed reference π₀, whereas equation (14) uses current policy πₖ without deriving this objective change. Please provide detailed comparisons with these methods, controlled experiments, and a derivation connecting the local update to the actual global objective.
  2. It is still unclear how the limitations of this method extend to real robots, and whether its advantages can still hold. I noticed that the author lacks verification of this method on real-world robots and tasks, which is crucial for supporting the motivation of this paper and revealing potential limitations.
  3. Appendix C, lines 609-647, reports a 102758-step difference and different training histories/settings between the "matched-step" checkpoints, confounding attribution to the update method. Best-under-budget selection uses observed success without a verified independent validation split, and its 6.03-point SR gain has an interval of [−1.01, 13.07] with p=0.1263. Appendix K requires all-state KL control and post-distillation feasibility, yet Appendix N omits actual d, δ, ε and per-constraint checks, preventing verification that implemented updates meet these conditions.
Questions for authors
  1. After a component-wise comparison with FOCOPS/CVPO, what optimization problem, update, or guarantee is new here? Can a comparison within the same VLA implementation and budget demonstrate the benefit of that contribution?
  2. How does the fixed-π₀ regularizer in Eq. (8) lead to the moving-πₖ regularizer in Eq. (14)? Does the algorithm optimize the fixed-reference regularized objective, or perform proximal policy iteration for the original CMDP? This changes what the theory should establish.
  3. Please provide the actual d, δ, ε, cost scales, advantage estimation procedure, and post-distillation constraint/KL checks. How does finite-batch clipping/backtracking relate to Appendix K's all-state condition? How are the action sums and statewise normalization implemented?
  4. Can the authors provide comparisons with matched initialization, training settings, and budgets, checkpoint selection on an independent validation set, and multiple training seeds, together with precise definitions of the Table 2 ablations? Otherwise, will the main claims be restricted to the historical-checkpoint comparison supported by Appendix C?
  5. What are the record counts, overlaps, and SR/CC weights for hard25, hard49, and nonhard? How do the subgroup results aggregate to the overall results in Table 1? Can matched baseline runtime and update counts directly substantiate the efficiency claim?
  6. Can the author conduct experiments on real-world robots to demonstrate the potential for application in the real world or further clarify the limitations of this method?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.