acceptodds
Withdrawn from ICLR 2027

AEGIS: Action-Endpoint Geometry for Nuisance-Invariant and State-Equivariant VLA Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Robust vision-language-action (VLA) models must remain stable under task-equivalent language and task-irrelevant visual changes while adapting their actions to control-relevant changes in robot state, object configuration, or task progress. Per-sample mean squared error (MSE) supervision anchors individual predictions to demonstrations, but its fixed quadratic metric does not adapt residual weighting or explicitly constrain how predictions should relate across these conditions. We present AEGIS, a demonstration-anchored action-endpoint geometry for VLA models that predict action chunks from single-timestep observations. It combines three complementary objectives in a shared action field: Bounded Heteroscedastic Flow Matching (BHFM) uses bounded heteroscedastic scales to adapt residual weighting; Nuisance Endpoint Invariance (NEI) encourages endpoint agreement under action-preserving language and visual variations; and State Endpoint Equivariance (SEE) encourages local endpoint displacements to match demonstrated action differences across progress-matched states. AEGIS achieves 91.58% success on LIBERO-Plus and 28.24% on RoboCasa365, improving over the π₀.₅ baseline by 6.74 and 7.24 percentage points, respectively. Across three real-world manipulation tasks, it achieves 57.33% average success, a gain of 12.33 percentage points over the baseline. These results demonstrate improved task performance in both simulated and physical environments.

Primary area:
applications to robotics, autonomy, planning
Keywords:
VLA, Vision-Language-Action (VLA) models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Sv4dOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes AEGIS, an action-endpoint geometry learning framework for improving VLA robustness. The authors argue that robust VLA policies should preserve their actions under task-irrelevant language and visual changes while responding appropriately to control-relevant changes in robot states, object configurations, and task progress. AEGIS introduces three training objectives based on π0.5\pi_{0.5}. BHFM uses bounded heteroscedastic modeling to adjust action prediction residual weights. NEI encourages endpoint agreement under task-equivalent inputs. SEE aligns endpoint displacements between progress-matched states with demonstrated action differences. Experiments on two simulation benchmarks and real-world robot tasks show clear improvements over the baseline.

Critical strengths
  1. The authors attempt to address two complementary requirements within a unified training framework, namely invariance to task-irrelevant changes and sensitivity to control-relevant changes. The research objective is well motivated. The method does not require changes to the original inference procedure, which is advantageous for practical applications. 
  2. The experimental evaluation is reasonably comprehensive, covering two simulation benchmarks, real-world robot experiments, results across three random seeds for some experiments, and a complete component ablation. The method achieves notable improvements under Camera, Language, and Noise perturbations. These results provide meaningful empirical support for the effectiveness of the proposed method.
Critical weaknesses
  1. The authors identify the lack of explicit relational constraints as a key limitation of standard MSE. However, with data augmentation or additional demonstrations under perturbed conditions, accurate per-sample predictions can already satisfy the relationships required by NEI and SEE. Equations (10) and (13) further show that both objectives essentially constrain differences between velocity prediction errors. The paper does not sufficiently establish the specific limitations of MSE or explain why the proposed relational regularizers address them. SEE also treats demonstrated action differences between progress-matched states from different trajectories as state-equivariant displacements. Such differences may reflect different action modes or demonstration biases, and their control relevance is not sufficiently justified.

  2. The endpoint estimate eθ=xt−tvθe_\theta=x_t-tv_\theta is a one-step extrapolation based on the current velocity and does not necessarily equal the final action generated by integrating the full flow ODE. The paper lacks evidence that constraints on extrapolated endpoints reflect the properties of actually generated actions. The MSE-optimal one-step endpoint estimator also has a conditional-mean interpretation. The potential effects of additional endpoint constraints on the multimodal action generation capability of flow-based policies are not discussed.

  3. The overall performance improvements are clear, but on LIBERO-Plus, the largest gains come from Camera, Language, and Noise perturbations. The full model does not outperform the baseline under Robot or Layout variations. Adding SEE to BHFM+NEI improves average success by only 0.40 percentage points. The endpoint diagnostics are also closely related to the training objectives and do not sufficiently demonstrate improved state equivariance in actually generated actions.

  4. The fairness of the baselines deserves further examination. The paper lacks a standard MSE baseline trained with the same paired data and additional views as AEGIS. BHFM is also not compared with simpler residual-reweighting methods. These missing comparisons make it difficult to attribute the performance improvements to the proposed objectives.

  5. Figures 1 and 2 are overly crowded, with some text, labels, and axes difficult to read. Some annotations and formatting also appear unusual, raising concerns about whether generative AI tools may have been used in figure preparation beyond what is disclosed in the AI Use Statement. I would appreciate clarification from the authors on this point. Additionally, some diagnostic tables are not presented clearly, making it difficult to understand the comparison targets and the conclusions supported by the results.

Questions for authors
  1. I recommend that the authors consider PAIR-VLA [1]. This work also discusses selective responses to task-irrelevant and control-relevant changes, although it follows a different technical approach. What broader scientific insight does AEGIS aim to establish? Why is action-endpoint geometry an appropriate framework for this problem rather than simply a combination of three loss functions?

  2. Could the authors provide additional evidence showing under what conditions standard MSE fails to learn the relationships required by NEI and SEE, and what specific problems the additional constraints address? Since NEI and SEE can be expressed as constraints on differences between velocity prediction errors, what is their unique contribution beyond standard supervised learning? Why should demonstrated action differences between progress-matched states from different trajectories represent meaningful state-equivariant displacements rather than demonstration biases or pairing errors?

  3. What is the mathematical relationship between extrapolated endpoints and the final actions generated by integrating the full flow ODE? Why should constraints on the former effectively influence the latter? Given the conditional-mean interpretation of the MSE-optimal one-step endpoint estimator, could additional endpoint-level constraints introduce a bias toward the mean and reduce the ability of flow-based policies to represent multiple valid action modes? I would appreciate a theoretical explanation or direct empirical evidence.

  4. The current results mainly demonstrate nuisance robustness, while SEE provides limited incremental gains. What direct evidence shows that the model responds more appropriately to control-relevant changes, rather than simply reducing prediction differences across inputs? Can improvements in endpoint diagnostics be linked to actual generated actions or closed-loop success rates?

  5. Could the authors provide a standard MSE baseline using the same paired data and additional views to isolate the benefits of relational regularization? Could BHFM also be compared with simpler robust regression or residual-reweighting methods?

[1] What to Ignore, What to React: Visually Robust RL Fine-Tuning of VLA Models.

Reviewer_GQcVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents AEGIS, a training method built on π0.5\pi_{0.5} that consists of three objectives: BHFM, NEI, and SEE. BHFM reweights flow-matching residuals using bounded heteroscedastic scales. NEI encourages the same demonstration to produce consistent action endpoints under instruction paraphrases or visual augmentations. SEE requires the difference between predicted endpoints from two progress-matched demonstrations of the same task to agree with the corresponding difference between demonstrated actions. All three objectives are used only during training and do not change the inference interface.

Compared with the authors' reproduced π0.5\pi_{0.5} baseline, AEGIS improves performance by 6.74 percentage points on LIBERO-Plus, 7.24 points on RoboCasa365, and 12.33 points on average across three real-world tasks. The paper also evaluates all eight combinations of the three components on LIBERO-Plus and reports simulation results for seeds 7, 17, and 27.

Critical strengths
  1. BHFM, NEI, and SEE are training-only objectives. They neither change the inference interface nor add deployment-time cost.
  2. The experiments cover two simulation benchmarks and three real-world tasks. AEGIS outperforms the authors' reproduced π0.5\pi_{0.5} baseline in all of these settings.
  3. The experimental reporting is fairly comprehensive: simulation results include means and standard deviations over three seeds, and the LIBERO-Plus study ablates all combinations of the three components.
Critical weaknesses
  1. The gains from the proposed objectives are not separated from additional computation and view-level supervision

NEI and SEE process more views per update than the baseline (p.4 l.181–191; App. p.1 l.044–049), but the paper does not clarify whether the global batch size of 256 counts samples, pairs, or expanded views. The current ablation therefore cannot separate the effects of the proposed losses from those of additional supervision and computation.

This is particularly relevant to the language results because NEI introduces an additional paraphrased instruction. RoVLA (Luo et al., 2026) shows that paraphrase augmentation alone can substantially improve language performance. Cross-View Action Consistency (Huang et al., 2026) provides a control that holds the paired data fixed while switching the consistency loss on and off. The authors should report the training cost and GPU-hours for each configuration and add controls matched for both view count and computation.

  1. The experiments do not establish whether the robustness gains extend beyond the training distribution

The paper does not identify the LIBERO-Plus training split or report the overlap between training paraphrases and evaluation instructions. RoVLA (Luo et al., 2026) states that the benchmark provides a perturbation-augmented training set with 15,874 demonstrations. If AEGIS uses this split, some evaluated perturbation families may already be present during training.

RoboCasa365 results are reported only at the split level, and the real-world experiments do not explicitly test variations excluded from training. The current evidence therefore supports robustness to perturbations covered during training, but not broad generalization to unseen variations.

  1. The distinctions from prior methods are insufficiently established, while the more distinctive SEE component has relatively weak evidence

BHFM follows established heteroscedastic weighting. EDM2 (Karras et al., 2024) uses the same error/exp⁡(s)+s\mathrm{error}/\exp(s)+s form. BHFM adds per-sample, per-element scales, bounds, and an MSE anchor, but is not compared with a simpler scale that depends only on flow time tt.

NEI and Cross-View Action Consistency (Huang et al., 2026) both use π0.5\pi_{0.5}, share tt and ϵ\epsilon across paired views, and enforce agreement between their predicted action flows. NEI mainly replaces velocity L2L_2 with a tt-weighted endpoint Huber loss and adds language pairs. Cross-View is a contemporaneous arXiv preprint and should not by itself determine the decision, but the paper should clearly explain the difference.

SEE is the most distinctive component, yet adding it to the otherwise complete model improves average success by only 0.40 points. The gains mainly appear in Language, Background, and Noise rather than Robot and Layout. Its target may also mix state differences with demonstration speed, style, and phase mismatch, while the pair audit covers only 40 examples. The current evidence is insufficient to support the claimed state-equivariant interpretation.

  1. All experiments use π0.5\pi_{0.5}, so cross-model applicability is not demonstrated

All experiments use π0.5\pi_{0.5}. Cross-dataset gains do not establish transfer to other VLA architectures, especially because BHFM requires an additional scale head and the endpoint definition depends on the current linear flow path. The main result should be validated on another flow-based VLA; otherwise, the claimed scope should be limited to fine-tuning π0.5\pi_{0.5}.

  1. The real-world experimental setup is underspecified

Section 4.3 provides little beyond the task names and success counts. The supplementary video appears to show two different robot arms being used for different tasks, but the paper does not identify their models or map the tasks to the platforms. For each setup, the authors should report the robot arm, gripper, camera configuration, control frequency, action space, number of demonstrations, and training recipe, as well as the initial-state randomization, success criteria, and number of independent training seeds.

Related work links:

Questions for authors
  • Which feature layer is used by the BHFM scale head? How is the head initialized, and do its gradients propagate into the backbone? Please also report the distribution of ss across flow time and action dimensions, and compare it with a weighting scheme that depends only on tt.
  • Does the global batch size of 256 count samples, pairs, or expanded views? Please report the number of forward passes, training updates, starting checkpoint, wall-clock time, and GPU-hours for each configuration. A control matched for both view count and computation is needed to separate the effect of paired augmentation from that of the consistency loss.
  • Which LIBERO-Plus training split is used? Do the training demonstrations already contain the evaluated perturbation types, and do the GPT-5.5 paraphrases overlap with the evaluation instructions?
  • Some published methods in Table 1 use numbers taken directly from their original papers, whereas the models in this work undergo LIBERO pretraining and LIBERO-Plus alignment. Any differences in training data should be clearly marked in the table.
  • Why does adding SEE to the otherwise complete model mainly improve Language, Background, and Noise rather than Robot and Layout? Please also clarify whether actions are absolute poses or delta commands, and explain how progress matching handles differences in demonstration speed and phase.
  • Can the main benefits of AEGIS be reproduced on another flow-based VLA? If not, the claimed scope should be explicitly limited to π0.5\pi_{0.5}.
  • The method introduces several hyperparameters without sensitivity analysis. At minimum, the paper should examine λNEI\lambda_{\mathrm{NEI}}, λSEE\lambda_{\mathrm{SEE}}, and the Huber threshold.
  • RoboCasa365 results are reported only as aggregates over three splits. Please provide per-task and per-seed success counts and rates for both π0.5\pi_{0.5} and AEGIS across all 50 tasks, especially the 16 Composite-Unseen tasks, to show whether the improvement is broad or concentrated in a small subset.
  • Which tasks correspond to each of the two robot arms shown in the supplementary video? For each platform, please report the robot and gripper models, camera configuration, control frequency, action space, number of demonstrations, training recipe, initial-state randomization, and success criteria, together with the number of independent training runs, checkpoint-selection procedure, and confidence intervals.
  • Are seeds 7, 17, and 27 independent training seeds or evaluation seeds?
  • The 73.5% in Figure 1(a) and the 73.3% in Figure 1(b) refer to different metrics. The caption should clarify that panel (b) reports the state-pair displacement residual.
Reviewer_ghkLOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes AEGIS, a training framework for VLA models that improves robustness to task-irrelevant perturbations while preserving sensitivity to control-relevant state changes. Built on π0.5, AEGIS introduces three objectives: BHFM for adaptive residual weighting, NEI for action consistency under equivalent inputs, and SEE for preserving demonstrated action differences across states. Experiments show overall improvements across simulation and real-world tasks, although ablation results suggest that SEE provides limited additional gains and does not consistently improve robustness to state-related perturbations.

Critical strengths
  1. The method addresses a practical distinction between invariance to irrelevant perturbations and sensitivity to control-relevant state changes, without changing the baseline inference procedure.
  2. AEGIS achieves substantial aggregate improvements on LIBERO-Plus (+6.74 pp), RoboCasa365 (+7.24 pp), and real-world tasks (+12.33 pp), supported by three-seed simulation results and eight-way component ablations.
Critical weaknesses
  1. SEE is mathematically a cross-sample residual-consistency regularizer. Although this can be useful, the distinctive contribution of the endpoint-geometry formulation beyond such regularization remains insufficiently justified.
  2. Full AEGIS slightly underperforms the baseline on Robot and Layout perturbations, while SEE adds only 0.40 pp over BHFM + NEI. The evidence for improved adaptation to control-relevant state changes is therefore mixed.
  3. State-pair construction and diversity are insufficiently characterized. The 39/40 broad-stage agreement audit does not establish adequate state coverage or distinguish state-dependent action differences from demonstration variability.
  4. The ablations do not clearly establish matched augmentation, sampling, and computational budgets. Consequently, improvements cannot be cleanly attributed to the proposed losses rather than additional data exposure or computation.
Questions for authors
  1. What distinctive benefit does the endpoint formulation provide beyond pairwise residual regularization?
  2. Why do SEE’s standalone Robot/Layout gains largely disappear in full AEGIS?
  3. How are progress-matched pairs constructed, and how diverse are their control-relevant states?
  4. Are augmentations, sampled states, evaluated views, and training budgets matched across all ablations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.