acceptodds
Under review as a conference paper at ICLR 2027

CAFE-VLA: Controllable Avoidance with Field Encoding for Compact Vision-Language-Action Models

Abstract

We present CAFE-VLA (Controllable Avoidance with Field Encoding), a controllable obstacle-avoidance framework for compact vision-language-action (VLA) models. The CAFE-VLA policy is conditioned on two complementary inputs: an avoidance-strength scalar and an end-effector-centric 3D geometry field. The avoidance-strength scalar specifies the requested degree of avoidance, while the 3D geometry field represents local obstacle distances and clearance directions at a fixed set of probes. We inject the scalar condition into the action expert via Feature-wise Linear Modulation (FiLM), while the 3D geometry field conditions the policy through cross-attention to field tokens and visual prompting encoding obstacle distances. This structure-aware design only increases the parameter count by less than 5%. A sweep over the avoidance-strength scalar confirms monotonic control over obstacle clearance (Spearman ρ = 0.953), while classifier-free guidance enables stable extrapolation to a strength 50% above the training maximum. On risk-stratified SafeLIBERO, CAFE-VLA achieves the highest task success rate at every risk level, with an average success rate of 74.32%, compared with 70.34% for the strongest safety-aware baseline. Meanwhile, CAFE-VLA retains a task success rate of 82.54% on the original LIBERO benchmark, the highest among the compared safety-aware methods.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Controllable Obstacle Avoidance, 3D Representation for Robotics, Compact Vision-Language-Action Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eLBpOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents CAFE-VLA, a controllable obstacle-avoidance framework for compact Vision-Language-Action (VLA) models. The method introduces a continuous avoidance-strength scalar and an end-effector-centric 3D geometry field to guide action generation. The scalar is incorporated through FiLM, while the geometry field is injected via sparse cross-attention and visual prompting. Experiments on SafeLIBERO and real-world robotic manipulation demonstrate improved obstacle avoidance while largely preserving the original manipulation capabilities.

Critical strengths
  1. Practical Motivation. The paper addresses an important problem in robotic manipulation: enabling VLA models to adapt their obstacle-avoidance behavior to different safety requirements.

  2. Simple and Effective Design. The proposed framework integrates continuous avoidance control and explicit geometric guidance into a compact VLA model with limited parameter overhead. The experimental results demonstrate its effectiveness in improving obstacle avoidance while maintaining task success.

  3. Comprehensive Experimental Validation. The paper provides extensive experiments across different safety levels, including detailed ablation studies and real-world evaluations. The results consistently support the effectiveness of the proposed components.

Critical weaknesses
  1. Joint Training and Capability Preservation. Although the paper evaluates capability retention on the original LIBERO benchmark, it remains unclear whether jointly training on the original LIBERO data and the newly generated obstacle-avoidance demonstrations could further improve the performance trade-off. The authors are encouraged to investigate different data-mixing ratios and examine whether the proposed method can maintain its avoidance capabilities while better preserving general manipulation performance.

  2. Data Construction and Scalability. The proposed method requires additional obstacle-avoidance demonstrations and a geometric annotation pipeline to construct the training conditions. While the avoidance-strength labels can be generated automatically, the overall data preparation process introduces additional complexity. It would be valuable to further discuss how this pipeline could scale to larger and more diverse robotic datasets, particularly in real-world and zero-shot settings.

  3. Generalization Across VLA Backbones. Although the paper includes comparisons with several safety-aware methods, the proposed framework is primarily evaluated on SmolVLA. It would be beneficial to further validate its effectiveness on stronger VLA architectures, such as π0.5 and VLANeXt, under comparable training and evaluation settings. Such experiments would provide stronger evidence of the method's general applicability.

  4. Extension to World Action Models. The proposed framework may also be applicable to emerging World Action Models (WAMs). The authors are encouraged to explore its effectiveness on representative WAM architectures, such as Fast-WAM. This would help assess whether the proposed conditioning strategy can generalize beyond conventional VLA models and improve obstacle avoidance in world-model-based action generation.

Questions for authors

See weakness.

Reviewer_bkuiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CAFE-VLA, a controllable obstacle-avoidance framework designed for compact Vision-Language-Action (VLA) models. Built upon SmolVLA, the approach decouples obstacle-avoidance preferences from local geometry. It represents the desired avoidance intensity via a continuous scalar ss, which is then used to modulate the action experts through FiLM. Local geometric information—specifically obstacle distances and avoidance directions—is captured by 78 probes centered on the end-effector, covering both the current position and the short-term motion trajectory; this data is subsequently integrated into the policy via two pathways: sparse cross-attention and visual image prompting. To handle avoidance intensities beyond the training distribution, the framework employs Classifier-Free Guidance (CFG) for extrapolation, while simultaneously utilizing teacher-matching and a "clean-sample" branch to preserve the model's original operational capabilities. The method was evaluated on the risk-stratified SafeLIBERO benchmark, the original LIBERO benchmark, and a real-world PiPER robotic arm. CAFE-VLA achieved an average task success rate of 74.32%, outperforming the strongest safety-focused baseline (70.34%), while maintaining an 82.54% success rate on the original LIBERO benchmark—a performance level comparable to the base model. Experiments in controlled, single-obstacle scenarios demonstrate a strong monotonic relationship between the specified intensity and the resulting clearance distance, with the framework supporting control at intensities beyond those encountered during training.

Critical strengths

The paper goes beyond mere collision reduction; it allows a single policy to dynamically adjust its obstacle-avoidance intensity based on the consequences of potential contact and the available space, effectively capturing the trade-offs between safety margins, task success rates, and operational efficiency.

By employing "probes" that cover the end-effector’s envelope and performing look-ahead sampling along the action sequence, the method directly provides distance and directional information between the end-effector and obstacles. This alleviates the burden on compact models to infer obstacle-avoidance relationships from the full geometric complexity of the scene.

The architecture utilizes FiLM for global scalar modulation, cross-attention to access probe-level geometric data, and visual prompting for alignment within the image space. Ablation studies confirm the complementary nature of these two geometric pathways, with the added parameters accounting for only about 1.32% of the total.

In addition to a safety-focused baseline—which leverages precise obstacle geometry in simulation, thereby lending credibility to the comparisons—the paper evaluates various geometric representations, conditioning mechanisms, and the use of fixed versus adaptive avoidance intensities. It further provides insights into performance variance across three independent training runs, sensitivity to geometric hyperparameters, computational overhead, and examples of actual failure cases.

The "retention branch" mechanism restores the success rate on the LIBERO benchmark from 80.08% (without the mechanism) to 82.54%, bringing it close to the 82.92% achieved by the base model. The authors also transparently acknowledge several limitations, including the assumption of quasi-static obstacles, the localized coverage of the end-effector, and the restricted extrapolation range for Classifier-Free Guidance (CFG).

Critical weaknesses

The core monotonicity result is derived from a controlled, single-obstacle scenario, with correlation coefficients calculated based on average clearances at various intensity levels; this does not necessarily demonstrate control reliability in complex tasks or along a single execution trajectory. Given that different obstacle layouts, narrow passages, and multi-obstacle environments may involve varying ranges of "effective intensity," the justification for uniformly applying the condition s≤15s \leq 15 requires further substantiation.

During training, "probes" are constructed using demonstrated future joint targets, whereas during deployment, the model relies on its most recently predicted action sequence. While joint-space perturbations are introduced, it remains unclear whether these are sufficient to account for the systematic prediction errors inherent in the actual policy. Furthermore, since the probes are based on raw (pre-aggregation) actions—while the actions actually executed undergo temporal aggregation—the discrepancy between the two may compromise the accuracy of the geometric cues.

The current probe mechanism covers only the envelope of the end-effector, leaving the rest of the robotic arm and any manipulated objects unaccounted for. Additionally, single-view RGB-D point clouds are insufficient to fully represent occluded surfaces. Missing depth data is encoded as the maximum sensor range, which could lead the system to interpret unknown regions as being far from any obstacles. Existing experiments have not yet adequately evaluated system behavior under conditions such as missed detections, occlusions, missing depth data, or calibration errors.

The framework defines "low-risk" settings to allow for contact, while "high-risk" settings treat any contact as a failure; consequently, the Task Success Rate (TSR) metrics for these different risk tiers incorporate distinct safety requirements. While significant variations in the Contact-Aware Success Rate (CAR) across risk tiers may indicate behavioral adaptation, they do not, in isolation, prove superior safety control. It is recommended to also report consistent metrics for overall task success, contact-free success, and the distribution of clearances.

The reliance on external perception modules and the associated system costs warrant a more thorough discussion. Although the policy network itself is compact, the overall system depends on external components like a Vision-Language Model (VLM) and SAM2. While the VLM API’s latency—approximately 1.2 seconds—can be managed asynchronously, it still entails overheads related to initialization, semantic updates, and the temporal validity of perception results. Given that the VLM offers only marginal gains over the existing rule-based mapping baseline, the necessity of its added complexity requires further justification.

With only 30 trials per method for each risk tier, some performance improvements over the NSG baseline amount to as few as one or two additional successful trials; therefore, the inclusion of confidence intervals or paired statistical analysis is necessary. The main text states that all 360 original demonstration examples were used for training, while the appendix mentions that 300 examples were used after filtering; this should be clarified for consistency.

Questions for authors

Can controllability be maintained across different tasks? For the full SafeLIBERO benchmark—including scenarios with multiple obstacles and narrow passages—could you report the intensity-versus-distance relationship for each task, the proportion of monotonicity violations, and the extrapolation range within which task success rates are sustained?

How sensitive is the look-ahead probe to prediction errors? Could you compare the geometric fields constructed using demonstration actions, policy-predicted actions, and temporally aggregated actions? Furthermore, is there an alignment between the perturbations introduced during training and the errors observed in the actual action sequences?

How is unknown geometry handled? If an obstacle is missed, depth data is missing, or an obstacle is occluded by a target object, does setting the distance to a maximum value compromise obstacle avoidance? Could you include tests for perception degradation, as well as statistics on collisions involving the robotic arm itself or the objects being manipulated?

Could safety and task-performance metrics be reported in a unified manner? In addition to risk-related Task-Specific Requirements (TSRs), please provide consistent metrics across all layers—such as task completion rates, success rates without contact, minimum distance distributions, and obstacle displacements—to distinguish between behavioral adaptability and actual safety gains.

What is the independent contribution of the VLM-based adaptive selection mechanism? Beyond the existing rule-based mapping baseline, could you report paired statistics, performance advantages in complex scenarios, as well as initialization latency, cache update frequency, and total system resource consumption? How is the actual reaction time—associated with the CFG (Classifier-Free Guidance) path—factored into the local radius settings?

2 comments
AuthorsOpenReview ↗

Response to Reviewer bkui Part 1 Controllability Action Aggregation and Perception

Thank you for the exceptionally detailed deployment-oriented review. We appreciate that you asked where controllability breaks, whether geometry remains causally aligned with executed actions, and how the system behaves under missing perception. Following your suggestions, we added complex-layout sweeps, causal aggregation replay, perception-corruption tests, and collision diagnostics.

Q1. Cross-task controllability

On full SafeLIBERO, including multi-obstacle and narrow-passage scenes, can the authors report per-task intensity–distance relationships, monotonicity-violation rates, and the extrapolation range over which task success is sustained?

A1. Controllability across tasks and complex scenes

We agree that a global statement such as s≤15s\leq15 requires task-wide support. We therefore run the full strength sweep on all 16 SafeLIBERO tasks and on multi-obstacle and narrow-passage variants.

The released and complex layouts obtain macro strength–clearance correlations of 0.946 and 0.962, respectively. Completion remains stable through s=15s=15 on released layouts and s=14s=14–1515 on complex layouts, while clearance remains monotonic through s=17s=17–1818 and s=16s=16–1818, respectively.

Rather than reducing non-monotonic behavior to a single violation count, we report the last strength level up to which the relationship remains monotonic for each task. The results support broad but bounded controllability: complex geometry preserves the strength ordering but can shorten the usable range.

Revision location: Sec. 4.7 and Appendix B.1, Table 14 and the per-task controllability figure.

Q2. Sensitivity of look-ahead probes to prediction error

Can the authors compare fields built from demonstration, policy-predicted, and temporally aggregated actions, and test whether training perturbations match observed action errors?

We clarify that no current predicted action exists before the current action is generated. Using a not-yet-generated action to construct its own conditioning would create a non-causal circular dependency.

A2. Causal action aggregation and prediction error

Demonstration actions are available during training, whereas deployment must construct the geometry field before the current policy call. A current or future predicted action therefore cannot condition its own generation. We instead use a causal replay protocol containing only information available at deployment time.

Before the current policy call, we compare the raw execution queue with the temporally aggregated predictions already available. After the new action chunk is generated, we compare the geometry fields and action directions before and after updating the aggregated queue.

Field comparisonProbe shift mean/p95 (cm)Distance difference mean/p95 (cm)Direction cosine mean/p10Training-noise coverage
Raw queue vs. causal pre-inference aggregation0.7 / 1.50.9 / 1.80.983 / 0.97598.2%
Pre-update vs. post-update aggregated queue1.1 / 1.81.6 / 2.00.969 / 0.95496.5%

The deployment discrepancy is measured directly and is largely covered by the training perturbation distribution. We also clarify that no same-step post-inference action constructs its own conditioning field.

Revision location: Appendix B.2, Table 15.

Q3. Unknown geometry

When obstacles are missed, depth is absent, or geometry is occluded, does encoding distance as the maximum range compromise avoidance, and can the authors test perception degradation and report arm and manipulated-object collisions?

A3. Missing depth, unknown geometry, and arm collision

We agree that encoding unknown space as far away is unsafe unless stress-tested. We therefore evaluate 10% and 20% mask dropout and depth holes. The uncorrupted CAR/TSR is 80.25/68.63%.

At 20% corruption:

  • Mask dropout gives 78.24% CAR, 67.13% TSR, 17.24% end-effector collision, and 5.43% arm collision.
  • Depth holes give 77.89% CAR, 66.71% TSR, 17.56% end-effector collision, and 6.01% arm collision.
  • Treating unknown mask regions as occupied recovers CAR/TSR to 78.95/68.15%.
  • Treating unknown depth regions as occupied recovers CAR/TSR to 78.64/67.98%.

The fallback does not reduce every collision metric simultaneously. Under 20% mask dropout, arm collision changes from 5.43% to 5.64%, although end-effector collision and overall CAR/TSR improve. This indicates that conservative fallback can redistribute contact across body parts. We therefore report end-effector, arm, and manipulated-object collisions separately and describe unknown-as-occupied as a conservative fallback rather than a universal solution.

Revision location: Appendix B.3, Table 16, and Appendix A.5.

AuthorsOpenReview ↗

Response to Reviewer bkui Part 2 Unified Metrics VLM Cost and Statistics

Continuing with the remaining points in your review, we agree that adaptive behavior cannot be inferred from risk-specific TSR alone and that the VLM should be evaluated as part of the complete system. We therefore add consistent metrics across all risk levels, isolate the VLM from the rule and oracle, report system cost, and expand the statistical and real-robot evaluations.

Q4. Unified metrics

Beyond risk-specific TSR, can the authors consistently report task completion, contact-free success, minimum-clearance distributions, and obstacle displacement across all risk levels?

A4. Unified safety and task metrics

We now report raw completion, contact-free completion, Any-contact rate, clearance p10/median, obstacle displacement, and risk-specific TSR at every risk level.

At low risk, CAFE-VLA deliberately accepts contact and therefore has lower contact-free success than NSG. At high risk, it achieves both higher raw completion and higher contact-free success.

Relative to NSG, the paired contact-free-success differences are:

  • Low: −18.13%, bootstrap 95% CI [−23.22, −13.22], p<0.001p<0.001.
  • Medium: −0.41%, bootstrap 95% CI [−5.11, 4.49], p=0.900p=0.900.
  • High: +7.08%, bootstrap 95% CI [2.54, 11.74], p<0.001p<0.001.

Thus, the high-risk advantage persists under risk-independent safety metrics, while the low-risk result reflects an explicit efficiency–safety trade-off. Our claim is controllable adaptation, not uniformly conservative behavior.

Revision location: Appendix B.4, Table 22.

Q5. Independent VLM contribution and system cost

Beyond the rule baseline, can the authors report paired statistics, complex-scene gains, initialization latency, cache-update frequency, total resource use, and how CFG reaction time enters the local-radius setting?

A5. VLM contribution and total system cost

We compare fixed s=8s=8, a three-level rule s=(2,6,12)s=(2,6,12), scene-level VLM selection, and a per-scene oracle. Their mean TSR values are 68.61%, 72.92%, 74.31%, and 75.59%. The rule captures most of the gain over fixed strength, confirming that the core benefit comes from controllable adaptation. The VLM adds 1.39% and remains 1.28% below the oracle. Unlike the rule, which requires manually supplied risk categories, the VLM automates scene understanding and strength selection.

The system-cost audit reports:

  • Obstacle-VLM API: approximately 1.2 s asynchronously.
  • VLM cache refresh: episode initialization.
  • SAM2 tracking: 15.2±2.115.2\pm2.1 ms/frame asynchronously.
  • Depth alignment, back-projection, and filtering: 6.1±0.86.1\pm0.8 ms.
  • Probe KNN and field construction: 4.0±0.34.0\pm0.3 ms.
  • Visual prompting: 2.2±0.42.2\pm0.4 ms.
  • Direct/CFG inference: approximately 65 ms / 118±8118\pm8 ms.
  • Direct/CFG end-to-end reaction: approximately 77 ms / 130 ms.
  • Peak inference/training GPU memory: 9.8/21.4 GB.

The direct-path p95 latency gives a 12 cm radius lower bound. Using the 130 ms CFG end-to-end latency gives approximately 13 cm, still below the deployed 20 cm radius.

Revision location: Appendix B.5, Table 23 for selector ablation and Table 14 for system cost.

Q6. Statistical sufficiency and data consistency

Some gains may represent only one or two additional successes, so confidence intervals or paired statistics are needed; the stated 360 demonstrations versus 300 after filtering must also be reconciled.

A6. Statistical analysis and demonstration count

We agree that 30 real-robot rollouts per entry could not rule out a one- or two-trial effect. We collected 70 additional rollouts for every method–risk pair, increasing each entry to 100 rollouts and the full evaluation to 900 trials.

CAFE-VLA completes 67, 64, and 62 trials at low, medium, and high risk, compared with 63, 55, and 55 for NSG and 61, 60, and 49 for fixed strength. Its mean TSR is 6.67% above NSG and 7.67% above fixed strength.

The contact-free analysis separately uses 480 paired outcomes and 10,000 bootstrap resamples. Differences against NSG are −18.13%, −0.41%, and +7.08% at low, medium, and high risk. The medium-risk difference is not significant; the high-risk gain has bootstrap 95% CI [2.54, 11.74] and p<0.001p<0.001.

The 100-rollout evaluation strengthens real-world task-success estimates, while the 480 paired outcomes test contact-free performance on matched initial states.

Finally, 360 is the number of raw demonstrations collected; 300 remain after quality filtering and are used for training.

Revision location: Sec. 4.6 and Appendix B.4.

Reviewer_4gjVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CAFE-VLA, a framework that enables compact Vision-Language-Action (VLA) models to adapt their obstacle avoidance behavior to scene-specific risks through a continuous avoidance-strength scalar and an end-effector-centric 3D geometry field. By integrating FiLM, cross-attention, visual prompting, and classifier-free guidance (CFG), the method achieves controllable obstacle avoidance, outperforming existing safety-aware baselines on SafeLIBERO and in real-world experiments while largely preserving the original manipulation capabilities.

Critical strengths
  1. The paper addresses an important problem of continuously controllable obstacle avoidance in VLA models. Compared with existing approaches that primarily rely on fixed safety constraints, the proposed formulation is well motivated and practically relevant.
  2. The method separates avoidance strength from local geometric information and employs FiLM and dual-pathway geometry injection to handle these distinct conditioning signals. The overall architecture is well motivated and balances obstacle avoidance performance with model efficiency.
  3. The experiments cover comparisons with safety-aware baselines, continuous controllability analysis, ablations of geometric representations and conditioning mechanisms, capability retention, and real-world validation, providing relatively comprehensive evidence of the method's effectiveness.
Critical weaknesses
  1. The core technical components, including FiLM, CFG, cross-attention, and local geometric fields, are largely based on existing techniques. The methodological novelty is therefore somewhat incremental, and the fundamental advantages over conventional geometry-based obstacle avoidance methods remain insufficiently established.
  2. The main evidence for continuous controllability comes from strength-clearance monotonicity experiments in a single-obstacle setting. It remains unclear whether this relationship generalizes consistently to more complex obstacle configurations and manipulation tasks.
  3. The baseline comparisons do not fully isolate the benefits of adaptive avoidance. In particular, it is unclear whether CAFE-VLA would retain its advantages if existing CBF-based methods were equipped with similar adaptive safety-margin adjustments. Additionally, the relatively limited real-world evaluation and lack of statistical significance analysis for key performance differences weaken the experimental evidence.
Questions for authors
  1. Could the authors evaluate the monotonic relationship between avoidance strength and actual clearance across a broader range of tasks and obstacle configurations, particularly multi-obstacle scenarios, to further establish the generalizability of continuous controllability?
  2. Would CAFE-VLA still outperform AEGIS or NSG if these methods were equipped with similar scene-adaptive mechanisms for dynamically adjusting their CBF safety margins? Could the authors also compare task success rates at matched safety levels?
  3. The VLM-based strength selection provides relatively modest improvements over a simple risk-level rule mapping. Could the authors further clarify the additional benefits of VLM-based scene understanding and whether these gains justify the reliance on an external VLM?

Comments on the paper

1 comment

Thanks for the reply, some of my concerns are solved. However, I still think that the scalability problem remains, which makes the method not practical when deploying in real world. I will maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.