acceptodds
Under review as a conference paper at ICLR 2027

MuseVLA: Adaptive Multimodal Sensing for Robotic Manipulation

Abstract

Humans use diverse senses to interact with the world, while most Vision-Language-Action (VLA) models for robotics rely solely on RGB observations. This limits their ability to perceive physical properties that are difficult or impossible to infer from RGB cameras, such as temperature, sound, or radar response. We present MuseVLA, an adaptive multimodal sensing VLA model that integrates novel sensors as on-demand tools for robotic manipulation. Given a task instruction and visual context, MuseVLA first generates a sensor token and target description that select the sensing modality to invoke and what to attend to, analogous to a tool call with arguments. It then converts the selected sensor measurement into a grounded sensor image, a unified intermediate representation that encodes heterogeneous readings for multimodal fusion and action generation. This design decouples sensor-specific processing from the VLA backbone, enabling efficient integration of diverse modalities. To reduce the need for expensive multisensory robot datasets, we further introduce a data synthesis pipeline that augments existing RGB video datasets with grounded sensor images, enabling generalization to unseen sensor-guided tasks. We evaluate MuseVLA on a real-world robot across challenging dexterous hand manipulation tasks that require multimodal sensing inputs, including temperature-guided pick-and-place, audio-driven object search, and radar-assisted hidden object retrieval. MuseVLA achieves 80.6% success rate on average, outperforming RGB-only and multisensory VLA baselines significantly, and exhibits strong zero-shot capabilities on unseen tasks.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robotics, vision-language-action model, sensing

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_PJQvOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MuseVLA integrates thermal, acoustic, and radar sensing into a VLA model through sensor selection and grounded sensor images. It also augments existing RGB demonstrations with synthetic sensor overlays. The method is evaluated on real-robot manipulation and unseen sensor–target combinations.

Critical strengths

I like the practical design. Mapping different sensor readings into a shared image representation is a simple way to reuse the vision backbone. The tasks also give clear reasons to use information beyond RGB. The paper is clear and easy to follow.

The real-robot experiments and controlled ablations are another positive. In particular, the synthetic-data comparison keeps the additional trajectories and training budget fixed. Unseen-task success improves from 39.6% without overlays to 66.7% with them, supporting the usefulness of the proposed augmentation.

Critical weaknesses

The demonstrated sensing capability is fairly narrow. Most tasks use the additional sensors to select the correct object, followed by picking, placing, or opening. Even the multi-stage tasks mainly combine successive target-selection steps. This is useful, but it says less about using sensory feedback to adapt the interaction itself. I’d like clearer evidence that sensed properties change how the robot manipulates an object, rather than mainly which object it chooses.

The synthetic data have a related limitation. The pipeline adds physical-property keywords and corresponding color overlays to existing demonstrations. It does not describe changing the demonstrated actions to reflect those properties. The authors acknowledge that it does not reproduce raw sensor physics. This seems useful for learning sensor-guided selection, but its value for richer relationships between physical properties, actions, and changing sensor readings remains unclear.

Generalization is encouraging, but still limited in scope. The unseen evaluation recombines existing sensors with new targets, using 16 trials per modality. The real-world training set contains seven objects, and the sensor registration assumes a fixed tabletop setup with limited depth variation. These results support transfer to new sensor–object combinations, but do not establish broader transfer across environments or interaction types. The alignment stress tests are useful, but are not a substitute for testing in different workspaces.

Questions for authors
  • How does MuseVLA compare with using sensors only to select the target, then relying on RGB for the remaining manipulation? This would help separate target-selection benefits from ongoing sensory feedback.
  • Are action labels kept unchanged during synthesis? How would the pipeline handle a physical-property change that requires a different manipulation strategy?
  • How many distinct objects and workspaces are included in the unseen evaluation? Are results available under different sensor placements or environmental conditions?
  • In multi-stage tasks, does the robot decide when to switch stages and sensors, or are updated instructions supplied externally?
4 comments
AuthorsOpenReview ↗

Response to Reviewer PJQv

We sincerely thank the reviewer for recognizing practical design, clear motivation, and effective real-robot experiments and ablations that support usefulness of the proposed augmentation.

We appreciate the valuable feedback that help strengthen our work, and we address each point below and will add them to our revised paper.

W1 & Q1. Broader and ongoing sensory feedback that changes how the robot manipulates

Our current evaluation intends to establish three capabilities: (1) selecting the appropriate sensor for the task, (2) using the selected sensor to identify and act on the correct target, and (3) completing challenging dexterous manipulations. We acknowledge that the evaluation tasks primarily involve sensor-based target selection followed by robotic manipulation. This structure allows us to evaluate sensing and manipulation separately and determine whether a failure originates from sensing or physical manipulation. We also would like to mention that target selection and pick-and-place tasks follow common practice for evaluation in many prior VLA work across the community, such as InstructVLA [1] and π0.5\pi_{0.5} [2].

Nevertheless, MuseVLA's method supports more than a one-time sensing decision or target selection. It already selects sensors, acquires grounded observations, and updates its actions repeatedly throughout a task. Therefore, there is no fundamental architectural barrier to using successive observations for finer-grained feedback control. With suitable training data, for example, a robot could repeatedly acquire thermal observations to act properly according to the real-time temperature of water or food.

Compare with using sensors only to select the target, then relying on RGB for the remaining manipulation

A separate high-level module could potentially select a target using the sensor before actions, but this would not provide the tightly coupled, timely sensor feedback, posing a fundamental system limit. Supporting such control externally would require an additional dynamic external module invocation process. MuseVLA instead jointly performs learned sensor selection, measurement grounding, and sensor-guided action generation within a unified policy. Nevertheless, we agree that more evidence is helpful to illustrate this and strengthen our work.

Clearer evidence that sensed properties change how the robot manipulates an object

To provide clearer evidence that sensed properties can change how the robot manipulates an object with MuseVLA, rather than mainly which object it chooses, we additionally build a fine-grained sensing guided manipulation task.

Task: The instruction is "Turn off the power when the kettle reaches 60 degrees." The water heats continuously; the robot must keep acquiring thermal observations during execution and press the power button at the moment the target temperature is reached. Because the task depends on real time temperature value, live sensor feedback is required. Moreover, (i) Our VLA conditions on the current observation and carries no clock or elapsed-time state, so it cannot execute a memorized countdown. (ii) Each trial begins from a naturally different starting water temperature, so the time needed to reach 60°C varies from trial to trial and no fixed actions can succeed. (iii) RGB-only baseline on the same task never succeeds (0%) due to no visible RGB cue like bubbling with the close-lid kettle in our setup.

Data and protocol: We collected 100 demonstrations of heating water. After adapting the pretrained MuseVLA model, we run 12 trials. A trial counts as a success only if the robot presses the button while the measured water temperature is within ±5°C of the 60°C target.

Result: MuseVLA succeeds on 10/12 trials (83.3%). This is a demanding criterion: the temperature rises continuously, so the model must read the measured value accurately at each step and commit to the press inside a narrow window. Achieving it therefore requires genuine fine-grained sensor feedback throughout manipulation/actions.

[1] Vision-Language-Action Instruction Tuning: From Understanding to Manipulation (InstructVLA). International Conference on Learning Representations, 2026.

[2] π0.5\pi_{0.5}: a Vision-Language-Action Model with Open-World Generalization. Conference on Robot Learning, 2025.

AuthorsOpenReview ↗

Response to Reviewer PJQv, cont'

W2 & Q2. Synthetic actions and physical-property-dependent manipulation

We acknowledge that our pipeline adds physical-property related task instruction and corresponding color overlays to existing demonstration actions. This enables scalable reuse of existing RGB datasets. To ensure factual actions matching the RGB video, we keep actions unchanged. However, this does not mean the synthesis pipeline is limited to this. We can change the sensing color overlays according to actions to achieve richer relationships between physical properties, actions, and changing sensor readings. In other words, instead of changing the manipulation actions based on physical-property change, we can change synthesized physical-property sensor measurement based on existing manipulation actions. For example, for an existing "pick the pot" task, we can have a time varying color overlay that slowly turns red when the robot arm starts to act/pick. This way, the corresponding task can be "pick the pot when finishing heating."

W3 & Q3. Generalization scope and workspace transfer

We agree with the importance of generalization scope and workspace transfer and will add related details to our revised paper. For our existing unseen task evaluation reported in the paper, we stress tested our model performance with both unseen objects and tabletop workspace environment changes:

(1) We have 5 distinct kinds of unseen objects across three sensing modalities, including plastic ice bags, bottles with unseen shape, two distinct-looking plush toys, two tote bags; Unseen objects mean new objects never appeared in training.

(2) we have another 10 distinct unseen objects as random distractors on the tabletop. This creates a messy realistic workspace as shown in Figure 7 and Figure 9. We randomly change the placement of the target basket (the destination for pick and place) and distractor objects every 2 task trials. This means 8 different tabletop environments with distinct object and destination arrangement per modality, 24 tabletop/environment configurations in total.

(3) After each trial, we also move the table aside to allow the robot arm move back to initial position. Moving the table is a safety procedure to avoid accident collision damage. So our table position does not stay fixed and has natural shifts across all task trials.

(4) Between our training data collection and unseen task performance evaluation, our real-world robot hardware was also repurposed for other projects, and our sensor module was reinstalled for evaluation. This creates a natural test for sensor placement robustness and over-time environmental condition changes.

Together, the strong performance of MuseVLA on unseen tasks already shows robust generalization under combination of unseen tasks and objects, changing tabletop environment configurations, table positions, and sensor installation.

Q4. Multi-stage switching

We use multi-stage tasks to demonstrate that multiple sensing capabilities can be composed for one longer task. The updated per-stage instructions are supplied externally, since MuseVLA currently does not support generating its own task instructions. However, the MuseVLA/robot decides sensor switching based on input instruction and observation fully autonomously.

Thanks so much for your time and effort on reviewing and providing feedback.

Dear Reviewer,

With the discussion period nearing its end, we would be grateful if you could let us know if there are any remaining concerns or questions. We would be happy to provide further clarification.

We hope our response has helped resolve your concerns. Thank you again for your time and thoughtful consideration!

Dear reviewer PJQv,

Thank you for taking the time to review our work. We sincerely appreciate your detailed feedback.

As the discussion period is ending soon, we would like to gently follow up and ask whether our earlier responses have adequately addressed your concerns.

If any part would benefit from further clarification, we would be glad to elaborate. Thanks again for your time and consideration.

Reviewer_wS8zOpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

MuseVLA adds thermal, acoustic, and mmWave sensing to a VLA model. It does this through task-conditioned sensor selection. The selected measurements are rendered onto the relevant object regions as grounded sensor images. This lets a shared vision encoder handle different modalities. The paper also introduces a data synthesis pipeline. It augments existing RGB demonstrations with sensor-conditioned instructions and overlays. The method is evaluated on real-world manipulation, component ablations, and unseen-task generalization.

Critical strengths
  1. The system is tested with three real sensing modalities. The experiments cover temperature-guided selection, sound-source localization, and radar-assisted hidden-object search with a dexterous hand. This shows it can work beyond one sensor or one task.

  2. The controlled ablations support the design. With the same backbone, adaptive selection raises success from 33.3% to 59.7%. Adding grounded sensor images pushes it to 76.4%. These comparisons go beyond the usual advantage of having more sensing than RGB-only policies.

  3. The synthetic-data approach is practical, and the control is useful. On unseen tasks, the full method gets 66.7% success. Using the same extra trajectories without sensor overlays gets 39.6%. This supports the value of the synthetic sensor representation, not just adding more manipulation data.

Critical weaknesses
  1. The scope of adaptive sensor selection is still limited. Many evaluated instructions directly mention the relevant modality with words like “hot” or “ringing.” So selection might be solvable from language cues alone. Tasks that require observation-dependent sensing decisions would better show the benefit of learned adaptive selection.

  2. A simpler modular baseline is missing. A system that uses sensor measurements to find a target object, then conditions a manipulation policy on its mask or location, would be a good comparison. It would clarify when rendering sensor values into the VLA input helps beyond sensor-assisted target selection.

  3. The efficiency evaluation is incomplete at the system level. The paper reports memory savings, but not end-to-end latency. That should include sensor processing, segmentation, and the sensor-selection stage. These measurements would help assess the practical overhead.

Questions for authors

please see the above weaknesses

5 comments
AuthorsOpenReview ↗

Response to Reviewer wS8z

We sincerely thank the reviewer for recognizing the value of real-world experiments across sensing modalities, controlled component ablations that support the design, and practical and effective synthetic-data approach.

We appreciate the valuable feedback that help strengthen our work. We address each point below and will revise the paper accordingly.

W1. Instruction keywords and observation-dependent sensor selection

We use keywords, like “hot” or “ringing”, related to physical properties in our task instructions. Our intention is to make the sensor selection and target description generation unambiguous and verifiable. Otherwise, a task instruction may have multiple plausible sensor selections.

Our learned adaptive selection approach is general to different styles of task instructions after training. But we agree that demonstrate the benefit of learned adaptive selection would strengthen the paper. We show this with new rule-based and lightweight classifier baselines and challenging new test sets.

New experiment: implicit and observation-dependent sensor selection.

Besides two original test set in the paper (seen and unseen sets), we construct two extra test sets for sensor selection and target description generation:

(1) The new implicit set contains 200 test samples covering 4 sensor selections (RGB, thermal, acoustic, and mmWave) and diverse objects. Example instructions include "Pick up the freshly cooked tomato," "Bring me the drink from the fridge," and "Pick up the plush toy that is interrupting our chat." Instead of explicit descriptive keywords, these instructions need indirect associations or reasoning, like "cooked", "fridge".

(2) The new observation-dependent set contains 120 test samples where model depends on visual observation to decide which sensor is most suited. For example, for the identical request "Pick the object in the box", visible contents in an open box require only RGB camera, whereas concealed contents in one of two boxes would call for mmWave radar tool.

Training data and configuration.

Because our pretraining dataset construction intentionally focused on explicit sensor keywords, we extend our training dataset to support new types of implicit and observation-dependent task instructions. Note that we still keep the new test sets unseen, which mean zero task instruction or video overlap in the training set. We sample 500 task videos from our training dataset and diversify their task instructions using our data pipeline. For example, training includes "Bring me the food just taken out of the stove," "Pick out the gray plush toy on the table playing a melody," and "Locate the box holding an item". For MuseVLA adaptation, we start from the pretrained checkpoint and use the same training config and loss function as in the paper.

Baselines.

We implement new baselines as comparison:

(1) Keyword rule based baseline is a fixed keyword matcher. We summarize physical sensing related keywords, like hot, cold, ringing,..., from our training set, and record their corresponding sensor selection.

(2) We train lightweight classifiers using our training dataset same as MuseVLA-adapted, including a text-only classifier (small MLP with SVD-compressed instruction features as input.), and an RGB+text classifier (CNN + linear layers) to predict among 4 sensor selections.

Result Table: Sensor selection / target-description accuracy (%).

MethodOriginal seen tasksOriginal unseen tasksNew implicit tasksNew obs-dependent tasks
PaliGemma-2 base0.0 / 13.00.0 / 9.50.0 / 12.50.0 / 26.7
Keyword rules100.0 / -91.5 / -7.0/ -50.0 / -
Lightweight text-only classifier91.5 / -73.5 / -56.0 / -50.0 / -
Lightweight RGB+text classifier91.5 / -53.0 / -45.5 / -71.7 / -
MuseVLA pretrained (original)100.0 / 93.5100.0 / 82.066.5 / 70.555.8 / 80.8
MuseVLA-adapted100.0 / 91.098.5 / 94.091.5 / 90.589.2 / 100.0

Result analysis:

  1. Keyword rules based baseline has degraded performance beyond seen tasks.
  2. Lightweight classifier-based sensor selection also show clear generalization gaps on unseen tasks and implicit tasks.
  3. MuseVLA after adaptation achieves a high accuracy for seen and unseen tasks and better performance on tasks with implicit and observation-dependent tasks compared to baselines.
  4. "MuseVLA pretrained (original)" is our pretrained version in the paper without any further training on new datasets, but still outperforms keyword rules and lightweight classifiers that have been exposed to new training datasets on New implicit tasks.

Together, the results show the benefit of learned adaptive selection: a generalizable learning approach that outperforms rule-based or lightweight classifier-based baselines.

AuthorsOpenReview ↗

Response to Reviewer wS8z, continued

W2. Modular baseline and Beyond sensor-assisted target selection

A simpler modular system baseline is possible for individual tasks, but would need hand-crafted task-specific logic for the mapping from sensor measurements to target objects. This approach would lack generalization across tasks, and may fail for new unseen tasks. In contrast, MuseVLA learns this association together with action generation, preserving sensor values across candidate regions instead of committing to one target beforehand. This provides a unified and generalizable learning system that internalizes such capabilities with high efficiency.

Our evaluation tasks primarily involve sensor-based target selection followed by robotic manipulation. This structure allows us to determine whether a failure originates from sensing or physical manipulation. We also would like to mention that target selection tasks follow common practice for evaluation setup in VLA work in the community, such as InstructVLA [1] and π0.5\pi_{0.5} [2].

Nevertheless, MuseVLA is not restricted to sensor-assisted target selection. There is no fundamental architectural barrier to using successive observations for other finer-grained feedback control. With suitable training data, for example, a robot could repeatedly acquire thermal observations while controlling the temperature of water or food for cooking.

To concretely demonstrate this, we evaluate MuseVLA with a new fine-grained sensing guided manipulation task beyond sensor-assisted target selection..

New result: fine-grained sensor-guided manipulation

Task. The instruction is "Turn off the power when the kettle reaches 60 degrees." The water heats continuously; the robot must keep acquiring thermal observations during execution and press the power button at the moment the target temperature is reached. The sensor value determines when to act, so live sensor feedback is needed. Moreover, (i) our model conditions on the current observation and carries no clock or elapsed-time state, so it cannot execute a memorized countdown. (ii) Each trial begins from a naturally different starting water temperature, so the time needed to reach 60°C varies from trial to trial and no fixed actions can succeed. (iii) RGB-only baseline on the same task never succeeds (0%) due to no visible RGB cue like bubbling for our closed-lit kettle.

Data and protocol. We collected 100 demonstrations of heating water. After adapting the pretrained MuseVLA model, we run 12 trials. A trial counts as a success only if the robot presses the button while the measured water temperature is within ±5°C of the 60°C target.

Result. MuseVLA succeeds on 10/12 trials (83.3%). This is a demanding criterion: the temperature rises continuously, so the policy must read the measured value accurately at each step and commit to the press inside a narrow window. Achieving it therefore requires genuine fine-grained sensor feedback throughout execution.

[1] Vision-Language-Action Instruction Tuning: From Understanding to Manipulation (InstructVLA). International Conference on Learning Representations, 2026.

[2] π0.5\pi_{0.5}: a Vision-Language-Action Model with Open-World Generalization. Conference on Robot Learning, 2025.

AuthorsOpenReview ↗

Response to Reviewer wS8z, continued 2

W3. End-to-end inference latency

We benchmarked MuseVLA FP32 inference on a single RTX 5090. We expect BF16 or lower bit inference to be faster. Each latency is reported as mean ± standard deviation over 100 measurements. The inference pipeline consists of three stages:

  • Stage 0: Sensor/target selection (asynchronous) MuseVLA takes the task instruction and current RGB observation to generate a sensor token and target description.
  • Stage 1: Grounded sensor image generation. SAM3 segments the target in the current RGB image, followed by a calibrated overlay of the selected sensor readings to form a grounded sensor image.
  • Stage 2: Action generation. MuseVLA takes the task instruction, sensor token and target description, RGB and the grounded sensor image, together with robot state, to predict actions.
StageLatency
Stage 0: Sensor/target selection143.7 ± 17.1 ms
Stage 1: SAM3 + sensor overlay54.1 ± 3.5 ms
Stage 2: Action generation109.1 ± 5.3 ms

As sensor/target selection does not need very fast adaptation, we cache sensor token and target description for action generation. We update them asynchronously with a configurable interval like 1 or 2 seconds, when the preceding action chunk is executed. Thus, generating an action chunk requires only Stages 1–2, with a combined end-to-end latency of 163.2 ms (54.1 + 109.1 ms) with cache.

MuseVLA also shows latency advantage when increasing the number of sensor input. We compare action generation between MuseVLA and MuseVLA-AllInput. MuseVLA always supplies RGB plus one selected sensor image to the VLA, whereas MuseVLA-AllInput jointly processes RGB and all M sensor images.

Sensing slots (M)MuseVLA Action GenMuseVLA-AllInput Action Gen
1109.1 ± 5.3 ms112.3 ± 4.7 ms
2109.1 ± 5.3 ms148.1 ± 4.6 ms
3109.1 ± 5.3 ms173.9 ± 3.6 ms
4109.1 ± 5.3 ms210.6 ± 4.5 ms
8109.1 ± 5.3 ms360.0 ± 4.1 ms

MuseVLA's action generation latency remains constant at 109 ms, whereas MuseVLA-AllInput incurs increasing latency as more image inputs are processed. For RGB plus all three real sensor images (thermal, acoustic, and mmWave), adaptive selection reduces VLA action-generation latency by 37.3%. At M=8, the reduction reaches 69.7%.

Thank you so much for time and effort on reviewing and providing feedback.

Reviewer_ahKfOpenReview ↗
Rating
3(+1)
Confidence
3(+1)
Informativeness
3(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MuseVLA, an adaptive multimodal sensing framework for robotic manipulation. The method dynamically selects sensing modalities through sensor tokens and target descriptions, and converts heterogeneous sensor information into unified Grounded Sensor Images. In addition, the authors introduce a multimodal data synthesis strategy to reduce the cost of collecting real sensory data. Real-world robotic experiments demonstrate that the method achieves average success rates of 80.6% and 66.7% on seen and unseen tasks, respectively.

Critical strengths
  1. The research problem has practical significance. This paper addresses the limited utilization of nonvisual sensing in existing VLA models by introducing a task-driven sensor selection mechanism, providing a meaningful exploration of multimodal robotic manipulation.
  2. The experiments provide reasonable evidence of effectiveness. The authors evaluate three types of sensor-guided tasks on a real robotic platform and demonstrate the effectiveness of the proposed method through component ablations, synthetic data comparisons, and robustness experiments.
Critical weaknesses
  1. The methodological novelty is relatively limited. Grounded Sensor Images are similar to the sensor representations used in OmniVLA, while the main contributions lie in sensor selection and data synthesis. However, the sensor types in the evaluated tasks can often be directly determined from keywords in the instructions, leaving the necessity of the learned selection mechanism insufficiently demonstrated.
  2. The experimental validation remains insufficient. A direct comparison with OmniVLA is missing, and the scale of real-world robotic evaluation is relatively small. The unseen tasks mainly involve recombinations of existing sensors and new objects. Furthermore, the efficiency analysis primarily focuses on GPU memory consumption, without evaluating end-to-end inference latency.
Questions for authors
  1. What are the substantive innovations of Grounded Sensor Images compared with OmniVLA? Could the authors provide a direct comparison under consistent experimental settings?
  2. Could the authors compare the proposed approach with rule-based or lightweight classifier-based sensor selection methods to demonstrate the necessity of the learned selection mechanism?
  3. Could the authors evaluate more challenging unseen tasks and provide baseline comparisons under the same synthetic data training conditions?
  4. Could the authors report end-to-end inference latency, including sensor invocation, SAM3 segmentation, and action generation?
7 comments
AuthorsOpenReview ↗

Response to Reviewer ahKf

We sincerely thank the reviewer for recognizing the practical significance, meaningful exploration of task-driven sensor selection, and reasonable evaluation of effectiveness on a real robotic platform.

We appreciate the valuable feedback that help strengthen our work. We address each point below and will revise the paper accordingly.

W1 & Q1. Contributions

The methodological novelty of MuseVLA lies on (1) adaptive sensing modality selection with learned sensor tokens and target description generation and (2) a scalable data synthesis pipeline to generalize such new capability, rather than Grounded Sensor Images alone.

Fundamentally, we present a new adaptive multimodal sensing capability and formulate this as a learning problem as in Section 3 of the paper. We hope to bring a new perspective to the community that robots can integrate novel sensors as generalizable on-demand tool calls, rather than conventional always-on integration.

These contributions are substantially different from OmniVLA. And if we compare specifically the sensor representation part only, we share similar RGB image-space sensor overlays. But Grounded Sensor Image significantly differs in terms of how we construct and use it. First, MuseVLA construct the sensor images based on VLA-predicted sensor token and target description, determining which measurement to acquire and which regions to ground internally. OmniVLA depends an external VLM (GPT-4o) to generate segmentation prompts to control construction. Second, MuseVLA turns this representation into a shared interface for real measurements and synthesized training data. This provides a concrete capability beyond rendering measurements alone. Under matched additional trajectories and training budget, synthesized grounding improves unseen-task success from 39.6% to 66.7%. This supports zero-shot transfer to unseen sensor-guided tasks, which OmniVLA does not achieve.

W2 & Q1. Comparison with OmniVLA

We would like to clarify that OmniVLA uses a different VLA backbone, SmolVLA, which is a compact model, pretrained on SO-100/101 robot arms only. MuseVLA used PaliGemma-2 VLM, which is 3B vs 0.5B in paramemeter sizes. This means that directly using OmniVLA on our robot hardware and tasks would not work. Our tests on OmniVLA with our robot arm showed near zero success rates and unsafe robot actions.

To compare under consistent experimental settings, we compare MuseVLA with MuseVLA-AllInput. MuseVLA-AllInput retains the same backbone as MuseVLA, but jointly processes all three sensing modalities instead of only the selected one, same as OmniVLA's approach. We keep the backbone, real-world training data, training budget, and evaluation protocol fixed. Both variants use grounded sensor images and are evaluated on 24 independent trials per modality.

MethodThermalAcousticmmWaveAverage
MuseVLA-AllInput (OmniVLA-method)50.0%54.2%66.7%56.9%
MuseVLA (Ours)83.3%58.3%87.5%76.4%

Adaptive selection improves end-to-end success in all three modalities, increasing average success from 56.9% to 76.4% while processing only the selected sensor image.

This comparison isolates the benefit of adaptive input selection beyond the representation itself. It is a matched test of the multi-input strategy used by OmniVLA within our backbone, indicating that selecting the relevant sensor image reduces interference from irrelevant sensor.

W2 & Q3. Unseen tasks

MuseVLA achieves zero-shot generalization to unseen tasks within the supported modalities. Different sensor devices can be calibrated into our unified representation and reuse the same pretrained model. Such generalization remains challenging and is not established by most prior VLA systems. To put this in the context of recent work, OmniVLA [1] adapts to new tasks using 25 demonstrations, while FD-VLA [2] trains with 50 demonstrations per task. The downstream improvement experiments in π0.6∗\pi_{0.6}^{*} [3] also use task-specific demonstrations for training. MuseVLA directly performs its evaluated unseen tasks without this adaptation training.

[1] OmniVLA: Physically-Grounded Multimodal VLA with Unified Multi-Sensor Perception for Robotic Manipulation. International Conference on Robotics and Automation, 2026.

[2] FD-VLA: Force-Distilled Vision-Language-Action Model for Contact-Rich Manipulation. International Conference on Robotics and Automation, 2026.

[3] π0.6∗\pi_{0.6}^{*}: a VLA That Learns From Experience. Robotics: Science and Systems, 2026.

AuthorsOpenReview ↗

Response to Reviewer ahKf, continued

W1 & Q2. Necessity of the learned selection mechanism

We use explicit keywords related to physical properties for our task instructions. Our intentions is to make the sensor selection and target description generation unambiguous and verifiable. Otherwise, a task instruction may have multiple plausible sensor selections.

But we agree that comparing with rule-based or lightweight classifier-based sensor selection methods to demonstrate the necessity of the learned selection mechanism would strengthen the paper.

New experiment: implicit instructions and observation-dependent sensor selection.

Besides two original test set in the paper (seen and unseen sets), we construct two extra test sets for sensor selection and target description generation:

(1) The new implicit set contains 200 test samples covering 4 sensor selections (RGB, thermal, acoustic, and mmWave) and diverse objects. Example instructions include "Pick up the freshly cooked tomato," "Bring me the drink from the fridge," and "Pick up the plush toy that is interrupting our chat." Instead of explicit descriptive keywords, these instructions need indirect associations or reasoning, like "cooked", "fridge".

(2) The new observation-dependent set contains 120 test samples where model depends on visual observation to decide which sensor is most suited. For example, for the identical request "Pick the object in the box", visible contents in an open box require only RGB camera, whereas concealed contents in one of two boxes would call for mmWave radar tool.

Training data and configuration.

Because our pretraining dataset construction intentionally focused on explicit sensor keywords, we extend our training dataset to support new types of implicit and observation-dependent task instructions. Note that we still keep the new test sets unseen, which mean zero task instruction or video overlap in the training set. We sample 500 task videos from our training dataset and diversify their task instructions using our data pipeline. For example, training includes "Bring me the food just taken out of the stove," "Pick out the gray plush toy on the table playing a melody," and "Locate the box holding an item".

Baselines.

We implement new baselines as comparison:

(1) Keyword rule based baseline is a fixed keyword matcher. We summarize physical sensing related keywords, like hot, cold, ringing,..., from our training set, and record their corresponding sensor selection.

(2) We train lightweight classifiers using our training dataset same as MuseVLA-adapted, including a text-only classifier (small MLP with SVD-compressed instruction features as input.), and an RGB+text classifier (CNN + linear layers) to predict among 4 sensor selections.

Result Table: Sensor selection / target-description accuracy (%).

MethodOriginal seen tasksOriginal unseen tasksNew implicit tasksNew obs-dependent tasks
PaliGemma-2 base0.0 / 13.00.0 / 9.50.0 / 12.50.0 / 26.7
Keyword rules100.0 / -91.5 / -7.0/ -50.0 / -
Lightweight text-only classifier91.5 / -73.5 / -56.0 / -50.0 / -
Lightweight RGB+text classifier91.5 / -53.0 / -45.5 / -71.7 / -
MuseVLA pretrained (original)100.0 / 93.5100.0 / 82.066.5 / 70.555.8 / 80.8
MuseVLA-adapted100.0 / 91.098.5 / 94.091.5 / 90.589.2 / 100.0

Result analysis:

  1. Keyword rules based baseline has degraded performance beyond seen tasks.
  2. Lightweight classifier-based sensor selection also show clear generalization gaps on unseen tasks and implicit tasks.
  3. MuseVLA after adaptation achieves a high accuracy for seen and unseen tasks and better performance on tasks with implicit and observation-dependent tasks compared to baselines.
  4. "MuseVLA pretrained (original)" is our pretrained version in the paper without any further training on new datasets, but still outperforms keyword rules and lightweight classifiers that have been exposed to new training datasets on New implicit tasks.

Together, the results show the benefit of learned adaptive selection: a generalizable learning approach that outperforms rule-based or lightweight classifier-based baselines.

AuthorsOpenReview ↗

Response to Reviewer ahKf, continued 2

W2 & Q4. End-to-end inference latency efficiency.

We benchmarked MuseVLA FP32 inference on a single RTX 5090. We expect BF16 or lower bit inference to be faster. Each latency is reported as mean ± standard deviation over 100 measurements. The inference pipeline consists of three stages:

  • Stage 0: Sensor/target selection (asynchronous) MuseVLA takes the task instruction and current RGB observation to generate a sensor token and target description.
  • Stage 1: Grounded sensor image generation. SAM3 segments the target in the current RGB image, followed by a calibrated overlay of the selected sensor readings to form a grounded sensor image.
  • Stage 2: Action generation. MuseVLA takes the task instruction, sensor token and target description, RGB and the grounded sensor image, together with robot state, to predict actions.
StageLatency
Stage 0: Sensor/target selection143.7 ± 17.1 ms
Stage 1: SAM3 + sensor overlay54.1 ± 3.5 ms
Stage 2: Action generation109.1 ± 5.3 ms

As sensor/target selection does not need very fast adaptation, we cache sensor token and target description for action generation. We update them asynchronously with a configurable interval like 1 or 2 seconds, when the preceding action chunk is executed. Thus, generating an action chunk with cached sensor selection requires only Stages 1–2, with a combined end-to-end latency of 163.2 ms (54.1 + 109.1 ms).

MuseVLA also shows latency advantage when increasing the number of sensor input. We compare action generation between MuseVLA and MuseVLA-AllInput. MuseVLA always supplies RGB plus one selected sensor image to the VLA, whereas MuseVLA-AllInput jointly processes RGB and all M sensor images.

Sensing/Image slots (M)MuseVLA Action GenMuseVLA-AllInput Action Gen
1109.1 ± 5.3 ms112.3 ± 4.7 ms
2109.1 ± 5.3 ms148.1 ± 4.6 ms
3109.1 ± 5.3 ms173.9 ± 3.6 ms
4109.1 ± 5.3 ms210.6 ± 4.5 ms
8109.1 ± 5.3 ms360.0 ± 4.1 ms

MuseVLA's action generation latency remains constant at 109 ms, whereas MuseVLA-AllInput incurs increasing latency as more image inputs are processed. For RGB plus all three real sensor images (thermal, acoustic, and mmWave), adaptive selection reduces VLA action-generation latency by 37.3%. At M=8, the reduction reaches 69.7%.

Thank you so much for time and effort on reviewing and providing feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Discussion Summary

Dear Area Chair and Reviewers,

Thank you all for your time and valuable feedback. As the discussion period concludes, we summarize the discussion during rebuttal to support your assessment of MuseVLA.

Strengths recognized by the reviewers.

The reviewers recognize the practical significance and motivation of adaptive multimodal sensing for robots, and the real-world dexterous-hand evaluation across thermal, acoustic, and mmWave modalities. They also recognize the controlled ablations demonstrating the effectiveness of adaptive sensor selection and synthetic-data pipeline, and matched-data comparison shows synthesized data enables zero-shot generalization to unseen tasks.

Key concerns addressed during rebuttal:

  1. Learned sensor selection beyond instruction keywords (ahKf, wS8z). We added implicit-instruction and observation-dependent test sets, together with keyword and lightweight classifier baselines. Without further training, the original pretrained MuseVLA outperforms keyword rules and trained lightweight classifiers on the new implicit test set. After adaptation using diversified instructions, MuseVLA achieves 91.5% and 89.2% sensor-selection accuracy respectively, outperforming baselines.

  2. Ongoing sensory feedback beyond target selection (wS8z, PJQv). We added a kettle task requiring the robot to turn off heating at 60°C ± 5°C, with varying initial water temperatures. After adaptation on demonstrations, MuseVLA succeeds in 10/12 trials (83.3%), versus 0% for RGB-only. This provides evidence that live sensor readings can determine when the robot acts, beyond identifying which object to manipulate.

  3. Clarifying the comparison with OmniVLA (ahKf). We clarified that MuseVLA-AllInput adopts OmniVLA's strategy of jointly processing all sensor images within our backbone. This provides a controlled comparison of sensor-input strategies using the same backbone, real-world training data, and training budget. MuseVLA’s adaptive selection improves success from 56.9% to 76.4%, supporting the benefit of selecting relevant sensor inputs.

  4. Inference latency (ahKf, wS8z). We added stage-wise timing: SAM3 segmentation and sensor overlay take 54.1 ms, and action generation takes 109.1 ms, totaling 163.2 ms with cached sensor selection. Selection takes 143.7 ms and is updated asynchronously. With three sensor modality inputs, adaptive selection reduces action-generation latency by 37.3% relative to MuseVLA-AllInput.

  5. Clarifying synthesized actions and robust generalization (PJQv). We explained that synthesis preserves demonstrated actions to match the RGB videos, and outlined how time-varying sensor overlays could extend this approach to represent changing physical properties. Our existing evaluation already tests unseen targets with 10 unseen distractors across 24 varying tabletop configurations, alongside table-position changes and sensor-module reinstallation. MuseVLA achieves 66.7% success under these combined variations, demonstrating robust generalization beyond the training objects and layouts.

Reviewer ahKf confirmed that the rebuttal addressed most concerns and raised the score to 3 (Weak Acceptance). Reviewer wS8z confirmed that questions were answered and retained score of 3 (Weak Acceptance) with higher confidence.

Manuscript revision. We have uploaded a revised manuscript with additions highlighted in blue, incorporating sensor-selection evaluations and baselines, the temperature-guided control experiment, latency analyses, MuseVLA-AllInput comparison explanation, asynchronous sensor selection, and other clarifications.

We sincerely appreciate the discussion and hope this summary helps you assess the strengthened evidence of our paper. Thank you for your time and consideration.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.