- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
MuseVLA integrates thermal, acoustic, and radar sensing into a VLA model through sensor selection and grounded sensor images. It also augments existing RGB demonstrations with synthetic sensor overlays. The method is evaluated on real-robot manipulation and unseen sensor–target combinations.
Critical strengths
I like the practical design. Mapping different sensor readings into a shared image representation is a simple way to reuse the vision backbone. The tasks also give clear reasons to use information beyond RGB. The paper is clear and easy to follow.
The real-robot experiments and controlled ablations are another positive. In particular, the synthetic-data comparison keeps the additional trajectories and training budget fixed. Unseen-task success improves from 39.6% without overlays to 66.7% with them, supporting the usefulness of the proposed augmentation.
Critical weaknesses
The demonstrated sensing capability is fairly narrow. Most tasks use the additional sensors to select the correct object, followed by picking, placing, or opening. Even the multi-stage tasks mainly combine successive target-selection steps. This is useful, but it says less about using sensory feedback to adapt the interaction itself. I’d like clearer evidence that sensed properties change how the robot manipulates an object, rather than mainly which object it chooses.
The synthetic data have a related limitation. The pipeline adds physical-property keywords and corresponding color overlays to existing demonstrations. It does not describe changing the demonstrated actions to reflect those properties. The authors acknowledge that it does not reproduce raw sensor physics. This seems useful for learning sensor-guided selection, but its value for richer relationships between physical properties, actions, and changing sensor readings remains unclear.
Generalization is encouraging, but still limited in scope. The unseen evaluation recombines existing sensors with new targets, using 16 trials per modality. The real-world training set contains seven objects, and the sensor registration assumes a fixed tabletop setup with limited depth variation. These results support transfer to new sensor–object combinations, but do not establish broader transfer across environments or interaction types. The alignment stress tests are useful, but are not a substitute for testing in different workspaces.
Questions for authors
- How does MuseVLA compare with using sensors only to select the target, then relying on RGB for the remaining manipulation? This would help separate target-selection benefits from ongoing sensory feedback.
- Are action labels kept unchanged during synthesis? How would the pipeline handle a physical-property change that requires a different manipulation strategy?
- How many distinct objects and workspaces are included in the unseen evaluation? Are results available under different sensor placements or environmental conditions?
- In multi-stage tasks, does the robot decide when to switch stages and sensors, or are updated instructions supplied externally?
4 comments
Response to Reviewer PJQv, cont'
W2 & Q2. Synthetic actions and physical-property-dependent manipulation
We acknowledge that our pipeline adds physical-property related task instruction and corresponding color overlays to existing demonstration actions. This enables scalable reuse of existing RGB datasets. To ensure factual actions matching the RGB video, we keep actions unchanged. However, this does not mean the synthesis pipeline is limited to this. We can change the sensing color overlays according to actions to achieve richer relationships between physical properties, actions, and changing sensor readings. In other words, instead of changing the manipulation actions based on physical-property change, we can change synthesized physical-property sensor measurement based on existing manipulation actions. For example, for an existing "pick the pot" task, we can have a time varying color overlay that slowly turns red when the robot arm starts to act/pick. This way, the corresponding task can be "pick the pot when finishing heating."
W3 & Q3. Generalization scope and workspace transfer
We agree with the importance of generalization scope and workspace transfer and will add related details to our revised paper. For our existing unseen task evaluation reported in the paper, we stress tested our model performance with both unseen objects and tabletop workspace environment changes:
(1) We have 5 distinct kinds of unseen objects across three sensing modalities, including plastic ice bags, bottles with unseen shape, two distinct-looking plush toys, two tote bags; Unseen objects mean new objects never appeared in training.
(2) we have another 10 distinct unseen objects as random distractors on the tabletop. This creates a messy realistic workspace as shown in Figure 7 and Figure 9. We randomly change the placement of the target basket (the destination for pick and place) and distractor objects every 2 task trials. This means 8 different tabletop environments with distinct object and destination arrangement per modality, 24 tabletop/environment configurations in total.
(3) After each trial, we also move the table aside to allow the robot arm move back to initial position. Moving the table is a safety procedure to avoid accident collision damage. So our table position does not stay fixed and has natural shifts across all task trials.
(4) Between our training data collection and unseen task performance evaluation, our real-world robot hardware was also repurposed for other projects, and our sensor module was reinstalled for evaluation. This creates a natural test for sensor placement robustness and over-time environmental condition changes.
Together, the strong performance of MuseVLA on unseen tasks already shows robust generalization under combination of unseen tasks and objects, changing tabletop environment configurations, table positions, and sensor installation.
Q4. Multi-stage switching
We use multi-stage tasks to demonstrate that multiple sensing capabilities can be composed for one longer task. The updated per-stage instructions are supplied externally, since MuseVLA currently does not support generating its own task instructions. However, the MuseVLA/robot decides sensor switching based on input instruction and observation fully autonomously.
Thanks so much for your time and effort on reviewing and providing feedback.
Dear Reviewer,
With the discussion period nearing its end, we would be grateful if you could let us know if there are any remaining concerns or questions. We would be happy to provide further clarification.
We hope our response has helped resolve your concerns. Thank you again for your time and thoughtful consideration!
Dear reviewer PJQv,
Thank you for taking the time to review our work. We sincerely appreciate your detailed feedback.
As the discussion period is ending soon, we would like to gently follow up and ask whether our earlier responses have adequately addressed your concerns.
If any part would benefit from further clarification, we would be glad to elaborate. Thanks again for your time and consideration.
Response to Reviewer PJQv
We sincerely thank the reviewer for recognizing practical design, clear motivation, and effective real-robot experiments and ablations that support usefulness of the proposed augmentation.
We appreciate the valuable feedback that help strengthen our work, and we address each point below and will add them to our revised paper.
W1 & Q1. Broader and ongoing sensory feedback that changes how the robot manipulates
Our current evaluation intends to establish three capabilities: (1) selecting the appropriate sensor for the task, (2) using the selected sensor to identify and act on the correct target, and (3) completing challenging dexterous manipulations. We acknowledge that the evaluation tasks primarily involve sensor-based target selection followed by robotic manipulation. This structure allows us to evaluate sensing and manipulation separately and determine whether a failure originates from sensing or physical manipulation. We also would like to mention that target selection and pick-and-place tasks follow common practice for evaluation in many prior VLA work across the community, such as InstructVLA [1] and π0.5 [2].
Nevertheless, MuseVLA's method supports more than a one-time sensing decision or target selection. It already selects sensors, acquires grounded observations, and updates its actions repeatedly throughout a task. Therefore, there is no fundamental architectural barrier to using successive observations for finer-grained feedback control. With suitable training data, for example, a robot could repeatedly acquire thermal observations to act properly according to the real-time temperature of water or food.
Compare with using sensors only to select the target, then relying on RGB for the remaining manipulation
A separate high-level module could potentially select a target using the sensor before actions, but this would not provide the tightly coupled, timely sensor feedback, posing a fundamental system limit. Supporting such control externally would require an additional dynamic external module invocation process. MuseVLA instead jointly performs learned sensor selection, measurement grounding, and sensor-guided action generation within a unified policy. Nevertheless, we agree that more evidence is helpful to illustrate this and strengthen our work.
Clearer evidence that sensed properties change how the robot manipulates an object
To provide clearer evidence that sensed properties can change how the robot manipulates an object with MuseVLA, rather than mainly which object it chooses, we additionally build a fine-grained sensing guided manipulation task.
Task: The instruction is "Turn off the power when the kettle reaches 60 degrees." The water heats continuously; the robot must keep acquiring thermal observations during execution and press the power button at the moment the target temperature is reached. Because the task depends on real time temperature value, live sensor feedback is required. Moreover, (i) Our VLA conditions on the current observation and carries no clock or elapsed-time state, so it cannot execute a memorized countdown. (ii) Each trial begins from a naturally different starting water temperature, so the time needed to reach 60°C varies from trial to trial and no fixed actions can succeed. (iii) RGB-only baseline on the same task never succeeds (0%) due to no visible RGB cue like bubbling with the close-lid kettle in our setup.
Data and protocol: We collected 100 demonstrations of heating water. After adapting the pretrained MuseVLA model, we run 12 trials. A trial counts as a success only if the robot presses the button while the measured water temperature is within ±5°C of the 60°C target.
Result: MuseVLA succeeds on 10/12 trials (83.3%). This is a demanding criterion: the temperature rises continuously, so the model must read the measured value accurately at each step and commit to the press inside a narrow window. Achieving it therefore requires genuine fine-grained sensor feedback throughout manipulation/actions.
[1] Vision-Language-Action Instruction Tuning: From Understanding to Manipulation (InstructVLA). International Conference on Learning Representations, 2026.
[2] π0.5: a Vision-Language-Action Model with Open-World Generalization. Conference on Robot Learning, 2025.