- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents Omni-Mono, a vision-language-action framework that integrates joint multimodal reasoning with modality-specific refinement for robotic manipulation. Motivated by the limitations of existing reasoning-based VLA approaches, which often process textual, visual, and action reasoning independently or in a predefined sequence, the proposed method aims to enable mutual information exchange across different reasoning modalities.
The framework consists of two main components. First, the Omni Reasoner jointly produces textual, visual, and action latent representations through grouped attention with modality relation masks, enabling structured information exchange across reasoning streams. Second, the Mono Reasoner employs dedicated modality-specific experts with adaptive guidance fusion to progressively refine these representations in text–vision–action order. The refined representations, together with the original task context, condition a flow-matching action head for action-chunk generation. The framework is trained with auxiliary supervision derived from textual plans, future visual features, and sparse reference actions.
The authors evaluate Omni-Mono on LIBERO, LIBERO-Plus, SimplerEnv, RoboTwin 2.0, and two real-world robotic manipulation tasks. The proposed method demonstrates strong performance across these benchmarks, with additional ablations and controlled interventions investigating the contributions of multimodal communication and modality-specific refinement.
Critical strengths
-
The paper addresses a meaningful challenge in reasoning-based robotic manipulation: enabling different reasoning modalities to exchange information rather than treating them as independent or strictly sequential processes. The proposed combination of an Omni Reasoner for joint multimodal information exchange and a Mono Reasoner for modality-specific refinement is conceptually well motivated. The grouped attention mechanism provides an explicit architectural structure for cross-modal communication, while the dedicated refinement experts aim to preserve modality-specific information. I find the overall design interesting, particularly the attempt to reconcile shared multimodal representations with specialized reasoning for semantic planning, future-state prediction, and motion generation.
-
The experimental results are impressive across several established robotic manipulation benchmarks. In particular, Omni-Mono achieves 92.4% success on LIBERO-Plus and 77.2% on RoboTwin Clean-to-Random, outperforming the strongest compared reasoning baseline by 5.2 and 6.0 percentage points, respectively. The improvements under distribution shifts are especially encouraging, given the importance of robustness for generalizable robot manipulation. The paper also includes real-world evaluations on a Franka robot, where Omni-Mono achieves an average success rate of 86.0%, compared with 75.0% for π0.5 under the reported evaluation protocol. Together, these results provide promising evidence for the effectiveness of the proposed framework across simulation and physical manipulation settings.
-
A notable strength of this paper is the effort to experimentally validate the proposed reasoning architecture beyond overall task success rates. In addition to standard Omni-only and Mono-only ablations, the authors provide capacity-controlled comparisons that retain four independent language backbones, helping distinguish architectural contributions from increased model capacity. The directional information-flow experiments further examine the effects of restricting cross-modal communication, while the controlled obstacle experiments evaluate task success, collision rates, and textual plan validity under conflicting spatial constraints. The visual-to-text intervention is particularly interesting, as it provides evidence that modifying visual guidance can influence textual representations and downstream execution. Although such interventions have limitations, I appreciate the authors' effort to investigate the proposed mechanism more directly.
Critical weaknesses
-
My primary concern is the computational complexity of the proposed framework and whether it is sufficiently justified by the additional performance gains. As described in Appendix A.1, the Omni Reasoner and three Mono experts employ four independent language backbones initialized from Qwen3-VL-4B-Instruct, with their language parameters independently fine-tuned. This results in a substantially larger architecture than conventional single-backbone VLA approaches. The authors have made a valuable effort to address the capacity concern through controlled comparisons. However, the improvements over the capacity-matched generic refinement baseline remain relatively modest: 1.7, 1.4, and 1.8 percentage points on LIBERO-Plus, SimplerEnv, and RoboTwin C2R, respectively. Furthermore, the reported inference latency of Omni-Mono is 680 ms on an A100 GPU, compared with 170 ms for the direct-action baseline under the same internal measurement protocol. This fourfold increase raises practical questions about the scalability and efficiency of the proposed architecture. While the controlled ablations provide evidence that the reasoning structure contributes beyond additional model capacity, the paper does not sufficiently explore whether comparable benefits could be achieved using a more parameter-efficient architecture, such as shared reasoning backbones or lightweight modality-specific refinement modules. A more explicit analysis of performance relative to model size and computational requirements would help establish the practical significance of the proposed approach.
-
Although the paper includes physical robot experiments, the evaluation is restricted to two relatively simple tabletop pick-and-place tasks: placing a milk box onto a plate and placing a bowl onto a plate.
These experiments demonstrate that the proposed framework can successfully execute physical manipulation tasks and improve success rates over the compared baselines. However, they do not directly evaluate the primary motivation of the paper: resolving inconsistencies or conflicts between textual, visual, and action reasoning during complex execution.
For example, the real-world tasks do not explicitly require obstacle-aware replanning, reasoning over multiple competing constraints, or revising an initial plan in response to unexpected environmental changes. The controlled obstacle experiments in simulation partially address this concern and represent a useful contribution. Nevertheless, extending such evaluations to physical robotic settings would provide stronger evidence that the proposed mutually informed reasoning mechanism remains effective under real-world perception uncertainty and execution errors. I believe that more challenging real-world tasks involving explicit reasoning dependencies or conflicting spatial constraints would substantially strengthen the paper's central claims.
- The proposed framework enables structured cross-modal communication through relation-masked attention and adaptive guidance fusion. However, the extent to which these mechanisms implement meaningful reasoning refinement, rather than improved multimodal feature integration, remains somewhat unclear. The directional information-flow ablations and visual-to-text intervention provide encouraging evidence that information from one modality can influence other representations and affect downstream behavior. Nevertheless, these experiments do not fully characterize how reasoning representations are revised when different modalities provide conflicting information. In particular, the visual-to-text intervention introduces representations from a different scene, which may create out-of-distribution feature combinations. Although the authors appropriately acknowledge this limitation, additional analyses would help clarify the causal relationship between cross-modal communication, plan consistency, and action generation. Furthermore, the paper jointly supervises textual, visual, and action representations at both reasoning stages, but does not sufficiently isolate the contribution of each individual auxiliary supervision objective. Such ablations would help determine whether the observed benefits arise primarily from the proposed information-flow architecture, the richer intermediate supervision, or their combination.
Questions for authors
-
Could the authors further justify the use of four independent Qwen3-VL-4B language backbones? Although the capacity-controlled experiments are appreciated, the improvements over generic refinement are relatively modest. Would it be possible to evaluate a shared-backbone architecture or a lighter modality-specific refinement design? A comparison of success rates, inference latency, and memory consumption across different model capacities would help clarify whether the proposed benefits require the full architectural complexity.
-
Could the authors extend the real-world experiments to tasks that more directly require cross-modal reasoning? For example, tasks involving obstacle avoidance, competing spatial constraints, multi-step planning, or unexpected environmental changes would be particularly relevant to the motivation presented in Figure 1. It would also be valuable to compare the full model with unidirectional reasoning or generic refinement on these tasks, to directly validate the benefits of mutual information exchange during physical execution.
-
Could the authors provide additional analyses demonstrating how the reasoning representations are modified in response to conflicting information from different modalities? For example, in the controlled obstacle scenarios, it would be interesting to examine whether the textual plan representation changes consistently when visual reasoning identifies an obstacle, and how such changes affect the generated actions. Additionally, could the authors independently ablate the textual, visual, and action auxiliary supervision losses to clarify their respective contributions to the overall performance gains? These experiments would help better distinguish the benefits of the cross-modal reasoning architecture from those of intermediate multimodal supervision.
7 comments
Response to Reviewer AJsV 2/4
-
Controlled comparisons with a shared backbone and lightweight refinement
We have added controlled experiments with a shared backbone and lightweight modality-specific refinement to evaluate the performance benefits and computational costs of four independent backbones. Specifically, we retain the full Omni Reasoner and construct two alternative configurations. The first has all three Mono experts share a single language backbone while retaining independent modality queries and adaptive guidance modules. The second replaces each of the three Mono experts with an independent 12-layer Transformer to reduce the parameter count and computational cost of refinement. Both variants retain the text→vision→action execution order, all auxiliary supervision objectives, and the final action head.
Model configuration LIBERO-Plus Avg.(%)↑ SimplerEnv Avg.(%)↑ Inference latency (ms) ↓ Relative peak GPU memory at inference ↓ Full Omni–Mono 92.4 78.3 680 1.00 Shared Mono backbone 90.5 75.6 544 0.58 Lightweight Mono 88.7 74.5 430 0.37 The results show that the full Omni–Mono model achieves the highest task success rates among the compared configurations. Sharing the Mono backbone reduces parameter storage overhead, but provides limited improvement in inference latency; lightweight Mono reduces computation while lowering success rates to some extent. We believe that sharing the language backbone restricts each expert's ability to independently adapt its parameters to different modalities, while reducing the number of refinement layers limits the processing of fine-grained information. Both simplifications may affect Mono's dedicated refinement of semantic plans, future-scene information, and motion details. These results therefore support using independent modality experts with sufficient capacity: among the architectures compared in this experiment, the full Omni–Mono design with joint cross-modal reasoning and modality-specific refinement achieves the highest task success rates.
Q2: Can the method be validated on real-robot tasks with greater cross-modal reasoning demands and compared with unidirectional reasoning or generic refinement?
"Could the authors extend the real-world experiments to tasks that more directly require cross-modal reasoning? For example, tasks involving obstacle avoidance, competing spatial constraints, multi-step planning, or unexpected environmental changes would be particularly relevant to the motivation presented in Figure 1. It would also be valuable to compare the full model with unidirectional reasoning or generic refinement on these tasks, to directly validate the benefits of mutual information exchange during physical execution."
A2: To strengthen the comparison in real-world settings, we have added two real-robot tasks involving object retrieval and placement and sequential stacking. The first task is Microwave-to-Plate Transfer: the robot must retrieve an object from inside a microwave and place it on a plate. The second task is Sequential Fruit Stacking: the robot must first place a banana on a plate and then place a peach on top of the banana. Real-world illustrations of these complex tasks are provided in Figure 7 of Appendix F.5 in the revised manuscript. These tasks involve object retrieval and placement and fruit stacking in a specified order, respectively. We include two structural controls on these tasks: Unidirectional and Generic refinement. Both retain four backbones, auxiliary supervision at both stages, and the final action head. Unidirectional restricts cross-modal information flow to text→vision, text→action, and vision→action, while Generic refinement replaces modality-specific experts with mixed-modality refinement modules.
| Method | Method type / control configuration | Sequential Fruit Stacking: SR(%)↑ | Microwave-to-Plate Transfer: SR(%)↑ |
|---|---|---|---|
| π0.5 | General-purpose VLA | 38 | 54 |
| Unidirectional | Four backbones, unidirectional cross-modal reasoning | 44 | 64 |
| Generic refinement | Four backbones, generic mixed-modality refinement | 42 | 68 |
| Omni–Mono | Full cross-modal interaction and modality-specific refinement | 56 | 76 |
The added unidirectional and generic-refinement controls further examine whether full cross-modal interaction and modality-specific refinement continue to improve real-world robot performance when the number of backbones and auxiliary supervision are held constant.
Response to Reviewer AJsV 3/4
Q3: How do reasoning representations adapt to cross-modal conflicts, and how does this affect plan consistency and action execution?
"Could the authors provide additional analyses demonstrating how the reasoning representations are modified in response to conflicting information from different modalities? For example, in the controlled obstacle scenarios, it would be interesting to examine whether the textual plan representation changes consistently when visual reasoning identifies an obstacle, and how such changes affect the generated actions."
A3: To further explain how cross-modal constraints affect reasoning representations and action execution, we have added an analysis of plan adaptation and execution consistency using the paired obstacle scenes in Appendix D.5. The experiment holds the task instruction, manipulated object, target location, and robot state fixed, changing only whether an obstacle blocks the transport path. We decode plans from the Omni and Mono textual states under clear-path and blocked-path conditions and record the actual execution trajectories. Decoding is used only for diagnosis, and the generated text is not fed back to the control policy. The experiment compares the full Omni–Mono model, Unidirectional, and LaRA-VLA, with LaRA-VLA serving as an external baseline that organizes textual latent reasoning and future visual prediction through unidirectional dependencies. We focus on whether the model can form a new plan that satisfies the obstacle constraints when the original transport plan no longer meets the current spatial constraints, and whether the corresponding adjustment is realized in action execution. For example, when a direct transport path is blocked, we check whether the plan introduces a feasible lifting or detouring operation and whether the actual trajectory reflects that adjustment.
| Method | Omni blocked-scene plan validity rate (%) ↑ | Final blocked-scene plan validity rate (%) ↑ | Final plan adaptation rate (%) ↑ | Plan–action consistency rate (%) ↑ |
|---|---|---|---|---|
| Unidirectional | 60 | 65 | 66 | 74 |
| LaRA-VLA | — | 66 | 64 | 72 |
| Omni–Mono | 66 | 74 | 78 | 84 |
The metrics are calculated as follows and are all reported as percentages:
-
Blocked-scene plan validity rate = number of blocked-scene samples with valid decoded plans / total number of blocked-scene samples × 100%. A valid plan must satisfy the instructed operation, object, and target location, and specify a feasible path adjustment under the current obstacle constraints. Omitting a required avoidance operation or proposing an infeasible path is counted as invalid.
-
Final plan adaptation rate = number of scene pairs requiring adjustment in which recomputation produces a valid final plan / total number of scene pairs requiring adjustment × 100%. Specifically, we first decode the original plan from the final textual state in the clear-path scene. A sample is counted as requiring adjustment only if this plan is valid in the clear-path scene but becomes invalid due to obstacle constraints when directly applied to the paired blocked-path scene. We then check whether the final plan obtained by recomputing the model from the blocked observation is valid. Samples whose original plan remains valid in both scenes are excluded from the denominator, and the model is not required to change its plan unnecessarily.
-
Plan–action consistency rate = number of blocked-scene samples in which the path adjustment required by a valid final plan is reflected in actual execution / number of blocked-scene samples with valid final plans × 100%. For example, if a plan requires lifting before transport, the trajectory should reflect the height adjustment needed to clear the obstacle; if a plan requires a detour, the trajectory should pass around the obstacle. An action change that does not satisfy the avoidance adjustment required by the plan is not counted as consistent. This metric evaluates the correspondence between valid plans and actual actions, rather than overall task success.
The results show that when an obstacle invalidates the original plan, Omni–Mono's textual reasoning representations can adapt: its decoded plans more effectively reflect avoidance requirements such as lifting or detouring, and the corresponding adjustments are reflected in actual actions. Compared with the unidirectional control, the higher plan adaptation and plan–action consistency rates support the interpretation that cross-modal interaction helps coordinate environmental constraints, plan updates, and action execution.
Response to Reviewer AJsV 1/4
We sincerely thank the reviewer for the thoughtful and constructive feedback. We have organized the reviewer's comments into the following questions and responses.
Q1: Do the performance benefits of four independent backbones justify their computational cost? Can shared-backbone or lighter refinement architectures be evaluated?
A1:
Rationale for independent backbones
Omni and Mono serve different modeling purposes. Omni organizes information exchange among textual, visual, and reference-action states through relation masks, allowing each modality to incorporate constraints from the others to form coordinated reasoning states. The three Mono experts refine semantic plans, future-scene information, and motion information, respectively, and adaptively select guidance from Omni and preceding experts based on their current query states. As described in Appendix A.1, the four reasoning backbones use the same Qwen3-VL-4B-Instruct initialization, but their language parameters are fine-tuned independently; the input visual encoder and task context are shared. Independent parameters allow each expert to learn transformations specific to its modality's objectives. This is the current implementation's design choice for preserving modality-specific details, while whether shared backbones or lightweight modules can achieve similar benefits requires direct comparison.
Latency and performance analysis
Compared with other methods, Omni–Mono achieves the best performance while maintaining relatively low latency. In the internal ablations, Omni–Mono has higher latency than Omni-only at 247 ms and Mono-only at 494 ms, but also increases the average LIBERO-Plus success rate from 87.4% and 88.0%, respectively, to 92.4%, gains of 5.0 and 4.4 percentage points. This shows that the additional computation from joint reasoning and modality-specific refinement is accompanied by improved task performance, making Omni–Mono an option that balances efficiency and performance.