acceptodds
Under review as a conference paper at ICLR 2027

Beyond Omni: A Unified MLLM for Embodied Sensory Understanding

Abstract

Recent embodied systems increasingly build on multimodal large language models (MLLMs) for their pretrained understanding and reasoning capabilities, but their perception of the physical world remains largely limited to basic modalities such as vision and audio. However, broadening MLLMs to more diverse sensory modalities from the physical environment is challenging: their representations are often weakly connected to the MLLM and vary substantially in structure and statistics, while their interpretation still requires shared high-level understanding and reasoning. We therefore propose UniSense, a unified embodied MLLM framework that supports diverse physical sensory modalities within a shared multimodal architecture. UniSense follows a two-stage training process: (i) aligning sensory representations with language to establish a language-accessible interface; and (ii) tuning modality-specific experts within the shared LLM to provide specialized processing for different sensory modalities. To support broad sensory learning, we curate UniSense–1M, a large-scale embodied sensory training corpus comprising 1M samples across 8 sensory modalities. We further evaluate UniSense on 12 benchmarks spanning depth, point cloud, IMU, tactile, thermal, radar, motion, and smell understanding. Across these heterogeneous sensing tasks, UniSense achieves strong performance and outperforms both generalist and specialized models on most benchmarks, demonstrating the potential of pretrained MLLMs as unified models for embodied sensory understanding.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Models, Cross-Modal Alignment

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_UeCROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to integrate different modalities, including depth, point clouds, IMU, tactile sensing, thermal imaging, radar, human motion, and smell signals, into a single MLLM.

It augments existing datasets with additional captions/descriptions and some additional QA pairs. It then designs an architecture comprising modality-specific tokenizers that convert raw signals into tokens, a shared encoder, modality-specific projectors, and a shared LLM with modality-specific residual experts. The alignment stage uses three main loss terms: a language-modeling loss for QA, a captioning loss, and a contrastive loss that pulls matched sensory and text representations together and pushes unmatched representations apart.

Critical strengths

The motivation is clear: to build a unified MLLM that can understand different modalities, which could be useful for applications such as robotics and other real-world scenarios.

Both the architecture design and the loss terms are convincing and supported by the reported experiments.

Critical weaknesses

The process for augmenting the data with new descriptions and additional QA pairs should be described more clearly. How much of this constitutes an original contribution is unclear. The authors should be more precise about their contribution in this regard and state whether the dataset and annotations will be publicly released.

The differences from existing work should be stated more clearly, especially the differences from OneLLM, ImageBind, and Octavius. Whether this work is primarily an extension of these methods or introduces substantial differences is not clearly explained.

The experimental comparisons are not sufficiently rigorous, as it is unclear whether the baselines were fine-tuned on the same dataset.

Limitations and potential extensions are not discussed in the main text. Such a discussion would be appreciated.

More details to support reproduction of the experiments would be helpful, including training time, GPU-hours for each stage, batch size, gradient accumulation steps, total optimization steps, total and trainable parameter counts of the added modules, GPU memory usage, training throughput, inference latency, and FLOPs.

Questions for authors

Please see the Weaknesses section.

I am curious whether joint multimodal training offers benefits over training separate modality-specific models. Is there any evidence to support this?

5 comments

We thank reviewer UeCR for the valuable time and constructive feedback.


Q1: Data construction, contribution, and availability.

A1: Sorry for the misunderstanding. As shown in Appendix B.2, UniSense–1M builds on existing sensory datasets. We retain the original annotations for some datasets and generate additional modality-specific captions and QA pairs for others. For example, we use Qwen3-Omni to generate depth-focused captions from RGB–D inputs (Appendix B.2.2). For radar, we use action labels to guide GPT-5.1 in refining existing HoloLLM captions and add questions on laterality and distinctions between similar actions (Appendix B.2.7).

We promise we will release UniSense–1M, including the augmented annotations, together with the annotation pipeline.


Q2: Differences from OneLLM, ImageBind, and Octavius.

A2: We thank the reviewer for raising these works. The key differences are as follows:

  • OneLLM uses a unified encoder and dynamic routing in a universal projection module before the LLM. UniSense also shares an encoder, but introduces modality-specific residual experts inside the LLM to specialize sensory processing.
  • ImageBind aligns modalities in an image-anchored embedding space. UniSense uses sensory–caption supervision through the downstream LLM’s frozen early layers for contrastive alignment and an auxiliary decoder sharing its token embeddings and language-modeling head for caption prediction.
  • Octavius uses separate modality encoders and question-based, example-level LoRA routing. UniSense uses a unified encoder and modality-specific expert pools with token-level routing.

We will clarify these differences in the revision.


Q3: Baseline training settings.

A3: Apologize for the unclear description. Some baselines are fine-tuned on the same official benchmark training split as our method. For example, Chat-Scene, LL3DA, and UniSense all use the official ScanQA training set (Table 3). For thermal QA, ImageBindLLM-7B, PandaGPT-7B, PandaGPT-13B, Infrared-LLaVA-7B, and UniSense all use the Infrared Template Benchmark training set (Table 6, Fine-Tuning column). ScentFormer and UniSense also use the same SmellNet-Base split, training on recordings from Days 1–5 and holding out Day 6 for testing (Table 7c). We will clarify which training splits each baseline uses where this information is available.

Q4: Limitations and potential extensions.

A4: Thank you for your helpful suggestion. UniSense still struggles to distinguish some fine-grained sensory details. Training on more examples that capture these distinctions could help address this limitation. For future work, we will propose a unified benchmark covering all eight sensory modalities to enable a more comprehensive evaluation. Adding action prediction could also extend UniSense to robotic tasks. We will discuss this limitation and both directions in the final manuscript.


Q5: Reproducibility and computational costs.

A5: We use two GPU configurations to train UniSense-9B, one with 16 H100 GPUs across two nodes and one with 8 H100 GPUs. The training hour, batch size, gradient accumulation steps, total optimization steps, total and trainable parameter counts of the added modules, GPU memory usage, training throughput, inference latency, and FLOPs of UniSense-9B are shown in the below table.

MetricSpeech alignmentMultimodal initializationSensory alignmentExpert tuning
GPU configuration2 × 8 H1002 × 8 H1001 × 8 H1001 × 8 H100
Training time (hours)~3~54~13~25
Batch size per GPU4188
Gradient accumulation steps1811
Effective global batch size641286464
Total optimization steps18,53924,39813,70227,404
Total parameters of added modules (M)221381,3991,432
Trainable parameters (M), active updates221381,3991,432
Peak GPU memory (GB/GPU)~48~50~45~45
Training throughput (tokens/s, all GPUs)~38.9K~20.7K~15.6K~15.9K

Q6: About joint multimodal training.

A6: Our core contribution is a single model that supports all eight embodied sensory modalities and achieves competitive results against published modality-specific models (Tables 2–7). To test whether multimodal training enables transfer, we vary the RGB-D training data while keeping general VQA-Speech training fixed. In specific, on SpatialBench-Speech, removing RGB-D-Speech lowers the score by only 1.3 points (66.4 to 65.1). Removing RGB-D-Text as well lowers it by only a further 1.7 points (65.1 to 63.4). This gap suggests that UniSense can transfer reasoning skills learned from written RGB-D questions to spoken ones.

ModelSpatialBenchSpatialBench-Speech
UniSense-9B70.166.4
UniSense-9B (w/o RGB-D-Speech)69.365.1
UniSense-9B (w/o RGB-D-Speech & RGB-D-Text)67.463.4

We also observe a benefit from joint reasoning at inference time. As shown in A2, adding RGB images to point clouds raises ScanQA CIDEr from 74.3 to 99.6. Overall, these results suggest that joint multimodal training facilitates cross-modal transfer.


We appreciate your thoughtful review and we hope to address your concerns. Please let us know if you'd like any further discussion.

We thank reviewer UeCR for your time and thoughtful feedback. We believe our rebuttal has addressed your concerns and would appreciate it if you could consider raising your score. If any questions remain, we would be happy to discuss them further.

Reviewer_5VHdOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UniSense, a multimodal model that integrates basic modalities, such as images and audio, with eight sensory modalities within a unified framework. Built on Qwen3.5, the model incorporates different sensor inputs through modality-specific tokenizers, a shared encoder, and projectors. It follows a two-stage training process: first aligning sensory representations with language through caption generation and contrastive learning, then training modality-specific residual experts within a frozen language model. The authors also organize existing sensor data into unified caption and instruction-based QA formats to construct the UniSense–1M dataset. Evaluations on multiple general multimodal and sensory understanding benchmarks demonstrate the model’s capabilities across different tasks.

Critical strengths
  • Broad coverage of sensory modalities. The model supports eight sensory modalities: depth, point clouds, IMU, tactile, thermal, radar, motion, and smell. These signals differ in structure and physical meaning. Integrating them into a single model has practical engineering value and broadens the model’s ability to understand physical environments.

  • Comprehensive experiments. The paper presents both 2B and 9B models, with extensive comparisons on general multimodal tasks and tasks involving all eight sensory modalities. The results show strong performance on the evaluated tasks and highlight the potential of combining sensor inputs with images and other modalities.

Critical weaknesses
  • The ablations do not fully explain the source of the performance gains. The full model assigns three experts to each of the eight modalities, while the shared-expert baseline has only three experts in total. It is difficult to tell whether the gains come from modality-specific specialization or simply from having more trainable parameters. Although the paper compares different loss combinations, it lacks a sensitivity analysis of the loss weights and ablations for the load-balancing and expert-diversity losses.

  • Some architecture and training details remain unclear. Appendix C.4 describes a caption decoder with a hidden dimension of 1024 that reuses Qwen’s frozen embeddings and output head, but does not explain how the dimensions are matched. Appendix C.2 also leaves unclear how non-image sensor tokens are assigned positional encodings and grouped for the visual merger. Important training details should also be provided.

Questions for authors

I am not very familiar with omni models or embodied AI, and clarifying the following points would help me better understand and assess the paper.

  1. How many parameters do the modality-specific residual experts add, and what proportions of the total and trainable parameters do they represent? Does their advantage over the shared expert pool remain when the total expert parameter counts are matched?
  2. For non-pixel modalities such as IMU, radar, and smell, how are positional encodings assigned in the shared vision encoder?
  3. How were the loss weights reported in Appendix C.6 selected, and how sensitive are the results to these choices?
6 comments

We thank reviewer 5VHd for the valuable time and constructive feedback.


Q1: Expert parameter counts and parameter-matched comparison.

A1: The modality-specific residual experts add 66.06M parameters, accounting for 0.55% of the total parameters and 4.61% of the trainable parameters during the expert tuning.

UniSense-9B uses 3 experts per modality and 24 experts per layer across eight modalities. We compare UniSense with two shared-expert baselines:

  • 3 shared experts per layer (also shown in Fig. 5(b)): All modalities share the same 3 experts. This matches the number of active experts per token but has fewer expert parameters than our method.
  • 24 shared experts per layer with top-3 routing (R-Table 1): Each token selects 3 experts from a shared pool of 24. This matches both the total expert parameter count and the number of active experts per token.

As shown in R-Table 1, UniSense outperforms both shared-expert baselines on all three benchmarks, retaining its advantage when expert parameter counts are matched.

R-Table 1. Parameter-matched expert comparison on UniSense-9B

SettingTotal experts per layerActive experts per sensory tokenRoboSpatial-Home Config. Acc. ↑ScanQA (XYZ + Image) CIDEr ↑MMFi-VQA Acc. ↑
Shared-3 (Fig. 5b)3379.697.663.0
Shared-24 top-324378.998.563.8
Modality-specific (Ours)24380.599.665.2

Q2: More ablations.

A2: We conduct a sensitivity analysis of the alignment and expert-diversity weights. We also ablate the load-balancing and expert-diversity losses separately. We evaluate UniSense-9B on three sensing modalities using RoboSpatial-Home for depth, ScanQA for point clouds, and MMFi-VQA for radar.

Alignment weight. R-Table 2 compares different alignment weights λalign\lambda_{\mathrm{align}} in Eqs. (5) and (6): 0.05, 0.08, and 0.10. The default value of 0.08 yields the best results on ScanQA and MMFi-VQA. Increasing the weight to 0.10 slightly improves RoboSpatial-Home accuracy but reduces scores on the other two benchmarks. Lowering it to 0.05 slightly reduces all three scores. These results suggest that performance is relatively stable over the tested range.

R-Table 2. Sensitivity to the alignment weight.

Alignment weightRoboSpatial-Home Config. Acc. ↑ScanQA (XYZ + Image) CIDEr ↑MMFi-VQA Acc. ↑
0.0579.296.764.0
0.08 (Ours)80.599.665.2
0.1081.398.264.7

Expert-diversity weight. We vary the value of λdiv\lambda_{\mathrm{div}} in Eq. (6) across 0.05, 0.07, and 0.10. As R-Table 3 shows, performance remains relatively stable across these settings. We use 0.07 as the default because it yields the highest scores on all three benchmarks.

R-Table 3. Sensitivity to the expert-diversity weight.

Diversity target weightRoboSpatial-Home Config. Acc. ↑ScanQA (XYZ + Image) CIDEr ↑MMFi-VQA Acc. ↑
0.0579.798.964.4
0.07 (Ours)80.599.665.2
0.1078.998.464.9

Ablations for the load-balancing and expert-diversity losses. Removing the Switch load-balancing loss lowers performance on all three benchmarks, while removing the expert-diversity loss lowers performance on two (R-Table 4). These results support retaining both losses in the training objective.

R-Table 4. Ablations for the load-balancing and expert-diversity losses.

SettingRoboSpatial-Home Config. Acc. ↑ScanQA (XYZ + Image) CIDEr ↑MMFi-VQA Acc. ↑
Ours80.599.665.2
w/o Load-balancing78.997.063.6
w/o Diversity80.496.563.4

Q3: Details about architecture and training.

A3: Caption decoder dimensions (Appendix C.4). At the decoder input, a trainable linear layer projects the 4,096-dimensional caption token embeddings from Qwen’s frozen text embedding layer to the decoder’s hidden size of 1,024. At the decoder output, a separate trainable linear layer projects the hidden states from 1,024 back to 4,096 dimensions to match the input size of Qwen’s frozen LM head.

Positional encoding and visual merger (Appendix C.2). For non-pixel inputs, modality-specific tokenizers produce token sequences for the shared vision encoder and visual merger. For IMU, radar, and smell, we assign each sensor token row and column indices based on its position in the sequence. The shared vision encoder uses these indices to add interpolated 2D absolute position embeddings and apply 2D RoPE. For the visual merger, we group every four consecutive sensor tokens into a 2×22 \times 2 block and combine their encoded representations into a single token. We will document the corresponding rules for the remaining non-pixel modalities in Appendix C.2.

Training details (Appendix C.6). For UniSense-9B, we use two nodes with eight H100 GPUs each for speech alignment and multimodal initialization. For sensory alignment and expert tuning, we use eight H100 GPUs, BF16 precision, and an effective global batch size of 64. We will provide further stage-specific training details in the revision.


We appreciate your thoughtful review and we hope to address your concerns. Please let us know if you'd like any further discussion.

We thank reviewer 5VHd for your time and thoughtful feedback. We believe our rebuttal has addressed your concerns and would appreciate it if you could consider raising your score. If any questions remain, we would be happy to discuss them further.

Reviewer_VJHxOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

UniSense is a method that aims at unifying the pipeline of aligning various sensor modalities (depth, point cloud, IMU, tactile, thermal, radar, skeleton motion, and smell) into a language model's representation to enable multi-modal reasoning tasks. The authors designed tokenizers specifically for each modality and trained them via alignment with synthetic captions + contrastive loss. A SwiGLU LoRA based MoE architecture was also used for training the attention layers itself.

Critical strengths

the obvious strength of the paper is the substantial empirical efforts that went into this work. The curated 1M samples across 8 sensor types, evaluated across multiple sensor benchmarks and general multimodal benchmarks are quite comprehensive, and combined with ablation studies, show the effectiveness of the proposed design.

Critical weaknesses

Although the composition of methods on this paper is novel, the individual components that was used here (contrastive loss for multi-modal alignment, MoE for multi-tasking) are not new, so a big part of the contribution depends on if the unified understanding is actually achieved in this design. Exhaustive experiments showed that the individual benchmarks on each modality are assessed, but analysis on actual joint reasoning that involves multiple modalities, or genuine modality transfer is less investigated. Some experimental flaws also exist such as window-level holdouts instead of session level isolation between training/testing (IMU evaluation).

Questions for authors

IMU Split Protocol (Material to assessment): In Appendix B.2.4, are the 2.5k held-out evaluation windows for HHAR, MotionSense, Shoaib, and UCI-HAR strictly subject-disjoint / recording-session-disjoint from the 23.8k training windows, or were windows sampled randomly across the pooled sliding windows?

4 comments

We thank reviewer VJHx for the valuable time and constructive feedback.


Q1: The novelty of our method.

A1: We thank the reviewer for recognizing the novelty of our framework design. First, our core contribution is a unified framework that integrates eight heterogeneous embodied sensory modalities into a single MLLM. Prior work typically focused on individual modalities or limited combinations, while broader sensory unification remains underexplored. Moreover, we conduct a comprehensive evaluation of the unified model across 12 benchmarks covering 8 embodied modalities. To our knowledge, few prior studies have evaluated embodied sensory understanding across such a wide range of modalities.

Second, our design differs from prior work in two architectural choices: i) Sensory representation alignment. Prior methods align modalities in shared embedding spaces (e.g., ImageBind and LanguageBind), or combine contrastive learning with caption prediction during encoder pretraining (e.g., CoCa). In contrast, UniSense uses sensory and caption features derived from the same frozen early layers of the downstream LLM’s frozen early layers for contrastive alignment. This encourages the downstream LLM to produce similar representations for a sensory input and its corresponding textual description. ii) Modality-specific expert tuning. Previous methods either route experts before the LLM (e.g., OneLLM) or select an expert mixture for the whole example (e.g., Octavius). UniSense assigns each sensory modality its own expert pool inside the shared LLM, with expert weights that can vary across tokens within the same example.

Third, Table 8 presents ablation results by first adding sensory representation alignment and then introducing modality-specific experts. Both components consistently improve performance across all five benchmarks. For example, RoboSpatial-Home Configuration accuracy increases from 74.0 to 77.2 and then to 80.5, while MMFi-VQA accuracy increases from 60.1 to 63.5 and then to 65.2. These results show that sensory alignment improves performance, and modality-specific experts provide further gains beyond alignment alone.

We will clarify these contributions and their distinctions from prior approaches in the revised manuscript.


Q2: Analysis on joint reasoning and modality transfer.

A2: Thank you for raising this question. UniSense processes tokens from different modalities in a shared backbone, allowing them to interact through self-attention. On ScanQA, adding RGB images to point clouds improves UniSense-9B’s CIDEr score from 74.3 to 99.6 (Table 3). This 25.3-point gain shows that RGB input improves performance on 3D question answering.

Our depth-training data include RGB-D-Text samples with written questions and RGB-D-Speech samples with synthesized spoken questions. With general VQA-Speech training retained, removing RGB-D-Speech alone lowers the SpatialBench-Speech score by only 1.3 points. Removing RGB-D-Text as well lowers it by a further 1.7 points. These results suggest that general VQA-Speech training helps UniSense transfer the reasoning ability on RGB-D-Text to RGB-D-Speech.

ModelSpatialBenchSpatialBench-Speech
UniSense-9B70.166.4
UniSense-9B (w/o RGB-D-Speech)69.365.1
UniSense-9B (w/o RGB-D-Speech & RGB-D-Text)67.463.4

Q3: IMU evaluation details.

A3: Thank you for raising this question. We randomly sample evaluation windows from each dataset’s preprocessed window pool, without enforcing a strictly subject-disjoint or recording-session-disjoint split. Sampling is stratified by activity class, following LLaSA’s class-balanced evaluation procedure. We use non-overlapping windows and verify that no evaluation window path appears in the training set. The IMU results in Table 5 reflect within-dataset activity recognition under this window-level protocol. We will clarify the split protocol and evaluation scope in the revision.


We appreciate your thoughtful review and we hope to address your concerns. Please let us know if you'd like any further discussion.

We thank reviewer VJHx for your time and thoughtful feedback. We hope our responses have helped address your concerns. We would welcome any further feedback and would be happy to provide additional clarification.

Thanks for the clarification. I consider my main concerns addressed and now recognized the novelty of the work. I will raise my initial rating to clear accept.

Thank you for reconsidering our work and for raising your rating. We’re glad our clarifications addressed your main concerns. We will incorporate these clarifications into the final paper. Thank you again for your time and feedback.

Comments on the paper

1 comment

We thank all the reviewers for their time and positive comments: “clear motivation” (UeCR); “broad coverage of sensory modalities” and “practical engineering value” (5VHd); “substantial empirical efforts” (VJHx) and “Comprehensive experiments” (5VHd); and “Both the architecture design and the loss terms are convincing and supported by the reported experiments” (UeCR).

Reviewer VJHx asks about novelty, evidence for joint reasoning and cross-modal transfer, and the IMU evaluation protocol. We explain how our sensory alignment and expert routing differ from prior approaches. To assess joint reasoning and cross-modal transfer, we show that adding RGB images improves point-cloud question answering and that training on written RGB-D questions improves performance on spoken ones. We clarify that the IMU evaluation holds out windows without enforcing a subject- or session-disjoint split.

Reviewer 5VHd asks the gains from modality-specific experts as well as more training and architecture detailds. Our comparison shows that modality-specific experts outperform shared experts on all three benchmarks. We also examine the effects of varying the loss weights and removing each auxiliary expert loss. Morevoer, we clarify the architecture and training details.

Reviewer UeCR asks the data annotation and training details, differences from prior works, baseline training details, limitation and potential extensions. We explain which original annotations UniSense–1M retains and how we generate additional captions and QA pairs. More training details for each stage are also reported. We clarify how UniSense differs from OneLLM, ImageBind, and Octavius in sensory alignment and expert routing, and identify baselines that use the same benchmark training splits as UniSense. On joint training, we show that UniSense outperforms some published modality-specific models and is competitive with others, and we provide evidence of transfer from written to spoken RGB-D questions.

We believe these results and clarifications address the reviewers’ main concerns, and we welcome further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.