- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper aims to integrate different modalities, including depth, point clouds, IMU, tactile sensing, thermal imaging, radar, human motion, and smell signals, into a single MLLM.
It augments existing datasets with additional captions/descriptions and some additional QA pairs. It then designs an architecture comprising modality-specific tokenizers that convert raw signals into tokens, a shared encoder, modality-specific projectors, and a shared LLM with modality-specific residual experts. The alignment stage uses three main loss terms: a language-modeling loss for QA, a captioning loss, and a contrastive loss that pulls matched sensory and text representations together and pushes unmatched representations apart.
Critical strengths
The motivation is clear: to build a unified MLLM that can understand different modalities, which could be useful for applications such as robotics and other real-world scenarios.
Both the architecture design and the loss terms are convincing and supported by the reported experiments.
Critical weaknesses
The process for augmenting the data with new descriptions and additional QA pairs should be described more clearly. How much of this constitutes an original contribution is unclear. The authors should be more precise about their contribution in this regard and state whether the dataset and annotations will be publicly released.
The differences from existing work should be stated more clearly, especially the differences from OneLLM, ImageBind, and Octavius. Whether this work is primarily an extension of these methods or introduces substantial differences is not clearly explained.
The experimental comparisons are not sufficiently rigorous, as it is unclear whether the baselines were fine-tuned on the same dataset.
Limitations and potential extensions are not discussed in the main text. Such a discussion would be appreciated.
More details to support reproduction of the experiments would be helpful, including training time, GPU-hours for each stage, batch size, gradient accumulation steps, total optimization steps, total and trainable parameter counts of the added modules, GPU memory usage, training throughput, inference latency, and FLOPs.
Questions for authors
Please see the Weaknesses section.
I am curious whether joint multimodal training offers benefits over training separate modality-specific models. Is there any evidence to support this?
5 comments
Q4: Limitations and potential extensions.
A4: Thank you for your helpful suggestion. UniSense still struggles to distinguish some fine-grained sensory details. Training on more examples that capture these distinctions could help address this limitation. For future work, we will propose a unified benchmark covering all eight sensory modalities to enable a more comprehensive evaluation. Adding action prediction could also extend UniSense to robotic tasks. We will discuss this limitation and both directions in the final manuscript.
Q5: Reproducibility and computational costs.
A5: We use two GPU configurations to train UniSense-9B, one with 16 H100 GPUs across two nodes and one with 8 H100 GPUs. The training hour, batch size, gradient accumulation steps, total optimization steps, total and trainable parameter counts of the added modules, GPU memory usage, training throughput, inference latency, and FLOPs of UniSense-9B are shown in the below table.
| Metric | Speech alignment | Multimodal initialization | Sensory alignment | Expert tuning |
|---|---|---|---|---|
| GPU configuration | 2 × 8 H100 | 2 × 8 H100 | 1 × 8 H100 | 1 × 8 H100 |
| Training time (hours) | ~3 | ~54 | ~13 | ~25 |
| Batch size per GPU | 4 | 1 | 8 | 8 |
| Gradient accumulation steps | 1 | 8 | 1 | 1 |
| Effective global batch size | 64 | 128 | 64 | 64 |
| Total optimization steps | 18,539 | 24,398 | 13,702 | 27,404 |
| Total parameters of added modules (M) | 22 | 138 | 1,399 | 1,432 |
| Trainable parameters (M), active updates | 22 | 138 | 1,399 | 1,432 |
| Peak GPU memory (GB/GPU) | ~48 | ~50 | ~45 | ~45 |
| Training throughput (tokens/s, all GPUs) | ~38.9K | ~20.7K | ~15.6K | ~15.9K |
Q6: About joint multimodal training.
A6: Our core contribution is a single model that supports all eight embodied sensory modalities and achieves competitive results against published modality-specific models (Tables 2–7). To test whether multimodal training enables transfer, we vary the RGB-D training data while keeping general VQA-Speech training fixed. In specific, on SpatialBench-Speech, removing RGB-D-Speech lowers the score by only 1.3 points (66.4 to 65.1). Removing RGB-D-Text as well lowers it by only a further 1.7 points (65.1 to 63.4). This gap suggests that UniSense can transfer reasoning skills learned from written RGB-D questions to spoken ones.
| Model | SpatialBench | SpatialBench-Speech |
|---|---|---|
| UniSense-9B | 70.1 | 66.4 |
| UniSense-9B (w/o RGB-D-Speech) | 69.3 | 65.1 |
| UniSense-9B (w/o RGB-D-Speech & RGB-D-Text) | 67.4 | 63.4 |
We also observe a benefit from joint reasoning at inference time. As shown in A2, adding RGB images to point clouds raises ScanQA CIDEr from 74.3 to 99.6. Overall, these results suggest that joint multimodal training facilitates cross-modal transfer.
We appreciate your thoughtful review and we hope to address your concerns. Please let us know if you'd like any further discussion.
We thank reviewer UeCR for your time and thoughtful feedback. We believe our rebuttal has addressed your concerns and would appreciate it if you could consider raising your score. If any questions remain, we would be happy to discuss them further.
We thank reviewer UeCR for the valuable time and constructive feedback.
Q1: Data construction, contribution, and availability.
A1: Sorry for the misunderstanding. As shown in Appendix B.2, UniSense–1M builds on existing sensory datasets. We retain the original annotations for some datasets and generate additional modality-specific captions and QA pairs for others. For example, we use Qwen3-Omni to generate depth-focused captions from RGB–D inputs (Appendix B.2.2). For radar, we use action labels to guide GPT-5.1 in refining existing HoloLLM captions and add questions on laterality and distinctions between similar actions (Appendix B.2.7).
We promise we will release UniSense–1M, including the augmented annotations, together with the annotation pipeline.
Q2: Differences from OneLLM, ImageBind, and Octavius.
A2: We thank the reviewer for raising these works. The key differences are as follows:
We will clarify these differences in the revision.
Q3: Baseline training settings.
A3: Apologize for the unclear description. Some baselines are fine-tuned on the same official benchmark training split as our method. For example, Chat-Scene, LL3DA, and UniSense all use the official ScanQA training set (Table 3). For thermal QA, ImageBindLLM-7B, PandaGPT-7B, PandaGPT-13B, Infrared-LLaVA-7B, and UniSense all use the Infrared Template Benchmark training set (Table 6, Fine-Tuning column). ScentFormer and UniSense also use the same SmellNet-Base split, training on recordings from Days 1–5 and holding out Day 6 for testing (Table 7c). We will clarify which training splits each baseline uses where this information is available.