- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Authors propose ROMA, an LLM-based system that decides when vision is not enough, and then interacts with objects to collect audio, tactile and force feedback. The paper builds a dataset, ROMI-2K. It has a handheld part with 1,657 object-content combinations and a robot-arm part with 500 tabletop scenes (400 for training and 100 for testing). Each object is recorded under six actions (lift, press, collide, shake, rotate, squeeze) with four synchronised modalities. The labels include whether a property needs interaction and which modalities can show what is inside an object. ROMA-7B is built on Qwen2.5-Omni with special action and modality tokens, and is trained by alignment and then SFT. The robot side uses AnyGrasp with point cloud completion and grasp refinement. On ROMA Bench (2,100 tasks) and 132 real-robot tasks, ROMA-7B is better than GPT-5.4, Gemini 3.5 Flash and two Qwen-Omni models, and uses fewer interactions than trying all actions on all objects.
Critical strengths
This paper makes a useful contribution to multi-sensory active perception. ROMI-2K covers many objects and many types of interaction with four synchronised modalities, which existing datasets do not provide together. The labels of whether a property needs interaction, and which modalities can show what is inside an object, are especially useful. They are the labels needed to learn when an interaction is worth doing. I expect the dataset and ROMA Bench to be used by other works. In addition, the whole system runs on a real robot, and the paper gives enough details to reproduce it, such as point cloud completion for transparent objects, grasp depth compensation and the two-stage gripping force.
Critical weaknesses
I have the following concerns that the authors should address in the final version:
- The evaluation is small. The real-robot test has 132 tasks from 8 scenes with a single run, and the 325 object instances in ROMA Bench come from only 56 test objects. Repeated runs, error bars, and the number of tasks of each type would help to show whether the improvements over the baselines are reliable.
- The object labels are generated by Gemini 3.5 Flash and the intent-driven questions by GPT-5.4, and both models are also used as baselines. I wonder whether this affects the comparison. The paper should also explain how the labels were checked by hand.
- To my knowledge, choosing exploratory actions has been studied before: “Multi-modal Predicate Identification using Dynamically Learned Robot Controllers” (IJCAI 2018) learns which actions to take from real-robot data, considering both the information gained and the cost of the actions. “Chat with the Environment” (IROS 2023) uses an LLM to choose actions for collecting information and to reason over vision, sound, touch and proprioception. The statement that existing systems only combine sensory inputs should be revised, and these works should be discussed.
Questions for authors
- Section 3 says the test scenes include 56 additional unseen objects, while Appendix D.6 says the test objects do not overlap with the training objects. Could you clarify this?
- In Table 9, removing force improves Inside on single-chain tasks (59.0 → 61.5) but lowers it on multi-chain tasks (75.1 → 64.3). Could you explain this?
- Could you report the inference time and token cost of ROMA-7B compared with the API baselines?
8 comments
[Annotation by baseline models] The object labels are generated by Gemini 3.5 Flash and the intent-driven questions by GPT-5.4, and both models are also used as baselines. I wonder whether this affects the comparison. The paper should also explain how the labels were checked by hand.
Thank you for raising this concern. In our work, Gemini 3.5 Flash pre-labeled a subset of the attributes. One annotator verified all pre-labels against the recorded interactions using a shared rubric and labeled inside contents from scratch.
Table R4: Correction rates of the Gemini pre-labels in ROMI-2K (%).
| Annotation Field | Handheld Subset | Tabletop Training Set | ROMA Bench |
|---|---|---|---|
| Object name | 72.5 | 13.9 | 10.5 |
| Roughness | 39.0 | 19.2 | 21.8 |
| Texture | 22.3 | 2.4 | 3.1 |
| Hardness (grasp location) | 36.9 | 21.4 | 16.6 |
| Material (grasp location) | 18.1 | 8.5 | 17.5 |
| Material (collision location) | 18.3 | 8.9 | 17.5 |
| Grasp location | 7.8 | 1.1 | 0.0 |
| Collision location | 3.7 | 0.2 | 0.0 |
| Bounding box | 0.0 | 0.8 | 0.0 |
As shown in Table R4, manual verification corrected many pre-labels for contact-dependent attributes such as roughness, hardness, and material. Bounding boxes rarely needed correction. GPT-5.4 generated Intent-Driven QA pairs from the verified annotations and the initial scene image, without access to interaction feedback. Human review checked formatting and obvious issues, and at most 10 pairs required edits. Their involvement in dataset construction may give these baselines some advantage, yet ROMA-7B still outperforms both on ROMA Bench. We have detailed the verification protocol in Appendices C.3 and D.5, the correction rates in Table 6 in the paper, and the QA generation process in Appendix D.6 of the revised version. Thank you again for raising this concern!
[Missing related work on exploratory action selection] To my knowledge, choosing exploratory actions has been studied before: "Multi-modal Predicate Identification using Dynamically Learned Robot Controllers" (IJCAI 2018) learns which actions to take from real-robot data, considering both the information gained and the cost of the actions. "Chat with the Environment" (IROS 2023) uses an LLM to choose actions for collecting information and to reason over vision, sound, touch and proprioception. The statement that existing systems only combine sensory inputs should be revised, and these works should be discussed.
Thank you for pointing out these important works. Both works are meaningful exploratory efforts toward advancing active perception. Amiri et al. [1] learn MOMDP controllers from real-robot exploration data. They trade off information gain against action cost to identify predicates of a single object, using per-predicate classifiers. Matcha [2] uses an LLM to select information-gathering actions such as knock, touch, and weigh in CoppeliaSim simulation. In Matcha, each modality is first converted into short language descriptions by separate modules.
Both are important early works on selecting exploratory actions, and ROMA differs from them in three aspects. First, ROMA operates in the real world, and its multi-sensory LLM directly takes raw visual, audio and tactile feedback, as well as processed force signals as input. Second, it handles open-vocabulary, multi-object tasks. These tasks require choosing the object to interact, reusing evidence across chains, and inferring targets from implicit intents. Third, it comes with large-scale data covering about 2,000 object-content combinations, together with a training framework.
We have discussed both works in Section 2 and revised the corresponding statements in the revised version. We also note the explicit cost modeling of Amiri et al. [1] as an inspiring future direction. Thank you again for pointing these out!
[1] Amiri, Saeid, et al. "Multi-modal predicate identification using dynamically learned robot controllers." Proceedings of the 27th international joint conference on artificial intelligence (IJCAI-18). 2018.
[2] Zhao, Xufeng, et al. "Chat with the environment: Interactive multimodal perception using large language models." 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2023.
Section 3 says the test scenes include 56 additional unseen objects, while Appendix D.6 says the test objects do not overlap with the training objects. Could you clarify this?
Thank you for pointing this out. By "additional unseen objects", we mean newly collected objects that are entirely held out from training. The 100 test scenes are built only from 56 newly collected objects, which form 325 instances with different contents. None of them appear in the tabletop training scenes or the handheld subset. We have clarified this in Section 3 in the revised version.
In Table 9, removing force improves Inside on single-chain tasks (59.0 → 61.5) but lowers it on multi-chain tasks (75.1 → 64.3). Could you explain this?
Thank you for the careful observation. First, Single-Chain Inside contains only 78 tasks, so the change from 59.0 to 61.5 corresponds to just 2 tasks and is not a meaningful difference. Second, we would like to clarify the meaning of the Multi-Chain columns in Table 12 in the paper. Each multi-chain task involves two target attributes. Each attribute column reports the accuracy over all multi-chain tasks that contain this attribute, so every multi-chain task is counted in two columns. The Multi-Chain Inside column (446 tasks) therefore covers questions that combine Inside with another attribute, such as "Is the heaviest object also empty?". The final answer depends on both attributes. Many of these tasks pair Inside with Weight, where force is the key evidence. When force is removed, the weight-related chain fails, and the whole task fails even if the inside evidence is acquired correctly. Consistently, the accuracy of multi-chain Weight itself drops sharply. Third, many inside contents are light (e.g., candies and beads). Their force difference from the empty container is close to the sensor noise, and may occasionally mislead the model in single-chain tasks, where audio is the primary cue. We also acknowledge that the integration of force in ROMA has only been preliminarily explored. Deeper integration of force, such as modeling full force/torque trajectories during interactions, is a promising direction that deserves further investigation. We discuss the force ablation in Appendix G of the revised version (Table 12 in the paper). Thank you again for your careful observation!
We would like to express our sincere appreciation for your constructive suggestions. We sincerely appreciate your recognition of our contribution to multi-sensory active perception, the value of ROMI-2K and especially its interaction-necessity and content-modality labels, and the reproducible details of our real-robot system.
Thank you for your valuable suggestions. First, we report the number of tasks of each type in ROMA Bench and the real-world evaluation.
Table R1: Task statistics of ROMA Bench and the real-world evaluation.
As shown in Table R1, all three task types are well covered in both evaluations, and Multi-Chain tasks account for about half of ROMA Bench.
Second, we evaluated ROMA-7B in three runs on the same 132 real-robot tasks. Each task takes about 2–3 minutes, including interactions and scene resets, so evaluating ROMA-7B and the two frontier baselines once each takes about 16 hours. The baseline results are from the original evaluation.
Table R2: Real-world evaluation results. Results of ROMA-7B are reported as mean ± std over three runs, and results of the baselines are from Table 2 in the paper.
As shown in Table R2, ROMA-7B's mean accuracy across runs still exceeds both frontier baselines on every task type, with the largest lead on Multi-Chain tasks. Intent-Driven accuracy varies the most between runs, while the variation in overall accuracy is small relative to the lead over the original baseline results.
Third, we further evaluated generalization under two new settings. We reran all 132 original real-robot tasks once with a green tablecloth, which changes the visual background and muffles the sounds produced by
<collide>. For the unseen counting task (offline), we generate five questions for each of the 95 ROMA Bench scenes with at least two objects. The 475 questions are evenly distributed across the target attributes (e.g., "How many objects are harder than [object_name]?"). This question type never appears in training.Table R3: Generalization to new settings.
As shown in Table R3, ROMA-7B leads both frontier baselines on the unseen counting task. Its accuracy drops slightly with the green tablecloth compared with the original setting, but it retains a clear lead despite the changes to visual and acoustic feedback. We report generalization in Section 5.4 (Table 4 in the paper) and task counts and repeated real-robot results in Appendix H (Table 13 in the paper). Thank you again for your valuable suggestions!