acceptodds
Under review as a conference paper at ICLR 2027

ROMA: LLM System for Real-World Object-Centric Multi-Sensory Active Perception

12 views

Abstract

Humans inherently understand the physical world through an active process. When sensory evidence is insufficient to infer physical properties, we naturally interact with the environment by deciding what information is missing, how to acquire it, and when sufficient evidence has been obtained. In stark contrast, existing multi-sensory robot systems mainly integrate sensory inputs rather than actively acquiring missing evidence through interactions. In this work, we introduce ROMA, an LLM-based system for Real-World Object-Centric Multi-Sensory Active Perception. ROMA integrates vision, audio, tactile, and force sensing into a reasoning-interaction-feedback loop. The model identifies missing evidence and determines the target objects, interactions, and modalities, while a physical interface executes the selected interactions and collects the multi-sensory feedback. To support this capability, we construct ROMI-2K, a large-scale real-world multi-sensory object interaction dataset covering nearly 2,000 objects and 6 atomic interactions with synchronized sensory feedback. Building on these data, we develop a two-stage training framework that aligns sensory modalities and equips the LLM to assess evidence sufficiency, select informative interactions, and reason over the multi-sensory feedback. We further characterize active perception as perception chains, where acquired evidence guides subsequent interactions and reasoning, and establish ROMA Bench to evaluate single-attribute, long-horizon multi-attribute, and intent-driven active perception. Experiments show that ROMA can actively acquire missing evidence and solve complex, long-chain multi-sensory perception tasks that existing methods struggle to handle, laying a strong perceptual foundation for active multi-sensory embodied agents.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Active Perception, Multi-modal Embodied AI, Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_ibbxOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Authors propose ROMA, an LLM-based system that decides when vision is not enough, and then interacts with objects to collect audio, tactile and force feedback. The paper builds a dataset, ROMI-2K. It has a handheld part with 1,657 object-content combinations and a robot-arm part with 500 tabletop scenes (400 for training and 100 for testing). Each object is recorded under six actions (lift, press, collide, shake, rotate, squeeze) with four synchronised modalities. The labels include whether a property needs interaction and which modalities can show what is inside an object. ROMA-7B is built on Qwen2.5-Omni with special action and modality tokens, and is trained by alignment and then SFT. The robot side uses AnyGrasp with point cloud completion and grasp refinement. On ROMA Bench (2,100 tasks) and 132 real-robot tasks, ROMA-7B is better than GPT-5.4, Gemini 3.5 Flash and two Qwen-Omni models, and uses fewer interactions than trying all actions on all objects.

Critical strengths

This paper makes a useful contribution to multi-sensory active perception. ROMI-2K covers many objects and many types of interaction with four synchronised modalities, which existing datasets do not provide together. The labels of whether a property needs interaction, and which modalities can show what is inside an object, are especially useful. They are the labels needed to learn when an interaction is worth doing. I expect the dataset and ROMA Bench to be used by other works. In addition, the whole system runs on a real robot, and the paper gives enough details to reproduce it, such as point cloud completion for transparent objects, grasp depth compensation and the two-stage gripping force.

Critical weaknesses

I have the following concerns that the authors should address in the final version:

  1. The evaluation is small. The real-robot test has 132 tasks from 8 scenes with a single run, and the 325 object instances in ROMA Bench come from only 56 test objects. Repeated runs, error bars, and the number of tasks of each type would help to show whether the improvements over the baselines are reliable.
  2. The object labels are generated by Gemini 3.5 Flash and the intent-driven questions by GPT-5.4, and both models are also used as baselines. I wonder whether this affects the comparison. The paper should also explain how the labels were checked by hand.
  3. To my knowledge, choosing exploratory actions has been studied before: “Multi-modal Predicate Identification using Dynamically Learned Robot Controllers” (IJCAI 2018) learns which actions to take from real-robot data, considering both the information gained and the cost of the actions. “Chat with the Environment” (IROS 2023) uses an LLM to choose actions for collecting information and to reason over vision, sound, touch and proprioception. The statement that existing systems only combine sensory inputs should be revised, and these works should be discussed.
Questions for authors
  1. Section 3 says the test scenes include 56 additional unseen objects, while Appendix D.6 says the test objects do not overlap with the training objects. Could you clarify this?
  2. In Table 9, removing force improves Inside on single-chain tasks (59.0 → 61.5) but lowers it on multi-chain tasks (75.1 → 64.3). Could you explain this?
  3. Could you report the inference time and token cost of ROMA-7B compared with the API baselines?
8 comments

We would like to express our sincere appreciation for your constructive suggestions. We sincerely appreciate your recognition of our contribution to multi-sensory active perception, the value of ROMI-2K and especially its interaction-necessity and content-modality labels, and the reproducible details of our real-robot system.

[Small evaluation scale] The evaluation is small. The real-robot test has 132 tasks from 8 scenes with a single run, and the 325 object instances in ROMA Bench come from only 56 test objects. Repeated runs, error bars, and the number of tasks of each type would help to show whether the improvements over the baselines are reliable.

Thank you for your valuable suggestions. First, we report the number of tasks of each type in ROMA Bench and the real-world evaluation.

Table R1: Task statistics of ROMA Bench and the real-world evaluation.

SplitSingle-ChainMulti-ChainIntent-DrivenTotal
ROMA Bench7411,0363232,100
Real World545127132

As shown in Table R1, all three task types are well covered in both evaluations, and Multi-Chain tasks account for about half of ROMA Bench.

Second, we evaluated ROMA-7B in three runs on the same 132 real-robot tasks. Each task takes about 2–3 minutes, including interactions and scene resets, so evaluating ROMA-7B and the two frontier baselines once each takes about 16 hours. The baseline results are from the original evaluation.

Table R2: Real-world evaluation results. Results of ROMA-7B are reported as mean ± std over three runs, and results of the baselines are from Table 2 in the paper.

ModelSingle-ChainMulti-ChainIntent-DrivenTotal
GPT-5.442.629.455.640.2
Gemini 3.5 Flash50.029.448.141.7
ROMA-7B56.8 ± 4.6660.1 ± 4.0866.7 ± 6.4260.1 ± 1.58

As shown in Table R2, ROMA-7B's mean accuracy across runs still exceeds both frontier baselines on every task type, with the largest lead on Multi-Chain tasks. Intent-Driven accuracy varies the most between runs, while the variation in overall accuracy is small relative to the lead over the original baseline results.

Third, we further evaluated generalization under two new settings. We reran all 132 original real-robot tasks once with a green tablecloth, which changes the visual background and muffles the sounds produced by <collide>. For the unseen counting task (offline), we generate five questions for each of the 95 ROMA Bench scenes with at least two objects. The 475 questions are evenly distributed across the target attributes (e.g., "How many objects are harder than [object_name]?"). This question type never appears in training.

Table R3: Generalization to new settings.

ModelNew Tabletop Setting (Real World)Unseen Task Type (Counting, Offline)
GPT-5.439.443.8
Gemini 3.5 Flash43.245.7
ROMA-7B57.665.3

As shown in Table R3, ROMA-7B leads both frontier baselines on the unseen counting task. Its accuracy drops slightly with the green tablecloth compared with the original setting, but it retains a clear lead despite the changes to visual and acoustic feedback. We report generalization in Section 5.4 (Table 4 in the paper) and task counts and repeated real-robot results in Appendix H (Table 13 in the paper). Thank you again for your valuable suggestions!

[Annotation by baseline models] The object labels are generated by Gemini 3.5 Flash and the intent-driven questions by GPT-5.4, and both models are also used as baselines. I wonder whether this affects the comparison. The paper should also explain how the labels were checked by hand.

Thank you for raising this concern. In our work, Gemini 3.5 Flash pre-labeled a subset of the attributes. One annotator verified all pre-labels against the recorded interactions using a shared rubric and labeled inside contents from scratch.

Table R4: Correction rates of the Gemini pre-labels in ROMI-2K (%).

Annotation FieldHandheld SubsetTabletop Training SetROMA Bench
Object name72.513.910.5
Roughness39.019.221.8
Texture22.32.43.1
Hardness (grasp location)36.921.416.6
Material (grasp location)18.18.517.5
Material (collision location)18.38.917.5
Grasp location7.81.10.0
Collision location3.70.20.0
Bounding box0.00.80.0

As shown in Table R4, manual verification corrected many pre-labels for contact-dependent attributes such as roughness, hardness, and material. Bounding boxes rarely needed correction. GPT-5.4 generated Intent-Driven QA pairs from the verified annotations and the initial scene image, without access to interaction feedback. Human review checked formatting and obvious issues, and at most 10 pairs required edits. Their involvement in dataset construction may give these baselines some advantage, yet ROMA-7B still outperforms both on ROMA Bench. We have detailed the verification protocol in Appendices C.3 and D.5, the correction rates in Table 6 in the paper, and the QA generation process in Appendix D.6 of the revised version. Thank you again for raising this concern!

[Missing related work on exploratory action selection] To my knowledge, choosing exploratory actions has been studied before: "Multi-modal Predicate Identification using Dynamically Learned Robot Controllers" (IJCAI 2018) learns which actions to take from real-robot data, considering both the information gained and the cost of the actions. "Chat with the Environment" (IROS 2023) uses an LLM to choose actions for collecting information and to reason over vision, sound, touch and proprioception. The statement that existing systems only combine sensory inputs should be revised, and these works should be discussed.

Thank you for pointing out these important works. Both works are meaningful exploratory efforts toward advancing active perception. Amiri et al. [1] learn MOMDP controllers from real-robot exploration data. They trade off information gain against action cost to identify predicates of a single object, using per-predicate classifiers. Matcha [2] uses an LLM to select information-gathering actions such as knock, touch, and weigh in CoppeliaSim simulation. In Matcha, each modality is first converted into short language descriptions by separate modules.

Both are important early works on selecting exploratory actions, and ROMA differs from them in three aspects. First, ROMA operates in the real world, and its multi-sensory LLM directly takes raw visual, audio and tactile feedback, as well as processed force signals as input. Second, it handles open-vocabulary, multi-object tasks. These tasks require choosing the object to interact, reusing evidence across chains, and inferring targets from implicit intents. Third, it comes with large-scale data covering about 2,000 object-content combinations, together with a training framework.

We have discussed both works in Section 2 and revised the corresponding statements in the revised version. We also note the explicit cost modeling of Amiri et al. [1] as an inspiring future direction. Thank you again for pointing these out!

[1] Amiri, Saeid, et al. "Multi-modal predicate identification using dynamically learned robot controllers." Proceedings of the 27th international joint conference on artificial intelligence (IJCAI-18). 2018.

[2] Zhao, Xufeng, et al. "Chat with the environment: Interactive multimodal perception using large language models." 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2023.

Section 3 says the test scenes include 56 additional unseen objects, while Appendix D.6 says the test objects do not overlap with the training objects. Could you clarify this?

Thank you for pointing this out. By "additional unseen objects", we mean newly collected objects that are entirely held out from training. The 100 test scenes are built only from 56 newly collected objects, which form 325 instances with different contents. None of them appear in the tabletop training scenes or the handheld subset. We have clarified this in Section 3 in the revised version.

In Table 9, removing force improves Inside on single-chain tasks (59.0 → 61.5) but lowers it on multi-chain tasks (75.1 → 64.3). Could you explain this?

Thank you for the careful observation. First, Single-Chain Inside contains only 78 tasks, so the change from 59.0 to 61.5 corresponds to just 2 tasks and is not a meaningful difference. Second, we would like to clarify the meaning of the Multi-Chain columns in Table 12 in the paper. Each multi-chain task involves two target attributes. Each attribute column reports the accuracy over all multi-chain tasks that contain this attribute, so every multi-chain task is counted in two columns. The Multi-Chain Inside column (446 tasks) therefore covers questions that combine Inside with another attribute, such as "Is the heaviest object also empty?". The final answer depends on both attributes. Many of these tasks pair Inside with Weight, where force is the key evidence. When force is removed, the weight-related chain fails, and the whole task fails even if the inside evidence is acquired correctly. Consistently, the accuracy of multi-chain Weight itself drops sharply. Third, many inside contents are light (e.g., candies and beads). Their force difference from the empty container is close to the sensor noise, and may occasionally mislead the model in single-chain tasks, where audio is the primary cue. We also acknowledge that the integration of force in ROMA has only been preliminarily explored. Deeper integration of force, such as modeling full force/torque trajectories during interactions, is a promising direction that deserves further investigation. We discuss the force ablation in Appendix G of the revised version (Table 12 in the paper). Thank you again for your careful observation!

Reviewer_iM1sOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposed a new active perception problem, how to leverage multi-sensory perception for an object to reason its properties via active object interactions. Towards this goal, the paper first presented a new ROMI-2K dataset which involves about 2000 everyday objects with 6 atomic interactions (lift, rotate, press, shake, collide, squeeze) and records synchronized visual, audio, tactile, and force feedback and annotated object identities, physical and interaction-specific attributes. Then, the paper proposes a ROMA system that is able to actively select the objects and interactions so as to reason the object properties of interests. The benchmark on the ROMI-2K dataset demonstrated that the ROMA system using a fine-tuned ROMA-7B LLM model achieves better performance than those general LLM models.

Critical strengths

First of all, the paper indeed proposed a new set up for active object-centric perception. The active reasoning for object properties using multi-sensory perception was not investigated thoroughly before.

The proposed ROMI-2K dataset was constructed in a very thorough way. The efforts of collecting synchronized visual, audio, tactile, force sensing using robot arms, etc. are quite valuable to the research community.

The proposed ROMA system is reasonable and demonstrates competitive performance over the general LLM models.

Overall the novelty of this active perception problem and the efforts to define the problem with a new dataset and a specific baseline are sufficient technical contributions .

Critical weaknesses

The active perception SFT (stage 2) in the proposed ROMA system needs more detailed explanation, e.g., how to generate the SFT samples, why the Beta-distribution-based sampling is needed. How the shift from training distribution to real-world scenes is achieved? Please justify this claim.

The interactions of objects undergoing some actions, e.g., shake, press, squeeze, are highly determined by how much force in these actions, as well as the feedback from the multi-sensors. How to reason how much force are needed to reveal the object properties in these actions?

Questions for authors

Please explain more about the active perception SFT and discuss the impact of different levels of force in the actions.

Regarding the example in Fig.1, how confident the visual evidence can tell whether it is a transparent bottle with candy inside or a picture on the bottle with candy on the picture, without shaking the bottle?

2 comments

We would like to express our sincere appreciation for your constructive suggestions. We sincerely appreciate your recognition of the novel setup for object-centric active perception, the thorough construction and community value of the ROMI-2K dataset, and the reasonable design and competitive performance of the ROMA system.

[Details of active perception SFT] The active perception SFT (stage 2) in the proposed ROMA system needs more detailed explanation, e.g., how to generate the SFT samples, why the Beta-distribution-based sampling is needed. How the shift from training distribution to real-world scenes is achieved? Please justify this claim.

Thank you for your valuable comments. First, the SFT samples come from handheld interaction-conditioned QA, pseudo-scenes composed of 1–3 handheld objects, and real tabletop scenes. For single-chain and multi-chain tasks, the perception chains are generated from human-verified annotations by a sophisticated program with controlled randomness. Each chain selects interactions and modalities according to the target attribute and the current evidence. For example, for inside contents, the chain skips interaction if the contents are visible (the "Require Interaction" label). Otherwise, it shakes or rotates the object with audio, and further squeezes it with vision, touch, and force if nothing is heard. For hardness, it squeezes the object with touch and adds a press if the evidence is ambiguous. The reasoning text is composed from the annotation fields, and evidence shared across chains is reused. For intent-driven tasks, GPT-5.4 generates the QA pairs and their perception chains from the human-verified annotations and the initial scene image, and the generated examples are checked for formatting and obvious issues.

Second, the "progressive shift" in Section 4.2 refers to training sample order. Handheld and pseudo-scene data teach what feedback different actions produce across many objects, while real scenes add force, realistic layouts, and the robot viewpoint. We draw positions from Beta(1, α) for handheld and pseudo-scene samples and from Beta(α, 1) for real-scene samples (α = 2.5), then sort the samples by position. This places more handheld and pseudo-scene examples early and more real-scene examples later, with a gradual transition. We also use HOTS examples early in training to improve tabletop object detection (Appendix F). We evaluate this dynamic sampling strategy on ROMA Bench.

Table R1: Ablation of the data sampling strategy on ROMA Bench (mean ± std over three SFT seeds).

Sampling StrategySingle-ChainMulti-ChainIntent-DrivenTotal
Uniform Mixing (− Sampling)70.1 ± 0.6172.1 ± 1.2270.1 ± 1.0971.1 ± 0.38
Beta Sampling (Ours)71.3 ± 0.3173.4 ± 1.9073.6 ± 0.6472.7 ± 0.74

As shown in Table R1, Beta sampling improves mean accuracy on all three task types across seeds. The largest gain is on Intent-Driven tasks, consistent with giving more weight late in training to real tabletop scenes, the source of these tasks. We describe sample construction in Appendices C.4 and D.6, the training schedule in Section 4.2 and Appendix F, and the sampling ablation in Appendix G (Table 11 in the paper). Thank you again for your valuable comments!

[Force level of interactions] The interactions of objects undergoing some actions, e.g., shake, press, squeeze, are highly determined by how much force in these actions, as well as the feedback from the multi-sensors. How to reason how much force are needed to reveal the object properties in these actions?

Thank you for this insightful question. In the current system, the interaction primitives use fixed and safe parameters, chosen to be informative while avoiding damage to everyday objects and the robotic arm. Squeeze closes the gripper by 8 mm, press moves the object down by 15 mm at a controlled speed, shake runs 7 reciprocating cycles, and collide taps the object on the table 4 times. The grip force itself is adaptive: a two-stage grasp measures the object's resistance and sets Ffinal=2⋅FmaxF_{final} = 2·F_{max} (Eq. 15 in the paper), so the applied force scales with object size and stiffness. We believe that reasoning about how much force to apply is indeed a promising future direction for active perception. Our action-token interface naturally supports this extension through parameterized primitives (e.g., <squeeze:strong>). These would allow ROMA to increase the force adaptively when the evidence remains ambiguous, while avoiding damage to fragile objects. Appendices D.3 and D.4 describe the adaptive grasp and interaction parameters, and Appendix B discusses extensions to the interaction skills. Thank you again for your insightful question!

Regarding the example in Fig.1, how confident the visual evidence can tell whether it is a transparent bottle with candy inside or a picture on the bottle with candy on the picture, without shaking the bottle?

Thank you for this interesting question. In Fig. 1 in the paper, the model infers that the transparent bottle contains candy from its appearance. ROMI-2K does annotate whether interaction is required to determine the contents of an object. ROMA-7B is thus trained to interact whenever appearance alone is unreliable. Nevertheless, a single image can be misleading. This is precisely why a shake is worth taking when in doubt. More broadly, mistaking a printed picture for real contents is an interesting form of multi-modal hallucination: visual appearance suggests a physical state that does not exist. We believe that resolving such hallucinations through active physical interaction is a promising direction for embodied intelligence. Multi-sensory active perception provides a natural way to verify visual hypotheses with non-visual evidence.

To test whether models seek additional evidence in this setting, we place two bottles side by side on the tabletop. One contains real candy, and the other is empty with a candy sticker on it. We ask each model “Which bottle is empty?” and run ten trials per model. For each bottle, we report the fraction of trials in which the model interacted with it before answering.

Table R2: Interaction rates for each bottle in the real-candy and candy-sticker experiment.

ModelBottle with Real CandyEmpty Bottle with Candy Sticker
GPT-5.41/101/10
Gemini 3.5 Flash4/104/10
ROMA-7B5/104/10

As shown in Table R2, ROMA-7B and Gemini 3.5 Flash interact with both bottles more often than GPT-5.4 and show similar interaction rates. One possible explanation is that Gemini annotated whether interaction was required, and those labels were used to train ROMA-7B. Distinguishing a small candy sticker from real candy is visually challenging even for frontier models. All three models sometimes interact with the bottle containing real candy as well, suggesting that visible contents do not always give them enough confidence to answer from vision alone. These observations highlight room for improving MLLMs' visual perception in tasks that hinge on subtle visual details. We report this experiment in Appendix H (Fig. 25 and Table 17 in the paper). Thank you again for your interesting question!

Reviewer_tnaWOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents ROMA, a system in which a multimodal LLM (ROMA-7B, fine-tuned from Qwen2.5-Omni) runs a reasoning–interaction feedback loop on a real robot for Q&A. At each step it chooses: a target object, given as a bounding box; one of six scripted interaction primitives (lift, press, collide, shake, rotate, squeeze); and which sensory modalities to read (wrist RGB, audio, GelSight tactile, and mean gravity force as text). It stops when it judges the evidence sufficient. Evaluation is based on whether the final answer is correct, with tasks also counted as failures when a grasp cannot be localized to any object (IoU < 70% for the predicted bounding box).

The authors make three contributions:

  1. ROMI-2K, a real-world dataset with two parts: A handheld UMI-style subset with 1,657 object–content combinations, each with 6 interactions, and a robot tabletop subset with 400 training and 100 test scenes. Annotations were pre-labelled with Gemini and then manually corrected.
  2. A two-stage training recipe, stage 1 aligns audio and tactile with vision and text, and stage 2 is supervised fine tuning on “perception chains”. The grasping pipeline is AnyGrasp plus heuristics.
  3. ROMA Bench, consisting of 2,100 offline replay tasks, plus a real-robot evaluation of 132 tasks from 8 scenes.
Critical strengths
  1. Relevant problem with a real closed loop. Moving from passive multi-sensory fusion to deciding what evidence to collect is interesting and well motivated.
  2. Potentially useful resource and considerable engineering work. A paired dataset of objects, interactions, images, audio, tactile, and force, with content variation inside containers, could be useful. The planned release of data, code and checkpoints add real value.
  3. Behavioral analysis beyond accuracy. Table 3 (grasps and interactions per scene) and Fig. 23 (action distributions) show qualitatively different exploration strategies across models.
Critical weaknesses
  1. The ML contribution is thin, and the “active” claim is mostly asserted rather than tested. Methodologically, ROMA-7B is standard supervised fine tuning of an omni-model with added special tokens, modality alignment by instruction tuning, and a data-ordering schedule for training. The perception chains used as supervision are built from annotations: an attribute maps to the interactions and modalities known to reveal it. The model therefore imitates a templated acquisition policy. Nothing in the method models uncertainty, information gain, or the cost of acting. “Recognizing when sufficient evidence has been obtained” is learned only as the point where the templated chain ends, which limits the generalization capabilities of the method.

  2. The comparison with MLLM baselines is biased.

  • ROMA is trained on the benchmark’s own QA templates, token format, robot, camera, and annotation scheme. Baselines are zero-shot with a system prompt, making the comparisons unfair.
  • Different baselines receive different data modalities. All baselines receive GelSight images as ordinary RGB images. The modality asymmetry makes comparisons hard to interpret. Also, a table expliciting which modalities each baseline received would be welcomed.
  • Localization dominates the gaps: with oracle boxes, GPT-5.4 rises from 43.5% (Table 1) to 63.3% (Table 3), about 20 points. ROMA rises only from 72.9% to 76.0%. Much of ROMA’s advantage over GPT-5.4 and Gemini is therefore grounding and bounding-box-format compliance, not active perception.
  1. Ablations are within statistical error. All the results presented in the ablation displayed on Table 8, for testing different components of the method, are within statistical error, making it unclear how each component of the method actually contributes to the final results. This also reinforces the notion that the better performance of ROMA against the baselines is more related to data alignment than to the method itself.

  2. Scope and generality. The evaluation is limited to one robot, one gripper, six hand-scripted primitives, a single tabletop setup, and a fixed set of six attributes. More importantly, the acquired knowledge is used only to answer questions. Active perception is a compelling research direction, but its value ultimately lies in informing decisions: choosing a grasp, adapting force, or selecting an object for a task. The intent-driven tasks move in this direction, but they still end in a verbal answer rather than an action. The authors acknowledge this in Appendix B, yet without even a small downstream experiment, the practical utility of the system remains unclear.

Minor weaknesses:

  1. The paper says that existing datasets are either limited in object diversity or interaction diversity. Then, the authors claim to bridge this gap with ROMI-2K, which presents only 6 primitive interactions. I believe it fits the category lacking in “interaction diversity”.

  2. The offline benchmark is replay, which limits what “active” means. In ROMA Bench, feedback comes from pre-recorded interactions taken in a fixed order with a fixed grasp. The agent’s choices index into a static table. Actions do not change state, and a poor grasp has no consequence. This is fine as a cheap proxy, but it should be stated clearly as a limitation, along with how often the real-robot grasp differs from the recorded one.

  3. Annotation circularity and label quality.

  • Gemini 3.5 Flash pre-annotates attributes and GPT-5.4 generates the intent-driven Q&A. Both are then evaluated as baselines.
  • Missing quality numbers, please report the fraction of labels corrected by humans.
  • Comparative questions for hardness, roughness and weight with “sufficiently clear” margins are a sensible design, but the margin criterion is not specified.
  1. Force is reduced to a scalar given as text. Weight then becomes nearly trivial once the right action is taken. This supports the reading that the benchmark mostly tests whether the right primitive and modality is requested plus localization, rather than deep multi-sensory reasoning.
Questions for authors
  1. How does a rule-based attribute→(primitive, modality) policy, paired with ROMA-7B as the reasoner, compare to learned selection?
  2. What is the accuracy of a passive model that receives all six interactions’ feedback for the queried objects, using a per-object budget to fit the context?
  3. Which modalities did each baseline receive? What coordinate convention were they given, and what fraction of baseline failures are format errors rather than wrong boxes?
  4. What are the task completion rates per model? Can you report CIs, or results over several SFT seeds?
  5. How many unique physical objects are in ROMI-2K? What fraction of LLM-generated annotations were corrected, and what is inter-annotator agreement?
  6. Is there an ablation of the “Dynamic Scene Sampling” proposed in Section 4.2 against alternative strategies, such as uniform mixing?
  7. In real-world runs, how often do grasps fail or objects slip during shake or rotate, and how are those tasks scored?
  8. Have the authors considered framing the paper primarily as a datasets-and-benchmarks contribution? In my view, the strongest and most durable contributions are ROMI-2K, the real-robot interaction pipeline, and ROMA Bench, while ROMA-7B, whose training is standard supervised fine-tuning, mainly serves as a strong reference baseline. The current framing places the methodological contribution at the center and the data and benchmark in a supporting role, which I think undersells the paper’s real strengths and invites comparisons the method cannot fully support. A research question such as “how do omni-models reason about the physical world when allowed to interact with it?” would better match the evidence presented.
14 comments

We would like to express our sincere appreciation for your detailed and constructive suggestions. We sincerely appreciate your recognition of the relevant problem with a real closed loop, the useful paired multi-sensory dataset, the considerable engineering work, and our behavioral analysis beyond accuracy. Your thoughtful suggestion on framing has also helped us clarify our contributions in the revised version.

[Scope and generality] The evaluation is limited to one robot, one gripper, six hand-scripted primitives, a single tabletop setup, and a fixed set of six attributes. More importantly, the acquired knowledge is used only to answer questions. Active perception is a compelling research direction, but its value ultimately lies in informing decisions: choosing a grasp, adapting force, or selecting an object for a task. The intent-driven tasks move in this direction, but they still end in a verbal answer rather than an action. The authors acknowledge this in Appendix B, yet without even a small downstream experiment, the practical utility of the system remains unclear.

Thank you for your constructive suggestion. We conducted three new experiments, one offline and two on the real robot:

  1. Unseen task type (offline). We use 95 ROMA Bench scenes with at least two objects each and generate five counting questions per scene. The 475 questions are evenly distributed across the target attributes (e.g., "How many objects are harder than [object_name]?"). This question type never appears in training.
  2. New tabletop setting (real world). We reran all 132 real-world tasks once with a green tablecloth. The cloth changes the visual background and muffles the sounds produced when objects strike the table during <collide>.
  3. Downstream manipulation (real world). We randomly sampled 50 questions from the original 132 real-world tasks whose answers each identify a single target object. Without retraining, we extracted the target from the model's final answer and used the existing grasping interface to pick it up and place it in a nearby basket. This provides a simple example of using the model's answer to guide a physical action.

Table R1: Generalization and downstream manipulation experiments.

ModelOriginal (Real-World)New Tabletop (Real-World)Downstream Manipulation (Real-World)Unseen Task Type (Offline)
GPT-5.440.239.436.043.8
Gemini 3.5 Flash41.743.240.045.7
ROMA-7B61.457.656.065.3

As shown in Table R1, ROMA-7B still performs best on unseen counting tasks, which require aggregating evidence across objects in a question format absent from training. Its accuracy drops slightly with the green tablecloth, yet it retains a clear lead over both frontier models despite the changes to visual and acoustic feedback. This shows that its advantage generalizes to the changed tabletop setting. ROMA-7B also achieves the highest placement success, showing that its answers can guide object selection for downstream pick-and-place.

Besides, several aspects of ROMA potentially support transfer across embodiments. Audio and tactile feedback primarily reflect local object–contact events, which are less directly tied to the arm's kinematics. We also subtract the empty-gripper contribution from the force readings. ROMI-2K combines robot-arm data with data collected using a handheld UMI device equipped with a Pika gripper and a different fisheye camera. The handheld data improve performance on the robot-arm benchmark (Table 11 in the paper), suggesting that the model can learn from interactions collected on different hardware. However, testing on another embodiment requires redesigning and fabricating sensor flanges and fingertips, which we could not complete during a short period. Dual-arm robots and dexterous hands are particularly interesting directions because they may explore objects and acquire evidence through interaction patterns that differ from those of a single arm.

We have added these experiments to Section 5.4 of the revised version (Fig. 6 and Table 4 in the paper). Thank you again for your constructive suggestion!

[Thin ML contribution and the "active" claim] The ML contribution is thin, and the "active" claim is mostly asserted rather than tested ...... "Recognizing when sufficient evidence has been obtained" is learned only as the point where the templated chain ends, which limits the generalization capabilities of the method.

Thank you for your comments. Our methodological contribution is the ROMA system as a whole. ROMA-7B selects the object, interaction, and modality, and the physical interface executes that choice and returns the evidence. Active perception requires grasps that hold during shaking and rotation and maintain stable contact for tactile sensing. To meet these requirements, we use symmetry-based point cloud completion for transparent and reflective objects, geometry-aware grasp refinement, and adaptive grip force. We also transform force readings into the robot base frame so the model can interpret them.

ROMA-7B provides a new training paradigm for real-world multi-sensory active perception. Dynamic scene sampling progressively shifts training from handheld data and pseudo-scenes toward real tabletop scenes, so the model progresses from learning what individual interactions reveal to deciding which interactions a scene requires. Dual multi-sensory alignment connects audio and touch with both text and paired vision, since visual context helps resolve ambiguity in what the robot hears or feels.

Perception chains are also conditioned on observations. Visible contents can make an interaction unnecessary, while an ambiguous squeeze can prompt a press (Fig. 11 in the paper) and a silent shake can prompt squeezing and a force reading (Fig. 16 in the paper). The chains can also reuse evidence and skip distractor objects. Controlled randomization in Single-Chain and Multi-Chain examples and GPT-5.4-generated Intent-Driven examples provide diverse chain structures. Moreover, ROMA-7B retains its lead on held-out Intent-Driven tasks (Tables 1 and 2 in the paper) and on counting, a task type absent from training (Table R1).

We agree that explicitly modeling uncertainty, information gain, and action cost is a promising direction. Prior controllers such as Amiri et al. [1] use per-predicate classifiers for a closed set of properties. ROMA Bench provides a setting to study cost-aware reinforcement learning for open-vocabulary queries over raw multi-sensory feedback.

We have clarified these contributions in the Introduction and Sections 4.1 and 4.2 of the revised version. Thank you again for your comments!

[1] Amiri, Saeid, et al. "Multi-modal predicate identification using dynamically learned robot controllers." Proceedings of the 27th international joint conference on artificial intelligence (IJCAI-18). 2018.

[Baseline fairness] ROMA is trained on the benchmark's own QA templates, token format, robot, camera, and annotation scheme. Baselines are zero-shot with a system prompt, making the comparisons unfair.

Thank you for your comments. Frontier models cannot be fine-tuned on tactile and audio interaction data. This is part of the motivation for ROMA-7B. To make the comparison as fair as possible, we evaluate the baselines under three additional settings:

  1. absolute pixel coordinates for bounding boxes, the same convention as ROMA-7B;
  2. few-shot perception-chain exemplars, together with an explanation of GelSight images;
  3. a Qwen2.5-Omni [2] audio-to-text captioner that makes up for GPT-5.4's lack of audio input.

Table R2: Comparison with strengthened baselines on ROMA Bench.

ModelAcc.
GPT-5.4 (Relative Bbox)43.5
GPT-5.4 (Absolute Bbox)42.9
GPT-5.4 (Relative Bbox) + Few-shot + GelSight Explanation44.1
GPT-5.4 + Audio-to-Text Captioner (Qwen2.5-Omni)42.0
Gemini 3.5 Flash (Relative Bbox)53.0
Gemini 3.5 Flash (Absolute Bbox)28.9
Gemini 3.5 Flash (Relative Bbox) + Few-shot + GelSight Explanation38.3
ROMA-7B72.9

As shown in Table R2, both frontier models perform better with relative coordinates. Adding few-shot examples and a GelSight explanation gives GPT-5.4 a small gain and lowers Gemini's accuracy. GPT-5.4's accuracy also drops when it receives audio captions from Qwen2.5-Omni, whose audio encoder is initialized from Whisper-large-v3 [2, 3].

ROMA-7B retains its lead across the tested coordinate conventions, few-shot prompts, and audio captions. We have added these results to Appendix H of the revised version (Table 15 in the paper). Thank you again for your comments!

[2] Xu, Jin, et al. "Qwen2.5-Omni Technical Report." arXiv preprint arXiv:2503.20215 (2025).

[3] Radford, Alec, et al. "Robust speech recognition via large-scale weak supervision." International conference on machine learning. PMLR, 2023.

[Different modalities across baselines] Different baselines receive different data modalities. All baselines receive GelSight images as ordinary RGB images. The modality asymmetry makes comparisons hard to interpret. Also, a table expliciting which modalities each baseline received would be welcomed.

Thank you for your suggestion. This difference comes from the input limits of current models, not from our design choice. It is exactly why a model with native support for all four modalities, such as ROMA-7B, is needed. Adding a Qwen2.5-Omni audio captioner to make up for the missing audio input of GPT-5.4 does not help (Table R2). This is because the captioner's understanding of object interaction sounds is limited, and its low-quality captions may even mislead GPT-5.4. The modalities and box conventions of each model are listed below. The box convention of each model was selected through comparison.

Table R3: Modalities and box conventions received by each model.

ModelWrist RGBAudioTactileForce (Text)Box Convention
GPT-5.4✓✗ (unsupported)As RGB images✓Normalized (0–1)
Gemini 3.5 Flash✓✓As RGB images✓Normalized (0–1)
Qwen2.5-Omni✓✓As RGB images✓Absolute pixels
Qwen3-Omni✓✓As RGB images✓Normalized (0–1)
ROMA-7B✓✓Tactile encoder✓Absolute pixels

As shown in Table R3, all baselines receive tactile signals as ordinary RGB images, and GPT-5.4 cannot receive audio at all, whereas ROMA-7B natively takes all four modalities as input. We describe the baseline inputs in Appendix F and the coordinate conventions in Appendix H of the revised version. Thank you again for your suggestion!

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to All Reviewers

We would like to express our sincere gratitude to the reviewers for their thoughtful reviews and constructive suggestions. We are also grateful for their kind recognition of the novelty and significance of studying multi-sensory active perception (Reviewer tnaW, iM1s and ibbx), the value of our ROMI-2K dataset and ROMA Bench with various objects, interactions and paired modalities (Reviewer tnaW, iM1s and ibbx), the competitive performance of our ROMA system (Reviewer iM1s and ibbx) and its comprehensive engineering work (Reviewer tnaW and ibbx), the behavioral analysis beyond accuracy (Reviewer tnaW), and the clarity of our presentation (Reviewer tnaW, iM1s and ibbx).

We list our responses to several key concerns here:

[Contributions of the ROMA system] Our methodological contribution is the ROMA system as a whole, rather than the SFT of ROMA-7B alone. The physical interface turns the model's decisions into reliable real-world interactions through an interaction pipeline designed for active perception, and returns multi-sensory feedback that the model can interpret. ROMA-7B identifies missing evidence and selects the object, interaction, and modality to acquire it. It provides a training paradigm for multi-sensory active perception with heterogeneous interaction data (dynamic scene sampling) and richer modality-paired data (dual multi-sensory alignment), and is deployable on-robot. Together, they form the complete multi-sensory active perception system that operates in the real world and jointly leverages vision, audio, touch, and force.

[Generalization and downstream manipulation] ROMA-7B leads both frontier baselines on unseen counting tasks and retains a clear lead when we rerun all 132 real-robot tasks with a new green tablecloth that changes the visual background and muffles collision sounds. In a simple downstream application, we use each model's final answer to select the object to grasp and place in a basket, with no retraining. ROMA-7B achieves the highest placement success on 50 randomly sampled tasks with a single target object.

[Active perception and baseline fairness] ROMA decides whether to interact, adds interactions after ambiguous feedback, and reuses evidence across chains. Our ROMA-7B with interaction planning outperforms rule-based action selection and passive feedback overall, with the fewest interactions and the shortest context. The best frontier result across the tested coordinate conventions, few-shot prompts, and audio-caption settings is 53.0%, compared with 72.9% for ROMA-7B. Wrong boxes are more common than format errors for the frontier baselines, and ROMA-7B completes 99.4% of tasks. Under relaxed localization, ROMA retains its largest leads on attributes that require non-visual sensing and on Multi-Chain and Intent-Driven tasks.

[Statistical reliability] We report task counts by type and mean ± std for three SFT seeds on ROMA Bench and three runs on the same 132 real-robot tasks. ROMA-7B achieves 72.7 ± 0.74% on ROMA Bench and 60.1 ± 1.58% on the robot, compared with 53.0% and 41.7% for the strongest baseline in the original evaluations. The full model also has the highest mean accuracy on every task type in the three-seed ablations.

[Annotation quality] Gemini 3.5 Flash pre-labels a subset of attributes. A human annotator verifies every pre-label, corrects many contact-dependent labels, and labels inside contents from scratch. GPT-5.4 generates intent-driven QA pairs from the verified annotations and the initial scene image without access to interaction feedback. Human review checks formatting and obvious issues.

[Related work and dataset positioning] We discuss two important prior works on exploratory action selection [1, 2] and compare ROMI-2K with existing multi-sensory interaction datasets. ROMI-2K provides dense coverage of object–interaction pairs, recording all six exploratory interactions for each of nearly 2,000 object–content combinations. It also annotates which actions and modalities provide evidence for each property.

[1] Amiri, Saeid, et al. "Multi-modal predicate identification using dynamically learned robot controllers." Proceedings of the 27th international joint conference on artificial intelligence (IJCAI-18). 2018.

[2] Zhao, Xufeng, et al. "Chat with the environment: Interactive multimodal perception using large language models." 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2023.

Area_Chair_twXSOpenReview ↗

Reminder: Please Review Author Rebuttals and Participate in Discussions

Dear Reviewers,

Thank you for your time on the reviews. The authors have now submitted their rebuttals, and I encourage you to read their responses and participate in the discussion.

Please consider whether the rebuttals address your concerns and, where clarification is still needed, engage with the authors through the discussion forum. I also encourage you to exchange perspectives with your fellow reviewers and update your review or rating if appropriate.

Your participation will help ensure a thorough and fair evaluation of each paper. Thank you again for your time and effort.

Best regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.