acceptodds
Under review as a conference paper at ICLR 2027

OmniSmartHome: A Multimodal Reasoning Benchmark for Smart-Home Agents

Abstract

Smart-home assistants are expected to handle diverse, realistic requests that arise in daily life. In such interactions, users often rely on the surrounding multimodal context—pointing at objects or referring to what they see or hear, leaving their requests underspecified in language alone. Existing smart-home benchmarks, however, express user requests solely through language, leaving context-dependent real-world requests underexplored. To bridge this gap, we introduce OmniSmartHome, a multimodal smart-home benchmark where each spoken request is paired with the surrounding visual and spatial-audio context, providing complementary cues to disambiguate underspecified requests. OmniSmartHome comprises 1,360 synthetic and 272 real-world episodes. We evaluate 16 omnimodal large language models (Omni-LLMs) and reveal that, while they perform strongly when speech alone sufficiently conveys the user's intent, performance drops substantially when resolving it requires reasoning over multimodal contextual cues. As a simple agent baseline, we provide PROME (PROcedural Memory for multimodal Evidence gathering), which equips agents with specialized audio-visual perception tools and procedural memory for orchestrating their use. PROME generally improves performance across six Omni-LLMs. Demos and examples are available at https://omni-smart-home.github.io.

Primary area:
applications to robotics, autonomy, planning
Keywords:
smarthome, LLM, agent

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_gCvpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OmniSmartHome: pairing visual and spatial audio with voice requests from smart home agents, using G0-G4 to distinguish explicit/implicit references, covering 1360 synthesized+272 real episodes. There are three contributions: the first multimodal context disambiguation benchmark; Reviewed 16 Omni LLMs and found that when the voice is clear, it is strong, and when relying on gestures/sounds/directions/identities, it drops significantly; Propose PROME baseline using perception tools and program memory. In terms of verification, the manual level is very high, and PROME has generally improved on six models, and ablation, memory transfer, and real set testing have also been conducted.

Critical strengths

First, the problem is well motivated and genuinely new: existing smart-home benchmarks are language-only, while OmniSmartHome is the first to require resolving underspecified requests from gesture, sound, space, and speaker identity. This matters because it targets a real gap in evaluating Omni-LLMs, not just another leaderboard. Second, the benchmark and validation are substantial: synthetic plus real episodes, human verification, 16 models, and a simple PROME baseline that improves across backbones and transfers to real data. That combination makes the reported gap credible and gives the community a concrete starting point.

Critical weaknesses

One reason is that the benchmark volume is relatively small, with only 272 items in the real world, and the synthesized data is a static graph. The true generalization conclusion is not hard enough, and the statistical persuasiveness is limited. The second reason is that PROME is more like engineering assembly, with uneven improvement. G4 and some models have very little gain, and rely on external perception tools without real-time delay evaluation, which weakens the contribution of the "baseline can narrow the gap" method. Overall, it looks more like a dataset report, with insufficient innovative methods.

Questions for authors

How are the 272 real sets distributed on G0-G4 and Q1-Q4, and is it enough to support the conclusion of 'real generalization'? Has a significance test been conducted on the improvement of PROME, especially the small gains of G4 and some models? What is the consistency and error rate of manually verified annotations?

6 comments

Dear reviewer gCvp,

We sincerely appreciate your efforts and comments. All revisions are highlighted in blue in the updated PDF.


[W1, Q1] Real-world benchmark size and generalization.

OmniSmartHome includes 1,360 synthetic and 272 real-world episodes collected across five homes and five participants. As described in Line 224, eight episodes are included for each grounding–query/feasibility type. Real-world evaluations in simulation-centric embodied and audio-visual benchmarks are often limited in scale, involving a single office [1] or a single scanned lab [2], while some rely entirely on simulation [3]. Similarly, existing smart-home benchmarks, including HomeBench [4], SimuHome [5], SMH-Bench [6], and MIST [7], lack real-world evaluations. In this context, our 272 real-world episodes across five homes represent a meaningful addition to existing benchmarks.

We clarify that “Real-world generalization” in Section 4.2 (line 407) refers specifically to whether PROME, whose procedural memory is learned entirely from synthetic data, remains effective on real-world episodes, rather than claiming that our 272 episodes establish broad generalization across real-world smart-home environments. We acknowledge that the original heading could imply a broader claim and revise it to “Synthetic-to-real transfer.”

To complement this analysis, we further assess the sim-to-real predictivity of OmniSmartHome's synthetic evaluations using the Sim-vs-Real Correlation Coefficient (SRCC)[2]. Across 22 model configurations, we obtain high SRCC values for both grounding accuracy (0.949) and goal accuracy (0.944). These results demonstrate that OmniSmartHome's synthetic evaluations meaningfully reflect relative model performance in real-world settings, supporting its utility as a benchmark for multimodal smart-home agents.


[W2] Novelty of PROME.

We agree that PROME is a relatively simple, tool-augmented baseline rather than a fundamentally new methodological framework. Our primary contribution is the benchmark, the first multimodal smart-home benchmark in which spoken requests are accompanied by surrounding visual and spatial-audio context. PROME is intended to demonstrate that existing models can partially bridge the observed performance gaps through structured perception and tool use, rather than to claim methodological novelty. We acknowledge its uneven gains across models and clarify its role and scope in the revised paper.


[Q2] Statistical significance of PROME

We conduct exact McNemar tests on paired per-episode success. The table below reports accuracy improvements over each base model with the corresponding p-values. PROME yields statistically significant improvements for all six models on both metrics (p < 0.05).

ModelΔ Ground Acc. (pp)Δ Goal Acc. (pp)
Gemini-2.5-Flash+4.5 (p<0.001)+6.1 (p<0.001)
Gemini-2.5-Pro+4.0 (p<0.001)+6.1 (p<0.001)
Qwen3-Omni-Think+4.3 (p<0.001)+5.5 (p<0.001)
Nemotron3-Omni+2.5 (p=0.045)+4.6 (p<0.001)
Gemma4+23.4 (p<0.001)+34.1 (p<0.001)
Qwen2.5-Omni+5.2 (p<0.001)+4.8 (p<0.001)

For G4 Identity (n = 384), PROME also yields positive improvements across all six models, with statistically significant gains for four of them (p ≤ 0.001).


[Q3] Annotation Quality Control

We employed a two-stage quality-control process combining automatic filtering and human verification.

Automatic filtering assessed TTS quality using ASR-based transcription errors and applied task-specific criteria to reduce ambiguity and unintended cues, such as device proximity.

Human verification involved two annotators checking task-specific validity, including full-body and finger visibility, pointing direction, head-orientation clarity, and relative audio levels between TTS and device noise. Only samples approved by both annotators were retained.

Overall, 49.2% of candidate samples were excluded, and the final dataset was selected from the remaining samples to balance target devices and task types.


[References]
[1] Sim-to-Real Transfer for Vision-and-Language Navigation, CoRL, 2020
[2] Sim2Real Predictivity: Does Evaluation in Simulation Predict Real-World Performance?, IEEE Robotics and Automation Letters, 2020
[3] Grounded, or a Good Guesser? A Per-Question Balanced Dataset to Separate Blind from Grounded Models for Embodied Question Answering, ACL, 2025
[4] HomeBench: Evaluating LLMs in Smart Homes with Valid and Invalid Instructions Across Single and Multiple Devices, ACL, 2025
[5] SimuHome: A Temporal- and Environment-Aware Benchmark for Smart Home LLM Agents, ICLR, 2026
[6] SMH-Bench: Benchmarking LLM Agents for Environment-Grounded Reasoning and Action in Smart Homes, arXiv, 2026
[7] MIST: Multimodal Interactive Speech-based Tool-calling Conversational Assistants for Smart Homes, arXiv, 2026

Thanks the authors for their detailed response and supplementary analysis. The authors more accurately defined “real-world generalization” as “transfer from synthetic to real-world settings” and provided a correlation analysis between simulated and real-world performance, which alleviated my concerns regarding the scope of their conclusions. The newly added paired significance test also provides stronger statistical support for PROME’s overall improvement; the significant gains observed for G4 across all four models partially address my concerns regarding limited gains in that category. Therefore, I am raising my rating from 2 (weak rejection) to 3 (weak acceptance).

Dear Reviewer gCvp,

We are grateful that our response has addressed your concerns, and we sincerely appreciate your decision to raise your score.

Thank you again for your constructive feedback. We have carefully incorporated your suggestions into the manuscript and will continue to refine these revisions in the final version.

Sincerely,

The Authors

Reviewer_fK6yOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Existing smart-home assistants primarily rely on spoken commands to control various household devices. This paper addresses the challenge of resolving ambiguous spoken references to target devices by incorporating multimodal contextual cues, including gestures, sounds, spatial relationships, and user identity. To this end, the authors introduce OmniSmartHome, a benchmark comprising 1,360 synthetic and 272 real-world episodes, covering five grounding types and four query types. Evaluations on this benchmark reveal that existing multimodal LLMs struggle to identify target devices when their references depend on multimodal context. The authors further propose PROcedural Memory for multimodal Evidence gathering (PROME), a ReAct-based agent framework that integrates specialized audio-visual perception tools with procedural memory to explicitly guide the collection of fine-grained perceptual evidence. Experimental results demonstrate that PROME consistently improves the performance of existing multimodal LLMs on the benchmark.

Critical strengths

(1) Novel Problem. The paper addresses a practically relevant research problem: resolving ambiguous spoken requests in smart-home environments by incorporating multimodal contextual cues, including gestures, sounds, spatial relationships, and user identity. The proposed benchmark covers diverse tasks, including device state inquiries, device control, workflow scheduling, and implicit-intent requests, as well as both feasible and infeasible scenarios. This comprehensive design enables the evaluation of an agent's ability to identify target devices, execute appropriate control workflows, and recognize when user requests cannot be fulfilled due to environmental or operational constraints.

(2) Comprehensive Experiments. The paper evaluates 16 multimodal LLMs, demonstrating the challenging nature of the proposed OmniSmartHome benchmark. Moreover, PROME consistently improves performance across six evaluated models, demonstrating that more effective perceptual evidence gathering and utilization is a promising direction for improving multimodal agents and inspiring future research. The authors also conduct real-world transfer experiments, further demonstrating the potential of the proposed approach in practical scenarios.

Critical weaknesses

(1) Evaluation Reliability. In the experiments, all Q1 (state inquiry) requests and infeasible requests are evaluated using an LLM-as-a-Judge, but the authors do not report the agreement between LLM-based evaluations and human judgments. The authors are encouraged to report human–judge agreement on a representative subset to validate the reliability of the evaluation protocol. Furthermore, would the main model rankings and the performance gains achieved by PROME remain consistent under human evaluation?

Questions for authors

(a) The authors define Q3 (Workflow Scheduling) as scheduling device operations at a future time or in coordination with another event. However, Section 3.3 states that Goal Accuracy for feasible Q2–Q4 requests is evaluated by checking whether the final simulator state matches the ground-truth goal. Could the authors clarify whether Q3 evaluation verifies the correctness of scheduled workflows, including their timing and triggering conditions, rather than simply checking the final device state?

3 comments

Dear reviewer fK6y,

We sincerely appreciate your efforts and comments. All revisions are highlighted in blue in the updated PDF.


[W1]: LLM Evaluation reliability.

We adopt the LLM-as-a-Judge evaluation protocol of SimuHome[1], which reports Cohen's k of 0.913 between human annotators and 0.826 between human labels and the LLM judge. To validate its reliability on OmniSmartHome, we additionally evaluated 100 samples and obtained comparable agreement scores of 0.908 and 0.825, respectively. This close agreement supports the reliability of our evaluation. Furthermore, the model rankings and PROME improvements were preserved in human evaluation on 100 randomly sampled examples across three models and PROME, providing strong evidence that these trends would remain consistent under broader human evaluation.


[Q1]: Workflow scheduling evaluation.

We thank the reviewer for raising this important question. We follow the simulator based scheduling evaluation protocol of SimuHome [1]. For Q3, the simulator advances its virtual time to the target execution time, executes the scheduled workflows, and verifies whether the required device states are achieved at the specified time. For requests involving event dependencies, the target time is determined by the corresponding event completion time and the requested temporal constraints. Thus, the evaluation checks whether workflows produce the correct outcomes under the required timing conditions, rather than merely checking the device states immediately after the agent finishes.


[References]
[1] SimuHome: A Temporal- and Environment-Aware Benchmark for Smart Home LLM Agents, ICLR, 2026

Reviewer_fK6yOpenReview ↗

Reply to Authors Rebuttal

Thank you for the rebuttal and additional experimental evidence. The human-LLM judge agreement results and clarification of the workflow scheduling evaluation address my concerns.

Considering the novelty of the research problem, the proposed method, and the comprehensive experimental results, I maintain my original rating.

Best, Reviewer fK6y

Dear Reviewer fK6y,

Thank you very much for your thoughtful follow-up and continued positive assessment. We are pleased that our rebuttal has addressed your concerns. We sincerely appreciate your constructive feedback, which has helped us strengthen the paper.

Sincerely,

The Authors

Reviewer_QoqkOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces OmniSmartHome, a benchmark for smart-home agents that must interpret spoken requests using surrounding visual and acoustic context. The benchmark contains 1,360 synthetic and 272 real-world episodes, covering explicit speech, gestures, sounds, speaker-relative spatial references, and user identity. It separately evaluates target-device grounding and downstream goal completion. The authors benchmark 16 omnimodal models and introduce PROME, an agent baseline that combines specialized perception tools with procedural memory to guide evidence gathering.

The paper addresses a relevant problem: identifying the intended device when a request is underspecified in language alone. Its main contribution is the evaluation resource and associated diagnostic framework, supported by a practical agent baseline.

Critical strengths

A well-motivated evaluation target. Requests involving pointing, environmental sounds, speaker-relative directions, and household identity are recognizable examples of context-dependent interaction. The grounding taxonomy provides a useful structure for studying these capabilities.

Separation of grounding and task completion. Reporting both grounding accuracy and goal accuracy helps distinguish incorrect target identification from failures in subsequent device operation. This makes the benchmark more informative than an aggregate success-rate comparison.

Substantial experimental coverage. The paper evaluates multiple model families and includes synthetic and real-world observations. I appreciate the additional analyses: the text-only ablation, tools-only comparison, oracle-perception experiment, comparisons with AWM and MemP, and memory-transfer results.

A useful starting point for future methods. PROME provides a practical baseline and improves overall performance across several backbones. Its synthetic-to-real results provide encouraging evidence that the approach is useful beyond the synthetic observations used for memory construction.

Critical weaknesses

Weaknesses

W1. The dependence on multimodal evidence is not sufficiently characterized.

The central motivation is that the intended device cannot be resolved from language alone. However, Figure C.2 shows that some nominally multimodal categories remain highly solvable using the request transcript and available simulator information. For example, on G1 episodes with one same-type candidate, text-only grounding accuracy is 87.5%, compared with 82.6% using full multimodal input. On G2 episodes with multiple same-type candidates, text-only accuracy remains 81.2%.

The same figure also provides convincing evidence of multimodal dependence in other subsets, particularly G1 and G3 with multiple same-type candidates. The concern is therefore how these different cases contribute to the overall evaluation. Without subgroup counts and clearer analysis, it is difficult to determine how much the aggregate scores measure contextual grounding, as opposed to candidate availability, linguistic cues, or device-state information.

W2. Differences in spatial-audio access complicate the interpretation of failures and improvements.

Appendix C.1 states that only two evaluated models accept FOA audio, while the other models receive a mono downmix. For episodes requiring sound-source localization, performance may therefore reflect information lost before inference, alongside limitations in perception or reasoning.

The same issue affects the interpretation of PROME. If its localization tool accesses the original FOA recording while the backbone receives mono audio, some improvement may come from access to additional spatial information. This is a reasonable system design, but the paper should distinguish this benefit from improved orchestration of the same evidence.

The oracle-tool experiment is helpful, although an agent having access to an oracle tool does not guarantee that it calls the tool and receives the relevant evidence.

W3. The incremental contribution of learned procedural guidance remains unclear.

Table 3b shows improvements over tools alone. However, Appendix D.1 also introduces a substantial manually specified cost prior that encourages listing devices, inspecting candidate states, and declaring a target. This prior directly addresses several failure patterns identified in the paper.

The current comparisons do not establish how much additional benefit comes from the learned utility bonuses and policy update beyond the tools and this initial procedure. A comparison with the initial policy or a fixed checklist, using the same tools and comparable interaction budgets, would make the results easier to interpret. This is particularly relevant to Gemma4, which exhibits the largest improvement but is not included in the main tools-only ablation. Uncertainty estimates would also help assess the smaller gains on stronger models.

W4. The judge configuration and human-score aggregation require clarification.

The Q1 judge prompt in Appendix E.2 assumes that the user identifies the device only through sound or position and never provides its name, type, or room. This appears inconsistent with other grounding categories, especially G0. It is unclear whether this prompt was used for all Q1 episodes or represents a specialized configuration.

The human results also require a clearer account of sampling and aggregation. Appendix F.2 reports a balanced subset of 400 episodes, whereas model results are reported on the full benchmark. The human overall goal accuracy of 82.3% is difficult to reconcile with the per-grounding-type values under the benchmark’s usual category weights. Different subset weights could explain this, so the underlying counts and aggregation procedure should be provided.

Questions

Multimodal dependence: Could the authors provide the sample counts for the Figure C.2 subgroups and report performance on the subset where multiple plausible targets remain after considering the transcript and available device information? A small paired diagnostic, in which the request is fixed and a relevant perceptual cue changes the target, would provide additional evidence.

Information access: Which audio representation is available to each backbone and perception tool? For localization-dependent episodes, can the authors separate improvements from additional spatial information from improvements in how the agent uses that information?

PROME attribution: Could the authors compare full PROME with its initial policy or a fixed procedure on one representative backbone, using identical tools and comparable interaction budgets? This would clarify the contribution of learning beyond the predefined procedure.

Evaluation reliability: Was the Appendix E.2 prompt used for every Q1 category? Please clarify the judge model and settings, provide any available human validation, and explain the human-score aggregation. Model results on the same 400 human-evaluated episodes would make that comparison more interpretable.

Questions for authors

I find the problem important and believe that OmniSmartHome could become a useful evaluation resource. My current reservation concerns confidence in what the aggregate scores measure and how the reported improvements should be interpreted.

I would be willing to raise my score to marginally above the acceptance threshold and support acceptance if the rebuttal provides convincing evidence that the core evaluation meaningfully requires multimodal grounding, clarifies the spatial-audio information available to the compared systems, and resolves the judge and human-evaluation concerns. A targeted comparison against the initial procedural policy would further strengthen my confidence.

These points appear addressable through focused analyses, clarification of the existing evaluation, and appropriately scoped claims. I place the greatest weight on establishing the benchmark’s validity and reliability, given that this is the paper’s primary contribution.

AI assistance was used for manuscript analysis, checking related work, and drafting and revising the review. But I have checked all details.

4 comments

Dear reviewer Qoqk,

We sincerely appreciate your efforts and comments. All revisions are highlighted in blue in the updated PDF.


[W1, Q1]: Dependence on multimodal evidence.

We thank the reviewer for raising this important question. We agree that episodes vary in their dependence on multimodal evidence. Our benchmark includes cases where (1) multimodal evidence is essential for grounding and (2) it provides useful cues but the target device can also be inferred from the speech instruction and home state. The former corresponds to cases where multiple plausible target devices remain after considering the instruction, home device information, and current states, whereas the latter generally involves cases where only one plausible target remains. We retain the latter because multimodal evidence is not always essential in realistic situated interaction, and judging when it is essential is itself part of the task. This is in line with prior multimodal benchmarks, which also evaluate instances where multimodal evidence is not strictly essential [1, 2]. We clarify this distinction in the paper.

1. Candidate-based analysis of modality dependence. Our benchmark contains 90/114 feasible episodes for G1, 0/204 for G2, 131/73 for G3, and 139/133 for G4, with each pair indicating multiple/single plausible targets, respectively. (Note that, following the reviewer's suggestion, these counts refine those in Figure C.2, which considers only device types mentioned in the instruction, by additionally accounting for current device states.) G2 contains only single-target cases by design. If multiple devices produce the same sound, a request like "Turn off that sound" would naturally refer to all of them, rather than requiring the agent to disambiguate a single target among multiple candidates.

The table below reports the change in grounding accuracy from full multimodal input to text-only input across three models, grouped by the number of plausible targets for feasible episodes. Removing multimodal evidence leads to substantial accuracy drops, particularly when multiple candidates exist, consistent with the varying degrees of dependence on multimodal evidence.

ModelCandidatesG1G2G3G4
Gemini 3.1 Pro≥266.7 → 34.4 (−32.3)—87.8 → 52.7 (−35.1)58.3 → 32.4 (−25.9)
191.2 → 93.9 (+2.7)97.5 → 86.3 (−11.2)89.0 → 72.6 (−16.4)77.4 → 66.2 (−11.2)
Gemini 2.5 Pro≥258.9 → 36.7 (−22.2)—81.7 → 57.3 (−24.4)51.8 → 34.5 (−17.3)
197.4 → 95.6 (−1.8)87.7 → 79.4 (−8.3)98.6 → 89.0 (−9.6)82.7 → 56.4 (−26.3)
Qwen3.8 Omni Flash≥256.7 → 35.6 (−21.1)—77.9 → 53.4 (−24.5)43.2 → 20.1 (−23.1)
197.4 → 97.4 (0.0)92.6 → 92.2 (−0.4)100.0 → 80.8 (−19.2)72.9 → 51.1 (−21.8)

2. Paired counterfactual analysis of modality dependence. Following the reviewer's suggestion, we constructed 30 counterfactual pairs, each keeping the spoken instruction and home state unchanged while modifying only the relevant multimodal cue to indicate a different target device.

The table below reports grounding accuracy on original and counterfactual episodes, along with retention (the proportion of correct original predictions that remain correct after the target changes). Accuracy remains comparable after the target change, suggesting that models rely on multimodal cues rather than solely on the unchanged speech instruction and home state.

ModelOriginalCounterfactualRetention
Gemini 3.1 Pro70.073.371.4
Gemini 2.5 Pro63.360.063.2
Qwen3.8 Omni Flash63.370.073.7

[W2,Q2]: Disentangling spatial-audio access from PROME's contribution.

1. Spatial-audio access vs. PROME. To disentangle PROME's contribution from access to spatial audio information, we evaluate a setting where oracle spatial audio information (Direction of Arrival; DoA) is directly provided in the prompt of the vanilla agent.

The table below reports Grounding/Goal Acc. PROME consistently outperforms the oracle DoA prompting baseline across all backbones for large margin. These results suggest that PROME's improvements cannot be explained solely by access to spatial information, but also from selecting and integrating complementary evidence, such as device states and visual cues.

ModelVanillaOracle DoAPROME
Gemini-2.5-Flash76.5/57.976.8/59.680.1/63.3
Qwen3-Omni-Think66.4/47.967.5/49.670.9/54.1
Nemotron3-Omni64.2/46.063.5/44.266.1/48.8
Gemma 455.6/28.258.8/28.175.7/60.1

Results are reported on the synthetic dataset only to enable oracle information, and thus differ from Table 2.

2. Clarification of spatial audio access. As clarified in Line 1197, only Spatial-Omni and Qwen3.8-Omni-Flash natively support spatial audio; the remaining backbones receive mono audio but can access spatial information through PROME's DoA perception tool.

3. Statistical significance of PROME. We conduct exact McNemar tests on paired per-episode success. The table below reports accuracy improvements over each backbone model with the corresponding p-values. PROME yields statistically significant improvements for all six models(p<0.05).

ModelΔ Ground Acc.(pp)Δ Goal Acc.(pp)
Gemini-2.5-Flash+4.5 (p<0.001)+6.1 (p<0.001)
Gemini-2.5-Pro+4.0 (p<0.001)+6.1 (p<0.001)
Qwen3-Omni-Think+4.3 (p<0.001)+5.5 (p<0.001)
Nemotron3-Omni+2.5 (p=0.045)+4.6 (p<0.001)
Gemma4+23.4 (p<0.001)+34.1 (p<0.001)
Qwen2.5-Omni+5.2 (p<0.001)+4.8 (p<0.001)

[W3,Q3] Disentangling tool availability and manually specified procedures from PROME's contribution

To disentangle PROME's contribution from tool availability and procedural guidance, we conduct two additional comparisons. First, we extend the tools-only evaluations in Tables 3(b) and D.2 to Gemma4 and Nemotron3-Omni. Second, we explicitly prompt the backbone to follow a manually specified procedure—listing devices, inspecting candidate states, and identifying the target-without PROME's learned orchestration.

The table below reports Grounding/Goal Acc. PROME outperforms both the tools-only and manually specified procedure baselines across most backbones, with the exception of Gemma 4, where tools-only performance is comparable. We attribute this to Gemma 4's heavy reliance on ASR (87.1% of tools-only cases, vs. 64.1% for Gemini-2.5-Pro and 0.5% for Qwen3-Omni-Think). PROME also primarily recommends ASR for Gemma 4 (66.7% of first-step recommendations, vs. 10.3% for Gemini-2.5-Pro), suggesting limited additional procedural learning beyond the tools-only baseline.

ModelVanillaTools OnlyManual ProcedurePROME
Gemini-2.5-Flash73.7/56.275.2/58.974.5/57.478.2/62.3
Qwen3-Omni-Think65.9/48.465.7/47.766.1/48.570.2/53.9
Nemotron3-Omni62.7/44.561.3/43.562.4/45.865.2/49.1
Gemma450.6/25.275.4/58.974.0/56.574.0/59.3

[W4,Q4]: Judge configuration and human evaluation.

Judge prompt and reliability The Q1 judge prompt in Appendix E.2 is an example for G2 and G3, not all categories. Following [3], we use GPT-5 mini as the judge. To assess reliability, we evaluated 100 samples, obtaining Cohen's κ of 0.908 between human evaluators and 0.825 between human and LLM judgments.

Human evaluation subset composition The 400 human-evaluated episodes closely match the full benchmark in composition. We thank the reviewer for identifying a reporting error in Table 2: the correct overall human goal accuracy is 85.0%, not 82.3%.

CategoryHuman subset(400)Full benchmark(1632)
G096 (24.0%)384 (23.5%)
G172 (18.0%)288 (17.6%)
G272 (18.0%)288 (17.6%)
G364 (16.0%)288 (17.6%)
G496 (24.0%)384 (23.5%)
Synthetic332 (83.0%)1,360 (83.3%)
Real68 (17.0%)272 (16.7%)
Infeasible117 (29.3%)476 (29.2%)

Subset representativeness Consistent model performance across the human subset and full benchmark supports its representativeness. Results for all models are in the paper.

ModelHuman subset(400)Full benchmark(1632)
Gemini-2.5-Flash73.8/55.873.7/56.2
Qwen3-Omni-Think64.8/49.365.9/48.4
Nemotron3-Omni60.5/44.562.7/44.5
Gemma454.0/25.550.6/25.2

[References]
[1] SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversations, EMNLP, 2021
[2] HEMM: Holistic Evaluation of Multimodal Foundation Models, NeurIPS, 2024
[3] SimuHome: A Temporal- and Environment-Aware Benchmark for Smart Home LLM Agents, ICLR, 2026

Reviewer_QoqkOpenReview ↗

Minor Concerns.

Thank you for the rebuttal. Your response has addressed my concerns,

I have one minor suggestion for the related work section. IMHO, some works, such as CUHK-X (ACM MobiSys 2026), should be included and discussed in the related work section. Both benchmarks evaluate how models use multimodal cues to interpret human behavior in the home. It focuses on recognizing everyday activities and anticipating a person’s next action, while OmniSmartHome uses visual and spatial audio cues to resolve ambiguities in spoken requests. I think it is helpful for the impact of this paper, please make this distinction explicit in the revision.

Dear Reviewer Qoqk,

We sincerely thank you for your decision to raise your score and for your constructive and thoughtful feedback.

Following your suggestion, we have revisited the related literature and incorporated CUHK-X and several other relevant works that we had previously overlooked into a new paragraph titled Context-aware AI agents in everyday life in the Related Work section. We have updated the manuscript accordingly, with the revisions highlighted in blue.

Sincerely,
The Authors

Comments on the paper

2 comments

Dear Reviewers,

The authors have responded to your questions. If you haven’t already done so, please review their responses and let them know whether your concerns have been addressed. Please use the discussion to raise any remaining concerns or follow-up questions.

Thank you for your time!

Best,

AC

Dear AC and Reviewers,

Our work makes three core contributions, each positively recognized by the reviewers:

  1. OmniSmartHome benchmark: Described as well-motivated (Qoqk, gCvp), novel (fK6y), new (gCvp), comprehensive (fK6y), useful, and informative (Qoqk).
  2. Evaluation: Recognized for the comprehensiveness of our experiments (Qoqk, fK6y, gCvp).
  3. Simple agentic baseline (PROME): Recognized for its performance improvements across backbones (Qoqk, fK6y, gCvp), practicality (Qoqk), and value as a promising and concrete starting point for future research (Qoqk, fK6y, gCvp).

Reviewers feedback has substantially improved the paper. In particular, we have strengthened the following aspects:

  • Multimodal dependence of OmniSmartHome: Experiments across three models with restricted multimodal information confirm that our benchmark meaningfully requires multimodal grounding (Qoqk).
  • Effectiveness of PROME': We demonstrated that PROME's gains persist after accounting for spatial-audio access, tool availability, and manually specified procedures, and confirmed their statistical significance (Qoqk, gCvp).
  • Reliability of LLM-as-a-Judge and human verification: We demonstrated high agreement between LLM and human judgments and clarified the benchmark's human verification protocol (Qoqk, fK6y, gCvp).

We are grateful that all reviewers acknowledged that their concerns had been addressed.  Reviewer fK6y reaffirmed their initial positive assessment, and Qoqk and gCvp expressed their intention to raise their scores to Weak Accept in their comments.

We sincerely thank all reviewers for their thoughtful feedback and constructive engagement, which has helped strengthen our work.

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.