- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes OmniSmartHome: pairing visual and spatial audio with voice requests from smart home agents, using G0-G4 to distinguish explicit/implicit references, covering 1360 synthesized+272 real episodes. There are three contributions: the first multimodal context disambiguation benchmark; Reviewed 16 Omni LLMs and found that when the voice is clear, it is strong, and when relying on gestures/sounds/directions/identities, it drops significantly; Propose PROME baseline using perception tools and program memory. In terms of verification, the manual level is very high, and PROME has generally improved on six models, and ablation, memory transfer, and real set testing have also been conducted.
Critical strengths
First, the problem is well motivated and genuinely new: existing smart-home benchmarks are language-only, while OmniSmartHome is the first to require resolving underspecified requests from gesture, sound, space, and speaker identity. This matters because it targets a real gap in evaluating Omni-LLMs, not just another leaderboard. Second, the benchmark and validation are substantial: synthetic plus real episodes, human verification, 16 models, and a simple PROME baseline that improves across backbones and transfers to real data. That combination makes the reported gap credible and gives the community a concrete starting point.
Critical weaknesses
One reason is that the benchmark volume is relatively small, with only 272 items in the real world, and the synthesized data is a static graph. The true generalization conclusion is not hard enough, and the statistical persuasiveness is limited. The second reason is that PROME is more like engineering assembly, with uneven improvement. G4 and some models have very little gain, and rely on external perception tools without real-time delay evaluation, which weakens the contribution of the "baseline can narrow the gap" method. Overall, it looks more like a dataset report, with insufficient innovative methods.
Questions for authors
How are the 272 real sets distributed on G0-G4 and Q1-Q4, and is it enough to support the conclusion of 'real generalization'? Has a significance test been conducted on the improvement of PROME, especially the small gains of G4 and some models? What is the consistency and error rate of manually verified annotations?
6 comments
Thanks the authors for their detailed response and supplementary analysis. The authors more accurately defined “real-world generalization” as “transfer from synthetic to real-world settings” and provided a correlation analysis between simulated and real-world performance, which alleviated my concerns regarding the scope of their conclusions. The newly added paired significance test also provides stronger statistical support for PROME’s overall improvement; the significant gains observed for G4 across all four models partially address my concerns regarding limited gains in that category. Therefore, I am raising my rating from 2 (weak rejection) to 3 (weak acceptance).
Dear Reviewer gCvp,
We are grateful that our response has addressed your concerns, and we sincerely appreciate your decision to raise your score.
Thank you again for your constructive feedback. We have carefully incorporated your suggestions into the manuscript and will continue to refine these revisions in the final version.
Sincerely,
The Authors
Dear reviewer gCvp,
We sincerely appreciate your efforts and comments. All revisions are highlighted in blue in the updated PDF.
[W1, Q1] Real-world benchmark size and generalization.
OmniSmartHome includes 1,360 synthetic and 272 real-world episodes collected across five homes and five participants. As described in Line 224, eight episodes are included for each grounding–query/feasibility type. Real-world evaluations in simulation-centric embodied and audio-visual benchmarks are often limited in scale, involving a single office [1] or a single scanned lab [2], while some rely entirely on simulation [3]. Similarly, existing smart-home benchmarks, including HomeBench [4], SimuHome [5], SMH-Bench [6], and MIST [7], lack real-world evaluations. In this context, our 272 real-world episodes across five homes represent a meaningful addition to existing benchmarks.
We clarify that “Real-world generalization” in Section 4.2 (line 407) refers specifically to whether PROME, whose procedural memory is learned entirely from synthetic data, remains effective on real-world episodes, rather than claiming that our 272 episodes establish broad generalization across real-world smart-home environments. We acknowledge that the original heading could imply a broader claim and revise it to “Synthetic-to-real transfer.”
To complement this analysis, we further assess the sim-to-real predictivity of OmniSmartHome's synthetic evaluations using the Sim-vs-Real Correlation Coefficient (SRCC)[2]. Across 22 model configurations, we obtain high SRCC values for both grounding accuracy (0.949) and goal accuracy (0.944). These results demonstrate that OmniSmartHome's synthetic evaluations meaningfully reflect relative model performance in real-world settings, supporting its utility as a benchmark for multimodal smart-home agents.
[W2] Novelty of PROME.
We agree that PROME is a relatively simple, tool-augmented baseline rather than a fundamentally new methodological framework. Our primary contribution is the benchmark, the first multimodal smart-home benchmark in which spoken requests are accompanied by surrounding visual and spatial-audio context. PROME is intended to demonstrate that existing models can partially bridge the observed performance gaps through structured perception and tool use, rather than to claim methodological novelty. We acknowledge its uneven gains across models and clarify its role and scope in the revised paper.
[Q2] Statistical significance of PROME
We conduct exact McNemar tests on paired per-episode success. The table below reports accuracy improvements over each base model with the corresponding p-values. PROME yields statistically significant improvements for all six models on both metrics (p < 0.05).
For G4 Identity (n = 384), PROME also yields positive improvements across all six models, with statistically significant gains for four of them (p ≤ 0.001).
[Q3] Annotation Quality Control
We employed a two-stage quality-control process combining automatic filtering and human verification.
Automatic filtering assessed TTS quality using ASR-based transcription errors and applied task-specific criteria to reduce ambiguity and unintended cues, such as device proximity.
Human verification involved two annotators checking task-specific validity, including full-body and finger visibility, pointing direction, head-orientation clarity, and relative audio levels between TTS and device noise. Only samples approved by both annotators were retained.
Overall, 49.2% of candidate samples were excluded, and the final dataset was selected from the remaining samples to balance target devices and task types.