- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents a new benchmark SMH-Bench, for evaluating LLMs on smart home interaction tasks. Compared to existing similar benchmarks, SMH-Bench supports new evaluations which cover more realistic user requests, including corrections, multi-turn dialogue, and user preferences. SMH-Bench is constructed by combining LLM-based task generation and human audit, and is devided into different categories based on difficulty and query type. The paper also evaluate mainstream closed- and open-source models on this benchmark and provide insights into their performance across different task categories.
Critical strengths
- Clear new features that differ from existing related benchmarks.
- A dedicated and detailed evaluation of how current LLMs perform on different tasks.
Critical weaknesses
- Limited technical novelty compared to SimuHome (https://arxiv.org/abs/2509.24282), which uses a similar pipeline.
- Missing several critical ablations to verify whether the tasks effectively evaluate the intended capabilities. (Please see Q3 for details)
Questions for authors
- What are the criteria for determining the ground truth, especially for ambiguous requests where multiple valid answers may exist?
- Could the authors declare potential biases or gaps between LLM-generated tasks/instructions and real-world user requests?
- Could the authors provide ablation studies to show how performance on TC5 and TC6 changes respectively when the provided dialogue history and user memory are removed?
4 comments
Response to vcry (2/3)
W2/Q3: The role of history and memory, and context-removal experiments
TC5 and TC6 address requests whose execution goals depend on dialogue history or user memory. History establishes referents, supports continuation of previous operations, and conveys user corrections; memory supplies past settings and user preferences. The tasks therefore provide the context needed to interpret the current request, evaluating whether models can extract and apply this information to identify target devices, determine action parameters, and execute the task.
To examine the role of this information, we conducted context-removal experiments with DeepSeek-V3.2. For TC5, we removed dialogue history. For TC6, we removed historical context from short-term memory tasks and user preferences from long-term memory tasks. The experiments evaluate the same target turns, with the current request, initial environment, goal, and scoring rules unchanged. The table below uses the paper's results with context retained as the baseline.
| Setting | Task | With context | Without context | Drop |
|---|---|---|---|---|
| DR | TC5 | 80.0% | 47.0% | 33.0 pp |
| EIA | TC5 | 85.0% | 41.5% | 43.5 pp |
| DR | TC6 | 68.0% | 33.0% | 35.0 pp |
| EIA | TC6 | 74.0% | 26.0% | 48.0 pp |
Removing the relevant context substantially reduces success rates on both TC5 and TC6 under DR and EIA. These results support the substantive role of history and memory in task completion and provide empirical evidence for TC5/TC6's evaluation of contextual understanding, application, and execution capabilities.
The context-removal setup, results, and findings are included in Appendix F.5 (Table 24) of the revised manuscript.
Q1: GT criteria and valid alternative outcomes
GT is determined from the complete user requirements, relevant dialogue history or preferences, and the device states and executable capabilities in the current environment, and is expressed as state constraints that execution results must satisfy. Exact values are checked when explicitly requested; directional or range constraints are checked when only a direction of change or a range is specified. Goal constraints undergo execution checks in the environment and human review to verify their feasibility and consistency with user intent.
(1) Accepting multiple valid outcomes when no unique value is specified.
For example, suppose the user requests “Turn down the living-room air conditioner's temperature a bit,” and its current setpoint is 26°C. The intent is satisfied by identifying the correct air conditioner, lowering its setpoint to a supported value below 26°C, and meeting the requirement to leave other states unchanged. Thus, either 25°C or 24°C can pass evaluation. Such requests already specify the target device, attribute, and direction of change, allowing direct evaluation against state constraints.
(2) Resolving the execution goal through subsequent interaction for ambiguous requests requiring clarification.
For example, the initial user request “The room is too dark” does not specify the target room or the required adjustments:
| Item | Content |
|---|---|
| Initial user request | “The room is too dark.” |
| Clarification question | “Which room would you like to make brighter?” |
| Subsequent user reply | “Open the bedroom curtains and set the lights in both the bedroom and kitchen area to 80% brightness.” |
| Final state constraints | Lights in the bedroom and kitchen area at 80% brightness; bedroom curtains fully open |
The final state constraints are determined by the subsequent user reply. Clarification text is evaluated by an LLM judge; the execution stage provides the complete annotated dialogue and verifies outcomes through deterministic state checks.
(3) Accepting different execution methods that achieve the same target state.
For example, the bedroom curtains can be opened using either open() or set_position(100). Both methods fully open the curtains and therefore pass evaluation.
Thus, GT defines state constraints that satisfy the complete user requirements, and different outcomes and execution methods satisfying these constraints can all pass evaluation.
Table 15 in Appendix C now includes the initial request, clarification, and user reply for all three ambiguous-request examples, together with an explanation of clarification and execution evaluation.
Response to vcry (3/3)
Q2: Distributional and interaction differences from real household use
We agree that LLM-generated requests may differ from real user requests even after human review. SMH-Bench is designed to systematically cover capabilities in an executable environment with explicit, verifiable success criteria; it is not sampled according to the frequency of requests in real households. We identify three potential sources of bias and mismatch.
(1) Task distribution.
Requests are generated under predefined task categories, templates, and environment constraints. The proportions of task categories and household complexity levels serve capability coverage. Consequently, the prevalence of compositional control, ambiguity handling, and history- or memory-dependent requests in the benchmark may differ from their frequency in actual household use.
(2) Language and intent interpretation.
Instructions and reference responses are both generated by GPT-5 and may share linguistic and interpretive preferences. Template constraints may also produce more regular request structures than naturally occurring user requests. Although the benchmark includes colloquial language, repetition, corrections, and ambiguity, these generated texts cannot guarantee coverage of the full variation in real users’ wording, information omission, and expression of intent.
(3) Real-time interaction.
Real user requests often develop incrementally: users may pause before supplying additional parameters, or interrupt, correct, or withdraw a request while the assistant is responding or executing an action. Systems must therefore understand the request, determine whether the user has finished speaking, decide when to respond, and adapt ongoing actions to subsequent input. SMH-Bench includes textual corrections and ambiguous expressions, but does not cover end-of-turn detection in continuous speech, real-time interruptions, or changes to requests during execution. These factors may make practical use more challenging than the current text-based evaluation.
HomeEnv consistency checks and human review assess executability, instruction naturalness, and consistency among user requirements, context, and goals. These measures support instance quality but do not establish distributional equivalence with real user requests. Accordingly, SMH-Bench scores characterize model performance under the specified controlled task conditions, rather than average success over real household requests.
Response to Authors
Thanks to the authors for the detailed response. For W1, the authors emphasize the novelty in designing benchmark tasks and the value of this benchmark. For W2/Q3, the authors well addressed my concerns with additional ablation results. The authors have also clarified the GT criteria for Q1 and added a discussion on synthetic data bias for Q2. I appreciate the authors' efforts to improve the completeness of the paper. However, for W1, I still have concerns about whether the technical novelty is sufficient to claim a significant contribution rather than an incremental extension of SimuHome (which introduces a dynamic smart-home simulator). Therefore, I will maintain my current score.
Response to vcry (1/3)
Thank you for recognizing the benchmark's distinctive features and the value of its category-specific evaluation. We have incorporated the supplementary experiments and their findings into the revised manuscript. All revisions are marked in blue. We address your comments below.
W1: Differences in evaluation design from SimuHome
SMH-Bench's novelty lies in turning the process of determining task goals from the environment, dialogue history, and user preferences into a systematic evaluation with explicit task categories and success conditions. Smart home requests often leave target devices, action parameters, or execution conditions incompletely specified. Models must integrate the relevant information to determine the goal and then execute the correct actions. SMH-Bench designs its task categories and verification criteria around this process.
Both SimuHome and SMH-Bench use environment construction, instruction generation, and human validation, and verify task outcomes through executable environments. Their differences primarily concern the following three evaluation dimensions.
(1) Systematically evaluating how dialogue history and user memory affect task goals.
SimuHome's tasks mainly cover environment queries, implicit intent inference, explicit device control, and temporal scheduling. SMH-Bench organizes continuation of previous operations, reference resolution, user corrections, topic shifts, and memory application into explicit evaluation categories, testing whether models can use information beyond the current request to determine the correct devices and action parameters. For example, “Set the other one the same way as before” requires resolving the device reference and recovering settings from history; “Set it to my usual preferred temperature” requires applying user preferences; and user corrections require updating the goal accordingly. These tasks turn the effects of context on target devices, action parameters, and execution requirements into directly testable success conditions.
(2) Combining device selection and state constraints to evaluate goal determination and execution across household complexity levels.
SimuHome's state goals cover device attributes, directions of change in environmental variables, and device states at specified times. SMH-Bench systematically constructs target-selection tasks combining room relationships, device states, ranking, and exclusion conditions. These tasks span simple, medium, and complex homes, testing whether models can correctly determine target device sets from these combined conditions across different layouts and device inventories. For example, “Set the two brightest lights in the living room to 50%” requires jointly identifying the room, comparing device states, and applying a quantity constraint. Evaluation checks both whether target devices meet the requirements and whether non-target devices remain unchanged, jointly testing target selection and execution so that omissions, incorrect selections, and changes to unrelated devices are reflected in task outcomes.
(3) Verifying both the trigger logic and action effects of automation rules.
SimuHome's scheduling tasks focus on device states at future times and temporal coordination between actions. SMH-Bench's automation tasks cover time triggers, state triggers, and their combinations. For example, “Every day at two in the afternoon, if the kitchen temperature exceeds 28°C, turn on the kitchen air conditioner” requires expressing a recurring time, a state condition, and a device action. Evaluation checks whether the rule correctly specifies when and under what conditions to execute, as well as the action effects after triggering, testing whether models accurately translate natural-language requirements into condition–action rules.
These designs establish context-dependent goal determination, compositional device selection across household complexity levels, and automation rule construction as evaluation dimensions with explicit categories and verification criteria, complementing SimuHome's focus on environmental dynamics and temporal coordination.