acceptodds
Under review as a conference paper at ICLR 2027

SMH-Bench: Benchmarking LLM Agents for Environment-Grounded Reasoning and Action in Smart Homes

Abstract

Smart homes are evolving toward complex state-dependent living environments, requiring Large Language Models (LLMs) to reason over user intent, preferences, and multi-device interactions. However, existing smart-home benchmarks often focus on static instruction-to-API mapping or limited simulations, failing to evaluate whether LLMs can reason, interact, and act reliably in realistic household scenarios. To address these limitations, we introduce SMH-Bench, a comprehensive benchmark for evaluating LLMs in smart-home environments. Built upon HomeEnv, an executable and verifiable smart-home simulator, SMH-Bench contains 1,100 high-quality tasks spanning 7 categories and 22 fine-grained subcategories. It further stratifies tasks across simple, medium and complex homes, ranging from small apartments to dense multi-room environments with 135 devices. Experiments show that although frontier LLMs achieve strong performance on explicit control and query tasks, they still exhibit significant weaknesses in automation task scheduling, ambiguity handling and personalized reasoning, especially as home complexity increases. We hope SMH-Bench will facilitate the development of more reliable, context-aware, and practically deployable smart-home agents.

Primary area:
datasets and benchmarks
Keywords:
Smart Home, LLM Agents

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vcryOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a new benchmark SMH-Bench, for evaluating LLMs on smart home interaction tasks. Compared to existing similar benchmarks, SMH-Bench supports new evaluations which cover more realistic user requests, including corrections, multi-turn dialogue, and user preferences. SMH-Bench is constructed by combining LLM-based task generation and human audit, and is devided into different categories based on difficulty and query type. The paper also evaluate mainstream closed- and open-source models on this benchmark and provide insights into their performance across different task categories.

Critical strengths
  • Clear new features that differ from existing related benchmarks.
  • A dedicated and detailed evaluation of how current LLMs perform on different tasks.
Critical weaknesses
  • Limited technical novelty compared to SimuHome (https://arxiv.org/abs/2509.24282), which uses a similar pipeline.
  • Missing several critical ablations to verify whether the tasks effectively evaluate the intended capabilities. (Please see Q3 for details)
Questions for authors
  1. What are the criteria for determining the ground truth, especially for ambiguous requests where multiple valid answers may exist?
  2. Could the authors declare potential biases or gaps between LLM-generated tasks/instructions and real-world user requests?
  3. Could the authors provide ablation studies to show how performance on TC5 and TC6 changes respectively when the provided dialogue history and user memory are removed?
4 comments
AuthorsOpenReview ↗

Response to vcry (1/3)

Thank you for recognizing the benchmark's distinctive features and the value of its category-specific evaluation. We have incorporated the supplementary experiments and their findings into the revised manuscript. All revisions are marked in blue. We address your comments below.

W1: Differences in evaluation design from SimuHome

SMH-Bench's novelty lies in turning the process of determining task goals from the environment, dialogue history, and user preferences into a systematic evaluation with explicit task categories and success conditions. Smart home requests often leave target devices, action parameters, or execution conditions incompletely specified. Models must integrate the relevant information to determine the goal and then execute the correct actions. SMH-Bench designs its task categories and verification criteria around this process.

Both SimuHome and SMH-Bench use environment construction, instruction generation, and human validation, and verify task outcomes through executable environments. Their differences primarily concern the following three evaluation dimensions.

(1) Systematically evaluating how dialogue history and user memory affect task goals.

SimuHome's tasks mainly cover environment queries, implicit intent inference, explicit device control, and temporal scheduling. SMH-Bench organizes continuation of previous operations, reference resolution, user corrections, topic shifts, and memory application into explicit evaluation categories, testing whether models can use information beyond the current request to determine the correct devices and action parameters. For example, “Set the other one the same way as before” requires resolving the device reference and recovering settings from history; “Set it to my usual preferred temperature” requires applying user preferences; and user corrections require updating the goal accordingly. These tasks turn the effects of context on target devices, action parameters, and execution requirements into directly testable success conditions.

(2) Combining device selection and state constraints to evaluate goal determination and execution across household complexity levels.

SimuHome's state goals cover device attributes, directions of change in environmental variables, and device states at specified times. SMH-Bench systematically constructs target-selection tasks combining room relationships, device states, ranking, and exclusion conditions. These tasks span simple, medium, and complex homes, testing whether models can correctly determine target device sets from these combined conditions across different layouts and device inventories. For example, “Set the two brightest lights in the living room to 50%” requires jointly identifying the room, comparing device states, and applying a quantity constraint. Evaluation checks both whether target devices meet the requirements and whether non-target devices remain unchanged, jointly testing target selection and execution so that omissions, incorrect selections, and changes to unrelated devices are reflected in task outcomes.

(3) Verifying both the trigger logic and action effects of automation rules.

SimuHome's scheduling tasks focus on device states at future times and temporal coordination between actions. SMH-Bench's automation tasks cover time triggers, state triggers, and their combinations. For example, “Every day at two in the afternoon, if the kitchen temperature exceeds 28°C, turn on the kitchen air conditioner” requires expressing a recurring time, a state condition, and a device action. Evaluation checks whether the rule correctly specifies when and under what conditions to execute, as well as the action effects after triggering, testing whether models accurately translate natural-language requirements into condition–action rules.

These designs establish context-dependent goal determination, compositional device selection across household complexity levels, and automation rule construction as evaluation dimensions with explicit categories and verification criteria, complementing SimuHome's focus on environmental dynamics and temporal coordination.

AuthorsOpenReview ↗

Response to vcry (2/3)

W2/Q3: The role of history and memory, and context-removal experiments

TC5 and TC6 address requests whose execution goals depend on dialogue history or user memory. History establishes referents, supports continuation of previous operations, and conveys user corrections; memory supplies past settings and user preferences. The tasks therefore provide the context needed to interpret the current request, evaluating whether models can extract and apply this information to identify target devices, determine action parameters, and execute the task.

To examine the role of this information, we conducted context-removal experiments with DeepSeek-V3.2. For TC5, we removed dialogue history. For TC6, we removed historical context from short-term memory tasks and user preferences from long-term memory tasks. The experiments evaluate the same target turns, with the current request, initial environment, goal, and scoring rules unchanged. The table below uses the paper's results with context retained as the baseline.

SettingTaskWith contextWithout contextDrop
DRTC580.0%47.0%33.0 pp
EIATC585.0%41.5%43.5 pp
DRTC668.0%33.0%35.0 pp
EIATC674.0%26.0%48.0 pp

Removing the relevant context substantially reduces success rates on both TC5 and TC6 under DR and EIA. These results support the substantive role of history and memory in task completion and provide empirical evidence for TC5/TC6's evaluation of contextual understanding, application, and execution capabilities.

The context-removal setup, results, and findings are included in Appendix F.5 (Table 24) of the revised manuscript.

Q1: GT criteria and valid alternative outcomes

GT is determined from the complete user requirements, relevant dialogue history or preferences, and the device states and executable capabilities in the current environment, and is expressed as state constraints that execution results must satisfy. Exact values are checked when explicitly requested; directional or range constraints are checked when only a direction of change or a range is specified. Goal constraints undergo execution checks in the environment and human review to verify their feasibility and consistency with user intent.

(1) Accepting multiple valid outcomes when no unique value is specified.

For example, suppose the user requests “Turn down the living-room air conditioner's temperature a bit,” and its current setpoint is 26°C. The intent is satisfied by identifying the correct air conditioner, lowering its setpoint to a supported value below 26°C, and meeting the requirement to leave other states unchanged. Thus, either 25°C or 24°C can pass evaluation. Such requests already specify the target device, attribute, and direction of change, allowing direct evaluation against state constraints.

(2) Resolving the execution goal through subsequent interaction for ambiguous requests requiring clarification.

For example, the initial user request “The room is too dark” does not specify the target room or the required adjustments:

ItemContent
Initial user request“The room is too dark.”
Clarification question“Which room would you like to make brighter?”
Subsequent user reply“Open the bedroom curtains and set the lights in both the bedroom and kitchen area to 80% brightness.”
Final state constraintsLights in the bedroom and kitchen area at 80% brightness; bedroom curtains fully open

The final state constraints are determined by the subsequent user reply. Clarification text is evaluated by an LLM judge; the execution stage provides the complete annotated dialogue and verifies outcomes through deterministic state checks.

(3) Accepting different execution methods that achieve the same target state.

For example, the bedroom curtains can be opened using either open() or set_position(100). Both methods fully open the curtains and therefore pass evaluation.

Thus, GT defines state constraints that satisfy the complete user requirements, and different outcomes and execution methods satisfying these constraints can all pass evaluation.

Table 15 in Appendix C now includes the initial request, clarification, and user reply for all three ambiguous-request examples, together with an explanation of clarification and execution evaluation.

AuthorsOpenReview ↗

Response to vcry (3/3)

Q2: Distributional and interaction differences from real household use

We agree that LLM-generated requests may differ from real user requests even after human review. SMH-Bench is designed to systematically cover capabilities in an executable environment with explicit, verifiable success criteria; it is not sampled according to the frequency of requests in real households. We identify three potential sources of bias and mismatch.

(1) Task distribution.

Requests are generated under predefined task categories, templates, and environment constraints. The proportions of task categories and household complexity levels serve capability coverage. Consequently, the prevalence of compositional control, ambiguity handling, and history- or memory-dependent requests in the benchmark may differ from their frequency in actual household use.

(2) Language and intent interpretation.

Instructions and reference responses are both generated by GPT-5 and may share linguistic and interpretive preferences. Template constraints may also produce more regular request structures than naturally occurring user requests. Although the benchmark includes colloquial language, repetition, corrections, and ambiguity, these generated texts cannot guarantee coverage of the full variation in real users’ wording, information omission, and expression of intent.

(3) Real-time interaction.

Real user requests often develop incrementally: users may pause before supplying additional parameters, or interrupt, correct, or withdraw a request while the assistant is responding or executing an action. Systems must therefore understand the request, determine whether the user has finished speaking, decide when to respond, and adapt ongoing actions to subsequent input. SMH-Bench includes textual corrections and ambiguous expressions, but does not cover end-of-turn detection in continuous speech, real-time interruptions, or changes to requests during execution. These factors may make practical use more challenging than the current text-based evaluation.

HomeEnv consistency checks and human review assess executability, instruction naturalness, and consistency among user requirements, context, and goals. These measures support instance quality but do not establish distributional equivalence with real user requests. Accordingly, SMH-Bench scores characterize model performance under the specified controlled task conditions, rather than average success over real household requests.

Reviewer_vcryOpenReview ↗

Response to Authors

Thanks to the authors for the detailed response. For W1, the authors emphasize the novelty in designing benchmark tasks and the value of this benchmark. For W2/Q3, the authors well addressed my concerns with additional ablation results. The authors have also clarified the GT criteria for Q1 and added a discussion on synthetic data bias for Q2. I appreciate the authors' efforts to improve the completeness of the paper. However, for W1, I still have concerns about whether the technical novelty is sufficient to claim a significant contribution rather than an incremental extension of SimuHome (which introduces a dynamic smart-home simulator). Therefore, I will maintain my current score.

Reviewer_DPCnOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

SMH-Bench evaluates language-model agents in an executable smart-home simulator. It covers device control, automation, ambiguous requests, dialogue context, memory, and queries across homes of different sizes. The paper compares direct prediction with interactive tool use.

Critical strengths
  1. Evaluating final environment states is more meaningful than checking whether an API call matches a reference sequence.
  2. The benchmark covers a useful range of smart-home tasks.
  3. The analysis by home complexity and error type is informative.
Critical weaknesses
  1. The comparison between direct and interactive reasoning is not controlled. One setting receives the full home state, while the other starts with partial information and has a limited interaction budget.
  2. There is no ablation that matches state visibility across the two settings or varies the interaction budget.
  3. The simulator uses deterministic state updates and leaves out physical dynamics, noisy sensors, and device failures.
  4. The dialogue tasks appear to use fixed histories rather than actual back-and-forth interaction with a user.
Questions for authors

see weakness

2 comments
AuthorsOpenReview ↗

Response to DPCn (1/2)

Thank you for recognizing the value of our state-based evaluation, task coverage, and experimental analyses. We have incorporated the supplementary experiments and their findings into the revised manuscript. All revisions are marked in blue. We address your comments below.

W1/W2: State visibility and interaction budget in the DR/EIA comparison

The DR/EIA comparison evaluates task completion under two complete execution protocols. DR provides the full environment context and tests whether a model can integrate the available information and directly generate correct actions. EIA completes tasks through on-demand queries, action execution, and feedback. Observability, output format, execution feedback, and interaction budget jointly define the task completion conditions of each protocol.

To address the concerns about initial state visibility and interaction budget, we conducted EIA-full experiments and a tool-budget sensitivity analysis with DeepSeek-V3.2.

(1) Providing the same complete initial environment information as DR.

On the same 1,100 tasks, we provided EIA with the complete initial environment JSON used by DR for each task, creating the EIA-full setting. We retained tool interaction, execution feedback, and the 10-round tool interaction budget, using the same scoring rules.

Task categoryDR (%)EIA (%)EIA-full (%)
TC1 Atomic Control79.387.390.7
TC2 Compositional Control60.852.858.4
TC3 Ambiguous Intent65.385.384.0
TC4 Automated Task Scheduling52.064.060.0
TC5 Context-Aware Multi-turn Interaction80.085.083.0
TC6 Personalized Memory68.074.074.0
TC7 Environment-Grounded Query89.094.094.0
Overall69.575.075.6

EIA-full achieves an overall success rate of 75.6%, an increase of 0.6 percentage points over EIA, and remains 6.2 percentage points above DR. Both EIA settings outperform DR in six of the seven categories and score lower on TC2 (Compositional Control). With the same complete initial environment information, interactive execution and direct generation still yield different task performance.

(2) Varying the EIA tool interaction budget.

The main evaluation uses a uniform budget of at most 10 tool interaction rounds, with multiple tool calls allowed within a round. We additionally compared DeepSeek-V3.2 with budgets of 10, 20, and 40 rounds:

Maximum tool interaction roundsSuccess rateChange from EIA@10
1075.0%—
2075.6%+0.6 pp
4076.3%+1.3 pp

Increasing the budget from 10 to 40 rounds improves the success rate by 1.3 percentage points, with a 95% confidence interval of [−0.6, +2.7] pp. Although the point estimate increases with the budget, the current analysis does not detect a reliable overall improvement. The main results use the uniform EIA@10 protocol; the larger-budget results characterize budget sensitivity.

Across the DeepSeek-V3.2 settings examined, neither providing the complete initial state nor increasing the interaction budget changes the overall pattern of EIA achieving a higher success rate than DR.

The setups, results, and findings of the initial-state visibility and tool-budget experiments are included in Appendix F.4 (Tables 22–23) of the revised manuscript.

AuthorsOpenReview ↗

Response to DPCn (2/2)

W3: Evaluation scope and research value of the deterministic simulator

SMH-Bench focuses on decision-making and task execution at the agent policy layer. A smart home system can be divided into an agent policy layer and a real-device execution layer: the former determines goals and actions from user requests, context, and environment states, while the latter handles device communication, physical execution, and state feedback. We connect models to an executable environment through a unified device interface, evaluating whether they correctly interpret intent, select devices, determine parameters, and execute actions. Task outcomes are verified against the resulting environment states.

For example, for “Set the living-room air conditioner to 25°C,” the benchmark checks whether the model selects the correct air conditioner, changes its temperature setpoint to 25°C, and leaves non-target devices unchanged. When the actual room temperature reaches 25°C depends on physical processes such as cooling and heat exchange. The device interface abstracts low-level execution details, while preserving the decision problem of translating user intent and environment information into correct actions. Tasks involving reference resolution, corrections, conditional filtering, multi-device coordination, and automation rule construction all require this translation and warrant dedicated evaluation.

Deterministic state transitions and execution conditions allow models to be evaluated in consistent environments, so task outcomes more directly reflect these decision-making capabilities. The current environment provides execution results and error feedback at the device interface, but does not simulate continuous physical dynamics, sensor noise, or stochastic device failures. Introducing these factors requires specifying dynamic processes, observation errors, and failure mechanisms; task performance would then also involve handling action delays, reasoning under uncertainty, and recovering from failures.

End-to-end deployment in real homes requires joint evaluation of both layers for adaptation and recovery under these conditions.

W4: Fixed dialogue histories and evaluation across dialogue turns

SMH-Bench provides annotated dialogue histories and their corresponding initial states to establish the context needed for corrections, reference resolution, and continuation of previous operations. It evaluates whether models use this information to understand and correctly execute the current request. The historical context is supplied through annotations; interpretation, decision-making, and actions at the target turn are performed by the evaluated model and verified through environment execution.

For example, “Set the living-room air conditioner to 26°C” establishes the previous operation, while “No, change it to 24°C” is the target turn for evaluating correction. Preceding turns may themselves belong to other categories, such as atomic control. Providing consistent histories and initial states allows category scores to focus on correction, reference resolution, or continuation at the target turn.

Following MT-Bench-101 (Bai et al., 2024), we evaluate multi-turn dialogues using annotated dialogue history as golden context, with the corresponding initial environment state supplied at each turn.

Beyond the category-specific target-turn evaluation, we additionally summarized DeepSeek-V3.2's task completion across all five turns of 50 short-term memory dialogues, reporting both success at each turn and the proportion of dialogues in which all turns up to that point pass.

Turn tDR SRDR CSREIA SREIA CSR
184.0%84.0%90.0%90.0%
280.0%70.0%84.0%78.0%
392.0%64.0%92.0%72.0%
494.0%60.0%90.0%66.0%
566.0%44.0%70.0%46.0%

SR (Success Rate) is task success at turn t; CSR (Cumulative Success Rate) is the proportion of dialogues in which all of the first t turns pass. Each turn includes 50 tasks, with the corresponding annotated history and initial state provided.

The target-turn success rates are 66.0% / 70.0% for DR/EIA, respectively, while the rates of passing all five turns are 44.0% / 46.0%. Target-turn evaluation assesses correction, reference resolution, and continuation in a specific context, while per-turn and joint evaluation additionally examines task completion across the turns of the same dialogue. Together, they characterize how models apply dialogue context to complete tasks.

The five-turn evaluation setup, complete results, and findings are included in Appendix F.5 (Table 25) of the revised manuscript.

References

Bai et al. (2024). MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues. ACL 2024, 7421–7454. https://aclanthology.org/2024.acl-long.401/

Reviewer_BMzEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SMH-Bench, a benchmark of 1100 smarthome tasks across seven categories, built on the executable HomeEnv simulator. Tasks cover control, ambiguity, automation, dialogue context, memory, and queries. Thirteen LLMs are evaluated through one-shot Direct Reasoning and tool-based Environment-Interactive Agent settings. Evaluation combines state-based verification with validated LLM judging.

Critical strengths
  1. Executable, state-based evaluation
  • HomeEnv verifies resulting device states rather than matching action sequences, while requiring irrelevant states to remain unchanged. This supports evaluation across different agent implementations.
  1. Careful validation of semantic judging
  • The authors report 98% agreement with blind human judgements on 200 instances and 96.4% agreement with an independent-family judge, including checks for model-family bias.
Critical weaknesses
  1. The evaluation does not isolate the causes of performance differences
  • DR and EIA differ in observability, output format, execution feedback, and interaction budget. Their comparison cannot isolate the benefit of tool use.
  • Request/parsing failures and budget exhaustion affect results. Claude has 58 DR request/parsing errors, while excessive tool calls account for 29% of DeepSeek's categorized EIA errors (table 17). Their impact on model ranking is not assessed through retries or budget sensitivity experiments.
  1. Some conclusions exceed what the benchmark establishes
  • Underspecified instructions can be scored against overly specific targets. For example, the critic for “the room is too dark” requires exactly brightness 80 on two lights and curtain position 100 (Table 15). Unless these values come from a clarification reply, the construction checks do not establish why these exact values are necessary, and other reasonable solutions would fail.
  • Dialogue history and memory are supplied as context, limiting conclusions about end-to-end conversation and memory management.
  • Different instances populate the home-complexity tiers, so household size is not isolated from task difficulty.
Questions for authors
  1. How are alternative valid outcomes handled for underspecified requests, such as lighting example in table 15?
  2. Do model rankings and DR-EIA differences persist with larger interaction budgets and retries for infrastructure failures?
  3. Is clarification followed by a user response during evaluation, and if so, how is that response produced, particularly in the one-shot DR setting?
5 comments
AuthorsOpenReview ↗

Response to BMzE (1/3)

Thank you for recognizing the value of our state-based evaluation and validation of semantic judging. We have incorporated the supplementary experiments and their findings into the revised manuscript. All revisions are marked in blue.

W1.1: DR/EIA protocol design and initial state visibility

The DR/EIA comparison evaluates task completion under two complete execution protocols. DR provides the full environment context and tests whether a model can integrate the available information and directly generate correct actions. EIA completes tasks through on-demand queries, action execution, and feedback. Observability, output format, execution feedback, and interaction budget jointly define the task completion conditions of each protocol.

To examine whether initial environment information affects this comparison, we conducted an EIA-full experiment with DeepSeek-V3.2: on the same 1,100 tasks, we provided EIA with the complete initial environment JSON used by DR for each task, while retaining tool interaction, execution feedback, and the 10-round tool interaction budget, using the same scoring rules.

Task categoryDR (%)EIA (%)EIA-full (%)
Atomic Control79.387.390.7
Compositional Control60.852.858.4
Ambiguous Intent65.385.384.0
Automated Task Scheduling52.064.060.0
Context-Aware Multi-turn Interaction80.085.083.0
Personalized Memory68.074.074.0
Environment-Grounded Query89.094.094.0
Overall69.575.075.6

EIA-full achieves an overall success rate of 75.6%, an increase of 0.6 percentage points over EIA, and remains 6.2 percentage points above DR. Both EIA settings outperform DR in six of the seven categories and score lower on Compositional Control.

EIA can progressively acquire the environment information needed for a task through tool calls, obtaining states missing from the initial context. The small overall change from providing the complete initial environment in this experiment is consistent with this mechanism of acquiring information on demand. After aligning initial state visibility, the model's overall relative performance under DR and EIA remains unchanged.

The experimental setup, category-level results, and findings are included in Appendix F.4 (Table 22) of the revised manuscript.

W1.2/Q2: Effects of retries and tool budgets on performance and ranking

We conducted a retry experiment for Claude Sonnet 4.6's DR request/parsing errors and a tool-budget sensitivity analysis for DeepSeek-V3.2.

(1) Claude Sonnet 4.6: Effects of retrying request/parsing errors.

For the original 58 request/parsing error events, we reissued requests or provided format-correction prompts to obtain successfully returned, parseable model outputs. With at most 3 retries, 24 previously failed tasks passed evaluation under the original task success criteria. Retries provide neither reference answers nor task-scoring feedback, and all other task results remain unchanged.

After retries, DR success increases from 76.7% to 78.9% (+2.2 pp), moving Claude from 4th to 3rd relative to the other models' original DR results. The gap to its own EIA score (84.1%) narrows from 7.4 to 5.2 pp. Thus, retries change its position in the ranking and narrow the DR–EIA gap, while EIA retains a higher overall score than DR.

The retry setup and its effects on success rate, ranking, and the DR–EIA gap are included in Appendix E.2 of the revised manuscript.

(2) DeepSeek-V3.2: Effects of tool budgets on success rate and relative performance.

The main evaluation uses a uniform budget of at most 10 tool interaction rounds, with multiple tool calls allowed within a round. The 29.0% in Table 17 is the share of categorized error events attributed to budget exhaustion. To examine this issue, we compared DeepSeek-V3.2 under different interaction budgets:

Maximum tool interaction roundsSuccess rateChange from EIA@10
1075.0%—
2075.6%+0.6 pp
4076.3%+1.3 pp

Increasing the budget from 10 to 40 rounds improves the success rate by 1.3 percentage points, with a 95% confidence interval of [−0.6, +2.7] pp. Although the point estimate increases with the budget, the current analysis does not detect a reliable overall improvement. EIA@40 remains above DR's 69.5%, so increasing the budget does not change the overall relative performance of the two protocols.

Regarding model ranking, although the EIA@40 point estimate exceeds some models' EIA@10 scores, these are comparisons across different budgets. Model rankings remain based on the uniform EIA@10 protocol; the larger-budget results characterize budget sensitivity.

The tool-budget sensitivity results and analysis are included in Appendix F.4 (Table 23) of the revised manuscript.

AuthorsOpenReview ↗

Response to BMzE (2/3)

W2.1/Q1/Q3: Goal specification, clarification, and valid alternative outcomes for ambiguous requests

The exact state targets in Table 15 follow the user's requirements after clarification. The original manuscript showed only the initial requests and execution targets; the revised manuscript now includes the clarification and subsequent user reply for all three TC3 examples, making the source of the target states explicit.

(1) Source of the exact targets.

The dialogue and goal for the example in Table 15 are as follows:

Dialogue stageContent
Q1: Initial request“The room is too dark.”
A1: Clarification question“Which room would you like to make brighter?”
Q2: User reply“Open the bedroom curtains and set the lights in both the bedroom and kitchen area to 80% brightness.”
Target stateBedroom curtains fully open; lights in both areas at 80% brightness

The 80% brightness for both areas comes from the explicit requirement in Q2, while curtain position 100% denotes “fully open” in the device interface.

(2) Clarification and execution evaluation for ambiguous requests.

For ambiguous requests requiring clarification, both DR and EIA use two evaluation stages: clarification and execution. First, the model generates a clarification based on the initial request. An LLM judge assesses its appropriateness and its role in eliciting missing information. The execution stage then provides the annotated Q1/A1/Q2, asks the model to perform the operations, and verifies the result through deterministic state checks.

A1 and the subsequent user reply Q2 are fixed annotations; they do not change dynamically with the model-generated clarification. This gives all models the same explicit requirements. DR generates the corresponding output separately at each stage, while EIA uses tool queries and execution as appropriate to the stage.

(3) Acceptance of valid alternative outcomes.

GT defines valid outcomes from the complete user requirements: when an exact value is specified, that value is checked; when only a direction of change or a range is specified, different final states satisfying the corresponding constraints are accepted. For example, if the living-room air conditioner's current setpoint is 26°C and the user asks to “turn it down a bit,” setting it to either 25°C or 24°C can pass, provided that the correct device is operated, the value is within its supported range, and the other task constraints are satisfied.

Different execution methods that achieve the same goal can also pass: for example, both open() and set_position(100) can fully open the curtains. In Table 15, the user explicitly requests 80% brightness, so other brightness values do not satisfy the requirements of that execution stage. The criterion is always whether the execution result satisfies the complete user requirements.

Table 15 in Appendix C now includes these dialogues, followed by an explanation of clarification and execution evaluation.

AuthorsOpenReview ↗

Response to BMzE (3/3)

W2.2: Evaluation objectives with supplied history and memory, and supplementary results across dialogue turns

TC5 and TC6 evaluate whether models can apply dialogue history and user memory to understand and execute the current request. History supplies referents, previous operations, and user corrections; memory supplies past device settings, relevant facts, and user preferences. Models must use this information to determine target devices and action parameters. These scores reflect the ability to understand and apply contextual information and execute the resulting actions. Memory writing, retrieval, updating, and forgetting are distinct system-management processes.

Tasks are organized by the capability required at the target turn. Preceding turns establish the context for correction, reference resolution, or continuation, and may themselves involve atomic control or other task types. Providing annotated histories and corresponding initial states focuses category scores on the target-turn capability, reducing the combined influence of failures on different task types in preceding turns.

Following MT-Bench-101 (Bai et al., 2024), we evaluate multi-turn dialogues using annotated dialogue history as golden context, with the corresponding initial environment state supplied at each turn.

To complement the category-specific target-turn scores, we additionally examined task completion across all turns of the same dialogue. Specifically, we summarized DeepSeek-V3.2's execution results across all five turns of 50 short-term memory dialogues, reporting both success at each turn and the proportion of dialogues in which all turns up to that point pass:

Turn tDR SRDR CSREIA SREIA CSR
184.0%84.0%90.0%90.0%
280.0%70.0%84.0%78.0%
392.0%64.0%92.0%72.0%
494.0%60.0%90.0%66.0%
566.0%44.0%70.0%46.0%

SR (Success Rate) is task success at turn t; CSR (Cumulative Success Rate) is the proportion of dialogues in which all of the first t turns pass. Each turn includes 50 tasks, with the corresponding annotated history and initial state provided.

The target-turn success rates are 66.0% / 70.0% for DR/EIA, respectively, while the rates of passing all five turns are 44.0% / 46.0%. The target-turn metric measures performance on a specific context-dependent task, while the all-five-turn success rate measures whether every turn in the same dialogue is completed successfully. They provide complementary evidence on category-specific capabilities and task completion across an entire dialogue.

The five-turn evaluation setup, complete results, and findings are included in Appendix F.5 (Table 25) of the revised manuscript.

W2.3: Household scaling with fixed tasks

To examine the effect of household environment size on task completion, we conducted a paired experiment with DeepSeek-V3.2 that keeps tasks fixed while progressively expanding the environment. We selected 100 tasks spanning seven categories from simple homes, constructed medium and complex home versions for each task, and evaluated them under both DR and EIA. The medium and complex versions were expanded to 11 rooms with 35 devices and 31 rooms with 135 devices, respectively.

All three versions retain the same user requests, history and memory, original room and device states, target outcomes, and scoring rules; only rooms and devices are added. We checked task executability after expansion and verified filtering and ranking results to ensure that the correct targets remained unchanged.

DeepSeek-V3.2's success rates across the three household sizes on these 100 paired tasks are:

ProtocolSimple homesMedium homesComplex homesComplex − Simple
DR80.0%76.0%64.0%−16.0 pp
EIA85.0%79.0%70.0%−15.0 pp

On the same set of tasks, DeepSeek-V3.2's success rate decreases as the household environment expands under both DR and EIA, dropping by 16.0 and 15.0 percentage points, respectively, from simple to complex homes. These paired results demonstrate the effect of household environment size on task completion, providing additional evidence for the household-complexity analysis under fixed-task conditions.

The fixed-task household-scaling setup, results, and findings are included in Appendix F.6 (Table 26) of the revised manuscript.

References

Bai et al. (2024). MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues. ACL 2024, 7421–7454. https://aclanthology.org/2024.acl-long.401/

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.