acceptodds
Under review as a conference paper at ICLR 2027

MUSE-Home: Evaluating and Aligning LLM Agents for Multi-User Smart Homes

Abstract

Smart-home assistants work in shared environments, where residents have different needs, permissions, routines, and observations, while their actions affect one another through shared spaces and devices. We introduce Multi-User Smart-home Evaluation (MUSE-Home), an executable benchmark that studies four challenges created by this setting: multi-user conflict, socially conditioned proactivity, cross-resident environment forensics, and robustness against unnecessary coordination. MUSE-Home contains 6,790 training instances and a manually audited out-of-distribution test set of 721 instances. The best evaluated model achieves only 45.35% task success. We further propose Aggregating Local Interests with Global Need-weighting (MUSE-Align), a judge-based alignment framework that combines observation-bounded resident judgments with full-context weights for current needs. We use this signal to select SFT and RL tasks, and to provide a soft reward for GRPO. MUSE-Align improves Qwen3-4B from 26.63% to 31.35% under SFT and from 20.39% to 38.56% under GRPO. These results provide a benchmark and training framework for developing LLM agents that act appropriately in shared homes.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
smart home, large language model, finetuning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Wn8COpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MUSE-Home, an executable benchmark for LLM agents in smart homes shared by multiple residents, covering four reasoning categories: multi-user conflict, socially conditioned proactivity, cross-resident environment forensics, and robustness against unnecessary coordination. It contains 6,790 training instances and a 721-instance human-audited out-of-distribution test set, on which the best model reaches 45.35% task success. The authors also propose MUSE-Align, which combines observation-bounded resident judgments with full-context need weights for data selection and GRPO rewards, improving Qwen3-4B to 31.35% under SFT and 38.56% under GRPO.

Critical strengths
  1. A useful executable benchmark for an underexplored setting. The benchmark targets multi-user decisions that existing single-user smart-home benchmarks do not cover. This gives the community a chance to think the capability gap.

  2. Evidence that multi-user context is essential. Removing resident memory and household history lowers performance by 24.13 points on average (Table 2), showing the tasks cannot be reduced to mapping the latest request to a tool call.

Critical weaknesses
  1. Social Validity of the Success Criteria

1.1 Predefined target states may exclude reasonable alternatives. The paper allows multiple valid interaction paths (p. 4), but multiple paths to the same target state do not imply coverage of multiple reasonable solutions. In the MC example of Figure 1, lowering the TV volume to 0.15 is marked correct and turning off the TV is marked wrong. Would the checker accept a trajectory in which the agent suggests that Viktor use headphones or watch in another room? Similarly, the RB example marks a compromise of 22 to 24°C as wrong, indicating that the range of acceptable solutions is fixed in advance.

1.2 Authorization and negotiation are not evaluated. In the SP example, the agent stops the oven timer without any request, and this is labeled as correct behavior. In the MC example, the agent lowers the volume first and notifies Viktor afterward. Notification may satisfy transparency, but it does not constitute authorization. For non-urgent conflicts, appropriate behavior may involve asking or proposing, and residents may refuse, counter-propose, or withdraw consent. The paper does not specify whether residents respond to the agent's messages within an episode, nor how such responses would change the set of acceptable subsequent actions.

1.3 Reference decisions lack external validation. The scenario, reference decision, and success criteria are produced jointly by the same generator (Figure 2), and the human audit checks scenario plausibility, outcome validity, and non-triviality (p. 5). This audit clearly improves benchmark quality, but annotators assess internal consistency rather than whether residents would accept the intervention. What the paper validates is therefore whether models follow the household norms encoded by the benchmark, not whether those norms are endorsed by real residents. Since the scenarios targeted by this benchmark often admit many correct solutions and are inherently subjective, I believe evaluation data from real human users is necessary, rather than reliance on model-generated references.

  1. Justification for Satisfaction Aggregation

2.1 Equation (1) computes a weighted average of 1 to 4 satisfaction levels, which implicitly assumes that the levels are equally spaced and that scores are comparable across residents. Neither assumption is justified in the paper.

2.2 An average cannot protect individual minimum guarantees. When an important need of one resident is seriously harmed, high scores from other residents may mask the harm. It is unclear which minimum guarantees are enforced by the executable checker and which are handled only indirectly through the average.

2.3 The weights are not independent of the satisfaction scores. According to the description on p. 6, the assigner sees all resident scores and rationales when determining weights, and the weights vary across trajectories. Different trajectories for the same task may therefore be evaluated under different priority orderings, which is in tension with the claimed separation of local satisfaction from global prioritization.

  1. Methodological Contribution and Experimental Design

3.1 The contribution of the factorized design is not isolated. The components of MUSE-Align, namely perspective-restricted LLM scoring, full-context weighting, and weighted averaging, each have close counterparts in the LLM-as-a-Judge and multi-stakeholder evaluation literature, and GRPO itself is unchanged. The methodological novelty therefore lies mainly in the specific decomposition. Section 4.1 states that this design avoids two simple alternatives, a single global judge and an equal average, yet Table 3 only compares against the former. An equal-weight variant and a variant in which each resident judge receives the full context are needed to assess the contributions of observation bounding and need weighting separately.

3.2 Alignment is only performed on Qwen3-4B. At least one larger open-source model, for example Qwen3-8B, which is already evaluated in the paper, is needed to show whether the gains persist as base model capability increases.

3.3 A rejection sampling baseline is missing. The SFT targets of MUSE-Align are selected from trajectories sampled from Seed-2.0-Pro, Qwen3.8-Max, Kimi-K2.5, and GPT-5.5, while the SFT baseline uses the reference trajectories from data generation. The improvement from 26.63% to 31.35% therefore conflates distillation from stronger models with judge-based selection. A baseline that uses the same sampled trajectories, filtered only by the executable checker, would isolate the contribution of the judge.

3.4 Gains are unevenly distributed across categories. With GRPO, MUSE-Align raises RB from 24.49% to 83.67%, far above all API models (the best is 59.18%), while MC and EF improve by only about 8 to 10 points. Per-category sample sizes and their contributions to the overall gain should be reported, together with an analysis of whether the high RB score reflects a tendency to execute directly without coordination.

Questions for authors

I would ask the authors to primarily respond to the points raised in Critical Weaknesses, as these most strongly determine my assessment. Beyond those points, I have the following clarification questions:

  1. Would the checker accept reasonable alternatives to the encoded target state, such as suggesting headphones or another room in the MC example of Figure 1? Can residents respond to the agent's messages within an episode?

  2. What are the per-category sample sizes, and how much of the overall GRPO gain comes from RB?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Wn8C (1/2)

W1.1/Q1 The RB Example and Coverage of Reasonable Alternatives

Predefined target states may exclude reasonable alternatives, including in the MC and RB examples of Figure 1.

Would the checker accept suggestions such as headphones or another room?

There may be a misunderstanding of the RB example in Figure 1. The two requests do not represent a current conflict. Hiroshi's and Amara's requests occur one day apart, so the earlier request does not justify compromising the current one. Setting the AC to 22–24°C would introduce an unnecessary concession instead of following the current request for 27°C. This example tests temporal reasoning and resistance to unwarranted coordination.

Coverage of alternative solutions is a separate concern. Please see General Response 1, and an example in Appendix E.3. Specifically regarding headphones or another room, we will exclude the headphones, another actionable rooms, or other alternatives in the view of the agents in the first place during the dataset construction process.

W1.2/Q1 Authorization and Negotiation

Notification after an action does not constitute authorization, and residents may refuse, counter-propose, or withdraw consent.

Can residents respond within an episode?

We agree that, for non-urgent conflicts, notification after an action does not constitute authorization. The SP example involves an immediate safety risk, in which waiting for authorization could delay intervention. Non-urgent situations require separate consideration of consent.

The current benchmark focuses on understanding and reasoning about multi-resident needs and conflicts, rather than evaluating authorization and negotiation. As a result, this paper assumes a authorization-free setting to isolate the multi-resident reasoning capability. Meanwile, too offensive and impractical cases are filtered during human review.

Residents do not respond to the agent’s messages within an episode. These interactions are difficult to evaluate in a benchmark setting and deviate from the core evaluation objective.

W1.3 Human Validation and Real-Resident Acceptance

The human audit may establish internal consistency without showing that real residents would accept the reference interventions.

Please see General Response 2. The audit evaluates whether interventions are appropriate for the residents specified in household memory, beyond internal consistency. The cases in Appendices E.1 and E.2 show how human reviewers challenged the generator's assumptions and revised or rejected its decisions.

W2.1/W2.2 Justification of Satisfaction Judge

A weighted average of 1–4 satisfaction levels assumes equal spacing and comparability across residents.

We use the 1–4 scale and weighted aggregation as a practical training signal, which is a pragmatic and common pactice. The shared scoring rubric provides a common operational scale, while the weights represent the situational importance of current needs.

We agree that an average alone cannot enforce individual minimum guarantees. Technically, the scroes and weights combined can achieve such guarantee.

Table 3 provides empirical support for judge, where removing the judge reward reduces task success from 38.56% to 34.67%. The human-agreement evaluation in General Response 2 separately examines its consistency with human judgments.

W2.3 Functional Separation and Trajectory-Dependent Weights

The weights are not independent of the satisfaction scores.

Thank you for pointing this out, we have change the expression from "separation of local satisfaction from global prioritization" to "two-stage treatment of local satisfaction and global prioritization". We empirically find that offering more information (e.g., judge scores and following trajectories of the agents) can improve the judge quality as shown in the Table below. Ablation in Table 3 still shows that the two-step-design significantly improves performance.

Judge modelSatisfaction MAE ↓Satisfaction weighted κ ↑Need-importance τ-b ↑
original0.16310.63940.8951
w/o scores0.17700.58090.8949
w/o scores & trajs0.20440.42860.8138
AuthorsOpenReview ↗

Response to Reviewer Wn8C (2/2)

W3.1 Isolating Observation Bounding and Need Weighting

An equal-weight variant and a variant in which each resident judge receives the full context are needed.

We have added an ablation in Table 3. Results show an equal judge improves average accuracy from 20.39 of vanilla GRPO to 24.97, but still falls far behind 38.56 of MUSE-Align.

W3.2 Alignment on a Larger Base Model

Does the improvement persist on a larger model such as Qwen3-8B?

Due to resource constraints, we are unable to train Qwen3-8B, and we apologize for this limitation. Our experiments show that the original Qwen3-8B and Qwen3-4B perform similarly (19.14 vs. 17.48), while our aligned Qwen3-4B achieves 38.56, already substantially surpassing the original Qwen3-8B. This demonstrates that the gains are not merely due to base model scale and can exceed the original model’s capability.

W3.3 Rejection-Sampling Baseline

A checker-only selection baseline using the same sampled trajectories is needed to isolate judge-based selection from stronger-model distillation.

We collect successful trajectories from sampled rollouts of GPT-5.5, Doubao-2.0-Pro, Kimi-K2.5 and Qwen-3.6-Plus. Overall, we have 4,861 out of 6,790 tasks to train on. For every task which has more than one successful trajectories, we prefer the shortest one in steps.

Training on the filtered tasks, Qwen3-4B SFT achieves 29.54, outperforming 26.63 from the vanilla SFT baseline, but still falls behind 31.35 from MUSE-Align filtered SFT and 38.56 from MUSE-Align GRPO.

W3.4/Q2 Category Sizes and RB's Contribution to the Overall Gain

What are the per-category sample sizes, and how much of the overall GRPO gain comes from RB?

Table 6 reports the category sizes. Combining these counts with the GRPO success rates in Table 1 gives the following breakdown:

Category# Train# TestMUSE-Align GRPOContribution to overall gain
MC1,68622329.15+2.64
SP1,68317726.55+4.30
EF1,64822337.67+3.19
RB1,7739883.67+8.04
Total6,79072138.56+18.17

Each category's contribution is its success-rate gain weighted by its share of the 721 test episodes. The RB improvement is substantial, but the majority of the additional successes occur outside RB.

Reviewer_3EzqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets benchmarking LLM models for multi-user smart home tasks. It systematically builds a new benchmark, MUSE-Home, supporting four types of tasks specific to multi-user scenarios, and then presents MUSE-Align, which provides a new data selection and reward design for SFT/GRPO to improve LLM performance on the benchmark, leveraging judgments from individual users and their importance. In the experiments, the paper evaluates multiple mainstream models on MUSE-Home and verifies the effectiveness of MUSE-Align.

Critical strengths
  • Idenities multi-user-specific challenges and builds corresponding benchmark tasks.
  • Proposes MUSE-Align beyond model evaluation.
Critical weaknesses
  • Insufficient technical contributions.
  • More probing experiments are needed.

Please see questions for details.

Questions for authors
  1. Please highlight the technical novelty in building MUSE-Home and designing MUSE-Align.
  2. Since the best evaluated model achieves only 45.35% task success, a detailed failure analysis would be helpful for understanding where and why current models fail.
  3. What drives the improvements in SFT/GRPO, more reasonable importance weighting or other factors?
  4. The household assigner relies on LLM-generated weights without human annotations. How is their reliability verified?
  5. Could the authors discuss potential biases in synthetic multi-user scenarios and how to handle the subjectivity of real users?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 3Ezq

W1/Q1 Technical Novelty and Contributions

Please highlight the technical novelty in building MUSE-Home and designing MUSE-Align.

Our contributions are an executable evaluation of multi-user smart-home decisions and a supervision framework tailored to the resulting alignment problem.

MUSE-Home. To our knowledge, this is the first benchmark centered on end-to-end multi-user smart-home decisions, where evidence, consequences, and interaction requirements span residents. Beyond carefully design data construction pipeline, two annotators independently review all test candidates. The human quality control requires more than 0.3 person-hours per released test task on average. This provides a carefully validated evaluation resource covering four distinct multi-user reasoning challenges.

MUSE-Align. The methodological contribution is the decomposition of multi-user supervision into observation-bounded resident satisfaction and full-context weighting of current needs. This introduces a simple but effective method for multi-user scenarios with only LLM judge shaping. The methodology is proved highly effective in both training data filtering and GRPO rewarding.

W2/Q2 Failure Analysis

Since the best evaluated model achieves only 45.35% task success, a detailed failure analysis would be helpful for understanding where and why current models fail.

To further analyze model failures, we categorize unsuccessful cases into state errors (unmatch device state) and interaction errors (unmatch resident interaction). State errors are further divided into incomplete target-state satisfaction and unintended state changes. Since a single case may exhibit multiple error types simultaneously, the percentages are not mutually exclusive and may sum to more than 100%.

ModelState ErrorsPartial State MatchUnintended State ChangesApp Interaction Mismatch
Seed-2.0-Pro69.0%58.1%14.7%26.9%
DeepSeek-V467.2%51.3%12.2%33.5%
GPT-5.565.5%54.8%25.9%21.9%
Qwen3-4B MUSE-Align SFT83.4%65.9%27.1%20.3%
Qwen3-4B MUSE-Align RL84.4%69.8%13.3%18.6%

The results show that models make both device-operation and user-interaction errors, with device-operation errors being substantially more prevalent. Most operation failures arise from mismatched target states or omitted device controls, while a smaller proportion results from unnecessary operations that cause unintended state changes.

Q3 What Drives the SFT and GRPO Improvements?

Are the improvements driven by more reasonable importance weighting or other factors?

Table 3 shows that both factorized data selection the factorized judge reward contribute to the GRPO improvement. Relative to the full method, replacing factorized filtering leads to a major performance drop, while the soft reward provides an additional benefit. These ablations support the combined supervision design.

Q4 Reliability of LLM-Generated Need Weights

The household assigner relies on LLM-generated weights without human annotations. How is their reliability verified?

Please see General Response 2. The downstream results provide direct evidence of the supervision's practical utility (Table 1), and we also directly evaluate human agreement.

Q5 Synthetic Biases and Subjective User Preferences

Could the authors discuss potential biases in synthetic multi-user scenarios and how to handle the subjectivity of real users?

Please see General Response 1. We ground scenarios in resident-specific preferences and circumstances, and human annotators assess interventions from those residents' perspectives.

Reviewer_3EzqOpenReview ↗

Response to Authors

I thank the authors for the clarification on each point and the additional experimental results. My concerns on Q3 and Q4 are fully addressed.

For W1/Q1, I agree with the contributions of the multi-user benchmark design and MUSE-Align as a simple but effective method. However, for the technical contribution, MUSE-Align relies on a straightforward strategy and has limited new design on the training algorithm.

For W2/Q2, the failure analysis is useful and shows where models fail. However, I still have concerns about the coverage of the failure categories, as partial state match may result from incorrect target device selection, failure to resolve multi-user preference conflicts, incomplete execution of required actions, or omissions in complex tasks. The current experimental design does not distinguish these cases, which is important for understanding failures and how to improve model performance. Nevertheless, the current analysis still provides some useful insights.

For Q5, I understand that, for a benchmark, the authors try to ensure a unique reference decision through human filtering for verifiable evaluation. However, this may exclude real-world scenarios where multiple reasonable solutions exist, as multi-user decisions do not necessarily have a unique correct solution. This could itself introduce a bias.

Overall, my concerns are partially addressed, but I still have some concerns about the overall concept of the paper.

AuthorsOpenReview ↗

Follow-Up Response

Thank you for your follow-up comments. We really appreciate your efforts in helping us improve our paper.

W1/Q1

First of all, our main contribution is MUSE-Home, the first benchmark focused on multi-user decision-making in smart homes. We believe it provides a useful resource for the community.

For MUSE-Align, we acknowledge that it does not propose a fundamentally new training algorithm. Its contribution lies in adapting data filtering and rewarding design to multi-user scenarios, enabling a 4B model to outperform larger models such as GPT-4o and GLM-5.3 on our benchmark. We believe the pragmatic methodology and strong results offer useful insights for related tasks.

W2/Q2

1. Analysis in main paper

To "understanding failures and how to improve model performance", the main category-wise results already provide some indications. Full table is shown in Table 1.

ModelMulti-user Conflict (MC)Proactivity (PI)Environment Forensics (EF)Robustness (RB)
API-accessed Large Models (Avg.)29.49%32.48%50.11%54.21%
Qwen3-4B MUSE-Align GRPO29.15%26.55%37.67%83.67%

Averaged across the eight API-accessed large models, success rates are substantially lower on multi-user conflict and proactivity than on environment forensics, indicating that the limitation lies more in action boundary rather than complex multi-step information gathering. Moreover, MUSE-Align GRPO significantly improves robustness, showing that using GRPO to improve appropriateness is promising.

2. New fine-grained analysis

We offer a new analysis with finer granulariy compared to that in the first response. It compares the sets of devices targeted by reference and model actions, excluding read-only queries and human interactions:

ModelAdditional Devices OnlyOmitted Devices OnlyBoth Additional and OmittedSame Device Set
Seed-2.0-Pro22.3%21.6%12.4%43.7%
DeepSeek-V4 Flash17.2%29.4%11.9%41.5%
GPT-5.536.4%14.7%12.4%36.6%

The denominator is each model’s unsuccessful trajectories. This breakdown reveals different behavioral patterns: GPT-5.5 more frequently targets additional devices (over-action), while DeepSeek more frequently omits reference devices (under-action). Nevertheless, 36.6–43.7% of unsuccessful trajectories target the same device set as the reference, showing that device-set differences alone cannot explain all failures.

In conclusion, we do not claim a single dominant semantic cause or definitive prevalence estimates from this small inspection. Rather, the expanded analysis identifies limitations across multi-user task categories, device selection, and execution, with model-specific patterns.

Q5

We appreciate your point. We agree that many real-world multi-user scenarios admit multiple reasonable solutions, and that enforcing a unique reference decision can exclude such cases. This is a real trade-off. Our goal with MUSE-Home is not to cover the full space of real-world multi-user behavior, but to provide a verifiable and executable benchmark subset.

During annotation, we found these scenarios to be subtle. Small changes in background information can change the desired behavior, while omitting such information can make a sample unsolvable or admit multiple valid answers. We therefore invested substantial human effort (on average 0.3 person-hours per sample) to filter and revise them. Specifically, we removed 654 samples, revised 113, and retained 721.

We thus position MUSE-Home as an executable and benchmarkable subset of real-world multi-user smart-home scenarios. While ambiguous and unresolvable cases are excluded, the retained executable cases remain challenging for current models. We believe this is a useful early step to this problem.


Thank you for your diligent work once again. We hope our response has addressed your remaining concerns, and might help you reassess the value of our paper.

Reviewer_9g9sOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper recasts smart-home assistance as a multi-principal decision problem: one action propagates to several co-residents whose needs, permissions, routines, and observations differ, and the correct behavior sometimes consists of not acting at all. The authors release MUSE-Home — 7,511 executable episodes (6,790 train / 721 test) over 24 floor plans, with a house-level split and a fully human-audited test set — organized into four reasoning categories (multi-user conflict, socially conditioned proactivity, cross-resident environment forensics, robustness against unnecessary coordination). They further propose MUSE-Align, which decomposes multi-user feedback into (i) observation-bounded per-resident satisfaction scores and (ii) full-context need-importance weights assigned by a household-level assigner, aggregating them into a single score used for SFT data selection, RL task selection, and a soft GRPO reward mixed with a binary task-success reward.

Critical strengths
  1. The reframing is itself a contribution — evidence, consequences, and interaction requirements all span residents, which is novel and important.
  2. disciplined test construction : two independent annotators over all 1,375 candidates, removal reasons and counts disclosed, cost reported, full prompt templates released.
Critical weaknesses
  1. Although multi-user co-residence is the paper's central claim, the benchmark provides no analysis of how performance scales with the number of residents or the volume of associated context : Table 2 ablates only at the level of information source (memory / history / both), removing each source wholesale for all residents at once; there is no per-resident, per-content-type, or quantity-controlled condition, nor any scaling analysis over household size or context volume. Without these, the reported drops cannot distinguish genuine multi-user reasoning from the trivial effect of deleting task-relevant context, leaving the paper's central claim under-supported
Questions for authors
  1. What is the agreement between resident-judge scores and human ratings, and how much does the result move with a different judge model?
  2. GRPO+MUSE-Align reaches 83.67% on RB, far above all API models. It is unclear whether this reflects a genuine ability to resist unnecessary coordination or a learned shortcut of defaulting to inaction in multi-resident scenes. Since RB rewards not intervening, a policy that simply under-acts could achieve high RB scores at the cost of other categories. Please provide a difficulty-stratified analysis within RB (e.g., episodes where inaction is correct vs. incorrect) or failure cases showing that the model actually distinguishes warranted from unwarranted coordination.
3 comments
AuthorsOpenReview ↗

Response to Reviewer 9g9s

W1 Scaling with Household Size and Context Volume

How does performance scale with the number of residents or the volume of associated context?

We evaluate task success stratified by the number of residents, rooms, and devices. The results have been included in the appendix.

ModelOverallUsers <=2Users >2Rooms <=10Rooms >10Devices <=100Devices >100
Seed-2.0-Pro45.3550.1041.7945.1046.3145.7345.21
DeepSeek-V443.2748.4637.2044.0640.2743.2243.30
GPT-5.540.5045.2037.2038.8146.9839.2041.00
Qwen3-4B MUSE-Align SFT31.3536.8623.4432.1728.1938.6928.54
Qwen3-4B MUSE-Align RL38.5643.8931.7039.5134.9044.2236.40

The results show that the resident scale does influence the model performance, while tasks of relative small sclae in MUSE-Home also pose challenges to LLMs. Larger household scale influences smaller models, but poses little additional challenges to API-accessed large models.

Q1 Agreement with Human Ratings and Sensitivity to Judge Models

What is the agreement between resident-judge scores and human ratings, and how much does the result move with a different judge model?

Please see General Response 2, which describes the human evaluation of satisfaction scores and need-importance rankings, together with the comparison across judge models.

Q2 Does the RB Improvement Reflect an Inaction Shortcut?

Does the high RB score reflect a learned shortcut of defaulting to inaction?

RB does not uniformly reward inaction. It tests whether the agent avoids unnecessary coordination when the available evidence indicates no actual conflict. In Figure 1, the two temperature requests occur on different days. The correct response is to execute the current request and set the AC to 27°C. Doing nothing or imposing a 22–24°C compromise would fail. Such cases test whether an agent introduces unwarranted coordination merely because multiple residents appear in the context.

The improvements also extend beyond RB. Table 1 reports the following GRPO results:

Category# Train# TestMUSE-Align GRPOContribution to overall gain
MC1,68622329.15+2.64
SP1,68317726.55+4.30
EF1,64822337.67+3.19
RB1,7739883.67+8.04
Total6,79072138.56+18.17

These gains can dispel specific concern that RB improves at the expense of the other categories through generalized under-action. They support improved task-dependent behavior. We have clarified the distinction between avoiding unnecessary coordination and avoiding action in the main results.

Thanks authors for the rebuttal. The revision addresses most of my concerns, and I maintain my weak accept rating.

AuthorsOpenReview ↗

Follow-Up Response

We really appreciate your positive feedback and efforts. Thank you for your response.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response (1/2)

We sincerely thank you all for your thoughtful comments and constructive suggestions. We appreciate your recognition of the importance of multi-user smart-home decision-making, the value of an executable benchmark for this setting, and the quality ensurance with extensive human review of MUSE-Home.

We appreciate your patience regarding the timing of our response, as we have devoted considerable time and effort to extensive manual verification, careful manuscript revision, and additional experiments so as to address the concerns as thoroughly as possible. We address all the concerns, and accordingly revise the paper. The revisions include the following updates:

  • Section 1 Introducetion: Adjust the expression from "separation" to "two-stage treatment".
  • Section 3.3 Human quality control: clarify that annotators assess decisions from the perspectives of the residents described in household memory, rather than using their own preferences.
  • Section 5.2 Main results: clarify that RB evaluates resistance to unnecessary coordination, which can require direct execution rather than inaction, and report its contribution to the overall improvement.
  • Section 5.4 Methodology component ablation: add new ablation experimental results, isolating the improvement from weighting over equal weights.
  • Appendix E: add concrete human-audit case examples.
  • Appendix F: add additional experiment results of fine-grained judge and performance analysis.

We address the shared concerns in the general response and provide point-by-point responses below.

AuthorsOpenReview ↗

General Response (2/2)

GR1 Potential Biases and Alternatives in MUSE-Home (@Wn8C, @3Ezq)

We minimize the potential biases and multiple alternative trajectories throughout the benchark construction, by resident-specific scenario generation and independent human review.

Resident-specific generation. Scenarios are conditioned on household memory, which specifies each resident's preferences, routines, health and privacy constraints, and interpersonal relations. This grounds decisions in the residents' stated circumstances rather than generic assumptions about acceptable household behavior.

Human review from resident perspectives. Annotators are instructed to adopt the perspectives of the residents described in that memory and assess the intended intervention against their needs and circumstances, rather than the annotators' own preferences. The given background must support and only support the reference decision. The audit can therefore revise or reject the generator's decisions, to ensure grounded an answer and exclude reasonable alternatives. Examples illustrate this process:

  • In living_room_tv_privacy_switch_during_guest_visit (P.PL.1), the reference decision interrupted a portfolio display for privacy protection despite the resident explicitly requesting it for a client meeting. The decision, trajectory, and success criteria were revised to preserve the display (Appendix E.1).
  • In bathroom_main_water_heater_kept_off_for_late_night_wash (E.MC.4), the reference decision denied a hot-water request to maintain an energy-saving schedule, assuming alternatives would suffice without evidence of the resident's acceptance. This sample was removed (Appendix E.2).
  • In direct_execution_leave_home_climate (E.RB.3), the reference decision allowed either activating away mode or maintaining cooling at 26°C, but the checker accepted only away mode. The sample was flagged for overly narrow criteria and removed (Appendix E.3).

This process reduces unsupported assumptions and other alternatives in synthetic references.

GR2 Reliability of LLM-Based Satisfaction Scoring and Need Weighting (@9g9s, @3Ezq)

The reliability of satisfaction scores and need weights are ensured by the following three aspects.

1. Downstream effectiveness. MUSE-Align improves test success from 26.63% to 31.35% under SFT and from 20.39% to 38.56% under GRPO (Table 1). These gains establish the practical value of the combined supervision.

2. Agreement with human judgments. We evaluate 200 randomly sampled task–trajectory pairs (with 764 persons to judge in total), with annotators working independently and without access to model outputs. We assess the two components according to their distinct roles:

  • Satisfaction scoring: annotators receive the same observation-bounded view as each resident judge and rate need satisfaction on the same 1–4 scale. We measure agreement using mean absolute error (MAE) and linearly weighted Cohen's κ.
  • Need weighting: annotators receive the full context available to the household assigner and weight residents' current needs by importance. We measure agreement between human rankings and model-induced rankings using mean Kendall's τ-b.

3. Sensitivity to the judge model. We compare the original GPT-5.5 judge with Seed-2.0-Pro and DeepSeek-V4 on the same samples using the same protocol. This comparison assesses how human agreement changes when the judge is replaced.

Judge modelSatisfaction MAE ↓Satisfaction weighted κ ↑Need-importance τ-b ↑
GPT-5.5 (original)0.16310.63940.8951
DeepSeek-V40.14890.65560.8479
Seed-2.0-Pro0.19860.45780.9031

In conclusion, the existing results already support MUSE-Align as an effective training signal, and the additional human evaluation directly shows its satisfaction scores and need rankings reflect human judgments, directly addressing your reliability concerns.

AuthorsOpenReview ↗

Looking forward to your feedback

Dear reviewers,

We would like to once again thank you for your time and effort devoted to reviewing our paper. Accordingly, after conducting additional experiments and revising the manuscript, we have posted concise but informative responses addressing all the concerning points.

As the discussion period is approaching its end, we kindly wish to check whether our responses have dispelled your concerns. If any questions remain, we would greatly appreciate hearing from you at your earliest convenience, so that we can address them promptly within the discussion window.

AuthorsOpenReview ↗

Summary of the author-reviewer discussion period

We would like to express our sincere gratitude to all reviewers for the time and effort you have devoted. Considering that the discussion period is approaching its end, we would like to provide a brief summary of the key points regarding the discussion.

  • Acknowledged strengths
    • The relavance and importance of the first multi-user smart home task framing (@9g9s, @3Ezq, @Wn8C).
    • Careful test construction with extensive quality control and disclosed details (@9g9s).
    • Methodology proposed in addition to the benchmark (@3Ezq).
    • Detailed analysis proving that the tasks cannot be reduced to mapping the latest request to a tool call (@Wn8C).
  • Dispelled concerns during the discussion period
    • The reliability of LLM satisfaction scoring and need weighting (@9g9s, @3Ezq).
    • More analysis on environment scaling (@9g9s), improvement attribution (@9g9s, @3Ezq, @Wn8C), and component ablation (@Wn8C).
    • Additional baseline (@Wn8C).
  • Remaining concerns up to now
    • RC.1 Methodology novelty and finer-grained failure analysis (@3Ezq).
    • RC.2 The authenticity of the benchmark (@3Ezq, @Wn8C).

In response to RC.1, we have added new analysis in finer granularity.

In response to RC.2, we acknowledge that there exists a gap between our benchmark and real-world scenarios, but argue that we provide an executable and benchmarkable subset that preserves the core challenges of multi-user decision-making without relying on expensive or unstable external environments.

We sincerely hope that the clarifications and revisions we have provided adequately address the reviewers’ concerns. We are grateful for the thoughtful feedback and constructive suggestions throughout this process.

Thank you once again for your careful review of our manuscript. If you have any questions during the end of the discussion phase, please feel free to comment anytime.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.