- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces MUSE-Home, an executable benchmark for LLM agents in smart homes shared by multiple residents, covering four reasoning categories: multi-user conflict, socially conditioned proactivity, cross-resident environment forensics, and robustness against unnecessary coordination. It contains 6,790 training instances and a 721-instance human-audited out-of-distribution test set, on which the best model reaches 45.35% task success. The authors also propose MUSE-Align, which combines observation-bounded resident judgments with full-context need weights for data selection and GRPO rewards, improving Qwen3-4B to 31.35% under SFT and 38.56% under GRPO.
Critical strengths
-
A useful executable benchmark for an underexplored setting. The benchmark targets multi-user decisions that existing single-user smart-home benchmarks do not cover. This gives the community a chance to think the capability gap.
-
Evidence that multi-user context is essential. Removing resident memory and household history lowers performance by 24.13 points on average (Table 2), showing the tasks cannot be reduced to mapping the latest request to a tool call.
Critical weaknesses
- Social Validity of the Success Criteria
1.1 Predefined target states may exclude reasonable alternatives. The paper allows multiple valid interaction paths (p. 4), but multiple paths to the same target state do not imply coverage of multiple reasonable solutions. In the MC example of Figure 1, lowering the TV volume to 0.15 is marked correct and turning off the TV is marked wrong. Would the checker accept a trajectory in which the agent suggests that Viktor use headphones or watch in another room? Similarly, the RB example marks a compromise of 22 to 24°C as wrong, indicating that the range of acceptable solutions is fixed in advance.
1.2 Authorization and negotiation are not evaluated. In the SP example, the agent stops the oven timer without any request, and this is labeled as correct behavior. In the MC example, the agent lowers the volume first and notifies Viktor afterward. Notification may satisfy transparency, but it does not constitute authorization. For non-urgent conflicts, appropriate behavior may involve asking or proposing, and residents may refuse, counter-propose, or withdraw consent. The paper does not specify whether residents respond to the agent's messages within an episode, nor how such responses would change the set of acceptable subsequent actions.
1.3 Reference decisions lack external validation. The scenario, reference decision, and success criteria are produced jointly by the same generator (Figure 2), and the human audit checks scenario plausibility, outcome validity, and non-triviality (p. 5). This audit clearly improves benchmark quality, but annotators assess internal consistency rather than whether residents would accept the intervention. What the paper validates is therefore whether models follow the household norms encoded by the benchmark, not whether those norms are endorsed by real residents. Since the scenarios targeted by this benchmark often admit many correct solutions and are inherently subjective, I believe evaluation data from real human users is necessary, rather than reliance on model-generated references.
- Justification for Satisfaction Aggregation
2.1 Equation (1) computes a weighted average of 1 to 4 satisfaction levels, which implicitly assumes that the levels are equally spaced and that scores are comparable across residents. Neither assumption is justified in the paper.
2.2 An average cannot protect individual minimum guarantees. When an important need of one resident is seriously harmed, high scores from other residents may mask the harm. It is unclear which minimum guarantees are enforced by the executable checker and which are handled only indirectly through the average.
2.3 The weights are not independent of the satisfaction scores. According to the description on p. 6, the assigner sees all resident scores and rationales when determining weights, and the weights vary across trajectories. Different trajectories for the same task may therefore be evaluated under different priority orderings, which is in tension with the claimed separation of local satisfaction from global prioritization.
- Methodological Contribution and Experimental Design
3.1 The contribution of the factorized design is not isolated. The components of MUSE-Align, namely perspective-restricted LLM scoring, full-context weighting, and weighted averaging, each have close counterparts in the LLM-as-a-Judge and multi-stakeholder evaluation literature, and GRPO itself is unchanged. The methodological novelty therefore lies mainly in the specific decomposition. Section 4.1 states that this design avoids two simple alternatives, a single global judge and an equal average, yet Table 3 only compares against the former. An equal-weight variant and a variant in which each resident judge receives the full context are needed to assess the contributions of observation bounding and need weighting separately.
3.2 Alignment is only performed on Qwen3-4B. At least one larger open-source model, for example Qwen3-8B, which is already evaluated in the paper, is needed to show whether the gains persist as base model capability increases.
3.3 A rejection sampling baseline is missing. The SFT targets of MUSE-Align are selected from trajectories sampled from Seed-2.0-Pro, Qwen3.8-Max, Kimi-K2.5, and GPT-5.5, while the SFT baseline uses the reference trajectories from data generation. The improvement from 26.63% to 31.35% therefore conflates distillation from stronger models with judge-based selection. A baseline that uses the same sampled trajectories, filtered only by the executable checker, would isolate the contribution of the judge.
3.4 Gains are unevenly distributed across categories. With GRPO, MUSE-Align raises RB from 24.49% to 83.67%, far above all API models (the best is 59.18%), while MC and EF improve by only about 8 to 10 points. Per-category sample sizes and their contributions to the overall gain should be reported, together with an analysis of whether the high RB score reflects a tendency to execute directly without coordination.
Questions for authors
I would ask the authors to primarily respond to the points raised in Critical Weaknesses, as these most strongly determine my assessment. Beyond those points, I have the following clarification questions:
-
Would the checker accept reasonable alternatives to the encoded target state, such as suggesting headphones or another room in the MC example of Figure 1? Can residents respond to the agent's messages within an episode?
-
What are the per-category sample sizes, and how much of the overall GRPO gain comes from RB?
2 comments
Response to Reviewer Wn8C (2/2)
W3.1 Isolating Observation Bounding and Need Weighting
An equal-weight variant and a variant in which each resident judge receives the full context are needed.
We have added an ablation in Table 3. Results show an equal judge improves average accuracy from 20.39 of vanilla GRPO to 24.97, but still falls far behind 38.56 of MUSE-Align.
W3.2 Alignment on a Larger Base Model
Does the improvement persist on a larger model such as Qwen3-8B?
Due to resource constraints, we are unable to train Qwen3-8B, and we apologize for this limitation. Our experiments show that the original Qwen3-8B and Qwen3-4B perform similarly (19.14 vs. 17.48), while our aligned Qwen3-4B achieves 38.56, already substantially surpassing the original Qwen3-8B. This demonstrates that the gains are not merely due to base model scale and can exceed the original model’s capability.
W3.3 Rejection-Sampling Baseline
A checker-only selection baseline using the same sampled trajectories is needed to isolate judge-based selection from stronger-model distillation.
We collect successful trajectories from sampled rollouts of GPT-5.5, Doubao-2.0-Pro, Kimi-K2.5 and Qwen-3.6-Plus. Overall, we have 4,861 out of 6,790 tasks to train on. For every task which has more than one successful trajectories, we prefer the shortest one in steps.
Training on the filtered tasks, Qwen3-4B SFT achieves 29.54, outperforming 26.63 from the vanilla SFT baseline, but still falls behind 31.35 from MUSE-Align filtered SFT and 38.56 from MUSE-Align GRPO.
W3.4/Q2 Category Sizes and RB's Contribution to the Overall Gain
What are the per-category sample sizes, and how much of the overall GRPO gain comes from RB?
Table 6 reports the category sizes. Combining these counts with the GRPO success rates in Table 1 gives the following breakdown:
| Category | # Train | # Test | MUSE-Align GRPO | Contribution to overall gain |
|---|---|---|---|---|
| MC | 1,686 | 223 | 29.15 | +2.64 |
| SP | 1,683 | 177 | 26.55 | +4.30 |
| EF | 1,648 | 223 | 37.67 | +3.19 |
| RB | 1,773 | 98 | 83.67 | +8.04 |
| Total | 6,790 | 721 | 38.56 | +18.17 |
Each category's contribution is its success-rate gain weighted by its share of the 721 test episodes. The RB improvement is substantial, but the majority of the additional successes occur outside RB.
Response to Reviewer Wn8C (1/2)
W1.1/Q1 The RB Example and Coverage of Reasonable Alternatives
There may be a misunderstanding of the RB example in Figure 1. The two requests do not represent a current conflict. Hiroshi's and Amara's requests occur one day apart, so the earlier request does not justify compromising the current one. Setting the AC to 22–24°C would introduce an unnecessary concession instead of following the current request for 27°C. This example tests temporal reasoning and resistance to unwarranted coordination.
Coverage of alternative solutions is a separate concern. Please see General Response 1, and an example in Appendix E.3. Specifically regarding headphones or another room, we will exclude the headphones, another actionable rooms, or other alternatives in the view of the agents in the first place during the dataset construction process.
W1.2/Q1 Authorization and Negotiation
We agree that, for non-urgent conflicts, notification after an action does not constitute authorization. The SP example involves an immediate safety risk, in which waiting for authorization could delay intervention. Non-urgent situations require separate consideration of consent.
The current benchmark focuses on understanding and reasoning about multi-resident needs and conflicts, rather than evaluating authorization and negotiation. As a result, this paper assumes a authorization-free setting to isolate the multi-resident reasoning capability. Meanwile, too offensive and impractical cases are filtered during human review.
Residents do not respond to the agent’s messages within an episode. These interactions are difficult to evaluate in a benchmark setting and deviate from the core evaluation objective.
W1.3 Human Validation and Real-Resident Acceptance
Please see General Response 2. The audit evaluates whether interventions are appropriate for the residents specified in household memory, beyond internal consistency. The cases in Appendices E.1 and E.2 show how human reviewers challenged the generator's assumptions and revised or rejected its decisions.
W2.1/W2.2 Justification of Satisfaction Judge
We use the 1–4 scale and weighted aggregation as a practical training signal, which is a pragmatic and common pactice. The shared scoring rubric provides a common operational scale, while the weights represent the situational importance of current needs.
We agree that an average alone cannot enforce individual minimum guarantees. Technically, the scroes and weights combined can achieve such guarantee.
Table 3 provides empirical support for judge, where removing the judge reward reduces task success from 38.56% to 34.67%. The human-agreement evaluation in General Response 2 separately examines its consistency with human judgments.
W2.3 Functional Separation and Trajectory-Dependent Weights
Thank you for pointing this out, we have change the expression from "separation of local satisfaction from global prioritization" to "two-stage treatment of local satisfaction and global prioritization". We empirically find that offering more information (e.g., judge scores and following trajectories of the agents) can improve the judge quality as shown in the Table below. Ablation in Table 3 still shows that the two-step-design significantly improves performance.