acceptodds
Under review as a conference paper at ICLR 2027

Counterfactuals, Proactivity, and Personalization: Personal HomeBench for Evaluating Smart Home Agents

Abstract

We introduce PERSONALHOMEBENCH (PHB), a benchmark for personalized smart-home agents under partial observability, comprising 9168 reactive and proactive instances grounded in multi-resident personas, long-term histories, evolving household state, heterogeneous appliances, and real-home multimodal observations. We further introduce PERSONALHOMETOOLS, a unified toolbox for context retrieval, memory access, multimodal perception, and appliance interaction. PHB evaluates models under Sole-Reasoning and Agentic settings, separating reasoning over available context from actively acquiring and integrating personalized information. Across models, we observe a pronounced reasoning-to-interaction gap, including reduced counterfactual robustness and a mismatch between plausible personalized plans and valid executable actions. Failures arise primarily from incomplete context acquisition, tool hallucination, and execution errors, while gains from scaling, explicit reasoning, and multimodal perception remain task-dependent. Overall, PHB provides a unified setting to probe how personalization, reasoning, perception, and tool use come together in realistic smart-home agents.

Primary area:
datasets and benchmarks
Keywords:
personalization, personalized agents, multi-modal agents, agent evaluation, grounded agents, tool use, function calling, planning in agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hQ2KOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PersonalHomeBench to study personalized smart-home agents that do not have access to all household information at once. It also introduces PersonalHomeTools, a toolbox for retrieving context, accessing memory, perceiving multimodal input, and interacting with appliances. The results show a clear gap between sole-reasoning and agentic settings: models often fail to acquire the right context, use tools incorrectly, or produce plans that look reasonable but are not executable.

Critical strengths
  • The paper provides a fairly complete evaluation of personalized smart-home agents. The dataset is large, with many personas, devices/APIs, and tasks.
  • The paper gives a detailed analysis of tool errors, trajectories, and plan validity, and it validates the Role-Playing Judge with human annotations.
  • The paper includes many reproducibility details and data analyses.
Critical weaknesses
  • The evaluated models are outdated. Among proprietary models, the newest ones are only GPT-5 and Gemini 2.5 Pro. The paper should evaluate newer models such as GPT-6, GPT-5.6, Gemini 3.8 Flash, Claude Opus/Fable, and so on. The same applies to open-source models and multimodal models.
  • Most of the data are model-generated. The real-home videos contain only 100 scenarios, and the safety scenarios are scripted simulations, so the external validity is limited.
  • The proactive tasks fix the intervention timing in advance, so they do not actually test when the agent should intervene.
Questions for authors
  • How would the RPJ score and plan validity change if the agent were allowed to decide when to intervene?
1 comment

Thank you for the rebuttal. However, the additional results of GPT-5.6 still cannot show the performance of latest frontier models. GPT-5.6 is released in July, even two months before the submission deadline. There have been dozens of new models (closed-source, open-source, multimodal) since then.

Reviewer_hx1BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces PersonalHomeBench, a benchmark of 9168 personalized smart-home tasks across 1100 households, including 100 grounded in real-home videos. Five task types cover reactive reasoning and proactive assistance. Models are evaluated with full context and through tool-mediated acquisition. The paper also introduces PersonalHomeTools and a role-playing judge (RPJ) for personalized plan quality.

Critical strengths
  1. Well-motivated benchmark settings
  • Combining resident profiles, memories, household context and device capabilities captures useful challenges in personalized smart-home assistance.
  1. Useful diagnostic evaluation
  • The Sole-Reasoning and Agentic settings expose substantial differences between answering with supplied context and acquiring information through tools.
Critical weaknesses
  1. Benchmark construction and validation remain insufficiently established
  • The manual-review subset is unquantified , and Table 9, which appears to be the cited result, reports generic MC accuracies of 84–94%, which do not support the overall >99% solvability claim.
  • Long-term memory generation and proactive-task validation remain unclear. Video collection is described inconsistently as unprompted versus scenario-directed and coached.
  1. The evaluation limits interpretation of the main findings
  • The Agentic setting adds context retrieval, interface discovery, tool-call formatting, and budget constraints simultaneously. The comparison therefore does not isolate which factors cause the lower scores.
  • RPJ's human validation shows annotators its scores and rationales, risking anchoring. Its sample size is inconsistently reported.
  • Intervention points are fixed in advance, so the benchmark evaluates what assistance to provide, rather than whether or when to intervene.
Questions for authors
  1. Can you report the size and the error rate of the manual label audit, including validation of proactive tasks?
  2. Does the reasoning-to-interaction gap persist with device interfaces supplied, stronger structured tool calling, and a larger interaction budget?
  3. Can you validate RPJ against independent human ratings collected without showing its scores and rationales?
1 comment

Thanks for the response. The blind RPJ study addresses my concern. However, without an experiment that provides the device APIs upfront, it is unclear whether the plan-validity drop reflects API discovery rather than personalization, so I am keeping my score.

Reviewer_4i3rOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work introduces PersonalHomeBench, a benchmark for personalized smart-home agents in multi-resident households. It generates 1100 households with structured personas, long-term memories, appliances, and household state, derives 9168 reactive and proactive task instances, and provides PersonalHomeTools for retrieving context and controlling devices.

Critical strengths

I think the problem is worth studying. Assistance in a shared home depends on resident histories, current household state, and device capabilities at the same time, and existing benchmarks test these pieces separately. The benchmark is large, the tool layer is useful on its own, and the evaluation covers a reasonable range of models in both settings.

Critical weaknesses
  1. The first major concern is whether the tasks test what the title and the introduction claim.

1.1. The central claim of the finding is about the “agentic-realization gap”. However, I do not find this new or surprising. Models doing worse when context must be retrieved under a 15-turn budget than when it is pasted in is the expected outcome of the design. The interesting part is the magnitude: PG validity drops from 0.87/0.84/0.79 to 0.02/0.02/0.03 for GPT-4, GPT-4o, and Gemini, while GPT-5 keeps 0.68. But Section 4.2 attributes much of this to reasoning traces leaking into JSON, tool hallucination, and exhausted budgets, and nothing separates these harness effects from the capability the benchmark means to measure.

1.2. The paper says that conditioning on “persona, long-term memories, and household state” introduces reasoning demands that generic benchmarks do not capture (line 375), but I could not find evidence that the instances depend on long-term memory. Tasks are generated from “a compact set of contextually relevant memories” (line 186) and no instance is annotated with the evidence it needs. The personalization test in Table 3 replaces names with IDs, which measures entity grounding rather than memory. The task examples in Appendix J.1 (“What is the woman wearing on her left foot for support?”) need no memory at all; only the example in Fig. 1a does. Without specified tasks, many instances may not need long-term memory, and then the benchmark cannot support the claim. The same question applies to counterfactuals, which lead the title. CF is one of five tasks, built by single-factor perturbations of an IG question, with easy counterfactuals defined as “superficial or irrelevant change; answer remains unchanged” (Appendix H.2). This is an invariance test on retrieval QA rather than counterfactual reasoning, it plays no role in the proactive tasks or the tool layer, and the authors’ own analysis in Appendix E.4 attributes the CF drop under tools to retrieval “rather than counterfactual reasoning itself."

  1. Another major concern is about the validation of the proposed work.

2.1. The data validation is not well described. Line 183 says memories “are validated for temporal and logical consistency ... using a combination of manual review and LLM-based validation,” but the paper never defines either kind of consistency and gives no procedure, prompt, or statistics (how many memories were generated and rejected, how large the manual sample was, how the manual and LLM judgments agree). Figure 3 shows no validation stage, and Appendix H has prompts for persona and task generation but none for memory generation or validation. The same holds for the manual review of task instances at line 224. The 99% solvability figure is measured on generic questions with full context, so it validates the generic QA pairs, not the personalized or counterfactual instances the benchmark is about.

2.2. The task set is not well motivated. The five tasks are called “complementary” and loosely mapped to the three questions at the start of Section 4, but the paper does not motivate why this decomposition captures personalized household intelligence, which task exercises which ingredient (multi-resident, long-term memory, evolving state, heterogeneous appliances), or what is left out. MC is IG in a constrained format. FR asks the model to rank ten LLM-selected features against Gemini’s own ranking by MAP@1; if the inference candidate list is these ten features, chance MAP@1 is 0.30, and the Sole-Reasoning scores in Table 2 can be at chance.

2.3. “Project researchers subsequently annotate a guidance subset” (line 239). Who are the annotators, how many trajectories, and how well do they agree? More importantly, can a researcher judge what a particular resident would want? The RPJ validation has four annotators judging from written persona profiles whether they agree with the judge’s scores, so the humans are role-playing just as the LLM is, and the persona-level agreement is lower than the household-level one. The main text says each annotator evaluates 100 plans; footnote 3 says 50. I would like to see a discussion of whether annotators should resemble the personas (age, role, caregiver status), or a small study with real residents.

Questions for authors

(1.1) How much of the Agentic drop comes from harness effects (JSON format errors, hallucinated tools, the 15-turn budget) rather than model capability, and what remains of the gap if the API inventory is given up front or the budget is raised? (1.2) What evidence shows that the instances actually require long-term memory and counterfactual reasoning, rather than entity grounding and retrieval? For example, what fraction of instances need a memory to answer, and does accuracy change when the memory block is removed? (2.1) How was the data validated: what do temporal and logical consistency mean for memories, what procedure and prompts were used, how large was the manual sample, and what were the rejection rates and agreement? (2.2) Why these five tasks, and what does each contribute? In particular, what does MC add beyond IG, and how many candidates does the FR model rank at inference (and thus what is chance-level MAP@1)? (2.3) Who annotated the intervention points and RPJ judgments, how many and with what agreement, and should annotators resemble the personas or be real residents to judge personalized preferences?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.