- The first major concern is whether the tasks test what the title and the introduction claim.
1.1. The central claim of the finding is about the “agentic-realization gap”. However, I do not find this new or surprising. Models doing worse when context must be retrieved under a 15-turn budget than when it is pasted in is the expected outcome of the design. The interesting part is the magnitude: PG validity drops from 0.87/0.84/0.79 to 0.02/0.02/0.03 for GPT-4, GPT-4o, and Gemini, while GPT-5 keeps 0.68. But Section 4.2 attributes much of this to reasoning traces leaking into JSON, tool hallucination, and exhausted budgets, and nothing separates these harness effects from the capability the benchmark means to measure.
1.2. The paper says that conditioning on “persona, long-term memories, and household state” introduces reasoning demands that generic benchmarks do not capture (line 375), but I could not find evidence that the instances depend on long-term memory. Tasks are generated from “a compact set of contextually relevant memories” (line 186) and no instance is annotated with the evidence it needs. The personalization test in Table 3 replaces names with IDs, which measures entity grounding rather than memory. The task examples in Appendix J.1 (“What is the woman wearing on her left foot for support?”) need no memory at all; only the example in Fig. 1a does. Without specified tasks, many instances may not need long-term memory, and then the benchmark cannot support the claim. The same question applies to counterfactuals, which lead the title. CF is one of five tasks, built by single-factor perturbations of an IG question, with easy counterfactuals defined as “superficial or irrelevant change; answer remains unchanged” (Appendix H.2). This is an invariance test on retrieval QA rather than counterfactual reasoning, it plays no role in the proactive tasks or the tool layer, and the authors’ own analysis in Appendix E.4 attributes the CF drop under tools to retrieval “rather than counterfactual reasoning itself."
- Another major concern is about the validation of the proposed work.
2.1. The data validation is not well described. Line 183 says memories “are validated for temporal and logical consistency ... using a combination of manual review and LLM-based validation,” but the paper never defines either kind of consistency and gives no procedure, prompt, or statistics (how many memories were generated and rejected, how large the manual sample was, how the manual and LLM judgments agree). Figure 3 shows no validation stage, and Appendix H has prompts for persona and task generation but none for memory generation or validation. The same holds for the manual review of task instances at line 224. The 99% solvability figure is measured on generic questions with full context, so it validates the generic QA pairs, not the personalized or counterfactual instances the benchmark is about.
2.2. The task set is not well motivated. The five tasks are called “complementary” and loosely mapped to the three questions at the start of Section 4, but the paper does not motivate why this decomposition captures personalized household intelligence, which task exercises which ingredient (multi-resident, long-term memory, evolving state, heterogeneous appliances), or what is left out. MC is IG in a constrained format. FR asks the model to rank ten LLM-selected features against Gemini’s own ranking by MAP@1; if the inference candidate list is these ten features, chance MAP@1 is 0.30, and the Sole-Reasoning scores in Table 2 can be at chance.
2.3. “Project researchers subsequently annotate a guidance subset” (line 239). Who are the annotators, how many trajectories, and how well do they agree? More importantly, can a researcher judge what a particular resident would want? The RPJ validation has four annotators judging from written persona profiles whether they agree with the judge’s scores, so the humans are role-playing just as the LLM is, and the persona-level agreement is lower than the household-level one. The main text says each annotator evaluates 100 plans; footnote 3 says 50. I would like to see a discussion of whether annotators should resemble the personas (age, role, caregiver status), or a small study with real residents.
Thank you for the rebuttal. However, the additional results of GPT-5.6 still cannot show the performance of latest frontier models. GPT-5.6 is released in July, even two months before the submission deadline. There have been dozens of new models (closed-source, open-source, multimodal) since then.