- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors provide CoHabit Arena. CoHabit Arena evaluates whether an embodied agent can infer a user’s preferences, retain them across tasks, and reflect them in later actions. It uses 576 task sessions with synthetic personas and retrospectively matches trajectories to preference-sensitive choices. An E-ToM baseline maintains a preference vector from structured feedback. Its recorded estimates improve, but behavioral comparisons and several metrics remain unresolved. The benchmark separates mental alignment, measured against a synthetic preference vector, from behavioral regret and task success. Physical state resets between tasks, while interaction history persists. Thus the long-horizon aspect concerns accumulated knowledge rather than continuous household dynamics. This is a reasonable controlled setting, but its usefulness depends on valid alternatives, consistent readouts, and reliable trajectory scoring.
Critical strengths
- The paper tackles an important question: completing a household task is not the same as respecting a user’s preferences. Separating task success, preference estimation, and actual choices is a useful starting point for a benchmark.
- The authors are clear about the limitations. In particular, they acknowledge that better preference-vector scores do not establish better behavior. This makes the current evidence easier to interpret. The proposed choice–consequence–reaction records could also support useful error analysis. If the matching and alternatives are audited, they would let researchers inspect whether a failure came from interpreting feedback, retaining preferences, or selecting an action. That diagnostic aim is a strength worth preserving in a revised evaluation.
Critical weaknesses
- About the real novelty. The benchmark's distinct contribution over existing personalization work is not yet demonstrated. MEMENTO already evaluates embodied use of personal knowledge acquired from earlier interactions, while Hypothetical Minds maintains and refines hypotheses about other agents. These are not identical settings: CoHabit's potential addition is inferring unstated preferences from consequences and tracing their realization through diagnostic choices. However, the paper does not show a controlled task family or behavioral failure that specifically requires this addition rather than ordinary persistent memory. E-ToM itself is also a limited algorithmic advance: Eqs. (6)–(7) collapse to a weighted pseudo-count accumulation rule. A new benchmark need not introduce a new learner, but here its distinctive evaluation capability must carry the contribution, and that capability is still unvalidated.
- Several reported measurements contradict their definitions or cannot be reconciled across experiments. SPL exceeds SR in three rows of Table 2 (p. 7). Appendix B.2.3, Eq. (13), defines each SPL contribution as the success indicator multiplied by L/max(P,L), a factor at most one. Therefore SPL must be at most SR on the same tasks with the same aggregation. Yet the DeepSeek rows report (SR, SPL) = (0.108, 0.134) for ToT, (0.068, 0.072) for ToT+E-ToM, and (0.109, 0.116) for Hyp-Minds. These gaps exceed rounding error; overlapping confidence intervals do not resolve a per-trajectory mathematical constraint. The results require corrected computation or an explicit explanation of differing samples or metric definitions.
- The nominally same full configuration has unreconciled results. For ReAct+E-ToM with DeepSeek, Table 2 reports SR/SPL = 0.391/0.326 and Ep@Thr = 17.4, whereas Table 3 reports 0.382/0.316 and 3.1. Different runs could explain some variation, but the paper does not identify differing runs, samples, or aggregation rules. In particular, the adaptation-cost comparison cannot be interpreted without that information.
- I'm not sure but, the worked example does not preserve the task constraints. Appendix E.2 (p. 35) requires moving objects to the kitchen cabinet, but offers a living-room cabinet or any cabinet as alternatives. The preference comparison therefore does not hold task satisfaction fixed. The same example reports task completion of 0.9 versus 0.1 under “same interaction, different attribution,” without identifying separate runs or an evaluation unit. Retrospectively assigning feedback cannot change the success of an already completed trajectory, so these numbers do not validate an execution benefit from attribution.
Questions for authors
- Why the three SPL > SR pairs in Table 2? Please provide corrected SR/SPL computed from the same trajectories and weights (I'm not sure if I'm misunderstanding this. Could you please clarify?)And also, do the full rows in Tables 2 and 3 use the same runs and samples? Please reconcile Ep@Thr = 17.4 versus 3.1, the SR/SPL differences, and specify the indexing unit, aggregation, and confidence-interval procedure if possible.
- What task or failure mode does CoHabit isolate beyond embodied memory personalization such as MEMENTO? Since Eqs. (6)–(7) reduce to weighted evidence accumulation, does E-ToM outperform a persistent-memory baseline with the same input fields, candidate budget, and preference readout?