acceptodds
Under review as a conference paper at ICLR 2027

CoHabit Arena: Evaluating Personal Preference Realization in Long-Horizon Human-Agent CoHabitation

Abstract

Completing a household task does not ensure that an embodied agent respects a user’s unstated preferences. We formalize Personal Preference Realization (PPR) as the ability to infer a user’s latent preferences from interaction, retain them across tasks, and express them in subsequent embodied decisions. To evaluate this capability, we introduce CoHabit Arena, a diagnostic evaluation environment built on Habitat 3.0. Each session preserves a synthetic user persona and interaction history across tasks without providing its ground-truth preference profile to the agent. Executed trajectories are retrospectively matched to preference-sensitive choices and associated with persona-conditioned reactions, allowing preference evidence to be traced to prior choices and their consequences. CoHabit Arena comprises 576 sessions of 15 tasks across 12 scenes and 36 synthetic personas, totaling 8,640 task instances derived from 1,080 source tasks. Its evaluation separates preference inference (mental alignment), preference-sensitive executed choices (behavioral alignment), and task success. We include E-ToM, a diagnostic baseline that maintains a persistent belief over preference axes and uses it to select subsequent plans. Experiments with standard LLM planners and a Theory-of-Mind baseline across two LLM backbones show that E-ToM-augmented planners obtain higher mental-alignment scores and lower preference-estimation error, while task-success gains are inconsistent. CoHabit Arena thus provides a controlled setting for studying cross-task preference acquisition and testing whether inferred preferences translate into embodied behavior beyond instruction following.

Primary area:
datasets and benchmarks
Keywords:
embodied agents, personal preference realization, human-agent interaction, long-horizon interaction, personalization, theory of mind, embodied AI, evaluation benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_QmfeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors provide CoHabit Arena. CoHabit Arena evaluates whether an embodied agent can infer a user’s preferences, retain them across tasks, and reflect them in later actions. It uses 576 task sessions with synthetic personas and retrospectively matches trajectories to preference-sensitive choices. An E-ToM baseline maintains a preference vector from structured feedback. Its recorded estimates improve, but behavioral comparisons and several metrics remain unresolved. The benchmark separates mental alignment, measured against a synthetic preference vector, from behavioral regret and task success. Physical state resets between tasks, while interaction history persists. Thus the long-horizon aspect concerns accumulated knowledge rather than continuous household dynamics. This is a reasonable controlled setting, but its usefulness depends on valid alternatives, consistent readouts, and reliable trajectory scoring.

Critical strengths
  • The paper tackles an important question: completing a household task is not the same as respecting a user’s preferences. Separating task success, preference estimation, and actual choices is a useful starting point for a benchmark.
  • The authors are clear about the limitations. In particular, they acknowledge that better preference-vector scores do not establish better behavior. This makes the current evidence easier to interpret. The proposed choice–consequence–reaction records could also support useful error analysis. If the matching and alternatives are audited, they would let researchers inspect whether a failure came from interpreting feedback, retaining preferences, or selecting an action. That diagnostic aim is a strength worth preserving in a revised evaluation.
Critical weaknesses
  • About the real novelty. The benchmark's distinct contribution over existing personalization work is not yet demonstrated. MEMENTO already evaluates embodied use of personal knowledge acquired from earlier interactions, while Hypothetical Minds maintains and refines hypotheses about other agents. These are not identical settings: CoHabit's potential addition is inferring unstated preferences from consequences and tracing their realization through diagnostic choices. However, the paper does not show a controlled task family or behavioral failure that specifically requires this addition rather than ordinary persistent memory. E-ToM itself is also a limited algorithmic advance: Eqs. (6)–(7) collapse to a weighted pseudo-count accumulation rule. A new benchmark need not introduce a new learner, but here its distinctive evaluation capability must carry the contribution, and that capability is still unvalidated.
  • Several reported measurements contradict their definitions or cannot be reconciled across experiments. SPL exceeds SR in three rows of Table 2 (p. 7). Appendix B.2.3, Eq. (13), defines each SPL contribution as the success indicator multiplied by L/max(P,L), a factor at most one. Therefore SPL must be at most SR on the same tasks with the same aggregation. Yet the DeepSeek rows report (SR, SPL) = (0.108, 0.134) for ToT, (0.068, 0.072) for ToT+E-ToM, and (0.109, 0.116) for Hyp-Minds. These gaps exceed rounding error; overlapping confidence intervals do not resolve a per-trajectory mathematical constraint. The results require corrected computation or an explicit explanation of differing samples or metric definitions.
  • The nominally same full configuration has unreconciled results. For ReAct+E-ToM with DeepSeek, Table 2 reports SR/SPL = 0.391/0.326 and Ep@Thr = 17.4, whereas Table 3 reports 0.382/0.316 and 3.1. Different runs could explain some variation, but the paper does not identify differing runs, samples, or aggregation rules. In particular, the adaptation-cost comparison cannot be interpreted without that information.
  • I'm not sure but, the worked example does not preserve the task constraints. Appendix E.2 (p. 35) requires moving objects to the kitchen cabinet, but offers a living-room cabinet or any cabinet as alternatives. The preference comparison therefore does not hold task satisfaction fixed. The same example reports task completion of 0.9 versus 0.1 under “same interaction, different attribution,” without identifying separate runs or an evaluation unit. Retrospectively assigning feedback cannot change the success of an already completed trajectory, so these numbers do not validate an execution benefit from attribution.
Questions for authors
  • Why the three SPL > SR pairs in Table 2? Please provide corrected SR/SPL computed from the same trajectories and weights (I'm not sure if I'm misunderstanding this. Could you please clarify?)And also, do the full rows in Tables 2 and 3 use the same runs and samples? Please reconcile Ep@Thr = 17.4 versus 3.1, the SR/SPL differences, and specify the indexing unit, aggregation, and confidence-interval procedure if possible.
  • What task or failure mode does CoHabit isolate beyond embodied memory personalization such as MEMENTO? Since Eqs. (6)–(7) reduce to weighted evidence accumulation, does E-ToM outperform a persistent-memory baseline with the same input fields, candidate budget, and preference readout?
Reviewer_PpUBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Personal Preference Realization (PPR), an evaluation setting that examines whether embodied agents can infer users' latent preferences from previous interactions, retain these preferences across tasks, and translate them into appropriate future actions. Unlike conventional embodied instruction-following benchmarks that primarily emphasize task completion, PPR explicitly distinguishes preference inference from preference-sensitive behavior and task execution.

The paper additionally introduces Embodied Theory-of-Mind (E-ToM) as a reference baseline. E-ToM maintains a persistent Dirichlet-based belief over user preference dimensions, updates this belief using structured feedback, and uses the resulting estimates to guide subsequent plan selection.

Overall, the paper presents a relevant benchmark direction and a structured evaluation framework, but the empirical evidence remains insufficient to validate its central objective of measuring and improving preference-conditioned embodied behavior.

Critical strengths
  1. A meaningful evaluation objective that distinguishes preference realization from task completion

The proposed PPR formulation makes an explicit distinction between inferring preferences, retaining them, and realizing them through subsequent decisions. This distinction is consequential because it motivates evaluation criteria that go beyond conventional task-success metrics.

In particular, separating Mental Alignment, Behavioral Alignment, and instruction-following success provides a useful conceptual basis for studying situations where an agent performs the requested task correctly but makes choices that are undesirable for a particular user.

  1. A structured, simulator-backed framework for examining preference-sensitive decisions

The combination of persistent personas, preference-diagnostic alternatives, attribution windows, and structured outcome signals offers a potentially inspectable evaluation protocol.

Critical weaknesses
  1. The central claim of behavioral preference realization is not empirically established

The most consequential weakness is the absence of a controlled, comparable evaluation of actual preference-conditioned behavior.

The paper defines PPR not merely as estimating a user's latent preference vector, but as expressing the inferred preferences in subsequent embodied decisions. However, the reported experiments provide limited evidence for this essential behavioral component.

First, D-Regret is reported only for E-ToM configurations. The baseline methods have missing matched-option measurements, and the number and distribution of valid scored decisions are not provided. Consequently, it is impossible to determine whether E-ToM achieves better preference realization than the baselines, or whether its reported regret values reflect meaningful behavior.

Second, the paper does not include a controlled intervention that isolates the influence of acquired preference history on subsequent actions. For example, holding the future task and environment fixed while varying the preceding user interaction history would directly test whether agents make different decisions because they have acquired different preferences.

Third, the case studies do not adequately compensate for this missing evidence. In particular, Case Study 2 includes preference alternatives that do not preserve the original instruction's destination constraint, raising concerns about whether the annotated options constitute valid preference-sensitive alternatives within the same task.

As a result, the experiments demonstrate improvements in an explicit preference representation, but do not establish corresponding improvements in executed preference-sensitive decisions.

  1. Inconsistent measurements undermine the reliability of the reported comparisons

The numerical results raise concerns:

  • In Table 2, several DeepSeek configurations report SPL greater than SR, violating the standard inequality SPL≤SR\mathrm{SPL}\leq\mathrm{SR}.
  • In Table 3, the frozen-state ablation reports Ep@Thr = 1.0 despite having no preference-state movement, contradicting the metric's stated convergence criterion.
  • The full ReAct+E-ToM configuration reports different SR, SPL, and Ep@Thr values between Tables 2 and 3, without a documented explanation.
  • Some ablations achieve slightly better preference-estimation or regret point estimates than the complete system, limiting conclusions about the necessity of particular components.

The paper acknowledges these problems, but acknowledgment does not resolve their implications for experimental validity.

Questions for authors
  1. Can the authors provide a method-independent behavioral evaluation showing that preference history improves subsequent decisions?

The central PPR claim requires evidence that acquired preferences influence later behavior. Can the authors report D-Regret for all baseline methods using an evaluator applied independently to their executed trajectories, together with the number of encountered, matched, excluded, and scored diagnostic choices?

More importantly, can the authors provide a controlled comparison in which the same agent faces an identical future task and environment after receiving different preference histories, including an appropriate no-history control? Evidence that preferences systematically influence valid future choices would substantially strengthen the paper.

  1. What information is available to each method, and how are baseline preference estimates obtained?

Can the authors specify the exact observation and feedback fields accessible to the base planners, E-ToM wrapper, and candidate-plan scorer, particularly whether preference-axis labels, consequence vectors, or ground-truth-derived metadata are exposed?

  1. Can the authors reconcile the experimental inconsistencies and provide sufficient implementation details to verify the measurements?

  2. How is the validity of preference-diagnostic choices and synthetic feedback established?

The benchmark relies on generated preference-sensitive alternatives and persona-conditioned outcome signals. Can the authors clarify how they verify that annotated alternatives are physically feasible, satisfy the same explicit task constraints, and genuinely differ in preference implications?

Reviewer_BSLMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper defines Personal Preference Realization (PPR): inferring a user's latent preferences, retaining them across tasks, and acting on them later. It builds CoHabit Arena on Habitat 3.0 (12 scenes, 36 synthetic personas, 576 sessions of 15 tasks). Executed trajectories are matched retrospectively to preference-diagnostic choices, and synthetic persona reactions are attached to those choices. The evaluation reports three components separately: preference inference (Mental Alignment, MA), behavioral realization (Decision Regret, D-Regret), and task success (SR/SPL). The proposed E-ToM wrapper maintains a Dirichlet preference state. With two LLM backbones, it improves MA over CoT/ReAct/ToT/Hyp-Minds without a consistent change in SR.

Critical strengths

Splitting the evaluation into inference (MA), realization (D-Regret), and execution (SR/SPL) is a sound benchmark design. It can expose "knows but doesn't act" failures that benchmarks scoring only task success cannot detect. I also appreciate how openly the authors discuss the limitations of their current results.

Critical weaknesses

W1. The behavioral metric is not yet evaluated for the baselines, and some reported numbers need reconciliation.

D-Regret is the benchmark's distinctive metric, but it is reported only for E-ToM (Table 2). Without values for the baselines, or reference points such as a random or oracle policy, it is hard to judge whether inferred preferences translate into behavior. Several entries also need checking:

  • three rows report SPL > SR;
  • the frozen-state Ep@Thr = 1.0 conflicts with Eq. 17;
  • the full configuration differs between Table 2 and Table 3.

Resolving these would substantially strengthen the paper as an evaluation resource.

W2. The MA comparison may overstate the inference gain.

The CoT and ReAct MA, L2, and α0 values are identical on both backbones and appear to equal the prior. If so, E-ToM is effectively being compared against a constant vector. In addition, E-ToM consumes outcome signals annotated with "affected preference axes". Since those annotations are derived from the ground truth, part of the MA gain may reflect access to structured labels rather than inference.

The following controls would make the result more convincing:

  • a common elicitation probe applied to all methods;
  • a memory-matched baseline;
  • a history-swap test.
Questions for authors
  1. Could you report D-Regret for all methods with a common matcher, together with coverage counts and random/oracle references?
  2. Which signal fields (axes, severity, decision ID) are available to E-ToM but not to the base planners?
  3. What explains SPL > SR in the three DeepSeek rows, and the difference between the full-configuration rows in Tables 2 and 3?
  4. How are the personas and signals generated? If an LLM is used, is it from the same family as the evaluated backbones?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.