- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces HABIT, a bottom-up taxonomy of 23 behavioral axes that characterizes how LLM agents perform everyday tasks beyond simple task success. And AgentHABIT, a benchmark that uses these axes to build behavioral profiles from agent trajectories. The taxonomy is derived from 408 trajectories across 17 domains, while the benchmark evaluates 18 models on 86 open-ended tasks and aggregates trajectory-level annotations into model-level behavioral profiles. Validation shows that HABIT distinguishes models more effectively than prior value/action taxonomies, and reveals that some behavioral tendencies are steerable by prompting, while others remain resistant even after SFT or DPO fine-tuning.
Critical strengths
- The paper studies an interesting question: not only focusing on task success, it examines how agents behave during task execution and provides a systematic taxonomy to describe behavioral differences.
- The empirical study is extensive and produces several findings. By evaluate 18 models over 86 tasks, show that behavioral profiles are relatively stable and find meaningful differences across model family.
Critical weaknesses
- AgentHABIT uses only one constructed task per subdomain, and the 86 tasks are balanced across subdomains rather than sampled according to the real distribution of user requests. This may have limited external validity of the behavioral profiles.
- The paper demonstrates that HABIT can distinguish models and that its axes can be annotated consistently, but it does not show that these behavioral differences correspond to actual user preferences, satisfaction, task efficiency, or downstream outcomes. So the practical meaning of the behavior taxonomy is not sufficiently validated.
Questions for authors
- How sensitive are the reported behavioral profiles to the particular task chosen for each subdomain? For example, have the authors evaluated multiple independent task instantiations within the same subdomain, or reweighted tasks according to real usage frequencies?
- Is there any direct evidence that similarity in HABIT profiles predicts similarity in human-perceived behavior or user preference? If such evidence exists, it would substantially strengthen the practical interpretation of the benchmark.
4 comments
Rebuttal part [2/2] - Alignment of HABIT profiles with real users’ preferences and perceived similarity
W2. The paper demonstrates that HABIT can distinguish models and that its axes can be annotated consistently, but it does not show that these behavioral differences correspond to actual user preferences, satisfaction, task efficiency, or downstream outcomes. So the practical meaning of the behavior taxonomy is not sufficiently validated.
Q2. Is there any direct evidence that similarity in HABIT profiles predicts similarity in human-perceived behavior or user preference? If such evidence exists, it would substantially strengthen the practical interpretation of the benchmark.
A2. Thank you for raising this point. We agree that distinguishing models and annotating axes consistently do not by themselves show that these behavioral differences matter to users. To test this directly, we recruited 50 participants on Prolific to complete 10 AgentHABIT tasks with three agents (GPT-5.6 Sol, Claude Opus 5, and Gemini 3.7 Flash). After the conversations, each participant rated their satisfaction with each agent, ranked the three agents by which one they would use again, rated how similarly each pair of agents worked, and completed a survey in which, for each of the 23 HABIT axes, they ranked the possible agent behaviors from most to least preferred.
| Participant's own preferences | Another participant's preferences | |
|---|---|---|
| Predicted agent is the participant's first choice | 86% (43/50) | 52% |
| Predicted order of all three agents is correct | 72% (36/50) | 38% |
We first asked whether HABIT profiles can predict which agent a user prefers. For each participant, we built a preference profile from these survey rankings and ordered the three agents by how close their benchmark profiles are to it. As the table above shows, the predicted agent is the one the participant actually ranked first for 43 of the 50 participants (86%), and the predicted order of all three agents is correct for 72%. With another participant's preferences, these rates drop to 52% and 38% (). Satisfaction shows the same pattern, as participants rated the best-matching agent 6.18 out of 7 and the worst-matching agent 4.84. In short, simply by indicating their preferences on each HABIT axis, users can be recommended the agent that best fits them.
| Agent pair | Distance between benchmark profiles | Perceived similarity |
|---|---|---|
| GPT-5.6 Sol and Gemini 3.7 Flash | 0.196 | 4.28 |
| Claude Opus 5 and GPT-5.6 Sol | 0.280 | 4.10 |
| Claude Opus 5 and Gemini 3.7 Flash | 0.319 | 3.50 |
HABIT also measures how close or far two agents are from each other, so we next asked whether this distance matches what people actually feel (Q2). Participants rated how similarly each pair of agents worked, from 1 (very different) to 7 (very similar). As the table above shows, the closer two agents' profiles are, the more similar participants found them. The profiles also tell us where the difference lies. Claude Opus 5 and Gemini 3.7 Flash, the pair that participants found least similar, differ most in whether they ask the user questions (95% versus 29% of benchmark runs), how they handle missing information, and whether they suggest next steps. We find this meaningful because it suggests that HABIT captures the behaviors by which users tell agents apart.
In addition, since real users took the place of the LLM user simulator in this study, we could also check whether the profiles themselves still hold. We built each agent's profile from its conversations with participants, and each one is closest to its own benchmark profile (a distance of 0.16–0.17 on a scale from 0 to 1, versus 0.24–0.42 to the other agents' profiles). We have added these results to Section 3.4 and Appendix Q. Thank you again for this suggestion, which has strengthened the paper.
Thank you for the detailed responses
Thank you for the detailed responses. I really appreciate that.
Thank you for reading our replies! If anything else comes up before the discussion closes, we’d be glad to answer. The points you raised in your review pushed us to test whether HABIT profiles hold under real usage distributions and whether they align with what real users prefer and perceive, so thank you again for the thoughtful engagement.
Rebuttal part [1/2] - Sensitivity to task choice and real usage
We thank the reviewer for recognizing the discriminability and reliability of HABIT, and for the thoughtful questions.
A1. Thank you for these suggestions. Following them, we built a 120-task version of AgentHABIT whose tasks are sampled according to real usage, and we also reweighted the original 86 tasks by real usage. In both cases, the profiles barely change.
As you suggested, we constructed a version of AgentHABIT in which the number of tasks follows real usage. It keeps the original 86 tasks and adds 34 new ones in the most heavily used subdomains of the Anthropic Economic Index, which cuts the distance between the task distribution and real usage nearly in half (from 0.260 to 0.142). We ran all 14 models on the new tasks (1,428 new trajectories). As the table above shows, the 120-task profiles are nearly identical to the original profiles (mean TV distance of 0.024), and reweighting the original 86 tasks by real usage gives the same picture (0.039–0.050). For comparison, two different models are 0.20 apart on average, and even a model's nearest neighbor is 0.10 away. Since you also asked about independent tasks within the same subdomain, we note that the 34 new tasks are exactly such tasks, and profiles built from them alone also remain close to the original profiles (0.084).
These results indicate that the original 86 tasks already reflect real usage well, so we keep them as the main benchmark and release the 120-task version alongside it. Regarding external validity more broadly, we also tested the profiles with real users, which we describe in A2 below. We have added these results to Section 3.3 and Appendix L.