acceptodds
Under review as a conference paper at ICLR 2027

AgentHabit: Characterizing Distinct Behaviors of Agents on Everyday Tasks

Abstract

Large language model (LLM) agents assist users with everyday tasks that can be completed in many reasonable ways. Even when their answers are useful, how agents carry out these tasks may not match users' preferences and needs. For example, agents differ in whether they ask clarifying questions or search the web. We introduce HABIT, a taxonomy of 23 behavioral axes in five categories, which three authors and three LLMs derive bottom-up from 408 agent trajectories across 17 domains. On held-out tasks, HABIT distinguishes models more clearly than existing taxonomies of human values and agent actions while supporting comparably consistent annotation. Building on HABIT, we construct AgentHABIT, a benchmark that profiles each agent's behavioral tendencies from its trajectories on 86 everyday tasks. Profiling 18 models with AgentHABIT reveals a range of distinctive tendencies. For example, most GPT and Claude models state their assumptions and offer alternatives when requirements conflict, whereas Qwen and Google's models more often leave assumptions or changes to requirements unstated. These profiles remain recognizable even when built from entirely different sets of tasks, indicating that they reflect general tendencies rather than task-specific behavior. Prompting agents to adopt specific behaviors shifts some axes readily but barely changes others, while fine-tuning on another model's trajectories changes only part of a model's profile and leaves much of it intact. Overall, HABIT and AgentHABIT provide a systematic framework for characterizing how agents carry out everyday tasks beyond task success, offering insights to guide the development of agents whose behavior better fits users' needs.

Primary area:
datasets and benchmarks
Keywords:
LLM Agents, Agent Behavior, Behavioral Profiling, Agent Evaluation, Benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MdNjOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces HABIT, a bottom-up taxonomy of 23 behavioral axes that characterizes how LLM agents perform everyday tasks beyond simple task success. And AgentHABIT, a benchmark that uses these axes to build behavioral profiles from agent trajectories. The taxonomy is derived from 408 trajectories across 17 domains, while the benchmark evaluates 18 models on 86 open-ended tasks and aggregates trajectory-level annotations into model-level behavioral profiles. Validation shows that HABIT distinguishes models more effectively than prior value/action taxonomies, and reveals that some behavioral tendencies are steerable by prompting, while others remain resistant even after SFT or DPO fine-tuning.

Critical strengths
  1. The paper studies an interesting question: not only focusing on task success, it examines how agents behave during task execution and provides a systematic taxonomy to describe behavioral differences.
  2. The empirical study is extensive and produces several findings. By evaluate 18 models over 86 tasks, show that behavioral profiles are relatively stable and find meaningful differences across model family.
Critical weaknesses
  1. AgentHABIT uses only one constructed task per subdomain, and the 86 tasks are balanced across subdomains rather than sampled according to the real distribution of user requests. This may have limited external validity of the behavioral profiles.
  2. The paper demonstrates that HABIT can distinguish models and that its axes can be annotated consistently, but it does not show that these behavioral differences correspond to actual user preferences, satisfaction, task efficiency, or downstream outcomes. So the practical meaning of the behavior taxonomy is not sufficiently validated.
Questions for authors
  1. How sensitive are the reported behavioral profiles to the particular task chosen for each subdomain? For example, have the authors evaluated multiple independent task instantiations within the same subdomain, or reweighted tasks according to real usage frequencies?
  2. Is there any direct evidence that similarity in HABIT profiles predicts similarity in human-perceived behavior or user preference? If such evidence exists, it would substantially strengthen the practical interpretation of the benchmark.
4 comments
AuthorsOpenReview ↗

Rebuttal part [1/2] - Sensitivity to task choice and real usage

We thank the reviewer for recognizing the discriminability and reliability of HABIT, and for the thoughtful questions.


W1. AgentHABIT uses only one constructed task per subdomain, and the 86 tasks are balanced across subdomains rather than sampled according to the real distribution of user requests. This may have limited external validity of the behavioral profiles.

Q1. How sensitive are the reported behavioral profiles to the particular task chosen for each subdomain? For example, have the authors evaluated multiple independent task instantiations within the same subdomain, or reweighted tasks according to real usage frequencies?

A1. Thank you for these suggestions. Following them, we built a 120-task version of AgentHABIT whose tasks are sampled according to real usage, and we also reweighted the original 86 tasks by real usage. In both cases, the profiles barely change.

Profile built fromMean TV distance to original 86-task profileSpearman ρ\rho of pairwise model distances
120 tasks, usage-sampled (86 original + 34 new)0.0240.993
86 tasks, usage-weighted (platforms equal)0.0390.978
86 tasks, usage-weighted (platforms by traffic)0.0500.964
34 new tasks only0.0840.941
Reference: distance between two different models0.200–

As you suggested, we constructed a version of AgentHABIT in which the number of tasks follows real usage. It keeps the original 86 tasks and adds 34 new ones in the most heavily used subdomains of the Anthropic Economic Index, which cuts the distance between the task distribution and real usage nearly in half (from 0.260 to 0.142). We ran all 14 models on the new tasks (1,428 new trajectories). As the table above shows, the 120-task profiles are nearly identical to the original profiles (mean TV distance of 0.024), and reweighting the original 86 tasks by real usage gives the same picture (0.039–0.050). For comparison, two different models are 0.20 apart on average, and even a model's nearest neighbor is 0.10 away. Since you also asked about independent tasks within the same subdomain, we note that the 34 new tasks are exactly such tasks, and profiles built from them alone also remain close to the original profiles (0.084).

These results indicate that the original 86 tasks already reflect real usage well, so we keep them as the main benchmark and release the 120-task version alongside it. Regarding external validity more broadly, we also tested the profiles with real users, which we describe in A2 below. We have added these results to Section 3.3 and Appendix L.

AuthorsOpenReview ↗

Rebuttal part [2/2] - Alignment of HABIT profiles with real users’ preferences and perceived similarity

W2. The paper demonstrates that HABIT can distinguish models and that its axes can be annotated consistently, but it does not show that these behavioral differences correspond to actual user preferences, satisfaction, task efficiency, or downstream outcomes. So the practical meaning of the behavior taxonomy is not sufficiently validated.

Q2. Is there any direct evidence that similarity in HABIT profiles predicts similarity in human-perceived behavior or user preference? If such evidence exists, it would substantially strengthen the practical interpretation of the benchmark.

A2. Thank you for raising this point. We agree that distinguishing models and annotating axes consistently do not by themselves show that these behavioral differences matter to users. To test this directly, we recruited 50 participants on Prolific to complete 10 AgentHABIT tasks with three agents (GPT-5.6 Sol, Claude Opus 5, and Gemini 3.7 Flash). After the conversations, each participant rated their satisfaction with each agent, ranked the three agents by which one they would use again, rated how similarly each pair of agents worked, and completed a survey in which, for each of the 23 HABIT axes, they ranked the possible agent behaviors from most to least preferred.

Participant's own preferencesAnother participant's preferences
Predicted agent is the participant's first choice86% (43/50)52%
Predicted order of all three agents is correct72% (36/50)38%

We first asked whether HABIT profiles can predict which agent a user prefers. For each participant, we built a preference profile from these survey rankings and ordered the three agents by how close their benchmark profiles are to it. As the table above shows, the predicted agent is the one the participant actually ranked first for 43 of the 50 participants (86%), and the predicted order of all three agents is correct for 72%. With another participant's preferences, these rates drop to 52% and 38% (p=0.001p=0.001). Satisfaction shows the same pattern, as participants rated the best-matching agent 6.18 out of 7 and the worst-matching agent 4.84. In short, simply by indicating their preferences on each HABIT axis, users can be recommended the agent that best fits them.

Agent pairDistance between benchmark profilesPerceived similarity
GPT-5.6 Sol and Gemini 3.7 Flash0.1964.28
Claude Opus 5 and GPT-5.6 Sol0.2804.10
Claude Opus 5 and Gemini 3.7 Flash0.3193.50

HABIT also measures how close or far two agents are from each other, so we next asked whether this distance matches what people actually feel (Q2). Participants rated how similarly each pair of agents worked, from 1 (very different) to 7 (very similar). As the table above shows, the closer two agents' profiles are, the more similar participants found them. The profiles also tell us where the difference lies. Claude Opus 5 and Gemini 3.7 Flash, the pair that participants found least similar, differ most in whether they ask the user questions (95% versus 29% of benchmark runs), how they handle missing information, and whether they suggest next steps. We find this meaningful because it suggests that HABIT captures the behaviors by which users tell agents apart.

In addition, since real users took the place of the LLM user simulator in this study, we could also check whether the profiles themselves still hold. We built each agent's profile from its conversations with participants, and each one is closest to its own benchmark profile (a distance of 0.16–0.17 on a scale from 0 to 1, versus 0.24–0.42 to the other agents' profiles). We have added these results to Section 3.4 and Appendix Q. Thank you again for this suggestion, which has strengthened the paper.

Reviewer_MdNjOpenReview ↗

Thank you for the detailed responses

Thank you for the detailed responses. I really appreciate that.

Thank you for reading our replies! If anything else comes up before the discussion closes, we’d be glad to answer. The points you raised in your review pushed us to test whether HABIT profiles hold under real usage distributions and whether they align with what real users prefer and perceive, so thank you again for the thoughtful engagement.

Reviewer_FbxwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposed Habit and AgentHabit -- Taxonomy and benchmark to model each agent’s behavioral tendencies from its trajectories on 86 everyday tasks.

Critical strengths

-- Generally well-written and easy to understand the motivation and study setup -- wide collection of LLM evaluation

Critical weaknesses

-- My biggest concern is the evaluation setup and validation of the taxonomy and the dataset. The task annotation was done using an agent and a user simulator, but overall paper motivation starts with the vision of 'helping real users with everyday tasks'. It was unclear if the performance will transfer in real-world setup. For a everyday task domain like this, atleast evaluating with real users would have been essential.

-- My second concern is about the data quantity. Since the data was collected synthetically, was there a reason the authors chose to consider 86 everyday tasks in AgentHabit?

-- Some of the recent works, which are highly relevant are missing: Cocoa: Co-Planning and Co-Execution with AI Agents, Modeling Distinct Human Interaction in Web Agents

Questions for authors

-- see above

8 comments
AuthorsOpenReview ↗

Rebuttal part [1/4] - Validation with real users

We thank the reviewer for recognizing the motivation of characterizing how agents help real users with everyday tasks, and for the constructive feedback.


W1. My biggest concern is the evaluation setup and validation of the taxonomy and the dataset. The task annotation was done using an agent and a user simulator, but overall paper motivation starts with the vision of 'helping real users with everyday tasks'. It was unclear if the performance will transfer in real-world setup. For a everyday task domain like this, atleast evaluating with real users would have been essential.

A1. Thank you for raising this concern. We agree that a benchmark motivated by helping real users should be evaluated with real users, and we have now conducted a new study with 50 human participants. Before presenting these new results, we first summarize how the submission already validates each stage of the pipeline against real usage and human judgment rather than fully-automated annotation by an agent and a user simulator.

Although an agent and a simulated user generate the trajectories, every stage is checked by humans:

  • Tasks. The domains come from real-world Claude usage (Anthropic Economic Index). An LLM drafts each task, and we manually review and revise every one.
  • Taxonomy. Three annotators each spend more than 20 hours independently reading all 408 trajectories to propose axes, and three LLMs from different model families review the same trajectories to catch differences that humans may overlook. Three authors then jointly review the merged proposals and refine them into the final 23 axes. We test these axes on 17 held-out tasks that were not used to build the taxonomy, where they distinguish models better than existing taxonomies (92.2% versus 72.8–87.2% matching accuracy).
  • LLM judge. Three human annotators independently label 17 trajectories on all 23 axes. Among four candidate judges, we select the one that agrees best with these labels. It matches the human-majority label in 83.5% of cases (Cohen's κ=0.83\kappa =0.83).
  • User simulator. We sample its writing style and feedback behavior from real user turns in WildChat-1M. Three annotators rate its utterances 4.55–4.68 out of 5 for naturalness, contextual appropriateness, and role consistency.

Nevertheless, real users may behave differently from a simulator, so we believe it is important to observe how the agents behave with them. To test this directly, we recruited 50 Prolific participants to complete 10 AgentHabit tasks with three agents (GPT-5.6 Sol, Claude Opus 5, and Gemini 3.7 Flash). After the conversations, each participant rated their satisfaction with each agent, ranked the three agents by which one they would use again, and ranked the behaviors on each of the 23 HABIT axes by their own preference.

Profile with real usersBenchmark: GPT-5.6 SolBenchmark: Claude Opus 5Benchmark: Gemini 3.7 Flash
GPT-5.6 Sol0.1710.3220.293
Claude Opus 50.3610.1680.421
Gemini 3.7 Flash0.2380.3530.163

We annotated their conversations in the same way as the benchmark trajectories and derived each agent's profile from them. As the table above shows, each agent's profile with real users is closest to its own benchmark profile, where the distance ranges from 0 (identical) to 1 (no overlap). We further compared each agent's profile from real-user conversations with the benchmark profiles of all 14 models in our main analysis, and the closest benchmark profile was still their own. This suggests that the model-specific behavioral patterns captured by the benchmark remain recognizable when agents interact with real users instead of the simulator.

AuthorsOpenReview ↗

Rebuttal part [2/4] - Validation with real users

A1. (continued)

The benchmark profiles also agree with users' own judgments. For each participant, we predicted the agent whose benchmark profile best matches the behaviors the participant said they prefer, and compared the result with using another participant's preferences.

Participant's own preferencesAnother participant's preferencesUniform chance
Predicted agent is the participant's first choice86% (43/50)52%33%
Predicted order of all three agents is correct72% (36/50)38%17%

As the table above shows, the agent predicted from a participant's own preferences is the one they would actually use again for 86% of participants (43 of 50), and the predicted order of all three agents is correct for 72%. With another participant's preferences, these rates fall to 52% and 38% on average. In 1,000 random reassignments of preferences among participants, none reached the accuracy obtained with participants' own preferences ($p = 0.001$ for both). This shows that the behavioral differences captured in benchmark profiles carry over to which agent real users actually prefer.

Agent pairDistance between benchmark profilesPerceived similarity
GPT-5.6 Sol and Gemini 3.7 Flash0.1964.28
Claude Opus 5 and GPT-5.6 Sol0.2804.10
Claude Opus 5 and Gemini 3.7 Flash0.3193.50

Participants also rated how similarly each pair of agents worked, from 1 (very different) to 7 (very similar). As the table above shows, pairs with closer benchmark profiles were rated as more similar. Because every participant rated all three pairs, we can compare two pairs within the same participants. The closest pair (GPT-5.6 Sol and Gemini 3.7 Flash, 4.28) was rated as more similar than the most distant pair (Claude Opus 5 and Gemini 3.7 Flash, 3.50).

Best-matching agentSecond-best-matching agentWorst-matching agent
Mean satisfaction (1–7)6.185.324.84

Finally, we compared the satisfaction ratings, given on a seven-point scale from 1 (not at all) to 7 (extremely), across the three agents, ordered as above by how well their benchmark profiles match the participant's stated preferences. As the table above shows, participants were more satisfied with the agent that best matches their stated preferences (6.18) than with the one that matches them least (4.84), and a difference of this size is unlikely to arise by chance (Wilcoxon signed-rank test over the 50 participants, p<0.001p<0.001). The best-matching agent also received the participant's highest satisfaction rating in 90% of cases (45/50). The benchmark profiles therefore relate not only to which agent a user would use again, but also to how satisfied the user is with the result. We have added these results to Section 3.4 and Appendix Q. Thank you again for this suggestion, which has strengthened the paper.

AuthorsOpenReview ↗

Rebuttal part [3/4] - Rationale for 86 tasks and profile stability with additional tasks

W2. My second concern is about the data quantity. Since the data was collected synthetically, was there a reason the authors chose to consider 86 everyday tasks in AgentHabit?

A2. Thank you for raising this concern. Our goal was for AgentHabit to cover the full range of everyday tasks that people bring to AI assistants, so we started from the Anthropic Economic Index (AEI), which summarizes what people actually ask Claude. Clustering its 1,058 open-ended request categories yielded 86 subdomains, and we built one task for each so that the benchmark covers real usage evenly (Appendix A). The number 86 is thus not predefined, but follows directly from this clustering, which raises the question of why we built one task per subdomain rather than several. We made this choice for three reasons.

First, and most importantly, each model behaves consistently across tasks, so its profile stabilizes well before 86 tasks (Section 3.3, Table 15). Even when evaluated against profiles built from the disjoint remaining 60 tasks, a profile built from only 26 randomly drawn tasks matches the exact same model 89.3% of the time and the correct model family 99.99% of the time.

Second, regarding synthetic collection, AgentHabit is not purely automated. While an LLM helps draft each task and a simulator plays the user, we manually review and revise every task for realism, consistency with the user’s private context, and overlap with existing tasks, while checking that held-back information can be obtained by asking (Section 3.1; Appendix B).

Third, evaluation cost grows in proportion to the number of tasks. Running the full benchmark cost about $1,450 in API fees and 160 GPU-hours, so adding a second task per subdomain would double this cost for every model profiled while changing the resulting profiles very little. In fact, we tested this directly by adding 34 new tasks in the most heavily used AEI subdomains (120 tasks total) and running all 14 models in our main analysis on them (1,428 new trajectories). The resulting 120-task profiles are nearly identical to the original 86-task profiles (mean TV distance of 0.024, with a Spearman correlation of 0.993 across pairwise model distances), whereas two different models are 0.20 apart on average, and even profiles built from the 34 new tasks alone remain close to the original profiles (0.084). We therefore keep the 86-task set as the main benchmark and release the 120-task version alongside it. We have added this rationale and analysis to Sections 3.1 and 3.3 and Appendix L.

Reviewer_EdhVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces HABIT, a taxonomy of 23 behavioral axes grouped into five categories, derived by having human annotators and LLMs summarize differences across 408 agent trajectories. Using an LLM-as-a-judge pipeline, the authors construct AgentHABIT (86 tasks across 18 models) to aggregate trajectory labels into model "behavioral profiles." The paper evaluates this taxonomy via pairwise model identification, cross-subset profile matching, self-report questionnaires, prompt steering, and basic SFT/DPO fine-tuning.

Critical strengths

The paper is clearly written and well-organized, and the engineering effort in curating multi-turn trajectories with a persona-driven user simulator and explicit annotation rubrics is tidy. Separating the tasks used to propose the rubric from the held-out evaluation tasks is a reasonable practice for descriptive dataset construction. However, these strengths are primarily organizational and do not overcome the lack of fundamental scientific or methodological contribution.

Critical weaknesses
  1. Lack of conceptual and methodological innovation: The framework is a straightforward application of standard LLM-as-a-judge rubric annotation.
    At its core, the paper's methodology consists of prompting models on multi-turn tasks, summarizing observable differences into 23 categorical rubrics (e.g., whether an agent uses emojis, how often it calls search, whether it asks questions before answering), and using an LLM judge to count label frequencies. Inductively proposing axes from model outputs (already established in works like VibeCheck) and categorizing agent actions (as in Act·ONOMY) are well-trodden ideas. Packaging 23 hand-curated rubrics into a descriptive checklist does not constitute a substantive conceptual or algorithmic advance for an ICLR paper. Furthermore, validating this taxonomy primarily by showing that a linear classifier can identify which model generated a trajectory (92.2% matching accuracy) is a trivial bar: models naturally exhibit distinct stylistic and formatting fingerprints across tasks, and high model discriminability does not demonstrate that HABIT provides deeper scientific insight than surface trajectory statistics (e.g., turn count, length, raw tool-call counts) or task competence.

  2. The empirical findings are largely predictable and conflate basic task competence with "behavioral tendencies."
    Because the methodology is a direct descriptive pipeline, the paper relies on its empirical findings to justify its contribution—yet the reported takeaways yield little new insight beyond common knowledge:

  • Conflating capability with behavioral style: A large fraction of the 23 axes—such as Artifact completeness (delivering a complete output vs. fragments/no output), Constraint preservation across outputs (retaining vs. dropping user constraints), Scope adherence, and Constraint conflict handling (explaining conflicts vs. fabricating feasibility)—directly measure instruction-following competence rather than neutral stylistic preferences. Observing that frontier models (e.g., GPT-5.6, Claude Opus 5) preserve constraints and flag conflicts while smaller or weaker models silently drop requirements simply redescribes well-known capability differences under the label of "behavioral profiles."
  • Predictable secondary analyses: Showing that models from the same training family have similar output distributions, that multiple-choice self-reports do not reliably match actual execution (a well-documented know–do gap in prior LLM psychometrics), and that prompting easily steers surface formatting (Decorative symbol use) much more than complex reasoning or sourcing behaviors are all intuitive, unsurprising observations.
  • Uninformative fine-tuning analysis: The claim that behavioral profiles resist SFT and DPO (Section 4.3, Appendix P) is based on a minimalist setup (258 trajectories / 27 optimizer updates for SFT, and 774 unfiltered model-level pairs / 75 updates for DPO) without verifying that the training signal actually targets specific behavioral differences or achieves convergence. Observing no significant profile shift from 27–75 steps of coarse LoRA tuning does not establish a meaningful scientific finding about behavioral persistence.

Overall, the paper applies a standard rubric-grading pipeline to confirm intuitive expectations about model families and instruction-following capabilities, falling well short of the conceptual novelty and empirical depth expected at ICLR.

Questions for authors
  1. Substantive advance over surface features and competence: Beyond showing that 23 rubric labels can fingerprint model identity across tasks, what non-obvious scientific insight or new capability does HABIT enable that cannot already be explained by (a) general instruction-following competence and (b) directly observable trajectory metadata (e.g., response length, number of turns, raw tool-invocation counts)?

  2. Disentangling capability failures from behavioral habits: Why are clear execution failures—such as dropping user constraints across turns (Constraint preservation), failing to supply a requested artifact (Artifact completeness), or fabricating feasibility (Constraint conflict handling)—framed as alternative "behavioral tendencies" rather than standard instruction-following deficits? If competence-related axes are removed, do the main cross-family conclusions still hold?

  3. Validity of the training-based steering claim: In Appendix P, what evidence demonstrates that 27 SFT updates and 75 DPO updates on unfiltered whole-trajectory pairs successfully learned the training distribution in the first place, so as to justify framing the null result as evidence of "behavioral persistence" rather than an ineffective training setup?

9 comments
AuthorsOpenReview ↗

Rebuttal part [1/7] - Contribution beyond surface features and competence

W1. Lack of conceptual and methodological innovation: The framework is a straightforward application of standard LLM-as-a-judge rubric annotation. At its core, the paper's methodology consists of prompting models on multi-turn tasks, summarizing observable differences into 23 categorical rubrics (e.g., whether an agent uses emojis, how often it calls search, whether it asks questions before answering), and using an LLM judge to count label frequencies. Inductively proposing axes from model outputs (already established in works like VibeCheck) and categorizing agent actions (as in Act·ONOMY) are well-trodden ideas. Packaging 23 hand-curated rubrics into a descriptive checklist does not constitute a substantive conceptual or algorithmic advance for an ICLR paper. Furthermore, validating this taxonomy primarily by showing that a linear classifier can identify which model generated a trajectory (92.2% matching accuracy) is a trivial bar: models naturally exhibit distinct stylistic and formatting fingerprints across tasks, and high model discriminability does not demonstrate that HABIT provides deeper scientific insight than surface trajectory statistics (e.g., turn count, length, raw tool-call counts) or task competence.

Q1. Substantive advance over surface features and competence: Beyond showing that 23 rubric labels can fingerprint model identity across tasks, what non-obvious scientific insight or new capability does HABIT enable that cannot already be explained by (a) general instruction-following competence and (b) directly observable trajectory metadata (e.g., response length, number of turns, raw tool-invocation counts)?

A1. Thank you for sharing your perspective. We believe that systematically understanding agent behavior is highly important to users choosing an assistant, model developers seeking to control its behavior, and ML researchers studying how behavioral tendencies generalize and respond to intervention. A behavioral framework therefore needs to be assessed by what it measures and what new findings and decisions those measurements support. Our contribution lies in what HABIT characterizes and what behavioral patterns and real-user decisions it explains. Below, we (1) illustrate some potentially insightful information HABIT provides to users, model developers, and ML researchers, (2) present findings and empirical evidence of its practical utility, (3) distinguish this information from existing taxonomies and surface metadata, and (4) clarify how its axes were derived.

1. Concrete insights for users, model developers, and ML researchers.

HABIT makes differences in agents' working habits explicit and comparable. For users, these differences help anticipate how an assistant will collaborate: GPT-5.6 Sol proceeds without intermediate process narration in 99% of trajectories, compared with 60% for Claude Opus 5 (Figure 12). Users who prefer uninterrupted execution versus updates on ongoing work thus have information beyond an overall capability ranking.

For model developers, HABIT distinguishes specific behaviors that might otherwise be grouped under broad goals such as reliability or transparency. Claude Opus 5 visibly checks its initial result using tools in 79% of trajectories, compared with 36% for GPT-5.6 Sol; it also explicitly admits mistakes when revising more often (51% versus 23%; Figures 12 and 15). These are distinct aspects of collaboration: checking a result and acknowledging a correction. Measuring them separately helps developers identify what to change and assess whether an intervention changes the intended behavior.

For ML researchers, HABIT exposes exceptions to broad intuitions about model behavior. Although GPT-5.6, Qwen 3.7, and Gemma 4 exhibit strong within-family similarity, Claude's within-family similarity is lower than the average across all model pairs (0.77 versus 0.80; Section 3.2). Behavioral resemblance within a model family is therefore an empirical question rather than a reliable assumption. Together, these examples illustrate the concrete information that HABIT makes available for choosing, developing, and studying agents.

AuthorsOpenReview ↗

Rebuttal part [2/7] - Contribution beyond surface features and competence

A1. (continued)

2. What HABIT reveals and enables (Q1).

HABIT provides a common set of measurements for studying three questions: whether behavioral profiles help users select an agent that matches their preferences, whether agents accurately describe and can change their own behavior, and how behavioral similarity relates to model family and capability.

First, HABIT can provide practical help in selecting an agent that fits an individual user's preferences. If HABIT captures genuine collaborative habits rather than surface fingerprints, its benchmark profiles should predict which assistant a real user actually prefers and is satisfied with. To test this directly, we recruited 50 Prolific participants to complete 10 AgentHabit tasks with three agents (GPT-5.6 Sol, Claude Opus 5, and Gemini 3.7 Flash). After the conversations, each participant rated their satisfaction with each agent, ranked the three agents by which one they would use again, and ranked the behaviors on each of the 23 HABIT axes by their own preference. For each participant, we predicted the agent whose benchmark profile best matches the behaviors the participant said they prefer:

Participant's own preferencesAnother participant's preferencesUniform chance
Predicted agent is the participant's first choice86% (43/50)52%33%
Predicted order of all three agents is correct72% (36/50)38%17%

As the table shows, the predicted agent is the one the participant actually ranked first for 86% of participants, and the predicted order of all three agents is correct for 72%, compared with 52% and 38% when using another participant's preferences (p=0.001p=0.001 for both). User satisfaction shows the same pattern: participants rated the best-matching agent 6.18 out of 7 versus 4.84 for the worst-matching agent (p<0.001p<0.001), with the best-matching agent receiving the participant's highest satisfaction rating in 90% (45/50) of cases. A single competence ranking recommends the same agent to every user regardless of preference. HABIT instead connects differences in how agents collaborate to differences in what individual users prefer. These results provide empirical evidence that HABIT profiles can support personalized agent selection: matching benchmark profiles to a user's stated behavioral preferences identifies the agent that user actually prefers and finds more satisfying.

Second, models' descriptions of their habits differ from their execution, and explicit instructions change different behaviors to markedly different degrees. Raw turn counts, lengths, and tool counts do not directly measure our Section 4 findings: when queried directly on these 23 operational habits, models converge on a shared "ideal assistant" script that matches their own actual behavior for only 2 of 14 models, and explicit prompt instructions shift multi-turn Ask-user timing by +46 to +54 percentage points across families while moving sentence-level Self-correction framing and Certainty framing by less than 4 percentage points despite these latter behaviors requiring only wording changes rather than changes to turn structure or tool use (see A3). These findings identify both a gap between stated and enacted operational habits and a difference in steerability that is not captured by a simple distinction between surface wording and multi-turn behavior.

Third, behavioral similarity does not simply follow capability differences. In Section 3.2, frontier flagships from different labs, such as GPT-5.6 Sol and Claude Opus 5, diverge more in their behavioral profiles (0.28 TV distance) than an average model pair (0.20), whereas within GPT-5.6, Qwen 3.7, and Gemma 4 even the smallest, weakest sibling stays within 0.07 to 0.11 of its own flagship despite wide capability gaps. This pattern challenges an account in which behavioral profiles merely reflect a single ranking of general competence: much weaker siblings preserve their flagship's working style, while similarly capable frontier models take different collaborative approaches.

AuthorsOpenReview ↗

Rebuttal part [3/7] - Contribution beyond surface features and competence

3. How HABIT differs from VibeCheck, Act·ONOMY, and surface metadata (W1).

VibeCheck inductively discovers and quantifies qualitative differences between a pair of models' responses, including differences in tone, formatting, and other output characteristics, while Act·ONOMY logs which low-level subactions occur in an execution trace. In contrast, HABIT characterizes how an agent handles recurring multi-turn situations in everyday collaboration where multiple approaches are possible. The distinction is therefore in what is measured, rather than in claiming that inductive axis discovery or LLM-based annotation is itself new. HABIT provides a shared inventory of operational behaviors for comparing agents across models and tasks, including when they seek clarification, what role executed code plays, and how they revise their work after feedback.

Even axes whose topics sound familiar capture a much broader multi-turn specification than a binary check or raw action count. For instance, rather than simply checking whether an agent asks questions before answering, Ask-user timing tracks across the full interaction whether substantive clarifications occur only before the first deliverable, only after an initial attempt, in both phases, or not at all, covering both prose and tool calls while excluding generic polite closings. Likewise, rather than merely counting code invocations, Execution tool role specifies the functional role that executed code plays in producing the final result.

To test empirically whether surface statistics or action tags already capture these axes, we trained classifiers to predict HABIT labels from 13 judge-free trajectory metadata features (round count, deliverable and non-deliverable lengths, and web-search, code-execution, and ask_user counts before vs. after the first deliverable) and from Act·ONOMY's 46 subaction annotations across 408 held-out trajectories on unseen tasks:

PredictorsMean κ (all 23 axes)Mean κ (20 non-tool axes)Axes with κ <0.40
Trajectory metadata (13 features)0.240.1419 / 23
Act·ONOMY subactions (46 features)0.230.1920 / 23
Both combined (59 features)0.310.2217 / 23

Aside from the three tool-count and tool-timing axes (Execution tool use, Web search frequency, and Web search timing), which naturally track raw tool logs, agreement is substantially lower: across the remaining 20 axes, mean Cohen's κ is just 0.14 from metadata, 0.19 from Act·ONOMY, and 0.22 even with all 59 features combined, indicating that these features do not fully recover HABIT's behavioral distinctions.

Concrete situations show why surface counts and action tags miss this signal. When a user's request omits needed details, two trajectories can share the exact same turn count, response length, zero clarification calls, and identical Act·ONOMY reasoning tags, yet one agent explicitly flags its assumptions and placeholders while the other states unverified guesses as fact. When two runs execute Python the exact same number of times, raw call counts and Act·ONOMY's computation tag are identical, whereas HABIT distinguishes whether code was used to compute the core answer, verify an independently drafted result, render a user-facing file, or merely print prose. Similarly, after user criticism, two polite revisions of comparable length differ sharply in whether the agent openly owns its earlier mistake, explains the change without accepting fault, or silently replaces the output.

4. Clarifying how HABIT was derived (W1).

Rather than hand-picking 23 ad hoc rubrics in advance, three authors and three LLMs surfaced 716 observed differences across 408 trajectories and retained every difference that (i) concerns how the work is carried out rather than domain-specific answer correctness, (ii) recurs across task domains, and (iii) can be scored reliably from a trajectory with a fixed rubric and an explicit N/A trigger when the situation does not arise. Human curation was part of this process, but the resulting axes were derived systematically from observed behavioral differences and explicit selection criteria, rather than a predefined checklist.

We have added the live human study to Section 3.4 and Appendix Q, clarified our axis-selection criteria in Section 2.2, expanded the discussion of VibeCheck and Act·ONOMY in Section 5, and added the metadata and Act·ONOMY recoverability analyses to Section 2.3 and Appendix F.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Review Summary (2/2)

The reviewers pointed out several weaknesses and suggestions, which we have carefully addressed through detailed responses and additional experiments:

  • Validation with Real Users (Reviewers Fbxw, MdNj): Reviewers raised concerns that profiles collected with a user simulator may not transfer to real users (Fbxw), and that we had not shown they correspond to users’ actual preferences (MdNj). We clarified that humans validate every stage of the pipeline (tasks, taxonomy, judge, and simulator), and conducted a new study with 50 participants who completed AgentHABIT tasks with three commercial agents.
    • Each agent’s profile from real-user conversations remained closest to its own benchmark profile.
    • Benchmark profiles identified the agent each participant actually preferred for 86% of participants, against 33% by chance, and predicted their satisfaction.
    • Agent pairs with closer profiles were also perceived as more similar.
  • Number and Sampling of Tasks (Reviewers Fbxw, MdNj): Reviewers raised concerns that 86 tasks may be too few (Fbxw), and that using one task per subdomain, without weighting by real usage, may limit how well the profiles generalize (MdNj). We clarified that the 86 tasks follow directly from clustering real user requests into subdomains, and that 86 tasks are sufficient: profiles built from only 26 tasks already identify the same model 89.3% of the time. We also built a 120-task version that follows real usage more closely and reweighted the original tasks by real usage. In both cases, the profiles remained nearly unchanged.
  • Contribution beyond Surface Features and Prior Work (Reviewer EdhV): Reviewer EdhV raised concerns that HABIT applies a standard rubric-annotation pipeline with hand-curated rubrics, without going beyond surface trajectory statistics or prior work such as VibeCheck and Act·ONOMY, and that identifying models is a trivial bar for validation. We clarified that the axes were derived systematically from 716 observed differences under explicit selection criteria, and that HABIT differs from prior work in measuring how agents handle recurring multi-turn situations. We showed that trajectory metadata and Act·ONOMY annotations cannot predict most HABIT labels, so HABIT captures behaviors that surface statistics and action tags miss. The profiles also predict which agent each real user prefers, so they reflect differences that matter to users rather than stylistic fingerprints that merely identify models.
  • Behavioral Tendencies vs. Competence (Reviewer EdhV): Reviewer EdhV noted that some axes may measure instruction-following competence. Following the reviewer’s suggestion, we excluded the competence-related axes. User-preference prediction, family clustering, and profile stability all remained intact, indicating that our conclusions are not driven by differences in competence. We also showed that behavioral similarity does not follow capability: GPT-5.6 Sol and Claude Opus 5 differ more than an average model pair, while in three families even the weakest model stays close to its flagship.
  • Informativeness of Findings (Reviewer EdhV): Reviewer EdhV commented that some secondary findings were predictable. We revised the paper to lead with finer-grained findings: family resemblance is not universal, self-reports are systematically idealized, and steerability does not follow behavioral complexity. We replicated the steering result on a model from another family.
  • Fine-tuning Analysis (Reviewer EdhV): Reviewer EdhV commented that the fine-tuning setup was too limited to support claims of behavioral persistence. We revised our claim to a more modest one and replicated the experiment on two additional models, where the training objective was fitted but the behavioral profile still did not transfer.
  • Contribution as a Datasets and Benchmarks Paper (Reviewer EdhV): Reviewer EdhV's remaining concern is that the work is a descriptive profiling pipeline with limited methodological novelty. We respectfully note that this paper is submitted to the datasets and benchmarks area, and we believe its contribution lies in what it measures and enables: a validated taxonomy and benchmark whose profiles hold with real users, predict each user’s preferred agent, and reveal findings that surface statistics, prior taxonomies, and self-reports do not.
  • Related Work (Reviewer Fbxw): We added the suggested work on human–agent collaboration and clarified that ours complements it by profiling the agents rather than the users.
AuthorsOpenReview ↗

Review Summary (1/2)

Dear Area Chair and Reviewers,

We sincerely appreciate the time and effort you have dedicated to reviewing our paper. We are grateful that Reviewer MdNj acknowledged our detailed responses, that Reviewer Fbxw engaged further with our new human study, and that Reviewer EdhV confirmed that our new analyses address the technical questions on surface features, competence-related axes, and fine-tuning.

The following summarizes the contributions of our work as highlighted by the reviewers (with reviewers’ quotes in bold):

  • Reviewers found that our work addresses “an interesting question” beyond task success and “provides a systematic taxonomy to describe behavioral differences” (Reviewer MdNj).
  • The taxonomy is built with “explicit annotation rubrics” and validated on “held-out evaluation tasks” (Reviewer EdhV).
  • The “extensive” empirical study shows that behavioral profiles are “relatively stable”, with “meaningful differences across model family” (Reviewer MdNj). It is supported by a “wide collection of LLM evaluation” (Reviewer Fbxw).
  • The paper is “clearly written and well-organized” (Reviewer EdhV), and it is “easy to understand the motivation and study setup” (Reviewer Fbxw).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.