acceptodds
Under review as a conference paper at ICLR 2027

On the Behavioral Traits of LLM Agents

Abstract

Users increasingly describe different AI agents as distinct colleagues to work with. AI personality research aims to quantify such impressions by attributing human-like “traits” to agents. However, existing measures fall short: models’ self-reports (S-data) diverge from their actual behavior, while informant ratings from LLM judges (I-data) are costly to scale and cover few everyday scenarios. In this paper, we propose A-B-D to infer traits bottom-up from behavioral data (B-data), namely how agents act on their environment and communicate with users, as recorded in existing trajectories. From 345,667 real-world trajectories spanning 80 models, 12 tasks, and 50 harnesses, we extract 318 candidate features that capture both the actions an agent takes at each step (functional) and the language accompanying them (linguistic). We retain only features that show instance-level stability, cross-task consistency, and model discriminability. Factor analysis of the remaining 79 features uncovers six stable, model-attributable factors, two functional and four linguistic. For example, Kimi-K3 exhibits the most planfulness, whereas GPT-5.5 and GPT-5.6 are the least energetic. Moreover, we quantify the “knowledge–action gap” in the wild: these factors correlate only weakly with self-reported Big Five scores, even for conceptually matched pairs such as extroversion and energetic (r = 0.07, p = 0.58). Our work offers a new lens for understanding AI personality, with implications for users, developers, and researchers from both computer science and social science.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
agent behavioral analysis, personality and trait, large language model, computer-using agent, factor analysis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_b6tMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether stable behavioral traits can be recovered from agent trajectories. It curates 27,803 analysis trajectories from an initial collection of 345,667, constructs functional and linguistic features, and applies context adjustment and three reliability filters. Factor analysis of the retained 79 features yields six behavioral factors. The paper then relates these factors to self-reported Big Five scores, task success, and model release date.

Critical strengths
  1. The behavioral question has practical value. Task success alone does not explain why agents offer different collaboration experiences. Examining both action sequences and language adds a useful dimension to comparisons of how agents work.

  2. The feature-to-factor pipeline has clear selection criteria. Stability, cross-task consistency, and model discriminability address different sources of incidental variation. Separate functional and linguistic analyses help explain where the factors come from.

Critical weaknesses
  1. The link from behavioral differences to practical value needs more validation. The introduction motivates model selection, version tracking, and working-style matching, but the results mainly provide factors, rankings, and correlations. Frequently stating a plan does not directly show better collaboration or less rework. It would help to establish what these factors predict beyond task success and simple behavior counts. A focused preference or behavioral-outcome check would be more informative than additional profile examples.

  2. Some factors measure expression more directly than the behavior their names suggest. Planful rewards repeatedly stating a plan, while Responsible includes polite and reader-directed language. An agent can announce plans frequently but execute them poorly, or use few social words while behaving reliably. The operational definitions are helpful, but verbalized planning, following a plan, and social wording should remain distinct. Otherwise, the interpretation moves too quickly from textual features to behavioral traits. This matters for how readers use the resulting scores.

  3. Context adjustment relies on a separability assumption that needs checking. Section 2.3 removes additive task and harness offsets. Models may nevertheless respond differently to the same harness, and reasoning visibility can vary with model and logging source. A common offset does not remove these model-by-context interactions. This is especially important when functional annotation reads reasoning: not displaying analysis is different from not performing it. Comparisons with shared tasks, harnesses, and observable output channels would provide stronger evidence for model attribution.

  4. The scores are relative to the current model cohort, which complicates version tracking. Section 2.4 standardizes features across models and subsequently standardizes factor scores. If these quantities are refitted after adding new models, an unchanged model can receive a different score. That limits the proposed use for tracking behavioral changes across versions. The paper should explain whether feature filters, loadings, and normalization references are frozen for deployment, and how new models are projected onto the same coordinates.

Questions for authors

Are reasoning visibility and logging granularity comparable across models? Do the main loadings and rankings remain on a shared task-and-harness subset using only commonly observable actions and messages?

8 comments
AuthorsOpenReview ↗

Official Response to Reviewer b6tM (1/3)

We thank the reviewer for the detailed and insightful comments. We appreciate your recognition of the practical value of our research question and our principled feature selection criteria. Please find our point-by-point responses below, with all changes incorporated into the revised manuscript.

[W1] Focused preference check for personalization

[W1] The link from behavioral differences to practical value needs more validation. The introduction motivates model selection, version tracking, and working-style matching, but the results mainly provide factors, rankings, and correlations. Frequently stating a plan does not directly show better collaboration or less rework. It would help to establish what these factors predict beyond task success and simple behavior counts. A focused preference or behavioral-outcome check would be more informative than additional profile examples.

Thank you for suggesting a preference check to validate the practical value of our framework. Users with different personalities may prefer agents with different behavioral traits. To test this, we simulate 40 users whose personalities are taken from real participants of the BFI-2 data [1], following the persona design of [2], which has been shown to reproduce how a person's Big Five traits shape their judgments. Each user is shown pairs of agents that solved the same task under the same harness, and asked which agent they would rather have work on their tasks. Since both agents in a pair succeeded on the same task, the preference cannot come from task success, task difficulty, or the harness. For each user, we regress the choices on the differences in A-B-D factor scores between the two agents, which gives how strongly the user prefers agents high on each factor.

The detailed setting and results are in Appendix B.9 of the revised manuscript. We find that users with different personalities tend to prefer agents with different A-B-D profiles. For example, more conscientious users prefer less Talkative agents, and more agreeable users prefer more Planful and less Absolutist agents. As a comparison, we repeat the analysis with the models' self-reported Big Five scores in place of A-B-D. As shown in Figure 5, page 30, the users' preferences are more consistently tied to A-B-D (median absolute correlation 0.20) than to the self-reported Big Five (0.15), i.e., to how an agent behaves rather than to how the model describes its own personality. This suggests that the factors carry information about how an agent is received that task success does not capture, and supports working-style matching, i.e., personalization, as a potential application of A-B-D. We acknowledge that there could be a gap between simulated and real user preferences, and leave a study with real users for future work.

[1] The next Big Five Inventory (BFI-2). JPSP 2017.

[2] Designing AI-Agents with Personalities: A Psychometric Approach. Personality Science 2026.

[W2] Factor phrasing

[W2] Some factors measure expression more directly than the behavior their names suggest. Planful rewards repeatedly stating a plan, while Responsible includes polite and reader-directed language. An agent can announce plans frequently but execute them poorly, or use few social words while behaving reliably. The operational definitions are helpful, but verbalized planning, following a plan, and social wording should remain distinct. Otherwise, the interpretation moves too quickly from textual features to behavioral traits. This matters for how readers use the resulting scores.

We acknowledge that the operational definitions need to be precise for readers to use the resulting scores. As the Planful factor is mostly composed of planning related features as shown in Table 12, we would like to keep the factor name as planful. We do agree that verbalized planning and following a plan can be two different things, so we refine the definition for planful in Section 3.2 as: "A planful model states what it will do next at almost every step and returns to planning after each evaluation, reflection, or decision; its steps also mix more kinds of behavior. The factor captures how likely a model is to state its plan, but not whether it follows the proposed plan."

For the factor "Responsible", we agree that the term is misleading. As this factor is mainly composed of linguistic features of social and reader-directed wording, namely clout, social references, politeness, affiliation, prosocial words, and the pronouns "we" and "you", we have renamed it Affiliative, after the LIWC affiliation category that loads on it. In Section 3.2 of the updated version, we define it as: "An affiliative model writes as if working alongside the reader: we, you, please, thanks. The factor captures how much a model's wording makes the reader feel like a collaborator in the task, rather than an audience for its report." Any other factor naming suggestions will also be appreciated.

AuthorsOpenReview ↗

Official Response to Reviewer b6tM (2/3)

[Q1] Reasoning visibility

[Q1] Are reasoning visibility and logging granularity comparable across models?

We agree that reasoning visibility does differ across models, but we view this difference as part of the behavior A-B-D aims to measure instead of a confound. A-B-D characterizes an agent from behavioral data (B-data), i.e., what an observer can see the agent do and say. When the model's reasoning is invisible, users who work with that agent cannot see it either, so the hidden reasoning cannot shape how the agent comes across to them. Conversely, when a model exposes more reasoning, e.g., under a higher reasoning effort, users do see a different working style. Our factors therefore describe the agent as it is observed, which is the impression users have in mind when they describe different agents as distinct colleagues.

[W3, Q2] Reasoning visibility and harness effects

[W3] Context adjustment relies on a separability assumption that needs checking. Section 2.3 removes additive task and harness offsets. Models may nevertheless respond differently to the same harness, and reasoning visibility can vary with model and logging source. A common offset does not remove these model-by-context interactions. This is especially important when functional annotation reads reasoning: not displaying analysis is different from not performing it. Comparisons with shared tasks, harnesses, and observable output channels would provide stronger evidence for model attribution.

[Q2] Do the main loadings and rankings remain on a shared task-and-harness subset using only commonly observable actions and messages?

We agree that the additive context adjustment relies on a separability assumption, and that models may respond differently to the same harness. To further validate the effects of context adjustment and the model-harness interactional effects, we decompose the variance of the derived factors into the effects of task instance, harness, model, and model × harness. With the context adjustment, the harness explains only 1-4% of the variance in the factor scores, while the model explains 7-22%. The model × harness effect remains as expected, but is small in the derived factors: it explains 4-5% of the variance without and 4-8% with the context adjustment, and less than the model for five of the six factors.

FactorHarness (w/o adj.)Model (w/o adj.)Model × Harness (w/o adj.)Harness (w/ adj.)Model (w/ adj.)Model × Harness (w/ adj.)
Planful0.120.080.050.010.110.07
Deliberate0.050.080.040.020.100.05
Talkative0.380.040.050.010.070.08
Affiliative0.080.100.050.010.160.08
Energetic0.160.180.050.010.220.06
Absolutist0.060.070.040.040.070.04

We do not remove this interaction in the context adjustment, since how a model responds to a harness is part of the model's own behavior, and our factor scores aim to describe a model's overall behavior across the harnesses it is run with. Its effect on the model score is further limited: a model's score averages its trajectories over the harnesses it is run with, and while the model effect is the same under every harness, its deviations under different harnesses partly cancel. The experiment details and results are updated to the revised manuscript in Appendix B.6.

AuthorsOpenReview ↗

Official Response to Reviewer b6tM (3/3)

[W4] Analyze factors for new model

[W4] The scores are relative to the current model cohort, which complicates version tracking. Section 2.4 standardizes features across models and subsequently standardizes factor scores. If these quantities are refitted after adding new models, an unchanged model can receive a different score. That limits the proposed use for tracking behavioral changes across versions. The paper should explain whether feature filters, loadings, and normalization references are frozen for deployment, and how new models are projected onto the same coordinates.

We agree that it’s important to deploy the pipeline for new models. To validate whether the new models can be projected onto the same coordinates derived on the current corpus, we conducted a model hold-out experiment. We split all models into 8 folds of 10 models each and repeated the split with 3 random partitions, giving 24 folds. In each fold, we fit the entire pipeline on the remaining models, from the context adjustment to the factor analysis, and freeze all fitted quantities (means, standard deviations, offsets, and scoring weights). We then project the held-out models with the frozen quantities to obtain their factor scores. We use the factor scores derived from all models as the “ground truth” for the held-out models. This setting simulates the scenario when new incoming models use the frozen parameters in the pipeline to derive their behavioral factor scores.

Our results showed that the factor scores for the held-out model derived with the frozen parameters of the "training set" and the whole corpus has an average Pearson correlation of 0.91. This result demonstrated the capability for deploying our pipeline for analysing behavioral traits of new models with the frozen parameters fitted on the current corpus. The detailed experiment setting and results are updated to the revised manuscript in Appendix B.8.

Factorrρ
Planful0.92 ± 0.110.92 ± 0.10
Deliberate0.92 ± 0.150.85 ± 0.23
Talkative0.91 ± 0.150.89 ± 0.14
Affiliative0.95 ± 0.040.91 ± 0.07
Energetic0.92 ± 0.100.91 ± 0.09
Absolutist0.86 ± 0.190.80 ± 0.17
Average0.910.88
Reviewer_knb5OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This work proposed Agent Behavioral Data, a framework for identifying behavioral traits of LLM agents from their actual trajectories instead of self-reported personality questionnaires. The authors collect over 300k+ real-world trajectories that span across 80 models, 12 tasks, and 50 harnesses. The framework comprises 3 stages, feature extraction, feature selection, and factor analysis. They identify 6 stable, traits and factors, including 2 functional and 4 linguistic.

Critical strengths

This paper is clear, simple, interesting, and I like the overarching idea and topic.

  • I think being able to analyse personality traits more fruitfully from naturally occurring agent behavior and interactions allows us to better understand these AI tools in human-AI interaction. This is a more "in-the-wild" setup, although I understand that a potentially more impactful setup is to go beyond analysing these data from benchmarks.
  • The empirical investigation is also large, covering a lot of trajectories, and models. This is pretty convincing in terms of the setup.
  • I think the behavioral traits filtering is sound, simple and principled, and I also appreciate the interesting analysis in table 1; which can potentially inspire research into how the factors can correlate with task success. That would be interesting to see - how the personality etc. relates to task performance.
Critical weaknesses

Although I do like the paper's idea, I think there are a few weaknesses. There is one in particular (reliability of the behavioral annotations) which is the most pressing concern that prevents me from accepting the paper.

  • It seems that the underlying behavioral annotations, done by GPT-OSS-120B has weak reliability, when the agreement study with the human annotators are conducted. In Appendix A.3., the micro F1 seems to be just 0.45 for group and 0.30 for sub-action. This is rather concerning, because although the authors have gone through exceeding lengths to curate the data and setup the framework, the underlying LLM driven annotations may still be weak.
  • I think some of the framings and phrasings can be more precise. For instance, it is easy to correlate "personality" with the actual behavior/ performance of the model. An "responsible" model in this context "writes as if working alongside the reader", as quoted by the authors, but a first glance of the paper may imply that personality = responsible behavior (i.e. better guardrails etc.). The framing has to be more precise.
  • Although the paper examines harnesses and models, in the main text, the authors do not explicitly seperate the effects of harnesses from the models themselves. This may confound some of the results as we aren't sure whether some of the models' personalities are due to the harness or the base model (i.e. perhaps the model has a different personality altogether without the harness).
Questions for authors

Main concerns are stated above and should be addressed, particularly the annotation reliability, and the harness-model results separation etc.

4 comments
AuthorsOpenReview ↗

Official Response to Reviewer knb5 (1/2)

We thank the reviewer for the thoughtful and encouraging feedback. We appreciate your recognition of our in-the-wild research setup and the large-scale empirical study. We respond to each of your concerns below and have reflected the corresponding changes in the revised manuscript.

[W1] Human annotations for functional behavior

[W1] It seems that the underlying behavioral annotations, done by GPT-OSS-120B has weak reliability, when the agreement study with the human annotators are conducted. In Appendix A.3., the micro F1 seems to be just 0.45 for group and 0.30 for sub-action. This is rather concerning, because although the authors have gone through exceeding lengths to curate the data and setup the framework, the underlying LLM driven annotations may still be weak.

To validate the agreement between human annotators and LLM driven annotation, we increase both annotators and annotated trajectories. We now have two annotators perform annotations on the trajectory set including 20 trajectories and 182 steps in total. We also adjust the annotation procedure to strictly align with Act-onomy, with fixed text span from LLM and only ask the annotators to annotate the behaviors.

We describe the details of the new annotation process and results in appendix A.3. The resulting group-level kappa between the two annotators is 0.75, and between human and gpt-oss-120b is 0.51. This results suggests moderate reliability for our LLM-driven annotation pipeline.

Annotator pairMicro-F1 (Group)Micro-F1 (Sub-Action)Jaccard (Group)Jaccard (Sub-Action)κ (Group)
Human – Human0.790.670.740.610.75
GPT-OSS-120B – Human0.600.460.490.360.51

[W2] Separate model and harness effects

[W2] Although the paper examines harnesses and models, in the main text, the authors do not explicitly separate the effects of harnesses from the models themselves. This may confound some of the results as we aren't sure whether some of the models' personalities are due to the harness or the base model (i.e. perhaps the model has a different personality altogether without the harness).

We agree that it’s important to separate the effects of harnesses from the model themselves. In fact, the context adjustment in Section 2.3 already separates the harness effects from the model effects: before the factor analysis, we regress every trait feature on task and harness indicators jointly with a model random effect, and remove the estimated task and harness offsets.

To answer how much of the models' personalities are due to the harness or the base model, we further analyze the model and harness effects on the derived factors. We regress each trajectory's factor score on the task instance, harness and model indicators, and measure the share of each effect as the variance it explains beyond the task instance and the other effects. We also re-run the whole pipeline without the context adjustment step to validate the effectiveness of context adjustment. Without the context adjustment, the harness explains 5-38% of the variance in the factor scores, and more than the model for Planful and Talkative. With the context adjustment, the harness share falls to 1-4%, while the model share rises from 4-18% to 7-22%. These results suggest that the derived factors mainly reflect the models instead of the harnesses. The results also demonstrate that the context adjustment effectively removes the harness effects in the factor analysis. The experiment details and results are updated to the revised manuscript in Appendix B.6.

FactorHarness (w/o ctx-adj.)Model (w/o ctx-adj.)Harness (w/ ctx-adj.)Model (w/ ctx-adj.)
Planful0.120.080.010.11
Deliberate0.050.080.020.10
Talkative0.380.040.010.07
Affiliative0.080.100.010.16
Energetic0.160.180.010.22
Absolutist0.060.070.040.07
AuthorsOpenReview ↗

Official Response to Reviewer knb5 (2/2)

[W3] Factor phrasing

[W3] I think some of the framings and phrasings can be more precise. For instance, it is easy to correlate "personality" with the actual behavior/ performance of the model. An "responsible" model in this context "writes as if working alongside the reader", as quoted by the authors, but a first glance of the paper may imply that personality = responsible behavior (i.e. better guardrails etc.). The framing has to be more precise.

We agree that it’s important to have precise phrasing for the derived factors, and the factor "Responsible" can be misleading. As this factor is mainly composed of linguistic features of social and reader-directed wording, namely clout, social references, politeness, affiliation, prosocial words, and the pronouns "we" and "you", we have renamed it Affiliative, after the LIWC affiliation category that loads on it. In Section 3.2 of the updated version, we define it as: "An affiliative model writes as if working alongside the reader: we, you, please, thanks. The factor captures how much a model's wording makes the reader feel like a collaborator in the task, rather than an audience for its report." Any other factor naming suggestions will also be appreciated.

Reviewer_knb5OpenReview ↗

Reviewer's Response

I appreciate the response provided by the users. Given the context provided by the reviewers on the human-human annotator score, it does seem that the human-LLM agreement of 0.5 is more reasonable. I believe the paper is interesting, and could benefit the community, while I also acknowledge that AI conferences tend to be slightly on the more lenient side when it comes to kappa agreement.

Therefore, I shall raise my score to 3.

We sincerely appreciate your recognition of the value of our work. Thank you for your follow-up and for raising your score. We hope our work can offer a new perspective and inspire further research on understanding LLM agents through their actual behavior.

Reviewer_o8VLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces A-B-D, a framework for profiling LLM agents from recorded task behavior. Its corpus covers 80 models, 12 task categories and 50 harnesses. The method extracts functional and linguistic features, adjusts for task and harness effects, and screens features for reliability. Factor analysis yields two functional and four linguistic factors. The authors repeat the analysis with each task category removed to check stability. They also compare factor scores with Big Five self-reports, model release dates and task success. Agreement with self-reports is weak.

Critical strengths

[+] The corpus spans 80 models and many task and harness settings, giving a broad view of agent behavior.

[+] Context adjustment and reliability screening help separate model differences from task and harness effects.

[+] The functional factors remain consistent when the full analysis is repeated after removing each task category.

Critical weaknesses

[-] The functional labels are validated on only 56 steps, with group F1 of 0.45 and kappa of 0.30. This makes the behavioral interpretation uncertain.

[-] The BFI protocol is unspecified, and the personality matches lack independent validation. Measurement mismatch could explain the reported knowledge-action gap.

[-] The planfulness-success correlation uses an outcome-balanced sample. It does not establish that planfulness directly improves success.

Questions for authors
  1. Do the functional factors remain similar with more independent human annotations?
  2. How were BFI scores collected, and do the correlations persist after accounting for their reliability?
  3. Does the planfulness-success correlation hold on matched tasks and harnesses before outcome balancing?
5 comments
AuthorsOpenReview ↗

Official Response to Reviewer o8VL (1/2)

We thank the reviewer for their time and effort. We appreciate your recognition of our provided broad view of agent behaviors and designed analysis pipeline. We address your questions and concerns in the following and have updated the changes to the revised manuscript.

[W1 & Q1] Human annotations for functional behavior

[W1] The functional labels are validated on only 56 steps, with group F1 of 0.45 and kappa of 0.30. This makes the behavioral interpretation uncertain.

[Q1] Do the functional factors remain similar with more independent human annotations?

To validate the agreement between human annotators and LLM driven annotation, we increase both annotators and annotated trajectories. We now have two annotators perform annotations on the trajectory set including 20 trajectories and 182 steps in total. We also adjust the annotation procedure to strictly align with Act-onomy, with fixed text span from LLM and only ask the annotators to annotate the behaviors.

We describe the details of the new annotation process and results in appendix A.3. The resulting group-level kappa between the two annotators is 0.75, and between human and gpt-oss-120b is 0.51. This results suggests moderate reliability for our LLM-driven annotation pipeline.

Annotator pairMicro-F1 (Group)Micro-F1 (Sub-Action)Jaccard (Group)Jaccard (Sub-Action)κ (Group)
Human – Human0.790.670.740.610.75
GPT-OSS-120B – Human0.600.460.490.360.51

[W2.1 & Q2.1] BFI collection protocol

[W2.1] The BFI protocol is unspecified

[Q2.1] How were BFI scores collected

We administer the 44-item BFI with the implementation of PsychoBench [1], keeping all prompts and items unchanged. Models are instructed to rate each statement on a 5-point scale and answer the items in a multi-turn conversation, with previous answers kept in context. Each model completes the inventory twice, and we average the items of each domain and use the mean of the two runs. We have added the details to Appendix B.5.

[1] On the Humanity of Conversational AI: Evaluating the Psychological Portrayal of LLMs, ICLR 2024

[W2.2 & Q2.2] Statistical analysis of the knowledge-action gap

[W2.2] The personality matches lack independent validation. Measurement mismatch could explain the reported knowledge-action gap.

[Q2.2] Do the correlations persist after accounting for their reliability?

We conceptually match a A-B-D factor to a BFI domain only when the behavior that defines the factor directly expresses a characteristic in the definition of that domain. This yields four matches. The detailed description is in Section 3.3.

To account for the noise in BFI measurement when correlating it with our derived factors, we additionally repeat the BFI collection 10 times for each model, each time with a different random item order. For each run, we compute the correlation between every factor and every BFI domain. We report the 2.5-97.5 percentile range of these correlations across the 10 runs as the 95% CI. For all conceptually matched pairs, the CI includes 0. The CIs that exclude 0 come only from conceptually unrelated pairs, e.g., Planful-Neuroticism, Deliberate-Neuroticism. This result demonstrates that the correlation between a model's self-reported personality and its behavior does not persist after accounting for their reliability, validating the knowledge-action gap. We added these additional experiments and results to Appendix B.5.

OpennessConscientiousnessExtraversionAgreeablenessNeuroticism
Planful−0.15 [−0.23, −0.05]0.05 [−0.14, 0.20]−0.10 [−0.27, 0.14]0.07 [−0.09, 0.22]−0.17 [−0.29, −0.03]
Deliberate−0.10 [−0.29, 0.11]0.03 [−0.07, 0.19]0.11 [−0.14, 0.28]0.06 [−0.09, 0.19]−0.09 [−0.18, −0.03]
Talkative−0.15 [−0.31, −0.00]−0.05 [−0.17, 0.12]0.05 [−0.11, 0.17]−0.02 [−0.20, 0.13]0.01 [−0.07, 0.11]
Responsible0.07 [−0.07, 0.31]−0.19 [−0.37, 0.08]0.10 [−0.01, 0.31]−0.17 [−0.41, 0.01]0.27 [0.12, 0.40]
Energetic−0.02 [−0.13, 0.15]0.02 [−0.06, 0.17]0.02 [−0.28, 0.28]0.02 [−0.12, 0.17]0.05 [−0.06, 0.17]
Absolutist0.09 [−0.11, 0.22]0.00 [−0.20, 0.11]0.11 [−0.11, 0.27]0.09 [−0.09, 0.20]0.04 [−0.11, 0.16]
AuthorsOpenReview ↗

Official Response to Reviewer o8VL (2/2)

[W3 & Q3] Planfulness-success correlation without outcome-balancing

[W3] The planfulness-success correlation uses an outcome-balanced sample. It does not establish that planfulness directly improves success.

[Q3] Does the planfulness-success correlation hold on matched tasks and harnesses before outcome balancing?

We thank the reviewer for pointing out the outcome-balancing problem on the correlation between A-B-D factors and task success. We recompute each model's success rate on all its trajectories before outcome-balanced sampling, from the same harnesses and task types as in our analyzed subset. We found that the planfulness-success correlation remains significantly positive without outcome-balancing. The changes have been updated to Section 3.3 in our revised manuscript.

PlanfulnessDeliberationTalkativeAffiliativeEnergeticAbsolutist
Task success0.37***0.040.03-0.27*-0.66***-0.11
Release date0.42***-0.24*-0.36**-0.68***-0.53***-0.06
AuthorsOpenReview ↗

Official Response to Reviewer o8VL (follow up)

Hi Reviewer o8VL,

We sincerely thank you for reading our manuscript and giving us those valuable comments. As the discussion period is closing in 24 hours, we want to check in politely to ask whether our rebuttal has addressed your concerns, and whether there is anything further we could clarify or run in the remaining time.

We would especially like to highlight the new analysis for [W2.2 & Q2.2]. To account for the noise in BFI measurement, we repeated the BFI collection 10 times per model with shuffled item orders. For all four conceptually matched pairs, the 95% CIs of the factor–domain correlations across the 10 runs include 0, while the CIs that exclude 0 come only from conceptually unrelated pairs. The result shows that after accounting for the reliability of the BFI measurement, the mismatch between a model's self-reported personality and its behavior persists, supporting the knowledge–action gap. (Appendix B.5).

On [W1 & Q1], two annotators now label 182 steps from 20 trajectories, with a procedure strictly aligned to Act-onomy. The group-level κ is 0.75 between the two annotators and 0.51 between GPT-OSS-120B and the human annotators, suggesting moderate reliability of our annotation pipeline (Appendix A.3). On [W2.1 & Q2.1], we have specified the BFI protocol (44-item PsychoBench implementation; Appendix B.5). On [W3 & Q3], with each model's success rate before outcome-balanced sampling, Planful remains significantly correlated with task success (r = 0.37, p < 0.001; Section 3.3).

If any of these responses fall short, we would be very glad to hear it while we can still act on it. Thank you again for your time and care.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers and the AC

We truly appreciate all reviewers and the AC for their effort and time. We are encouraged to receive the following recognition:

  1. Important, practically motivated research question: profiling agents from how they actually behave (knb5, b6tM)
  2. Large-scale, in-the-wild study: 345,667 real-world trajectories spanning 80 models, 12 tasks, and 50 harnesses (o8VL, knb5)
  3. Sound and principled pipeline: context adjustment plus three filters that separate model differences from task and harness effects (o8VL, knb5, b6tM)
  4. Robust functional factors under leave-one-task-out re-analysis, and complementary functional vs. linguistic views of agent behavior (o8VL, b6tM)
  5. Clear presentation and interesting analyses, e.g., factor–success relations in Table 1 (o8VL, knb5, b6tM)

The comments have largely improved our study. As suggested, we have made the following improvements, with all revision marked blue in the manuscript:

  1. Expanded the human validation of functional annotations to two annotators on 182 steps from 20 trajectories, with a procedure strictly aligned to Act-onomy; human–human κ = 0.75 and GPT-OSS-120B–human κ = 0.51 (Appendix A.3; o8VL, knb5)
  2. Recomputed the factor–success correlation on each model's success rate before outcome-balanced sampling, on the same harnesses and task types (Section 3.3; o8VL)
  3. Added a variance decomposition into task instance, harness, model, and model × harness effects: context adjustment reduces the harness share from 5–38% to 1–4%, while the model explains 7–22% (Appendix B.6.1; knb5, b6tM)
  4. Recomputed the model factor scores on a shared task-and-harness subset (Terminal-Bench 2 under Terminus-2, 32 models): they agree with the full-corpus scores for all factors with high correlations (Appendix B.6.3; b6tM)
  5. Added a model hold-out experiment with all pipeline parameters frozen: held-out models projected onto the existing coordinates correlate at r = 0.91 with their full-corpus scores, supporting version tracking of new models (Appendix B.8; b6tM)
  6. Added a simulated user-preference study (40 personas from real BFI-2 participants, pairs of agents that both succeeded on the same task and harness, controlling for simple behavior counts): user preferences are more consistently tied to A-B-D factors (median |r| = 0.21) than to self-reported Big Five (0.14) (Appendix B.9; b6tM)

We thank again for all reviewers and the AC for their invaluable service to our community.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.