- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper asks whether stable behavioral traits can be recovered from agent trajectories. It curates 27,803 analysis trajectories from an initial collection of 345,667, constructs functional and linguistic features, and applies context adjustment and three reliability filters. Factor analysis of the retained 79 features yields six behavioral factors. The paper then relates these factors to self-reported Big Five scores, task success, and model release date.
Critical strengths
-
The behavioral question has practical value. Task success alone does not explain why agents offer different collaboration experiences. Examining both action sequences and language adds a useful dimension to comparisons of how agents work.
-
The feature-to-factor pipeline has clear selection criteria. Stability, cross-task consistency, and model discriminability address different sources of incidental variation. Separate functional and linguistic analyses help explain where the factors come from.
Critical weaknesses
-
The link from behavioral differences to practical value needs more validation. The introduction motivates model selection, version tracking, and working-style matching, but the results mainly provide factors, rankings, and correlations. Frequently stating a plan does not directly show better collaboration or less rework. It would help to establish what these factors predict beyond task success and simple behavior counts. A focused preference or behavioral-outcome check would be more informative than additional profile examples.
-
Some factors measure expression more directly than the behavior their names suggest. Planful rewards repeatedly stating a plan, while Responsible includes polite and reader-directed language. An agent can announce plans frequently but execute them poorly, or use few social words while behaving reliably. The operational definitions are helpful, but verbalized planning, following a plan, and social wording should remain distinct. Otherwise, the interpretation moves too quickly from textual features to behavioral traits. This matters for how readers use the resulting scores.
-
Context adjustment relies on a separability assumption that needs checking. Section 2.3 removes additive task and harness offsets. Models may nevertheless respond differently to the same harness, and reasoning visibility can vary with model and logging source. A common offset does not remove these model-by-context interactions. This is especially important when functional annotation reads reasoning: not displaying analysis is different from not performing it. Comparisons with shared tasks, harnesses, and observable output channels would provide stronger evidence for model attribution.
-
The scores are relative to the current model cohort, which complicates version tracking. Section 2.4 standardizes features across models and subsequently standardizes factor scores. If these quantities are refitted after adding new models, an unchanged model can receive a different score. That limits the proposed use for tracking behavioral changes across versions. The paper should explain whether feature filters, loadings, and normalization references are frozen for deployment, and how new models are projected onto the same coordinates.
Questions for authors
Are reasoning visibility and logging granularity comparable across models? Do the main loadings and rankings remain on a shared task-and-harness subset using only commonly observable actions and messages?
8 comments
Official Response to Reviewer b6tM (2/3)
[Q1] Reasoning visibility
[Q1] Are reasoning visibility and logging granularity comparable across models?
We agree that reasoning visibility does differ across models, but we view this difference as part of the behavior A-B-D aims to measure instead of a confound. A-B-D characterizes an agent from behavioral data (B-data), i.e., what an observer can see the agent do and say. When the model's reasoning is invisible, users who work with that agent cannot see it either, so the hidden reasoning cannot shape how the agent comes across to them. Conversely, when a model exposes more reasoning, e.g., under a higher reasoning effort, users do see a different working style. Our factors therefore describe the agent as it is observed, which is the impression users have in mind when they describe different agents as distinct colleagues.
[W3, Q2] Reasoning visibility and harness effects
[W3] Context adjustment relies on a separability assumption that needs checking. Section 2.3 removes additive task and harness offsets. Models may nevertheless respond differently to the same harness, and reasoning visibility can vary with model and logging source. A common offset does not remove these model-by-context interactions. This is especially important when functional annotation reads reasoning: not displaying analysis is different from not performing it. Comparisons with shared tasks, harnesses, and observable output channels would provide stronger evidence for model attribution.
[Q2] Do the main loadings and rankings remain on a shared task-and-harness subset using only commonly observable actions and messages?
We agree that the additive context adjustment relies on a separability assumption, and that models may respond differently to the same harness. To further validate the effects of context adjustment and the model-harness interactional effects, we decompose the variance of the derived factors into the effects of task instance, harness, model, and model × harness. With the context adjustment, the harness explains only 1-4% of the variance in the factor scores, while the model explains 7-22%. The model × harness effect remains as expected, but is small in the derived factors: it explains 4-5% of the variance without and 4-8% with the context adjustment, and less than the model for five of the six factors.
| Factor | Harness (w/o adj.) | Model (w/o adj.) | Model × Harness (w/o adj.) | Harness (w/ adj.) | Model (w/ adj.) | Model × Harness (w/ adj.) |
|---|---|---|---|---|---|---|
| Planful | 0.12 | 0.08 | 0.05 | 0.01 | 0.11 | 0.07 |
| Deliberate | 0.05 | 0.08 | 0.04 | 0.02 | 0.10 | 0.05 |
| Talkative | 0.38 | 0.04 | 0.05 | 0.01 | 0.07 | 0.08 |
| Affiliative | 0.08 | 0.10 | 0.05 | 0.01 | 0.16 | 0.08 |
| Energetic | 0.16 | 0.18 | 0.05 | 0.01 | 0.22 | 0.06 |
| Absolutist | 0.06 | 0.07 | 0.04 | 0.04 | 0.07 | 0.04 |
We do not remove this interaction in the context adjustment, since how a model responds to a harness is part of the model's own behavior, and our factor scores aim to describe a model's overall behavior across the harnesses it is run with. Its effect on the model score is further limited: a model's score averages its trajectories over the harnesses it is run with, and while the model effect is the same under every harness, its deviations under different harnesses partly cancel. The experiment details and results are updated to the revised manuscript in Appendix B.6.
Official Response to Reviewer b6tM (3/3)
[W4] Analyze factors for new model
[W4] The scores are relative to the current model cohort, which complicates version tracking. Section 2.4 standardizes features across models and subsequently standardizes factor scores. If these quantities are refitted after adding new models, an unchanged model can receive a different score. That limits the proposed use for tracking behavioral changes across versions. The paper should explain whether feature filters, loadings, and normalization references are frozen for deployment, and how new models are projected onto the same coordinates.
We agree that it’s important to deploy the pipeline for new models. To validate whether the new models can be projected onto the same coordinates derived on the current corpus, we conducted a model hold-out experiment. We split all models into 8 folds of 10 models each and repeated the split with 3 random partitions, giving 24 folds. In each fold, we fit the entire pipeline on the remaining models, from the context adjustment to the factor analysis, and freeze all fitted quantities (means, standard deviations, offsets, and scoring weights). We then project the held-out models with the frozen quantities to obtain their factor scores. We use the factor scores derived from all models as the “ground truth” for the held-out models. This setting simulates the scenario when new incoming models use the frozen parameters in the pipeline to derive their behavioral factor scores.
Our results showed that the factor scores for the held-out model derived with the frozen parameters of the "training set" and the whole corpus has an average Pearson correlation of 0.91. This result demonstrated the capability for deploying our pipeline for analysing behavioral traits of new models with the frozen parameters fitted on the current corpus. The detailed experiment setting and results are updated to the revised manuscript in Appendix B.8.
| Factor | r | ρ |
|---|---|---|
| Planful | 0.92 ± 0.11 | 0.92 ± 0.10 |
| Deliberate | 0.92 ± 0.15 | 0.85 ± 0.23 |
| Talkative | 0.91 ± 0.15 | 0.89 ± 0.14 |
| Affiliative | 0.95 ± 0.04 | 0.91 ± 0.07 |
| Energetic | 0.92 ± 0.10 | 0.91 ± 0.09 |
| Absolutist | 0.86 ± 0.19 | 0.80 ± 0.17 |
| Average | 0.91 | 0.88 |
Official Response to Reviewer b6tM (1/3)
We thank the reviewer for the detailed and insightful comments. We appreciate your recognition of the practical value of our research question and our principled feature selection criteria. Please find our point-by-point responses below, with all changes incorporated into the revised manuscript.
[W1] Focused preference check for personalization
Thank you for suggesting a preference check to validate the practical value of our framework. Users with different personalities may prefer agents with different behavioral traits. To test this, we simulate 40 users whose personalities are taken from real participants of the BFI-2 data [1], following the persona design of [2], which has been shown to reproduce how a person's Big Five traits shape their judgments. Each user is shown pairs of agents that solved the same task under the same harness, and asked which agent they would rather have work on their tasks. Since both agents in a pair succeeded on the same task, the preference cannot come from task success, task difficulty, or the harness. For each user, we regress the choices on the differences in A-B-D factor scores between the two agents, which gives how strongly the user prefers agents high on each factor.
The detailed setting and results are in Appendix B.9 of the revised manuscript. We find that users with different personalities tend to prefer agents with different A-B-D profiles. For example, more conscientious users prefer less Talkative agents, and more agreeable users prefer more Planful and less Absolutist agents. As a comparison, we repeat the analysis with the models' self-reported Big Five scores in place of A-B-D. As shown in Figure 5, page 30, the users' preferences are more consistently tied to A-B-D (median absolute correlation 0.20) than to the self-reported Big Five (0.15), i.e., to how an agent behaves rather than to how the model describes its own personality. This suggests that the factors carry information about how an agent is received that task success does not capture, and supports working-style matching, i.e., personalization, as a potential application of A-B-D. We acknowledge that there could be a gap between simulated and real user preferences, and leave a study with real users for future work.
[1] The next Big Five Inventory (BFI-2). JPSP 2017.
[2] Designing AI-Agents with Personalities: A Psychometric Approach. Personality Science 2026.
[W2] Factor phrasing
We acknowledge that the operational definitions need to be precise for readers to use the resulting scores. As the Planful factor is mostly composed of planning related features as shown in Table 12, we would like to keep the factor name as planful. We do agree that verbalized planning and following a plan can be two different things, so we refine the definition for planful in Section 3.2 as: "A planful model states what it will do next at almost every step and returns to planning after each evaluation, reflection, or decision; its steps also mix more kinds of behavior. The factor captures how likely a model is to state its plan, but not whether it follows the proposed plan."
For the factor "Responsible", we agree that the term is misleading. As this factor is mainly composed of linguistic features of social and reader-directed wording, namely clout, social references, politeness, affiliation, prosocial words, and the pronouns "we" and "you", we have renamed it Affiliative, after the LIWC affiliation category that loads on it. In Section 3.2 of the updated version, we define it as: "An affiliative model writes as if working alongside the reader: we, you, please, thanks. The factor captures how much a model's wording makes the reader feel like a collaborator in the task, rather than an audience for its report." Any other factor naming suggestions will also be appreciated.