acceptodds
Under review as a conference paper at ICLR 2027

COGENT: A Modular Cognitive Architecture for Agentic User Simulation

Abstract

LLM-based user simulation can support interface design, accessibility research, and interactive-agent evaluation, but standard computer-use agents are optimized for task completion rather than reproducing how people perceive, reason, and act. We introduce COGENT (COGnitive agENT), a modular cognitive architecture for simulating user interaction with desktop interfaces. COGENT draws on established cognitive theories, such as Norman’s action cycle and ACT-R, to enable explain- able, intervenable, and modular user simulation by decomposing the simulation process into cognitive modules with explicit representations of perception, memory, user state, intentions, and action selection. We begin by presenting six interven- ability demonstrations, each tracing how a single edit to memory, a user attribute, or a processing module propagates through the cognitive pipeline to produce a behavioral change. We evaluate COGENT’s behavioral and attributional fidelity across three benchmarks – AgentNetBench, A11y-CUA, and SENSE-42 – finding that it achieves statistically significant fidelity gains over computer-use baselines in reproducing recorded human behavior and recovers the relational structure among human self-reported psychological states Together, these demos and evaluations demonstrate COGENT’s utility toward more faithful, explainable, intervenable, and modular LLM-based cognitive simulations.

Primary area:
applications to neuroscience & cognitive science
Keywords:
simulation, user simulation, agents, cognitive architecture, cognitive modeling, computer use, computer use agents

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FjxpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a cognitive architecture for simulating user interactions with desktop interfaces. The architecture uses cognitive theories such as Norman's action cycle and ACT-R to develop explainable, intervenable simulation. Evaluation on three benchmarks show gains over baseline computer use agents.

Critical strengths
  • Reasonable architecture based on cognitive theories that is explainable and intervenable.
  • Experiments show that it outperform the baselines.
Critical weaknesses
  • The baselines are weak and are not prompted to think before acting -- thinking before acting has become standard in agentic architectures.
  • Performance is still far from human performance and no clear analysis the weaknesses of current the cognitive architecture is provided.
Questions for authors
  • How would the baselines do if they follow the standard agentic architecture and are prompted to think before acting?
  • What are the situations where human perform well but the cognitive architectures fail?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Fjxp, part 1: baseline.

We agree that a deliberative agent baseline is an important comparison. Our original CUA-User baseline primarily emphasized simulating human behavior, rather than explicitly encouraging structured reasoning before acting, as our evaluation focuses on behavioral fidelity rather than task completion. However, we recognize that a standard agentic think-before-act approach may also improve behavioral fidelity, even without COGENT's cognitive decomposition.

Although unable to add additional baseline with exact think-before-act approach. We added a compute-matched baseline that uses persona prompt, generating five independent actions and finally decide based on majority voting. We reported its performance in updated table 2.

AgentTasksJS Distance ↓Sequence Distance ↓HNG ↓
COGENT430.272[0.239,0.320]\mathbf{0.272}_{[0.239, 0.320]}0.511[0.467,0.557]\mathbf{0.511}_{[0.467, 0.557]}2.020[1.572,2.546]2.020_{[1.572, 2.546]}
CUA430.370[0.330,0.432]0.370_{[0.330, 0.432]}0.595[0.527,0.660]0.595_{[0.527, 0.660]}2.198[1.761,2.682]2.198_{[1.761, 2.682]}
CUA-User430.418[0.362,0.493]0.418_{[0.362, 0.493]}0.612[0.546,0.681]0.612_{[0.546, 0.681]}2.153[1.814,2.506]2.153_{[1.814, 2.506]}
UA-User-CB430.389[0.351,0.437]0.389_{[0.351, 0.437]}0.532[0.478,0.580]0.532_{[0.478, 0.580]}1.890[1.577,2.280]1.890_{[1.577, 2.280]}
AuthorsOpenReview ↗

Response to Reviewer Fjxp, part 2: Human vs. cognitive architecture

We appreciate the interesting question raised by the reviewer. We see two interpretations: (A) how well humans could accomplish the main goal of our work, which is faithfully simulating the behavior of (other) humans; or (B) the situations in which human performance and cogntive architecture diverges, especially cases when human performs well and cognitive architecture does not.

For (A), to our knowledge, there are no prior empirical studies in which human participants are explicitly instructed to predict the discrete CUA-interface behaviors of other humans; so there is no available data to make a comparison against. The most relevant comparison with available data would be the human-normalized gap metric we introduce in sec 5 (eqn 1), with results in table 2.

For (B), we could only provide anecdotal evidences through examination of the resutls. One example is that human seems to able to forecast the outcome of predicted action better than computer or more confidently. In our experiment, we see that after clicking on the second page of a Google search results, agent often have trouble verifying that the newly displayed result is indeed the second page of search results (in reasoning step). It often rely on additional action (sometime check the url) to verify the outcome of the action.

Thanks for the rebuttal. I still do not find the work compelling, so will retain my score.

Reviewer_SmrFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors proposed COGENT, which decomposes desktop-user simulation into explicit perception, memory, state, control, and action processes. It evaluates intervention traces, human-action prediction, interaction sequences, and relationships among psychological attributes.

Critical strengths

I like the typed interfaces and traceable intervention paths. The human-reference evaluation is substantive: in AgentNetBench, exact-action agreement improves from 53.7% to 59.8% over CUA-USER. I also feel that the repeated module analyses and transparent reporting of negative attribute results are also strengths.

Critical weaknesses

However, I do have some concerns towards this work:

first, I think the additional inference is confounded with architectural organization. COGENT uses more cognition, candidate-generation, selection, and grounding calls than the direct baselines. While I appreciate the authors did acknowledge this, but a comparable-compute deliberative baseline is still needed to attribute the gain to the proposed decomposition.

Second, the aggregate ablations do not validate the central memory/state pathways. I found the Appendix D uses empty authored memory and attributes, inactive retrieval/appraisal, and no consolidation. Based on my understanding, these experiments therefore leave the active mechanisms largely untested. The surviving lagged-feedback effect is useful evidence, while the selected intervention demonstrations establish editability more clearly than correspondence with human intervention responses.

Finally, attribute covariance which I feel is insufficient evidence of calibrated cognitive dynamics. SENSE-42 uses one partial simulation run. The strong correlation ordering could partly reflect semantic relationships among attribute definitions; a simple appraisal baseline is missing. Magnitude error remains substantial, and neither tested agent attribute–behavior association is significant.

Questions for authors

Pls refer to the weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer SmrF part 1: Additional baseline.

We added a persona-prompt baseline under a matched model-call budget for A11y dataset and reported the results in updated table 2 (also reproduced below). The matched model-call use CUA-User persona prompt, and generate five independents actions using both the screenshots and past actions. The final action is decided by majority voting.

AgentTasksJS Distance ↓Sequence Distance ↓HNG ↓
COGENT430.272[0.239,0.320]\mathbf{0.272}_{[0.239, 0.320]}0.511[0.467,0.557]\mathbf{0.511}_{[0.467, 0.557]}2.020[1.572,2.546]2.020_{[1.572, 2.546]}
CUA430.370[0.330,0.432]0.370_{[0.330, 0.432]}0.595[0.527,0.660]0.595_{[0.527, 0.660]}2.198[1.761,2.682]2.198_{[1.761, 2.682]}
CUA-User430.418[0.362,0.493]0.418_{[0.362, 0.493]}0.612[0.546,0.681]0.612_{[0.546, 0.681]}2.153[1.814,2.506]2.153_{[1.814, 2.506]}
UA-User-CB430.389[0.351,0.437]0.389_{[0.351, 0.437]}0.532[0.478,0.580]0.532_{[0.478, 0.580]}1.890[1.577,2.280]1.890_{[1.577, 2.280]}
AuthorsOpenReview ↗

Response to Reviewer SmrF part 2: core architecture contribution to action fidelity

We thank the reviewer for this observation. We would distinguish two complementary contributions of COGENT: out-of-the-box behavioral fidelity under the default configuration and configurable cognitive simulation, where researchers can introduce memories, psychological attributes, and alternative processing mechanisms to study specific behavioral dynamics.

The default configuration intentionally contains no authored long-term memory or psychological attributes. These components are designed as intervention points for modeling different users and testing cognitive hypotheses, rather than mechanisms expected to improve generic action prediction without relevant configuration. Accordingly, Appendix D evaluates the active processing mechanisms in the default configuration, while Section 4 demonstrates how configuring memory, attributes, and modules can produce traceable behavioral changes.

AuthorsOpenReview ↗

Response to Reviewer SmrF part 3: SENSE-42 results

We thank the reviewer for these observations. Regarding the partial run, SENSE-42 contains many repeated task blocks. Our simulation includes 11 repetitions of each of five substantive task types (55 tasks), rather than the full 45 repetitions per type. We conducted a convergence analysis indicating that 11 repetitions are sufficient to obtain stable attribute-correlation estimates, and will include this analysis in the appendix F. We also plan additional independent simulation runs to assess robustness. We acknowledge that semantic relationships among attribute definitions may contribute to the observed correlations. However, importantly, these attributes were configured once in a task-agnostic manner, rather than tailored to individual tasks. The agent must independently interpret its evolving task experience and update attribute values throughout the interaction. For example, model tend to update frustration level based on task failures, and update mental demand when the task is complex. So the value updates does not soly come from maintaining the correlation of attributes based on semantic knowledge. We also acknowledge that the attribute-behavior association is statistically insignificant. We believe that these SENSE-42 results demonstrate promising relational agreement rather than fully calibrated cognitive dynamics at this stage.

The added appendix information is reproduced below:

We ran one simulated participant on the task sequence of human participant~001 and stopped after 11 blocks as an exploratory study, by which point the agent had completed 11 repetitions of each of the five substantive task types (55 tasks and 750 state updates). A sensitivity analysis of the 42 human participants supports this budget for an exploratory comparison. When we truncated each human session to its first 11 blocks and recomputed the cohort-level estimates, the ordering of the 21 attribute correlations closely matched the full-session ordering (ρs=0.940\rho_s=0.940). Thus, the broad human pattern was already visible at this point, although agreement between an individual's first 11 blocks and their own full session was lower (median ρs=0.688\rho_s=0.688). The single simulated trajectory showed agreement with the full human reference both across updates (ρs=0.860\rho_s=0.860) and across task means (ρs=0.830\rho_s=0.830).

Reviewer_SmrFOpenReview ↗

reply to rebuttal

Thanks for the response and additional analyses. The matched-call baseline helps address the compute concern, and the distinction between the default setup and configurable simulation is clearer.

My remaining concerns are mainly about validation. The correspondence between the configured memory/state interventions and human responses remains unclear. For SENSE-42, the shorter human-session analysis is useful, but it does not establish robustness across simulated runs. The simulation still relies on one trajectory, and a simple appraisal baseline is missing to assess how much the correlations reflect semantic relationships among attributes.

I appreciate the more cautious framing of these findings. Given these remaining gaps, I will keep my current assessment.

Reviewer_v2f6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduced COGENT, a LLM-based user simulation established on a modular cognitive architecture. Then it evaluates COGENT on three benchmarks to show its advantage over computer-user baselines.

Critical strengths

Evaluating a user simulator by fidelity to recorded human traces rather than task success is an interesting idea.

Typed information boundaries between modules give a concrete mechanism for tracing an intervention.

Critical weaknesses
  1. The task domain is not motivated. Why is computer use the right domain for a cognitive user simulator? These are a lot of possible tasks in the literature, which look more relevant to cognitive user simulation. Also, the abstract mentioned three applications (interface design, accessibility research, agent evaluation) but it appears no design or accessibility question is answered with the simulator. COGENT is also not used to evaluate another agent.

  2. The novelty of this work is incremental, and the closest prior work is missing. The architecture is fundamentally a perceive–think–act loop, which has been widely used. Modular cognitive architecture for LLM agents also exists in the existing work like CoALA, Generative Agents, Concordia, and Webb.

  3. LLM-agent user simulators for usability testing are not cited: SimUser (Xiang et al., CHI 2024) and UXAgent (Lu et al., 2025), which already has a persona generator, a dual-process reasoning loop, memory, and inspectable reasoning traces. The thesis “architecture, not a longer persona” is tested only against two minimal CUA prompts, never against an existing user-simulation architecture.

  4. The central causal claim is not supported by evidence. My main concern is the distance between what the paper claims and what its experiments can attribute. The claim is that organizing the simulated user as a cognitive architecture, rather than prompting a persona, is what produces the fidelity gains. The appendices argue against this. Appendix D is, to my reading, the most informative part of the paper. In the AgentNetBench setting, the memory and attribute stores are empty, so retrieval and appraisal never run. Of the components that do run, removing the attention mask, clearing working memory, and collapsing the executive to the bare objective change nothing obvious; the objective-only variant is in fact slightly better. Restricting candidates to interface affordances hurts, but Table 8 shows the loss comes almost entirely from adapter errors. The one component that matters is lagged outcome assessment, worth 3.7 points, and 106 of the 223 lost matches are stopping decisions. What moves exact-action agreement is then checking whether the last action worked and deciding when to stop. That is sound agent engineering, but it is not memory, perception, user state, or intention maintenance, which the paper’s argument rests on.

Questions for authors
  1. Why is desktop computer use the right domain for a cognitive user simulator, given that the paper’s own motivating evidence comes from conversational tasks? None of the three applications named in the abstract is executed.

  2. How does COGENT differ from existing modular cognitive architectures for LLM agents beyond its application to desktop interfaces?

  3. Why are SimUser (Xiang et al., CHI 2024) and UXAgent (Lu et al., 2025) not discussed, and can you compare against UXAgent or a persona prompt with the same model-call budget?

  4. Appendix D shows that memory, perception, and intention maintenance contribute nothing detectable and that lagged outcome assessment alone drives the gain; how do you reconcile this with attributing the gain to the cognitive organization? What differs between Table 1 (59.8%) and the Appendix D full configuration (57.3%)?

5 comments
AuthorsOpenReview ↗

Response to Reviewer v2f6 part 1: task domain

We thank the reviewer for raising these points. We chose desktop computer use for two reasons: Cognitive observability: Computer use involves a continuous perception–reasoning/planning–action–feedback loop, requiring users to perceive interfaces, retrieve knowledge, plan actions, and adapt to feedback. Both interaction traces and evolving environment states are observable, making it particularly suitable for studying how cognitive processes translate into situated behavior. Tractability: Compared with physical-world activities, computer use has a relatively well-defined action space while retaining substantial cognitive complexity.

Regarding the applications mentioned in the abstract, we acknowledge that these are potential downstream applications rather than fully validated use cases. However, our demonstrations illustrate relevant mechanisms. For example, Demo (a) shows how prior interface knowledge affects navigation behavior, consistent with findings on icon recognition and visual search (e.g.,: Wu et al., 2022). Demo (f) illustrates how altered color perception affects information-seeking strategies, highlighting implications for accessible interface design.

Wu, J., Jiao, D., Lu, C., Li, C., Huang, X., & Weng, S. (2022). How Do Older Adults Process Icons in Visual Search Tasks? The Combined Effects of Icon Type and Cognitive Aging. International journal of environmental research and public health, 19(8), 4525. https://doi.org/10.3390/ijerph19084525

AuthorsOpenReview ↗

Response to Reviewer v2f6 part 2: Novelty

We thank the reviewer for this comment. We agree that the four mentioned works (CoALA, Generative Agents, Concordia, and Webb et al.) are highly relevant. All four are already discussed in Section 2, where we explain how COGENT builds upon them. We acknowledge that the distinctions could be clearer and have added further clarification of COGENT's contributions at the beginning of Section 3.

CoALA provides a general framework for organizing LLM agents; Generative Agents emphasizes memory, reflection, and emergent social behavior; Concordia supports configurable agent-based simulation; and Webb et al. focus on planning through specialized cognitive modules.

Architecturally, COGENT explicitly distinguishes objective observations from subjective perception and situation comprehension, and separates cognitive action selection from constrained motor execution. Practically, these distinctions provide a more traceable and controllable framework for investigating how memory, psychological attributes, and cognitive processes influence human behavior. For example, as demonstrated in Section 4, manipulating specific cognitive mechanism produces specific behavioral difference, illustrating how cognitive differences can be linked to observable behavioral outcomes.

AuthorsOpenReview ↗

Response to Reviewer v2f6 part 3: SimUser and UXAgent, model-call budget baseline

We thank the reviewer for this comment. We agree that both works are relevant and have added them to the related work section.

SimUser (Xiang et al., 2024) combines user and application agents to simulate interactions and generate usability feedback. UXAgent (Lu et al., 2025) incorporates persona generation, memory, a dual-loop architecture for interaction and reflection, and inspectable reasoning traces. We recognize that these works have established important capabilities for LLM-based user simulation.

COGENT differs in its emphasis on explicit, independently intervenable cognitive processes. Specifically, COGENT (1) separates objective observations from subjective perception and situation comprehension, (2) models psychological-state updates and executive control as distinct processes, and (3) separates cognitive action selection from constrained motor execution. Unlike UXAgent, which primarily uses these cognitive capabilities to simulate and evaluate user interactions, COGENT enables targeted interventions on specific cognitive mechanisms and traces how these changes propagate to observable behavior. For example, our demonstrations examine how changes in prior interface knowledge or color perception influence navigation and information-seeking behavior.

We also agree that compute-matched comparisons would help isolate architectural benefits from additional inference. We added a persona-prompt baseline under a matched model-call budget for A11y dataset and reported the results in updated table 2 (also reproduced below). The matched model-call use CUA-User persona prompt, and generate five independents actions using both the screenshots and past actions. The final action is decided by majority voting.

AgentTasksJS Distance ↓Sequence Distance ↓HNG ↓
COGENT430.272[0.239,0.320]\mathbf{0.272}_{[0.239, 0.320]}0.511[0.467,0.557]\mathbf{0.511}_{[0.467, 0.557]}2.020[1.572,2.546]2.020_{[1.572, 2.546]}
CUA430.370[0.330,0.432]0.370_{[0.330, 0.432]}0.595[0.527,0.660]0.595_{[0.527, 0.660]}2.198[1.761,2.682]2.198_{[1.761, 2.682]}
CUA-User430.418[0.362,0.493]0.418_{[0.362, 0.493]}0.612[0.546,0.681]0.612_{[0.546, 0.681]}2.153[1.814,2.506]2.153_{[1.814, 2.506]}
UA-User-CB430.389[0.351,0.437]0.389_{[0.351, 0.437]}0.532[0.478,0.580]0.532_{[0.478, 0.580]}1.890[1.577,2.280]1.890_{[1.577, 2.280]}

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.