- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a cognitive architecture for simulating user interactions with desktop interfaces. The architecture uses cognitive theories such as Norman's action cycle and ACT-R to develop explainable, intervenable simulation. Evaluation on three benchmarks show gains over baseline computer use agents.
Critical strengths
- Reasonable architecture based on cognitive theories that is explainable and intervenable.
- Experiments show that it outperform the baselines.
Critical weaknesses
- The baselines are weak and are not prompted to think before acting -- thinking before acting has become standard in agentic architectures.
- Performance is still far from human performance and no clear analysis the weaknesses of current the cognitive architecture is provided.
Questions for authors
- How would the baselines do if they follow the standard agentic architecture and are prompted to think before acting?
- What are the situations where human perform well but the cognitive architectures fail?
3 comments
Response to Reviewer Fjxp, part 2: Human vs. cognitive architecture
We appreciate the interesting question raised by the reviewer. We see two interpretations: (A) how well humans could accomplish the main goal of our work, which is faithfully simulating the behavior of (other) humans; or (B) the situations in which human performance and cogntive architecture diverges, especially cases when human performs well and cognitive architecture does not.
For (A), to our knowledge, there are no prior empirical studies in which human participants are explicitly instructed to predict the discrete CUA-interface behaviors of other humans; so there is no available data to make a comparison against. The most relevant comparison with available data would be the human-normalized gap metric we introduce in sec 5 (eqn 1), with results in table 2.
For (B), we could only provide anecdotal evidences through examination of the resutls. One example is that human seems to able to forecast the outcome of predicted action better than computer or more confidently. In our experiment, we see that after clicking on the second page of a Google search results, agent often have trouble verifying that the newly displayed result is indeed the second page of search results (in reasoning step). It often rely on additional action (sometime check the url) to verify the outcome of the action.
Thanks for the rebuttal. I still do not find the work compelling, so will retain my score.
Response to Reviewer Fjxp, part 1: baseline.
We agree that a deliberative agent baseline is an important comparison. Our original CUA-User baseline primarily emphasized simulating human behavior, rather than explicitly encouraging structured reasoning before acting, as our evaluation focuses on behavioral fidelity rather than task completion. However, we recognize that a standard agentic think-before-act approach may also improve behavioral fidelity, even without COGENT's cognitive decomposition.
Although unable to add additional baseline with exact think-before-act approach. We added a compute-matched baseline that uses persona prompt, generating five independent actions and finally decide based on majority voting. We reported its performance in updated table 2.