- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2(+1)
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduces TAMER, a benchmark for evaluating VLM agents on tasks that require retrieving, reasoning over information across multiple applications and over a persistent timeline. The benchmark constructs a synthetic enterprise environment containing employees, teams, projects, and historical interactions across multiple web applications. It provides more than 200 tasks covering both read and write operations, and evaluates several existing VLM-based agents in this environment.
Critical strengths
- The enterprise setting is interesting and provides a more structured and persistent environment than many existing web-agent benchmarks.
- The benchmark explicitly tests tasks requiring information to be gathered across multiple applications and timelines, which is a meaningful challenge for current agents.
- The results show that agents struggle substantially when tasks require coordination across multiple applications, suggesting that current agents have difficulties with long-horizon enterprise workflows.
Critical weaknesses
- The novelty relative to existing computer-use benchmarks is not sufficiently justified.
The benchmark appears conceptually similar to existing environments such as OSWorld: agents interact with applications through a computer interface and are evaluated on completing multi-step tasks. The main differences appear to be the persistent enterprise storyline, the synthetic organizational environment, and the emphasis on cross-application information. However, I am not convinced that the paper sufficiently establishes why these changes constitute a fundamentally new evaluation capability rather than a new domain or instantiation of existing computer-use tasks. For example, it would be useful to have a clearer comparison with OSWorld and other related benchmarks along dimensions such as:
- persistence of state and history,
- number of applications involved in a task,
- cross-application dependencies,
- temporal reasoning
More importantly, the authors should demonstrate what agents fail on in TAMER that would not be captured by existing benchmarks. Without such evidence, the main contribution risks appearing to be an enterprise-specific reimplementation of an existing computer-use evaluation paradigm.
- The presentation is very difficult to follow.
The presentation quality of the paper is below the standard I would expect for an academic conference paper. In particular, several figures are not up to academic standards: they are difficult to read (i.e. Figure 3, sub title, sub title?) and do not communicate the key results clearly. More importantly, regarding to the text, the paper also presents a large amount of implementation detail in every almost every section without sufficiently indicating which details are important for understanding the main contribution. As a result, I found it difficult to identify what information should be prioritized when reading the paper. The authors mention that AI was used in drafting the text. Regardless of how the text was produced, the manuscript would benefit substantially from further editing and restructuring. The presentation should be made more concise, the figures should be substantially improved, and the main contribution should be made much easier to identify.
- The empirical takeaway appears relatively similar to prior computer-use/VLM benchmarks.
The main empirical conclusion is that current agents struggle with complex, multi-step interaction and that performance decreases as tasks become more complicated. While this is an important observation, similar conclusions have already been demonstrated in prior computer-use benchmarks. The paper therefore needs to make a stronger case that the observed failures are specifically attributable to the enterprise, persistent, cross-application, and temporal aspects of TAMER, rather than simply being another manifestation of the general difficulty of computer-use tasks. An ablation or controlled comparison would be particularly useful here.
Questions for authors
See weakness
5 comments
3.2: Response to Critical Weakness 1 (Comment 2/3)
(Response 3.1, continued)
Convention: ✗ in "Unrequested writes checked" means the paper describes only checks of the requested outcome.
Judgment calls (†):
- OSWorld, history ✗: no simulated event history; the only time mechanism is live web values fetched at evaluation.
- OSWorld, Unrequested writes ✗: evaluators check only the requested outcome (convention above).
- WebArena, Observation: the environment offers an a11y tree, HTML and screenshots; the paper's baselines use the a11y tree.
- WebArena, history ∘: the paper does not name it, but 93 tasks contain explicit dates or periods; there is no shared dated history.
- WebArena, Written answers ∘: string answers are graded by exact, must-include or fuzzy match, not fact by fact; ✓ if the column only asks whether free-text answers are graded.
- VisualWebArena, history ∘: about 10% of intents use dates or recency ("most recent", a specific date); no shared history.
- WorkArena++, Shared persistent world ∘: one shared instance with standard data, but each task creates and deletes its own records (∘ vs ✗).
- WorkArena++, Unrequested writes ✗: validators query only the target records (convention above).
- TheAgentCompany, Shared persistent world ✓: one company shared by all tasks; its sites are reset before each task.
- TheAgentCompany, history ∘: some tasks are date-based (attendance, scheduling, sprint cycles); no dated history.
- TheAgentCompany, Unrequested writes ✗: evaluators check only that target checkpoints were reached (convention above).
- WorkBench, history ✓: a fixed current date, and tasks require relative-date reasoning over dated records.
- Gaia2, Apps per task: every scenario loads all 12 apps; neither paper gives a per-task app count or a multi-app share.
How TAMER differs. No single column is unique to it. API benchmarks such as AppWorld and Gaia2 already track time and unrequested writes, and TheAgentCompany has a shared company. However importantly TAMER is the only one that combines them in a computer-use setting. The agent operates several apps through screenshots alone, inside one organization whose five-month history every task shares. It sees only one employee's data, its written answers are graded fact by fact, and every app it could reach is checked for writes it was not asked to make (3.1.A, 3.1.B).
What agents fail on beyond computer use. Three controlled comparisons (new Section 5.4, change 2.1.C):
- Replacing screenshots with the accessibility tree, which removes visual grounding and coordinate clicking, does not help the two Claude models (Opus 5 31.2% → 29.7%; Sonnet 5 38.8% → 34.0%). Models limited by grounding improve sharply (GPT-5.6 Terra 9.0% → 40.9%).
- Given the reference facts with no environment, every model passes 65–91% of the 150 rubric-graded tasks, against 0–29% inside the environment. Most of the difficulty is therefore in finding and reconciling information across the organization.
- Cross-app coordination has an effect beyond task type. Within read-write tasks alone, Sonnet 5 passes 65.5% at one app, 45.5% at two and 17.9% at three, with either observation (2.2.A, Table 9).
Some failures are also invisible to a check of the target state alone. 6.2% of failed attempts (414 of 6,703) make an unrequested write, and for 1.6% (110) that write is their only deterministic error (Appendix). Read-only tasks, about a quarter of the benchmark, are graded only on the written answer. OSWorld reports a related single- versus multi-app gap (13.7% vs 6.6% for GPT-4V with set-of-mark prompting); TAMER makes app count a controlled variable over one shared organization. A persistence ablation that hands the agent the relevant records directly is planned for the camera-ready version.
Changes 3.1 (also 2.1.C, 2.2.A and 2.2.B)
3.1.A Related work: TheAgentCompany is not API-only
Where: Section 2, first paragraph, sentence starting "In all of these benchmarks".
Was:
In all of these benchmarks, the agent reaches the applications through an API rather than through their interfaces, and none binds it to a named employee inside a simulated organization.
Now:
In most of these benchmarks the agent calls the applications through an API. TheAgentCompany is the exception: its agent browses a simulated company's web apps through their page structure and talks to simulated colleagues. None of them restricts the agent to one employee's view of the organization.
3.1: Response to Critical Weakness 1 (Comment 3/3)
(Response 3.1, continued)
3.1.B Related work: comparison table
Where: Section 2, end of the paragraph on persistent worlds; new Table 1 (the table in Response 3.1).
Was:
We generate one organization and share it across tasks, so the same history is available to multiple tasks and any run can be repeated against this same state.
Now:
We generate one organization and share it across tasks, so the same history is available to multiple tasks and any run can be repeated against this same state. Table 1 compares TAMER with these benchmarks; no single property is unique to it, and what it adds is their combination in a setting where the agent operates the apps through screenshots alone.
[see Table 1 which is also the table in Response 3.1.]
3.1: Response to Critical Weakness 1 (Comment 1/3)
NOTE: Where comments overlap we decided to refer to our existing responses instead of copying them again somewhere else. Please consult these for the full response. E.g. for comment 3.3. we write "Changes 3.3: 2.1.C (new Section 5.4), 2.2.A (app-count paragraph), 2.2.B (Table 4)". All our responses are numbered by Reviewer, Comment/Response and Subresponse in this way. Every change in the revised PDF is highlighted and labelled in the margin: changes made in response to a specific comment carry that response's number (for example 1.2.A).
Comment 3.1
The benchmark appears conceptually similar to existing environments such as OSWorld: agents interact with applications through a computer interface and are evaluated on completing multi-step tasks. The main differences appear to be the persistent enterprise storyline, the synthetic organizational environment, and the emphasis on cross-application information. However, I am not convinced that the paper sufficiently establishes why these changes constitute a fundamentally new evaluation capability rather than a new domain or instantiation of existing computer-use tasks. For example, it would be useful to have a clearer comparison with OSWorld and other related benchmarks along dimensions such as:
More importantly, the authors should demonstrate what agents fail on in TAMER that would not be captured by existing benchmarks. Without such evidence, the main contribution risks appearing to be an enterprise-specific reimplementation of an existing computer-use evaluation paradigm.
Response 3.1
We added a comparison table against OSWorld, WebArena, VisualWebArena, WorkArena, TheAgentCompany and WorkBench on the dimensions suggested: persistence of state and history, applications per task, cross-application dependencies and temporal reasoning, plus scoped employee access, grading of written answers and checks for writes outside the target app. To show what TAMER measures that a computer-use benchmark does not, we ran all six models with an accessibility-tree observation instead of screenshots, which removes most of the difficulty of operating a GUI. The two strongest models do not improve (Opus 5: 31.2% to 29.7%; Sonnet 5: 38.8% to 34.0%), while models limited by visual grounding do (GPT-5.6 Terra: 9.0% to 40.9%). For the strongest models, then, TAMER's remaining difficulty is in finding, reconciling and writing up information across a persistent organization, not in computer use as such.
Changes 3.1
We add the comparison you suggested as Table 1 (3.1.B). ✓ yes, ∘ partly, ✗ no, – not stated in the paper, † = judgment call).