acceptodds
Under review as a conference paper at ICLR 2027

TAMER: Testing Agents in complex Multi-tool Enterprise knowledge worker Roles

Abstract

We introduce TAMER, a benchmark for evaluating vision-language-model agents on realistic knowledge work in an enterprise setting. TAMER contains 238 tasks requiring an agent to coordinate actions in a simulated environment consisting of eight web apps it observes only as screenshots and operates through coordinate- based tools, grounded in a synthetic organization of 54 employees across 19 teams and 21 projects spanning five months of interlinked history. Each task ships a hybrid verifier combining deterministic scripts with LLM-judged rubrics. Measured difficulty splits the set into 24 easy, 98 medium, and 116 hard tasks. Across six models, the highest per-attempt accuracy on the full set is 42.3 ± 3.6%, falling to 6.0% on the hard subset. Pass rate also falls with the number of web apps a task spans.

Primary area:
datasets and benchmarks
Keywords:
AI evaluation, agentic AI, benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eeNpOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2(+1)
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces TAMER, a benchmark for evaluating VLM agents on tasks that require retrieving, reasoning over information across multiple applications and over a persistent timeline. The benchmark constructs a synthetic enterprise environment containing employees, teams, projects, and historical interactions across multiple web applications. It provides more than 200 tasks covering both read and write operations, and evaluates several existing VLM-based agents in this environment.

Critical strengths
  • The enterprise setting is interesting and provides a more structured and persistent environment than many existing web-agent benchmarks.
  • The benchmark explicitly tests tasks requiring information to be gathered across multiple applications and timelines, which is a meaningful challenge for current agents.
  • The results show that agents struggle substantially when tasks require coordination across multiple applications, suggesting that current agents have difficulties with long-horizon enterprise workflows.
Critical weaknesses
  1. The novelty relative to existing computer-use benchmarks is not sufficiently justified.

The benchmark appears conceptually similar to existing environments such as OSWorld: agents interact with applications through a computer interface and are evaluated on completing multi-step tasks. The main differences appear to be the persistent enterprise storyline, the synthetic organizational environment, and the emphasis on cross-application information. However, I am not convinced that the paper sufficiently establishes why these changes constitute a fundamentally new evaluation capability rather than a new domain or instantiation of existing computer-use tasks. For example, it would be useful to have a clearer comparison with OSWorld and other related benchmarks along dimensions such as:

  • persistence of state and history,
  • number of applications involved in a task,
  • cross-application dependencies,
  • temporal reasoning

More importantly, the authors should demonstrate what agents fail on in TAMER that would not be captured by existing benchmarks. Without such evidence, the main contribution risks appearing to be an enterprise-specific reimplementation of an existing computer-use evaluation paradigm.

  1. The presentation is very difficult to follow.

The presentation quality of the paper is below the standard I would expect for an academic conference paper. In particular, several figures are not up to academic standards: they are difficult to read (i.e. Figure 3, sub title, sub title?) and do not communicate the key results clearly. More importantly, regarding to the text, the paper also presents a large amount of implementation detail in every almost every section without sufficiently indicating which details are important for understanding the main contribution. As a result, I found it difficult to identify what information should be prioritized when reading the paper. The authors mention that AI was used in drafting the text. Regardless of how the text was produced, the manuscript would benefit substantially from further editing and restructuring. The presentation should be made more concise, the figures should be substantially improved, and the main contribution should be made much easier to identify.

  1. The empirical takeaway appears relatively similar to prior computer-use/VLM benchmarks.

The main empirical conclusion is that current agents struggle with complex, multi-step interaction and that performance decreases as tasks become more complicated. While this is an important observation, similar conclusions have already been demonstrated in prior computer-use benchmarks. The paper therefore needs to make a stronger case that the observed failures are specifically attributable to the enterprise, persistent, cross-application, and temporal aspects of TAMER, rather than simply being another manifestation of the general difficulty of computer-use tasks. An ablation or controlled comparison would be particularly useful here.

Questions for authors

See weakness

5 comments
AuthorsOpenReview ↗

3.1: Response to Critical Weakness 1 (Comment 1/3)

NOTE: Where comments overlap we decided to refer to our existing responses instead of copying them again somewhere else. Please consult these for the full response. E.g. for comment 3.3. we write "Changes 3.3: 2.1.C (new Section 5.4), 2.2.A (app-count paragraph), 2.2.B (Table 4)". All our responses are numbered by Reviewer, Comment/Response and Subresponse in this way. Every change in the revised PDF is highlighted and labelled in the margin: changes made in response to a specific comment carry that response's number (for example 1.2.A).

Comment 3.1

The benchmark appears conceptually similar to existing environments such as OSWorld: agents interact with applications through a computer interface and are evaluated on completing multi-step tasks. The main differences appear to be the persistent enterprise storyline, the synthetic organizational environment, and the emphasis on cross-application information. However, I am not convinced that the paper sufficiently establishes why these changes constitute a fundamentally new evaluation capability rather than a new domain or instantiation of existing computer-use tasks. For example, it would be useful to have a clearer comparison with OSWorld and other related benchmarks along dimensions such as:

  • persistence of state and history,
  • number of applications involved in a task,
  • cross-application dependencies,
  • temporal reasoning

More importantly, the authors should demonstrate what agents fail on in TAMER that would not be captured by existing benchmarks. Without such evidence, the main contribution risks appearing to be an enterprise-specific reimplementation of an existing computer-use evaluation paradigm.

Response 3.1

We added a comparison table against OSWorld, WebArena, VisualWebArena, WorkArena, TheAgentCompany and WorkBench on the dimensions suggested: persistence of state and history, applications per task, cross-application dependencies and temporal reasoning, plus scoped employee access, grading of written answers and checks for writes outside the target app. To show what TAMER measures that a computer-use benchmark does not, we ran all six models with an accessibility-tree observation instead of screenshots, which removes most of the difficulty of operating a GUI. The two strongest models do not improve (Opus 5: 31.2% to 29.7%; Sonnet 5: 38.8% to 34.0%), while models limited by visual grounding do (GPT-5.6 Terra: 9.0% to 40.9%). For the strongest models, then, TAMER's remaining difficulty is in finding, reconciling and writing up information across a persistent organization, not in computer use as such.

Changes 3.1

We add the comparison you suggested as Table 1 (3.1.B). ✓ yes, ∘ partly, ✗ no, – not stated in the paper, † = judgment call).

BenchmarkObservationApps per taskShared persistent worldCohesive environment historyWritten answers gradedUnrequested writes checkedReference
OSWorldscreenshot / a11y1; 27% multi-app✗✗†✗✗†Xie et al., NeurIPS 2024
WebArenaa11y / HTML / screenshot†1; 6% cross-site∘∘†∘†✗Zhou et al., ICLR 2024
VisualWebArenascreenshot + a11y1; 5% multi-site∘∘†∘✗Koh et al., ACL 2024
WorkArena++a11y + screenshot1 platform∘†∘∘✗†Boisvert et al., NeurIPS 2024; Drouin et al., ICML 2024
TheAgentCompanya11y + screenshot + shell1–4 of 4 sites; 35% ≥2✓†∘†∘✗†Xu et al., 2024 (arXiv:2412.14161)
WorkBenchAPI1–5; 30% multi✓✓†✗✓Styles et al., COLM 2024
AppWorldAPI (code)1–6; avg 1.8✓✓∘✓Trivedi et al., ACL 2024
Gaia2API1+ of 12; multi-app share not stated†∘✓✓✓Froger et al., ICLR 2026
TAMERscreenshot only1–8; 52% ≥2✓ one org, 5 months✓✓ per fact✓ every appthis paper
AuthorsOpenReview ↗

3.2: Response to Critical Weakness 1 (Comment 2/3)

(Response 3.1, continued)

Convention: ✗ in "Unrequested writes checked" means the paper describes only checks of the requested outcome.

Judgment calls (†):

  • OSWorld, history ✗: no simulated event history; the only time mechanism is live web values fetched at evaluation.
  • OSWorld, Unrequested writes ✗: evaluators check only the requested outcome (convention above).
  • WebArena, Observation: the environment offers an a11y tree, HTML and screenshots; the paper's baselines use the a11y tree.
  • WebArena, history ∘: the paper does not name it, but 93 tasks contain explicit dates or periods; there is no shared dated history.
  • WebArena, Written answers ∘: string answers are graded by exact, must-include or fuzzy match, not fact by fact; ✓ if the column only asks whether free-text answers are graded.
  • VisualWebArena, history ∘: about 10% of intents use dates or recency ("most recent", a specific date); no shared history.
  • WorkArena++, Shared persistent world ∘: one shared instance with standard data, but each task creates and deletes its own records (∘ vs ✗).
  • WorkArena++, Unrequested writes ✗: validators query only the target records (convention above).
  • TheAgentCompany, Shared persistent world ✓: one company shared by all tasks; its sites are reset before each task.
  • TheAgentCompany, history ∘: some tasks are date-based (attendance, scheduling, sprint cycles); no dated history.
  • TheAgentCompany, Unrequested writes ✗: evaluators check only that target checkpoints were reached (convention above).
  • WorkBench, history ✓: a fixed current date, and tasks require relative-date reasoning over dated records.
  • Gaia2, Apps per task: every scenario loads all 12 apps; neither paper gives a per-task app count or a multi-app share.

How TAMER differs. No single column is unique to it. API benchmarks such as AppWorld and Gaia2 already track time and unrequested writes, and TheAgentCompany has a shared company. However importantly TAMER is the only one that combines them in a computer-use setting. The agent operates several apps through screenshots alone, inside one organization whose five-month history every task shares. It sees only one employee's data, its written answers are graded fact by fact, and every app it could reach is checked for writes it was not asked to make (3.1.A, 3.1.B).

What agents fail on beyond computer use. Three controlled comparisons (new Section 5.4, change 2.1.C):

  1. Replacing screenshots with the accessibility tree, which removes visual grounding and coordinate clicking, does not help the two Claude models (Opus 5 31.2% → 29.7%; Sonnet 5 38.8% → 34.0%). Models limited by grounding improve sharply (GPT-5.6 Terra 9.0% → 40.9%).
  2. Given the reference facts with no environment, every model passes 65–91% of the 150 rubric-graded tasks, against 0–29% inside the environment. Most of the difficulty is therefore in finding and reconciling information across the organization.
  3. Cross-app coordination has an effect beyond task type. Within read-write tasks alone, Sonnet 5 passes 65.5% at one app, 45.5% at two and 17.9% at three, with either observation (2.2.A, Table 9).

Some failures are also invisible to a check of the target state alone. 6.2% of failed attempts (414 of 6,703) make an unrequested write, and for 1.6% (110) that write is their only deterministic error (Appendix). Read-only tasks, about a quarter of the benchmark, are graded only on the written answer. OSWorld reports a related single- versus multi-app gap (13.7% vs 6.6% for GPT-4V with set-of-mark prompting); TAMER makes app count a controlled variable over one shared organization. A persistence ablation that hands the agent the relevant records directly is planned for the camera-ready version.

Changes 3.1 (also 2.1.C, 2.2.A and 2.2.B)

3.1.A Related work: TheAgentCompany is not API-only

Where: Section 2, first paragraph, sentence starting "In all of these benchmarks".

Was:

In all of these benchmarks, the agent reaches the applications through an API rather than through their interfaces, and none binds it to a named employee inside a simulated organization.

Now:

In most of these benchmarks the agent calls the applications through an API. TheAgentCompany is the exception: its agent browses a simulated company's web apps through their page structure and talks to simulated colleagues. None of them restricts the agent to one employee's view of the organization.

AuthorsOpenReview ↗

3.1: Response to Critical Weakness 1 (Comment 3/3)

(Response 3.1, continued)

3.1.B Related work: comparison table

Where: Section 2, end of the paragraph on persistent worlds; new Table 1 (the table in Response 3.1).

Was:

We generate one organization and share it across tasks, so the same history is available to multiple tasks and any run can be repeated against this same state.

Now:

We generate one organization and share it across tasks, so the same history is available to multiple tasks and any run can be repeated against this same state. Table 1 compares TAMER with these benchmarks; no single property is unique to it, and what it adds is their combination in a setting where the agent operates the apps through screenshots alone.

[see Table 1 which is also the table in Response 3.1.]

Reviewer_eVFROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

This paper introduces TAMER, a benchmark for evaluating agents in complex, multi-tool enterprise knowledge-worker roles. It contains 238 tasks of varying difficulty, task types, and numbers of required web applications, set in a simulated organization of 54 employees across 19 teams and 21 projects with five months of interconnected history. Unlike many existing web benchmarks that provide access to the DOM, accessibility trees, or APIs, TAMER restricts agents to screenshot-based observations and coordinate-based interactions to evaluate their ability to navigate and operate graphical interfaces. During benchmark construction, the authors employ task authoring, human auditing, and verifier calibration to reduce task ambiguity and ensure that evaluation criteria can distinguish correct executions from those with missing, incorrect, or unintended actions. Experimental results show that Claude Sonnet 5 achieves the highest per-attempt success rate of 42.3%, outperforming the other evaluated models. Overall, performance declines as tasks involve more web applications, while tasks involving email show particularly low success rates.

Critical strengths

This paper has two clear strengths:

  1. The idea of evaluating agents from the perspective of a project manager operating within an organization is interesting. The authors construct a simulated organization of 54 employees across 19 teams and 21 projects, with five months of interconnected history. This creates a more realistic and challenging setting for multi-step enterprise tasks. Agents also operate under persona-specific access permissions, requiring them to retrieve and act on information available to their assigned roles rather than relying on unrestricted access to the underlying data.

  2. The authors put considerable effort into task authoring, human auditing, and verifier calibration to reduce ambiguity and improve evaluation reliability. Each task is validated through human execution, and its verifier is tested against one correct and three deliberately incorrect executions to identify overly strict or permissive criteria. The evaluation also combines deterministic checks with rubric-based judgments from five LLMs using majority voting. In addition, the paper reports per-attempt accuracy, pass@3, and confidence intervals, providing a relatively thorough account of model performance and evaluation reliability.

Critical weaknesses

This paper has two major weaknesses:

  1. The paper presents TAMER as a benchmark for evaluating agents in complex enterprise knowledge-worker roles. However, real-world project managers can often use browser search and other interface features to locate information more efficiently than through visual navigation alone. They also typically have some prior knowledge of the projects they manage, rather than exploring an unfamiliar organization from scratch. While the screenshot-only setting is reasonable for evaluating visual computer-use agents, it is less clear how well it reflects the broader enterprise knowledge-work capabilities emphasized in the paper. More importantly, the current evaluation does not clearly distinguish failures caused by insufficient exploration or information retrieval from those caused by incorrect reasoning or action execution. Although Section 5.3 reports performance breakdowns, it lacks a systematic analysis of these failure modes. This makes it difficult to identify which capabilities TAMER primarily evaluates and where current agents actually struggle.

  2. While some imbalance in task distribution is understandable, several categories are too small to support strong conclusions. Only 12 of the 238 tasks require four or more web applications, and just 6 require five or more web applications. It is difficult to determine whether the reported 0% success rate on tasks involving five or more applications reflects a genuine increase in difficulty or simply the limited sample size. Similarly, there are only 20 write-only tasks, substantially fewer than the read-only and read-write tasks. These imbalances limit the reliability of comparisons across task types and application counts, particularly given the paper's emphasis on complex cross-application coordination.

Questions for authors
  1. Have you considered incorporating additional observation modalities beyond screenshots, such as browser text search or other built-in navigation features that human knowledge workers commonly use? If so, how could these be integrated into TAMER?

  2. Have you analyzed the failed trajectories to distinguish between navigation, information retrieval, reasoning, and action execution failures? Which of these contributes most to the overall failure rate?

  3. Given that only 12 tasks require four or more applications and only 20 tasks are write-only, how reliable are the conclusions drawn from these relatively small subsets?

  4. Why does Claude Sonnet 5 outperform Claude Opus 5 on TAMER (42.3% vs. 31.5%), despite Opus 5 generally performing better on several public benchmarks? Have you investigated the reasons for this difference?

  5. I suggest replacing or supplementing Qwen3-VL-235B with a more recent agent-capable multimodal model in future evaluations. Qwen3-VL achieves 0% success across all tasks, making it difficult to draw meaningful insights from this baseline. A model such as Qwen3.5-122B-A10B, which shows stronger performance on several public agentic benchmarks despite having fewer parameters, could provide a more informative comparison.

11 comments
AuthorsOpenReview ↗

2.1: Response to Critical Weakness 1 (Comment 1/5)

NOTE: Where comments overlap we decided to refer to our existing responses instead of copying them again somewhere else. Please consult these for the full response. E.g. for comment 3.3. we write "Changes 3.3: 2.1.C (new Section 5.4), 2.2.A (app-count paragraph), 2.2.B (Table 4)". All our responses are numbered by Reviewer, Comment/Response and Subresponse in this way. Every change in the revised PDF is highlighted and labelled in the margin: changes made in response to a specific comment carry that response's number (for example 1.2.A).

Comment 2.1

The paper presents TAMER as a benchmark for evaluating agents in complex enterprise knowledge-worker roles. However, real-world project managers can often use browser search and other interface features to locate information more efficiently than through visual navigation alone. They also typically have some prior knowledge of the projects they manage, rather than exploring an unfamiliar organization from scratch. While the screenshot-only setting is reasonable for evaluating visual computer-use agents, it is less clear how well it reflects the broader enterprise knowledge-work capabilities emphasized in the paper. More importantly, the current evaluation does not clearly distinguish failures caused by insufficient exploration or information retrieval from those caused by incorrect reasoning or action execution. Although Section 5.3 reports performance breakdowns, it lacks a systematic analysis of these failure modes. This makes it difficult to identify which capabilities TAMER primarily evaluates and where current agents actually struggle.

Response 2.1

Realism. The screenshot setting withholds the DOM, the accessibility tree and APIs, not search. The apps' own search boxes and filters (for example in the wiki, the goal tracker and the ticketing system) are part of the rendered interface. About half of attempts on lookup tasks use them: 51% with screenshots, 55% with an accessibility-tree observation (see Changes 2.1.A). We agree that a long-tenured project manager has context our agent lacks. The prompts carry the context a colleague's request normally would, and the limitations section now states this (2.1.B). Written documentation of the organization (2.1.D) does not help the Claude models and more than doubles the pass rate of the GPT-5.6 models, mostly through fewer turn-limit hits. A second arm, in progress and due to complete during the review period, gives agents only an index of where information lives, not its content. It tests whether knowing where to look is enough or whether agents need the content itself, which separates the cost of navigation from the cost of information access.

Failure attribution. A new Section 5.4 separates four sources (2.1.C).

  • Navigation and visual grounding. Replacing screenshots with the accessibility tree lifts GPT-5.6 Terra from 9.0% to 40.9% per attempt and Sol from 13.2% to 38.9%. It does not lift the Claude models (Opus 5 31.2% → 29.7%; Sonnet 5 38.8% → 34.0%). Terra reaches the 200-turn limit in 81% of screenshot attempts, the Claude models in about 5%.
  • Composing the answer. Holding exploration fixed, we replayed only the final answering turn five times. Attempts that failed only on the rubric pass in 4.9% of replays, against 85.4% for attempts that originally passed. Per model, rubric-only failures pass in 1.8–9.5% of replays (Opus 5 4.7%, Sonnet 5 4.6%, GPT-5.6 Sol 9.5%, GPT-5.6 Terra 7.1%, Kimi 2.5 3.2%, Qwen3-VL 1.8%; appendix table, 2.1.G).
  • Reasoning over the right facts. Given the reference facts with no environment, every model passes 65–91% of the 150 rubric-graded tasks, against 0–29% on the same tasks inside the environment.
  • Retrieval and synthesis in the environment. This is what remains. With the accessibility tree, 71–77% of the rubric-only failures of the Claude and GPT-5.6 models had already seen at least 80% of the reference facts in page text before answering. Of all 62 Claude attempts that leave the apps in the correct state but fail a content criterion, hand-labelled, 63% misstate, omit or wrongly reconcile information the agent had seen, 32% never reached the source, and none are navigation failures (Appendix; 2.1.I).

A per-model failure profile assigns every failed attempt one primary category (Table 5, 2.1.C). For the two Claude models, failures come mostly from selecting and reconciling information found across the organization, not from operating the interface or from sampling the final answer.

AuthorsOpenReview ↗

2.1: Response to Critical Weakness 1 (Comment 2/5)

(Response 2.1, continued) Changes 2.1

2.1.A In-app search remains available

Where: Section 4.1, last sentence of the observation paragraph.

Was:

We withhold the accessibility tree, the DOM, and web-app APIs from the agent as a design choice for the benchmark.

Now:

Agents act through a Playwright MCP browser harness and observe only screenshots of the active tab. They act through coordinate-based mouse actions, keyboard input, tab switching, and a few utility tools. The accessibility tree, the DOM, and web-app APIs are withheld, so agents must navigate the interface rather than match text in the page structure. The apps’ own search boxes and filters remain available as part of the interface, and agents search in about half of their attempts on lookup tasks.

2.1.B New limitation: the agent starts without experience

Where: Appendix A, new paragraph before "Scope".

Was: none (new).

Now:

The agent starts without experience. A project manager who has run a project for months knows where its records live; our agent starts each task without that history. Task prompts carry the context a colleague's request normally would, and in-app search is available (Section 4), but TAMER measures an agent working in an organization it has not explored before, not one with accumulated familiarity.

AuthorsOpenReview ↗

2.1: Response to Critical Weakness 1 (Comment 3/5)

(Response 2.1, continued)

2.1.C New Section 5.4: what drives failures

Where: new Section 5.4 after Section 5.3, with new Tables 4 and 5; the accessibility-tree version (table 14) goes in the appendix.

Was: none (new section).

Now:

5.4 What drives failures

How we attribute failures. We classify every failed attempt on two axes: how the attempt ended, and where it first went wrong (Appendix H). Harness and infrastructure faults are identified first and reported as their own category, not as model failures. Before any model-assigned tag, code computes four objective signals from each trace: whether the agent never opened an app the task requires, whether it repeated an identical action three or more times in a row, whether a tool call returned an error or did not match the tool’s schema, and whether it declared the task complete while the verifier rejected the final state. Error tags assigned by a language model are reported only where they agree with these signals. Tasks that every model fails on every attempt are analysed separately.

Each model fails in a characteristic way. Table 5 gives the primary outcome of every attempt. Failed attempts by Claude Opus 5 and Claude Sonnet 5 most often fail only a rubric criterion (281 and 276 of 714 attempts). GPT-5.6 Terra and Qwen3-VL most often reach the turn limit or overflow the context (578 and 467), GPT-5.6 Sol divides between the turn limit (284) and failed write checks (171), and Kimi 2.5 most often fails a write check (372). Harness and environment faults, which are not model failures, end 73 attempts for Claude Opus 5 and 38 for Claude Sonnet 5. Table 5 assigns every failed attempt one primary category, checked in a fixed order: harness or environment fault, turn limit or context overflow, login as the wrong user, a failed deterministic check on a write, and otherwise a failed rubric criterion on the written content. The same categories and order apply to every model. The accessibility tree changes the ranking. We reran all six models on the same 238 tasks with the page’s accessibility tree in place of screenshots, keeping tasks, world and grading fixed (Table 4). This removes visual grounding and coordinate clicking. GPT-5.6 Sol and Terra rise from 13.2% and 9.0% to 38.9% and 40.9% per attempt, while Claude Opus 5 and Claude Sonnet 5 change little (31.2% to 29.7% and 38.8% to 34.0%).

Outcomes are set before the final answer. Holding each agent’s exploration fixed, we replayed only its final answering turn five times from the identical context (round 1, all six models, both observations). Attempts that originally failed only on the rubric pass in 4.9% of replays (1.8–9.5% across models; Appendix, Table 15), against 85.4% for attempts that originally passed, and 90% of these failures never pass in any replay. With the accessibility tree, 71–77% of the rubric-only failures of the Claude and GPT-5.6 models had already seen at least 80% of the reference facts in page text before answering.

Without the environment, every model answers most tasks. Given the reference facts directly, with no environment and no tools, every model passes 65–91% of the 150 rubric-graded tasks, against 0–29% on the same tasks in the environment (round 1). Most of TAMER’s difficulty is in finding and reconciling information across the organization’s apps and history rather than in composing the answer. For the two Claude models a read-only gap remains with the facts given (Sonnet 5: 60.0% against 78.7%; Opus 5: 54.3% against 69.4%), mostly on conciseness and format criteria.

Documentation of the organization helps agents that struggle to navigate. Giving agents written documentation of the organization does not help the Claude models (Sonnet 5: 38.2% to 38.7%; Opus 5: 30.7% to 27.7%; one round on all 238 tasks for every model), while GPT-5.6 Sol rises from 15.1% to 34.9% and Terra from 12.2% to 28.6%, mostly through fewer turn-limit hits. Agents already use the apps’ own search (§4.1). The gains come mostly from fewer turn-limit hits, the most common screenshot outcome for both GPT-5.6 models (Table 5).

The number of attempts changes the result: three attempts instead of one add 5.3 to 8.8 points for every model that solves at least one task, so single-attempt accuracy and pass@k should be reported separately. A benchmark that grades only state changes would also rank the models differently, because read-only tasks, graded on a written answer, reorder them (§5.3). Performance further declines with the number of web apps a task requires, and holding the interface or the task type fixed does not remove that decline (§5.4).

Reviewer_UUChOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TAMER is a benchmark for evaluating vision-language agents on enterprise knowledge work through screenshot-based, coordinate-driven interaction. It comprises 238 tasks across eight simulated web applications, grounded in a synthetic organization with 54 employees, 21 projects, and five months of interconnected history. Agents operate under employee-specific access permissions. Across six evaluated models, the best configuration achieves 42.3% per-attempt success and 53.4% pass@3.

Critical strengths
  1. Relevant evaluation setting. The benchmark evaluates cross-application workflows that require reasoning over historical information. Employee-scoped access adds realistic constraints to enterprise task execution.
  2. The benchmark includes automated consistency checks, human task execution, and verifier calibration against correct and deliberately incorrect executions. These procedures provide useful evidence of task feasibility and basic verifier functionality.
Critical weaknesses
  1. The benchmark is not currently open-source. Without access to the code and data, I cannot inspect the evaluation pipeline or verify how the grading procedures are implemented in practice.
  2. Read, write, and read-write tasks differ in retrieval requirements, content complexity, application composition, and grading conditions. Their performance differences therefore do not establish that grading mode or text generation explains the gap.
Questions for authors

Refer to weakness.

4 comments
AuthorsOpenReview ↗

1.1: Response to Critical Weakness 1

NOTE: Where comments overlap we decided to refer to our existing responses instead of copying them again somewhere else. Please consult these for the full response. E.g. for comment 3.3. we write "Changes 3.3: 2.1.C (new Section 5.4), 2.2.A (app-count paragraph), 2.2.B (Table 4)". All our responses are numbered by Reviewer, Comment/Response and Subresponse in this way. Every change in the revised PDF is highlighted and labelled in the margin: changes made in response to a specific comment carry that response's number (for example 1.2.A).

Comment 1.1

The benchmark is not currently open-source. Without access to the code and data, I cannot inspect the evaluation pipeline or verify how the grading procedures are implemented in practice.

Response 1.1

We agree that the grading pipeline must be inspectable. As stated in Section 4.2 of the original manuscript, upon acceptance we will release the evaluation harness, the grading pipeline (verifiers and judge prompts) and the 44 tasks that use only the chat, email and document-sharing apps, and we will maintain a leaderboard for the remaining tasks. We cannot release code before acceptance. The revision now states the faithfulness criterion in full (1.1.A) and reports judge agreement under the corrected rubric (1.1.B).

1.1.A Faithfulness criterion

Where: Appendix E, paragraph on the general criteria, from "One criterion in every rubric" to "read as invention."

One criterion in every rubric checks faithfulness. It fails when the response contains a factual contradiction, a wrong identifier, a claim that a forthcoming event has occurred, an entity that does not exist, or a specific detail (person, date, number, identifier, status, decision) that the reference answer does not state, even if that detail is true of the environment. It does not penalize paraphrase, salutations, filler, reordering, or formatting choice. Under this criterion an answer must stay within the scope the request sets; in a sample of 97 attempts that failed it, 77% of the disputed claims were true statements the reference answer leaves out.

1.1.B Judge agreement under the corrected rubric

Where: Appendix A, paragraph "The judge's stability is measured between models".

The judge's stability is measured between models. Judges agree closely on rubric criteria (mean per-criterion Gwet's AC1 = 0.883 on the rubric development set; Gwet, 2008). The five judges are Claude Sonnet 4.5, DeepSeek-V3.2, Kimi K2.5, GLM-5 and Qwen3-Next, spanning five disjoint model families. Reported verdicts are the majority vote (≥3 of 5) of this panel (Verga et al., 2024; Wang et al., 2026), so no single model family determines a verdict and none can favor a model that shares its family. The deterministic layer is unaffected.

1.1.C Release sentence changed to clarify

Where: Section 4.2, sentence on the 44 public tasks.

We release the evaluation harness, the grading pipeline (verifiers and judge prompts), the 44 tasks that require only chat, email, and document sharing, as well as code for running each of those 3 gyms locally via Docker.

AuthorsOpenReview ↗

1.2: Response to Critical Weakness 2 (Comment 1/2)

Comment 1.2

Read, write, and read-write tasks differ in retrieval requirements, content complexity, application composition, and grading conditions. Their performance differences therefore do not establish that grading mode or text generation explains the gap.

Response 1.2

To clarify our interpretation of this comment, your concern is that there is no causal relationship shown between a task being a read task as opposed to a read-write or write-only task and that the observed differences in difficulty is not sufficient evidence of this. We agree that the raw comparison does not causally isolate grading mode, and we have removed the sentence attributing the gap to it (1.2.A–1.2.C). The performance differences are still informative to a reader in that they highlight the unintuitive differences in pass rates between these categories. Particularly, one would expect read-only tasks to be easier than read-write tasks due to the added UI difficulties in the latter if the read-only part of the task were held constant, however we intentionally generated more difficult read-only tasks to compensate. Write-only tasks are naturally the easiest since they completely eliminate any necessary information synthesis from the gym environments. All information needed to execute a write-only task is necessarily present in the prompt.

Two controls we have implemented now narrow what the comparison can support.

First, at a fixed number of required apps, read-only tasks still fail far more often than read-write tasks for both Claude models: Sonnet 5 at one app 7.2% vs 65.5%, Opus 5 3.6% vs 60.9%.

Second, the paper cited the reversed gap for the GPT-5.6 models as evidence. That reversal appears only with screenshots, where those models fail read-write tasks mostly by reaching the turn limit. With an accessibility-tree observation, all six models pass read-only tasks less often than read-write tasks.

Part of the gap comes from how written answers are graded rather than from retrieval. Given the reference facts with no environment, the two Claude models still pass read-only tasks less often (Sonnet 5 60.0% vs 78.7%; Opus 5 54.3% vs 69.4%), mostly on conciseness and format criteria. Each rubric also fails true details that the request did not ask for (1.1.B).

Because the task types also differ in retrieval and content, the revision states only the observation: for the two Claude models, tasks graded on a written answer fail far more often than state-graded tasks with the same number of apps.

Changes 1.2 (see also 1.1.A-C) [CONTINUES IN NEXT COMMENT]

AuthorsOpenReview ↗

1.2: Response to Critical Weakness 2 (Comment 2/2)

(Response 1.2 Changes, continued)

1.2.A Section 5.3, first paragraph: the read-only gap, narrowed

Where: Section 5.3, the whole first paragraph.

Tasks graded on a written answer and tasks that change state rank models differently. With screenshots, Claude Opus 5 and Claude Sonnet 5 pass read-only tasks far less often than read-write tasks (2.4% against 36.4% and 4.8% against 45.1% per attempt), while GPT-5.6 Sol and Terra pass read-only tasks more often (14.3% against 11.7% and 11.9% against 6.6%) and fail read-write tasks mainly at the turn limit. With the accessibility tree, all six models pass read-only tasks less often, and the gap holds at a fixed number of required web apps. Part of the gap comes from how written answers are graded rather than from retrieval: given the reference facts with no environment, read-only tasks still pass less often (Claude Opus 5: 54.3% against 69.4%; Claude Sonnet 5: 60.0% against 78.7%), mostly on conciseness and format criteria, and the faithfulness criterion fails true details the request did not ask for. Because the task types also differ in retrieval and content, we do not attribute the gap to grading mode. Write-only tasks pass at 76.7% pooled and only 3 fall in the hard subset, so we report them separately and draw no conclusion from that row alone.

1.2.B Discussion paragraph

Where: end of Section 5, discussion paragraph.

The number of attempts changes the result: three attempts instead of one add 5.3 to 8.8 points for every model that solves at least one task, so single-attempt accuracy and pass@k should be reported separately. A benchmark that grades only state changes would also score the two Claude models substantially higher, because they fail most often when the task calls for a written answer. Performance further declines with the number of web apps a task requires, and holding the interface or the task type fixed does not remove that decline (Section 5.4).

1.2.C Conclusion, second paragraph

Where: Section 6, second paragraph.

Two conclusions follow from the evaluation of six models. First, grading state changes alone gives an incomplete picture: read-only tasks, scored on written content, make up 34.3% of the hard tier and none of the easy one and rank models differently from tasks that change state, and pass rate falls further with each additional web app. Second, retry budget matters: three attempts instead of one add 5.3 to 8.8 points for every model that solves at least one task, so single-attempt reliability and pass@k belong in separate columns. Finally, the benchmark is far from saturated: no model exceeds 11.2% pass@3 on the 134 hard tasks.

Thank you for your detailed response. It addresses some of my concerns. I will keep my original score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comments

Rescored results. After initial submission we audited the verifiers and rescored every attempt under a corrected task version. This improves verifier reliability and the prompts of two tasks. The most significant change is with the faithfulness criterion: it fails any specific detail that the reference answer does not state, even a true one. The changes were manually reviewed to be valid for their tasks and they improve the judge panel agreement to 0.883. We reran the attempts that had failed only because the harness did not capture an app's final state, and the attempts on the two repaired tasks, on the same app versions as the original runs; every other attempt keeps its original trace. The five-judge panel and the 3-of-5 pass rule are unchanged. All numbers in our replies and in the revised PDF use these revised numbers. Every change in the revised PDF is highlighted and labelled in the margin: changes made in response to a specific comment carry that response's number (for example 1.2.A).

General changes

G.A Table 3 (results): rescored, with intervals

Where: Table 3, caption; all cells regenerated ; new appendix tables 9 and 13 with an interval for every cell - see individual review comments for details.

Was:

Reference results over the 238 tasks, partitioned three ways; each partition is exhaustive, so subset sizes sum to 238 within a block.

Now:

Reference results over the 238 tasks, partitioned three ways; each partition is exhaustive, so subset sizes sum to 238 within a block. Context overflows and turn-limit hits count as failures, including when a later retry of the same attempt completes. Per-attempt 95% task-level bootstrap intervals for every cell are in the appendix [...]

G.B Abstract

Where: Abstract, last three sentences.

Was:

Measured difficulty splits the set into 24 easy, 98 medium, and 116 hard tasks. Across six models, the highest per-attempt accuracy on the full set is 42.3 ± 3.6%, falling to 6.0% on the hard subset. Pass rate also falls with the number of web apps a task spans.

Now:

Measured difficulty splits the set into 19 easy, 85 medium, and 134 hard tasks. Across six models, the highest per-attempt accuracy on the full set is 38.8% (95% CI 35.3–42.4), falling to 7.2% on the hard subset. Each additional web app a task requires roughly halves the odds of success.

G.C Difficulty tiers recomputed

Where: Section 3.5, last sentence.

Was:

This yields 24 easy, 98 medium and 116 hard tasks of 238, with 90 tasks stumping all six models.

Now:

This yields 19 easy, 85 medium and 134 hard tasks of 238, with 99 tasks stumping all six models.

G.D Table 2 under the recomputed tiers

Where: Table 2, rows Easy, Medium and Hard; the All row is unchanged.

Was:

TierReadRead-writeWrite1 app2 apps3 apps4+ appsTotal
Easy51452040024
Medium176714552811498
Hard348113933368116

Now:

TierReadRead-writeWrite1 app2 apps3 apps4+ appsTotal
Easy01361630019
Medium10641153229185
Hard4685345403811134

G.E Hard-subset paragraph

Where: Section 5.2, paragraph "No model exceeds 11.2% of the hard subset".

Was:

The highest pass@3 on the hard subset is 11.2%, with per-attempt accuracy of 6.0%; the same configuration reaches 53.4% pass@3 on the full corpus.

Now:

The highest pass@3 on the 134 hard tasks is 11.2%, with per-attempt accuracy of 7.2%; the same model reaches 46.2% pass@3 on the full task set.

G.F Retry gain

Where: Section 5.2, paragraph on retries, first sentence.

Was:

Moving from one attempt to three adds between 0.0 and 12.2 points across the six models on the full task set, and between 11.1 and 12.2 points for the two highest-scoring (Figure 7).

Now:

Moving from one attempt to three adds between 0.0 and 8.8 points across the six models on the full task set, and between 5.3 and 8.8 points for every model that solves at least one task (Figure 7).

G.G Pass@k figure

Where: Figure 7 (Appendix G), regenerated; caption.

Was:

(a) all 238 tasks; (b) the 116 hard tasks, …

Now:

(a) all 238 tasks; (b) the 134 hard tasks, …

G.H Turn limit and context overflow

Where: Section 4.1, paragraph on limits, first two sentences.

Was:

We set max turn limit to 200. Hitting a max-turn limit is considered a failure.

Now:

Each attempt is capped at 200 turns. An attempt that reaches the cap, or that overflows the model's context window, counts as a failure, including when a later retry of the same attempt completes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.