acceptodds
Desk rejected at ICLR 2027

ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Language-model agents increasingly act as persistent coworkers that work alongside users across many days, during which the surrounding environment keeps changing on its own and key evidence arrives in many modalities. Existing benchmarks miss this regime, scoring agents within a single static, largely text-centric episode. We introduce ClawMark, a benchmark for coworker agents built around multi-turn, multi-day tasks whose sandboxed service state evolves between turns, with fully rule-based verification and no LLM-as-judge. ClawMark comprises 100 tasks across 13 professional scenarios, scored by 1,537 deterministic checkers over post-execution service state. Benchmarking seven frontier systems shows the setting is far from solved: the strongest reaches 75.8 weighted score but only 20.0% strict Task Success, and performance drops sharply after the first exogenous environment change. ClawMark thus isolates adaptation to changing state as a central open challenge; we release the benchmark, evaluation harness, and construction pipeline to support reproducible coworker-agent evaluation.

Primary area:
datasets and benchmarks
Keywords:
Agentic Evaluation, Dynamic Environment, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zwM9OpenReview ↗
Rating
1(−1)
Confidence
3(−1)
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

ClawMark evaluates agents on 100 tasks across 13 professional scenarios spanning 2–6 simulated workdays. It combines persistent state, multimodal evidence, and external changes with deterministic scoring. Results from seven models reveal challenges in end-to-end completion and performance declines after the first environment update.

Critical strengths
  • Tests persistent workflows in which external state changes independently of the agent.

  • Combines multiple services, multimodal evidence, and cross-turn dependencies.

  • Uses explicit checks on post-execution state, supported by an initial manual audit of 96 trajectories.

Critical weaknesses
  • In Figure 4, Qwen 3.6 Plus scores 63.4 on Day 1 and 62.7 on Day 3, corresponding to a decline of 0.7 percentage points. However, Appendix F reports a 0.2-percentage-point increase from Day 1 to Day 3 and accordingly states that only six models perform below their Day-1 baselines.

  • The subtitle of Figure 4(a) states that “all models dip on Day 2,” whereas Qwen’s score actually increases from 63.4 to 64.0.

  • The current evaluation conflates adherence to monitoring responsibilities, change detection, evidence interpretation, and subsequent action execution. Failure rates grouped by verifier identifiers and rubric metadata identify challenging verifier families, but do not establish that deficits in the corresponding capabilities are the underlying causes of failure. For silent mutations in particular, the paper should clarify whether the requirements enforced by the verifiers are adequately communicated in the task instructions and distinguish failures to detect a change from failures to respond correctly after detecting it.

  • Because different workdays involve different task requirements and evaluation criteria, and the paper lacks a comparison between changing and static environments under otherwise closely matched conditions, it remains unclear whether the Day-2 performance decline is primarily attributable to environmental changes or to the greater difficulty of the tasks on that day.

  • The manual audit of 96 trajectories provides preliminary validation, but primarily confirms whether tasks succeed or fail. It does not sufficiently establish the validity of partial-credit scores, scoring weights, or failure-cause classifications, and lacks cross-validation by multiple annotators.

  • The main experiments do not include representative models that were already available at the time of submission, so it is unclear whether the reported limitations also apply to these models. I suggest adding results for newer models(GPT-6 Astra, Opus 5.5, Gemini 4 Argon, DeepSeek-V4-Pro, Grok 4.7, GLM-5.3, and Kimi K3), and checking whether they have already overcome the challenges tested by this benchmark.

Questions for authors

1.The paper does not sufficiently show that the tasks reflect real professional workflows. Real documents alone do not establish realism; the authors should explain how domain experts validated the changes, conflicts, and monitoring requirements built into the tasks.

2.The supplementary material includes the task code and verifiers, but some checks produce reproducible semantic misjudgments, and one silent-change check passes even without any agent action. In addition, the verification logic for the temporal safety case and the task-stage statistics differ from the paper’s description. The authors should investigate these issues and explain their impact on the reported results.

3.After examining the tasks in the supplementary material, I found that 10 tasks have multiple stages occurring on the same day. For example, the three stages of one clinical task take place at 14:00, 14:30, and 15:00 on the same day, rather than across three working days. This conflicts with Appendix A’s explicit definition that one turn corresponds to one simulated working day.

4.Several core design elements have already been explored in prior work, such as ClawArena [1], SentinelBench [2], PM-Bench [3], and AppWorld [4]. The paper should explain more clearly what additional capabilities ClawMark can evaluate by combining these elements that existing benchmarks cannot.

[1] Haonian Ji et al. "ClawArena: Benchmarking AI Agents in Evolving Information Environments".

[2] Matheus Kunzler Maldaner et al. "SentinelBench: A Benchmark for Long-Running Monitoring Agents".

[3] Genglin Liu et al. "PM-Bench: Evaluating Prospective Memory in LLM Agents".

[4] Harsh Trivedi et al. "AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents".

3 comments
AuthorsOpenReview ↗

Response to Reviewer zwM9

Thank you for the detailed feedback.

Q1. Domain expertise

Professional tasks were developed and checked by PhD or master's students with backgrounds in the corresponding domains. Each task underwent 3–5 rounds of revision, focusing on the task description and rubric. Developers checked that the instructions, expected outcomes, and scoring criteria were consistent, and inspected model trajectories to identify ambiguous requirements and incorrect scoring.

During development, we also provided hints in trial runs to check whether models could complete the intended workflow, helping assess task feasibility and diagnose problems. These hints were used only for development and debugging. Formal evaluation was conducted separately after the tasks were finalized, without these hints.

Q2. Verifier issues and impact

We identified scoring issues in the following five tasks:

TaskIssue
HR task5Missing schedule CSV credited as updated
Executive assistant task5Incoming email credited as cover update
Clinical assistant task4“li” in “Clinic” matched patient Li
Insurance task7Injected area flag credited as detection
Legal assistant task1Injected calendar change credited as agent work

Deleting these five items and renormalizing lowers means by 0.021–0.133 pp, preserving all seven positions.

We will correct these faulty tasks in the next version.

Q3. Turns versus days

In most tasks, one simulated workday corresponds to one interaction turn, which is why the original manuscript used “Day” to describe turns. However, a small number of tasks include multiple interactions within the same day, so turns and workdays do not always correspond one-to-one.

In the revised manuscript, we define turns as successive interaction stages with persistent environment state and explicitly distinguish interaction turns from simulated workdays.

Q4. Contribution

ClawArena studies evolving evidence, SentinelBench monitoring, PM-Bench prospective memory and AppWorld persistent multi-app state. ClawMark contributes executable workflows checking whether new evidence updates existing deliverables and decisions. A new inspection must update the existing report, not just add a database entry.

W1. Qwen figure/appendix mismatch

We have corrected Figure 4 to match the results reported in the appendix.

W2. Turn-2 decline

Figure 4(a) and the accompanying text now state that six of seven models decline on Turn 2, with Qwen as the exception.

W3. Monitoring duties and failure categories

Our original Silent-change detection category included both types of failure. Additional trajectory analysis separates them: 83% of errors in this category involved undetected changes, and 17% involved detected changes followed by failed action. Missed changes therefore account for most errors in this category, demonstrating the challenge that ClawMark's silent environment changes pose to the evaluated systems.

W4. Update timing versus difficulty

We compared two settings: original ClawMark introduces environmental updates over subsequent turns; the control moves all subsequent updates to the start of the task, so the agent encounters the updated state from turn 1. Subsequent task requirements remain the same.

We tested Sonnet 4.6 on five tasks. The original setting's score was 33.33 points lower than the control providing updates earlier.

This supports the conclusion that receiving updates later adds difficulty for some systems.

W5. Partial scores and human review

Three additional PhD students uninvolved in development each rechecked the original cases; extra review covered omitted score bands. They reviewed the detailed scores and feedback from the revised verifier; the results also support the correctness of the tasks.

Equal-check weighting preserves order; doubling 55 red-line-named items only swaps Gemini/K2.6.

W6. Model coverage

We added full-benchmark results for DeepSeek V4 Pro (April-preview route): a weighted score of 68.531 and a strict success rate of 9% across all 100 tasks.

Reviewer_zwM9OpenReview ↗

Response to Authors

Thank you for adding the DeepSeek V4 Pro results. Could you also evaluate more recent versions of representative models, as requested in my original review, to check whether the reported limitations still hold? I also encourage you to recheck the scoring code for reliability, particularly the reproducible errors discussed above.

Thank you for the follow-up. We have added evaluations of GLM-5.3 and Kimi K3 and further examined the scoring issues you raised.

Additional model results. Given the remaining time, we randomly sampled 10 tasks and evaluated GLM-5.3 and Kimi K3 on this subset. For comparison, we also report the existing results for Sonnet 4.6, DeepSeek V4 Pro, and GPT-5.4 on the same 10 tasks. Scores are averaged equally across tasks on a 0–100 scale. We will add complete results in the next version

ModelMean score on the same 10 tasks
Claude Sonnet 4.680.60
GPT-5.475.84
GLM-5.378.60
Kimi K377.31
DeepSeek V4 Pro72.79

These additional results indicate that the newer models have not saturated this subset: substantial room for improvement remains in completing ClawMark’s workflows.

Scoring reliability. During the rebuttal, we further inspected the task and scoring code and reproduced faulty scoring behavior. In Q2 of our earlier response to you, we listed five affected tasks and their specific issues. To assess their impact, we removed the five faulty checks and recomputed scores with the remaining weights renormalized. And the ordering of all seven evaluated systems remained unchanged.

We appreciate your identification of these issues. We will incorporate the corresponding task and verifier corrections into the next version and document the changes.

Reviewer_vrHUOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary
  1. ClawMark proposes an benchmark to test whether AI agents can complete multi-day, multi-turn, multimodal office collaboration tasks.
  2. It includes 100 tasks across 13 professional scenarios, with each task spanning 2–6 simulated workdays and involving files, email, calendars, knowledge bases, and spreadsheets. The environment undergoes explicit or silent changes between turns.
  3. Among the seven models, Sonnet achieved the highest weighted score of 75.8, and Opus achieved the highest complete task success rate of 20%.
Critical strengths
  1. Evaluation is closer to ongoing office work: It jointly evaluates maintaining state across days, external changes, and multimodal integration.
  2. Checks actual execution results: It determines completion based on files and service states and can identify cases where an agent says it has completed something but has not actually executed it.
  3. Stable scoring and easy diagnosis: Deterministic verifiers make reruns easy and can identify specific failed checks.
Critical weaknesses
  1. Limited sample size: The 100 tasks are unevenly distributed, and EDA has only one task, making domain-level conclusions insufficiently robust.
  2. Conclusions depend on the evaluation configuration: Only OpenClaw is used, and most models lack repeated experiments, making it difficult to establish stable, generalizable model rankings. And the difference between the frontier model is small.
Questions for authors

None.

1 comment
AuthorsOpenReview ↗

Response to Reviewer vrHU

Thank you for recognizing persistent workflows and checks on actual execution. We clarify task coverage and add system-level comparisons.

W1. Task count and domain coverage

ClawMark's 100 executable multi-turn workflows are comparable in task count to established execution-based agent benchmarks.

BenchmarkTasks
Terminal-Bench 2.089
ScienceAgentBench102
ClawMark100

ClawMark covers 13 domains to broaden the range of workflows evaluated. Our primary conclusions concern aggregate performance and ranking across all 100 tasks. Although EDA is represented by a single task, ClawMark is not designed specifically to measure EDA capability. Small categories also appear in other broad benchmarks: Terminal-Bench 2.0, Figure 4 includes one task each in Games, Video Processing, and Data Querying. Similarly, our goal is broad workflow coverage rather than separate capability leaderboards for every domain.

W2. Harness choice, transfer and repeatability

We evaluate the agent system formed by a model and its harness. OpenClaw is a third-party open-source assistant framework supporting multiple models, persistent sessions and tool use, matching the coworker setting. Using it consistently provides a shared framework for model comparisons. OpenClaw

Harness effects also appear elsewhere: Terminal-Bench 2.0 uses shared Terminus 2 for model comparison; its Table 2 reports Gemini 2.5 Pro at 32.6% with Terminus 2 and 15.7% with OpenHands. Rankings therefore refer to specified system configurations.

We additionally ran GPT-5.4 and Sonnet 4.6 in Hermes on five tasks, twice per task. We average repeats within tasks, then the five tasks equally, and compare with the original OpenClaw records for the same tasks:

ModelOpenClawHermes
GPT-5.457.6362.14
Sonnet 4.665.9955.27

These scores describe the respective system configurations. ClawMark remains executable under Hermes and challenging for both evaluated systems.

For small gaps, we added 10,000 paired task-bootstrap resamples over all 100 tasks for all 21 model pairs:

DifferenceMean (pp)Nominal 95% interval
Sonnet−Opus1.154[−2.935,5.421]
Opus−GPT2.677[−2.062,7.340]

Across these resamples, mean Spearman correlation with the original ranking is 0.949, and the original top-three set is retained in 92.33%. This supports stable overall ordering and leading-group membership under task resampling.

Reviewer_qmhnOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces CLAWMARK, a benchmark for persistent multimodal “coworker” agents operating in environments that evolve across multiple turns and simulated workdays. It contains 100 tasks spanning 13 professional scenarios, five stateful services, and raw evidence in images, scanned PDFs, audio, video, and spreadsheets. Between turns, the environment changes through both announced events and unannounced mutations. Outcomes are evaluated using 1,537 deterministic Python verifiers that inspect the resulting filesystem and service states. Seven frontier models are evaluated using a fixed OpenClaw-based agent harness. The best weighted score is 75.8%, whereas the highest strict task-success rate is 20%. The paper additionally reports lower scores after the first environment update and attributes many failures to silent-change detection and backend writeback. Its main contribution is an executable testbed that combines persistent state tracking, adaptation to external changes, multimodal evidence integration, and state-based evaluation.

Critical strengths
  1. A consequential and underexplored evaluation setting. CLAWMARK targets an important gap between conventional agent benchmarks and realistic persistent assistants. Its combination of multi-day interaction, independently changing external state, multiple stateful services, and raw multimodal evidence is substantially more realistic than evaluating isolated tool-use episodes. This is consequential to my assessment because the resulting benchmark infrastructure could support meaningful research on long-horizon agents even if some of the present diagnostic claims require stronger validation.

  2. Executable and failure-diagnostic state-based evaluation. The use of deterministic verifiers over actual post-execution service state is a meaningful methodological strength. It can detect cases in which an agent produces plausible text but fails to update the appropriate backend or violates a procedural constraint. The rerun checks, verifier-level diagnostics, and explicit red-line conditions make the benchmark potentially more reproducible and informative than evaluations based only on final textual answers or an LLM judge.

Critical weaknesses
  1. The central post-update degradation claim is not identified by the experimental design.

The paper interprets the Day-1 to Day-2 decline in Figure 4 as evidence that exogenous environment changes impair agent performance. However, the comparison simultaneously changes several other factors: later turns can contain harder requirements, different verifier sets and weights, longer interaction histories, and errors accumulated from previous actions. Scores assigned on different days therefore need not measure the same underlying difficulty. The paper appropriately acknowledges in the introduction and conclusion that this analysis is descriptive and does not isolate the causal contribution of external mutation. Nevertheless, the abstract and Section 5 use stronger language, including the claim that environment changes “trigger” the decline. The evidence currently establishes that later turns are harder, but not that exogenous mutation is the cause. A persuasive test would require matched task variants in which the prompts, turn structure, and evaluation criteria are preserved while the relevant mutations are disabled, explicitly announced, or replaced by unchanged state. Constructing such controls may require modifying both service-side mutations and the associated prompts and verifiers; simply disabling the file-injection layer would not necessarily produce a valid counterfactual.

  1. The validity and statistical robustness of the reported evaluation remain under-established. Deterministic execution demonstrates verifier stability, but not semantic correctness or completeness. The calibration study uses one annotator and samples trajectories only from scores below 0.20, between 0.40 and 0.65, and exactly 1.0. It therefore omits the intervals [0.20, 0.40) and (0.65, 1.0), where partially correct solutions and verifier disagreements may be particularly common. The paper reports neither blinded assessment nor domain-expert validation for its clinical, legal, insurance, and HR workflows. The quantitative comparisons also have limited uncertainty analysis. Five models appear to have only one complete sweep. For Kimi K2.6, the three reported runs span 2.8 percentage points, which is of similar magnitude to several leaderboard gaps. Thus, the precise ranking in Table 3 is not statistically established. All results are additionally conditional on one OpenClaw harness with provider-specific compatibility transformations. While the authors acknowledge this limitation, model-level conclusions should be phrased as system-level results unless robustness across scaffolds or compatibility choices is demonstrated. Finally, the failure taxonomy in Table 5 is constructed from author-defined verifier identifiers and metadata rather than independent trajectory annotation. Because 67% of failures remain in the residual “scenario-specific” category, the taxonomy provides limited coverage. The high failure rates for silent-change detection and backend writeback are informative descriptive statistics, but they do not yet establish that these are the principal underlying mechanisms of failure.
Questions for authors
  1. How are the Day-1, Day-2, and Day-3 scores in Figure 4 calculated? How do the number, weight, and difficulty of the corresponding verifiers vary across turns?

  2. Who conducted the 96-trajectory audit, and was the annotator independent of benchmark construction and blinded to verifier outcomes? Why were trajectories in the score intervals [0.20, 0.40) and (0.65, 1.0) excluded? Were specialized tasks reviewed by relevant domain experts?

  3. Which reference models and Codex configuration were used during task development? Do any overlap with the evaluated model families, and how frequently were tasks or verifiers revised in response to their trajectories?

  4. How are the per-task values from Equation (1) aggregated into the Table 3 score? Are tasks weighted equally? How sensitive are the rankings to alternative verifier weights, task-level macro-averaging, red-line weights, and provider-specific harness transformations?

  5. Will the complete task code, verifier implementations, prompts, service images, multimodal artifacts, model configurations, and artifact provenance or license metadata be released?

  6. Can the authors provide matched static-versus-mutating runs that preserve the substantive requirements of each turn while varying only whether the external state changes?

  7. Can the authors report repeated sweeps or uncertainty intervals for all evaluated systems? Task-level paired bootstrap intervals would at least clarify whether the reported model differences are meaningful.

1 comment
AuthorsOpenReview ↗

Response to Reviewer qmhn

Thank you for recognizing the value of persistent-workflow evaluation. We clarify the protocol and provide additional evidence.

Q1. Turn-level scores

1. Score calculation. Each task has checks and weights for each turn. Its turn score is 100 × passed-check weight / total turn-check weight; task scores are then averaged equally for Figure 4. For example, passing checks with weight 7 out of a total of 10 gives 70 points. Final-only checks are excluded.

2. Average check counts and weights. The following statistics use the 73 tasks in Figure 4:

TurnChecks per taskTotal weight per taskWeight per check
Turn 16.419.991.56
Turn 24.457.041.58
Turn 34.386.711.53

All entries are averages.

3. Difficulty is not directly comparable across turns. Requirements and checks change, and difficulty is not held constant.

Q2. Human audit and sampling

Domain PhD/master's students authored and checked professional tasks. One PhD student uninvolved in development audited 96 trajectories, checking partial scores and failure reasons with verifier results visible.

This provided further calibration after developer checks: 32 cases each below 0.20, at 0.40–0.65 and at 1.0 covered different completion levels. Three additional PhD students, uninvolved in development, each rechecked the original cases; supplementary auditing covered [0.20,0.40) and (0.65,1.0). They checked revised-verifier scores/feedback; the results also support the correctness of the tasks.

Q3. Development models and revisions

Every task was trialled with Claude Code/Sonnet 4.5 and a local Codex client/GPT-5, which also reviewed trajectories. GPT-5 used high reasoning, with no developer-set debugging time/call cap. Each task underwent 3–5 description/rubric revisions to address inconsistencies and grading errors.

Q4. Scoring and ranking stability

1. Computing the total. Each task receives a percentage score: passed-check weight / full rubric weight × 100. Table 3 averages the 100 task scores equally, so tasks with more checks do not receive more weight overall.

2. Changing the weights. We rescored the same runs under two alternatives:

ChangeRanking
Equal weight for every checkUnchanged
Increase 55 red-line-named items from weight 2 to 4Only Gemini/K2.6 swap

The main ordering is stable under these two changes, although one adjacent pair is sensitive.

3. Run configuration. The main table compares model systems under shared OpenClaw. We also tested Hermes; please see our response to Reviewer vrHU, W2, for cross-harness results, repeatability analysis and evidence from other benchmarks. Provider-specific interface adaptations have not been tested in isolation.

Q5. Reproducibility

All tasks, checkers, prompts, assets and build definitions are public. We will also release all model trajectories, full model/harness configurations, environment images/build files and asset provenance/license metadata. Email/calendar run locally; Notion/Sheets require account authorization.

Q6. Update-timing comparison

We compared two settings: original ClawMark introduces environmental updates over subsequent turns; the control moves all subsequent updates to the start of the task, so the agent encounters the updated state from turn 1. Subsequent task requirements remain the same.

We tested Sonnet 4.6 on five tasks. The original setting's score was 33.33 points lower than the control providing updates earlier.

This supports the conclusion that receiving updates later adds difficulty for some systems.

Q7. Ranking uncertainty

Overall ordering and top-three membership are stable under task resampling. In 10,000 paired bootstrap resamples of 100 tasks, using the same task draw for all seven models, mean Spearman correlation with the original ranking is 0.949. The original top-three set—Sonnet 4.6, Opus 4.6 and GPT-5.4—is retained in 92.33% of resamples.

We also computed intervals for all 21 model pairs. Results among the top three:

DifferenceMean (pp)Nominal 95% interval
Sonnet−Opus1.154[−2.935,5.421]
Sonnet−GPT3.831[0.239,7.430]
Opus−GPT2.677[−2.062,7.340]

W2. Verifier validity and failure categories

To examine verifier accuracy, we identified a faulty check in each of five tasks. Removing these five checks and recomputing scores using the remaining checks preserved the ranking of all seven systems. See Reviewer zwM9, Q2.

To address your concern about failure attribution, we analyzed the trajectories. Of silent-change detection errors, 83% involved undetected changes and 17% involved detected changes followed by failed action. This identifies detecting silent environmental changes as the main difficulty within this error category. See our response to Reviewer zwM9, W3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.