- Rating
- 1(−1)
- Confidence
- 3(−1)
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ClawMark evaluates agents on 100 tasks across 13 professional scenarios spanning 2–6 simulated workdays. It combines persistent state, multimodal evidence, and external changes with deterministic scoring. Results from seven models reveal challenges in end-to-end completion and performance declines after the first environment update.
Critical strengths
-
Tests persistent workflows in which external state changes independently of the agent.
-
Combines multiple services, multimodal evidence, and cross-turn dependencies.
-
Uses explicit checks on post-execution state, supported by an initial manual audit of 96 trajectories.
Critical weaknesses
-
In Figure 4, Qwen 3.6 Plus scores 63.4 on Day 1 and 62.7 on Day 3, corresponding to a decline of 0.7 percentage points. However, Appendix F reports a 0.2-percentage-point increase from Day 1 to Day 3 and accordingly states that only six models perform below their Day-1 baselines.
-
The subtitle of Figure 4(a) states that “all models dip on Day 2,” whereas Qwen’s score actually increases from 63.4 to 64.0.
-
The current evaluation conflates adherence to monitoring responsibilities, change detection, evidence interpretation, and subsequent action execution. Failure rates grouped by verifier identifiers and rubric metadata identify challenging verifier families, but do not establish that deficits in the corresponding capabilities are the underlying causes of failure. For silent mutations in particular, the paper should clarify whether the requirements enforced by the verifiers are adequately communicated in the task instructions and distinguish failures to detect a change from failures to respond correctly after detecting it.
-
Because different workdays involve different task requirements and evaluation criteria, and the paper lacks a comparison between changing and static environments under otherwise closely matched conditions, it remains unclear whether the Day-2 performance decline is primarily attributable to environmental changes or to the greater difficulty of the tasks on that day.
-
The manual audit of 96 trajectories provides preliminary validation, but primarily confirms whether tasks succeed or fail. It does not sufficiently establish the validity of partial-credit scores, scoring weights, or failure-cause classifications, and lacks cross-validation by multiple annotators.
-
The main experiments do not include representative models that were already available at the time of submission, so it is unclear whether the reported limitations also apply to these models. I suggest adding results for newer models(GPT-6 Astra, Opus 5.5, Gemini 4 Argon, DeepSeek-V4-Pro, Grok 4.7, GLM-5.3, and Kimi K3), and checking whether they have already overcome the challenges tested by this benchmark.
Questions for authors
1.The paper does not sufficiently show that the tasks reflect real professional workflows. Real documents alone do not establish realism; the authors should explain how domain experts validated the changes, conflicts, and monitoring requirements built into the tasks.
2.The supplementary material includes the task code and verifiers, but some checks produce reproducible semantic misjudgments, and one silent-change check passes even without any agent action. In addition, the verification logic for the temporal safety case and the task-stage statistics differ from the paper’s description. The authors should investigate these issues and explain their impact on the reported results.
3.After examining the tasks in the supplementary material, I found that 10 tasks have multiple stages occurring on the same day. For example, the three stages of one clinical task take place at 14:00, 14:30, and 15:00 on the same day, rather than across three working days. This conflicts with Appendix A’s explicit definition that one turn corresponds to one simulated working day.
4.Several core design elements have already been explored in prior work, such as ClawArena [1], SentinelBench [2], PM-Bench [3], and AppWorld [4]. The paper should explain more clearly what additional capabilities ClawMark can evaluate by combining these elements that existing benchmarks cannot.
[1] Haonian Ji et al. "ClawArena: Benchmarking AI Agents in Evolving Information Environments".
[2] Matheus Kunzler Maldaner et al. "SentinelBench: A Benchmark for Long-Running Monitoring Agents".
[3] Genglin Liu et al. "PM-Bench: Evaluating Prospective Memory in LLM Agents".
[4] Harsh Trivedi et al. "AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents".
3 comments
Response to Authors
Thank you for adding the DeepSeek V4 Pro results. Could you also evaluate more recent versions of representative models, as requested in my original review, to check whether the reported limitations still hold? I also encourage you to recheck the scoring code for reliability, particularly the reproducible errors discussed above.
Thank you for the follow-up. We have added evaluations of GLM-5.3 and Kimi K3 and further examined the scoring issues you raised.
Additional model results. Given the remaining time, we randomly sampled 10 tasks and evaluated GLM-5.3 and Kimi K3 on this subset. For comparison, we also report the existing results for Sonnet 4.6, DeepSeek V4 Pro, and GPT-5.4 on the same 10 tasks. Scores are averaged equally across tasks on a 0–100 scale. We will add complete results in the next version
| Model | Mean score on the same 10 tasks |
|---|---|
| Claude Sonnet 4.6 | 80.60 |
| GPT-5.4 | 75.84 |
| GLM-5.3 | 78.60 |
| Kimi K3 | 77.31 |
| DeepSeek V4 Pro | 72.79 |
These additional results indicate that the newer models have not saturated this subset: substantial room for improvement remains in completing ClawMark’s workflows.
Scoring reliability. During the rebuttal, we further inspected the task and scoring code and reproduced faulty scoring behavior. In Q2 of our earlier response to you, we listed five affected tasks and their specific issues. To assess their impact, we removed the five faulty checks and recomputed scores with the remaining weights renormalized. And the ordering of all seven evaluated systems remained unchanged.
We appreciate your identification of these issues. We will incorporate the corresponding task and verifier corrections into the next version and document the changes.
Response to Reviewer zwM9
Thank you for the detailed feedback.
Q1. Domain expertise
Professional tasks were developed and checked by PhD or master's students with backgrounds in the corresponding domains. Each task underwent 3–5 rounds of revision, focusing on the task description and rubric. Developers checked that the instructions, expected outcomes, and scoring criteria were consistent, and inspected model trajectories to identify ambiguous requirements and incorrect scoring.
During development, we also provided hints in trial runs to check whether models could complete the intended workflow, helping assess task feasibility and diagnose problems. These hints were used only for development and debugging. Formal evaluation was conducted separately after the tasks were finalized, without these hints.
Q2. Verifier issues and impact
We identified scoring issues in the following five tasks:
Deleting these five items and renormalizing lowers means by 0.021–0.133 pp, preserving all seven positions.
We will correct these faulty tasks in the next version.
Q3. Turns versus days
In most tasks, one simulated workday corresponds to one interaction turn, which is why the original manuscript used “Day” to describe turns. However, a small number of tasks include multiple interactions within the same day, so turns and workdays do not always correspond one-to-one.
In the revised manuscript, we define turns as successive interaction stages with persistent environment state and explicitly distinguish interaction turns from simulated workdays.
Q4. Contribution
ClawArena studies evolving evidence, SentinelBench monitoring, PM-Bench prospective memory and AppWorld persistent multi-app state. ClawMark contributes executable workflows checking whether new evidence updates existing deliverables and decisions. A new inspection must update the existing report, not just add a database entry.
W1. Qwen figure/appendix mismatch
We have corrected Figure 4 to match the results reported in the appendix.
W2. Turn-2 decline
Figure 4(a) and the accompanying text now state that six of seven models decline on Turn 2, with Qwen as the exception.
W3. Monitoring duties and failure categories
Our original Silent-change detection category included both types of failure. Additional trajectory analysis separates them: 83% of errors in this category involved undetected changes, and 17% involved detected changes followed by failed action. Missed changes therefore account for most errors in this category, demonstrating the challenge that ClawMark's silent environment changes pose to the evaluated systems.
W4. Update timing versus difficulty
We compared two settings: original ClawMark introduces environmental updates over subsequent turns; the control moves all subsequent updates to the start of the task, so the agent encounters the updated state from turn 1. Subsequent task requirements remain the same.
We tested Sonnet 4.6 on five tasks. The original setting's score was 33.33 points lower than the control providing updates earlier.
This supports the conclusion that receiving updates later adds difficulty for some systems.
W5. Partial scores and human review
Three additional PhD students uninvolved in development each rechecked the original cases; extra review covered omitted score bands. They reviewed the detailed scores and feedback from the revised verifier; the results also support the correctness of the tasks.
Equal-check weighting preserves order; doubling 55 red-line-named items only swaps Gemini/K2.6.
W6. Model coverage
We added full-benchmark results for DeepSeek V4 Pro (April-preview route): a weighted score of 68.531 and a strict success rate of 9% across all 100 tasks.