- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper's main contribution includes a benchmark for evaluating always-on personal assistants with access to anything relevant in the user’s digital world. This paper aims to evaluate and train agents as personal assistants that have access to the context and personal information of the user over a long period of time, such as several months. In this way, this paper can evaluate the agent using long-horizon tasks and make the benchmark more similar to real-life agent use. This is a good idea and very practical.
However, the task design is vague according to the provided examples. Benchmark tasks should not be unsolvable. The task examples provided in the paper are not very clear, and they are not fully solvable by human experts. Also, this paper's contribution is more focused on a data synthesis pipeline for collecting training data, not on a concrete and clear benchmark. As shown in Table 12, Claw-Anything-Qwen3.5-27B only gains 1% on Claw-Eval, which raises concerns about whether the training data provided by Claw-Anything can lead to improvement in general agent ability.
Critical strengths
-
I appreciate the idea of this paper. It is interesting, important, and valuable. Developing a benchmark for evaluating personal assistant agents that have been working for months and collecting training data is important. To accomplish a task, the agents need to answer three questions: What is the previous context given? What is the current state, like needing attention or having accomplished something? What is needed to do now?
-
This paper contributes to a data synthesis pipeline. This pipeline is very valuable to the community, as such a pipeline can produce valuable training data for training capable agents at lower cost.
-
This paper provides an examination of the synthesized data's quality. This examination can ensure the quality of the data. This paper also provides comprehensive experiments, including evaluation on Claw-Anything's eval and Claw-Eval.
Critical weaknesses
-
This paper mentions, "Our primary contribution is Claw-Anything, a benchmark of 200 human-verified interactive task environments that jointly expands access across time, services, and devices, with heartbeat-conditioned proactivity as a distinct evaluation axis." However, this paper's main contribution should be a training data synthesis pipeline rather than a benchmark. Table2 provides the overall score, and I think this score does not differentiate between models.
-
Generalizability. According to Table 12, training on Claw-Anything data does not significantly improve the performance on Claw-Eval. This raises concerns about this improvement. Claw-Eval contains 161 general personal assistant tasks. 112 / 161 = 69.565%, and 113 / 161 = 70.186%, where is 69.7 come from? 114 / 161 = 70.807%, where is 70.7 come from?
-
According to the tasks provided in the code, the tasks are vague. For example, https://anonymous.4open.science/r/Claw-Anything/benchmark/skill/T01_T01_unresolved_training_budget_decision_disc/task.yaml, execution_date: '2026-04-05', "This signals analysis paralysis and fear of committing to the wrong strategy, especially with the sprint demo deadline approaching on January 28". These two are misaligned. The execution date is used by https://anonymous.4open.science/r/Claw-Anything/src/claw_anything/runner/system_prompt.py. Benchmark data should be high-quality, solvable, and clear.
-
Table 11 looks strange. MiniMax-M2.7 outperforms Claude Opus4.7. This looks strange. According to https://openrouter.ai/compare/anthropic/claude-opus-4.7/minimax/minimax-m2.7, it is difficult to find a benchmark where MiniMax-M2.7 outperforms ClaudeOpus4.7.
Questions for authors
-
This paper mentions, "Our primary contribution is Claw-Anything, a benchmark of 200 human-verified interactive task environments that jointly expands access across time, services, and devices, with heartbeat-conditioned proactivity as a distinct evaluation axis." However, this paper's main contribution should be a training data synthesis pipeline rather than a benchmark. Table2 provides the overall score, and I think this score does not differentiate between models. Question: Make the main contribution clearer?
-
Generalizability. According to Table 12, training on Claw-Anything data does not significantly improve the performance on Claw-Eval. This raises concerns about this improvement. Claw-Eval contains 161 general personal assistant tasks. 112 / 161 = 69.565%, and 113 / 161 = 70.186%, where is 69.7 come from? 114 / 161 = 70.807%, where is 70.7 come from? Question: Why does Table 12 contain scores that cannot be calculated from 161 tasks?
-
According to the tasks provided in the code, the tasks are vague. For example, https://anonymous.4open.science/r/Claw-Anything/benchmark/skill/T01_T01_unresolved_training_budget_decision_disc/task.yaml, execution_date: '2026-04-05', "This signals analysis paralysis and fear of committing to the wrong strategy, especially with the sprint demo deadline approaching on January 28". These two are misaligned. The execution date is used by https://anonymous.4open.science/r/Claw-Anything/src/claw_anything/runner/system_prompt.py. Benchmark data should be high-quality, solvable, and clear. Question: Why does this task have conflicts?
-
Table 11 looks strange. MiniMax-M2.7 outperforms Claude Opus4.7. This looks strange. According to https://openrouter.ai/compare/anthropic/claude-opus-4.7/minimax/minimax-m2.7, it is difficult to find a benchmark where MiniMax-M2.7 outperforms ClaudeOpus4.7. Question: Provide an analysis of how MiniMax-M2.7 outperforms Opus 4.7.