acceptodds
Withdrawn from ICLR 2027

Claw-Anything: Benchmarking Always-On Personal Assistants with Broader Access to User's Digital World

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language model agents are increasingly envisioned as always-on personal assistants with access to anything relevant in the user's digital world. Yet current systems operate over only narrow slices of that world, limiting context-sensitive reasoning and effective assistance. Existing benchmarks similarly provide only partial user state and therefore fail to capture performance in this broader setting. To address this gap, we introduce Claw-Anything, a benchmark that expands agent context along three dimensions: long-horizon activity histories, interdependent backend services, and integrated GUI and CLI interaction across multiple devices. To instantiate this setting, we simulate months of user activity through multi-round event injection, producing complex world states and realistic noise, including irrelevant events and conflicting signals. Agents must reason over rich contextual environments while remaining robust to such noise. This expanded scope also enables evaluation of heartbeat-conditioned proactive assistance, requiring agents to infer what to recommend at a fixed check time. Experiments show that GPT-5.5 achieves only 34.5% Pass@1, underscoring a gap between current agent capabilities and the demands of always-on personal assistance. Alongside the benchmark, we release an automated data-generation pipeline that yields 2,000 training environments; fine-tuning Qwen3.5-27B on successful trajectories improves Pass@1 by 23.7 percentage points, demonstrating the pipeline's utility for generating training data.

Primary area:
datasets and benchmarks
Keywords:
Agentic, Benchmark, LLM

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Hvb7OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper's main contribution includes a benchmark for evaluating always-on personal assistants with access to anything relevant in the user’s digital world. This paper aims to evaluate and train agents as personal assistants that have access to the context and personal information of the user over a long period of time, such as several months. In this way, this paper can evaluate the agent using long-horizon tasks and make the benchmark more similar to real-life agent use. This is a good idea and very practical.

However, the task design is vague according to the provided examples. Benchmark tasks should not be unsolvable. The task examples provided in the paper are not very clear, and they are not fully solvable by human experts. Also, this paper's contribution is more focused on a data synthesis pipeline for collecting training data, not on a concrete and clear benchmark. As shown in Table 12, Claw-Anything-Qwen3.5-27B only gains 1% on Claw-Eval, which raises concerns about whether the training data provided by Claw-Anything can lead to improvement in general agent ability.

Critical strengths
  1. I appreciate the idea of this paper. It is interesting, important, and valuable. Developing a benchmark for evaluating personal assistant agents that have been working for months and collecting training data is important. To accomplish a task, the agents need to answer three questions: What is the previous context given? What is the current state, like needing attention or having accomplished something? What is needed to do now?

  2. This paper contributes to a data synthesis pipeline. This pipeline is very valuable to the community, as such a pipeline can produce valuable training data for training capable agents at lower cost.

  3. This paper provides an examination of the synthesized data's quality. This examination can ensure the quality of the data. This paper also provides comprehensive experiments, including evaluation on Claw-Anything's eval and Claw-Eval.

Critical weaknesses
  1. This paper mentions, "Our primary contribution is Claw-Anything, a benchmark of 200 human-verified interactive task environments that jointly expands access across time, services, and devices, with heartbeat-conditioned proactivity as a distinct evaluation axis." However, this paper's main contribution should be a training data synthesis pipeline rather than a benchmark. Table2 provides the overall score, and I think this score does not differentiate between models.

  2. Generalizability. According to Table 12, training on Claw-Anything data does not significantly improve the performance on Claw-Eval. This raises concerns about this improvement. Claw-Eval contains 161 general personal assistant tasks. 112 / 161 = 69.565%, and 113 / 161 = 70.186%, where is 69.7 come from? 114 / 161 = 70.807%, where is 70.7 come from?

  3. According to the tasks provided in the code, the tasks are vague. For example, https://anonymous.4open.science/r/Claw-Anything/benchmark/skill/T01_T01_unresolved_training_budget_decision_disc/task.yaml, execution_date: '2026-04-05', "This signals analysis paralysis and fear of committing to the wrong strategy, especially with the sprint demo deadline approaching on January 28". These two are misaligned. The execution date is used by https://anonymous.4open.science/r/Claw-Anything/src/claw_anything/runner/system_prompt.py. Benchmark data should be high-quality, solvable, and clear.

  4. Table 11 looks strange. MiniMax-M2.7 outperforms Claude Opus4.7. This looks strange. According to https://openrouter.ai/compare/anthropic/claude-opus-4.7/minimax/minimax-m2.7, it is difficult to find a benchmark where MiniMax-M2.7 outperforms ClaudeOpus4.7.

Questions for authors
  1. This paper mentions, "Our primary contribution is Claw-Anything, a benchmark of 200 human-verified interactive task environments that jointly expands access across time, services, and devices, with heartbeat-conditioned proactivity as a distinct evaluation axis." However, this paper's main contribution should be a training data synthesis pipeline rather than a benchmark. Table2 provides the overall score, and I think this score does not differentiate between models. Question: Make the main contribution clearer?

  2. Generalizability. According to Table 12, training on Claw-Anything data does not significantly improve the performance on Claw-Eval. This raises concerns about this improvement. Claw-Eval contains 161 general personal assistant tasks. 112 / 161 = 69.565%, and 113 / 161 = 70.186%, where is 69.7 come from? 114 / 161 = 70.807%, where is 70.7 come from? Question: Why does Table 12 contain scores that cannot be calculated from 161 tasks?

  3. According to the tasks provided in the code, the tasks are vague. For example, https://anonymous.4open.science/r/Claw-Anything/benchmark/skill/T01_T01_unresolved_training_budget_decision_disc/task.yaml, execution_date: '2026-04-05', "This signals analysis paralysis and fear of committing to the wrong strategy, especially with the sprint demo deadline approaching on January 28". These two are misaligned. The execution date is used by https://anonymous.4open.science/r/Claw-Anything/src/claw_anything/runner/system_prompt.py. Benchmark data should be high-quality, solvable, and clear. Question: Why does this task have conflicts?

  4. Table 11 looks strange. MiniMax-M2.7 outperforms Claude Opus4.7. This looks strange. According to https://openrouter.ai/compare/anthropic/claude-opus-4.7/minimax/minimax-m2.7, it is difficult to find a benchmark where MiniMax-M2.7 outperforms ClaudeOpus4.7. Question: Provide an analysis of how MiniMax-M2.7 outperforms Opus 4.7.

Reviewer_fVRfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Personal always-on assistants often require to digest events, services and devices over a long time period rather than solving a single isolated task. This paper proposed a benchmark for evaluating this task which is neglected in existing studies. The paper introduces Claw-Anything, containing 200 bounded task envs with about three months of synthetic history, covering both CLI-only and CLI+GUI tasks.

Critical strengths
  • the overall construction pipeline covers an empty space in existing CLI/GUI agents tasks; the pipeline also contains useful safeguards and human verification; making the benchmark and data generation method a clear contribution
  • the ablation section on context scaling, data pipeline etc supports most of the claimed motivation and design choices
Critical weaknesses
  • evalutaion of training lack results on general CLI/GUI ability preservation: although the training results show large improvements, it is essentially evaluated in-domain; it is unclear whether this finetuning will corrupt model's orignal ability on other single task performance e.g., OS-world
  • Some analysis are performed on too few tasks e.g., Table 3, only 15 instances; the variance may be too large as for showing real trending
  • Some scores seem to be inconsistent in tables and figures; e.g., the Qwen3.6-27B results on cross-devices, shown as 16.0 iin Table 3 but shown as 24.0? in Figure 5
Questions for authors

N/A

Reviewer_jqJXOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces Claw-Anything, a benchmark for personal-assistant agents with access to long activity histories, multiple services, and CLI/GUI interfaces. It develops an automated pipeline to generate evaluation and training environments, evaluates several LLMs, and demonstrates improvements from fine-tuning on the generated trajectories.

Critical strengths
  1. The paper constructs a synthetic dataset for evaluating personal-assistant agents.
  2. The authors conduct experiments with several LLMs to evaluate their performance on the dataset.
Critical weaknesses
  1. The real-world relevance of the synthetic data remains unclear. The benchmark relies on generated user histories and needs, without sufficient evidence that the resulting recommendations reflect assistance real users would want.
  2. Insufficient comparison with existing proactive benchmarks. The main comparison excludes several directly relevant benchmarks, leaving the contribution beyond existing work unclear.
  3. The proactive evaluation is too limited. Every proactive task contains a recommendation target, so the benchmark does not evaluate whether agents can avoid unnecessary assistance.
Questions for authors
  1. The pipeline generates user histories, needs, and expected recommendations. The human checks assess whether these examples are plausible and consistent, but what evidence from real user behavior or feedback supports the usefulness of the recommendations?
  2. Table 1 highlights proactivity as a distinguishing feature but excludes Proactive Agent (ICLR 2025), FingerTip 20K (ICLR 2026), and ContextAgentBench (NeurIPS 2025). Although these works are discussed elsewhere, can the authors provide a clearer comparison showing what Claw-Anything adds beyond them? PARE-Bench, which also evaluates proactive assistance in simulated multi-app environments, should also be discussed.
  3. Only 15 tasks are proactive, and each contains a recommendation target at a fixed check time. Can the authors include cases where assistance is unnecessary and evaluate whether agents appropriately remain silent?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.