- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
Existing datasets in the area of proactive agent suffer from two critical deficiencies: (1) reliance on LLM-synthesized data that fails to capture authentic human decision-making patterns, and (2) focus on isolated tasks rather than continuous workflows, missing the pre-assistance behavioral context essential for learning proactive intervention signals. So in this paper, a rigorous benchmark, ProAgentBench, is introduced to evaluate proactive agents in working scenarios. This benchmark captures over 28,000 events from 500+ hours of continuous working sessions, and the authors formalize a “When + How” hierarchical task framework to express the proactive assistance. Experimentally, extensive main experiments are conducted across diverse models and methods, and ablation study is designed to explore three main research questions, namely impact of historical observation sequence length, user context, and real-world data.
Critical strengths
Real datasets have value. A large-scale privacy-compliant real-world dataset is constructed, which captures over 28,000 events from 500+ hours of authentic user activities, preserving critical pre-assistance behavioral patterns.
Critical weaknesses
1.Limited novelty. Previous works in the area of proactive agent such as ProactiveBench, ContextAgent, and FingerTip 20K have studied real user data, historical context, or actively assisted decision-making. And, the task decomposition of "When+How" is also similar to existing methods. Therefore, the description of the defects of the existing datasets and the statement of "the first rigid benchmark" in this paper are not convincing claims.
2.Unconvincing experiment design. The longest context length in the experiment is only 10 minutes, making it difficult to fully evaluate long-term task dependencies. In the experiment about impact of real-world data, only one scheme is used for data synthesis, lacking a more comprehensive comparison. What’s more, in reality, the actual moment when users take LLM interaction may not necessarily be the most suitable time for active intervention.
Questions for authors
-
Which evaluation capabilities does ProAgentBench offer compared to prior work that was previously unattainable?
-
In this article, the moment when users use LLM is considered equivalent to the need for active assistance from the agent. However, the moment when users actually use LLM may not be the best time to intervene, and not using LLM does not mean that users do not need help. Have the author verified or theoretically analyzed that these proxy tags can accurately reflect users' real proactive help needs?
2 comments
Response to Reviewer jC3u (2/2)
Q1. What can researchers evaluate with ProAgentBench?
The combination of logged requests and continuous desktop activity supports four concrete evaluations.
- Anticipating a request. A timestamped query gives an observable target for evaluating whether an agent can anticipate when a user will seek help.
- Identifying evidence before a request. The dataset contains 28,528 events over more than 500 hours, including 7,222 LLM-related events. Event-level screenshots, OCR, and metadata let researchers inspect what was visible before the user asked.
- Measuring false alarms. Negative examples come from moments that resemble genuine requests in context but at which the user did not ask for help. They reveal how often a policy would trigger at such moments and whether higher recall comes with more false alarms.
- Choosing a history length. Researchers can vary the preceding context while keeping the model and test events fixed. The results above show why this matters: extending the window from 30 to 60 minutes improved F1 for Qwen3.8-Max, while DeepSeek-V4-Pro had already reached its highest observed F1 at 30 minutes.
Q2. Does a logged query identify when help would have been most useful?
We agree that a user may have welcomed help before they asked for it. However, for a benchmark, the prediction target must be clear enough to score consistently. The moment a user would welcome an intervention is subjective and difficult to infer from a recording. A user-initiated query, however, gives us a timestamped action through which the user explicitly expresses a request for help. We therefore evaluate whether an agent can anticipate that request using only the activity visible before the user opens the LLM.
The logs also let us check how closely this target relates to the user’s work. We examined all 3,410 query events to assess how closely requests were connected to the ongoing workflow. Of the 3,254 queries with visible preceding activity, 90.7% directly continued or clearly related to that activity, and only 5.1% were unrelated. In the motivation analysis, 94.3% of queries were work-motivated and 5.7% included a curiosity component. These findings show that most recorded requests were grounded in the user’s current work.
Response to Reviewer jC3u (1/2)
We thank the reviewer for the detailed assessment and for recognizing the value of collecting large-scale, real-world interaction data with privacy safeguards. We address W1–W2 and Q1–Q2 below.
W1. What does ProAgentBench add to prior work on proactive assistance?
Prior work has studied proactive assistance using real interaction data. ProAgentBench contributes a specific pairing: a timestamped record of when a user asked for help and a continuous multimodal record of the desktop activity leading up to that request. This lets us evaluate whether an agent could have anticipated an observed request from the information available beforehand.
The When + How formulation builds on prior work. ProAgentBench’s distinctive contribution is that it grounds both questions in the same observed interaction. Among the resources compared above, ProAgentBench uniquely pairs continuous multimodal desktop activity with the time and content of a user’s actual request. The timestamp records when the user sought help, and the query itself records what help they sought. This pairing lets us test whether an agent could anticipate both from the activity preceding the request, how much history it needed to do so, and how often it would also trigger at similar moments when the user continued working without asking. We will replace “first rigorous benchmark” with this more precise account of what ProAgentBench makes possible.
W2. History length, synthetic data, and the interpretation of query timing
History length. The original 10-minute window reflected the models’ context budget. The continuous logs let us evaluate longer histories on the same 6,000 decisions, comprising 3,000 positive and 3,000 negative cases. All runs below used text-only inputs. Values are percentages.
For Qwen3.8-Max, extending the history from 10 to 60 minutes raised recall from 27.2% to 31.5% and F1 from 38.1% to 42.6%. The longer history helped the model anticipate requests it had previously missed. DeepSeek-V4-Pro reached its highest F1 at 30 minutes (54.7%), with no further gain at 60 minutes (54.5%).
Synthetic data. We gave each generator the same candidate timestamps as in the human records, so the timing comparison holds the opportunities to seek help fixed. Qwen-Max produced a burstiness coefficient of B=0.166. Repeating the experiment with GPT-5.1 Terra gave B=0.233, also below the range observed across the 11 participants with sufficient data (0.646–0.835). In a separate training comparison, real data yielded higher scores than synthetic data in all four model–adapter settings, including 74.0 versus 62.1 and 63.5 versus 54.8.
Query timing. The reviewer raises an important distinction between when a user asks for help and when an intervention would be most useful. Q2 explains what the logs establish and how we will revise the task definition.