acceptodds
Under review as a conference paper at ICLR 2027

ProAgentBench: Evaluating LLM Agents for Proactive Assistance with Real-World Data

Abstract

Proactive agents that anticipate user intentions without explicit prompts represent a significant evolution in human-AI interaction, promising to reduce cognitive load and streamline workflows. However, existing datasets suffer from two critical deficiencies: (1) reliance on LLM-synthesized data that fails to capture authentic human decision-making patterns, and (2) focus on isolated tasks rather than continuous workflows, missing the pre-assistance behavioral context essential for learning proactive intervention signals. To address these gaps, we introduce **ProAgentBench**, a rigorous benchmark for proactive agents in working scenarios. We study whether an agent can anticipate when a user will initiate an LLM interaction and what assistance the user will subsequently request, using only the workflow observations available before the interaction. Our contributions include: (1) a hierarchical task framework that decomposes proactive assistance into timing prediction and assist content generation; (2) a privacy-compliant dataset with 28,000+ events from 500+ hours of real user sessions, preserving bursty interaction patterns (burstiness =0.787) absent in synthetic data; and (3) extensive experiments that evaluates LLM- and VLM-based baselines. Numerically, we showed that long-term memory and historical context significantly enhance prediction accuracy, while real-world training data substantially outperforms synthetic alternatives. We release our dataset and code at https://anonymous.4open.science/r/ProAgentBench-6BC0

Primary area:
datasets and benchmarks
Keywords:
Proactive agent, Large language model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_jC3uOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Existing datasets in the area of proactive agent suffer from two critical deficiencies: (1) reliance on LLM-synthesized data that fails to capture authentic human decision-making patterns, and (2) focus on isolated tasks rather than continuous workflows, missing the pre-assistance behavioral context essential for learning proactive intervention signals. So in this paper, a rigorous benchmark, ProAgentBench, is introduced to evaluate proactive agents in working scenarios. This benchmark captures over 28,000 events from 500+ hours of continuous working sessions, and the authors formalize a “When + How” hierarchical task framework to express the proactive assistance. Experimentally, extensive main experiments are conducted across diverse models and methods, and ablation study is designed to explore three main research questions, namely impact of historical observation sequence length, user context, and real-world data.

Critical strengths

Real datasets have value. A large-scale privacy-compliant real-world dataset is constructed, which captures over 28,000 events from 500+ hours of authentic user activities, preserving critical pre-assistance behavioral patterns.

Critical weaknesses

1.Limited novelty. Previous works in the area of proactive agent such as ProactiveBench, ContextAgent, and FingerTip 20K have studied real user data, historical context, or actively assisted decision-making. And, the task decomposition of "When+How" is also similar to existing methods. Therefore, the description of the defects of the existing datasets and the statement of "the first rigid benchmark" in this paper are not convincing claims.

2.Unconvincing experiment design. The longest context length in the experiment is only 10 minutes, making it difficult to fully evaluate long-term task dependencies. In the experiment about impact of real-world data, only one scheme is used for data synthesis, lacking a more comprehensive comparison. What’s more, in reality, the actual moment when users take LLM interaction may not necessarily be the most suitable time for active intervention.

Questions for authors
  1. Which evaluation capabilities does ProAgentBench offer compared to prior work that was previously unattainable?

  2. In this article, the moment when users use LLM is considered equivalent to the need for active assistance from the agent. However, the moment when users actually use LLM may not be the best time to intervene, and not using LLM does not mean that users do not need help. Have the author verified or theoretically analyzed that these proxy tags can accurately reflect users' real proactive help needs?

2 comments
AuthorsOpenReview ↗

Response to Reviewer jC3u (1/2)

We thank the reviewer for the detailed assessment and for recognizing the value of collecting large-scale, real-world interaction data with privacy safeguards. We address W1–W2 and Q1–Q2 below.

W1. What does ProAgentBench add to prior work on proactive assistance?

Prior work has studied proactive assistance using real interaction data. ProAgentBench contributes a specific pairing: a timestamped record of when a user asked for help and a continuous multimodal record of the desktop activity leading up to that request. This lets us evaluate whether an agent could have anticipated an observed request from the information available beforehand.

ProAgentBenchProactiveBenchContextAgentBenchFingerTip 20K
Real multimodal✓ screenshots + logs✕ text events◐ Lite only✓ screenshots
Desktop, continuous✓◐✕✕
Timing label source✓ logged queries◐ simulated✕ authored✕ task demos

The When + How formulation builds on prior work. ProAgentBench’s distinctive contribution is that it grounds both questions in the same observed interaction. Among the resources compared above, ProAgentBench uniquely pairs continuous multimodal desktop activity with the time and content of a user’s actual request. The timestamp records when the user sought help, and the query itself records what help they sought. This pairing lets us test whether an agent could anticipate both from the activity preceding the request, how much history it needed to do so, and how often it would also trigger at similar moments when the user continued working without asking. We will replace “first rigorous benchmark” with this more precise account of what ProAgentBench makes possible.

W2. History length, synthetic data, and the interpretation of query timing

History length. The original 10-minute window reflected the models’ context budget. The continuous logs let us evaluate longer histories on the same 6,000 decisions, comprising 3,000 positive and 3,000 negative cases. All runs below used text-only inputs. Values are percentages.

HistoryQwen3.8-Max AccF1RecallDeepSeek-V4-Pro AccF1Recall
10 min55.938.127.261.753.844.5
20 min56.540.329.362.054.345.2
30 min56.641.030.261.754.746.2
60 min57.642.631.561.754.545.8

For Qwen3.8-Max, extending the history from 10 to 60 minutes raised recall from 27.2% to 31.5% and F1 from 38.1% to 42.6%. The longer history helped the model anticipate requests it had previously missed. DeepSeek-V4-Pro reached its highest F1 at 30 minutes (54.7%), with no further gain at 60 minutes (54.5%).

Synthetic data. We gave each generator the same candidate timestamps as in the human records, so the timing comparison holds the opportunities to seek help fixed. Qwen-Max produced a burstiness coefficient of B=0.166B=0.166. Repeating the experiment with GPT-5.1 Terra gave B=0.233B=0.233, also below the range observed across the 11 participants with sufficient data (0.646–0.835). In a separate training comparison, real data yielded higher scores than synthetic data in all four model–adapter settings, including 74.0 versus 62.1 and 63.5 versus 54.8.

GroupUsersMean BBSD across users
Undergraduate30.7860.078
Graduate80.7870.072
All110.787range 0.646–0.835

Query timing. The reviewer raises an important distinction between when a user asks for help and when an intervention would be most useful. Q2 explains what the logs establish and how we will revise the task definition.

AuthorsOpenReview ↗

Response to Reviewer jC3u (2/2)

Q1. What can researchers evaluate with ProAgentBench?

The combination of logged requests and continuous desktop activity supports four concrete evaluations.

  1. Anticipating a request. A timestamped query gives an observable target for evaluating whether an agent can anticipate when a user will seek help.
  2. Identifying evidence before a request. The dataset contains 28,528 events over more than 500 hours, including 7,222 LLM-related events. Event-level screenshots, OCR, and metadata let researchers inspect what was visible before the user asked.
  3. Measuring false alarms. Negative examples come from moments that resemble genuine requests in context but at which the user did not ask for help. They reveal how often a policy would trigger at such moments and whether higher recall comes with more false alarms.
  4. Choosing a history length. Researchers can vary the preceding context while keeping the model and test events fixed. The results above show why this matters: extending the window from 30 to 60 minutes improved F1 for Qwen3.8-Max, while DeepSeek-V4-Pro had already reached its highest observed F1 at 30 minutes.

Q2. Does a logged query identify when help would have been most useful?

We agree that a user may have welcomed help before they asked for it. However, for a benchmark, the prediction target must be clear enough to score consistently. The moment a user would welcome an intervention is subjective and difficult to infer from a recording. A user-initiated query, however, gives us a timestamped action through which the user explicitly expresses a request for help. We therefore evaluate whether an agent can anticipate that request using only the activity visible before the user opens the LLM.

The logs also let us check how closely this target relates to the user’s work. We examined all 3,410 query events to assess how closely requests were connected to the ongoing workflow. Of the 3,254 queries with visible preceding activity, 90.7% directly continued or clearly related to that activity, and only 5.1% were unrelated. In the motivation analysis, 94.3% of queries were work-motivated and 5.7% included a curiosity component. These findings show that most recorded requests were grounded in the user’s current work.

Reviewer_LWK6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper tries to introduce the ProAgentBench, a benchmark to evaluate whether LLM agents can anticipate when users will seek AI assistance and what help they will request, using only preceding computer activity? The paper also investigates whether context, user memory, and authentic training data improve this ability.

The benchmark is constructed from continuous desktop workflows recorded from 17 student participants. It evaluates two tasks: predicting assistance timing and inferring request content. The authors provide baseline comparisons across six models and experiments examining prompting strategies, context length, memory representations, input modalities, and real versus synthetic training data.

Critical strengths

The paper's strength lies in its use of observed human behavior in evaluating the proactive assitance. The benchmark they introduce in the paper links activity preceding AI use with subsequent requests, enabling assessment of timing prediction and intent inference without providing the request itself.

Critical weaknesses

A central weakness of the paper is that it does not sufficiently demonstrate the added benefit of its benchmark relative to existing resources. Although the authors emphasize continuous recordings, behavior, and context preceding assistance requests, these characteristics alone do not demonstrate improved evaluation validity or greater usefulness for developing proactive agents. The context length ablation supports the value of historical information, and the comparison between real and synthetic training data demonstrates benefits under the tested conditions. However, neither experiment isolates continuous recording as the source of improvement or establishes an advantage over existing benchmarks.

Questions for authors

How is the benchmark introduced help assess or develop proactive agent in a different way and can the authors justify this?

1 comment
AuthorsOpenReview ↗

Response to Reviewer LWK6

We thank the reviewer for recognizing the value of grounding proactive assistance in observed human behaviour and for asking what this enables us to evaluate. We address W1 and Q1 below.

W1. "No added benefit over existing resources, continuity not isolated"

ProAgentBench pairs a continuous multimodal record of desktop activity with the timestamp at which a user actually asks for help. This pairing lets us test whether an agent can anticipate the same request when given different amounts of preceding activity.

Our history-length experiment makes this comparison within the same test events. Extending the history from 10 seconds to 60 minutes improves Qwen3.8-Max’s accuracy by 1.7 percentage points and its F1 by 4.5 points. The main gain comes from recall, which rises from 27.2% to 31.5%: with more history, the model anticipates requests it previously missed. For DeepSeek-V4-Pro, performance plateaus at 30 minutes. These results show how much each model benefits from a longer history on ProAgentBench.

The existing resources support different comparisons:

ProAgentBenchProactiveBenchContextAgentBenchFingerTip 20K
Real multimodal✓ screenshots + logs✕ text events◐ Lite only✓ screenshots
Desktop, continuous✓◐✕✕
Timing label source✓ logged queries◐ simulated✕ authored✕ task demos

FingerTip 20K has real continuous mobile activity, but its episodes are demonstrations the user chose to record, so a timing label there is missed. ProactiveBench logs queries but converts the surrounding activity to text and trains on simulation. ContextAgentBench is LLM-authored. Ours is the only one where the label is a logged query on a continuous desktop workflow.

Q1. How does it help assess or develop proactive agents differently?

ProAgentBench supports four concrete evaluation and design decisions.

  1. What to predict. A logged query gives the agent an observable target: whether the user will ask for help at a given moment.
  2. What evidence to use. The dataset contains 28,528 events over more than 500 hours, including 7,222 LLM-related events. Event-level screenshots, OCR, and metadata let researchers examine what was visible to the agent before a request.
  3. When to stay silent. The benchmark includes moments that resemble genuine triggers but at which the user did not ask for help (line 899–902). These cases let researchers measure how often an agent would interrupt and check whether a gain in recall comes at the cost of more false alarms.
  4. How much history to retain. Varying the history up to 60 minutes reveals whether additional context helps recover missed requests and where those gains level off. Here are the results for the evalutions:
HistoryQwen3.8-Max AccF1RecallDeepSeek-V4-Pro AccF1Recall
10 min55.938.127.261.753.844.5
20 min56.540.329.362.054.345.2
30 min56.641.030.261.754.746.2
60 min57.642.631.561.754.545.8
Reviewer_954KOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper provides a potentially valuable real-world resource for proactive-assistance research by collecting continuous desktop activity and preserving context before users initiate LLM interactions. The pipeline covers event segmentation, quality filtering, privacy screening, automatic annotation, temporal splitting, short- and long-term context, real-versus-synthetic training data, multimodal inputs, and latency analysis. The observed difference in burstiness between human and synthetic interactions is also potentially interesting.

Critical strengths

The observed difference in burstiness between human and synthetic interactions is also potentially interesting.

Critical weaknesses
  1. This paper does not provide end-to-end evidence that proactive assistance improves user outcomes. It evaluates help-seeking prediction and query similarity, but not task success, completion time, or interruption cost.
  2. The benchmark distribution and sampling protocol are under-specified. The timing task uses an approximately 50% positive rate despite a much lower raw prevalence, and the paper does not report sufficient per-split or per-intent statistics.
  3. The automatically generated labels are weakly validated. The reported 97.6% annotation success rate is not clearly defined, and separate validation results for event detection, query extraction, and intention labeling are missing.
  4. The burstiness result is interesting but not fully supported. The paper does not report per-user uncertainty or sufficiently compare alternative temporal models, while the synthetic comparison relies on only one LLM-based generation protocol.
Questions for authors
  1. How was the 97.6% annotation success rate computed, and what are the separate validation results for event detection, query extraction, and intention labeling?
  2. Why do the Accuracy values in Table 2 differ from the TP+TN values implied by Table 7 in Appendx, and what is the exact definition of FAR?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 954K (1/2)

We thank the reviewer for their constructive feedback and for their appreciation of the discovery of burstiness. Here are we response to each concern.

W1. "No end-to-end evidence that proactive assistance improves outcomes"

We kindly point out that this topic lies beyond the scope of the paper. ProAgentBench studies when a user actively seeks help, which is an observable, well-defined signal of proactive need, recorded from the user's own behaviour rather than inferred by an annotator. End-to-end evidence such as task success or completion time is produced jointly by the agent and the user as it absorbs the user's own capability, effort and work habits into the score, so a difference between two models can no longer be attributed to the model's anticipation. We therefore measure the agent-side quantity on its own. The joint outcome requires a deployment study, which we identify as future work. Furthermore, we argue that interruption cost can be measured offline, and we have measured it with a false alarm (Table 3), which is an interruption at a moment that looks like a real trigger but is not.

W2. "50% positive rate despite lower prevalence: missing statistics"

The rate is under design to equally measure the performance of negative and positive cases. In the raw logs only 25.3% of events involve an LLM, so a test set that kept the natural ratio would let a model score well by always answering "no help needed" and fixing the ratio at 1:1 makes both classes count equally.

W3 & Q1. "97.6% undefined, per-stage validation missing"

It is the share of inspected LLM-related events whose annotation was judged correct. For each event the annotator compared the pipeline output against the screen content and counted the event as a success only if four conditions held: (1) the event was correctly identified as an LLM interaction, (2) the platform and interaction type were right, (3) the extracted conversation contained the user's queries, and (4) the output was valid, parseable JSON. A failure in any of the four counted the event as unsuccessful. The 2.4% failure cases are because the query of user is under input or get covered, making the failure in extracting the conversations.

W4. "Burstiness unsupported"

We thank you for find the burstiness results interesting.

On per-user uncertainty, we report the results as following:

Per-user values, computed on each participant's own inter-event times (dropping users with fewer than 30 intervals, which leaves 11 valid users):

GroupUsersMean BBSD across users
Undergraduate30.7860.078
Graduate80.7870.072
All110.787range 0.646–0.835

Moreover, the least bursty participant sits at B=0.646B = 0.646 against a Poisson baseline of 00, and the spread is small relative to that distance, so the pooled value describes every user rather than averaging quiet and bursty ones. The per-user table with a bootstrap interval for each participant will be added to the appendix.

On alternative temporal models, we would like to ponit it out that the burstiness score itself makes no distributional assumption, as BB is a function of the mean and standard deviation of the inter-event times alone, so the claim that human interaction is bursty holds whatever family one fits. Where a family is fitted, the relevant alternative is the exponential, because that is the inter-event distribution of a Poisson process, which is exactly the process our synthetic baseline instantiates, and the likelihood-ratio test between the two is the standard heavy-tail comparison[1]. It gives 2951.48 for the human data and −59.36 for the synthetic, which is the comparison that decides between bursty and Poisson. We will add Weibull and log-normal fits as a robustness check.

On the generation protocol, the synthetic series comes from one generator (Qwen-Max, following Sun et al.). The comparison is controlled on the sampling axis as the generator decided at exactly the same candidate timestamps as the human records, so the two series share one timeline and the drop to B=0.166B = 0.166 reflects the decisions rather than the spacing. We have repeat it with GPT-5.1 Terra on the same timestamps, and the results turns to be B=0.233B=0.233, which also supports our claim. We will report this in the revision.

[1] Clauset, Aaron, Cosma Rohilla Shalizi, and Mark EJ Newman. "Power-law distributions in empirical data." SIAM review 51.4 (2009): 661-703.

AuthorsOpenReview ↗

Response to Reviewer 954K (1/2)

Q2. Table 2 vs Table 7, and FAR

The mismatch is due to the different treatments for failed parsed outputs. Each test sample asks the model for two decisions: whether assistance should be triggered, and if so which intent. The evaluator reads that decision out of a fixed output format, so an answer that does not contain it in the expected form cannot be scored. CoT makes this frequent, because a long reasoning trace sometimes ends without the required decision line. Table 2 scores those answers as "no assistance triggered", i.e. as a miss, and computes accuracy over all samples. Table 7 uses the successfully parsed samples only: it takes their precision and recall, and reconstructs TP/TN/FP/FN assuming a single 50/50 denominator. That assumption puts back the answers that were unparseable, so wherever parsing failed the reconstructed accuracy is higher than the accuracy in Table 2.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of rebuttal phase

Dear Area Chair,

Thank you for considering ProAgentBench. The ratings are 2 (weak rejection) from reviewer 954K and reviewer LWK6, and 1 (clear rejection) from jC3u. All three reviewers identified value in the dataset or its findings. We responded to all three but none has posted a public follow-up.

1. What can ProAgentBench evaluate that existing benchmarks cannot? Reviewer LWK6 asked what continuous recording adds to evaluation. Reviewer jC3u questioned the novelty of the “when and how” formulation and whether ten minutes of history was sufficient. We clarified that our contribution is pairing an actual request’s time and content with the desktop activity preceding it. This lets us vary how much history an agent sees while keeping the request fixed. We conduct a history-length experiment on the same 6,000 decisions. Extending history from 10 to 60 minutes raises Qwen3.8-Max’s recall from 27.2% to 31.5% and F1 from 38.1% to 42.6%. DeepSeek-V4-Pro reaches its highest F1 at 30 minutes. We also agreed to replace the broad novelty claim with this specific contribution.

2. Does a logged request tell us when help would be useful? Reviewer jC3u asked whether users might welcome help before they ask. Reviewer 954K asked whether proactive assistance improves task outcomes or justifies an interruption. We clarified that the benchmark measures whether an agent can anticipate an observed request. We examined all 3,410 query events. Among the 3,254 with visible preceding activity, 90.7% continued or clearly related to that activity. We also explained that false alarms measure how often an agent would interrupt when the user continued working without asking. Establishing the best intervention time or its effect on task completion requires a deployment study.

3. How robust is the comparison between real and synthetic data? Reviewer 954K asked whether the burstiness finding holds across users and temporal models. Both reviewer 954K and jC3u questioned reliance on one generator. We repeat the timing experiment with a second generator using the same candidate timestamps. Burstiness was 0.166 for Qwen-Max and 0.233 for GPT-5.1 Terra, below every one of the 11 participants with sufficient data (0.646–0.835). We also reported that training on real data outperformed synthetic data in all four model–adapter settings.

4. How reliable are the annotations and reported metrics? Reviewer 954K asked about the balanced test set, the 97.6% annotation success rate, and discrepancies between the accuracy tables. We explained that 25.3% of raw events involved an LLM and that we balanced the timing test to give requests and non-requests equal weight. We specified the four checks each inspected event had to pass to count toward 97.6%. We also traced the accuracy discrepancy to how the two tables handled unparseable outputs.

Best regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.