acceptodds
Under review as a conference paper at ICLR 2027

Ping: Learning Proactive Agents Over User Trajectories

Abstract

Most existing AI agents are reactive: they wait for users to specify a task, leaving users to notice problems and decide what help to request. On the other hand, a helpful human colleague may offer advice without being asked. In an ideal world, agents should also be able to do the same, offering useful proactive assistance that users might not have considered. Such assistance could help users catch overlooked problems and avoid unnecessary work, without requiring them to ask an agent to check for each possibility. Thus, we introduce Ping, a framework for benchmarking the relevance and timeliness of proactive assistance over user trajectories in software engineering and productivity domains. We derive taxonomies from GitHub comments and meeting transcripts to identify the kinds of proactive assistance agents could offer. We evaluate agents against references that describe the assistance they should proactively offer. Each reference has an assistance window specifying when that assistance should be offered. We use the GitHub taxonomy to extract references for SWE-chat coding trajectories and the meeting taxonomy to categorize existing references in PARE's productivity scenarios. We evaluate five baseline LLMs on 100 SWE-chat test instances and 143 PARE instances. Most references remain unmatched: the highest macro recall among these baselines is 7.0% on SWE-chat and 20.3% on PARE. We also fine-tune agents on user trajectories, using examples of when to offer assistance and when to remain silent. This training increases macro recall on SWE-chat by up to 13.5 percentage points over the corresponding base models. We hope Ping will support the development of proactive agents that can identify useful assistance and offer it at the right time, without waiting for users to ask.

Primary area:
datasets and benchmarks
Keywords:
proactive agent, user trajectory, coding, productivity

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_gTErOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tackles a critical real-world problem: supervisors (i.e., users) can make mistakes during human-in-the-loop interaction. Evaluating agents in such settings is well-motivated. The experimental results shows that frontier LLMs suffer from this scenario.

Critical strengths
  • This paper tackles a critical real-world problem: supervisors (i.e., users) can make mistakes during human-in-the-loop interaction. Evaluating agents in such settings is well-motivated.
  • The paper is well-structured.
Critical weaknesses
  • One limitation of offline benchmarks is that it is difficult to evaluate agents in environments that capture real-world complexity. This is why current evaluation frameworks are mostly based on end-to-end simulation environments. To make a clear contribution to the agent research field as an offline benchmark, this paper should conduct more thorough experiments and show significant results. Currently, however, the experiments mainly focus on the low scores of frontier LLMs. I suggest that the authors conduct more meaningful experiments (e.g., how serious a problem this could be in real-world deployment, and whether it can be mitigated by means other than SFT) and, if possible, move the table from the appendix to Section 5.
  • The setting of this paper treats the agent as an observer, not an actor that actually carries out the tasks delegated by the user. This lacks realism in the benchmark scenario, because real-world agents do not only monitor; they should also perform actions in real time.
  • If my understanding is correct, the agent only outputs responses in the (content, intervention) format. I think this limits analysis of the experimental results from diverse perspectives.
  • It is unclear how this paper builds the live context, workspace, and knowledge pool. These should be described in more detail.
  • It is surprising that even frontier LLMs such as GPT-5.6-sol and GPT-5.4 show low performance on this dataset, because the example scenarios in Figures 1-3 seem easy enough to recognize that the user’s instruction is somewhat wrong. One major concern is that this benchmark may contain many unsolvable or ambiguous cases. To address this concern, the authors should clearly describe the data synthesis process and show that it reliably generates feasible scenarios.
Questions for authors
  • Please add the agent prompt to the appendix.
  • Please add example cases from the generated dataset, and clearly state what the input to the agent consists of.
  • Scores should be reported in percentages (e.g., 0.054 → 5.4%).
4 comments
AuthorsOpenReview ↗

Rebuttal 1/2

Thank you for recognizing the importance of evaluating proactive assistance and the structure of our paper. We respond to your concerns below.

W.1: One limitation of offline benchmarks is that it is difficult to evaluate agents in environments that capture real-world complexity. This is why current evaluation frameworks are mostly based on end-to-end simulation environments. To make a clear contribution to the agent research field as an offline benchmark, this paper should conduct more thorough experiments and show significant results. Currently, however, the experiments mainly focus on the low scores of frontier LLMs. I suggest that the authors conduct more meaningful experiments (e.g., how serious a problem this could be in real-world deployment, and whether it can be mitigated by means other than SFT) and, if possible, move the table from the appendix to Section 5.

We agree with your concern that low scores alone do not tell us how to improve proactive assistance. Thus, we want to highlight our finding that providing workspace and knowledge-pool information improves GPT-5.4's recall and precision on PARE without training. Adding these sources to live context increases recall from 2.1% to 7.7% and precision from 1.7% to 6.2% (Appendix D.5). Additionally, full event details increase recall for all four tested models on PARE, for example by adding email bodies to event summaries (Appendix D.4). These findings suggest that low scores partly reflect missing information in the input. To address your concern, we have highlighted these findings in Section 5.3 of the main text.

W.2: The setting of this paper treats the agent as an observer, not an actor that actually carries out the tasks delegated by the user. This lacks realism in the benchmark scenario, because real-world agents do not only monitor; they should also perform actions in real time.

We agree that executing actions is important for a proactive agent. In a separate experiment with PARE, we evaluate whether agents can execute actions to complete tasks. GPT-5.4 completes 17.5% of tasks according to PARE's validators. Appendix C describes this experiment, which uses a simulated user to accept or reject proposed actions before execution.

Monitoring the user's activity is a first step toward proactive assistance. The agent needs to use what it observes to decide what help to offer and when, without waiting for a user request. Our main evaluation studies this ability by having agents observe the same user trajectories, so we can compare the relevance and timeliness of their interventions.

W.3: If my understanding is correct, the agent only outputs responses in the (content, intervention) format. I think this limits analysis of the experimental results from diverse perspectives.

Thank you for raising this point. The output records whether the agent intervenes, what it says, and when, allowing us to analyze both relevance and timeliness. We measure recall, precision, match rate, in-window recall, and intervention rate, and compare results across types of assistance. In PARE, an intervention also specifies an application, an action, and the inputs needed to carry it out (Section 4).

To address your concern, we have expanded the discussion in Section 5.3 to examine whether higher scores simply come from more frequent interventions. We compare equal numbers of saved interventions from each run, with the full analyses in Appendices D.3–D.4.

W.4: It is unclear how this paper builds the live context, workspace, and knowledge pool. These should be described in more detail.

Thank you for pointing out that the inputs need a clearer explanation. Section 4 and Figure 3 explain the three context sources, and Appendices B.5–B.6 describe their preparation and use in each domain. The three sources serve the same roles across domains, while their implementation follows the available data.

Live context contains the user trajectory up to the current decision point. In SWE-chat, this includes developer messages and the coding agent's responses and actions. In PARE, it includes user actions and environment events.

Workspace describes the initial working environment. For SWE-chat, we provide an overview of the repository and its files. For PARE, we provide the initial application state, such as inbox contents and calendar entries.

The knowledge pool contains additional pull requests (PRs) and issues in SWE-chat, or records of past application activity in PARE. We exclude the SWE-chat discussions used to construct the instance's references.

AuthorsOpenReview ↗

Rebuttal 2/2

W.5: It is surprising that even frontier LLMs such as GPT-5.6-sol and GPT-5.4 show low performance on this dataset, because the example scenarios in Figures 1-3 seem easy enough to recognize that the user’s instruction is somewhat wrong. One major concern is that this benchmark may contain many unsolvable or ambiguous cases. To address this concern, the authors should clearly describe the data synthesis process and show that it reliably generates feasible scenarios.

We agree that we should explain how references are constructed and why the assistance is feasible. SWE-chat references come from existing PR and issue comments, which can identify problems the developer and coding agent missed. The proactive agent must identify useful assistance from the context available at the decision point and offer it while it can still help the user.

During construction, an LLM checks whether each candidate reference is supported by the available context, addresses the user's task, and describes assistance the user could act on while it is still useful. Only candidates that pass these checks become references (Section 3.2).

Figure 2 in Section 3.2 illustrates how a PR comment becomes a reference and how assistance windows are used in evaluation.

We agree that annotation quality needs to be checked directly. Two annotators each completed 100 judgments across SWE-chat and PARE. They assessed reference validity, intervention matching, and usefulness at selected decision points.

Both annotators agreed with all sampled matching labels and judged all sampled in-window points useful, supporting the matching decisions and the usefulness of assistance within the sampled windows.

For the sampled references, both annotators confirmed supporting evidence for 93.8% and judged 81.3% to go beyond explicit requests. At least one annotator judged 93.8% useful. Overall agreement across the evaluation was 89.0%. Appendix E.3 reports the procedure and full results.

Q.1: Please add the agent prompt to the appendix.

Thank you for this suggestion. The SWE-chat decision prompt asks the agent to offer assistance only when it is useful at the current turn and supported by the supplied context. The PARE prompt asks whether there is a grounded, actionable opportunity that the user has not already handled, and requests the proposed application action and its inputs. Appendix B.5 describes the retrieval and decision procedure. We will include the full decision and retrieval prompts with the code release.

Q.2: Please add example cases from the generated dataset, and clearly state what the input to the agent consists of.

Thank you for this suggestion. Section 4 and Figure 3 illustrate the three context sources and the resulting intervention using the API-field rename example. Section 5.4 examines one SWE-chat case and one PARE case, and Appendix D.4 shows how email details help agents identify eligible expenses in a PARE reimbursement task. Appendices B.5–B.6 describe the agent inputs and observation rules. References and assistance windows are used only for evaluation and are not given to the agent.

Q.3: Scores should be reported in percentages (e.g., 0.054 → 5.4%).

Thank you for pointing this out. We now report rates as percentages in Section 5 and the appendix result tables, using consistent metric names.

We hope these clarifications and revisions address your concerns. If so, we would appreciate it if you could reconsider your rating.

Thank you for your reply. I will keep my scores.

Thank you again for your time and for considering our responses. If possible, we would appreciate any further feedback on the concerns that remain, particularly regarding the additional experiments and human evaluation. Your feedback would help us better understand your assessment and improve the paper. We would be happy to provide further clarification if helpful.

Reviewer_vPNmOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents a framework Ping that benchmarks AI assistance on the i) relevance and ii) timeliness dimensions for both software engineering and productivity domains. To label or categorize the type of proactive assistance, the authors proposed a taxonomy for both domains using a LLM. These taxonomies are used as a “reference” label for user trajectories to evaluate whether the agent provide proactive assistance to the user. In addition to that, the authors also show that an agent’s proactive intervention is considered beneficial only if the intervention occurs in a timely window period which the authors refer to as “assistance window”.

Critical strengths
  1. The paper addresses an important and underexplored problem of the need to benchmark and quantify the proactiveness aspect of an AI agent.
  2. The paper proposes two interesting dimensions (relevance and timeliness) for evaluating an agent’s proactiveness. The two dimensions consider an agent contribution to be proactive only if it occurs at the right time and goes beyond what the user requests.
Critical weaknesses
  1. The empirical results are limited, with some results for the dataset missing (refer to question section below) which makes one question whether the findings are generalizable.
  2. The quantitative results show very low agent proactive scores performance, even after supervised finetuning which seem abit strange. This makes me question the correctness of the benchmark (see further details in the question section below).
Questions for authors
  1. List of contributions are not properly articulated. Please clarify how your benchmark differs from existing works and what novel contributions it offers beyond prior work?
  2. Authors should move the related work from the appendix to the main text. Additionally, the paper would benefit from a broader discussion of related works. The current paper only mentioned six references in the related work section.
  3. Clarification: L269. Authors claimed that the match rate metric is the fraction of instances in which at least one intervention matches a reference. But a single instance could have many references. To compute this metric, do we consider this instance to be successful if all the references for that instance are matched at least once? In other words, the recall score for that instance should be 1 to contribute to the match rate, otherwise the contribution to match rate would be zero? Or does an instance count as successful as long as at least one intervention matches at least one reference? Which interpretation is correct for the match rate metric? Please clarify this.
  4. Why gpt4.1 family models are missing in Table 3?
  5. Based on Table 4, SFT does not seem to help much on the SWE-chat test dataset. Why is that so? The low scores even after fine-tuning somewhat suggest that perhaps, the reference ground truth and assistance window annotation may not be correct. How do the authors verify that the ground truths for the benchmark are indeed correct, especially for SWE-chat as the ground truths are generated by LLM (Line 281)?
  6. Table 4 is also missing SFT results for PARE dataset.
  7. How many times did the authors run the experiments for the same test case? Due to non-determinism of LLM, it is standard practice to run every testcase multiple times.
  8. Code and dataset are not provided.
3 comments
AuthorsOpenReview ↗

Rebuttal 1/2

Thank you for recognizing the importance of proactive agent evaluation and the value of measuring both relevance and timeliness. We address your concerns about the contribution, evaluation, and reproducibility below.

W.1: The empirical results are limited, with some results for the dataset missing (refer to question section below) which makes one question whether the findings are generalizable.

Q.6: Table 4 is also missing SFT results for PARE dataset.

We agree that testing SFT in both domains would help assess generalization. Our SFT results in Table 4 cover SWE-chat only, while Table 3 evaluates five baseline models in both SWE-chat and PARE. To address your concern, we have specified the SFT training and evaluation domains in Section 5.2 and the Table 4 caption.

W.2: The quantitative results show very low agent proactive scores performance, even after supervised finetuning which seem abit strange. This makes me question the correctness of the benchmark (see further details in the question section below).

Q.5: Based on Table 4, SFT does not seem to help much on the SWE-chat test dataset. Why is that so? The low scores even after fine-tuning somewhat suggest that perhaps, the reference ground truth and assistance window annotation may not be correct. How do the authors verify that the ground truths for the benchmark are indeed correct, especially for SWE-chat as the ground truths are generated by LLM (Line 281)?

Thank you for raising this concern. We would like to clarify that SFT increases GPT-4.1-mini's recall from 4.3% to 17.8% (Table 4).

We agree that we should explain how references are constructed and why the assistance is feasible. SWE-chat references come from existing PR and issue comments, which can identify problems the developer and coding agent missed. The proactive agent must identify useful assistance from the context available at the decision point and offer it while it can still help the user.

We agree that annotation quality needs to be checked directly. Two annotators each completed 100 judgments across SWE-chat and PARE. They assessed reference validity, intervention matching, and usefulness at selected decision points.

Both annotators agreed with all sampled matching labels and judged all sampled in-window points useful, supporting the matching decisions and the usefulness of assistance within the sampled windows.

For the sampled references, both annotators confirmed supporting evidence for 93.8% and judged 81.3% to go beyond explicit requests. At least one annotator judged 93.8% useful. Overall agreement across the evaluation was 89.0%. Appendix E.3 reports the procedure and full results.

Figure 2 in Section 3.2 illustrates how a PR comment becomes a reference.

Q.1: List of contributions are not properly articulated. Please clarify how your benchmark differs from existing works and what novel contributions it offers beyond prior work?

Q.2: Authors should move the related work from the appendix to the main text. Additionally, the paper would benefit from a broader discussion of related works. The current paper only mentioned six references in the related work section.

We agree that Ping's contribution should be clearer. To address your concerns, we have moved the related work from the appendix to Section 6 of the revised main text and added the comparison below (Table 5).

FeatureProactive AgentPROBEPAREPing
Assistance without explicit user requests✓✓✓✓
Decisions as user activity progresses✓✗✓✓
References derived from real-world task feedback✗✗✗✓
Assistance categories identified from real discussions✗✗✗✓
Recall with and without assistance windows✗✗✗✓

Table 5. Comparison of proactive assistance benchmarks.

Instead of asking agents to predict the user's next action and do it in advance, Ping evaluates whether they can offer useful help that users may not think to request. We construct our taxonomies from the proactive help people offer each other in real GitHub discussions and meetings, grouping contributions such as pointing out overlooked problems or sharing useful resources.

In SWE-chat, review feedback can reveal problems that the developer and coding agent missed. We use the GitHub taxonomy to construct references from this feedback and annotate assistance windows to evaluate whether agents offer that help while it is still useful.

We have also expanded our discussion of related work on next-action prediction and code-review evaluation, and added work on LLM-based taxonomy construction to explain how we identify assistance types and guide benchmark construction.

We would also welcome any related work you think would strengthen this comparison.

AuthorsOpenReview ↗

Rebuttal 2/2

Q.3: Clarification: L269. Authors claimed that the match rate metric is the fraction of instances in which at least one intervention matches a reference. But a single instance could have many references. To compute this metric, do we consider this instance to be successful if all the references for that instance are matched at least once? In other words, the recall score for that instance should be 1 to contribute to the match rate, otherwise the contribution to match rate would be zero? Or does an instance count as successful as long as at least one intervention matches at least one reference? Which interpretation is correct for the match rate metric? Please clarify this.

Thank you for asking us to clarify this. The second interpretation is correct. An instance counts toward match rate when at least one intervention matches at least one reference. In other words, match rate measures how often an agent provides at least one useful intervention during a user trajectory, as assessed against the references. Recall measures the fraction of references matched, while precision measures the fraction of interventions that match a reference.

For example, suppose an instance has four references and two interventions. If one intervention matches exactly one reference and the other matches none, the instance is counted as a match, with 25% recall and 50% precision. To address your concern, we have clarified these definitions and added this example in Section 3.3.

Q.4: Why gpt4.1 family models are missing in Table 3?

The base GPT-4.1 models are reported in Table 4, beside their SFT counterparts so that readers can compare performance before and after training. Table 3 reports the five models evaluated in both domains. To address your concern, we have made the location of the GPT-4.1-family results explicit in Section 5.1.

Q.7: How many times did the authors run the experiments for the same test case? Due to non-determinism of LLM, it is standard practice to run every testcase multiple times.

Thank you for raising this concern. We evaluated each model once under each input setting to keep the computational cost manageable. We agree that repeated runs would provide a stronger assessment of result stability. To address your concern, we have clarified the run count in Section 5.1 and discussed its scope in Appendix A.

Q.8: Code and dataset are not provided.

Thank you for raising this point. We will release both the code and dataset upon acceptance of the paper for reproducibility.

We hope these clarifications and revisions address your concerns. If so, we would appreciate it if you could reconsider your rating.

Reviewer_vPNmOpenReview ↗

Official comment from Reviewer vPNm

Thank you for your rebuttal. However since the code and dataset are still not provided, together with my concern of low scores even after fine-tuning, I decided to maintain my rating.

Reviewer_gFkMOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces Ping, a benchmark for proactive assistance over user trajectories in software engineering and productivity. For software engineering, references (expected proactive contributions) are extracted from GitHub PR/issue comments linked to real developer–coding-agent sessions (SWE-chat), with LLM-audited assistance windows specifying when each reference should be offered; for productivity, the existing PARE environment is reused, with its scripted expected assistance serving as references under windows [a, T]. Five baseline LLMs are evaluated on reference-matching recall/precision and in-window (timeliness) recall, and GPT-4.1-family models are fine-tuned on the training split, improving SWE-chat macro recall by up to 13.5 points.

Critical strengths

The SWE-chat half is anchored in real collaboration data — real developer–coding-agent conversations, with references derived from real PR/issue comments rather than synthesized targets — and the appendix self-auditing (second-judge checks, window-source comparisons, budget-matched analyses) documents ground-truth fragility more openly than is typical for benchmark papers.

Critical weaknesses
  1. Limited novelty. The framing follows prior proactive-agent work, and both components are external: SWE-chat provides the trajectories, while PARE's references and timing anchors are inherited wholesale ("PARE already specifies when the assistant should offer help").

  2. The ground truth is weakly validated. References, windows, and match judgments are produced and gated by LLMs; human validation covers 35 items, with prelabels visible and one annotator an author. Yet 183 of 306 test windows change between the paper's own two audits, and no ablation establishes agreement with humans — the evaluated objective is never crisply defined.

  3. Hard to read, uninformative, and interruption cost is unmeasured. The main text has no related-work section, contribution list, or limitations (24 of 28 tables sit in the appendix), and the single-run comparisons have overlapping bootstrap intervals (Table 23), so model orderings are not separated. Intervention frequency is never priced — its diagnostics are appendix-only and "without a preferred direction" — so the recall-winning fine-tuned model, intervening at 96.5% of decision points, is indistinguishable from a well-timed assistant.

Questions for authors
  1. What, precisely, is the delta of Ping relative to prior proactive-agent evaluation work[1-3] — in reference construction, timing annotation, environment, and user simulation? Please move this comparison into the main text.

  2. What is the agreement between the LLM-constructed ground truth (references, windows, match judgments) and blinded human judgment at a meaningful scale? The current validation covers 35 items with AI prelabels visible and an author annotator.

  3. Only 92 of the 306 SWE-chat test references come from accounts typed as users (Table 8). What is the account-type composition of the remaining 214, and were automated or AI-agent accounts screened during reference extraction?

  4. Why is intervention frequency treated only as a descriptive diagnostic? Would the reported conclusions — including the fine-tuning gains — survive an evaluation that prices over-intervention?

[1] Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance (Lu et al., ICLR 2025)

[2] Beyond Reactivity: Measuring Proactive Problem Solving in LLM Agents (Pasternak et al., arXiv:2510.19771)

[3] Proactive Agent Research Environment: Simulating Active Users to Evaluate Proactive Assistants (Nathani et al., arXiv:2604.00842)

3 comments
AuthorsOpenReview ↗

Rebuttal 1/2

Thank you for recognizing the use of real collaboration data and the transparency of our validation analyses. We address your concerns about the contribution, human validation, and intervention frequency below.

W.1: Limited novelty. The framing follows prior proactive-agent work, and both components are external: SWE-chat provides the trajectories, while PARE's references and timing anchors are inherited wholesale ("PARE already specifies when the assistant should offer help").

Q.1: What, precisely, is the delta of Ping relative to prior proactive-agent evaluation work[1-3] — in reference construction, timing annotation, environment, and user simulation? Please move this comparison into the main text.

Thank you for highlighting these three related works. To address your concern, we have moved the related work from the appendix to Section 6 of the revised main text and expanded the discussion of Proactive Agent, PROBE, and PARE. We have also added the comparison below (Table 5). We would also welcome any additional related work you think would strengthen this comparison.

FeatureProactive AgentPROBEPAREPing
Assistance without explicit user requests✓✓✓✓
Decisions as user activity progresses✓✗✓✓
References derived from real-world task feedback✗✗✗✓
Assistance categories identified from real discussions✗✗✗✓
Recall with and without assistance windows✗✗✗✓

Table 5. Comparison of proactive assistance benchmarks.

Regarding your concern about novelty, we would like to clarify what Ping contributes beyond the existing datasets and what it reuses.

Instead of asking agents to predict the user's next action and do it in advance, Ping evaluates whether they can offer useful help that users may not think to request. We construct our taxonomies from the proactive help people offer each other in real GitHub discussions and meetings, grouping contributions such as pointing out overlooked problems or sharing useful resources.

In SWE-chat, review feedback can reveal problems that the developer and coding agent missed. We use the GitHub taxonomy to construct references from this feedback and annotate assistance windows to evaluate whether agents offer that help while it is still useful. For PARE, we reuse its scenarios, references, environment, and user simulation, and use the meeting taxonomy to analyze assistance types. We determine the window's start from the expected assistant response's position in the event sequence and set its end to the trajectory's end. To address your question about timing annotation, we have clarified this construction in Section 3.3.

W.3: Hard to read, uninformative, and interruption cost is unmeasured. The main text has no related-work section, contribution list, or limitations (24 of 28 tables sit in the appendix), and the single-run comparisons have overlapping bootstrap intervals (Table 23), so model orderings are not separated. Intervention frequency is never priced — its diagnostics are appendix-only and "without a preferred direction" — so the recall-winning fine-tuned model, intervening at 96.5% of decision points, is indistinguishable from a well-timed assistant.

We agree that the main text should make the contribution and findings easier to understand. To address your concern, we have clarified the contribution in Section 1, moved and expanded related work in Section 6, and brought together the evaluation scope and limitations in Appendix A, referenced from Section 7. In Section 5.3, we highlight our finding that providing workspace and knowledge-pool information improves GPT-5.4's recall and precision on PARE without training (Appendix D.5).

Regarding your concern about result stability, we evaluated each model once under each input setting to keep the computational cost manageable. We agree that repeated runs would provide a stronger assessment of result stability. To address your concern, we have clarified the run count in Section 5.1 and discussed its scope in Appendix A. We therefore report observed scores without claiming stable model rankings. We address intervention frequency in Q.4 below.

AuthorsOpenReview ↗

Rebuttal 2/2

W.2: The ground truth is weakly validated.

Q.2: What is the agreement between the LLM-constructed ground truth (references, windows, match judgments) and blinded human judgment at a meaningful scale? The current validation covers 35 items with AI prelabels visible and an author annotator.

We agree that annotation quality needs to be checked directly. Two annotators each completed 100 judgments across SWE-chat and PARE. They assessed reference validity, intervention matching, and usefulness at selected decision points.

The annotators evaluated the same 50 items spanning 42 user trajectories. Each recorded their own yes, no, or unable-to-judge answers from the supplied context. The saved viewing records show that neither annotator opened the optional original-label or model-judgment panels.

Both annotators agreed with all sampled matching labels and judged all sampled in-window points useful, supporting the matching decisions and the usefulness of assistance within the sampled windows.

For the sampled references, both annotators confirmed supporting evidence for 93.8% and judged 81.3% to go beyond explicit requests. At least one annotator judged 93.8% useful. Overall agreement across the evaluation was 89.0%. Appendix E.3 reports the procedure and full results.

The 183 differences come from comparing preliminary transcript-only windows with windows that use both the transcript and supporting context to assess when help is useful. The reported timing scores use the latter, and the human evaluation above checks usefulness at sampled points (Appendix E.3).

We also compare the results under both window procedures with agent outputs and reference matches held fixed. The five models have the same ordering in macro in-window recall, while recall, precision, and match rate are unchanged (Appendix E.1). Section 3.3 defines relevance through reference matching and timeliness through matches within assistance windows.

Q.3: Only 92 of the 306 SWE-chat test references come from accounts typed as users (Table 8). What is the account-type composition of the remaining 214, and were automated or AI-agent accounts screened during reference extraction?

Thank you for asking us to clarify this. Our references are extracted from existing PR and issue discussions associated with the coding sessions. The source comments are written directly by users, written by users with AI assistance, or generated by automated code-review tools. We use a contribution as a reference only if it passes the reference-construction checks described in Section 3.2. Appendix B.4 describes the sources and reports an evaluation restricted to references from accounts typed as users.

Q.4: Why is intervention frequency treated only as a descriptive diagnostic? Would the reported conclusions — including the fine-tuning gains — survive an evaluation that prices over-intervention?

We agree with your concern that frequent interventions may be costly to users. We report intervention rate as a diagnostic because frequency alone does not tell us whether an intervention offers the expected assistance. Instead, precision measures the fraction of interventions that match a reference within each instance, averaged across instances. Adding non-matching interventions lowers precision unless it is already zero.

To test whether GPT-4.1-mini's SFT gains come only from making more interventions, we calculate the expected scores for 599 interventions selected uniformly at random without replacement from its 1,295 saved interventions. This approximately matches the base intervention count. If each intervention carries similar cost, the two conditions would also incur approximately the same total cost.

SettingRecallPrecisionIn-window recallIntervention rate
Base (reported)4.30%3.90%2.90%44.6%
SFT (all 1,295)17.82%17.15%12.81%96.5%
SFT (599 selected)11.76%12.76%6.16%44.6%

GPT-4.1-mini on 100 SWE-chat test instances. Recall, precision, and in-window recall are macro-averaged. The last row reports expected scores for 599 randomly selected SFT interventions; bold values exceed the base model's scores.

SFT still gives higher recall, precision, and in-window recall at approximately the same intervention count. The improvement therefore cannot be explained solely by producing more interventions. To address your concern, we have added this comparison to Section 5.2. This comparison controls intervention frequency, but does not measure users' actual interruption costs.

We hope these clarifications and revisions address your concerns. If so, we would appreciate it if you could reconsider your rating.

Thank you for your response. Two annotators assessing the same 50 items provide insufficient evidence to validate the timing windows. The requested source breakdown for the remaining 214 references is also missing. I will therefore maintain my rating.

Reviewer_cr5sOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Ping, a benchmark for evaluating proactive AI agents across software engineering and productivity scenarios. The benchmark assesses agents along two key dimensions: relevance and timeliness of proactive interventions. Through extensive experiments with multiple LLMs, the paper highlights the limitations of existing models in providing useful and timely proactive assistance.

Critical strengths
  1. Proactive agents are receiving increasing attention as AI systems evolve from reactive assistants toward more autonomous and context-aware agents. Developing a systematic benchmark to evaluate proactive capabilities is therefore both timely and important. This work addresses a meaningful gap in existing agent evaluation and could provide valuable guidance for future research in this direction.
  2. The paper introduces well-defined quantitative metrics to evaluate proactive assistance, particularly in terms of relevance and timeliness. These metrics provide a more systematic and objective basis for assessing proactive capabilities, rather than relying solely on qualitative observations or case studies. The distinction between identifying useful assistance and delivering it at the appropriate time is particularly valuable.
  3. The paper considers both software engineering and productivity scenarios, covering different types of user interactions and proactive assistance. Evaluating multiple models across these distinct domains strengthens the empirical evidence and provides a more comprehensive understanding of current limitations in proactive agent capabilities.
Critical weaknesses
  1. In Section 5.2, the authors report supervised fine-tuning results for the GPT-4.1 family, with particularly notable improvements for GPT-4.1-mini. However, since GPT-4.1 is a closed-source model, the implementation details of the fine-tuning procedure are unclear. Was the fine-tuning conducted through OpenAI's official fine-tuning API? The authors should provide a more detailed description of the training procedure, including the fine-tuning interface, training configuration, and hyperparameters, to improve the clarity and reproducibility of the experiments. Additionally, it is unclear why the authors chose to fine-tune only the GPT-4.1 family rather than including open-weight models such as Qwen or Gemma, which are already considered in the baseline experiments. Fine-tuning open-weight models would enable more transparent and reproducible experiments and help assess whether the observed improvements generalize beyond a specific proprietary model family. A clearer justification for the model selection, or additional fine-tuning experiments on open-weight models, would strengthen this section.
  2. The proactive capabilities of an agent depend not only on the underlying LLM but also on the design of the agent harness, including context management, information retrieval, and intervention mechanisms. Although the paper investigates the effects of different context configurations, the influence of the broader harness design remains insufficiently explored. It would be valuable to conduct a more systematic analysis or ablation study of the harness components to better distinguish the contributions of the underlying LLM from those of the surrounding agent system. This could also provide more actionable insights into how proactive agent performance can be improved.
Questions for authors

See the weakness part.

1 comment

Author Response:

Thank you for your encouraging feedback! We appreciate your recognition of the importance of evaluating proactive assistance, the relevance and timeliness metrics, and the evaluation across software engineering and productivity. We also thank you for your suggestions on the fine-tuning procedure and agent harness, and respond to each point below.

W.1: In Section 5.2, the authors report supervised fine-tuning results for the GPT-4.1 family, with particularly notable improvements for GPT-4.1-mini. However, since GPT-4.1 is a closed-source model, the implementation details of the fine-tuning procedure are unclear. Was the fine-tuning conducted through OpenAI's official fine-tuning API? The authors should provide a more detailed description of the training procedure, including the fine-tuning interface, training configuration, and hyperparameters, to improve the clarity and reproducibility of the experiments. Additionally, it is unclear why the authors chose to fine-tune only the GPT-4.1 family rather than including open-weight models such as Qwen or Gemma, which are already considered in the baseline experiments. Fine-tuning open-weight models would enable more transparent and reproducible experiments and help assess whether the observed improvements generalize beyond a specific proprietary model family. A clearer justification for the model selection, or additional fine-tuning experiments on open-weight models, would strengthen this section.

Thank you for pointing out that the training procedure needs more detail. We performed supervised fine-tuning (SFT) on the SWE-chat training split and evaluated the models on the held-out test split. All three models were trained for 3 epochs with a batch size of 32 and a learning-rate multiplier of 0.1. We omit the platform name to preserve anonymity during review and will provide it in the final public version.

Given our available training resources, we focused on the GPT-4.1 family and compared all three models using the same training configuration. We agree that including Qwen or Gemma would help assess whether the gains extend to other model families. Section 5.2 explains the training-data construction, model selection, and data split. The hyperparameters are given above, and we will release the data construction and training configuration with the code.

W.2: The proactive capabilities of an agent depend not only on the underlying LLM but also on the design of the agent harness, including context management, information retrieval, and intervention mechanisms. Although the paper investigates the effects of different context configurations, the influence of the broader harness design remains insufficiently explored. It would be valuable to conduct a more systematic analysis or ablation study of the harness components to better distinguish the contributions of the underlying LLM from those of the surrounding agent system. This could also provide more actionable insights into how proactive agent performance can be improved.

We agree that proactive assistance depends on the agent harness as well as the underlying model. Thus, we want to highlight our finding that providing workspace and knowledge-pool information improves GPT-5.4's recall and precision on PARE without training. Adding these sources to live context increases recall from 2.1% to 7.7% and precision from 1.7% to 6.2% (Appendix D.5). Additionally, full event details increase recall for all four tested models on PARE, for example by adding email bodies to event summaries (Appendix D.4). These findings suggest that low scores partly reflect missing information in the input. To address your concern, we have highlighted these findings in Section 5.3 of the main text.

To address your question about the harness, we compare different context inputs using the same model and PARE user trajectories. With full event details, all four models also have a higher expected number of matched references at equal intervention counts (Appendix D.4). Section 4 and Appendices B.5–B.6 describe context preparation, retrieval, and the information available at each decision point. We agree that broader harness ablations would be valuable and will consider further comparisons of retrieval strategies and intervention mechanisms in future work.

Comments on the paper

1 comment

To all reviewers and the AC:

Thank you all for your valuable time and feedback. We appreciate your recognition of the importance of proactive assistance, the use of collaboration data, and the evaluation of both relevance and timeliness.

We have revised our submission in response to your comments.

  • We have added human evaluation with 100 judgments per annotator on reference validity, intervention matching, and usefulness across SWE-chat and PARE. Both annotators agreed with all sampled matching labels and judged all sampled in-window points useful (Appendix E.3).
  • We have clarified Ping's contribution in Section 1 and moved related work to Section 6 of the main text, with a new comparison table (Table 5).
  • We report a separate experiment evaluating whether agents can execute actions to complete tasks in PARE (Appendix C), complementing Ping's evaluation of what help to offer and when.
  • We have highlighted how workspace and knowledge-pool information improves GPT-5.4's recall and precision on PARE without training, and how full event details increase recall for all four tested models (Section 5.3; Appendices D.4–D.5).
  • We have added a comparison showing that GPT-4.1-mini's SFT gains persist at approximately the base model's intervention count, with higher expected recall, precision, and in-window recall (Section 5.2).

As the discussion period closes today, we thank you again for your time and feedback. If our responses and revisions have addressed your concerns, we would appreciate it if you could reconsider your ratings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.