- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper tackles a critical real-world problem: supervisors (i.e., users) can make mistakes during human-in-the-loop interaction. Evaluating agents in such settings is well-motivated. The experimental results shows that frontier LLMs suffer from this scenario.
Critical strengths
- This paper tackles a critical real-world problem: supervisors (i.e., users) can make mistakes during human-in-the-loop interaction. Evaluating agents in such settings is well-motivated.
- The paper is well-structured.
Critical weaknesses
- One limitation of offline benchmarks is that it is difficult to evaluate agents in environments that capture real-world complexity. This is why current evaluation frameworks are mostly based on end-to-end simulation environments. To make a clear contribution to the agent research field as an offline benchmark, this paper should conduct more thorough experiments and show significant results. Currently, however, the experiments mainly focus on the low scores of frontier LLMs. I suggest that the authors conduct more meaningful experiments (e.g., how serious a problem this could be in real-world deployment, and whether it can be mitigated by means other than SFT) and, if possible, move the table from the appendix to Section 5.
- The setting of this paper treats the agent as an observer, not an actor that actually carries out the tasks delegated by the user. This lacks realism in the benchmark scenario, because real-world agents do not only monitor; they should also perform actions in real time.
- If my understanding is correct, the agent only outputs responses in the (content, intervention) format. I think this limits analysis of the experimental results from diverse perspectives.
- It is unclear how this paper builds the live context, workspace, and knowledge pool. These should be described in more detail.
- It is surprising that even frontier LLMs such as GPT-5.6-sol and GPT-5.4 show low performance on this dataset, because the example scenarios in Figures 1-3 seem easy enough to recognize that the user’s instruction is somewhat wrong. One major concern is that this benchmark may contain many unsolvable or ambiguous cases. To address this concern, the authors should clearly describe the data synthesis process and show that it reliably generates feasible scenarios.
Questions for authors
- Please add the agent prompt to the appendix.
- Please add example cases from the generated dataset, and clearly state what the input to the agent consists of.
- Scores should be reported in percentages (e.g., 0.054 → 5.4%).
4 comments
Rebuttal 2/2
W.5: It is surprising that even frontier LLMs such as GPT-5.6-sol and GPT-5.4 show low performance on this dataset, because the example scenarios in Figures 1-3 seem easy enough to recognize that the user’s instruction is somewhat wrong. One major concern is that this benchmark may contain many unsolvable or ambiguous cases. To address this concern, the authors should clearly describe the data synthesis process and show that it reliably generates feasible scenarios.
We agree that we should explain how references are constructed and why the assistance is feasible. SWE-chat references come from existing PR and issue comments, which can identify problems the developer and coding agent missed. The proactive agent must identify useful assistance from the context available at the decision point and offer it while it can still help the user.
During construction, an LLM checks whether each candidate reference is supported by the available context, addresses the user's task, and describes assistance the user could act on while it is still useful. Only candidates that pass these checks become references (Section 3.2).
Figure 2 in Section 3.2 illustrates how a PR comment becomes a reference and how assistance windows are used in evaluation.
We agree that annotation quality needs to be checked directly. Two annotators each completed 100 judgments across SWE-chat and PARE. They assessed reference validity, intervention matching, and usefulness at selected decision points.
Both annotators agreed with all sampled matching labels and judged all sampled in-window points useful, supporting the matching decisions and the usefulness of assistance within the sampled windows.
For the sampled references, both annotators confirmed supporting evidence for 93.8% and judged 81.3% to go beyond explicit requests. At least one annotator judged 93.8% useful. Overall agreement across the evaluation was 89.0%. Appendix E.3 reports the procedure and full results.
Q.1: Please add the agent prompt to the appendix.
Thank you for this suggestion. The SWE-chat decision prompt asks the agent to offer assistance only when it is useful at the current turn and supported by the supplied context. The PARE prompt asks whether there is a grounded, actionable opportunity that the user has not already handled, and requests the proposed application action and its inputs. Appendix B.5 describes the retrieval and decision procedure. We will include the full decision and retrieval prompts with the code release.
Q.2: Please add example cases from the generated dataset, and clearly state what the input to the agent consists of.
Thank you for this suggestion. Section 4 and Figure 3 illustrate the three context sources and the resulting intervention using the API-field rename example. Section 5.4 examines one SWE-chat case and one PARE case, and Appendix D.4 shows how email details help agents identify eligible expenses in a PARE reimbursement task. Appendices B.5–B.6 describe the agent inputs and observation rules. References and assistance windows are used only for evaluation and are not given to the agent.
Q.3: Scores should be reported in percentages (e.g., 0.054 → 5.4%).
Thank you for pointing this out. We now report rates as percentages in Section 5 and the appendix result tables, using consistent metric names.
We hope these clarifications and revisions address your concerns. If so, we would appreciate it if you could reconsider your rating.
Thank you for your reply. I will keep my scores.
Thank you again for your time and for considering our responses. If possible, we would appreciate any further feedback on the concerns that remain, particularly regarding the additional experiments and human evaluation. Your feedback would help us better understand your assessment and improve the paper. We would be happy to provide further clarification if helpful.
Rebuttal 1/2
Thank you for recognizing the importance of evaluating proactive assistance and the structure of our paper. We respond to your concerns below.
We agree with your concern that low scores alone do not tell us how to improve proactive assistance. Thus, we want to highlight our finding that providing workspace and knowledge-pool information improves GPT-5.4's recall and precision on PARE without training. Adding these sources to live context increases recall from 2.1% to 7.7% and precision from 1.7% to 6.2% (Appendix D.5). Additionally, full event details increase recall for all four tested models on PARE, for example by adding email bodies to event summaries (Appendix D.4). These findings suggest that low scores partly reflect missing information in the input. To address your concern, we have highlighted these findings in Section 5.3 of the main text.
We agree that executing actions is important for a proactive agent. In a separate experiment with PARE, we evaluate whether agents can execute actions to complete tasks. GPT-5.4 completes 17.5% of tasks according to PARE's validators. Appendix C describes this experiment, which uses a simulated user to accept or reject proposed actions before execution.
Monitoring the user's activity is a first step toward proactive assistance. The agent needs to use what it observes to decide what help to offer and when, without waiting for a user request. Our main evaluation studies this ability by having agents observe the same user trajectories, so we can compare the relevance and timeliness of their interventions.
Thank you for raising this point. The output records whether the agent intervenes, what it says, and when, allowing us to analyze both relevance and timeliness. We measure recall, precision, match rate, in-window recall, and intervention rate, and compare results across types of assistance. In PARE, an intervention also specifies an application, an action, and the inputs needed to carry it out (Section 4).
To address your concern, we have expanded the discussion in Section 5.3 to examine whether higher scores simply come from more frequent interventions. We compare equal numbers of saved interventions from each run, with the full analyses in Appendices D.3–D.4.
Thank you for pointing out that the inputs need a clearer explanation. Section 4 and Figure 3 explain the three context sources, and Appendices B.5–B.6 describe their preparation and use in each domain. The three sources serve the same roles across domains, while their implementation follows the available data.
Live context contains the user trajectory up to the current decision point. In SWE-chat, this includes developer messages and the coding agent's responses and actions. In PARE, it includes user actions and environment events.
Workspace describes the initial working environment. For SWE-chat, we provide an overview of the repository and its files. For PARE, we provide the initial application state, such as inbox contents and calendar entries.
The knowledge pool contains additional pull requests (PRs) and issues in SWE-chat, or records of past application activity in PARE. We exclude the SWE-chat discussions used to construct the instance's references.