- Rating
- 3
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a unified framework for proactive LLM agents based on three objectives (3T): Task Capability (TC), Temporal Allocation (TA), and Trust (TR). It introduces a five-dimensional design space and Proactivity-Gym, a simulation-based benchmark featuring multi-day scenarios, stateful environments, and persona-conditioned users. Experiments across 23 model–harness configurations reveal substantial performance gaps in temporal allocation and intervention alignment. A complementary human study with 30 participants further highlights the importance of appropriate intervention timing and user trust.
Critical strengths
-
The paper addresses an underexplored aspect of LLM agents by considering not only task completion but also computational resource allocation and user trust in proactive assistance.
-
The proposed 3T framework provides a clear and structured perspective on proactive agent design, connecting task capability, temporal allocation, and trust with concrete design dimensions.
Critical weaknesses
- The paper motivates joint consideration of TC, TA, and TR but lacks controlled ablations demonstrating its benefits. The experiments primarily reveal performance gaps across metrics rather than validate joint optimization.
2.TR-D measures intervention-depth agreement rather than actual user trust. Additionally, evaluating only activated tasks may introduce selection bias, since different agents can trigger different downstream task branches.
- The human study relies on 30 participants evaluating hypothetical interaction scenarios. Its findings on trust dynamics may not fully generalize to real-world, long-term agent interactions.
Questions for authors
-
How do the authors ensure fair comparisons when different agents activate different task branches? Could end-to-end scenario success rates be reported?
-
How well do TR-D and LLM-judged trust correlate with actual human trust judgments on the same agent trajectories?
4 comments
W2-2, Q1 (continued)
| Model | Reach@3 (%) | Avg. Reach@1 (%) |
|---|---|---|
| Qwen 3.5 2B | 85.59 | 77.48 |
| Qwen 3.5 9B | 76.58 | 75.97 |
| Qwen 3.5 27B | 81.08 | 77.28 |
| Gemma 4 12B | 77.48 | 75.97 |
| Gemma 4 31B | 77.48 | 76.38 |
| GPT 5.6 Luna | 88.29 | 77.98 |
| GPT 5.6 Sol | 89.19 | 81.18 |
| Claude Sonnet 5 | 83.78 | 80.68 |
| Claude Opus 5 | 95.50 | 85.59 |
Moreover, the rankings between models and harnesses remain mostly constant when we test for bias directly by computing the performance of each agent (model-harness configuration) on the 84 commonly reached tasks only (Table 15; averaged across harnesses in the table below). Compared with the performance that we have reported previously in the paper (Table 7), Kendall’s τ is 1, 1, 0.992, and 0.937 for TC, TA, TR-J, and TR-D, respectively. Beyond rankings, the absolute scores stay close to those reported before, with a maximum difference of 1.91 points across all cells.
| Model | TC (0–100) | TA (%) | TR-D (%) | TR-J (1–5) | E2E TC (0–100) |
|---|---|---|---|---|---|
| Qwen 3.5 2B | 20.79 | 0.19 | 41.96 | 1.84 | 14.88 |
| Qwen 3.5 9B | 39.95 | 2.41 | 45.41 | 3.10 | 28.47 |
| Qwen 3.5 27B | 44.11 | 6.30 | 47.08 | 3.51 | 32.01 |
| Gemma 4 12B | 22.40 | 0.93 | 38.88 | 2.87 | 16.01 |
| Gemma 4 31B | 44.51 | 3.70 | 45.73 | 3.24 | 32.04 |
| GPT 5.6 Luna | 46.11 | 10.84 | 46.96 | 3.61 | 33.27 |
| GPT 5.6 Sol | 54.65 | 17.78 | 52.17 | 3.96 | 40.58 |
| Claude Sonnet 5 | 55.80 | 16.67 | 50.79 | 3.85 | 41.93 |
| Claude Opus 5 | 65.36 | 51.67 | 53.35 | 4.24 | 52.59 |
Finally, regarding end-to-end scenario success rates, we report two results: the branch coverage above (Reach@3 and Avg. Reach@1) and an end-to-end version of TC with a fixed denominator (E2E TC), where all 111 tasks are counted for every agent and a task on an unreached branch receives a score of 0. The scores of all agents decrease (by 4.0 - 14.2 %p) as some branches remain unreached, but the ranking of the 23 configurations is identical (Kendall’s τ=1) to that of Table 7, which was originally reported. These results show that our findings remain consistent despite different branch activations.
W3. Generalization of scenario-based human study
We agree with the reviewer’s point, where the measurement of trust in the real-world requires an additional longitudinal deployment study, which we originally note as future work (Sec. 6, App. H). The table below shows which combinations of sample and judge we evaluated, where the three evaluated cells reveal that LLM judges are relatively insensitive to intervention-depth misalignment. For our human study, participants rate cumulative week-long logs adapted from Proactivity-Gym scenarios, where the trust patterns are robust and consistent across different scenarios and analyses (App. G.5). Hence, we expect the direction of our findings to carry over, while their magnitude in long-term use remains to be measured.
| Sample | Human | LLM judge |
|---|---|---|
| Full Proactivity-Gym trajectories | - | ✓ |
| Human-study logs | ✓ | ✓ |
Q2. Alignment of LLM judges with human judgments on same agent trajectories (Sec. 5.3, App. E.7)
The human–LLM discrepancy persists on matched logs. When an LLM judge rates the same trust logs as our human participants, with TC held constant, it shows relatively less sensitive behavior toward intervention-depth misalignment compared to humans.
In detail, we provide an LLM judge (Qwen3.8-27B) the same trust logs from our human study questionnaire (Sec. 5.3), which also is a subsample of the scenario in Proactivity-Gym along with an agent’s trajectory. We gave all 30 questionnaires that were used in the study.
The LLM judge tracked the direction of trust change, where LLM judges increased trust after intervention-aligned (A) actions and decreased it after misaligned (M) actions (human vs. LLM judge; A→M: −1.86 vs. −0.89; M→A: +1.27 vs. +0.91). However, the magnitude of trust loss and the ratio of trust loss to trust recovery were smaller than those of humans (1.46 vs 0.98). This behavior was also evident at the sequence level. In the A->M->A sequence, humans showed a 22.57% decrease in trust at the final aligned action relative to the first one, while the LLM judge only showed a 1.25% decrease. In the M->A->M sequence, human’s final trust after the misaligned action, despite the intermediate alignment, decreased by 26.20% compared to the initial trust, while LLM’s final trust rather increased by 2.10%. Consistent to our original findings, these results indicate the judge is lenient toward intervention-depth misalignment and less sensitive to it than humans.
Flag for Ethics Review
Lastly, we notice that the ethics review flag was raised. Could you please let us know if there is a specific concern we can address, or if everything is satisfactory, kindly consider removing the flag?
We once again thank you for the thoughtful assessment and suggestions.
Thanks for your clarification. I keep my rating at 3 and remove the ethics review flag.
Thank you for the follow-up. We once again appreciate your time and feedback.
We appreciate the reviewer’s thoughtful comments and suggestions, which help strengthen the paper’s claims. We also appreciate the reviewer for acknowledging the novelty and strength of the paper of uncovering a relatively underexplored and overlooked aspect of current proactive agents of temporal allocation and trust with concrete design formulations.
Below, we clarify our claim regarding the joint optimization of 3T which is supported by our human study, clarify the role of TR-D, show that our comparisons remain constant under different branch activations, discuss the scope of our human study, and compare LLM-judged trust with human trust judgments on the same logs. All additional analyses or revisions are added to the revised draft (Sec. 5.2-5.3, App. E.7, and F.7).
Our claim is that TC, TA, and TR must be considered together and our human study provides direct evidence for it. The study also serves as a controlled ablation, which the reviewer requests, where we vary one objective while holding the others fixed and show that users value TA and TR beyond TC.
We summarize the relevant results from our human study (Sec. 5.3 / App. G). As detailed in the original draft, the results support the importance of both TA and TR. For TA, participants accepted correct assistance that competed with their ongoing work in only 26.7% of cases (TC ↑ , TA↓ ), but accepted sleep-time assistance in 97.8% of cases even when it required correction (TC ↓ , TA ↑). For TR, with content quality held constant, participants preferred agents whose intervention depth matched their expectations in 88.3% of comparison. Also, a single misaligned intervention lowered trust by 1.86 points, while even after the agent behaved appropriately again, trust did not return to its initial level (4.43 → 3.43). For TC, participants preferred correct outputs in 92.2% of the cases. These findings suggest all three 3T components are perceived to be important and beneficial to human subjects, which should be considered for proactive agent design.
Existing benchmarks do not evaluate these three objectives, but rather focus mostly on TC. Proactivity-Gym, as a proof-of-concept, serves as an initial testbed with dynamic and action-dependent features along time horizons to properly assess proactivity (Sec. 5), where we show that contemporary agents fall short on TA and TR. Joint optimization is the next step that this foundation and testbed enable, rather than a claim that we developed such an agent (Sec. 6, App. H).
We clarify that TR-D is not intended to measure user trust directly. Rather, it is one of two complementary measures (TR-D and TR-J) we use to approximate it. Trust is user-specific and is not directly observable to the agent (Sec. 3.1). Hence, human trust cannot be measured directly in simulation environments, especially at scale. To overcome this, consistent with recent research trends, we use LLM judges to rate the five constructs of trust, defined in Sec. 3.1 and App. B (TR-J). To complement this judge-based estimate with an observable signal, we also report TR-D, a behavioral proxy (Isbell & Pierce, 2005; Kraus et al., 2021; 2023b) that measures whether an agent’s intervention depth matches the depth the user expects (i.e., how far the user allows the agent to proceed autonomously). Our human study further supports TR-D as a proxy, where participants closely tied their trust to whether the agent’s intervention depth matched their preferences.
We confirm that differences in branch activation do not change our comparisons. Our scenarios are designed so that a large majority of tasks are given to every agent in every run, and model rankings remain nearly identical on commonly reached tasks and identical when unreached branches are counted as failures. Such branching resembles real-world agent usage, where an agent’s earlier action affects future state. Hence, depending on its earlier actions, an agent might fail to reach a certain branch, where the evaluation does not include that unreached branch. Here, we explore whether our evaluation remains robust to these different branch coverage.
To ensure fair comparisons as much as we could, first, scenarios are designed such that a sufficient number of tasks are reached, where 84 of the maximum of 111 reachable tasks are reached by every agent in every run. As shown in the table below, every agent (harness: OpenClaw) reaches at least 76.58% of the 111 tasks across three runs (Reach@3) and at least 75.97% in a single run on average (Avg. Reach@1). Second, each configuration is run three times.