acceptodds
Under review as a conference paper at ICLR 2027

Foundations of Proactive Agents: Principles, Technical Layers, and Simulation Gym

Abstract

Proactive LLM agents can turn idle compute into useful support before users ask. Yet even correct work can misread user context, impose review costs, or undermine trust. This work proposes foundations for designing, realizing, and evaluating proactive LLM agents around three joint principles (3T): *Task Capability*, anticipating relevant needs and correctly performing useful work; *Temporal Allocation*, allocating compute according to resource availability and when results are needed; and *Trust*, sustaining users’ confidence and appropriate reliance on the agent. We connect these objectives to a design space organized around five dimensions: task scope, anticipation horizon, activation trigger, processing timing, and intervention depth, and specify the situation and system modeling needed to support its choices, including user and environment representations, backbone LLMs, and agent harnesses. Lastly, we propose PROACTIVITY-GYM a simulation-based evaluation testbed including multi-day scenarios, stateful environments, and persona-conditioned simulated users that can evaluate the consequences of proactive assistance across interactions. Evaluations across 23 model-harness configurations uncover substantial performance gaps across 3T and reveal that LLM judges often conflate task capability and trust. A human study with 30 participants demonstrates the importance of the joint 3T optimization: participants show sharp trust declines after intervention misalignment despite correct outcomes, and prefer sleep-time assistance, even when imperfect, to preserve ongoing focus. Together, these findings support designing and evaluating proactive agents through the joint consideration of useful work, compute allocation, and evolving user trust.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Proactivity, LLM Agent, Agentic AI, Simulation, Agent Evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_GqLAOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a unified framework for proactive LLM agents based on three objectives (3T): Task Capability (TC), Temporal Allocation (TA), and Trust (TR). It introduces a five-dimensional design space and Proactivity-Gym, a simulation-based benchmark featuring multi-day scenarios, stateful environments, and persona-conditioned users. Experiments across 23 model–harness configurations reveal substantial performance gaps in temporal allocation and intervention alignment. A complementary human study with 30 participants further highlights the importance of appropriate intervention timing and user trust.

Critical strengths
  1. The paper addresses an underexplored aspect of LLM agents by considering not only task completion but also computational resource allocation and user trust in proactive assistance.

  2. The proposed 3T framework provides a clear and structured perspective on proactive agent design, connecting task capability, temporal allocation, and trust with concrete design dimensions.

Critical weaknesses
  1. The paper motivates joint consideration of TC, TA, and TR but lacks controlled ablations demonstrating its benefits. The experiments primarily reveal performance gaps across metrics rather than validate joint optimization.

2.TR-D measures intervention-depth agreement rather than actual user trust. Additionally, evaluating only activated tasks may introduce selection bias, since different agents can trigger different downstream task branches.

  1. The human study relies on 30 participants evaluating hypothetical interaction scenarios. Its findings on trust dynamics may not fully generalize to real-world, long-term agent interactions.
Questions for authors
  1. How do the authors ensure fair comparisons when different agents activate different task branches? Could end-to-end scenario success rates be reported?

  2. How well do TR-D and LLM-judged trust correlate with actual human trust judgments on the same agent trajectories?

4 comments

We appreciate the reviewer’s thoughtful comments and suggestions, which help strengthen the paper’s claims. We also appreciate the reviewer for acknowledging the novelty and strength of the paper of uncovering a relatively underexplored and overlooked aspect of current proactive agents of temporal allocation and trust with concrete design formulations.

Below, we clarify our claim regarding the joint optimization of 3T which is supported by our human study, clarify the role of TR-D, show that our comparisons remain constant under different branch activations, discuss the scope of our human study, and compare LLM-judged trust with human trust judgments on the same logs. All additional analyses or revisions are added to the revised draft (Sec. 5.2-5.3, App. E.7, and F.7).

W1 Lack of ablations regarding different benefits of TA, TC, TR and joint optimization validation

Our claim is that TC, TA, and TR must be considered together and our human study provides direct evidence for it. The study also serves as a controlled ablation, which the reviewer requests, where we vary one objective while holding the others fixed and show that users value TA and TR beyond TC.

We summarize the relevant results from our human study (Sec. 5.3 / App. G). As detailed in the original draft, the results support the importance of both TA and TR. For TA, participants accepted correct assistance that competed with their ongoing work in only 26.7% of cases (TC ↑ , TA↓ ), but accepted sleep-time assistance in 97.8% of cases even when it required correction (TC ↓ , TA ↑). For TR, with content quality held constant, participants preferred agents whose intervention depth matched their expectations in 88.3% of comparison. Also, a single misaligned intervention lowered trust by 1.86 points, while even after the agent behaved appropriately again, trust did not return to its initial level (4.43 → 3.43). For TC, participants preferred correct outputs in 92.2% of the cases. These findings suggest all three 3T components are perceived to be important and beneficial to human subjects, which should be considered for proactive agent design.

Existing benchmarks do not evaluate these three objectives, but rather focus mostly on TC. Proactivity-Gym, as a proof-of-concept, serves as an initial testbed with dynamic and action-dependent features along time horizons to properly assess proactivity (Sec. 5), where we show that contemporary agents fall short on TA and TR. Joint optimization is the next step that this foundation and testbed enable, rather than a claim that we developed such an agent (Sec. 6, App. H).

W2-1 TR-D measures intervention-depth agreement rather than actual user trust

We clarify that TR-D is not intended to measure user trust directly. Rather, it is one of two complementary measures (TR-D and TR-J) we use to approximate it. Trust is user-specific and is not directly observable to the agent (Sec. 3.1). Hence, human trust cannot be measured directly in simulation environments, especially at scale. To overcome this, consistent with recent research trends, we use LLM judges to rate the five constructs of trust, defined in Sec. 3.1 and App. B (TR-J). To complement this judge-based estimate with an observable signal, we also report TR-D, a behavioral proxy (Isbell & Pierce, 2005; Kraus et al., 2021; 2023b) that measures whether an agent’s intervention depth matches the depth the user expects (i.e., how far the user allows the agent to proceed autonomously). Our human study further supports TR-D as a proxy, where participants closely tied their trust to whether the agent’s intervention depth matched their preferences.

W2-2, Q1 Fair comparison under different branch activations. (Sec. 5.2, App. F.7)

We confirm that differences in branch activation do not change our comparisons. Our scenarios are designed so that a large majority of tasks are given to every agent in every run, and model rankings remain nearly identical on commonly reached tasks and identical when unreached branches are counted as failures. Such branching resembles real-world agent usage, where an agent’s earlier action affects future state. Hence, depending on its earlier actions, an agent might fail to reach a certain branch, where the evaluation does not include that unreached branch. Here, we explore whether our evaluation remains robust to these different branch coverage.

To ensure fair comparisons as much as we could, first, scenarios are designed such that a sufficient number of tasks are reached, where 84 of the maximum of 111 reachable tasks are reached by every agent in every run. As shown in the table below, every agent (harness: OpenClaw) reaches at least 76.58% of the 111 tasks across three runs (Reach@3) and at least 75.97% in a single run on average (Avg. Reach@1). Second, each configuration is run three times.

W2-2, Q1 (continued)

ModelReach@3 (%)Avg. Reach@1 (%)
Qwen 3.5 2B85.5977.48
Qwen 3.5 9B76.5875.97
Qwen 3.5 27B81.0877.28
Gemma 4 12B77.4875.97
Gemma 4 31B77.4876.38
GPT 5.6 Luna88.2977.98
GPT 5.6 Sol89.1981.18
Claude Sonnet 583.7880.68
Claude Opus 595.5085.59

Moreover, the rankings between models and harnesses remain mostly constant when we test for bias directly by computing the performance of each agent (model-harness configuration) on the 84 commonly reached tasks only (Table 15; averaged across harnesses in the table below). Compared with the performance that we have reported previously in the paper (Table 7), Kendall’s τ is 1, 1, 0.992, and 0.937 for TC, TA, TR-J, and TR-D, respectively. Beyond rankings, the absolute scores stay close to those reported before, with a maximum difference of 1.91 points across all cells.

ModelTC (0–100)TA (%)TR-D (%)TR-J (1–5)E2E TC (0–100)
Qwen 3.5 2B20.790.1941.961.8414.88
Qwen 3.5 9B39.952.4145.413.1028.47
Qwen 3.5 27B44.116.3047.083.5132.01
Gemma 4 12B22.400.9338.882.8716.01
Gemma 4 31B44.513.7045.733.2432.04
GPT 5.6 Luna46.1110.8446.963.6133.27
GPT 5.6 Sol54.6517.7852.173.9640.58
Claude Sonnet 555.8016.6750.793.8541.93
Claude Opus 565.3651.6753.354.2452.59

Finally, regarding end-to-end scenario success rates, we report two results: the branch coverage above (Reach@3 and Avg. Reach@1) and an end-to-end version of TC with a fixed denominator (E2E TC), where all 111 tasks are counted for every agent and a task on an unreached branch receives a score of 0. The scores of all agents decrease (by 4.0 - 14.2 %p) as some branches remain unreached, but the ranking of the 23 configurations is identical (Kendall’s τ=1) to that of Table 7, which was originally reported. These results show that our findings remain consistent despite different branch activations.

W3. Generalization of scenario-based human study

We agree with the reviewer’s point, where the measurement of trust in the real-world requires an additional longitudinal deployment study, which we originally note as future work (Sec. 6, App. H). The table below shows which combinations of sample and judge we evaluated, where the three evaluated cells reveal that LLM judges are relatively insensitive to intervention-depth misalignment. For our human study, participants rate cumulative week-long logs adapted from Proactivity-Gym scenarios, where the trust patterns are robust and consistent across different scenarios and analyses (App. G.5). Hence, we expect the direction of our findings to carry over, while their magnitude in long-term use remains to be measured.

SampleHumanLLM judge
Full Proactivity-Gym trajectories-✓
Human-study logs✓✓

Q2. Alignment of LLM judges with human judgments on same agent trajectories (Sec. 5.3, App. E.7)

The human–LLM discrepancy persists on matched logs. When an LLM judge rates the same trust logs as our human participants, with TC held constant, it shows relatively less sensitive behavior toward intervention-depth misalignment compared to humans.

In detail, we provide an LLM judge (Qwen3.8-27B) the same trust logs from our human study questionnaire (Sec. 5.3), which also is a subsample of the scenario in Proactivity-Gym along with an agent’s trajectory. We gave all 30 questionnaires that were used in the study.

The LLM judge tracked the direction of trust change, where LLM judges increased trust after intervention-aligned (A) actions and decreased it after misaligned (M) actions (human vs. LLM judge; A→M: −1.86 vs. −0.89; M→A: +1.27 vs. +0.91). However, the magnitude of trust loss and the ratio of trust loss to trust recovery were smaller than those of humans (1.46 vs 0.98). This behavior was also evident at the sequence level. In the A->M->A sequence, humans showed a 22.57% decrease in trust at the final aligned action relative to the first one, while the LLM judge only showed a 1.25% decrease. In the M->A->M sequence, human’s final trust after the misaligned action, despite the intermediate alignment, decreased by 26.20% compared to the initial trust, while LLM’s final trust rather increased by 2.10%. Consistent to our original findings, these results indicate the judge is lenient toward intervention-depth misalignment and less sensitive to it than humans.

Flag for Ethics Review

Lastly, we notice that the ethics review flag was raised. Could you please let us know if there is a specific concern we can address, or if everything is satisfactory, kindly consider removing the flag?

We once again thank you for the thoughtful assessment and suggestions.

Thanks for your clarification. I keep my rating at 3 and remove the ethics review flag.

Thank you for the follow-up. We once again appreciate your time and feedback.

Reviewer_virCOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper organizes proactive LLM agents around task capability, temporal allocation, and trust, and introduces Proactivity-Gym with ten multi-day scenarios and three user personas. It evaluates some model–harness configs and conducts a study with 30 human. The results motivate considering task quality, timing, and intervention preferences together. I find the conceptual synthesis useful, but the claims about joint optimization, compute allocation, and human–ai or human-LLM differences in trust assessment exceed what the current evaluations establish.

Critical strengths
  1. The design distinctions are useful. I find the separation of anticipation horizon, processing timing, and intervention depth in the paper helpful.
  2. The testbed evaluates behavior across interactions. The paper includes persistent environments, scheduled events, conditional follow-up tasks, unavailable users, and cases where no intervention is appropriate. Earlier actions can therefore affect later opportunities. The wide configurations choice and repeated runs provide an initial characterization of the problem.
Critical weaknesses
  1. The claim that LLM judges conflate capability and trust is not established by a matched comparison. The paper show that judged trust (TR-J) correlates more strongly with task complete than with intervention-depth agreement (TR-D). However, Section 3.1 includes technical competence among the trust constructs, so some association with TC is expected. TR-D also treats unnecessary approval requests and unauthorized execution as mismatches, although their consequences can differ. More importantly, the human study uses constructed comparison scenarios, whereas the judges assess evaluated agent trajectories. I am not convinced that these results support the abstract's claim of conflation or the introduction's claimed human–LLM discrepancy. People and judges would need to assess the same logs with matched questions, separating under-intervention from over-intervention. Without that validation, the supported finding is a difference between the reported measures, with its cause unresolved.
  2. TA measures prioritization and deferral more directly than compute allocation. As described, TA as allocating available compute to useful work while meeting future needs. sec 5 operationalizes it through urgent-task prioritization and explicit deferral. The detailed scoring confirms that a proposal can earn immediate-attention credit without completion, and deferral can earn credit without later delivery. The paper acknowledges this as a first step, and it is a useful diagnostic. It does not establish how much compute is misallocated, whether idle resources are used productively, or how often deferred work meets its deadline. Linking the decisions to these outcomes would support the broader framing; otherwise, the conclusions should consistently retain the narrower scope.
  3. Action-dependent task coverage complicates the model comparisons? Sec 5 makes later opportunities depend on earlier actions, while fig 6 compare aggregate configuration scores. I am a little bit confused. The eval protocol confirms that tasks on unreached branches are excluded and scores are averaged over active tasks. Excluding genuinely inapplicable tasks is sensible, but the evaluated task mix can consequently differ across agents. An agent that progresses further could face harder follow-up work than one that never reaches it.
    1. Also, the paper does not show the extent of this difference or its effect on the comparisons. Reporting branch coverage alongside scenario-goal completion would help distinguish performance differences from differences in the work encountered. This is an uncertainty about interpreting the current results, not evidence that the rankings are wrong.
  4. The trust loss–recovery asymmetry needs a more controlled analysis. Section 5.3 and Figure 7 report larger trust losses after aligned-to-misaligned transitions than gains in the reverse direction, plus incomplete recovery in one sequence. These observations are interesting, but the numerical contrast can depend on initial ratings, the bounded scale, scenario assignment, and which transitions enter each average. More details are required.
  5. The joint-optimization claim goes beyond the study's evidence. At the beginning the paper states that the human study demonstrates the importance of joint 3T optimization. Sec 5.3 shows that participants value timing and intervention alignment alongside correctness, but does not evaluate an agent optimized for the joint objective or compare its outcomes with alternative designs. Moreover, the conclusion reached appropriately identifies joint optimization as future work. IMO, the current evidence motivates joint consideration of the three dimensions, while the stronger optimization claim remains untested.
Questions for authors
  1. Does the human–LLM discrepancy persist on matched logs and trust questions, separating technical competence and distinguishing under-intervention from over-intervention?
  2. How often do correct deferral decisions lead to feasible continuation and completion before the deadline in the existing runs?
  3. How were trust sequences assigned across participants and scenarios, and what is the uncertainty in the loss–recovery difference when repeated participant responses are kept together?
4 comments

We appreciate the thoughtful feedback from the reviewer, while recognizing the need for joint consideration of 3T, task capability (TC), temporal allocation (TA), and trust (TR) for proactive AI agents. To clarify our core contribution: whereas most existing proactive agent literature focuses narrowly on TC, we bring the overlooked dimensions of TA and TR to the forefront. Beyond 3T, we holistically integrate a theoretical framework including 3T and technical layers, and an initial simulation environment testbed, Proactivity-Gym, for evaluation along with human validation of these concepts and testbed to observe real-world user behavior, as noted by reviewer N8EB and GqLa. In short, the role of this paper is to establish that proactive agents must be designed and evaluated on the three objectives together, and Proactivity-Gym, a proof-of-concept paving a direction for proper evaluation, serves as an initial testbed that reveals current agents’ inabilities.

Below, we compare LLM judges with human participants on the same trust logs, link TA results to compute allocation outcomes such as idle-time utilization, show that our comparisons remain constant under different branch activations, report ablations, such as uncertainty or variance, for the trust-loss asymmetry, and clarify the scope of our joint-3T claim. All additional analyses are added to the revised draft (Sec. 5.2-5.4, App. E.7, F.4, F.7, and G.5).

W1-1, Q1.Alignment of LLM judges with human judgments on same agent trajectories (Sec. 5.3, App. E.7)

The human–LLM discrepancy persists on matched logs. When an LLM judge rates the same trust logs as our human participants, with task outcomes held correct, it shows relatively less sensitive behavior toward intervention-depth misalignment compared to humans.

In detail, we provide an LLM judge (Qwen3.8-27B) the same trust logs from our human study questionnaire (Sec. 5.3), which also is a subsample of the scenario in Proactivity-Gym along with an agent’s trajectory. We gave all 30 questionnaires that were used in the study. These logs keep task capability constant, separating task capability.

The LLM judge tracked the direction of trust change, where LLM judges increased trust after intervention-aligned (A) actions and decreased it after misaligned (M) actions (human vs. LLM judge; A→M: −1.86 vs. −0.89; M→A: +1.27 vs. +0.91). However, the magnitude of trust loss and the ratio of trust loss to trust recovery were smaller than those of humans (1.46 vs 0.98). This behavior was also evident at the sequence level. In the A->M->A sequence, humans showed a 22.57% decrease in trust at the final aligned action relative to the first one, while the LLM judge only showed a 1.25% decrease. In the M->A->M sequence, human’s final trust after the misaligned action, despite the intermediate alignment, decreased by 26.20% compared to the initial trust, while LLM’s final trust rather increased by 2.10%.

Moreover, these findings persist under both under- and over-intervention cases. For instance, a single A->M transition reduced trust by 1.25 (humans) vs. 0.78 (LLM) when the violation was under-intervention, and by 2.60 vs. 1.03 when it was over-intervention. Consistent to our original findings, these results indicate the judge is lenient toward intervention-depth misalignment and less sensitive to it than humans. Regarding the abstract, “conflate” refers to this pattern. LLM-judged trust under-reacts to intervention-depth misalignment, which now also holds on matched logs with TC held constant, while showing relatively high correlation to TC.

W1-2. TC–TR-J correlation may stem from the perceived technical competence construct (Sec. 5.2)

The TC–TR-J association is not driven by the “perceived technical competence” (PTC) construct. When we exclude PTC from TR-J, the remaining four constructs still correlate strongly with TC (r = 0.689) and weakly with TR-D (r = 0.235), nearly unchanged from full TR-J (0.70 and 0.23; Fig. 6(c)). We agree that some correlation with TC is expected, since an agent that does the wrong work cannot earn trust. Our point is that judges remain largely insensitive to intervention-depth misalignment, despite the high sensitivity for humans, observed in our human study (Sec. 5.3). Moreover, qualitatively, we observe LLM judges justify their high TR-J scores despite intervention-depth misalignment by stating that they “did the work correctly task-wise”. In the matched-log analysis (W1-1), TC is held constant, yet the judge is still far more lenient toward misalignment than humans (A→M: −0.89 vs. −1.86).

W2&Q2. More analysis on TA compute allocation (Sec. 5.2, App. F.4)

Following your valuable suggestion, we additionally link prioritization and deferral decisions to outcomes from additional analyses on TA, which show the same gap. Agents utilize 10.1% of idle time on average, while completing 15.2% of tasks with deadlines.

In detail, as the reviewer respectfully mentions, we tend to bring TA to the forefront for proactive agents. Proactivity-Gym, as an initial testbed for evaluation, embeds this TA aspect, and we agree with the reviewer that it is indeed a useful diagnostic, where most papers do not consider this. In this testbed, our main focus is to evaluate the status quo of contemporary agents, where we observe frequent failures. Following your valuable suggestion, we report how much compute has been misallocated, and how an ideal proactive agent with proper TA considerations embedded can have utilized idle resources productively ultimately leading to feasible completion.

In our gym, an ideal system would have utilized 72.9% of idle-time, where models on average utilize 10.1% of the idle time, with Claude Opus 5 showing the highest coverage at 50.5% coverage. Note that agents can schedule their own reactivation to resume work later, which we call a wake-up. For misallocation, we compute the fraction of actual wake-ups that do not occur at the correct, deferred time window (# of wake-ups not matched to the right time window / # of total wake-ups). We observe that agents misallocate their wake-ups 74.2% of the time. Moreover, among the task opportunities that require a wake-up, agents correctly woke up only 9.9% of the time. Lastly, due to these limitations of current agents, the average deadline completion rate is 15.2%. The detailed analyses for each model have been added to our revision in Table 12.

W3. Fair comparison under different branch activations (Sec. 5.2, App. F.7)

We confirm that differences in branch activation do not change our comparisons, where our scenarios are designed so that a large majority of tasks are given to every agent in every run and model rankings remain nearly identical on commonly reached tasks and identical when unreached branches are counted as failures. Such branching resembles real-world agent usage, where an agent’s earlier action affects future state. Hence, depending on its earlier actions, an agent might fail to reach a certain branch, where the evaluation does not include that unreached branch. Though sensible, as the reviewer respectfully mentions, this evaluation scheme might penalize agents that progressed further and faced harder work.

To limit such bias, first, scenarios are designed such that a sufficient number of tasks are reached, where 84 of the maximum of 111 reachable tasks are reached by every agent in every run. As shown in the table below, every agent (harness: OpenClaw) reaches at least 76.58% of the 111 tasks across three runs (Reach@3) and at least 75.97% in a single run on average (Avg. Reach@1). Second, each configuration is run three times.

ModelReach@3 (%)Avg. Reach@1 (%)
Qwen 3.5 2B85.5977.48
Qwen 3.5 9B76.5875.97
Qwen 3.5 27B81.0877.28
Gemma 4 12B77.4875.97
Gemma 4 31B77.4876.38
GPT 5.6 Luna88.2977.98
GPT 5.6 Sol89.1981.18
Claude Sonnet 583.7880.68
Claude Opus 595.5085.59

Moreover, the rankings between models and harnesses remain mostly constant when we test for bias directly by computing the performance of each agent (model-harness configuration) on the 84 commonly reached tasks only (Table 15; note that we average across harnesses in the table below). Compared with the performance that we have reported previously in the paper (Table 7), Kendall’s tau is 1, 1, 0.992, and 0.937 for TC, TA, TR-J, and TR-D, respectively. Beyond rankings, the absolute scores stay close to those reported before, with a maximum difference of 1.91 points across all cells.

ModelTC (0–100)TA (%)TR-D (%)TR-J (1–5)E2E TC (0–100)
Qwen 3.5 2B20.790.1941.961.8414.88
Qwen 3.5 9B39.952.4145.413.1028.47
Qwen 3.5 27B44.116.3047.083.5132.01
Gemma 4 12B22.400.9338.882.8716.01
Gemma 4 31B44.513.7045.733.2432.04
GPT 5.6 Luna46.1110.8446.963.6133.27
GPT 5.6 Sol54.6517.7852.173.9640.58
Claude Sonnet 555.8016.6750.793.8541.93
Claude Opus 565.3651.6753.354.2452.59

W3. Fair comparison under different branch activations (continued)

Finally, regarding scenario-goal completion, we report two results: the branch coverage above (Reach@3 and Avg. Reach@1) and an end-to-end version of TC with a fixed denominator (E2E TC in the table above), where all 111 tasks are counted for every agent and a task on an unreached branch receives a score of 0. The scores of all agents decrease (by 4.0 - 14.2 percentage points) as some branches remain unreached, but the ranking of the 23 configurations is identical (Kendall’s τ=1) to that of Table 7, which was originally reported. These consistencies show that differences in branch activations did not bias our findings.

W4 & Q3 Additional Trust loss-recovery asymmetry Analyses (Sec. 5.3, App. G.5)

The loss–recovery asymmetry, which we also found as a very intriguing aspect of human behavior, holds under each control the reviewer lists. Beyond the detailed survey analyses in App. G.5 and G.6, we conducted additional analyses following your suggestion, as detailed below.

To recap, the average A \rightarrow M transition was -1.86, while the M \rightarrow A was +1.27. The A \rightarrow M transition includes AMM sequences’ first transition, MAM sequences’ second transition, and AMA sequences’ first transition. The M \rightarrow A transition includes MAA sequences’ first transition, MAM sequences’ first transition,and AMA sequences’ second transition.

Scenario Assignment. Sequences and scenarios are assigned to different participants, such that scenarios and sequences (M: misaligned intervention; A: aligned intervention; MMM, AAA, AMA, MAM, MAA, AMM) are evenly distributed. Every sequence type appears in every scenario.

Uncertainty in the loss-recovery difference. To compute the requested uncertainty, we compute the participant-level cluster bootstrap of 20K resamples. We observe that the trust loss exceeds the recovery value by 0.59 points with a 95% CI of [0.14, 1.03]. In the A->M->A sequence, humans showed a 22.57% decrease in trust at the final aligned action relative to the first one, where trust dropped by 1.00 point on average (95% CI [-1.57, -0.43]).

Initial ratings and bounded scale. To further verify that trust losses exceed trust gains beyond differences in initial ratings and the bounded 1–5 scale, we perform three additional analyses. Trust drops and rises start from different initial ratings, so the two can have different amounts of room to vary on a 5 point scale. To complement this difference, we fit a regression predicting the size of each change from its direction, its remaining room (starting rating − 1 for drops; 5 − starting rating for rises), and the scenario, with a separate baseline for each participant so that repeated responses from the same person are not treated as independent responses. With remaining room held fixed, drops were still 0.592 points larger than rises (95% CI [0.11, 1.07]). Moreover, among the participants who experienced both a drop and a rise, when we scale each change by its remaining range, the drop still exceeds the rise within the participant (Wilcoxon signed-rank test, p = .021).

Variance between Scenarios. The pattern of trust increases and decreases was consistent across scenarios. Average trust changes across sequences were strongly correlated across scenarios (mean pairwise Pearson r = 0.80; Spearman ρ = 0.82). The magnitude of change was also similarly small across scenarios. The standard deviation of average trust change for each transition type (A→A, A→M, M→A, M→M) was 0.32 points.

W5 Joint optimization claim

Our main claim is that TC, TA, and TR must be considered jointly, as the reviewer respectfully mentions. Originally, “joint optimization” referred to the goal for future agent development, motivated by demonstrating the distinct importance of each dimension, rather than a claim that we already trained such an agent. While prior work has focused largely on TC, we aim to establish this foundation by formulating the objective, providing the technical layers, and designing an initial testbed. Proactivity-Gym serves as a proof-of-concept, initial testbed that evaluates all three objectives together with proper simulation environments, which no existing benchmark does.

We show that agents are relatively strong on TC, while falling short on TA and TR. Moreover, our human study shows that users value TA and TR beyond TC. Eq. 1 is introduced “to guide agent design and evaluation” (Sec. 3.2) and building agents that optimize it is the next step that this work enables (Sec. 6 & App. H). To avoid ambiguity, we have revised the phrase in the abstract from “demonstrates the importance of the joint 3T optimization” to “supports the importance of jointly considering 3T” such that it helps to better clarify the paper's claims.

We once again thank the reviewer for the valuable suggestions.

AuthorsOpenReview ↗

Gentle Reminder to Reviewer virC

Dear Reviewer virC,

Thank you again for your detailed feedback. We are sending a gentle reminder regarding the response that we have posted for your review. In summary, we showed that:

  • On matched logs for trust, the human–LLM discrepancy in trust-loss recovery and magnitude persists for both under- and over-intervention (W1-1, Q1).

  • The TC–TR-J association holds without the perceived technical competence construct (r = 0.689) (W1-2).

  • For additional TA analyses, agents use 10.1% of idle time and complete 15.2% of deadline tasks (W2, Q2).

  • Rankings are robust to unreached branches, where the rankings are unchanged on commonly reached tasks. End-to-end ranking is also identical (Kendall's τ = 1) (W3).

  • The trust loss–recovery asymmetry holds robust under various ablational experiments such as, participant-clustered bootstrap (0.59, 95% CI [0.14, 1.03]), scenarios, and bounded scales. (W4, Q3).

  • We revised the abstract to "supports the importance of jointly considering 3T" for better clarity. (W5).

Could you let us know whether this addresses your concerns and if you have any additional comments? If so, we would appreciate it if you would consider updating your assessment.

Best regards,

Authors of Submission 39312

Reviewer_N8EbOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Agents can work 24/7, but they are often reactive instead of proactive. This paper proposes that the design of proactive agents be governed by task capability, temporal allocation, and trust. The authors run a human study to validate their proposal. In addition, the authors contribute a simulation-based evaluator called Proactivity-Gym to empirically test how to design around these three dimensions.

Critical strengths

The framework is well-motivated and clearly explained. A particular strength of the work is the holistic nature of the contribution. Rather than focusing solely on a conceptual or algorithmic contribution, the paper brings together a theoretical framework, a simulation environment for systematic evaluation, and a human-subject study that examines the approach in practice. These components complement one another well: the theoretical framework provides a principled foundation, the simulation environment enables controlled and reproducible experimentation, and the human study offers evidence about how the proposed ideas translate to real-world interactions and user behavior.

Critical weaknesses

Measuring the 3Ts at scale is inherently challenging. The authors address this challenge by relying on an LLM-based judge, which is a reasonable approach, but I think additional validation is needed to establish that the resulting scores are reliable measures of the intended constructs. The analysis in Appendix E provides some useful evidence in this direction, but it remains somewhat limited. In particular, I would have liked to see agreement evaluated across multiple frontier models to assess the robustness of the LLM-based judgments to the choice of evaluator. More importantly, comparing LLM judgments against human annotations and reporting inter-annotator agreement among human annotators would provide stronger evidence that the proposed evaluation procedure is well-calibrated.

Questions for authors
  1. Are the paper’s main conclusions robust to alternative evaluation procedures or judges? Can you provide agreement scores between LLM judges and human judges?
3 comments

We sincerely appreciate the reviewer’s highly encouraging assessment and recognition of our work’s holistic contribution of integrating the theoretical aspects with empirical simulation, which shows how proposed ideas can translate to real-world systems, and human study for proposing an overlooked concept in current agentic systems. Below, we provide analyses to address your main concerns regarding judge validity, robustness, and alignment with human judges. The details are also added to the revised draft on Sec. 5.3-5.4 and App E.6, E.7, and G.7.

Robustness across different LLM judges (Sec. 5.4, App. E.6)

Our conclusions are robust to our judge choices, where six additional frontier judges agree closely with our reported scores (configuration-level r ≥ 0.989, 0.925, and 0.907 for TC, TA, and TR-J), and the main findings hold under every judge.

In detail, as respectfully mentioned by the reviewer, measuring the 3T dimensions at scale is challenging, which is why we utilize two models from different families, Qwen 3.8-27B and Gemini-3.8-Flash, to avoid model-specific bias. We agree with the reviewer that the reliability of this evaluation scheme is an important factor for our work and further validation could strengthen our claims. Hence, we evaluate agents’ run trajectories (for OpenClaw Harness including all nine backbone models) with 6 additional frontier models, Claude 5.5 - Sonnet, Opus, GPT-6.1-Sol, Terra, Gemini 4 Argon, and DeepSeek V4 Pro. We report the correlations between the newly tested models’ judgment and the numbers that we report in the paper based on two judges. The numbers in each cell correspond to Pearson and Spearman correlations, respectively.

JudgeTC-HybridTATR-J
Claude Sonnet 5.50.997 / 1.0000.977 / 0.8560.985 / 0.983
Claude Opus 5.50.998 / 1.0000.946 / 0.8560.977 / 0.950
GPT-6.1-Terra0.991 / 1.0000.978 / 0.8770.907 / 0.767
GPT-6.1-Sol0.990 / 1.0000.986 / 0.9020.911 / 0.767
Gemini 4 Argon1.000 / 1.0000.981 / 0.9130.967 / 0.967
DeepSeek V4 Pro0.989 / 0.9830.925 / 0.8460.995 / 0.983

Consistent with Table 6, where the two original judges show high configuration-level correlations of r ≥ 0.95 on every metric, the frontier judges agree closely with the original scores (configuration-level r ≥ 0.989 for TC, ≥ 0.925 for TA, and ≥ 0.907 for TR-J) with a minimum pairwise correlation of 0.829 across the newly tested frontier judges. Importantly, the main findings from the paper hold under every judge, for instance TR-J continues to track TC more closely than TR-D, while models show weak performance on TA.

LLM Judgment vs Human Judge (Sec. 5.4, App. E.7)

Our LLM judges align well with human judgments on TC and TA (TC: Pearson r = 0.94; TA: 95% agreement with Cohen’s κ ≈ 0.6).

In detail, we further explore LLM judge alignment with human judgments for two of the LLM-judged metrics: TC and TA. Two annotators with ample experience in LLM agents rate in total 100 randomly sampled task outcomes and 100 decision windows for TC and TA judge validation, without access to the LLM scores. For both criteria, we observe high correlation between the reported judge scores and human scores. For TC, Pearson correlation reaches 0.94, while for TA, human and LLM judges show a 95% agreement with Cohen’s kappa approximately at 0.6.

MetricMeasureQwen 3.8-27BGemini 3.8-FlashReported Score
TCr / ρ.891 / .831.908 / .923.937 / .917
TAAgreement / κ.940 / .593.960 / .646.950 / –

Inter-annotator agreement (IAA) (Sec. 5.3, App. G.7)

Participants in our human study show substantial agreement, with Gwet’s AC1 ≥ 0.62 on all three dimensions and 73% overall pairwise agreement.

In detail, since participants differ in how they use the rating scale (e.g., one participant may rate only between 2 and 3 while another uses the full 1–5 range), we compute agreement on relative responses rather than raw scores. For A/B comparisons, we observe which option was chosen and which was rated higher and for TR ratings, we group the rating scales into whether trust increased, stayed the same, or decreased between consecutive checkpoints. We report pairwise agreement (the fraction of participant pairs giving the same response) and Gwet’s AC1 (Gwet, British Journal of Mathematical and Statistical Psychology, 2008), which corrects for chance agreement and remains stable under skewed responses.

MeasureTCTATROverall
Pairwise agreement83%71%72%73%
Gwet's AC10.810.620.630.65

We observe high IAA across all three dimensions (AC1 ≥ 0.62), where human participants showed substantial agreement of the importance of the 3T components. The agreement was the highest for TC, where correctness is most objective. The lower agreement on TR and TA is expected, as appropriate timing and intervention depth partly reflect personal preference (App. G.6). On raw 1–5 scores, participants rarely gave identical scores (AC1 = 0.26) but usually gave close ones (AC2 with quadratic weights = 0.64; Gwet, Advanced Analytics, 2012), indicating that their ratings follow similar trends even when their exact scores differ.

We once again thank the reviewer for their thoughtful assessment and suggestions.

AuthorsOpenReview ↗

Gentle Reminder to Reviewer N8EB

Dear Reviewer N8EB,

As the discussion period progresses, we would like to briefly follow up on our rebuttal response and thank you once again for your thoughtful comments and encouraging assessment.

In our rebuttal, we show that,

  • Six additional frontier judges, including Claude Opus 5.5, Gemini 4 Argon, and GPT 6.1 Sol, agree closely with our reported scores (configuration-level r ≥ 0.907), where our main findings hold under every judge.

  • Our LLM judges align with human annotations (TC: r = 0.94; TA: 95% agreement).

  • Human-study participants show substantial inter-annotator agreement. (Gwet's AC1 ≥ 0.62).

Please let us know if you have any additional questions regarding the paper or our response and if so, we would be happy to clarify them.

Best regards,

Authors of Submission 39312

Comments on the paper

2 comments

Thank you for your sparing your time for the follow-up and re-assessment.

Term "conflate" seems stronger than evidence

We would like to clarify that “conflate” in our abstract, refers to a pattern that we observe. LLM-judged trust largely follows task capability and gives little weight to intervention-depth alignment, which humans relatively perceive as an important factor of trust. It does not mean that the judges cannot tell the two apart at all. Three results supports this, where:

  • TR-J correlates strongly with TC (r = 0.70) but weakly with TR-D (r = 0.23). This holds even when perceived technical competence construct is excluded from TR-J (r = 0.69).
  • Contemporary LLM agents score high in TR-J, despite low TR-D (e.g., Claude Opus 5: 4.25 out of 5 for TR-J, but 52.4% on TR-D). When we ask the LLM judges the rationale behind these high TR-J scores, they justify that despite the intervention misalignment, the task accuracy was high enough.
  • On matched logs with TC held constant, LLM judge reacts far less than humans to misalignment (LLM judge vs human => A→M: −0.89 vs. −1.86; A→M→A: −1.25% vs. −22.57%), for both under- and over-intervention.

Nonetheless, to avoid such confusion, we edited our abstract from “reveal that LLM judges often conflate task capability and trust.” to “reveal that LLM-judged trust closely tracks task capability while remaining relatively insensitive to intervention-depth misalignment” to better-align the abstract with our claims.

More TA analyses

Idle-time utilization, where we used the term “idle-time coverage” in our draft (App. F.4) measures the percentage of a user’s idle time (as defined in Sec. 3.1) that an agent spends doing useful work (i.e. user’s idle time doing useful work / user’s total idle time) . Following your valuable suggestion, we observe what actually happens after a correct deferral. To do so, we collect the cases where an agent correctly defers its work. For each case, we inspect every subsequent turn, tool call, and file change after the deferral until the end of the scenario. Among them, only 16.4% actually resumed the deferred task, and only 4.1% completed the task with all the requirements met. Agents explicitly told the user in 49.3% of the cases that they will come back in the future to finish the deferred work, but never did so. Hence, even a correct deferral decision rarely becomes follow-through. Deciding to defer and returning to finish the deferred work are distinct capabilities, and PROACTIVITY-GYM is able to separate them because it tracks the deferred task across the rest of the multi-day run, which also shows another reason why this multi-day simulation gym is a must for proper proactivity evaluation.

We hope our response has addressed your additional concerns. If you have any concerns remaining, please let us know and we would be happy to clarify further. Otherwise, we would greatly appreciate it if you would consider updating your assessment. We look forward to hearing from you.

AuthorsOpenReview ↗

Gentle Reminder to Reviewer virC

Dear Reviewer virC,

As the discussion period progresses and closes soon, we are sending a gentle reminder regarding our response.

We first explain what we originally meant by "conflate" in our paper, while following your comments, we revised the abstract to replace “conflate” with "LLM-judged trust closely tracks task capability while remaining relatively insensitive to intervention-depth misalignment" that better aligns with our evidence. We also added additional analyses of correct deferrals, along with the definition of idle time utilization, where we use the term idle-time coverage in the paper.

We would appreciate it if you could let us know whether our response addresses your remaining concerns. If it does, we would be grateful if you would consider reflecting this in your assessment. Please also let us know if you have any further questions. We once again thank you for your time and suggestions.

Best regards,

Authors of Submission 39312

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.