acceptodds
Under review as a conference paper at ICLR 2027

When Should AI Chime In? Benchmarking Timely Contribution of Proactive Agents in Multi-Party Chat

Abstract

Proactive agents can support group collaboration by surfacing useful information without waiting to be asked, but even relevant contributions can be disruptive if poorly timed. In this work, we isolate the timing question: given a fixed candidate contribution such as a clarification or relevant fact, when should an AI agent chime in? We introduce PITCH (Proactive Intervention Timing in CHat), to our knowledge the first benchmark with large-scale human judgments of proactive agent timing collected from participants in the same unscripted multi-party conversations they took part in. To collect high-quality judgments, we design a novel interface in which 2–4 people chat synchronously, author candidate AI contribution messages, place their own and others’ contributions into the chat history, and then deliberate as a group about the best placement for each message. PITCH contains 384 candidate contributions and 1,091 judgments, each marking an ideal turn, an acceptable window, or that a contribution should not appear, from 89 participants engaged in 35 conversations. Evaluating 15 models across 28 configurations, we find that while models can place contributions well when given the entire conversation history, they systematically chime in too early when asked to decide live as the conversation unfolds: best-turn accuracy falls from 57.8% to 19.0%, with nearly half of placements occurring before every participant’s acceptable window. In comparison, lay humans deciding live are more accurate (23.7% vs 19.0%), and fewer of their placements are premature (39.7% vs 46.4%). Model reasoning traces show that 93% of intervention justifications cite relevance, while almost none consider waiting. By providing a human-grounded benchmark, our work reveals gaps in agents’ ability to integrate into group settings and points towards proactive agents that can reason explicitly about appropriate timing.

Primary area:
datasets and benchmarks
Keywords:
agent, MULTI-PARTY CONVERSATIONS, real-human dataset

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8eexOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces PITCH, a benchmark for evaluating the timing of proactive agent interventions in multiparty conversations. By fixing the candidate contribution, the work effectively isolates timing as the primary variable of interest. It collects retrospective timing preferences (ideal turns and acceptable windows) directly from the conversation participants, yielding 1,091 judgments across 35 conversations and 384 contributions. The evaluation of 28 model configurations reveals a substantial offline–online performance gap and a strong tendency for models to intervene prematurely, a finding supplemented by an independent online human baseline study.

Critical strengths
  1. The benchmark’s grounding in actual participant preferences is a major strength. By retaining acceptable timing windows and acknowledging annotator disagreement, it offers a much more realistic and nuanced evaluation framework than approaches that assume a single, universally correct insertion point. This is a substantive contribution to the evaluation of proactive agents.

  2. Holding the contribution content fixed effectively isolates timing as the central evaluation target. The consistent offline–online gap observed across diverse model configurations provides compelling evidence that retrospective selection does not trivially translate to effective online decision-making.

Critical weaknesses
  1. There are notable discrepancies in reported statistics that hinder reproducibility. Specifically, the endorsement rates in Figure 3 (11.8% ideal-turn and 34.5% window for replay; 37.1% and 76.8% for full-transcript) appear to be swapped with the row assignments in Table 6. Furthermore, the human baseline study reports conflicting sample sizes: Section 5 mentions 81 recruits, 56 contributing annotators, and 192 placements, whereas Appendix D.3 and Table 4 cite 109 recruits and 331 retained observations. While this may stem from different filtering criteria or study subsets, the manuscript must explicitly reconcile these figures to ensure the human-baseline comparisons are traceable.

  2. The human-versus-model comparison requires more rigorous statistical backing. Comparing an aggregate human accuracy of 23.7% to a pooled model average of 19.0% is misleading, especially since the strongest replay configuration achieves 32.8%. Furthermore, the models are evaluated on all 384 messages, while the human study reports a different, inconsistently documented number of placements. Without a matched comparison on the exact same items, with appropriate uncertainty estimates that account for the nested structure of the data (i.e., shared conversations and repeated annotators), the conclusion regarding a general human advantage remains unsubstantiated.

Questions for authors
  1. Can you reconcile Table 6 with Figure 3 and provide consistent recruitment, exclusion, and placement counts identifying which human sample produced each reported result?

  2. On exactly the same candidate messages and with identical weighting, what are the human, pooled-model, and strongest-model results, including uncertainty accounting for shared conversations and repeated annotators? Does the claimed human advantage remain under this comparison?

  3. How reliable are the LLM-assigned firing-justification labels against human annotation? Where preceding NO justifications are available, do they contain consideration of waiting that is absent from the final firing justification? Would examining these earlier decisions support the broader interpretation about waiting, or restrict the conclusion to the justifications expressed at commitment?

2 comments

Hi Reviewer 8eex,

Thank you for the review. We traced each flagged inconsistency and recomputed all human-model comparisons. No correction changes the main finding about the offline-online gap or the early-commitment skew. However, we do see that the corrected lay-human baseline performances are all lower, and we've softened the human-advantage claim.

Q1: Reconciling Table 6 with Figure 3, and consistent recruitment, exclusion, and placement counts.

Appendix Table 6 accidentally swapped Tasks A and C; we've corrected it to 11.8%/34.5% for replay, 14.1%/36.2% for delayed replay, and 37.1%/76.8% for full-transcript selection.

For the counts, we separated the primary corpus from the lay-human replay study for clarity:

  • The primary annotated corpus contains 83 unique group participants, 96 participant-conversation memberships (several participants participated multiple times), 35 conversations, 384 candidate messages, and 1,091 judgments. The previous "89 participants" count included testing accounts and accounts for no-show participants; we've corrected the count throughout the paper, and no analysis was affected.
  • After collecting the main benchmark dataset, we additionally recruited 108 participants to review a subset of the collected conversations and decide where to place each message, serving as the human replay baseline. Collectively, they produced 330 judgments on 193 messages in 33 conversations.

Q2: Human, pooled-model, and strongest-model results on identical items with identical weighting.

While preparing this response, we found an inconsistency in the lay-human scoring: humans were scored against the minimum-to-maximum envelope of participants' windows, whereas models were scored against the true disjoint union of windows over all decisions. We thus recalculated the human baseline performance with the identical canonical scorer used for the model performance: human performance is 21.5% ideal-turn, 48.2% anyone-window, and 29.5% unanimous-window accuracy, with a 36.1% early-placement rate (replacing 23.7%, 53.7%, 33.2%, and 39.7%).

On the 193 messages covered by humans and all 28 arms:

  • humans: 21.5% ideal / 48.6% window;
  • model fixed-arm mean: 17.2% / 51.4%;
  • strongest observed arm (Gemini 3.6 Flash with reasoning): 33.5% / 72.0%.

We modified the claim about general human advantage to more accurately reflect the finding. The revised manuscript states: the strongest configuration outperforms lay humans on the matched items, while most configurations are premature more often than lay humans (19 of 28 arms were early more often than lay humans). Under bootstrap intervals crossed at participant and conversation level, no arm was reliably above lay humans on best-turn accuracy and 5 were reliably below. On window accuracy, 5 were reliably above and 4 reliably below. The general human advantage does not hold. However, this doesn't change the paper's main argument: the retrospective human label reflects an ideal expectation for agents chiming into group chats, and current models systematically chime in too early. The lay-human baseline shows the online task is hard for humans too, so agents should step beyond simply replicating human heuristics and aim for superhuman performance on this task.

Q3: Reliability of the firing-justification labels, and whether preceding NOs contain waiting that is absent at commitment.

We validated the judge directly against human annotation. One researcher hand-labeled 25 commitment justifications from a stratified validation sample. Human-judge agreement is 92% (κ\kappa = 0.75) on whether the justification explicitly weighs waiting, and 88% (κ\kappa = 0.76) on whether it argues the message's relevance. We then applied another LLM judge (DeepSeek V4 Pro) with a matched label prompt to all 3,746 immediately-preceding NO segments: it labels 21.5% (95% CI [18.1, 25.0], conversation-clustered bootstrap) as weighing waiting, that is, conceding the message's relevance while preferring a later moment. The rate is essentially unchanged (21.4%) in the subset most relevant to your question: of the 3,194 preceding-NO segments whose eventual firing justification mentions only relevance, 682 had explicitly weighed waiting one turn before the model committed. On 29 preceding-NO segments hand-labeled by one author blind to the judge's labels, this judge agrees 86% of the time (κ\kappa = 0.70). In short, models weigh waiting in roughly one in five immediately-preceding NO decisions, yet only 6 of 4,349 commitment justifications (0.14%) weigh waiting. This agrees with the wait-aware ablation (reply to ughR, W2), which changes placements without reversing the early trend. One plausible interpretation is that models' consideration of waiting could be outweighed by their consideration of topical relevance when they commit.

Dear authors,

Thank you for the corrections and additional analyses. The consistent scoring, matched human–model comparison, and revised claims substantially address my concerns about the reported results.

My remaining concern is the interpretation of premature commitment. The preceding-NO analysis shows that models sometimes explicitly consider waiting, while human validation remains limited to small samples labeled by one researcher. This provides useful descriptive evidence about the justifications, but limited support for the proposed explanation that relevance outweighs timing considerations.

I will keep my current score.

Reviewer_ughROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PITCH, a benchmark for when a proactive agent should make an intervention (a fixed, helpful message) in a multi-party text chat. Prior timing benchmarks have two gaps. They use synthetic conversations, or infer the preferred timing with an LLM judge or outside annotators, so no actual participant of the conversation labels when an intervention is welcome; and they assume a single gold intervention turn. PITCH addresses both: it collects 35 real, unscripted chats (89 participants), takes timing labels from each chat's own participants, and asks for a best intervention turn plus an acceptable range of turns. Participants wrote 384 candidate interventions after their chat and labeled the timing of each intervention in their group, first independently and then after seeing others' choices (1,091 judgments). Across 15 LLMs in 28 configurations, model interventions match some participant's best turn 57.8% of the time when the model sees the full transcript and 19.0% when it decides turn by turn, where 46.4% of interventions come before every acceptable range.

Critical strengths
  1. The first benchmark whose intervention timing is labeled by the conversation's own participants. This is the main reason the dataset is worth having: it records when the people in a conversation would have welcomed a message, which labels from LLM judges or outside annotators cannot provide.
  2. The first multi-party chat benchmark to label acceptable ranges instead of a single gold turn. It captures how participants actually agree: they usually disagree on the exact best turn but often share an acceptable range.
Critical weaknesses

W1. Why the acceptable range matters is not explained. The paper motivates ranges by the ambiguity of timing and by messages that fit several points (Sec. 3, 4, 6), and concedes that they cannot show whether an intervention improves the discussion (Sec. 6, l.428–430). It does not say how the range should guide the design or evaluation of an agent, or which range a model should target: any annotator's range is lenient (a random turn falls inside it 25.5% of the time, Table 3), and the range shared by all annotators contains no turn for 88 of 384 interventions. This weighs most in my rating, because the labels are the paper's main contribution.

W2. The early-intervention result is tested with only one prompt. A prompt that asks the model to say "yes" only at the right moment, and tells it that it will be asked again later, might reduce early intervention. The authors list prompt ablations as future work (Sec. 6, l.436–437), but the abstract's claim that models systematically intervene too early depends on them, so they belong in this paper, or the claim should be softened.

Questions for authors
  1. (W2) How does the early-intervention rate change in the turn-by-turn settings when the prompt asks the model to say "yes" only at the right moment, tells it that it will be asked again later, and drops the "yes" example? A drop would show that per-turn asking pushes models toward "yes", and no drop would support the paper's explanation that models treat relevance as a sufficient reason to speak; either answer would raise my score.
  2. (W1, clarification) Which range should a model target, any annotator's range or the range shared by all annotators, and is there evidence, for example from participants' optional rationales (l.799–800), that interventions inside a range help the group?
1 comment

Hi Reviewer ughR,

Thank you for your thoughtful review. To respond to Q1 and W2, we've added an ablation experiment to test an augmented wait-aware prompt, and observed a similar trend of models chiming in early. To respond to your other question, which target (acceptable range) an agent should use: no single target fits every deployment, so we further clarified the four possible targets based on the different ways to aggregate the human annotations, and calibrate each against chance.

Q1 and W2: The wait-aware prompt ablation shifts placements but does not remove the early skew.

We ran the ablation on all 384 items for Tasks A and B. The revised prompt states that each message can be inserted only once, that the model will be asked again at later turns, and that speaking now is irreversible and should be chosen only if a better moment later is unlikely. We applied it to three arms spanning the original replay accuracy range: Kimi K3 standard (4th of 28 on original replay best-turn accuracy), GPT-5.6 Sol reasoning (8th), and GPT-OSS 120B reasoning (28th). Each cell reads original prompt → wait-aware prompt.

Despite explicit prompting, models still commit early. On Task A the early rate (placements before every participant's window) fell for Sol (40.6% to 35.4%), rose for Kimi (31.5% to 44.3%) and stayed flat for GPT-OSS (78.6% to 78.4%). On Task B it fell for Sol (42.2% to 36.5%) and rose for Kimi (30.7% to 45.8%) and GPT-OSS (71.1% to 75.8%). The full ablation (prompt text, per-task results, placement histograms) will appear as an appendix section in the revision.

Model configurationTaskHits some participant's best turn (higher = better)Inside some participant's acceptable window (higher = better)Too early for every participant (lower = better)
GPT-5.6 Sol (reasoning)A (replay)22.4% → 26.0%56.5% → 62.5%40.6% → 35.4%
GPT-5.6 Sol (reasoning)B (delayed replay)25.3% → 25.5%56.2% → 61.7%42.2% → 36.5%
Kimi K3 (standard)A (replay)27.1% → 19.0%66.7% → 54.2%31.5% → 44.3%
Kimi K3 (standard)B (delayed replay)30.5% → 22.4%68.0% → 53.4%30.7% → 45.8%
GPT-OSS 120B (reasoning)A (replay)6.2% → 6.0%20.8% → 20.8%78.6% → 78.4%
GPT-OSS 120B (reasoning)B (delayed replay)10.2% → 9.4%27.6% → 23.4%71.1% → 75.8%

W1 and Q2: (a) Which range should an agent target, and (b) do in-range interventions help the group?

(a) The paper argues that the same message can be appropriate at multiple distinct points, and that ground truth should preserve this (previously in Discussion). We will make this explicit in the paper by adding the following definitions of four targets, each for a different deployment context, to RQ1:

  • Anyone's best turn is the union of the exact best-turn votes. It exists for all 384 messages and tests whether a message's placement falls on at least one participant's ideal point. A random turn matches it 3.7% of the time.
  • Anyone's acceptable window is the true union of windows. This metric is useful for ruling out turns that nobody accepts, but is deliberately lenient: a random turn satisfies it 25.5% of the time.
  • Everyone's acceptable window is the intersection of all windows. It exists for 296/384 messages (77.1%) and is a conservative group-safe target. Among messages where it exists, a random turn satisfies it 7.6% of the time.
  • Everyone's best turn requires exact consensus and exists for only 114/384 messages (29.7%); among those, a random turn matches it 2.2% of the time.

We state the main empirical findings (the offline-online gap and the early-commitment skew) against the two targets that exist for every message, and note that the stricter consensus targets suit deployments where an ill-timed interruption costs more than a missed one, such as a workplace channel or a sensitive negotiation.

(b) You asked whether in-range interventions help the group. Unfortunately, this is not within the scope of what our collected data can provide evidence for. The revision states this limitation. Our labels came after each conversation ended. Measuring group outcomes needs a deployment study, because each message inserted into a live conversation alters the exchange that follows. The 143 optional rationales participants submitted, covering 126 messages, explained retrospective timing preferences but contained no evidence on the group outcomes. We expect that future work that deploys proactive agents in actual groups can provide more evidence on the downstream group outcomes. Our paper provides participant-grounded preferences about the timing of a fixed contribution, which provides a valuable evaluation asset for developing such proactive group agents.

Reviewer_GSNoOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a benchmark for measuring how well agents do at proactively engaging in group chats. This is a tricky problem. On the one hand, it can be quite useful for proactive agents to jump in, engage, and provide useful information without waiting to be asked. On the other hand, this can be very annoying and disruptive if the timing is not right. PITCH is a benchmark that tries to measure that timing question: that is, given a candidate contribution (clarification, relevant fact), decide when in a conversation an AI agent should provide it.

Critical strengths

I found this paper to be well written, interesting, and a nice dataset/study contribution. Some of the empirical findings may also be quite useful (such as the performance gap between decisions make in hindsight vs online, and the observation that the online errors are overwhelmingly premature rather than late).

Critical weaknesses
  1. The human references in Fig. 4 and Table 3 are confusing. First, isn't the 40.6% reference in Figure 3 for participants after they have had the conversation? This is much closer to task (C), though with even more information, since its is after the Phase 4 deliberation where they saw the other peer placements? So I don't see how this supports the "under online evaluation, models achieve less than one-third of the human best-turn share and less than half of the window share" claim. In Table 6, for example, after the full conversation but before deliberation, participants score at 31.3/67.7, which models in task (C) already beat at 37.1/76.8 (by the way, note that rows (A) and (C) appear to be accidentally swapped in Table 6 too?).

  2. I'm not convinced that there isn't a bias in the task setup towards penalizing chime-in timing that is early, but still reasonable. The ground-truth labels are all retrospective (collected after the full conversation is over). For any message, either the best timing is the first reasonable moment it comes up, or it is later in the conversation... but in a way that is possibly unknowable to an online agent (e.g., that the group will circle back to the topic in N turns with an even more relevant question). Even if the first moment was completely reasonable in real time, AFAIC each participant only labels a single window? So if they pick the later moment in hindsight, the early intervention gets scored as "premature." Section 6 mentions that multiple valid timing windows can cause disagreement among annotators (which supports, at least that this possibility does exist) and briefly notes that the reference is retrospective, but doesn't really address how this systematically biases the online evaluation against early interventions (which would also explain why online human annotators are "premature" ~40% of the time, or that is, about 86% of the incorrect placements being premature).

  3. Relatedly, looking at the prompts in Appendix G. Its not clear that there is anything in the prompts that suggests that the model should, e.g., weigh weighting for a better moment to chime in. If so, its not surprising that nearly all of the chiming in reasoning is about relevance only.

Questions for authors

Why is the average model performance something we should be caring about? Most of the comparisons in the abstract and text use the median arm performance (e.g., 19%). But some of the models are really quite bad. If, instead, we look at the best models, they outperform the lay humans in this benchmark.

I would be curious to see if you tried a prompt ablation for replay that, e.g., explicitly tells the model that YES is its only chance to send the message (L242-243), and asks it to consider whether a better moment may come later.

2 comments

Hi Reviewer GSNo,

Thank you for the careful and constructive review. We interpret the core question as whether the early-commitment trend is an artifact (1) of retrospective labels that unfairly penalize early placements, or (2) of a prompt that doesn't require models to consider waiting. We added two analyses addressing these concerns.

W2: The retrospective labels are not drastically different from the placements participants would have wanted during the conversation.

We understand the concern that retrospective labels are elicited after the conversation, but they provide a reasonable proxy of what participants would have desired during the conversation. To contextualize the potential bias in the benchmark labels, we analyzed the placement of private bookmarks recorded during the conversation (bookmarks were recorded via a button next to the message draft box during the live conversation in stage 1; the feature was optional, but we explicitly introduced it to participants as a recall tool for constructing and annotating messages in the next stage). We found that of the 404 bookmarks participants created during the conversation, only 3.5% of them precede every window the same participant later marked retrospectively, and 70.3% fall inside at least one window the same participant later marked. The bookmarks show that the retrospective labels are not drastically different from what participants would have wanted during the conversation.

W3 & Q2: Telling the model explicitly to consider waiting does not fix the early bias.

We ran a wait-aware prompt ablation, where the revised prompt explicitly tells the model that each message can be inserted only once, that speaking now is irreversible, and that it should choose to speak only if waiting is unlikely to offer a better moment. We applied it to three model configurations spanning the original replay accuracy range: Kimi K3 standard (4th of 28 on original replay best-turn accuracy), GPT-5.6 Sol reasoning (8th), and GPT-OSS 120B reasoning (28th). The early-commitment trend persists across all three tested model configurations on both online tasks (Task A. Replay, Task B. Replay with five-turn lookahead). Specifically, Task A early rates (placements before every participant's window) went from 40.6% to 35.4% for GPT-5.6 Sol, 31.5% to 44.3% for Kimi K3 and 78.6% to 78.4% for GPT-OSS 120B. With this prompt, all three still placed messages, on average, 18% to 37% of conversation length before the human best turn. The early bias is therefore not explained by the absence of an instruction to consider waiting. The full design and result table are in our response to Reviewer ughR (W2).

(Response continues in the next comment.)

(Continued from the previous comment.)

W1: The swapped Table 6 rows and the human references.

Thank you for pointing out the inconsistency. We've fixed Table 6 (now Table 7) in the Appendix to accurately characterize model and human baseline performance on all Tasks. These are endorsement rates characterizing how well a retrospectively injected AI message would have been received by the group. You are right that the 40.6%/76.1% (best-turn/window) endorsement represents the "best" human effort at deciding when to interject a message post-deliberation. Given the amount of information available for that decision, it is closest to Task C, where models on average reach 37.1% endorsement against participants' best-turn selections and 76.8% against the windows participants considered acceptable.

However, in practice it is unrealistic for agents to obtain such hindsight. Tasks A and B therefore test the more realistic setup, where the agent must decide online whether to interject. There, agents' decisions are received far less well: on average, only 11.8% of participants would consider a placement aligned with their own best turn, and only 34.5% would find it within their acceptable window.

To state our position more accurately: one of the paper's points is that post-deliberation placement is the ground truth an ideal proactive agent should target, because it is where the conversation's own participants converge, after deliberation, on when a contribution would have served the group best. Task C shows this target is largely recoverable with hindsight, so the A/C and B/C performance gaps measure the cost of deciding from earlier turns only and committing to one insertion, which live agents cannot avoid.

Q1: Average vs. strongest configurations against lay humans.

We removed the general human-advantage claim. Following reviewer 8eex's suggestion, we reran the comparison on only the 193 messages with human replay annotations. The strongest model configuration beats lay humans, but 19 of 28 commit early more often than lay humans (also in 8eex Q2). Lay humans reach 21.5% best-turn and 48.6% window accuracy, and the strongest configuration (Gemini 3.6 Flash with reasoning) reaches 33.5% and 72.0%. By point estimate, 7 of 28 configurations exceed lay humans on best-turn accuracy (21 are at or below, ties not counted as exceeding) and 18 of 28 on window accuracy. However, under bootstrap intervals crossed at participant and conversation level, none is reliably above lay humans on best-turn accuracy and 5 are reliably below. On window accuracy, 5 are reliably above and 4 below. This comparison shows that most model configurations are not reliably different from lay humans, who find the online task hard.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.