- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces PITCH, a benchmark for evaluating the timing of proactive agent interventions in multiparty conversations. By fixing the candidate contribution, the work effectively isolates timing as the primary variable of interest. It collects retrospective timing preferences (ideal turns and acceptable windows) directly from the conversation participants, yielding 1,091 judgments across 35 conversations and 384 contributions. The evaluation of 28 model configurations reveals a substantial offline–online performance gap and a strong tendency for models to intervene prematurely, a finding supplemented by an independent online human baseline study.
Critical strengths
-
The benchmark’s grounding in actual participant preferences is a major strength. By retaining acceptable timing windows and acknowledging annotator disagreement, it offers a much more realistic and nuanced evaluation framework than approaches that assume a single, universally correct insertion point. This is a substantive contribution to the evaluation of proactive agents.
-
Holding the contribution content fixed effectively isolates timing as the central evaluation target. The consistent offline–online gap observed across diverse model configurations provides compelling evidence that retrospective selection does not trivially translate to effective online decision-making.
Critical weaknesses
-
There are notable discrepancies in reported statistics that hinder reproducibility. Specifically, the endorsement rates in Figure 3 (11.8% ideal-turn and 34.5% window for replay; 37.1% and 76.8% for full-transcript) appear to be swapped with the row assignments in Table 6. Furthermore, the human baseline study reports conflicting sample sizes: Section 5 mentions 81 recruits, 56 contributing annotators, and 192 placements, whereas Appendix D.3 and Table 4 cite 109 recruits and 331 retained observations. While this may stem from different filtering criteria or study subsets, the manuscript must explicitly reconcile these figures to ensure the human-baseline comparisons are traceable.
-
The human-versus-model comparison requires more rigorous statistical backing. Comparing an aggregate human accuracy of 23.7% to a pooled model average of 19.0% is misleading, especially since the strongest replay configuration achieves 32.8%. Furthermore, the models are evaluated on all 384 messages, while the human study reports a different, inconsistently documented number of placements. Without a matched comparison on the exact same items, with appropriate uncertainty estimates that account for the nested structure of the data (i.e., shared conversations and repeated annotators), the conclusion regarding a general human advantage remains unsubstantiated.
Questions for authors
-
Can you reconcile Table 6 with Figure 3 and provide consistent recruitment, exclusion, and placement counts identifying which human sample produced each reported result?
-
On exactly the same candidate messages and with identical weighting, what are the human, pooled-model, and strongest-model results, including uncertainty accounting for shared conversations and repeated annotators? Does the claimed human advantage remain under this comparison?
-
How reliable are the LLM-assigned firing-justification labels against human annotation? Where preceding NO justifications are available, do they contain consideration of waiting that is absent from the final firing justification? Would examining these earlier decisions support the broader interpretation about waiting, or restrict the conclusion to the justifications expressed at commitment?
2 comments
Dear authors,
Thank you for the corrections and additional analyses. The consistent scoring, matched human–model comparison, and revised claims substantially address my concerns about the reported results.
My remaining concern is the interpretation of premature commitment. The preceding-NO analysis shows that models sometimes explicitly consider waiting, while human validation remains limited to small samples labeled by one researcher. This provides useful descriptive evidence about the justifications, but limited support for the proposed explanation that relevance outweighs timing considerations.
I will keep my current score.
Hi Reviewer 8eex,
Thank you for the review. We traced each flagged inconsistency and recomputed all human-model comparisons. No correction changes the main finding about the offline-online gap or the early-commitment skew. However, we do see that the corrected lay-human baseline performances are all lower, and we've softened the human-advantage claim.
Q1: Reconciling Table 6 with Figure 3, and consistent recruitment, exclusion, and placement counts.
Appendix Table 6 accidentally swapped Tasks A and C; we've corrected it to 11.8%/34.5% for replay, 14.1%/36.2% for delayed replay, and 37.1%/76.8% for full-transcript selection.
For the counts, we separated the primary corpus from the lay-human replay study for clarity:
Q2: Human, pooled-model, and strongest-model results on identical items with identical weighting.
While preparing this response, we found an inconsistency in the lay-human scoring: humans were scored against the minimum-to-maximum envelope of participants' windows, whereas models were scored against the true disjoint union of windows over all decisions. We thus recalculated the human baseline performance with the identical canonical scorer used for the model performance: human performance is 21.5% ideal-turn, 48.2% anyone-window, and 29.5% unanimous-window accuracy, with a 36.1% early-placement rate (replacing 23.7%, 53.7%, 33.2%, and 39.7%).
On the 193 messages covered by humans and all 28 arms:
We modified the claim about general human advantage to more accurately reflect the finding. The revised manuscript states: the strongest configuration outperforms lay humans on the matched items, while most configurations are premature more often than lay humans (19 of 28 arms were early more often than lay humans). Under bootstrap intervals crossed at participant and conversation level, no arm was reliably above lay humans on best-turn accuracy and 5 were reliably below. On window accuracy, 5 were reliably above and 4 reliably below. The general human advantage does not hold. However, this doesn't change the paper's main argument: the retrospective human label reflects an ideal expectation for agents chiming into group chats, and current models systematically chime in too early. The lay-human baseline shows the online task is hard for humans too, so agents should step beyond simply replicating human heuristics and aim for superhuman performance on this task.
Q3: Reliability of the firing-justification labels, and whether preceding NOs contain waiting that is absent at commitment.
We validated the judge directly against human annotation. One researcher hand-labeled 25 commitment justifications from a stratified validation sample. Human-judge agreement is 92% (κ = 0.75) on whether the justification explicitly weighs waiting, and 88% (κ = 0.76) on whether it argues the message's relevance. We then applied another LLM judge (DeepSeek V4 Pro) with a matched label prompt to all 3,746 immediately-preceding NO segments: it labels 21.5% (95% CI [18.1, 25.0], conversation-clustered bootstrap) as weighing waiting, that is, conceding the message's relevance while preferring a later moment. The rate is essentially unchanged (21.4%) in the subset most relevant to your question: of the 3,194 preceding-NO segments whose eventual firing justification mentions only relevance, 682 had explicitly weighed waiting one turn before the model committed. On 29 preceding-NO segments hand-labeled by one author blind to the judge's labels, this judge agrees 86% of the time (κ = 0.70). In short, models weigh waiting in roughly one in five immediately-preceding NO decisions, yet only 6 of 4,349 commitment justifications (0.14%) weigh waiting. This agrees with the wait-aware ablation (reply to ughR, W2), which changes placements without reversing the early trend. One plausible interpretation is that models' consideration of waiting could be outweighed by their consideration of topical relevance when they commit.