acceptodds
Under review as a conference paper at ICLR 2027

Retrieval Apophenia: How Growing Candidate Pools Make New Events Look Familiar

Abstract

Streaming event induction asks whether an arriving document belongs to an event already in memory or starts a new one. Most systems answer this question by retrieving historical candidates, taking the candidate with the highest compatibility score, and comparing that score with a fixed threshold. We show that this maximum rule fails in a specific and predictable way: as memory grows, the maximum is taken over more candidates, making a genuinely new event increasingly appear familiar. We call this phenomenon candidate-pool false familiarity. Prior work has studied threshold drift in streams and score inflation in large candidate galleries, but these effects have not been connected within streaming event memory, where an erroneous merge changes the state against which subsequent documents are judged. We measure this effect on annotated non-coreferent candidates and distinguish between two possible explanations. If the drift were merely an uncalibrated score scale, normalising each query by its candidate pool should eliminate it. However, this removes only a small portion of the effect, indicating that the decision must examine the candidates themselves rather than rely on a single score. We therefore propose C2E-Judge, which keeps the retriever, candidate budget, and memory fixed while allowing a language-model judge to inspect the entire bounded pool. It separates the two error types: adding candidates improves repeated-event coverage with nearly no increase in false merges, whereas replacing score thresholding with whole-pool judging removes most false merges. This visibility effect persists under a stricter prompt, a second judge endpoint, and a second dataset.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
streaming event induction, cross-document event coreference, LLM as judge

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_MBMpOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies streaming event induction, where each incoming event-bearing document must either be assigned to an event already represented in memory or start a new event. The paper focuses on a common decision rule that retrieves historical candidates, takes the maximum compatibility score, and compares it against a fixed threshold. The authors identify what they call candidate-pool false familiarity: as the candidate pool grows, the maximum similarity among non-coreferent candidates tends to increase, making genuinely new events increasingly likely to appear familiar. The paper first diagnoses this phenomenon using first mentions of gold events and historical candidates that are explicitly annotated as non-coreferent. It studies the maximum score MKM_K over candidate pools of different sizes and shows that both the mean maximum and the fixed-threshold false-match rate increase substantially with KK. The authors further introduce a pool-relative extremeness score to test whether this effect is merely a score-calibration artifact and report that this normalization removes only part of the observed drift. Motivated by this diagnosis, the paper proposes C2E-JUDGE, which retains the same source-excluded BGE-M3 retrieval and bounded candidate set but replaces maximum-score thresholding with a language-model judge that jointly inspects the retrieved candidate reports and outputs either a candidate identifier or UNSEEN. On WEC, the proposed system improves B-cubed F1, ARI, new-event false-merge risk, and repeated-event coverage relative to the WEC-score baseline. The paper further performs a same-judge visibility sweep over different numbers of visible candidates, evaluates a stricter prompt and a second judge endpoint, and reports a transfer experiment on a binary ECB+ projection. Overall, the paper provides a useful empirical diagnosis of maximum-based decisions in streaming event memory and shows that candidate-conditioned language-model judgment can substantially improve the evaluated streaming clustering system.

Critical strengths
  1. The strongest part of the paper is the diagnostic experiment. The authors restrict the analysis to first mentions of gold events and historical candidates annotated as belonging to different gold events. This makes every comparison in the diagnostic explicitly non-coreferent. Moreover, they provide a paired analysis on the same 523 queries, rather than relying only on different valid-query subsets at different candidate counts. On these queries, the mean maximum increases substantially as the candidate pool grows, and the associated fixed-threshold false-match probability also increases. This provides convincing empirical evidence that a maximum-based decision statistic can become increasingly permissive as additional non-coreferent alternatives are introduced.
  2. The paper does more than compare a conventional scorer against an LLM judge. Importantly, it reruns the same streaming system using the same judge, prompt, retriever, and memory while varying only the number of visible candidates. This is a useful dose-response experiment and substantially strengthens the empirical claim that additional candidate evidence can improve repeated-event recovery. The paper also reports uncertainty intervals, a stricter prompt, a second judge endpoint, and a transfer check on ECB+.
Critical weaknesses
  1. My main concern is a mismatch between the diagnostic mechanism and the operational system. The central narrative is that, as streaming memory grows, a new event becomes increasingly likely to look familiar because the maximum is taken over more candidates. However, the diagnostic establishes this by explicitly varying the number of candidates entering the maximum, K∈{4,16,64,256}K\in\{4,16,64,256\}, whereas the actual end-to-end system uses a fixed top-eight candidate budget. These are not equivalent interventions. With a fixed top-eight retriever, increasing the size of the historical memory does not increase the number of scores entering the final maximum; it instead changes which eight candidates survive retrieval. The relevant operational mechanism is therefore closer to max⁡c∈TopK(Ht)s(xt,c)\max_{c\in TopK(H_t)}s(x_t,c) with K=8K=8 and ∣Ht∣↑|H_t|\uparrow, rather than max⁡1≤i≤Ksi\max_{1\le i\le K}s_i with K↑K\uparrow. The latter is the phenomenon directly demonstrated in the diagnostic, while the former is what the deployed pipeline actually experiences. A larger memory can certainly produce harder retrieved negatives even at fixed KK, but that is a different mechanism and is not directly isolated by the current experiment. I therefore think the key missing experiment is a fixed-KK, growing-gallery analysis: hold K=8K=8 constant, expose the same new-event queries to increasingly large historical memory prefixes, retrieve the top eight from each prefix, and measure whether the maximum score and false-match probability increase. Ideally this should again be performed on identical queries. Without this experiment, the title and central framing about “growing candidate pools” or growing event memory are somewhat stronger than what the diagnostic strictly establishes. I am also not fully convinced by the subsequent interpretation that the remaining drift after pool-relative normalization demonstrates “genuine semantic proximity” rather than a calibration issue. The paper tests one specific z-score-type normalization, ZK=(mK−μ−1)/σ−1Z_K=(m_K-\mu_{-1})/\sigma_{-1}, and observes that it removes only about 24% of the drift. However, failure of this normalization does not rule out richer calibration schemes for maxima, including conditional quantile calibration, empirical-CDF calibration, extreme-value calibration, or calibration conditioned jointly on gallery size and query statistics. Since maxima are order statistics, standardizing by the mean and variance of the remaining scores is not in general expected to make their tail probability invariant to pool size. Thus, the evidence supports the narrower conclusion that this particular normalization is insufficient, but not yet the stronger conclusion that the residual drift must primarily reflect semantic proximity and “cannot be fixed” by score calibration.

  2. The second major concern is that the headline system improvement conflates a stronger decision model, richer textual context, and listwise candidate conditioning. The baseline uses the official WEC pair scorer and maximum-score thresholding, whereas C2E-JUDGE uses a language-model judge operating on whole reports. The authors acknowledge this limitation, and the same-judge visibility sweep is a useful control. However, that sweep reveals an important distinction that should be reflected more clearly in the claims. With the same language-model judge, top-one already achieves 0.0% new-event false-merge risk, while top-eight obtains 0.4%. Thus, the dramatic reduction from the baseline risk of 16.8% to approximately zero is essentially achieved by replacing the original scorer/threshold decision with the language-model judge; it is not caused by exposing the judge to multiple candidates. Increasing candidate visibility primarily improves repeated-event coverage. Therefore, statements suggesting that whole-pool visibility itself “removes most false merges” risk conflating two separate interventions. The decomposition also differs substantially across clustering metrics. For B-cubed F1, top-one to top-eight contributes 13.1 points out of the 18.9-point baseline-to-top-eight gain, which is the reported 69%. However, for ARI, visibility contributes only about 19.345.8≈42%\frac{19.3}{45.8}\approx42\% of the full system-level improvement. A substantial fraction of the ARI gain therefore arises before listwise visibility is introduced. Reporting the 69% B-cubed decomposition without an analogous discussion of ARI gives a somewhat incomplete picture of what drives the improvement. A cleaner control would use the same language model and exactly the same whole-report inputs in both decision rules: evaluate every candidate independently with the LLM as a pairwise SAME/DIFFERENT judge and then aggregate those pairwise outputs using a maximum or calibrated rejection rule, versus presenting the same candidates jointly to C2E-JUDGE. This would directly test whether joint candidate conditioning, rather than simply using a substantially stronger semantic model, provides the gain. There is also an important issue with the operational metrics. The paper defines repeated-event coverage as P(SAME∣repeated)P(\mathrm{SAME}\mid\mathrm{repeated}) rather than the probability that the system attaches the document to the correct existing event. Consequently, a repeated-event document that is attached to the wrong historical event is still counted as “covered.” The cluster-level B-cubed and ARI metrics will eventually penalize such mistakes, so this issue does not invalidate the overall clustering results. However, it weakens the mechanistic interpretation that increasing visibility improves coverage at “near-zero false-merge risk.” The paper should separately report (1) correct-repeat attachment rate, (2) wrong-event attachment rate for repeated events, and (3) missed-repeat rate.

Questions for authors

AI Usage Disclosure

I used ChatGPT to assist with organizing and polishing the language of my review. The main technical concerns are summarized in Chinese below. These notes were reconstructed from my existing English review draft to document my assessment, rather than copied verbatim from an earlier prompt. I also include an illustrative polishing instruction. The AI assistance was used for language and presentation, and the submitted review reflects the arguments and concerns I endorse.

Chinese Review Notes (Reconstructed)

我觉得这篇文章关注的问题挺有意思的,就是历史事件越来越多的时候,新事件更容易被错误地当成以前出现过的事件。文章前面的诊断实验做得比较好,作者只用确定不是同一个事件的候选来比较,而且还用了相同的523个query做对照。结果确实说明候选数量增加以后,最高分和错误匹配概率都会增加。另外,作者固定同一个LLM,只改变它能看到的候选数量,这个实验也比较有价值。其他prompt、模型和数据集上的验证也算比较充分。

不过我有几个比较主要的疑问。

首先,诊断实验和实际系统里的情况好像不完全一样。诊断实验是把候选数量K从4增加到256,但实际系统一直只取top 8。也就是说,memory变大不代表最后参与判断的分数数量变多,而是检索出来的8个候选可能发生变化。这两种情况不能直接等同。我觉得应该补一个实验,固定K=8,只增加历史memory的大小,然后用相同的一批新事件query,看最高分和错误匹配率是否仍然上升。不然文章关于streaming memory增长的核心解释还不够直接。

另外,作者发现z-score normalization只能消除24%的drift,就认为剩下的主要是真实的语义相似性,而不是calibration问题,我觉得这个结论有点强。毕竟只测试了一种归一化方式,最大值的分布本身就会随着候选数量变化。其他基于分位数或者分布的校准方法可能还有作用,目前只能说明这种normalization不够有效。

第二个问题是,系统提升到底来自更强的LLM,还是来自同时看多个候选,感觉没有完全分开。Baseline和新方法改变的不只是候选数量,模型和输入内容也变了。尤其Table 2里面,LLM只看top 1的时候,错误合并率已经是0%,top 8则是0.4%。所以从baseline的16.8%降到接近0%,主要看起来是换成LLM带来的,而不是多候选联合判断带来的。增加候选更明显的作用应该是改善重复事件的识别。

作者还强调增加候选可见性贡献了69%的B-cubed F1提升,但如果看ARI,这个比例只有大约42%。我觉得两个指标应该一起讨论。

这里最好再做一个对照:固定相同的LLM、完整文章输入和候选集合,一种让LLM逐对判断再汇总结果,另一种让LLM一次看完所有候选再判断。这样才能更清楚地知道joint candidate conditioning本身到底有多少贡献。

最后,coverage的定义也有点问题。现在只要重复事件被判成SAME就算covered,但它可能匹配到了错误的历史事件。所以coverage增加不一定意味着真正找对的事件更多。我觉得应该分别报告正确匹配、错误匹配以及漏掉重复事件的比例。虽然B-cubed和ARI会惩罚错误聚类,但单独用coverage来解释机制还是不够准确。

总体来说,我觉得这篇文章的诊断实验和控制实验有价值,但希望作者进一步证明固定top 8时memory增长的影响,同时更明确地区分LLM本身和多候选联合判断的贡献,并把重复事件的正确匹配率报告清楚。

Illustrative Prompt for AI Language Polishing

请把上面的中文评审想法整理成正式、自然的英文ICLR review,分成Strengths、Weaknesses和Questions for Authors三个部分。

这些是我自己对论文的主要判断和疑问,中文比较口语化,请帮我翻译并适当整理逻辑,让表达更加清楚、专业和严谨。不要改变我的观点和批评力度,不要添加新的技术问题、实验数据或结论。保留所有关键数字、公式和实验设置,特别注意区分实验已经证明的现象和我认为还需要进一步验证的解释。语气保持客观、有建设性,使用自然的第一人称审稿表达,不要过度润色成模板化的AI风格。最后把主要疑问整理成三个具体的作者问题。

2 comments

We thank the reviewer for the positive assessment of our work and for the constructive comments, which have helped us improve the paper. We add the requested same-query, fixed-K, growing-memory experiment and decompose the system gain into three separately measurable factors.

W1: Fixed K=8 with growing memory. Fixing K=8 and growing the historical prefix from 25% to 100%, on the same 954 annotated new-event queries (pooled over queries common to every prefix), mean top-8 WEC pair score rises .0867 to .1113 and false-match risk rises .1279 to .1656. Paired differences are .0245 [95% CI .0172,.0319] and .0375 [.0168,.0587]; per-order score/risk increases are .0197/.0503, .0242/.0189, and .0300/.0440. With the budget fixed at eight, the top score still rises as memory grows, and a false match becomes more likely. So the deployed pipeline runs into the same problem by a different route: memory does not add points to the best score, it changes which eight documents produce it. The slots are refilled with different documents rather than with more of them, which is why the risk grows at constant budget (Appendix H.1).

W2: Scope of calibration. We add four controls that replace z-score normalization with another calibration family: pool-relative extremeness, PoolCal, PoolCal+, and a score-only control. All four bring the drift down a little, and under all four the drift still grows with memory. The leftover growth therefore belongs to this whole family of rescaling, not to the z-score we happened to test. What memory changes is which documents sit in the pool, not the scale of the scores. The gradient reported in the paper is measured under all four (Appendix H.8).

W3: Model, text, and listwise conditioning. Holding the LLM endpoint, full reports, BGE-M3 retriever, source exclusion, and candidate pool fixed, joint top-8 gives B3/ARI .7147/.6749, .7172/.6696, and .7073/.6667, while independent one-candidate judgments give .7810/.7939, .7618/.7564, and .7725/.7908. Same model, same reports, same candidates: judging one candidate at a time scores higher in all three orders. Most of the gain therefore comes from the semantic judge, and reading the whole list at once mainly adds coverage. Reported separately, 4.5 to 6.6 B3 points and 10 to 12 ARI points sit in the decision rule (Appendix H.2).

W4: Correct, wrong, and missed repeats. Joint obtains 62.93% correct, .06% wrong-target, and 37.00% missed; pairwise obtains 71.31%, .17%, and 28.52%. Wrong-target counts are 2/1/0 and 3/1/4 out of 1,571 documents. Of the 8.5 points that stop being counted as missed, 8.4 turn into correct links while wrong-target rises by .11 points. Showing the judge more candidates turns misses into correct links, not into wrong ones. (Appendix H.3)

Q1: Fixed K with growing memory. Yes. W1 reports it on identical queries with paired bootstrap intervals: Yes. With K fixed at eight and the same queries, the top score rises from .0867 to .1113 and false-match risk from .1279 to .1656, and both paired intervals stay above zero. (Appendix H.1)

Q2: Pairwise aggregation versus joint judgment. Yes, the pairwise controls above share the model and the inputs. Pairwise wins: .7810/.7939, .7618/.7564, .7725/.7908 against .7147/.6749, .7172/.6696, .7073/.6667. The gain comes from the judge, not from the joint decision rule. Pairwise consumes about twice the input tokens (7.14M versus 3.49M on order 17) and far more output tokens, so joint is the deployed default and both frozen prediction sets are released (Appendix H.2; token counts in H.7).

Q3: Correct versus merely covered repeats. The three outcomes above are mutually exclusive and reported per stream. Showing more candidates moves 8.4 of the 8.5 lost points from missed to correct, and attaching to the wrong event stays at or below .3% in all six joint/pairwise orders., and the second model reproduces the pattern at a mean incorrect-event attachment rate of .11% (Appendix H.3 and H.6).

Thank you again for your thoughtful comments. We would appreciate it if you could let us know whether our rebuttal has adequately addressed the concerns you raised. If anything remains unclear or you have any new questions, we would be very happy to continue the discussion.

Reviewer_pymSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies a system that reads news documents one at a time and must decide, for each one, whether it describes an event it has already seen or a new event. The decision cannot be undone later.

The usual approach is: find the most similar stored events, take the highest similarity score, and if it is above a threshold, say already seen. The authors show a problem with this. As the system stores more events, it compares each new document against more candidates, so the highest score tends to get higher just by chance. New events are therefore more and more often mistaken for old ones. In their test, this error grows from about 20% to 55% as the number of candidates grows from 4 to 256. Rescaling the score fixes only about a quarter of the problem.

Their solution gives the 8 best candidates to a language model, which either picks the matching event or answers new. This clearly improves clustering quality and almost eliminates wrong merges of new events.

Strengths

  1. As more past events are stored, new events are more often mistaken for old with false matches rise from about 20% to 55%. The test uses only pairs known to be different events, so every match is clearly an error. Simpler agjustments show they do not solve the problem, such as rescaling or adjusting the score and threshold, making the problem convincing and is a key reason for my positive score.

  2. The study shows that seeing more candidates steadily improves results by keeping the same model and setup and changing only how many candidates the judge sees. This explains 69% of the overall gain and it is the strongest evidence that looking at all the candidates together, rather than at one score.

Weaknesses 1.The baseline and the new method are different in how they decide, in what text they read such as a short passage vs. the full article and in the model they use such as a small trained scorer vs. a LLM. It is therefore not possible to tell which change causes the large gain. This is the main reason my score is borderline and not a clear accept.

  1. If most documents in the data are actually new events, a system that always answers new could also score well. The paper does not report how many documents are new vs. repeated, almost never linking a document to an old event, or how such a simple system would perform, so part of the gain may come from being cautious rather than from looking at the candidates.

Questions

  1. What share of documents are new vs. repeated events and what scores do an always new system and a stricter version of the baseline get?
  2. Can you compare the two systems when they read the same text?
  3. Can you show the error curve at threshold 0,18?
  4. Does the 24% figure hold with a better rescaling method?
  5. Does the effect also appear for documents early in the stream?
  6. How much time and money does the method cost compared with the baseline? WHat about a smaller and cheaper model?
  7. Can you give full results and not only differences, for the 2nd language mode?
Critical strengths
  1. The paper shows that as more past events are stored, new events are more often mistaken for old ones (false matches rise from about 20% to 55%). The test uses only pairs known to be different events, so every match is clearly an error. The authors also try simpler fixes, such as rescaling the score or adjusting the threshold, and show they do not solve the problem. This makes the problem convincing and is a key reason for my positive score.

  2. Keeping the same model and setup and changing only how many candidates the judge sees, the paper shows that seeing more candidates steadily improves results. This explains 69% of the overall gain. It is the strongest evidence that looking at all the candidates together, rather than at one score, is what matters.

Critical weaknesses

1.The baseline and the new method differ in how they decide, in what text they read (a short passage vs. the full article) and in the model they use (a small trained scorer vs. a large language model). It is therefore not possible to tell which change causes the large gain. This is the main reason my score is borderline and not a clear accept.

  1. The new method almost never links a document to an old event. If most documents in the data are actually new events, a system that always answers new could also score well. The paper does not report how many documents are new vs. repeated, or how such a simple system would perform, so part of the gain may come from being cautious rather than from looking at the candidates.
Questions for authors
  1. What share of documents are new vs. repeated events? What scores do an "always new" system and a stricter version of the baseline get?
  2. Can you compare the two systems when they read the same text?
  3. Please show the error curve at threshold 0.18.
  4. Does the 24% figure hold with a better rescaling method?
  5. Does the effect also appear for documents early in the stream?
  6. How much time and money does the method cost compared with the baseline? Have you tried a smaller, cheaper model?
  7. Please give full results (not only differences) for the second language model.
  8. Please add the missing references and confirm the status of Chen et al. (2026).
1 comment

We thank the reviewer for the positive assessment of our work and for the constructive comments, which have helped us improve the paper. We address each point with results obtained under the same protocol as the main experiments, including complete three-stream evidence for the second model requested.

W1: Source of gain. Same-judge top-1 gives B3/ARI .6020/.5143 and top-8 joint gives .7331/.7077 across complete orders 17/29/43 (.7147/.6749, .7172/.6696, .7073/.6667). With the same LLM, full reports, retriever, source exclusion, and top-8 candidates, independent pairwise per-order B3/ARI is .7810/.7939, .7618/.7564, and .7725/.7908. With model, text, retriever and pool all fixed, pairwise is higher in every order; with visibility fixed, the judge alone is worth 4.5 to 6.6 B3 points. The gain is a stronger judge plus wider evidence rather than a tangled change. (Appendix H.2)

W2: Conservative UNSEEN policy. 17.0% of the documents are new and 83.0% are repeats. Always-new gets B3 .2907 and ARI 0; joint C2E links 62.93% of the repeats correctly with .06% wrong-target. The gain comes from telling events apart, not from predicting the majority class. Pairwise reaches 71.31 / 0.17 / 28.52 on the same split (Appendix H.3).

Q1: Class balance and stricter baseline. Split 17.0/83.0. Always-new gets B3 .2907 and ARI 0. The frozen .18 baseline gets B3/ARI .5290/.0678 at risk .3458, which sends 29% of repeats to the wrong event. Its correct/wrong/missed split is .4087/.2905/.3009 (Appendix H.5).

Q2: Same-text comparison. C2E-Judge and the visibility controls read the same full reports through the same BGE-M3 retrieval and the same candidate IDs. The only differences are how many candidates are visible and how they are combined.

Q3: Threshold .18. Appendix H.5 of the revised supplementary material carries the complete .10--.30 curve and the class balance. At .18 the row is B3 .5290, ARI .0678, risk .3458: clustering is middling, and a third of new events are merged into an existing event.

Q4: Better rescaling. All four rescaling families bring the drift down, and all four leave it growing with memory. The 24% therefore describes this whole class of correction, not the z-score on its own. The controls are pool-relative extremeness, PoolCal, PoolCal+, and score-only (Appendix H.8).

Q5: Early stream. Yes. Inside the early segment the top score still rises from .0885 to .1138 and risk from .1318 to .1723. The drift is there from the first prefix; it does not need a long history to build up. Over the full prefix range the changes are .0867 to .1113 and .1279 to .1656 at K=8 (Appendix H.1).

Q6: Time, cost, and smaller models. The three full runs cost 5,673 calls and 10,443,394 input / 106,739 output tokens, against no LLM calls for the embedding baseline; on order 17 pairwise uses 7.14M against 3.49M input. Phi-3 on 100 documents gets B3 .8469, ARI -.0004 and no correct repeat. Luna completes all three 1,893-document streams at mean .8319/.8351. Costs are reported in tokens because provider pricing changed during the run (Appendix H.7).

Q7: Full second-model results. GPT-5.6-Luna uses the same retrieval, top-8, balanced prompt and joint decision: B3/ARI .8729/.8792, .8038/.7988, and .8191/.8272, mean .8319/.8351, with new-event risk 1.24/1.24/0%. False merges are 4/322, 4/322, and 0/322; correct/wrong-target/missed counts are 1332/1/238, 1179/2/390, and 1229/2/340; correct-repeat rates are 84.79/75.05/78.23% (Appendix H.6).

Q8: Related works. Verified: We cite Chen et al., ACL 2026, pages 29151--29168, DOI 10.18653/v1/2026.acl-long.1345 in related work for its temporal graph and ordered-time-segment design. The revision also adds the two lines our method extends: cohort score normalisation from speaker verification (Auckenthaler et al., 2000), which is the family that zKz_K belongs to, and new-event detection (Allan et al., 1998; Kumaran and Allan, 2004), which supplies the single-threshold decision we replace. Recent work is added where it bears on the mechanism: listwise judges lose accuracy as the candidate list grows and need cross-set calibration (Ren et al., WWW 2025), long contexts are read unevenly (Liu et al., TACL 2024), and incremental clustering now enforces an explicit memory budget with dual thresholds (Luo et al., Findings of ACL 2026).

Reviewer_a3kvOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies candidate‑pool false familiarity: in streaming event induction, as the historical candidate pool grows, maximum similarity scores over non‑coreferent candidates inflate, causing fixed‑threshold systems to wrongly merge new events into existing clusters. The authors demonstrate that score normalization can only eliminate ~24% of this inflation, indicating the problem is not purely a calibration issue. They propose C2E‑JUDGE: reuse the original retriever and candidate budget, feed the complete retrieved candidate set to an LLM judge, and let the judge select a matching candidate or output UNSEEN for new events. Experiments are conducted on WEC and a short prefix subset of ECB+.

Critical strengths
  1. The observed candidate‑pool false familiarity is a real pitfall for streaming event coreference systems relying on max‑score plus fixed‑threshold logic. The non‑coreferent diagnostic setup is carefully designed to measure score inflation.
  2. The controlled dose‑response ablation that varies visible candidate count keeps retrieval, memory and prompt fixed, which offers suggestive evidence for the value of multi‑candidate joint reading.
  3. The paper clearly defines online streaming protocol, avoids label leakage, and explicitly states its own limitations.
Critical weaknesses
  1. The primary baseline is a simple fixed‑threshold embedding matching system. The paper fails to compare against existing mature streaming event coreference systems with adaptive thresholds, online clustering and streaming memory management. Even the pool‑aware threshold baseline the authors implemented yields marginal improvement, but there is no in‑depth discussion on conformal prediction and adaptive threshold approaches that are highly relevant to this streaming feedback scenario. Without these comparisons, the practical severity and novelty of the studied problem remain unclear.
  2. The large performance leap of C2E‑JUDGE mixes two factors: (1) joint reading of multiple candidates; (2) stronger semantic reasoning capability of LLM compared with embedding‑based pairwise scoring. Although the K‑visibility ablation shows performance grows as visible candidates increase, it cannot fully separate these two factors. At present, we cannot rule out that most improvement comes from LLM’s stronger pairwise matching rather than the proposed multi‑candidate decision mechanism. 3.Experiments mainly rely on WEC dataset. ECB+ result only uses short document prefixes and serves merely as a superficial transfer check. There is no evaluation on other real‑world streaming event benchmarks. Results are obtained under one specific retriever, one main LLM endpoint. While the authors run one alternative prompt and one secondary judge, robustness across different retrievers, model sizes and prompt perturbations remains insufficiently validated. The generalizability of the conclusion is not well established.
Questions for authors
  1. Could you report results against at least two published streaming cross‑document event coreference baselines? How much residual false‑merge risk still exists in those existing systems?
  2. Please add a control: use the same LLM judge, but only feed one candidate at a time. How much performance drops compared with joint multi‑candidate reading? This will help us disentangle gains from multi‑candidate joint inspection vs pure LLM pairwise superiority.
  3. Do you have preliminary explorations on smaller open‑source models or distillation to lower token cost for real‑time streaming?
2 comments

We thank the reviewer for the positive assessment of our work and for the constructive comments, which have helped us improve the paper. We add two published baselines on complete streams, a same-model aggregation control, a complete second-provider judge, and repeat-level error decomposition.

W1: Published streaming baselines and adaptive methods. On complete WEC orders 17/29/43 the official WEC pair scorer, with its threshold chosen on the order-17 development split (1,250 documents, disjoint from every test document) and frozen for test, obtains B3/ARI .5506/.2000, .5492/.2703, and .5326/.2800, with new-event false-merge risk 18.94%, 16.77%, and 14.60%. The published xCoRe checkpoint, run in causal mode with gold event spans and max length 512, obtains B3/ARI .2402/.0015 and risk 71.74% on complete order 17, with correct/wrong-target/missed repeat rates 5.16%/71.16%/23.68%. On the same streams, the WEC scorer still merges 15 to 19 points of new events into an existing event, xCoRe merges 72, and C2E-Judge merges 0.4. Most new events in existing systems land in the wrong cluster, so the hard part is keeping identity over a long history, not scoring a single pair. xCoRe's split shows the shape of that failure: it attaches 76% of its repeats and 93% of those attachments point at the wrong event. We also report pool-aware threshold, PoolCal, PoolCal+, strict-threshold, and always-new controls; each isolates one rescaling choice, and the score and risk gradient with memory survives all of them (Appendix H.4; controls in H.8).

W2: Joint reading versus a stronger LLM. Fixing the model, full reports, retriever, and top-8 pool, same-judge top-1 gives B3/ARI .6020/.5143 and top-8 joint gives .7331/.7077 across complete orders 17/29/43 (.7147/.6749, .7172/.6696, .7073/.6667). Independent one-candidate pairwise judgments give .7810/.7939, .7618/.7564, and .7725/.7908. With the model, reports, retriever and pool all fixed, pairwise is higher in every order. Most of the jump therefore comes from the stronger judge, and the wider view mainly raises coverage (50.4% to 65.6%). (Appendix H.2)

W3: Generalization. A stricter prompt and the immutable-witness variant preserve the direction. A second provider gives .8729/.8792, .8038/.7988, and .8191/.8272 on the three complete streams at mean risk 0.83%, so the effect holds across endpoints, prompts and model families. On ECB+ B3/ARI rises from .6861/.1570 to .8199/.3936 over three 500-document prefixes (Appendix H.6 and F).

Q1: At least two published baselines and residual risk. WEC official scorer: three complete orders, risk 18.94/16.77/14.60%. xCoRe: complete order 17, risk 71.74%, repeat split 5.16/71.16/23.68. Two published systems, complete streams: they still merge 15 to 19 points and 72 points of new events, where C2E-Judge merges 0.4. An earlier order-17 500-document deepseek-v4-pro pilot gives B3/ARI .8460/.7079 (Appendix H.4).

Q2: Same LLM, one candidate at a time. Yes, with the same full reports and top-8 pool. One candidate at a time gains 4.5 to 6.6 B3 points on identical reports and candidates. The joint rule is the weaker half of the system, and the pairwise semantic judgment carries the gain. Pairwise consumes about twice the input tokens (7.14M versus 3.49M on order 17) and far more output tokens, so joint is the deployed default and both frozen prediction sets are released (Appendix H.2; token counts in H.7).

Q3: Smaller models and cost. The three full primary-judge runs use 5,673 calls and 10,443,394 input / 106,739 output tokens. Phi-3 on 100 documents reaches B3 .8469 but ARI -.0004 with no correct repeat, while Luna reaches .8319/.8351 at 0.11% wrong-target. What separates events is how well the judge reasons, not model size on its own. (Appendix H.7)

Thank you again for your thoughtful comments. We would appreciate it if you could let us know whether our rebuttal has adequately addressed the concerns you raised. If anything remains unclear or you have any new questions, we would be very happy to continue the discussion.

Comments on the paper

1 comment

We thank the reviewers for all comments. Additional experiments conducted in response to the reviews are collected in Appendix H of the revised supplementary material, following the original appendices A through G. We address each reviewer's questions individually in the replies below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.