- Rating
- 1(−1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies streaming event induction, where each incoming event-bearing document must either be assigned to an event already represented in memory or start a new event. The paper focuses on a common decision rule that retrieves historical candidates, takes the maximum compatibility score, and compares it against a fixed threshold. The authors identify what they call candidate-pool false familiarity: as the candidate pool grows, the maximum similarity among non-coreferent candidates tends to increase, making genuinely new events increasingly likely to appear familiar. The paper first diagnoses this phenomenon using first mentions of gold events and historical candidates that are explicitly annotated as non-coreferent. It studies the maximum score over candidate pools of different sizes and shows that both the mean maximum and the fixed-threshold false-match rate increase substantially with . The authors further introduce a pool-relative extremeness score to test whether this effect is merely a score-calibration artifact and report that this normalization removes only part of the observed drift. Motivated by this diagnosis, the paper proposes C2E-JUDGE, which retains the same source-excluded BGE-M3 retrieval and bounded candidate set but replaces maximum-score thresholding with a language-model judge that jointly inspects the retrieved candidate reports and outputs either a candidate identifier or UNSEEN. On WEC, the proposed system improves B-cubed F1, ARI, new-event false-merge risk, and repeated-event coverage relative to the WEC-score baseline. The paper further performs a same-judge visibility sweep over different numbers of visible candidates, evaluates a stricter prompt and a second judge endpoint, and reports a transfer experiment on a binary ECB+ projection. Overall, the paper provides a useful empirical diagnosis of maximum-based decisions in streaming event memory and shows that candidate-conditioned language-model judgment can substantially improve the evaluated streaming clustering system.
Critical strengths
- The strongest part of the paper is the diagnostic experiment. The authors restrict the analysis to first mentions of gold events and historical candidates annotated as belonging to different gold events. This makes every comparison in the diagnostic explicitly non-coreferent. Moreover, they provide a paired analysis on the same 523 queries, rather than relying only on different valid-query subsets at different candidate counts. On these queries, the mean maximum increases substantially as the candidate pool grows, and the associated fixed-threshold false-match probability also increases. This provides convincing empirical evidence that a maximum-based decision statistic can become increasingly permissive as additional non-coreferent alternatives are introduced.
- The paper does more than compare a conventional scorer against an LLM judge. Importantly, it reruns the same streaming system using the same judge, prompt, retriever, and memory while varying only the number of visible candidates. This is a useful dose-response experiment and substantially strengthens the empirical claim that additional candidate evidence can improve repeated-event recovery. The paper also reports uncertainty intervals, a stricter prompt, a second judge endpoint, and a transfer check on ECB+.
Critical weaknesses
-
My main concern is a mismatch between the diagnostic mechanism and the operational system. The central narrative is that, as streaming memory grows, a new event becomes increasingly likely to look familiar because the maximum is taken over more candidates. However, the diagnostic establishes this by explicitly varying the number of candidates entering the maximum, , whereas the actual end-to-end system uses a fixed top-eight candidate budget. These are not equivalent interventions. With a fixed top-eight retriever, increasing the size of the historical memory does not increase the number of scores entering the final maximum; it instead changes which eight candidates survive retrieval. The relevant operational mechanism is therefore closer to with and , rather than with . The latter is the phenomenon directly demonstrated in the diagnostic, while the former is what the deployed pipeline actually experiences. A larger memory can certainly produce harder retrieved negatives even at fixed , but that is a different mechanism and is not directly isolated by the current experiment. I therefore think the key missing experiment is a fixed-, growing-gallery analysis: hold constant, expose the same new-event queries to increasingly large historical memory prefixes, retrieve the top eight from each prefix, and measure whether the maximum score and false-match probability increase. Ideally this should again be performed on identical queries. Without this experiment, the title and central framing about “growing candidate pools” or growing event memory are somewhat stronger than what the diagnostic strictly establishes. I am also not fully convinced by the subsequent interpretation that the remaining drift after pool-relative normalization demonstrates “genuine semantic proximity” rather than a calibration issue. The paper tests one specific z-score-type normalization, , and observes that it removes only about 24% of the drift. However, failure of this normalization does not rule out richer calibration schemes for maxima, including conditional quantile calibration, empirical-CDF calibration, extreme-value calibration, or calibration conditioned jointly on gallery size and query statistics. Since maxima are order statistics, standardizing by the mean and variance of the remaining scores is not in general expected to make their tail probability invariant to pool size. Thus, the evidence supports the narrower conclusion that this particular normalization is insufficient, but not yet the stronger conclusion that the residual drift must primarily reflect semantic proximity and “cannot be fixed” by score calibration.
-
The second major concern is that the headline system improvement conflates a stronger decision model, richer textual context, and listwise candidate conditioning. The baseline uses the official WEC pair scorer and maximum-score thresholding, whereas C2E-JUDGE uses a language-model judge operating on whole reports. The authors acknowledge this limitation, and the same-judge visibility sweep is a useful control. However, that sweep reveals an important distinction that should be reflected more clearly in the claims. With the same language-model judge, top-one already achieves 0.0% new-event false-merge risk, while top-eight obtains 0.4%. Thus, the dramatic reduction from the baseline risk of 16.8% to approximately zero is essentially achieved by replacing the original scorer/threshold decision with the language-model judge; it is not caused by exposing the judge to multiple candidates. Increasing candidate visibility primarily improves repeated-event coverage. Therefore, statements suggesting that whole-pool visibility itself “removes most false merges” risk conflating two separate interventions. The decomposition also differs substantially across clustering metrics. For B-cubed F1, top-one to top-eight contributes 13.1 points out of the 18.9-point baseline-to-top-eight gain, which is the reported 69%. However, for ARI, visibility contributes only about of the full system-level improvement. A substantial fraction of the ARI gain therefore arises before listwise visibility is introduced. Reporting the 69% B-cubed decomposition without an analogous discussion of ARI gives a somewhat incomplete picture of what drives the improvement. A cleaner control would use the same language model and exactly the same whole-report inputs in both decision rules: evaluate every candidate independently with the LLM as a pairwise SAME/DIFFERENT judge and then aggregate those pairwise outputs using a maximum or calibrated rejection rule, versus presenting the same candidates jointly to C2E-JUDGE. This would directly test whether joint candidate conditioning, rather than simply using a substantially stronger semantic model, provides the gain. There is also an important issue with the operational metrics. The paper defines repeated-event coverage as rather than the probability that the system attaches the document to the correct existing event. Consequently, a repeated-event document that is attached to the wrong historical event is still counted as “covered.” The cluster-level B-cubed and ARI metrics will eventually penalize such mistakes, so this issue does not invalidate the overall clustering results. However, it weakens the mechanistic interpretation that increasing visibility improves coverage at “near-zero false-merge risk.” The paper should separately report (1) correct-repeat attachment rate, (2) wrong-event attachment rate for repeated events, and (3) missed-repeat rate.
Questions for authors
AI Usage Disclosure
I used ChatGPT to assist with organizing and polishing the language of my review. The main technical concerns are summarized in Chinese below. These notes were reconstructed from my existing English review draft to document my assessment, rather than copied verbatim from an earlier prompt. I also include an illustrative polishing instruction. The AI assistance was used for language and presentation, and the submitted review reflects the arguments and concerns I endorse.
Chinese Review Notes (Reconstructed)
我觉得这篇文章关注的问题挺有意思的,就是历史事件越来越多的时候,新事件更容易被错误地当成以前出现过的事件。文章前面的诊断实验做得比较好,作者只用确定不是同一个事件的候选来比较,而且还用了相同的523个query做对照。结果确实说明候选数量增加以后,最高分和错误匹配概率都会增加。另外,作者固定同一个LLM,只改变它能看到的候选数量,这个实验也比较有价值。其他prompt、模型和数据集上的验证也算比较充分。
不过我有几个比较主要的疑问。
首先,诊断实验和实际系统里的情况好像不完全一样。诊断实验是把候选数量K从4增加到256,但实际系统一直只取top 8。也就是说,memory变大不代表最后参与判断的分数数量变多,而是检索出来的8个候选可能发生变化。这两种情况不能直接等同。我觉得应该补一个实验,固定K=8,只增加历史memory的大小,然后用相同的一批新事件query,看最高分和错误匹配率是否仍然上升。不然文章关于streaming memory增长的核心解释还不够直接。
另外,作者发现z-score normalization只能消除24%的drift,就认为剩下的主要是真实的语义相似性,而不是calibration问题,我觉得这个结论有点强。毕竟只测试了一种归一化方式,最大值的分布本身就会随着候选数量变化。其他基于分位数或者分布的校准方法可能还有作用,目前只能说明这种normalization不够有效。
第二个问题是,系统提升到底来自更强的LLM,还是来自同时看多个候选,感觉没有完全分开。Baseline和新方法改变的不只是候选数量,模型和输入内容也变了。尤其Table 2里面,LLM只看top 1的时候,错误合并率已经是0%,top 8则是0.4%。所以从baseline的16.8%降到接近0%,主要看起来是换成LLM带来的,而不是多候选联合判断带来的。增加候选更明显的作用应该是改善重复事件的识别。
作者还强调增加候选可见性贡献了69%的B-cubed F1提升,但如果看ARI,这个比例只有大约42%。我觉得两个指标应该一起讨论。
这里最好再做一个对照:固定相同的LLM、完整文章输入和候选集合,一种让LLM逐对判断再汇总结果,另一种让LLM一次看完所有候选再判断。这样才能更清楚地知道joint candidate conditioning本身到底有多少贡献。
最后,coverage的定义也有点问题。现在只要重复事件被判成SAME就算covered,但它可能匹配到了错误的历史事件。所以coverage增加不一定意味着真正找对的事件更多。我觉得应该分别报告正确匹配、错误匹配以及漏掉重复事件的比例。虽然B-cubed和ARI会惩罚错误聚类,但单独用coverage来解释机制还是不够准确。
总体来说,我觉得这篇文章的诊断实验和控制实验有价值,但希望作者进一步证明固定top 8时memory增长的影响,同时更明确地区分LLM本身和多候选联合判断的贡献,并把重复事件的正确匹配率报告清楚。
Illustrative Prompt for AI Language Polishing
请把上面的中文评审想法整理成正式、自然的英文ICLR review,分成Strengths、Weaknesses和Questions for Authors三个部分。
这些是我自己对论文的主要判断和疑问,中文比较口语化,请帮我翻译并适当整理逻辑,让表达更加清楚、专业和严谨。不要改变我的观点和批评力度,不要添加新的技术问题、实验数据或结论。保留所有关键数字、公式和实验设置,特别注意区分实验已经证明的现象和我认为还需要进一步验证的解释。语气保持客观、有建设性,使用自然的第一人称审稿表达,不要过度润色成模板化的AI风格。最后把主要疑问整理成三个具体的作者问题。
2 comments
Thank you again for your thoughtful comments. We would appreciate it if you could let us know whether our rebuttal has adequately addressed the concerns you raised. If anything remains unclear or you have any new questions, we would be very happy to continue the discussion.
We thank the reviewer for the positive assessment of our work and for the constructive comments, which have helped us improve the paper. We add the requested same-query, fixed-K, growing-memory experiment and decompose the system gain into three separately measurable factors.
W1: Fixed K=8 with growing memory. Fixing K=8 and growing the historical prefix from 25% to 100%, on the same 954 annotated new-event queries (pooled over queries common to every prefix), mean top-8 WEC pair score rises .0867 to .1113 and false-match risk rises .1279 to .1656. Paired differences are .0245 [95% CI .0172,.0319] and .0375 [.0168,.0587]; per-order score/risk increases are .0197/.0503, .0242/.0189, and .0300/.0440. With the budget fixed at eight, the top score still rises as memory grows, and a false match becomes more likely. So the deployed pipeline runs into the same problem by a different route: memory does not add points to the best score, it changes which eight documents produce it. The slots are refilled with different documents rather than with more of them, which is why the risk grows at constant budget (Appendix H.1).
W2: Scope of calibration. We add four controls that replace z-score normalization with another calibration family: pool-relative extremeness, PoolCal, PoolCal+, and a score-only control. All four bring the drift down a little, and under all four the drift still grows with memory. The leftover growth therefore belongs to this whole family of rescaling, not to the z-score we happened to test. What memory changes is which documents sit in the pool, not the scale of the scores. The gradient reported in the paper is measured under all four (Appendix H.8).
W3: Model, text, and listwise conditioning. Holding the LLM endpoint, full reports, BGE-M3 retriever, source exclusion, and candidate pool fixed, joint top-8 gives B3/ARI .7147/.6749, .7172/.6696, and .7073/.6667, while independent one-candidate judgments give .7810/.7939, .7618/.7564, and .7725/.7908. Same model, same reports, same candidates: judging one candidate at a time scores higher in all three orders. Most of the gain therefore comes from the semantic judge, and reading the whole list at once mainly adds coverage. Reported separately, 4.5 to 6.6 B3 points and 10 to 12 ARI points sit in the decision rule (Appendix H.2).
W4: Correct, wrong, and missed repeats. Joint obtains 62.93% correct, .06% wrong-target, and 37.00% missed; pairwise obtains 71.31%, .17%, and 28.52%. Wrong-target counts are 2/1/0 and 3/1/4 out of 1,571 documents. Of the 8.5 points that stop being counted as missed, 8.4 turn into correct links while wrong-target rises by .11 points. Showing the judge more candidates turns misses into correct links, not into wrong ones. (Appendix H.3)
Q1: Fixed K with growing memory. Yes. W1 reports it on identical queries with paired bootstrap intervals: Yes. With K fixed at eight and the same queries, the top score rises from .0867 to .1113 and false-match risk from .1279 to .1656, and both paired intervals stay above zero. (Appendix H.1)
Q2: Pairwise aggregation versus joint judgment. Yes, the pairwise controls above share the model and the inputs. Pairwise wins: .7810/.7939, .7618/.7564, .7725/.7908 against .7147/.6749, .7172/.6696, .7073/.6667. The gain comes from the judge, not from the joint decision rule. Pairwise consumes about twice the input tokens (7.14M versus 3.49M on order 17) and far more output tokens, so joint is the deployed default and both frozen prediction sets are released (Appendix H.2; token counts in H.7).
Q3: Correct versus merely covered repeats. The three outcomes above are mutually exclusive and reported per stream. Showing more candidates moves 8.4 of the 8.5 lost points from missed to correct, and attaching to the wrong event stays at or below .3% in all six joint/pairwise orders., and the second model reproduces the pattern at a mean incorrect-event attachment rate of .11% (Appendix H.3 and H.6).