- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies dynamic hypergraphs made up of a timeline of records. The main concern the paper addresses is protocol leakage (information beyond the historical record that is unfairly provided to a model that inflates scores) and what state a model should keep. The stream likelihood is factorized where each task is condition of it and leakage is defined as conditional mutual information. The paper claims that under a specific model of group dynamics, a small set of counters, including pair-terms, is sufficient. The authors' proposal is the incidence state machine (ISM), which they evaluate on a new benchmark from Stack Exchange. dump-time information is found by the leakage audit to inflate scores. ISM is reported the best performer, compared to temporal graph networks.
Critical strengths
-
The audit is useful and has practical impact. It found that data dumps add features that provide after-the-fact information, like the final number of answers in a thread, which raises role-prediction macro-F1 by up to 0.19. This difference is larger than the difference between any two models. It also found that a smaller leak is detected when processing same day events in the order the file present.
-
The evaluation is rigorous. The benchmark was carefully built and evaluated with known leaks, stratified negatives, using multiple random seeds, careful treatment of time dependence, and honest disclosures in the appendix. These make the benchmark and leakage reports credible.
Critical weaknesses
Although I see potential in the paper's directions, my concern is that the paper as it stands has major flaws whose possible fixes would produce a substantially different paper. In light of this, I'd argue for rejection in its current form. My main concerns are:
-
By construction, the theory is true. Proposition 2 shows sufficiency only for a process constructed from the stored statistics, ie, persistent processes with exponential kernels, Markov roles and parameters known in advance. This matters because we estimate the parameters in practice, and the sufficiency result won't hold as stated. Fixing it with a new theorem that holds broadly would require a major rework of the paper. Reframing the claims would produce a narrower paper, different from the submission. The paper presents theory based on textbook standard definitions, such as results from conditional mutual/usable information, Pinsker's inequality. I believe the paper would be stronger if it demotes its core from theory to a benchmark scaffolded by information theory.
-
Performance claims don't match results. The abstract claims "best overall" and "outperforms" but the authors' own results in the appendix show a different picture. In D.1 on aligned update schedules, ISM is 014-.039 below TPNet in one direction and at most .008 ahead in the other. The authors conclusion "parity with TPNet on T-JOIN, not a margin." goes against the abstract.
-
A single data source. That matters, because the pair-terms the authors investigate only account for 0.003 MRR on joining and no more than 0.030 AT on leaving. A dataset where pair-terms matter is needed to support the theory. Novelty is also a concern. The paper should compare its proposal with prior work like DyNAM-i (not cited) and RHEM (cited without further comparison), which already model group events from relational statistics.
Questions for authors
The weaknesses discussed are larger than missing details. Addressing them would require a new theorem, a new platform, or a reframed paper. So, I don't have questions about the paper's details. I'm open to reconsider my assessment if I misunderstood any assumptions of Proposition 2 or the results in D.1.