- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces PolyEvent, a large-scale benchmark for cold-start relational event forecasting built from Polymarket’s on-chain CLOB data. The authors aggregate roughly 1B order-book fills into ~280M (market, block) events spanning ~1.5M markets and ~650K platform-defined event groups, and define two tasks: discrete block-gap prediction and three-way next-event price-mark prediction (UP/FLAT/DOWN). The main relational setting asks whether a target market’s prediction improves when strictly earlier histories of sibling markets from the same event group are provided, with whole event groups held out from training. The main methodological contribution is the evaluation protocol: block-based event construction, group-disjoint cold-start splitting, strictly earlier information sets, paired comparisons on identical target rows, and reporting under both event-micro and market-macro weighting. The authors also provide extensive construction and leakage audits. The primary empirical result is that sibling context improves next-mark prediction on a sealed whole-group holdout. Additional development analyses show that the effect survives an XGBoost implementation and that relative-threshold role features can further help on a subset of threshold-based event groups.
Critical strengths
- A large, well-documented resource for an under-benchmarked domain. PolyEvent links official event groups, market metadata, block-level event sequences, split manifests, and reference baselines. Prediction markets offer a natural relational structure that is not directly covered by many existing event-sequence benchmarks, making the resource potentially valuable to the community.
- Exceptionally careful benchmark construction and leakage auditing. The paper pays close attention to what is actually being evaluated. Aggregating events by (market, block) avoids the concentration of zero-time transitions associated with raw fills, while group-disjoint splits prevent contamination through shared event groups. The authors also audit timestamp ordering, contract-generation changes, calendar splitting, sibling contamination, and the effects of alternative evaluation weightings.
Critical weaknesses
- The baseline set is limited, the relational model description is terse, and NLL is only loosely translated into practical forecasting terms. The empirical comparison centers on GRU, Transformer, XGBoost, and a simple mean/max sibling-pooling model, with no established neural TPP or temporal-graph/relational baseline. The sibling encoder, aggregation and Transformer are also described only briefly, which makes it hard to understand the concrete architecture. NLL differences are translated only into rough geometric-mean probability changes, not into accuracy, calibration, or decision-relevant forecasting metrics. It is therefore unclear whether the benchmark separates modern relational methods or whether the reported gains are practically meaningful.
- The relational gain is not sufficiently isolated from the choice of target-only baseline and from context-distribution confounds. The sealed experiment shows that sibling context improves NLL relative to the selected target-only GRU, but that baseline omits the target’s absolute price and elapsed blocks since its previous trade. In the post-hoc XGBoost experiments, adding these inputs substantially reduces NLL, and sibling context still helps but by less; the analogous price/time-augmented neural comparison is not reported on the sealed split. Moreover, the different-event context placebo is evaluated only on development data and does not adequately match temporal position, activity density, or recency. Consequently, the sealed result establishes a predictive benefit relative to the chosen baseline, but does not convincingly establish how much of the improvement is specific to same-group context rather than to missing own-history information or differences between context distributions.
Questions for authors
- Could the authors include or justify omitting at least one recent, task-compatible relational event or temporal-graph baseline? Would an adaptation of existing event-sequence methods, such as a EBES or HoTPP-style approach, be applicable here?
- Do the authors have results for a target-only neural baseline augmented with the last price and elapsed blocks, or a better-matched different-group context control?
- Could the authors report a metric beyond NLL (e.g., accuracy or calibration) that makes the gain easier to interpret?