acceptodds
Under review as a conference paper at ICLR 2027

PolyEvent: A Large-Scale Benchmark for Inductive Relational Event Forecasting

Abstract

How should we evaluate forecasters on unseen event groups whose markets have trading histories? Reviewed benchmarks do not report this comparison. We introduce PolyEvent, an open event-sequence benchmark using Polymarket's official event groups. It aggregates 1.03B order-book fills into 281M (market, block) events across 652,016 groups; experiments use a fixed 10M-event subset. Tasks are binned block-gap prediction and three-way price-change prediction (UP, FLAT, DOWN); the relational track evaluates the latter. Separately trained own-history and context-augmented models are frozen and compared on identical targets from held-out multi-market groups, using strictly earlier-block information. Scores average over events (event-micro) or markets (market-macro). Construction choices—fill-level events, timestamp order, order-direction marks and calendar splitting—each change what is evaluated. As a use case, context lowers negative log-likelihood on a sealed whole-group holdout (primary market-macro ΔNLL = −0.0490 nats), exceeding the 0.010-nat margin fixed before any sealed prediction. On development data, an audited placebo using other groups with matched sibling count yields ΔNLL = −0.0018 nats, its interval containing zero; it does not isolate event-group identity. In a post-hoc study of 156 development threshold groups, features encoding threshold order improve over pooling and a capacity-matched shuffled-role control (market-macro ΔNLL = −0.0250 nats against the control, simultaneous 95% CI [−0.0377,−0.0122]); event-micro gains are smaller. Data and code links will follow publication.

Primary area:
datasets and benchmarks
Keywords:
benchmark, event sequence modeling, relational learning, temporal point processes, prediction markets

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NW6vOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PolyEvent, a large-scale benchmark for cold-start relational event forecasting built from Polymarket’s on-chain CLOB data. The authors aggregate roughly 1B order-book fills into ~280M (market, block) events spanning ~1.5M markets and ~650K platform-defined event groups, and define two tasks: discrete block-gap prediction and three-way next-event price-mark prediction (UP/FLAT/DOWN). The main relational setting asks whether a target market’s prediction improves when strictly earlier histories of sibling markets from the same event group are provided, with whole event groups held out from training. The main methodological contribution is the evaluation protocol: block-based event construction, group-disjoint cold-start splitting, strictly earlier information sets, paired comparisons on identical target rows, and reporting under both event-micro and market-macro weighting. The authors also provide extensive construction and leakage audits. The primary empirical result is that sibling context improves next-mark prediction on a sealed whole-group holdout. Additional development analyses show that the effect survives an XGBoost implementation and that relative-threshold role features can further help on a subset of threshold-based event groups.

Critical strengths
  1. A large, well-documented resource for an under-benchmarked domain. PolyEvent links official event groups, market metadata, block-level event sequences, split manifests, and reference baselines. Prediction markets offer a natural relational structure that is not directly covered by many existing event-sequence benchmarks, making the resource potentially valuable to the community.
  2. Exceptionally careful benchmark construction and leakage auditing. The paper pays close attention to what is actually being evaluated. Aggregating events by (market, block) avoids the concentration of zero-time transitions associated with raw fills, while group-disjoint splits prevent contamination through shared event groups. The authors also audit timestamp ordering, contract-generation changes, calendar splitting, sibling contamination, and the effects of alternative evaluation weightings.
Critical weaknesses
  1. The baseline set is limited, the relational model description is terse, and NLL is only loosely translated into practical forecasting terms. The empirical comparison centers on GRU, Transformer, XGBoost, and a simple mean/max sibling-pooling model, with no established neural TPP or temporal-graph/relational baseline. The sibling encoder, aggregation and Transformer are also described only briefly, which makes it hard to understand the concrete architecture. NLL differences are translated only into rough geometric-mean probability changes, not into accuracy, calibration, or decision-relevant forecasting metrics. It is therefore unclear whether the benchmark separates modern relational methods or whether the reported gains are practically meaningful.
  2. The relational gain is not sufficiently isolated from the choice of target-only baseline and from context-distribution confounds. The sealed experiment shows that sibling context improves NLL relative to the selected target-only GRU, but that baseline omits the target’s absolute price and elapsed blocks since its previous trade. In the post-hoc XGBoost experiments, adding these inputs substantially reduces NLL, and sibling context still helps but by less; the analogous price/time-augmented neural comparison is not reported on the sealed split. Moreover, the different-event context placebo is evaluated only on development data and does not adequately match temporal position, activity density, or recency. Consequently, the sealed result establishes a predictive benefit relative to the chosen baseline, but does not convincingly establish how much of the improvement is specific to same-group context rather than to missing own-history information or differences between context distributions.
Questions for authors
  1. Could the authors include or justify omitting at least one recent, task-compatible relational event or temporal-graph baseline? Would an adaptation of existing event-sequence methods, such as a EBES or HoTPP-style approach, be applicable here?
  2. Do the authors have results for a target-only neural baseline augmented with the last price and elapsed blocks, or a better-matched different-group context control?
  3. Could the authors report a metric beyond NLL (e.g., accuracy or calibration) that makes the gain easier to interpret?
Reviewer_GuhUOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces a large-scale benchmark targeting next price-change prediction on previously unseen groups. Incorporating historical context from related markets within the same group reduces the negative log-likelihood on a held-out test set, while a placebo experiment using unrelated-group context shows no detectable improvement.

Critical strengths
  1. The benchmark is large-scale, with a whole-group holdout and a strictly causal information set.
  2. The paper provides useful audits of how data-construction choices change the evaluation.
Critical weaknesses
  1. The main finding cannot be attributed to event-group identity. The placebo does not control for timing or activity, so the result reduces to the effect that related assets are predictive.
  2. The baselines are underpowered. The main models do not use price-level information, which is the most basic signal for predicting price changes. Table 2 shows that adding price information improves prediction far more than sibling context does. The reported context gain may therefore shrink with stronger baselines.
  3. The readability is poor. Nearly every sentence is hedged, which makes the main contribution hard to extract. Figure captions also repeatedly disclaim the figures themselves.
Questions for authors
  1. Does the context gain still hold in neural models that include price inputs?
  2. Could the authors provide a concise task definition and a primary metric for submitters?
Reviewer_EaFoOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PolyEvent, a large benchmark for relational event forecasting using Polymarket transaction data. It aggregates approximately 1.03 billion fills into 281 million market-block events from 1.54 million markets and 652,016 official event groups. The benchmark includes block-gap prediction and three-class next-price-change prediction, with the relational track asking whether earlier histories of sibling markets improve prediction for event groups absent from training. The reference context model encodes the target history and pooled sibling histories. On a separately constructed group holdout, context reduces market-macro NLL by 0.0490 nats and event-micro NLL by 0.0132 nats relative to the target-only model. The paper also provides extensive analyses of event construction, split design, evaluation weighting, context controls, and threshold-order features. The main contribution is the dataset and evaluation protocol rather than the forecasting architecture.

Critical strengths
  1. Large and carefully constructed benchmark. PolyEvent provides a large transaction-level resource with official event-group structure, group-disjoint splits, and strictly historical sibling context. The construction audits also show that event aggregation, timestamps, split design, and evaluation weighting materially affect the resulting task.
  2. Careful and transparent evaluation. The main comparison uses paired predictions, multiple training seeds, group-clustered uncertainty estimates, and a separately constructed holdout evaluated with frozen models. The authors also report useful controls, negative results, implementation corrections, and limitations. This makes the narrow predictive result more credible.
Critical weaknesses
  1. The benchmark target has limited practical and scientific interpretation. The task predicts the sign of the next recorded trade-price change rather than event outcomes, calibrated probabilities, economically meaningful movements, or trading performance. These short-horizon labels may also contain market-microstructure effects such as execution order and bid-ask bounce. More evidence is needed that the benchmark measures useful forecasting rather than properties of recorded trades.
  2. The main comparison does not cleanly isolate the value of relational information. The target-only GRU omits absolute price, current elapsed time, volume, and order-book information, while the context model receives additional activity and recency features and has greater capacity. The different-event control is useful, but its matching does not adequately control time, activity, recency, and history length, and it is not evaluated on the sealed split. The post-hoc XGBoost experiment partly addresses this concern, but a comparably strong neural target-only baseline on the sealed split is missing.
  3. The generalization claim is limited to unseen groups under observed market regimes. Validation and test overlap in calendar time, and the sealed split has no future-period restriction. Therefore, the results do not establish generalization to future market periods. In addition, the context improvement is much larger under market-macro evaluation and is concentrated among markets with short histories. An event-group-macro result and sealed stratified results would help clarify the scope of the finding.
Questions for authors
  1. Can the authors evaluate a stronger target-only/context pair on the sealed split in which both models receive the target’s absolute price and current elapsed time? If possible, can volume or order-book information also be included? This would help determine whether sibling history remains useful relative to a stronger target-only model.
  2. Can the authors report a capacity-matched different-event context control on the sealed split, with stronger matching on absolute time, activity, recency, and history length? If this is not possible, what evidence supports interpreting the improvement as relational rather than as additional market-activity information?
  3. How sensitive are the results to the definition of the price-change label? For example, do the conclusions remain under a minimum price-change threshold, a volume-weighted price, or another construction less sensitive to individual trade execution and bid-ask bounce?
  4. Can the authors provide an evaluation on a strictly future calendar period, without validation/test temporal overlap? If not, the paper should consistently limit its generalization claim to unseen event groups under the observed regimes.
  5. Can the authors report event-group-macro results and sealed results stratified by target-history length and group structure? This would clarify whether the improvement is broadly distributed or mainly comes from sparse markets and groups with many markets.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.