- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents a continuously synchronized dataset suite for studying the full lifecycle of decentralized prediction markets, using Polymarket as the primary setting. The main contribution is a unified data infrastructure that links three heterogeneous layers—market metadata, fill-level trading records, and oracle/resolution events—through canonical identifiers, on-chain recovery, and incremental synchronization. The September 2026 snapshot contains approximately 3.29M market/protocol identities, 1.90B deduplicated OrderFilled events, and 21.03M oracle and resolution events, including more than 2.4M markets with both observed trading activity and verified final outcomes. 24811_Unlocking_the_Forecastin 24811_Unlocking_the_Forecastin The paper validates the utility of the dataset through several downstream studies. In NBA markets, pre-game market probabilities are shown to be broadly well calibrated, while additional trading features do not yield persistent out-of-sample alpha. 24811_Unlocking_the_Forecastin For CPI markets, the authors reconstruct continuous market-implied inflation expectations from bucket contracts and show that they provide a high-frequency complement to conventional nowcasts, although they are not uniformly more accurate. 24811_Unlocking_the_Forecastin The paper also introduces a supervised benchmark for predicting final market resolution from partial trading histories, comparing tabular, regular time-series, and irregular event-sequence models; notably, the latest market price remains the strongest probabilistic baseline in terms of Log Loss and Brier score. 24811_Unlocking_the_Forecastin Overall, the work is primarily a dataset and research-infrastructure contribution aimed at enabling reproducible studies of forecasting, market microstructure, information aggregation, and oracle dynamics.
Critical strengths
-
A genuinely valuable full-lifecycle prediction-market dataset at substantial scale.
The paper goes beyond collecting market prices or metadata by linking market definitions, fill-level trading activity, and oracle/resolution events under a common identity structure. The resulting snapshot contains roughly 3.29M market/protocol identities, 1.90B deduplicated fills, and 21.03M oracle-related events, with more than 2.4M identities having both trading activity and verified resolution evidence. This is consequential because it enables analyses that require connecting information formation, trading behavior, and realized outcomes, which are difficult to conduct reliably using existing fragmented data sources. -
The validation demonstrates practical research utility rather than merely dataset scale.
The downstream studies cover materially different use cases, including leakage-aware NBA probability calibration and trading evaluation, reconstruction of market-implied CPI expectations, and resolution prediction from partial trading histories. In particular, the evaluation uses point-in-time information and held-out temporal splits, and the paper reports negative results when more complex models or trading signals fail to outperform the market baseline. This strengthens my confidence that the dataset can support rigorous empirical research rather than serving only as a large descriptive archive.
Critical weaknesses
-
The cross-source linkage and data-quality validation are not yet sufficiently rigorous for a dataset whose main contribution depends on correctness of entity resolution.
The paper carefully describes how market metadata, outcome tokens, executions, and oracle events are linked using canonical identifiers and fallback matching rules, but it provides limited quantitative evaluation of linkage accuracy itself. In particular, there is no systematic estimate of precision/recall, large-scale manual audit, or breakdown of matching reliability across different protocol generations and fallback cases. This is consequential because even a small rate of incorrect market–trade–resolution associations could materially affect downstream calibration, forecasting, and behavioral analyses. Since the primary contribution is the dataset rather than a new modeling method, stronger evidence that the reconstructed lifecycle links are correct is important for acceptance. -
The downstream validation is relatively limited compared with the scale and breadth of the dataset, which weakens the evidence for its claimed general research utility.
Despite containing millions of market identities and billions of executions, the supervised learning benchmark uses only 120 resolved markets, with only 18 markets in the test split, while the CPI study covers only 13 target months. In addition, only about 73% of catalog identities contain both observed trading activity and verified final-resolution evidence. These experiments demonstrate that the dataset can support several interesting analyses, but they do not yet establish that the proposed infrastructure yields robust conclusions across market categories, time periods, and protocol variants. A substantially larger and more diverse benchmark evaluation would make the empirical case for the dataset considerably stronger.
Questions for authors
-
How was the accuracy of the cross-source entity resolution quantitatively validated?
Since the main contribution relies on correctly linking market metadata, outcome tokens, fill-level executions, and oracle/resolution events, could the authors report the proportion of links obtained through direct identifiers versus fallback matching, together with an estimate of linkage precision/recall or a manual audit on a representative sample? It would also be useful to know whether matching accuracy varies across protocol generations or market types. The answer would materially affect my confidence in the correctness of the reconstructed lifecycle dataset. -
Why is the supervised-learning benchmark restricted to only 120 resolved markets, despite the dataset containing millions of linked market lifecycles?
Could the authors clarify the filtering criteria that reduce the available data to this subset, and whether these 120 markets are representative of the broader dataset in terms of category, liquidity, duration, and time period? In particular, the test set contains only 18 markets, which leads to fairly wide confidence intervals. Understanding whether this small benchmark is due to intrinsic task constraints or implementation choices would materially affect my assessment of the empirical validation. -
How should the approximately 73% market-level coverage of both trading activity and verified final resolution be interpreted with respect to selection bias?
Could the authors provide a breakdown of the missing lifecycle components by time period, market category, protocol generation, and liquidity level? If markets with incomplete lifecycles are systematically older, less liquid, or structurally different from fully linked markets, this could affect the validity of downstream analyses and the claimed generality of the dataset.
Final Author Rebuttal Summary: Key Revisions, Linkage Reliability, Dataset Scale, and Practical Impact
Dear reviewer,
We thank the Area Chair and reviewers for their feedback. We summarize the concerns raised and our revisions below.
1. Dataset scale (Reviewer 45Aa: insufficient evidence of scale and utility).
Our continuously updated, full-lifecycle dataset contained 3.29M markets, 1.9B+ executions, and 21M+ oracle events in September 2026, and now exceeds 7M markets and 2.1B executions. This surpasses Marketlens (349M trades; https://marketlens.trade/data) and predmktdata (865M trades; https://predmktdata.com) in reported scale, although coverage differs. Crucially, we integrate market metadata, fill-level executions, and oracle-resolution histories across protocol generations since 2020. Public downloads: https://prediction-market-data-engine.vercel.app/
2. Linkage correctness (Ya4q Q1 and 45Aa Q1: accuracy and audit evidence; AgTX Q2: lifecycle reconstruction).
Our cross-source joins use contract-defined identifiers (
conditionId,clobTokenIds,questionId), following Polymarket's official APIs and smart contracts (https://gamma-api.polymarket.com/docs), rather than primarily heuristic matching. 1,901,194,698 of 1,901,194,769 OrderFilled records map to catalog identities; the remaining 71 concern 21 combinatorial conditions outside the snapshot catalog. Only 10,395 of 21,028,317 lifecycle events (0.0494%) require fallback matching. A manual audit of 1,000 fallback-linked markets found no incorrect matches in the sample. Section 2.1 and Appendix C document identifier examples, transaction-level reconciliation, and intermediate events. Similar identifier-based approaches appear in predmktdata (https://predmktdata.com/docs) and poly_data (https://github.com/warproxxx/poly_data).3. Benchmark scale and temporal validity (Ya4q Q2 and 45Aa Q4: limited sample; AgTX Q3: look-ahead leakage).
We expanded the prediction benchmark 1,000-fold, from 120 to 120,000 markets, adding metrics and 95% confidence intervals. MLP retains the highest AUROC at both scales (0.827 originally; 0.791, 95% CI [0.785, 0.796] when expanded). We explain differences in Log Loss/Brier between cohorts. The task is retrospective: final settlement time defines post-hoc stages and labels, not model inputs; features use only information available at simulated prediction times.
4. Lifecycle completeness and selection bias (Ya4q Q3 and 45Aa Q2: coverage and missingness; AgTX Q2: lifecycle definition).
2,402,991 markets (73.02%) have observed trading and verified final resolution, well above the below-50% coverage reported for related datasets (Qin & Yang, 2026; Akey et al., 2026), although definitions differ. We do not claim every market has a complete historical record. We added missingness statistics by year (2020–2023: 28.31%; 2024: 95.99%) and category (Crypto: 89.97%; Sports: 61.79%), discussed legacy AMM/FPMM mechanisms and snapshot cutoffs, and proposed stratified sensitivity analyses.
5. CPI methodology (AgTX Q1 and 45Aa Q3: inconsistent methods for Figure 5 and Tables 14–15).
We clarify two distinct estimators. Figure 5 (Gaussian fitting) uses 24-hour rolling windows, trade-notional (
price × size) weights, complementary NO prices, excludes buckets with probability ≤0.10, and takes the fitted mean as CPI. Tables 14–15 (share-weighted buckets) use hourly share-quantity weights, carry-forward for no-trade periods, normalized bucket probabilities, and probability-weighted representative values; Table 14 covers 13 months. These are complementary procedures, not competing descriptions of one estimator.6. Downstream adoption and public access (45Aa: practical utility; Ya4q Q1: linkage correctness; AgTX Q4: data/code access).
Our linked records already support public tools, notably Polymonitor (https://polymonitor.club/) for market intelligence and Backtest Lab (https://backtest.polymonitor.club/) for historical replay and strategy research. These applications provide practical corroboration of cross-source linkage usability (Ya4q) and demonstrate real downstream value at scale (45Aa). Further industry-associated recognition comes from Qin and Yang (2026), affiliated with quantitative trading firm Time Seventeen Capital, who cite our synchronized, cross-source data contribution (https://arxiv.org/abs/2606.04217).
We also clarify that the MAS is a proof-of-concept panel in the polymonitor (https://www.polymonitor.club/), not a new algorithmic claim (45Aa Q5).
Experiments use the September 1, 2026 snapshot; data are accessible through the Data Engine above, and anonymous code is available at https://anonymous.4open.science/r/polymonitor-DBC2. The frozen dataset is planned for release in Kaggle after acceptance.
We respectfully ask the Area Chair to consider these substantive improvements and the dataset's scientific and practical significance.
Best regards,
Authors of Submission 24811