- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
TopoPrimer adds two forms of frozen and precomputed population context into time series forecasters, a per-series spectral coordinate and a population level persistent homology fingerprint. The authors evaluate them with from-scratch Transformers and adapters for frozen foundation models. The relational coordinate improves across several public benchmark results. The fingerprint's strongest improvements appear in peak demand and cold start tests on an internal dataset. However, a public M5 cold start replication does not show overall improvement.
Critical strengths
The paper separates the roles of the two descriptors clearly, and explains under what circumstances a population level fingerprint can provide information. It makes the claims more precise than a general “topology helps” argument.
The evaluation includes meaningful controls and unfavorable results. These analyses are helpful in determining when the proposed prior hypotheses are valid and in avoiding overly generalized claims.
Critical weaknesses
Much of the public benchmark advantage seems to come from the adapter, or from the overall advantage in providing cross-series information. Therefore, the reason for a distinctive benefit of the proposed topological representation is modest.
The fingerprint's strongest evidence comes from an unreleased internal dataset, whereas the public cold-start replication is negative. Its applicability to other populations remains uncertain.
Questions for authors
-
Can the fingerprint's cold start benefit be demonstrated on a public benchmark with several structurally distinct cohorts and multiple training seeds?
-
Do the relational coordinate gains persist across multiple forecast origins?
6 comments
W2 and Q1, Cold-Start Claims and M4 (2/3)
W2: Scoping the Cold-Start Claims
W2. "The fingerprint's strongest evidence comes from an unreleased internal dataset, whereas the public cold-start replication is negative. Its applicability to other populations remains uncertain."
We agree, and have revised the claims to match the evidence:
- What we withdraw: Your request for more seeds was right. On our internal corpus, the paper gave every item in a cohort the same fingerprint. Re-tested at 5 seeds, that version's cold-start gain (13.3%) is not significant, and its peak-demand gain (9–15%) does not replicate. We remove both from the abstract and main claims.
- What replaces it (our new cold-start claim): the same fingerprint, computed for each new item from a comparable earlier item instead of once per cohort. An ID cannot stand in for it, because the new item is a different item.
- Against the plain model (10 seeds): 4.8% lower error in the launch week and 3.5% in week 1 (9/10 and 10/10 seeds). The gain fades as the item builds its own history.
- Against simpler encodings of the same data (10/10 seeds): summary statistics are 9.1% worse, the reference item's average early curve 5.2% worse, and a random ID per reference item 15.5% worse.
- Robust to TDA settings: other neighbor counts and landscape resolutions keep 53–97% of the gain (full table: Table 8 in Post 4 of the General Response).
Q1: Public Cold Start With Multiple Seeds
Q1. "Can the fingerprint's cold start benefit be demonstrated on a public benchmark with several structurally distinct cohorts and multiple training seeds?"
Yes: on public M4, the fingerprint beats both summary statistics and a learned domain ID on cold start, across 5 seeds. We made M4 Monthly (about 48,000 business and economic series across 6 domains) into a cold-start test by holding out series that start after a cut-off in 1996. The model is the paper's Transformer, with a learned domain ID in every arm, and each arm's settings chosen on a held-out validation cohort from the same grid. M4 has 6 domains and thousands of series entering over time, close to the setting you describe.
Public Cold-Start Replication: M4 Monthly, New Series Starting After 1996 (test sMAPE, 5 seeds)
| Model | sMAPE | Fingerprint better by | Seeds Where TDA Wins |
|---|---|---|---|
| + TDA fingerprint | 7.19 | - | - |
| + 125 summary statistics of the same data | 7.38 | 2.6% (interval excludes 0, resampling seeds and series) | 5/5 |
| Naive last-value forecast (no training) | 7.67 | 6.3% | 5/5 |
| Plain (learned domain ID only) | 8.50 | 15.5% | 5/5 |
At the 2001 cut-off the fingerprint still beats the learned domain ID but ties summary statistics; at 2010 it ties both. Most other public benchmarks are a single population or a few fixed groups, so a cold-start fingerprint has nothing to vary over.
A second public dataset: Iowa Liquor Sales (public state ordering data, 2012–2019, about 2,300 stores). Iowa has a structure M4 lacks. Each product is ordered across many stores, so a new product's early sales form their own population, and it usually has a comparable earlier product. This lets us compare two designs:
- The paper's design (one fingerprint per product category): does not help (−0.1% vs. plain, 2/5 seeds), as Appendix M predicts for a vector that can only shift its category.
- Per-item design (the same fingerprint, computed for each new product from its comparable earlier product instead of once per category): lowers cold-start error by 1.9–2.6% against the plain model in three setups (5/5 seeds, intervals exclude 0). It also edges out summary statistics in 4 of 5 setups (0.3–1.6%, not significant). A simple product label matches or beats it, since the related product usually appears in training.
- Fingerprint source: the earlier product's first 13 weeks across stores. A store that starts carrying an existing product gets that product's first weeks elsewhere.
On public cold start, the fingerprint beats summary statistics on M4, and the per-item version beats the plain model on Iowa Liquor Sales.
Q1 (continued) and Q2, Unseen Stations and Forecast Origins (3/3)
Q1 (continued): Rel on Unseen Series
TopoPrimer's per-series coordinate (Rel) also shows strong cold-start gains on public data. We held 602 Weather stations out of training and validation and projected each unseen station's history onto the frozen training basis. This significantly reduces 30-day block MAE across all 5 seeds:
Rel on 602 Unseen Weather Stations (30-day block MAE, 5 seeds)
| Coordinate Assigned to Unseen Station | 30-Day Block MAE Δ vs. Plain Model | 95% Bootstrap CI | Seeds Improved |
|---|---|---|---|
Rel (projected onto frozen training basis) | −4.0% | [−5.6%, −2.6%] | 5/5 |
Shuffled Rel (permuted across unseen stations) | +1.7% | - | 0/5 |
Rel vs. Shuffled Rel (station-specific signal) | −5.6% | [−7.3%, −4.0%] | 5/5 |
Takeaway: on stations the model never saw, Rel lowers error by 4.0% in every seed, while the same vectors shuffled across stations make it worse (+1.7%). The gain comes from each station's own coordinate, not from giving it just any vector.
Q2: Rel Across Forecast Origins
Q2. "Do the relational coordinate gains persist across multiple forecast origins?"
Yes for the ECL adapters and the Weather from-scratch model: Rel beats the plain model in 7–8 of 8 test months on ECL and at all 4 start dates on Weather. The Weather adapters gain only at shorter context, and on METR-LA the gain is clear only on top of Chronos-2's joint mode. First, we clarify an exposition error in our original text: our ECL foundation-model adapter evaluation already evaluated all 5,261 rolling hourly forecast origins across the entire 8-month test split (321 clients × 5,261 origins). Second, we ran new multi-origin evaluations on Weather (4 start dates spaced 30 days apart across the full 30-day forecast block) and METR-LA (572 rolling hourly origins):
Rel Across Forecast Origins (5 seeds per origin)
| Benchmark & Model Architecture | Forecast Origins Evaluated | Rel vs. Plain Baseline Across Origins | Consistency Across Time / Seeds |
|---|---|---|---|
| ECL Adapters (Chronos-2 & TimesFM-2.5) | All 5,261 rolling hourly origins (Nov 2018 – Jun 2019, split by month) | −4.0% [−6.0, −2.0] (Chronos-2) and −3.7% [−6.0, −1.2] (TimesFM-2.5) at default k=256; −7.8% / −7.7% at k=26 | Better in 7/8 months (MSE) and 8/8 months (MAE); 5/5 seeds |
| Weather (From-Scratch Transformer) | 4 start dates spaced 30 days apart (full 30-day block MAE) | −3.9% overall (−3.2%, −4.6%, −5.1%, −2.7% at the 4 dates) | 95% CI excludes 0 at all 4 dates; wins in 19/20 date × seed pairs |
| Weather Adapters (Chronos-2 & TimesFM-2.5) | 4 start dates spaced 30 days apart (full 30-day block MAE) | No gain at 300-day context (+0.8% / +0.4%); −14% gain at 90-day context (one start date) | At 90-day context, wins in 5/5 seeds (see our reply to Reviewer S1Lq, Q3) |
| METR-LA Traffic (Chronos-2 Adapter) | 572 rolling hourly origins (15-min MAE) | Univariate: −1.6% (p = 0.095, n.s.); On top of Multivariate Joint Mode: −1.5% | Wins in 5/5 seeds when paired with joint mode (see our reply to Reviewer a6vu, Q2) |
Takeaway: the gains are not an artifact of one forecast origin. On ECL and on the Weather from-scratch model, Rel wins across months, start dates and seeds.
Thank you again. We welcome any further questions.
W1, Adapter vs. Rel vs. Fingerprint (1/3)
We thank you for the clear review.
W1: Adapter vs.
Relvs. FingerprintWe agree completely that on single-population public benchmarks, predictive gains come from per-series cross-series coordinates (
Rel) rather than cohort-level persistent homology (Fingerprint). Rerunning ECL without non-negativity clipping across 5 seeds illustrates this division of labor cleanly:Per-Series vs. Per-Cohort Features on Unclamped ECL (Chronos-2 adapter, 96h horizon, 5 seeds)
Rel(Default / Population-scaled 8% of N)CorrRowcontrol)Fingerprint(TDA)Next, to test whether
Rel's advantage comes merely from adding an adapter to a frozen backbone or from arbitrary per-series conditioning, we ran five strict controls on unclamped ECL (k=26, 5 seeds) and held-out Weather stations:Controls for
Rel(5 seeds)RelAdapter on LoRA (r=2) / Fully Fine-Tuned Chronos-2Relvs. ShuffledRel(across 4 backbone setups)Relvs. Learned 26-dim Client EmbeddingRel(TimesFM-2.5 MSE interval crosses 0).Rel: −4.0% (5/5) vs. ShuffledRel: +1.7% (0/5)Reltransfers.Takeaway:
Rel's gain is not the adapter, the frozen backbone, or extra parameters. About 2% of it comes from the coordinates themselves rather than from giving each series its own vector, and onlyRelcarries over to series the model has never seen.Finally, where does the fingerprint beat simpler summaries? On cold start, it beats summary statistics and ID controls both per item on our internal corpus (10/10 seeds) and per domain on public M4 (5/5 seeds). Separately, its persistence statistics rank which internal cohorts will benefit, before any training:
Verified Settings Where Persistent Homology (
Fingerprint) Outperforms Non-Topological ControlsS1LqQ4)