acceptodds
Under review as a conference paper at ICLR 2027

The Window Lever: Observation-Window Length Outweighs Observed Scale and Architecture Effects in Customer Behavior Forecasting

Abstract

Forecasting a customer's next year: add customers, grow the model, switch architectures, or look further back? We test all four, one knob at a time. Looking back wins. Stretching the observation window from 8 to 16 weeks adds to log-space in all eight models at 1M, and the simplest gains most. Test-set composition is the obvious suspect, so we fix test customers and targets across windows (training cohorts still grow with the window). The gain gets bigger, not smaller, and the matched transformer gain beats all 1,000 permutation-null draws. The direction replicates on a public dataset. For sequence models, the other levers move an order of magnitude less. the customers shifts by at most , as does a larger transformer (three-seed mean), and on one seed, five sequence architectures land within at a matched objective at 16 weeks. All three are one-knob contrasts in small-model recipes; joint scaling is untested. We internally registered four predictions, with falsifiers, before the deciding evaluations. Three held. The one fully blind prediction failed: an activity-mixture reading called for from 24 to 32 weeks; we measured . The pooled weekly-aggregate probe plateaus near 22 to 24 weeks, but activity-specific refits keep climbing: no information ceiling is established. One limit remains: every test anchor falls in one eight-week band, so window length and calendar coverage move together. The direction is robust; the magnitude is not identified. Observation-window length belongs beside model selection as a first-class design axis: what a sequence model is shown matters more here than what it is.

Primary area:
learning on time series and dynamical systems
Keywords:
scaling laws, event-sequence forecasting, observation window, customer behavior, empirical evaluation, registered predictions

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_WhSjOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Summary

The authors analyze data from customer event streams from a single consumer platform to evaluate how the following components affect predictive accuracy:

  • Dataset size (number of customers)
  • Dataset size (window length)
  • Model size
  • Model architecture

Specifically, the authors change these elements one at a time in an attempt to determine which carries the largest impact on model performance. This leads to the claim that the number of time steps (the window length) is the largest “lever” of the group. This claim is only justified with respect to the specific dataset that the authors use (along with a short external validity check on one dataset, which does not replicate the findings), which is left unidentified and only described as “customer event streams from a single consumer platform, exported as fixed cohort carves”. No code or data is given, and thus there is no avenue for replicating the one empirical claim the paper rests on. I must also mention that the writing throughout the paper is incredibly unclear, disorganized, and relies on jargon terms which are not well-defined or commonly used in this literature. Each time a claim is made it carries a long list of qualifications using these jargon terms, and the details needed to piece together what was actually done are scattered across the paper with insufficient references or clarity as to how to piece it together.

Contributions

Under the most generous interpretation of this paper’s contributions, the authors have shown for the one “primary dataset” used, window length is the most impactful component of the four listed above—but nothing is provided with respect to the possible generalization of this finding or even the trustworthiness and/or replicability of this finding itself, other than the author’s statements. The authors claim to have pre-registered their study, but it seems their word must be taken at face value for this (which as I understand it, is not common practice for preregistration) since they only print their internal hash prefixes rather than following standard practice of having some sort of third-party registry/timestamp. The dataset is not adequately described for a full understanding of the setup used by the authors. Important elements of the dataset (w.r.t. external applicability) are never mentioned, such as what the “events” are. Given this specific application in customer event forecasting and the narrow empirical finding, the ceiling on the contributions of this paper are very low. The overall paper’s conclusion that “observation-window length belongs beside model selection as a first-class design axis” is already widely agreed upon in the literature. Lastly, their title finding that window length is crucial confounds their own limitation that they were unable to differentiate window length from calendar coverage.

Validation

Other than the findings on the “primary dataset”, the authors test for external validity on a dataset from Alibaba. The entirety of this external validity check, including its setup and results, occurs in one paragraph of main text with no accompanying figures or tables (I did not find the Appendix covering this to provide much clarity). The external check also does not replicate the author’s claims on their “primary dataset”, with R^2 very close to zero.

Critical strengths

The authors are clear about their claimed contributions and provide honest discussion of limitations.

Critical weaknesses
  • Limited potential for impact. Taken at face value, the finding applies to only one dataset, which is not identified or described in adequate detail to understand the setting. It is also restricted to small models and does not check for joint changing of the “knobs” (i.e., maybe they interact in certain ways)? The authors mention that their controlled setup is their contribution, but changing one thing at a time is hardly a new idea.
  • Claims are unable to be verified and do not have adequate support. Data and code are unavailable, which is crucial for a work whose main finding is an empirical result. Same goes for the preregistration, which is only self-verified and does not have any documentation that cannot be manipulated (thus not following the standard preregistration procedures). The external check only is able to replicate the sign, and even then has near-zero R^2. Calendar coverage confounds the window effect and takes away the identifiability of this titular claim.
  • Writing quality. The explanations and descriptions in this paper are full of jargon and string together incredibly vague sentences which obfuscate their meaning. Notation is not deeply thought out - for example, there is one sentence in Section 2 where “M” means three different things at once (parameters in millions, window variable in weeks, and a label prefix). All notes on replicability and experimental setup are written out in a way that makes it incredibly difficult to follow and understand their setting, and required many readings to piece together part of what occurred.
Questions for authors
  • Can you please provide more information regarding the primary dataset and why this is a fair/representative dataset to base the claim on/why such a finding on this dataset would generalize nicely?
  • Do you have any documentation of pre-registration in a publicly-available registry or third-party verified source?
  • Given the difference between the time horizons on the primary dataset and the Alibaba dataset, do you think these examples are comparable?
  • Do you have any empirical results demonstrating how window length interacts with the other components that you inspect?
15 comments
AuthorsOpenReview ↗

Reply to Reviewer WhSj (1/2)

(1/2)

Thank you for the close read and the concrete questions. You're right about the writing and about "preregistered", and, following your review, the revision changes both. Section, table and appendix numbers are the revised PDF's, and the transformer's numbers are for the model as it ran (the categorical inputs were one constant code, the numeric fields went in unstandardized, and the rotary encoding was as implemented; see our general reply).

Q1. What is the dataset, and why is it a fair base? Each row is one customer at one forecast week (the week the forecast is made; the submission's anchor week). The input is that customer's events in the 8, 12, 16 or 24 weeks before it, capped at 750 events for the sequence models. Each event is one of four kinds (adding an item to a cart or list, a purchase, a review, or a subscription order) with a type code, five categorical attribute codes and two numeric value fields (recipe table), but no model read the codes: the transformer got one constant code, and the baselines don't take them. The target is the customer's total activity over the next 364 days, a real-valued total, not a count, modeled as log(1+max(0,y)), and exactly zero for 4.8% to 30.4% of rows by slice (§2.2). Splits are disjoint in customers and forecast weeks: train runs 2024-01-29 to 2024-09-09, validation 2024-11-04 to 2024-12-30, and test is three forecast weeks from 2025-02-24 to 2025-04-21. The 364-day target periods overlap in calendar time, so the splits are out-of-time in forecast weeks, not in labels. The three tiers (×1, ×5, ×10) are non-overlapping customer samples of one export, with test splits of about 480K, 2.4M and 4.8M rows (§2.1, App. J). We don't name the platform; the data is proprietary. This is now in §2 and App. H.

On representativeness, we agree that one primary dataset is a real limit, and we don't claim the data is representative: the primary evidence is one platform, one target, one export era (§5). But the result isn't tied to one customer sample or one model: the window gain holds in all three customer-disjoint tiers (+0.069 to +0.085 across sequence models and tiers, seed-42 endpoints; §3.5), it shows up in all eight model families (Q4), and for ridge the bottom three activity deciles carry 69.8% of the matched 8-to-16-week error reduction (§3.2). All of it shares the three test forecast weeks, so none of it touches the calendar point below. A history gain also shows up for ridge and our transformer on all five public datasets we ran, four of them on our own time scale, and none was chosen by its result (Q3 and our follow-ups). Its size on other platforms is open. Scope: every row of the ×1 matched cohorts has an event in the 8 weeks before its forecast week (§2.3), so the matched gain is for recently active customers. The rows the cohort drops are mostly lapsed (at 16 weeks, 66.3% have no event in those 8 weeks and 30.4% have a zero target, against 4.8% matched; App. B).

Q2. Is there a public registry or third-party record? No, and we should have said so more plainly (App. L: "not deposits with an external registry, and not third-party timestamped"). So "preregistered" is now "internally registered". Each prediction and its falsifier were dated in an internal record before the deciding evaluations. The predictions, falsifiers and scored record are unchanged. A reader can still check each outcome against its printed falsifier, though not when it was committed: Table 1 says, in brief, what was already known at each commit time (App. N.2 adds that a matched ridge gain of +0.104, already over commitment (i)'s +0.05 bar, was on record when that bar was set), and the one fully blind prediction failed and is reported at the same volume as the hits. The window gain and the bounds don't depend on the record.

Q3. Are the public dataset and ours comparable? Taobao is comparable in direction, and loosely in shape, not in size or where the curve bends. Its windows are 3 to 7 days, its target is a 2-day purchase count that's about 69% zero, and levels sit near zero, so we never compare levels (§3.6, App. E). What's shared is the intervention: fix the target period, lengthen the history, score the same users. On 6,985 matched users ridge R² goes 0.0213, 0.0305, 0.0374, every paired difference excluding zero, and the three-seed transformer test agrees. What it can't close is the horizon gap: Taobao spans 9 days, and its longer windows admit more training users, so it doesn't separate more history from more training rows (§5). Our longer windows add up to 10% more, colder training customers too (Table 2); 10× the customers moves sequence-model R² by at most 0.006 (seed 42), far below the window gain, though that bound doesn't isolate them. You're right that it was thin on the page. §3.6 now also prints the four multi-year public gains (App. P), and App. E adds a results table (Table 10) to its extracts, entry rule, features, target, seeds and per-seed results.

AuthorsOpenReview ↗

Reply to Reviewer WhSj (2/2)

(2/2)

Q3, continued. Thank you for pressing on the horizon gap. The revision adds a public check on our time scale (App. P): Online Retail II (Chen, 2012, UCI Machine Learning Repository), a two-year log from one UK online retailer with many wholesale buyers. Setup: the paper's weekly ridge probe (§3.3) up to 24 weeks, 364-day target, one fixed set of 13,665 customer-forecast-week rows selected on the 8 weeks every compared window sees, reading rule fixed before any number existed. From 8 to 16 weeks R² goes 0.134 to 0.189, +0.055 [0.046, 0.062]; 24 weeks adds +0.068 [0.057, 0.079] over 8. Shuffling weeks 8 to 15 across customers gains nothing (−0.0013 [−0.0023, −0.0003]): the gain is each customer's own history, not more features. Returns diminish, but the sharp bend isn't shown: weeks 22 to 24 add 0.21 [0.13, 0.32] as much per week as weeks 8 to 16, and a bend needs that ratio's upper limit under a tenth. Unlike Taobao, every window fits and scores the same rows, and the weeks the 16-week window adds hold no November to January day, so the gain is neither more training rows nor added holiday weeks. Scope: one small retailer, ridge only, no level or gain compared with ours.

Q4. Does the window interact with the other factors? Three crossings are measured. Customers: the transformer's matched 8-to-16-week gain is +0.0939±0.0022 at ×1 and +0.0934±0.0005 at ×10 (3 seeds, §3.2), on disjoint cohorts with their own schedule points, so this bounds cohort, size and schedule together (App. I); ridge gains a bit more at ×10 (full split, +0.1050 vs +0.099). Model family: +0.070 to +0.107 across all eight families at ×1 (Table 7; the transformer in Table 4). Loss: under Huber, MSE and pinball the 3-seed full-split climb is +0.0701 to +0.0800 (§3.1) and +0.0863 to +0.0959 matched (seed 42, App. B); the loss moves levels by 0.03 to 0.05 and every loss climbs. You're right that we have no joint setting (a bigger model on more customers): capacity was probed only at ×1 with the small model's recipe, so we don't claim the ordering survives joint scaling (§5).

Code and data. We understand the concern, but we can't release either: the data is proprietary customer data, and company policy keeps the code internal. The paper documents the recipe (the recipe table, as corrected), the splits and direct-join disjointness audit (App. J), the matched cohort and its permutation control (§2.3, App. K), the bit-exactness ledger (App. H) and the registration digests (App. L). Every public check gives its windows, features, target and ridge setup (Apps. E and P). That doesn't replace a release.

Writing and notation. Agreed, and following your comment the revision uses its 10th page on it. In that sentence M did triple duty: window length, window-label prefix (M8 to M24) and tier-label suffix (1M/5M/10M). The revision's text and tables use W8 to W32 and ×1/×5/×10. Setup now comes before results (§2), the target, metric and matched cohort are each defined once, a notation paragraph pins down the labels (§2.4), and we replaced most of our internal shorthand in the main text with words; the rest is item 3 of our general reply.

Novelty. Agreed that window length mattering isn't new, and neither is changing one thing at a time; the intro says "our contribution is that comparison, not the question" (§1), and we should have made that clearer. The claim is narrower: on one population and target, with test customers and targets fixed, the window gain is about an order of magnitude above the scale, capacity and architecture contrasts for the sequence models (point estimates at the recipes tested), and through 16 weeks the raw comparison understates it. Related work (App. M) now covers the concurrent Dou et al. (2026).

Calendar coverage. Agreed, it's the main open limit (abstract, §5). We kept the title because its claim is about effect sizes at the recipes tested (§5), though we agree the confound leaves open how much of the window gain is history rather than calendar. The conclusion now says so: the window, confounded with calendar coverage (§5), is the dominant tested lever on this platform and target (§6). We ran the check Mhj2 asked for (App. O): the weekly ridge probe on held-out training customers, training forecast weeks whose windows never reach November or December against ones on the test weeks of the year. By its rule, written before the check computed any R², the gain persists outside the holiday season: the holiday-free 8-to-16-week gain is 0.72 [0.69, 0.75] of the test-calendar one. Returns diminish on both calendars, but the sharp 22 to 24 week bend, read on all held-out customers of those forecast weeks, shows only on the test calendar, so the revision scopes it there. Table 1's predictions and scores stand: they were made and scored on the test forecast weeks. App. O repeats it on the transformer, and it persists outside the holiday season. Summer and spring differ in more than the holidays.

AuthorsOpenReview ↗

The public checks in one table

The public ridge checks and the Taobao transformer in one table

Thank you for pointing out that our external check was one short paragraph with no table. Following your comment, and because the public evidence is now spread over several replies, here it is in one place (App. E and App. P of the revised PDF). A gain counts only if its 95% interval sits wholly above zero. The control shuffles the added weeks across customers.

DatasetSpanRows scoredR² at the shorter windowGain from the longer window [95%]Shuffled-weeks control [95%]
Taobao, ridge9 days6,985 users0.0213 (3 days)+0.0162 [+0.0090, +0.0229], 3 to 7 daysnot run
Taobao, transformer (3 seeds)9 days6,985 users−0.0073 (3 days, seed mean)+0.047 [+0.035, +0.059], 3 to 7 daysnot run
Online Retail II2 years13,6650.134+0.055 [0.046, 0.062]−0.0013 [−0.0023, −0.0003]
The Complete Journeyabout 2 years6,5240.4822+0.0315 [0.0245, 0.0384]−0.0017 [−0.0030, −0.0004]
CDNOW18 months3,4450.1188+0.0317 [0.0198, 0.0432]−0.0038 [−0.0070, −0.0009]
H&M2 years3,164,0240.1868+0.1024 [0.1017, 0.1030]+0.000002 [−0.000006, +0.000010]

The paper's transformer on Online Retail II, The Complete Journey and CDNOW is in App. P (Table 15) and in a follow-up below, each dataset read on its own.

The Taobao rows go from a 3-day to a 7-day window on a 2-day purchase-count target, which is about 69% zero, so their R² levels sit near zero (a negative level is below the cohort-mean predictor). The last four rows use the weekly ridge probe and a 364-day target, from 8 to 16 weeks, with the same rows fit and scored at every window, so the gain isn't more training rows. On Online Retail II and CDNOW the weeks the 16-week window adds hold no November to January day. On CDNOW the added weeks also carry when the customer joined (everyone joined in Q1 1997; the 16-week window holds the join date for 31% of rows). On CDNOW the gain also shows on the 2,375 rows whose 8- and 16-week windows fall wholly after joining, +0.0462 [0.0311, 0.0614]. On H&M the gain shows without holiday weeks in the added history: a refit on the five forecast Mondays whose 16-week history holds no November, December or January day, with each Monday's level taken out of the fit, gains +0.1054 [0.1045, 0.1062]. A first version without the per-Monday level failed its shuffled-weeks control, so this one adds it. Those five target years hold 82 to 194 days of the COVID period, and the per-Monday level doesn't capture how COVID changed buying differently across customers. Seven of H&M's eight target years run into the COVID period, which the pooled row doesn't separate. The Complete Journey's days carry no calendar dates, so it says nothing about season or holidays. None of this transfers a size to our data, and the table doesn't rank the datasets by gain (the Taobao rows use a different window and target). The cross-dataset comparisons we do make are the predictions in the Complete Journey, CDNOW and H&M follow-ups (App. P), each with its caveats.

Reviewer_Mhj2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper systematically compares extending the historical observation window against scaling customer count, model capacity, and sequence architecture in customer behavior forecasting, showing under a matched-cohort evaluation protocol that lookback window length dominates all other single-knob interventions before plateauing around five to six months.

Critical strengths
  1. The use of matched-cohort evaluation frameworks, combined with falsifiable pre-registration protocols, enables a rigorous separation of biases stemming from test-set composition—setting a commendable standard for empirical transparency.
  2. Such systematic, univariate controlled studies demonstrate that historical constraints and objective alignment are far more significant than architectural complexity, offering highly valuable practical insights for the field of industrial forecasting.
Critical weaknesses
  1. As all test benchmarks are concentrated within the February-to-April timeframe, it is not possible to determine the magnitude of the main effect or the saturation pattern occurring between weeks 22 and 24.
  2. There are flaws in the implementation and optimization of the core experimental comparisons. This is a significant issue, as such anomalies undermine the validity of both the compact architecture and the scaling boundaries.
Questions for authors
  1. Could the authors evaluate the cumulative weekly "spikes" (detections) for sub-segments of the training anchor weeks to clarify whether the 8–16 week "jumps" and 22–24 week "bends" persist when the anchor points are distributed throughout the calendar year?
  2. Why have the essential figures (Figures 2–5), key comparison tables (Tables 3–8), and definitions of core terms been relegated entirely to the appendices? How do the authors intend to restructure the main text to ensure the paper is self-contained?
7 comments
AuthorsOpenReview ↗

Reply to Reviewer Mhj2 (1/2)

(1/2)

Thank you for the careful read and for two concrete, helpful questions; we ran the check you proposed in Q1. Section, table and appendix numbers below are the revised PDF's.

Calendar (W1, Q1). We agree this is the main limit, and it's the right one to raise. §5 says the magnitude isn't identified for exactly this reason, and names one seasonal mechanism as a candidate for the 22 to 24 week bend as well as the 8 to 16 step. Our control that varies the forecast week (the submission's anchor week) shows the pattern a holiday explanation predicts, a gain several times larger at the earliest test forecast week than the latest, but it is void under its own criterion: at 8 weeks on MSE, its R² spread across the test forecast weeks is already 5.6 to 6.2 times the permitted bound, so we count it neither for nor against that explanation.

A calendar-balanced test set still can't be built from this export, but your suggestion doesn't need one: training forecast weeks run from late January to early September 2024 (§2.1), which is enough to split them by season. So, following your guidance, we did that:

  • Same weekly-aggregate ridge probe as §3.3 (one nested ridge per history length, k = 1..24), fit within each group on 80% of training customers and scored on the other 20%.
  • Holiday-free group: four forecast weeks, mid-June to early September 2024. No window up to 24 weeks reaches November or December.
  • Test-calendar group: three forecast weeks, one per test forecast week on the same week of the year, so the holidays enter their windows as they do on the test set (to within two edge days).
  • The groups are set by the calendar alone, and they and the reading rule were written down before the check computed any R². The gain persists outside the holiday season if the holiday-free gain's interval excludes zero and the holiday-free/test-calendar ratio's 95% interval sits at or above one half; it's mostly seasonal if that ratio's upper limit is below one half. The sharp bend holds in a group if the 95% upper limit of (per-week gain over weeks 22 to 24) / (per-week gain over weeks 8 to 16) is under a tenth.
  • Before either group was read, the probe had to reproduce the paper's own R²(k) at k = 8, 16, 22 and 24 on the matched test cohort. It did, exactly. Intervals are a customer-clustered bootstrap over held-out customers, so they cover scoring only, not fit noise, the split or variation across forecast weeks.

Result: the gain persists outside the holiday season. From 8 to 16 weeks, R² rises 0.0352 [0.0342, 0.0361] on test-calendar forecast weeks and 0.0253 [0.0246, 0.0259] on holiday-free ones, so the holiday-free gain is 0.72 [0.69, 0.75] of the test-calendar gain (customers resampled, forecast weeks fixed). Most of the gain shows with no holiday weeks in the window, though the rest may still be calendar-related. A shuffled-weeks control adds no R², and refitting with each forecast week's level taken out leaves the conclusion unchanged. On all held-out customers of test-calendar forecast weeks, weeks 22 to 24 add 0.071 [0.056, 0.086] as much per week as weeks 8 to 16, so the sharp bend holds there, but on that calendar it can't be told apart from a seasonal effect (§5). On holiday-free forecast weeks the ratio is 0.18 [0.16, 0.20], so it isn't shown, though returns still diminish (R² 0.435, 0.461, 0.467 at 8, 16 and 24 weeks). Returns diminish on every calendar we checked, and the revision scopes the sharp 22 to 24 week bend to the test calendar, read on all its held-out customers. On H&M, a public dataset, a ridge refit on three forecast weeks whose 24-week histories hold no holiday weeks finds that returns still diminish and the bend isn't shown (App. P; its three target years hold 138 to 194 days on or after 2020-03-11, and the fit absorbs each forecast week's level but not how COVID changed buying across customers). Table 1's predictions and scores stand: they were made and scored on the test forecast weeks.

This check has three limits. It's the linear probe on training-period forecast weeks and held-out training customers, not the transformer or the test numbers. And the two groups differ in season as well as holidays (summer vs. spring), so it shows the gain doesn't need holiday weeks in the window, but it can't pin the gap between the groups on the holidays alone. Two of the three test-calendar forecast weeks, 2024-02-26 and 2024-03-25, also have only partial event history from 17 and 21 weeks back (the 24-week extract's event history is partial before 2023-10-30), which covers both weeks the 22-to-24-week step adds, so part of that group's sharper bend may come from missing history rather than the calendar. The 8-to-16-week gains, and so the 0.72, use weeks 8 to 15, which sit after that boundary at every forecast week here. It's in the revision as App. O, summarized in §5 with a pointer in §3.3.

AuthorsOpenReview ↗

Reply to Reviewer Mhj2 (2/2)

(2/2)

Implementation and optimization (W2). Thank you for raising this; we take it seriously. We read it as the issues the paper discloses (if you had others in mind, we'd be glad to hear them):

  • (a) ContiFormer and Neural Hawkes are simplified stand-ins that ran with a zero inter-event interval, so they act as a plain causal attention stack and an LSTM (App. H). We don't count them as tests of continuous-time models. Following your comment, both have been rerun with a working clock, under a rule fixed before we saw either result (check 2 of our reading rules).
  • (b) The transformer ran near its peak learning rate while the recurrent and continuous-time models annealed. That could manufacture the architecture band. Among the baselines alone (three-seed means), where that asymmetry doesn't apply, the spread is 0.0059, wider than the abstract's 0.0052, both on the full test split (App. I). But the pair that sets it, GRU and ContiFormer, trained at different peak learning rates (one of the corrections below), so we don't pin that on architecture either. We don't claim the band survives matched schedules.
  • (c) We agree: the 54× capacity probe reused the small model's recipe unchanged, so it bounds capacity at fixed recipe only, and a hyperparameter retune of that model returns no verdict, since it selected on validation only and its confirmation runs never got all their seeds (§3.1). A separate retrain that corrected the schedule and switched to log-space selection together leaves the magnitude bound unchanged, but the 54× model's deficit in the original runs doesn't reproduce in the retrain's three-seed means (App. I). The capacity bound survives a 5-point learning-rate sweep of the 54× model, on corrected inputs (check 1 of our reading rules; our follow-up below).
  • (d) The ×1 and ×10 tiers sit at different points of the schedule, so the scale bound compares two optimization regimes (App. I).
  • (e) Checkpoints were picked on original-scale validation error. Re-selecting in log space moves validation scores by up to +0.009. At 8 weeks the scale bound is close to invariant to that; the architecture band has no such defense, so we read it as including selection sensitivity, not corrected for it (App. I).

None of this reaches the window gain where it's largest. Ridge (+0.099) and the one-feature linear model (+0.107) gain the most of the eight families from 8 to 16 weeks on the full test split at ×1 (Table 7; the transformer's are in Table 4), with no learning-rate schedule, no epochs to select and no sequence encoder. Inside the transformer, the 8- and 16-week models at ×1 run the same number of steps and end at essentially the same rate, so that delta is schedule-matched (App. I), though checkpoint selection still reaches it.

You're right that the bounds are claims about this small-model recipe, not about scale or architecture in general. The revision says so up front, in one "what the numbers show, and what they don't" paragraph in the intro, and a new Table 3 lists each lever's largest observed change with the seeds behind it, as point contrasts at the recipes tested. Our general reply lists four implementation corrections the revision makes: the transformer's categorical inputs were constant, its numeric fields weren't standardized (the recurrent and continuous-time models' were), its rotary map as implemented isn't in general a rotation, and ContiFormer and Neural Hawkes trained at lr 1e-4, not the 3e-4 we printed. These describe what ran, so the first three apply to every transformer number, its own window gain included, and the last to the two continuous-time models. They're implementation flaws of exactly the kind you raise, and they bear most on the architecture band, the one bound that compares families: the sequence baselines read no category codes either, but they took standardized numerics, so the input gap favored them. None of them touches ridge or the linear model.

Presentation (Q2). We should have handled this better: under the 9-page limit we moved figures and tables out to fit, and the submitted main text cited its Figures 2, 3 and 5 without saying they were in the appendix. To your question on restructuring: the revision uses the 10th page:

  • Setup (§2) now comes before the results, and the target, metric and matched cohort are each defined once there.
  • A notation paragraph (§2.4) pins down the labels (windows W8 to W32, tiers ×1/×5/×10, seed conventions) and says which figures and tables are in the appendix.
  • Figure 2 (the saturating law, the 24-week hit and the 32-week miss) is now in the main text, next to the new bounds table. The per-family window figure (Figure 3) stays in the appendix, with Figures 4 and 5 and Tables 4 to 15, since the main text is now at the 10-page limit.
  • We replaced most of our internal shorthand in the main text with words.

None of this is a re-run: every number in those pieces was already in the paper. Happy to clarify.

AuthorsOpenReview ↗

Follow-up: ContiFormer with a working clock

Follow-up to Mhj2's W2a: ContiFormer with a working clock

Thank you again for raising the continuous-time baselines; following your comment, we reran both with a working clock. This is the first of the two reruns. It ran under check 2 (the working-clock check) in our reading-rules comment. That rule was set before we saw any of its results.

ContiFormer was trained with the clock on (real event times, each sequence sorted by time, its decay settings as implemented) and with it off, at seeds 42, 123 and 456. Everything else followed the paper's recipe: 16 weeks of history, 10 epochs, AdamW at lr 1e-4, and the final model scored on the full test split of 480,673 rows. These runs used 8 GPUs instead of 1. Clock-on and clock-off runs read the same training rows: the per-GPU batch cap reads a 48,000-row prefix of each GPU's shard per epoch, which is a different subset from the paper's 1-GPU runs. Before the full runs, a 1-epoch clock-on run showed a nonzero clock, a finite loss and scoring on the full test split.

Result: no gain from a working clock detected for ContiFormer. Δ = R²(clock on) − R²(clock off), averaged over the three seeds, is +0.0012, with a 95% t interval of [−0.0009, +0.0033] (df 2). The interval covers seed-to-seed noise only.

Scope: the clock as implemented, including its time sort and ContiFormer's time-gap divisor of 1; 16 weeks at the ×1 tier only, 8 GPUs, 3 seeds. The revised PDF adds it to App. H. Neural Hawkes is the second rerun; its result follows as its own comment.

Comments on the paper

8 comments
AuthorsOpenReview ↗

Reading rules for three GPU checks (1/2)

Reading rules for three GPU checks (1/2)

Thank you, WhSj, for asking for third-party evidence that our plans came before the results. Here are the reading rules for three GPU checks, each fixed before any of its results was read, so each follow-up can be checked against the rule it ran under. All three checks' results were seen before posting, so this comment's date can't vouch for their rules; their internal signing dates are what back them. In all three the metric is test log-space R² at the ×1 tier, each interval is a 95% t interval on the three per-seed paired differences (df 2), so it covers seed noise only, and a failed precondition or a needed variant without three scored seeds by Nov 16 voids the check, unless its own rule below says otherwise (a void check reports status only, no number). Transformer runs use the rotary encoding as implemented.

1. The 54× transformer with a tuned learning rate (Mhj2's W2c)

  • d512/8L at peak lr 6.25e-5, 1.25e-4, 2.5e-4, 5e-4 and 1e-3, against d128/2L at 5e-4, seeds 42, 123, 456, 16 weeks, the same 20-epoch recipe.
  • These runs use corrected inputs (real categorical codes and standardized numeric fields, where the paper's runs had one constant code and raw values; see our general reply). So they measure the capacity gap on those inputs, their two 5e-4 models re-run the paper's base configurations with App. I's corrected schedule and selection, and no paper transformer score is printed beside them.
  • The d512 lr is picked on validation only (highest three-seed mean, a tie to the lower lr), among variants whose three seeds all finished without diverging, fixed when the runs finish or on Nov 14, whichever comes first, before any test score is computed. If the d512 or d128 5e-4 variant isn't eligible, the check is void. Every d512 variant's test R² is shown after the verdict; only the picked one enters Δ.
  • Δ = mean over seeds of R²(d512 at the picked lr) − R²(d128).
  • Checked first: if the picked variant's population SD of test R² across seeds is above 0.0043, no verdict yet; seeds 789 and 2024 run on it and on d128, and one final estimate at five seeds (df 4) gives a verdict only in the next two bands.
  • Δ ≤ −0.007: "after a 5-point learning-rate sweep the 54x model is still worse than the base by more than the paper's bound".
  • −0.007 < Δ < +0.007: "the capacity bound survives a 5-point learning-rate sweep". If another eligible d512 variant's validation mean differs from the 5e-4 variant's by more than twice the 5e-4 variant's population SD across seeds, we add that the sweep moved validation and the bound held at the selected lr; if none does, we add that this grid can't show learning rate was what held the large model back, so our concession that hyperparameters weren't re-tuned is narrowed, not retired.
  • Δ ≥ +0.007, interval excluding 0: d128's lr is picked the same way over 2.5e-4, 5e-4 and 1e-3, and we apply these bands once more, for a final verdict. If it lands here again, the paper's capacity bound is false at this width and depth once both models' learning rates are tuned.
  • Δ ≥ +0.007, interval including 0: d128's lr is picked the same way, both get seeds 789 and 2024, and one final estimate at five seeds gives a verdict only in the first two bands.
  • Follow-ups not done in time (d128's learning rate fixed on Nov 16 before any of its runs is scored; every follow-up scored by Nov 16): the first estimate, marked "follow-up not finished", with no verdict.
  • We call it "re-tuned over a pre-declared 5-point learning-rate grid at fixed budget", never "fully re-tuned" or "a hyperparameter search". If the picked lr is at an end of the grid, we say the sweep didn't bracket the best rate on that side; if epoch 20 is its best epoch on two or more seeds, we say the 20-epoch budget may bind.

2. ContiFormer and Neural Hawkes with a working clock (Mhj2's W2a)

  • Each model with the clock on (real event times, each sequence sorted by time, its decay settings as implemented) and off, seeds 42, 123, 456, 16 weeks, otherwise the paper's recipe (10 epochs, lr 1e-4, final model scored on the full test split), on 8 GPUs instead of 1. Both variants use the same training rows, a different subset from the paper's runs.
  • First, one 1-epoch run per model with the clock on has to show a nonzero clock, a finite loss and the full test split, or that model's runs stop. If it finds events out of order, the clock-on variant also changes event order, and we'll say so.
  • Δ = mean over seeds of R²(clock on) − R²(clock off). Wholly above 0: "a working clock helps <family>". Wholly below 0: "a working clock hurts <family>". Otherwise: "no gain from a working clock detected for <family>", with the interval.
AuthorsOpenReview ↗

Reading rules for three GPU checks (2/2)

Reading rules for three GPU checks (2/2)

3. The window gain on corrected inputs, and window × model size (WhSj's Q4)

  • d128/2L and d512/8L at 8 and 16 weeks, lr 5e-4 (untuned), seeds 42, 123, 456. The 16-week runs are check 1's 5e-4 runs, on the same corrected inputs, so no paper score or gain is printed beside them; at most we'll say in words whether the sign agrees with the paper's history finding. Check 1 read those 16-week runs' full-test-split scores, and this check's matched-cohort scores were also read before posting. A failure in the d512 runs only, in either check, voids just the interaction and G(d512); the d128 window gain is still read. Any other void of check 1 voids this check.
  • Scored on the paper's matched cohort: the 407,338 test customer-weeks present at 8, 12 and 16 weeks, with identical targets. If matched scoring can't be done or fails its checks, that doesn't void the check: we use the full test splits, prefix every verdict "on the full test splits (the matched cohort couldn't be scored):", and add that on the paper's inputs the full-split comparison understated the 8-to-16 gain, so a "not shown" there is weaker evidence than a matched one.
  • G = mean over seeds of R²(16 weeks) − R²(8 weeks) per size, and I = G(d512) − G(d128). G(d512) is shown but gets no verdict.
  • Checked first, for each question: if any configuration it uses has a population SD across seeds above 0.0043, that question gets no verdict and no more runs.
  • G(d128) wholly above 0: "on the activated inputs, 16 weeks of history still beats 8 for the base model". Wholly below 0: "on the activated inputs, 8 weeks beats 16 for the base model". Otherwise: "on the activated inputs, the 8-to-16-week gain isn't shown at 3 seeds", with the interval. ("Activated" means the corrected inputs.)
  • I wholly above 0: "the 54x model gains more from the longer window than the base model". Wholly below 0: "the 54x model gains less from the longer window than the base model". Otherwise we give the interval, plus "any interaction is smaller than the paper's capacity bound" if it lies within ±0.007.
  • If check 1 picks another lr for d512, the interaction at the tuned lr is untested, and we'll say so. If epoch 20 is the best epoch on two or more seeds of a configuration, we say the 20-epoch budget may bind.

2. ContiFormer and Neural Hawkes with a working clock, continued. A run that diverges isn't rerun; that pair is missing, which voids that family's verdict, and we report the divergence. If a family's interval is wholly above 0 and its Δ exceeds 0.005944 (the spread of the four baseline families' three-seed means, best minus worst), our architecture comparison could change, and we revisit it before changing any sentence about it.

Also under signed rules: the transformer on the public datasets. Each rule below was signed before any of its runs was scored (dates UTC), and a gain is read only if its 95% interval sits wholly above zero. The Online Retail II, The Complete Journey and CDNOW results were read before posting, so this comment's date can't vouch for that rule; its internal signing date is what backs it. H&M's runs were read before posting too.

  • Online Retail II, The Complete Journey and CDNOW, 8 and 16 weeks, 5 seeds; two intervals, one for seed noise and one for scoring noise, and both must clear zero (signed Nov 10).
  • H&M, 8 and 16 weeks, 3 seeds; the interval covers seed noise (signed Nov 10).

Each check's result is in its own follow-up, reported against these rules at the same volume as the rest.

AuthorsOpenReview ↗

Reading rules for the four added checks (1/2)

Reading rules for the four added checks (1/2)

Here are the reading rules for the four checks we added. Each was fixed before any of its results was read; our own dated records show every rule signed before its check's read (for check 7, see its first point). Each check's result follows in its own comment, read under these rules. All four use the paper's d128/2L transformer on the corrected inputs (real categorical codes and standardized numeric fields) with the rotary encoding as implemented, so no paper transformer score is printed beside them. The metric is log-space R², and each gain's interval is a 95% t interval on the per-seed paired differences, so it covers seed noise only.

4. The window gain with training rows held fixed (the AC's point that a longer window also changes who is eligible to train)

  • 8 and 16 weeks, seeds 42, 123, 456, each window trained twice: on its own training split ("own"), and on only the rows whose customer and forecast week are in both windows' training splits ("int", 4,314,853 rows, 14.4% fewer than either own split). The two int arms train and select on the same rows and targets; only the event history differs. All 12 are scored on the paper's 407,338-row matched cohort.
  • D_int = mean over seeds of R²(16, int) − R²(8, int), the primary read. E = mean over seeds of D_int − D_own, the change in the gain from holding training rows fixed.
  • D_int wholly above 0 and E covering 0: "Holding the training rows fixed leaves the transformer's 8-to-16-week gain unchanged within seed noise." E wholly below 0: "Part of the gain comes from the larger, colder training set." E wholly above 0: "Holding the training rows fixed raises the transformer's 8-to-16-week gain", with no reading of why. D_int reaching 0 or below: "With training rows fixed, the transformer's 8-to-16-week gain isn't shown at three seeds", and the eligibility answer then rests on the weekly ridge probe, which already trains both windows on one training set.
  • The int splits are 14.4% smaller in both windows, so E mixes who trains with how many train; D_int compares the two windows at equal size. Beside D_int and E we'll note that the store field was recoded at its source in December 2024, and two of the three 16-week test windows reach back before that date.
  • First the scorer must replay the paper's seed-42 matched 8-to-16 gain from its recorded predictions to 4 decimals, or nothing is read. An arm with fewer than three seeds in at the read voids the check.

5. The 32-week extract's early boundary, transformer half

  • In the 32-week extract, add events start only on Oct 30, 2023. For five of the nine training forecast weeks, some of the weeks that 32 adds over 24 fall before that date (55.6% of training rows); at test none do. A model could learn to use those far weeks less, which would make our 24-to-32 gain an understatement.
  • The 24- and 32-week models retrained three ways, seeds 42, 123, 456: on the full training split; "clean", only the four forecast weeks (June to September 2024) whose 32 weeks all fall after the boundary; and a size control that keeps the same share of every forecast week's rows, so it matches clean's size but keeps the truncated rows.
  • Scored on the 351,174-row matched cohort where the paper's 24-to-32 gain was read. D = the matched 24-to-32 gain, d = D(clean) − D(size control) per seed, and the read is the mean of d.
  • Wholly above 0: "Removing the truncated forecast weeks raises the 24-to-32-week gain beyond the size effect (or the later season does)." Wholly below 0: "Removing them lowers it beyond the size effect." Covering 0: "Removing them doesn't move the 24-to-32-week gain beyond the size effect." We'll also say whether the clean gain sits nearer our saturating prediction (+0.012) or the log-linear one (+0.041), which were made for the paper's inputs.
  • The clean weeks are also the latest four, so clean changes the season mix too, and the refit can't separate the truncated weeks from the later season. The full and size-control gains are shown, not read.
  • A missing clean or size-control run drops its seed, and fewer than two seeds void the check. One full-split run (24 weeks, seed 123) was preempted and resumed within its run in a way our resume check can't grade. Full-split runs enter only the descriptions, so its gain is shown as not checked, and the full-split mean is over the other two seeds.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.