- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
Summary
The authors analyze data from customer event streams from a single consumer platform to evaluate how the following components affect predictive accuracy:
- Dataset size (number of customers)
- Dataset size (window length)
- Model size
- Model architecture
Specifically, the authors change these elements one at a time in an attempt to determine which carries the largest impact on model performance. This leads to the claim that the number of time steps (the window length) is the largest “lever” of the group. This claim is only justified with respect to the specific dataset that the authors use (along with a short external validity check on one dataset, which does not replicate the findings), which is left unidentified and only described as “customer event streams from a single consumer platform, exported as fixed cohort carves”. No code or data is given, and thus there is no avenue for replicating the one empirical claim the paper rests on. I must also mention that the writing throughout the paper is incredibly unclear, disorganized, and relies on jargon terms which are not well-defined or commonly used in this literature. Each time a claim is made it carries a long list of qualifications using these jargon terms, and the details needed to piece together what was actually done are scattered across the paper with insufficient references or clarity as to how to piece it together.
Contributions
Under the most generous interpretation of this paper’s contributions, the authors have shown for the one “primary dataset” used, window length is the most impactful component of the four listed above—but nothing is provided with respect to the possible generalization of this finding or even the trustworthiness and/or replicability of this finding itself, other than the author’s statements. The authors claim to have pre-registered their study, but it seems their word must be taken at face value for this (which as I understand it, is not common practice for preregistration) since they only print their internal hash prefixes rather than following standard practice of having some sort of third-party registry/timestamp. The dataset is not adequately described for a full understanding of the setup used by the authors. Important elements of the dataset (w.r.t. external applicability) are never mentioned, such as what the “events” are. Given this specific application in customer event forecasting and the narrow empirical finding, the ceiling on the contributions of this paper are very low. The overall paper’s conclusion that “observation-window length belongs beside model selection as a first-class design axis” is already widely agreed upon in the literature. Lastly, their title finding that window length is crucial confounds their own limitation that they were unable to differentiate window length from calendar coverage.
Validation
Other than the findings on the “primary dataset”, the authors test for external validity on a dataset from Alibaba. The entirety of this external validity check, including its setup and results, occurs in one paragraph of main text with no accompanying figures or tables (I did not find the Appendix covering this to provide much clarity). The external check also does not replicate the author’s claims on their “primary dataset”, with R^2 very close to zero.
Critical strengths
The authors are clear about their claimed contributions and provide honest discussion of limitations.
Critical weaknesses
- Limited potential for impact. Taken at face value, the finding applies to only one dataset, which is not identified or described in adequate detail to understand the setting. It is also restricted to small models and does not check for joint changing of the “knobs” (i.e., maybe they interact in certain ways)? The authors mention that their controlled setup is their contribution, but changing one thing at a time is hardly a new idea.
- Claims are unable to be verified and do not have adequate support. Data and code are unavailable, which is crucial for a work whose main finding is an empirical result. Same goes for the preregistration, which is only self-verified and does not have any documentation that cannot be manipulated (thus not following the standard preregistration procedures). The external check only is able to replicate the sign, and even then has near-zero R^2. Calendar coverage confounds the window effect and takes away the identifiability of this titular claim.
- Writing quality. The explanations and descriptions in this paper are full of jargon and string together incredibly vague sentences which obfuscate their meaning. Notation is not deeply thought out - for example, there is one sentence in Section 2 where “M” means three different things at once (parameters in millions, window variable in weeks, and a label prefix). All notes on replicability and experimental setup are written out in a way that makes it incredibly difficult to follow and understand their setting, and required many readings to piece together part of what occurred.
Questions for authors
- Can you please provide more information regarding the primary dataset and why this is a fair/representative dataset to base the claim on/why such a finding on this dataset would generalize nicely?
- Do you have any documentation of pre-registration in a publicly-available registry or third-party verified source?
- Given the difference between the time horizons on the primary dataset and the Alibaba dataset, do you think these examples are comparable?
- Do you have any empirical results demonstrating how window length interacts with the other components that you inspect?
15 comments
Reply to Reviewer WhSj (2/2)
(2/2)
Q3, continued. Thank you for pressing on the horizon gap. The revision adds a public check on our time scale (App. P): Online Retail II (Chen, 2012, UCI Machine Learning Repository), a two-year log from one UK online retailer with many wholesale buyers. Setup: the paper's weekly ridge probe (§3.3) up to 24 weeks, 364-day target, one fixed set of 13,665 customer-forecast-week rows selected on the 8 weeks every compared window sees, reading rule fixed before any number existed. From 8 to 16 weeks R² goes 0.134 to 0.189, +0.055 [0.046, 0.062]; 24 weeks adds +0.068 [0.057, 0.079] over 8. Shuffling weeks 8 to 15 across customers gains nothing (−0.0013 [−0.0023, −0.0003]): the gain is each customer's own history, not more features. Returns diminish, but the sharp bend isn't shown: weeks 22 to 24 add 0.21 [0.13, 0.32] as much per week as weeks 8 to 16, and a bend needs that ratio's upper limit under a tenth. Unlike Taobao, every window fits and scores the same rows, and the weeks the 16-week window adds hold no November to January day, so the gain is neither more training rows nor added holiday weeks. Scope: one small retailer, ridge only, no level or gain compared with ours.
Q4. Does the window interact with the other factors? Three crossings are measured. Customers: the transformer's matched 8-to-16-week gain is +0.0939±0.0022 at ×1 and +0.0934±0.0005 at ×10 (3 seeds, §3.2), on disjoint cohorts with their own schedule points, so this bounds cohort, size and schedule together (App. I); ridge gains a bit more at ×10 (full split, +0.1050 vs +0.099). Model family: +0.070 to +0.107 across all eight families at ×1 (Table 7; the transformer in Table 4). Loss: under Huber, MSE and pinball the 3-seed full-split climb is +0.0701 to +0.0800 (§3.1) and +0.0863 to +0.0959 matched (seed 42, App. B); the loss moves levels by 0.03 to 0.05 and every loss climbs. You're right that we have no joint setting (a bigger model on more customers): capacity was probed only at ×1 with the small model's recipe, so we don't claim the ordering survives joint scaling (§5).
Code and data. We understand the concern, but we can't release either: the data is proprietary customer data, and company policy keeps the code internal. The paper documents the recipe (the recipe table, as corrected), the splits and direct-join disjointness audit (App. J), the matched cohort and its permutation control (§2.3, App. K), the bit-exactness ledger (App. H) and the registration digests (App. L). Every public check gives its windows, features, target and ridge setup (Apps. E and P). That doesn't replace a release.
Writing and notation. Agreed, and following your comment the revision uses its 10th page on it. In that sentence M did triple duty: window length, window-label prefix (M8 to M24) and tier-label suffix (1M/5M/10M). The revision's text and tables use W8 to W32 and ×1/×5/×10. Setup now comes before results (§2), the target, metric and matched cohort are each defined once, a notation paragraph pins down the labels (§2.4), and we replaced most of our internal shorthand in the main text with words; the rest is item 3 of our general reply.
Novelty. Agreed that window length mattering isn't new, and neither is changing one thing at a time; the intro says "our contribution is that comparison, not the question" (§1), and we should have made that clearer. The claim is narrower: on one population and target, with test customers and targets fixed, the window gain is about an order of magnitude above the scale, capacity and architecture contrasts for the sequence models (point estimates at the recipes tested), and through 16 weeks the raw comparison understates it. Related work (App. M) now covers the concurrent Dou et al. (2026).
Calendar coverage. Agreed, it's the main open limit (abstract, §5). We kept the title because its claim is about effect sizes at the recipes tested (§5), though we agree the confound leaves open how much of the window gain is history rather than calendar. The conclusion now says so: the window, confounded with calendar coverage (§5), is the dominant tested lever on this platform and target (§6). We ran the check Mhj2 asked for (App. O): the weekly ridge probe on held-out training customers, training forecast weeks whose windows never reach November or December against ones on the test weeks of the year. By its rule, written before the check computed any R², the gain persists outside the holiday season: the holiday-free 8-to-16-week gain is 0.72 [0.69, 0.75] of the test-calendar one. Returns diminish on both calendars, but the sharp 22 to 24 week bend, read on all held-out customers of those forecast weeks, shows only on the test calendar, so the revision scopes it there. Table 1's predictions and scores stand: they were made and scored on the test forecast weeks. App. O repeats it on the transformer, and it persists outside the holiday season. Summer and spring differ in more than the holidays.
The public checks in one table
The public ridge checks and the Taobao transformer in one table
Thank you for pointing out that our external check was one short paragraph with no table. Following your comment, and because the public evidence is now spread over several replies, here it is in one place (App. E and App. P of the revised PDF). A gain counts only if its 95% interval sits wholly above zero. The control shuffles the added weeks across customers.
| Dataset | Span | Rows scored | R² at the shorter window | Gain from the longer window [95%] | Shuffled-weeks control [95%] |
|---|---|---|---|---|---|
| Taobao, ridge | 9 days | 6,985 users | 0.0213 (3 days) | +0.0162 [+0.0090, +0.0229], 3 to 7 days | not run |
| Taobao, transformer (3 seeds) | 9 days | 6,985 users | −0.0073 (3 days, seed mean) | +0.047 [+0.035, +0.059], 3 to 7 days | not run |
| Online Retail II | 2 years | 13,665 | 0.134 | +0.055 [0.046, 0.062] | −0.0013 [−0.0023, −0.0003] |
| The Complete Journey | about 2 years | 6,524 | 0.4822 | +0.0315 [0.0245, 0.0384] | −0.0017 [−0.0030, −0.0004] |
| CDNOW | 18 months | 3,445 | 0.1188 | +0.0317 [0.0198, 0.0432] | −0.0038 [−0.0070, −0.0009] |
| H&M | 2 years | 3,164,024 | 0.1868 | +0.1024 [0.1017, 0.1030] | +0.000002 [−0.000006, +0.000010] |
The paper's transformer on Online Retail II, The Complete Journey and CDNOW is in App. P (Table 15) and in a follow-up below, each dataset read on its own.
The Taobao rows go from a 3-day to a 7-day window on a 2-day purchase-count target, which is about 69% zero, so their R² levels sit near zero (a negative level is below the cohort-mean predictor). The last four rows use the weekly ridge probe and a 364-day target, from 8 to 16 weeks, with the same rows fit and scored at every window, so the gain isn't more training rows. On Online Retail II and CDNOW the weeks the 16-week window adds hold no November to January day. On CDNOW the added weeks also carry when the customer joined (everyone joined in Q1 1997; the 16-week window holds the join date for 31% of rows). On CDNOW the gain also shows on the 2,375 rows whose 8- and 16-week windows fall wholly after joining, +0.0462 [0.0311, 0.0614]. On H&M the gain shows without holiday weeks in the added history: a refit on the five forecast Mondays whose 16-week history holds no November, December or January day, with each Monday's level taken out of the fit, gains +0.1054 [0.1045, 0.1062]. A first version without the per-Monday level failed its shuffled-weeks control, so this one adds it. Those five target years hold 82 to 194 days of the COVID period, and the per-Monday level doesn't capture how COVID changed buying differently across customers. Seven of H&M's eight target years run into the COVID period, which the pooled row doesn't separate. The Complete Journey's days carry no calendar dates, so it says nothing about season or holidays. None of this transfers a size to our data, and the table doesn't rank the datasets by gain (the Taobao rows use a different window and target). The cross-dataset comparisons we do make are the predictions in the Complete Journey, CDNOW and H&M follow-ups (App. P), each with its caveats.
Reply to Reviewer WhSj (1/2)
(1/2)
Thank you for the close read and the concrete questions. You're right about the writing and about "preregistered", and, following your review, the revision changes both. Section, table and appendix numbers are the revised PDF's, and the transformer's numbers are for the model as it ran (the categorical inputs were one constant code, the numeric fields went in unstandardized, and the rotary encoding was as implemented; see our general reply).
Q1. What is the dataset, and why is it a fair base? Each row is one customer at one forecast week (the week the forecast is made; the submission's anchor week). The input is that customer's events in the 8, 12, 16 or 24 weeks before it, capped at 750 events for the sequence models. Each event is one of four kinds (adding an item to a cart or list, a purchase, a review, or a subscription order) with a type code, five categorical attribute codes and two numeric value fields (recipe table), but no model read the codes: the transformer got one constant code, and the baselines don't take them. The target is the customer's total activity over the next 364 days, a real-valued total, not a count, modeled as log(1+max(0,y)), and exactly zero for 4.8% to 30.4% of rows by slice (§2.2). Splits are disjoint in customers and forecast weeks: train runs 2024-01-29 to 2024-09-09, validation 2024-11-04 to 2024-12-30, and test is three forecast weeks from 2025-02-24 to 2025-04-21. The 364-day target periods overlap in calendar time, so the splits are out-of-time in forecast weeks, not in labels. The three tiers (×1, ×5, ×10) are non-overlapping customer samples of one export, with test splits of about 480K, 2.4M and 4.8M rows (§2.1, App. J). We don't name the platform; the data is proprietary. This is now in §2 and App. H.
On representativeness, we agree that one primary dataset is a real limit, and we don't claim the data is representative: the primary evidence is one platform, one target, one export era (§5). But the result isn't tied to one customer sample or one model: the window gain holds in all three customer-disjoint tiers (+0.069 to +0.085 across sequence models and tiers, seed-42 endpoints; §3.5), it shows up in all eight model families (Q4), and for ridge the bottom three activity deciles carry 69.8% of the matched 8-to-16-week error reduction (§3.2). All of it shares the three test forecast weeks, so none of it touches the calendar point below. A history gain also shows up for ridge and our transformer on all five public datasets we ran, four of them on our own time scale, and none was chosen by its result (Q3 and our follow-ups). Its size on other platforms is open. Scope: every row of the ×1 matched cohorts has an event in the 8 weeks before its forecast week (§2.3), so the matched gain is for recently active customers. The rows the cohort drops are mostly lapsed (at 16 weeks, 66.3% have no event in those 8 weeks and 30.4% have a zero target, against 4.8% matched; App. B).
Q2. Is there a public registry or third-party record? No, and we should have said so more plainly (App. L: "not deposits with an external registry, and not third-party timestamped"). So "preregistered" is now "internally registered". Each prediction and its falsifier were dated in an internal record before the deciding evaluations. The predictions, falsifiers and scored record are unchanged. A reader can still check each outcome against its printed falsifier, though not when it was committed: Table 1 says, in brief, what was already known at each commit time (App. N.2 adds that a matched ridge gain of +0.104, already over commitment (i)'s +0.05 bar, was on record when that bar was set), and the one fully blind prediction failed and is reported at the same volume as the hits. The window gain and the bounds don't depend on the record.
Q3. Are the public dataset and ours comparable? Taobao is comparable in direction, and loosely in shape, not in size or where the curve bends. Its windows are 3 to 7 days, its target is a 2-day purchase count that's about 69% zero, and levels sit near zero, so we never compare levels (§3.6, App. E). What's shared is the intervention: fix the target period, lengthen the history, score the same users. On 6,985 matched users ridge R² goes 0.0213, 0.0305, 0.0374, every paired difference excluding zero, and the three-seed transformer test agrees. What it can't close is the horizon gap: Taobao spans 9 days, and its longer windows admit more training users, so it doesn't separate more history from more training rows (§5). Our longer windows add up to 10% more, colder training customers too (Table 2); 10× the customers moves sequence-model R² by at most 0.006 (seed 42), far below the window gain, though that bound doesn't isolate them. You're right that it was thin on the page. §3.6 now also prints the four multi-year public gains (App. P), and App. E adds a results table (Table 10) to its extracts, entry rule, features, target, seeds and per-seed results.