- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper observes that several RelBench regression targets are defined by SQL of the form "sum a value column over future events joined to the entity", and that when the value column is a static per-entity unit (a price) the target factorises as price × count, with the price known at query time. TASK-ALGEBRA is a small SQLGlot-based compiler that detects this pattern in the task manifest, trains the model on the count (or on a price-scaled residual over an anchor) and multiplies back. Because the monetary absolute error is the price times the count error, a monetary MAE objective becomes a price-weighted count MAE, implemented with sample weights. A proposition shows that under a uniform price and a count whose conditional median does not depend on price, a constant count head is Bayes-optimal for MAE while any piecewise-constant monetary predictor with finitely many pieces has a positive excess error. Experiments on rel-amazon item-ltv with a CatBoost head report a mean MAE reduction of about 2.4 over four development origins (Table 1), followed by controls on tree size, CatBoost's quantile tolerance, a residual MLP, a history-selected population, scale transport and price trimming, and an H&M item-sales experiment.
Critical strengths
- The observation is correct and cleanly stated: if the target is price × count and the price is known, predicting the count and multiplying back is the better-conditioned problem, and the loss correspondence is exact rather than heuristic.
- Both heads keep the price as an input feature, which avoids the classic mistake of regressing only on the ratio.
- The compiler's refusal checks and the replay checks against raw history are sensible engineering, and the scale-transport experiment is a nice sanity check that the factorised head is equivariant to a change of currency.
- The authors are forthright about limitations (the conclusion itself says that broader baselines, multiple seeds and official validation/test evaluation remain necessary), and the finding that CatBoost's default quantile tolerance does not scale with the target is genuinely useful, even though it undercuts the paper's own story.
Critical weaknesses
The idea is standard offset / exposure modelling presented as new, the theoretical result is a classical approximation rate whose bound is negligible for the trained models, and the empirical claim is made on training-period windows with no official split, no published baseline, one seed, and a set of controls that largely erase the gain. The authors are candid about most of these gaps, which I respect, but candour about a missing experimental programme is not a substitute for it.
1. This is offset / exposure modelling, and the paper does not position it as such. Fitting a rate and multiplying by a known per-row scale is the textbook GLM offset, and it is the standard device in actuarial pricing, where the exposure-weighted deviance is exactly the weighted loss derived here. It has been done with boosted trees for years (Henckaerts et al. 2021; TDboost), every major GBDT library exposes it directly (init_score in LightGBM, baseline in CatBoost, base_margin in XGBoost), the Poisson and Tweedie deviances satisfy the same scaling identity, DeepAR normalises each series by a known item scale and weights accordingly, and the M5 competition was explicitly "predict units, evaluate with dollar-weighted error". None of this is cited. The related-work paragraph on known scales cites two papers, and one of them (CANN) is precisely the anchor-plus-residual head used here. What is actually new is deriving the scale automatically from the task SQL, a parser that currently accepts one aggregation pattern on three hand-written manifests, and the identity-link MAE case, which is one line of algebra. The paper needs to say "offset modelling applied to RelBench" and then argue why that is an ICLR contribution.
2. The proposition is a known approximation rate and does not explain the experiments. That a piecewise-constant function with L pieces approximates a linear one with error of order 1/L is classical, and the squared-error version appears in existing tree lower bounds (Tan et al. 2022). More importantly the bound does not bite: CatBoost quantises each feature into a few hundred bins, so the whole ensemble is piecewise constant in price with a bounded number of pieces, and the resulting bound is a few cents, two orders of magnitude below the gain claimed. There is a deeper problem. Since price is an input feature, the conditional median of the monetary target is simply price times the conditional median of the count, so both heads target the same function and the weighted loss has the same population minimiser as the monetary loss. Any gain is therefore a finite-capacity or optimiser effect, which is consistent with the gain shrinking for the larger ensemble in Table 3. If the story is "trees cannot represent the linear factor", the honest baseline is a tree with linear leaves (LightGBM linear_tree, Cubist), which needs no knowledge of the task SQL. It is not run.
3. The evaluation never touches the official validation or test split and never compares to a published RelBench number. All four origins in Table 1 are the last four official train timestamps, and Appendix B says the official validation/test outcomes are not used in this draft. The same windows serve for development and for reporting. The paper cites RelBench, RDL, KumoRFM-2 and others but reports none of their item-ltv or item-sales numbers, and does not run the LightGBM baseline that ships with the benchmark. Those numbers exist, and the recent systems put the item-ltv test bar at roughly 41–42, so a 42 on training-period windows cannot be placed against anything. RDL's own validation-to-test gap on this task is about twice the headline gain, so I have no idea whether the gain survives a period shift. RelBench scores the masked test labels through its own evaluator, so not using the frozen split is a choice, not a constraint. For an empirical paper on a public benchmark this is disqualifying on its own.
4. The headline gain is fragile under the paper's own controls, and the official roster helps the factor head. With the quantile tolerance set to zero (the numerically clean setting) the October gain flips sign and the two-origin mean has an interval covering zero; trimming the top 1% of prices removes most of the gain; for the residual MLP the factor head is worse; on the history-selected population, the only roster that can actually be built at prediction time, a plain direct MLP beats the factor tree at both origins; and on H&M unit normalisation adds nothing over the ordinary residual baseline. Read together, the gain is real only for CatBoost at default tolerance on the full price tail. Separately, the official item-ltv roster only contains products with at least one future review, so every label is at least the price. The count head only has to learn "at least one", which a constant already does, while the direct tree has to learn a floor that varies with price. Part of the gain is therefore an artefact of a roster that requires future knowledge. I would want the gain broken down by price decile before believing anything about the mechanism.
Questions for authors
- What are the official RelBench validation and test MAEs of both heads on item-ltv and item-sales, in one table with the published LightGBM, RDL and KumoRFM-2 numbers?
- Please report Tables 1 and 3 with five seeds per arm, with the tolerance set to zero as the primary configuration, matched rows in both arms, and per-head tuning on a held-out origin.
- How does the method differ from CatBoost with a Poisson/Tweedie objective and log-price as
baseline, or from a linear-leaf tree, scored by monetary MAE? Please run both on the same features. - Since price is an input feature, do you agree that the weighted count loss and the monetary loss have the same population minimiser? If so, what is the proposition explaining, and what is its numerical value in dollars for the trained ensemble?