- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper investigates when adding a 'feature cross' can improve a trained model, even when the model can already represent crosses. To make this more concrete: suppose you want to predict whether or not someone will pay back their loan. It might be worth adding a feature that records the combination of (1) how long someone has had a bank account and (2) their profession. Now, if you have already added these two features separately, the model can already represent this interaction, but this does not imply that it will learn this interaction from a finite dataset. The paper studies whether an additive correction to a predictor can help.
It first develops a general analysis of when adding a correction helps an already trained model . It does so through a second-order taylor expansion around the already trained model , which is exact for squared loss. When we add to the predictions of through an linear smoother, derives a condition for which this addition helps (under a fixed-design Gaussian model). It then focuses the analysis to feature crosses, showing that adding more combinations of features can hurt - when these combinations capture the same signal, adding more combinations increases the estimation noise.
The paper proposes a method called CROSSAUDIT, which fits corrections based on and adds an correction only when the lower confidence bound on improvement on the validation loss is positive. The experiments verify the theoretical claims in a synthetic settings, and a semi-synthetic setting using covariates from the Avazu and Criteo datasets. In these experiments, they investigate logistic regression, Multilayer perceptrons, and a model called FM (which I believe is a factorization machine).
Critical strengths
-
The research question is interesting: why can adding feature cross help when the model could learn these crosses? There is a trade-off here betwen (1) the capacity of the model to represent this feature cross, and (2) its ability to estimate the usefulness of this cross from data. By studying the residuals of a trained model, the authors provide a concrete way to address this question.
-
The analysis provides useful intuitions for when a cross can help. In particular, the trade-off between (1) adding more features and thus estimation noise (increasing variance) and (2) adding information that is useful for the predictions (decreasing bias). The comparison between atomic and hierarchical estimators illustrates that this depends on the training procedure and the representation.
Critical weaknesses
-
I found this paper extremely hard to read and understand. Below, I give several concrete examples to illustrate the broader presentation problem. My intention is to offer constructive feedback that I hope will help the authors make the paper accessible to readers interested in feature crosses.
- The abstract introduces many technical terms, which are not necessarily obvious or expected to be known by interested readers. Examples include (but are not limited to) “post-selection certification,” “sufficient exposure–cost frontier,” and “nested Gaussian projection learners”. Instead, it would be good to use plain English and explain that the method fits candidate feature crosses and checks whether they improve validation loss.
- In a similar vein, the terms “positive certificate,” “the refinement effect,” “atomic representations,” and “main-effect repairs” appear in the abstract. These terms become (more, although not completely) understandable later but they make the abstract difficult to follow.
- In the introduction, it is not initially clear what is meant with “the update representation” and “a candidate repair to a particular predictor”. Similarly, it is not clear what is meant with “An atomic table assigns one coefficient to each joint state”.
- As another example, “These oracle quantities remain graded under dependent covariates” (Section 3.2) - what is meant with graded? what dependent covariates? This list is by no means exhaustive but serves to illustrate a broader point.
-
The theoretical results need more intuition and explanation. Sections 2–3 study a correction to a trained model; their results are not specific to feature crosses. This is not generally clear, and I would encourage the authors to state the structure of their argument more explicitly. As another example, the transition from studying the expected risk to a fixed-Gaussian design is very abrupt - this design needs to be introduced in much more detail.
-
Some of the theoretical results do not need the current degree of emphasis. Lemma 2.1 is simply a Taylor expansion - which is fine to use, but does not need this much attention in the paper. I believe this does not aid with exposing the theory - I believe it would be better to have a more concise derivation and then instead spend much more words on explaining what these results mean. Similarly, Proposition 3.1 is a standard quadratic bias–variance calculation. This does not invalidate the analysis, but it distracts from the main theoretical contributions of the paper.
-
My primary concern for the CROSSAUDIT method is that it is not compared to simply (1) estimating the correction on training data and (2) selecting the correction based on its performance on a validation set (e.g. via cross-validation). I appreciate that the paper develops a guarantee that, under certain conditions, adding the correction reduces the expected loss. However, I believe a similar guarantee could be obtained by calculating (standard) confidence intervals for the validation loss. The paper does compare different confidence bounds in a synthetic experiment, but it is not clear what the practical advantage of CROSSAUDIT is over simpler procedures.
-
Relatedly, the five-way split used in CROSSAUDIT seems quite demanding for a method motivated by finite-data limitations (as then the model will not learn the feature cross). The reported CTR experiments use 200K rows, which seems to me like quite a lot. Experiments varying the sample size and allocation along the five splits would help establish when the procedure is useful.
-
The related-work part is way too short: I would encourage the authors to discuss more explicitly how their approach relates to fitting corrections to model residuals (e.g. gradient boosting), the bias–variance trade-off when adding features, and methods for selecting feature crosses.
Questions for authors
Why does the MLP benefit more from the corrections than the other models? Since it also has a higher loss before adding the corrections, could this be because it has not been trained sufficiently? It would be useful to see whether adding the corrections has a similar effect when we train the MLP for longer.
2 comments
I appreciate the effort of the authors. I will go over each original concern, and briefly discuss the extent to which it has been addressed.
In terms of presentation; while I appreciate the authors attempt to improve the presentation, I believe the paper remains hard to understand. While the device-country example helps, and the writing has improved, I still think many terms used are hard to understand. Not too continue nitpicking, but more to illustrate; let's just take line 40-41 from the manuscript:
We ask whether an additive update , estimated from a finite fitting sample, reduces risk and whether independent validation can support that claim
What is an additive update? what is a 'finite fitting sample'? what is 'independent validation'?
I believe the paper needs to adopt much more 'plain English' when describing these things. To illustrate; I believe line 40-41 basically describe; 'We ask whether adding a correction based on a feature cross (e.g. interactions of device type and country) can help with performance of the model, and whether we can estimate the usefulness of this correction can be established using a validation set'.
This example from line 40-41 is not the only one but I am using it to describe a broader problem I have with the paper.
In terms of the comparison to selection based on validation; I understand that what the authors are doing is more sophisticated than simply selecting based on some validation criterion. I have a couple of outstanding questions regarding the comparison;
- Presumably, you can use more samples when using plain selection based on the validation set (no five-fold split is needed, as for your procedure). Is this taken into account?
- 'Validation-mean selection yields larger observed D4 gains' does this mean it is better?
I appreciate the authors adding an experiment where they vary the validation split, but again have genuine trouble understanding what the result of this; in plain english, if you vary the size of the allocation across the the five splits, do the results still hold? when yes/no?
I also appreciate the authors redoing the MLP experiments but I do share the concern of other reviewers that it remains undertuned.
Overall, the main concern I have against acceptance of the paper still holds (clarity of writing), hence why I retain my score.
Response to Reviewer X5ZT
Thank you for the detailed presentation and methodology suggestions.
Presentation. We rewrote the abstract and introduction around a device–country example: a model may represent this interaction but lack enough observations in individual cells to estimate a reliable correction in the revised paper. We define atomic tables, main-effect repairs, and positive certificates where first introduced. Sections 2–3 present the general analysis of corrections to a fitted predictor, while Section 4 specializes it to feature crosses. We shortened the Taylor and quadratic derivations, explain the fixed-design Gaussian model and dependent-covariate exposure, and expand the connections to boosting, bias–variance, interaction screening, hierarchical sharing, and held-out risk control.
1. Comparison with simpler validation procedures. CROSSAUDIT combines pre-validation fitting and bounds, simultaneous inference across candidates, and an explicit null decision. We compared it with validation-mean selection, split-validation (select-then-certify), and simultaneous Hoeffding bounds using the same fitted update family. Validation-mean selection chooses the highest D3 mean gain and selects a non-null update in every run. Split validation selects on the first 15K D3 observations and certifies the locked choice on the remaining 15K; it obtains 0/3 certificates on each dataset.
For the extended MLP, the full-30K-D3 comparison is shown below. Certificate-based methods use joint-eight correction; validation-mean selection is reported descriptively. Entries give certificate counts and mean null-inclusive D4 log-loss gains:
Validation-mean selection yields larger observed D4 gains, while empirical Bernstein supplies simultaneous positive-gain certificates for some selected updates; neither tested Hoeffding variant certifies an update in these runs. The EB/Hoeffding comparison reuses the same fitted updates, D2-derived bounds, and D3 gains. For gains in
−Bj,Bj, the one-sided Hoeffding penalty is B_j√(2log(M/δ)/n), and EB uses L=log(2M/δ). Per-family, joint-two, and joint-eight use M=H, 2H, and 8H (four checkpoints × permissive/shuffled families), respectively, at δ=0.1 within each dataset–seed. Families select separately under each simultaneous penalty. Per-family EB certifies 3/3 Avazu and 2/3 Criteo runs, while both Hoeffding variants certify 0/3 in either dataset.
The independent search stress test (Figure 2/Table 5) provides complementary error-rate evidence: over 1,000 repetitions per setting with n=3,000 and δ=0.05, naive maximization has FWER 1.000, while both simultaneous methods have zero observed false certificates under the strict null (95% exact-binomial upper bound ≈0.0030). With one gain-0.02 candidate among H=1,000, EB detects it in 30.8% of repetitions versus 11.0% for Hoeffding. Together, the controlled and CTR comparisons show how simultaneous certification and variance-adaptive bounds affect the decision to deploy a correction.
2. Validation budget and split roles. Holding D0/D1/D2/D4 fixed, we evaluated prespecified 10K/20K/30K prefixes of D3. Under per-family correction, extended-MLP certificates are 0/3, 0/3, 3/3 on Avazu (null-inclusive gains 0, 0, 0.000828) and 0/3, 0/3, 2/3 on Criteo (gains 0, 0, 0.000235). The 30K endpoint uses per-family correction; the table above uses joint-eight. D0 trains the checkpoint, D1 forms candidates, D2 fits corrections, and D3 selects and certifies. D4 provides independent reporting and is not needed for CROSSAUDIT itself. The prefix experiment isolates how validation budget influences certification; optimizing the full allocation across splits is a separate design question.
3. MLP optimization sensitivity. We increased its maximum training budget from 30 to 100 epochs and early-stopping patience from 5 to 15, with D0-only learning-rate selection and scheduling. All six runs chose initial LR 0.0003; the selected extended refits used 13–14 epochs on Avazu and 9–10 on Criteo. Mean base log-loss improved 0.409673 → 0.408322 and 0.484401 → 0.479195, and AUC improved 0.738309 → 0.739421 and 0.760586 → 0.763125 (Avazu/Criteo). Remaining joint-eight null-inclusive pair gains decreased from 0.001052 → 0.000560 and 0.000523 → 0.000082, with 3/3 and 1/3 extended-model certificates. Appendix I gives the training schedules, validation comparisons, candidate manifests, selection locks, and D4 results.