- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes OMID, a method for managing overfitting in deep networks with large embedding tables. Its central motivation is that small embedding weights do not necessarily imply low influence: downstream layers can amplify them, weakening the protection provided by embedding L2 regularization. OMID adds a Fisher-information-based sensitivity correction to the training objective and learns feature-specific regularization strengths during training. The authors connect this correction to hierarchical likelihood and interpret it as controlling effective model complexity. Experiments on synthetic data and Criteo show improved predictive performance and more stable training than the reported L2 baseline.
Critical strengths
-
This approach includes a statistically motivated approach. Connecting sensitivity regularization with hierarchical likelihood and learned prior variances provides a useful perspective on adaptive regularization. The discussion of minibatch bias and diagonal curvature approximations also identifies important implementation challenges.
-
The paper addresses an important research problem. Selecting an appropriate early-stopping point can be challenging and often requires monitoring a held-out validation set. A method that reduces this dependence could offer practical value.
-
The empirical results demonstrate the effectiveness of the proposed method. The reported improvements in predictive performance and training stability suggest that OMID can mitigate overfitting in the evaluated settings.
Critical weaknesses
-
This paper should include sufficient citations for established methods and background claims. For example, the discussion of dropout, L2 regularization, early stopping in the introduction, and the work with the third term on page 2 should cite foundational or representative studies. Adding these references would better contextualize the proposed method within the existing literature.
-
Limited experimental diversity. The evaluation is restricted to synthetic data and one real-world dataset, Criteo, using primarily embedding-based MLP architectures. Although Avazu is mentioned, no quantitative results are provided. Broader evaluation across datasets and substantially different architectures is needed to support the method’s general applicability.
-
The motivation illustrated in Figure 1 is incomplete. The scaling argument assumes that L2 regularization penalizes only the embeddings: shrinking them while amplifying downstream weights preserves predictions and reduces the embedding penalty. However, L2 regularization is commonly applied to both embedding and dense-layer weights, in which case downstream amplification also incurs a penalty. The authors should clarify when this failure mode persists under joint regularization and demonstrate why the proposed correction offers additional protection.
-
The practical advantage of avoiding a validation set is insufficiently established. When training data are abundant, reserving a small subset for validation may have little impact on model performance. Thus, eliminating validation-based tuning alone does not establish superiority over conventional approaches.
-
A comparison with a standard validation-based pipeline is needed. The authors should compare OMID’s final-epoch performance against a baseline whose regularization strength and stopping epoch are selected using a held-out validation set, with both evaluated on the same untouched test set. This would establish whether OMID offers an advantage over conventional tuning and early stopping.
Questions for authors
Please see the weaknesses.
Thank you for a careful and constructive review. Each of the five points changed the paper, and the fifth gave it the comparison it most needed. The revision is a major one, made to address the reviews, and we believe it addresses each of your points. If time is short, these are the places most relevant to them:
The points below follow the order of your review.
1. Citations. The bibliography goes from 9 to 36 entries. Weight decay, dropout, early stopping, the third term's source in the hierarchical likelihood (Lee and Nelder; REML) and the evidence framework that uses the same log-determinant (Immer et al., 2021) are now cited (Sec. 1 and Related work).
2. Experimental diversity. Avazu now has numbers (App. C, Table 6, page 16): 8M rows, three seeds, L2 bracketed and early-stopped on validation 0.7627 AUC / 0.3894 log-loss; OMID at its last epoch 0.7681 / 0.3867, ahead on every seed. Appendix C adds DCNv2 on Criteo (OMID ahead at every size, one seed each). The largest addition is a benchmark that is not ours: the one-epoch collapse of Zhang et al. (CIKM 2022) on Amazon Books, with the authors' released code and preprocessing. That code peaks at 0.742 AUC and collapses; OMID on the same rows ends at 0.764 after 25 epochs (Sec. 4.6 and Table 2, pages 8 to 9; App. A, page 13). On "substantially different architectures": the revision states its scope as embedding-dominated networks and gives the reason in Sec. 2; normalized architectures are left to future work.
3. Figure 1 under joint L2. In every experiment both arms already carry a tuned penalty on the dense weights, and the collapse persists; the revision adds the analysis (Sec. 3, page 5) and a sweep (App. D, pages 16 to 17). A dense penalty pins the overall scale of the pair (e, w), but it is paid once and shared by every row of the table, so it cannot tell a row supported by ten examples from one supported by a hundred thousand. The OMID term is per entry and grows with the evidence behind it. On the known-truth example of Figure 2, with the dense penalty swept over two decades in three independent worlds, the L2 turn persists at every penalty but the strongest, and goes there only for the model that matches the generating process; OMID ends within 0.02 of its best at every dense penalty with nothing tuned.
4. The practical advantage of avoiding a validation set. Agreed: when data are abundant a held-out split costs little, and the paper should not rest on the data saved. The revision makes the claim that the evidence supports: the model that early stopping selects is worse than the model OMID reaches at a fixed budget (point 5), and training can continue far past the baseline's stop. The abstract now says what is eliminated, the search over the penalty strength and the stopping rule, not that validation data leave the pipeline. One setting of OMID itself remains, the discount k of the diagonal approximation: it is 1 on Criteo and Avazu, where nothing is chosen, and is chosen on validation on the one-epoch and flooding benchmarks.
5. The standard validation-based pipeline. Done as you describe; it is now Table 4 (App. B.2, page 14). Criteo_x1 at 3.3M and 33M rows, three seeds each: L2 with its prior scale chosen from a seven-value bracket and its stopping epoch chosen on the official validation split, scored on the untouched test set; OMID at a fixed 40-pass budget with nothing selected. At 3.3M: L2 0.7934 AUC / 0.4564 log-loss, OMID 0.7972 / 0.4530. At 33M: L2 0.8076 / 0.4439, OMID 0.8127 / 0.4390. OMID is ahead on every seed at both sizes. The same protocol runs on Avazu (App. C) and on the Amazon Books benchmark (App. A, Table 3). We also report the limit: continued to 100 passes, OMID on the tables alone drifts down from 0.7972 to 0.7885; with the same term on the dense layers as well, the drift is gone (0.799 at pass 100; App. B).
Thank you again for the time and care in the review. If any further comparison would raise your confidence in the assessment, we have the protocol in place and will run it within the discussion period.