acceptodds
Under review as a conference paper at ICLR 2027

Overfitting Management in Deep Networks: A Sensitivity Correction to the Training Loss

Abstract

Overfitting in deep neural networks (DNNs) is handled by a combination of techniques: an L2 penalty and dropout, each with its own hyperparameter, and early stopping on a held-out validation set. All are set from outside the fit: tuned or monitored on held-out data rather than estimated within it. In classical statistics, by contrast, the hyperparameter controlling L2 can be learned from the data. Empirical Bayes estimates it as a variance component while fitting the model, and the penalized objective is minimized to convergence with no stopping rule. This paper pursues the same goal in a DNN: estimating from the training data, within SGD, the strength at which a fit neither overfits nor underfits. The classical construction does not transfer unchanged, and the adaptation and extension it requires are the substance of the work. The difficulty is that the very expressive power that makes a DNN useful enables it to hide overfitting. A DNN can satisfy an L2 penalty by holding a weight small while amplifying the gain applied downstream, so influence grows while magnitude does not, and the penalty stops binding. The same freedom opens the door to a kind of overfitting that neither the penalty nor its classical estimate can see, and that is why in DNNs L2 is left as a hyperparameter and monitoring is needed to decide when to stop. We add a sensitivity correction to the L2 penalty, pricing what that penalty cannot see: the sensitivity the network grants itself. The adjusted loss has its roots in the hierarchical likelihood of Lee and Nelder (in a linear mixed model it reduces exactly to REML, so the correction is an extension of REML to DNNs). Regularization strength is then estimated during ordinary training, and no validation set is needed to set it or to stop training. The quantity controlled is the model’s effective degrees of freedom, the capacity a fit uses rather than the parameters it owns. We call the approach OMID, for Overfitting Management in Deep Networks, and focus first on embedding-dominated architectures. We test OMID in simulation, where the data-generating process is known, and on the public Criteo and Avazu click-through-rate benchmarks. Both held-out loss and ranking accuracy improve. Under tuned L2 with early stopping, held-out loss reaches its best within a few epochs and rises sharply thereafter, and that best is not the best available. With the correction, loss stops turning: it keeps falling as training continues and ends below that value at every data size tested, with the largest gains where data is scarcest. Training and held-out performance no longer diverge, and further passes bring diminishing returns rather than damage. Expected calibration error drops by factors of three to twenty. We close with the extension now under way, to sequence models and large language models, where transformer and self-attention architectures change how sensitivity must be addressed.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
overfitting, regularization, empirical Bayes, recommendation systems, click-through rate prediction, memorization, generalization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_sKz4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes OMID, a method for managing overfitting in deep networks with large embedding tables. Its central motivation is that small embedding weights do not necessarily imply low influence: downstream layers can amplify them, weakening the protection provided by embedding L2 regularization. OMID adds a Fisher-information-based sensitivity correction to the training objective and learns feature-specific regularization strengths during training. The authors connect this correction to hierarchical likelihood and interpret it as controlling effective model complexity. Experiments on synthetic data and Criteo show improved predictive performance and more stable training than the reported L2 baseline.

Critical strengths
  1. This approach includes a statistically motivated approach. Connecting sensitivity regularization with hierarchical likelihood and learned prior variances provides a useful perspective on adaptive regularization. The discussion of minibatch bias and diagonal curvature approximations also identifies important implementation challenges.

  2. The paper addresses an important research problem. Selecting an appropriate early-stopping point can be challenging and often requires monitoring a held-out validation set. A method that reduces this dependence could offer practical value.

  3. The empirical results demonstrate the effectiveness of the proposed method. The reported improvements in predictive performance and training stability suggest that OMID can mitigate overfitting in the evaluated settings.

Critical weaknesses
  1. This paper should include sufficient citations for established methods and background claims. For example, the discussion of dropout, L2 regularization, early stopping in the introduction, and the work with the third term on page 2 should cite foundational or representative studies. Adding these references would better contextualize the proposed method within the existing literature.

  2. Limited experimental diversity. The evaluation is restricted to synthetic data and one real-world dataset, Criteo, using primarily embedding-based MLP architectures. Although Avazu is mentioned, no quantitative results are provided. Broader evaluation across datasets and substantially different architectures is needed to support the method’s general applicability.

  3. The motivation illustrated in Figure 1 is incomplete. The scaling argument assumes that L2 regularization penalizes only the embeddings: shrinking them while amplifying downstream weights preserves predictions and reduces the embedding penalty. However, L2 regularization is commonly applied to both embedding and dense-layer weights, in which case downstream amplification also incurs a penalty. The authors should clarify when this failure mode persists under joint regularization and demonstrate why the proposed correction offers additional protection.

  4. The practical advantage of avoiding a validation set is insufficiently established. When training data are abundant, reserving a small subset for validation may have little impact on model performance. Thus, eliminating validation-based tuning alone does not establish superiority over conventional approaches.

  5. A comparison with a standard validation-based pipeline is needed. The authors should compare OMID’s final-epoch performance against a baseline whose regularization strength and stopping epoch are selected using a held-out validation set, with both evaluated on the same untouched test set. This would establish whether OMID offers an advantage over conventional tuning and early stopping.

Questions for authors

Please see the weaknesses.

1 comment

Thank you for a careful and constructive review. Each of the five points changed the paper, and the fifth gave it the comparison it most needed. The revision is a major one, made to address the reviews, and we believe it addresses each of your points. If time is short, these are the places most relevant to them:

  • Point 1, citations: Related work, page 10; references, pages 11 to 12.
  • Point 2, further datasets and networks. The main addition: the one-epoch benchmark of Zhang et al. (CIKM 2022), who report the collapse of test performance after one epoch on production click models (Amazon Books): Sec. 4.6, Figure 3 and Table 2, pages 8 to 9. Also Avazu and DCNv2: App. C and Table 6, page 16.
  • Point 3, Figure 1 under joint L2: the new paragraph in Sec. 3, page 5; the sweep, App. D and Table 7, pages 16 to 17.
  • Point 4, what is claimed about validation data: Abstract, page 1.
  • Point 5, the validation-based pipeline: App. B.2 and Table 4, page 14; the same protocol on Amazon Books, Table 3, page 13.

The points below follow the order of your review.

1. Citations. The bibliography goes from 9 to 36 entries. Weight decay, dropout, early stopping, the third term's source in the hierarchical likelihood (Lee and Nelder; REML) and the evidence framework that uses the same log-determinant (Immer et al., 2021) are now cited (Sec. 1 and Related work).

2. Experimental diversity. Avazu now has numbers (App. C, Table 6, page 16): 8M rows, three seeds, L2 bracketed and early-stopped on validation 0.7627 AUC / 0.3894 log-loss; OMID at its last epoch 0.7681 / 0.3867, ahead on every seed. Appendix C adds DCNv2 on Criteo (OMID ahead at every size, one seed each). The largest addition is a benchmark that is not ours: the one-epoch collapse of Zhang et al. (CIKM 2022) on Amazon Books, with the authors' released code and preprocessing. That code peaks at 0.742 AUC and collapses; OMID on the same rows ends at 0.764 after 25 epochs (Sec. 4.6 and Table 2, pages 8 to 9; App. A, page 13). On "substantially different architectures": the revision states its scope as embedding-dominated networks and gives the reason in Sec. 2; normalized architectures are left to future work.

3. Figure 1 under joint L2. In every experiment both arms already carry a tuned penalty on the dense weights, and the collapse persists; the revision adds the analysis (Sec. 3, page 5) and a sweep (App. D, pages 16 to 17). A dense penalty pins the overall scale of the pair (e, w), but it is paid once and shared by every row of the table, so it cannot tell a row supported by ten examples from one supported by a hundred thousand. The OMID term is per entry and grows with the evidence behind it. On the known-truth example of Figure 2, with the dense penalty swept over two decades in three independent worlds, the L2 turn persists at every penalty but the strongest, and goes there only for the model that matches the generating process; OMID ends within 0.02 of its best at every dense penalty with nothing tuned.

4. The practical advantage of avoiding a validation set. Agreed: when data are abundant a held-out split costs little, and the paper should not rest on the data saved. The revision makes the claim that the evidence supports: the model that early stopping selects is worse than the model OMID reaches at a fixed budget (point 5), and training can continue far past the baseline's stop. The abstract now says what is eliminated, the search over the penalty strength and the stopping rule, not that validation data leave the pipeline. One setting of OMID itself remains, the discount k of the diagonal approximation: it is 1 on Criteo and Avazu, where nothing is chosen, and is chosen on validation on the one-epoch and flooding benchmarks.

5. The standard validation-based pipeline. Done as you describe; it is now Table 4 (App. B.2, page 14). Criteo_x1 at 3.3M and 33M rows, three seeds each: L2 with its prior scale chosen from a seven-value bracket and its stopping epoch chosen on the official validation split, scored on the untouched test set; OMID at a fixed 40-pass budget with nothing selected. At 3.3M: L2 0.7934 AUC / 0.4564 log-loss, OMID 0.7972 / 0.4530. At 33M: L2 0.8076 / 0.4439, OMID 0.8127 / 0.4390. OMID is ahead on every seed at both sizes. The same protocol runs on Avazu (App. C) and on the Amazon Books benchmark (App. A, Table 3). We also report the limit: continued to 100 passes, OMID on the tables alone drifts down from 0.7972 to 0.7885; with the same term on the dense layers as well, the drift is gone (0.799 at pass 100; App. B).

Thank you again for the time and care in the review. If any further comparison would raise your confidence in the assessment, we have the protocol in place and will run it within the discussion period.

Reviewer_wAKNOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a new training loss regularization term with the goal of controlling memorization and generalization intrinsically within the loss without requirement to extra validation set for hyperparameter tuning. In particular, a correction term is proposed to be added to the empirical training loss and the L2 penalty which is related to the sensitivity of the neural network output to embedding parameter changes. They call this learning algorithm OMID, and show empirically that it is able to outperform the loss + L2 penalty algorithm in a controlled setting.

Critical strengths
  1. Identifying the extended role of the correction term originally introduced in Lee et al. (2006) in deep neural networks is an interesting topic that could have nice practical implications. Any algorithm that introduces a regularization without hyper-parameter tuning is highly valuable in practice, particularly in those areas where gathering labeled validation sets is expensive or extremely difficult.
  2. It is interesting to see how well OMID is able to outperform the empirical loss + L2 penalty in Figure 2. When it is a controlled setting the proposed method outperforms the baseline algorithm without requiring any early stopping which is very interesting. This is further highlighted when the paper looks at data regimes where data is even more scarce.
Critical weaknesses
  1. The paper offers very limited contributions. The proposed algorithm is not novel. The provided numerical results are extremely limited. There is near zero evidence how well this proposed algorithm does compared to other competing algorithms. Empirical results on state of the art datasets and neural networks are missing. This makes it difficult to assess how well the algorithm is generalizable to other datasets or using other SOTA deep neural networks. Furthermore, it is not clear how well the algorithm performs compared to other competing algorithms and regularization terms.
  2. The paper is overall not easy to read and not well-written. Abstract is unnecessarily long and contains details which make it distracting. The introduction is also long, has some redundant background information, and also includes some statements which are problematic. For example: “ A model can be highly sensitive to a handful of examples, which is memorization rather than generalization”. This is not true. There are hard/difficult examples on the tail distribution which the ideal model should be sensitive to, but that does not contradict with the generalizability of the model.
Questions for authors

Please refer to the weaknesses. But the main question is how well the proposed algorithm performs compared to the competing regularization algorithms in sota empirical settings.

1 comment

Thank you for your review, and for the strengths you noted in the paper. Your two main concerns were that the method is not novel and that the evidence is too limited. We took both seriously. The revision is a major one, written to address them: it now says plainly what is new and what is not, and it compares OMID with competing regularizers on several datasets, as you asked. We believe it addresses each of your points. If time is short, these are the most relevant places:

  • The main addition: the one-epoch benchmark of Zhang et al. (CIKM 2022), who report the collapse of test performance after one epoch on production click models (Amazon Books): Sec. 4.6, Figure 3 and Table 2, pages 8 to 9.
  • What is new, stated as one missing link in a chain of known results: Introduction, pages 1 to 3.
  • The test of that claim, the prior-estimation-only arm: Table 2, page 9; also Table 4, page 14, and Table 6, page 16.
  • The sentence on memorization, corrected: Introduction, page 2.
  • Competing regularizers, each selected on validation: flooding, Table 1, page 7; L2, dropout, MEDA and prior estimation only, Table 4, page 14.
  • Further datasets and networks: Avazu and DCNv2, App. C, page 16.

The points below follow the order of your review.

1. "The proposed algorithm is not novel." The term itself is not ours, and the paper says so: it is the log-determinant of Lee and Nelder's adjusted profile likelihood, REML in the linear case. What is ours is the role it is given. In the classical model, and in the evidence framework (Immer et al., 2021), that term enters only the update of the prior variance, and the weights are trained on data loss plus L2. In OMID it is in the weights' objective, with a gradient through the curvature. The revised introduction gives the argument for that role as a chain of known results (Zhang et al., ICLR 2019; Efron, 1986; Lee and Nelder) with one missing link (Sec. 1 and 2).

That change of role is tested. A prior-estimation-only arm (the same diagonal Fisher and log-determinant as OMID, used to estimate the prior variance and detached from the weights; the role Immer et al. give the term) runs next to OMID on three datasets, three seeds each. One-epoch benchmark (Table 2): it peaks at 0.743 AUC at the end of epoch 1 and is at 0.722 at epoch 25, where OMID is at 0.764. Criteo 3.3M (Table 4): it peaks at 0.791 and collapses to 0.753; OMID is at 0.797. Avazu (Table 6): it falls to 0.733 to 0.735 by pass 40; OMID is at 0.768. The two roles are separated by data, not by argument.

2. The sentence on memorization. You are right, and the point matters to us: a model should be sensitive to the rare examples that carry real signal (Feldman, 2020), and nothing in our objective is meant to prevent that. What the paper means, and now says, is that a fit can satisfy the L2 penalty while its predictions covary with the noise in its own training labels, which is overfitting in Efron's (1986) sense. The distinction is not sensitivity itself but the evidence behind it, and that is what the OMID correction measures per entry: dfj=Hj/(Hj+1/τ2)\mathrm{df}_j = H_j/(H_j+1/\tau^2) is near one when the data determine an entry and near zero when the prior does. A rare row that the data pin down keeps its degrees of freedom (Sec. 1).

3. Writing. The abstract is about 40% shorter (about 270 words against 475). The introduction now opens with the recent work that recognizes the problem, and the classical background is cut back.

4. Your main question: competing regularizers in SOTA settings. The revision adds the following, with every baseline selected on a validation split and scored on the untouched test set (test AUC):

  • Criteo_x1 at 3.3M rows (App. B, Table 4): L2 0.7934, dropout 0.7889, MEDA-style re-initialization (Fan et al., 2024) 0.7838, prior estimation only 0.7910; OMID at the 40-pass budget, nothing chosen, 0.7972, ahead of every arm on every seed. At 33M rows: L2 0.8076, OMID 0.8127.
  • Flooding (Ishida et al., 2020) on its authors' own benchmark (Sec. 4.2, Table 1): OMID's test NLL is below flooding's in all 6 cells.
  • Avazu at 8M rows (App. C, Table 6): L2 0.7627, OMID 0.7681, ahead on every seed.
  • Amazon Books, the one-epoch benchmark of Zhang et al. (CIKM 2022), with the authors' released code and preprocessing (Sec. 4.6; App. A): best baseline stop 0.736, OMID 0.756, ahead on every seed.
  • DCNv2 with OMID on Criteo (App. C): ahead of the validation-stopped DCNv2 baseline at every size from 500k to 10M rows (one seed each).

We would welcome a pointer to any further regularizer you consider a necessary comparison. Thank you for pressing on both points; the paper is clearer about what it claims because of it.

Reviewer_YsMjOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes an approach loosley inspired by empirical Bayes to control overfitting of DNNs directly from the training loss by adding an additional term.

Critical strengths
  • Devising new ways to control overfitting, without relying on validation sets, is an interesting research avenue.
Critical weaknesses
  • The paper is missing relevant related work [1], both from the point of view of referencing and experimental comparison. Furthermore, it contains as few as 9 total references, which is extremely low for a conference paper.
  • The experimental benchsuite is minimal, with an experiment on synthetic data that could be useful to measure the ground truth distribution, but zero extension to even slightly more interesting (toy) datasets, e.g., MNIST, CIFAR10. This point particularly clashes with the overly broad title.
  • The paper doesn't provide justifications for why it focuses on categorical data (beginning of Section 2).
  • The writing clarity is very low. For example, the abstract is long and verbose, quite difficult to read and follow at times.
  • Some claims are not correct or are extremely superficial and trivial. For instance: (1) "Overfitting in deep neural networks (DNNs) is handled by a combination of techniques: an L2 penalty and dropout, each with its own hyperparameter, and early stopping on a held-out validation set.", this claim is overly restrictive; (2) "two of which every deep network already has.", referring to the two terms of the equation introduced in the introduction, not necessarily true; (3) "An L2 penalty therefore does not control how much influence an input feature has", a trivial statement for a paper under review at ICLR.

[1] Do We Need Zero Training Loss After Achieving Zero Training Error?, ICML 2020

Questions for authors

I do not have any particular question that would likely affect my assessment.

1 comment

Thank you for a direct and useful review. The revision is a major one, made specifically to address your feedback: the comparison with flooding, the longer bibliography, the stated focus and the corrected sentences are there because you asked for them. We believe it addresses each of your points. If time is short, these are the places to look:

  • The main addition: the one-epoch benchmark of Zhang et al. (CIKM 2022), who report the collapse of test performance after one epoch on production click models (Amazon Books): Sec. 4.6, Figure 3 and Table 2, pages 8 to 9.
  • Abstract, rewritten and about 40% shorter: page 1.
  • The three sentences you quoted, corrected: Introduction, pages 1 to 2.
  • Flooding [1] on its authors' own benchmark: Sec. 4.2 and Table 1, pages 6 to 7.
  • Why embedding tables, and the stated focus: Sec. 2, page 3.
  • 36 references in place of 9: Related work, page 10; references, pages 11 to 12.

The points below follow the order of your review.

1. Related work and flooding [1]. You are right on both counts. The bibliography goes from 9 to 36 entries, and flooding (Ishida et al., 2020) is now compared on its authors' own benchmark: their three synthetic sets at 1% and 5% label noise, their network, their 51-value grid for the flood level, ten paired trials per cell, against plain training, tuned L2 and OMID. With the setting and epoch chosen by validation NLL, OMID's test NLL is below flooding's in all 6 cells (by 0.03 to 0.12; ahead in 9 or 10 of the 10 paired trials per cell); without early stopping it is below in all 6 cells (by 0.08 to 0.16). (Sec. 4.2, Table 1; App. B.3.)

2. Benchmarks, MNIST/CIFAR-10, and the title. The request is a fair one. The OMID principle is not specific to embedding tables: the term applies to any weight of a network, and the revision shows it on dense layers in two places. The flooding benchmark above is a fully dense network with no tables, with OMID on every weight matrix (Table 1), and Appendix B puts the same term on the dense layers of the click model. This paper focuses on embedding tables because that is where these models overfit: each row is supported only by the examples that carry its level. Dense and convolutional networks at the scale of CIFAR-10 need their own numerical techniques, since every weight is touched by every example and the diagonal form used here over-counts (Sec. 4.4). Those techniques do not fit in this paper and are the subject of a follow-up publication, so here we state the focus (abstract, Sec. 2) rather than add these datasets. If you and the AC prefer, we will narrow the title to "Overfitting Management in Deep Networks with Embedding Tables: ...". Within that focus the main addition is a benchmark that is not ours: the one-epoch phenomenon of Zhang et al. (CIKM 2022) on Amazon Books, with the authors' released code and preprocessing. That code peaks at 0.742 test AUC at the end of epoch 1 and collapses; OMID on the same rows ends at 0.764 after 25 epochs without a turn. With every baseline chosen on a validation split: best baseline stop 0.736, OMID 0.756. (Sec. 4.6, Table 2; App. A to C.)

3. Why categorical data. Sec. 2 now writes the training loss over all parameters, dense and embedding, and says why the tables are the focus: every dense parameter is present in every example, while a row of a table is touched only by the examples carrying its level, so the tables are where these models overfit. Appendix B puts the same term on the dense layers as well.

4. Writing. The abstract is about 40% shorter (about 270 words against 475) and keeps its claims to those the evidence supports.

5. The three claims. (1) Agreed, too restrictive as written, and the sentence is gone; the introduction now opens with weight decay as the field has found it (Zhang et al., ICLR 2019; Li and Arora, ICLR 2020; Zhang et al., CIKM 2022). (2) Now: "two of which, the data loss and the L2 penalty, are standard in deep networks." (3) We agree it is elementary, and the revision no longer presents it as ours: it is attributed to Zhang et al. (ICLR 2019) and Dinh et al. (2017). What the paper adds is the consequence: if the same function can be written at many weight scales, the training data cannot locate the prior variance, so the classical estimate of the penalty strength fails, and the identified quantity is the effective degrees of freedom.

Soundness. The new comparisons use three seeds unless noted and validation-based selection for every baseline. OMID runs to a fixed budget with no stopping rule; its one remaining setting, the discount k, is 1 on Criteo and Avazu and is chosen on validation on the one-epoch and flooding benchmarks. We would be grateful to know which part of the argument or of the new evidence you find least convincing.

Thank you again; the paper is better for these points.

Comments on the paper

1 comment

We thank the three reviewers: the reviews were specific and constructive, and what is new in this paper is there because they asked for it. The revision is a major one, and we kindly ask that the paper be read in its revised form. The method is unchanged. What is new is the evidence that it is useful: we now report it on several more datasets, in particular the Amazon Books benchmark of Zhang et al. (CIKM 2022), which those authors used to study exactly this problem, overfitting beyond the first epoch. Every baseline is now selected on a validation split, and OMID is compared with competing regularizers (flooding, dropout and a prior-estimation-only arm). The abstract and introduction are rewritten, and the references go from 9 to 36. The revised PDF is uploaded; the lists below say where each change is, and the replies to each reviewer take the individual points.

The main addition is that benchmark, which is published and not ours. Zhang et al. (CIKM 2022) report that, on production click models, test performance collapses at the start of the second epoch, and neither weight decay nor dropout prevents it. We ran OMID on the Amazon Books benchmark of that paper, with the authors' released code and preprocessing. Where that code collapses after one epoch, OMID trains for 25 epochs without a turn and ends above that code's peak (item 1 below; Figure 3 and Table 2, page 9).

New experiments. Three seeds unless noted. Every baseline selects its strength and its stopping epoch on a validation split and is scored on the untouched test set; OMID runs to a fixed budget.

  1. A benchmark that is not ours (Sec. 4.6 and Table 2, pages 8 to 9; App. A, page 13). The one-epoch phenomenon of Zhang et al. (CIKM 2022) on Amazon Books, with the authors' released code and preprocessing. That code peaks at 0.742 test AUC at the end of epoch 1 and collapses; OMID on the same rows runs 25 epochs without a turn and ends at 0.764. With every baseline chosen on validation: best baseline stop 0.736, OMID 0.756, ahead on every seed.

  2. Flooding (Reviewer YsMj; Sec. 4.2 and Table 1, pages 6 to 7; App. B.3, page 15). On its authors' own benchmark (Ishida et al., 2020), ten paired trials per cell: OMID's test NLL is below flooding's in all 6 cells, both with validation-chosen stopping (by 0.03 to 0.12) and without early stopping (by 0.08 to 0.16).

  3. The validation pipeline and competing regularizers (Reviewers sKz4, wAKN; App. B and Table 4, page 14). Criteo_x1 at 3.3M and 33M rows: L2 over a bracket of prior scales, early-stopped on validation, reaches 0.7934 and 0.8076 AUC; OMID at a fixed 40-pass budget with nothing selected reaches 0.7972 and 0.8127, ahead on every seed. Dropout (0.789), MEDA-style re-initialization (0.784) and a prior-estimation-only arm (0.791; the same log-determinant used only to update the prior variance, its role in Immer et al., 2021) each peak below OMID and collapse afterwards.

  4. Avazu and DCNv2 (Reviewer sKz4; App. C and Table 6, page 16). Avazu at 8M rows: L2 at its validation-chosen scale and epoch 0.7627 AUC / 0.3894 log-loss; OMID at pass 40, 0.7681 / 0.3867, ahead on every seed. DCNv2 on Criteo: OMID ahead at every size from 500k to 10M rows (one seed each).

  5. Joint L2 (Reviewer sKz4; Sec. 3, page 5; App. D, pages 16 to 17). The analysis of what a penalty on the dense weights does and does not fix, with a known-truth sweep of the dense penalty.

Changes to the text.

  • Abstract: about 40% shorter (about 270 words against 475). It opens with weight decay as the field's own work has found it (Zhang et al., ICLR 2019; Li and Arora, ICLR 2020; Zhang et al., CIKM 2022). The calibration sentence had no table behind it and is withdrawn.
  • Introduction: the sentences Reviewers YsMj and wAKN quoted are corrected, and the contribution is stated as one missing link in a chain of known results.
  • Sec. 2 states the scope, embedding-dominated networks, and the reason for it.
  • 9 references become 36.
  • Main text: 10 pages, the tenth permitted during the discussion for new results. Appendices A to E follow the bibliography.

What we did not change: the method, the title, Figures 1-2 and the synthetic study. One setting remains, the discount k of the diagonal approximation: it is 1 on Criteo and Avazu, and is chosen on validation on the one-epoch and flooding benchmarks.

We are grateful for the reviewers' time and will answer any further question during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.