- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the one-epoch phenomenon in recommendation models, where additional training reduces training loss but worsens generalization. The authors propose self-influence asymmetry as an explanation: previously observed labels affect sparse embeddings, allowing the shared consumer network to exploit information that is unavailable when predicting fresh examples. The paper develops an exact scalar model and a local influence analysis to characterize this effect. Based on these observations, it introduces Uncertainty-Weighted Sensitivity Regularization (UWSR), which penalizes the consumer's sensitivity to uncertain embedding representations. Experiments on MovieLens-20M, Amazon Electronics, and Avazu compare UWSR with existing multi-epoch training methods and report improvements in test performance.
Critical strengths
- The paper provides a useful perspective on the one-epoch phenomenon.
The connection between repeated training, self-influence, and the prequential principle offers a meaningful explanation of why sparse embeddings may encourage overfitting. The scalar model provides an exact characterization under simplified assumptions, while the local influence analysis extends the intuition to neural recommendation models. This goes beyond treating the phenomenon as ordinary overfitting.
- The update-order intervention provides evidence for the proposed mechanism.
Section 3.3 directly examines the interaction between embedding updates and consumer updates. Table 1 shows that updating embeddings before the consumer substantially increases score sensitivity and worsens validation performance, while reversing the order produces much smaller changes. This intervention provides useful evidence that same-label self-inclusion can contribute to generalization degradation.
- The proposed method achieves competitive results against relevant baselines.
The evaluation includes MEDA, AdamAR, embedding dropout, and other training strategies. UWSR achieves clear improvements on Amazon Electronics and Avazu, with paired confidence intervals reported in Appendix D. The comparisons with uniform sensitivity regularization and expected-Fisher weighting also provide evidence that uncertainty weighting contributes to the observed gains.
Critical weaknesses
- The proposed mechanism is not sufficiently established as the primary explanation for multi-epoch generalization degradation.
The scalar analysis uses a simplified two-stage model, while the neural analysis relies on local approximations and row-wise effects. The update-order intervention in Section 3.3 shows that self-inclusion can induce sensitivity inflation, but it does not quantify how much of the degradation during ordinary multi-epoch training is caused by this mechanism. The intervention also changes the optimization sequence, making it difficult to isolate the effect of label reuse from other changes in training dynamics. The experiments support self-influence asymmetry as a contributing factor, but not necessarily as the dominant cause of the one-epoch phenomenon.
- The connection between the theoretical uncertainty measure and its practical approximation remains insufficiently validated.
The theoretical formulation relies on inverse row information derived from a local generalized Gauss-Newton model. In practice, UWSR uses normalized, row-wise accumulated squared gradients as an approximation. This approximation does not directly measure the covariance assumed in the analysis, and the paper does not establish how accurately it captures the uncertainty or self-influence of individual embedding rows. Although the expected-Fisher and uniform-weight comparisons are useful, they do not directly validate the proposed uncertainty estimates. It remains unclear whether the improvements follow from the specific theoretical mechanism or from an effective form of frequency-dependent sensitivity regularization.
- The empirical evidence for generalization across model architectures is insufficient.
The main experiments use a compact WuKong-inspired consumer architecture with 32-dimensional embeddings. The effectiveness of sensitivity regularization may depend on the consumer structure, embedding dimension, and interaction patterns. The paper refers to additional architecture experiments in Appendix G, but these results are absent from the supplied manuscript. Without this evidence, it is difficult to determine whether UWSR addresses a general limitation of sparse-embedding recommendation models or performs particularly well under the evaluated architecture.
- The computational cost of the proposed method is not adequately substantiated.
UWSR requires second-order automatic differentiation to compute the sensitivity penalty, which introduces additional computation beyond standard first-order training. The paper claims minimal overhead and refers to wall-clock measurements in Appendix H, but this appendix is not included in the supplied manuscript. No directly verifiable comparison of training time, peak memory usage, or throughput is available. Since the method targets recommendation systems with potentially very large embedding tables, computational efficiency is important for assessing its practical contribution.
- The reported improvements are not equally reliable across the three benchmarks.
The results on Amazon Electronics and Avazu are encouraging, but MovieLens-20M provides weaker evidence. Appendix D reports that the 95% confidence intervals for both the AUC and cross-entropy improvements at epoch four include zero. Moreover, UWSR reaches its highest MovieLens AUC at epoch two, after which performance declines. Thus, the method does not consistently prevent degradation during additional training epochs. The results support substantial benefits in some settings, but the evidence for robust improvement across all three datasets is less convincing than the main conclusions suggest.
Questions for authors
Can the authors quantify how much of the ordinary multi-epoch generalization degradation is attributable to self-influence asymmetry? How can the effect of label reuse be separated from the optimization changes introduced by the update-order intervention?
Is there empirical evidence that the accumulated squared-gradient weights accurately reflect embedding uncertainty or generalized leverage? How strongly do these estimates correspond to the quantities used in the theoretical analysis?
Can the authors provide the additional architecture experiments referenced in Appendix G? In particular, does UWSR retain its advantage across consumer architectures with different interaction structures and embedding dimensions?
What are the training-time and peak-memory overheads of UWSR relative to MEDA and AdamAR under identical hardware and training settings? Can the authors provide the wall-clock measurements referenced in Appendix H?
How should the MovieLens-20M results be interpreted given that the confidence intervals include zero and that AUC declines after the second epoch? Does the method consistently outperform the strongest baseline when each method is selected at its best validation checkpoint?
11 comments
On the accuracy of our uncertainty approximation
Is there empirical evidence that the accumulated squared-gradient weights accurately reflect embedding uncertainty or generalized leverage? How strongly do these estimates correspond to the quantities used in the theoretical analysis?
Our weighting makes several simplifications, each with established precedent:
-
A local quadratic/Laplace approximation and diagonal precision. We model the embeddings' posterior as Gaussian with precision and approximate the precision matrix by its diagonal. Elastic Weight Consolidation (EWC; Kirkpatrick et al., 2017) uses these same two approximations. Our manuscript focuses on one embedding row for clarity; the same quadratic formulation can be written jointly over the parameters.
-
An optional substitution of empirical Fisher statistics for expected Fisher. The expected-Fisher variant gives nearly identical results (Table 2; construction in Appendix B.3). RWalk (Chaudhry et al., ECCV 2018) provides precedent for this choice.
-
An optional reduction to one scalar per embedding row before shrinkage and inversion. This is inspired by sparse embedding optimizers such as row-wise AdaGrad in TorchRec/FBGEMM. Our per-coordinate ablation performs comparably (Appendix F, Table 5). Shrinkage toward the table mean stabilizes the row-information estimates before inversion, following a common strategy for regularizing estimated covariance.
We will update the manuscript to cite these established approximations.
The practical construction is intended as an efficient surrogate for the theory-motivated regularizer, rather than as a precisely calibrated estimator of each embedding row's covariance or self-influence.
It remains unclear whether the improvements follow from the specific theoretical mechanism or from an effective form of frequency-dependent sensitivity regularization.
For the inverse-frequency count ablation, please see the section “Inverse-Frequency Based UWSR Ablation” in our response to Reviewer 1. This includes an experiment demonstrating why inverse-frequency weighting cannot generally be assumed sufficient and where gradient-based information improves on count weighting.
We would like to share our perspective on this framing. Under constant information per occurrence, our theory directly predicts inverse-frequency-weighted sensitivity regularization. It follows from the manuscript’s asymptotic analysis often relating row information and covariance to support count .
If a novelty concern rests on treating it as a pre-existing method, or on a prior theoretical account explaining its effectiveness, we would welcome any corresponding references. However, to the best of our knowledge, no prior work has proposed frequency-dependent Jacobian regularization for learned sparse embedding rows, nor are we aware of another prior theoretical account motivating this particular construction.
Performance Benchmarking
What are the training-time and peak-memory overheads of UWSR relative to MEDA and AdamAR under identical hardware and training settings? Can the authors provide the wall-clock measurements referenced in Appendix H.
While the penalty can seem to be computationally expensive, the gradient computation only needs hessian vector products which is an optimized computation in pytorch. Thus, the full hessian is neither computed nor materialized in the training step, resulting in minimal increase in the training time and peak memory. Please find below the implementations benchmarked in our paper, benchmarked serially in isolation on an A100 for Amazon with Wukong:
| Method | Four-pass training | Peak GPU memory |
|---|---|---|
| Adam | 198.6 s | 0.200 GiB |
| MEDA | 198.5 s | 0.200 GiB |
| AdamAR | 207.3 s | 0.200 GiB |
| Empirical Fisher | 196.6 s | 0.371 GiB |
| Expected Fisher | 211.3 s | 0.377 GiB |
The original harness implementation was not been optimized for memory. Our production prototype allocates temporary buffers only for the embedding rows accessed by each minibatch, avoiding an additional buffer the size of the full table.
Update: We adapted the active-row differentiation approach in our productionisation prototype to the test harness. Equivalence checks confirmed that the optimized implementation preserved the original loss to numerical precision. On Amazon Electronics, we expanded the embedding tables 32× to 8.2 million rows, keeping active IDs fixed, to evaluate scaling at larger table sizes.
All methods used torch.compile and ran serially on the same A100 for four full training passes.
| Method | Four-pass total | Replay passes 2–4 | Peak allocated memory |
|---|---|---|---|
| Adam | 200.6 s | 134.8 s | 4.954 GiB |
| MEDA | 181.8 s | 132.0 s | 4.955 GiB |
| AdamAR | 184.6 s | 137.9 s | 4.955 GiB |
| UWSR | 183.6 s | 142.7 s | 5.128 GiB |
Movielens Confidence Intervals
Thank you for identifying this issue. Please see our response to reviewer #2 with regards to the Movielens ablations / large percent cold user cohort.
How should the MovieLens-20M results be interpreted given that the confidence intervals include zero and that AUC declines after the second epoch?
We select regularization strengths using validation BCE separately at each applicable epoch in the manuscript. The visualization in our “Architecture Ablations” response makes the comparisons easier to interpret.
Under the validation selection method requested, UWSR or its expected-Fisher variant achieves the best mean test BCE and AUC on all three datasets.
Over 90% of MovieLens test examples contain at least one user or item embedding that was never trained. This helps explain why uniform sensitivity regularization, which reduces reliance on embeddings across all rows, is competitive on this dataset. It is also consistent with embedding dropout’s comparatively strong performance on MovieLens relative to the other two datasets.
Regarding the AUC decline after epoch two, UWSR is a theoretically motivated, computationally tractable regularizer intended to mitigate replay-related overfitting and enable useful additional training. We do not claim that it prevents overfitting indefinitely. AdamAR (Li & Lyu, ICLR 2026) likewise reports settings in which performance peaks after two or three epochs.
Requested Test Performance at the Best Validation Checkpoint
Amazon Electronics (5 seeds)
| Method | Test BCE | Test AUC | ΔBCE vs one epoch |
|---|---|---|---|
| Adam | 0.475737 | 0.8568 | +0.28% |
| MEDA | 0.457353 | 0.8658 | +4.14% |
| Freeze until final pass | 0.477334 | 0.8517 | -0.05% |
| Active-row AdamW | 0.475737 | 0.8568 | +0.28% |
| Embedding dropout | 0.473639 | 0.8567 | +0.72% |
| AdamAR | 0.460100 | 0.8661 | +3.56% |
| UWSR Uniform | 0.468553 | 0.8591 | +1.79% |
| UWSR | 0.444786 | 0.8753 | +6.77% |
| UWSR Expected Fisher | 0.444016 | 0.8756 | +6.93% |
Avazu (5 seeds)
| Method | Test BCE | Test AUC | ΔBCE vs one epoch |
|---|---|---|---|
| Adam | 0.389732 | 0.7405 | +0.00% |
| MEDA | 0.388556 | 0.7414 | +0.30% |
| Freeze until final pass | 0.388997 | 0.7414 | +0.19% |
| Active-row AdamW | 0.391082 | 0.7348 | -0.35% |
| Embedding dropout | 0.389732 | 0.7405 | +0.00% |
| AdamAR | 0.384169 | 0.7486 | +1.43% |
| UWSR Uniform | 0.384534 | 0.7474 | +1.33% |
| UWSR | 0.379938 | 0.7558 | +2.51% |
| UWSR Expected Fisher | 0.379873 | 0.7561 | +2.53% |
MovieLens-20M (15 seeds)
| Method | Test BCE | Test AUC | ΔBCE vs one epoch |
|---|---|---|---|
| Adam | 0.652441 | 0.6684 | +0.05% |
| MEDA | 0.643272 | 0.6768 | +1.44% |
| Freeze until final pass | 0.646041 | 0.6714 | +1.01% |
| Active-row AdamW | 0.646849 | 0.6758 | +0.89% |
| Embedding dropout | 0.643787 | 0.6798 | +1.36% |
| AdamAR | 0.643060 | 0.6804 | +1.47% |
| UWSR Uniform | 0.637289 | 0.6881 | +2.35% |
| UWSR | 0.635796 | 0.6896 | +2.58% |
| UWSR Expected Fisher | 0.635904 | 0.6891 | +2.57% |
Thank you for reviewing our manuscript. Please find our response below and let us know if there are any other concerns we can address.
1. On causal attribution
Unfortunately, this is difficult to quantify. Validation-loss change across an additional epoch is a net effect: e.g. beneficial optimization during epoch two can offset harmful overfitting effects. The one epoch phenomenon describes a loss curve pattern, which is the sum of all effects.
We establish self-influence asymmetry as a contributing mechanism, without claiming it is the dominant cause in the manuscript.
Our contribution is the exact scalar model, mechanistic analysis, its experimentally tested predictions, and the theoretically motivated regularizer we propose, which compares favorably with existing methods such as AdamAR (Li & Lyu, ICLR 2026) and MEDA (Fan et al., 2024) in our experiments.
This was an interesting question to think about.
Self-influence describes the effect of a label on its own fitted prediction, studied in the context of statistical optimism (Efron, 1986; 2004). Our manuscript defines self-influence asymmetry as the resulting change in the effective training objective and its optimum from a model’s fit on an example. Appendix B develops this connection to classical optimism.
A causal attribution requires a specified counterfactual and a quantity whose contribution is being measured. By necessity, testing label self-inclusion requires a comparison with and without the label-dependent model state. We cannot hold the consequences of the sample’s influence on the model and optimizer fixed while measuring the effect of removing that training label from the training procedure. A nontrivial intervention must therefore change the label-dependent state in one way or another.
Our update-order intervention supplies a non-trivial way to test label inclusion, and optimizing the labels before the network is the independent variable being tested.
Both staged arms and the standard training control match initialization, examples, minibatches, optimizer configuration, and update counts. The arms differ in whether embeddings were already updated using the labels the rest of the model is then being fit on. The consumer-first ordering arm controls for the split training procedure. We have clarified this motivation and the reverse-order control’s role in Section 3.3 of the revised manuscript (Figures 2–3).
Zhang et al. found that training the embeddings for multiple epochs after freezing the consumer prevented the one-epoch phenomenon, demonstrating that the issue is not intrinsic to training the embeddings for multiple epochs. Additionally, our benchmarks show that freezing the embeddings until the last pass, while not competitive, prevents catastrophic degradation, demonstrating that it is not intrinsically caused by training the downstream network for multiple epochs.
Our intervention, however, shows that degradation is triggered when the embeddings are updated on the labels before the rest of the network in a single epoch. Our analysis predicts this, alongside a rise in the embeddings’ gradient magnitudes, which is observed in both multi-epoch training and our intervention (Figures 2–3 and 5).
In the spirit of the question, we draw a comparison of magnitudes: the MovieLens contrast between the staged arms is approximately 180% of the ordinary validation-BCE increase from the first to the second epoch in the archived benchmarks. This shows that self-inclusion can induce a penalty large enough to explain the observed net increase, and the effect is non-trivial.
Please refer to the predictive test reported to Reviewer 1 under “Explanatory power of Self-Influence Asymmetry,” now included in Figure 7 and Appendix B of the revised manuscript. Your question also motivated it.