acceptodds
Under review as a conference paper at ICLR 2027

Randomly Masked Linear Autoencoders: A Unified Closed-Form Framework

Abstract

Randomly masked linear autoencoders (LAEs) reconstruct data from partially observed inputs. Existing formulations, however, do not jointly provide emphasis weighting, independent diagonal control, and an explicit whole-matrix solution. We introduce a unified masked objective that combines emphasis weighting with separately tunable diagonal and full-matrix regularization. Conditioning on whether each target is retained or dropped exposes a shared quadratic structure, yielding a whole-matrix closed form through one shared inverse and diagonal rescaling. Experiments on nine recommendation datasets show that our unified model outperforms existing LAE baselines on most evaluation metrics and achieves competitive accuracy against more complex deep learning methods. Our solver also matches DEQL's column-wise solution numerically at the free-diagonal endpoint while reducing runtime and peak incremental memory. We additionally derive two closed-form low-rank approximations with factorized storage. Our code is available at https://anonymous.4open.science/r/ICLR_2027_Unified_Recommendation-E5F2.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
linear autoencoders, recommender system, closed-form solution, low-rank approximation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UH3DOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper studies a randomly masked LAE. Based on a prior work DEQL, it introduces an independent diagonal regularization and derives a whole-matrix closed-form solution, which has computational advantage over baselines. Furthermore, the authors propose a related low-rank approximation and conduct experimental evaluation on recommendation tasks.

Critical strengths
  1. The empirical results in the paper demonstrate a significant efficiency improvement. Under fair settings, the proposed solution achieves a significant speedup of approximately 1.5–2.6 times compared to DEQL.
  2. The paper provides relatively comprehensive experimental results involving multiple recommendation datasets.
Critical weaknesses
  1. The Reviewer think that the contribution beyond DEQL is not adequately discussed in the paper. Compared to DEQL, the main modifications in the paper focus on computational implementation and independent diagonal penalties, while the discussion of the low-rank part mainly refers to some existing techniques. This makes the paper's current positioning closer to an extension of the existing DEQL framework with only provide some incremental results.

  2. Some related works are not sufficiently discussed. This paper does not include [1], which also introduces low-rank designs in the LAE model, making its comparison with existing works incomplete.

  3. Some experimental results show that the proposed solution has limited accuracy advantages or even disadvantages compared to some strong baselines. For example, compared to DEQL, only 3 out of 12 comparisons reached significance, while the Recall advantage of Games was approximately +6.48% in the main text, but reversed to approximately -0.84% ​​in the appendix. This suggests that the gains reported in the main table may be sensitive to data partitioning.

  4. The article should briefly discuss some practical application scenarios (such as cold-start) that the proposed scheme might have in recommender systems. Furthermore, another suggestion from reviewers is to place comparisons of related work in the main text rather than the appendix.

[1] Pu, Y., Fan, R., Chen, J., Zhu, Z., Lian, D., & Chen, E. (2025). Automated Sparse and Low-Rank Shallow Autoencoders for Recommendation. ACM Transactions on Recommender Systems, 3(3), 1-26.

Questions for authors

Please refer to Weakness.

5 comments

Thank you for your constructive comments. We have added an exact analysis of the diagonal penalty, significance tests after retuning, and a discussion of Pu et al. (2025). All references below are to the revised manuscript.

W1: Our main contribution relative to DEQL is an exact characterization of how emphasis and diagonal control affect predictions, including conditions under which the diagonal penalty is not redundant (Section 4.3).

DEQL derives column-wise solutions under general emphasis (Section 1). Our whole-matrix closed form answers two questions it leaves open: how the diagonal changes cross-item predictions and whether a separate diagonal penalty yields new solutions.

Theorem 4.3 and Corollary 4.4 show that, for an unobserved item (Ru,i=0R_{u,i}=0), the score under WL2∗W_{L_2}^{\ast} equals γi\gamma_i times the score under the zero-diagonal solution W∞∗W_{\infty}^{\ast}, where γi=(p+ηi)/(p+ηi+qα2ϵi)\gamma_i=(p+\eta_i)/(p+\eta_i+q\alpha^2\epsilon_i) and ηi=s(λD+λW)/(qα2Ni,i)\eta_i=s(\lambda_D+\lambda_W)/(q\alpha^2N_{i,i}).

Here, q=1−pq=1-p, s=qα2+ps=q\alpha^2+p, Ni,iN_{i,i} is item popularity, and ϵi\epsilon_i measures its regularized prediction difficulty from other items. Holding (p,α,λW)(p,\alpha,\lambda_W) fixed, increasing λD\lambda_D reduces the fitted diagonal and moves each column toward the zero-diagonal endpoint.

Diagonal control therefore scales each item's prediction from other items by a factor γi\gamma_i shared across users. Differences in these factors across items can change a user's ranking. With a positive total penalty, more popular items undergo stronger shrinkage at equal prediction difficulty. When λW>0\lambda_W>0, the effect of a finite λD\lambda_D on the multiplier is largest at intermediate popularity (Figure 1; Appendix C.4).

Corollary 4.5 holds the masking rate pp and normalized ridge λ~W\tilde\lambda_W fixed, keeping the shared inverse unchanged. If two co-occurring items differ in popularity, every finite λD>0\lambda_D>0 produces off-diagonal coefficients that DEQL(L2L_2) with the same pp and λ~W\tilde\lambda_W cannot reproduce at any emphasis ratio, and in general not even up to a rank-preserving rescaling. If all such items have equal popularity, a smaller emphasis ratio reproduces the coefficients. The added freedom thus comes exactly from variation in popularity and carries no guarantee of better rankings.

We agree that the low-rank part builds on existing full-rank-teacher techniques (Jin et al., 2021; Steck & Garcia, 2021; Li et al., 2026). We present it as a secondary result in Appendix D, outside the listed contributions. Our weighted approximation uses the metric induced by the objective under general emphasis and diagonal regularization. DEQL's exact low-rank closed form applies only when the two emphasis weights are equal (Appendix A). In that equal-weight case without diagonal regularization, our weighted approximation is itself an exact rank-constrained minimizer and, without ridge, recovers an optimal encoder-decoder product of the linear masked autoencoder of Bisulco et al. (2025) (Appendix D.1).

W2: We now cite and position Pu et al. (2025) in the related work on low-rank and scalable linear recommenders (Appendix A).

AutoS²AE+ (Pu et al., 2025), together with ELSA, SANSA, SVD-AE, and ImplicitSLIM, pursues compact or sparse item transformations. Our approximations project a full-rank closed-form teacher without low-rank training, so they require the full-rank solve first (Appendix E).

W3: We retuned both methods on ten random splits and tested significance with Holm correction (Appendix C.2). The Games gain in the main table is sensitive to the split.

Accuracy is comparable across the two methods. Differences are small and dataset-dependent, and none of the 12 dataset-metric comparisons is significant after Holm correction. Table 8 reports the mean differences, Uni-L2L_2 minus DEQL(L2L_2), and Holm-adjusted p-values from a one-sided corrected t-test:

DatasetΔR@20 (adj. p)ΔN@20 (adj. p)
Games+0.0070 (0.96)+0.0010 (1.00)
Beauty-0.0011 (1.00)+0.0007 (1.00)
Gowalla-1+0.0003 (1.00)+0.0001 (1.00)
ML-20M+0.0006 (0.73)+0.0014 (0.072)
Netflix+0.0002 (0.27)+0.0001 (0.29)
MSD0.0000 (1.00)0.0000 (1.00)

Before correction, the same three comparisons as in the original five-split test have p-values below 0.05. We have updated Section 5.2 and moderated the abstract's accuracy claim accordingly.

On Games, the main-table R@20 gain of 6.48% comes from a single split. The five-split appendix comparison (-0.84%) reused the configurations selected on that split without retuning. After retuning on ten splits, the mean gain is positive but small relative to the variation across splits. We therefore agree that the single-split gain is sensitive to the partition and does not establish a reliable advantage on Games.

Uni-L2L_2 is also not uniformly best in the main tables: EDLAE is slightly better on MSD, while deep or graph models lead on Yelp2018 and Gowalla (Tables 3 and 4; Section 5.2).

The sweeps in Appendix C.3 suggest that full-matrix regularization accounts for most of the improvement over the unregularized model, while λD\lambda_D mainly redistributes accuracy across popularity groups. The evidence supports an interpretable extension with comparable accuracy.

W4: Appendix E now discusses cold-start and long-tail recommendation. Sections 2 and 4.2 also compare our model with the most closely related work.

For previously unseen users with observed interactions, the model provides recommendations through fold-in without retraining the item-to-item matrix. Corollary 4.4 explicitly covers these inputs. The strong generalization setting in Table 3 evaluates this scenario because test users are disjoint from training users, although it does not separately measure performance for users with very few interactions.

For long-tail items, Corollary 4.4 shows that a positive total penalty shrinks the scores of less popular items less at equal prediction difficulty. Relative to the zero-diagonal limit, the selected λD\lambda_D improves tail accuracy and catalog coverage on the strong generalization datasets (Tables 13 and 14). Relative to the free-diagonal endpoint λD=0\lambda_D=0, tail accuracy does not improve on most datasets (Table 12). The tail benefit over the zero-diagonal limit therefore mainly reflects allowing a nonzero diagonal; increasing λD\lambda_D does not consistently improve tail accuracy.

As new interactions arrive, the model can be refitted in closed form. Tuning α\alpha and λD\lambda_D reuses one factorization per (p,λ~W)(p,\tilde\lambda_W). Zero-interaction cold start, covering users without interactions and new items without training interactions, remains outside our scope. Handling these cases requires side information, such as item content, or a popularity fallback (Appendix E).

Section 2 now compares how EASE, EDLAE, RLAE, RDLAE, and DEQL treat the diagonal, while Section 4.2 relates our objective to linear masked autoencoders.

Thank you for your detailed explanations and revisions on this manuscript, which has addressed some of my main concerns. I therefore update my soundness score to 2.

However, at the same time, the manuscript has also been extensively changed, which includes substantial additions of contents. Within the limited discussion time for about 3-4 days, I'm afraid that I'm not able to examine all these additions carefully enough to fully check their correctness. I feel sorry but want to maintain my original overall score. I believe the revised manuscript would be benefited from another round of full review in a future top venue.

Reviewer_htZyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors develop a unified formulation for masked linear recommenders, or linear autoencoders (LAEs), combining emphasis weighting with full-matrix and diagonal regularization. The resulting framework interpolates between several existing formulations and recovers multiple LAE methods as special or limiting cases.

Critical strengths
  • The derivations are clean, and the relationships between existing methods are clearly presented, particularly in Table 1 and throughout the text.
  • The computational simplification and whole-matrix reformulation of the column-wise DEQL solution is a concrete practical contribution.
  • The authors evaluate the proposed Uni-L2L_2 model across several datasets and report modest improvements over existing LAE baselines.
Critical weaknesses

Limited novelty

The methodological novelty appears relatively limited. DEQL already covers the emphasized free-diagonal objective, while RLAE/RDLAE already provide an interpolation between zero-diagonal and unconstrained solutions, although without general emphasis. The main additional ingredient is therefore the combination of these ideas through an independently tunable diagonal regularizer while preserving a whole-matrix closed-form solution.

Small empirical benefit and hyperparameter-selection cost

The empirical benefit of the new method appears relatively small. The authors explicitly state that most of the gain comes from λW\lambda_W (see Figure 3 and the corresponding discussion), while the additional benefit obtained from λD\lambda_D is generally small. However, full-matrix or ridge regularization is already present in strong baselines such as DEQL(L2L_2) and EASE. Therefore, the improvement attributable specifically to the newly introduced diagonal regularization appears limited.

Moreover, Uni-L2L_2 requires selecting several hyperparameters,

(p,α,λW,λD).(p,\alpha,\lambda_W,\lambda_D).

The reported computational advantage concerns the solver for a fixed configuration. It is therefore unclear how much of this advantage remains when accounting for the complete hyperparameter-selection procedure.

In addition, in the five-split experiments, Uni-L2L_2 is statistically significantly better than DEQL(L2L_2) in only 3 of the 12 dataset-metric comparisons reported in Table 7.

Limited connection to masked representation learning

The broader connection to masked learning in vision and language appears speculative. The main model is a full item-item reconstruction matrix without an explicit bottleneck representation. Moreover, for α>0\alpha > 0, the objective reconstructs both masked and unmasked entries rather than restricting the reconstruction loss to masked targets.

There is already theoretical work studying masked autoencoders much more directly in representation-learning settings. For example, Bisulco et al. (2025), arXiv:2508.15404, theoretically analyze linear masked autoencoders with structured image-patch masking and study downstream representation quality through both linear probing and fine-tuning.

In contrast, the present paper does not evaluate representation quality or downstream tasks beyond recommendation. The claimed connection to masked learning in images and language therefore seems insufficiently supported by the current results.

Questions for authors
  1. Novelty relative to prior work. What is the main conceptual advance over DEQL and RLAE/RDLAE beyond combining existing regularization choices while preserving closed-form solvability?

  2. End-to-end computational cost. Could the authors report the total runtime, including hyperparameter selection, for the methods shown in Tables 3 and 4? Since Uni-L2L_2 tunes (p,α,λW,λD)(p,\alpha,\lambda_W,\lambda_D), it would be useful to understand whether the reported solver-level computational advantage translates into an end-to-end computational advantage.

  3. Practical role of λD\lambda_D. Given that the authors state that most of the gain comes from λW\lambda_W, while λD\lambda_D typically provides only a small improvement over the better endpoint, what is the main practical and theoretical benefit of introducing the additional diagonal interpolation?

  4. Baseline tuning. The paper states that each baseline is tuned over the range used in its original work, but the exact search spaces are not reported. Could the authors provide the hyperparameter grids used for EASE and DEQL(L2L_2), in particular their ridge/L2L_2 regularization parameters, and confirm that these were independently selected using the same validation protocol as Uni-L2L_2?

  5. Representation learning. Which result in the current paper supports the claimed relevance to representation learning or downstream tasks beyond recommendation?

  6. Masked-only reconstruction. What happens at α=0\alpha = 0? This regime corresponds to applying the reconstruction loss only to masked entries and is therefore particularly natural from the perspective of masked autoencoders. Is there a finite-λD\lambda_D extension or limiting solution in this regime, and how does it perform empirically?

  7. Sensitivity to the masking probability. Since the masking probability pp is central to the formulation and is selected through validation, could the authors provide performance as a function of pp across the datasets considered in the paper?

    It would also be informative to provide a two-dimensional sensitivity analysis over (p,λW)(p,\lambda_W) for a fixed or selected λD\lambda_D. This would help disentangle the effect of the masking probability from the effect of full-matrix regularization.

7 comments

Thank you for your careful and constructive review. We are especially grateful for the pointer to Bisulco et al. (2025). This was a very insightful suggestion: studying their linear analysis led us to a precise relationship between their linear masked autoencoder and our framework, which we believe strengthens the paper. All references below are to the revised manuscript.

On novelty, the whole-matrix closed form gives an exact characterization of how the diagonal penalty affects predictions and establishes when it is not redundant with emphasis (Q1).

We agree that the aggregate empirical gains are modest. After retuning on ten random splits, none of the comparisons with DEQL(L2L_2) is significant after Holm correction. Section 5.2 now reports this result, and we have moderated the abstract's accuracy claim. The diagonal penalty mainly redistributes accuracy and exposure across popularity groups (Q3). Tuning it adds evaluations while reusing the same factorizations (Q2).

We have also clarified the connection to masked representation learning:

  1. With singleton patches and the same rank constraint, the linear MAE objective of Bisulco et al. (their Eq. 1) coincides with ours at α=1\alpha=1 and λD=λW=0\lambda_D=\lambda_W=0. Identifying their masking ratio mm with our pp, their matrix VV becomes our K=qG+pNK=qG+pN. In this setting, the weighted low-rank solution of Proposition D.1 exactly minimizes the original rank-constrained problem and recovers an optimal encoder-decoder product from their Theorem 1: W^k=K−1GQkQk⊤\widehat W_k=K^{-1}GQ_kQ_k^\top, where QkQ_k contains the top-kk eigenvectors of GK−1GGK^{-1}G. The new "Exact case" paragraph in Appendix D.1 states this correspondence, which is also summarized in Section 4.2.

  2. Their linear objective reconstructs all coordinates with equal weight (their Eq. 1 and footnote 1), corresponding to our α=1\alpha=1. Reconstructing retained targets is therefore shared by the two objectives. Our framework covers both this equal-weight setting and masked-only reconstruction at α=0\alpha=0 (Q6).

  3. Our analysis jointly treats unequal weights on retained and dropped targets, an independently tunable diagonal penalty, and full-matrix regularization in one closed form. Theorem 4.3 and Corollary 4.4 show how emphasis and diagonal control change cross-feature predictions even when the target coordinate is absent from the input.

  4. We agree that no result in our paper establishes downstream representation quality in vision or language (Q5). Their multi-coordinate patch masking is outside our theory, which assumes independent coordinate masking. With unequal emphasis or positive diagonal regularization, our teacher-based low-rank constructions need not solve the rank-constrained problem exactly. The revised Conclusion and the new "Scope Beyond Recommendation" paragraph in Appendix E therefore present vision and language as future directions.

Q1: Beyond combining existing regularizers, we characterize exactly how emphasis and diagonal control act on predictions and prove when the diagonal penalty is not redundant (Section 4.3).

As the review notes, DEQL covers the emphasized free-diagonal objective, while RLAE/RDLAE interpolate the diagonal without emphasis (Section 2). Our whole-matrix closed form answers two questions left open by DEQL: how the diagonal changes cross-item predictions and whether a separate diagonal penalty yields new solutions under general emphasis.

Theorem 4.3 and Corollary 4.4 show that, for an unobserved item (Ru,i=0R_{u,i}=0), the score under WL2∗W_{L_2}^{\ast} equals γi\gamma_i times the score under the zero-diagonal solution W∞∗W_{\infty}^{\ast}, where γi=(p+ηi)/(p+ηi+qα2ϵi)\gamma_i=(p+\eta_i)/(p+\eta_i+q\alpha^2\epsilon_i) and ηi=s(λD+λW)/(qα2Ni,i)\eta_i=s(\lambda_D+\lambda_W)/(q\alpha^2N_{i,i}).

Here, q=1−pq=1-p, s=qα2+ps=q\alpha^2+p, Ni,iN_{i,i} is item popularity, and ϵi\epsilon_i measures its regularized prediction difficulty from other items. Diagonal control scales each item's prediction from other items by a factor shared across users. Differences in these factors across items can change a user's ranking. With a positive total penalty, more popular items undergo stronger shrinkage at equal prediction difficulty (Section 4.3; Figure 1).

Corollary 4.5 holds the masking rate pp and normalized ridge λ~W\tilde\lambda_W fixed. If two co-occurring items differ in popularity, every finite λD>0\lambda_D>0 produces off-diagonal coefficients that DEQL(L2L_2) with the same pp and λ~W\tilde\lambda_W cannot reproduce at any emphasis ratio, and in general not even up to a rank-preserving rescaling. If all such items have equal popularity, a smaller emphasis ratio reproduces the coefficients. The added freedom thus comes from variation in popularity and carries no guarantee of better rankings.

Q2: Does the solver-level advantage translate into an end-to-end advantage once hyperparameter selection is included?

We agree that the earlier comparison used a fixed configuration, while Uni-L2L_2 tunes (p,α,λW,λD)(p,\alpha,\lambda_W,\lambda_D). With the same solver and base grid, searching λD\lambda_D covers six times as many configurations and takes 1.1 to 2.7 times as long.

Appendix C.1, Table 7 is a controlled cost benchmark, separate from the searches behind Tables 3 and 4, whose grids differ across methods (Appendix C.7). Both methods search the same (p,λ~W,α)(p,\tilde\lambda_W,\alpha) grid from scratch on one GPU across the six strong-generalization datasets, using the same solver and evaluation code; Uni-L2L_2 also searches λD\lambda_D. Settings sharing (p,λ~W)(p,\tilde\lambda_W) reuse one factorization and zero-diagonal score matrix (Corollary 4.4), so both methods perform the same number of factorizations. The extra search therefore adds only validation evaluations. End-to-end times (s) include data loading, Gram construction, all solves and validation evaluations, and model selection:

DatasetDEQL(L2L_2)Uni-L2L_2Ratio
Games2.366.132.60
Beauty198.3264.01.33
Gowalla-1683.8756.91.11
ML-20M357.0580.21.63
Netflix539.51451.72.69
MSD2136.63339.61.56

Peak memory is essentially the same (Table 7). This comparison is conservative for Uni-L2L_2: in this benchmark, DEQL(L2L_2) already uses our whole-matrix solver and factorization reuse, while its original column-wise solver is several times slower per solve (Table 6). Reuse keeps the additional tuning cost modest.

Q3: Corollary 4.5 establishes the popularity-dependent control provided by λD\lambda_D (Q1). Empirically, λD\lambda_D redistributes accuracy and exposure across popularity groups (Appendix C.4).

We fix (p,α,λW)(p,\alpha,\lambda_W) and compare the validation-selected λD\lambda_D with both endpoints on the test set, using paired bootstrap confidence intervals (Tables 12 and 13). Relative to λD=0\lambda_D=0, mid-popularity accuracy improves on five of six datasets while head accuracy decreases on four, leaving small aggregate changes; overall gains are significant on only two datasets. Compared with the zero-diagonal endpoint, tail R@20 is higher on all six datasets, significantly so on five. The share of users whose Top-20 set changes varies widely across datasets, typically with only a few items replaced. The table summarizes Tables 12 to 15 at fixed (p,α,λW)(p,\alpha,\lambda_W). Tail share is the percentage of Top-20 slots occupied by tail items, reported at the selected λD\lambda_D / λD=∞\lambda_D=\infty.

DatasetMid ΔR@20 vs 0Overall ΔR@20 vs 0Tail ΔR@20 vs ∞Tail share (%)Changed Top-20 vs 0 (%)
Games+0.0776+0.0092+0.014120.6 / 14.2100.0
Beauty0.0000-0.0027+0.001816.1 / 12.744.6
Gowalla-1+0.0023-0.0004+0.016625.4 / 18.728.5
ML-20M+0.0019+0.0006+0.00030.09 / 0.0617.7
Netflix+0.0011+0.0001+0.00662.6 / 2.216.2
MSD+0.00030.0000+0.003030.0 / 29.25.3

The zero-diagonal limit also reduces catalog coverage on all six datasets (Table 14). The selected penalty lies between the endpoints. Relative to λD=0\lambda_D=0, tail accuracy and tail share do not increase on most datasets. The tail benefit over the zero-diagonal limit therefore mainly reflects allowing a nonzero diagonal; increasing λD\lambda_D does not consistently improve these tail metrics (Appendix E).

To address the significance concern, we retuned both methods on ten random splits (Appendix C.2). Accuracy is comparable: differences are small and dataset-dependent, and none of the 12 dataset-metric comparisons is significant after Holm correction. Table 8 reports mean differences, Uni-L2L_2 minus DEQL(L2L_2), with Holm-adjusted p-values from a one-sided corrected t-test:

DatasetΔR@20 (adj. p)ΔN@20 (adj. p)
Games+0.0070 (0.96)+0.0010 (1.00)
Beauty-0.0011 (1.00)+0.0007 (1.00)
Gowalla-1+0.0003 (1.00)+0.0001 (1.00)
ML-20M+0.0006 (0.73)+0.0014 (0.072)
Netflix+0.0002 (0.27)+0.0001 (0.29)
MSD0.0000 (1.00)0.0000 (1.00)

Before correction, the same three comparisons as in the original five-split test have p-values below 0.05. We have updated Section 5.2 and moderated the abstract's accuracy claim accordingly. The sweeps in Appendix C.3 suggest that full-matrix regularization accounts for most of the improvement over the unregularized model, while λD\lambda_D mainly redistributes accuracy across popularity groups. The evidence supports an interpretable extension with comparable accuracy.

Q4: Appendix C.7 now reports the search grids. EASE and DEQL(L2L_2) were tuned independently under the same validation protocol as Uni-L2L_2.

Each method selects its hyperparameters independently on the same validation split, and the selected configuration is evaluated once on the test set (Section 5.1). Baseline search ranges cover at least those of their original papers. Table 18 summarizes each grid as the union across datasets; the ellipsis does not indicate a uniform step. For EASE and DEQL(L2L_2), the grids are:

MethodSearch grid
EASEλ ∈ {1, 2, …, 800, 1000}
DEQL(L2L_2)p ∈ {0.1, 0.2, …, 0.8, 0.9}, α ∈ {0.1, 0.25, …, 4, 5}, λW\lambda_W ∈ {10, 20, …, 300, 500}

Table 18 also lists the grids for the other baselines. The selected values for every method and dataset are released with the code.

Q5: The paper does not establish downstream representation quality. The supported connection is an exact correspondence with the linear MAE of Bisulco et al. (2025).

As detailed in our response to the representation-learning concern above, the revised discussion presents vision and language as future directions. In the singleton-patch, equal-weight setting, their linear MAE objective is a special case of ours, and our weighted rank-kk solution recovers an optimal encoder-decoder product from their Theorem 1 (Section 4.2; Appendix D.1). Downstream representation quality, structured patch masking, and unequal emphasis under a rank constraint remain future work (Appendix E).

Q6: At α=0\alpha=0, the diagonal penalty does not affect cross-feature predictions, and any positive penalty gives a zero-diagonal solution (Section 4.2; Appendix C.5).

With α=0\alpha=0, only dropped entries are reconstructed, so the diagonal is absent from the data term. With any positive penalty, Proposition 4.2 gives the same unique minimizer for every λD\lambda_D: the zero-diagonal minimizer with ridge λW\lambda_W. There is therefore no finite-λD\lambda_D interpolation. Without penalties, the diagonal is arbitrary while the off-diagonal part is unique (Proposition 3.6). The diagonal penalty thus creates no distinct family of cross-feature predictors in the masked-only regime.

Empirically, validation accuracy is unchanged across λD\lambda_D. Table 16 reports the following test R@20 values:

Datasetα=0\alpha=0λD→∞\lambda_D\to\inftyUni-L2L_2
Games0.26480.25460.2891
Beauty0.13190.13220.1359
Gowalla-10.22660.22660.2276
ML-20M0.39300.39310.3936
Netflix0.36530.36530.3659
MSD0.33420.33420.3341

Masked-only results essentially match the zero-diagonal limit except on Games, where the selected configurations differ. Compared with Uni-L2L_2, masked-only reconstruction is slightly lower on five datasets, with the largest gap on Games, and essentially matches it on MSD.

Q7: Appendix C.6 now reports sensitivity to pp and to (p,λW)(p,\lambda_W) on all nine datasets.

Figure 7 varies pp while holding the other selected values fixed. Figure 8 varies pp and raw λW\lambda_W jointly. Table 17 summarizes validation R@20 differences relative to the selected configuration. A negative value in the last column means that the selected configuration outperforms every point in the two-dimensional grid.

DatasetSelected pΔR@20 at lower / upper pBest 2-D grid point ΔR@20
Games0.8-0.0389 / -0.0290-0.0047
Beauty0.33-0.0009 / -0.0029+0.0029
Gowalla-10.5-0.0033 / -0.0002+0.0004
ML-20M0.25+0.0006 / -0.0011+0.0009
Netflix0.25-0.0005 / -0.0005-0.0003
MSD0.1-0.0006 / +0.0002+0.0003
Gowalla (weak)0.7-0.0012 / -0.0054-0.0002
Yelp20180.8-0.0098 / -0.01050.0000
Amazon-Book0.3-0.0018 / +0.0001+0.0004

Accuracy is stable under small changes in pp on seven of the nine datasets; Games and Yelp2018 are more sensitive. The selected configurations are close to the best points in the two-dimensional grid. On the three largest strong-generalization datasets, the heatmaps are relatively flat over a broad range of pp and λW\lambda_W.

Reviewer_rw3EOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a unified objective for Randomly Masked Linear Autoencoders (LAEs) by integrating emphasis weighting with independently tunable diagonal regularization. Specifically, the proposed framework introduces a diagonal penalty alongside full-matrix L2L_2 regularization, allowing continuous control over the degree of self-reconstruction. The authors derive a whole-matrix closed-form solution that encompasses existing approaches, including DEQL and zero-diagonal LAEs, as special or limiting cases. Experiments on nine recommendation datasets demonstrate modest performance improvements over existing LAE baselines. Additionally, the proposed whole-matrix solver (Uni-Sol) reduces runtime and memory usage compared to the existing DEQL solver while maintaining numerical equivalence.

Critical strengths
  1. Flexible Control through Independent Diagonal Regularization The proposed framework introduces an independently tunable diagonal penalty alongside the emphasis weighting used in DEQL, enabling continuous control over the degree of self-reconstruction. This formulation unifies approaches that allow a free diagonal with those that impose a zero-diagonal constraint. Furthermore, rather than relying on a heuristic approach, the authors provide mathematical derivations and proofs establishing a whole-matrix closed-form solution and its relationships to existing methods. This offers a mathematically grounded and flexible framework for LAEs.
  2. Improved Computational Efficiency over DEQL As shown in Table 2, the proposed whole-matrix solver (Uni-Sol) achieves approximately 1.51.5–2.6×2.6\times speedup over the existing column-wise DEQL solver across nine datasets while maintaining numerical equivalence. It also reduces incremental memory usage across the evaluated datasets. These results demonstrate a practical contribution by improving computational efficiency without changing the underlying solution.
Critical weaknesses
  1. Insufficient Explanation and Limited Empirical Benefits of Diagonal Regularization Although the proposed framework enables independent control of the diagonal penalty, the paper does not sufficiently explain why such control leads to more effective cross-item prediction or improved recommendation performance. Further theoretical or empirical analysis would be helpful to establish the underlying mechanism. Moreover, while the proposed method shows performance improvements over DEQL(L2L_2) on several datasets, the improvements are generally modest, particularly on Netflix, MSD, and ML-20M in Table 3. In Appendix Table 7, the statistical analysis over five random splits shows statistically significant improvements over DEQL(L2L_2) in only 3 out of 12 dataset–metric comparisons at the uncorrected p<0.05p < 0.05 significance level. Furthermore, the authors acknowledge that most of the performance gains come from full-matrix regularization (λW\lambda_W), while diagonal regularization (λD\lambda_D) mainly provides fine-grained improvements. Therefore, the empirical evidence does not sufficiently establish the practical benefits of diagonal regularization, which is a central methodological contribution of this paper.

  2. Incomplete Computational Efficiency Evaluation of the Main Proposed Model Although the paper emphasizes the computational efficiency of the whole-matrix closed-form solver, the runtime and memory comparisons in Table 2 are conducted only for Uni-Sol, where λD=λW=0\lambda_D = \lambda_W = 0. This model computes the same solution as the original DEQL method. However, the main proposed model demonstrating improved recommendation performance is Uni-L2L_2, which incorporates both diagonal and full-matrix regularization. The paper does not provide a direct comparison of runtime and peak memory usage between Uni-L2L_2 and DEQL(L2L_2). Additionally, the computational cost of hyperparameter selection, including the additional diagonal regularization parameter λD\lambda_D, is not evaluated. Since the proposed method also retains O(n3)O(n^3) time complexity, further evaluation is necessary to establish whether the computational efficiency improvements extend to the final proposed model in practical settings.

Questions for authors

Q1. Underlying Mechanism of Diagonal Regularization While the proposed diagonal regularization allows control over self-reconstruction coefficients, it remains unclear how this mechanism leads to more effective cross-item prediction and improved recommendation performance. Could the authors provide a clearer explanation or additional theoretical or empirical evidence supporting this relationship?

Q2. Statistical Significance and Consistency of Performance Improvements The performance improvements over DEQL(L2L_2) are generally modest, and the statistical analysis over five random splits shows significant improvements in only a limited number of comparisons. Could the authors provide additional evidence demonstrating that diagonal regularization consistently yields statistically significant improvements over DEQL(L2L_2)?

Q3. Computational Efficiency of the Final Proposed Model Table 2 compares the computational efficiency of Uni-Sol and the original DEQL solver. Could the authors provide a direct comparison of runtime and peak memory usage between Uni-L2L_2 and DEQL(L2L_2)? Additionally, does the overall computational advantage remain when accounting for the additional hyperparameter selection cost?

5 comments

Thank you for your constructive comments. We have added an exact analysis of the diagonal penalty and controlled experiments. All references below are to the revised manuscript.

W1/Q1 (a): We now characterize exactly how diagonal regularization changes the scores of unobserved items (Section 4.3).

Theorem 4.3 and Corollary 4.4 show that, for an unobserved item (Ru,i=0R_{u,i}=0), the score under WL2∗W_{L_2}^{\ast} equals γi\gamma_i times the score under the zero-diagonal solution W∞∗W_{\infty}^{\ast}, where γi=(p+ηi)/(p+ηi+qα2ϵi)\gamma_i=(p+\eta_i)/(p+\eta_i+q\alpha^2\epsilon_i) and ηi=s(λD+λW)/(qα2Ni,i)\eta_i=s(\lambda_D+\lambda_W)/(q\alpha^2N_{i,i}).

Here, q=1−pq=1-p, s=qα2+ps=q\alpha^2+p, Ni,iN_{i,i} is item popularity, and ϵi\epsilon_i measures its regularized prediction difficulty from other items. Holding (p,α,λW)(p,\alpha,\lambda_W) fixed, increasing λD\lambda_D reduces the fitted diagonal and moves each column toward the zero-diagonal endpoint.

Diagonal control therefore scales each item's prediction from other items by a factor γi\gamma_i shared across users. Differences in these factors across items can change a user's ranking. With a positive total penalty, more popular items undergo stronger shrinkage at equal prediction difficulty. When λW>0\lambda_W>0, the effect of a finite λD\lambda_D on the multiplier is largest at intermediate popularity (Figure 1; Appendix C.4).

W1/Q1 (b): We prove when the diagonal penalty adds control beyond emphasis and ridge (Corollary 4.5) and isolate its effect empirically (Appendix C.4).

Corollary 4.5 holds the masking rate pp and normalized ridge λ~W\tilde\lambda_W fixed, keeping the shared inverse unchanged. If two co-occurring items differ in popularity, every finite λD>0\lambda_D>0 produces off-diagonal coefficients that DEQL(L2L_2) with the same pp and λ~W\tilde\lambda_W cannot reproduce at any emphasis ratio, and in general not even up to a rank-preserving rescaling. If all such items have equal popularity, a smaller emphasis ratio reproduces the coefficients. The added freedom thus comes exactly from variation in popularity and carries no guarantee of better rankings.

To isolate the empirical effect, we fix (p,α,λW)(p,\alpha,\lambda_W) and compare the validation-selected λD\lambda_D with both endpoints on the test set, using paired bootstrap confidence intervals (Tables 12 and 13). Compared with λD=0\lambda_D=0, mid-popularity accuracy improves on five of six datasets, while head accuracy decreases on four. Aggregate accuracy consequently changes little, with significant overall gains on only two datasets. Compared with the zero-diagonal endpoint, tail accuracy improves on five datasets. These gains mainly reflect allowing a nonzero diagonal: relative to λD=0\lambda_D=0, tail accuracy does not improve on most datasets. The share of users whose Top-20 set changes varies widely across datasets, from a small minority to all users, typically with only a few items replaced (Table 15).

W1/Q2: We retuned both methods on ten random splits, tested significance with Holm correction (Appendix C.2), and separated the contribution of the diagonal penalty.

Accuracy is comparable across the two methods. Differences are small and dataset-dependent, and none of the 12 dataset-metric comparisons is significant after Holm correction. Table 8 reports the mean differences, Uni-L2L_2 minus DEQL(L2L_2), and Holm-adjusted p-values from a one-sided corrected t-test:

DatasetΔR@20 (adj. p)ΔN@20 (adj. p)
Games+0.0070 (0.96)+0.0010 (1.00)
Beauty-0.0011 (1.00)+0.0007 (1.00)
Gowalla-1+0.0003 (1.00)+0.0001 (1.00)
ML-20M+0.0006 (0.73)+0.0014 (0.072)
Netflix+0.0002 (0.27)+0.0001 (0.29)
MSD0.0000 (1.00)0.0000 (1.00)

Before correction, the same three comparisons as in the original five-split test have p-values below 0.05. The largest mean gain, on Games, is small relative to the variation across splits. We have revised Section 5.2 and the abstract to reflect these results.

The sweeps in Appendix C.3 suggest that full-matrix regularization accounts for most of the gain over the unregularized model, while λD\lambda_D mainly redistributes accuracy across popularity groups. Taken together, the results support an interpretable extension with comparable accuracy.

W2/Q3 (a): Does the computational advantage extend to the final regularized model, Uni-L2L_2?

Yes. Appendix C.1, Table 6 now benchmarks Uni-L2L_2 directly against DEQL(L2L_2), each at its own validation-selected configuration, on CPU in double precision. Memory is measured as peak incremental RSS in GiB.

DatasetDEQL time (s)Uni time (s)DEQL memoryUni memory
Games0.1560.0620.080.10
Beauty51.539.3010.044.05
Gowalla-1247.440.9132.1612.74
ML-20M91.9515.5915.556.19
Netflix69.8711.6312.164.85
MSD699.1107.464.7825.52
Amazon-Book3882.61373.6320.62250.32
Yelp2018497.778.5355.4021.85
Gowalla642.5103.464.3025.34

Uni-L2L_2 is faster on all nine datasets and uses less peak incremental memory on eight. On Games, memory increases by 0.02 GiB. At the identical DEQL(L2L_2) configuration with λD=0\lambda_D=0, our solver reproduces the DEQL(L2L_2) solution to numerical precision at nearly the same cost as the final model. Adding diagonal regularization requires no additional factorization.

These timings cover only the solver. Part of the speedup comes from the implementation: our regularized solver uses Cholesky inversion except on Amazon-Book, while the column-wise implementation uses general inversion. On Amazon-Book, where both use general inversion, the speedup is 2.83 times. Table 2 also compares the unregularized solvers using the same inversion routine. Both methods retain O(n3)O(n^3) time and O(n2)O(n^2) memory complexity (Appendix E); the savings come from constant factors and reuse.

W2/Q3 (b): Does the advantage remain after accounting for the extra hyperparameter search?

With the solver and base grid held fixed, searching λD\lambda_D covers six times as many configurations and takes 1.1 to 2.7 times as long. Appendix C.1, Table 7 reports this controlled cost benchmark; it is separate from the searches used for the accuracy tables.

On the six strong-generalization datasets, both methods search the same (p,λ~W,α)(p,\tilde\lambda_W,\alpha) grid from scratch on one GPU, using the same solver and evaluation code. Uni-L2L_2 also searches λD\lambda_D. All settings sharing (p,λ~W)(p,\tilde\lambda_W) reuse one factorization and zero-diagonal score matrix (Corollary 4.4), so both methods perform the same number of factorizations. The extra search therefore adds only validation evaluations. End-to-end time includes data loading, Gram construction, all solves and validation evaluations, and model selection.

DatasetDEQL time (s)Uni time (s)RatioPeak RSS, GiB (DEQL / Uni)Peak GPU memory, GiB
Games2.366.132.604.76 / 4.751.57
Beauty198.3264.01.337.53 / 7.542.63
Gowalla-1683.8756.91.1113.67 / 13.685.24
ML-20M357.0580.21.639.22 / 9.223.99
Netflix539.51451.72.698.37 / 8.765.68
MSD2136.63339.61.5623.74 / 23.7315.93

Memory use is essentially the same for both methods. This comparison is conservative for Uni-L2L_2: in this benchmark, DEQL(L2L_2) already uses our whole-matrix solver and factorization reuse, while its original column-wise solver is several times slower per solve (Table 6). Reuse keeps the additional cost of tuning λD\lambda_D modest, as also summarized in Section 5.2.

Thank you for your response. In particular, I appreciate the clear explanation, through Theorem 4.3 and Corollaries 4.4 and 4.5, of how diagonal regularization works and how the proposed method differs theoretically from DEQL. The added analyses of the final model’s computational efficiency and the additional cost of hyperparameter search also address gaps in the original experimental evaluation. However, I remain concerned about the practical utility of diagonal regularization, a central component of the proposed method. It remains unclear whether the additional flexibility it offers translates into sufficient practical benefit. I will therefore maintain my original assessment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Changes

We sincerely thank all three reviewers for their careful and constructive feedback. Your questions helped us clarify what diagonal regularization adds and test where it makes a difference in practice. All additions and modifications in the revised manuscript are highlighted in blue.

  • How diagonal control changes predictions (Section 4.3, proofs in Appendix B.8). In response to all three reviewers, we show how each column of the solution combines a rescaled zero-diagonal predictor with a self-reconstruction weight (Theorem 4.3). For an unobserved item, only the first term contributes. Its score equals the zero-diagonal score times an item-specific factor that depends on popularity and prediction difficulty (Corollary 4.4). At fixed masking rate and normalized ridge, Corollary 4.5 establishes when tuning emphasis in DEQL(L2L_2) can reproduce the off-diagonal coefficients and when a separate diagonal penalty yields new predictors.

  • Effects across popularity groups (Appendix C.4 and Figure 1 in Section 5.2). At the request of Reviewers rw3E and htZy, we varied λD\lambda_D while holding all other hyperparameters fixed. We relate the multipliers in Corollary 4.4 to accuracy by popularity group, with bootstrap confidence intervals. We also report exposure, catalog coverage, and the fraction of users whose Top-20 set changes. Aggregate accuracy changes little, but accuracy and exposure shift across popularity groups.

  • Selecting for tail recall (Appendix C.9). Following the discussion with Reviewers rw3E and htZy, we selected each model for the highest validation tail R@20 among configurations reaching at least 99.5% of the best DEQL(L2L_2) validation R@20. The reference could choose either diagonal endpoint and included DEQL(L2L_2) at every emphasis value on the common grid. On four of seven datasets, an intermediate λD\lambda_D gave significantly higher test tail R@20 than the selected endpoint, with small changes in overall accuracy. We also compare with post-hoc power-law popularity rescaling, which gives a different trade-off between tail recall and overall accuracy. The four gains illustrate the practical significance of Corollary 4.5 under this selection rule.

  • Connection to linear masked autoencoders (Section 4.2 and Appendix D.1). We thank Reviewer htZy for pointing us to Bisulco et al. (2025). With singleton patches, equal emphasis, the same rank constraint, and λD=λW=0\lambda_D=\lambda_W=0, the two objectives coincide. Our weighted low-rank solution is then an exact minimizer and recovers an optimal encoder-decoder product from their analysis. Appendix E and the Conclusion present applications to vision and language as future directions.

  • Accuracy across random splits (Appendix C.2, abstract, and Section 5.2). To address all three reviewers' concerns about the size and consistency of the gains, we retuned Uni-L2L_2 and DEQL(L2L_2) on ten random splits. None of the 12 comparisons was significant after a corrected resampled t-test with Holm correction. The abstract and Section 5.2 now describe the two methods as comparable in accuracy.

  • Solver and search costs (Appendix C.1 and Section 5.2). For Reviewer rw3E, we benchmarked the final Uni-L2L_2 and DEQL(L2L_2) solvers for runtime and peak memory and clarified how the choice of inversion routine affects the speedup. For Reviewers rw3E and htZy, Table 7 compares end-to-end search costs for these two methods on a common base grid. Searching λD\lambda_D adds validation evaluations while reusing the same factorizations.

  • Masked-only reconstruction (Proposition 4.2 and Appendix C.5). For Reviewer htZy, we analyzed and tested the case α=0\alpha=0. The diagonal penalty leaves cross-item predictions unchanged. Any positive total penalty gives a zero-diagonal solution. With both penalties zero, the diagonal is arbitrary.

  • Sensitivity to masking (Appendix C.6). For Reviewer htZy, we varied pp alone and jointly with λW\lambda_W on all nine datasets.

  • Hyperparameter selection (Section 5.1 and Appendix C.7). We report the search grids requested by Reviewer htZy and clarify that each method, including each baseline, is tuned independently under the same validation protocol.

  • Related work (Section 2 and Appendix A). In response to Reviewers htZy and UH3D, Section 2 compares how EASE, EDLAE, RLAE, RDLAE, and DEQL treat the diagonal. Appendix A now discusses Pu et al. (2025) alongside other low-rank and scalable linear recommenders.

  • Cold-start recommendation (Appendix E). Following Reviewer UH3D's suggestion, we clarify how fold-in supports unseen users with observed interactions and why zero-interaction cold start remains outside the model's scope.

Please let us know if any of your concerns remain unaddressed. We will update the revision accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.