acceptodds
Under review as a conference paper at ICLR 2027

Orthogonal Auto-Selective Encoding Tucker Factorization

Abstract

Neural Tucker factorization provides a promising framework for learning representations of high-order tensor data by modeling multiplicative interactions among mode-specific embeddings. The outer-product construction from mode-specific embeddings defines a latent interaction space whose coordinates are not explicitly organized according to their relevance to downstream tasks. A central challenge is to adapt this space so that task-relevant interaction directions can be identified and retained. We propose Orthogonal Auto-Selective Encoding Tucker Factorization (OASENT), which combines an orthogonal coordinate transformation with a task-driven selection module. A Householder-parameterized layer learns a norm-preserving coordinate transformation. A Gumbel-Softmax selector then forms a compact -channel task representation using continuous masks during training and deterministic coordinate selections at inference. The nonlinear decoder then maps this compact representation to the original interaction dimension for prediction. Our analysis shows that a full-dimensional orthogonal transformation alone leaves the represented decoder function class unchanged. Subsequent selection forms a bottleneck with effective width at inference. Across seven real-world tensor datasets, OASENT significantly outperforms all evaluated baselines. All findings indicate that adapting the interaction representation space before selective encoding and decoding can improve higher-order tensor representation learning.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Neural Tensor Factorization, Tensor Representation Learning, Orthogonal Parameterization, Subspace Selection

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WdG2OpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes OASENT, a neural tensor factorization framework for tensor completion. It applies a Householde orthogonal transformation and a Gumbel-Softmax selector to form a compact task representation. The orthogonal basis and selection are jointly optimized. The experiments verify the performance gains across diverse datasets and metrics.

Critical strengths

The experiments are thoroughly designed with rigorous logic, extensive coverage, and comprehensive evaluation metrics.

Critical weaknesses
  • As mentioned in section 3.5, the full-dimensional orthogonal transformation alone can be absorbed into an unconstrained first decoder layer and therefore does not change the represented decoder function class. This makes the Householder layer look like an unnecessary one.
  • Are the Gumbel-Softmax logits generated from fixed parameters? If it does not depend on the input sample, the model picks the same coordinates for every observation, which is not truly adaptive in the usual sense.
  • Does the theoretical analysis prove that the learned basis makes the selected coordinates more useful than arbitrary coordinates?
  • Computational cost is not compared in practice. How about efficiency and computational cost? Suggest conducting experiments to demonstrate the increase in computational complexity as rank increases, such as the real runtime and memory usage of OASENT compare with the baselines.
  • How stable is OASENT without dataset-specific tuning? Is Hyperparameter sensitivity high?
Questions for authors
  • If the orthogonal layer only works because selection follows it, what is the real architectural benefit of rotating the space instead of just selecting from the original space?
  • Are the Gumbel-Softmax logits generated from fixed parameters? Does the auto-selective design learn an input-dependent feature?
  • Does the theoretical analysis prove that the learned basis makes the selected coordinates more useful than arbitrary coordinates?
  • How about efficiency and computational cost in practice? Suggest conducting experiments to demonstrate the increase in computational complexity as rank increases, such as the real runtime and memory usage of OASENT compare with the baselines.
  • Is Hyperparameter sensitivity high?
5 comments
AuthorsOpenReview ↗

Response 1

We thank reviewer for the thoughtful review, constructive feedback, and recognition of our work’s strengths.

Weakness 1 As mentioned in section 3.5, the full-dimensional orthogonal transformation alone can be absorbed into an unconstrained first decoder layer and therefore does not change the represented decoder function class. This makes the Householder layer look like an unnecessary one.

Question 1 If the orthogonal layer only works because selection follows it, what is the real architectural benefit of rotating the space instead of just selecting from the original space?

Thank you for the question. The orthogonal transformation lets each selected channel retain a learned linear combination of the original interaction coordinates. With direct selection, z=xP⊤z=xP^\top, each channel retains a single original coordinate. With the learned basis, z=xQP⊤z=xQP^\top, each channel retains the projection of xx onto a learned basis direction, which can combine multiple original coordinates. From the view of information flow, jointly learning Q\mathbf{Q} and the selector thus lets the objective guide the transformation to allow more essential task-relevant information passed through the selection bottleneck, yielding performance gain.

The comparison between the two model variants ν2\nu_2 and ν1\nu_1 in Section 4.3 verifies the benefit directly. Both variants retain the selector and nonlinear decoder under the same channel budget. The former selects from the original coordinates, while the latter selects with the learned orthogonal basis. Learning the basis help reducing mean RMSE by 6.8% on NYCTaxi, 31.3% on PCTemp, and 2.7% on QoS-RT. These results show that selection after space rotation improved prediction relative to selecting directly from the original coordinates in the tested settings.

We will improve the presentation in Sections 3.5 and 4.3 to clarify the architectural benefit.

AuthorsOpenReview ↗

Response 2

Weakness 2 Are the Gumbel-Softmax logits generated from fixed parameters? If it does not depend on the input sample, the model picks the same coordinates for every observation, which is not truly adaptive in the usual sense.

Question 2 Are the Gumbel-Softmax logits generated from fixed parameters? Does the auto-selective design learn an input-dependent feature?

Thanks for the question. The Gumbel–Softmax logits are generated from trainable parameters that are updated during training. It allows the free exploration of different subset of direction selections.

When training complete, it is held fixed at inference. Every observation uses the same deterministic selection matrix P\mathbf{P} at inference. The selected indices are therefore shared, while the selected feature values depend on each observation through z=xQP⊤\mathbf{z}=\mathbf{x}\mathbf{Q}\mathbf{P}^{\top}.

In OASENT, auto-selective refers to task-supervised global coordinate selection. The prediction loss jointly learns the orthogonal basis Q\mathbf{Q} and the selector, determining which interaction directions are retained within the KK-channel budget.

Weakness 3, Question 3 Does the theoretical analysis prove that the learned basis makes the selected coordinates more useful than arbitrary coordinates?

We thank the reviewer to raise this insightful question. Our theoretical analysis only establishes the isometry of the orthogonal transform and characterizes the effective width of selection. Establishing when a learned basis help retaining more task-relevant information than an arbitrary basis remains an open question. A predictive guarantee would require additional assumptions relating the target to the retained subspace. Recent work on sufficient dimension reduction and orthogonal multi-index models provides starting points for such an analysis [1, 2]. We view extending our work along this as a promising future research direction.

Empirically, Section 4.3 shows that learning an orthogonal basis before selection yields lower mean RMSE on all three tested datasets compared with selecting directly from the original tensor-product coordinates. Both variants retain the selector and nonlinear decoder, so this comparison provides evidence for the usefulness of the learned basis.

References

[1] T. Pautrel and F. Portier. “Riemannian Stochastic Optimization for Sufficient Dimension Reduction.” ICML, 2026.

[2] Y. Ren and J. D. Lee. “Learning Orthogonal Multi-Index Models: A Fine-Grained Information Exponent Analysis.” NeurIPS, 2025.

AuthorsOpenReview ↗

Response 3

Weakness 4 Computational cost is not compared in practice. ...

Question 4 How about efficiency and computational cost in practice? ...

Thank you for the question and suggestion.

Comparison with baselines at optimal ranks. On PCTemp, the default Householder OASENT at R=5R=5 achieved 18.56% lower mean RMSE than CoATF at its validation-selected rank, with approximately 4.0× the total training time and 2.0× the test-set evaluation time.

ModelRRRMSETotal Training Time (min)Per Epoch Training Time (s)Inference Time (s)Peak training GPU allocation (MiB)
CPD82.7761 ± 0.11091.09 ± 0.010.58910.4630.46
CoSTCo320.8923 ± 0.00852.91 ± 0.290.62010.76121.48
HOCTC200.8823 ± 0.20603.65 ± 0.060.94212.778176.91
CoATF300.6508 ± 0.01802.92 ± 0.630.68611.07040.54
NEAT81.1246 ± 0.03173.19 ± 0.060.63710.95418.12
ConvNTC320.7544 ± 0.01793.71 ± 0.010.74111.31653.99
ANTucF50.6861 ± 0.05682.86 ± 0.380.63110.82119.33
OASENT (Householder)50.5300 ± 0.009711.63 ± 0.022.32521.80942.95

Rank scaling within OASENT. We vary the orthogonal parameterization while retaining the selection and decoding design. Householder achieved the lowest mean RMSE at every tested rank. Cayley and Gram–Schmidt offered lower measured runtime and peak GPU allocation at those same ranks. In the Householder implementation, the transform applies D=R3D=R^3 reflections to DD-dimensional interaction vectors before selection. For a batch of size ss, this stage costs O(sD2)=O(sR6)\mathcal{O}(sD^2)=\mathcal{O}(sR^6).

ModelRRRMSETotal Training Time (min)Per Epoch Training Time (s)Inference Time (s)Peak training GPU allocation (MiB)
OASENT (Householder)50.5300 ± 0.009711.63 ± 0.022.32521.80942.95
OASENT (Cayley)50.6059 ± 0.02483.43 ± 0.010.68411.62320.55
OASENT (Gram-Schmidt)50.6623 ± 0.05013.49 ± 0.050.69811.89322.34
OASENT (Householder)60.4941 ± 0.007217.97 ± 0.143.59429.599139.38
OASENT (Cayley)60.5460 ± 0.01623.49 ± 0.060.69811.51124.32
OASENT (Gram-Schmidt)60.5577 ± 0.02674.16 ± 0.020.83014.15124.14
OASENT (Householder)80.4450 ± 0.001141.73 ± 0.018.34464.0491571.25
OASENT (Cayley)80.5180 ± 0.00564.17 ± 0.360.83415.28238.00
OASENT (Gram-Schmidt)80.4755 ± 0.01914.72 ± 0.030.94316.33237.00

Above results also suggest that choose a different orthogonal parameterization in OASENT can help reducing the computational cost while maintain its accuracy in a certain level. How to improve the computation efficiency with parallel computing is also an open problem worth of investigation in future.

We will add these results to Appendix E.2 and explain the accuracy–cost trade-off in the discussion.

Reviewer_YhTZOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes OASENT, a neural tensor factorization framework that couples a Householder-parameterized orthogonal coordinate transformation with Gumbel-Softmax-based coordinate selection for high-order interaction tensor completion. The theoretical analysis of isometry and the effective width of hard selection is clear, and the empirical study on seven real-world tensor datasets is promising.

Critical strengths

The paper introduces supervisory information that depends on the performance of downstream tasks during the tensor decomposition (compression) process. This idea has intuitive plausibility.

The paper conducts a reliable theoretical analysis of several key viewpoints regarding the modeling motivation and the algorithm implementation.

The experiments in the paper are quite detailed, and to a certain extent, they support the main idea of the paper.

Critical weaknesses

No code link or data-split files are provided. The effect of the model seems to be more sensitive to E_target. The E_target is selected per dataset on the validation split; the selection protocol and its isolation from the test set should be stated clearly.

Questions for authors

Would you please provide code, split files, and a complete hyperparameter table.

2 comments
AuthorsOpenReview ↗

Response 1

We thank the reviewer for his time and valuable feedback. We appreciate all the positive comments on the soundness of our work. We are glad that the reviewer found our theoretical analysis and empirical results are promising.

Weakness 1 No code link or data-split files are provided. The effect of the model seems to be more sensitive to E_target. The E_target is selected per dataset on the validation split; the selection protocol and its isolation from the test set should be stated clearly.

Question 1 Would you please provide code, split files, and a complete hyperparameter table.

Thank you for the suggestion. We have provided the code, fixed train/validation/test split files, and PCTemp configuration files in an anonymized repository at https://anonymous.4open.science/r/review-OASENT-63E8/. Due to upload size limits of anonymous repository, only the split data files of PCTemp dataset is currently included. The remaining datasets and their splits will be released later in Github after acceptance.

Below is the complete hyperparameter table with the dataset-specific hyperparameters. Here, λ\lambda denotes the L2L_2 regularization coefficient, ϵ\epsilon a model-specific scaling factor, and η\eta the dropout rate. For OASENT, EtargetE_{\mathrm{target}} is the target epoch at which the temperature reaches τmin⁡\tau_{\min}.

The table will be added as Table 3 in Appendix B.2.

ModelHyperparameterD1: HZMetroD2: NYCTaxiD3: PCTempD4: NAPrecD5: QoS-RTD6: QoS-TPD7: Alibaba
CPDλ\lambda10−410^{-4}10−410^{-4}10−410^{-4}10−410^{-4}10−410^{-4}10−410^{-4}10−410^{-4}
CoSTCoλ\lambda10−410^{-4}10−410^{-4}10−410^{-4}10−410^{-4}10−410^{-4}10−410^{-4}10−410^{-4}
HOCTCϵ\epsilon0.50.50.50.50.50.50.5
HOCTCλ\lambda10−410^{-4}10−410^{-4}10−410^{-4}10−410^{-4}10−410^{-4}10−410^{-4}10−410^{-4}
CoATFη\eta0.30.30.30.30.30.30.3
CoATFλ\lambda10−410^{-4}10−410^{-4}10−410^{-4}10−410^{-4}10−410^{-4}10−410^{-4}10−410^{-4}
NEATλ\lambda0010−410^{-4}10−410^{-4}000
ConvNTCϵ\epsilon0.50.50.50.50.50.50.5
ConvNTCλ\lambda10−410^{-4}10−410^{-4}10−410^{-4}10−410^{-4}10−410^{-4}10−410^{-4}10−410^{-4}
ANTucFλ\lambda0010−410^{-4}10−410^{-4}000
OASENTEtargetE_{\mathrm{target}}19020520010010075175

Many thanks again for your time and consideration, please let us know if we have addressed the concerns and increased your confidence in our work.

AuthorsOpenReview ↗

To Reviewer YhTZ

Dear Reviewer YhTZ,

We also sincerely appreciate the time and effort you have devoted to reviewing our work. We are especially grateful for your positive comments, which mean a great deal to us.

If you have any further questions, please let us know. We are committed to refining our work based on your suggestions.

Best Regards,

Authors of Submission 24107

Reviewer_hgasOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Orthogonal Auto-Selective Encoding Tucker Factorization (OASENT) for tensor completion. It constructs interaction features from mode-specific embeddings, learns an orthogonal transformation, and uses Gumbel-Softmax to learn which features to select. A nonlinear decoder then maps the selected features to a representation used for prediction. The theoretical analysis explains the geometry preserved by the transformation and how repeated selections affect the number of distinct features retained. Experiments on seven real-world datasets compare OASENT with seven baselines and report the lowest mean errors across all 21 dataset–metric combinations.

Critical strengths

S1. The method has a clear and coherent design. OASENT first learns an orthogonal coordinate system for interaction features, then selects a compact set of features for nonlinear prediction. The stages serve distinct, understandable purposes.

S2. The theoretical analysis clarifies the roles of transformation and selection. It shows that the orthogonal transformation preserves distances and inner products, and explains how repeated selections determine the effective representation width.

S3. OASENT achieves consistently strong results across seven datasets. It obtains the lowest mean error against seven baselines across all 21 dataset–metric combinations. Results over ten independent runs, with reported standard deviations, help assess the consistency of these gains.

S4. The additional experiments provide useful evidence for the design choices. The paper compares alternative orthogonal transformations, encoding mechanisms, and selection with and without duplicates. It also examines sensitivity to the selection budget and temperature schedule.

Critical weaknesses

W1. The ablation does not isolate the contribution of feature selection. Removing the selection module also removes the nonlinear decoder, so the performance drop cannot be attributed to feature selection alone. Appendix C.2 compares different encoders, but all reduce the number of features before decoding. A comparison with a nonlinear decoder that directly uses all features is needed to establish the additional benefit of feature selection.

W2. The benefit of selecting the same feature multiple times remains unclear. The paper interprets the reduced number of distinct selected features as adaptation to each dataset. However, Table 4 shows that preventing duplicates worsens performance on D1 but improves performance on D2 and D3. These mixed results do not clarify when duplicate selection provides useful compression and when it limits prediction accuracy.

W3. The comparison does not show whether OASENT’s advantage holds when each model’s embedding rank is tuned independently. All models use a rank of five, although different architectures may perform best at different ranks. The same rank also does not give CP decomposition and neural Tucker models equal representational capacity. Although the paper mentions hyperparameter tuning, it does not report the search ranges or final settings. Each model should select its rank on validation data to make the comparison more convincing.

W4. The computational cost of OASENT’s accuracy gains remains unclear. The model transforms the full interaction vector before selecting features, and this cost grows rapidly with the embedding rank. Although the appendix analyzes computational complexity, the experiments use only a rank of five and provide no measured runtime or memory comparisons. Reporting training time, inference time, and peak memory across ranks would help determine whether the accuracy gains justify the computational cost.

Questions for authors
  • Q1. Does feature selection improve prediction compared with feeding all features directly into a nonlinear decoder? Please compare OASENT with a variant that retains the orthogonal transformation and a comparable nonlinear decoder but uses all features without selection or dimensionality reduction.

  • Q2. Why does preventing duplicate selections worsen performance on D1 but improve it on D2 and D3? Please explain when duplicate selection is beneficial and how the number of distinct selected features relates to prediction accuracy.

  • Q3. Does OASENT still outperform the baselines when each model’s embedding rank is tuned independently? Please allow each model to select its rank on validation data and report the hyperparameter search ranges and final settings.

  • Q4. How do OASENT’s runtime and memory usage compare with the baselines as the embedding rank increases? Please report training time, inference time, and peak memory alongside accuracy at several ranks to clarify the trade-off between accuracy and computational cost.

5 comments
AuthorsOpenReview ↗

Response 1

We thank the reviewer for their time and valuable feedback. We would like to take this opportunity to clarify several key points in our paper, which we believe directly address the main concerns raised.

Weakness 1 The ablation does not isolate the contribution of feature selection. Removing the selection module also removes the nonlinear decoder, so the performance drop cannot be attributed to feature selection alone. Appendix C.2 compares different encoders, but all reduce the number of features before decoding. A comparison with a nonlinear decoder that directly uses all features is needed to establish the additional benefit of feature selection.

Question 1 Does feature selection improve prediction compared with feeding all features directly into a nonlinear decoder? Please compare OASENT with a variant that retains the orthogonal transformation and a comparable nonlinear decoder but uses all features without selection or dimensionality reduction.

Thanks for your valuable suggestion. We agree that the original ablation removed the selector and the nonlinear decoder together, so it could not isolate the contribution of selection. To address this question, we added an all-feature nonlinear decoder control. It retains the orthogonal transformation and feeds the entire DD-dimensional transformed interaction vector into a two-layer MLP with width DD throughout. Both layers have width 125, so the control uses the full interaction vector throughout decoding. The control uses the same data splits and training settings as OASENT.

The table below reports the all ablation results compares this control with the existing ablations.

VariantNYCTaxiPCTempQoS-RT
OASENT w/o orthogonal transformation, feature selection, and decoder7.0598 ± 0.09361.1403 ± 0.00221.8368 ± 0.0087
OASENT w/o feature selection and decoder6.9573 ± 0.07371.1406 ± 0.00231.8217 ± 0.0059
OASENT w/o feature selection5.7657 ± 0.06790.6342 ± 0.03401.7904 ± 0.0074
OASENT w/o orthogonal transformation6.0977 ± 0.13560.7711 ± 0.05671.7856 ± 0.0063
OASENT5.6824 ± 0.05230.5300 ± 0.00971.7380 ± 0.0201

With nonlinear decoding retained, OASENT achieves lower mean RMSE than the all-feature control on all three datasets, with reductions of 1.44% on NYCTaxi, 16.43% on PCTemp, and 2.93% on QoS-RT. Adding nonlinear decoding to the full orthogonal features lowers mean RMSE on all three datasets. OASENT further lowers mean RMSE with a 15-dimensional decoder input and fewer decoder parameters.

VariantDecoder widths, from input to outputDecoder parameters
OASENT w/o feature selection125 → 125 → 12531,500
OASENT15 → 31 → 62 → 12510,355

All these results support an additional benefit from task-driven feature selection when nonlinear decoding is retained. We will add this control to Section 4.3 with updated Figure 2 and revise the discussion to distinguish the gain from adding nonlinear decoding from the further gain observed with the selection architecture.

AuthorsOpenReview ↗

Response 2

Weakness 2 The benefit of selecting the same feature multiple times remains unclear. The paper interprets the reduced number of distinct selected features as adaptation to each dataset. However, Table 4 shows that preventing duplicates worsens performance on D1 but improves performance on D2 and D3. These mixed results do not clarify when duplicate selection provides useful compression and when it limits prediction accuracy.

Question 2 Why does preventing duplicate selections worsen performance on D1 but improve it on D2 and D3? Please explain when duplicate selection is beneficial and how the number of distinct selected features relates to prediction accuracy.

We thank the reviewer for the insightful comments. Allowing repeats lets the model use fewer than K distinct directions at inference. On D1, the default selector retains an average of 3.7 distinct directions across 15 channels, showing that a narrow representation can suffice under the tested training setup. The results on D2 and D3 suggest that retaining more directions may help on those datasets. One possible explanation is that a small set of learned directions captures sufficient predictive information on D1, while a wider representation helps on D2 and D3.

The forced-distinct variant also changes mask construction during training as dictated in Appendix B.5, so the comparison cannot isolate the effect of effective width on accuracy. We agree it is an interesting open problem worth of further investigation in the future.

However, repeated selection has a precise consequence at inference. When several hard-selection channels retain the same direction, their weights can be combined in the decoder’s first layer to give a JeffJ_{\mathrm{eff}}-dimensional input with identical predictions (Lemma 1). This makes JeffJ_{\mathrm{eff}} a measure of the number of distinct directions used by the decoder.

In practice, both variants are available choices and we recommend selecting between them using validation error under the desired constraint on effective width.

We will revise Section 4.5 to clarify these findings.

AuthorsOpenReview ↗

Response 3

Weakness 3 The comparison does not show whether OASENT’s advantage holds when each model’s embedding rank is tuned independently. All models use a rank of five, although different architectures may perform best at different ranks. The same rank also does not give CP decomposition and neural Tucker models equal representational capacity. Although the paper mentions hyperparameter tuning, it does not report the search ranges or final settings. Each model should select its rank on validation data to make the comparison more convincing.

Question 3 Does OASENT still outperform the baselines when each model’s embedding rank is tuned independently? Please allow each model to select its rank on validation data and report the hyperparameter search ranges and final settings.

Thank you for raising this question. On PCTemp, OASENT at (R=5) still achieved the lowest mean MAE, MRE, and RMSE after each of the six non-Tucker baselines selected its embedding rank on the validation split. Its mean RMSE was 18.56% lower than CoATF’s, the lowest among the six tuned baselines. The rank search covers these six baselines on PCTemp. We will add these results and discussion in Table 10 of Appendix E.1.

ModelCandidate ranksRank usedMAE ↓MRE ↓RMSE ↓
CPD5, 6, 8, 10, 1282.3679 ± 0.10580.0944 ± 0.00442.7761 ± 0.1109
CoSTCo5, 8, 16, 32, 40320.6423 ± 0.00850.0267 ± 0.00030.8923 ± 0.0085
HOCTC5, 20, 30, 40200.6512 ± 0.14020.0263 ± 0.00590.8823 ± 0.2060
CoATF5, 20, 30, 40300.4908 ± 0.01330.0198 ± 0.00060.6508 ± 0.0180
NEAT5, 6, 8, 10, 1280.8546 ± 0.02530.0345 ± 0.00101.1246 ± 0.0317
ConvNTC5, 8, 16, 32, 40320.5639 ± 0.01510.0227 ± 0.00060.7544 ± 0.0179
OASENTFixed at 550.4110 ± 0.00770.0164 ± 0.00030.5300 ± 0.0097

The common training configuration is held fixed across all models, using Adam with a learning rate of 10−310^{-3}, and a batch size of 1024. All experiments are evaluated over 10 independent runs. For OASENT, the interaction dimension is D=R3=125D=R^3=125, and the temperature is annealed from the initial value τ0=15\tau_0=15 to the minimum value τmin⁡=0.1\tau_{\min}=0.1. The details on the hyperparameter setting are reported in Table 3 of Appendix B.2.

Also, we want to clarify that our original experiments answer how effectively different architectures use compact mode embeddings at R=5R=5. Recent neural tensor factorization methods illustrate how different architectures can capture nonlinear structure from compact factors [1-4]. OASENT jointly learns an orthogonal basis and uses the task objective to select relevant directions. The ν3\nu_3 control also illustrates the value of this design at a fixed rank since the selective representation obtains lower error with a smaller decoder. A compact embedding budget also matters in applications. In large-scale recommendation, embedding tables can make training memory-bound, motivating systems that offload embeddings to host memory [5]. In on-device federated recommendation, exchanging updates to large embedding tables creates communication overhead, motivating methods that reduce the payload of those updates [6]. These settings motivate asking how much predictive accuracy an architecture can achieve at a given embedding dimension.

References

[1] P. Tang, X. Luo, and J. Woodcock. “Auto-Encoding Neural Tucker Factorization.” IEEE Transactions on Knowledge and Data Engineering, vol. 37, no. 10, pp. 5795–5807, 2025.

[2] D. Ahn, U. S. Saini, E. E. Papalexakis, and A. Payani. “Neural Additive Tensor Decomposition for Sparse Tensors.” Proceedings of the 33rd ACM International Conference on Information and Knowledge Management (CIKM), pp. 14–23, 2024.

[3] X. He, L. Liao, H. Zhang, L. Nie, X. Hu, and T.-S. Chua. “Neural Collaborative Filtering.” Proceedings of the 26th International Conference on World Wide Web (WWW), pp. 173–182, 2017.

[4] X. Luo, H. Wu, Z. Wang, J. Wang, and D. Meng. “A Novel Approach to Large-Scale Dynamically Weighted Directed Network Representation.” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 12, pp. 9756–9773, 2022.

[5] I. Park et al. “Toward 100TB Recommendation Models with Embedding Offloading.” 18th ACM Conference on Recommender Systems (RecSys), 2024.

[6] P. Mai, Y. Ding, Z. Lyu, M. Du, and Y. Pang. “SecEmb: Sparsity-Aware Secure Federated Learning of On-Device Recommender System with Large Embedding.” Proceedings of the 42nd International Conference on Machine Learning (ICML), PMLR 267, pp. 42642–42667, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.