acceptodds
Under review as a conference paper at ICLR 2027

SOLAR: SVD-Optimized Lifelong Attention for Recommendation

Abstract

Attention mechanism remains the defining operator in Transformers since it provides expressive global credit assignment, yet its quadratic cost in sequence length makes long-context modeling expensive and often forces truncation or other heuristics. Linear attention reduces complexity to by reordering computation through kernel feature maps, but this reformulation drops the softmax mechanism and shifts the attention score distribution. Lifelong recommendation requires efficient attention as well, for large-scale sequence modeling with user histories and candidate items under tight latency and resource constraints. We introduce SVD-Attention, a novel attention mechanism, and SOLAR, a set-aware framework built on it for lifelong recommendation. SVD-Attention factorizes low-rank embeddings into principal components, computes candidate-to-interest scores in compact space, and applies softmax over those scores. Its bilinear reduction is exact on the rank- reconstruction, while the normalized output approximates token-level softmax with an explicitly bounded residual. The resulting computation reduces the cost from to , and supports 12,000 behaviors and 3,000 candidates per request without filtering. SOLAR achieves the best among compared methods on RecFlow and MIND, delivers 0.8531 AUC at approximately 95th-percentile latency in an industrial evaluation, and yields business gains with a relative lift in Video Views in the real-world online A/B test.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
SVD, Attention Mechanism, Transformer, User Behavior Modeling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_bEQjOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces a recommendation model for handling long user histories and large candidate sets, namely SOLAR. The main component is SVD-Attention. It applies low-rank decomposition to historical representations and calculates attention scores over a small number of principal interests. The model also considers interactions among candidate items. Experiments on RecFlow, MIND, and industrial data evaluate its prediction performance and efficiency.

Critical strengths
  1. The idea is interesting. Handling 12,000 historical behaviors and 3,000 candidates under a serving-latency constraint is a meaningful setting for lifelong recommendation.
  2. The evaluation covers several relevant aspects.
Critical weaknesses
  1. There is something unclear in Figure 3. Both attention branches are simply labeled as “Attn”, making it difficult to distinguish SVD-Attention from candidate-set attention. The key operations of SVD-Attention are also missing in the figure.
  2. It seems that the softmax approximation relies on assumptions about principal-interest grouping and limited variation within groups. However, it is unclear whether SVD can satisfy these assumptions or whether they hold in the experimental datasets.
  3. The authors state that the code will only be released upon acceptance. This limits reproducibility of the work.
  4. As RecFlow and MIND use only 50 historical behaviors, and the evaluation with 12,000 behaviors and 3,000 candidates relies entirely on private industrial data, the public experiments may not adequately reflect the lifelong recommendation setting.
Questions for authors
  1. Could the authors clarify the two attention branches in Figure 3? Could the key operations of SVD-Attention be better illustrated?
  2. How are the principal-interest groups defined in the softmax approximation analysis?
  3. Could the authors provide experiments with longer user histories on public datasets?
8 comments
AuthorsOpenReview ↗

Response to Reviewer bEQj (Part 1/2)

Thank you for the time you spent examining our method and for your detailed comments on the analysis and evaluation. Below, we clarify the architecture, approximation assumptions, and experimental setting.

W1 / Q1. What do the two “Attn” branches in Figure 3 compute?

Response.{\color{green}\mathbf{Response.}} The upper branch is candidate-set attention, producing ZC=SetAttn⁡(C)Z_C=\operatorname{SetAttn}(C). The lower branch is SVD cross-attention over the user history, producing a separate history representation for each candidate. Its operations are:

H&\approx U\Sigma V^\top, & Q&=CW_Q,\\\\ K_r&=\Sigma V^\top W_K, & V_r&=\Sigma V^\top W_V,\\\\ Z_H&=\operatorname{softmax}_r\\!\left(\frac{QK_r^\top}{\sqrt d}\right)V_r. \end{aligned}$$ Here $V_r$ denotes the compact value matrix. Equation (11) fuses both branches as $s_i=g(c_i,(Z_H)_{i:},(Z_C)_{i:})$. The history branch recovers candidate-relevant historical evidence. The candidate branch lets the score reflect the current choice set. Table 4 tests both contributions: | Variant | RecFlow AUC | Industrial AUC | |---|---:|---:| | Candidate-set branch only | 0.6293 | 0.8012 | | User-history branch only | 0.6644 | 0.8508 | | SOLAR | **0.6812** | **0.8531** | **Our theoretical analysis explains the expressive advantage of set-wise modeling over independent point-wise modeling under contextual preference reversals (Proposition 4.1).** Building on this theoretical foundation, we introduce SVD-Attention to make set-wise modeling practical under strict industrial latency constraints. SVD-Attention exploits the empirically observed low-rank structure of history embeddings to reduce the cost of candidate-to-history attention. SOLAR retains candidate-set conditioning and processes the full lifelong history through this compact representation. This enables efficient joint modeling of lifelong histories and very large candidate sets in industrial recommendation systems. Table 4 supports this design: candidate-set context improves AUC over the history-only branch by 0.0168 on RecFlow and 0.0023 industrially. --- > #### **W2 / Q2. How are principal-interest groups defined, and does SVD ensure the approximation assumptions?** ${\color{green}\mathbf{Response.}}$ In Proposition C.1, $\\{\mathcal G_a\\}_{a=1}^r$ is an analytical partition of historical tokens associated with the compact components. The forward pass computes the SVD factors directly. The partition is used in the error analysis to express sufficient conditions for a close approximation. For $n_a=|\mathcal G_a|$ and $S=QK_r^\top/\sqrt d$, Eqs. (48)–(50) define: $$\begin{aligned} \epsilon_K&=\max_{c,a,t\in\mathcal G_a}|(SU^\top)_{ct}-S_{ca}|,\\\\ \epsilon_V&=\max_{a,t\in\mathcal G_a}\\|(UV_r)_{t:}-(V_r)_{a:}\\|_2,\\\\ \epsilon_n&=\min_{\alpha}\max_a|\log n_a-\alpha|. \end{aligned}$$ The spectral tail controls reconstruction error. Proposition C.1 separately bounds the normalized-output error through logit variation, value variation, and group-size imbalance. Table 8 evaluates the approximation directly: | Operator | $E_{\mathrm{total}}$ | $E_{\mathrm{softmax}}$ | $E_{\mathrm{solver}}$ | AUC | |---|---:|---:|---:|---:| | Full token softmax on $H$ | 0 | — | — | 0.8539 | | Token softmax on exact $H_r$ | 0.008 | 0 | — | 0.8539 | | Exact SVD-Attention | 0.033 | 0.031 | 0 | 0.8538 | | Randomized SVD-Attention | 0.035 | 0.031 | 0.002–0.004 | 0.8537 | Token softmax on the exact rank-$r$ reconstruction retains an AUC of 0.8539 with a relative output deviation of 0.008. Moving softmax to the compact components changes AUC by 0.0001. Randomized factorization contributes a further 0.002–0.004 solver deviation. These measurements separate the three sources of approximation and show their effect on prediction quality. Equation (46) gives the exact residual: $$O_U-O_{\mathrm{SVD}}= \left[\operatorname{softmax}(SU^\top)U-\operatorname{softmax}(S)\right]V_r.$$ **Eq. (9) provides the exact bilinear reduction. Eq. (10) defines the normalized approximation evaluated in Table 8.** The conditional analysis explains which properties control the residual, and the controlled experiment measures its size in the evaluated setting.
AuthorsOpenReview ↗

Response to Reviewer bEQj (Part 2/2)

W3. Releasing code only after acceptance limits reproducibility.

Response.{\color{green}\mathbf{Response.}} You can access the anonymous implementation here: SOLAR anonymous repository. Algorithm 1 specifies the randomized solver, and Eqs. (8)–(10) give the forward operator. Table 6 provides the dataset-specific training settings:

DatasetRankEmbedding dimensionHeadsTraining batch size
RecFlow86442,048
MIND86442,048
Industrial2712844,096

All three settings use Adam. The learning rate is 0.001 for RecFlow and MIND, and the industrial setting uses cosine annealing from 0.01. Together with the dataset-construction appendix, these details specify the factorization, compact attention, and training setup available for inspection.


W4 / Q3. Can longer histories be evaluated on public data?

Response.{\color{green}\mathbf{Response.}} The public experiments evaluate RecFlow and MIND with length-50 histories. To examine the contribution of longer histories, Figure 5(b) varies history length from 50 to 12,000 in the industrial pipeline:

History length501,0006,0008,00012,000
AUC0.84240.84380.84900.85100.8535

The 0.0111 AUC increase shows the value of retaining long-history evidence. The gains continue beyond 6,000 behaviors, supporting the motivation for lifelong modeling. Table 3 then evaluates the full 12,000-history/3,000-candidate serving workload, where SOLAR achieves 0.8531 AUC and 19 ms p95 latency.

This workload requires long histories paired with request-level candidates and feedback. TransAct V2 [1] also evaluates its joint lifelong-feature setting on industrial data. Our public results test ranking quality across datasets, and the industrial results quantify the benefits and serving cost at the intended scale.

[1] TransAct V2: Lifelong User Action Sequence Modeling on Pinterest Recommendation.

Reviewer_bEQjOpenReview ↗

Official Response to Authors

I appreciate the authors' efforts to address my concerns. The revised manuscript provides clearer explanations of the proposed method, additional theoretical analysis, and further experimental results. The anonymous code repository is also a positive improvement.

However, some concerns remain. First, the theoretical approximation bound relies on assumptions about principal-interest grouping, whose validity in practical scenarios has not been sufficiently demonstrated. Then, the long-history evaluation is still limited to private industrial data, leaving the generalizability of the proposed method insufficiently validated on public benchmarks. Finally, Figure 3 remains rather schematic, and the two attention branches are still not sufficiently distinguished.

Overall, although the revision has improved the presentation and experimental analysis, I do not think the main concerns regarding theoretical validity and experimental generalizability have been fully resolved. I therefore prefer to maintain my original rating.

Reviewer_4nuQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes an efficient recommendation model that compresses users' long behavior histories into principal interests using SVD and then applies softmax attention. Its key advantage is that it achieves approximately 4× faster inference while maintaining nearly the same recommendation accuracy. The model also incorporates interactions among candidate items, accounting for the fact that users' relative preferences may change depending on which other items are recommended alongside them.

Critical strengths

S1. The main strength of this paper is its use of SVD to compress long user behavior histories, substantially reducing the computational complexity of attention. The proposed method achieves a 4× improvement in inference speed while maintaining recommendation accuracy, and reduces CPU core usage by approximately 52%.

S2. The paper also incorporates interactions among candidate items, considering that users' relative preferences may vary depending on the other items in the candidate set. This candidate-set contextualization contributes to further performance improvements.

Critical weaknesses

W1. Compressing user behavior histories using SVD results in the loss of sequential order information, but the paper does not provide a mechanism to compensate for this information loss.

Questions for authors

Q1. As mentioned in W1, applying SVD to user behavior histories results in the loss of sequential order information. Do the authors believe that preserving or incorporating this information could further improve recommendation performance? If so, have they considered any potential approaches to addressing this limitation?

Q2. Do the authors believe that the optimal SVD rank may vary across users? In other words, rather than using a fixed rank of 27 for all users, could adaptively selecting the rank for each user lead to further performance improvements?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 4nuQ

Thank you for taking the time to review our work and for your helpful suggestions on temporal information and rank selection. We clarify both design choices below.

W1 / Q1. Does SVD remove sequential information, and can temporal information help?

Response.{\color{green}\mathbf{Response.}} Your concern about order is valid. Our implementation includes calendar-time information before SVD: the month and date are discretized, mapped to learned embeddings, and incorporated into each behavior embedding. Writing this as ht=Embed⁡(xt,mt,dt)h_t=\operatorname{Embed}(x_t,m_t,d_t), the factorization operates on joint item–time features.

For any row permutation PP,

(PH)⊤(PH)=H⊤H.(PH)^\top(PH)=H^\top H.

This invariance concerns reordering already constructed embeddings. Changing an item's associated month or date changes HH and can change the retained interest representation. Thus, the compressed representation can retain calendar-time cues. Your suggestion of finer sequence modeling is compatible with the same operator: position or recency embeddings, or an order-sensitive encoder, can enrich HH before factorization.


Q2. Should the SVD rank adapt to each user?

Response.{\color{green}\mathbf{Response.}} Yes, effective rank can vary across users. In our observational study, 27 was the largest effective rank observed. This provides an empirical basis for the shared industrial budget r=27r=27. RecFlow and MIND use r=8r=8 (Table 6).

Industrial rank rr1624273264
AUC, Table 70.84280.85350.85370.85340.8535

The accuracy sweep reaches a plateau around ranks 24–27. In our randomized solver, the target rank determines the sketch size before factorization. Per-user adaptation therefore requires a rank-estimation or iterative stopping mechanism. The shared budget provides predictable serving cost and reaches the observed accuracy plateau.

Thank you for the rebuttal. I appreciate the authors’ explanations regarding temporal information and rank selection. I have no major concerns about the paper and will maintain my original score.

We sincerely thank you for taking the time to review our rebuttal. We are very glad that our clarifications were helpful !

Reviewer_8VFEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies full candidate-to-history interaction for lifelong recommendation. In the industrial setting each request has 12,000 behaviors and 3,000 candidates, so dense cross-attention costs O(N_C N_L d). SVD-Attention factorizes the history with a rank-r randomized SVD, cancels the history-length-dependent left singular factor by U^T U = I_r in Eq. (9), and applies softmax over r principal interests in Eq. (10). The bilinear reduction is exact on the rank-r reconstruction. The paper states that Eq. (9) does not by itself exactly reformulate token-level softmax. Proposition C.1 bounds that residual. SOLAR adds a candidate-set branch. In Table 3, SOLAR has AUC 0.8531 against 0.8533 for full softmax, p95 latency falls from 76 ms to 19 ms, peak core usage falls by 52.38%, and throughput is 3.36 times that of full softmax. A seven-day online A/B test on a service with more than 400 million DAU reports a 0.68% relative increase in Video Views (p < 0.05).

Critical strengths
  1. Eq. (9) gives an exact bilinear reduction on the rank-r reconstruction and removes the history-length factor, while Eq. (10) keeps softmax over r interests. The forward cost is O((N_L + N_C) d r). Table 3 shows that this stays within 0.0002 AUC of full softmax while cutting p95 latency from 76 ms to 19 ms, cutting peak core usage by 52.38%, and raising throughput to 3.36 times that of full softmax, inside the 25 ms p95 target. That quality-efficiency result is the main reason to accept.
  2. The same operator is run on 12,000 behaviors and 3,000 candidates per request, and a seven-day A/B test on a service with more than 400 million DAU reports a 0.68% relative increase in Video Views (p < 0.05). This business result is obtained at the operating point the paper claims, and it supports acceptance.
Critical weaknesses
  1. The claimed regime is not tested on public data. Appendix Table 5 uses length-50 histories on both RecFlow and MIND. Section 5.1 states that SIM keeps 20 of those 50 behaviors. The reported gains at a history length of 12,000 come from non-public industrial data. This keeps the central scalability claim from being independently checked and is the main reason not to give a clear acceptance.
  2. In Table 2 the industrial AUC gain over IFA is 0.0013 (0.8531±0.00589 versus 0.8518±0.00569). Section 5.1 measures these values over a 24-hour window after streaming convergence, but it does not state the sampling unit of the standard deviations and does not report a standard error of the paired difference. RecFlow and MIND report no standard deviations. Longformer (w = 256) reaches AUC 0.8537 with p95 latency 27 ms, 2 ms above the 25 ms target. The offline quality comparison is therefore not established tightly enough for a clear acceptance. This weakness does not apply to the online Video Views result, which reports p < 0.05.
Questions for authors
  1. Are the AUC gains over IFA in Table 2 statistically significant? Please state the sampling unit of the reported standard deviations and provide a paired test, or the standard error of the difference, for the industrial and MIND results.
  2. The method is motivated by histories of 12,000 behaviors, while RecFlow and MIND use length 50. Can the authors provide a reproducible comparison at this claimed history length?
  3. Table 3 leaves the end-to-end latency of exact SVD blank. What is its end-to-end p95 latency? The latency column in Table 8 is the controlled attention time, approximately 500 ms for exact SVD and 340 ms for randomized SVD, not end-to-end serving latency.
  4. The online result covers a seven-day window with Video Views as the reported business metric. Does the Video Views gain persist over a longer window? A novelty-driven effect would change the weight this review places on the online result.
3 comments
AuthorsOpenReview ↗

Response to Reviewer 8VFE (Part 1/2)

Thank you for your time and thoughtful feedback on our evaluation. We address your questions on scalability, statistical evidence, and serving performance below.

W1 / Q2. Public evaluation does not cover the 12,000-behavior regime.

Response.{\color{green}\mathbf{Response.}} RecFlow and MIND evaluate ranking quality with length-50 histories. The industrial experiments evaluate the target setting of 12,000 behaviors and 3,000 candidates per request (Table 5). To examine the benefit of longer histories directly, Figure 5(b) varies history length within the same industrial pipeline:

History length501,0006,0008,00012,000
AUC0.84240.84380.84900.85100.8535

These controlled results show a 0.0111 AUC gain from extending the history to 12,000 behaviors. AUC improves at every evaluated length, including beyond 6,000 behaviors. Table 3 evaluates serving at the full 12,000-history/3,000-candidate setting and reports 19 ms p95 latency. Algorithm 1 and Eqs. (8)–(10) explain this scaling through a dominant cost of O((NL+NC)dr)\mathcal{O}((N_L+N_C)dr).

This setting requires lifelong histories together with request-level candidates and feedback. TransAct V2 [1] uses industrial data for a similar joint feature requirement. Our public benchmarks establish ranking quality across datasets, and the industrial ablation measures the contribution of long histories under the target workload.

[1] TransAct V2: Lifelong User Action Sequence Modeling on Pinterest Recommendation.


W2 / Q1. Are the AUC gains over IFA statistically significant?

Response.{\color{green}\mathbf{Response.}} Table 2 shows consistent AUC improvements over IFA across all three datasets:

DatasetIFA AUCSOLAR AUCAbsolute difference
RecFlow0.67690.6812+0.0043
MIND0.66970.6713+0.0016
Industrial0.8518±0.005690.8518\pm0.005690.8531±0.005890.8531\pm0.00589+0.0013

Industrial training runs continuously on over 100 million interactions per day. Consumed behaviors enter the training stream within 10 minutes to 3 hours. Each AUC reporting epoch covers approximately 30 million interactions. The industrial ±\pm values denote standard deviations, and Section 5.1 summarizes performance over 24 hours after streaming convergence.

The AUC comparisons above are descriptive effect estimates. They also agree with the Logloss reductions in Table 2, from 0.0615 to 0.0610 on RecFlow and from 0.1060 to 0.1052 on MIND. The reported p<0.05p<0.05 applies to the separate seven-day Video Views experiment.


W2. Longformer achieves higher AUC and misses the latency target by only 2 ms.

Response.{\color{green}\mathbf{Response.}} Longformer at w=256w=256 achieves 0.8537 AUC with 27 ms p95 latency. Our deployment criterion requires p95 latency at or below 25 ms. On a platform serving several hundred million daily active users, even a 2 ms overrun is operationally consequential. The additional window-size measurements show how reducing the window affects both metrics:

MethodAUCp50 / p95 latency (ms)Meets 25 ms p95 target
Longformer, w=256w=2560.853719 / 27No
Longformer, w=128w=1280.842420 / 26No
Longformer, w=64w=640.836618 / 24Yes
SOLAR, r=27r=270.853113 / 19Yes

The w=128w=128 and w=64w=64 rows are additional measurements. The other rows are from Table 3. Among the evaluated Longformer configurations, w=64w=64 meets the latency target. SOLAR achieves 0.0165 higher AUC and 5 ms lower p95 latency than this configuration. SOLAR combines near-full-softmax quality with 6 ms of margin below the serving deadline.


Q3. What is the end-to-end p95 latency of exact SVD?

Response.{\color{green}\mathbf{Response.}} Exact SVD has 35 ms end-to-end p95 latency, with p50 of 25 ms, in the industrial serving configuration:

Attention operatorAUCp50 / p95 latency (ms)Throughput (QPS)
Full token softmax0.853348 / 7622
Exact SVD-Attention0.8532525 / 3539
Randomized SVD-Attention0.853113 / 1974

These are the end-to-end measurements under the Table 3 configuration. Randomized SVD reduces p95 from 35 to 19 ms at an AUC difference of 0.00015. It also increases throughput from 39 to 74 QPS. Table 8's approximately 500/340 ms values measure attention computation in a separate controlled benchmark.

AuthorsOpenReview ↗

Response to Reviewer 8VFE (Part 2/2)

Q4. Does the online gain persist beyond seven days?

Response.{\color{green}\mathbf{Response.}} The seven-day randomized A/B test assigned 6.14% of traffic to each arm. After full rollout, reverse testing checks whether withdrawing SOLAR reduces Video Views. Its protocol uses an evaluation window of more than 14 days:

EvaluationEvaluation windowVideo Views result
Production A/B test7 consecutive days+0.68% relative lift, p<0.05p<0.05
Post-rollout reverse testMore than 14 days under the test protocolVV decreases when SOLAR is withdrawn

The observed reverse-test decline agrees with the direction of the original gain and supports the benefit after rollout. The +0.68% estimate and p<0.05p<0.05 refer to the seven-day randomized test.

Dear Reviewer 8VFE,

Thank you once again for your valuable comments on our submission. As the discussion phase is approaching its end, we would kindly ask whether you have had a chance to review our rebuttal.

We believe we have provided detailed responses and additional evidence addressing your other concerns, including statistical reporting, exact-SVD latency, and longer-term online performance. The main remaining concern may only in lack of public long-history evaluation.

However, to our knowledge, no public dataset jointly provides lifelong histories at this scale, large request-level candidate sets, and corresponding feedback. We therefore provide complementary evidence from two perspectives, as further explained in our follow-up responses.

  1. On public datasets, we conducted additional comparisons between SVD-Attention and other attention mechanisms under the same set-aware framework:
AttentionRecFlow AUC
Softmax Attention (Vaswani et al., 2017)0.6829
MALA (Fan et al., 2025)0.6759
Longformer (Beltagy et al., 2020)0.6708
Hiformer (Gui et al., 2023)0.6718
Linear Attention (Katharopoulos et al., 2020)0.6762
SVD-Attention without Softmax0.6757
SVD-Attention (Ours)0.6812

These results demonstrate that SVD-Attention achieves competitive performance compared with existing attention mechanisms, approaching full softmax attention while outperforming the other evaluated alternatives. Meanwhile, comparisons with other sequence modeling methods and ablation studies further support the effectiveness of set-wise modeling, theoretically motivated by our Proposition 4.1.

  1. On industrial data, we demonstrate not only AUC improvements but also substantial reductions in computational resources and serving latency under the full-scale workload of 12,000 behaviors and 3,000 candidates. Importantly, these efficiency benefits are further supported by reproducible experiments in Figure 5(a) and our theoretical runtime complexity analysis, rather than relying solely on private industrial evaluations.

We have provided more detailed clarifications regarding this shared concern in our follow-up responses to Reviewer bEQj. We believe that these complementary theoretical, public, and industrial results provide meaningful evidence of the practical effectiveness and computational value of our approach.

We sincerely hope these additional explanations help address your remaining concerns. As the discussion phase is approaching its end, we would greatly appreciate knowing whether our responses have sufficiently addressed your questions. Should there be any remaining concerns requiring further clarification, please do not hesitate to let us know. If you find our responses satisfactory, we would be grateful if you could reconsider your evaluation score accordingly.

Thank you very much for your time and thoughtful consideration!

Best regards,

The Authors

Comments on the paper

2 comments

Dear Area Chair,

Thank you very much for your time and effort in coordinating the review process. As the discussion phase is approaching its end, we would kindly ask for your help in encouraging the reviewers to revisit our rebuttal and follow-up responses.

We have made efforts to address the reviewers' main concerns. We are pleased that Reviewer 4nuQ has acknowledged our clarifications and confirmed that no major concerns remain. At this stage, we understand the only remaining concern is about the private long-history evaluation. However, we believe our additional clarifications and experimental evidence provide meaningful support for addressing this concern.

To our knowledge, no public dataset jointly provides lifelong histories at this scale, large request-level candidate sets, and corresponding feedback. Consequently, many influential industrial recommendation studies also rely on private industrial datasets for evaluation at this scale. We provide complementary clarification with evidence from two perspectives in response for addressing the concerns:

  • On public datasets, comparisons with different attention mechanisms demonstrate the competitive performance of SVD-Attention, approaching full softmax attention while outperforming the other evaluated alternatives. Meanwhile, comparisons with other sequence modeling methods and ablation studies support the effectiveness of set-wise modeling, theoretically motivated by Proposition 4.1, which, to the best of our knowledge, is also the first work that provides a theoretical explanation for why set-wise modeling is superior to independent point-wise modeling in this setting.

  • On industrial dataset, we demonstrate not only AUC improvements but also substantial reductions in computational resources and serving latency under the full-scale workload. These efficiency benefits are further supported by reproducible experiments in Figure 5(a) and our theoretical runtime complexity analysis, rather than relying solely on private industrial evaluations.

We have provided more detailed explanations in responses to Reviewer bEQj and Reviewer 8VFE. We believe our proposed methods will also benefit other domains while recommendation provides a particularly suitable setting, and would especially appreciate it if Reviewers 8VFE and bEQj could revisit our latest responses and let us know whether their remaining concerns have been addressed. If our clarifications are satisfactory, we would be grateful if they could reconsider their ratings accordingly.

Thank you again for your time, effort, and consideration!

Best regards,

The Authors

AuthorsOpenReview ↗

General Response

General Response.{\color{green}\mathbf{General\ Response.}} We sincerely thank the Area Chair and all reviewers for their thorough and constructive comments. We are glad that the reviewers recognize the key contributions of our work, including the efficiency of SVD-Attention in compressing lifelong user histories while preserving competitive prediction quality (Reviewer 8VFE, Reviewer 4nuQ), the effectiveness of SOLAR in modeling candidate-set interactions (Reviewer 4nuQ), and the practical significance of handling large-scale user histories and candidate sets under strict industrial serving constraints (Reviewer 8VFE, Reviewer bEQj). Based on the reviewers' valuable feedback, we have conducted additional experiments and provided further theoretical and empirical clarifications to address their concerns. The major additional experiments and improvements are summarized as follows:

  1. To address the concerns about temporal information and adaptive rank selection (Reviewer 4nuQ), we clarified how temporal information is incorporated into behavior embeddings before SVD and provided further analysis of rank selection. The results show that performance reaches a plateau around ranks 24–27, supporting our choice of a fixed rank for efficient industrial serving. We are very pleased that Reviewer 4nuQ has acknowledged our clarifications, confirmed that no major concerns remain, and maintained their positive score.

  2. To address the concerns about theoretical validity and industrial scalability (Reviewers bEQj and 8VFE), we provided additional comparisons with different attention mechanisms on public datasets, further analysis of softmax approximation fidelity, and industrial experiments on history-length scalability and end-to-end serving latency. We also clarified the two attention branches and statistical reporting details, supplemented the online A/B test with longer-term evidence from post-rollout reverse testing, and released an anonymous implementation to improve reproducibility. These additional experiments and clarifications strengthen the evidence for the effectiveness, efficiency, and practical value of SOLAR.

We also recognize that Reviewer bEQj's Q3, Reviewer 8VFE's Q2, or some other questions(which may related with theoretical validity and industrial scalability concerns above) may arise from a shared concern regarding the lack of public long-history evaluation. This concern primarily reflects a limitation in the availability of suitable public benchmarks, rather than a fundamental issue with our theoretical foundations or other experimental results, for which we have provided additional evidence and clarifications. To the best of our knowledge, no publicly available dataset currently supports the full scale and request-level structure of our industrial setting, making such an evaluation infeasible at present. Privacy, data governance, and storage requirements make such datasets difficult to release, and many influential industrial recommendation studies similarly rely on private industrial datasets for full-scale evaluation.

To deal with that concern, we have provided further follow-up clarifications and complementary evidence from two perspectives:

● On public datasets, our comparisons with different attention mechanisms demonstrate that SVD-Attention achieves competitive prediction quality, approaching full softmax attention while outperforming the other evaluated alternatives. Meanwhile, comparisons with other sequence modeling methods and ablation studies support the effectiveness of set-wise modeling, theoretically motivated by Proposition 4.1, which, to the best of our knowledge, is also the first work that provides a theoretical explanation for why set-wise modeling is superior to independent point-wise modeling in this setting.

● On industrial dataset, we demonstrate not only improvements in predictive performance but also substantial reductions in computational resources and serving latency under the full-scale workload of 12,000 behaviors and 3,000 candidates. Importantly, these efficiency benefits are also supported by reproducible experiments in Figure 5(a) and our theoretical runtime complexity analysis, rather than relying solely on private industrial evaluations.

We sincerely appreciate that our clarifications have helped address Reviewer bEQj's concerns and encouraged them to raise their score. We also hope that these further explanations and experimental results, including those shared in our follow-up discussion with Reviewer bEQj, can help Reviewer 8VFE address the remaining concerns and reconsider the evaluation accordingly. We would be very happy to provide any further clarifications if needed, and we believe the proposed methods have potential applications beyond recommendation systems. Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.