- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces a recommendation model for handling long user histories and large candidate sets, namely SOLAR. The main component is SVD-Attention. It applies low-rank decomposition to historical representations and calculates attention scores over a small number of principal interests. The model also considers interactions among candidate items. Experiments on RecFlow, MIND, and industrial data evaluate its prediction performance and efficiency.
Critical strengths
- The idea is interesting. Handling 12,000 historical behaviors and 3,000 candidates under a serving-latency constraint is a meaningful setting for lifelong recommendation.
- The evaluation covers several relevant aspects.
Critical weaknesses
- There is something unclear in Figure 3. Both attention branches are simply labeled as “Attn”, making it difficult to distinguish SVD-Attention from candidate-set attention. The key operations of SVD-Attention are also missing in the figure.
- It seems that the softmax approximation relies on assumptions about principal-interest grouping and limited variation within groups. However, it is unclear whether SVD can satisfy these assumptions or whether they hold in the experimental datasets.
- The authors state that the code will only be released upon acceptance. This limits reproducibility of the work.
- As RecFlow and MIND use only 50 historical behaviors, and the evaluation with 12,000 behaviors and 3,000 candidates relies entirely on private industrial data, the public experiments may not adequately reflect the lifelong recommendation setting.
Questions for authors
- Could the authors clarify the two attention branches in Figure 3? Could the key operations of SVD-Attention be better illustrated?
- How are the principal-interest groups defined in the softmax approximation analysis?
- Could the authors provide experiments with longer user histories on public datasets?
8 comments
Response to Reviewer bEQj (Part 2/2)
W3. Releasing code only after acceptance limits reproducibility.
You can access the anonymous implementation here: SOLAR anonymous repository. Algorithm 1 specifies the randomized solver, and Eqs. (8)–(10) give the forward operator. Table 6 provides the dataset-specific training settings:
| Dataset | Rank | Embedding dimension | Heads | Training batch size |
|---|---|---|---|---|
| RecFlow | 8 | 64 | 4 | 2,048 |
| MIND | 8 | 64 | 4 | 2,048 |
| Industrial | 27 | 128 | 4 | 4,096 |
All three settings use Adam. The learning rate is 0.001 for RecFlow and MIND, and the industrial setting uses cosine annealing from 0.01. Together with the dataset-construction appendix, these details specify the factorization, compact attention, and training setup available for inspection.
W4 / Q3. Can longer histories be evaluated on public data?
The public experiments evaluate RecFlow and MIND with length-50 histories. To examine the contribution of longer histories, Figure 5(b) varies history length from 50 to 12,000 in the industrial pipeline:
| History length | 50 | 1,000 | 6,000 | 8,000 | 12,000 |
|---|---|---|---|---|---|
| AUC | 0.8424 | 0.8438 | 0.8490 | 0.8510 | 0.8535 |
The 0.0111 AUC increase shows the value of retaining long-history evidence. The gains continue beyond 6,000 behaviors, supporting the motivation for lifelong modeling. Table 3 then evaluates the full 12,000-history/3,000-candidate serving workload, where SOLAR achieves 0.8531 AUC and 19 ms p95 latency.
This workload requires long histories paired with request-level candidates and feedback. TransAct V2 [1] also evaluates its joint lifelong-feature setting on industrial data. Our public results test ranking quality across datasets, and the industrial results quantify the benefits and serving cost at the intended scale.
[1] TransAct V2: Lifelong User Action Sequence Modeling on Pinterest Recommendation.
Official Response to Authors
I appreciate the authors' efforts to address my concerns. The revised manuscript provides clearer explanations of the proposed method, additional theoretical analysis, and further experimental results. The anonymous code repository is also a positive improvement.
However, some concerns remain. First, the theoretical approximation bound relies on assumptions about principal-interest grouping, whose validity in practical scenarios has not been sufficiently demonstrated. Then, the long-history evaluation is still limited to private industrial data, leaving the generalizability of the proposed method insufficiently validated on public benchmarks. Finally, Figure 3 remains rather schematic, and the two attention branches are still not sufficiently distinguished.
Overall, although the revision has improved the presentation and experimental analysis, I do not think the main concerns regarding theoretical validity and experimental generalizability have been fully resolved. I therefore prefer to maintain my original rating.
Response to Reviewer bEQj (Part 1/2)
Thank you for the time you spent examining our method and for your detailed comments on the analysis and evaluation. Below, we clarify the architecture, approximation assumptions, and experimental setting.
Response. The upper branch is candidate-set attention, producing ZC=SetAttn(C). The lower branch is SVD cross-attention over the user history, producing a separate history representation for each candidate. Its operations are:
H&\approx U\Sigma V^\top, & Q&=CW_Q,\\\\ K_r&=\Sigma V^\top W_K, & V_r&=\Sigma V^\top W_V,\\\\ Z_H&=\operatorname{softmax}_r\\!\left(\frac{QK_r^\top}{\sqrt d}\right)V_r. \end{aligned}$$ Here $V_r$ denotes the compact value matrix. Equation (11) fuses both branches as $s_i=g(c_i,(Z_H)_{i:},(Z_C)_{i:})$. The history branch recovers candidate-relevant historical evidence. The candidate branch lets the score reflect the current choice set. Table 4 tests both contributions: | Variant | RecFlow AUC | Industrial AUC | |---|---:|---:| | Candidate-set branch only | 0.6293 | 0.8012 | | User-history branch only | 0.6644 | 0.8508 | | SOLAR | **0.6812** | **0.8531** | **Our theoretical analysis explains the expressive advantage of set-wise modeling over independent point-wise modeling under contextual preference reversals (Proposition 4.1).** Building on this theoretical foundation, we introduce SVD-Attention to make set-wise modeling practical under strict industrial latency constraints. SVD-Attention exploits the empirically observed low-rank structure of history embeddings to reduce the cost of candidate-to-history attention. SOLAR retains candidate-set conditioning and processes the full lifelong history through this compact representation. This enables efficient joint modeling of lifelong histories and very large candidate sets in industrial recommendation systems. Table 4 supports this design: candidate-set context improves AUC over the history-only branch by 0.0168 on RecFlow and 0.0023 industrially. --- > #### **W2 / Q2. How are principal-interest groups defined, and does SVD ensure the approximation assumptions?** ${\color{green}\mathbf{Response.}}$ In Proposition C.1, $\\{\mathcal G_a\\}_{a=1}^r$ is an analytical partition of historical tokens associated with the compact components. The forward pass computes the SVD factors directly. The partition is used in the error analysis to express sufficient conditions for a close approximation. For $n_a=|\mathcal G_a|$ and $S=QK_r^\top/\sqrt d$, Eqs. (48)–(50) define: $$\begin{aligned} \epsilon_K&=\max_{c,a,t\in\mathcal G_a}|(SU^\top)_{ct}-S_{ca}|,\\\\ \epsilon_V&=\max_{a,t\in\mathcal G_a}\\|(UV_r)_{t:}-(V_r)_{a:}\\|_2,\\\\ \epsilon_n&=\min_{\alpha}\max_a|\log n_a-\alpha|. \end{aligned}$$ The spectral tail controls reconstruction error. Proposition C.1 separately bounds the normalized-output error through logit variation, value variation, and group-size imbalance. Table 8 evaluates the approximation directly: | Operator | $E_{\mathrm{total}}$ | $E_{\mathrm{softmax}}$ | $E_{\mathrm{solver}}$ | AUC | |---|---:|---:|---:|---:| | Full token softmax on $H$ | 0 | — | — | 0.8539 | | Token softmax on exact $H_r$ | 0.008 | 0 | — | 0.8539 | | Exact SVD-Attention | 0.033 | 0.031 | 0 | 0.8538 | | Randomized SVD-Attention | 0.035 | 0.031 | 0.002–0.004 | 0.8537 | Token softmax on the exact rank-$r$ reconstruction retains an AUC of 0.8539 with a relative output deviation of 0.008. Moving softmax to the compact components changes AUC by 0.0001. Randomized factorization contributes a further 0.002–0.004 solver deviation. These measurements separate the three sources of approximation and show their effect on prediction quality. Equation (46) gives the exact residual: $$O_U-O_{\mathrm{SVD}}= \left[\operatorname{softmax}(SU^\top)U-\operatorname{softmax}(S)\right]V_r.$$ **Eq. (9) provides the exact bilinear reduction. Eq. (10) defines the normalized approximation evaluated in Table 8.** The conditional analysis explains which properties control the residual, and the controlled experiment measures its size in the evaluated setting.