- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes RED-Rec, a hierarchical two-tower LLM retrieval framework for multi-scenario recommendation across Homefeed, Search, and Advertising. To address cross-scenario signal interference, RED-Rec introduces two main mechanisms:
Temporal Axis: Scenario-aware 2-D dense mixing: Enforces per-scenario history quotas and orders chronological blocks of size via exact dynamic programming to jointly minimize normalized temporal distortion and consecutive same-scenario block repetition, paired with a 2-D positional encoding.
Semantic Axis: Query-based MoE readout with hybrid training supervision & max-score serving: Appends 5 learnable query tokens per scenario bank to the mixed history with blockwise attention masking. During Stage I, Item LLM and User LLM are jointly trained with an item-aware hybrid routing gate and diversity/load-balancing regularizers. During Stage II, the Item LLM is frozen and user queries are refined with future-window spherical k-means + Hungarian matching NCE and an angular specialization loss. At serving time, candidate-dependent gating is replaced by parallel ANN retrieval per expert followed by parameter-free max similarity fusion.
Critical strengths
Training-serving decoupling for multi-scenario, multi-Interest two-tower retrieval: A central bottleneck in applying LLM-based multi-interest and multi-scenario models to industrial retrieval is serving latency over billion-item corpora. By enforcing a blockwise attention mask the model shares cross-scenario representations in the backbone while allowing non-target scenario query banks to be dropped at serving time with zero change to retained activations or position IDs. Pairing item-aware hybrid routing + future-window Hungarian matching during training with candidate-independent max-similarity fusion at inference makes the architecture practically deployable.
Within-backbone factorial ablations and representation diagnostics: In LLM-for-recommendation papers, gains are frequently confounded by backbone scale or pretraining differences. Here, the authors isolate their contributions on an identical Qwen2.5-1.5B backbone via factorial study of the 2-D Mixer and Query MoE, scale sweeps across 0.5B/1.5B/7B, and matched-backbone adaptations of MIND and ComiRec-SA. Furthermore, the representation diagnostics on 10,000 test users directly verify why the two components are complementary.
Critical weaknesses
Under-specification of fixed global scenario quotas under user-level sparsity: The 2-D mixer assumes fixed per-scenario quotas summing to 𝐿=128, and the active-user cohort only requires ≥30 valid Homefeed clicks and ≥5 valid Ads clicks (with no minimum on Search clicks, Section 3). Consequently, many users have far fewer than 32 Ads or Search interactions. The paper does not specify whether unfilled scenario quotas are padded (wasting up to nearly half of the 𝐿=128 sequence budget) or dynamically reallocated to Homefeed. Moreover, Table 10 shows that shifting the quota ratio from 1:1:1 to 1:2:2 causes Ads HR@10 to collapse, demonstrating that performance is highly sensitive to manual quota tuning rather than adapting to per-user activity distributions.
Potential supervision & compute imbalance against baselines and ambiguity in single-scenario public benchmarks:
RED-Rec uses a Qwen2.5-1.5B backbone plus a two-stage training pipeline (Stage I joint alignment + Stage II user-side refinement with future-window (𝑊=10) positives, Hungarian matching, and three regularizers), whereas it is unclear whether LLM baselines (HLLM, LLM4CDSR) and sequential baselines (HSTU) were trained with matched backbone capacity, matched multi-positive look-ahead windows (𝑊=10), or single next-item targets. Second, on the single-scenario Amazon Books benchmark, where the scenario-aware 2-D mixer and multi-scenario banks cannot operate, RED-Rec-1.5B-Qwen2.5 still outperforms HLLM-1B (29.87 vs. 27.22 HR@200), yet the paper never specifies which variant of RED-Rec (base vs. MoE, Stage I vs. Stage II) is evaluated in Table 6a/6b.
Questions for authors
See the weakness #1, #2 above
3 comments
thanks for the response
We thank the authors for the thorough rebuttal and clarifications. Overall, the rebuttal addresses most of our concerns, and we maintain our positive assessment.
W1: Thank you for clarifying that unfilled slots are masked rather than reallocated, and for reporting the history distribution. This resolves our main question. Two remaining notes:
The statistics describe the filtered active-user evaluation cohort and may not reflect sparser users at serving time. It would help to report performance on users who cannot fill their Ads/Search quotas
At 1:2:2, the Ads and Search quotas likely exceed many users' histories, so some of the drop to 2.71 may come from more masked slots rather than reduced Homefeed context alone. Reporting the average number of filled slots per ratio would separate the two.
W2: the new data support the readout design. However, the main baselines still differ in backbone capacity, and only RED-Rec receives Stage II multi-positive supervision. A matched-backbone HLLM run, with and without Stage II-style supervision, would show how much of the 4.08 → 5.13 gain comes from the method rather than from capacity and supervision. Please also confirm whether Stage II steps are included in the "same number of examples."
Thank You for Quick Follow-Up
Dear Reviewer yDAV,
We greatly appreciate your follow-up and your continued positive assessment.
W1. How much of the test history is masked?
For the Table A5 cohort, we measured how many of the 128 history slots contained valid events before each prediction cutoff. Overall percentages are weighted by the indicated quota ratios:
| Homefeed:Ads:Search quota ratio | Homefeed filled (%) | Ads filled (%) | Search filled (%) | Overall filled (approx. %) | Overall masked (approx. %) | Ads HR@10 (%) |
|---|---|---|---|---|---|---|
| 4:1:1 | 99.9 | 99.2 | 99.8 | 99.77 | 0.23 | 4.92 |
| 2:1:1 | 99.9 | 99.1 | 99.4 | 99.575 | 0.425 | 5.13 |
| 1:1:1 | 99.9 | 92.4 | 97.9 | 96.73 | 3.27 | 5.04 |
| 1:2:2 | 99.9 | 87.4 | 96.8 | 93.66 | 6.34 | 2.71 |
The four allocations filled approximately 128, 128, 124, and 120 of 128 slots on average. The default 2:1:1 allocation retained nearly all 128 events and achieved the highest Ads HR@10 in the sweep. Moving from 1:1:1 to 1:2:2 reduced the Homefeed quota from roughly 43 slots to 26 and left about four more slots empty. Within the original cohort, fully filled and partially padded histories had similar observed Ads HR@10.
W1. Retrieval with shorter histories.
We would like to provide some further evidence on this point. We have done some further evaluation with Homefeed+Ads as our sparse-scenario example (because Search interactions are more frequent). In the original cohort, 98.6% of users exceeded the 32-event Search quota, compared with 92.7% for Ads. We evaluated the existing checkpoint on two additional groups sampled from the same period:
| Evaluation cohort | Test set size | Valid events in the indicated scenario (mean ± SD) | Ads HR@10 (%) | Ads HR@100 (%) |
|---|---|---|---|---|
| Original active-user cohort (Table 2) | 10,000 users | — | 5.13 | 20.42 |
| Short Ads history | 10,000 examples | 14.23 ± 4.51 Ads events | 4.67 | 18.44 |
| Short Homefeed history | 10,000 examples | 49.14 ± 5.82 Homefeed events | 3.78 | 17.91 |
The short Ads group filled less than half its quota on average and scored 0.46 percentage points below the original cohort on Ads HR@10. The gap was 1.35 points for short Homefeed histories. The smaller gap suggests that Ads retrieval holds up reasonably well with short Ads histories in this cohort, although the comparison may also reflect a greater tendency among less active users to explore unfamiliar content. If you have a particular combination of history lengths across scenarios in mind, we would be happy to sample a corresponding test set and evaluate it quickly with the existing checkpoint.
W2. Training exposure and the HLLM control.
Yes. The example count for the main three-epoch comparison includes both Stage I and Stage II of RED-Rec. We also compared HLLM and RED-Rec-MoE on the same TinyLlama-1.1B backbone (We kindly refer you to our previous response to Reviewer WjAy.). HLLM was trained for more epochs than the TinyLlama version of RED-Rec-MoE. The table summarizes the reported results on the same mixed-scenario Ads test set (Table 2 and §5.5). Table A1(a) also compares RED-Rec-1.1B-LLaMA with HLLM-1B on Amazon Books.
| Training setting | Model | Ads HR@10 (%) | Ads HR@100 (%) | Ads HR@1000 (%) |
|---|---|---|---|---|
| TinyLlama (in Table 2) | HLLM | 4.08 | 19.92 | 43.27 |
| TinyLlama | HyMiRec | 4.12 | 15.31 | 38.97 |
| TinyLlama | RED-Rec-MoE | 4.31 | 17.28 | 46.37 |
| Qwen (in Table 2) | RED-Rec-MoE | 5.13 | 20.42 | 48.67 |
For some further discussion, we report the two stages together because both directly optimize the same retrieval task. Stage II retains the item-level loss and refines the queries against future-item clusters in the space learned in Stage I. Freezing the item tower gives the queries stable targets and avoids repeated item encoding. If with sufficient compute, both towers could continue training while the cluster targets are refreshed. We greatly appreciate your suggestion to test how much Stage II-style supervision adds to HLLM when its backbone and training exposure are held fixed. We are running this ablation (though it may finish after the discussion deadline) and will add it to the paper revision. Nevertheless, the existing component ablation supports our conclusion that scenario-aware mixing and the user-side MoE provide complementary gains in multi-scenario retrieval (Table 3b).
Thank you again for your thoughtful suggestions. If any questions remain, please let us know.
Dear Reviewer yDAV,
We greatly appreciate your thoughtful review and your questions. We respond to Weaknesses 1 and 2 below:
Briefly, the history distribution shows that most evaluated users can fill the default quotas. The quota sweep shows that scenario allocation affects retrieval.
We have added the observed history distribution: in the final 10,000-user evaluation cohort, 98.6% had more than 32 valid Search events, 92.7% more than 32 valid Ads events, and 99.9% more than 128 valid Homefeed events before the prediction cutoff. Most evaluated users therefore have enough history to fill the default quotas. These statistics are now reported in §3. The quotas are 64 Homefeed, 32 Ads, and 32 Search events. If a scenario has fewer events, its unused slots are masked, without reallocating them to another scenario (§4.2 and Appendix G.1). Our quota design retains substantial Homefeed context while reserving capacity for Ads and Search, preventing high-volume behavior from crowding out other scenarios. Table A5 supports this balance: Ads HR@10 remains at 4.92–5.13 for ratios of 4:1:1, 2:1:1, and 1:1:1, but falls to 2.71 when more capacity is shifted away from Homefeed at 1:2:2. The default 2:1:1 allocation achieves the highest observed HR@10 among the tested settings.
We have clarified the shared training settings and the method-specific architectural and supervision differences. The industrial HLLM, LLM4CDSR, and HSTU comparisons use their official implementations with the same training data, 128-event history limit, and three-epoch exposure, processing the same number of examples as RED-Rec. For item-level supervision, all four sample an offset uniformly from 1 to 10 after the cutoff and use that item as the positive. Each baseline retains its native loss. These settings align training-data exposure and the sampled target horizon. The revised setup and Appendix G.1 state these differences explicitly, and we also report the common-backbone readout controls below to help assess the contribution.
These results are for Homefeed + Search + Ads → Ads (Table 2). To examine readout design under a common backbone and future-window group supervision, Table 4b compares MIND, ComiRec-SA, and RED-Rec-MoE on Qwen2.5-1.5B for 20,000 steps. The adapted baselines also use the ten-event window, Stage II group matching, and regularization. Ads HR@10 is 3.58, 3.81, and 4.61, respectively. These controls provide additional evidence for the user-side readout while retaining group supervision. To avoid ambiguity, the 4.61 result belongs to this 20,000-step comparison; 5.13 is the full-training result.
Both panels of Table A1 use RED-Rec-MoE after Stage II, with a ten-interaction training window and one bank of five queries. On Amazon Books, the queries read a chronological history, with no cross-scenario interleaving; each test example has one held-out book. On Clothing–Sports, a single model shares the five-query bank across both domains, and metrics are reported by the held-out item's domain. We have made these configurations explicit in the revised caption and Appendix B.
On Books, HR@200 is 29.87 for RED-Rec-1.5B versus 27.22 for HLLM-1B, with numerically comparable NDCG@200 (8.88 versus 8.89). We interpret these results as evidence for the complete model's effectiveness without cross-scenario mixing. Because the Books backbones differ, we use the industrial controls to assess the specific contribution of readout design. On Clothing–Sports, starred results are from LLM4CDSR, and our runs follow its split, candidate-pool construction, and negative-sampling protocol.
We greatly appreciate your thoughtful review and constructive feedback. If any questions or concerns remain, please let us know.