acceptodds
Under review as a conference paper at ICLR 2027

Scenario-Aware Multi-Interest Retrieval with Hierarchical Language Models

Abstract

Feed, search, and advertising capture different aspects of user interests, but their interaction histories differ substantially in frequency and temporal structure. Combining these heterogeneous signals requires deciding both which events enter the model and how they are represented for retrieval. We present RED-Rec (Recommender Engine for Diversified Scenarios), a hierarchical language-model framework for multi-scenario recommendation. RED-Rec first employs a scenario-aware 2-D mixing policy that combines per-scenario history quotas with blockwise temporal ordering. Learnable interest queries then extract multiple user representations from the mixed sequence. During training, an item-aware gate distributes supervision across queries; at serving time, each query independently retrieves candidates from precomputed item embeddings, and the retrieved results are merged by maximum similarity. This design enables cross-scenario interest modeling while preserving efficient two-tower retrieval. Controlled experiments isolate the effects of the mixer and interest queries on a common backbone, while matched-budget comparisons demonstrate improvements over adapted multi-interest readouts. Experiments on public benchmarks and an industrial dataset show consistent gains in retrieval accuracy, with the largest cross-scenario improvements observed for advertising. In a one-week online A/B test, RED-Rec increases advertiser value by 0.8864% and advertising spend by 0.3401%.

Primary area:
learning on time series and dynamical systems
Keywords:
Multi-Scenario Recommendation, Sequential Recommendation, Multi-Interest Recommendation, Cross-Scenario Modeling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_yDAVOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RED-Rec, a hierarchical two-tower LLM retrieval framework for multi-scenario recommendation across Homefeed, Search, and Advertising. To address cross-scenario signal interference, RED-Rec introduces two main mechanisms:

Temporal Axis: Scenario-aware 2-D dense mixing: Enforces per-scenario history quotas and orders chronological blocks of size via exact dynamic programming to jointly minimize normalized temporal distortion and consecutive same-scenario block repetition, paired with a 2-D positional encoding.

Semantic Axis: Query-based MoE readout with hybrid training supervision & max-score serving: Appends 5 learnable query tokens per scenario bank to the mixed history with blockwise attention masking. During Stage I, Item LLM and User LLM are jointly trained with an item-aware hybrid routing gate and diversity/load-balancing regularizers. During Stage II, the Item LLM is frozen and user queries are refined with future-window spherical k-means + Hungarian matching NCE and an angular specialization loss. At serving time, candidate-dependent gating is replaced by parallel ANN retrieval per expert followed by parameter-free max similarity fusion.

Critical strengths

Training-serving decoupling for multi-scenario, multi-Interest two-tower retrieval: A central bottleneck in applying LLM-based multi-interest and multi-scenario models to industrial retrieval is serving latency over billion-item corpora. By enforcing a blockwise attention mask the model shares cross-scenario representations in the backbone while allowing non-target scenario query banks to be dropped at serving time with zero change to retained activations or position IDs. Pairing item-aware hybrid routing + future-window Hungarian matching during training with candidate-independent max-similarity fusion at inference makes the architecture practically deployable.

Within-backbone factorial ablations and representation diagnostics: In LLM-for-recommendation papers, gains are frequently confounded by backbone scale or pretraining differences. Here, the authors isolate their contributions on an identical Qwen2.5-1.5B backbone via factorial study of the 2-D Mixer and Query MoE, scale sweeps across 0.5B/1.5B/7B, and matched-backbone adaptations of MIND and ComiRec-SA. Furthermore, the representation diagnostics on 10,000 test users directly verify why the two components are complementary.

Critical weaknesses

Under-specification of fixed global scenario quotas under user-level sparsity: The 2-D mixer assumes fixed per-scenario quotas summing to 𝐿=128, and the active-user cohort only requires ≥30 valid Homefeed clicks and ≥5 valid Ads clicks (with no minimum on Search clicks, Section 3). Consequently, many users have far fewer than 32 Ads or Search interactions. The paper does not specify whether unfilled scenario quotas are padded (wasting up to nearly half of the 𝐿=128 sequence budget) or dynamically reallocated to Homefeed. Moreover, Table 10 shows that shifting the quota ratio from 1:1:1 to 1:2:2 causes Ads HR@10 to collapse, demonstrating that performance is highly sensitive to manual quota tuning rather than adapting to per-user activity distributions.

Potential supervision & compute imbalance against baselines and ambiguity in single-scenario public benchmarks:

RED-Rec uses a Qwen2.5-1.5B backbone plus a two-stage training pipeline (Stage I joint alignment + Stage II user-side refinement with future-window (𝑊=10) positives, Hungarian matching, and three regularizers), whereas it is unclear whether LLM baselines (HLLM, LLM4CDSR) and sequential baselines (HSTU) were trained with matched backbone capacity, matched multi-positive look-ahead windows (𝑊=10), or single next-item targets. Second, on the single-scenario Amazon Books benchmark, where the scenario-aware 2-D mixer and multi-scenario banks cannot operate, RED-Rec-1.5B-Qwen2.5 still outperforms HLLM-1B (29.87 vs. 27.22 HR@200), yet the paper never specifies which variant of RED-Rec (base vs. MoE, Stage I vs. Stage II) is evaluated in Table 6a/6b.

Questions for authors

See the weakness #1, #2 above

3 comments

Dear Reviewer yDAV,

We greatly appreciate your thoughtful review and your questions. We respond to Weaknesses 1 and 2 below:

Weakness 1: Fixed quotas under user-level sparsity and sensitivity to history allocation. & How are unfilled scenario quotas handled, and what do the history distribution and quota sweep show?

Briefly, the history distribution shows that most evaluated users can fill the default quotas. The quota sweep shows that scenario allocation affects retrieval.

We have added the observed history distribution: in the final 10,000-user evaluation cohort, 98.6% had more than 32 valid Search events, 92.7% more than 32 valid Ads events, and 99.9% more than 128 valid Homefeed events before the prediction cutoff. Most evaluated users therefore have enough history to fill the default quotas. These statistics are now reported in §3. The quotas are 64 Homefeed, 32 Ads, and 32 Search events. If a scenario has fewer events, its unused slots are masked, without reallocating them to another scenario (§4.2 and Appendix G.1). Our quota design retains substantial Homefeed context while reserving capacity for Ads and Search, preventing high-volume behavior from crowding out other scenarios. Table A5 supports this balance: Ads HR@10 remains at 4.92–5.13 for ratios of 4:1:1, 2:1:1, and 1:1:1, but falls to 2.71 when more capacity is shifted away from Homefeed at 1:2:2. The default 2:1:1 allocation achieves the highest observed HR@10 among the tested settings.

Weakness 2(a): Potential supervision and compute imbalance against baselines. & How are baseline capacity, training exposure, and future-window supervision controlled?

We have clarified the shared training settings and the method-specific architectural and supervision differences. The industrial HLLM, LLM4CDSR, and HSTU comparisons use their official implementations with the same training data, 128-event history limit, and three-epoch exposure, processing the same number of examples as RED-Rec. For item-level supervision, all four sample an offset uniformly from 1 to 10 after the cutoff and use that item as the positive. Each baseline retains its native loss. These settings align training-data exposure and the sampled target horizon. The revised setup and Appendix G.1 state these differences explicitly, and we also report the common-backbone readout controls below to help assess the contribution.

ModelEncoder and training objectiveAds HR@10 (%)
HLLMJointly trained item and user LLMs with a contrastive retrieval loss4.08
LLM4CDSRSASRec encoders with LLM-derived features, binary cross-entropy, and contrastive alignment3.82
HSTUHSTU sequential transducer trained on the sampled target with its retrieval loss3.92
RED-Rec-MoEQwen2.5-1.5B, with joint item/user training in Stage I and user-side cluster matching in Stage II5.13

These results are for Homefeed + Search + Ads → Ads (Table 2). To examine readout design under a common backbone and future-window group supervision, Table 4b compares MIND, ComiRec-SA, and RED-Rec-MoE on Qwen2.5-1.5B for 20,000 steps. The adapted baselines also use the ten-event window, Stage II group matching, and regularization. Ads HR@10 is 3.58, 3.81, and 4.61, respectively. These controls provide additional evidence for the user-side readout while retaining group supervision. To avoid ambiguity, the 4.61 result belongs to this 20,000-step comparison; 5.13 is the full-training result.

Weakness 2(b): Ambiguity in the public-benchmark variants and interpretation of single-scenario gains. & Which RED-Rec variants are evaluated on the public benchmarks, and how should the Books gains be interpreted without cross-scenario mixing?

Both panels of Table A1 use RED-Rec-MoE after Stage II, with a ten-interaction training window and one bank of five queries. On Amazon Books, the queries read a chronological history, with no cross-scenario interleaving; each test example has one held-out book. On Clothing–Sports, a single model shares the five-query bank across both domains, and metrics are reported by the held-out item's domain. We have made these configurations explicit in the revised caption and Appendix B.

On Books, HR@200 is 29.87 for RED-Rec-1.5B versus 27.22 for HLLM-1B, with numerically comparable NDCG@200 (8.88 versus 8.89). We interpret these results as evidence for the complete model's effectiveness without cross-scenario mixing. Because the Books backbones differ, we use the industrial controls to assess the specific contribution of readout design. On Clothing–Sports, starred results are from LLM4CDSR, and our runs follow its split, candidate-pool construction, and negative-sampling protocol.

We greatly appreciate your thoughtful review and constructive feedback. If any questions or concerns remain, please let us know.

Reviewer_yDAVOpenReview ↗

thanks for the response

We thank the authors for the thorough rebuttal and clarifications. Overall, the rebuttal addresses most of our concerns, and we maintain our positive assessment.

W1: Thank you for clarifying that unfilled slots are masked rather than reallocated, and for reporting the history distribution. This resolves our main question. Two remaining notes:

The statistics describe the filtered active-user evaluation cohort and may not reflect sparser users at serving time. It would help to report performance on users who cannot fill their Ads/Search quotas

At 1:2:2, the Ads and Search quotas likely exceed many users' histories, so some of the drop to 2.71 may come from more masked slots rather than reduced Homefeed context alone. Reporting the average number of filled slots per ratio would separate the two.

W2: the new data support the readout design. However, the main baselines still differ in backbone capacity, and only RED-Rec receives Stage II multi-positive supervision. A matched-backbone HLLM run, with and without Stage II-style supervision, would show how much of the 4.08 → 5.13 gain comes from the method rather than from capacity and supervision. Please also confirm whether Stage II steps are included in the "same number of examples."

AuthorsOpenReview ↗

Thank You for Quick Follow-Up

Dear Reviewer yDAV,

We greatly appreciate your follow-up and your continued positive assessment.

W1. How much of the test history is masked?

For the Table A5 cohort, we measured how many of the 128 history slots contained valid events before each prediction cutoff. Overall percentages are weighted by the indicated quota ratios:

Homefeed:Ads:Search quota ratioHomefeed filled (%)Ads filled (%)Search filled (%)Overall filled (approx. %)Overall masked (approx. %)Ads HR@10 (%)
4:1:199.999.299.899.770.234.92
2:1:199.999.199.499.5750.4255.13
1:1:199.992.497.996.733.275.04
1:2:299.987.496.893.666.342.71

The four allocations filled approximately 128, 128, 124, and 120 of 128 slots on average. The default 2:1:1 allocation retained nearly all 128 events and achieved the highest Ads HR@10 in the sweep. Moving from 1:1:1 to 1:2:2 reduced the Homefeed quota from roughly 43 slots to 26 and left about four more slots empty. Within the original cohort, fully filled and partially padded histories had similar observed Ads HR@10.

W1. Retrieval with shorter histories.

We would like to provide some further evidence on this point. We have done some further evaluation with Homefeed+Ads as our sparse-scenario example (because Search interactions are more frequent). In the original cohort, 98.6% of users exceeded the 32-event Search quota, compared with 92.7% for Ads. We evaluated the existing checkpoint on two additional groups sampled from the same period:

Evaluation cohortTest set sizeValid events in the indicated scenario (mean ± SD)Ads HR@10 (%)Ads HR@100 (%)
Original active-user cohort (Table 2)10,000 users—5.1320.42
Short Ads history10,000 examples14.23 ± 4.51 Ads events4.6718.44
Short Homefeed history10,000 examples49.14 ± 5.82 Homefeed events3.7817.91

The short Ads group filled less than half its quota on average and scored 0.46 percentage points below the original cohort on Ads HR@10. The gap was 1.35 points for short Homefeed histories. The smaller gap suggests that Ads retrieval holds up reasonably well with short Ads histories in this cohort, although the comparison may also reflect a greater tendency among less active users to explore unfamiliar content. If you have a particular combination of history lengths across scenarios in mind, we would be happy to sample a corresponding test set and evaluate it quickly with the existing checkpoint.

W2. Training exposure and the HLLM control.

Yes. The example count for the main three-epoch comparison includes both Stage I and Stage II of RED-Rec. We also compared HLLM and RED-Rec-MoE on the same TinyLlama-1.1B backbone (We kindly refer you to our previous response to Reviewer WjAy.). HLLM was trained for more epochs than the TinyLlama version of RED-Rec-MoE. The table summarizes the reported results on the same mixed-scenario Ads test set (Table 2 and §5.5). Table A1(a) also compares RED-Rec-1.1B-LLaMA with HLLM-1B on Amazon Books.

Training settingModelAds HR@10 (%)Ads HR@100 (%)Ads HR@1000 (%)
TinyLlama (in Table 2)HLLM4.0819.9243.27
TinyLlamaHyMiRec4.1215.3138.97
TinyLlamaRED-Rec-MoE4.3117.2846.37
Qwen (in Table 2)RED-Rec-MoE5.1320.4248.67

For some further discussion, we report the two stages together because both directly optimize the same retrieval task. Stage II retains the item-level loss and refines the queries against future-item clusters in the space learned in Stage I. Freezing the item tower gives the queries stable targets and avoids repeated item encoding. If with sufficient compute, both towers could continue training while the cluster targets are refreshed. We greatly appreciate your suggestion to test how much Stage II-style supervision adds to HLLM when its backbone and training exposure are held fixed. We are running this ablation (though it may finish after the discussion deadline) and will add it to the paper revision. Nevertheless, the existing component ablation supports our conclusion that scenario-aware mixing and the user-side MoE provide complementary gains in multi-scenario retrieval (Table 3b).

Thank you again for your thoughtful suggestions. If any questions remain, please let us know.

Reviewer_WjAyOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RED-Rec combines scenario-specific history quotas and temporal mixing with multiple interest queries in a hierarchical language-model recommender. Item-aware routing supplies training supervision, while serving uses independent retrieval from precomputed item embeddings and maximum-score fusion. The paper evaluates public benchmarks, industrial retrieval, and an online advertising experiment.

Critical strengths
  1. The paper addresses a concrete industrial problem: combining uneven feed, search, and advertising histories under a limited input budget. Candidate-independent user encoding and precomputed item representations make the retrieval design practically coherent.

  2. The evaluation includes useful controls beyond component removal. The mixer–query factorial comparison and matched-backbone, matched-interest-count comparisons with MIND and ComiRec help separate component gains from backbone capacity.

  3. The online experiment provides evidence of practical utility, with reported gains in advertiser value and advertising spend. Attention, routing, and retrieval-overlap diagnostics also provide some characterization of the learned representations.

Critical weaknesses
  1. The incremental technical contribution remains insufficiently distinguished from closely related work. The paper itself identifies HyMiRec’s learnable queries and future-window matching, but does not compare against it. Comparisons with older multi-interest readouts leave the closest architectural overlap unresolved.

  2. The evidence for the mixing mechanism is incomplete. Selection quotas, ordering, and positional encoding all affect the representation. Table 3 and the attention diagnostics support the overall design, but do not clearly establish the benefit of block interleaving when selected events and time information are identical.

  3. The serving-cost evidence is incomplete relative to the deployment emphasis.

Questions for authors
  1. What remains novel relative to HyMiRec under the same backbone and training budget, and can that difference be evaluated directly?

  2. Does blockwise mixing improve over chronological ordering on exactly the same selected events and positional features?

  3. What are the end-to-end latency, throughput, and incremental serving costs?

5 comments

Dear Reviewer WjAy,

We sincerely thank you for your detailed review and constructive suggestions. We respond to the three concerns raised in your review below:

Weakness 1 / Question 1: The technical contribution is insufficiently distinguished from HyMiRec. & What remains novel relative to HyMiRec under the same backbone and training budget, and can that difference be evaluated directly?

We appreciate this important point. The proposed RED-Rec combines scenario-aware history mixing with a user-side query-based MoE. The mixer allocates history slots by scenario and optimizes block interleaving. The MoE allocates interest experts to scenario-specific banks and uses hybrid item-content/user–item-similarity routing to distribute item-level supervision within each bank. These experts operate at the User LLM readout; the requested bank produces candidate-independent user vectors for parallel retrieval. We have revised the description to make the shared elements and technical differences clearer.

Some features such as learnable queries and future-window Hungarian matching are shared with HyMiRec. The supervision differs: HyMiRec uses cluster centroids for Hungarian assignment, then applies contrastive supervision to the positive items in each assigned cluster. RED-Rec combines hybrid-routed item-level NCE with an additional cluster-centroid NCE term in Stage II. The revised abstract, Introduction, §4.2, and §5.5 make these contributions explicit. We added a direct comparison with TinyLlama-1.1B, 20,000 training steps per method and setting, a 128-event history limit, and a ten-interaction target window. Both methods use the same evaluation users, candidates, and positives. Backbone and step count are matched; total training query count, history construction, and objectives follow each method. For Homefeed + Search + Ads → Ads, HyMiRec orders events chronologically, while RED-Rec applies scenario quotas and block interleaving:

ModelHR / NDCG @10HR / NDCG @100HR / NDCG @1000MRR
HyMiRec (TinyLlama-1.1B)4.12 / 1.3115.31 / 3.2738.97 / 4.511.92
RED-Rec-MoE (TinyLlama-1.1B)4.31 / 1.3417.28 / 3.3546.37 / 4.882.08

HR@100 improves by 1.97 percentage points and HR@1000 by 7.40 points; all other reported metrics also improve. The same backbone and 20,000-step protocol yields gains on every reported metric in both single-scenario settings:

ScenarioModelHR / NDCG @10HR / NDCG @100HR / NDCG @1000MRR
HomefeedHyMiRec2.13 / 1.2112.17 / 1.9829.42 / 3.661.21
HomefeedRED-Rec-MoE2.28 / 1.2713.42 / 2.1133.21 / 3.971.29
AdsHyMiRec4.01 / 1.2415.13 / 3.0138.91 / 4.511.94
AdsRED-Rec-MoE4.33 / 1.4516.72 / 3.3741.28 / 5.032.13

These complete-method results appear in §5.5 and Table 5, with the protocol in Appendix B. The Qwen2.5-1.5B controls separately test the proposed components:

  • Expert allocation: with 15 total queries, scenario-specific banks achieve 5.13 Ads HR@10, compared with 4.91 for a shared bank (Appendix E.1).
  • Hybrid supervision: with max fusion and Stage II group matching retained, hybrid routing achieves 5.13, versus 5.02 for similarity-only routing, 4.86 for content-only routing, and 4.94 for item-level hard top-1 matching (Table A6(a)).
  • Interaction with mixing: adding MoE to the mixer raises HR@10 from 4.72 to 5.13 and HR@100 from 18.33 to 20.42. Adding the mixer to MoE raises these metrics from 4.51 to 5.13 and 17.78 to 20.42, respectively (Table 3b).

Together, the direct HyMiRec comparison and component controls support RED-Rec's central contribution: coupling scenario-aware history construction with user-side expert allocation and hybrid supervision to improve multi-scenario retrieval.

Weakness 2 / Question 2: The evidence for the mixing mechanism does not isolate block interleaving. & Does blockwise mixing improve over chronological ordering on exactly the same selected events and positional features?

Thank you for emphasizing the importance of this control. Briefly, yes, under the controlled block-level comparison described below. All Table 3b variants share selected events, quotas, scenario tokens, cutoff-relative time gaps, and the positional-encoding scheme; sequence position IDs follow the resulting order. The 1D-position baseline cycles through Homefeed → Search → Ads, taking each scenario's events from oldest to newest and skipping exhausted histories. To isolate the repetition penalty, we compare chronological block ordering (4.40 Ads HR@10) with optimized block interleaving (4.72), without MoE. These settings preserve the same within-scenario chronology and use identical blocks of up to four events, differing only in β = 0 versus β = 0.2 in Eq. (2). The 0.32-point gain is the controlled interleaving result. Eventwise chronological ordering gives 2.10 and also differs in block construction.

Weakness 3 / Question 3: The serving-cost evidence is incomplete. & What are the end-to-end latency, throughput, and incremental serving costs?

To make the deployment evidence easier to assess, we would like to firstly clarify the execution paths and the scope of each measurement below. The deployed system separates representation refresh from request-time retrieval. Every ten valid clicks trigger a user-vector refresh using cached item embeddings. Retrieval requests reuse the cached user vectors, search the expert item indexes, and merge candidates by maximum similarity. Thus, User LLM encoding is amortized across retrieval requests, and item encoding runs asynchronously. The revised §5.6 and Appendix G.2 quantify these execution paths as follows.

  • Production user-vector refresh throughput and GPU allocation. Peak monitoring records 4,000–6,000 user-vector refreshes/s across 20 L20 GPUs, or 200–300 refreshes/s per GPU. At refresh rate q, the allocated GPU time per million refreshes is 20 × 10⁶ / (3,600q), giving 0.93–1.39 L20 GPU-hours. This quantifies the user-encoding allocation at observed peak throughput. Each resulting cached representation can serve multiple retrieval requests.
  • Item maintenance resources. Existing items are re-encoded every seven days using 64 H800 GPUs, and new items are indexed daily using five L20 GPUs. These background allocations keep item content encoding outside request-time retrieval and are reported separately from user refresh.
  • Measured scoring latency. On one L20, scoring one million candidates with five experts, max fusion, and top-K selection achieves 15.2 ms p95. Under the same scoring benchmark, soft fusion takes 16.1 ms, giving a 0.9 ms latency reduction for max fusion (Table A6(a)). User and item vectors are precomputed; this benchmark excludes user encoding, cache I/O, and ANN retrieval.

We would like to further note that in the whole system the item encoding, user-vector refresh, and retrieval follow separate execution paths without continuous serial execution. With item embeddings precomputed, the full online service pipeline—from the refresh trigger through user encoding and candidate retrieval to writing back the retrieved results—completes within approximately 0.6 s (but this is the end-to-end latency of the nearline recall pipeline, excluding downstream coarse and fine ranking).

We greatly appreciate your thoughtful review and constructive feedback. If any questions or concerns remain, please let us know.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer WjAy,

As the discussion period is ending soon, we kindly ask you to revisit our rebuttal and revisions. We have added a direct comparison with HyMiRec and clarified the controlled block-interleaving results, serving latency, throughput, and resource costs. We hope these updates address your main concerns and would appreciate your reconsideration of your assessment. Please let us know if any questions remain.

We sincerely thank you for your time and consideration.

Reviewer_2gvEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RED-Rec for multi-scenario retrieval using feed, search, and advertising histories. It uses a hierarchical framework with an Item LLM and a User LLM, selects historical events under scenario quotas, organizes them through blockwise interleaving, and extracts multiple interest vectors using scenario-specific queries. Training uses an item-aware hybrid gate to distribute supervision. The second stage freezes the item side and adds clustering of future-window items and Hungarian matching supervision. At inference, the interest vectors independently retrieve precomputed item representations, and the results are merged by maximum similarity. Evaluation includes public and industrial data, matched-backbone and matched-budget comparisons, ablations, and a one-week online A/B test.

Critical strengths
  1. The design coherently addresses practical constraints in multi-scenario retrieval. History quotas and scenario-specific queries allocate input capacity and interest readout capacity, respectively, while keeping user encoding independent of candidate items. This supports precomputed item representations and multi-vector retrieval. Industrial deployment and the online test provide evidence of practical value.

  2. The experimental evidence supports gains beyond backbone size alone. Table 3b contains a 2×2 factorial comparison of the mixer and multi-interest queries. Table 4b compares readouts with the same backbone, number of interests, and training steps. Appendix E.1 fixes the total query budget, and Appendix E.4 compares fusion rules with a fixed checkpoint and candidate pool.

Critical weaknesses
  1. The benefit attributable specifically to blockwise interleaving remains unclear. Although component comparisons are provided, Table 3b does not sufficiently specify whether Naive, 1D-position, 1D-timestamp, and 2D use the same historical events, scenario quotas, and time encodings. The current results support the overall history construction design, but clearer controls are needed to determine the additional contribution of the interleaving optimization in Equation (2) over simpler ordering strategies.

  2. The candidate protocol for the main industrial results is incomplete. Appendix G.2 states that approximately one million candidates are drawn from items active during the evaluation period, but does not fully specify the sampling distribution, inclusion of positives, or candidate-pool sharing across models. The fixed-pool statement in Table 11a concerns the fusion comparison and does not establish the protocol used for the main industrial comparisons.

Questions for authors
  1. For the same user and cutoff, do the mixing strategies in Table 3b use the same event set, scenario quotas, scenario tokens, and positional/time encodings? Please define the two 1D strategies and specify which variables each comparison changes.

  2. Is the candidate pool for the main industrial results constructed per user or globally? What sampling distribution is used, and how are subsequent positive items included? Do all models use the same candidate pool and positive-inclusion rules?

2 comments

Dear Reviewer 2gvE,

We greatly appreciate your careful review and constructive suggestions. We address each point below:

Weakness 1 / Question 1: The additional benefit of blockwise interleaving is unclear. & Do the mixing strategies use the same events, quotas, and temporal features, and how is each strategy defined?

We have clarified the comparison along the following four points:

  • The selected histories and scenario capacity are held fixed. For the same user and prediction cutoff, all mixer rows in Table 3b use the same selected events, scenario tokens, and quotas (64 Homefeed, 32 Ads, and 32 Search events). Changing the ordering policy therefore does not add or remove historical events. Section 5.3 and the revised Table 3 caption now state these shared controls explicitly.
  • Event time gaps are preserved, while position IDs follow the ordering. Each event's time gap is the prediction cutoff minus its timestamp, so this feature is unchanged across strategies. All strategies use the same positional-encoding scheme, combining sequence-position and time-gap features. The absolute position ID changes according to the event's location in the resulting sequence. We have made this distinction explicit in §4.2, alongside the positional-encoding definition, and reiterated it in §5.3 and the Table 3 caption.
  • The ordering unit and rule differ across strategies. Appendix G.1, Table A7 summarizes the ordering policies. For completeness, we describe the procedures below, including the exact within-scenario order and cycling rule for 1D-position.
Table 3b strategyHow the history is ordered
Naive CombinationConcatenate Homefeed, Search, then Ads, skipping empty histories. Events within each scenario stay in chronological order.
By TimestampSort individual events across all three scenarios by timestamp.
1D on positionOrder each scenario's selected events from oldest to newest. Repeatedly take one event from Homefeed, then Search, then Ads; skip exhausted histories and continue until all selected events are placed.
1D on timestampSplit each history into chronological blocks of up to four events and sort the blocks by their last timestamps.
2D MixingUse the same blocks and minimize Eq. (2) with β = 0.2, which penalizes consecutive blocks from the same scenario.

The 1D-position row interleaves individual events by their within-scenario indices while preserving each scenario's chronological order. By Timestamp instead sorts all events globally. The two block-based strategies share the same chronological blocks of up to four events. To make these comparisons easier to read, the revised Table 3b row labels and Table 3 caption explicitly distinguish event-level and block-level ordering.

  • The two block-based rows isolate the additional benefit of optimized interleaving. Sorting by block-end time minimizes Eq. (2) at β = 0. With the same blocks and MoE disabled, changing the repetition weight to β = 0.2 raises Ads HR@10 from 4.40 to 4.72 (+0.32 percentage points) and NDCG@10 from 1.25 to 1.33. This comparison retains the selected events, temporal features, and block construction, and measures the added benefit of the repetition penalty. The larger comparison between eventwise timestamp sorting and 2D mixing, 2.10 versus 4.72, also changes block construction. Section 5.3 now explains this distinction directly, with the corresponding results in Table 3b. We hope this makes the evidence attributable to interleaving clearer.

Weakness 2 / Question 2: The candidate protocol for the main industrial results is incomplete. & Is the industrial candidate pool global or user-specific, how is it sampled, and are positives and candidates shared across models?

Every user and model is evaluated against one shared global pool of approximately one million distinct items. The pool combines uniformly sampled eligible active items with all eligible held-out positives. Eligibility is assessed at the dataset sampling date: items must have been published within the preceding three months, been available for at least 15 days, and had CTR above 0.01% during their first 15 days. CTR is an eligibility criterion; sampling among eligible items is uniform.

About 99% of distinct held-out positive item IDs qualify. Ineligible positives are removed from ground truth, and users without any remaining positives are excluded. The final evaluation uses the same 10,000 users, filtered positive sets, and candidate pool for every model. This pool is separate from training negatives. Appendix G.2 now specifies this protocol for the main industrial results, in addition to the fixed-pool fusion comparison.

We greatly appreciate your thoughtful review and constructive feedback. If any questions or concerns remain, please let us know.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer 2gvE,

We sincerely thank you for your positive assessment. As the discussion period is ending soon, we would greatly appreciate your review of our updates. Please let us know if you have any further questions or concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Rebuttal

Dear ACs and Reviewers,

We sincerely thank you for the time and care you have devoted to reviewing our work. Your constructive suggestions have helped us present the contribution more clearly and make the experimental protocols easier to assess. We welcome the recognition of RED-Rec's overall design and practical value (2gvE, WjAy), deployability through training–serving decoupling (yDAV), and technical contribution plus controlled evidence beyond backbone scaling (all three reviewers).

Our central contribution is the joint design of scenario-aware history mixing and a user-side query-based MoE. History quotas and optimized block interleaving preserve evidence across scenarios; scenario-specific expert banks and hybrid routing allocate interest capacity and supervision. Together, these mechanisms improve retrieval while preserving candidate-independent user encoding.

To address the reviewers' concerns, we added a direct HyMiRec comparison using the same TinyLlama-1.1B backbone and 20,000 steps per method, with gains across every reported metric in three industrial settings (§5.5). We clarify the controlled interleaving gain (§5.3), candidate sampling, quota handling, baseline supervision, and public-benchmark configurations (Appendices B, E, and G). Our responses also clarify the full online serving pipeline—from the refresh trigger through user encoding and candidate retrieval to writing back the retrieved results and the overall end-to-end time consuming. We hope these additions directly strengthen the evidence for the method's technical contribution and clarify its evaluation and deployment.

We kindly refer you to the revised paper for the updated results and discussions. Once again, we greatly appreciate your thoughtful review and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.