acceptodds
Under review as a conference paper at ICLR 2027

Your Best Retriever May Not Be Best for Your Reranker: Adaptive Coordination for LLM Recommendation Cascades

Abstract

Large-scale recommender systems typically adopt a two-stage retrieve-then-rerank cascade, in which a retriever narrows a massive item pool to a small candidate set, and a reranker produces the final ranking. This creates an objective misalignment between recall-oriented retrieval and precision-oriented reranking: the retriever aims to ensure that relevant items are included, whereas the reranker must accurately distinguish them from competing candidates and place them at the top of the final ranking. Retrieval should therefore provide not only relevant items, but candidate sets that the downstream reranker can reliably resolve into high-quality final rankings. Prior work often addresses this misalignment through joint optimization, but discrete candidate selection limits end-to-end gradient flow, while the retrieval and reranking objectives may induce competing optimization directions. We propose R4Rec (Retrieve–Rerank–Reallocate–Refine), a cascaded framework that uses two trainable feedback mechanisms to coordinate pre-trained retrievers and an LLM-based reranker, aligning candidate generation with final reranking without updating the underlying retriever or reranker components. Across-Sample Reallocation learns context-dependent retriever quotas, assigning larger budgets to retrievers whose candidates better support the downstream reranker. It derives each retriever’s contribution from closed-form, reranker-aligned Shapley values and uses this feedback to train a contextual bandit. Within-Sample Refinement instead operates at the request level: a separate contextual bandit decides whether to stop or refine the current candidate set and, when refinement is needed, generates a diagnostic query to retrieve additional candidates before reranking. Experiments on Amazon and MovieLens show that R4Rec improves accuracy at practical inference cost, with a statistically significant 6.43% relative CTR lift over the production baseline in an online A/B test.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM Recommendation, Recommendation Cascades, Agentic Recommendation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VWWpOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes R4Rec, a framework for coordinating multiple frozen retrievers with a frozen LLM reranker. It addresses the observation that higher retrieval recall does not necessarily translate into better final rankings, since candidate composition also affects reranking success. The framework introduces two contextual-bandit mechanisms: across-sample reallocation adjusts retriever quotas using closed-form rank-weighted Shapley credits gated by downstream ranking success; within-sample refinement selectively generates diagnostic queries, retrieves additional items, and partially replaces the candidate set before reranking. The authors evaluate R4Rec on five Amazon domains and MovieLens using multiple LLM reranker backbones. Validation includes comparisons with recommendation and cascade baselines, component ablations, recall-versus-reranking analyses, candidate-order sensitivity experiments, and latency measurements. An approximately 12-week online A/B test reports a statistically significant 6.43% relative CTR improvement over the strongest fixed-allocation baseline.

Critical strengths

A meaningful focus on downstream candidate quality. The paper highlights that retrieval recall alone does not determine final recommendation quality. Its matched-gold analysis on MovieLens compares candidate sets that all contain the target and varies candidate input order

Critical weaknesses
  1. Limited methodological novelty and a mismatch between the attribution objective and the claimed mechanism. The rank-weighted Shapley formulation is a specialization of the classical max/airport game, whose closed-form solution is already known. More importantly, the coalition value depends only on the target item’s retrieval ranks. Downstream reranking enters through a single success indicator shared by all retrievers, without evaluating coalition-specific reranking outcomes. Consequently, the attribution cannot directly capture how non-target candidates help or interfere with ranking—the candidate-composition effects emphasized by the paper.
  2. The experiments do not isolate the contribution of Shapley attribution or establish sufficiently competitive matched baselines. Table 4 compares different Shapley rewards but omits simple alternatives such as success-gated equal sharing and normalized rank-discount rewards under the same controller. It also lacks Recall+Gate, leaving the effects of rank weighting and success gating incompletely separated. Some cascade baselines are substantially weaker than conventional recommenders: on MovieLens, Static-Global with TWiCERec achieves HR@10 of 6.98, versus 16.78 for GRU4Rec. This requires further explanation and validation of baseline adaptation and tuning. Overall, the results support the usefulness of adaptive coordination, but do not yet establish that the proposed Shapley design is necessary for the gains.
  3. The two core modules are primarily connected through the initial candidate set in a sequential pipeline, without explicit bidirectional feedback or a shared optimization objective. The quota controller’s reward depends on pre-refinement reranking success, while the refinement process does not use Shapley credit to adjust retrieval allocation. Consequently, the framework appears to combine two independently replaceable strategies. The existing ablations do not sufficiently establish synergy between the modules, weakening the novelty claim of a unified coordination framework.
Questions for authors

Why is adaptive retriever-quota allocation preferable to a conventional lightweight coarse-ranking stage between retrieval and the LLM reranker? A natural baseline would retrieve a larger candidate pool using fixed quotas, apply a user-conditioned coarse ranker to select the same 150 candidates, and pass them to the same LLM reranker. Under comparable retrieval and end-to-end inference budgets, does the proposed quota allocation provide additional benefits in candidate recall, final ranking quality, or latency? This comparison would help establish whether the proposed allocation mechanism offers advantages beyond standard candidate filtering.

2 comments
AuthorsOpenReview ↗

Response to Reviewer VWWp

We appreciate the reviewer's careful assessment and critical feedback. We have updated the manuscript in response to the reviews and marked the revisions in red. We have addressed the concerns raised and provide our detailed responses below.


W1. Candidate Composition: Captured Indirectly Through the Gate, Not Modeled Directly.

Takeaway. We agree that our credit does not directly attribute candidate-composition effects to individual coalitions. Instead, the gate incorporates downstream reranking success as an indirect feedback signal, and our experiments show improved rerankability.

  • Novelty and Scope. We acknowledge that the closed-form solution builds on the known max/airport game (Eq. 1 is the Shapley value of the classical airport (max) game (Littlechild & Owen, 1973). We now cite it. Our contribution is not a new solution to this game, but its integration into an adaptive retrieval–reranking cascade, combining rank-based credit with downstream success gating and online allocation. Prop. 2 establishes efficient attribution for this proxy, not for reranking interactions.

  • Computational Trade-off. Directly evaluating all nonempty coalitions of 12 retrievers would require up to 212−1=4,0952^{12}-1=4,095 LLM reranker calls per request, rather than a single call. Even sampling-based approximations would substantially increase reranking cost, so we use gold rank as a tractable proxy.

  • Indirect Path. The gate assigns credit only when the reranker succeeds on the full candidate set. Across requests, retrievers whose candidates co-occur with reranking success continue receiving credit, while those associated with failures do not. This is a correlational signal, not coalition-specific attribution.

  • Empirical Evidence:

    • On the matched-gold subset, R4Rec achieves higher HC@10 than Static-Global across candidate-order permutations (+0.049 [0.028, 0.071]; [x6g8 Q2]).

    • Our allocation achieves the highest conditional hit rate (b=HR@10/Recallb = \mathrm{HR@10}/\mathrm{Recall}) in all six domains ([VWWp Q1]).

    • These results suggest that our allocation produces candidate sets that are easier to rerank, although they do not isolate the causal contribution of individual non-target candidates.


W2-1. Addressed in Global Response [G1].

  • We added the requested success-gated equal sharing (Recall+Gate) and normalized rank-discount baselines (DGR and DGR+Gate), using the same controller and candidate budget with refinement disabled.
  • Rank+Gate outperforms both Recall+Gate and DGR+Gate across all six domains on NDCG@10 and NDCG@20. We also revised the claim regarding the effect of success gating in §4.3.

W2-2. MovieLens Gap to GRU4Rec: Understanding the Limits of Static Allocation.

Takeaway. The weak Static-Global cascade on MovieLens reflects a limitation of fixed retrieval allocation. With the same retrievers and reranker, adaptive allocation and selective refinement improve both retrieval coverage and reranking success, ultimately surpassing GRU4Rec.

  • Where the Cascade Loses (Table 3, a × b): Static-Global + TWiCERec retrieves the gold item within its 150 candidates for only 25.2% of users (a), and the reranker places it in the top-10 27.7% of the time when retrieved (b): 25.2% × 27.7% ≈ 6.98% HR@10.
  • Same Components, Adaptively Coordinated: GRU4Rec is one of the 12 retrievers in our retriever bank. With the same retrievers, initial candidate budget, reranker, and prompt, adaptive allocation and selective refinement raise a to 41.1% and b to 46.0%, yielding 18.90% HR@10, above GRU4Rec's 16.78%.
  • Across Backbones (Table 2): Under Static-Global, all six LLM cascades trail GRU4Rec on MovieLens; with R4Rec, three surpass it (18.2–18.9%). The weakest Static-Global backbones show the largest gains (+110% to +171%).
  • Tuning Transparency: Static-Global uses the best fixed allocation selected per dataset on the validation split by NDCG@10. All baselines were tuned on the same validation split.
  • Shapley Attribution: We separately evaluate the contribution of the proposed credit design in Global Response [G1]. Under the same gate and controller, Rank+Gate outperforms Recall+Gate and DGR+Gate across all six domains on both NDCG@10 and NDCG@20.

W3. Module Coupling: Two Complementary Mechanisms Linked Through the Candidate Set.

Takeaway. We agree that the modules are not jointly optimized, but allocation directly shapes refinement decisions. A new 2×2 ablation further shows positive interaction effects across all six domains.

  • Scope Clarified. Allocation learns from pre-refinement reranking success (Eq. 2), while refinement outcomes do not update allocation. We revised §3 to describe R4Rec as two complementary mechanisms rather than a jointly optimized framework.

  • How Allocation Shapes Refinement:

    • What to Replace: Refinement retains candidates with higher weighted-RRF scores s(d)s(d) (App. F.3), which depend on allocation weights (Eq. 7).

    • Whether to Refine: The refinement gate uses features derived from s(d), including score gaps and summary statistics (Table 13).

    • Shared Context: Both modules use the same request features z(x) (App. D).

  • New 2×2 Ablation: We compare static and adaptive allocation, each with and without refinement. Refinement without adaptive allocation uses Static-Global quotas, and its refinement gate is retrained under these quotas (same features and refinement budget), so each arm is evaluated with a gate fitted to its own allocation. The combined method achieves the highest HR@10 in all six domains, with positive additive interaction effects (+0.35 to +3.55 percentage points). This provides evidence that the two modules complement each other beyond their individual contributions.

HR@10 (%)FashionClothingGroceryHealthCDsMovieLens
Static-Global0.800.202.400.301.306.98
+ Allocation (No-Refine)0.880.342.380.511.5613.41
+ Refinement (No-Allocation)0.770.312.140.471.6110.12
+ Allocation + Refinement (R4Rec)1.201.275.671.102.2318.90

Same retriever bank, budget, and reranker in all rows.


Q1. (New Experiments) Adaptive Allocation vs. Coarse Ranking.

Takeaway. Coarse ranking achieves higher recall than our allocation in four domains, but adaptive allocation achieves better final ranking quality in five.

  • Implemented the Requested Baseline: Static-Global fixed quotas → 300 retrieved items → user-conditioned coarse ranking (embedding similarity based on user history) → top 150 candidates → same LLM reranker (refinement disabled).

    • Final Quality: Our allocation outperforms coarse ranking in 5 of 6 domains on HR@10, NDCG@10, and NDCG@20.
    • Conditional Hit Rate: We measure b=HR@10/Recall@150b = \mathrm{HR@10}/\mathrm{Recall@150}, the fraction of retrieved gold items placed in the top-10 by the reranker. Our allocation achieves the highest bb in all six domains, while coarse ranking falls below Static-Global in four. This suggests that our allocation produces candidate sets that are easier to rerank, rather than simply maximizing recall.
  • Revised: Added the comparison to the Appendix.

Table. Allocation vs. a Coarse-Ranking Stage.

MetricMethodFashionClothingGroceryHealthCDsMovieLens
Recall@150 (%)Static-Global3.302.1015.702.254.3025.19
Coarse3.732.4416.272.884.5629.31
Ours3.212.1115.312.454.7033.87
HR@10 (%)Static-Global0.800.202.400.301.306.98
Coarse0.860.252.410.371.268.39
Ours0.880.342.380.511.5613.41
b = HR@10 / Recall (%)Static-Global24.249.5215.2913.3330.2327.71
Coarse23.0610.2514.8112.8527.6328.63
Ours27.4116.1115.5520.8233.1939.59
NDCG@10 (%)Static-Global0.570.131.270.161.114.03
Coarse0.5460.1261.4270.1771.0454.511
Ours0.6500.2401.2800.3151.1947.860
NDCG@20 (%)Static-Global0.7120.1511.7510.2131.4454.789
Coarse0.7140.1492.0190.2441.4565.471
Ours0.8000.2701.8700.4401.5899.470

Refinement off; Bold: best per domain.

Reviewer_g73FOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the objective gap between retrieval and LLM reranking in recommendation cascades. The authors propose R4Rec, which coordinates frozen retrievers and a frozen reranker through two contextual controllers: across-sample reallocation learns retrieval quotas from rank-weighted Shapley credit, while within-sample refinement selectively rewrites retrieval queries and revises candidate sets. Experiments on public datasets, together with an online A/B test, demonstrate the potential of the proposed framework.

Critical strengths
  • This paper studies an important coordination problem in recommendation cascades without retraining the underlying components.
  • The experiments show improvements over baselines on public datasets and an online A/B test.
Critical weaknesses
  • The benefit of refinement over larger single-pass candidate pools at comparable total cost is not established.
  • The incremental benefits of rank-weighted credit and uncertainty-driven exploration are not isolated.
Questions for authors

I find the proposed framework interesting, but I have some questions regarding the experimental design and ablation studies:

  • Refinement Budget. While the Random control in Table 5 matches the average number of refinement calls, it does not test whether simply enlarging the initial candidate pool in a single pass yields similar gains. To justify the multi-step overhead, the authors should compare refinement at KD=150K_D=150 against a single-pass baseline with a larger pool (e.g., KD=300K_D=300), using the same allocation controller and reranker. Furthermore, please report the actual computational cost or wall-clock latency to evaluate performance at a comparable total cost.

  • Credit Assignment. Given that personalized retrieval allocation has already been studied in prior work [R1], the incremental value of the proposed credit design needs to be strictly isolated. Currently, Table 4 omits the Recall+Gate variant, leaving the specific benefit of rank weighting under cascade gating unclear. Please include this control, i.e., using the same controller and candidate budget, with refinement disabled, and compare it directly against Rank+Gate.

  • Exploration Mechanism. Table 4 does not isolate the impact of the LinUCB uncertainty bonus, and Table 11 only evaluates positive exploration coefficients. Since the softmax formulation already permits multi-retriever allocation, the necessity of this explicit bonus remains unclear. The authors should compare against an exploitation-only baseline (αacr=0\alpha_{\mathrm{acr}}=0) by retraining the allocation controller under the same reward, features, and candidate budget (with refinement disabled).

References:

  • [R1] Unleashing the Potential of Multi-Channel Fusion in Retrieval for Personalized Recommendations. WWW 2025.
3 comments
AuthorsOpenReview ↗

Response to Reviewer g73F

We appreciate the reviewer's thoughtful comments and suggestions. The revised manuscript reflects our rebuttal, with the changes marked in red. Our responses to each point are provided below.


W1/Q1. Addressed in Global Response [G3].

  • We added Big-Candidates, a larger single-pass baseline using the same allocator and reranker with a comparable average candidate budget (KD=210K_{D}=210).
  • R4Rec achieves 25–90% higher HR@10 across all six domains, with 16% higher mean latency (2.99 vs. 2.57 s). We also report measured end-to-end p50/p95 latencies in Global Response [G3].

W2/Q2. Addressed in Global Response [G1].

  • We added the requested Recall+Gate baseline under the same controller, candidate budget, and cascade gate, with refinement disabled.
  • Rank+Gate outperforms Recall+Gate across all six domains on both NDCG@10 and NDCG@20, demonstrating the additional benefit of our rank-weighted Shapley credit.
  • [R1] paper is already in our comparison: its CEM and PG methods are baselines in Table 1 (with the same LLM reranker), and R4Rec outperforms both in all six domains.

W2/Q3. (New Experiments) Exploration Bonus: Without It, Most Retrievers Stop Being Evaluated.

Takeaway. Softmax gives every retriever a share, but that is not enough: low-weight retrievers get cut from the final candidates and stop receiving feedback. The bonus keeps them in the learning loop.

  • Retrained the allocation controller with α∈\alpha \in {0, 0.1, 0.5, 1.0} (α\alpha = 0: no exploration bonus; same reward, features, and budget; refinement off).
  • Why softmax is not enough: a retriever receives feedback only if at least one of its items survives truncation to the final 150 candidates. Low-weight retrievers are cut, get no feedback, and stay low-weight; the bonus gives them another chance.
  • Starvation at α\alpha = 0: 10 of 12 retrievers enter the final candidates in < 5% of training requests.
  • Accuracy at α\alpha = 0: significantly lower in all six domains on both NDCG@10 and @20 (NDCG@10 −15% to −38%; paired t-test, p < 0.01).
  • Recovery: increasing α\alpha reduces starvation and monotonically improves NDCG@10 and NDCG@20 in all six domains.
  • Revised: both tables added to the Appendix; the α\alpha search range in Table 11 now includes 0.

Table. Accuracy by exploration strength α\alpha.

α\alpha (NDCG@10 / @20, %)FashionClothingGroceryHealthCDsMovieLens
00.541† / 0.671†0.204† / 0.245†0.911† / 1.345†0.211† / 0.288†0.744† / 1.099†5.671† / 7.290†
0.10.612† / 0.735†0.224 / 0.2611.111† / 1.609†0.243† / 0.377†1.017† / 1.388†7.001† / 8.588†
0.50.639 / 0.7810.231 / 0.2661.240 / 1.8310.291† / 0.411†1.033† / 1.5117.633 / 9.241
1.0 (Ours)0.650 / 0.8000.240 / 0.2701.280 / 1.8700.315 / 0.4401.194 / 1.5897.860 / 9.470

Refinement off. †: significantly lower than α\alpha = 1.0 (paired t-test, p < 0.01).

Table. Retriever participation during training.

α\alpha00.10.51.0
Lowest retriever participation3.2%4.5%7.2%13.6%
Retrievers below 5% (of 12)10500

Participation of retriever k = share of training requests in which at least one of k's items survives in the final 150 candidates (i.e., k receives a feedback update).

Thanks for the detailed rebuttal. The authors' responses have addressed most of my concerns, and I have raised my score accordingly.

Thank you so much for your kind follow-up and for raising your score! We truly appreciate your thoughtful engagement and constructive feedback.

Reviewer_x6g8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

R4Rec coordinates a bank of frozen retrievers and an LLM reranker. One controller learns context-dependent retrieval quotas using cascade-gated, rank-weighted Shapley credit. Another decides whether to generate a diagnostic query, replace some candidates, and rerank. Evaluation includes five Amazon domains and MovieLens, six reranker backbones, component ablations, candidate-order tests, and an online experiment reporting a 6.43% relative CTR lift over Static-Global.

Critical strengths

The accuracy gains extend across rerankers and into production. Table 2 reports improvements in most dataset–backbone combinations while retaining the same retriever bank, reranker, and prompt within each pair (p. 7). For TWiCERec on MovieLens, HR@10 rises from 6.98 to 18.90. The reported online CTR increase from 1.7853% to 1.9001% adds practical relevance. Appendix G accounts for repeated impressions and multiple comparisons, although the adjusted p-value of 0.0493 and odds-ratio interval [1.0002, 1.1353] warrant caution about the benefit’s size (Table 16, p. 26).

The refinement ablation includes a meaningful budget control. The learned gate has higher reported NDCG@10 than both No-Refine and Random in all six datasets, with Random using the same average refinement budget (Table 5, p. 8). This supports useful request selection rather than an explanation based only on additional calls; it does not imply improvement on every metric.

Critical weaknesses

The methodological advance is narrower than the adaptive-versus-static framing suggests. According to Appendix C.2, PG already learns user-specific channel quotas, and RRCM learns when and what to retrieve using final-ranking rewards (p. 18). R4Rec’s specific additions are cascade-gated rank-game attribution and selective query-based partial replacement after observing a reranking outcome. LinUCB and RRF are reused components. The new coupling could be valuable, but Table 4 does not distinguish the benefit of Shapley credit from simpler cascade-gated rank supervision. The proof establishes efficient attribution for the chosen proxy, not contributions to reranking interactions; Appendix I acknowledges this limitation (p. 30). This limits the methodological significance I can assign, without negating the integration’s empirical gains. My novelty comparison is restricted to the supplied descriptions of prior work.

Small controller overhead is supported; small overall inference overhead is not established. A candidate size of 150 does not match cumulative work when R4Rec can execute four additional retrieval/reranking rounds. Table 6 supplies component latencies but leaves end-to-end p50/p95 blank and does not compare the learned policy with Static-Global (p. 9). Table 5 and Figure 4 establish selective call usage, not the full system’s latency advantage. This is a reporting gap, not evidence that R4Rec is necessarily inefficient, but it weakens a central part of the practical contribution.

Questions for authors

Targeted attribution evidence: With refinement disabled, does Shapley credit outperform credit proportional to discounted gold-item rank, normalized to sum to one over target-surfacing retrievers, using the same cascade gate and allocator? This would clarify the distinctive attribution mechanism’s value.

Rerankability clarification and reanalysis: Were Table 14’s omissions censored jointly across conditions or separately? Please provide censoring rates and HC@10 on the full matched-gold subset, counting omissions as misses, with paired uncertainty estimates. This would show whether the result survives output-dependent exclusion.

Cost clarification: Does the stated equal retrieval budget refer to candidate size or cumulative work? What are measured end-to-end p50/p95 latencies for the learned policy and Static-Global on identical hardware? These measurements would materially affect my assessment of practical significance.

1 comment
AuthorsOpenReview ↗

Response to Reviewer x6g8

Thank you for your constructive feedback. We have incorporated the changes described in our rebuttal into the revised manuscript and highlighted them in red. We address your comments point by point below.


W1/Q1. Addressed in Global Response [G1], [G2], and VWWp W1.

  • Under the same gate and allocator, Rank+Gate outperforms Discounted Gold-item Rank (DGR) and DGR+Gate across all six domains on both NDCG@10 and NDCG@20, with refinement disabled.

W2/Q3. Addressed in Global Response [G3].

  • We clarify that "equal budget" refers to candidate size per pass, and report new end-to-end latency measurements against Static-Global and other baselines on identical hardware (e.g., (p50 / p95) Static-Global: 2.12 / 2.77, No-Refine: 2.23 / 2.89, Always: 12.37 / 13.33, R4Rec: 2.44 / 11.34).
  • We also introduce a candidate-matched single-pass baseline, showing that selective refinement improves HR@10 across all six domains at a 16% increase in mean processing time.

Q2. (New Analysis) Censoring in Table 14: Results Hold When Omissions Count as Misses.

Takeaway. Omissions were censored jointly. Counting them as misses on the full matched-gold subset barely changes HC@10, and the conclusions of both panels hold.

  • Joint censoring: a request was excluded from all rows of a panel if the gold item was omitted in any of the 10 permutations under any condition, so all rows use the same requests.
  • Rates (previously unreported): per-condition omission rates are 13–15%; the joint rule censored 16% (Panel A) and 15% (Panel B) of requests. The joint rate is close to the per-condition maximum, so omissions fall on largely the same requests regardless of the condition.
  • Omission = miss on the full matched-gold subset, with paired bootstrap 95% CIs (resampling requests):
    • HC@10 drops by at most 0.017; the ordering is unchanged.
    • R4Rec vs. Static-Global: +0.049 [0.028, 0.071].
    • After vs. before refinement: +0.046 [0.002, 0.098]. The interval excludes zero but is wide (n = 168), so we claim the direction of the gain rather than its size.
  • Revised: Table 14 (Appendix) now reports omission and censoring rates, omission-as-miss HC@10, and paired CIs.

Table. HC@10 with and without censoring.

PanelConditionOmission rateHC@10 (censored)HC@10 (omission = miss)Δ vs. ref. [95% CI]
AStatic-Global (ref.)0.150.5500.533—
AUniform0.140.5620.551+0.018 [−0.001, 0.039]
AShapley+Gate (R4Rec)0.130.5940.582+0.049 [0.028, 0.071]
BBefore refinement (ref.)0.150.3830.377—
BAfter refinement0.140.4300.423+0.046 [0.002, 0.098]

Omission rate: share of requests with the gold item omitted in at least one permutation under that condition. Joint censoring rate: 16% (A), 15% (B). Δ is computed on omission-as-miss HC@10.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Global Response (2/2)

[G3] (New Experiments) Candidate-Matched Comparison: Selective Refinement Outperforms a Larger Single Pass.


Takeaway. At a comparable average reranker candidate budget, selective refinement outperforms a larger single pass across all six domains, with 16% higher mean processing time.


  • Budget Clarified. The paper's “equal budget” meant candidate size (150), not cumulative work. Measured as the average number of candidates the LLM reranker processes per request, R4Rec uses ≈210 (0.371 refinement rounds on average).
  • Added to Table 6: Static-Global, No-Refine, Always (4 rounds), and Big-Candidates, a single pass with 210 candidates (same allocator and reranker), all on identical hardware.
  • Matched Work: R4Rec beats Big-Candidates in all 6 domains (HR@10 +25–90%) at +16% mean time (2.99 vs. 2.57 s).
  • More Candidates ≠ Better: relative to No-Refine, a larger pool helps unevenly (0 to +97%, −21% on MovieLens), while refinement helps every domain (+36% to +274%).
  • vs. Static-Global: mean +37% (2.19 → 2.99 s), p50 2.12 → 2.44 s; the p95 of 11.34 s comes from the few requests that use all four rounds.
  • Serving: Recommendations are recomputed asynchronously upon user-history updates and cached. Clients read from the cache, so the additional processing time does not directly increase user-facing response latency, although it affects computation cost and potentially recommendation freshness.
  • Revised: Table 6 and §4.5.

Revised Table 6. Latency breakdown, p50 / p95 (s).

ComponentStatic-GlobalNo-RefineAlways (4 rounds)(New) Big-CandidatesR4Rec (Ours)
Controller - weight allocation0 / 00.03 / 0.070.03 / 0.070.03 / 0.070.03 / 0.07
Controller - decision state (per round)0 / 00 / 00.01 / 0.040 / 00.01 / 0.04
Retrieval - multi-retriever execute1.76 / 2.441.76 / 2.441.76 / 2.442.01 / 2.671.76 / 2.44
Retrieval - candidate merge0.05 / 0.080.05 / 0.080.05 / 0.080.07 / 0.090.05 / 0.08
Reranking - LLM reranker0.37 / 0.500.37 / 0.500.37 / 0.500.44 / 0.600.37 / 0.50
Refinement - query generation (per round)0 / 00 / 00.49 / 0.670 / 00.49 / 0.67
Refinement - additional rerank (per round)0 / 00 / 02.04 / 2.560 / 02.04 / 2.56
Refinement rounds per request00400.371 (avg.)
End-to-end2.12 / 2.772.23 / 2.8912.37 / 13.332.54 / 3.292.44 / 11.34

4×H200; p50/p95 are measured from per-request end-to-end wall-clock times.

Table. Accuracy and latency at comparable candidate budgets.

Static-GlobalNo-RefineAlways (4 rounds)Big-CandidatesR4Rec (Ours)
Avg. reranker candidates150150750210210
Mean latency (s)2.192.2012.112.572.99
HR@10 Fashion0.800.880.550.881.20
HR@10 Clothing0.200.341.080.671.27
HR@10 Grocery2.402.384.112.985.67
HR@10 Health0.300.510.680.881.10
HR@10 CDs1.301.561.361.782.23
HR@10 MovieLens6.9813.415.6210.5618.90

Latency and candidate counts averaged over all six domains. Bold: best HR@10 per domain.

AuthorsOpenReview ↗

Global Response (1/2)

We thank all reviewers for their constructive feedback. We have incorporated the changes described in our rebuttal into the revised manuscript and highlighted them in red. Below, we address the common concerns raised by multiple reviewers.

[G1] (New Experiments) Credit Ablation: Simpler Credits Do Not Reproduce the Gains. (x6g8 W1/Q1, g73F W2/Q2, VWWp W2)


Takeaway. The gain comes from combining rank-weighted Shapley credit with the cascade gate, not from simpler credits.


  • Added to Table 4 (refinement off; same allocator/candidate budget, and LLM reranker):
    • (1) Success-gated equal sharing (SES) is equivalent to Recall+Gate (Prop. 1). (VWWp W2, g73F Q2)
    • (2) Discounted Gold-Item Rank (DGR): credit J(r)=1/log⁡2(r+1)J(r) = 1/\log_2(r+1); credit ∝J(r)\propto J(r) of each retriever's gold rank, normalized to sum to 1 over gold-surfacing retrievers (x6g8 W1/Q1).
  • With the Same Gate, Shapley Wins. Rank+Gate beats DGR+Gate and Recall+Gate in all six domains on both NDCG@10 and NDCG@20.
  • Rank+Gate is Best in 10 of 12 Cells (exceptions: Health @10, by 0.001; Clothing @20).
  • Revised Manuscript: Table 4 (three new columns); the claim in §4.3 that the gate benefits all credit schemes has been narrowed accordingly (main paper).

Revised Table 4: Ablation of across-sample reallocation.

DomainMetricSingleUniformRecall(New) Recall+Gate (SES)(New) DGR(New) DGR+GateRankRank+Gate (Ours)
FashionNDCG@100.2260.5580.5700.5840.5980.6180.5400.650
NDCG@200.2650.6530.7170.7300.6680.7410.6610.800
ClothingNDCG@100.0510.1890.1570.1660.1420.1400.1540.240
NDCG@200.0730.3080.2360.2140.1800.1790.2800.270
GroceryNDCG@100.3170.6910.6930.5220.7710.8521.0571.280
NDCG@200.5341.0131.1681.0471.4121.4351.4241.870
HealthNDCG@100.0740.1420.2220.2930.2740.2930.3160.315
NDCG@200.1230.1420.3260.3950.3240.3670.4150.440
CDsNDCG@100.2250.3210.3450.3630.5210.7910.9311.194
NDCG@200.4190.6250.5280.5670.9720.9961.3631.589
MovieLensNDCG@103.7303.1792.9292.9854.4934.6186.1627.860
NDCG@204.9064.1164.0224.1275.7446.0957.7539.470

Refinement disabled. New methods: Recall+Gate (success-gated equal sharing; SES), DGR, and DGR+Gate. Bold indicates the best result for each domain and metric.

[G2] Novelty Clarified: Contribution Restated, Airport Game Credited. (x6g8 W1, g73F Q2, VWWp W1)


Takeaway. We agree the contribution is narrower than our framing suggested. The novelty lies in how frozen retrievers and a frozen LLM reranker are coordinated, not in the individual components.


  • Airport Game. Eq. 1 is the Shapley value of the classical airport (max) game (Littlechild & Owen, 1973). We now cite it.
  • Why the Credit Matters: [G1] now isolates it. Under the same gate, Shapley credit beats simpler cascade-gated rank credit (DGR+Gate) and success-gated equal sharing in all six domains.
  • vs. prior work (all already baselines in Table 1, with the same reranker):
    • PG / CEM learn global or user-specific channel quotas, but optimize candidate fusion before the ranking stage; neither assigns per-retriever credit from the reranker's outcome ([R1] Unleashing the Potential of Multi-Channel Fusion in Retrieval for Personalized Recommendations. WWW 2025; g73F Q2).
    • RRCM uses final-ranking rewards, but trains the LLM policy with GRPO; R4Rec keeps every component frozen and learns only lightweight bandit policies.
    • None of them revises the current candidate set after observing the realized reranking outcome.
AuthorsOpenReview ↗

A gentle reminder before the discussion closes

Dear Reviewers,

We know the discussion period ends on November 18 and that this is a busy time. We put a lot of work into this rebuttal. Beyond running the additional experiments you suggested, we provided detailed explanations and proofs for the points you raised, and revised the main text accordingly. If you have a few minutes before the deadline, we'd really appreciate it if you could take a look at our responses.

Thank you for your time!

AuthorsOpenReview ↗

Summary of Revisions and Response to Reviewers

Dear Area Chair and Reviewers,

Thank you for handling our submission. Below is a brief summary of the updates made during the rebuttal period. All revisions have been incorporated into the updated manuscript and are highlighted in red.


We conducted all six analyses requested by the reviewers, and the results consistently support our design:

  • Is Shapley credit assignment necessary? (All three reviewers) We added the requested baselines (Recall+Gate, DGR, and DGR+Gate). Our credit assignment outperforms all three across all six domains on both NDCG@10 and NDCG@20.
  • Is the budget comparison fair? (x6g8, g73F) Given a comparable number of reranker candidates, R4Rec outperforms a larger single pass by 25–90% in HR@10 across all six domains, with only a 16% increase in mean latency. We also now report the full end-to-end cost.
  • Is the exploration bonus needed? (g73F) Removing it leaves 10 out of 12 retrievers receiving feedback in less than 5% of training requests, causing NDCG@10 to drop by 15–38% across all six domains (p<0.01p < 0.01).
  • Why not use a coarse ranker? (VWWp) We implemented the reviewer's suggested baseline. While it retrieves more gold items in four domains, our allocation achieves stronger ranking performance in five out of six domains because the reranker converts our selected candidates into top-10 hits at the highest rate across all six.
  • Are the two modules merely independent add-ons? (VWWp) A 2×22\times 2 ablation demonstrates a positive interaction across all six domains, with the full system consistently achieving the best results.
  • Does the matched-gold analysis hold under censoring? (x6g8) Yes. Treating omitted items as misses does not alter any of our conclusions (Table 14).

Where the reviewers raised valid points, we revised the framing accordingly. We now explicitly credit the classical airport game (Littlechild & Owen, 1973) for the closed-form solution, clarify that our credit formulation serves as a tractable proxy, and present R4Rec as two complementary mechanisms rather than a single jointly optimized framework.

We would greatly appreciate it if you could take these updates into account in your final assessment.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.