acceptodds
Under review as a conference paper at ICLR 2027

ECB: Evidence-Conditioned Continuous Bandit for Recommendation Agents

Abstract

Recommendation agents must decide where to retrieve evidence, how much evidence to acquire, and when further exploration is no longer worthwhile. This creates two fundamental challenges. First, reranking an initial candidate list cannot recover items that were never retrieved, making candidate coverage dependent on the retrieval workflow itself. Second, retrieval and candidate scoring consume limited resources, so exploration must balance the discovery of new items against its additional cost. We propose Evidence-Conditioned Continuous Bandit (ECB), a budgeted controller for adaptive workflow control. ECB introduces a gate-before-scoring mechanism that first examines the base recommender's output and determines whether further evidence should be acquired. When exploration is triggered, an Evidence-aware Ranker integrates user history, contextual information, item attributes, and construction evidence to produce candidate-level retrieval signals. These signals condition a joint continuous policy that controls interest focus, evidence mixing, and retrieval-source allocation. The policy therefore coordinates what to retrieve, how broadly to explore, and when to stop, while the base recommender ranks the expanded candidate pool. ECB learns this policy offline from sampled workflow trajectories using a target-conditioned proxy reward that jointly accounts for recommendation outcomes and resource costs. A likelihood-ratio update enables policy learning through the discrete retrieval executor. Experiments on multiple recommendation benchmarks show that ECB improves recommendation quality and target coverage, demonstrating the value of evidence-conditioned exploration and budget-aware workflow control.

Primary area:
reinforcement learning
Keywords:
contextual bandits, policy optimization, recommender systems, offline policy learning, budgeted exploration, adaptive information retrieval

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_t8LjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to efficiently retrieve evidence for explorations of items in the recommendation tasks. Specifically, the paper considers an agentic recommendation setting where agents retrieves so called "evidence" about each items to input them as a prompt for some LLM-based recommendation models (e.g., OneRec).The paper proposes a gating mechanism to determine if further evidence retrieval is necessary for provided inquiry to reduce the cost of evidence retrieval. The experiments are conducted on several recommendation datasets, against variety of baselines for recommendation.

Critical strengths
  • The motivation of considering retrieval-cost aware agentic use for recommendation task makes sense, and sufficiently explained in the paper.

  • The experiment results is good, it clearly demonstrates the benefits of the proposed method combined on top of the base model, which is OneRec. The choice of recommendation models compared are also representative choice. (While there are also some limitation, see Weaknesses).

Critical weaknesses

I feel that there might be several LLM-generated hallucinations in the paper, which makes the paper not easy to follow. I will detail specific points below.

(1) Abstract and Introductions (and problem formulation) are somewhat related, but provides different first impression in the topic. When I look at Figure 1 and read the Introduction, it somehow focuses on tool use, and it provides me the first impression that the action space of the decision problem is "which tools and how many times we should use" for some recommendation task.

Then, in the problem formulation section specifies action to be "(joint set of) interest focus, evidence mixing, retrieval-source allocation", although the explanation about what these variables actually refer to was never explained in the paper. This already sounds a bit contradictory, but if I assume that retrieval-source allocation means selection over which tool to use, then I can barely see connection to the Intro.

However, it is quite confusing that later in Section 4 (i.e., in the proposed method section), the paper writes "A recommender can rank only items that retrieval makes available", and this sounds like the first stage model is performing candidate item retrieval to pass to late-stage decision model, and this time, the paper does not consider tool choose as the action space.

I feel that probably LLMs or authors are somehow mixing the two different problem, "candidate item retrieval" and "tool selection". If they understand that these are distinct topics, then, the presentation of the paper matters. In any case, the fundamental problem setting is really unclear from the manuscript.

(2) Some important details are not explained. For example, (2-1) how the gating function is trained, (2-2) how the utility function (or value function) is defined for "joint action" over "interest focus, evidence mixing, retrieval-source allocation" in G(⋅)G(\cdot) right after the equation, (2-3) what is "training target" in this setting

(3) Related work picks representative papers from each field, but the connection to the proposed method is not really explained. For instance, "Learning from limited feedback" cites many counterfactual learning paper from logged data, but counterfactual learning methods are not used in the proposed method, and why these papers are specifically discussed is questionable.

(4) Some other minor writing issues; no citation for OneRec in the experiment section, sudden introduction of Proposition 1 without any connection sentence, to name a few.


Other than the above issues in writing, following points are counted as weaknesses.

  • The paper only reports mean of the performance, and does not report variance-related quantity in the experiment. And if I did not overlook, it appears that the paper report the results with only a single random seed or single split of train/test data. This is not reliable performance reporting.

  • If the paper considers effective tool use as the main contribution, related work should be more detailed in this point, not just listing some representative papers for tool-use or agentic recommendation. The idea of each paper, and its connection to the proposed method should be explained. (And this is also important for evaluating the novelty of the paper.)

Questions for authors

Please write rebuttals with author(s)' own words, as LLM-generated sentence may be hallucinated and is hard to follow.

  • How are the action space defined?

  • How is the value function learned for the evidence retrieval? How is the reward defined?

  • How does the proposed method learn the gating function?

2 comments
AuthorsOpenReview ↗

Response to Reviewer t8Lj

We are very sorry that the original text blurred tool use and candidate expansion, which could make ECB look like a choice among recommendation tools. We have rewritten the setup using the actual sequence of operations.

  1. Response to W1/Q1 (problem and action space): what happens in one round.
    1. OneRec returns the current ranked list and its score distribution.
    2. The gate reads the observed history, request context, current list, retained pool, and remaining budget, then chooses STOP or UPGRADE before evidence scoring.
    3. On UPGRADE, the Evidence-aware Ranker scores the current candidates. The joint policy then produces (lambda, rho, omega), controlling interest focus, evidence mixing, and retrieval allocation across dense, lexical, and sequential sources.
    4. The executor retrieves and merges new candidates, and OneRec ranks the merged pool. Only after this process finishes does the evaluator read the held-out item.
  2. Response to W2/Q2/Q3 (training objectives and updates). The gate learns whether another upgrade is worthwhile. We compare the outcome of an executed upgrade with stopping at that point and fit the advantage with a Huber loss. The three action heads share the current state and form a conditional joint distribution. For their updates, we use the cumulative return after the current action, subtract a learned state-value baseline, and apply the likelihood ratio because retrieval and ranking contain discrete operations.
  3. Supplementary result: three-seed stability. Table 1 compares the original OneRec, initial-pool reranking, fixed retrieval, and full ECB. OneRec and fixed retrieval use deterministic cached outputs; the learned reranker and ECB use three training seeds. These new seed-level results will replace the corresponding entries in the paper's main results table.

Table 1: NDCG@5, mean ± sample SD over three training seeds

MethodBeautyYelpMoviesToys
OneRec, fixed cached output0.15470.01490.02410.0211
Rerank initial pool0.1582 ± 0.00080.0162 ± 0.00020.0266 ± 0.00040.0234 ± 0.0003
Fixed retrieval, deterministic0.16060.01750.03010.0248
ECB0.1754 ± 0.00100.0221 ± 0.00030.0360 ± 0.00050.0288 ± 0.0004
  1. Supplementary result: candidate-expansion analysis. We additionally measured target coverage, newly exposed targets reaching the final top five, and retained/final pool sizes under the initial pool, fixed retrieval, and full ECB. Table 2 will be added to the paper as a new analysis.

Table 2: Candidate coverage, final hits, and pool sizes

QuantityBeautyYelpMoviesToys
Initial candidate coverage20.41%2.90%5.07%3.86%
Fixed-retrieval coverage22.77%3.87%6.78%5.27%
ECB final coverage24.35%4.64%7.80%5.88%
ECB NDCG@5 when the target is initially covered0.80120.56380.53720.5746
Newly exposed targets reaching final top-5 / all requests1.64%0.69%1.03%0.78%
Mean initial retained-pool size72.058.964.567.7
Mean final candidate-pool size96.894.195.595.9

The retained pool can exceed the 20-item displayed shortlist. For these reference runs, final pool size = initial retained-pool size + 20 × mean upgrades.

  1. Response to W3/W4 (paper revisions). We have moved Proposition 1 directly after the shortlist-coverage motivation, cited OneRec in the experimental setup, and shortened the counterfactual-learning discussion. The experiments are described as controlled offline workflow replay. Table 1 will update the paper's main results, and Table 2 will be added as a new candidate-expansion analysis. The corresponding method changes are in Sections 2 and 5, Table 2, and Appendix A.

Thank you for the detailed response. The rebuttal comments are useful, however, provided the quality of the manuscript, I will maintain my current evaluation score.

Reviewer_BruGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces the Evidence-Conditioned Continuous Bandit (ECB) framework to optimize the retrieval workflow and candidate coverage in recommendation agents. The proposed architecture uses a gate-before-scoring mechanism to dynamically determine whether the expected benefit of acquiring new candidate evidence justifies the computational cost. When expansion is triggered, the system utilizes a joint continuous policy to allocate retrieval budgets across different sources and evidence mixtures. The model is trained offline via a likelihood-ratio update relying on a frozen, target-conditioned proxy reward.

Critical strengths
  • The gate-before-scoring mechanism effectively forces the agent to balance recommendation improvement against explicit computational overhead.

  • The authors present a solid experimental design with thorough ablations. The matched-round gating and budget sensitivity analyses successfully demonstrate that the system's performance gains stem from dynamic context-awareness rather than merely scaling up computational resources.

  • Coordinated Action Space: Factoring the continuous action space into interest focus, evidence mixing, and source allocation allows for a highly adaptive retrieval strategy compared to static heuristics.

Critical weaknesses
  • While the framework is functionally effective, the core methodology relies heavily on combining a standard likelihood-ratio update with a gating mechanism. This presents more as a clever systems-engineering optimization rather than a fundamental algorithmic breakthrough in reinforcement learning for recommendation systems.

  • The continuous policy dynamically shifts evidence mixtures and retrieval budgets, but it does so as a black box. There are no proper frameworks for explainability provided. For an agent making autonomous decisions about budget allocation and information retrieval, a transparent mechanism to audit why specific paths were chosen is necessary for adequate oversight.

  • The evaluation relies heavily on aggregate offline metrics and a single successful case study detailing the recovery of an initially missing item. The manuscript lacks a detailed qualitative study of the system's failure modes. Understanding when the gate triggers a premature stop, or when the continuous policy collapses into suboptimal retrieval allocations, is important for assessing the system's robustness.

  • The policy learns entirely offline using a frozen target-conditioned proxy reward model. The paper does not sufficiently analyze how sensitive the learned continuous policy is to the inherent biases or calibration errors of this proxy reward.

Questions for authors

1- Can you provide a qualitative error analysis detailing common failure modes? Specifically, in what contexts does the gate prematurely stop, and when does the continuous policy misallocate the retrieval budget?

2- How can practitioners interpret or audit the continuous policy's decisions? Are there plans to integrate a framework for explainability to ensure proper oversight of the agent's autonomous routing choices?

3- How sensitive is the final policy to the calibration of the frozen proxy reward model used during offline training?

1 comment
AuthorsOpenReview ↗

Response to Reviewer BruG

Thank you for recognizing the value of the budgeted workflow and the matched-round control.

  1. Response to W1 (contribution). The contribution is the recommendation-specific coordination of candidate acquisition, evidence use, and stopping. Every upgrade trace records the gate input, selected action, retrieved candidates, and rank change.
  2. Response to W2/W3/Q1/Q2 (failure analysis and audit). We first select failures with fixed rules over all test traces and then inspect individual cases. The table below gives explicit denominators; the last row follows directly from the coverage decomposition. | Failure measurement | Beauty | Yelp | Movies | Toys | |---|---:|---:|---:|---:| | Premature STOP later rescued by diagnostic continuation / all requests | 4.2% | 5.3% | 4.8% | 5.0% | | Upgrade rounds without an NDCG@5 gain / all upgrade rounds | 18.7% | 22.4% | 24.1% | 21.8% | | Newly exposed target outside final top-5 / all requests | 2.30% | 1.05% | 1.70% | 1.24% |

Here are three example traces:

  • Premature STOP: In one Beauty trace, the gate stopped with predicted advantage −0.002. Diagnostic continuation exposed the target and placed it third, showing that the gate can stop too early near part of the decision boundary.
  • Upgrade without improvement: one Movies trace needed a larger pool, but repeated retrieval still missed the target. In the action (0.63, 0.41; 0.46, 0.31, 0.23), the first two values control interest focus and evidence mixing, and the last three allocate dense, lexical, and sequential budget. Three upgrades expanded the pool from 27 to 87, while NDCG stayed at zero.
  • Exposure without a top-5 hit: One Toys trace exposed the target in round two but ranked it seventh. Acquisition succeeded, and the final ranking still missed the top five.
  1. Response to W4/Q3 (reward reliability and sensitivity). Both reward tables now appear as shared Table S2, because they also answer Reviewer 5Ry2's training-signal question. On 820 train-internal contexts excluded from reward fitting, the four-domain averages are 0.81 ROC-AUC and 0.68 Spearman. Each perturbation retrains and freezes the gate and action policy before evaluating the same requests. Small perturbations preserve the main comparison; larger systematic errors reduce quality and change more stopping decisions. Full calibration details are in Appendix D.
Reviewer_5Ry2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ECB (Evidence-Conditioned Continuous Bandit) is a controller that improves recommendations by adaptively expanding the candidate pool. It first decides whether further exploration is worthwhile; if so, an evidence-aware scorer informs what to retrieve next. The base recommender, OneRec, ranks the expanded pool. The central idea is that reranking alone cannot recover relevant items that were never retrieved. Main contributions:

  • A gate-before-scoring workflow that avoids acquiring additional evidence when its expected benefit does not justify its cost.
  • An evidence-conditioned continuous policy jointly controlling interest focus, evidence mixing, and allocation across semantic, lexical, and sequential retrieval.
  • Offline learning using a frozen, target-conditioned proxy reward minus scoring/retrieval costs, with a likelihood-ratio policy gradient through discrete retrieval operations. The paper also derives a finite-horizon stopping rule. Offline next-item experiments cover Beauty, Yelp, Movies, and Toys. Reported NDCG@5 improvements over OneRec are 13.4%, 48.3%, 49.4%, and 36.5%, respectively. Candidate-expansion controls show increased target coverage beyond reranking or fixed retrieval. The learned gate uses 41–59% fewer expansion rounds than always executing three rounds, with similar recommendation quality, and outperforms random gating at matched average rounds.
Critical strengths

The paper addresses a consequential limitation of reranking: relevant items outside the candidate pool cannot be recovered. ECB connects candidate-level evidence to adaptive retrieval while retaining the base recommender for final ranking. The expansion controls substantiate this motivation: ECB improves both target coverage and NDCG@5 over initial-pool reranking and fixed retrieval across all four datasets. This supports the central claim that learning how to acquire candidates provides value beyond improving their ordering.

The experiments meaningfully isolate the benefit of adaptive resource allocation. The learned gate outperforms random gating at matched average expansion counts and achieves similar quality to always continuing with 41–59% fewer rounds. ECB also outperforms fixed retrieval using larger per-round budgets. These comparisons strengthen the acceptance case because they suggest the gains arise from directing retrieval effort effectively, rather than simply spending more resources.

Critical weaknesses

The central learning mechanism is insufficiently specified and validated. The paper relies on a learned, target-conditioned reward model to train both retrieval decisions and stopping, but does not adequately describe its architecture, training targets, or predictive accuracy. Important implementation details—including the default evidence scorer, action distributions, and training configuration—are also missing. Because these components determine the controller’s behavior, the omissions materially limit reproducibility and confidence that the proposed learning mechanism explains the reported gains.

The evidence for cost-effective control does not establish actual system efficiency. Resource comparisons primarily report expansion counts, relative retrieval budgets, and scorer-only timing; they do not report end-to-end latency or total scoring and retrieval costs. Matching average rounds does not necessarily match computational expense, especially when candidate-pool sizes and retrieval sources vary. Since balancing recommendation quality against acquisition cost is a central contribution, the absence of a measured quality–cost comparison leaves that claim only partially supported.

Questions for authors

How are the frozen reward model and default evidence scorer instantiated and trained? Specifically, what supervision does the reward model receive, and what evidence shows that its scores predict actual recommendation improvements on held-out data?

What are the absolute retrieval budgets and scoring/retrieval cost penalties, and how do they relate to measured computational costs? Under comparable total cost or end-to-end latency, does ECB retain its advantage over fixed retrieval and always continuing?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 5Ry2

Thank you for identifying the missing training and cost details. We respond to W/Q as follows.

  1. Response to W1/Q1 (training and reward). Shared Table S1 specifies the reward model, evidence scorer, gate, action distributions, data sizes, optimizers, and freezing order. The training target is labeled 1 and sampled negatives 0. The Evidence-aware Ranker uses the same labels with point-wise binary cross-entropy and separate parameters, producing candidate logits for retrieval configuration. The gate learns a one-step advantage: U_base and U_full are the q_phi scores immediately before and after the same upgrade, and C_upgrade is that upgrade's realized cost. The action uses the multi-round return Rt=qphi(T)−Σk=tTCupgrade,kR_t=q_phi(T)−Σ_{k=t}^T C_{upgrade,k}, covering the terminal target score and all current and future upgrade costs. Each controller uses 20,480 rollout records. Shared Table S2 reports reward reliability and sensitivity on 820 additional train-internal contexts excluded from reward fitting, averaging 0.81 ROC-AUC and 0.68 Spearman correlation.
  2. Response to W2/Q2 (absolute budget and measured cost). Each upgrade retrieves up to 20 new candidates, with one reference quota of 8/6/6 across dense, lexical, and sequential sources and at most three upgrades. Training cost counts only retrieval calls and candidate scoring that actually occur; avoiding an unnecessary upgrade avoids both costs. Table 1 below reports realized counts and latency. NDCG follows the common protocol, and cost uses Beauty. “Scored candidates” counts Evidence-aware Ranker evaluations after an UPGRADE; OneRec proposal is already included in end-to-end time.

Table 1: Full-pipeline quality and measured cost

MethodNDCG@5, Beauty / Yelp / Movies / ToysMean upgradesRetrieval callsScored candidatesEnd-to-end p50 / p95 (s)
Always stop.1547 / .0149 / .0241 / .02110000.071 / 0.108
Fixed retrieval.1606 / .0175 / .0301 / .02481.003.0020.00.164 / 0.247
Random gate, matched upgrades.1655 / .0188 / .0315 / .02531.243.7224.80.193 / 0.301
Always continue.1750 / .0221 / .0367 / .02903.009.0060.00.376 / 0.583
ECB learned gate.1754 / .0221 / .0360 / .02881.243.7224.80.191 / 0.292

Relative to always continuing, ECB uses substantially fewer upgrades while retaining similar quality. The matched-round random gate answers the complementary question: learned gating gives higher NDCG@5 in every domain, and all four paired CIs exclude zero. 3. Response to W2/Q2 (threshold and quality–cost trade-off). Table 2 keeps the action policy and all other settings fixed and varies only the gate's predicted-advantage threshold.

Table 2: Gate threshold and quality–cost trade-off

Gate thresholdNDCG@5Mean upgradesEnd-to-end p50 (s)
−0.0100.17552.420.301
−0.0050.17541.830.246
0.000 (default)0.17541.240.191
+0.0050.17490.920.157
+0.0100.17380.610.119

Lowering the threshold yields only a marginal quality gain while increasing upgrades and latency. Higher thresholds reduce latency smoothly. A moderately higher threshold also exceeds fixed retrieval in quality at lower measured latency. Always-Continue provides the complementary comparison: ECB retains similar quality with substantially less computation. Complete per-component timing, retry counts, and four-domain latency are in Appendix E.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Shared implementation and evaluation details

We thank all three reviewers for their careful reviews. The original manuscript was not as clear as it should have been, and different readers may have formed different interpretations of the workflow. We have rewritten it to explain what happens in practice. In short, ECB studies how to acquire valuable additional candidates on top of the fixed OneRec recall process. The gate only chooses STOP or UPGRADE. Only after an UPGRADE do we use (lambda, rho, omega) to jointly control interest focus, evidence mixing, and retrieval-source allocation. All comparisons use the same requests, initial candidates, base recommender, and complete evaluation denominator. Learned variants use training seeds 17, 43, and 91. We report the sample standard deviation across these three training runs. For paired intervals, we first align predictions by request, average the three seed-specific predictions for each method, and then run 10,000 request-level paired bootstrap resamples. Thus, the SD describes training variation and the CI describes paired request variation.

Table S1. Implemented components and training protocol

ComponentImplementation used in the revised description
Base and retrievalOneRec-1.7B; fixed dense, lexical, and sequential retrievers; at most three upgrades; up to 20 new candidates per upgrade with reference quota 8/6/6 and up to 60 over three upgrades
Gate512→128→1 MLP with GELU; Huber regression on the executed continuation advantage A=(U_full−C_upgrade)−U_base; default decision threshold 0
Evidence-aware RankerPoint-wise 512→128→1 scorer over frozen item-text embeddings, history/request features, structured attributes, and construction evidence; one scalar per exposed candidate
Reward model q_phi1024→256→128→1 MLP with GELU; point-wise relevance objective; 3,276 training contexts for fitting and 820 disjoint train-internal contexts for evaluation and calibration
Joint actionlambda and rho use Beta(2,2) distributions on [0,1]; omega uses Dirichlet(2,2,2) over the three retrieval sources
Controller training4,096 train contexts × 5 replay epochs = 20,480 rollout records per seed; batch 256; AdamW for 400 updates
Learning ratesGate 1e−3; action and value heads 3e−4; reward/evidence scorers 3e−4; weight decay 1e−4; gradient clipping 1.0
Freezing and evaluationFit reward/evidence scorers, train controller and gate, select the operating point on validation, freeze all components, then run held-out evaluation

Both scorers are fitted for five epochs with batch 256 and early stopping on train-internal validation. The Evidence-aware Ranker is trained with point-wise binary cross-entropy: the training target is labeled 1 and sampled negatives are labeled 0. Its output logit is used only to score exposed candidates for retrieval configuration. The reward model uses the same labels in a separate point-wise relevance objective and outputs q_phi∈[0,1] to estimate trajectory utility during training; the two scorers have separate parameters and serve different roles.

Each upgrade retrieves up to 20 new candidates. One reference allocation is 8 dense, 6 lexical, and 6 sequential; the joint action can redistribute this total across the three sources. A full reference upgrade also scores 20 exposed candidates. Training cost counts only the retrieval calls and candidate scoring that actually occur, so avoiding an unnecessary upgrade avoids both costs.

Table S2. Reward reliability and sensitivity

The reliability table uses 820 train-internal contexts excluded from reward fitting. The four-domain averages are 0.81 ROC-AUC and 0.68 Spearman after rounding. ROC-AUC means ROC-AUC for target relevance, and Spearman correlation means Spearman correlation with ranking utility.

DomainROC-AUCSpearman correlation
Beauty0.820.70
Yelp0.790.66
Movies0.810.68
Toys0.810.67

For each perturbation, we retrain the gate and action policy, freeze the resulting controller, and evaluate the same requests. Gate changes are measured against the unperturbed controller with the same seed.

Reward perturbation during trainingBeauty NDCG@5Movies NDCG@5Gate decisions changed
None0.17540.0360—
Gaussian noise, σ = 0.05 × training-score SD0.17460.03573.8%
Gaussian noise, σ = 0.10 × training-score SD0.17320.03527.6%
Positive logit bias, +0.10 × training-score SD0.17410.03555.1%
Temperature calibration on train-internal validation0.17500.03591.9%

Small perturbations preserve the main comparison. Larger systematic errors reduce quality and change more stopping decisions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.