acceptodds
Under review as a conference paper at ICLR 2027

RSCB: REGIONAL SEMANTIC CONTEXTUAL BANDIT FOR OPTIMAL SELECTION OVER LLM-GENERATED DYNAMIC CANDIDATES

Abstract

Large Language Models (LLMs) increasingly generate multiple candidate solutions for complex tasks such as bug localization, code repair, and tool selection. Selecting among these candidates remains challenging because the available arm set changes across rounds and newly generated arms may have no interaction history. Identity-based or disjoint contextual bandits cannot transfer feedback to such arms, while globally shared semantic linear bandits may suffer negative transfer when heterogeneous context-arm relationships cannot be captured by a single reward parameter. We propose Regional Semantic Contextual Bandits (RSCB), which maintains independent linear reward models over semantic interaction regions and routes each context-arm pair to its corresponding regional model. RSCB selects arms using region-specific upper-confidence scores, transferring feedback among semantically related pairs within each region while restricting transfer across heterogeneous regions. For semantic regions, we prove a regret bound of under regional linear realizability and correct fixed routing. Consequently, for fixed , RSCB retains the regret rate independent of the number of distinct arms. We further establish an lower bound for arm-independent learning. Finally, under a semantic-continuity assumption, we bound the prediction uncertainty of unseen arms within a correctly routed region. We evaluate RSCB on SWE-bench and an Android/Java security bug dataset, using LLM-generated root-cause hypotheses as arms and objective rewards derived from ground-truth patches. On held-out bugs, RSCB reduces cumulative regret compared to SC-LinUCB and kernelized baselines, with statistically significant improvements on the security dataset. Additionally, RSCB also learns a multi-region structure () on the public BugsInPy benchmark, achieving the lowest regret among all methods.

Primary area:
reinforcement learning
Keywords:
Contextual bandits, LLM candidate selection, semantic representations, regional feedback sharing, regret analysis

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_SUGWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper formulates selecting among LLM-generated candidates as a contextual bandit problem. RSCB partitions context–candidate pairs into regions via k-means on semantic features and maintains a separate LinUCB in each region, with a regret bound under correct routing and an evaluation on bug localization.

Critical strengths
  • The reward is computed from ground-truth patches rather than an LLM judge.
Critical weaknesses
  • The motivation is not supported. The premise that LLMs increasingly generate multiple candidates is given without references or evidence, and the reasons for dismissing existing practice (inconsistent self-ranking, expensive validation) are not justified. Existing approaches for selecting among LLM candidates, such as best-of-N with verifiers or reward models, self-consistency, and rerankers, are neither discussed nor compared. The claimed applicability to tool selection and code repair is also not evaluated.
  • The bandit formulation is not realistic. The reward requires the ground-truth patch, which is unavailable when a candidate must be selected. Since rewards for all training candidates can be computed offline, a supervised reranker is the natural baseline but is missing.
  • The technical contribution is limited. The method is k-means clustering followed by one LinUCB per cluster, and the theory sums standard OFUL bounds under assumed correct routing, plus standard lower-bound and Lipschitz arguments.
  • The empirical evidence is weak. The only significant gain over SC-LinUCB is on a private dataset; on both public benchmarks, the improvement is not significant. There is no comparison with a random partition of the same size, and several baselines perform close to random.
Questions for authors

see above

Reviewer_gJm3OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper is interested in selecting good candidate solutions generated from LLMs, where each candidate is associated to the general context. The proposed approach, RSCB, uses a learned router that send each context-candidate pair to one of HH separate semantic contextual bandit (SCB). The paper first present theoretical results on upper-bounds on the regret, first for a single SCB, and then for the routed setting. They evaluate the method empirically on three benchmarks, one of them being private.

Critical strengths
  • Selecting among LLM-generated candidates is a practical and timely problem. Splitting it into region-specific bandits with a learned router is a reasonable design.
  • The semantic context bandit approaches perform well against the baselines across the benchmarks. This gives practical support for the general direction, even though it is unclear how much of the gain comes from routing itself (see Weaknesses).
Critical weaknesses
  • The major weakness is the limited novelty over SC-LinUCB (Müller, 2025), in both theory and method.
    • The main theoretical analysis covers a single SCB, which is the same setting as SC-LinUCB. It is unclear what is new in these results.
    • The extension to the routed setting (RSCB) is very simple because it directly assumes perfect routing.
    • Regarding the method itself, as the paper notes (l. 171–175), setting H=1H=1 recovers the single global model, i.e., SC-LinUCB. Algorithm 1 (Appendix A) is the same as in Müller (2025).
    • In the SWE-bench experiment,  H=1H=1 and the results match SC-LinUCB.
    • In general, only H∈{1,2,3}H\in \{1,2,3\} is tested, which stays very close to the single-bandit setting. As a result, the contribution specific to routing is not clearly shown.
  • The pipeline has several components that are not described clearly, and many are never ablated. For example, the paper does not show whether the PCA step is needed or how much it matters. This makes it hard to tell which design choices drive the performance.
Questions for authors
  • Which parts of the theoretical results are new? The theoretical section is too short for me to judge. A clear statement to describe the theoretical results and what they tell us would help.
  • How does performance change with HH ? Can you test values above H=3H=3, or at least compare results over different values ? Do more bandits (finer regions) improve results?
  • Why is PCA always used for dimensionality reduction? How much does the information lost through PCA affect results? An ablation with and without PCA, or with different numbers of components, would clarify this.
Reviewer_34H7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • This paper formalizes root-cause hypotheses as a contextual bandit.
  • Specifically, the authors implement a router (based on an element-wise product of the context and candidate embeddings), reduce dimensions, and assign context-candidate pair to one of H regions
  • The authors provide a bound for regret, and empirical experiments show that the semantic bandits outperform on a SWE-bench and proprietary security dataset
Critical strengths
  • The design is adaptive and enables online updates
  • The authors show that reward is uncorrelated with LLM judge, which confirms there is minimal bias from self-preference in the evaluation.
  • The validation approach is sound. Test data is not seen during training and region count is tuned and frozen over a separate validation split.
Critical weaknesses
  • The authors only show that performance gains compared to SC-LinUCB are only statistically significant over the proprietary dataset.
  • The regret theory does not cover selection of H in the routing mechanism. As a result, it does not clearly establish whether the key contribution (i.e. the clustering method) affects performance. There is also no ablation of the routing controls.
Questions for authors
  • larification - Do the baselines all use consistent features, preprocessing, and hyper parameter tuning budgets? Are model parameters updated during test?
  • Clarification - What is the reward signal in deployment when there is no golden patch?
  • Suggestion - Is it possible to re-run with a stronger generator and choice of K? With same H, but random assignment?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.