acceptodds
Under review as a conference paper at ICLR 2027

Pilot Priors: Safely Gating Warm Starts for Contextual Bandits

Abstract

Warm-starting contextual bandits has become an increasingly attractive method for reducing early-stage regret. Warm start data can come from historical logs, transfer pipelines, or be generated synthetically via simulators or large language models. However, the quality of such priors is often unknown: priors aligned with target tasks can reduce downstream regret, while misaligned priors can increase regret relative to cold starts. This paper studies the pre-deployment setting in which a single candidate prior is available, but only a short amount of real target-task data can be collected before source trust is fixed for the next deployment phase. We propose the Pilot Priors framework, which uses a short target-task pilot to certify a candidate warm prior relative to a cold-start baseline. The framework produces accept, reject, or inconclusive decisions from certified bounds on prior error, and satisfies a correctness guarantee for hard decisions about prior-error ordering. We further derive a sufficient condition for decisiveness through a margin-versus-width condition that links pilot informativeness to prior quality. Finally, for the inconclusive regime, we introduce a soft trust-calibration rule that interpolates between warm and cold starts rather than forcing a brittle all-or-nothing decision. The framework is evaluated in both a synthetic setting and two deployment scenarios based on real-world data. We show that the certificate becomes decisive as pilot information increases in synthetic experiments and detects harmful LLM-generated priors in one real-world setting. In another real-world setting, the certificate remains appropriately conservative while the soft calibration reduces harm from a reward-permuted null.

Primary area:
reinforcement learning
Keywords:
contextual bandits, reinforcement learning, decision making, uncertainty quantification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_VAF7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies pre-deployment assessment of a single warm prior for linear contextual bandits. A shared target-task pilot produces a confidence ellipsoid, which is converted into intervals for the warm and cold prior errors B0B_0. Interval separation yields accept/reject decisions with a high-probability guarantee on B0B_0 ordering, together with a margin-versus-width sufficient condition for decisiveness. An endpoint-clamped sigmoid rule interpolates sufficient statistics when certification is inconclusive.

The framework is evaluated in three settings: a synthetic linear bandit with known θ⋆\theta^\star, a COVID-19 conjoint study in which GPT-4o-generated CBLI priors are compared against real human choices, and KuaiRand, with priors fit on historical adaptive logs and SNIPS replay evaluation on the random-intervention stream. The authors report that the certificate becomes decisive as the pilot grows in the synthetic setting, rejects corrupted LLM priors in COVID, and stays inconclusive in KuaiRand. They also report that the soft rule preserves about 90% of the aligned COVID warm-start gain, removes the cost of the corrupted COVID prior, and reduces the KuaiRand reward-shuffled SNIPS loss by 69% relative to ungated warm start.

Critical strengths
  1. Deciding before deployment whether to trust an LLM-generated or log-derived warm start is a practical and well-motivated question.
  2. The framework is simple and cheap, with a clear high-probability guarantee for the hard decisions and complete proofs.
  3. The empirical protocol is careful (SNIPS replay on the random-intervention stream, a reward-shuffled null prior).
Critical weaknesses
  1. The given guarantee appears misaligned with the deployment objective, as stated by the authors. Thm. 4 guarantees the ordering of B0(m)\mathcal{B}_0^{(m)} versus B0cold\mathcal{B}_0^{\text{cold}}, not regret or policy value. Several results suggest this ordering often disagrees with downstream benefit in the experiments (e.g., p=0.20p=0.20 prior in Table 5 improves regret by 1.949, yet sufficiently informative pilots make SOFT discard it).

  2. The reported downstream gains are largely not driven by the certificate. When all decisions are inconclusive (KuaiRand), SOFT is indistinguishable from a fixed ω=0.30\omega = 0.30 blend, with differences of at most 0.0006 SNIPS.

  3. The theoretical part seems largely builds on prior works, and as the author stated, there is no regret guarantee for the gated or soft polices.

  4. The guarantee is misaligned with the deployment objective. Thm. 4 guarantees the ordering of B0(m)B_0^{(m)} versus B0coldB_0^{\text{cold}} instead of regret. Several results show that this ordering can disagree with downstream benefit: e.g., the p=0.20p = 0.20 prior in Table 5 improves regret by 1.949, yet once the pilot is informative enough (Tpilot≥5,000T_{\text{pilot}} \ge 5{,}000), SOFT discards it.

  5. The reported downstream gains are largely not driven by the certificate. When all decisions are inconclusive (KuaiRand), SOFT is indistinguishable from a fixed ω=0.30\omega = 0.30 blend, with differences of at most 0.0006 SNIPS.

  6. The downstream gains are largely not driven by the certificate. When all decisions are inconclusive (KuaiRand), SOFT is indistinguishable from a fixed ω=0.30\omega = 0.30 blend, with differences of at most 0.0006 SNIPS, so the 69% reduction in the abstract comes from blending. The certificate does matter when it reaches REJECT, but this is the main case where it changes the outcome.

  7. The theoretical contribution is modest. Theorem 1 applies the self-normalized bound of Abbasi-Yadkori et al. (2011), and the remaining results convert it into intervals and a sufficient separation condition. There is no regret guarantee for the gated policy.

Questions for authors

See Weakness.

1 comment
AuthorsOpenReview ↗

Reviewer VAF7 Response

We thank the reviewer for recognizing the practical motivation of our work, the correctness of the hard certificate, and the care of our empirical evaluation. We have grouped your concerns and addressed them below.

W1 and W4

The given guarantee appears misaligned with the deployment objective

The example in Table 5 shows a limitation we acknowledge in Section 2.3. A prior can reduce deployment regret even when its prior error B0(m)\mathcal{B}_0^{(m)} is larger than B0cold\mathcal{B}_0^{\mathrm{cold}}. Our certificate does not guarantee that ACCEPT or REJECT selects the policy with lower regret.

To examine this relationship, we derived a new regret bound UBT\mathrm{UB}_T for the LinUCB algorithm used in our experiments. Under the stated linear-bandit assumptions, the bound is non-decreasing in the prior error B0\mathcal{B}_0 and non-increasing in the minimum eigenvalue λmin⁡(A0)\lambda_{\min}(A_0) of the prior precision matrix. For prior initializations satisfying A0(m)⪰A0cold=IA_0^{(m)} \succeq A_0^{\mathrm{cold}} = I, a correct ACCEPT decision implies

B0(m)≤B0cold⟹UBT(m)≤UBTcold.\mathcal{B}_0^{(m)} \le \mathcal{B}_0^{\mathrm{cold}} \quad\Longrightarrow\quad \mathrm{UB}_T^{(m)} \le \mathrm{UB}_T^{\mathrm{cold}}.

This result establishes that WARM has a no-worse regret upper bound when the certificate ACCEPTs. It does not guarantee lower realized regret.

We also constructed exact two-dimensional counterexamples. In one example, WARM has five times the prior error of COLD (B0(m)=5\mathcal{B}_0^{(m)} = 5 versus B0cold=1\mathcal{B}_0^{\mathrm{cold}} = 1) but achieves lower regret. These results clarify the relationship between prior-error certification and deployment performance. The regret analysis appears in Appendix A.8, while the numerical counterexamples are given in our response to yz4x-W1/Q1.

W2, W5, and W6

The reported downstream gains are largely not driven by the certificate

We acknowledge that certification may not be the largest driver of downstream gains in all settings. However, our additional experiments show that certification and SOFT interpolation can each provide independent benefits.

To isolate the contribution of certification, we developed a joint-gap certificate. On independent synthetic seeds, it produced 93 additional regret-beneficial ACCEPT decisions relative to exact separate bounds. This reduced CONSERVATIVE-policy mean regret by 0.040 (95% CI: 0.034–0.046), without SOFT interpolation (see Appendix B.10.1; Table 19).

We also evaluated certification in a new trust-then-verify experiment. On held-out synthetic seeds, joint-gap certified monitoring reduced regret by 0.11–0.20 relative to a calibrated point-estimate monitor using the same observation schedule. These differences were statistically significant.

For continuous weighting, SOFT achieved significantly lower regret than an independently calibrated fixed-weight baseline with identical hard decisions (HARD+FIXED) in all 27 helpful-prior synthetic configurations (mean improvement of 0.21). The KuaiRand comparison does not establish that continuous weighting is generally redundant: FIXED(0.30) was chosen to approximate SOFT's weights ω(m)\omega^{(m)} on the real datasets, so similar performance in this setting is expected. We do not claim a statistically significant difference on KuaiRand (see Uk7M-W3/Q3).

W3 and W7

theoretical contribution

While our analysis builds on established concentration results, our contribution extends these tools to pre-deployment certification of warm-start priors. The framework provides a finite-sample guarantee for ACCEPT/REJECT decisions, an explicit INCONCLUSIVE outcome, and a condition characterizing when certification becomes decisive.

To further strengthen our contribution, we derived a new joint-gap certificate that directly optimizes the difference in squared prior errors, G(m)(θ)G^{(m)}(\theta), over the shared pilot confidence ellipsoid CTpilotpilot(δ)\mathcal{C}^{\mathrm{pilot}}_{T_{\mathrm{pilot}}}(\delta) , rather than bounding each prior error separately. It preserves the original finite-sample correctness guarantee while increasing synthetic decisiveness from 56.5% to 65.8%.

These results demonstrate how the original certification framework can be extended to obtain tighter decisions. The joint-gap derivation appears in Appendix A.7, with experimental comparisons in Appendix B.10.1 and Table 19.

Reviewer_yz4xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether a candidate warm-start prior can be trusted before deploying a linear contextual bandit. Its Pilot Priors framework uses a short pilot dataset from the target environment to compare a warm prior with a cold-start baseline.

The main contribution is a confidence-based certification procedure. Using a high-probability confidence ellipsoid for the unknown reward parameter, the authors bound the prior error

B0(m)=∥θ0(m)−θ⋆∥A0(m).B_0^{(m)}=\|\theta_0^{(m)}-\theta^\star\|_{A_0^{(m)}}.

Comparing the warm and cold error intervals yields ACCEPT, REJECT, or INCONCLUSIVE. With probability at least 1−δ1-\delta, every decisive classification correctly orders the prior errors. The authors also give a sufficient margin-versus-width condition for decisive certification as pilot information grows.

For inconclusive cases, a soft calibration rule interpolates between warm and cold initialization through a logistic transformation of the normalized difference between interval midpoints.

The framework is evaluated on synthetic bandits with controlled prior corruption, a COVID-19 conjoint dataset with LLM-generated priors for human preferences, and KuaiRand recommendation logs. The results show that more pilot information improves decisiveness and that gating reduces negative transfer. Soft calibration also preserves some gains from useful warm starts while limiting losses from corrupted priors.

Critical strengths

S1. A practical and interpretable approach to pre-deployment prior assessment.

The paper addresses a meaningful problem: whether to trust auxiliary information whose compatibility with the target task is unknown. Its ACCEPT/REJECT/INCONCLUSIVE framework offers an interpretable decision rule, with a 1−δ1-\delta guarantee that clarifies the statistical validity of decisive outcomes.

Unlike methods that exploit auxiliary information or adjust source weights during deployment, pre-deployment certification is useful when trust decisions must be fixed in advance.

S2. Evaluation across complementary synthetic and real-data settings.

The evaluation covers controlled synthetic corruption, LLM-generated preferences compared with human responses, and recommendation logs under policy shift, capturing distinct forms of prior mismatch.

The experiments demonstrate both rejection of harmful priors and the hard certificate’s conservativeness. Appendix comparisons involving fixed-weight baselines, pilot reuse, and online adaptation further strengthen the analysis. Together, these results support the framework’s implementability and the relevance of its statistical decisions to downstream performance.

Critical weaknesses

W1. Limited theoretical novelty and alignment with deployment performance.

The central guarantee certifies whether [ B_0^{(m)}\leq B_0^{\mathrm{cold}}, ] rather than whether warm-starting improves regret or policy value. As Section 2.3 acknowledges, regret also depends on initialization geometry, confidence radius, action-selection trajectory, and deployment horizon. A correct prior-error ordering therefore does not ensure that ACCEPT identifies a better deployment policy or that REJECT prevents negative transfer.

The certificate combines a standard confidence ellipsoid, a generalized eigenvalue bound for norm conversion, and triangle inequalities. Correctness then follows from interval separation. These steps appear sound but are a relatively straightforward application of established concentration tools. Likewise, the margin-versus-width condition follows generically from interval separation rather than providing a bandit-specific account of certification difficulty.

The contribution would be stronger with a formal connection to downstream regret, even under additional assumptions, or evidence that certification offers advantages over simpler pilot-based selection methods.

W2. The benefit of soft calibration over simpler interpolation and gating remains unclear.

The logistic weighting rule uses confidence-interval information but remains heuristic, with no theoretical justification for its particular form in minimizing downstream regret. The empirical results also suggest that simple interpolation or hard rejection explains much of the benefit.

In Appendix B.8.3, SOFT performs nearly identically to fixed interpolation with (\omega=0.30) on KuaiRand, where all certificates are inconclusive. SNIPS lifts are +0.0103 versus +0.0105 for old_full, and −0.0059 versus −0.0062 for shuffled_old; neither difference is statistically significant.

For the corrupted COVID prior, the advantage over fixed interpolation mainly arises when the hard certificate triggers REJECT and removes the source. This supports hard gating but does not isolate the value of the soft weighting formula.

Appendix B.3.4 also reports rejection of priors that improve downstream performance. At preference-flip rate (p=0.20), warm-starting reduces regret by 1.949 units, yet the gated policies revert to cold with sufficiently large pilots. Although consistent with the prior-error guarantee, this highlights its mismatch with deployment performance.

Together, these findings leave unclear whether the full method substantially improves on simpler pilot-based heuristics.

Questions for authors

Q1. Relationship between prior error and deployment regret

Under what conditions does B0(m)<B0coldB_0^{(m)}<B_0^{\mathrm{cold}} imply lower downstream regret, or a meaningfully tighter regret bound, in the same environment? Conversely, can a warm prior with substantially larger B0B_0 consistently outperform cold start? A characterization or counterexample would clarify the certificate’s practical meaning and limitations.

Q2. Incremental value of soft calibration

Appendix B.8.3 reports nearly identical KuaiRand performance for SOFT and fixed interpolation with ω=0.30\omega=0.30 when all decisions are inconclusive. Can the authors identify settings where the continuous trust score materially improves performance independently of hard ACCEPT/REJECT decisions? A useful baseline would retain the same certified endpoints but use a fixed weight for inconclusive cases.

Q3. Validity of assumptions in real-data experiments

The certificate assumes a correctly specified linear reward model, conditionally sub-Gaussian residuals, and ∥θ⋆∥2≤S\|\theta^\star\|_2\leq S. How is S=2S=2 justified for the COVID and KuaiRand feature representations? How does model misspecification affect empirical coverage and certification decisions? Violations could invalidate the stated guarantee in these experiments.

Q4. Pilot cost relative to deployment

Pilot budgets sometimes exceed deployment horizons: up to 25,000 observations for 2,000 deployment rounds in the synthetic experiments, and 20,000 for 10,000 rounds on KuaiRand. How should pilot collection costs enter the total regret or reward objective? Does the method remain beneficial when accounting for these costs and allowing competing methods to learn from the same pilot data?

4 comments
AuthorsOpenReview ↗

Reviewer yz4x Response Part 1

We thank the reviewer for their questions and patience. Please see our responses below.

W1 and Q1

Limited theoretical novelty and alignment with deployment performance

Relationship between prior error and deployment regret

Thank you for this question. A smaller prior error alone does not guarantee lower realized regret. To explain, we have derived a new regret bound that establishes a formal connection between prior-error ordering and downstream performance.

Our bound UBT\mathrm{UB}_T depends on the initial prior error B0\mathcal{B}_0 and the precision A0A_0 of the initialization. We proved that the bound is non-decreasing in B0\mathcal{B}_0 and non-increasing in the minimum eigenvalue λmin⁡(A0)\lambda_{\min}(A_0) of the prior precision matrix. When τpre≥1\tau_{\mathrm{pre}}\ge1, the ridge-type warm initialization satisfies A0(m)⪰A0cold=IA_0^{(m)}\succeq A_0^{\mathrm{cold}}=I. Under this condition, a correct ACCEPT decision implies

B0(m)≤B0cold⟹UBT(m)≤UBTcold.\mathcal{B}_0^{(m)} \le \mathcal{B}_0^{\mathrm{cold}} \quad\Longrightarrow\quad \mathrm{UB}_T^{(m)} \le \mathrm{UB}_T^{\mathrm{cold}}.

Under these assumptions, ACCEPT certifies that WARM has a no-worse regret upper bound than COLD. For a specified horizon TT, the condition $\alpha\ge\overline\beta_T(A_0,\delta_{\mathrm{dep}}) +\mathcal{B}_0$ eliminates the linear-in-TT under-exploration term in the regret bound (Appendix A.8). A fixed α\alpha does ntot need to satisfy this condition as TT grows. This establishes a formal connection to downstream regret bounds, although it does not guarantee lower realized regret.

To address the second part of your question, we constructed exact counterexamples using the same LinUCB algorithm in two-dimensional linear bandits. In the first, WARM has five times the prior error of COLD (B0(m)=5\mathcal{B}_0^{(m)} = 5 versus B0cold=1\mathcal{B}_0^{\mathrm{cold}} = 1), yet achieves zero cumulative regret compared with 0.999 for COLD over 1,000 rounds. In the second, WARM has smaller prior error (B0(m)=1\mathcal{B}_0^{(m)} = 1 versus B0cold=1.408\mathcal{B}_0^{\mathrm{cold}} = 1.408) but incurs regret 10 compared with 0 for COLD. These counterexamples highlight the difficulty of deriving regret guarantees from prior-error ordering alone, as acknowledged in Section 2.3 of our paper.

Q3

Validity of assumptions in real-data experiments

Thank you for raising this concern. The choice S=2S = 2 is an experimental configuration, not a restriction of our theory. Our confidence bound holds for any valid S≥∥θ⋆∥2S \ge \|\theta^\star\|_2 under the stated linear-model assumptions. Increasing SS widens the confidence intervals and generally produces more INCONCLUSIVE decisions.

Regarding model misspecification, a departure from the assumed linear reward model can be represented by an additional term η(xt)\eta(x_t) in the conditional reward, rt=xt⊤θ⋆+η(xt)+ξtr_t=x_t^\top\theta^\star+\eta(x_t)+\xi_t. In principle, if the resulting estimation bias can be bounded with high probability, this bound could be incorporated into an enlarged confidence radius, allowing a corresponding finite-sample certification guarantee under modified assumptions. Without an explicit bound on this bias, model misspecification can reduce confidence-set coverage and lead to incorrect decisive certifications. Establishing this extension is beyond the scope of the present work.

Q4

Pilot cost relative to deployment

Pilot collection cost is important. For a fixed interaction budget Ttotal=Tpilot+TdeployT_{\mathrm{total}} = T_{\mathrm{pilot}} + T_{\mathrm{deploy}}, we evaluate

Rtotal=Rpilot(Tpilot)+Rdeploy(Tdeploy).R_{\mathrm{total}} = R_{\mathrm{pilot}}(T_{\mathrm{pilot}}) + R_{\mathrm{deploy}}(T_{\mathrm{deploy}}).

We conducted additional fixed-budget experiments in which pilot regret was included, pilot observations were reused during deployment, and competing COLD and WARM policies could learn over the same total horizon TtotalT_{\mathrm{total}}. We used a COLD LinUCB pilot, so PILOT-ONLY is equivalent to uninterrupted COLD learning.

Under these conditions, GATED+PILOT reduced total regret relative to COLD in 31/36 helpful-prior synthetic configurations. On COVID with an aligned prior, a pilot of Tpilot=200T_{\mathrm{pilot}} = 200 tasks within a total budget of Ttotal=2,000T_{\mathrm{total}} = 2{,}000 tasks reduced regret by 62.1 relative to uninterrupted COLD (95% CI: 49.5–75.1), including pilot costs.

We also evaluated a balanced synthetic mixture of helpful and harmful priors. GATED+PILOT reduced total regret by 23–34 relative to uninterrupted WARM, but generally did not outperform uninterrupted COLD. The additional regret over COLD was 0.17–2.13 in the shorter-pilot configurations. One possible explanation is that the COLD pilot delays use of beneficial WARM priors, while SOFT interpolation further reduces their influence.

Pilot reuse can preserve useful transfer and reduce negative transfer under a common interaction budget, though its net benefit depends on source quality and available horizon. The detailed fixed-budget comparisons are provided in our response to 35Zk-W4.

AuthorsOpenReview ↗

Reviewer yz4x Response Part 2

Part 2 of our response to Reviewer yz4x

W2 and Q2

The benefit of soft calibration over simpler interpolation and gating

Incremental value of soft calibration

Thank you for this comment. To isolate the value of continuous weighting, we compared SOFT against an independently calibrated fixed-weight baseline with identical ACCEPT/REJECT decisions (HARD+FIXED; see Uk7M-W3/Q3). On 100 held-out synthetic seeds with helpful priors, SOFT achieved significantly lower regret in all 27 configurations (mean improvement of 0.21). On KuaiRand, SOFT and FIXED(0.30) remained statistically indistinguishable.

The fixed weight ω=0.30\omega = 0.30 in Appendix B.8.3 was chosen because it approximates the weights ω(m)\omega^{(m)} produced by SOFT on the real datasets. It is heuristic and does not optimize for downstream regret. To address this, Appendix A.9 derives a separate certificate-optimized weighting rule that selects the interpolation weight by optimization rather than using the fixed weight or sigmoid employed in the main experiments.

For information-form interpolation, we proved the exact identity

(B0hybrid,(m))2=(1−ω)(B0cold)2+ω(B0(m))2−J(m)(ω),\bigl(\mathcal{B}_0^{\mathrm{hybrid},(m)}\bigr)^2 = (1-\omega)\bigl(\mathcal{B}_0^{\mathrm{cold}}\bigr)^2 + \omega\bigl(\mathcal{B}_0^{(m)}\bigr)^2 - J^{(m)}(\omega),

where B0hybrid,(m):=∥θ0hybrid,(m)−θ⋆∥A0hybrid,(m)\mathcal{B}_0^{\mathrm{hybrid},(m)} := \lVert \theta_0^{\mathrm{hybrid},(m)} - \theta^\star \rVert_{A_0^{\mathrm{hybrid},(m)}} is the prior error of the hybrid prior with weight ω\omega (Appendix A.6), and J(m)(ω)≥0J^{(m)}(\omega) \ge 0 is an explicit gain from blending the two priors. Let G‾(m)(δ)\overline{G}^{(m)}(\delta) denote the maximum squared prior-error difference ∥θ0(m)−θ∥A0(m)2−∥θ0cold−θ∥A0cold2\lVert \theta_0^{(m)} - \theta \rVert_{A_0^{(m)}}^2 - \lVert \theta_0^{\mathrm{cold}} - \theta \rVert_{A_0^{\mathrm{cold}}}^2 over the pilot confidence ellipsoid θ∈CTpilotpilot(δ)\theta \in \mathcal{C}^{\mathrm{pilot}}_{T_{\mathrm{pilot}}}(\delta) (the upper endpoint of the joint-gap certificate in Uk7M-W2/Q2). We then select

ω⋆=arg⁡min⁡ω∈[0,1]{ω G‾(m)(δ)−J(m)(ω)}.\omega^\star = \arg\min_{\omega \in [0,1]} \Bigl\lbrace \omega \thinspace \overline{G}^{(m)}(\delta) - J^{(m)}(\omega) \Bigr\rbrace.

This is a one-dimensional convex optimization problem. Because ω=0\omega = 0 is feasible, the selected hybrid is certified to have no greater squared prior error than COLD, with probability at least 1−δ1-\delta under the original assumptions.

This rule can select a positive warm weight even when the hard certificate is INCONCLUSIVE. In 144 exploratory synthetic INCONCLUSIVE comparisons, 72 admitted a hybrid certified to have lower prior error than COLD. This provides a theoretically justified alternative to the heuristic sigmoid and fixed ω=0.30\omega = 0.30 interpolation. The new guarantee applies to squared prior error, while the regret improvements reported above concern the submitted SOFT method.

Thank you for the detailed rebuttal and additional results. The responses addressed some of my concerns with the new regret analysis and fixed-budget experiments.

But I do still have two concerns remaining: The new regret bound provides a useful connection to deployment performance, but only under additional conditions. It remains for me a question of how plausible these conditions are. Moreover, the optimized interpolation is an interesting addition, but seems that its guarantee does not apply to the original SOFT method?

Overall I am not fully convinced and will keep my score as is.

AuthorsOpenReview ↗

Response to Official Comment by Reviewer yz4x

We thank Reviewer yz4x for their follow-up. We would like to clarify the scope and practical relevance of the new guarantees.

Regret bound plausibility

Appendix A.8 requires (i) standard boundedness and conditional-noise assumptions already used in our framework, (ii) a common horizon and exploration coefficient for WARM and COLD, and (iii) A0(m)⪰IA_0^{(m)}\succeq I.

Condition (ii) ensures a controlled comparison rather than imposing restrictions on deployment. Without this assumption, it is difficult to compare WARM and COLD. Enforcing (ii) isolates the effect of initialization from differences in exploration settings.

Regarding (iii), this condition holds by construction in all our experiments. We used τpre=1\tau_{\mathrm{pre}}=1, so A0(m)=Xm⊤Xm+I⪰IA_0^{(m)}=X_m^\top X_m+I\succeq I, regardless of source-data alignment. The stronger horizon-dependent exploration condition is not required for the ACCEPT-based regret upper-bound comparison. It is only needed to eliminate the linear under-exploration term.

Guarantee applying to the original SOFT

The exact interpolation identity in Appendix A.9 applies to the original sigmoid-selected SOFT weight. The certificate inequality in A.9 provides a high-probability bound on its squared prior-error difference relative to COLD. We agree that the stronger cold-relative guarantee applies specifically to the separately optimized weight ω⋆\omega^{\star} and not the original SOFT method, though the empirical comparisons in Appendix B.8.4 support SOFT's downstream performance.

Reviewer_Uk7MOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines the reliability of warm-start priors in contextual bandits. The authors primarily consider a scenario in which a prior has already been derived from historical data or data generated by a large language model (LLM), but it is unclear whether it is suitable for the specific task; using it directly could actually lead to greater regret.

To address this problem, the authors propose the ‘Pilot Priors’ framework, which uses a small amount of pilot data from the actual task to compare the prior errors of the ‘warm-start’ and ‘cold-start’ approaches. Specifically, the authors first construct a confidence ellipsoid for the target parameter, then they calculate the upper and lower bounds of the prior errors for both approaches, and, based on the relative relationship between these intervals, they decide whether to accept, reject, or provisionally defer the decision. The work proves the correctness of the hard certification decisions with probability at least 1−δ and provides sufficient conditions for the intervals to be disjoint. For cases where it is not possible to make a clear decision, the authors also developed a ‘soft calibration’ method that combines both initialization approaches by adjusting the weights.

Experiments were conducted primarily with synthetic data, COVID-19 preference data, and KuaiRand. The results show that this method is capable of identifying inappropriate priors in some cases, while the soft calibration retains some of the advantages of beneficial ‘warm-start’ initializations, reducing the losses caused by incorrect initializations.

Critical strengths
  1. In my opinion, the topic this paper addresses is of practical importance. For contextual bandits, prior knowledge about the target task may not be useful, especially if it comes from historical data or data generated by an LLM, because the distributions may differ. The authors attempt, before deployment, to decide whether to use this prior knowledge rather than relying solely on subsequent online updates. I find this problem formulation to be reasonable and practically useful.

  2. The theoretical analysis of the paper is very comprehensive. The authors use pilot data to construct confidence intervals and to provide a correctness guarantee for hard certification decisions. I have checked Theorems 1, 2, and 4, and Corollary 5, and I have not found any obvious theoretical errors. Although the mathematical tools used are mostly standard, the authors provide a clear theoretical explanation of the certification framework, rather than merely recommending a heuristic. In addition, the synthetic and COVID experiments also provide some supporting evidence that the method can sometimes identify harmful priors and reduce losses resulting from negative transfer.

Critical weaknesses
  1. **There is still a gap between the theoretical certification criterion and the actual optimization target. **What the authors certify is the ordering of prior errors, not the regret after actual deployment. The main text (see Table 1) shows that in KuaiRand there is a clear difference in downstream performance between beneficial and harmful priors, but Section 5.3 did not provide a definitive hard-certification decision. Although this does not violate the theoretical guarantees, whether the certification criterion can provide effective guidance for deployment decisions remains unclear. Therefore, more experimental evidence is needed.

  2. There are practical usability issues with hard certification and problems with sample costs. Figures 1 and 2 show that as the amount of pilot data increases, the certification becomes clearer. However, in KuaiRand, even using 20,000 pilot samples, all results remained INCONCLUSIVE (Table 13). Additionally, in some experiments the pilot length exceeded the deployment horizon. I believe the authors should also evaluate whether certification remains beneficial under a fixed total data budget, including the sample costs required for certification.

  3. The independent contribution of soft calibration is not yet clear. The main text (Table 2) shows that soft calibration preserves some of the benefits of beneficial warm-start priors, which is a positive result. However, Table 16 shows that the soft calibration method and a fixed-weight mixture with weight 0.30 perform similarly on KuaiRand, and the difference is not statistically significant. Therefore, the current experiments are not sufficient to demonstrate the additional benefit of adaptive weights compared to a simple fixed-weight mixture. Additional experiments with different fixed weights could be performed. A baseline that retains hard certification but uses a fixed weight for INCONCLUSIVE cases could also help separate the effects of hard rejection from those of adaptive weighting.

  4. The benefit of additional certification, compared to using pilot data directly, is still not clear. In the main experimental study, pilot data were used for certification rather than directly for subsequent initialization. Although the authors added relevant experiments in Table 17, the additional benefit remains unclear. The results show that, using 20,000 pilot samples on KuaiRand, GATED+PILOT achieves only about a 0.0003 improvement in SNIPS compared to PILOT ONLY. I believe that the actual value of the additional certification step needs to be further clarified.

Questions for authors
  1. Table 1 in the main text shows that downstream performance differs significantly across different priors, yet hard certification still cannot distinguish them. Could the authors further analyze the relationship between prior-error ordering and actual regret improvement? For example, by calculating the agreement rate between certification decisions and downstream performance improvements under different experimental conditions, as well as the proportion of beneficial priors that are rejected.

  2. Figures 1 and 2 in the main text show the effect of pilot data size on certification results, but in KuaiRand even using 20,000 samples still fails to yield a clear verdict. Could the authors report certification rates and interval widths for longer pilot lengths? Also, could they compare final performance across different pilot lengths under a fixed total data budget to demonstrate whether certification gains offset data collection costs?

  3. The main text's Table 2 shows that soft calibration can alleviate negative transfer, but in Table 16 its performance is close to that of the fixed-weight method. Could the authors include comparative experiments with different fixed weights? In particular, could they add a baseline that retains hard certification but uses a fixed weight in INCONCLUSIVE cases, to disentangle the contributions of hard rejection and adaptive weighting?

  4. Table 17 shows that on KuaiRand using 20,000 pilot samples, the improvement of GATED+PILOT over PILOT ONLY is very limited. Is this difference statistically significant? Could the authors further compare these two methods under different pilot lengths and source sizes, and report the corresponding confidence intervals to indicate under which conditions the additional certification is truly beneficial?

2 comments
AuthorsOpenReview ↗

Reviewer Uk7M Response Part 1

Thank you for your feedback, questions, and patience. We appreciate your assessment of our theoretical results. We have addressed your concerns below.

W1 and Q1

Gap between the theoretical certification criterion and the actual optimization target

Analyze the relationship between prior-error ordering and actual regret improvement

Thank you for raising this concern. As stated in Section 2.3 of our paper, our certificate guarantees prior-error ordering, not lower deployment regret. To further examine this relationship, we conducted additional synthetic experiments with 100 seeds and known target parameters θ⋆\theta^\star.

Prior-error and regret orderings agreed in 766/1,000 distinct seed–candidate comparisons (76.6%; 95% Wilson CI: 73.9–79.1%), as reported in Appendix B.3.5. We observed no incorrect decisive prior-error decisions or regret-harmful ACCEPT decisions in these tests. However, regret-beneficial priors were sometimes rejected, increasing from 9.8% at Tpilot=1,000T_{\mathrm{pilot}} = 1{,}000 to 47.5% at Tpilot=25,000T_{\mathrm{pilot}} = 25{,}000.

We also tested whether tighter certification could improve deployment. In an independent synthetic replication, joint-gap certification (described in W2) produced 93 additional regret-beneficial ACCEPT decisions relative to exact separate bounds. The resulting CONSERVATIVE policy reduced mean regret by 0.040 (95% CI for the reduction: 0.034–0.046) relative to CONSERVATIVE with exact separate bounds, without SOFT interpolation (Appendix B.10.1).

Regarding KuaiRand, we found that 77–79% of the helpful old_full prior's measured squared prior error comes from invariant coordinates. These coordinates cancel in direct arm-score comparisons, helping explain the disagreement between global parameter error and action quality.

W2 and Q2

Practical usability issues with hard certification and problems with sample costs

Certification rates and interval widths for longer pilot lengths

Appendix B.7.3 (Table 13) examines KuaiRand certification. To assess whether more target-task information resolves them, we extended the pilot from Tpilot=20,000T_{\mathrm{pilot}} = 20{,}000 to 102,898102,898 randomized-intervention observations while keeping the WARM prior fixed (100,000 source observations). We found that the warm interval width W(m)(δ)\mathcal{W}^{(m)}(\delta) decreased by 74%, leading to a decrease in INCONCLUSIVE decisions.

To improve our certificate, we have derived a new joint-gap certificate. Rather than bounding the squared prior errors separately, it directly bounds their difference,

G(m)(θ):=∥θ0(m)−θ∥A0(m)2−∥θ0cold−θ∥A0cold2,G(m)(θ⋆)=(B0(m))2−(B0cold)2,G^{(m)}(\theta) := \lVert \theta_0^{(m)} - \theta \rVert_{A_0^{(m)}}^2 - \lVert \theta_0^{\mathrm{cold}} - \theta \rVert_{A_0^{\mathrm{cold}}}^2, \qquad G^{(m)}(\theta^\star) = \bigl(\mathcal{B}_0^{(m)}\bigr)^2 - \bigl(\mathcal{B}_0^{\mathrm{cold}}\bigr)^2,

over the same pilot confidence ellipsoid CTpilotpilot(δ)\mathcal{C}^{\mathrm{pilot}}_{T_{\mathrm{pilot}}}(\delta) . We compute its minimum G‾(m)(δ)\underline{G}^{(m)}(\delta) and maximum G‾(m)(δ)\overline{G}^{(m)}(\delta) over the ellipsoid using two trust-region optimization problems. This preserves the finite-sample prior-error correctness guarantee under the stated assumptions. Appendix A.7 gives the derivation, and Appendix B.10.1 and Table 19 report the experimental comparisons.

Table: Certification results using matched pilot data.

CertificateSynthetic decisive rateKuaiRand REJECTs
Analytic separate56.5%17/792
Exact separate bounds59.6%130/792
Joint-gap65.8%691/792

Joint-gap increased synthetic decisiveness by 9.3 percentage points, with no observed incorrect prior-error decisions.

We also evaluated fixed-total-budget performance with reusable pilot data (see 35Zk-W4). Pilot reuse reduced negative transfer after accounting for pilot costs, but did not consistently outperform uninterrupted COLD.

AuthorsOpenReview ↗

Reviewer Uk7M Response Part 2

This is part 2 of our response to Reviewer Uk7M.

W3 and Q3

Independent contribution of soft calibration is not yet clear

Disentangle the contributions of hard rejection and adaptive weighting

Appendix B.8.3 compares fixed weights ω∈{0.25,0.30,0.50,0.75}\omega \in \{0.25, 0.30, 0.50, 0.75\}. Appendix B.8.4 additionally evaluates HARD+FIXED, which retains SOFT's ACCEPT and REJECT decisions but uses an independently calibrated fixed weight when certification is INCONCLUSIVE.

We calibrated weights on separate data and evaluated them on 100 held-out synthetic seeds. We used leave-group-out calibration for helpful priors and pooled calibration for a balanced mixture.

Table: SOFT relative to fixed-weight baselines. Lower regret and higher SNIPS favor SOFT.

ComparisonResult
Helpful synthetic priors (leave-group-out HARD+FIXED)lower regret in 27/27 configurations, mean difference −0.21
Balanced synthetic mixture (pooled HARD+FIXED)Mean regret difference +0.58
KuaiRand (FIXED 0.30)SNIPS difference −0.0003; 95% CI [−0.0013, +0.0009]

SOFT achieved significantly lower regret for helpful synthetic priors, but this advantage did not extend to all source regimes. On KuaiRand, the difference was not statistically significant.

These results separate the benefit of hard REJECT decisions from the contribution of adaptive interpolation. We also derived an exact theoretical identity for information-form interpolation, presented in our response to 35Zk-W5/Q3.

W4 and Q4

The benefit of additional certification, compared to using pilot data directly, is still not clear

Could the authors further compare these two methods under different pilot lengths and source sizes

Thank you for raising your concerns. In our paper, Appendix B.9.1 and Table 17 already show that retaining source information can improve performance over PILOT-ONLY. To address your question about statistical significance, we extended these comparisons during rebuttal using matched seeds and deployment windows. The table below reports the performance differences and their 95% confidence intervals.

Table: GATED+PILOT relative to PILOT-ONLY. Lower regret and higher SNIPS are better.

Dataset / TpilotT_{\mathrm{pilot}}Difference95% CI
COVID / 1,000−50.1 regret[−64.1, −36.8]
COVID / 2,000−32.6 regret[−42.0, −23.7]
KuaiRand / 5,000+0.0032 SNIPS[+0.0010, +0.0054]
KuaiRand / 20,000+0.0003 SNIPS[−0.0003, +0.0009]

The COVID results show that GATED+PILOT retains useful source information beyond PILOT-ONLY. On KuaiRand, the improvement is small and is not significant after multiple-comparison correction.

Reviewer_35ZkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces Pilot Priors, a framework to assess the quality of some warm-start priors for linear contextual bandits before deployment. The approach uses a short pilot phase with real target-task data to construct confidence intervals for the prior error of the prior and as a comparison additional of a cold-start baseline. If the confidence intervals do not intersect, the methods decides between ACCEPT, REJECT or INCONCLUSIVE about the prior with a guarantees corectness with probability 1−δ1-\delta. The authors additionally propose a soft trust-calibration mechanism that interpolates between warm and cold starts. The proposed approach is evaluated in synthetic experiments and two real-world scenarios involving LLM-generated preferences and historical recommendation data, demonstrating its potential to identify bad priors.

Critical strengths
  • relevant and well-motivated problem, in particular with the (possibly) increasing use of LLM-generated priors
  • clear and intuitive framework in a UCB-style
  • theoretical correctness guarantee with high probability is provided and seems reasonable
  • I like to have the option of using a soft trust-calibrated prior if the test was inconclusive
  • experimental evaluation on different datasets and with different prior sources
Critical weaknesses
  • lack of pseudo code: from an algorithmic perspective it would improve the understanding a lot if the procedure would be shown in an algorithmic overview. It was not completely clear to me when during the deployment of an algorithm which test is done and for how long
  • limited theoretical novelty: the theoretical results make sense, but seem to rely primarily on established self-normalized confidence bound. The framework to use these for the comparison of priors is useful, but the underlying theoretical contribution seems somewhat incremental, in particular for a top-tier conference like ICLR
  • restrictive modeling assumptions: the framework focus on contextual bandits with linear reward model only. It is unclear whether and how the approach can be extended to other bandit setting, or at least to other reward models like a generalized linear model (which is much less restrictive and also commonly used in bandit literature like logistic bandits)
  • pilot overhead and opportunity costs: the framework introduces a separate pilot phase and only the remaining budget afterwards can be used for deployment and learning of the real target algorithm. To me, this raises the question whether under a fixed total budget, your approach actually outperforms simply running the target algorithm from a cold start for the entire available budget. Or whether at least the observations gathered in the pilot phase can be used? Or whether an online-testing of the prior quality would make more sense from an efficiency perspective?
  • heuristic self trust calibration: it is not clear to me why convex interpolation between two 'somehow good' priors should consistently yield a new useful prior for deployment. Couldn't it happen, that both, the tested prior and the cold start prior were different local optima, thus hard to distinguish, but the midpoint in between is much worse?
  • lack of live online evaluation: COVID uses sequential evaluations on a fixed held-out set of human responses and Kuairand relies on offline clustered arm replays. It remains unclear how your approach performs in an actual adaptive deployment environment
Questions for authors
  • as already mentioned in the weaknesses: I am wondering whether you can easily adapt your framework and the theoretical guarantees to a generalized linear model or to other kinds of reward models in contextual bandits?
  • how sensitive is you certification framework to the policy used during the pilot phase? Since the policy decides how often which action is played and how much to explore and to exploit, i expect that it might also effect the evaluation of the priors?
  • what is the intuition behind the interpolation? Is there a theoretical guarantee that this is a better approach than just choosing one of the tested priors uniformly at random?
2 comments
AuthorsOpenReview ↗

Reviewer 35Zk Response Part 1

Thank you for your careful review and feedback. We address your weaknesses and questions below.

W1

Lack of pseudo code

Thank you for this suggestion. Algorithm 1 in the revised paper now summarizes the complete procedure, including pilot collection, certification, initialization, and deployment. The steps are reproduced below:

  1. Pilot: Collect TpilotT_{\mathrm{pilot}} target-task observations using a shared pilot policy πpilot\pi_{\mathrm{pilot}}.
  2. Certify: Construct the certified intervals [B‾(m)(δ),B‾(m)(δ)][\underline{\mathcal{B}}^{(m)}(\delta), \overline{\mathcal{B}}^{(m)}(\delta)] and [B‾cold(δ),B‾cold(δ)][\underline{\mathcal{B}}^{\mathrm{cold}}(\delta), \overline{\mathcal{B}}^{\mathrm{cold}}(\delta)] for the WARM and COLD prior errors.
  3. Decide:
    • ACCEPT if B‾(m)(δ)≤B‾cold(δ)\overline{\mathcal{B}}^{(m)}(\delta) \le \underline{\mathcal{B}}^{\mathrm{cold}}(\delta).
    • REJECT if B‾(m)(δ)>B‾cold(δ)\underline{\mathcal{B}}^{(m)}(\delta) > \overline{\mathcal{B}}^{\mathrm{cold}}(\delta).
    • INCONCLUSIVE otherwise.
  4. Initialize: Use WARM for ACCEPT (ω(m)=1\omega^{(m)}=1), COLD for REJECT (ω(m)=0\omega^{(m)}=0), or SOFT interpolation with weight ω(m)\omega^{(m)} for INCONCLUSIVE.
  5. Deploy: Run LinUCB using the selected initialization.

Appendix B.9 provides additional details on pilot reuse and early stopping.

W3 and Q1

Restrictive modeling assumptions

Whether you can easily adapt your framework and the theoretical guarantees to a generalized linear model

The framework presented in our paper requires a valid confidence set for the target parameter θ⋆\theta^\star. We agree that extensions to generalized linear models may be possible using model-specific confidence regions. However, establishing the corresponding theoretical guarantees would require separate analysis and is beyond the scope of this paper.

W4

Whether the observations gathered in the pilot phase can be used / online testing

Thank you for raising your concerns. Our framework targets settings where source trust must be selected before deployment, with limited target-task data. Appendix B.9 and Table 17 show that pilot observations can be reused and that retaining useful source information can improve performance over pilot-only learning. We also study early stopping (Appendix B.9.2) and online adaptation with ARRoW-CB (Appendix B.10.3, Table 18), finding no consistent advantage for either approach.

Appendix B.9.1 now includes fixed-total-budget experiments that charge pilot regret. We summarize these results in the table below. We used total budgets of Ttotal=Tpilot+Tdeploy∈{2,000, 10,000, 25,000}T_{\mathrm{total}} = T_{\mathrm{pilot}} + T_{\mathrm{deploy}} \in \{2{,}000,\ 10{,}000,\ 25{,}000\} rounds, with pilots using 10–50% of the budget (5–25% for mixed-source experiments). For COVID, we used Ttotal=2,000T_{\mathrm{total}} = 2{,}000 tasks, with Tpilot=200T_{\mathrm{pilot}} = 200 for the aligned prior and Tpilot=1,000T_{\mathrm{pilot}} = 1{,}000 for the corrupted prior. Lower regret is better. We find that GATED+PILOT reduced regret relative to the specified comparator in most tested settings, even after accounting for pilot costs.

Table: Fixed-total-budget experiments with reusable pilot data.

DatasetComparisonResult
Synthetic (helpful priors)GATED+PILOT vs. COLDLower regret in 31 of 36 tested configurations
Synthetic (harmful priors)GATED+PILOT vs. WARMLower regret in all 72 tested configurations
Synthetic (mixed priors)GATED+PILOT vs. WARM23–34 fewer regret units
COVID (aligned prior)GATED+PILOT vs. COLD62.1 fewer regret units (95% CI: 49.5–75.1)
COVID (corrupted prior)GATED+PILOT vs. WARM317.2 fewer regret units

W5 and Q3

Heuristic self trust calibration

The intuition behind the interpolation, is there a theoretical guarantee

Thank you for raising this question. SOFT interpolates the prior precision matrices and information vectors (A0,b0)(A_0,b_0) rather than averaging parameter estimates θ0\theta_0 (Appendix A.6). Appendix A.9 now establishes the following exact interpolation identity. For any fixed weight ω∈[0,1]\omega\in[0,1],

(B0hybrid,(m))2=(1−ω)(B0cold)2+ω(B0(m))2−J(m)(ω),J(m)(ω)≥0,\bigl(\mathcal{B}_0^{\mathrm{hybrid},(m)}\bigr)^2 = (1-\omega)\bigl(\mathcal{B}_0^{\mathrm{cold}}\bigr)^2 + \omega\bigl(\mathcal{B}_0^{(m)}\bigr)^2 - J^{(m)}(\omega), \\ J^{(m)}(\omega) \ge 0,

where B0hybrid,(m):=∥θ0hybrid,(m)−θ⋆∥A0hybrid,(m)\mathcal{B}_0^{\mathrm{hybrid},(m)} := \lVert \theta_0^{\mathrm{hybrid},(m)} - \theta^\star \rVert_{A_0^{\mathrm{hybrid},(m)}} is the prior error of the hybrid prior with weight ω\omega, and J(m)(ω)J^{(m)}(\omega) is a nonnegative term. For any fixed weight ω\omega, interpolation has no greater squared prior error than the expected squared prior error from randomly selecting WARM with probability ω\omega and COLD otherwise.

We also compared SOFT against uniformly random endpoint selection using 100 synthetic seeds across 30 configurations. SOFT achieved lower regret in 27 of 30 configurations (Appendix B.8.4). As noted in our paper, however, we do not establish a general regret guarantee.

AuthorsOpenReview ↗

Reviewer 35Zk Response Part 2

Q2

How sensitive is the certification framework to the policy used during the pilot phase

Theorem 1 allows for adaptive pilot actions under the stated linear reward and conditional-noise assumptions. As aforementioned, the pilot policy πpilot\pi_{\mathrm{pilot}} affects confidence-set geometry and therefore decisiveness.

To assess this, we evaluated five pilot policies: uniform exploration, COLD LinUCB, COLD epsilon-greedy, and greedy policies using aligned or misaligned warm priors (Appendix B.3). Across 100 synthetic seeds and 10 candidate priors per seed, with Tpilot=500T_{\mathrm{pilot}} = 500, decisiveness ranged from 41.2% to 42.5%. All policies achieved 100/100 observed confidence-set coverage, with no incorrect decisive prior-error decisions across 1,000 evaluations per policy.

These findings suggest limited sensitivity to pilot policy in this synthetic setting, although pilot design remains important when feature directions are poorly observed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.