acceptodds
Under review as a conference paper at ICLR 2027

Learning Feature Interfaces to Steer Prescribed Contextual Bandits

Abstract

Many online decision systems are modular: an upstream component controls the features exposed to a downstream learner but cannot change the learner's reward or learning rule. While the learner optimizes its prescribed reward, the upstream component may optimize a different system-level objective. We study feature-interface optimization in repeated linear contextual-bandit deployments. Before each deployment, a leader applies a positive coordinate-wise rescaling to the raw features exposed to a prescribed contextual-bandit learner, which then learns from bandit feedback. Even this simple intervention leaves the follower's reward task unchanged while altering its finite-sample estimates, exploration, and collected feedback. As a result, the leader objective can depend nontrivially on the feature interface. We propose CILOT, which optimizes the interface using an unbiased trajectory-gradient estimator that differentiates through the follower's feedback-dependent online updates using only selected-action feedback. A single follower calibration guarantees regret uniformly over all admissible interfaces, while the leader optimization admits an projected-stationarity guarantee. Experiments on synthetic and real-world datasets demonstrate the benefit of optimizing the feature interface.

Primary area:
learning theory
Keywords:
bandit, linear bandit

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2avKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies feature-interface optimization in repeated linear contextual-bandit deployments. Before each deployment, an upstream leader applies a coordinate-wise rescaling v∈[α,1]dv\in[\alpha,1]^d to the raw features seen by a prescribed softmax LinUCB follower, aiming to maximize a discounted system-level utility that may differ from the follower's reward. The rescaling leaves the follower's reward task and optimal actions unchanged. However, it is equivalent to replacing the follower's ridge penalty with λ,Diag(v−2)\lambda\\,\mathrm{Diag}(v^{-2}), so it alters finite-sample estimation and exploration. The proposed method, CILOT, optimizes vv across deployments by projected stochastic ascent on an unbiased likelihood-ratio gradient that differentiates the follower's policy through its regression state, using only selected-action feedback. The theory shows that a single follower calibration yields O~(dH)\tilde O(d\sqrt H) regret uniformly over admissible interfaces, and that the leader's optimization reaches O~(M−1/2)\tilde O(M^{-1/2}) projected stationarity. Experiments on a synthetic environment and on Last.FM and MovieLens compare CILOT only with the identity interface. CILOT improves leader utility in all three, while follower regret rises by about 51% on the synthetic environment and falls on both real datasets.

Critical strengths
  1. A clean formulation that pins down how an upstream interface can influence a fixed learner. The paper targets a common pattern in modular decision systems: an upstream component controls what a downstream bandit learner sees but cannot change its reward or update rule. The leader is restricted to invertible coordinate-wise rescalings, which keep the follower's reward task and reward-optimal actions unchanged. Any effect on the leader's objective must therefore come through the follower's finite-sample estimation and exploration, not through representation learning or reward shaping. The raw-coordinate equivalence makes this mechanism explicit. Along any history, rescaling is the same as running ridge regression with the diagonal penalty λ,Diag(v−2)\lambda\\,\mathrm{Diag}(v^{-2}), so the interface acts only through the regularizer.

  2. An unbiased leader gradient that accounts for both how the interface changes the follower's estimates and which feedback it collects. The interface affects the leader objective through two channels:

    • Along a fixed history, it changes the follower's estimates and confidence bonuses.
    • Across histories, it changes the follower's action probabilities, and therefore which feedback gets collected and what later updates see.

    CILOT differentiates the follower's conditional policy through its regression state to capture the first channel, and uses likelihood-ratio terms to capture the second. The resulting gradient is unbiased.

Critical weaknesses

W1. The theory does not support the claim of compatible guarantees at the two levels.

The interface leaves the follower-optimal action a⋆(x)a^\star(x) unchanged, so the leader can only move its utility on rounds where the follower makes errors. For any v,v′v,v' and nonnegative weights ωt\omega_t,

∣J(v)−J(v′)∣≤∑tωt[Pv(at≠at⋆)+Pv′(at≠at⋆)].|J(v)-J(v')|\le\sum_t\omega_t\big[\mathbb P_v(a_t\ne a^\star_t)+\mathbb P_{v'}(a_t\ne a^\star_t)\big].

In expectation, suboptimal rounds with gap at least ε\varepsilon have number at most RegH/ε\mathrm{Reg}_H/\varepsilon. Under a margin condition P(Δ(x)<ε)≤C0εκ\mathbb P(\Delta(x)<\varepsilon)\le C_0\varepsilon^\kappa, the leader's influence on the undiscounted average is therefore

O((sup⁡vRegH/H)κ/(1+κ))→0.O\big((\sup_v\mathrm{Reg}_H/H)^{\kappa/(1+\kappa)}\big)\to0.

That is, the stronger the follower guarantee, the less the leader can achieve.

The fixed-γ\gamma objective sidesteps this. With γ=0.99\gamma=0.99 and H=200H=200, 73% of the weight falls on the first 100 rounds and 46% on the first 50. That is fewer observations than d=50d=50, so the follower is still prior-dominated. In that window Theorem 2 is vacuous: assuming λ=S=1\lambda=S=1 and σY=0\sigma_Y=0 (not reported), its bound exceeds the trivial 2H2H until at least H≈3.5×105H\approx3.5\times10^5 at α=0.1\alpha=0.1.

Theorem 2's uniformity over V\mathcal V is also immediate, since the bound does not depend on vv. It places no limit on how much the leader can degrade the follower relative to Identity: synthetic regret rose 51% while the theorem held. "Competence-preserving" thus has no content in the regime where the leader acts.

W2. The experiments do not isolate steering toward a different objective.

The only baseline is the identity interface, and the main text reports total leader utility without separating follower reward from platform value.

Synthetic. Features and parameters are drawn from symmetric distributions. Flipping (Φ,Θ⋆)↦(−Φ,−Θ⋆)(\Phi,\Theta^\star)\mapsto(-\Phi,-\Theta^\star) leaves every follower trajectory unchanged but maps ca↦1−cac_a\mapsto1-c_a. So, averaged over environment draws, no interface can raise the expected platform value. Yet CILOT raises leader utility from 0.520 to 0.590 while follower regret goes up, and the learned interface correlates with ww at 0.958±0.0060.958\pm0.006 in every seed. The authors should reconcile this with the stated generative model.

Real data. On Last.FM and MovieLens, the platform value is built from the same interaction data whose SVD defines the features, so it is largely aligned with the follower's reward. That fits with CILOT also lowering follower regret on both datasets. The gains may come from better regularization rather than steering, and Table 4 omits λ\lambda, βt\beta_t and τ0\tau_0, so this cannot be checked. Three runs would separate the explanations: a scalar interface, a run with the leader utility set to the follower's reward, and a setting where the two objectives conflict.

Questions for authors
  1. Leader leverage versus follower competence. Do the authors agree with the bound in W1? If so, How does the gain sup⁡vJγ,H(v)−Jγ,H(1)\sup_vJ_{\gamma,H}(v)-J_{\gamma,H}(\mathbf 1) behave as γ→1\gamma\to1? How large can sup⁡vRegH(v)−RegH(1)\sup_v\mathrm{Reg}_H(v)-\mathrm{Reg}_H(\mathbf 1) be?

  2. Steering or better regularization? CILOT's gain over Identity could come from steering the follower toward high-platform-value arms, or from making the follower better at its own task in early rounds, since the leader utility also rewards YtY_t. Two facts make the second plausible: the interface is equivalent to a diagonal ridge penalty, and calibrating βt\beta_t for comparator norm S/αS/\alpha may make the identity follower over-explore. Please:

    • (a) report λ\lambda, SS, δ0\delta_0, βt\beta_t and τ0\tau_0, and state whether βt\beta_t uses the class-wide calibration;
    • (b) report the gain of a scalar interface v=κ1v=\kappa\mathbf 1 optimized the same way, which is equivalent to tuning a single ridge parameter;
    • (c) train the interface with the leader utility replaced by the follower's reward alone, and report its gain under the original leader utility;
    • (d) on Last.FM and MovieLens, report the follower-reward and platform-value components of UtU_t separately.

    If (b) or (c) recovers most of the gain, the improvement reflects better regularization of the follower rather than steering toward the platform value.

Reviewer_NhemOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how a leader can influence the learning behavior of a linear contextual bandit through feature scaling, without modifying the follower’s reward function or learning algorithm. The proposed method, CILOT, learns a positive diagonal feature transformation to improve the leader’s utility. Its trajectory-gradient estimator accounts for how this transformation affects both the regression state constructed from past observations and subsequent action probabilities. Under suitable confidence and temperature schedules, the paper establishes a O~(dH)\widetilde O(d\sqrt H) expected follower regret bound uniformly over admissible interfaces. For an averaged and damped outer update, it also provides a O~(M−1/2)\widetilde O(M^{-1/2}) bound on the expected squared norm of the projected gradient mapping. Experiments on synthetic and semi-synthetic environments demonstrate improvements in leader utility over the Identity interface, while the effects on follower performance vary across environments.

Critical strengths

The problem formulation clearly distinguishes the leader’s objective from the follower’s learning process. The connection between feature scaling and diagonal ridge regularization provides a concrete explanation of how reward-preserving transformations can alter finite-time learning behavior. The uniform regret guarantee over admissible interfaces is particularly relevant when the leader changes the interface between deployments. The gradient analysis also carefully accounts for the dependence of the regression state on the interface. Proposition 5 illustrates why ignoring this dependence can produce a biased gradient, providing useful justification for differentiating through the regression state rather than treating it as fixed.

Critical weaknesses
  1. The theoretical contribution relative to closely related bandit meta-learning methods needs clearer positioning. Section 6 of Kveton et al., Meta-Learning Bandit Policies by Gradient Ascent, already considers learning feature transformations and differentiating their effects on the regression state. The present paper differs by introducing distinct leader and follower objectives and a common regret guarantee over admissible interfaces. The paper should explain more explicitly which technical difficulties they introduce and which results require arguments beyond a direct adaptation of existing analyses.

  2. The correspondence between the experimental algorithm and the analyzed algorithm is unclear. Algorithm 1 and Theorem 4 use gradient averaging and a damped update, whereas Appendix L describes a “direct projected-ascent update” with η=20\eta=20. It is unclear whether the experiments implement all components of equation (8) or a simpler direct projection. Moreover, the exact settings of λ,βt,τt\lambda,\beta_t,\tau_t, the initial interface, and the averaging and damping schedules are not sufficiently specified. These details are important for reproducibility and for determining whether the experiments satisfy the conditions of the theoretical guarantees.

  3. The experiments do not sufficiently isolate the benefits of the proposed method. Comparison with Identity demonstrates the potential value of interface learning, but does not establish the importance of coordinatewise scaling or differentiation through the regression state. The full-information Oracle is useful as a reference, but does not replace comparisons with methods operating under the same interface and feedback constraints. Similarly, CILOT-ZERO examines the effect of baseline subtraction rather than the importance of differentiating through the regression state. Comparisons with scalar scaling and a stopped-state gradient would help clarify these contributions. In addition, follower regret increases in the synthetic experiment. Although this does not contradict the regret theorem, “competence-preserving” should be clearly distinguished from preserving performance relative to Identity.

  4. The treatment of a gradient-related term in the simplified convergence bound needs clarification. Theorem 4 includes a Gγ,H2/MG_{\gamma,H}^2/\sqrt M term. Using the stated bound Gγ,H2=O~(d)G_{\gamma,H}^2=\widetilde O(d) appears to leave an additional O~(d/M)\widetilde O(d/\sqrt M) contribution, whereas Appendix I presents the simplified bound as O~(dΔ1/M+d/(BM))\widetilde O(d\Delta_1/\sqrt M+d/(B\sqrt M)). It is unclear how the former term is absorbed or otherwise controlled. This concern does not necessarily affect the convergence rate in MM, but it matters when the dependence on batch size BB is explicitly retained.

Questions for authors
  1. How does CILOT differ technically from the feature-transformation approach in Section 6 of Kveton et al., Meta-Learning Bandit Policies by Gradient Ascent? In particular, which new technical challenges and guarantees arise from the distinct leader and follower objectives and the uniform guarantee over admissible interfaces? Which parts of the analysis follow by adapting existing results?

  2. Do the experiments implement all averaging and damping steps in equation (8), or do they use direct projected ascent without these components? Please specify the follower parameters, initial interface, and outer-update schedules. Do the experimental settings satisfy the common calibration conditions in Theorem 2?

  3. How does CILOT compare with a learned scalar interface v=c1v=c\mathbf 1 and a stopped-state gradient variant under the same feedback budget and follower settings? Could leader utility also be decomposed into follower reward and platform value to clarify the source of the utility improvement and the increase in follower regret?

  4. How is the Gγ,H2/MG_{\gamma,H}^2/\sqrt M term from Theorem 4 handled in Appendix I? Is the resulting d/Md/\sqrt M contribution absorbed into a constant that depends on BB, or is it controlled by an additional argument? Please clarify the simplified bound when the dependence on BB is kept explicit.

Reviewer_78NPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper considers repeated deployments of a fixed Soft-LinUCB learner. An upstream leader chooses a positive diagonal feature rescaling before each deployment and optimizes a possibly different discounted utility. The rescaling preserves reward realizability while changing the follower's finite-sample behavior. CILOT estimates a trajectory likelihood-ratio gradient through the follower's regression state and applies averaged, damped projected updates. The analysis gives a common follower-regret bound over the interface class, sensitivity and variance bounds, and convergence to projected stationarity. My assessment focuses on these theoretical contributions.

Critical strengths

The operational restriction is understandable: changing an upstream feature interface can affect a fixed downstream learner even when all reward means remain unchanged. Proposition 1 gives a useful exact representation, and the analysis carefully distinguishes derivatives along a fixed history from derivatives of its probability. The Fisher-based bounds also correctly handle utility multipliers correlated with trajectory scores, rather than incorrectly canceling weighted cross terms.

Critical weaknesses
  1. The intervention and gradient reduce to a narrow instance of existing bandit-policy meta-optimization. Proposition 1 shows that feature scaling is exactly equivalent, along every realized history, to choosing the diagonal ridge matrix

    Λ(v)=λdiag⁡(v1−2,…,vd−2).\Lambda(v)=\lambda\operatorname{diag}(v_1^{-2},\ldots,v_d^{-2}).

    The conditional policy is therefore an ordinary history-dependent policy parameterized by a bounded diagonal regularizer. Theorem 1 is the return-to-go likelihood-ratio identity for that policy, with its conditional action probabilities differentiated correctly. Boutilier et al. (2020, Theorem 2 and Section 5.3) already optimize history-dependent bandit policies using this identity, including recurrent policies whose internal states depend on the optimized parameters. A different upstream utility changes the return multiplier, while positive coordinate scaling restricts the policy class. Those are legitimate modeling choices, but I do not see a new gradient principle or a new obstacle resolved by calling the parameter a feature interface. The comparison to stopped-state differentiation shows that an incomplete derivative is biased; it does not establish an advance over correctly differentiated policy-gradient methods.

  2. The remaining guarantees do not establish a substantial additional theoretical advance. The follower guarantee follows from the comparator norm bound ∥ψθ,v∗∥≤S/α\|\psi^*_{\theta,v}\|\le S/\alpha, standard linear-bandit confidence analysis, and the softmax loss max⁡aqt(a)−Ea∼πtqt(a)≤τtlog⁡K\max_a q_t(a)-\mathbb E_{a\sim\pi_t}q_t(a)\le\tau_t\log K. Appendix C explicitly obtains the same deterministic expected-regret bound for each fixed interface and then takes the supremum. This is a useful common calibration, but requires no new uniform concentration argument. The leader guarantee combines explicit smoothness/variance bounds with a projected stochastic-optimization argument, and controls stationarity of the selected interface objective. The raw-coordinate sensitivity calculation and discount-weighted bounds are the main technical additions I can identify. Although carefully worked out, I find their significance too limited to support acceptance as a learning-theory contribution. The common regret envelope also permits worse follower regret on particular instances, as the paper itself correctly acknowledges in Proposition 3; it is a limited form of competence preservation.

References

Boutilier et al. (2020), Differentiable Meta-Learning of Bandit Policies, especially Theorem 2 and Section 5. The relevant connection is the history-dependent policy-gradient mechanism; the original task and policy classes differ.

Questions for authors
  1. Under the exact reparameterization by Λ(v)\Lambda(v), what theoretical capability does CILOT add to existing differentiable bandit-policy optimization using the full conditional policy derivative? Please make the comparison at the level of assumptions and guarantees.
  2. Which result should be regarded as the principal technical advance beyond this specialization: the sensitivity bounds, the discount-dependent variance analysis, or the outer optimization result? What materially stronger conclusion does it enable relative to applying existing tools to this same policy class?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.