acceptodds
Under review as a conference paper at ICLR 2027

COMAS: Co-Optimizing Model and Policy in Stochastic Contextual Bandits

Abstract

The performance of contextual bandit algorithms critically depends on the model configuration used to estimate the reward function, including its family, architecture, and regularization. However, selecting an effective configuration requires data collected by the action-selection policy, which itself depends on the reward estimator, creating a circular dependency between model configuration and policy learning. We introduce COMAS, a framework that co-optimizes model configuration and action selection by treating configuration as a sequential decision variable. At each episode, COMAS selects a model configuration, trains the corresponding reward estimator on historical interaction data, and uses a Gaussian process surrogate to model the episode-level reward as a function of the configuration, thereby guiding subsequent configuration selection. Under suitable assumptions, we show that COMAS achieves a sub-linear regret upper bound that separately accounts for regret due to configuration selection and policy learning. Experiments on synthetic and real-world contextual bandit problems demonstrate the effectiveness of co-optimizing heterogeneous model configurations and action-selection policies.

Primary area:
learning theory
Keywords:
Contextual Bandits, Model Selection, Bayesian Optimization, Co-optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gxtNOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces COMAS, a framework that jointly optimizes model configuration and action selection in contextual bandits by treating model configuration as a sequential decision variable and using a Gaussian process to guide configuration selection. The authors provide a sublinear regret analysis under some additional assumptions and demonstrate that COMAS effectively selects heterogeneous model configurations on synthetic and real datasets.

Critical strengths
  1. This paper studies an interesting and practical problem: how to jointly select the optimal reward model and configuration at the same time.
  2. The presentation of this work is mostly clear.
  3. This paper provides both theoreical analysis and a suite of empirical results on synthetic and real datasets.
Critical weaknesses
  1. I may overlook but it seems that there is a discrepancy between the described algorithm and its implementation: the main text describes sampling from the joint GP posterior while the appendix samples each dimension independently from their marginal distritbuion.
  2. I feel the experiments do not fully disentable the benefit of the GP meta controller proposed in this work from the benefit of shared interaction data. I think COMAS selects configuration using the entire shared Dt refit while the old method such as CORRAL only uses the selected rounds. Based on the no-share ablation, it seems the data sharing has a substantial impact. Therefore, I think a matched comparison shuold be conducted as this work is practically driven.
  3. Assumption 1 is also a bit strong. It would be great if the authors could give more explanation why this is required.
Questions for authors
  1. Can the authors clarify the discrepancy between the joint GP posterior sampling described in the main text and the independent marginal sampling used in the experiments?
  2. Can the authors provide some more ablation that separates the effect of shared interaction data from the GP meta-controller?
  3. How restrictive is the assumption 1 in practice?
4 comments
AuthorsOpenReview ↗

Response [1/2]

We thank the reviewer for the careful and constructive review. We appreciate the reviewer’s recognition of the importance and practical relevance of the problem, the clarity of the presentation, and the combination of theoretical analysis with extensive experiments on synthetic and real-world datasets. We have carefully addressed each concern and question below to further strengthen the paper and clarify our contributions.

I may overlook but it seems that there is a discrepancy between the described algorithm and its implementation: the main text describes sampling from the joint GP posterior while the appendix samples each dimension independently from their marginal distritbuion.

Thank you for identifying this inconsistency. In finite-pool COMAS (TS), we independently sample from each configuration's GP posterior marginal and select the configuration with the largest sampled value. These marginals are obtained from the full-covariance GP, preserving kernel-based information transfer across configurations; only the correlation among acquisition draws is removed. By contrast, the mixed-continuous implementation uses joint posterior sampling within the fitted local candidate sets.

We have revised Eq. (3) and Appendix C.1 to clearly distinguish these two procedures. Theorem 1 and the COMAS (UCB) results remain unaffected, as UCB relies on posterior means and standard deviations rather than posterior sampling.

I feel the experiments do not fully disentable the benefit of the GP meta controller proposed in this work from the benefit of shared interaction data. I think COMAS selects configuration using the entire shared Dt refit while the old method such as CORRAL only uses the selected rounds. Based on the no-share ablation, it seems the data sharing has a substantial impact. Therefore, I think a matched comparison shuold be conducted as this work is practically driven.

We agree that shared-data access explains part of the original performance gap: the Smooth-CORRAL/COMAS (TS) median-regret ratios decrease from 12.7× to 7.3× on Cosine and from 7.2× to 2.5× on MovieLens.

To further disentangle the contributions of shared-data access and cross-configuration covariance modeling, we conducted two complementary experiments on the Cosine and MovieLens datasets, using 10 seeds and 10,000 rounds per method.

Matched-data comparison. We modified Smooth-CORRAL and EXP3.P to refit their selected base learners using all observations collected within each run, matching COMAS's data access. All other experimental settings, including environments, configuration pools, base learners, episode scheduler, horizons, and environment seeds, were held fixed. The resulting median final cumulative regrets are:

Final cumulative regret, medianCosineMovieLens
COMAS (TS), full meta-GP116.2221.9
Shared-data Smooth-CORRAL853.3552.2
Shared-data EXP3.P982.7623.8

COMAS (TS) retains 7.3× and 2.5× median-regret advantages over shared-data Smooth-CORRAL on Cosine and MovieLens, respectively. It also achieves lower regret than each shared-data baseline on 10/10 Cosine seeds and 9/10 MovieLens seeds. Thus, unequal data access accounts for only part of the observed advantage.

Covariance ablation. We further compared learned full and diagonal meta-GPs within COMAS (TS), keeping base-level data sharing and the independent-marginal acquisition rule fixed while removing cross-configuration covariance. The two variants perform comparably on Cosine (mean final regret: 120.9 vs. 119.2; median: 116.2 vs. 118.4). On MovieLens, the full GP achieves lower mean (313.1 vs. 328.9) and median (221.9 vs. 321.7) regret, winning on 8/10 paired seeds. These results provide suggestive evidence that cross-configuration covariance is beneficial on MovieLens, although the effect is less pronounced on Cosine. Notably, even the diagonal variant retains median-regret advantages of 7.2× on Cosine and 1.7× on MovieLens over shared-data Smooth-CORRAL.

Finally, our original no-sharing ablation provides separate evidence for the benefit of shared training data: removing data sharing increases COMAS (UCB)'s median Cosine regret from 148.9 to 229.4, with higher regret on all 10 paired seeds.

Together, these experiments clarify the complementary contributions of shared-data access and covariance modeling. Full details are provided in Appendices B.6.1–B.6.3.

↓ ↓ ↓ Continued below ↓ ↓ ↓

AuthorsOpenReview ↗

Response [2/2]

Assumption 1 is also a bit strong. It would be great if the authors could give more explanation why this is required.

Assumption A1 controls the learning-induced drift introduced by shared-data refitting. Specifically, on a common high-probability event, it bounds the deviation of the episode-averaged pre-round conditional expected reward along deployed episodes from a deterministic reference g(m)g(m) by ∣bq∣≤Cd(τq+1)−α|b_q|\le C_d(\tau_q+1)^{-\alpha}. Without controlling this drift, learning-induced bias may persist, and the stated GP confidence and configuration-selection regret bounds are not guaranteed.

Assumption A1 is used in the proof at two places. First, it bounds the cumulative drift in the regret decomposition (Lemmas 2 and 5). Second, it controls the drift contribution CbD2α(T)C_b\sqrt{D_{2\alpha}(T)} to the GP confidence radius (Lemma 9). Thus, the analysis explicitly accounts for retraining-induced non-stationarity rather than treating the meta-level feedback as stationary. Moreover, Appendix B.5 shows that sublinear configuration-selection overhead remains possible under the slower-decay condition α>β/2\alpha>\beta/2 for a fixed finite pool and fixed regularity constants, provided the remaining assumptions hold. We discuss practical sufficient conditions and empirical diagnostics for Assumption A1 in our response to the final question below.

Can the authors clarify the discrepancy between the joint GP posterior sampling described in the main text and the independent marginal sampling used in the experiments?

Please see our response to the sampling discrepancy above. The revised Eq. (3) and Appendix B.1 now clearly distinguish the two procedures: independent posterior-marginal sampling for the finite-pool implementation and joint posterior sampling within fitted local candidate sets for the mixed-continuous implementation. Theorem 1 and the COMAS (UCB) results remain unaffected.

Can the authors provide some more ablation that separates the effect of shared interaction data from the GP meta-controller?

Both comparisons are reported above and address complementary aspects of COMAS. The shared-data baselines evaluate meta-selection under matched data-access conditions, showing that COMAS's advantage cannot be explained solely by unequal data access. The full-versus-diagonal ablation retains shared training data while removing GP covariance transfer, providing suggestive evidence of an additional covariance benefit on MovieLens. Full results are reported in Appendices B.6.2–B.6.3.

How restrictive is the assumption 1 in practice?

Sufficient conditions. Assumption A1 does not require a correctly specified reward model or an optimal limiting policy. Proposition 3 in Appendix A.8 establishes a general sufficient condition: uniform convergence of scores to deterministic limits at rate O(t−r)O(t^{-r}), together with a margin condition of exponent κ\kappa, yields a drift-decay exponent α=rκ\alpha=r\kappa. Proposition 4 establishes A1 for well-specified ridge learners under persistent excitation and a vanishing exploration bonus. Proposition 5 extends the result to misspecified ridge learners under deterministic design and cross-moment convergence, with a vanishing bonus. Under the stated O~(t−1/2)\tilde O(t^{-1/2}) score-convergence rates and the margin condition, both ridge results yield α=κ/2\alpha=\kappa/2, up to logarithmic factors.

Finite-horizon evidence. Under COMAS (UCB), frequently deployed Polynomial configurations exhibit small median late-stage deviations in policy value from their run-specific reference values: 2.46×10−42.46\times10^{-4} on Cosine and 9.21×10−79.21\times10^{-7} on Square. These configurations account for 78.8% and 88.0% of the analyzed post-warm-up episodes, respectively. Notably, the Polynomial learners are misspecified on Cosine.

These results provide finite-horizon evidence of stabilization among frequently deployed configurations, complementing the sufficient conditions in Appendix A.8. However, the diagnostics use episode-start policy values and run-specific late-stage references; they do not establish A1 over the full horizon. Details are provided in Appendix B.6.4.

We thank the reviewer again for the careful review, which will help us improve the paper. We have corrected the sampling description, added matched-data comparisons, and clarified the role and sufficient conditions of Assumption A1. We would be happy to provide any further clarification or results during the discussion period. We hope these additions address your concerns and strengthen your view of our work.

Thank you for your responses. Most of my concerns have been resolved, but I am still a bit conservative about A1’s applicability to the empirical setting. I think a more systematic study of when A1 holds under adaptive shared-data refitting would help clarify the practical scope of the theory. I tend to keep my current score while I am fine with the current work with its rebuttal if it is accepted.

Thank you for your constructive suggestion. To better assess the practical applicability of Assumption A1, we extended our empirical study of adaptive shared-data refitting to two environments, Cosine and Square, and three model families, Linear, Polynomial, and Neural.

On the theoretical side, Appendix B.8 already establishes sufficient conditions for A1. Specifically, high-probability uniform convergence of fitted action scores, together with a margin condition, yields a polynomial bound on expected-reward drift under shared-data refitting.

For each configuration, we used the recorded expected-reward values to define an empirical reference level, g^(m)\widehat g(m), and examined deviations from it. We report both the median and upper quantiles to characterize variation beyond typical behavior. The table also reports empirical polynomial envelopes of the form C(τq+1)−αC(\tau_q+1)^{-\alpha}, with α=0.30\alpha=0.30 selected after inspecting the observed deviations. The exponent alpha=0.30alpha=0.30 lies within the general vanishing-drift regime analyzed in Appendix B.5.

Settingg^(m)\widehat g(m) rangeα\alphaCCDeviation: medianDeviation: q75Deviation: q90
Cosine / Linear0.4890–0.54150.302.000.0151700.0263500.088666
Cosine / Polynomial0.3775–0.99100.300.500.001092520.0057850.0237379
Cosine / Neural0.6580–0.80050.300.800.006250390.0207950.0588186‾\underline{0.0588186}
Square / Linear0.077–0.1020.300.500.004874040.0069100.0104606
Square / Polynomial0.4156–0.41590.300.350.000003668210.0000400.000225669
Square / Neural0.102–0.3560.300.220.004242640.0104000.0254938

The results show small median deviations across all six settings, particularly for Polynomial configurations, including the misspecified Polynomial learners on Cosine. The upper quantiles also reveal greater variability in the Linear and Neural configurations. This broader assessment provides empirical support for stabilization under adaptive shared-data refitting while quantifying how its strength varies across environments and model families. Across all six settings, run-level coverage of the calibrated drift envelopes reaches at least 80%, with 100% coverage in both Polynomial settings, providing direct empirical evidence for Assumption A1's simultaneous drift condition.

Together with the sufficient conditions established in Appendix B.8, these findings clarify the theoretical basis and empirical plausibility of Assumption A1 under adaptive shared-data refitting.

Reviewer_oH4XOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a combined framework of "heterogeneous configuration selection + shared data training + GP guidance."
Under the assumption that configuration performance gradually stabilizes, provide regret guarantees for a finite configuration pool and UCB version, while allowing relearning of GP hyperparameters. In addition, this paper provides sufficient conditions for certain ridge learners to satisfy the stability assumption. The novelty lies mainly in the specific combinations and the corresponding theoretical analysis. The experiment can verify most of the authors’ claims.

Critical strengths

S1: The research question is of practical significance. Model configuration affects action selection, and actions in turn influence training data. The paper focuses on this feedback relationship and addresses the practical difficulty in online decision-making of determining an appropriate model in advance.

S2: The method design has good integration. By combining heterogeneous configuration selection, shared data training, and GP-guided exploration, different configurations can reuse interaction experience and utilize the correlations between configurations to assist in selection.

S3: It provides further support for key stability assumptions. The authors not only propose the assumption that configuration performance gradually stabilizes but also provide sufficient conditions for some ridge learners, establishing a connection between theory and specific learning algorithms.

S4: The experiments cover a wide range and provide mechanism-level verification. In six synthetic and real data-derived environments, different model configurations are examined, reporting results from multiple random seeds, and using shared data to conduct ablation tests for one of the core designs. The results support the effectiveness of the entire method in the tested settings.

Critical weaknesses

W1: The novelty of the method has not been fully established. There are existing studies on online model selection, double-layer parameter tuning, and GP-guided search. The paper proposes a meaningful combination, but does not adequately explain: compared to the closest method, what new capabilities cannot be directly obtained by this combination.

W2: It is difficult to separate the independent benefits of the configuration selection mechanism in experiments. COMAS uses shared interaction data, while the main baseline only uses the data collected by each configuration itself. Therefore, the performance advantage is simultaneously influenced by the selection mechanism and the way training data is utilized. The existing shared data ablation only covers Cosine and is not sufficient to prove the benefits of GP association modeling alone.

W3: The theoretical guarantee and experimental implementation have a distance. The main theorem is for the UCB version of a finite configuration pool and requires theoretical exploration coefficients; the experiments use a fixed coefficient of 1.5. TS and continuous space experiments cannot directly obtain the same guarantee. In addition, the joint GP sampling described in the main text is inconsistent with the independent edge sampling actually used in the appendix, and clarification is needed.

Questions for authors

Q1: The proposed framework combines several established techniques, including online model selection, hyperparameter optimization, and Gaussian process-based exploration. How does COMAS differ fundamentally from existing approaches such as CORRAL, Syndicated Bandits, and CDT? In particular, which aspects of the framework represent new methodological contributions, and what specific shortcomings of previous methods do they overcome?

Q2: The experimental comparison appears to involve different training data conditions. COMAS allows its candidate models to learn from a common interaction history, while the competing methods maintain separate datasets for individual configurations. To what extent can the observed improvements be attributed to this difference rather than the proposed selection strategy? Additional comparisons under comparable data-sharing conditions, together with an ablation removing cross-configuration covariance from the GP, would help establish the contribution of each component.

Q3: There appears to be a discrepancy between the theoretical analysis and the experimental implementation regarding the exploration parameter. While the regret analysis relies on Assumption 4, the experiments adopt a constant value of αq=1.5\alpha_q=1.5. What justifies this choice, and can it be shown to satisfy the required theoretical conditions? It would also be useful to examine how the algorithm's performance changes under different exploration coefficients, particularly to determine whether the empirical results depend strongly on this parameter.

5 comments
AuthorsOpenReview ↗

Response [1/3]

We thank the reviewer for the careful review and for recognizing the practical significance of the problem. We also appreciate the reviewer’s recognition of the method’s cohesive design, which enables configurations to share interaction experience while leveraging cross-configuration correlations to inform selection. We are grateful for the positive assessment of our theoretical analysis and extensive experiments. To further clarify the paper and address the reviewer’s concerns, we respond to each weakness and question below:

W1: The novelty of the method has not been fully established. There are existing studies on online model selection, double-layer parameter tuning, and GP-guided search. The paper proposes a meaningful combination, but does not adequately explain: compared to the closest method, what new capabilities cannot be directly obtained by this combination.

COMAS jointly selects reward-model configurations and their induced action-selection policies under shared-data retraining. Its selection space spans learner families, architectures, regularization, and exploration parameters. Unlike conventional hyperparameter optimization within a fixed learner family, COMAS refits selected configurations on a common interaction history and uses an episodic GP surrogate to transfer information across heterogeneous configurations while accounting for episode-length-dependent feedback noise. This enables joint selection across both model and policy dimensions.

The theoretical contribution addresses the resulting non-stationary meta-level feedback. Our regret analysis explicitly accounts for learning-induced drift, while the GP confidence analysis accommodates predictably relearned kernels and bounded scalar-likelihood noise via a fixed reference-kernel comparison. Consequently, standard stationary GP-UCB analysis cannot be directly applied to establish our guarantee. We provide a method-by-method comparison and technical details in our response to your Q1.

W2: It is difficult to separate the independent benefits of the configuration selection mechanism in experiments. COMAS uses shared interaction data, while the main baseline only uses the data collected by each configuration itself. Therefore, the performance advantage is simultaneously influenced by the selection mechanism and the way training data is utilized. The existing shared data ablation only covers Cosine and is not sufficient to prove the benefits of GP association modeling alone.

To isolate the effects of configuration selection and data sharing, we conducted additional experiments on Cosine and MovieLens, giving EXP3.P and Smooth-CORRAL access to the same shared interaction data as COMAS. All methods use 10 seeds and 10,000 rounds, with identical environments, configuration pools, base learners, episode schedulers, horizons, and environment seeds.

Results. Matching data access narrows the performance gap, indicating that data sharing accounts for part of COMAS's original advantage. Specifically, the Smooth-CORRAL/COMAS (TS) median-regret ratio decreases from 12.7× to 7.3× on Cosine and from 7.2× to 2.5× on MovieLens. However, COMAS (TS) still achieves lower final regret on 10/10 Cosine seeds and 9/10 MovieLens seeds.

Final cumulative regret, medianCosineMovieLens
COMAS (TS), full meta-GP116.2221.9
Shared-data Smooth-CORRAL853.3552.2
Shared-data EXP3.P982.7623.8

The results demonstrate that unequal data access contributes to the original performance gap but does not fully explain COMAS's overall performance advantage.

**Additional ablations. **

  • Full vs. diagonal meta-GP. We further compared the full and diagonal meta-GP variants within COMAS (TS), keeping shared base-level data and the same independent-marginal acquisition rule. On Cosine, the two variants perform similarly (mean final regret: 120.9 vs. 119.2; median: 116.2 vs. 118.4). On MovieLens, the full GP achieves lower mean (313.1 vs. 328.9) and median (221.9 vs. 321.7) regret, outperforming the diagonal GP on 8/10 paired seeds. These results provide evidence that modeling cross-configuration covariance can improve performance, particularly on the MovieLens dataset. Importantly, even without covariance modeling, the diagonal variant retains median-regret advantages of 7.2× on Cosine and 1.7× on MovieLens over shared-data Smooth-CORRAL.
  • Shared-data ablation. Our original ablation provides complementary evidence of the benefit of shared training data. Removing data sharing from COMAS (UCB) increases median Cosine regret from 148.9 to 229.4, with higher regret on all 10 paired seeds (Appendix B.6.1).

The new experiments and ablation results are reported in Appendices B.6.2–B.6.3.

↓ ↓ ↓ Continued below ↓ ↓ ↓

AuthorsOpenReview ↗

Response [2/3]

W3: The theoretical guarantee and experimental implementation have a distance. The main theorem is for the UCB version of a finite configuration pool and requires theoretical exploration coefficients; the experiments use a fixed coefficient of 1.5. TS and continuous space experiments cannot directly obtain the same guarantee. In addition, the joint GP sampling described in the main text is inconsistent with the independent edge sampling actually used in the appendix, and clarification is needed.

Theorem 1 provides a conditional guarantee for finite-pool COMAS (UCB) under Assumptions 1–4. The TS and mixed-continuous variants are evaluated empirically; the theorem is learner-class agnostic provided its assumptions hold. The fixed exploration coefficient and sensitivity study are addressed in our response to your Q3.

We also corrected the description of the sampling procedures. Finite-pool COMAS (TS) independently samples from each configuration's posterior marginal, whereas mixed-continuous TS samples jointly within fitted local candidate sets. Importantly, GP inference in the finite pool still uses the full kernel, preserving cross-configuration information transfer; only the correlation between acquisition draws is removed. The revised Eq. (3) and Appendix C.1 clarify these distinctions. Theorem 1 and the COMAS (UCB) results remain unchanged, as UCB relies on posterior means and standard deviations rather than posterior sampling.

Q1: The proposed framework combines several established techniques, including online model selection, hyperparameter optimization, and Gaussian process-based exploration. How does COMAS differ fundamentally from existing approaches such as CORRAL, Syndicated Bandits, and CDT? In particular, which aspects of the framework represent new methodological contributions, and what specific shortcomings of previous methods do they overcome?

The key contribution is heterogeneous, shared-data configuration selection, together with a theoretical analysis of the non-stationary feedback it induces. The relevant distinctions from existing methods are as follows:

MethodSelection scopeBase feedback and training
CORRALFinite collection of base algorithmsImportance-weighted feedback passed to the bases
Syndicated BanditsHyperparameter grids within a specified learner familyOne base learner updated online
CDTContinuous tuning for generalized linear banditsZooming TS with restarts and an online GLB learner
COMASLearner families, architectures, regularization, and exploration parametersSelected estimator refit on common interaction data, with episodic GP selection

The revised related work section distinguishes the feedback mechanisms of the published frameworks from those of the shared-data baseline variants evaluated in this work.

Shared-data refitting mitigates data starvation among infrequently deployed configurations (Appendix B.6.1), while heterogeneous selection avoids committing to a single learner family when no family performs best across all tested environments. The empirical mixed-continuous backend further avoids restricting selection to a fixed finite hyperparameter grid.

COMAS's shared-data lifecycle introduces two technical challenges, which motivate our theoretical contributions:

  1. Non-stationary meta-feedback. Lemma 2 decomposes regret into calibration, pool mismatch, configuration-selection regret, retraining-induced drift, and noise. Lemma 9 incorporates this drift into the GP confidence radius. Appendix A.8 establishes sufficient stability conditions for ridge learners, including misspecified settings.
  2. Adaptive GP fitting. Relearning the kernel and bounded scalar likelihood noise invalidates a direct application of the usual fixed-kernel determinant argument. We address this by comparing learned posteriors with a fixed reference kernel (Lemmas 6, 8, and 10) and establishing uniform noise concentration over the likelihood-noise interval (Lemma 7).

Our UCB analysis also identifies conditions under which exploiting covariance structure improves the pool-size dependence of the regret upper bound relative to a diagonal reference. Specifically, under the stated UCB assumptions, for growing finite pools with pool-uniform regularity constants and α≥1/2\alpha\geq 1/2, this improvement holds when ΓTref=o(K)\Gamma_T^{ref}=o(\sqrt K) (Sec. 4; Appendices A.5–A.6).

We build on established GP-UCB and posterior-sampling acquisition rules rather than claim new acquisition algorithms. Our contributions lie in the heterogeneous shared-data selection framework, and the theoretical analysis of the coupled, non-stationary feedback, and adaptive GP fitting it requires.

↓ ↓ ↓ Continued below ↓ ↓ ↓

AuthorsOpenReview ↗

Response [3/3]

Q2: The experimental comparison appears to involve different training data conditions. COMAS allows its candidate models to learn from a common interaction history, while the competing methods maintain separate datasets for individual configurations. To what extent can the observed improvements be attributed to this difference rather than the proposed selection strategy? Additional comparisons under comparable data-sharing conditions, together with an ablation removing cross-configuration covariance from the GP, would help establish the contribution of each component.

The matched comparisons in our response to W2 show that shared-data access explains part of the original performance gap: the Smooth-CORRAL/COMAS (TS) median-regret ratios decrease from 12.7× to 7.3× on Cosine and from 7.2× to 2.5× on MovieLens. Nevertheless, COMAS retains an advantage when both methods refit on their respective shared interaction histories.

The full-versus-diagonal ablation provides a complementary comparison. By keeping base-level data sharing and the acquisition rule fixed while removing cross-configuration covariance, it isolates the contribution of covariance modeling. The results suggest a benefit on MovieLens, while the two variants perform comparably on Cosine. Both comparisons are reported in Appendices B.6.2–B.6.3.

Q3: There appears to be a discrepancy between the theoretical analysis and the experimental implementation regarding the exploration parameter. While the regret analysis relies on Assumption 4, the experiments adopt a constant value of (\alpha_q=1.5). What justifies this choice, and can it be shown to satisfy the required theoretical conditions? It would also be useful to examine how the algorithm's performance changes under different exploration coefficients, particularly to determine whether the empirical results depend strongly on this parameter.

Assumption 4 provides a sufficient condition for GP-UCB optimism while controlling exploration cost. The coefficients used in our experiments were chosen empirically to achieve competitive regret, rather than calibrated to satisfy the theoretical confidence requirement; we have not established that these choices satisfy A4.

To assess sensitivity to this choice, we conducted a seven-coefficient sweep on Cosine and Square environments, using 10 seeds and 10,000 rounds per setting. We varied beta_W ∈0.25,0.5,1,1.5,2.5,4,6\in{0.25,0.5,1,1.5,2.5,4,6}, with the acquisition multiplier set to αq=\alpha_q=beta_W$^{1/2}$. The original settings were beta_W=1.5 for Cosine and beta_W=2.5 for Square. The results identify alternative settings that substantially reduce across-seed variability while maintaining competitive median regret. Full results for all seven settings are provided in the revised Appendix B.6.5, and Appendix B.1 clarifies the parameterization.

We thank the reviewer again for the careful review, which will help us improve the paper. We would be happy to provide any further clarification or results during the discussion period. We hope these additions address your concerns and strengthen your view of our work.

Reviewer_7X9TOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper addresses online selection of the reward-model configuration (learner family, architecture, regularization, exploration scale) in contextual bandits. COMAS selects a configuration at the start of each episode using a Gaussian-process surrogate over configurations, refits the chosen learner on a dataset shared by all configurations, and feeds the episode-average reward back to the surrogate. Under stability and GP regularity assumptions, the paper proves a sublinear regret bound, and experiments on synthetic, classification, and recommendation environments show lower regret than online model-selection baselines.

Critical strengths
  1. The paper shows convincingly that configuration choice matters a great deal: committing to a random configuration yields 2.3–455× the regret of the best one, and no learner family is best across environments.

  2. Refitting every configuration on the shared interaction data avoids the data starvation that cripples per-configuration model selection, and an ablation confirms this (median regret 149 vs. 229, worse in 10 of 10 seeds).

Critical weaknesses
  1. Only COMAS refits configurations on shared data; the baselines (EXP3.P, Smooth-CORRAL) refit each configuration only on its own rounds. The paper’s own ablation shows that shared data alone accounts for a large part of the gap. As a result, the headline improvement over the baselines (3.3–35× lower regret) cannot be attributed to the GP-based selection, which is the paper’s distinguishing component. Baselines with shared-data refitting, and a comparison against an independent (non-GP) meta-controller, are needed.

  2. The analysis assumes that each configuration’s performance converges to a fixed limit despite repeated refitting on data collected by other configurations. This closed-loop stability is precisely the coupling the paper motivates as its key difficulty, and it is verified only for well-specified ridge learners. End-to-end sublinear regret also requires that some configuration be well specified, which contradicts the claim that COMAS does not need one. In addition, the experiments use a fixed exploration coefficient, Thompson sampling, and neural learners, none of which are covered by the theorem.

Questions for authors

Please see weakness.

2 comments
AuthorsOpenReview ↗

Response [1/2]

We thank the reviewer for the careful review and appreciate the recognition that configuration choice has a substantial impact on performance and that no single learner family performs best across all environments. We also appreciate the reviewer’s acknowledgment that refitting every configuration on shared interaction data mitigates the data starvation that can undermine per-configuration model selection. We have addressed each of the reviewer’s concerns as follows:

Only COMAS refits configurations on shared data; the baselines (EXP3.P, Smooth-CORRAL) refit each configuration only on its own rounds. The paper's own ablation shows that shared data alone accounts for a large part of the gap. As a result, the headline improvement over the baselines (3.3–35× lower regret) cannot be attributed to the GP-based selection, which is the paper's distinguishing component. Baselines with shared-data refitting, and a comparison against an independent (non-GP) meta-controller, are needed.

We agree that COMAS refits every configuration using the shared interaction data DtD_t, whereas EXP3.P and Smooth-CORRAL, by design, refit each configuration only on data from their own rounds. To isolate the impact of this difference, we conducted additional experiments on Cosine and MovieLens, evaluating EXP3.P and Smooth-CORRAL with access to the same interaction data as COMAS. All methods use 10 seeds and 10,000 rounds, with the same environments, configuration pools, base learners, episode scheduler, horizons, and environment seeds.

Results. Shared-data access explains part of the original performance gap: the Smooth-CORRAL/COMAS (TS) median-regret ratio decreases from 12.7× to 7.3× on Cosine and from 7.2× to 2.5× on MovieLens. Nevertheless, COMAS (TS) achieves lower final regret than the shared-data baselines on 10/10 Cosine seeds and 9/10 MovieLens seeds.

Final cumulative regret, medianCosineMovieLens
COMAS (TS), full meta-GP116.2221.9
Shared-data Smooth-CORRAL853.3552.2
Shared-data EXP3.P982.7623.8

These shared-data baselines provide the requested non-GP meta-controller comparisons under a matched data-access protocol. The results show that unequal data access contributes to the original gap but does not, by itself, explain COMAS's overall performance advantage.

Additional ablation.

  • Full vs. diagonal meta-GP. We further compared the learned full and diagonal meta-GPs within COMAS (TS), keeping shared base-level data and the same independent-marginal acquisition rule. The two variants perform similarly on Cosine (mean final regret: 120.9 vs. 119.2; median: 116.2 vs. 118.4). On MovieLens, however, the full GP achieves lower mean (313.1 vs. 328.9) and median (221.9 vs. 321.7) regret, outperforming the diagonal GP on 8/10 paired seeds. This provides suggestive evidence that modeling cross-configuration covariance is beneficial on MovieLens. Notably, even without this covariance modeling, the diagonal variant retains median-regret advantages of 7.2× on Cosine and 1.7× on MovieLens over shared-data Smooth-CORRAL.

  • Shared-data ablation. Our original ablation provides complementary evidence for the importance of shared training data: removing data sharing from COMAS (UCB) increases median Cosine regret from 148.9 to 229.4, with higher regret on all 10 paired seeds (Appendix B.6.1).

The new results are reported in Appendices B.6.2–B.6.3.

↓ ↓ ↓ Continued below ↓ ↓ ↓

AuthorsOpenReview ↗

Response [2/2]

The analysis assumes that each configuration's performance converges to a fixed limit despite repeated refitting on data collected by other configurations. This closed-loop stability is precisely the coupling the paper motivates as its key difficulty, and it is verified only for well-specified ridge learners. End-to-end sublinear regret also requires that some configuration be well specified, which contradicts the claim that COMAS does not need one. In addition, the experiments use a fixed exploration coefficient, Thompson sampling, and neural learners, none of which are covered by the theorem.

Shared-data stability (A1). We agree that Theorem 1 is conditional on A1. Refitting on shared interaction data makes meta-level performance non-stationary. Over deployed episodes, A1 bounds the learning-induced drift in the episode-averaged pre-round conditional expected reward around a deterministic reference g(m)g(m) as ∣bq∣≤Cd(τq+1)−α|b_q|\le C_d(\tau_q+1)^{-\alpha}. Our analysis explicitly accounts for this drift in both the regret decomposition (Lemmas 2 and 5) and the GP confidence radius (Lemma 9). Appendix A.8 provides sufficient conditions for well-specified ridge under persistent excitation and misspecified ridge under deterministic design- and cross-moment limits, together with the required score-convergence, exploration-bonus, and margin conditions. Furthermore, for a fixed finite pool and fixed regularity constants, Appendix B.5 establishes that configuration-selection overhead can remain sublinear even for slower drift decay, α>β/2\alpha>\beta/2, under the remaining assumptions. Empirically, frequently deployed Polynomial configurations under COMAS (UCB) exhibit small median late-stage policy-value deviations from run-specific references: 2.46×10−42.46\times10^{-4} on Cosine and 9.21×10−79.21\times10^{-7} on Square. These configurations account for 78.8% and 88.0% of analyzed post-warm-up episodes, respectively, and are misspecified on Cosine. While these observations do not establish A1, they provide finite-horizon evidence of stabilization that complements the analytical sufficient conditions (Appendix B.6.4).

Well-specification and regret guarantees. COMAS does not require a well-specified candidate to operate, nor does Theorem 1. The theorem explicitly retains MTpoolM_T^{pool}, separating the regret due to configuration selection from the regret due to the quality of the candidate pool. Corollary 1 establishes end-to-end sublinear regret under i.i.d. contexts, oracle alignment g⋆=μ⋆g^\star=\mu^\star, and the other stated conditions (Appendix A.9). Importantly, the policy-value construction in Appendix A.8 shows that alignment does not require the limiting score to equal the reward function: it is sufficient for the score to preserve the true action ranking and thereby select optimal actions almost surely. Thus, a well-specified configuration with an optimal limiting policy is a sufficient route to alignment but not necessary. We will clarify the distinction between the configuration-selection guarantee of Theorem 1 and the end-to-end guarantee of Corollary 1.

Theoretical and empirical scope. Theorem 1 covers finite-pool COMAS (UCB) under Assumptions 1–4 and is learner-class-agnostic under these conditions. Appendix B.8 provides explicit sufficient-stability conditions for ridge learners. We have not established A1 analytically for the neural configurations used in our experiments. Likewise, the fixed exploration coefficients are empirical choices that yielded competitive regret; we have not established that they satisfy A4. To further characterize their empirical robustness, we added a seven-coefficient sensitivity study (Appendix B.6.5) and corrected the parameterization to αq=\alpha_q=beta_W, with the original settings beta_W=1.5 for Cosine and 2.52.5 for Square. TS and mixed-continuous variants are evaluated empirically, and we will explicitly distinguish these empirical results from the theoretical guarantees in the revision.

We thank the reviewer again for the careful review, which will help us improve the paper. We have added the requested matched-data comparisons and clarified the stability assumptions and scope of the guarantees. We would be happy to provide any further clarification or results during the discussion period. We hope these additions address your concerns and strengthen your view of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.