- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces COMAS, a framework that jointly optimizes model configuration and action selection in contextual bandits by treating model configuration as a sequential decision variable and using a Gaussian process to guide configuration selection. The authors provide a sublinear regret analysis under some additional assumptions and demonstrate that COMAS effectively selects heterogeneous model configurations on synthetic and real datasets.
Critical strengths
- This paper studies an interesting and practical problem: how to jointly select the optimal reward model and configuration at the same time.
- The presentation of this work is mostly clear.
- This paper provides both theoreical analysis and a suite of empirical results on synthetic and real datasets.
Critical weaknesses
- I may overlook but it seems that there is a discrepancy between the described algorithm and its implementation: the main text describes sampling from the joint GP posterior while the appendix samples each dimension independently from their marginal distritbuion.
- I feel the experiments do not fully disentable the benefit of the GP meta controller proposed in this work from the benefit of shared interaction data. I think COMAS selects configuration using the entire shared Dt refit while the old method such as CORRAL only uses the selected rounds. Based on the no-share ablation, it seems the data sharing has a substantial impact. Therefore, I think a matched comparison shuold be conducted as this work is practically driven.
- Assumption 1 is also a bit strong. It would be great if the authors could give more explanation why this is required.
Questions for authors
- Can the authors clarify the discrepancy between the joint GP posterior sampling described in the main text and the independent marginal sampling used in the experiments?
- Can the authors provide some more ablation that separates the effect of shared interaction data from the GP meta-controller?
- How restrictive is the assumption 1 in practice?
4 comments
Response [2/2]
Assumption 1 is also a bit strong. It would be great if the authors could give more explanation why this is required.
Assumption A1 controls the learning-induced drift introduced by shared-data refitting. Specifically, on a common high-probability event, it bounds the deviation of the episode-averaged pre-round conditional expected reward along deployed episodes from a deterministic reference by . Without controlling this drift, learning-induced bias may persist, and the stated GP confidence and configuration-selection regret bounds are not guaranteed.
Assumption A1 is used in the proof at two places. First, it bounds the cumulative drift in the regret decomposition (Lemmas 2 and 5). Second, it controls the drift contribution to the GP confidence radius (Lemma 9). Thus, the analysis explicitly accounts for retraining-induced non-stationarity rather than treating the meta-level feedback as stationary. Moreover, Appendix B.5 shows that sublinear configuration-selection overhead remains possible under the slower-decay condition for a fixed finite pool and fixed regularity constants, provided the remaining assumptions hold. We discuss practical sufficient conditions and empirical diagnostics for Assumption A1 in our response to the final question below.
Can the authors clarify the discrepancy between the joint GP posterior sampling described in the main text and the independent marginal sampling used in the experiments?
Please see our response to the sampling discrepancy above. The revised Eq. (3) and Appendix B.1 now clearly distinguish the two procedures: independent posterior-marginal sampling for the finite-pool implementation and joint posterior sampling within fitted local candidate sets for the mixed-continuous implementation. Theorem 1 and the COMAS (UCB) results remain unaffected.
Can the authors provide some more ablation that separates the effect of shared interaction data from the GP meta-controller?
Both comparisons are reported above and address complementary aspects of COMAS. The shared-data baselines evaluate meta-selection under matched data-access conditions, showing that COMAS's advantage cannot be explained solely by unequal data access. The full-versus-diagonal ablation retains shared training data while removing GP covariance transfer, providing suggestive evidence of an additional covariance benefit on MovieLens. Full results are reported in Appendices B.6.2–B.6.3.
How restrictive is the assumption 1 in practice?
Sufficient conditions. Assumption A1 does not require a correctly specified reward model or an optimal limiting policy. Proposition 3 in Appendix A.8 establishes a general sufficient condition: uniform convergence of scores to deterministic limits at rate , together with a margin condition of exponent , yields a drift-decay exponent . Proposition 4 establishes A1 for well-specified ridge learners under persistent excitation and a vanishing exploration bonus. Proposition 5 extends the result to misspecified ridge learners under deterministic design and cross-moment convergence, with a vanishing bonus. Under the stated score-convergence rates and the margin condition, both ridge results yield , up to logarithmic factors.
Finite-horizon evidence. Under COMAS (UCB), frequently deployed Polynomial configurations exhibit small median late-stage deviations in policy value from their run-specific reference values: on Cosine and on Square. These configurations account for 78.8% and 88.0% of the analyzed post-warm-up episodes, respectively. Notably, the Polynomial learners are misspecified on Cosine.
These results provide finite-horizon evidence of stabilization among frequently deployed configurations, complementing the sufficient conditions in Appendix A.8. However, the diagnostics use episode-start policy values and run-specific late-stage references; they do not establish A1 over the full horizon. Details are provided in Appendix B.6.4.
We thank the reviewer again for the careful review, which will help us improve the paper. We have corrected the sampling description, added matched-data comparisons, and clarified the role and sufficient conditions of Assumption A1. We would be happy to provide any further clarification or results during the discussion period. We hope these additions address your concerns and strengthen your view of our work.
Thank you for your responses. Most of my concerns have been resolved, but I am still a bit conservative about A1’s applicability to the empirical setting. I think a more systematic study of when A1 holds under adaptive shared-data refitting would help clarify the practical scope of the theory. I tend to keep my current score while I am fine with the current work with its rebuttal if it is accepted.
Thank you for your constructive suggestion. To better assess the practical applicability of Assumption A1, we extended our empirical study of adaptive shared-data refitting to two environments, Cosine and Square, and three model families, Linear, Polynomial, and Neural.
On the theoretical side, Appendix B.8 already establishes sufficient conditions for A1. Specifically, high-probability uniform convergence of fitted action scores, together with a margin condition, yields a polynomial bound on expected-reward drift under shared-data refitting.
For each configuration, we used the recorded expected-reward values to define an empirical reference level, , and examined deviations from it. We report both the median and upper quantiles to characterize variation beyond typical behavior. The table also reports empirical polynomial envelopes of the form , with selected after inspecting the observed deviations. The exponent lies within the general vanishing-drift regime analyzed in Appendix B.5.
| Setting | range | Deviation: median | Deviation: q75 | Deviation: q90 | ||
|---|---|---|---|---|---|---|
| Cosine / Linear | 0.4890–0.5415 | 0.30 | 2.00 | 0.015170 | 0.026350 | 0.088666 |
| Cosine / Polynomial | 0.3775–0.9910 | 0.30 | 0.50 | 0.00109252 | 0.005785 | 0.0237379 |
| Cosine / Neural | 0.6580–0.8005 | 0.30 | 0.80 | 0.00625039 | 0.020795 | |
| Square / Linear | 0.077–0.102 | 0.30 | 0.50 | 0.00487404 | 0.006910 | 0.0104606 |
| Square / Polynomial | 0.4156–0.4159 | 0.30 | 0.35 | 0.00000366821 | 0.000040 | 0.000225669 |
| Square / Neural | 0.102–0.356 | 0.30 | 0.22 | 0.00424264 | 0.010400 | 0.0254938 |
The results show small median deviations across all six settings, particularly for Polynomial configurations, including the misspecified Polynomial learners on Cosine. The upper quantiles also reveal greater variability in the Linear and Neural configurations. This broader assessment provides empirical support for stabilization under adaptive shared-data refitting while quantifying how its strength varies across environments and model families. Across all six settings, run-level coverage of the calibrated drift envelopes reaches at least 80%, with 100% coverage in both Polynomial settings, providing direct empirical evidence for Assumption A1's simultaneous drift condition.
Together with the sufficient conditions established in Appendix B.8, these findings clarify the theoretical basis and empirical plausibility of Assumption A1 under adaptive shared-data refitting.
Response [1/2]
We thank the reviewer for the careful and constructive review. We appreciate the reviewer’s recognition of the importance and practical relevance of the problem, the clarity of the presentation, and the combination of theoretical analysis with extensive experiments on synthetic and real-world datasets. We have carefully addressed each concern and question below to further strengthen the paper and clarify our contributions.
Thank you for identifying this inconsistency. In finite-pool COMAS (TS), we independently sample from each configuration's GP posterior marginal and select the configuration with the largest sampled value. These marginals are obtained from the full-covariance GP, preserving kernel-based information transfer across configurations; only the correlation among acquisition draws is removed. By contrast, the mixed-continuous implementation uses joint posterior sampling within the fitted local candidate sets.
We have revised Eq. (3) and Appendix C.1 to clearly distinguish these two procedures. Theorem 1 and the COMAS (UCB) results remain unaffected, as UCB relies on posterior means and standard deviations rather than posterior sampling.
We agree that shared-data access explains part of the original performance gap: the Smooth-CORRAL/COMAS (TS) median-regret ratios decrease from 12.7× to 7.3× on Cosine and from 7.2× to 2.5× on MovieLens.
To further disentangle the contributions of shared-data access and cross-configuration covariance modeling, we conducted two complementary experiments on the Cosine and MovieLens datasets, using 10 seeds and 10,000 rounds per method.
Matched-data comparison. We modified Smooth-CORRAL and EXP3.P to refit their selected base learners using all observations collected within each run, matching COMAS's data access. All other experimental settings, including environments, configuration pools, base learners, episode scheduler, horizons, and environment seeds, were held fixed. The resulting median final cumulative regrets are:
COMAS (TS) retains 7.3× and 2.5× median-regret advantages over shared-data Smooth-CORRAL on Cosine and MovieLens, respectively. It also achieves lower regret than each shared-data baseline on 10/10 Cosine seeds and 9/10 MovieLens seeds. Thus, unequal data access accounts for only part of the observed advantage.
Covariance ablation. We further compared learned full and diagonal meta-GPs within COMAS (TS), keeping base-level data sharing and the independent-marginal acquisition rule fixed while removing cross-configuration covariance. The two variants perform comparably on Cosine (mean final regret: 120.9 vs. 119.2; median: 116.2 vs. 118.4). On MovieLens, the full GP achieves lower mean (313.1 vs. 328.9) and median (221.9 vs. 321.7) regret, winning on 8/10 paired seeds. These results provide suggestive evidence that cross-configuration covariance is beneficial on MovieLens, although the effect is less pronounced on Cosine. Notably, even the diagonal variant retains median-regret advantages of 7.2× on Cosine and 1.7× on MovieLens over shared-data Smooth-CORRAL.
Finally, our original no-sharing ablation provides separate evidence for the benefit of shared training data: removing data sharing increases COMAS (UCB)'s median Cosine regret from 148.9 to 229.4, with higher regret on all 10 paired seeds.
Together, these experiments clarify the complementary contributions of shared-data access and covariance modeling. Full details are provided in Appendices B.6.1–B.6.3.
↓ ↓ ↓ Continued below ↓ ↓ ↓