acceptodds
Under review as a conference paper at ICLR 2027

Regulating Human-in-the-Loop Learning for Dynamic LLMs

Abstract

Large language model (LLM) platforms increasingly host multiple models whose response quality evolves through user interactions and online adaptation. This creates a human-in-the-loop model-selection problem in which myopic users favor the best immediate option, while the platform seeks to maximize long-run user satisfaction. We formulate this problem as a parametric stochastic rising bandit with myopic users, using %low-dimensional saturation families to capture adaptation-driven reward growth. For a platform-controlled benchmark, we propose GAP-UCB, a growth-aware prospective UCB algorithm that fits reward trajectories via nonlinear least squares and compares models using optimistic prospective rewards. %Under the parametric formulation, GAP-UCB achieves \(O(T\log T)\) regret, improving over the existing \(O(T^2/3)\) guarantee for general stochastic rising rested bandits. We then return to the user-driven setting and develop iGAP-UCB, an ex-post budget-balanced payment mechanism that regulates users toward the platform-preferred action while preserving the same regret order up to a fee-dependent factor. Experiments on real LLM fine-tuning trajectories validate the effectiveness of GAP-UCB in the platform-controlled setting and show that iGAP-UCB closely tracks the platform-preferred benchmark under budget-balanced user regulation.

Primary area:
reinforcement learning
Keywords:
large language model platforms, online model selection, human-in-the-loop learning, stochastic rising bandits, decision regulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Yr9bOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies online LLM selection when repeated interactions improve model quality but users prioritize immediate rewards. It models this setting using parametric stochastic rising bandits and proposes GAP-UCB, which fits saturation curves and compares models at a common prospective adaptation state. It further introduces iGAP-UCB, a fee-and-compensation mechanism with ex-post budget balance. The paper presents regret guarantees and evaluates both methods through simulations based on four LLMs’ GSM8K fine-tuning trajectories.

Critical strengths
  1. The formulation captures a useful interaction between learning and incentives: selecting a model both reveals its quality and improves its future performance, while individual users may not benefit from supporting this improvement. Connecting these effects provides an informative perspective on dynamic model selection.

  2. The combination of prospective reward estimation and budget-balanced intervention is appealing. The stated analysis separates statistical uncertainty from delayed-adaptation costs, while the experiments provide encouraging evidence that the incentive mechanism can closely track the platform-controlled policy.

Critical weaknesses
  1. The headline regret claim needs clearer qualification. Theorem 3.3 includes a growth-dependent term, and Corollary 3.5 contains an additional term proportional to (T^{1-a_{\mathrm{poly,min}}}), which can dominate the square-root statistical term. The abstract and conclusion should specify the conditions under which the stated square-root regret rate applies. This matters because the improved regret guarantee is a central contribution.

  2. The empirical support has a limited scope. The experiments simulate selection using four models’ fine-tuning trajectories on a single task, with user choices following the assumed utility model. These results support the framework within the evaluated setting, but provide limited evidence about robustness to imperfect saturation fits or deviations from the assumed user behavior. This limits the strength of the practical applicability claims.

Questions for authors
  1. What restrictions on the candidate family library are required for the overall square-root regret rate, and which parameter ranges were used in the experiments?

  2. How sensitive are the reported gains to trajectory misspecification and observation noise? Please clarify whether rewards are drawn directly from empirical trajectories or from fitted curves, and report variability across repeated simulations.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Yr9b (1/2)

W1. Qualification of the Regret Guarantee.

We thank the reviewer for this careful observation. As already characterized in Corollary 3.5, exponential and hyperbolic saturation contribute only logarithmic growth-dependent terms, whereas polynomial saturation introduces an additional O(Kapoly,minT1−apoly,min)O(K^{a_{\mathrm{poly,min}}}T^{1-a_{\mathrm{poly,min}}}) term when apoly,min<1a_{\mathrm{poly,min}}<1. When apoly,min≥1/2a_{\mathrm{poly,min}}\geq1/2, this additional term does not dominate the square-root statistical term (for fixed KK), and the overall regret achieves O~(T)\widetilde O(\sqrt T). For 0<apoly,min<1/20<a_{\mathrm{poly,min}}<1/2, the growth-dependent term may dominate, and the overall guarantee becomes O~(T1−apoly,min)\widetilde O(T^{1-a_{\mathrm{poly,min}}}).

We have revised the Abstract and Conclusion to explicitly state the complete regret bound and clarify the reward-family-dependent conditions under which the overall square-root regret rate holds and present them below as well. These revisions are highlighted in blue in the updated manuscript.

W2: Experimental Scope and Robustness

We thank the reviewer for this important concern. We have substantially extended our evaluation and theoretical analysis in three aspects.

(1) Additional task and LLMs. We conduct additional model-selection experiments using reward trajectories of five candidate LLMs from an independent text summarization task reported by Xia et al. (2024). The trajectories are digitized from their published results and extended for longer-horizon online simulations. We evaluate both platform-controlled and user-driven settings, with detailed results presented in Appendix E.4.

(2) Heterogeneous user preferences. We extend the baseline user model to allow individual valuations to deviate from the platform's published estimates. We consider two scenarios: random preferences and model-specialized preferences, in which arriving users may favor different LLMs. As shown in Figure 12(b) of Appendix G.4, at p=0.1p=0.1, iGAP-UCB maintains a final regret of approximately 155 across all tested heterogeneity levels ζ∈[0,0.2]\zeta\in[0,0.2] under random preferences. Under model-specialized preferences, its regret increases only slightly, from approximately 155 to 159, whereas iRedUCB's regret increases from approximately 206 to 337, shown in Figure 13(b). Moreover, at p=0.15p=0.15, iGAP-UCB maintains nearly constant regret across both scenarios (Figure 13(c)). These results demonstrate its empirical robustness to heterogeneous user valuations under sufficiently large fee caps. The corresponding theoretical guarantee is established in Theorem G.2.

(3) Robustness to model misspecification. We further relax the exact realizability assumption by allowing the true expected reward trajectories to deviate from the candidate saturation families by at most ϵ\epsilon. The robustness analysis in Appendix F establishes an additional O(ϵT)O(\epsilon T) regret term, explicitly quantifying the effect of model misspecification. Moreover, our empirical trajectories are not perfectly fitted by the candidate families. For example, Table 2 reports a best-fitting R2R^2 of 0.368 for Yi-6B, yet GAP-UCB achieves favorable performance on these trajectories. This provides empirical evidence that accurate curve fitting is not a prerequisite for favorable finite-horizon performance.

Together, these additions provide broader experimental evidence and theoretical robustness guarantees beyond the original single-task setting, while acknowledging that the experiments remain simulation-based rather than a deployed human--AI system.

References

[1] Yu Xia, Fang Kong, Tong Yu, Liya Guo, Ryan A. Rossi, Sungchul Kim, and Shuai Li. Which LLM to Play? Convergence-Aware Online Model Selection with Time-Increasing Bandits. Proceedings of the ACM Web Conference (WWW), 2024.

AuthorsOpenReview ↗

Response to Reviewer Yr9b (2/2)

Q1: Conditions for Square-Root Regret and Experimental Parameter Ranges

The overall square-root regret rate, up to logarithmic factors, holds when the reward-growth-dependent term satisfies Γ(T)=O~(T)\Gamma(T)=\widetilde O(\sqrt T). As characterized in Corollary 3.5, this condition is satisfied by exponential and hyperbolic saturation families, as well as polynomial saturation with exponent apoly,min≥1/2a_{\mathrm{poly,min}}\ge 1/2. For polynomial saturation with 0<apoly,min<1/20<a_{\mathrm{poly,min}}<1/2, the additional growth-dependent term may dominate, resulting in an overall regret bound of O~(T1−apoly,min)\widetilde O(T^{1-a_{\mathrm{poly,min}}}).

Regarding the experimental parameter ranges, we use:

  • GSM8K: c∈[0,0.6]c\in[0,0.6], a∈[0.01,0.1]a\in[0.01,0.1].

  • XSum text summarization: c∈[0,0.8]c\in[0,0.8], a∈[0.02,0.1]a\in[0.02,0.1]

These ranges specify the feasible parameter domains used for online fitting. In particular, the polynomial fitting ranges include exponents below 1/21/2; therefore, the overall square-root regret guarantee does not hold uniformly over these parameter domains. Nevertheless, the complete family-dependent bound in Theorem 3.3 remains applicable under its stated assumptions.

Q2: Model Misspecification, Observation Noise, and Experimental Robustness

We thank the reviewer for this question. We clarify the reward generation process and the robustness of our approach as follows.

Reward observations. In the GSM8K experiments, rewards are taken directly from the empirical fine-tuning trajectories, where each adaptation step is evaluated on 273 held-out problems. We do not generate rewards from fitted saturation curves. Similarly, in the additional XSum text summarization experiments (Appendix E.4), we use reward trajectories digitized from Xia et al. (2024), with their late-stage segments repeated to construct a longer simulation horizon. The saturation families are used only for online reward estimation and model selection, not for generating the experimental rewards.

Model misspecification and observation noise. Our stochastic reward model accommodates observation noise under the assumed sub-Gaussian noise conditions. To further investigate deviations from the candidate saturation families, we have added a robustness analysis in Appendix F, establishing that bounded model misspecification of magnitude ϵ\epsilon introduces an additional O(ϵT)O(\epsilon T) regret term. In particular, when ϵ=O(1/T)\epsilon=O(1/\sqrt{T}), the additional regret is O(T)O(\sqrt{T}), preserving the original square-root regret order whenever the family-dependent growth term satisfies Γ(T)=O~(T)\Gamma(T)=\widetilde O(\sqrt{T}). Moreover, when ϵ=O(1/T)\epsilon=O(1/T), the additional regret is only O(1)O(1).

Our empirical results provide complementary evidence of robustness to imperfect saturation fits. For example, Table 2 reports a best-fitting R2R^2 of 0.368 for Yi-6B, yet GAP-UCB achieves favorable performance on the empirical trajectories. This suggests that accurate fitting of every observed reward fluctuation is not necessary for effective finite-horizon model selection.

Variability across simulations. Our experiments replay fixed empirical reward trajectories rather than generating new fine-tuning trajectories for each simulation. Consequently, repeated runs with identical reward sequences and algorithmic randomness produce identical results. We clarify this experimental setup and distinguish variability in online model selection from variability arising from independent LLM fine-tuning runs.

References

[1] Yu Xia, Fang Kong, Tong Yu, Liya Guo, Ryan A. Rossi, Sungchul Kim, and Shuai Li. Which LLM to Play? Convergence-Aware Online Model Selection with Time-Increasing Bandits. Proceedings of the ACM Web Conference (WWW), 2024.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer Yr9b,

Thank you again for your careful and constructive comments. We hope that our revisions and additional analysis have addressed your main concerns. We would be happy to provide further explanations or analyses where helpful.

We look forward to your feedback and appreciate your time and consideration. Thanks.

Best Regards,

Authors of Submission43597

Reviewer_9eNvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how current model selection affects both immediate response quality and future model capabilities: the selected model receives a training opportunity, thereby influencing subsequent cumulative quality.

The authors formulate this problem as a rising rested bandit, in which only the selected model advances its training. GAP-UCB fits model improvement trajectories using exponential, polynomial, and hyperbolic saturation families, compares predicted quality at a common hypothetical adaptation state, and incorporates an upper confidence bound for selection. iGAP-UCB further considers myopic users, collecting fees to accumulate a budget and compensating users for the immediate quality loss incurred by choosing the platform-recommended model.

The experiments simulate 1,800 rounds of model selection using LoRA fine-tuning trajectories from four models on GSM8K. GAP-UCB outperforms three baselines in the platform-controlled setting, and iGAP-UCB performs close to GAP-UCB. However, the supplementary experiments show that iRedUCB substantially outperforms iGAP-UCB.

Critical strengths
  1. The research problem is reasonably motivated. Current selections affect future training progress, distinguishing this problem from routing among fixed models. Users’ immediate preferences may also conflict with the platform’s long-term objective.

  2. The method is consistent with the problem formulation. Growth-curve prediction, prospective comparison, and budget-constrained compensation form a clear framework. The paper also provides analyses of confidence bounds, regret bounds, and budget properties.

  3. The experiments use actual fine-tuning trajectories and provide some analysis of the mechanisms. Model-selection counts and fixed-family ablations help explain the algorithms’ behavior. The authors also disclose the stronger performance of iRedUCB.

Critical weaknesses
  1. The experiment most closely aligned with the paper’s central application scenario does not demonstrate superiority over existing methods.

    The paper emphasizes user participation, fees, and compensation. However, in supplementary Figure 6, under the same compensation mechanism and fee cap of p=0.03p=0.03, iRedUCB achieves a final regret of approximately 10, compared with approximately 167 for iGAP-UCB.

    The authors explain this result by suggesting that the budget constraint happens to suppress RedUCB’s excessive exploration, but this explanation is not validated across additional environments. Thus, the available results support the potential usefulness of the compensation mechanism, but do not sufficiently establish the necessity or advantage of using GAP-UCB within that mechanism. The existence of a theoretical upper bound does not resolve this empirical concern.

  2. There is a substantial gap between the experimental setting and real human–AI interaction.

    The experiments rely on GSM8K data with ground-truth answers, use an effective training batch size of 16, and evaluate all 273 held-out problems after every adaptation step. The model-selection policy receives the accuracy measured over this entire evaluation set.

    This evaluates a resource-allocation problem based on supervised fine-tuning trajectories, but does not establish how a platform would obtain reliable training targets and reward signals after real user queries. Missing corrections, erroneous feedback, delayed feedback, and performance degradation are not examined. The paper’s conclusions about practical human-in-the-loop systems therefore extend beyond the scope of its experimental evidence.

  3. The experimental scope and evidence of robustness are insufficient.

    Both the main paper and the supplementary material focus on one task, four models, and the same collection of growth trajectories. The appendix specifies a fixed random seed, but does not report means, variances, or confidence intervals across multiple seeds.

    The concern is not merely the number of models, but the lack of environmental variation needed to establish the method’s applicability: different tasks, training-data orders, growth rates, crossing trajectories, distribution shifts, and time horizons. Given the substantial advantage of iRedUCB in one comparison, the available evidence is insufficient to establish whether the relative ranking of the methods is stable.

  4. The algorithmic novelty primarily consists of an incremental combination of existing ideas, and its distinct value has not been sufficiently established.

    Improvement induced by selection is already studied in rising bandits. Comparing models at a common prospective adaptation state is also not introduced by this paper, as the main text explicitly acknowledges. The principal technical additions are online fitting and selection within a finite library of saturation families, together with the associated confidence bounds and fee-and-compensation analysis.

    These additions may be valuable, but their significance depends on demonstrating sufficiently substantial and consistent improvements. The ablation supports adaptive family selection over fixing a single family, but does not adequately establish whether this design remains reliable when the true growth dynamics fall outside the candidate function families.

  5. The characterization of the theoretical advantage requires more explicit qualifications.

    The bound in Theorem 3.3 is actually

    O(KTlog⁡(MKT))+Γ(T),O\left(\sqrt{KT\log(MKT)}\right)+\Gamma(T),

    rather than the square-root term alone. In Corollary 3.5, the polynomial family introduces an additional term

    Kamin⁡T1−amin⁡.K^{a_{\min}}T^{1-a_{\min}}.

    The experimental setup in the appendix specifies amin⁡=0.01a_{\min}=0.01. If this common lower bound also applies to the polynomial family, substituting it into the bound yields a K0.01T0.99K^{0.01}T^{0.99} term. This does not imply that the actual regret necessarily grows as T0.99T^{0.99}, but it does mean that the overall guarantee under this configuration cannot simply be characterized as having square-root dependence on the horizon. The presentation of the theoretical advantage should remain consistent with the complete bound and its applicable conditions.

  6. The user model and evaluation metrics are insufficient to support conclusions about practical platform value.

    Users are assumed to choose models according to the platform’s published quality estimates. Quality, fees, and compensation enter the same linear utility function, and ties are resolved in favor of the platform recommendation. The experiments do not examine heterogeneous preferences, willingness to pay, or user departure.

    Moreover, the primary metric is response-quality regret, which does not directly reflect the fees borne by users or account for inference, fine-tuning, and feedback-acquisition costs. Consequently, lower regret under higher fees does not establish a better overall user experience. Likewise, a nonnegative compensation budget does not establish the financial sustainability of the platform’s overall operations.

Overall Assessment

The paper presents a reasonable research motivation and a clear methodological framework. However, the current technical advances, the scope of the theoretical advantages, and the experimental evidence are insufficient to establish a convincing contribution. In particular, the gap between the experimental environment and real human–AI interaction, together with the insufficiently demonstrated advantage in the central user-driven setting, limits the strength of the conclusions. Overall, the submission does not meet the acceptance threshold for this conference, and I recommend rejection.

Questions for authors

Under the same compensation mechanism, iRedUCB substantially outperforms iGAP-UCB. How do the authors characterize the principal advantage of their method in the user-driven setting, and how do the existing experiments support that advantage?

9 comments
AuthorsOpenReview ↗

Response to Reviewer 9eNv (1/6)

W1 & Q1: Performance of iGAP-UCB versus iRedUCB

We thank the reviewer for raising this important concern. We sincerely apologize for an implementation error in our original user-driven experiments and the misleading empirical conclusion it led to. Specifically, the myopic user's decision was incorrectly based on prospective reward estimates rather than estimates at each LLM's current adaptation state, r^i,t(Ni,t)\widehat r_{i,t}(N_{i,t}). We have corrected this error, rerun the affected experiments, and updated the corresponding results. In addition, we have also substantially expanded our evaluation across different fee caps and tasks. The revised results are provided in Appendix E.2 and E.4.

Main findings from the revised experiments. Our additional experiments on GSM8K and an independent text summarization benchmark provide stronger evidence of the advantage of iGAP-UCB. Specifically, iGAP-UCB achieves lower regret than iRedUCB under larger fee caps on GSM8K and across most tested fee caps on the text summarization benchmark, while exhibiting more stable performance as the fee cap varies. These results demonstrate that the favorable performance of iRedUCB in the original instance does not generalize consistently across different budget conditions and tasks.

Additional results on GSM8K. we evaluate the algorithms across fee caps ranging from p=0.01p=0.01 to 0.200.20. As shown in Figure 7 of Appendix E.2, the performance of iRedUCB is highly sensitive to pp. Although iRedUCB still achieves lower regret at p=0.03p=0.03 after correcting the implementation, its regret increases substantially as pp becomes larger. In contrast, iGAP-UCB exhibits more stable performance and approaches the platform-controlled GAP-UCB benchmark under larger fee caps (p≥0.08p\ge 0.08), while iRedUCB's regret increases to approximately 338338 at p=0.20p=0.20. The detailed results are provided in Appendix E.2.

Additional evaluation on text summarization task. We further evaluate the algorithms using reward trajectories digitized from Figure 5(a) of Xia et al. (2024) on the XSum text summarization task. In the platform-controlled setting, GAP-UCB achieves the lowest final regret among the compared algorithms (approximately 530530, versus 1,610 for RedUCB and 910 for TI-UCB) (see Figure 9). In the user-driven setting with p=0.10p=0.10, iGAP-UCB achieves a final regret of approximately 540, compared with 1,190 for iRedUCB and 920 for iTI-UCB (see Figure 10), closely tracking its platform-controlled counterpart. Moreover, the fee-cap sensitivity results in Figure 11 show that iGAP-UCB maintains stable regret of approximately 540 for p≥0.03p\ge 0.03, whereas iRedUCB's regret increases substantially under larger fee caps. See Appendix E.4 for the complete results.

Explanation and principal advantage. The relatively low regret of iRedUCB under certain small fee caps can be attributed to an instance-specific interaction between its aggressive exploration and the compensation budget constraint. Limited budget may incidentally suppress excessive exploration, whereas larger budgets allow more exploratory recommendations to be implemented. In contrast, iGAP-UCB explicitly exploits the increasing-and-saturating structure of LLM adaptation to balance exploration and long-term reward growth, while providing an ex-post budget-balanced incentive mechanism with a theoretical regret guarantee. The revised experiments demonstrate the benefits of this design across the evaluated tasks and fee settings.

References

[1] Yu Xia, Fang Kong, Tong Yu, Liya Guo, Ryan A. Rossi, Sungchul Kim, and Shuai Li. Which LLM to Play? Convergence-Aware Online Model Selection with Time-Increasing Bandits. Proceedings of the ACM Web Conference (WWW), 2024.

AuthorsOpenReview ↗

Response to Reviewer 9eNv (2/6)

W2: Gap Between the Experimental Setup and Real Human-AI Interaction

We thank the reviewer for highlighting the practical limitations of our experimental setting. We clarify the following points.

(1) Training targets and reward feedback. Our framework assumes that user interactions can provide training signals for adapting the selected LLM. One possible implementation is an iterative interaction process in which a user continues refining the response until a satisfactory answer is obtained. The resulting query--response pair can then serve as a supervision signal for subsequent fine-tuning. This does not require every user query to have a predefined ground-truth answer, although the quality of such user-provided supervision is not guaranteed. In our experiments, GSM8K provides a controlled setting in which both adaptation data and reward observations can be reliably constructed. The full evaluation after each adaptation step is used to generate empirical reward trajectories for the online simulations, rather than implying that a deployed platform must evaluate 273 labeled queries after every interaction.

(2) Delayed and erroneous feedback. We acknowledge that real user feedback may be delayed or noisy. Our stochastic reward model already accommodates random observation noise under the assumed concentration conditions. Bounded feedback delays can also be studied through delayed bandit updates; under appropriate assumptions, a delay-dependent regret overhead can be derived. We regard the explicit treatment of delayed feedback as a natural extension of the present framework.

(3) Performance deviations and model misspecification. Our theoretical analysis models the expected reward through increasing-and-saturating adaptation functions, while allowing stochastic fluctuations in observed rewards. To examine robustness beyond exact realizability, we have added a bounded model misspecification analysis in Appendix F, establishing that deviations of magnitude at most ϵ\epsilon introduce an additional O(ϵT)O(\epsilon T) regret term. The original guarantee is recovered when ϵ=0\epsilon=0. When ϵ=O(1/T)\epsilon=O(1/\sqrt{T}), the additional term is only O(T)O(\sqrt{T}), preserving the the overall square-root regret order whenever the original family-dependent term is at most O~(T)\tilde{O}(\sqrt{T}). Moreover, when ϵ=O(1/T)\epsilon=O(1/T), the additional regret is only O(1)O(1). Generally, for a fixed nonzero ϵ\epsilon, the average regret is controlled up to an O(ϵ)O(\epsilon) approximation floor. This provides a quantitative robustness guarantee when the expected reward trajectory deviates from the assumed saturation families.

Scope of the study. Our work focuses on the fundamental learning-and-incentive challenges in human-in-the-loop LLM selection, rather than developing an end-to-end deployed system. The framework captures adaptation-dependent rewards, myopic user decisions, and budget-balanced monetary incentives, while providing theoretical regret guarantees. We acknowledge that real deployments may involve imperfect supervision, delayed feedback, and more complex user behavior. We have clarified these practical considerations and the scope of our assumptions in Section 2.1and Section 6 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 9eNv (3/6)

W3: Experimental Diversity and Robustness

We thank the reviewer for this valuable suggestion. We have substantially expanded our experiments, as detailed in Appendix E.2 and E.4.

Diversity of reward trajectories. Even in the original GSM8K experiments, the four candidate LLMs exhibit different initial qualities, growth rates, and saturation levels, with their reward trajectories crossing during adaptation (Figure 2). These characteristics create a nontrivial online selection problem in which the optimal model may change as adaptation progresses.

Additional task and time horizon. We further evaluate our algorithms on an independent XSum text summarization benchmark using reward trajectories of five candidate LLMs digitized from Xia et al. (2024). Unlike GSM8K, which considers T=1800T=1800 rounds, we evaluate online model selection algorithms over T=10000T=10000 rounds on the extended XSum trajectories with different reward-growth patterns. The details are presented in Appendix E.4.

Additional robustness evaluation. We conduct both platform-controlled and user-driven experiments on the additional benchmark, together with fee-cap sensitivity analyses on both tasks. On GSM8K, iGAP-UCB achieves a final regret of approximately 155 under larger fee caps, closely matching platform-controlled GAP-UCB, whereas iRedUCB's regret increases to approximately 338338 at p=0.20p=0.20. On XSum, GAP-UCB achieves the lowest final regret among the compared algorithms (approximately 530530, versus 1,610 for RedUCB and 910 for TI-UCB) (see Figure 9). In the user-driven setting with p=0.10p=0.10, iGAP-UCB achieves a final regret of approximately 540, compared with 1,190 for iRedUCB and 920 for iTI-UCB (see Figure 10). Furthermore, iGAP-UCB maintains nearly constant regret across p∈[0.03,0.20]p\in[0.03,0.20] on XSum, whereas iRedUCB's regret increases substantially. These results demonstrate that iRedUCB's favorable performance under certain small fee caps is not consistent across the evaluated environments, while iGAP-UCB exhibits more stable performance.

We acknowledge that variations in training-data ordering and distribution shifts are not explicitly examined in the current experiments and remain interesting directions for future work.

References

[1] Yu Xia, Fang Kong, Tong Yu, Liya Guo, Ryan A. Rossi, Sungchul Kim, and Shuai Li. Which LLM to Play? Convergence-Aware Online Model Selection with Time-Increasing Bandits. Proceedings of the ACM Web Conference (WWW), 2024.

Reviewer_NCVvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies online LLM selection in which interactions reveal model quality and improve selected models through adaptation, while users maximize immediate estimated utility. It formulates the problem as a parametric rising rested bandit and proposes GAP-UCB, which combines nonlinear trajectory fitting with optimistic selection, achieving a regret bound with an O(KTlog⁡(MKT))O(\sqrt{KT\log(MKT)}) statistical term and growth-dependent terms. Its incentive-based extension, iGAP-UCB, uses model-independent fees to fund compensation, maintaining a nonnegative budget and preserving the regret order up to a fee-dependent factor under the stated assumptions. Simulations using recorded GSM8K LoRA adaptation trajectories from four LLMs show lower regret than competing baselines and improved alignment with the platform-controlled benchmark as the fee cap increases, though no live user interactions are evaluated.

Critical strengths
  1. The paper studies LLM selection when user interactions both reveal and improve model quality, creating a tension between users’ immediate preferences and the platform’s long-term objective. This coupling of model adaptation and user choice makes the problem compelling and motivates studying learning and incentives jointly.

  2. The paper develops GAP-UCB for growth-aware model selection and extends it to iGAP-UCB through a budget-balanced payment mechanism. The corresponding analysis provides regret bounds and establishes budget feasibility and recommendation implementability under the stated assumptions. These guarantees strengthen the contribution by explaining when the proposed approach can achieve effective long-term learning.

Critical weaknesses
  1. Each model’s expected quality must follow a curve exactly contained in the predefined saturation library, with improvement determined solely by its own selection count. This excludes model misspecification, nonmonotonic adaptation, and variation in the usefulness of interaction data. Since the regret guarantees depend on this structure, the absence of robustness analysis or misspecified experiments limits their relevance to realistic LLM adaptation.

  2. The experiments simulate selection using precomputed fine-tuning trajectories from four models on a single task. Users are assumed to maximize the platform’s estimated utility and follow its recommendation when indifferent, while fees and compensation are expressed directly in reward units. Without experiments involving heterogeneous preferences, imperfect responses to incentives, or a justified mapping from quality to monetary utility, the results provide limited evidence that iGAP-UCB would successfully regulate real user choices.

Questions for authors
  1. How sensitive is GAP-UCB to reward trajectories outside the candidate saturation library? Do you have analysis or experimental evidence showing whether its advantage persists under modest misspecification? This would help establish whether the reported gains depend critically on the realizability assumption.

  2. What justifies assuming that users value model quality according to the platform’s published estimates and that monetary transfers enter utility on the same scale? Can you provide evidence or sensitivity results showing whether iGAP-UCB remains effective when users’ valuations differ from these estimates? This is central to assessing the applicability of the mechanism beyond simulated users.

3 comments
AuthorsOpenReview ↗

Response to Reviewer NCVv (1/2)

W1 & Q1: Robustness to Model Misspecification and Non-Monotonic Adaptation

We thank the reviewer for raising this important concern. We have extended our reward model to accommodate both non-monotonic adaptation and model misspecification, and established a corresponding regret guarantee in Appendix F.

Extended reward model and theoretical robustness. In Section 2.2, the original reward model assumes that each LLM's expected reward follows ri(n)=μi+fmi∗(n;θi∗)r_i(n)=\mu_i+f_{m_i^{*}}(n;\theta_i^*), where the improvement function fmi∗f_{m_i^*} belongs to the predefined saturation library in Eq. (2). We relax this assumption by introducing a bounded deviation:

\qquad \sup_{n\ge 0}|d_i(n)|\le \epsilon.$$ Importantly, the deviations $d_i(n)$ are not required to be monotone, allowing the true expected reward trajectories to exhibit non-monotonic adaptation and depart from the candidate saturation families. Thus, our extended model covers a broader class of reward dynamics than the original realizable setting. Under this extended model, we establish that bounded misspecification introduces an additional $O(\epsilon T)$ regret term. The original guarantee is recovered when $\epsilon=0$. When $\epsilon=O(1/\sqrt{T})$, the additional term is only $O(\sqrt{T})$, preserving the original asymptotic regret order. Moreover, when $\epsilon=O(1/T)$, the additional regret is only $O(1)$. Generally, for a fixed nonzero $\epsilon$, the average regret is controlled up to an $O(\epsilon)$ approximation floor. The detailed analysis and proofs are provided in Appendix F. **Empirical evidence.** We clarify our experiments use directly measured LLM fine-tuning trajectories rather than rewards generated from fitted saturation functions. The empirical fitting results in Table 2 further show that the candidate families approximate different LLMs with varying accuracy. For example, the best-fitting exponential family for Yi-6B achieves an $R^2$ of only 0.368, yet GAP-UCB achieves favorable performance on these empirical trajectories. This indicates that exact fitting of empirical observations is not necessary for favorable finite-horizon performance, consistent with the motivation for our bounded misspecification analysis. Overall, the extended model and corresponding regret analysis demonstrate that GAP-UCB can accommodate bounded deviations from the candidate saturation families, including non-monotonic expected reward trajectories, with an explicitly quantified approximation cost. This substantially broadens the scope of our original theoretical framework.
AuthorsOpenReview ↗

Response to Reviewer NCVv (2/2)

W2 & Q2: User Valuation, Monetary Incentives, and Heterogeneous Preferences

We thank the reviewer for raising these important concerns. We have clarified the user valuation and utility model in Sections 2.3 and 4.2, and extended our framework to accommodate heterogeneous user preferences in Appendix G.

User valuation and monetary incentives. We have clarified the user valuation model in Section 2.3 and utility model in Section 4.2. Let vi,tv_{i,t} denote the arriving user's valuation of LLM ii. In our baseline model, users evaluate LLMs according to the platform's published estimates of their current response quality, i.e., vi,t=r^i,t(Ni,t)v_{i,t}=\widehat r_{i,t}(N_{i,t}). We further relax the homogeneous valuation assumption to accommodate user-specific preferences, with theoretical and empirical results provided in Appendix G.

Following the standard quasi-linear utility framework widely adopted in economics and mechanism design (Mas-Colell et al., 1995; Nisan et al., 2007), we define

ui(t)=vi,t+λt(xt(i)−pt),u_i(t)=v_{i,t}+\lambda_t(x_t(i)-p_t),

where xt(i)x_t(i) and ptp_t denote the compensation and usage fee, respectively, and λt>0\lambda_t>0 caputure the user's sensitivity to monetary transfers. Similar models of myopic, self-interested users whose exploration decisions can be influenced by monetary compensation have been extensively studied in incentivized bandits (Frazier et al., 2014; Wang and Huang, 2018).

Our main analysis adopts the normalized setting λt=1\lambda_t=1, under which response quality and monetary transfers are expressed in a common normalized utility scale. We emphasize that this is a standard modeling abstraction rather than an empirically calibrated mapping between LLM response quality and real-world monetary willingness to pay.

Theoretical robustness. We, in Appendix G.1, extend the model to heterogeneous users with valuations vi,t=r^i,t(Ni,t)+ξi,tv_{i,t}=\widehat r_{i,t}(N_{i,t})+\xi_{i,t}, where ξi,t\xi_{i,t} captures the arriving user's model-specific preference deviation, satisfying ∣ξi,t∣≤ζ|\xi_{i,t}|\le \zeta. The extended mechanism preserves ex-post budget balance, individual rationality, and action-level implementability (presented in Appendix G.2). Moreover, in Appendix G.3, Theorem G.2 establishes that bounded user heterogeneity introduces an additional O(ζT/p)O(\zeta T/p) regret term, recovering the original guarantee when ζ=0\zeta=0.

Additional experiments with heterogeneous users. We conduct two new experiments using the real GSM8K fine-tuning trajectories: (i) random preferences, where each arriving user's model-specific deviations are independently sampled from U(−ζ,ζ)\mathcal U(-\zeta,\zeta), and (ii) model-specialized preferences, where each user randomly favors one LLM over the others. We evaluate iGAP-UCB, iRedUCB, iTI-UCB, and an unregulated Greedy baseline under different heterogeneity levels and fee caps. As shown in Figures 12--13 of Appendix G, iGAP-UCB exhibits stable regret under heterogeneous preferences, particularly when the fee cap is sufficiently large. In contrast, the competing algorithms generally show greater sensitivity to preference heterogeneity. These results provide additional empirical evidence of the mechanism's robustness under the tested scenarios.

We acknowledge that imperfect user compliance and empirically calibrated monetary valuations remain beyond the scope of our current model.

References

[1] Andreu Mas-Colell, Michael D. Whinston, and Jerry R. Green. Microeconomic Theory. Oxford University Press, 1995.

[2] Noam Nisan, Tim Roughgarden, Éva Tardos, and Vijay V. Vazirani (eds.). Algorithmic Game Theory. Cambridge University Press, 2007.

[3] Peter Frazier, David Kempe, Jon Kleinberg, and Robert Kleinberg. Incentivizing Exploration. ACM Conference on Economics and Computation (EC), 2014.

[4] Siwei Wang and Longbo Huang. Multi-armed Bandits with Compensation. Advances in Neural Information Processing Systems (NeurIPS), 2018.

AuthorsOpenReview ↗

Following-up on Our Rebuttal

Dear Reviewer NCVv,

We would like to kindly follow up on our rebuttal and thank you again for your thoughtful comments.

We hope that our revisions, additional analyses, and experiments have addressed your main concerns. As the discussion period is drawing to a close, we would be grateful for any further feedback you may have to help us improve our paper. We sincerely thank you for your time, thoughtful feedback, and careful consideration of our work.

Best Regards,

Authors of Submission43597

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.