acceptodds
Withdrawn from ICLR 2027

Online Serving Problem with Costly Small-model Adaptation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Edge-cloud collaboration with large language models offers a promising way to balance model quality and deployment cost: large models provide strong capabilities but incur substantial inference costs, while smaller models are more efficient and can potentially be improved through continuous adaptation. However, existing routing approaches typically treat model capabilities as fixed and only decide which model should serve each request. We study an online serving problem where the learner jointly decides whether to fine-tune a small model and how to serve incoming requests, while adaptation incurs an immediate cost but changes future model performance. We characterize the fundamental difficulty of online adaptation through lower bounds and propose an adaptive algorithm combining convergence detection, optimistic adaptation, and confidence-based serving. The resulting regret guarantee is instance-adaptive and nearly matches the lower bounds. Experiments on edge-cloud LLM serving benchmarks demonstrate that the proposed method effectively exploits adaptation when it improves the small model while avoiding unnecessary fine-tuning when adaptation provides limited benefits.

Primary area:
learning theory
Keywords:
Sequential Decision Making, Multi-armed Bandits, LLM routing, Edge-cloud Collaboration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_whwwOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies an online model-serving problem in edge-cloud collaborative inference, where the system must jointly decide whether to serve incoming requests using a cloud-based large language model or a local small language model, and whether to fine-tune the small model to improve its future performance. The authors propose an adaptive algorithm that combines convergence detection, optimistic adaptation, and confidence-based model selection to balance inference quality, serving costs, and fine-tuning overhead. They also provide theoretical regret bounds to characterize the effectiveness of their approach. Experiments using Qwen3.5-Plus and Qwen3-4B on the MATH and XSum datasets show that the proposed method achieves lower average loss on MATH while maintaining performance close to cloud-only serving on XSum.

Critical strengths
  1. The paper goes beyond traditional model routing by considering both inference decisions and the cost of fine-tuning the local model. This provides an interesting perspective on balancing model performance and serving costs over time.
  2. The authors provide regret bounds to support their proposed algorithm and evaluate it on two datasets using real LLMs. The results show some improvements over existing baselines, particularly on MATH, which helps demonstrate the potential effectiveness of the approach.
Critical weaknesses
  1. The theoretical analysis assumes that the small model's loss decreases along a relatively simple learning curve and eventually converges. However, real-world online fine-tuning is much more complex, especially when different task clusters share the same model parameters. It is unclear whether the theoretical guarantees still hold under more realistic training conditions.
  2. The experiments only cover two datasets and one pair of models. The improvement on MATH is relatively small (0.338 vs. 0.346 for Always Cloud), while the method does not outperform Always Cloud on XSum. More extensive experiments, stronger baselines, and ablation studies are needed to demonstrate the effectiveness and generalizability of the proposed approach.
  3. Although the paper focuses on reducing serving and adaptation costs, the experiments mainly use simplified loss and cost metrics. Important factors such as actual fine-tuning time, GPU resource consumption, communication overhead, and end-to-end inference latency are not adequately evaluated. On XSum, the method still spends 40.7% of requests on training even though the authors claim that adaptation provides no benefit, which further raises concerns about its practical efficiency.
  4. I found the paper quite difficult to follow. The Introduction lacks citations and contains too many short paragraphs, making the background and motivation unclear. There is also no system architecture or motivation figure to help readers quickly understand the problem and proposed solution. The experimental results are presented with relatively limited visual analysis, making it harder to evaluate the main contributions and understand why the method works.
Questions for authors
  1. Could the authors explain why continuous decisions about fine-tuning are necessary in real-world edge-cloud collaborative inference systems? In practice, these systems often focus on cooperation between large and small models rather than continuously adapting the smaller model. What specific deployment scenario motivates this problem, and why would the proposed approach be more suitable than conventional model routing with periodic fine-tuning?
  2. The theoretical analysis assumes that the small model's loss follows a decreasing-then-converging learning curve. However, actual online fine-tuning may produce unstable performance, especially when different task clusters share the same model parameters. How do the authors justify these assumptions, and to what extent would the theoretical guarantees remain valid under more realistic training conditions?
  3. In the XSum experiments, the proposed method allocates 40.7% of requests to training, even though the authors state that adaptation provides no benefit. Could the authors explain why the algorithm continues training so frequently in this case? How is the training cost reflected in the reported average loss of 0.909, which is almost identical to the Always Cloud baseline of 0.908?
  4. The improvement over Always Cloud on MATH appears relatively small (0.338 vs. 0.346), while no improvement is observed on XSum. Could the authors clarify whether these improvements are statistically significant and provide stronger evidence that the proposed joint optimization is more effective than simpler routing or periodic fine-tuning strategies?
  5. The proposed approach uses the local model's average token entropy to cluster incoming requests. Does this require the local model to generate a response before the routing decision is made? If so, how are the additional inference latency and computational costs accounted for? Also, how much overhead does online fine-tuning introduce when inference and training share limited edge resources?
Reviewer_ej18OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies online model serving where the learner decides among large model, small model, or fine-tuning the small model while serving with the large model. It formulates this as sequential decision-making with costly, cluster-dependent adaptation, defines a useful adaptation complexity measure, gives worst-case and budget-dependent regret lower bounds, and proposes an adaptive algorithm combining convergence detection, optimistic adaptation, and confidence-based serving. The claimed instance-adaptive regret nearly matches the lower bounds. Experiments on MATH and XSum with Qwen3.5-Plus and Qwen3-4B show gains over serving baselines when adaptation helps.

Critical strengths
  1. The problem is interesting and practical: jointly considering serving and costly online adaptation is more realistic than standard routing, since fine-tuning decisions affect future serving.
  2. The algorithm is intuitive, and “useful adaptation complexity” gives a nice angle for instance-dependent regret analysis.
  3. The experiments offer useful insights. The method gets the lowest average loss on MATH, and the per-cluster analysis helps show how adaptation/serving decisions evolve.
Critical weaknesses
  1. The theory has real gaps. Lemma 1 says convergence is detected within ω updates after the true convergence point, but the detector only starts after 2ω updates. If the true point comes early enough, both windows can be flat, so the detector can’t see convergence. The proof also needs a separation condition involving adaptation slope, window size, and noise, but this isn’t stated in the theorem.
  2. The useful adaptation delay analysis isn’t convincing. Lemma 2 bounds each delayed update by its marginal improvement, but the learner can be several updates behind the oracle, so the gap can accumulate. The adaptation probability uses marginal gain versus immediate fine-tuning cost, but ignores the remaining serving horizon. For slow-improving curves, this can delay beneficial adaptation too much. The O(B*) dependence isn’t really established.
  3. The theory–experiment connection is weak. The theory assumes independent per-cluster adaptation curves, but the experiments seem to fine-tune a shared small model, and entropy-based cluster assignments change during training. So fine-tuning one cluster may affect others, and cluster boundaries may shift the request distribution. The implementation also uses batched GRPO updates, not the immediate transitions assumed in theory.
  4. The experiments are not comprehensive enough. On XSum, the method fine-tunes 40.7% of requests even though the paper says adaptation isn’t beneficial; the cost accounting is unclear, and the average loss is very close to always-cloud. There are no multiple seeds, component ablations, or sensitivity analyses for adaptation cost and convergence threshold. The MATH gain over always-cloud is small, so statistical validation matters.
Questions for authors
  1. Can you clarify when the convergence detector actually satisfies Lemma 1? What happens if the true convergence point is before the first detection window, and what minimum adaptation slope is needed to guarantee detection within ω updates?
  2. Can you justify the Lemma 2 delay bound more rigorously? How does the analysis handle the accumulated gap when the learner is several updates behind the oracle? And how does the adaptation probability avoid long delays when marginal improvements are small but cumulative benefit is large?
  3. How are cluster-specific adaptation curves implemented? Do you keep separate model parameters per cluster, or fine-tune a shared small model? If shared, how do you justify the independence assumptions when entropy-based cluster assignments also change during adaptation?
Reviewer_qMWXOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies online LLM serving with costly small-model adaptation. At each round, the learner chooses to serve with a large model, serve with a small model, or serve with the large model while fine-tuning the small model on the current request cluster. The authors establish lower bounds and propose an algorithm combining optimistic adaptation, convergence detection, and confidence-based serving, with an instance-adaptive regret bound. Experiments with Qwen3.5-Plus and Qwen3-4B show the lowest average loss on MATH and parity with always-cloud on XSum.

Critical strengths
  1. The paper provides a solid theoretical treatment: lower bounds for both the unrestricted and the budget-restricted classes, and an instance-adaptive upper bound expressed through the useful-adaptation complexity B⋆B^\star.
  2. The algorithm is modular and interpretable, and the per-cluster analyses (Fig. 2, Table 3) clearly illustrate where adaptation helps.
Critical weaknesses
  1. Motivation of the setting. The importance of the setting could be made more persuasive. In particular, the method performs online test-time training of the small model during serving (GRPO with 8 generations per prompt), and it is not fully clear whether this is realistic for edge deployment in terms of compute, latency, and cost. A concrete deployment scenario would help show that the formulation reflects a practical need rather than mainly a setting for applying bandit tools.

  2. Analysis of test-time adaptation. The effects of the adaptation itself are not analyzed. Since a single shared LoRA is updated across clusters, adaptation on one cluster may affect others or degrade general capability (catastrophic forgetting). This is not evaluated: the MATH stream repeats the same training problems for three epochs, and no held-out or general benchmark is reported.

  3. Ablations of individual components. More component-wise ablations would strengthen the paper. In particular, a routing-only baseline (per-cluster LCB between local and cloud without training) would help isolate the contribution of adaptation, since on MATH the small model already outperforms the cloud on the easiest cluster before training. Sensitivity to cc, λ\lambda (set differently per dataset), and ω\omega, as well as results over multiple seeds, would also be helpful.

  4. Gap between theory and experiments. The analysis assumes independent per-cluster learning curves, exogenous cluster identities, and a linear-then-flat curve. In the experiments, a shared LoRA is updated, clusters are defined by the evolving model's entropy, updates are batched, and an additional consecutive-detection rule is used. The λ\lambda condition in Lemma 1 also appears to require knowledge of αk\alpha_k. A discussion of how far the guarantees carry over to the actual system would be valuable.

Questions for authors
  1. How would this setting be deployed in practice? Who performs the fine-tuning, on what hardware, and at what cost relative to cloud inference?
  2. Test-time fine-tuning may decrease as well as increase performance. Could it lead to catastrophic forgetting on other clusters or on general tasks, and how would the method detect or handle this?
  3. On XSum, where B⋆=0B^\star = 0, the method spends 40.7% of rounds on training, yet its loss is within 0.001 of always-cloud. Could the authors clarify whether Table 1 includes the adaptation cost cc?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.