- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies an online model-serving problem in edge-cloud collaborative inference, where the system must jointly decide whether to serve incoming requests using a cloud-based large language model or a local small language model, and whether to fine-tune the small model to improve its future performance. The authors propose an adaptive algorithm that combines convergence detection, optimistic adaptation, and confidence-based model selection to balance inference quality, serving costs, and fine-tuning overhead. They also provide theoretical regret bounds to characterize the effectiveness of their approach. Experiments using Qwen3.5-Plus and Qwen3-4B on the MATH and XSum datasets show that the proposed method achieves lower average loss on MATH while maintaining performance close to cloud-only serving on XSum.
Critical strengths
- The paper goes beyond traditional model routing by considering both inference decisions and the cost of fine-tuning the local model. This provides an interesting perspective on balancing model performance and serving costs over time.
- The authors provide regret bounds to support their proposed algorithm and evaluate it on two datasets using real LLMs. The results show some improvements over existing baselines, particularly on MATH, which helps demonstrate the potential effectiveness of the approach.
Critical weaknesses
- The theoretical analysis assumes that the small model's loss decreases along a relatively simple learning curve and eventually converges. However, real-world online fine-tuning is much more complex, especially when different task clusters share the same model parameters. It is unclear whether the theoretical guarantees still hold under more realistic training conditions.
- The experiments only cover two datasets and one pair of models. The improvement on MATH is relatively small (0.338 vs. 0.346 for Always Cloud), while the method does not outperform Always Cloud on XSum. More extensive experiments, stronger baselines, and ablation studies are needed to demonstrate the effectiveness and generalizability of the proposed approach.
- Although the paper focuses on reducing serving and adaptation costs, the experiments mainly use simplified loss and cost metrics. Important factors such as actual fine-tuning time, GPU resource consumption, communication overhead, and end-to-end inference latency are not adequately evaluated. On XSum, the method still spends 40.7% of requests on training even though the authors claim that adaptation provides no benefit, which further raises concerns about its practical efficiency.
- I found the paper quite difficult to follow. The Introduction lacks citations and contains too many short paragraphs, making the background and motivation unclear. There is also no system architecture or motivation figure to help readers quickly understand the problem and proposed solution. The experimental results are presented with relatively limited visual analysis, making it harder to evaluate the main contributions and understand why the method works.
Questions for authors
- Could the authors explain why continuous decisions about fine-tuning are necessary in real-world edge-cloud collaborative inference systems? In practice, these systems often focus on cooperation between large and small models rather than continuously adapting the smaller model. What specific deployment scenario motivates this problem, and why would the proposed approach be more suitable than conventional model routing with periodic fine-tuning?
- The theoretical analysis assumes that the small model's loss follows a decreasing-then-converging learning curve. However, actual online fine-tuning may produce unstable performance, especially when different task clusters share the same model parameters. How do the authors justify these assumptions, and to what extent would the theoretical guarantees remain valid under more realistic training conditions?
- In the XSum experiments, the proposed method allocates 40.7% of requests to training, even though the authors state that adaptation provides no benefit. Could the authors explain why the algorithm continues training so frequently in this case? How is the training cost reflected in the reported average loss of 0.909, which is almost identical to the Always Cloud baseline of 0.908?
- The improvement over Always Cloud on MATH appears relatively small (0.338 vs. 0.346), while no improvement is observed on XSum. Could the authors clarify whether these improvements are statistically significant and provide stronger evidence that the proposed joint optimization is more effective than simpler routing or periodic fine-tuning strategies?
- The proposed approach uses the local model's average token entropy to cluster incoming requests. Does this require the local model to generate a response before the routing decision is made? If so, how are the additional inference latency and computational costs accounted for? Also, how much overhead does online fine-tuning introduce when inference and training share limited edge resources?