acceptodds
Under review as a conference paper at ICLR 2027

Cloud on the Clock: Online LLM Routing with Early Edge Takeover

Abstract

An edge–cloud LLM system routes each query to either a local edge model or a cloud model to balance answer quality against user waiting time. In practice, many routers are trained offline and do not adapt to new service feedback. Some recent routers learn model choices online, but cloud exploration can incur prolonged waiting and highly variable utility feedback, making exploration less efficient. The deadline-only service runs the selected model until delivery or a final deadline, which cannot resolve the difficulty of exploring cloud and learning the quality-delay trade-off under such variability. Therefore, we propose an online edge-cloud routing framework that combines early edge takeover with kernel upper confidence bound (UCB) routing based on semi-adaptive (SA) confidence calibration. During service, the framework interrupts a cloud request that remains unfinished at a threshold and starts an edge generation attempt. After each round, terminal quality–delay feedback updates the selected initial route's weighted kernel utility estimate, while the corresponding prediction error calibrates the SA confidence bonus. Both updates guide UCB routing for the next query. When cloud delays exhibit heavy tails, we analyze the framework using a conditional Pareto model motivated by Pareto execution-time models in cloud straggler analysis. Our analysis establishes conditions for vanishing average regret and for a strictly smaller variance-dependent cumulative regret bound than deadline-only service. Comparing with the strongest baselines on six benchmarks, our method achieves a mean relative utility gain of 27.9% and reduces p95 latency by 36.5%.

Primary area:
reinforcement learning
Keywords:
LLM Routing, Online Reinforcement Learning, Edge-Cloud Collaboration, Heavy-tail Latency

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rGnzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates online routing between edge and cloud LLMs to balance service latency and utility. To achieve this, the authors propose an online edge-cloud routing framework that implements a confidence-based interruption in the cloud process and initiates an early edge takeover. This framework uses a conditional Pareto model to optimize the objective. Experiments are conducted with six benchmarks and show a mean relative utility gain of 27.9% and a 36.5% reduction in p95 latency.

Critical strengths
  1. This problem is interesting and somehow practical. It considers service latency, heavy tails, and the quality-delay trade-off.
  2. The ablation study verifies the SA calibration from the takeover.
  3. The appendix is informative.
Critical weaknesses
  1. The baseline methods cannot independently evaluate the specific contributions of the proposed approach. The performance gain is mainly from the "early takeover" mechanism. When running these baselines (such as RouteLLM, GAPG, and PAK-UCB), the strategies employed were limited to "deadline-only" or "final-moment" takeover approaches. To conduct a rigorous comparative study, these baselines should use the same early takeover mechanism to determine whether SA-calibrated kernel-UCB still outperforms them.

  2. The conclusion regarding "strictly smaller regret" compares upper bounds on regret rather than the actual regret itself. Corollary 1 compares two upper-bound functions, RN(vˉ0)R_N(\bar{v}_0) and RN(vˉs)R_N(\bar{v}_s), where the regret for each service is measured against its respective oracle. This does not imply that the takeover strategy yields lower actual regret or superior realized utility. The convergence rate of this bound, O((log⁡N)d+1/N)O((\log N)^{d+1}/\sqrt{N}), depends on the embedding dimension dd, and the bound is quite loose for values ​​of dd encountered in practical applications.

  3. The assumptions made in the paper are strong and lack empirical validation. The analysis relies on the following conditions: a) a Pareto distribution with a tail index between 1 and 2 and a bounded scale parameter; b) a normalized utility function realizable within the unit ball (≤ 1) of RKHS; and c) the existence of a uniform upper bound on marginal service times. Although the paper justifies the use of the Pareto distribution by citing other systems, it does not assess the goodness-of-fit against actual cloud latency data.

  4. The core data presented somewhat exaggerates the magnitude of the performance gains. The average relative gain of 27.9% is calculated by normalizing against the absolute utility of the strongest baseline. However, for several tasks, the utility values ​​of this baseline are close to zero or negative (e.g., -0.0603 versus -0.0098 for the MATH-Lighteval task), meaning the relative gain figures are heavily influenced by extremely small denominators. In addition, the performance of some baselines is very close in some cases, implying that the specific "strongest baseline" varies depending on the task.

Questions for authors
  1. How do baseline methods perform when the same early takeover is applied?
  2. Is it possible to test the Pareto hypothesis against the recorded cloud latency data? How well does it fit?
  3. If the time condition is fulfilled, is it possible to assess whether the act of takeover actually reduces the regret ultimately experienced, or merely lowers the upper limit of potential regret?
  4. What are the implications when cancellation and restart costs are non-zero?

I will consider improving the score if the response solves my concerns.

3 comments

Thank you for your sincere review and constructive feedback. We address an important practical problem: useful LLM answers within acceptable waits when cloud is slow, combining on-demand edge fallback with routing learned from quality and total waiting.

(1) To Question 1 and Weakness 1: All six methods share models, s=30s=30 s, a 60s deadline and task-specific utility weights. Ours leads in mean utility on 4/6 tasks (Table R1). Its six-task mean utility exceeds PAK-UCB and AlwaysCloud by 3.12% and 9.66%. On TriviaQA, waiting cost savings do not offset the rating loss. Versus AlwaysCloud, mean waiting falls 49.59%, rating drops 0.8622 to 0.5239, and utility is 11.67% lower. For a chemist-name query, Ours selects edge, which is wrong in 0.163 s; cloud correctly answers “Otto Hahn” in 2.051 s. On MBPP, extra waiting cost slightly exceeds the rating gain. Both answers pass a last-digit coding request; Ours selects cloud while PAK-UCB selects edge.

Table R1. Mean utility.

TaskAlwaysEdgeAlwaysCloudRouteLLMGAPGPAK-UCBOurs
MATH-Lighteval−0.1340−0.0532−0.0849−0.0854−0.0351−0.0098
MATH-500−0.1429−0.1693−0.1172−0.1529−0.0980−0.0922
MMLU0.53040.53120.53120.53010.55910.5601
TriviaQA0.18120.23530.18200.21890.20800.2079
SQuAD 1.10.52570.55620.52570.54410.55060.5674
MBPP0.33470.34140.33470.33690.34830.3474

(2)To Question 3 and Weakness 2: Corollary 1 gives a “strictly tighter regret bound.” Each oracle follows its service rule and selects the highest-mean-utility route; regret measures loss against it. Theorem 1 uses kernel information ΓN\Gamma_N. Its worst-case rate O((log⁡N)d+1/N)O((\log N)^{d+1}/\sqrt N) can be loose in high dimensions. Corollary 1 reduces the variance factor at common ΓN\Gamma_N, preserving the order. For four queries and two routes, Appendix F.2 (F.1) gives ΓN=O(log⁡N)\Gamma_N=O(\log N) and average regret O(log⁡N/N)O(\log N/\sqrt N). Appendix F.3 computes regret from known conditional means and actual routes. All four settings show lower average regret:

Table R2. Average regret RN/NR_N/N.

NTail indexNo takeoverTakeoverReduction
1,5001.250.026880.0238711.20%
1,5001.750.092800.083799.72%
10,0001.250.022810.0199612.48%
10,0001.750.049840.047305.10%

(3) To Question 2 and Weakness 3: We use 7,000 untruncated cloud records, including 366 above 60 s. Kolmogorov–Smirnov (KS) distance selects the tail start and measures the largest gap between empirical and fitted cumulative probabilities.

Table R3. Pareto fits; smaller KS is better.

TaskStart (s)Tail nTail indexTail KSFull KS
MATH-Lighteval5.8017510.8720.03490.2089
MATH-50032.293761.3980.05730.1736
MMLU5.5066010.8100.04950.1816
TriviaQA2.7024511.1220.04520.2609
SQuAD 1.12.8603391.6060.04910.3236
MBPP19.7832260.9190.09150.1954

Tail deviations are 3.49–9.15 percentage points. These are marginal fits; Section 3.1 assumes query-conditional distributions. MATH-500, TriviaQA and SQuAD have fitted indices in (1,2); the others are below 1. Under original Table 1's settings, Ours leads on all six tasks, motivating extensions of our theoretical guarantees to broader latency distributions. Appendix F.2 meets the assumptions: Pareto indices 1.25/1.75, edge time 5 s, takeover 30 s, deadline 60 s; both normalized means' squared RKHS norms are below 0.77.

(4) To Question 4: We add 0, 0.5, 1, 2 or 5 s per takeover beyond the included edge initialization and generation time. All methods share the original task-specific thresholds ss, models, utility weights and a 60-s deadline within each task.

Table R4. Six-task mean utility.

Extra delay (s)AlwaysEdgeAlwaysCloudRouteLLMGAPGPAK-UCBOurs
00.215850.262220.256160.253930.266550.27756
0.50.215850.257180.253640.249770.264850.27300
10.215850.252140.251120.246310.262370.27093
20.215850.242070.246070.238110.259480.26421
50.215850.211830.230160.216780.244280.25208

Ours leads at all five delays (+1.82%–4.13% over PAK-UCB, the strongest baseline at each delay). For unchanged answers still delivered on time, extra delay τ\tau reduces utility by βτ\beta\tau. If 4max⁡x+,de+τmax⁡≤s≤Dmax⁡/24\max\\{x_+,d_e+\tau_{\max}\\}\le s\le D_{\max}/2, Proposition 1 retains variance reduction ≥ks>0\ge k_s>0. Direct edge has no switching cost; its bound stays de2/4d_e^2/4. Both routes retain their original conservative variance bounds. With Assumption 2 for both means and v0>de2/4v_0>d_e^2/4, the strict regret-bound comparison holds. Appendix F.2's s=30,de=5s=30,d_e=5 s allow τmax⁡≤2.5\tau_{\max}\le2.5 s; larger delays require a new Lemma B.1 bound.

We welcome further discussion. If our evidence addresses your concerns, we would appreciate reconsideration of your assessment. Thank you

Thank you for the detailed response and the additional experiments. While several of my initial questions have been addressed, the updated results introduce several new concerns. Overall, while the rebuttal and additional runs provide transparency, the demonstrated gains are substantially narrower than originally claimed. Consequently, I am maintaining my score at 2.

  1. Once takeover is applied to the baselines, Table R1 shows an average utility gain across the six tasks of only 3.12% over PAK-UCB (ranging from 1.82% to 4.13% in Table R4), down sharply from the 27.9% claimed in the abstract. Furthermore, the margin is virtually zero on MMLU, TriviaQA, and MBPP, with more than half of the mean improvement driven solely by MATH-Lighteval.

  2. Table R1 reports scores for "Ours" of 0.2079 on TriviaQA and 0.5674 on SQuAD, whereas the main paper reports 0.2799 and 0.5799, respectively. Additionally, while Table R4 at delay 0 aligns with the paper's macro-average (0.2776), it deviates from the Table R1 mean (0.2635). The baseline numbers also differ across these tables.

  3. Appendix D notes that configurations in Table 3 were selected "on the evaluation data, including searches based on gains over baselines," yet no baseline tuning procedure is detailed. Please report results on a proper held-out split with fairly matched baseline tuning. Moreover, Table 1 and Tables R1/R4 lack standard deviations, and replaying the same cache across the 15 request orders fails to capture realistic variance in latency and response quality.

  4. Table R2 evaluates only four queries and two routes, making the setup effectively tabular. Additionally, the regret certificate in Table 9 yields bounds of ~4.7 to 5.0 per round, barely below the trivial cap of 5.8, while empirical realized regret is only 0.02 to 0.09. This loose gap renders the theoretical bound uninformative in practice.

  5. Three out of the six tasks exhibit fitted tail indices below 1, violating the assumed (1, 2) regime, while the full-distribution KS distances range from 0.17 to 0.32. Combined with the unmet timing condition, the theoretical guarantees fail to cover the majority of the empirical settings.

Thank you again for taking the time to review our response and examine the additional experiments. Your comments helped us identify where the paper needs clearer comparisons and more precise claims. We are grateful for this guidance and will use it to improve the revision.

(1) Why the reported gain changes. The paper and Table R1 use different takeover settings. In the paper, our method uses task-specific early takeover, while the baselines use deadline-only or last-minute takeover. In R1, all six methods use the same 30-s threshold. The percentages also use different comparisons. The paper’s 27.9% averages per-task relative gains over each task’s strongest original baseline, using its absolute utility as the denominator. R1’s 3.12% is the increase in six-task mean utility over PAK-UCB, from 0.25550 to 0.26348. Under this common rule, our method ranks first on four tasks. As you observed, MATH-Lighteval accounts for 52.9% of the net gain over PAK-UCB. SQuAD and MATH-500 also improve, while the other three differences are small. We will clarify what the 27.9% measures in the abstract.

(2) Why the utility values differ. Only two thresholds change for our method. R1 uses 30 s on both TriviaQA and SQuAD, giving utilities of 0.2079 and 0.5674. The paper and R4 at zero overhead use 4.75 s and 15.985 s, giving 0.2799 and 0.5799. The other four values are identical, so these two changes fully explain the difference between 0.2635 and 0.2776. The baselines in R4 also use the task-specific early thresholds. Across its five overhead levels, our six-task mean exceeds PAK-UCB’s by 1.82%–4.13%.

(3) Tuning and uncertainty. Appendix D reports that evaluation data informed the original parameter choices. Algorithm 1 updates routes online at each fixed threshold. The threshold itself is chosen by comparing mean utility across runs. In a separate sensitivity check, we used 232 MATH history queries disjoint from the target queries and tested three values of one parameter per adaptive method. Our method ranked second in six-task mean utility (0.26953 vs 0.27069 for PAK-UCB), ahead of the other four baselines. Some base settings came from the earlier exploration, so a fully independent test of the tuning procedure remains open.

We also computed mean ± SD for all methods in Table 1 and R1/R4. In R1, our method scores 0.26348 ± 0.00524 versus 0.25550 ± 0.00416 for PAK-UCB and leads in 13 of 15 paired replays. These SDs measure variation across request orders and learner runs on the same recorded outcomes. Measuring latency and answer variability across service sessions requires independent generations, which are still missing for the full six-task comparison.

(4) Tightness of the regret bound. We agree that Table 9’s certificates are loose at 1,500 rounds (4.70–4.99 per round, versus a ceiling of 5.8 and measured average regret of 0.02–0.09). Corollary 1 gives a strictly tighter variance-dependent bound for the same learner under takeover, with each service measured against its own oracle. Four queries and two routes let us compute both oracles exactly. Measured average regret falls by 5.10%–12.48% in all four controlled settings. The practical tightness of the bound in high dimensions remains open.

(5) Scope beyond Pareto delays. Pareto gives an explicit lower bound on the variance reduction from takeover. Lemma B.1, Eq. (B.3), expresses that reduction in terms of truncated delay moments. If another delay model satisfies the lemma and yields a uniformly positive lower bound over queries and histories, Eq. (C.9) gives the same strict regret-bound comparison under Corollary 1’s remaining learning and variance conditions.

Three fitted marginal tail indices fall below 1, full-distribution Kolmogorov–Smirnov distances range from 0.17 to 0.32, and some settings fall outside the timing conditions. In paired replay of 7,000 cloud-initiated requests, takeover reduced within-task delay variance by 40.6%–98.1%. The empirical lower bound in (B.3) was positive on four tasks, including MMLU and MBPP, both with tail indices below 1. These task-level measurements support checking delay moments directly in more services. Applying the guarantee to a particular service requires verifying the uniform conditional bound.

Thank you again for helping us improve the paper.

Reviewer_dDSrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper considers LLM serving in an edge computing system. When a request arrives, the server needs to choose between (i) serving the request using the edge server, which is faster but less accurate; or (ii) forwarding the request to a faraway cloud server, which is more accurate but takes more time. The system also has a built-in early takeover that invokes the edge server if it has been waiting for the cloud server for s time without getting a response.

The learning problem is to figure out whether one should use the edge server or the cloud server to serve a request. This is a standard bandit problem and the paper proposes a UCB algorithm to solve it.

Critical strengths

The problem has clear real world utility.

Critical weaknesses

The paper has two major contributions: (i) the edge-cloud system; (ii) the UCB algorithm. Neither of them appears to be novel and the connection between them appears to be weak. See detailed comments below:

Novelty of the edge-cloud system: The choice between edge and cloud is a well-studied problem. The trick of takeover is not new either. Perhaps the only thing that differentiates the work from the literature of edge/cloud service is that this paper focuses on LLM serving. However, it is not clear if "LLM" plays a meaningful role in the system and algorithm design. It seems one can replace "LLM" with any other kinds of AI workload, or even generic computation workload, and there would be no change in the system and algorithm design. The related work primarily surveys LLM serving only. It is recommended to broaden the scope of the related work to include pre-LLM studies on edge computing, handling of stragglers in distributed system literature, etc., as the paper clearly borrows a lot of mature ideas from this literature.

Novelty of the UCB algorithm and its connection to the edge-cloud system: It is not clear if there is anything new in the UCB algorithm. It is fundamentally a standard contextual bandit problem: In each step, the system sees a job with context z_n, and it chooses a binary action a_n from {edge, cloud}, and observes the reward in the form of U_n. This is a problem that is readily solvable by existing algorithms. It is not clear if the paper is doing anything beyond existing literature. It also seems that many details, such as the takeover mechanism, does not have a meaningful influence on the algorithm design.

Minor comments:

  1. The paper seems to assume that the threshold s(z_n) used to execute early takeover is given. In practice, this parameter plays an important role. The paper does not discuss how to optimize s(z_n)

  2. Another common trick to address straggler is to use redundancy. In the context of this paper, it means the following: When a job arrives, the system will allocate it to BOTH the edge server and the cloud server. If the cloud server responds within time d, then the system uses the cloud server's answer. Otherwise, it uses the edge server's answer. With a properly chosen d, it seems this naive solution would be better than the paper's solution.

Questions for authors

Does your paper contain a learning algorithm that can learn the optimal s(z_n)?

3 comments

Thank you for your sincere feedback. We address an important practical problem in LLM serving: delivering useful answers within acceptable waiting times when the cloud is slow. We provide on-demand edge fallback and continually improve routing by learning from final answer quality and total waiting.

(1) To Major Weakness 1: LLM serving shapes our utility target and takeover design: switching models can change answer quality as well as total waiting. Sections 3.1–3.2 use query-dependent utility to judge whether cloud quality gains justify waiting or edge already suffices. Takeover starts edge generation. We reserve time for it and assign its rating, prior cloud waiting and edge execution time to the initial cloud decision. The router learns the value of trying cloud with fallback.

MAUI (Cuervo et al., 2010) studies energy-saving code offloading; Dolly (Ananthanarayanan et al., 2013) replicates tasks to reduce stragglers; TMO (Yuan et al., 2025) studies inference placement and modality selection. We will link these studies to our service and feedback design in Section 2.

(2) To Major Weakness 2: Hu et al. (2025) learn model ratings with separate kernel predictors. We learn complete-service utility: both cloud completion and takeover update the initial cloud route with the final rating and all waiting. Jun and Kim (2024) provide weighted regression and residual-based SA confidence. Terminal utility updates the route predictor; its prediction error updates shared scale qnq_n. The next choice uses $ a_{n+1}\in\arg\max_a[\hat g_n(a,z_{n+1})+\sqrt{q_n}\,u_{n+1}(a,z_{n+1})], $ where aa is edge or cloud and uu is kernel uncertainty.

Under Section 3.1's conditional Pareto model and timing conditions, Lemma B.1 and Proposition 1 account for added random edge execution and prove net delay-variance reduction. This yields utility-variance bound vˉs\bar v_s, allowing rating–delay dependence. Appendix A verifies confidence coverage; Appendix C.4 controls estimation error in residuals using a weighted ridge-regression identity, giving $ q_t\le c_v(\bar v_s/B^2)\Gamma_N+2\lambda+2H. $ Theorem 1 adds kernel uncertainty to bound regret. With common kernel/learning settings, utility weights and deadline, both service means satisfying RKHS realizability, and v0>de2/4v_0>d_e^2/4, Corollary 1 gives a strictly tighter regret bound $ \mathcal R_N(\bar v_s)<\mathcal R_N(\bar v_0). $

(3) To Minor Comment 1 and the Question: We combine threshold online routing with adaptive, utility-guided threshold fine-tuning for each task. Corollary 1 gives the strict bound comparison under its conditions; Table 2 shows utility gains on all six tasks over SA without takeover.

Using 30 s as reference, we reinitialize Algorithm 1 at each candidate ss, learn routes from final ratings and total waiting, and refine the search by mean utility. On calibration records, the best tested thresholds are 4.75 s for TriviaQA and 15.985 s for SQuAD, improving utility over 30 s by 34.64% and 2.20%.

An extension is that we can treat direct edge and cloud with different thresholds as actions, learning routes and waiting budgets from terminal feedback. Under the same constraints, this set includes fixed-threshold policies, so its optimal expected utility is at least as high; finite-round guarantees need further analysis.

(4) To Minor Comment 2: Our original method starts a second model only when takeover is needed, reducing model starts by 42.0%–49.6% relative to running both models for every request. We test six tasks at eight common caps of 1.00–2.00 model starts per request. Both methods share models, evaluation queries, task utility weights and a 60-s deadline; ss and parallel threshold dd are chosen on a separate calibration set. Selective parallel execution chooses edge-only, cloud-only or dual execution by query. Our Budget-gated SA extension permits takeover while extra-start budget remains; otherwise it waits for cloud until the deadline. Table R2 compares equal actual starts on MATH-500 (s=d=15s=d=15 s). Mean waiting falls by 31.60% and 30.83%, respectively.

Starts/request (both)Parallel utilityBudget-gated SA utilityGainMean delay: parallel → SA (s)
1.05−0.2431−0.1516+37.63%14.391 → 9.843
1.10−0.2079−0.1168+43.81%13.649 → 9.441

Across all 48 settings, Budget-gated SA leads in 14 and parallel in 34; at cap 2, parallel leads on all six tasks, which means that on-demand execution thus helps in some settings with limited model starts.

We welcome further discussion and would be happy to address any remaining questions. If these replies and additional results address your concerns, we would be grateful if you could reconsider your assessment and rating. Thank you again.

The rebuttal unfortunately does not address my concerns and hence I am unable to raise the score.

Thank you again for taking the time to review our response and examine the additional experiments so carefully. Your comments have helped us see where our comparisons need clearer explanation and where our claims need stronger evidence. We are grateful for your guidance and will use it to strengthen both the analysis and its presentation as we revise the paper.

Reviewer_zMGAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies online routing between an edge LLM and a cloud LLM using a utility that balances answer quality and waiting time. It combines early edge takeover with kernel prediction and semi-adaptive confidence calibration. The paper derives regret guarantees under specified latency and realizability assumptions and evaluates the method through recorded-service replay on six benchmarks and a controlled Pareto model.

Critical strengths
  1. Early takeover addresses the cost of waiting for slow cloud responses. Treating cloud initiation and possible takeover as one service action gives the router a consistent target to learn from terminal feedback.

  2. The experiments separate confidence calibration from takeover and report how quality and waiting time contribute to utility. These comparisons help explain where the observed gains come from.

Critical weaknesses
  1. The learning method builds on kernel UCB and semi-adaptive calibration. The paper needs to distinguish its new algorithmic contribution more clearly from these existing components.

  2. Table 1 changes both the router and the takeover policy. The comparison therefore does not isolate the routing method's contribution, despite the internal ablations in Table 2.

  3. The regret result concerns learning under a prescribed takeover rule. It does not provide a condition for when takeover improves expected utility after accounting for answer-quality loss.

  4. The utility and latency gains use different comparisons, making it difficult to assess the full system's improvement against the same baseline.

Questions for authors
  1. Which learning updates or theoretical results are new beyond the cited kernel-routing and confidence-calibration methods?

  2. Does the routing advantage remain when PAK-UCB and AlwaysCloud use the same early-takeover rule?

  3. How should takeover time depend on answer quality and waiting cost? When does takeover improve actual utility?

  4. Can you report quality, utility, and tail latency for the complete method and baselines under matched service conditions?

2 comments

Thank you for your sincere review. We address an important practical problem in LLM serving: delivering useful answers within acceptable waiting times when the cloud is slow. Our framework combines on-demand edge takeover with online learning from final answer quality and total waiting time.

(1) To Weakness 1 and Question 1: Building on Hu et al.'s (2025) separate kernel predictors for ratings, we learn complete-service utility. Cloud completion and edge takeover both update the initial cloud route with the final rating and all waiting. Using Jun and Kim's (2024) weighted regression and residual-based SA confidence, utility updates the route predictor; its prediction error updates the shared scale for UCB decisions.

Under section 3.1's conditional Pareto model and timing conditions, Lemma B.1 and Proposition 1 account for random edge execution and prove lower delay variance, yielding utility-variance bound vˉs\bar v_s with rating–delay dependence. Appendix A proves confidence coverage; Appendix C.4 uses a weighted ridge-regression identity to control estimation error in residuals $ q_t\le c_v(\bar v_s/B^2)\Gamma_N+2\lambda+2H. $ Theorem 1 combines this with kernel uncertainty to bound regret. With common kernel/learning settings, utility weights and deadline, RKHS realizability for both means, and v0>de2/4v_0>d_e^2/4, Corollary 1 gives a strictly tighter regret bound $ \mathcal R_N(\bar v_s)<\mathcal R_N(\bar v_0). $

(2) To Question 2 and Weakness 2: All methods share s=30s=30 s, a 60s deadline and task-specific waiting-cost weights β\beta; Ours leads in utility on 4/6 tasks (Table R1). Six-task mean utility gains are 3.12% over PAK-UCB and 9.66% over AlwaysCloud.

Routing helps most when some queries suit fast edge answers and others justify waiting for cloud quality. On TriviaQA, Ours cuts mean waiting by 49.59%, but rating falls from 0.8622 to 0.5239, giving 11.67% lower utility than AlwaysCloud. For a chemist-name query, selected edge is wrong in 0.163 s; cloud answers “Otto Hahn” correctly in 2.051 s. On MBPP, extra waiting cost slightly exceeds the rating gain: Ours selects cloud for a last-digit coding request, while PAK-UCB selects edge. Both pass, taking 6.140 s and 0.428 s.

(3) To Question 3 and Weakness 3: In theory, from Sections 3.2 and 4.1, for a cloud request unfinished at ss, let ws(z)w_s(z) be expected waiting saved and ℓs(z)\ell_s(z) expected rating lost for query zz. Takeover improves expected utility if βws(z)>ℓs(z)\beta w_s(z)>\ell_s(z). Section 3.1's model/timing conditions and Appendix B.3 give ws(z)≥s/2−dew_s(z)\ge s/2-d_e, so ℓs(z)<β(s/2−de)\ell_s(z)<\beta(s/2-d_e) suffices. In experiments, online utility learning guides adaptive task-level threshold fine-tuning. At each candidate ss, routes learn from ratings and waiting under common settings. Mean utility guides finer search and the best tested choice. Joint learning could use direct edge and cloud with different thresholds as actions, adapting waits to queries. Under the same constraints, it includes fixed-threshold policies, so its optimal expected utility is at least as high.

(4) To Question 4 and Weakness 4: Table R1 reports mean quality, utility and p95 total service latency under matched service conditions. Ours improves quality and utility over AlwaysEdge and RouteLLM on all six tasks. It exceeds GAPG in utility on five and lowers p95 on four. Versus PAK-UCB, rating and utility improve on four tasks each; versus AlwaysCloud, utility improves on five and p95 falls on all six (1.62%–49.13%).

Table R1. Matched-service results; bold marks the best value.

DatasetMetricAlwaysEdgeAlwaysCloudRouteLLMGAPGPAK-UCBOurs
MATH-LightevalQuality ↑0.43930.86870.52530.70280.65810.6904
Utility ↑-0.1340-0.0532-0.0849-0.0854-0.0351-0.0098
p95 (s) ↓11.24140.88313.56338.25830.69632.295
MATH-500Quality ↑0.40800.84600.48320.62510.60870.6863
Utility ↑-0.1429-0.1693-0.1172-0.1529-0.0980-0.0922
p95 (s) ↓11.02040.48214.38638.70333.56239.824
MMLUQuality ↑0.54530.87800.61040.70560.72610.7214
Utility ↑0.53040.53120.53120.53010.55910.5601
p95 (s) ↓0.41230.3836.40624.72428.19626.938
TriviaQAQuality ↑0.19610.86220.19750.72630.52790.5239
Utility ↑0.18120.23530.18200.21890.20800.2079
p95 (s) ↓0.16913.0420.17210.4346.5436.634
SQuAD 1.1Quality ↑0.54700.91050.54700.75780.73990.7572
Utility ↑0.52570.55620.52570.54410.55060.5674
p95 (s) ↓0.4946.7780.4945.1474.9855.025
MBPPQuality ↑0.35800.56000.35800.47670.46610.4679
Utility ↑0.33470.34140.33470.33690.34830.3474
p95 (s) ↓5.26634.7795.26633.48933.11833.346

We welcome further discussion to address any question. If our evidence addresses your concerns, we would appreciate reconsideration of your assessment. Thank you again

Thank you again for your thoughtful review. Your question prompted us to compare answer quality, utility, and tail latency for our method and all five baselines under the same takeover rule. We included these results in our earlier response. When you have a chance to look them over, we would welcome your thoughts and would be happy to discuss any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.