- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates online routing between edge and cloud LLMs to balance service latency and utility. To achieve this, the authors propose an online edge-cloud routing framework that implements a confidence-based interruption in the cloud process and initiates an early edge takeover. This framework uses a conditional Pareto model to optimize the objective. Experiments are conducted with six benchmarks and show a mean relative utility gain of 27.9% and a 36.5% reduction in p95 latency.
Critical strengths
- This problem is interesting and somehow practical. It considers service latency, heavy tails, and the quality-delay trade-off.
- The ablation study verifies the SA calibration from the takeover.
- The appendix is informative.
Critical weaknesses
-
The baseline methods cannot independently evaluate the specific contributions of the proposed approach. The performance gain is mainly from the "early takeover" mechanism. When running these baselines (such as RouteLLM, GAPG, and PAK-UCB), the strategies employed were limited to "deadline-only" or "final-moment" takeover approaches. To conduct a rigorous comparative study, these baselines should use the same early takeover mechanism to determine whether SA-calibrated kernel-UCB still outperforms them.
-
The conclusion regarding "strictly smaller regret" compares upper bounds on regret rather than the actual regret itself. Corollary 1 compares two upper-bound functions, and , where the regret for each service is measured against its respective oracle. This does not imply that the takeover strategy yields lower actual regret or superior realized utility. The convergence rate of this bound, , depends on the embedding dimension , and the bound is quite loose for values of encountered in practical applications.
-
The assumptions made in the paper are strong and lack empirical validation. The analysis relies on the following conditions: a) a Pareto distribution with a tail index between 1 and 2 and a bounded scale parameter; b) a normalized utility function realizable within the unit ball (≤ 1) of RKHS; and c) the existence of a uniform upper bound on marginal service times. Although the paper justifies the use of the Pareto distribution by citing other systems, it does not assess the goodness-of-fit against actual cloud latency data.
-
The core data presented somewhat exaggerates the magnitude of the performance gains. The average relative gain of 27.9% is calculated by normalizing against the absolute utility of the strongest baseline. However, for several tasks, the utility values of this baseline are close to zero or negative (e.g., -0.0603 versus -0.0098 for the MATH-Lighteval task), meaning the relative gain figures are heavily influenced by extremely small denominators. In addition, the performance of some baselines is very close in some cases, implying that the specific "strongest baseline" varies depending on the task.
Questions for authors
- How do baseline methods perform when the same early takeover is applied?
- Is it possible to test the Pareto hypothesis against the recorded cloud latency data? How well does it fit?
- If the time condition is fulfilled, is it possible to assess whether the act of takeover actually reduces the regret ultimately experienced, or merely lowers the upper limit of potential regret?
- What are the implications when cancellation and restart costs are non-zero?
I will consider improving the score if the response solves my concerns.
3 comments
Thank you for the detailed response and the additional experiments. While several of my initial questions have been addressed, the updated results introduce several new concerns. Overall, while the rebuttal and additional runs provide transparency, the demonstrated gains are substantially narrower than originally claimed. Consequently, I am maintaining my score at 2.
-
Once takeover is applied to the baselines, Table R1 shows an average utility gain across the six tasks of only 3.12% over PAK-UCB (ranging from 1.82% to 4.13% in Table R4), down sharply from the 27.9% claimed in the abstract. Furthermore, the margin is virtually zero on MMLU, TriviaQA, and MBPP, with more than half of the mean improvement driven solely by MATH-Lighteval.
-
Table R1 reports scores for "Ours" of 0.2079 on TriviaQA and 0.5674 on SQuAD, whereas the main paper reports 0.2799 and 0.5799, respectively. Additionally, while Table R4 at delay 0 aligns with the paper's macro-average (0.2776), it deviates from the Table R1 mean (0.2635). The baseline numbers also differ across these tables.
-
Appendix D notes that configurations in Table 3 were selected "on the evaluation data, including searches based on gains over baselines," yet no baseline tuning procedure is detailed. Please report results on a proper held-out split with fairly matched baseline tuning. Moreover, Table 1 and Tables R1/R4 lack standard deviations, and replaying the same cache across the 15 request orders fails to capture realistic variance in latency and response quality.
-
Table R2 evaluates only four queries and two routes, making the setup effectively tabular. Additionally, the regret certificate in Table 9 yields bounds of ~4.7 to 5.0 per round, barely below the trivial cap of 5.8, while empirical realized regret is only 0.02 to 0.09. This loose gap renders the theoretical bound uninformative in practice.
-
Three out of the six tasks exhibit fitted tail indices below 1, violating the assumed (1, 2) regime, while the full-distribution KS distances range from 0.17 to 0.32. Combined with the unmet timing condition, the theoretical guarantees fail to cover the majority of the empirical settings.
Thank you again for taking the time to review our response and examine the additional experiments. Your comments helped us identify where the paper needs clearer comparisons and more precise claims. We are grateful for this guidance and will use it to improve the revision.
(1) Why the reported gain changes. The paper and Table R1 use different takeover settings. In the paper, our method uses task-specific early takeover, while the baselines use deadline-only or last-minute takeover. In R1, all six methods use the same 30-s threshold. The percentages also use different comparisons. The paper’s 27.9% averages per-task relative gains over each task’s strongest original baseline, using its absolute utility as the denominator. R1’s 3.12% is the increase in six-task mean utility over PAK-UCB, from 0.25550 to 0.26348. Under this common rule, our method ranks first on four tasks. As you observed, MATH-Lighteval accounts for 52.9% of the net gain over PAK-UCB. SQuAD and MATH-500 also improve, while the other three differences are small. We will clarify what the 27.9% measures in the abstract.
(2) Why the utility values differ. Only two thresholds change for our method. R1 uses 30 s on both TriviaQA and SQuAD, giving utilities of 0.2079 and 0.5674. The paper and R4 at zero overhead use 4.75 s and 15.985 s, giving 0.2799 and 0.5799. The other four values are identical, so these two changes fully explain the difference between 0.2635 and 0.2776. The baselines in R4 also use the task-specific early thresholds. Across its five overhead levels, our six-task mean exceeds PAK-UCB’s by 1.82%–4.13%.
(3) Tuning and uncertainty. Appendix D reports that evaluation data informed the original parameter choices. Algorithm 1 updates routes online at each fixed threshold. The threshold itself is chosen by comparing mean utility across runs. In a separate sensitivity check, we used 232 MATH history queries disjoint from the target queries and tested three values of one parameter per adaptive method. Our method ranked second in six-task mean utility (0.26953 vs 0.27069 for PAK-UCB), ahead of the other four baselines. Some base settings came from the earlier exploration, so a fully independent test of the tuning procedure remains open.
We also computed mean ± SD for all methods in Table 1 and R1/R4. In R1, our method scores 0.26348 ± 0.00524 versus 0.25550 ± 0.00416 for PAK-UCB and leads in 13 of 15 paired replays. These SDs measure variation across request orders and learner runs on the same recorded outcomes. Measuring latency and answer variability across service sessions requires independent generations, which are still missing for the full six-task comparison.
(4) Tightness of the regret bound. We agree that Table 9’s certificates are loose at 1,500 rounds (4.70–4.99 per round, versus a ceiling of 5.8 and measured average regret of 0.02–0.09). Corollary 1 gives a strictly tighter variance-dependent bound for the same learner under takeover, with each service measured against its own oracle. Four queries and two routes let us compute both oracles exactly. Measured average regret falls by 5.10%–12.48% in all four controlled settings. The practical tightness of the bound in high dimensions remains open.
(5) Scope beyond Pareto delays. Pareto gives an explicit lower bound on the variance reduction from takeover. Lemma B.1, Eq. (B.3), expresses that reduction in terms of truncated delay moments. If another delay model satisfies the lemma and yields a uniformly positive lower bound over queries and histories, Eq. (C.9) gives the same strict regret-bound comparison under Corollary 1’s remaining learning and variance conditions.
Three fitted marginal tail indices fall below 1, full-distribution Kolmogorov–Smirnov distances range from 0.17 to 0.32, and some settings fall outside the timing conditions. In paired replay of 7,000 cloud-initiated requests, takeover reduced within-task delay variance by 40.6%–98.1%. The empirical lower bound in (B.3) was positive on four tasks, including MMLU and MBPP, both with tail indices below 1. These task-level measurements support checking delay moments directly in more services. Applying the guarantee to a particular service requires verifying the uniform conditional bound.
Thank you again for helping us improve the paper.
Thank you for your sincere review and constructive feedback. We address an important practical problem: useful LLM answers within acceptable waits when cloud is slow, combining on-demand edge fallback with routing learned from quality and total waiting.
(1) To Question 1 and Weakness 1: All six methods share models, s=30 s, a 60s deadline and task-specific utility weights. Ours leads in mean utility on 4/6 tasks (Table R1). Its six-task mean utility exceeds PAK-UCB and AlwaysCloud by 3.12% and 9.66%. On TriviaQA, waiting cost savings do not offset the rating loss. Versus AlwaysCloud, mean waiting falls 49.59%, rating drops 0.8622 to 0.5239, and utility is 11.67% lower. For a chemist-name query, Ours selects edge, which is wrong in 0.163 s; cloud correctly answers “Otto Hahn” in 2.051 s. On MBPP, extra waiting cost slightly exceeds the rating gain. Both answers pass a last-digit coding request; Ours selects cloud while PAK-UCB selects edge.
Table R1. Mean utility.
(2)To Question 3 and Weakness 2: Corollary 1 gives a “strictly tighter regret bound.” Each oracle follows its service rule and selects the highest-mean-utility route; regret measures loss against it. Theorem 1 uses kernel information ΓN. Its worst-case rate O((logN)d+1/N) can be loose in high dimensions. Corollary 1 reduces the variance factor at common ΓN, preserving the order. For four queries and two routes, Appendix F.2 (F.1) gives ΓN=O(logN) and average regret O(logN/N). Appendix F.3 computes regret from known conditional means and actual routes. All four settings show lower average regret:
Table R2. Average regret RN/N.
(3) To Question 2 and Weakness 3: We use 7,000 untruncated cloud records, including 366 above 60 s. Kolmogorov–Smirnov (KS) distance selects the tail start and measures the largest gap between empirical and fitted cumulative probabilities.
Table R3. Pareto fits; smaller KS is better.
Tail deviations are 3.49–9.15 percentage points. These are marginal fits; Section 3.1 assumes query-conditional distributions. MATH-500, TriviaQA and SQuAD have fitted indices in (1,2); the others are below 1. Under original Table 1's settings, Ours leads on all six tasks, motivating extensions of our theoretical guarantees to broader latency distributions. Appendix F.2 meets the assumptions: Pareto indices 1.25/1.75, edge time 5 s, takeover 30 s, deadline 60 s; both normalized means' squared RKHS norms are below 0.77.
(4) To Question 4: We add 0, 0.5, 1, 2 or 5 s per takeover beyond the included edge initialization and generation time. All methods share the original task-specific thresholds s, models, utility weights and a 60-s deadline within each task.
Table R4. Six-task mean utility.
Ours leads at all five delays (+1.82%–4.13% over PAK-UCB, the strongest baseline at each delay). For unchanged answers still delivered on time, extra delay τ reduces utility by βτ. If 4maxx+,de+τmax≤s≤Dmax/2, Proposition 1 retains variance reduction ≥ks>0. Direct edge has no switching cost; its bound stays de2/4. Both routes retain their original conservative variance bounds. With Assumption 2 for both means and v0>de2/4, the strict regret-bound comparison holds. Appendix F.2's s=30,de=5 s allow τmax≤2.5 s; larger delays require a new Lemma B.1 bound.
We welcome further discussion. If our evidence addresses your concerns, we would appreciate reconsideration of your assessment. Thank you