- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces EC²-Bench to study edge-cloud collaboration in long-horizon LLM agents. It compares four strategies on 128 tasks, measuring completion, resource use, and privacy. Stronger edge models narrow completion gaps, while privacy differences remain.
Critical strengths
-
The benchmark compares completion, resource use, and privacy in the same tasks, making the trade-offs between strategies easy to see.
-
The instruction-following experiment includes an edge-only control and supports the idea that training helps the model use cloud advice better.
Critical weaknesses
-
Keeping only tasks where edge and cloud perform differently may overstate the benefits of collaboration.
-
Results are mostly aggregated. Small strategy gaps need confidence intervals and a breakdown by task type.
-
Appendix A.8 tests another cloud model, but the edge models are all Qwen. It is unclear whether the findings hold for other edge-model families.
-
The privacy metric weights all sensitive units equally. The overall rate may hide differences between information categories.
-
End-to-end latency is not reported, so the recommendations for interactive use are not fully supported.
Questions for authors
-
How many tasks were filtered out, and do the gains hold on unfiltered or less filtered tasks?
-
Could you compare Advisory and Delegation by task category, with confidence intervals?
-
Does Advisory's advantage with stronger edge models hold in Appendix A.8 and with non-Qwen edge models?
-
Could you report exposure by information category and check whether risk weighting changes the strategy ranking?
-
Could you report end-to-end latency or explain how its absence limits the recommendations for interactive use?
6 comments
Response to Reviewer fa8R [2/3]
W3 & Q3: Cross-family Generalizability of Edge Models
We thank the reviewer for this suggestion. Our main experiments primarily use the Qwen3.5 series as edge models because it provides a particularly suitable and consistent capability ladder for our study, spanning 2B to 27B within the same model generation, with shared chat templates and tool-calling interfaces. This allows us to systematically vary edge-model capability while minimizing potential confounding factors beyond model capability, such as differences in training recipes and interaction interfaces. This controlled setup is essential for isolating how edge capability scaling affects edge–cloud collaboration. Moreover, Qwen is a widely adopted model family in the open-weight LLM ecosystem, making it a practically relevant choice for studying edge-agent deployment.
Cross-cloud evaluation. Our Appendix A.8 already includes a complete collaboration evaluation using DeepSeek-V4.1-Flash as the cloud model, with the same Qwen edge models. The results consistently support our main findings, including the advantage of Advisory with stronger edge models, demonstrating that this observation is not specific to GLM-5.3.
To further examine whether these findings generalize to other edge-model families, we additionally evaluated IBM Granite-4.2 models at 3B, 8B, and 30B scales. As shown in the table below, the results exhibit trends consistent with our main findings in two aspects: (1) the completion spread across collaboration strategies decreases from 6.54 pp at 3B to 2.97 pp at 30B; and (2) the relative effectiveness shifts from external routing toward edge-controlled invocation, and, within edge-controlled strategies, from Delegation toward Advisory. Specifically, Per-step Routing achieves the highest completion at 3B, whereas Advisory performs best at both 8B and 30B.
These additional results provide further evidence that Advisory's advantage with stronger edge models is not specific to the Qwen family. We will incorporate these experiments into the revised manuscript.
| Model | Metric | Local-only | Query Routing | Step Routing | Delegation | Advisory |
|---|---|---|---|---|---|---|
| Granite-4.2-3B | Completion (%) | 21.98 | 39.33 | 41.36 | 38.17 | 34.82 |
| Cost ($) | 0.00 | 8.06 | 8.80 | 6.92 | 9.97 | |
| Granite-4.2-8B | Completion (%) | 42.17 | 51.19 | 50.41 | 50.81 | 54.12 |
| Cost ($) | 0.00 | 8.25 | 6.89 | 7.65 | 7.80 | |
| Granite-4.2-30B | Completion (%) | 58.39 | 60.19 | 60.74 | 62.07 | 63.16 |
| Cost ($) | 0.00 | 5.96 | 3.13 | 5.78 | 4.99 |
W4 & Q4: Privacy Exposure Measurement
We thank the reviewer for this suggestion. We intentionally adopt an equal-weighted privacy exposure metric to provide a transparent and reproducible measure of information exposure. Since the relative sensitivity of information depends on deployment contexts, introducing universal risk weights could impose subjective assumptions and potentially bias strategy comparisons.
To further examine the impact of risk weighting, we conduct a sensitivity analysis under three weighting schemes: (1) Personal Privacy Priority (personal: 3, organizational: 1), (2) Organizational Confidentiality Priority (personal: 1, organizational: 3), and (3) Selected Sensitive Category Priority (P1/P2/P4/P5/S2/S3: 5, others: 1). As shown in the table below, the privacy-exposure rankings of all four collaboration strategies remain unchanged across all three weighting schemes and edge models, supporting the robustness of our findings to these alternative risk assumptions.
We will include this risk-weighting sensitivity analysis in the revised manuscript while retaining equal-weighted privacy exposure as our default metric for its transparency and reproducibility.
| Edge Model | Original (Equal Weight) | Personal Privacy Priority | Organizational Confidentiality Priority | Sensitive Category Priority |
|---|---|---|---|---|
| 2B | D < A < Q < S | D < A < Q < S | D < A < Q < S | D < A < Q < S |
| 9B | Q < D < S < A | Q < D < S < A | Q < D < S < A | Q < D < S < A |
| 27B | Q < S < D < A | Q < S < D < A | Q < S < D < A | Q < S < D < A |
| Plus | Q < S < D < A | Q < S < D < A | Q < S < D < A | Q < S < D < A |
Q: Query Routing; S: Per-step Routing; D: Delegation; A: Advisory. Lower exposure is better.
Response to Reviewer fa8R [3/3]
W5 & Q5: End-to-end Latency
We thank the reviewer for raising this point. Our original study focuses on task completion, cloud expenditure, edge computation, and privacy exposure. We did not report end-to-end latency because it depends on multiple deployment-specific factors, including edge-device inference speed, cloud API response time, network latency, tool execution, and agent execution trajectories. Consequently, latency can vary substantially across deployment settings.
Nevertheless, we agree that latency is an important consideration for assessing the suitability of different collaboration strategies for interactive use. We therefore additionally measure end-to-end execution latency for all edge models and collaboration strategies under our experimental setup. The results are reported as mean ± standard deviation in seconds (table below). Overall, edge–cloud collaboration generally achieves higher task completion but incurs additional end-to-end latency compared with edge-only execution, highlighting a trade-off between task performance and response time. We will incorporate these results and clarify the corresponding deployment assumptions in the revised manuscript.
| Model | Edge-only | Query Routing | Per-step Routing | Delegation | Advisory |
|---|---|---|---|---|---|
| Qwen3.5-2B | 221.54±265.11 | 197.89±383.36 | 225.22±352.11 | 179.82±238.82 | 205.95±226.22 |
| Qwen3.5-9B | 133.50±166.58 | 155.57±360.67 | 138.53±246.38 | 210.29±335.47 | 243.07±349.44 |
| Qwen3.5-27B | 197.11±313.81 | 203.53±399.50 | 214.60±324.41 | 265.54±415.30 | 305.43±470.86 |
| Qwen3.7-Plus | 155.50±293.94 | 196.00±418.97 | 210.45±380.32 | 213.16±309.06 | 255.48±412.93 |
Response to Reviewer fa8R [1/3]
Dear reviewer fa8R,
Thank you for your thoughtful and constructive review, and for recognizing the value of our unified evaluation of task performance, resource usage, and privacy. Your suggestions have helped us further strengthen the empirical analysis and broaden the evidence supporting our findings. We address your comments point by point below.
We thank the reviewer for raising this concern. EC²-Bench is designed to focus on tasks where edge and cloud capabilities are complementary, rather than to represent the general distribution of agent tasks. Specifically, we exclude tasks that are either solved by all selected models (Qwen3.5-2B, Qwen3.5-9B, and GLM-5.3) or failed by all of them. This filtering removes 28 out of 156 tasks. Importantly, task selection relies exclusively on edge-only and cloud-only performance, without using the results of any collaboration strategy.
To examine the impact of this filtering, we reintroduced all 28 excluded tasks, expanding the evaluation set from 128 to 156 tasks, and repeated the experiments under the same settings (see the table below). The results remain consistent with our main findings: the completion spread across strategies continues to narrow as edge capability increases (22.84 → 14.79 → 4.81 → 3.95 pp). Moreover, Per-step Routing achieves the highest completion at 2B, whereas Advisory achieves the highest completion at 9B, 27B, and Plus.
These results suggest that the observed capability-dependent trends are not artifacts of the original task filtering. We will incorporate this additional analysis into the revised manuscript and clarify the scope of the benchmark.
We thank the reviewer for this suggestion. We additionally compare Advisory and Delegation across the six task categories and four edge-model configurations, reporting category-wise completion rates and differences (table below). Given the limited number of tasks within each category (10–36), we report the observed category-wise performance differences while using all 128 tasks to estimate the overall 95% confidence intervals. Specifically, we compute these intervals using task-level paired bootstrap with 10,000 resamples.
Overall, Advisory achieves higher average completion than Delegation at 9B and above. The category-wise breakdown further reveals differences in the relative performance of the two strategies across task categories. We will incorporate this analysis into the revised manuscript.