acceptodds
Under review as a conference paper at ICLR 2027

Empowering Edge Agents: A Systematic Analysis of Edge-Cloud Collaboration in LLM Agent Execution

Abstract

Edge–cloud collaboration offers a promising way to combine the lower inference overhead and greater data locality of edge-side execution with the substantially stronger capabilities of cloud-hosted frontier models. However, as LLM agents increasingly operate in long-horizon scenarios, cloud participation becomes interleaved throughout execution rather than a one-shot decision for each query. Such interleaved participation introduces trajectory-level effects that remain insufficiently understood. To address this gap, we introduce EC-Bench, a benchmark for evaluating edge–cloud collaboration in long-horizon agent execution. EC-Bench provides executable digital-workspace agent tasks and a unified evaluation framework covering task performance, resource usage, and privacy exposure. Using this benchmark, we empirically study diverse forms of edge–cloud collaboration. Our study reveals several key findings. First, edge–cloud collaboration provides an effective execution paradigm for long-horizon agents by achieving a favorable balance among task performance, resource usage, and privacy exposure compared with edge-only and cloud-only execution. Second, as edge models become stronger, collaborative execution approaches cloud-only performance, and performance differences across collaboration strategies become smaller, while privacy exposure remains strongly dependent on the chosen strategy. Third, stronger edge capability changes the preferred collaboration mode, shifting from externally controlled cloud participation and direct cloud execution toward more edge-controlled collaboration. Further analyses identify cloud-invocation decision-making and instruction following as important edge-side abilities underlying such effective collaboration. These findings establish a systematic understanding of edge–cloud collaboration in long-horizon agent execution and provide insights for designing future edge models that can effectively collaborate with cloud models. All code and datasets will be available on GitHub.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agent, Edge-Cloud Collaboration, Agent Evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fa8ROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces EC²-Bench to study edge-cloud collaboration in long-horizon LLM agents. It compares four strategies on 128 tasks, measuring completion, resource use, and privacy. Stronger edge models narrow completion gaps, while privacy differences remain.

Critical strengths
  1. The benchmark compares completion, resource use, and privacy in the same tasks, making the trade-offs between strategies easy to see.

  2. The instruction-following experiment includes an edge-only control and supports the idea that training helps the model use cloud advice better.

Critical weaknesses
  1. Keeping only tasks where edge and cloud perform differently may overstate the benefits of collaboration.

  2. Results are mostly aggregated. Small strategy gaps need confidence intervals and a breakdown by task type.

  3. Appendix A.8 tests another cloud model, but the edge models are all Qwen. It is unclear whether the findings hold for other edge-model families.

  4. The privacy metric weights all sensitive units equally. The overall rate may hide differences between information categories.

  5. End-to-end latency is not reported, so the recommendations for interactive use are not fully supported.

Questions for authors
  1. How many tasks were filtered out, and do the gains hold on unfiltered or less filtered tasks?

  2. Could you compare Advisory and Delegation by task category, with confidence intervals?

  3. Does Advisory's advantage with stronger edge models hold in Appendix A.8 and with non-Qwen edge models?

  4. Could you report exposure by information category and check whether risk weighting changes the strategy ranking?

  5. Could you report end-to-end latency or explain how its absence limits the recommendations for interactive use?

6 comments
AuthorsOpenReview ↗

Response to Reviewer fa8R [1/3]

Dear reviewer fa8R,

Thank you for your thoughtful and constructive review, and for recognizing the value of our unified evaluation of task performance, resource usage, and privacy. Your suggestions have helped us further strengthen the empirical analysis and broaden the evidence supporting our findings. We address your comments point by point below.

W1 & Q1: Rationale and Impact of Endpoint-based Filtering

We thank the reviewer for raising this concern. EC²-Bench is designed to focus on tasks where edge and cloud capabilities are complementary, rather than to represent the general distribution of agent tasks. Specifically, we exclude tasks that are either solved by all selected models (Qwen3.5-2B, Qwen3.5-9B, and GLM-5.3) or failed by all of them. This filtering removes 28 out of 156 tasks. Importantly, task selection relies exclusively on edge-only and cloud-only performance, without using the results of any collaboration strategy.

To examine the impact of this filtering, we reintroduced all 28 excluded tasks, expanding the evaluation set from 128 to 156 tasks, and repeated the experiments under the same settings (see the table below). The results remain consistent with our main findings: the completion spread across strategies continues to narrow as edge capability increases (22.84 → 14.79 → 4.81 → 3.95 pp). Moreover, Per-step Routing achieves the highest completion at 2B, whereas Advisory achieves the highest completion at 9B, 27B, and Plus.

These results suggest that the observed capability-dependent trends are not artifacts of the original task filtering. We will incorporate this additional analysis into the revised manuscript and clarify the scope of the benchmark.

MethodMetric2B9B27Bplus
Edge-onlyCompletion (%)17.4344.0264.4072.6
Cost ($)0.000.000.000.00
Query RoutingCompletion (%)25.2947.9166.7473.61
Cost ($)9.429.8512.0814.43
Per-step RoutingCompletion (%)40.6854.7768.7873.6
Cost ($)13.626.806.2312.67
DelegationCompletion (%)19.2456.1570.1674.33
Cost ($)1.533.495.895.2
AdvisoryCompletion (%)17.8462.7071.5577.55
Cost ($)2.5110.636.948.36

W2 & Q2: Task-category Analysis of Collaboration Strategies

We thank the reviewer for this suggestion. We additionally compare Advisory and Delegation across the six task categories and four edge-model configurations, reporting category-wise completion rates and differences (table below). Given the limited number of tasks within each category (10–36), we report the observed category-wise performance differences while using all 128 tasks to estimate the overall 95% confidence intervals. Specifically, we compute these intervals using task-level paired bootstrap with 10,000 resamples.

Task Category (n)2B (A/D, Δ)9B (A/D, Δ)27B (A/D, Δ)Plus (A/D, Δ)
Office (36)7.54/8.33 (-0.78)58.98/55.23 (+3.75)76.46/72.31 (+4.16)77.67/75.44 (+2.23)
Info Search (34)8.75/13.68 (-4.94)59.37/52.51 (+6.86)72.70/69.97 (+2.73)80.54/77.02 (+3.52)
Safety (21)36.75/33.95 (+2.80)56.81/51.78 (+5.03)56.94/60.48 (-3.55)66.61/64.62 (+1.99)
Dev & Ops (14)15.30/18.94 (-3.64)65.66/49.06 (+16.60)65.85/63.23 (+2.62)66.68/61.36 (+5.32)
Data Analysis (13)6.15/9.00 (-2.85)71.51/52.65 (+18.86)74.13/75.60 (-1.47)91.63/89.88 (+1.75)
Automation (10)25.31/16.99 (+8.32)55.56/53.64 (+1.92)67.14/71.99 (-4.85)77.73/78.34 (-0.61)
ModelComparisonDifference95% CI
2BAdvisory - Delegation-1.1[-3.72, 1.5]
9BAdvisory - Delegation+7.6[3.55, 11.61]
27BAdvisory - Delegation+1.1[−1.63, 4.40]
PlusAdvisory - Delegation+2.6[0.28, 4.97]

Overall, Advisory achieves higher average completion than Delegation at 9B and above. The category-wise breakdown further reveals differences in the relative performance of the two strategies across task categories. We will incorporate this analysis into the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer fa8R [2/3]

W3 & Q3: Cross-family Generalizability of Edge Models

We thank the reviewer for this suggestion. Our main experiments primarily use the Qwen3.5 series as edge models because it provides a particularly suitable and consistent capability ladder for our study, spanning 2B to 27B within the same model generation, with shared chat templates and tool-calling interfaces. This allows us to systematically vary edge-model capability while minimizing potential confounding factors beyond model capability, such as differences in training recipes and interaction interfaces. This controlled setup is essential for isolating how edge capability scaling affects edge–cloud collaboration. Moreover, Qwen is a widely adopted model family in the open-weight LLM ecosystem, making it a practically relevant choice for studying edge-agent deployment.

Cross-cloud evaluation. Our Appendix A.8 already includes a complete collaboration evaluation using DeepSeek-V4.1-Flash as the cloud model, with the same Qwen edge models. The results consistently support our main findings, including the advantage of Advisory with stronger edge models, demonstrating that this observation is not specific to GLM-5.3.

To further examine whether these findings generalize to other edge-model families, we additionally evaluated IBM Granite-4.2 models at 3B, 8B, and 30B scales. As shown in the table below, the results exhibit trends consistent with our main findings in two aspects: (1) the completion spread across collaboration strategies decreases from 6.54 pp at 3B to 2.97 pp at 30B; and (2) the relative effectiveness shifts from external routing toward edge-controlled invocation, and, within edge-controlled strategies, from Delegation toward Advisory. Specifically, Per-step Routing achieves the highest completion at 3B, whereas Advisory performs best at both 8B and 30B.

These additional results provide further evidence that Advisory's advantage with stronger edge models is not specific to the Qwen family. We will incorporate these experiments into the revised manuscript.

ModelMetricLocal-onlyQuery RoutingStep RoutingDelegationAdvisory
Granite-4.2-3BCompletion (%)21.9839.3341.3638.1734.82
Cost ($)0.008.068.806.929.97
Granite-4.2-8BCompletion (%)42.1751.1950.4150.8154.12
Cost ($)0.008.256.897.657.80
Granite-4.2-30BCompletion (%)58.3960.1960.7462.0763.16
Cost ($)0.005.963.135.784.99

W4 & Q4: Privacy Exposure Measurement

We thank the reviewer for this suggestion. We intentionally adopt an equal-weighted privacy exposure metric to provide a transparent and reproducible measure of information exposure. Since the relative sensitivity of information depends on deployment contexts, introducing universal risk weights could impose subjective assumptions and potentially bias strategy comparisons.

To further examine the impact of risk weighting, we conduct a sensitivity analysis under three weighting schemes: (1) Personal Privacy Priority (personal: 3, organizational: 1), (2) Organizational Confidentiality Priority (personal: 1, organizational: 3), and (3) Selected Sensitive Category Priority (P1/P2/P4/P5/S2/S3: 5, others: 1). As shown in the table below, the privacy-exposure rankings of all four collaboration strategies remain unchanged across all three weighting schemes and edge models, supporting the robustness of our findings to these alternative risk assumptions.

We will include this risk-weighting sensitivity analysis in the revised manuscript while retaining equal-weighted privacy exposure as our default metric for its transparency and reproducibility.

Edge ModelOriginal (Equal Weight)Personal Privacy PriorityOrganizational Confidentiality PrioritySensitive Category Priority
2BD < A < Q < SD < A < Q < SD < A < Q < SD < A < Q < S
9BQ < D < S < AQ < D < S < AQ < D < S < AQ < D < S < A
27BQ < S < D < AQ < S < D < AQ < S < D < AQ < S < D < A
PlusQ < S < D < AQ < S < D < AQ < S < D < AQ < S < D < A

Q: Query Routing; S: Per-step Routing; D: Delegation; A: Advisory. Lower exposure is better.

AuthorsOpenReview ↗

Response to Reviewer fa8R [3/3]

W5 & Q5: End-to-end Latency

We thank the reviewer for raising this point. Our original study focuses on task completion, cloud expenditure, edge computation, and privacy exposure. We did not report end-to-end latency because it depends on multiple deployment-specific factors, including edge-device inference speed, cloud API response time, network latency, tool execution, and agent execution trajectories. Consequently, latency can vary substantially across deployment settings.

Nevertheless, we agree that latency is an important consideration for assessing the suitability of different collaboration strategies for interactive use. We therefore additionally measure end-to-end execution latency for all edge models and collaboration strategies under our experimental setup. The results are reported as mean ± standard deviation in seconds (table below). Overall, edge–cloud collaboration generally achieves higher task completion but incurs additional end-to-end latency compared with edge-only execution, highlighting a trade-off between task performance and response time. We will incorporate these results and clarify the corresponding deployment assumptions in the revised manuscript.

ModelEdge-onlyQuery RoutingPer-step RoutingDelegationAdvisory
Qwen3.5-2B221.54±265.11197.89±383.36225.22±352.11179.82±238.82205.95±226.22
Qwen3.5-9B133.50±166.58155.57±360.67138.53±246.38210.29±335.47243.07±349.44
Qwen3.5-27B197.11±313.81203.53±399.50214.60±324.41265.54±415.30305.43±470.86
Qwen3.7-Plus155.50±293.94196.00±418.97210.45±380.32213.16±309.06255.48±412.93
Reviewer_4k4SOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents EC$^2$-Bench, a benchmark built to examine how edge and cloud resources can cooperate when LLM agents operate over long horizons. It contains 128 executable tasks set in a digital workspace: some are adapted from existing benchmarks, others were constructed from scratch, and 100 of them carry annotations marking sensitive information. A single shared framework logs both agent trajectories and the contexts that are visible to the cloud.

Four collaboration strategies are compared: query-level routing, per-step routing, edge-controlled cloud delegation, and edge-controlled cloud advisory. Each is evaluated across four edge models and two cloud models, with the stronger edge models standing in for potential future edge capabilities. The evaluation covers weighted task-completion scores, Pass$^3$ (success in all three independent runs with completion above 0.7), cloud token consumption and cost, edge FLOPs where these could be estimated, and exposure of annotated sensitive content.

The reported results indicate that collaboration raises completion relative to running on the edge alone, while lowering cloud cost and sensitive exposure relative to running entirely in the cloud. As edge capability grows, the completion gap between strategies generally narrow, whereas the differences in exposure remain large. The authors additionally report a shift toward edge-controlled and advisory approaches. Supporting analyses contrast delegation with random cloud invocation at a matched call rate, and investigate instruction-following training under a fixed advisory schedule, where advisory performance improves while edge-only performance stays essentially unchanged. Human evaluations are used to check annotation quality and the accuracy of exposure detection.

Critical strengths
  1. Privacy exposure is relevant because keeping sensitive workspace data local is an important motivation for on-device execution. EC$^2$-Bench tracks which annotated sensitive units become visible to the cloud throughout execution. Its exposure detector is validated against human judgments, achieving 96.9% F1 on 200 unit-context pairs; annotation precision and synthetic-content quality are assessed separately. These checks support the measurements without establishing annotation completeness or comprehensive privacy protection. Substantial exposure differences persist even when stronger edge models narrow completion gaps, showing why completion alone is insufficient to compare collaboration strategies.

  2. EC$^2$-Bench provides a common experimental framework by adapting executable tasks from heterogeneous benchmarks to a shared runtime and evaluation protocol. This enables systematic comparison of two meaningful design choices: externally routed versus edge-controlled cloud participation, and cloud-generated actions versus advisory support. Evaluations across several edge capability levels and two cloud models extend the evidence beyond a single model pairing, making the framework useful for studying how collaboration strategies perform as edge capability changes.

Critical weaknesses
  1. Unequal cloud usage and unreported uncertainty limit the interpretation of RQ3. With Qwen3.5-2B, Per-step Routing costs 12.80, compared with 1.19 for Delegation. Router calibration and the call-rate-matched random baseline provide useful controls, but do not establish budget-matched comparisons across all four strategies. The observed rankings therefore do not fully separate collaboration design from the amount of cloud assistance used. Completion scores are also averaged over three runs without reported variability, making small margins, such as Advisory's 1.07 percentage-point advantage over Delegation at 27B, difficult to assess. Matched-budget comparisons and uncertainty estimates would strengthen the recommendations about which strategy to use as edge capability increases.

  2. The implications for developing better collaboration policies are only partially validated. Instruction-following training improves advisory completion for Qwen3.5-2B under a fixed cloud-call schedule, but does not test improvements to autonomous invocation decisions or the resulting performance, cost, and exposure trade-offs. While the study compares exposure across existing strategies, it does not evaluate targeted privacy improvements, such as selectively sharing execution context with the cloud. The benchmark's usefulness for developing more effective and privacy-aware collaboration policies therefore remains only partially demonstrated.

Questions for authors
  1. Do the RQ3 rankings persist at matched cloud-token or monetary budgets? Please also clarify why Appendix A.9 describes step-level participation for Query Routing but task-level allocation for Per-step Routing.
  2. Can you report mean completion and standard deviation across the three runs for each configuration, together with uncertainty estimates for the key RQ3 comparisons?
  3. Which conclusions about improving autonomous invocation and privacy-aware collaboration are supported by the current experiments? Please distinguish what the fixed-schedule training analysis demonstrates from untested directions, such as selective context sharing to reduce exposure while preserving completion.
  4. The paper states that code and datasets will be released, but the PDF provides no benchmark repository link. Are anonymized executable tasks, evaluation checks, and collaboration configurations available for review?
  5. For Qwen3.7-Plus, Table 2 gives completion and privacy-exposure spreads of 4.764.76 and 66.2366.23 percentage points, whereas Figure 3 reports 3.673.67 and 56.2356.23. Could you clarify how these spreads were calculated?
  6. Task selection uses Qwen3.5-2B, Qwen3.5-9B, and GLM-5.3. How might this filtering affect the generality of the findings for stronger edge models?

Suggestions for future work and presentation, not conditions for acceptance:

  • An illustrative resource-cost estimate for a specific deployment scenario could make Table 2 easier to interpret. Combining cloud expenditure and estimated edge-computation cost under explicit hardware and pricing assumptions, alongside the separate measurements, would clarify the overall resource trade-offs.
  • Figure 1 is visually dense, and several labels are difficult to read at the displayed size. Larger, more consistent fonts and fewer secondary details would make the workflow and the distinctions between collaboration strategies easier to follow.
6 comments
AuthorsOpenReview ↗

Response to Reviewer 4k4S [1/3]

Dear reviewer 4k4S,

Thank you for your thorough and constructive review. Your insightful comments have helped us strengthen our empirical analysis, better characterize the trade-offs of different collaboration strategies, and clarify the scope and implications of our findings. We address your questions and suggestions below.

W1 & Q1: Cloud Usage and Budget-Matched Comparisons

We thank the reviewer for raising this concern. Our original experiments were designed to evaluate each collaboration strategy under its native execution mechanism, rather than under a fixed cloud budget. In edge-controlled strategies, cloud invocation is determined autonomously by the edge agent, making cloud usage itself part of the collaboration behavior under study. In contrast, routing-based strategies rely on an external router whose invocation threshold can be adjusted to control cloud participation. However, there is no unique budget reference for such calibration, as Delegation and Advisory naturally incur different cloud costs. We therefore calibrate both routing-based strategies to an approximately 15% cloud-participation target, following the calibration protocol described in Appendix A.9.

To better understand the cloud-invocation behavior of edge-controlled strategies, we analyze invocation patterns across edge models, as shown in the table below. Under Edge-Controlled Delegation, Qwen3.5-2B invokes the cloud in only 25.26% of tasks, compared with 49.22%–56.25% for stronger edge models, despite achieving only 12.75% edge-only completion. Moreover, its invocation decisions yield only a 1.2 pp improvement over call-rate-matched random invocation, compared with 3.0–4.4 pp for stronger models (Figure 4a). Together, these results suggest that weaker edge agents both underutilize cloud assistance and make less effective invocation decisions.

Model2b9b27bplus
Tasks with Cloud Invocation (%)25.2649.2255.4756.25

To further examine whether the advantage of routing at 2B is attributable to higher cloud expenditure, we conduct additional budget-matched comparisons. Specifically, we adjust the routing thresholds to match the cloud cost of Delegation (see the table below). Under comparable cloud costs, Query Routing and Per-step Routing achieve completion rates of 18.92% and 16.81%, respectively, compared with 15.86% for Delegation. For reference, Advisory achieves 14.75% completion at a higher cloud cost of $2.36 under its original setting. These results show that the advantage of external routing at 2B persists under comparable cloud expenditure.

We will incorporate the cloud-invocation analysis and budget-matched comparisons into the revised manuscript to clarify how cloud usage affects the interpretation of RQ3.

Method (Qwen3.5-2B)Completion (%)Cloud Cost ($)
Edge-Ctrl. Delegation15.861.19
Edge-Ctrl. Advisory14.752.36
Query Routing (budget-matched)18.921.25
Per-step Routing (budget-matched)16.811.21

W1 & Q2: Statistical Uncertainty and Reliability of RQ3 Findings

We thank the reviewer for highlighting the importance of reporting variability and uncertainty estimates. To address this concern, we provide the mean and standard deviation of task completion across three independent runs for each edge-only and collaborative configuration, as shown below.

MethodQwen3.5-2BQwen3.5-9BQwen3.5-27BQwen3.7-Plus
Edge-only12.75 ± 0.2839.12 ± 1.5461.79 ± 0.6371.34 ± 0.67
Query Routing23.98 ± 0.5443.60 ± 2.0265.53 ± 1.8673.17 ± 1.62
Per-step Routing39.97 ± 1.5152.71 ± 0.5068.42 ± 1.3872.08 ± 0.62
Delegation15.86 ± 0.6552.88 ± 0.3569.06 ± 0.3974.24 ± 1.04
Advisory14.75 ± 0.5260.46 ± 2.6270.13 ± 1.1376.84 ± 1.54

We further assess the uncertainty of key RQ3 comparisons using task-level paired percentile bootstrap with 95% confidence intervals (CIs). The results are summarized below.

ModelComparisonDifference95% CI
2BDelegation - Per-step Routing-24.1[-29.07, -19.25]
9BAdvisory - Delegation+7.6[3.55, 11.61]
27BAdvisory - Delegation+1.1[−1.63, 4.40]
PlusAdvisory - Delegation+2.6[0.28, 4.97]

The results further support the capability-dependent shift in collaboration effectiveness observed in RQ3. Per-step Routing achieves the highest mean completion at 2B, whereas Advisory achieves the highest mean completion at 9B, 27B, and Plus. The bootstrap confidence intervals provide statistical support for Advisory's advantage over Delegation at 9B and Plus. Although the difference at 27B is not statistically significant, the overall results remain consistent with our finding that stronger edge models increasingly favor edge-controlled advisory.

We will incorporate these uncertainty estimates into the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 4k4S [2/3]

W2 and Q3: Implications of the training and privacy analyses for collaboration policy development.

We thank the reviewer for raising this concern. Our goal is to identify key edge-side capabilities underlying effective collaboration and provide empirical insights for future policy development, rather than directly optimize collaboration policies. We clarify what our current experiments demonstrate and provide additional evidence for privacy-aware collaboration below.

Cloud-invocation decision-making. Our results demonstrate that effective autonomous invocation is an important capability for edge-controlled collaboration. Compared with a call-rate-matched random baseline, agent-controlled invocation improves completion by 1.2 pp with Qwen3.5-2B and 3.0–4.4 pp with stronger edge models (Figure 4a). Moreover, our invocation analysis (response to W1) shows that Qwen3.5-2B invokes the cloud in only 25.26% of tasks under Delegation, despite achieving only 12.75% edge-only completion. These findings suggest that weaker edge agents both underutilize cloud assistance and make less effective invocation decisions, highlighting the importance of invocation decision-making as an edge-side capability. These results characterize the contribution of autonomous invocation, but do not examine how to improve this capability through training.

Cloud guidance utilization. Our instruction-following training experiment examines a complementary capability: effectively utilizing cloud-provided guidance once it is received. To isolate this effect, we fix the advisory schedule and apply instruction-following training to Qwen3.5-2B. Advisory completion increases from 24.8% to 28.8%, while edge-only completion remains essentially unchanged (12.75% → 12.69%; Appendix A.10). This demonstrates that improving instruction-following ability can enhance the utilization of cloud guidance independently of invocation decisions.

Privacy-aware collaboration. While our main study characterizes privacy exposure across collaboration mechanisms rather than optimizing context-sharing policies, we agree that selective context sharing is a promising direction. We therefore conduct an additional experiment with Qwen3.5-9B under Cloud Advisory, comparing four context-sharing strategies: Full History (the complete execution history), Last 10 Turns (the 10 most recent interaction turns), Summary (an edge-LLM-generated summary of the execution history[1]), and Self-formulated Question (a focused question generated by the edge agent without sharing the execution history[2]).

Context sent to the cloudCompletion (%)Privacy exposure (%)Cloud cost ($)
Full history (Table 2)60.4645.468.73
Last 10 turns58.3244.427.82
Summary56.115.743.65
Self-formulated question52.565.392.40
Edge-only (reference)39.120.00–

These results show that different context-sharing strategies can substantially reduce privacy exposure and cloud cost, albeit with some performance degradation. Developing more effective strategies that minimize privacy exposure while preserving task performance is a promising future direction that EC²-Bench can support.

Together, these findings provide empirical evidence for identifying and improving key capabilities underlying effective edge–cloud collaboration, while highlighting promising directions for future policy development. We will incorporate the additional experiments and clarify the corresponding conclusions in the revised manuscript.

Q1 & Q5: Clarifications and Corrections

We appreciate the reviewer's careful reading and helpful comments.

(1) Cloud participation definitions (Appendix A.9). We thank the reviewer for pointing this out. Query Routing makes a single routing decision per task, so its cloud-participation rate is measured at the task level. Per-step Routing makes routing decisions at each interaction step, so its participation rate is measured at the step level. We will revise Appendix A.9 accordingly to align with Section 3.2 and our implementation.

(2) Completion and privacy spreads (Figure 3). Thank you for pointing this out. The correct spreads for Qwen3.7-Plus are 4.76 pp for completion (76.84 − 72.08) and 66.23 pp for privacy exposure (78.74 − 12.51), consistent with Table 2. We will correct Figure 3 accordingly in the revised manuscript.

Q4: Code and Benchmark Availability

Thank you for the suggestion. We have now released the code and dataset in an anonymized repository: https://anonymous.4open.science/r/ec2_anonymous-88ED/.

AuthorsOpenReview ↗

Response to Reviewer 4k4S [3/3]

Q6: Impact of Task Filtering on Stronger Edge Models

We thank the reviewer for raising this concern. Our task filtering aims to retain tasks that are informative for studying edge–cloud collaboration by excluding tasks that are either solved by all selected models (Qwen3.5-2B, Qwen3.5-9B, and GLM-5.3) or failed by all of them. Since Qwen3.5-27B and Qwen3.7-Plus were not involved in task selection, we further examine whether the filtering affects the observed trends for these stronger models.

To examine this, we reintroduced all tasks excluded during the endpoint-based filtering stage, expanding the evaluation set from 128 to 156 tasks, and repeated the experiments under the same settings (see the table below). The results remain consistent with our main findings: the completion spread across strategies continues to narrow as edge capability increases, from 22.84 pp at 2B to 14.79, 4.81, and 3.95 pp at 9B, 27B, and Plus, respectively. Moreover, Per-step Routing achieves the highest completion at 2B, whereas Advisory achieves the highest completion at 9B, 27B, and Plus.

These results suggest that our findings are not artifacts of the original task filtering. We will incorporate this additional analysis into the revised manuscript.

Method2B9B27Bplus
Edge-onlyCompletion (%)17.4344.0264.4072.6
Cost ($)0.000.000.000.00
Query RoutingCompletion (%)25.2947.9166.7473.61
Cost ($)9.429.8512.0814.43
Per-step RoutingCompletion (%)40.6854.7768.7873.6
Cost ($)13.626.806.2312.67
DelegationCompletion (%)19.2456.1570.1674.33
Cost ($)1.533.495.895.2
AdvisoryCompletion (%)17.8462.7071.5577.55
Cost ($)2.5110.636.948.36

S1: Illustrative Resource Cost Estimation

Thank you for this helpful suggestion. Following the inference-cost accounting framework of [3], we additionally estimate the edge-side inference cost based on computational FLOPs, GPU peak throughput, MFU, and GPU rental prices. Specifically, we consider an illustrative deployment scenario using four NVIDIA H800 GPUs, with a unified 10% MFU assumption for input and output processing.

We then combine the estimated edge-side cost with the actual GLM API expenditure to obtain the total edge–cloud cost per 128 tasks, as shown in the table below. This provides a deployment-oriented estimate of the overall resource trade-offs, complementing the separately reported cloud expenditure and edge FLOPs in Table2.

We will include the estimation methodology, assumptions, and results in the revised manuscript.

ModelMetricEdge-onlyQuery RoutingPer-step RoutingDelegationAdvisory
Qwen3.5-2BCompletion(%)12.75%23.98%39.97%15.86%14.75%
Estimated Total Cost ($)$0.83$9.60$13.14$2.19$3.75
Qwen3.5-9BCompletion(%)39.12%43.60%52.71%52.88%60.46%
Estimated Total Cost ($)$0.71$9.97$6.24$4.09$9.94
Qwen3.5-27BCompletion(%)61.79%65.53%68.42%69.06%70.13%
Estimated Total Cost ($)$3.28$13.46$5.80$7.58$8.98

S2: Figure 1 Readability

We thank the reviewer for this constructive suggestion. We will improve the readability of Figure 1 by increasing and standardizing font sizes, simplifying secondary details, and refining the layout to more clearly illustrate the agent workflow and distinctions among collaboration strategies.

[1] Kang M, Chen W N, Han D, et al. Acon: Optimizing context compression for long-horizon llm agents[J]. arXiv preprint arXiv:2510.00615, 2025.

[2] Zeng H, Liu X, Hu Y, et al. Learning to Seek Help: Dynamic Collaboration Between Small and Large Language Models[J]. arXiv preprint arXiv:2604.17827, 2026.

[3] Sardana, Nikhil, et al. "Beyond chinchilla-optimal: Accounting for inference in language model scaling laws." arXiv preprint arXiv:2401.00448 (2023).

Reviewer_H1KcOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a benchmark aggregation of six previously published benchmarks for edge-cloud LLM agent execution. The paper poses three research questions to better understand the edge-cloud setup through performance (RQ1) and the load dynamics of the edge-cloud paradigm (RQ2, RQ3).

The benchmark includes unified task and evaluation methods on top of the original benchmarks, and considers privacy costs and resource usage. For resource usage, only cloud price is taken into account, while energy expenditure at the edge, where energy is scarce, is not considered.

Models used to support the RQ and demonstrate the benchmark were the Qwen family as Edge devices and DeepSeek-v4-Pro, and GLM-5.3 as cloud agents.

Critical strengths

The problem is interesting and timely, and addresses the important problem of aggregating benchmarks and reflecting on the metrics and tasks.

Critical weaknesses

RQ2 and RQ3 are difficult to distinguish, too broad, and it is not clear how to falsify them. The paper does not discuss how the experiments address the research questions, and they seem to address more narrow questions than the ones posed.

The pure aggregation of benchmarks must be complemented by a study of why it makes sense to aggregate those benchmarks. For example, in what way do the different added benchmarks complement each other?

Only one model family was used for the edge devices. What about other model families (e.g., llama, mistral, etc)?

Only two large models were used as cloud models. What guarantees the community that the results that support the paper will not be different with some other model family?

The primary area of the paper should be datasets and benchmarks.

Code reproducing the results and how benchmark aggregation is performed is not shared with the reviewers.

Questions for authors

See weaknesses above.

7 comments
AuthorsOpenReview ↗

Response to Reviewer H1Kc [1/3]

Dear reviewer H1Kc,

Thank you for your thoughtful and constructive review. Your comments have helped us clarify the research questions, better articulate the rationale behind our benchmark design, and strengthen the empirical evidence for the generalizability of our findings. We address your comments point by point below.

W1: Clarification of RQ2 and RQ3

Thank you for this insightful comment. We would like to clarify that RQ2 and RQ3 are motivated by two distinct yet closely related aspects of edge–cloud collaboration, rather than broadly examining the impact of model scaling.

Our central motivation is to understand not only whether stronger edge models improve agent execution, but also whether and how they change the way edge and cloud agents collaborate. As edge models continue to advance rapidly [1], future edge models may attain capabilities currently associated with substantially larger models. Consequently, both the overall outcomes of edge–cloud collaboration and the most effective collaboration paradigms may evolve as edge capability increases. This motivates our RQ2 and RQ3, which respectively investigate these two aspects.

Specifically, RQ2 focuses on the outcome-level implications of edge capability scaling, examining how collaborative performance and associated trade-offs evolve as edge models become stronger. Section 6.2 and Figure 3 investigate whether collaborative performance approaches cloud-only performance, how completion gaps across collaboration strategies evolve, and whether privacy-exposure differences persist.

In contrast, RQ3 focuses on the strategy-level implications of edge capability scaling, examining whether and how the relative effectiveness of different collaboration strategies changes across edge-capability levels. Section 6.3 and Table 2 compare external routing with edge-controlled invocation, as well as delegation with advisory collaboration. Figure 4 further investigates the edge-side capabilities underlying these differences.

Both questions are empirically testable. For RQ2, the observed performance convergence would not be supported if the completion gap between collaborative and cloud-only execution, or the completion gaps across collaboration strategies, did not generally narrow as edge capability increased. Similarly, persistent privacy differentiation would be challenged if exposure differences across strategies diminished substantially. For RQ3, the observed strategy shift would not be supported if the relative effectiveness of different collaboration strategies remained unchanged across edge-capability levels.

We will revise the manuscript to clarify the distinct scopes and complementary roles of RQ2 and RQ3, and explicitly connect each question to its corresponding experimental settings, evaluation dimensions, and empirical findings.

W2: Benchmark Aggregation and Complementarity

We thank the reviewer for this comment. We would like to clarify that, as described in the Data Collection section (Lines 247–253), our primary motivation for integrating multiple benchmark sources is to achieve broader and more complementary task coverage, rather than simply increasing the number of tasks. To make this complementarity more explicit, we provide a breakdown of task categories across the selected benchmark sources in the table below. For example, Claw-Eval primarily focuses on Office, Information Search, and Data Analysis, while the additional sources extend the coverage, particularly in Safety & Security and Automation. This complementarity allows EC²-Bench to cover a wider range of agentic scenarios without relying on the task distribution of a single benchmark.

We would also like to clarify that EC²-Bench goes beyond simply aggregating existing benchmarks. On top of multi-source task collection, we establish a unified collaborative execution and evaluation framework, apply collaboration-oriented difficulty filtering, and introduce fine-grained privacy annotations covering 3,385 sensitive information units. These efforts result in a unified evaluation suite of 128 tasks for systematically studying edge–cloud collaboration. We will incorporate this analysis into the revised manuscript to better clarify the rationale for combining these benchmark sources and their complementary contributions.

SourceOfficeInfo SearchData AnalysisDev & OpsSafetyAutomation
Claw-Eval (68)282011621
WildClawBench (22)58–36–
QwenClawBench (21)–41448
Others (17)321191
Total (128)363413142110
AuthorsOpenReview ↗

Response to Reviewer H1Kc [2/3]

W3: Cross-family Generalizability

We thank the reviewer for this suggestion. Our main experiments primarily use the Qwen3.5 series as edge models because it provides a particularly suitable and consistent capability ladder for our study, spanning 2B to 27B within the same model generation, with shared chat templates and tool-calling interfaces. This allows us to systematically vary edge-model capability while minimizing potential confounding factors beyond model capability, such as differences in training recipes and interaction interfaces. This controlled setup is essential for isolating how edge capability scaling affects edge–cloud collaboration. Moreover, Qwen is a widely adopted model family in the open-weight LLM ecosystem, making it a practically relevant choice for studying edge-agent deployment.

We also evaluated Ministral-3 models at 3B, 8B, and 14B scales. However, their limited agentic capabilities in our long-horizon task setting resulted in consistently low edge-only task completion rates (9.31%, 14.58%, and 15.51%, respectively), with little differentiation across model sizes. Consequently, they did not provide a sufficiently informative capability range for investigating how collaboration effectiveness evolves with increasing edge capability, and we therefore did not include them in the submitted version of the paper.

To further examine cross-family generalizability, we additionally evaluated IBM Granite-4.2 models at 3B, 8B, and 30B scales. As shown in the table below, the results are consistent with our main findings in two aspects: (1) the completion spread across collaboration strategies decreases from 6.54 pp at 3B to 2.97 pp at 30B; and (2) the relative effectiveness shifts from external routing toward edge-controlled invocation, and, within edge-controlled strategies, from Delegation toward Advisory. Specifically, Per-step Routing achieves the highest completion at 3B, whereas Advisory performs best at both 8B and 30B.

These results provide further evidence that our findings on capability-dependent collaboration trends are not specific to the Qwen family. We will include these additional experiments in the revised manuscript.

ModelMetricLocal-onlyQuery RoutingStep RoutingDelegationAdvisory
Granite-4.2-3BCompletion (%)21.9839.3341.3638.1734.82
Cost ($)0.008.068.806.929.97
Granite-4.2-8BCompletion (%)42.1751.1950.4150.8154.12
Cost ($)0.008.256.897.657.80
Granite-4.2-30BCompletion (%)58.3960.1960.7462.0763.16
Cost ($)0.005.963.135.784.99

W4: Cloud Model Generalizability

We thank the reviewer for raising this concern. We would like to clarify that, in addition to GLM-5.3 (main results), we have already conducted a complete set of collaboration experiments with DeepSeek-V4.1-Flash, a cloud model from a different family (Appendix A.8), covering all four edge models and four collaboration strategies. Both sets of experiments consistently support our main findings. Given the substantial computational cost of long-horizon agent evaluation (128 tasks × 4 edge models × 4 strategies × 3 runs per cloud model), we initially focused on these two cloud models.

We agree that evaluating an additional cloud model would further strengthen the evidence for cross-family generalizability. We therefore additionally evaluated Kimi-K3.1-Pro with Qwen3.5-2B/9B/27B across all four collaboration strategies. Under cloud-only execution, Kimi-K3.1-Pro achieves 80.64% task completion, 86.90% privacy exposure, and $52.12 cloud cost, serving as a reference for the collaboration results. As shown in the table below, the results exhibit trends consistent with our main findings, including performance convergence, persistent differences in privacy exposure, and a shift in strategy preference from external routing to edge-controlled Advisory as edge capability increases.

These additional results provide further evidence that our findings are not specific to a particular cloud model family. We will extend the evaluation section accordingly and discuss the implications of these results for cloud-model generalizability.

ModelMetricQuery RoutingStep RoutingDelegationAdvisory
Qwen3.5-2BCompletion (%)30.5223.2218.0816.14
Cost ($)10.2611.272.602.81
Privacy Exposure (%)18.7019.647.265.74
Qwen3.5-9BCompletion (%)50.4852.5153.7458.49
Cost ($)9.8910.874.3616.08
Privacy Exposure (%)17.9537.8015.4545.86
Qwen3.5-27BCompletion (%)66.3264.7168.0370.90
Cost ($)9.145.139.2910.26
Privacy Exposure (%)15.9532.3751.6550.25
AuthorsOpenReview ↗

Response to Reviewer H1Kc [3/3]

W5: Clarification of the Primary Research Area

We thank the reviewer for the suggestion. While EC²-Bench is an important component of our work, we would like to clarify that our primary contribution lies in the systematic empirical study of edge–cloud collaboration for long-horizon LLM agents, with EC²-Bench serving as the evaluation framework that enables this investigation.

Specifically, we evaluate representative edge–cloud collaboration strategies through three research questions: (1) whether edge–cloud collaboration is effective, (2) how increasing edge-model capability affects collaborative execution, and (3) how the relative effectiveness of collaboration strategies changes with edge-model capability. Beyond comparing different strategies, we further identify two key edge-side capabilities underlying effective collaboration—cloud-invocation decision-making and guidance utilization—and investigate the latter through a controlled instruction-following training experiment (Appendix A.10).

Thus, EC²-Bench serves as an evaluation infrastructure for studying LLM capabilities and behaviors in edge–cloud collaboration, rather than being the sole contribution of the paper. Our findings provide insights into the capabilities required for effective collaboration and the development of future edge models. We have therefore selected “Foundation or Frontier Models, including LLMs” as the primary research area.

W6: Code reproducing the results and how benchmark aggregation is performed is not shared with the reviewers.

We thank the reviewer for pointing this out. We have now released the code and dataset in an anonymized repository (https://anonymous.4open.science/r/ec2_anonymous-88ED/), enabling the reproduction of our experimental results.

[1] Xiao, Chaojun, et al. "Densing law of llms." Nature Machine Intelligence 7.11 (2025): 1823-1833.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Rebuttal Summary for AC

Dear Area Chair,

We would like to sincerely thank you and the reviewers for the time and effort devoted to evaluating our submission (ID: 43300). The reviews highlighted the timeliness and importance of the edge–cloud collaboration problem, recognized the value of EC²-Bench in providing a unified framework for systematically comparing edge–cloud collaboration in long-horizon agent execution, and consistently praised the paper's clear and easy-to-follow presentation. The reviewers' constructive feedback has further helped us clarify the contributions of our work and strengthen the empirical evidence supporting our findings.

During the discussion phase, we provided detailed point-by-point responses and further revised the manuscript. The main clarifications and additions are summarized below:

  • Robustness across model families (H1Kc, fa8R). We added experiments with IBM Granite-4.2 edge models and Kimi-K3.1-Pro as an alternative cloud model. Both evaluations yield findings consistent with our main results, supporting the robustness of our empirical conclusions.

  • Effect of task filtering (4k4S, fa8R). We clarified the motivation behind our task-filtering design and added a full evaluation on the original, unfiltered set of 156 tasks, including all 28 previously excluded tasks. The results are consistent with those on the filtered set, demonstrating that our main conclusions are not materially affected by the designed task-filtering procedure.

  • Reliability of empirical findings (4k4S, fa8R). We conducted paired bootstrap analyses and a budget-matched experiment, further supporting the reliability of our empirical findings.

  • Reproducibility (H1Kc, 4k4S). We have made the code and benchmark resources available through an anonymized repository (https://anonymous.4open.science/r/ec2_anonymous-88ED/) to facilitate reproducibility.

We are also encouraged by Reviewer 4k4S's follow-up assessment, which noted that our additional experiments and clarifications had addressed most of the concerns raised and that the new evidence had changed the reviewer's assessment.

We sincerely appreciate all reviewers' engagement throughout the discussion. We believe that the additional evidence and clarifications have further strengthened the paper, and we hope they will be taken into consideration in your final assessment.

Thank you again for your time and consideration.

Best regards,
The Authors

Dear reviewers, as the deadline is approaching, please check the rebuttal and consider participating in the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.