acceptodds
Under review as a conference paper at ICLR 2027

How Extractable Are Edge LLMs?

Abstract

Edge large language models (Edge LLMs) are commonly quantized for deployment on devices with limited memory and compute. At the same time, their text interfaces allow an attacker to collect responses and train a student model, creating a risk of black-box behavioral extraction. For such attacks, collecting more target responses provides additional supervision; however, slow on-device inference makes large-scale data collection costly. We therefore investigate whether an attacker can extract instruction-following behavior more effectively under a limited target-query budget. To assess this risk, we introduce Coordinated Linguistic Instruction Querying (CLIQ), a structured probing framework designed to reduce repeated probing of similar behaviors by coordinating queries across semantic regions. Specifically, CLIQ partitions a public instruction pool and generates region-conditioned queries before observing any target response. We compare it with Original Queries (OQ), which are randomly sampled from the same pool and used without modification. Under a nominal 500-query budget, CLIQ improves a 4B student's BERT-F1 from 77.92 with OQ to 82.44 on Alpaca evaluation references. Furthermore, experiments with two phone-deployed Gemma teachers show improved student performance over OQ while target-side generation speeds remain comparable. Taken together, these results demonstrate that functional instruction-following behavior remains transferable from quantized edge models under the evaluated query budgets: the cost of collecting target responses does not, by itself, establish resistance to behavioral extraction.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
edge LLMs, quantization, model extraction, knowledge extraction, query-efficient distillation, structured querying, security

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Anp2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates functional transfer from quantized edge large language models under a limited text-only query budget. It proposes CLIQ, which partitions a public instruction pool into semantic regions, filters out regions with insufficient sample support, and uses an independent generator to produce an equal number of queries for each retained region. Responses collected from the target model are then used to fine-tune a student model via LoRA. The evaluation covers off-device Qwen and Llama configurations, as well as two Gemma teachers deployed on a smartphone. Student outputs are evaluated primarily using BERT-F1, BLEU, and ROUGE against the original Alpaca reference responses. The reported results show that CLIQ generally outperforms random querying across the evaluated configurations and exceeds two diversity-aware selection baselines in the main comparison using a 4B student.

Critical strengths
  1. The complete pipeline demonstrates clear and practically meaningful empirical gains under a limited query budget.** With the same target model and 500 training steps, CLIQ achieves a BERT-F1 score of 82.44±0.07 for the 4B student using 495 target queries, outperforming OQ (77.92±0.09) and K-Center Greedy (80.45±0.08), both of which use 500 queries. Table 4 reports consistent improvements across two Llama target configurations. These differences substantially exceed the reported training standard deviations across three runs with fixed query sets, providing valuable empirical evidence for the importance of query construction under limited target access.
  2. Real-device validation provides concrete evidence in an edge deployment setting.** The paper collects target responses on a Pixel 10 Pro Fold using the Tensor G5 NPU and LiteRT-LM, covering two Gemma teachers and two student sizes. Under a common 200-step training protocol, CLIQ outperforms OQ on all six reported metrics across all four teacher–student combinations.
Critical weaknesses
  1. The paper does not sufficiently explain the additional value of extracting the edge target given the resources available to the attacker. It uses Qwen3-30B-A3B-Instruct-2507 to generate queries, then collects supervisory responses from a 7B Qwen model or smaller phone-deployed Gemma models. Parameter count alone does not establish relative capability, and using an independent generator is consistent with the stated threat model. However, the experiments primarily evaluate student outputs against Alpaca references, without demonstrating that the target provides domain knowledge or customized behavior that the attacker’s existing generator cannot readily supply. Nor do they compare against directly training the student on the generator’s responses under a comparable training budget. Thus, the results show that target responses can improve student performance, but do not establish the practical benefit of accessing the edge target over using the generator already available to the attacker.
  2. The paper does not sufficiently establish the incremental value of CLIQ’s region-coordination design over ordinary generative querying. CLIQ combines semantic clustering, equal query allocation across regions, representative-example retrieval, and instruction generation. However, OQ, K-Means Representative, and K-Center Greedy in Table 2 all use existing instructions from the public pool, whereas only CLIQ constructs new queries using a generator. Its 1.99-point BERT-F1 advantage over K-Center Greedy therefore reflects changes in both query organization and query generation, making it difficult to distinguish the contribution of regional coordination from the benefits of generating new tasks. Given that prior work cited in the paper, such as Self-Instruct, has already established the use of synthetic instructions for training, a generative baseline using the same generator, comparable generation resources, and matched target-query and student-training budgets, but without regional coordination, is crucial for assessing the method’s incremental value. I believe the current evidence supports the superiority of the complete CLIQ pipeline over the evaluated selection-based baselines, but does not yet establish additional benefits from its core coordination design. This issue warrants further discussion.
Questions for authors
  1. Given that the attacker already has access to Qwen3-30B-A3B-Instruct-2507, what additional capability or practical benefit motivates extracting the evaluated edge targets? Have you compared training the student on target responses with training it on the generator’s responses to the same queries under a matched student-training budget? If this comparison is unavailable, please clarify what target-specific value the current evaluation establishes.
  2. Have you evaluated a generative-query baseline using the same generator, comparable generation resources, and matched target-query and student-training budgets, but without semantic-region coordination? If such results are available, how much of CLIQ’s advantage remains? Otherwise, please clarify whether the current evidence establishes a benefit from coordination beyond that of generating new queries.
Reviewer_SJvfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors study the problem of query-efficient knowledge distillation from edge LLMs. They propose CLIQ, a structured query generation method, to improve behavioral extraction efficiency compared to random query sampling.

Critical strengths

The paper provides comprehensive experimental evaluations across different model families, student sizes, and quantization settings. The authors further validate their approach on real mobile devices, demonstrating its effectiveness in practical edge deployment scenarios.

Critical weaknesses

Incomplete accounting of query generation costs. CLIQ relies on a relatively large model (Qwen3-30B-A3B) to generate queries, but the computational cost of this process is excluded from the query budget. A more comprehensive comparison of the overall cost efficiency would strengthen the paper.

Limited real-device evaluation. Although the mobile experiments are a strength, they are conducted on only one device, with single-run results and without diversity-aware selection baselines such as K-Center Greedy. This limits the strength and generalizability of the conclusions drawn from the real-device experiments.

Questions for authors

What is the main advantage of CLIQ over semantic clustering-based selection methods? Both K-Means and K-Center already demonstrate that improving semantic diversity can lead to more efficient knowledge distillation. The additional gains from CLIQ seem to come primarily from the higher-quality synthetic queries generated by another LLM, rather than the clustering strategy itself. Could the authors clarify what specifically contributes to CLIQ's improvement, ideally through an ablation study?

Why is BERT-F1 evaluated against the original Alpaca references rather than the teacher model's outputs? Since the paper focuses on behavioral extraction, shouldn't the evaluation measure how closely the student reproduces the teacher's behavior? Comparing against reference answers seems to evaluate general instruction-following quality rather than the effectiveness of model extraction.

Reviewer_mGdnOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focus on distillation attack on black-box models on edge device like mobile phones. Attack must be constrained by number of queries since the mobile devices are not capable to inference efficiently. Intuitively the instructions that is sent to attack should be as much diverse as possible. Therefore the paper propose CLIQ to enrich the diversity of attack instructions. CLIQ first partitions the embedded public instructions in Alpaca data set into clusters then select representative points as centroids. Use data points near centroids to formulate prompts for LLM generators (which is Qwen3-30B-A3B-Instruct-2507 in this research) to produce more diverse attack queries. After receive responses from the black-box teacher, performing SFT training students with query-response pairs. The main result indicate that CLIQ gains
4.52% increase compared to random query baseline. The results with the Gemma 4 teacher on mobile devices show a relatively modest average improvement of approximately 0.85 percentage points in BERT-F1, compared to the substantially larger gains observed in the main Qwen experiments.

Critical strengths
  1. The main experiments are adequate for model performance assessment. Experiments across multiple model families and scales show that the CLIQ method significantly outperforms the baseline.
Critical weaknesses

1.There are two main stage in CLIQ: first cluster the public queries then use LLM generator to produce diverse queries. However in the paper there is no ablation experiment on the two main stages. Lacks further discussion on the utility of clustering or query-generation separately in CLIQ.
2. All OQ experiments are based on the same batch of random queries which is not robust. In table 7 the seed of random query construction is fixed. The random choice of query can lead to different response sampling results. The OQ baseline may simply reflect an unlucky random draw. The research should discuss whether the CLIQ can beat the variance of random query choice.

Questions for authors

Q1. The research focus on edge LLMs . According to the results from mobile phone Gemma model, the extent of improvement is rather modest compared to main results from Qwen series model. Could the authors provide further insights into the potential factors underlying this discrepancy? For example, could differences in quantization configurations or teacher model families contribute to the observed gap?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.