acceptodds
Under review as a conference paper at ICLR 2027

Are Small Language Models Ready for On-Device Agents? Benchmarking Small Language Models as Smart-Home Agents

Abstract

Small language models are increasingly used to build agents that run locally on resource-constrained devices. We introduce EdgeBench-Home, an executable benchmark for evaluating such agents through natural-language interaction with smart-home environments. Each episode specifies an environment state, an observation, user turns, typed tools, a resource budget, and an executable task target. The taxonomy covers seven task families spanning device control, multi-device selection, intent resolution, preference evolution, environment queries, automation, and robust tool interaction. A target-first construction pipeline generates candidate episodes and validates them through executable replay; the evaluator scores goal satisfaction, preservation, prohibited effects, and budget compliance while accepting semantically equivalent valid trajectories. The current core release contains 1,342 balanced bilingual episodes across five populated families. Our evaluation covers three model-size bands (≤1B, 1–3B, and 3–7B) and includes experiments on a device powered by the Rockchip RK3588 system-on-chip (SoC), with analyses of task capability and on-device resource behavior under different model and runtime settings.

Primary area:
datasets and benchmarks
Keywords:
Large language models (LLMs), small language models, AI agents, agent benchmarks, interactive evaluation, tool use, on-device AI, smart homes

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_oYHnOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces EdgeBench-Home, a bilingual, outcome-based benchmark (seven smart-home task families, evaluated under both state-given and state-seeking interfaces) for testing whether small on-device language models can act as smart-home agents, with an added deployment check on physical RK3588 hardware. Its main finding is that current local models in the ≤7B range remain far below API-scale models and unreliable on strict state-constrained tasks — especially under partial observation, and notably on automation registration and read-only queries — so they are better suited to gated assistance than autonomous control.

Critical strengths
  1. Matched case IDs across two interfaces. State-Given and State-Seeking share identical case IDs and initial states, differing only in information visibility and interaction, so SG→SS gaps reflect information condition rather than task variation — and capability is disentangled from information-seeking. This requires one simulator to support both single-turn generation and multi-step tool loops.
Critical weaknesses
  1. Its novelty is compositional rather than frontier: every task family has precedent — valid/invalid instructions (HomeBench), personalization under partial observability (PersonalHomeBench), temporal dynamics (SimuHome), state-grounded reasoning (SMH-Bench) — leaving the systematic ≤7B small-model sweep as its only truly distinctive element.

  2. Floor effects destroy resolution: six of seven ≤1B models score 0.0%, and FAR and REQ are 0.0% for the entire bilingual local pool under both interfaces, so tied zeros cannot distinguish "never capable" from "occasionally capable."

  3. The safety framing is inflated: the introduction argues from "home control is safety-critical," yet the implementation reduces to three scoring add-ons — unrelated-state preservation, no prohibited effects, and no acting before clarifying — with no evaluation of any safety mechanism.

Questions for authors

NA.

1 comment
AuthorsOpenReview ↗

Response to Reviewer oYHn

Thank you for the clear assessment. We agree that the task families are compositional and that several have precedents. We have narrowed the contribution claim: we do not claim that the individual task families or the supplied-state/interactive distinction are unprecedented. The contribution is a bilingual, outcome-based comparison of small and locally deployed model/platform configurations across seven household task families, using matched case IDs and task criteria for SG and SS, plus a separate RK3588 capability evaluation. We acknowledge overlap with HomeBench, SMH-Bench, and PersonalHomeBench and do not compare their aggregate scores directly because their cases, model pools, and scoring contracts differ.

We also agree about the floor effect. Six of seven ≤1B configurations have 0.0%, and all tested local configurations have 0% on FAR and REQ, which limits within-group resolution. The four-configuration Server reference group’s nonzero SS means on those same families (43.8% FAR; 40.2% REQ) demonstrate separation for the evaluated systems and show the tasks are passable under the scoring contract. This supports the benchmark’s ability to distinguish these tested groups, not a claim that the results alone establish construct validity or explain why local models fail.

Finally, we agree that “safety-critical” overstated what the implementation evaluates. The benchmark checks task-level constraints such as preserving unrelated state, avoiding prohibited effects, asking for clarification when required, and registering a future rule without executing it immediately. It does not test a deployed safety mechanism, establish real-world home safety, or provide a safety certification. We have removed that implication and frame these as task-level reliability constraints; consequential actions still require user supervision.

Reviewer_MZjwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces EdgeBench-Home, a bilingual benchmark for evaluating small language models as smart-home agents. The benchmark contains 840 cases across seven task families, including device control, ambiguity resolution, automation, dialogue context, personalized memory, and environment queries.

The authors evaluate models under two interfaces: State-Given, where the relevant environment information is directly provided, and State-Seeking, where the model needs to interact with tools to collect information. The evaluation checks actual actions and final environment states, rather than only the generated responses.

The paper evaluates 22 models, including three local checkpoints deployed on RK3588 hardware. Results show that most small models have limited task success, especially under State-Seeking.

Critical strengths
  1. I think the benchmark focuses on an important problem. For smart-home agents, checking the final environment state is more meaningful than only checking whether the model generates a valid tool call. I especially like that the evaluation considers prohibited actions and preservation of unrelated device states, which are important for reliability.

  2. The benchmark covers several useful dimensions, including different task families, two languages, different model sizes, and two interaction interfaces. The physical RK3588 evaluation is also a useful addition, since it gives some evidence about small models running on actual edge hardware.

Critical weaknesses
  1. My biggest concern is that many models achieve zero success on entire task families. For example, all the evaluated local models have 0% success on read-only environment queries in Tables 1 and 2. I am not sure whether this means the models cannot solve these tasks, or whether they mainly fail because of the required tool format or evaluation protocol. The paper needs a more detailed failure analysis to make these results meaningful. Otherwise, it is difficult to understand what capability is actually measured.

  2. I think the benchmark validation is still somewhat limited. Each task family has only 60 cases per language, and the 3–7B bilingual group contains only two models. The ambiguity-resolution evaluation also relies on an LLM judge without completed human agreement results. In addition, the RK3588 experiments mainly measure task success, without a detailed latency, memory, or energy analysis. These limitations make the conclusions about on-device agents less general.

Questions for authors
  1. Can the authors provide a failure breakdown for the task families with nearly zero success? For example, how many failures are caused by invalid tool calls, incorrect reasoning, wrong final states, or output formatting?

  2. Have the authors tried a simple tool-use fine-tuning or few-shot prompting baseline? I wonder whether some of the zero-success results can be improved with better protocol alignment rather than requiring much larger models.

  3. How reliable is the LLM judge used for ambiguity resolution? Can the authors provide human agreement results or a small manual validation study?

  4. In Section 3.3, the paper treats overall micro success and the unweighted family mean as separate quantities. Since all seven families have equal numbers of cases, should these two values be mathematically equal for the same model and language? Could the authors clarify this?

1 comment
AuthorsOpenReview ↗

Response to Reviewer MZjw

Thank you for highlighting the interpretation limits of the zero scores. We agree that 0% on a family creates a floor effect: it cannot distinguish a near miss from failure at an early interaction step, and it does not rank local models within that family. Under the same SS tasks and scoring contract, the four-configuration Server reference group averages 43.8% on FAR and 40.2% on REQ while the tested local configurations score 0%. This demonstrates separation among the evaluated groups, but does not by itself validate every item or identify local failure causes.

The archived P0 materials do not support a reliable FAR/REQ-specific breakdown into invalid calls, reasoning errors, wrong final states, or formatting failures. We therefore do not attribute the zeros to one cause. The SS interface adds tool selection and context integration, which are plausible contributors but were not isolated experimentally. Our primary comparison did not include a tool-use few-shot or fine-tuned baseline; we now state that the shared prompt protocol measures these evaluated setups, not each model’s best achievable performance, and treat protocol-aligned prompting/fine-tuning as future work.

Human experts reviewed all 840 benchmark cases. Separately, the authors manually checked every recorded AIR judge decision against the rubric; the constrained clarification judgments were straightforward. We did not conduct an independent blinded double annotation or calculate an inter-rater agreement coefficient, and the rolling service alias remains a reproducibility limitation. We now distinguish the full manual check from a formal agreement study.

You are right about aggregation: with 60 cases in each of seven families per language, the 420-case micro rate equals the unweighted mean of the seven family rates, up to rounding. We clarified this in the methods. For Qwen3-1.7B-16K on RK3588 under SS, the corrected workbook/family breakdown gives 23/420 Chinese and 4/420 English, hence 27/840 = 3.2% bilingual. The submitted English aggregate (0.1%) and bilingual text/figure (2.8%) were erroneous; the revised manuscript reports 5.5%, 1.0%, and 3.2% with denominators. We cannot claim submission-time episode-log verification because those P0 logs are absent locally.

We also clarify the judge’s rolling alias, the two-configuration size-band limitation, and the absence of aligned RK3588 latency/memory/energy results. Size trends remain descriptive, without significance claims.

Reviewer_PgrQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces EdgeBench-Home, a bilingual (Chinese and English) benchmark for evaluating smart-home agents. The benchmark consists of 840 test cases covering seven task categories. The authors design two evaluation settings, State-Given (SG) and State-Seeking (SS), and determine task success based on tool execution, final environment states, and constraint satisfaction. The experiments evaluate 18 local small language models (SLMs) and four API-based large language models (LLMs). Three of the local models are additionally deployed on the RK3588 platform. The results indicate that local SLMs exhibit substantial limitations when handling complex state-constrained tasks, particularly under the State-Seeking setting, where their success rates are relatively low.

Critical strengths

1.Strict evaluation based on final environment states. The benchmark evaluates task success by examining final environment states while also checking for erroneous operations and unintended changes to unrelated devices. This is more closely aligned with practical smart-home agent requirements than simply measuring function-calling accuracy.

2.Broad task coverage and complementary evaluation settings. The benchmark covers seven task categories and provides both full-state-information and partially observable settings, enabling systematic comparisons of model capabilities under different interaction conditions.

3.Relatively broad empirical evaluation. The experiments cover multiple model sizes, both Chinese and English, and deployment on the RK3588 platform. These evaluations provide useful empirical observations, and the authors appropriately qualify the scope of their conclusions.

Critical weaknesses

Major Weaknesses

1.Table 2 reports an SS success rate of 3.2% for Qwen3-1.7B-16K, while Figure 4 appears to show a mean of 2.8%. It is unclear whether these results use the same evaluation and aggregation procedures. In addition, the reported English success rate of 0.1% cannot be directly obtained from a single set of 420 binary-scored cases. The paper should clarify how these metrics are computed.

2.Section 3.2 states that the dataset was designed by domain experts, with natural-language instances generated using LLM assistance. However, the paper does not provide sufficient analysis of sample diversity, human verification results, or cross-language task difficulty. Consequently, it is difficult to assess whether the resulting benchmark is sufficiently representative.

3.Figure 3 shows that local models achieve a success rate of 0% on both FAR and REQ tasks. However, the paper does not provide a breakdown of failure types, such as parsing failures, incorrect tool calls, or semantic understanding errors. Without such analysis, it remains unclear whether the extremely low success rates fully reflect limitations in model capabilities or are partially attributable to other factors in the evaluation pipeline.

4.Existing benchmarks, including HomeBench, SMH-Bench, and PersonalHomeBench, already cover some of the relevant capabilities. The main contribution of EdgeBench-Home appears to lie in task integration and unified evaluation. However, the paper lacks a systematic comparison of capability coverage and additional experiments demonstrating which previously unobserved insights can be uniquely revealed by the proposed benchmark.

Minor Weaknesses

1.Limited validation of LLM-as-a-Judge evaluation. The AIR task relies on LLM-as-a-Judge, but the paper does not report human agreement studies. Furthermore, the evaluation uses a rolling service alias for the judge model, which may negatively affect reproducibility (Section 3.1).

2.Missing deployment efficiency metrics. Although the paper emphasizes on-device deployment, the RK3588 experiments report only task success rates. Practical efficiency measures, such as inference latency, memory consumption, and energy usage, are not provided (Section 4.3).

3.The 3-7B model group contains only two models with complete bilingual evaluations, and the model families are not consistent across size groups. Therefore, the evidence supporting model-size trends is relatively weak, although the authors appropriately frame these conclusions as descriptive observations.

4.The manuscript contains several clearly unfinished references, including SAGE (?) in Section 2 and (?) in Section 3.4, as well as repeated author names in some citations. These issues affect the completeness of the manuscript and make it more difficult to verify the related work.

Questions for authors

1.Table 2 reports 3.2% SS success for Qwen3-1.7B-16K, while Figure 4 shows 2.8%. Are these results computed using the same evaluation cases and averaging procedure? How is the reported 0.1% English success rate calculated?

2.Figure 3 shows that local models achieve 0% success on FAR and REQ tasks. What are the main failure modes? Are these failures primarily caused by incorrect tool calls, parsing errors, or limitations in task understanding?

3.Section 3.2 mentions that LLMs were used to generate task instances. How much of the dataset was manually reviewed, and what quality checks were performed to identify repetitive or overly similar instances?

4.How does EdgeBench-Home differ from existing benchmarks such as HomeBench and SMH-Bench in terms of the capabilities being evaluated? A more direct comparison of task coverage would help clarify its contribution.

5.Since on-device deployment is a main focus of the paper, could the authors report inference latency and memory usage for the RK3588 experiments in Section 4.3?

1 comment
AuthorsOpenReview ↗

Response to Reviewer PgrQ

Thank you for the detailed and constructive review. We have revised the manuscript and clarify the points below.

Qwen3-1.7B-16K SS discrepancy. The submitted Table 2 value (3.2%) is consistent with the mean over seven equally sized task families. We found an error in the submitted English overall aggregate used for Figure 4 and the accompanying text: the seven family-level counts sum to 4/420 (1.0%), not 0.1%. The corrected results are 23/420 Chinese (5.5%), 4/420 English (1.0%), and 27/840 bilingual (3.2%). We corrected the manuscript’s platform comparison and report the denominators. Thus the submitted 2.8% bilingual figure resulted from an erroneous aggregate, not a different averaging procedure. The separate 0.1% result for LFM2.5-1.2B is its bilingual rate, 1/840, rather than an English-only rate. The submission-time episode-level RK3588 P0 logs are not available in the local archive, so we do not claim log-level re-verification.

Case review and diversity. Human experts individually reviewed all 840 final Chinese and English cases. They checked wording and naturalness in both languages, relative difficulty across language tracks, duplicate or near-duplicate cases, and whether each case specifies its initial state, expected outcome, preservation requirements, and task restrictions. Release checks separately validate IDs, schema, and file integrity. These checks support item-level quality control, but do not establish statistical representativeness or validate model outputs or the AIR judge. The language tracks are matched by family counts and case structure, not literal translations.

Zero FAR/REQ scores and failure causes. We agree that zeros create a floor effect and cannot rank the tested local configurations within these families. In the same SS evaluation, the four-configuration Server reference group averages 43.8% on FAR and 40.2% on REQ, versus 0% for the tested local configurations. This shows separation between the evaluated groups and that the tasks are passable under the scoring contract; it is evidence of discriminative value for this comparison, not proof of complete benchmark validity. The available P0 artifacts do not provide reliable item-level attribution of local failures to parsing, tool use, or semantic reasoning. SS requires information seeking and integration, which may contribute, but we have not isolated that cause. Local runs share a 16,384-token context limit. We therefore report the zero as strict task-level outcome and do not infer a dominant failure mode.

Related benchmarks and contribution. We agree the contribution is compositional. EdgeBench-Home is specifically designed to characterize small models (≤7B), especially configurations intended to run locally; the contribution is not a claim that prior work never tested small models. The overlap is specific: HomeBench studies valid/invalid instructions, including single- and multi-device settings; SMH-Bench covers 1,100 tasks in seven categories and 22 subcategories and, like our work, includes supplied-state and interactive evaluation; PersonalHomeBench covers personalized household tasks across proactive/reactive, partially observed, and multimodal settings. Our release is smaller (840 bilingual cases), and we do not claim larger scale or uniquely novel task families or interaction conditions. What we add is a controlled comparison of local small-model/platform configurations, the same language-specific case IDs and task criteria across SG/SS, strict outcome/state scoring, and a physical RK3588 inference evaluation with simulated household state. Because the benchmarks use different instances, model pools, and scoring contracts, their aggregate scores are not directly comparable.

RK3588 efficiency. The P0 board experiment measures task capability while household state remains simulated. It does not include aligned latency, memory, or energy measurements. We have scoped the claim accordingly and do not use telemetry from a different P2 prompt/schema-repair profile as a substitute.

We also corrected the unfinished references and repeated citation issue, clarified that the 3–7B bilingual band has only two local configurations, and keep size-band findings descriptive. AIR uses a versioned rubric prompt with a rolling provider alias. The authors manually checked every recorded AIR judge decision against the rubric and found these constrained judgments straightforward. This was a full output check, but not an independent blinded double-annotation study, so we do not report an inter-rater agreement coefficient. We will update the OpenReview abstract metadata to match the 840-case, seven-family paper.

Comments on the paper

1 comment

Dear authors and reviewers,

The author-reviewer discussion period closes on November 18 (AoE). Please engage in response periods so that we have our issues discussed thoroughly.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.