- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work introduces EdgeBench-Home, a bilingual, outcome-based benchmark (seven smart-home task families, evaluated under both state-given and state-seeking interfaces) for testing whether small on-device language models can act as smart-home agents, with an added deployment check on physical RK3588 hardware. Its main finding is that current local models in the ≤7B range remain far below API-scale models and unreliable on strict state-constrained tasks — especially under partial observation, and notably on automation registration and read-only queries — so they are better suited to gated assistance than autonomous control.
Critical strengths
- Matched case IDs across two interfaces. State-Given and State-Seeking share identical case IDs and initial states, differing only in information visibility and interaction, so SG→SS gaps reflect information condition rather than task variation — and capability is disentangled from information-seeking. This requires one simulator to support both single-turn generation and multi-step tool loops.
Critical weaknesses
-
Its novelty is compositional rather than frontier: every task family has precedent — valid/invalid instructions (HomeBench), personalization under partial observability (PersonalHomeBench), temporal dynamics (SimuHome), state-grounded reasoning (SMH-Bench) — leaving the systematic ≤7B small-model sweep as its only truly distinctive element.
-
Floor effects destroy resolution: six of seven ≤1B models score 0.0%, and FAR and REQ are 0.0% for the entire bilingual local pool under both interfaces, so tied zeros cannot distinguish "never capable" from "occasionally capable."
-
The safety framing is inflated: the introduction argues from "home control is safety-critical," yet the implementation reduces to three scoring add-ons — unrelated-state preservation, no prohibited effects, and no acting before clarifying — with no evaluation of any safety mechanism.
Questions for authors
NA.
Response to Reviewer oYHn
Thank you for the clear assessment. We agree that the task families are compositional and that several have precedents. We have narrowed the contribution claim: we do not claim that the individual task families or the supplied-state/interactive distinction are unprecedented. The contribution is a bilingual, outcome-based comparison of small and locally deployed model/platform configurations across seven household task families, using matched case IDs and task criteria for SG and SS, plus a separate RK3588 capability evaluation. We acknowledge overlap with HomeBench, SMH-Bench, and PersonalHomeBench and do not compare their aggregate scores directly because their cases, model pools, and scoring contracts differ.
We also agree about the floor effect. Six of seven ≤1B configurations have 0.0%, and all tested local configurations have 0% on FAR and REQ, which limits within-group resolution. The four-configuration Server reference group’s nonzero SS means on those same families (43.8% FAR; 40.2% REQ) demonstrate separation for the evaluated systems and show the tasks are passable under the scoring contract. This supports the benchmark’s ability to distinguish these tested groups, not a claim that the results alone establish construct validity or explain why local models fail.
Finally, we agree that “safety-critical” overstated what the implementation evaluates. The benchmark checks task-level constraints such as preserving unrelated state, avoiding prohibited effects, asking for clarification when required, and registering a future rule without executing it immediately. It does not test a deployed safety mechanism, establish real-world home safety, or provide a safety certification. We have removed that implication and frame these as task-level reliability constraints; consequential actions still require user supervision.