acceptodds
Under review as a conference paper at ICLR 2027

SynthDCB: World Distillation for Scalable Device Control Benchmarking

Abstract

Scaling device-control evaluation requires scaling both the worlds agents interact with and the tasks they must solve. We introduce SynthDCB, a scalable benchmark for environment awareness and device control built through world distillation. World distillation transforms knowledge of device behaviors, interaction constraints, and environmental effects into modular executable components, which are composed into interactive worlds with explicit states, actions, and temporal dynamics. A unified capability representation connects tool discovery, world execution, and task synthesis, allowing new capabilities to enter evaluation through a shared construction pipeline. SynthDCB instantiates Home, Car, and Phone domains with 114 device classes, 1,166 functions, and 95 events, supporting 5 task families with 32 subtypes. The stateful design enables efficient task validation and agent evaluation through state correctness and tool-call traces. We evaluate seven models on 2,560 test tasks; the best achieves 68.2% Pass@1. Error analysis reveals challenges in answer delivery, feasibility diagnosis, and temporal coordination. Construction experiments demonstrate scalability across domains and token efficiency: world distillation reduces token consumption by 0.9–66.0% and 46.2–92.5% relative to Codex low and xhigh, respectively, while achieving broader capability coverage. SynthDCB makes executable world construction the basis of scalable device-control benchmarking, connecting an expanding space of capabilities and interactions to verifiable agent evaluation. The code and data are available at https://anonymous.4open.science/r/SynthDCB_ICLR.

Primary area:
datasets and benchmarks
Keywords:
LLM Agents, Tool Use, Agent Benchmarking, Executable Environments, World Modeling, Benchmark Generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6ddTOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a methodology to create evaluation environments to simulate realistic settings in which device agents can be deployed in. This covers device management across Home, Car, and Phone. Reusable components of the environments are leveraged to create them at lower cost, and rich, detailed objects capture the environment states and configurations which the agent acts on. The authors evaluate 7 models on the generated tasks to identify weaknesses and trends.

Critical strengths

S1: The problem is important as realistic device control benchmarks are rare, especially ones that capture detailed environment state

S2: The presentation of the cost in terms of tokens used is useful

S3: The pipeline is well-designed and scalable

Critical weaknesses

W1: I am not sure what new agent capabilities are being evaluated as compared to existing benchmarks.

W2: My major concern is about the consistency of the states. There is physics and temporal modeling involved and I do not see sufficient external (not within generator samples) validation as the state changes in the pipeline. While the generated specifications, tests, tasks are validated, the environment dynamics, e.g. time to cool a room and how it interacts with rate of cooling which in turn interacts with different temperature configurations of the air conditioner, do not have validation. There is a lack of physics simulator or grounding protocol validation (which is done by the cited SimuHome).

W3: I also do not see how this information can be discovered by agents. While there are a lot of env factors to model, it is unclear whether exploration to determine them is expected, or if the interaction budget is sufficient. I cannot tell if task failures reflect planning/reasoning ability or inferring simulator-specific dynamics which may be inconsistent.

W4: Answer submission seems to be a big contributor to errors, which can be lowered with a revised submission format or better prompts. It would allow for a better understanding of actual model behavior errors.

Questions for authors

Q1: In reference to W1, why is there an expectation of a difference in performance wrt existing stateful/tool-use benchmarks (like AppWorld and SimuHome)?

Q2: In reference to W3, how are the environment state dynamics exposed to the agent? I do not see any specific instructions in the prompts.

Q3: In reference to W4, could the submission format be tuned to reduce the errors? Why do these models get the submission incorrect? Is the specified format too complex or a lack of instruction-following?

4 comments

Thank you very much for your questions, suggestions, and time. Since submission, we have substantially updated the manuscript and added experiments, and we have added further analyses in response to your feedback. We have marked these changes in the revised manuscript. The main updates are as follows:

  1. We redesigned the evaluation agent and context protocol, and added 15 models, bringing the total to 22 (Table 4). We also added inference-cost (Figure 3) and interaction-budget analyses (Figure 5).
  2. We redesigned the error analysis (Figure 6), organizing failure mechanisms into four groups and twelve categories (Table 20), and added calibration with human experts (Appendix D.2). We also added nine failure case studies (Appendix D.3).
  3. We expanded the construction comparison with task-coverage analysis (Figure 7) and implementation case studies (Figure 8), showing that WD captures device-protocol capabilities and that the task-construction pipeline covers instructions from a real reference corpus.

We address the questions raised in your review below.

Q1. What is the relationship to existing stateful tool-use benchmarks?

SynthDCB focuses on turning device behavior knowledge into reusable executable components, then connecting tool discovery, world execution, and task construction through a unified capability representation. This design supports combined checks on device operations, actual environmental goals, temporal coordination, and partially feasible requests. For example, correctly setting an air conditioner's target temperature and achieving the required room temperature at the evaluation time are different checks; a partially infeasible request also requires completing its independent feasible parts.

Our advantage relative to existing benchmarks is therefore scalability. Validated world distillation allows us to rapidly model new devices and construct test worlds and tasks, so that we can evaluate whether a model transfers its existing capabilities to new devices and functions. This construction mechanism continually expands the executable capabilities on which agents can be evaluated. Performance reflects how agents discover and use the capabilities of each unfamiliar world and satisfy its combined requirements.

Q2.1. Grounding and validation of state dynamics (W2).

We abstract the core mechanisms of environmental state changes and manage them centrally through the simulator (Appendix A.3). SimuHome specifies Matter-based device behaviors and environmental update equations, while WD automatically constructs device-specific state transitions and environmental effects through a shared generation harness. LLM-based reviews and executable checks assess realistic properties and ranges, behavioral consistency, and agreement between specifications and implementations. The resulting environment is a parallel world with its own explicit rules, where each device follows its generated dynamics. Because some state changes are abstracted, agents must actively probe the instantiated environment, observe how device actions affect shared state, and infer the resulting dynamics and interactions. This ability to discover and adapt to an unfamiliar environment is central to what SynthDCB evaluates.

Q2.2. Information available to the agent and the exploration budget.

We aim to simulate an agent entering a real, unfamiliar environment, where it must discover the available interfaces and explore the environment's behavior. We therefore expose observations and discoverable capabilities instead of providing the implementation of every dynamic rule upfront. The agent can query space and device states and use session.sleep_time to observe environmental evolution. The prompt explicitly instructs it to explore the world, use actual observations, and account for gradual changes in space properties.

Two traces on the same task illustrate the difference. In Home task task_0475204, the instruction asks the agent to reduce bedroom 1's temperature by 7–9 degrees and end at 20–22 degrees, starting from 29 degrees:

  • GPT-6 Astra. After observing that the initial cooling plan overshoots, Astra monitors the room, checks other running devices, and disables the air exchanger. From the subsequent observations, it infers a warming rate of 0.4 degrees per minute, waits another 15 minutes, observes 20.9 degrees, and turns the air conditioner off. It completes the task in 14 turns. The trace shows the agent inferring dynamics through observation and adjusting its plan without receiving the rate upfront.
  • GLM-5.3 Flash. The model sets the air conditioner's target temperature to 21°C, verifies the setting, and finishes at turn 7. However, an active air exchanger continues drawing in air at 10°C. During background simulation, the room temperature falls to 10.4°C at the evaluation horizon, violating the required 20–22°C range. The model overlooks the interaction between devices and treats the verified air-conditioner setting as sufficient to achieve the environmental goal.

The interaction-budget analysis (Figure 5) shows that agents often continue interacting after their existing actions already suffice for eventual goal attainment. Most selected models' Goal-success curves largely plateau by 20 turns: extending the cutoff to 50 turns improves each GPT model by only 0.13 percentage points and DeepSeek by 3.1 points.

Q3. Why do these models get the answer submission wrong?

We redesigned the prompt and input/output protocol in the updated experiments and observed substantially higher success rates in the updated evaluation. We also redesigned the error analysis and added case studies. An incorrect answer submission can arise from misunderstanding the user's intent and submitting additional answers.

For example, in Appendix D.3, Case 9, Astra correctly identifies the requested targets but also submits the intermediate baseline reading. Across its 46 Q6 tasks, 43 of the 44 failed first attempts contain the exact target list together with an extra baseline answer (lines 1830–1847). Case 4 shows the same mechanism in an aggregate task: all six requested answers match, but the agent also submits two intermediate readings (lines 1763–1773). These extra submissions lower answer accuracy and cause answer success to fail, while recall of the requested items is preserved. The main-text analysis also shows that GPT-5.6 Terra substantially outperforms Astra on counting and enumeration (90.6% and 85.5% versus 37.5% and 4.3%, respectively). Together with the case studies, this highlights how overinterpreting intermediate observations as additional output requirements can undermine task success even in stronger reasoning models.

The revised taxonomy distinguishes task-wise coordination (E6), which captures this misunderstanding of required outputs, from protocol adherence (E2) and tool-call construction (E3), which capture structural and type errors (Table 20). Cases 1 and 2 provide examples of the latter: submitting a string where a numeric answer is required, and repeatedly producing an invalid response structure (lines 1730–1750). This analysis identifies the underlying source of an answer-submission failure and distinguishes it from the model's ability to select the correct targets.

Thank you again for your time and effort. We are looking forward to your reply and further discussion.

Thank you for your response. I particularly like the Q2.2 and Q3 analyses and discussion. However, my main concerns regarding W1 and W2 remain unresolved because while the explanation is helpful, it lowers the novelty of the work (W1) or does not provide much new information that can ensure (near-)deterministic checks for reliable evaluation (W2). I will maintain my score.

Thank you for taking the time to follow up. We’re glad you found the analysis of agent exploration in Q2.2 and the discussion of answer-submission errors in Q3 helpful. As we revise the paper, we will highlight SynthDCB’s contribution more clearly, particularly how it enables us to build new device environments at scale and evaluate agents in unfamiliar settings. Sim-to-real transfer will remain a focus of our research as we develop future works of SynthDCB and study how well behavior observed in simulation carries over to real devices and environmental dynamics.

Reviewer_rhhiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents a pipeline that leverages a stateful code simulator with controllable state to synthetically generate configurable and executable device control environments. The authors use this pipeline to generate a benchmark across three domains (Home, Car, Phone), evaluate it using seven models, and analyze the results.

The motivation is clear, and the pipeline is clearly designed and documented. However, its usefulness and ecological validity remain uncertain, and the results are unclear. I recommend rejection.

Critical strengths
  • The motivations are clear and encouraging: it is important to efficiently build environments for evaluating device control agents in an automated and extensible manner and to ensure that these environments can be examined rather than function as black boxes.
  • The pipeline is meticulously designed in accordance with the proposal and clearly decomposed into stages that can be checked, observed, and analyzed. The results are appropriately analyzed.
Critical weaknesses
  • The primary concern is the approach itself, which resembles an overly naive and simple toy environment based on naive assumptions:
    • The worlds are structurally generated and lack real diversity; they are merely combinations of some given properties.
    • The tasks are templated sentences and lack ecological validity.
    • Its legitimacy is unclear: e.g. the source and quality of the world behavior knowledge remains questionable
  • Although the authors document and acknowledge the findings, the results remain unclear, with mixed signals across the multiple dimensions evaluated. The analyses lack sufficient depth for readers to understand the underlying causes. Overall, it is difficult to derive useful information from the results section beyond the observation that different models behave differently across different dimensions.
Questions for authors

Where is the “world behavior knowledge” sourced from? Is any quantitative or qualitative analysis provided to justify the effectiveness of such knowledge?

3 comments

Thank you very much for your questions, suggestions, and time. Since submission, we have substantially updated the manuscript and added experiments, and we have added further analyses in response to your feedback. We have marked these changes in the revised manuscript. The main updates are as follows:

  1. We redesigned the evaluation agent and context protocol, and added 15 models, bringing the total to 22 (Table 4). We also added inference-cost (Figure 3) and interaction-budget analyses (Figure 5).
  2. We redesigned the error analysis (Figure 6), organizing failure mechanisms into four groups and twelve categories (Table 20), and added calibration with human experts (Appendix D.2). We also added nine failure case studies (Appendix D.3).
  3. We expanded the construction comparison with task-coverage analysis (Figure 7) and implementation case studies (Figure 8), showing that WD captures device-protocol capabilities and that the task-construction pipeline covers instructions from a real reference corpus.

We address the questions raised in your review below.

Q1. Sources of world behavior knowledge.

We use reference knowledge from multiple real sources for world distillation. Section 3.2 describes how stage-specific retrieval supplies domain documents and expert specification, code, and test examples. The reference-material inventory is as follows.

DomainReference material type and sourceCount
HomeDevice knowledge summaries based on manufacturer documentation (Xiaomi, Aqara, Huawei, ORVIBO, Samsung, LG, and others)71
HomeProduct pages, manuals, support articles, and technical references cited in these summaries249
HomeProduct-category catalog compiled from Huawei, Xiaomi, Aqara, and ORVIBO, with JD.com product verification1
CarTesla Model Y owner's manual sections29
CarLi Auto L9 Max and L8 Pro owner's manual sections2
CarOfficial product/configuration pages from AITO, HIMA, XPeng, and ZEEKR4
CarGoogle Android Automotive API reference (VehiclePropertyIds)1
CarExpert-curated implementation excerpts from industry12
PhoneGoogle Android API references and developer guides23
PhoneHuawei HarmonyOS official API documents21
PhoneOpenHarmony official API documents27
RobotVacuumInternal voice-skill and command-protocol worksheets covering commands, parameters, temporal rules, multi-intent handling, semantic frameworks, and vocabularies14
RobotVacuumDistinct real industrial test-case instructions854

Due to intellectual property restrictions, we cannot disclose the specific contents of these source materials.

In Section 5, we use an internal RobotVacuum protocol for a more detailed analysis. Table 5 shows that WD achieves broader complete capability coverage than Codex low at both L2 and L3, including when the supplied reference is a concise functional overview. With the complete protocol at L3, WD fully covers 22/23 property/query dimensions, 19/29 functionalities, and 30/39 effect/event requirements (Table 5).

We have added a task-coverage comparison between 854 real industrial test-case instructions and 1,000 validated tasks generated under each construction setting. With complete protocol information at L3, WD achieves the broadest coverage: 81.6% of the reference instructions and 69.8% of their 172 categorized intents (Figure 7).

We also provide an L1–L3 comparison with Codex xhigh in Figure 8. At L2 and L3, WD translates the supplied vacuum-then-mop or sweep-then-mop behavior into executable phase transitions, while Codex xhigh retains a generic cleaning state.

Together, these results and analyses demonstrate that our pipeline captures real-world device capabilities and task requirements across both device-model abstraction and task construction.

Q2. How does world distillation ensure task and world diversity?

We use templates to make task construction efficient and standardized, allowing manufacturers to plug new device models into the shared framework and rapidly generate corresponding worlds and tasks. As discussed in Q1, the RobotVacuum experiments in Section 5 demonstrate that both the constructed device models and generated task instructions cover real industrial capabilities and requests. Our instruction-style ablation also shows broadly comparable model performance with template-based instructions and their explicit rewritten variants. This construction approach therefore addresses the practical industrial need to rapidly evaluate agents on new devices and capabilities.

We construct device models through two layers: metadata and executable code. The metadata layer extracts key properties, functions, constraints, and environmental effects from external knowledge; the code layer implements these specifications under a shared runtime contract. Across Home, Car, and Phone, the resulting library contains 114 device classes, 1,166 functions, and 95 events (Table 7), implemented in 40k+ code lines, with 1952 unit tests in 60k+ code lines.

World composition varies device configurations, initial states, and available capabilities. Task builders then use these capabilities to construct queries, controls, environmental goals, combined requests, and temporal workflows, including dependencies and partial feasibility. Each newly introduced capability can therefore expand both the executable worlds and the tasks generated from them.

Q3. What can we derive from the result analysis?

We have added/updated several analyses and case studies in the revised manuscript:

  • Cost and performance. GPT-6.1 Terra achieves a similar overall success rate to GPT-5.6 Sol (79.5% versus 79.1%), with higher interaction efficiency at about 31% of Sol's cache-discounted inference cost (Figure 3; Section 4.4, lines 361–369; Appendix E, Table 22).
  • Interaction budget. The interaction-budget analysis shows that agents often continue interacting after their existing actions already suffice for eventual goal attainment. Most selected models' Goal-success curves largely plateau by 20 turns: extending the cutoff to 50 turns improves each GPT model by only 0.13 percentage points and DeepSeek by 3.1 points, while Gemini gains 14.7 points (Figure 5; Section 4.6, lines 417–425).
  • Instruction style and implicit requests. Direct instructions achieve the highest average Pass@1. Implicit instructions reduce full-task success more than component-level accuracy and recall, showing that models still achieve parts of the user's intent when details are omitted (Figure 4; Section 4.5, lines 372–377 and 404–414; Tables 13–14).
  • Failure mechanisms. The revised error analysis distinguishes failures in interpreting complex requests and coordinating subtasks in stronger models from protocol and tool-call errors in smaller models. Task-wise coordination (E6) accounts for 73.5% and 75.2% of weighted failures for GPT-6.1 Sol and GPT-6 Astra, respectively; protocol adherence (E2) accounts for 96.8% for Qwen3.5-2B (Figure 6; Section 4.7, lines 428–436).
  • Case study of failure traces. We added nine case studies connecting task requirements, observations, actions, and failed checks. These include extra answers caused by treating intermediate observations as requested outputs, timing errors despite correct final states, and distinct runtime or reference errors (Appendix D.3, Cases 1–9, lines 1730–1847).

These analyses explain how similar overall scores can arise from different capabilities, interaction strategies, and failure mechanisms. We hope they help clarify the results and welcome further questions.

Thank you again for your time and effort. We look forward to your reply and further discussion.

Thanks authors for the additional experiments and clarifications. I am maintaining my current score as my main concerns relate to the underlying methodology and largely remain unresolved.

Reviewer_nxteOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces SYNTHDCB, a benchmark for environment-aware device control built through world distillation. Device knowledge is converted into typed capability specifications, executable device models, tests, and reusable world components, from which Home, Car, and Phone worlds and verifiable tasks are generated. The benchmark contains 114 device classes, 1,166 functions, 95 events, five task families with 32 subtypes, and 2,560 test tasks. Seven models are evaluated.

Critical strengths
  1. The evaluation goes beyond checking whether tool calls succeed. SYNTHDCB incorporates changes in environmental state, temporal constraints, infeasible requests, and partially feasible requests.
  2. The benchmark combines three device domains, explicit state and temporal dynamics, 114 reusable device classes, and five task families.
Critical weaknesses
  1. The evaluation relies entirely on simulated data, and the authors did not collect real user conversations or device data. It is unclear whether the original inputs reflect real-world scenarios.
  2. The range of evaluated models is too limited, covering only a few closed-source models and including no comparison with open-source models.
  3. The paper describes an automated validation pipeline and states that all human-involved checks were conducted by the authors. However, it provides no quantifiable review criteria, making it difficult to assess the quality of the dataset.
Questions for authors
  1. Why does this paper not report the evaluation results of any open-source models?
2 comments

Thank you very much for your questions, suggestions, and time. Since submission, we have substantially updated the manuscript and added experiments, and we have added further analyses in response to your feedback. We have marked these changes in the revised manuscript. The main updates are as follows:

  1. We redesigned the evaluation agent and context protocol, and added 15 models, bringing the total to 22 (Table 4). We also added inference-cost (Figure 3) and interaction-budget analyses (Figure 5).
  2. We redesigned the error analysis (Figure 6), organizing failure mechanisms into four groups and twelve categories (Table 20), and added calibration with human experts (Appendix D.2). We also added nine failure case studies (Appendix D.3).
  3. We expanded the construction comparison with task-coverage analysis (Figure 7) and implementation case studies (Figure 8), showing that WD captures device-protocol capabilities and that the task-construction pipeline covers instructions from a real reference corpus.

We address the questions raised in your review below.

Q1. Coverage of open models.

Before the rebuttal, we had already expanded the evaluation from 7 API models to 13 API models and 5 locally deployed models. During the rebuttal, we evaluated 4 additional locally deployed models. Table 4 now reports all 22 models, including 2 API open-weight models: DeepSeek V4.1 Flash and GLM-5.3 Flash, and 9 local open-weight models: Qwen3.5-0.8B/2B/4B/9B/27B, MiniCPM5-1B/2B, and Gemma-4-E2B-it/E4B-it. We also added Figure 3 to compare inference cost and performance across API models.

Q2. Real-world grounding of device knowledge and scenarios.

We use reference knowledge from multiple real sources for world distillation. Section 3.2 describes how stage-specific retrieval supplies domain documents and expert specification, code, and test examples. The reference-material inventory is as follows.

DomainReference material type and sourceCount
HomeDevice knowledge summaries based on manufacturer documentation (Xiaomi, Aqara, Huawei, ORVIBO, Samsung, LG, and others)71
HomeProduct pages, manuals, support articles, and technical references cited in these summaries249
HomeProduct-category catalog compiled from Huawei, Xiaomi, Aqara, and ORVIBO, with JD.com product verification1
CarTesla Model Y owner's manual sections29
CarLi Auto L9 Max and L8 Pro owner's manual sections2
CarOfficial product/configuration pages from AITO, HIMA, XPeng, and ZEEKR4
CarGoogle Android Automotive API reference (VehiclePropertyIds)1
CarExpert-curated implementation excerpts from industry12
PhoneGoogle Android API references and developer guides23
PhoneHuawei HarmonyOS official API documents21
PhoneOpenHarmony official API documents27
RobotVacuumInternal voice-skill and command-protocol worksheets covering commands, parameters, temporal rules, multi-intent handling, semantic frameworks, and vocabularies14
RobotVacuumDistinct real industrial test-case instructions854

Due to intellectual property restrictions, we cannot disclose the specific contents of these source materials.

In Section 5, we use an internal RobotVacuum protocol for a more detailed analysis. Table 5 shows that WD achieves broader complete capability coverage than Codex low at both L2 and L3, including when the supplied reference is a concise functional overview. With the complete protocol at L3, WD fully covers 22/23 property/query dimensions, 19/29 functionalities, and 30/39 effect/event requirements (Table 5).

We have added a task-coverage comparison between 854 real industrial test-case instructions and 1,000 validated tasks generated under each construction setting. With complete protocol information at L3, WD achieves the broadest coverage: 81.6% of the reference instructions and 69.8% of their 172 categorized intents (Figure 7).

We also provide an L1–L3 comparison with Codex xhigh in Figure 8. At L2 and L3, WD translates the supplied vacuum-then-mop or sweep-then-mop behavior into executable phase transitions, while Codex xhigh retains a generic cleaning state.

Together, these results and analyses demonstrate that our pipeline captures real-world device capabilities and task requirements across both device-model abstraction and task construction.

Q3. Dataset quality.

Our pipeline is designed for automated execution and efficient generation. Table 6 shows that 433 of 456 class-stage jobs (95.0%) complete within three executions, with automated checks and repair throughout device-model construction. Task construction likewise uses automated execution and replay to support efficient batch generation and validation.

The error analysis in Figure 6 further shows that task-design or evaluation issues (E12), including incorrect LLM-judge decisions, contribute to only 212 of the 56,320 first attempts (0.38%, 0.32% after contribution weighting). This provides quantitative evidence of the quality achieved by the automated pipeline.

The human checks mentioned in the Ethics Statement primarily concern targeted inspection and correction of flagged issues—for example, examining cases flagged as E12 and repairing confirmed task-design defects or evaluation errors. Human intervention is not a mandatory step in the generation of every device model or task.

Thank you again for your time and effort. We look forward to your reply and further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.