- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes a methodology to create evaluation environments to simulate realistic settings in which device agents can be deployed in. This covers device management across Home, Car, and Phone. Reusable components of the environments are leveraged to create them at lower cost, and rich, detailed objects capture the environment states and configurations which the agent acts on. The authors evaluate 7 models on the generated tasks to identify weaknesses and trends.
Critical strengths
S1: The problem is important as realistic device control benchmarks are rare, especially ones that capture detailed environment state
S2: The presentation of the cost in terms of tokens used is useful
S3: The pipeline is well-designed and scalable
Critical weaknesses
W1: I am not sure what new agent capabilities are being evaluated as compared to existing benchmarks.
W2: My major concern is about the consistency of the states. There is physics and temporal modeling involved and I do not see sufficient external (not within generator samples) validation as the state changes in the pipeline. While the generated specifications, tests, tasks are validated, the environment dynamics, e.g. time to cool a room and how it interacts with rate of cooling which in turn interacts with different temperature configurations of the air conditioner, do not have validation. There is a lack of physics simulator or grounding protocol validation (which is done by the cited SimuHome).
W3: I also do not see how this information can be discovered by agents. While there are a lot of env factors to model, it is unclear whether exploration to determine them is expected, or if the interaction budget is sufficient. I cannot tell if task failures reflect planning/reasoning ability or inferring simulator-specific dynamics which may be inconsistent.
W4: Answer submission seems to be a big contributor to errors, which can be lowered with a revised submission format or better prompts. It would allow for a better understanding of actual model behavior errors.
Questions for authors
Q1: In reference to W1, why is there an expectation of a difference in performance wrt existing stateful/tool-use benchmarks (like AppWorld and SimuHome)?
Q2: In reference to W3, how are the environment state dynamics exposed to the agent? I do not see any specific instructions in the prompts.
Q3: In reference to W4, could the submission format be tuned to reduce the errors? Why do these models get the submission incorrect? Is the specified format too complex or a lack of instruction-following?
4 comments
Q2.2. Information available to the agent and the exploration budget.
We aim to simulate an agent entering a real, unfamiliar environment, where it must discover the available interfaces and explore the environment's behavior. We therefore expose observations and discoverable capabilities instead of providing the implementation of every dynamic rule upfront. The agent can query space and device states and use session.sleep_time to observe environmental evolution. The prompt explicitly instructs it to explore the world, use actual observations, and account for gradual changes in space properties.
Two traces on the same task illustrate the difference. In Home task task_0475204, the instruction asks the agent to reduce bedroom 1's temperature by 7–9 degrees and end at 20–22 degrees, starting from 29 degrees:
- GPT-6 Astra. After observing that the initial cooling plan overshoots, Astra monitors the room, checks other running devices, and disables the air exchanger. From the subsequent observations, it infers a warming rate of 0.4 degrees per minute, waits another 15 minutes, observes 20.9 degrees, and turns the air conditioner off. It completes the task in 14 turns. The trace shows the agent inferring dynamics through observation and adjusting its plan without receiving the rate upfront.
- GLM-5.3 Flash. The model sets the air conditioner's target temperature to 21°C, verifies the setting, and finishes at turn 7. However, an active air exchanger continues drawing in air at 10°C. During background simulation, the room temperature falls to 10.4°C at the evaluation horizon, violating the required 20–22°C range. The model overlooks the interaction between devices and treats the verified air-conditioner setting as sufficient to achieve the environmental goal.
The interaction-budget analysis (Figure 5) shows that agents often continue interacting after their existing actions already suffice for eventual goal attainment. Most selected models' Goal-success curves largely plateau by 20 turns: extending the cutoff to 50 turns improves each GPT model by only 0.13 percentage points and DeepSeek by 3.1 points.
Q3. Why do these models get the answer submission wrong?
We redesigned the prompt and input/output protocol in the updated experiments and observed substantially higher success rates in the updated evaluation. We also redesigned the error analysis and added case studies. An incorrect answer submission can arise from misunderstanding the user's intent and submitting additional answers.
For example, in Appendix D.3, Case 9, Astra correctly identifies the requested targets but also submits the intermediate baseline reading. Across its 46 Q6 tasks, 43 of the 44 failed first attempts contain the exact target list together with an extra baseline answer (lines 1830–1847). Case 4 shows the same mechanism in an aggregate task: all six requested answers match, but the agent also submits two intermediate readings (lines 1763–1773). These extra submissions lower answer accuracy and cause answer success to fail, while recall of the requested items is preserved. The main-text analysis also shows that GPT-5.6 Terra substantially outperforms Astra on counting and enumeration (90.6% and 85.5% versus 37.5% and 4.3%, respectively). Together with the case studies, this highlights how overinterpreting intermediate observations as additional output requirements can undermine task success even in stronger reasoning models.
The revised taxonomy distinguishes task-wise coordination (E6), which captures this misunderstanding of required outputs, from protocol adherence (E2) and tool-call construction (E3), which capture structural and type errors (Table 20). Cases 1 and 2 provide examples of the latter: submitting a string where a numeric answer is required, and repeatedly producing an invalid response structure (lines 1730–1750). This analysis identifies the underlying source of an answer-submission failure and distinguishes it from the model's ability to select the correct targets.
Thank you again for your time and effort. We are looking forward to your reply and further discussion.
Thank you for your response. I particularly like the Q2.2 and Q3 analyses and discussion. However, my main concerns regarding W1 and W2 remain unresolved because while the explanation is helpful, it lowers the novelty of the work (W1) or does not provide much new information that can ensure (near-)deterministic checks for reliable evaluation (W2). I will maintain my score.
Thank you for taking the time to follow up. We’re glad you found the analysis of agent exploration in Q2.2 and the discussion of answer-submission errors in Q3 helpful. As we revise the paper, we will highlight SynthDCB’s contribution more clearly, particularly how it enables us to build new device environments at scale and evaluate agents in unfamiliar settings. Sim-to-real transfer will remain a focus of our research as we develop future works of SynthDCB and study how well behavior observed in simulation carries over to real devices and environmental dynamics.
Thank you very much for your questions, suggestions, and time. Since submission, we have substantially updated the manuscript and added experiments, and we have added further analyses in response to your feedback. We have marked these changes in the revised manuscript. The main updates are as follows:
We address the questions raised in your review below.
Q1. What is the relationship to existing stateful tool-use benchmarks?
SynthDCB focuses on turning device behavior knowledge into reusable executable components, then connecting tool discovery, world execution, and task construction through a unified capability representation. This design supports combined checks on device operations, actual environmental goals, temporal coordination, and partially feasible requests. For example, correctly setting an air conditioner's target temperature and achieving the required room temperature at the evaluation time are different checks; a partially infeasible request also requires completing its independent feasible parts.
Our advantage relative to existing benchmarks is therefore scalability. Validated world distillation allows us to rapidly model new devices and construct test worlds and tasks, so that we can evaluate whether a model transfers its existing capabilities to new devices and functions. This construction mechanism continually expands the executable capabilities on which agents can be evaluated. Performance reflects how agents discover and use the capabilities of each unfamiliar world and satisfy its combined requirements.
Q2.1. Grounding and validation of state dynamics (W2).
We abstract the core mechanisms of environmental state changes and manage them centrally through the simulator (Appendix A.3). SimuHome specifies Matter-based device behaviors and environmental update equations, while WD automatically constructs device-specific state transitions and environmental effects through a shared generation harness. LLM-based reviews and executable checks assess realistic properties and ranges, behavioral consistency, and agreement between specifications and implementations. The resulting environment is a parallel world with its own explicit rules, where each device follows its generated dynamics. Because some state changes are abstracted, agents must actively probe the instantiated environment, observe how device actions affect shared state, and infer the resulting dynamics and interactions. This ability to discover and adapt to an unfamiliar environment is central to what SynthDCB evaluates.