acceptodds
Under review as a conference paper at ICLR 2027

Choosing, Not Generating: Option Selection, Learned Preconditions, and Temporal Consistency Checks for Smart-Home Agents

Abstract

Language-model agents that operate a smart home by issuing one tool call at a time often fail on requests that involve timing, dependencies between devices, or a device or time that the user states incorrectly. Such an agent infers again at every step, from its growing history, what each command requires and when each action should happen. We observe that many of the decisions it makes are selections among options that the tool interface already lists, such as which device a phrase refers to, and that the times follow by computation from values the devices report. We therefore use text generation only to convert the request into a list of atomic goals, in one language-model call. A second model makes these selections: given a question and a list of options, it returns a probability distribution over the options instead of generating text. It maps each goal to a device, or to none when the named device does not exist, and chooses the goal's action and arguments. Code computes all times, checks with a simple temporal network whether the request's timing constraints are consistent, and either executes the request or explains the conflict. Skills learned from demonstrations, annotated with the preconditions of their actions, add the actions that others require first on SimuHome, a smart-home benchmark, and serve as the plans on TimeArena, a benchmark of concurrent time-consuming tasks. On SimuHome, the agent reaches 92.8% and 89.4% success with Gemma 4 12B and Qwen 3.5 9B, 30.5 and 31.6 points above the strongest baseline that uses the same demonstrations; with Gemma 4 12B, it makes about one backbone call per episode instead of about seven for ReAct and uses 13 times fewer backbone tokens. On TimeArena, it completes every episode that can be completed without violating the task instructions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Tool-use agent, Skill learning, Temporal planning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3bq7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework for smart-home agents that replaces repeated LLM-based decision-making with a combination of typed option selection, learned skill preconditions, and deterministic temporal reasoning. The key idea is that many decisions in tool-using agents can be formulated as selecting from predefined options rather than generating text. The framework uses an LLM to extract task goals, a specialized decision model (Jev) to select devices and actions, and rule-based components to handle temporal constraints and execution. Experiments on SimuHome and TimeArena demonstrate substantial improvements in task success and efficiency compared with existing agent baselines.

Critical strengths
  • Clear motivation and interesting perspective. The paper is well written, and the motivation is clearly presented. I appreciate the idea that not all agent decisions require LLM generation, and that separating different types of decisions can improve reliability and efficiency.

  • Strong experimental results. The proposed framework outperforms existing baselines, including ReAct, Reflexion, and ExpeL, particularly on tasks involving temporal constraints and dependencies.

  • Comprehensive evaluation and useful ablations. The paper evaluates different task types, including infeasible requests and concurrent scheduling. The ablation studies provide useful evidence about the contributions of individual components, particularly temporal consistency checking and the explicit none option.

Critical weaknesses
  • Limited methodological novelty. While the proposed framework achieves impressive results, the overall contribution feels more like an engineering effort than a novel algorithmic contribution. Most components are based on existing techniques, and the design choices, although effective, are relatively intuitive. I find the empirical improvements convincing, but the paper does not seem to provide particularly surprising or novel algorithmic insights beyond combining these components.

  • Heavy reliance on hand-crafted rules and unclear generalizability. The framework relies on numerous manually designed rules for device grounding, temporal parsing, execution ordering, and tool-call repair. It remains unclear how well this approach would generalize to new smart-home settings with different devices, tool interfaces, or operational requirements. Although the paper evaluates a held-out device type, a more systematic investigation of generalization and the dependence on domain-specific engineering would strengthen the contribution.

  • Limited contribution from some proposed components. The ablation studies suggest that the temporal consistency check and the explicit none option account for most of the performance improvements. In contrast, removing learned contracts only reduces overall success from 92.3% to 90.5%, while replacing the specialized decision model with LLM-based decisions results in a similarly small accuracy difference (92.3% vs. 91.8%), although it increases latency. This raises questions about how much the proposed skill-learning and typed-decision components contribute to task success relative to the simpler deterministic mechanisms.

Questions for authors

See weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 3bq7 (1/3)

We thank the reviewer for the positive assessment of our paper's clarity, motivation, and empirical results. We address each of the three concerns with new experiments below.

W1. Methodological novelty: Why the division of decisions is non-trivial

While individual building blocks (schema-constrained extraction, multiple-choice selection, skill compression, and Simple Temporal Networks) have roots in prior literature, how an agent decomposes and routes tool-use decisions across them is a fundamental architectural question. Prior hybrid agents (LLM+P, LLMCompiler, CodeAct) still rely on a generative LLM to synthesize formal plans, dependency DAGs, or executable code that jointly handles entity grounding, action mapping, and temporal constraints.

To test whether our performance is simply a consequence of combining an LLM with domain rules and an external temporal/execution engine, we ran a matched comparison (Table R1) where LLM+P, LLMCompiler, and CodeAct are given the exact same interface rules (Table 3, Appendices B.3 and B.6), the exact same STN (Section 3.4, Appendix B.5), and the exact same execution machinery (Section 3.5) and budget (400 tool calls, 900 s).

Table R1. SimuHome success (%) of structured-output, hybrid-planner, and code-generation baselines given the same interface knowledge (Table 3, including Appendices B.3 and B.6) and the same execution machinery (Section 3.4 and Appendix B.5; Section 3.5 without learned components). All methods use the same backbone and the same budget of 400 tool calls and 900 s per episode; means over three seeds (42–44). Superscripts follow Table 1a: S marks evaluation by the simulator's state and J by the panel of LLM judges.

MethodQT1 F^JQT1 IF^JQT2 F^SQT2 IF^JQT3 F^SQT3 IF^JQT4-1 F^SQT4-1 IF^JQT4-2 F^SQT4-2 IF^JQT4-3 F^SQT4-3 IF^JHeld-outOverall
Gemma 4 12B
LLM+P + rules & exec.88.077.392.078.090.768.783.398.776.798.776.782.090.884.2
LLMCompiler + rules & exec.78.774.791.375.396.780.782.094.766.734.064.748.074.773.9
CodeAct + rules & exec.90.088.090.781.394.790.779.396.792.094.768.081.393.787.3
Ours92.796.790.778.098.0100.097.3100.092.095.389.383.398.392.8
Qwen 3.5 9B
LLM+P + rules & exec.73.357.391.378.086.750.060.0100.082.792.050.071.383.974.4
LLMCompiler + rules & exec.84.074.775.363.381.390.045.392.761.349.340.746.064.467.0
CodeAct + rules & exec.86.094.788.762.795.392.068.789.388.775.352.769.383.380.3
Ours76.096.791.379.395.3100.090.7100.090.795.385.372.796.689.4

Interpretation of Table R1: Giving existing hybrid and code planners our exact rules, STN, and execution engine still leaves a gap of 5.5 to 18.9 points on Gemma and 9.1 to 22.4 points on Qwen (and up to +32.6 points on multi-device concurrent scheduling QT4-3 F). When an LLM is asked to generate PDDL plans, DAGs, or Python code, it conflates natural-language extraction with symbol grounding and temporal wiring. Our core algorithmic insight—that generation should only extract ungrounded literal mentions, while symbol grounding and action/argument binding should be posed as factored, closed-set selections over live interface primitives—is precisely what closes this gap.

AuthorsOpenReview ↗

Response to Reviewer 3bq7 (2/3)

W2. Dependence on hand-crafted rules and generalizability to new devices, tool interfaces, and operational requirements

We conducted three systematic experiments to quantify rule dependence and test zero-shot generalization across devices, interfaces, and physical requirements (Tables R2, R5, and R6).

1. Ablating all hand-crafted rules (Table R2). We removed every rule of Table 3 at once across 5 seeds.

Table R2. Contribution of the description rules (Table 3) on SimuHome, Gemma 4 12B, mean over five seeds (42–46; 600 episodes each). Columns report success on feasible (F), infeasible (IF) and infeasible scheduling (QT4 IF) episodes and overall (%), precision (%), and agent time per episode (s). "No description rules" removes every component of Table 3 at once and leaves all other components unchanged.

VariantFIFQT4 IFOverallPrecisionTime (s)
Ours92.892.993.192.890.511.5
No description rules (all of Table 3)60.382.488.971.352.711.6

Interpretation of Table R2: Even with zero description rules, our agent achieves 71.3% overall, 82.4% on infeasible episodes (IF), and 88.9% on infeasible scheduling (QT4 IF), outperforming every baseline in Table 1a (ExpeL 62.3%, Reflexion 52.7%, Workflow Memory 45.7%, ReAct 44.8%) by +9.0 to +26.5 points. Infeasible-request detection relies almost entirely on our STN and the none grounding option, while Table 3 rules address simulator-specific conventions on feasible episodes (such as Matter's hundredths unit scaling and working around SimuHome's broken ChangeChannel tool).

2. Generalization to unseen devices: Leave-one-device-type-out across all 16 device types (Table R5). Extending the paper's held-out dehumidifier experiment, we held out every device type one at a time by removing all demonstration trajectories and calls involving that type prior to learning.

Table R5. Leave-one-device-type-out generalization on SimuHome across all 16 device types. For each device type, every demonstration request or call that reads or operates that type is removed before learning, as for the dehumidifier in the paper. Success (%) is reported on the test episodes whose goals involve that type, mean over three seeds (42–44). Because multi-device requests involve more than one device type, the 16 device-type subsets (914 device-episode occurrences in total) span the 500 unique device-involving episodes in SimuHome. The rules of Table 3 are unchanged.

Held-out device typeEpisodesGemma 4 12BQwen 3.5 9B
dehumidifier (Table 1a)5898.396.6
on_off_light17698.596.6
laundry_dryer12889.183.9
laundry_washer10682.480.2
dishwasher8387.183.1
window_covering_controller5498.193.2
air_conditioner5389.389.9
air_purifier49100.0100.0
dimmable_light4397.791.5
refrigerator3391.991.9
fan2998.996.6
heat_pump2684.684.6
tv2679.575.6
freezer1998.282.5
rvc (robot vacuum)1782.482.4
humidifier14100.092.9
Mean over device types50092.288.8

Interpretation of Table R5: Across all 16 device types (500 unique device-involving episodes), holding out each device type from skill learning yields a mean success rate of 92.2% (Gemma 4 12B) and 88.8% (Qwen 3.5 9B)—within 0.6 percentage points of our full-demonstration agent (92.8% and 89.4%), and reaching 98.1%–100.0% on air_purifier, humidifier, fan, on_off_light, dehumidifier, freezer, and window_covering_controller. Even on multi-step cycle appliances (laundry_washer, dishwasher, laundry_dryer), the agent maintains 82.4%–89.1% without ever seeing that appliance type in demonstrations, because the learned state precondition (OperationalState.Start requires OnOff = true) is abstracted at the Matter cluster level rather than tied to a specific device name, and candidate actions are enumerated directly from the target device's live interface.

3. Generalization to modified tool interfaces and operational requirements (Table R6). With our agent's code and rules completely frozen, we evaluated two modified environments: (i) Modified interface, where tool names, argument keys, and attribute names are renamed and response formats altered; and (ii) Modified requirements, where the environment's public control rules and device operational requirements (e.g., valid setpoint ranges and power prerequisites) are mutated and ground-truth labels recomputed.

AuthorsOpenReview ↗

Response to Reviewer 3bq7 (3/3)

Table R6. Robustness of SimuHome success (%) to changes in the tool interface and in operational requirements, mean over three seeds. Original is the test set of Table 1a (Overall). Modified interface uses the same episodes with tool names, argument keys and attribute names renamed and response formats changed, with the same semantics. Modified requirements uses the same episodes with the environment's public control rules and device requirements changed (for example, setpoint ranges and power prerequisites), with labels recomputed. Both sets were created after the agent's code was frozen, and no rule was changed. Δ is the change from Original.

MethodOriginalModified interfaceΔModified requirementsΔ
Gemma 4 12B
Ours92.893.3+0.582.9−9.9

Interpretation of Table R6: Renaming the tool interface causes zero degradation (93.3%, +0.5%), because our agent dynamically populates its multiple-choice options from the interface specification rather than memorizing tool strings. Even when the physical operational requirements and setpoint bounds of the environment are mutated without updating our frozen rules or re-running skill learning (Modified requirements), our agent achieves 82.9%—still outperforming the strongest baseline on the unmodified environment (ExpeL, 62.3%) by +20.6 percentage points.


W3. Contribution of skill learning and typed decisions in the ablations

We scaled the single-seed ablations of Table 2a to five seeds (42–46; 3,000 episodes per variant) in Table R4.

Table R4. Ablations of Table 2a over five seeds (42–46), SimuHome, Gemma 4 12B, 600 episodes per seed; mean ± standard deviation. Columns as in Table 2a.

VariantFIFQT4 IFOverallPrecisionTime (s)
Ours92.8 ± 0.792.9 ± 2.293.1 ± 2.892.8 ± 0.890.5 ± 0.911.5 ± 1.3
LLM decisions (backbone answers the questions)92.7 ± 0.692.5 ± 1.693.7 ± 1.692.6 ± 0.690.4 ± 0.816.1 ± 2.1
No learned contracts89.1 ± 0.792.9 ± 1.893.1 ± 2.091.0 ± 0.887.9 ± 1.011.5 ± 1.4
No none option in grounding92.9 ± 0.666.5 ± 2.393.7 ± 1.679.7 ± 0.977.4 ± 0.811.8 ± 1.6
No temporal consistency check92.9 ± 0.649.3 ± 1.45.9 ± 0.771.1 ± 0.563.6 ± 0.611.6 ± 1.4
No read-back or repair91.5 ± 0.892.8 ± 1.892.9 ± 2.192.2 ± 0.790.4 ± 1.011.6 ± 1.6

Interpretation of Table R4:

  1. Skill learning (Learned contracts) eliminates 34% of remaining errors on Feasible SimuHome episodes and drives multi-task planning on TimeArena: Because 50% of SimuHome episodes are infeasible (IF) and are refused before any action is executed, learned preconditions and fallback repairs only act on feasible (F) episodes. Across 5 seeds, learned contracts boost F success from 89.1±0.7%89.1 \pm 0.7\% to 92.8±0.7%92.8 \pm 0.7\% (+3.7 points, p<0.001p < 0.001) and precision from 87.9±1.0%87.9 \pm 1.0\% to 90.5±0.9%90.5 \pm 0.9\% (+2.6 points), cutting the feasible failure rate by more than a third (from 10.9% to 7.2%). Furthermore, on TimeArena—where multi-step action dependencies dominate—the learned programs and contracts improve completion from 75.6% to 90.0% (+14.4 points) on two-task episodes and from 54.4% to 66.7% (+12.3 points) on three-task episodes (Table 2b).
  2. Typed decisions (Formulation + Specialized Decision Model): Our typed-decision mechanism contributes in two distinct ways:
    • First, formulating grounding and action selection as closed-set typed questions (used by both Ours and LLM decisions) accounts for the +5.3 to +18.7 point accuracy advantage over rule-and-STN-equipped CodeAct, LLM+P, and LLMCompiler (Table R1).
    • Second, answering those typed questions with the specialized decision model (Jev) instead of the 12B backbone (LLM decisions) cuts end-to-end episode latency by 29% (16.1±2.1 s→11.5±1.3 s16.1 \pm 2.1\text{ s} \to 11.5 \pm 1.3\text{ s} across 5 seeds, including the API round-trip), speeds up per-request decision inference by 24.5$\times$ (5.40 s→0.22 s5.40\text{ s} \to 0.22\text{ s}, Appendix F.3), reduces backbone calls from 2.68 to 1.06 per episode, and yields a calibrated probability distribution over actions to rank fallback repairs.

We hope these results fully resolve your concerns and support a positive reassessment of our work, and we warmly welcome any further questions during the discussion period.

Reviewer_CPmoOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper proposes a smart-home agent that uses one LLM call to extract atomic goals, a typed decision model to select devices and actions, and code to check timing constraints and execute plans. Learned skills supply prerequisites and fallbacks. It achieves 92.8% / 89.4% on SimuHome with two small LLMs while reducing LLM calls. A separate instantiation is evaluated on TimeArena.

Critical strengths
  1. Efficient design
  • Separating generation, selection, and computation reduces backbone calls from about seven to one per episode. The typed decision model also provides a measured latency benefit.
  1. Transparent reporting
  • The development protocol, hand-written rules, and limitations are documented clearly, making the system's assumptions assessable.
Critical weaknesses
  1. The comparisons do not isolate the source of the gains
  • Only the proposed agent receives extensive hand-written rules, which are never ablated. Baselines using structured output or code generation (e.g., CodeAct) with the same rules and checks are missing.
  • Replacing the decision model costs 0.5 points of success rate and removing learned contracts costs 1.8 points in single-seed ablations. Despite the latency benefits, the value of the decomposition over simpler alternatives remains unclear.
  1. Evidence for generality is limited
  • Development and test episodes in SimuHome are disjoint but come from the same generator and may share task and wording patterns. Robustness to different request wording or task structures is not evaluated.
  • TimeArena’s test of individual unseen tasks covers only three tasks and nine runs, all also completed by ReAct and Reflexion (Table 1b). Removing typed decisions also leaves completion rates unchanged (Table 2b).
Questions for authors
  1. Can you quantify how much the hand-written rules in Table 3 contribute to performance?
  2. Do you have comparisons with direct structured output or code generation using the same hand-written rules, temporal checks, and execution budget?
8 comments
AuthorsOpenReview ↗

Response to Reviewer CPmo (1/5)

We thank the reviewer for the clear, concrete questions and for recognizing the efficiency and transparency of our work. We address every question with new experiments below.

W1-1, Q1, & Q2. Quantifying hand-written rules (Table 3) and comparing against structured-output / code-generation baselines with identical rules, checks, and budgets

We ran both exact experiments requested in Q1 and Q2.

Answer to Q1 (Ablating all hand-written rules in Table 3): In Table R2, we remove all rules of Table 3 simultaneously from our agent across 5 seeds (3,000 episodes).

Table R2. Contribution of the description rules (Table 3) on SimuHome, Gemma 4 12B, mean over five seeds (42–46; 600 episodes each). Columns report success on feasible (F), infeasible (IF) and infeasible scheduling (QT4 IF) episodes and overall (%), precision (%), and agent time per episode (s). "No description rules" removes every component of Table 3 at once and leaves all other components unchanged.

VariantFIFQT4 IFOverallPrecisionTime (s)
Ours92.892.993.192.890.511.5
No description rules (all of Table 3)60.382.488.971.352.711.6

Interpretation of Table R2: Even when stripped of every rule in Table 3, our agent achieves 71.3% overall, 82.4% on infeasible episodes (IF), and 88.9% on infeasible scheduling (QT4 IF)—outperforming every baseline in Table 1a (ReAct 44.8%, Workflow Memory 45.7%, Reflexion 52.7%, and ExpeL 62.3%) by +9.0 to +26.5 percentage points. On infeasible requests (IF and QT4 IF), our performance remains above 82–88% without any Table 3 rules because conflict detection and nonexistent-device refusal are handled by our STN consistency check and typed grounding with the none option. Table 3 rules primarily bridge simulator-specific surface conventions on feasible episodes (F: 60.3% →\to 92.8%), such as Matter unit scaling (hundredths), SimuHome's broken ChangeChannel endpoint, and number-word normalization.

AuthorsOpenReview ↗

Response to Reviewer CPmo (2/5)

Answer to Q2 (Matched comparison with LLM+P, LLMCompiler, and CodeAct given the same rules, STN checks, and budget): In Table R1, we equip LLM+P, LLMCompiler, and CodeAct with every rule of Table 3 (including Appendices B.3 and B.6), the exact same STN and execution engine, and the same budget (400 tool calls and 900 s per episode).

Table R1. SimuHome success (%) of structured-output, hybrid-planner, and code-generation baselines given the same interface knowledge (Table 3, including Appendices B.3 and B.6) and the same execution machinery (Section 3.4 and Appendix B.5; Section 3.5 without learned components). All methods use the same backbone and the same budget of 400 tool calls and 900 s per episode; means over three seeds (42–44). Superscripts follow Table 1a: S marks evaluation by the simulator's state and J by the panel of LLM judges.

MethodQT1 F^JQT1 IF^JQT2 F^SQT2 IF^JQT3 F^SQT3 IF^JQT4-1 F^SQT4-1 IF^JQT4-2 F^SQT4-2 IF^JQT4-3 F^SQT4-3 IF^JHeld-outOverall
Gemma 4 12B
LLM+P + rules & exec.88.077.392.078.090.768.783.398.776.798.776.782.090.884.2
LLMCompiler + rules & exec.78.774.791.375.396.780.782.094.766.734.064.748.074.773.9
CodeAct + rules & exec.90.088.090.781.394.790.779.396.792.094.768.081.393.787.3
Ours92.796.790.778.098.0100.097.3100.092.095.389.383.398.392.8
Qwen 3.5 9B
LLM+P + rules & exec.73.357.391.378.086.750.060.0100.082.792.050.071.383.974.4
LLMCompiler + rules & exec.84.074.775.363.381.390.045.392.761.349.340.746.064.467.0
CodeAct + rules & exec.86.094.788.762.795.392.068.789.388.775.352.769.383.380.3
Ours76.096.791.379.395.3100.090.7100.090.795.385.372.796.689.4

Interpretation of Table R1: Even with identical rules, temporal checks, and execution budgets, LLM+P, LLMCompiler, and CodeAct trail our agent by 5.5 to 18.9 points on Gemma and 9.1 to 22.4 points on Qwen. On multi-device and future/concurrent scheduling (QT4-1 F and QT4-3 F), Ours beats the strongest rule-equipped baseline (CodeAct) by +18.0 and +21.3 points on Gemma and +22.0 and +32.6 points on Qwen. When generating code or formal plans in one or more open-ended steps, 9B–12B models frequently hallucinate device names, omit sub-goals, or mis-specify temporal arguments. Decomposing the task into a literal checklist followed by closed-set option selection over the live interface eliminates these failure modes.

AuthorsOpenReview ↗

Response to Reviewer CPmo (3/5)

W1-2. Multi-seed ablations and the value of the decomposition

We expanded the single-seed ablations of Table 2a to five seeds (42–46; 3,000 episodes per variant) in Table R4.

Table R4. Ablations of Table 2a over five seeds (42–46), SimuHome, Gemma 4 12B, 600 episodes per seed; mean ± standard deviation. Columns as in Table 2a.

VariantFIFQT4 IFOverallPrecisionTime (s)
Ours92.8 ± 0.792.9 ± 2.293.1 ± 2.892.8 ± 0.890.5 ± 0.911.5 ± 1.3
LLM decisions (backbone answers the questions)92.7 ± 0.692.5 ± 1.693.7 ± 1.692.6 ± 0.690.4 ± 0.816.1 ± 2.1
No learned contracts89.1 ± 0.792.9 ± 1.893.1 ± 2.091.0 ± 0.887.9 ± 1.011.5 ± 1.4
No none option in grounding92.9 ± 0.666.5 ± 2.393.7 ± 1.679.7 ± 0.977.4 ± 0.811.8 ± 1.6
No temporal consistency check92.9 ± 0.649.3 ± 1.45.9 ± 0.771.1 ± 0.563.6 ± 0.611.6 ± 1.4
No read-back or repair91.5 ± 0.892.8 ± 1.892.9 ± 2.192.2 ± 0.790.4 ± 1.011.6 ± 1.6

Interpretation of Table R4:

  1. Value of the decomposition (Choosing, Not Generating): Why does LLM decisions achieve 92.6±0.6%92.6 \pm 0.6\% while CodeAct + rules & exec. achieves 87.3%87.3\% and LLM+P + rules & exec. achieves 84.2%84.2\% (Table R1)? Because LLM decisions still retains our exact decomposition: 1-call atomic checklist extraction followed by closed-set multiple-choice questions over the interface's enumerated options (Appendix B.4). Thus, the +5.3 to +18.7 point gain over rule-equipped CodeAct, LLM+P, and LLMCompiler comes directly from decomposing the agent's decisions into explicit option selection rather than open-ended plan or code generation.
  2. Value of the specialized decision model (Jev): Once factored into typed questions, routing them to jev-1.13.0 instead of the 12B backbone cuts end-to-end episode latency by 29% (16.1 s→11.5 s16.1\text{ s} \to 11.5\text{ s} across 5 seeds, including the remote API round-trip), speeds up per-question-set inference by 24.5$\times$ (5.40 s→0.22 s5.40\text{ s} \to 0.22\text{ s}, Appendix F.3), reduces backbone calls from 2.68 to 1.06 per episode, and outputs a probability distribution over actions to rank fallback repairs (LLM decisions only outputs a single schema-constrained choice).
  3. Statistically significant impact of learned contracts on Feasible episodes: Because 50% of SimuHome episodes are infeasible (IF) and are refused prior to execution, preconditions and repairs only act on feasible (F) episodes. Across 5 seeds, learned contracts improve F success by +3.7 points (89.1±0.7%→92.8±0.7%89.1 \pm 0.7\% \to 92.8 \pm 0.7\%, p<0.001p < 0.001) and precision by +2.6 points (87.9±1.0%→90.5±0.9%87.9 \pm 1.0\% \to 90.5 \pm 0.9\%)—eliminating 34% of all remaining feasible errors—while driving a +14.4% and +12.3% completion gain on TimeArena (Table 2b).
Reviewer_qVB1OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents a hybrid architecture for smart-home and temporal-task agents. An LLM converts requests into atomic goals, a typed decision model selects devices and actions from explicit options, and deterministic code computes execution times, checks temporal consistency, and manages execution. Skills learned from demonstrations supply prerequisites and repairs on SimuHome, while learned task programs support concurrent scheduling on TimeArena. The system substantially improves SimuHome success over the evaluated baselines while reducing backbone inference, and it performs strongly on TimeArena. The results demonstrate the practical value of combining language understanding with structured decisions and explicit temporal reasoning.

Critical strengths
  1. The proposed system achieves substantial improvements over the evaluated baselines on SimuHome, reaching 92.8% and 89.4% success with Gemma and Qwen, respectively, compared with 62.3% and 57.8% for ExpeL. The improvements are particularly pronounced on temporal scheduling tasks. Meanwhile, the system substantially reduces backbone inference, requiring only 1.06 LLM calls and 3.35K backbone tokens per episode, compared with 6.85 calls and 44.69K tokens for ReAct. These results demonstrate the practical effectiveness and efficiency of the overall architecture, although the inference comparison does not fully account for the additional decision model.
Critical weaknesses
  1. The main comparison does not isolate the contribution from the hand-written domain logic. Only the proposed agent receives the rules in Table 3, including action filtering, unit conversions, grounding conventions, execution ordering, repair logic, a room-condition effect model, and checklist normalization. These rules are not ablated. Consequently, the large performance gap establishes the effectiveness of the complete system, but leaves unclear how much improvement comes from the proposed learning and selection mechanisms. A stronger comparison would give a conventional structured-output LLM or hybrid planner the same interface knowledge and execution machinery. This matters for positioning the contribution because prior systems such as LLM+P and LLMCompiler already combine language models with external planning or structured execution. The paper discusses these approaches but does not evaluate a matched comparison.
  2. The evidence for the learned-contract and specialized decision-model contributions is limited on SimuHome. Removing learned contracts reduces success by only 1.8 percentage points, while replacing the specialized decision model with the backbone answering the same typed questions reduces it by 0.5 points. These are single-seed ablations. More importantly, Appendix B.2 states that the deployed SimuHome agent does not use the learned composite programs, capability preconditions, effects, or learned unit scales during inference. Its actual use of the library consists of descriptions, one learned state precondition, heuristic power prerequisites, and a demonstrated repair alternative. The learning contribution is more substantial on TimeArena, and the specialized model improves latency, but neither observation establishes that these components explain the large SimuHome accuracy gains.
Questions for authors

Please address my concerns in weaknesses section.

6 comments
AuthorsOpenReview ↗

Response to Reviewer qVB1 (1/3)

e thank the reviewer for the thorough reading of the paper and Appendix B.2, and for recognizing the practical effectiveness and token/call efficiency of our architecture.

W1. Isolating the hand-written domain logic vs. our core decision architecture

To strictly isolate our architectural contribution from the domain interface knowledge in Table 3, we performed two complementary experiments: (1) giving strong structured-output, hybrid-planning, and code-generation baselines (LLM+P, LLMCompiler, CodeAct) the exact same rules and execution machinery as our agent, and (2) removing all of Table 3 from our agent.

1. Matched comparison with identical rules and execution machinery (Table R1). We provided LLM+P, LLMCompiler, and CodeAct with every rule in Table 3 (including the checklist normalization of Appendix B.3 and the room-condition effect model of Appendix B.6), our Simple Temporal Network (STN, Section 3.4 and Appendix B.5), and our execution engine (Section 3.5), under the same backbone and budget (400 tool calls and 900 s per episode).

Table R1. SimuHome success (%) of structured-output, hybrid-planner, and code-generation baselines given the same interface knowledge (Table 3, including Appendices B.3 and B.6) and the same execution machinery (Section 3.4 and Appendix B.5; Section 3.5 without learned components). All methods use the same backbone and the same budget of 400 tool calls and 900 s per episode; means over three seeds (42–44). Superscripts follow Table 1a: S marks evaluation by the simulator's state and J by the panel of LLM judges.

MethodQT1 F^JQT1 IF^JQT2 F^SQT2 IF^JQT3 F^SQT3 IF^JQT4-1 F^SQT4-1 IF^JQT4-2 F^SQT4-2 IF^JQT4-3 F^SQT4-3 IF^JHeld-outOverall
Gemma 4 12B
LLM+P + rules & exec.88.077.392.078.090.768.783.398.776.798.776.782.090.884.2
LLMCompiler + rules & exec.78.774.791.375.396.780.782.094.766.734.064.748.074.773.9
CodeAct + rules & exec.90.088.090.781.394.790.779.396.792.094.768.081.393.787.3
Ours92.796.790.778.098.0100.097.3100.092.095.389.383.398.392.8
Qwen 3.5 9B
LLM+P + rules & exec.73.357.391.378.086.750.060.0100.082.792.050.071.383.974.4
LLMCompiler + rules & exec.84.074.775.363.381.390.045.392.761.349.340.746.064.467.0
CodeAct + rules & exec.86.094.788.762.795.392.068.789.388.775.352.769.383.380.3
Ours76.096.791.379.395.3100.090.7100.090.795.385.372.796.689.4

Interpretation of Table R1: Even when given all of Table 3 and our exact STN and execution machinery, the baselines fall short of our agent by 5.5 to 18.9 percentage points on Gemma 4 12B (92.8% vs. 87.3% for CodeAct, 84.2% for LLM+P, and 73.9% for LLMCompiler) and by 9.1 to 22.4 percentage points on Qwen 3.5 9B (89.4% vs. 80.3%, 74.4%, and 67.0%). The gap is most pronounced on complex scheduling (QT4-1 F and QT4-3 F), where Ours outperforms the strongest rule-equipped baseline (CodeAct) by +18.0 and +21.3 points on Gemma and +22.0 and +32.6 points on Qwen, as well as on infeasible device and command grounding (reaching 96.7% on QT1 IF and 100.0% on QT3 IF and QT4-1 IF). Why do hybrid planners still fail when given the same rules and executor? Because LLM+P, LLMCompiler, and CodeAct still require the backbone LLM to generate formal problem specifications, task DAGs, or executable code that simultaneously resolves entity grounding, action/argument mapping, and temporal links. On 9B–12B backbones, open-ended generation frequently hallucinates device identifiers, drops sub-goals in multi-device requests, or mis-wires temporal dependencies. By contrast, our method restricts generation to extracting literal mentions into an atomic checklist, and poses grounding and action selection as factored, closed-set choices over explicit interface options.

AuthorsOpenReview ↗

Response to Reviewer qVB1 (2/3)

2. Direct ablation of all Table 3 rules (Table R2). Conversely, we removed every rule in Table 3 simultaneously from our agent across 5 seeds.

Table R2. Contribution of the description rules (Table 3) on SimuHome, Gemma 4 12B, mean over five seeds (42–46; 600 episodes each). Columns report success on feasible (F), infeasible (IF) and infeasible scheduling (QT4 IF) episodes and overall (%), precision (%), and agent time per episode (s). "No description rules" removes every component of Table 3 at once and leaves all other components unchanged.

VariantFIFQT4 IFOverallPrecisionTime (s)
Ours92.892.993.192.890.511.5
No description rules (all of Table 3)60.382.488.971.352.711.6

Interpretation of Table R2: Even with every single hand-written rule in Table 3 removed, our core architecture achieves 71.3% overall success, which still outperforms every baseline in Table 1a (ReAct 44.8%, Workflow Memory 45.7%, Reflexion 52.7%, and demonstration-equipped ExpeL 62.3%) by +9.0 to +26.5 percentage points. Notably, on infeasible episodes (IF: 82.4%) and infeasible scheduling (QT4 IF: 88.9%), removing all of Table 3 causes only a modest change (from 92.9% and 93.1%), proving that conflict detection and invalid-request refusal come overwhelmingly from our STN consistency check and typed grounding with the none option, rather than from hand-written rules. The rules in Table 3 primarily bridge simulator-specific surface conventions on feasible episodes (F: 60.3% →\to 92.8%), such as Matter unit scaling (hundredths of a degree) and bypassing SimuHome's broken ChangeChannel command. Together, Tables R1 and R2 demonstrate that our core algorithmic pipeline alone surpasses all prior baselines, and that factoring decisions into typed option selection yields substantial gains that existing hybrid planners cannot replicate even when given the exact same rules and execution engine. We will add Tables R1 and R2 to the paper.

AuthorsOpenReview ↗

Response to Reviewer qVB1 (3/3)

W2. Role and statistical significance of the learned contracts and specialized decision model

To ensure high statistical power, we reran all ablations of Table 2a across five seeds (42–46; 3,000 episodes per variant) in Table R4.

Table R4. Ablations of Table 2a over five seeds (42–46), SimuHome, Gemma 4 12B, 600 episodes per seed; mean ± standard deviation. Columns as in Table 2a.

VariantFIFQT4 IFOverallPrecisionTime (s)
Ours92.8 ± 0.792.9 ± 2.293.1 ± 2.892.8 ± 0.890.5 ± 0.911.5 ± 1.3
LLM decisions (backbone answers the questions)92.7 ± 0.692.5 ± 1.693.7 ± 1.692.6 ± 0.690.4 ± 0.816.1 ± 2.1
No learned contracts89.1 ± 0.792.9 ± 1.893.1 ± 2.091.0 ± 0.887.9 ± 1.011.5 ± 1.4
No none option in grounding92.9 ± 0.666.5 ± 2.393.7 ± 1.679.7 ± 0.977.4 ± 0.811.8 ± 1.6
No temporal consistency check92.9 ± 0.649.3 ± 1.45.9 ± 0.771.1 ± 0.563.6 ± 0.611.6 ± 1.4
No read-back or repair91.5 ± 0.892.8 ± 1.892.9 ± 2.192.2 ± 0.790.4 ± 1.011.6 ± 1.6

Interpretation of Table R4 and the complementary roles of each component:

  1. Why learned contracts yield a statistically significant gain on Feasible episodes (+3.7 points, p<0.001p < 0.001) and why the learner is unified across benchmarks: Because exactly 50% of SimuHome episodes are infeasible (IF) and are refused before any tool execution takes place, execution-time contracts (preconditions and fallback repairs) are never invoked on IF episodes (92.9%92.9\% with or without contracts). On the feasible (F) episodes where actions are actually executed, learned contracts improve success from 89.1±0.7%89.1 \pm 0.7\% to 92.8±0.7%92.8 \pm 0.7\% (+3.7 points) and boost precision by +2.6 points (87.9±1.0%→90.5±0.9%87.9 \pm 1.0\% \to 90.5 \pm 0.9\%)—eliminating 34% of all remaining feasible-episode errors (reducing the feasible failure rate from 10.9% to 7.2%). Furthermore, regarding Appendix B.2: our learner is designed as a domain-agnostic algorithm that extracts a complete contract specification (composites, capability/state preconditions, effects, and avoid sets) without manual feature selection. At runtime, each environment naturally consumes the contract primitives that its tool interface does not already expose: SimuHome's Matter tools already expose device capabilities dynamically (so the agent only needs the learned skill descriptions, state/composite power prerequisites, and avoid-set repairs), whereas TimeArena does not expose task programs or object dependencies, making the learned programs and dependency contracts responsible for a +14.4 point (two-task: 75.6% →\to 90.0%) and +12.3 point (three-task: 54.4% →\to 66.7%) jump in completion rate (Table 2b).
  2. The two-part role of typed decisions (Formulation vs. Fast System-1 Inference): Our "Choosing, Not Generating" paradigm separates how decisions are formulated (closed-set multiple choice over explicit interface options) from which model answers them (the specialized decision model Jev vs. the backbone LLM).
    • Formulation (Accuracy): As shown in Table R4 (LLM decisions: 92.6±0.6%92.6 \pm 0.6\%) and Table R1 (CodeAct: 87.3%87.3\%, LLM+P: 84.2%84.2\%), formulating grounding and action selection as factored typed questions is what unlocks >92%>92\% accuracy even when the 12B backbone answers the questions.
    • Specialized Decision Model (Speed, Efficiency, and Ranked Fallbacks): Routing those typed questions to jev-1.13.0 instead of the backbone cuts total end-to-end episode wall-clock time by 29% across 5 seeds (16.1±2.1 s→11.5±1.3 s16.1 \pm 2.1\text{ s} \to 11.5 \pm 1.3\text{ s} in Table R4; 21.5 s→12.0 s21.5\text{ s} \to 12.0\text{ s} in single-seed Table 2a), where agent time explicitly includes the external decision-model API round-trip. Per decision request, inference drops by 24.5$\times$ (from 5.40 s5.40\text{ s} with the backbone to 0.22 s0.22\text{ s} with Jev, Appendix F.3), backbone calls drop from 2.68 to 1.06 per episode, and Jev provides a calibrated probability distribution over actions to order fallback repairs (Section 3.3). As noted in Section 4.2 (Q3), Jev adds only 1.63 requests per episode (0.65 reaching the API, 1.9K input tokens each, costing $0.094 across all 1,800 episodes). We will include Table R4 and make this distinction between decision formulation and fast decision-model inference explicit in the Introduction and Section 4.2.

We hope these results fully resolve your concerns and support a positive reassessment of our work, and we warmly welcome any further questions during the discussion period.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their constructive feedback. To address every question with direct empirical evidence, we conducted six sets of new experiments (spanning over 30,000 additional evaluation episodes with frozen agent logic):

  1. Matched-component comparisons (Table R1): We equipped structured-output, hybrid-planner, and code-generation baselines (LLM+P, LLMCompiler, CodeAct) with the exact same interface rules (Table 3, Appendices B.3 and B.6), temporal network (Section 3.4, Appendix B.5), execution machinery (Section 3.5), and budget (400 calls, 900 s). Even with identical rules and execution machinery, our agent outperforms the strongest baseline (CodeAct) by +5.5 points on Gemma 4 12B (92.8% vs. 87.3%) and +9.1 points on Qwen 3.5 9B (89.4% vs. 80.3%), and by +18.0 to +32.6 points on complex scheduling (QT4-1 F, QT4-3 F).
  2. Full ablation of hand-written interface rules (Table R2): Removing all rules of Table 3 simultaneously across 5 seeds yields 71.3% overall success (and 82.4% on IF, 88.9% on QT4 IF), which still outperforms every baseline in Table 1a (ReAct 44.8%, Workflow Memory 45.7%, Reflexion 52.7%, ExpeL 62.3%) by +9.0 to +26.5 points.
  3. 5-seed statistical validation of all component ablations (Table R4): Scaling Table 2a from 1 seed to 5 seeds (3,000 episodes per variant) confirms low variance (±0.5%\pm 0.5\%–0.8%0.8\% overall), a +3.7 point (p<0.001p < 0.001) improvement on feasible episodes from learned contracts (eliminating 34% of remaining feasible errors), and a 29% end-to-end episode latency reduction (16.1 s→11.5 s16.1\text{ s} \to 11.5\text{ s}; 24.5×24.5\times per decision request) from the specialized decision model.
  4. Leave-one-device-type-out generalization across all 16 device types (Table R5): Holding out every device type one at a time from skill learning across all 500 device-involving episodes yields a mean success rate of 92.2% (Gemma) and 88.8% (Qwen)—within 0.6 points of our full-demonstration agent.
  5. Zero-shot robustness to paraphrased wording and unseen task structures (Table R3): On paraphrased requests (Reworded) and out-of-distribution task compositions never produced by the benchmark generator (New structures), our frozen agent outperforms all baselines by +29.5 to +46.2 points (Reworded) and +11.0 to +25.1 points (New structures).
  6. Zero-shot robustness to modified tool interfaces and operational requirements (Table R6): Renaming tool/attribute identifiers and response formats yields 93.3% (+0.5%), and mutating the environment's operational requirements and setpoint bounds yields 82.9% (+20.6 points above ExpeL on the unmodified environment).

Detailed tables and point-by-point responses are provided in the individual replies below.

AuthorsOpenReview ↗

Summary for the Area Chair

Dear AC,

Thank you for handling our paper. Below we summarize the strengths the reviewers mentioned, their concerns with our responses, and the responses still awaiting a reply. Tables R1–R7 are in our OpenReview responses.

1. Strengths mentioned by the reviewers

  • Motivation. Not every agent decision requires LLM generation; separating decision types improves reliability and efficiency (3bq7).
  • Results. Large gains, especially on temporal scheduling: 92.8% / 89.4% on SimuHome (Gemma 4 12B / Qwen 3.5 9B) vs. 62.3% / 57.8% for ExpeL (qVB1, 3bq7).
  • Efficiency. About one backbone call per episode instead of seven, and a measured latency benefit from the typed decision model (qVB1, CPmo).
  • Evaluation and presentation. Comprehensive evaluation with useful ablations (3bq7); clear writing (qVB1, 3bq7) and transparent reporting of the protocol, rules and limitations (CPmo).

2. Concerns and our responses

Reviewer qVB1

  1. The gains are not isolated from the hand-written rules; a matched comparison with LLM+P or LLMCompiler is missing. Given the same rules, STN, execution engine and budget, LLM+P, LLMCompiler and CodeAct still trail our agent by 5.5–18.9 points (Gemma) and 9.1–22.4 points (Qwen) (Table R1). Removing all rules of Table 3 still yields 71.3%, above every baseline of Table 1a (Table R2). Reviewer CPmo acknowledged the same point (item 3).
  2. Weak evidence for the learned contracts and the decision model (single-seed ablations, −1.8 and −0.5 points). Over five seeds, learned contracts raise feasible-episode success by 3.7 points (p < 0.001), removing 34% of the remaining feasible errors, and add 14.4 / 12.3 points of completion on TimeArena (Table R4, Table 2b). The closed-set formulation gives the accuracy (92.6% even when the backbone answers the typed questions, vs. 87.3% for CodeAct); the decision model cuts episode latency by 29%.

Reviewer CPmo

  1. The gains are not isolated: no rule ablation, no baselines with the same rules (Q1, Q2), single-seed ablations. The same concern as items 1, 2 and 6, answered with Tables R1, R2 and R4. Reviewer CPmo acknowledged this point: the experiments "resolve my questions on the rules and baselines," and the reviewer raised the rating; only item 4 remains.
  2. Limited evidence for generality (request wording, task structures, unseen TimeArena tasks). On two sets built after the agent was frozen, our agent leads all baselines by 29.5–46.2 points on paraphrased requests and by 11.0–25.1 points on new task structures (Table R3). With all 30 TimeArena tasks unseen, it reaches 75.6% / 54.4% completion on two-/three-task episodes, above all baselines. On the follow-up (error 7.2% → 33.9% on new structures), we gave the Table R1 baselines our request parser: they all drop as well (Gemma 55.9–61.2%, Qwen 56.1–64.4%), and our agent remains the best, ahead by 5.5–10.8 points (Gemma) and 2.2–10.5 points (Qwen) (Table R7).

Reviewer 3bq7

  1. Limited methodological novelty. With identical rules and execution machinery, generation-based planners still trail by 5.5–22.4 points (Table R1). This suggests that the gain comes from posing grounding and action selection as closed-set choices, not from the shared engineering.
  2. Reliance on hand-crafted rules; unclear generalization to new devices, tool interfaces and operational requirements. 71.3% without any rule of Table 3 (Table R2); 92.2% / 88.8% mean success when each of the 16 device types is held out (Table R5); 93.3% with a renamed tool interface and 82.9% with modified operational requirements, without any change to code or rules (Table R6). On the rules, Reviewer CPmo acknowledged the same point (item 3).
  3. Small contribution of skill learning and typed decisions. Over five seeds, learned contracts add 3.7 points on feasible episodes (p < 0.001), and the decision model cuts episode latency by 29% and backbone calls from 2.68 to 1.06 per episode (Table R4).

3. Responses awaiting a reply

  • Reviewer qVB1: kept the score without naming a remaining concern. We asked which concerns remain on Nov 18 and Nov 19, but have not yet received an answer.
  • Reviewer 3bq7: has not yet responded to our rebuttal (Nov 15), summary (Nov 18) or reminder (Nov 19) on items 5–7.
  • Reviewer CPmo: has not yet responded to our answer on generalization (item 4, Table R7, Nov 19).

We hope these responses can be considered during the AC–reviewer discussion period.

Thank you again for your time in handling our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.