acceptodds
Under review as a conference paper at ICLR 2027

TWILL: A Proactive Agent That Decides from Internal Proposals and Human Requests

Abstract

A household robot should recognize from its own observations what a situation requires. Instruction-driven manipulation systems and proactive assistants that anticipate human needs share a blind spot: the robot seldom forms its own judgment of the scene, let alone carries it through to execution. A robot with its own judgment must also arbitrate between that judgment and what the person asks. We build TWILL, a proactive robotic agent that closes the loop from perception to action. Its proposer forms a proposal from the scene alone. Its decision module, SHUTTLE, arbitrates between the proposal and the human request inside a pretrained vision-language model. SHUTTLE reads the scene and each intent through separate attention paths, so that the decision distinguishes the robot's intent from the person's. An off-the-shelf executor then carries out the decision. To evaluate the full loop, we introduce TWINE, which scores understanding and decisions on 360 cases in 180 controlled pairs, each differing in one consequential factor, and scores closed-loop success in simulated kitchens. The proposer reaches 89.88% accuracy in deciding whether and what to propose, 48.61 points above the untuned backbone. SHUTTLE raises the rate of fully correct decisions by 3.70 points over the same fine-tuned backbone without separate reads. In closed loop, TWILL reaches an 81.48% success rate on request-free episodes, against 50.00% for an untuned vision-language model commanding the same executor. TWILL thereby allows a household robot to act on proposals it forms from the scene and to arbitrate between these proposals and human requests according to context.

Primary area:
applications to robotics, autonomy, planning
Keywords:
embodied agent, proactive assistance, intent arbitration, agentic harness, autonomous robot

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7MywOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

TWILL addresses proactive household-robot decision making when an autonomous, scene-driven proposal must be reconciled with a human request. A scene-only proposer generates potential tasks, SHUTTLE separately reads scene context and both intent sources, and a frozen GR00T N1.6 executor acts in a closed loop. TWINE evaluates proposals, 360 decisions in 180 controlled pairs, and simulated execution; reported complete-decision accuracy is 71.85% for SHUTTLE versus 68.15% for Direct, with 81.48% request-free closed-loop success.

Critical strengths
  1. Dual-intent arbitration addresses an important problem in proactive robot assistance. Separating robot-generated proposals from human requests captures conflicts missed by instruction-only systems. Examples involving conditional requests, unsafe instructions, and additional goals make the formulation practically meaningful.
  2. TWILL provides a coherent perception-to-action design. A scene-only proposer, SHUTTLE decision module, and frozen GR00T N1.6 executor create a clear division of labor. Closed-loop replanning allows decisions to respond to new observations or requests.
  3. TWINE gives a useful diagnostic benchmark for dual-intent decisions. Its 180 controlled pairs change one consequential factor, distinguishing relevant reasoning from superficial instruction matching. Separate understanding probes and physical-obligation checks support focused error analysis.
  4. Experimental controls are stronger than simple backbone comparisons. Matched backbones, LoRA settings, data, and parameter-matched fusion baselines help isolate SHUTTLE. Three seeds, bootstrap confidence intervals, and human checks of automatic scoring improve credibility.
  5. SHUTTLE shows measurable gains in decision quality. Complete-decision pass increases from 68.15% for Direct to 71.85%, with a reported 95% CI of [+1.76, +5.65] points. Correctness on both members of a controlled pair also improves from 47.96% to 53.33%.
  6. Proposer shows strong restraint in scenes needing no action. Restraint improves from 23.16% to 96.84%, while false proposals fall from 88 to 3 relative to the untuned model. Avoiding unnecessary interventions is especially important for proactive agents.
Critical weaknesses
  1. Novelty of source-specific attention remains somewhat incremental. SHUTTLE adapts established multi-source attention and residual fusion, while its 1.94-point advantage over merged-text reads has Holm-adjusted p = 0.0604. Evidence for a distinct benefit from separate attention normalization is therefore less conclusive than the Direct comparison suggests.
  2. Closed-loop gains attributable to SHUTTLE cover only a narrow set of situations. Request-free success is identical to Direct (81.48%) because SHUTTLE is bypassed, and matched-request success is lower (88.15% versus 96.30%). Improvement on absent-object requests (42.22% versus 21.48%) is promising but does not yet demonstrate broad dual-intent advantages.
  3. Proposer experiments do not isolate the pairwise ranking objective. Reported joint accuracy of 89.88% is compared with an untuned VLM, rather than an equally fine-tuned model without the ranking term. Consequently, benefits beyond task-specific supervised adaptation remain unclear.
  4. Task coverage and physical generalization are limited. Proposals involve six predefined on/off or closure problems, and closed-loop tests use only three fixtures in simulated RoboCasa365 kitchens. Such a narrow action space restricts evidence for open-ended proactive behavior and real-world robustness.
  5. External closed-loop baselines do not fully match task-specific training. TWILL uses a supervised scene-state proposer, whereas the VLM commander is based on an untuned Qwen3-VL model. A task-adapted commander under a comparable training budget would better isolate the contribution of system decomposition.
Questions for authors
  1. Can authors report SHUTTLE-versus-Direct closed-loop results for conditional, unsafe, and added-goal requests, beyond the absent-object setting?
  2. How much does the pairwise ranking term improve the proposer over matched supervised fine-tuning without that term?
  3. What explains SHUTTLE’s lower matched-request success (88.15% versus 96.30%) despite higher offline decision accuracy?
  4. Would a task-adapted VLM commander narrow the closed-loop gap, and do the findings extend to unseen fixture types or physical robots?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 7Myw (7-Seed Significance, Ranking Loss, Commander & Closed-Loop)

We thank Reviewer 7Myw for the supportive and constructive comments.

Response to W1: Source-Specific Attention & 7-Seed Significance vs. Merged-text

To rule out seed noise in the original 3-seed comparison against Merged-text (Holm p=0.0604p = 0.0604, Table 12), we retrained all four variants across 7 seeds with a fixed final step and the Qwen3.8-Flash judge (Table 3, App. C.4):

Metric (7 seeds, Qwen3.8-Flash; Tables 25a, 26)SHUTTLEAdapterMerged-textDirectSHUTTLE −- Merged-text [95% CI] (pp)
TWINE complete-decision pass (%, 360 cases)69.1366.7165.5264.29+3.61 [+2.18, +5.08] (<0.0002< 0.0002)
Full-set arbitration-direction error (%)16.7919.7218.8520.28−2.06-2.06 [−3.33-3.33, −0.79-0.79] (0.00140.0014)
69 unseen kitchen families pass (%, 156 cases)69.2368.3266.5864.01+2.66 [+0.99, +4.41] (0.00160.0016)

In this 7-seed evaluation:

  • Merged-text gains nothing over the parameter-matched 5.78M adapter, so a shared [ℓp;ℓh][\ell^p; \ell^h] softmax does not separate the two intents.
  • Independent per-source softmaxes make SHUTTLE significantly better than Merged-text (+3.61 pp, p<0.0002p < 0.0002; +8.48 pp on the 150 conflict and revision cases, Table 25b) and the adapter.
  • SHUTTLE identifies the instruction source more accurately than Merged-text (87.50% vs. 84.72%, p=0.014p = 0.014; Table 16).

Response to W3 & Q2: Isolating the Pairwise Ranking Term

To isolate the ranking term, we compared λ=0\lambda = 0 and λ=0.1\lambda = 0.1 across 3 seeds on the same paired images (App. C.6, Table 28):

Proposer retraining (198 obs., 3-seed mean)Restraint (%)FP per seedJoint Acc. (%)
λ=0\lambda = 0 (supervised fine-tuning only)87.7214.786.04
λ=0.1\lambda = 0.1 (with pairwise ranking, Eq. 2)91.46 (+3.74)9.7 (−34%-34\%)86.84 (+0.80)

Cross-entropy already supervises positive detection; contrasting on/off states at λ=0.1\lambda = 0.1 raises restraint on every seed and cuts false proposals by 34% while preserving joint accuracy.

Response to W2, Q1 & Q3: Closed-Loop Arbitration & Matched-Request Analysis

  • Matched-request gap (Q3; App. C.7, Table 30): To explain why SHUTTLE completes fewer matched requests than Direct (88.15% vs. 96.30%) despite higher offline accuracy, we decomposed the 8.15 pp gap: 3.70 pp from stochastic frozen-executor timeouts on identical faucet commands, and 4.44 pp from drawer episodes in two cropped or shadowed views where the proposer misreads the state. There SHUTTLE weighs the conflicting proposal (decision accuracy 95.56% vs. 100.00%), whereas Direct copies the request regardless; that compliance helps when the request is valid but loses the hazard under conflict.
  • Conditional, unsafe, and added-goal requests (W2, Q1): Because frozen GR00T provides only three reliable fixture skills, we quantify absent-object conflict over 66 closed-loop episodes, verify conditional deferral in Fig. 5, and evaluate unsafe actions and added-goal ordering through the controlled 7-seed TWINE pairs of Table 25b, where SHUTTLE is significantly better in every conflict type:
Request typeEvaluation settingSHUTTLEDirectGain / pp
Absent object + hazardClosed loop: decision / completion (%)82.22 / 42.2259.63 / 21.48completion +20.74 pp (p=0.0056p = 0.0056)
Unsafe / forbidden extra actionTWINE pass (%)71.4353.81+17.62 pp
Added-goal orderingTWINE pass (%)80.0074.76+5.24 pp

Response to W4, W5 & Q4: Equal-Budget Tuned Commander & Scope

  • Task-adapted commander (W5, Q4; App. C.3): To isolate system decomposition under an equal training budget, we fine-tuned a single-module Qwen3-VL-8B commander on the same data and schedule using only image and request (Table 24). Fine-tuning lifts matched-request decision accuracy from 60.00% to 88.89% with 50.00% completion, but on absent object with a visible hazard it reaches only 28.89% decision accuracy and 0.00% completion (TWILL: 82.22% / 42.22%), showing that task fine-tuning alone cannot replace an explicit proposal channel.
  • Task scope and sim-to-real (W4, Q4): Closed-loop evaluation uses frozen GR00T N1.6's 3 reliable RoboCasa365 skills (88.89% oracle) to decouple arbitration from execution, whereas SHUTTLE trains on open-text proposals across 127 kitchen families and generalizes to 69 unseen families (+5.22 pp over Direct; Table 26). Because gϕg_\phi and pθp_\theta take only RGB and text with no simulator state, deployment only swaps eψe_\psi for a real-robot VLA and adapts gϕg_\phi on paired on/off images, leaving SHUTTLE unchanged; robustness to real-world clutter, occlusion, and lighting awaits physical validation (Sec. 7).

Thanks for the detailed response and additional experiments. The new comparisons, ablations, and generalization results address most of my concerns and strengthen the paper. Real-world validation remains an open limitation. I will maintain my rating for accept support.

AuthorsOpenReview ↗

Thank You for Your Constructive Feedback and Support

We sincerely thank you for the thoughtful feedback and support for our work. Your suggestions directly helped us strengthen the comparisons, ablations, and generalization analyses in the revision. We also agree on the importance of real-world validation, and evaluating TWILL on physical robots will be a key focus of our future work.

Reviewer_yVxNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces TWILL, a robotic agent that combines proactive assistance with human requests: it forms its own proposal from the scene and arbitrates between this proposal and a person’s request. The pipeline has three modules on a shared Qwen3-VL-8B backbone. First, a proposer, fine-tuned with LoRA on paired on/off images, maps a single RGB image to a structured proposal covering six household problems (e.g., running faucet, open drawer). The second module, SHUTTLE, is called only when a request is present and arbitrates between the proposal and the request: on top of LoRA fine-tuning, a zero-initialized residual module at one decoder layer reads the context, the proposal and the request through separate attention paths (shared query, per-source keys and values, one softmax per source). Finally, a frozen off-the-shelf VLA (GR00T N1.6) executes the resulting language decision. The authors introduce TWINE, which evaluates the proposer, the decision module (on 180 controlled pairs), and closed-loop execution on three fixtures in RoboCasa365 kitchens.

Critical strengths
  • Relevant, well-posed problem. The paper addresses a core issue for household robots: arbitrating between the robot’s own judgment of the scene and a person’s request. It formalizes this as dual-intent decision making, with concrete cases (conditional, unsafe, absent-object or added-goal requests).
  • Rigorous decision evaluation. TWINE uses controlled pairs where a single factor changes the correct decision, so a model ignoring that factor cannot pass both cases. It adds invariance tests and three training seeds, with scene-family bootstrap CIs and Holm-corrected tests. The LLM judge is validated against human annotators.
  • Complete, transparent system. The paper closes the loop from perception to execution and evaluates it in closed loop. It reports results unfavorable to the proposed module (e.g., matched requests in Table 2, the merged-text ablation in Table 11). The paper is generally easy to follow.
Critical weaknesses
  1. Limited novelty.
    • The technical contribution assembles established components, as the paper itself acknowledges (Sec. 2): per-source attention with separate normalization is the classic multi-source decoder strategy (Libovický et al., 2018), and zero-initialized residual branches are standard (ReZero, Flamingo, ControlNet). The proposer is LoRA fine-tuning of a VLM.
    • The “robot’s own judgment” is overstated: the proposer amounts to a closed-set state classifier (see Method clarification).
    • The motivation is fragile. The paper argues that VLMs read the proposal and the request as one undifferentiated stream, yet the Direct baseline (LoRA, with fields separated by headings in the chat template) already reaches 68%. Given that the native model has 36 layers of multi-head attention free to focus on each source, it is unclear why one extra read at a single layer should be needed.
  2. Method clarification.
    • Content and source of the shared context (Fig. 1b). Besides the image, the decision module receives a shared context containing a scene description, the current task and a capability policy, but no example of these inputs is given. In particular, it is unclear who produces the scene description at test time. If another model generates it, this component should be described and evaluated. If it is derived from the simulator state (e.g., the list of objects present), it is privileged information unavailable to a real robot. This would, for instance, make absent-object requests (the only closed-loop condition where SHUTTLE outperforms Direct) largely solvable by text matching. This uncertainty directly affects the soundness of the closed-loop conclusions.
    • Context-free, closed-set proposer. The proposer is effectively a fine-tuned VLM classifier over six predefined problems (on/off or open/closed states) plus “no action”, each mapped to a single fixed resolution (stop flow, turn off, or close). Its structured answer (flag, mode, skill family, operator) is a classification output, the proposal sentence appears templated, and the evaluation scores only the classification fields. It sees a single image, without the current task, history, or information about people, so it cannot tell whether a state is actually a problem: an open drawer or a running faucet may be intended by the person or required by the robot’s own task. Since the proposal is executed directly when no request is present, without passing through the decision module (the only component that sees the task), the robot acts on this context-free judgment unless the person intervenes. Hiding the request from the proposer does not require hiding the task or the scene context; a goal- or context-conditioned proposer should be discussed or evaluated.
    • Interface with the executor. The decision module outputs free-form, often composite decisions with conditions, orderings and waits (“Hold for now; turn off the faucet once the sink is free”), whereas the frozen GR00T executor was trained on short atomic instructions. It is unclear how a frozen VLA can accommodate such decisions. Either the “agent harness” converts them into the executor’s native instructions, in which case their expressiveness is lost at execution and handled by unspecified hand-written rules (splitting steps, checking conditions, possibly mapping to a fixed instruction list), or the executor receives out-of-distribution text. In both cases, the closed loop can only exercise the simple atomic skills the frozen VLA already masters, which may explain why the core arbitration cases are not evaluated in closed loop. Some behaviors are explicitly hard-coded (e.g., the cancellation channel).
    • Decoupled design and lack of joint training. The three components are trained separately, on different data and objectives, and barely interact at inference: the proposer sees one image without history, task or request; the decision module is skipped when no request is present; and the frozen executor provides no feedback. The decision module thus never learns from execution outcomes, its free-form decisions are never aligned with what the executor can carry out, and proposer errors are executed without downstream correction. No failure attribution across components is provided. Hierarchical systems that jointly train or co-train the language and action levels and handle human interjections (e.g., π0.5, π0.7, Hi Robot) are neither discussed nor compared. A jointly trained variant, or at least a justification of the decoupled design with a per-component failure analysis, is needed.
  3. Experiments.
    • Weak baselines.
      • Proposer: compared only with the untuned backbone, which scores below a trivial always-hold classifier (41.27% vs. 56.55% joint accuracy). Natural baselines for this closed-set task are missing: a supervised image classifier, a frontier VLM, and perception pipelines that localize fixtures with open-vocabulary detection or segmentation and then classify their state on the localized region (or query a VLM on it, e.g., with zoom-in tools), which is particularly relevant for subtle states such as a running faucet.
      • Decision: the baselines are fusion variants on the same backbone (Direct, matched adapter, hierarchical attention, DMRM-style), which are closer to ablations than to state-of-the-art comparisons. Missing are prompting or reasoning-based approaches, hierarchical VLM–VLA systems, and frontier VLMs (Gemini zero-shot appears only in the appendix, scored by itself).
      • Closed loop: the only VLM commander is the untuned backbone. The performance of the same backbone fine-tuned on the same data as a single module is missing, as well as a frontier-VLM commander, and a hierarchical system such as Hi Robot.
    • Small and partly insignificant gains. SHUTTLE improves over Direct by 3.70 points, of the same order as seed-to-seed variance (Direct ranges from 62.78 to 71.67%). Against the merged-text ablation, which isolates the paper’s central claim of separate per-source reads, the gain is not significant (+1.94, 95% CI [0.00, 3.98], Holm p = 0.06).
    • The closed loop does not support SHUTTLE.
      • Without a request, SHUTTLE is not called, so both TWILL variants share the same result (81.48%). The headline comparison in the abstract therefore measures the proposer.
      • With a matched request, SHUTTLE performs worse than Direct (88.15 vs. 96.30%) and no better than the instruction-driven VLA (88.89%). It helps only on absent-object requests, where it still fails most of the time (42.22%).
      • Cancellation is 100% for every system because of a shared hard-coded channel, so it is uninformative.
      • The core arbitration cases (conditional requests, ordering, added goals) are not evaluated in closed loop beyond a single qualitative episode, and the scale is small (3 fixtures, 11 scene resets).
    • Ad hoc and narrow evaluation. All results come from TWINE, a benchmark built by the authors on RoboCasa365 scenes rather than from the framework’s standard task suite. It covers six hand-picked close/turn-off problems, three fixtures in closed loop (11 scene resets), and template-generated requests, scored with author-defined obligations and an LLM judge. Since the training data come from the same templates and authors, the risk of fitting the method to its own evaluation design is high. Held-out scene families and styles do not compensate for this: no standard benchmark, no existing proactive-assistance benchmark, no real humans, and no real-robot experiment are reported. Evaluation on at least one external benchmark and broader closed-loop coverage (more fixtures and the core arbitration cases) are needed.
Questions for authors

I like the problem and the controlled-pair evaluation methodology. In the rebuttal, I would mainly expect answers to the following:

  • Stronger baselines: results for the proposer against a supervised classifier, detection/segmentation-based pipelines and a frontier VLM; and, for the decision and closed-loop levels, against the same backbone fine-tuned as a single module, a frontier-VLM commander and a hierarchical system such as Hi Robot.
  • Interface and context: how are free-form decisions converted into commands for the frozen GR00T executor, and which behaviors are hard-coded in the agent harness (step splitting, condition checking, instruction lists)? Who produces the scene description of the shared context at test time, and is it derived from the simulator state?
  • Decoupled design: what justifies training the three components separately rather than jointly? Can you provide a per-component failure analysis of the closed-loop results, and closed-loop evaluations of the core arbitration cases (conditional requests, ordering, added goals)?
7 comments
AuthorsOpenReview ↗

Response to Reviewer yVxN (Part 1: Context, Executor Interface & Architecture)

We thank Reviewer yVxN for the thorough review and address all points in three parts: Part 1 covers input transparency, executor grounding, and SHUTTLE's motivation; Part 2 covers the proposer and 7-seed significance; Part 3 covers frontier VLMs, closed-loop attribution, and benchmark generalization.

1. Shared Context and Executor Interface (Method Clarification & Q2)

  • No privileged simulator state in ctc_t: To ensure evaluation reflects realistic perception rather than text matching over simulator state, ctc_t contains only the RGB camera image and static prompt templates (verbatim in App. C.1). At test time, [Scene] is the fixed string "The attached image is the current scene." in TWINE and "The kitchen area that the robot currently sees." in closed loop, alongside the standing monitoring policy and nine supported command strings. No object list or simulator metadata enters ctc_t, so hazards and absent objects must be recognized visually.

  • Mapping free-form decisions to the frozen VLA: To bridge composite decisions yty_t with frozen GR00T N1.6's nine fixed executor commands without passing out-of-distribution text or losing conditional logic, all closed-loop systems share the same command-grounding interface, applied every 8 motor steps (App. C.2):

  • Per-cycle vocabulary alignment: Every 8 motor steps, regex rules hold if yty_t expresses waiting/holding or names no supported fixture; otherwise yty_t matches one of the nine native commands directly or maps to a fixture family by keyword (e.g., drawer, burner, cabinet door, faucet), with the frozen base VLM disambiguating multiple spatial instances from RGB.

  • Multi-cycle conditional execution: Rather than compressing multi-step or conditional decisions into a single motor call, each cycle executes only the active step or holds while a condition applies, then re-observes and re-decides on subsequent frames until two consecutive visual checks confirm completion or /cancel arrives (Fig. 5). Intent arbitration remains inside the decision module, while the harness applies identical command grounding across all baselines.

2. Architectural Motivation (Limited Novelty & Motivation)

Dual-intent arbitration requires decision tokens to preserve both the content and the source identity of ℓp\ell^p and ℓh\ell^h:

  • Shared-softmax competition: Native causal attention and Merged-text normalize ℓp\ell^p and ℓh\ell^h under a single softmax, so high attention weight on a salient request directly reduces weight on the proposal and obscures which source issued each constraint.

  • Multi-source decoders: Classic multi-source modules (Hierarchical, DMRM) add secondary cross-source attention to align separate encoders. Inside a pretrained VLM where Layers 1–18 already jointly encode vision and text, extra cross-source mixing blurs source boundaries, and these modules underperform Direct (66.48% and 67.50% vs. 68.15%; Table 9).

  • Single-layer decoupled reads (ours): While SHUTTLE builds on established per-source normalization and zero-initialized residuals, its goal is minimal, source-preserving adaptation inside a pretrained VLM. At Layer 18, SHUTTLE reads cc, ℓp\ell^p, and ℓh\ell^h with independent softmaxes and feeds their fixed-order concatenation to a zero-initialized MLP on decision tokens only (Eqs. 5–7), preserving explicit source slots while leaving prefix states and KV caches intact (0.71 vs. 0.70 s/case; Table 9).

To isolate separate normalization from added capacity, we compared SHUTTLE against a generic adapter and Merged-text across 7 seeds: generic parameters leave arbitration-direction errors unchanged, whereas independent softmaxes improve source attribution on swapped pairs (87.50% vs. 84.72%; Table 16) and reduce direction errors to drive 72% of SHUTTLE's 7-seed gain (Table 25a).

AuthorsOpenReview ↗

Response to Reviewer yVxN (Part 2: Proposer Grounding, Baselines & 7-Seed Significance)

3. Proposer Design, Baselines & Factorization (Proposer, Baselines & Q1)

  • Proposer safety and extensibility: To keep autonomous execution safe when no request is present, gϕg_\phi is currently bounded to six reversible fixture resets. To avoid overfitting scene layout and to suppress false triggers on clean scenes, Eq. 2 contrasts paired on/off images of the same scene, which in the 3-seed retraining of Table 28 cuts mean false proposals from 14.7 to 9.7 (−34%). For extensibility, the shared VLM emits natural-language proposals ℓp\ell^p rather than a fixed KK-way classifier, so pθ(y∣c,ℓp,ℓh)p_\theta(y \mid c, \ell^p, \ell^h) accommodates new rules or external detectors without changing SHUTTLE (App. C.6). We have also made a first check of decision transfer under a different renderer in AI2-THOR (Kolve et al., 2017); extending that evaluation to more rooms and fixture types is future work.

  • Benchmarking against classifiers, detectors, and cloud VLMs (Q1): To compare against specialized perception pipelines, we evaluated five front ends on the 198 held-out observations, training supervised baselines on the identical 1,416 paired images (App. C.6, Table 27):

Proposer variantRestraint (%)FPDirect NL ℓp\ell^pExtra vision model
TWILL proposer (ours, shared 8B)96.843YesNo
Supervised CLIP ViT-L/14-336 (3 seeds)91.7011.0No (7-way ID)Yes (428M)
Gemini-3.8-Flash zero-shot91.2310YesCloud API
OWLv2 + CLIP region classifier69.4738No (crop ID)Detector + CLIP
Qwen3-VL-8B-Instruct zero-shot23.1688YesNo

Cropping fixtures via OWLv2 loses the surrounding scene geometry needed to disambiguate open/closed states, and a global CLIP classifier lacks a paired-state margin; our proposer reaches the highest restraint on the shared backbone.

  • Why gϕ(ot)g_\phi(o_t) is decoupled from context: Separating visual candidate detection ℓtp=gϕ(ot)\ell_t^p = g_\phi(o_t) from context-conditioned arbitration yt∼pθ(⋅∣ct,ℓtp,ℓth)y_t \sim p_\theta(\cdot \mid c_t, \ell_t^p, \ell_t^h) serves three purposes: (i) Unbiased recall, since conditioning detection on ℓth\ell_t^h lets instruction-following priors suppress unmentioned hazards (our request-conditioned single-module commander completes 0/22 absent-object episodes; App. C.3); (ii) Sample efficiency, learning state boundaries from paired images without combinatorial scene–dialogue supervision; and (iii) Low-overhead monitoring on safe, reversible resets when ℓth=∅\ell_t^h = \emptyset (96.84% restraint). We agree that a single image cannot fully capture ongoing human activity or task history, and have listed multi-frame history and human context as key future work (Sec. 7).

4. Multi-Head Specialization & 7-Seed Significance (Motivation & Small Gains)

  • Why Direct fails under conflict: TWINE evaluates both cooperative and competing dual-intent interactions. On the 90 confirmation and local-revision cases, where ℓh\ell^h confirms or locally refines ℓp\ell^p, the two sources agree and no attention competition arises, so Direct is already near ceiling (96.98% across 7 seeds) and the average gap over all 360 cases narrows. Once ℓp\ell^p and ℓh\ell^h compete, sequence-level cross-entropy does not force native heads to partition by source and instruction-following priors favor ℓh\ell^h: on the 150 dual-intent conflict and revision cases, Direct drops to 62.48% against 70.10% for SHUTTLE (+7.62 pp; Table 25b), with direction errors causing 57.0% of Direct's failures (App. C.4).

  • Testing unconstrained attention heads: To test whether multi-head freedom suffices under a shared softmax and to reduce seed variance (3-seed Holm p=0.0604p = 0.0604, Table 12), we retrained 7 seeds (Table 3a, App. C.4) and evaluated Merged-text, which keeps per-source projections over a shared [ℓp;ℓh][\ell^p; \ell^h] softmax. Merged-text shows no significant gain over the source-blind 5.78M adapter (65.52% vs. 66.71%) and makes more source-attribution errors on swapped pairs (84.72% vs. 87.50%; Table 16). Restricting each softmax to its own source in SHUTTLE reaches 69.13%, significantly above Merged-text (+3.61 pp; +8.48 pp on conflict and revision cases), the adapter (+2.42 pp), and Direct (+4.84 pp).

AuthorsOpenReview ↗

Response to Reviewer yVxN (Part 3: Cloud VLMs, Closed-Loop Breakdown & Benchmarks)

5. Frontier VLM Baselines & Judge Verification (Decision Baselines & Q1)

To benchmark against frontier cloud VLMs and check for judge self-preference, we evaluated zero-shot GPT-6-Luna and Gemini-3.8-Flash on all 360 TWINE cases (App. C.6, Table 29):

Decision model (360 TWINE cases, Gemini-3.8-Flash judge)Pass (%)Median latency (s)Local execution
TWILL + SHUTTLE (8B, ours)71.851.96Yes
GPT-6-Luna zero-shot (med. reasoning)71.1126.65No (cloud API)
Gemini-3.8-Flash zero-shot (med. reasoning)46.117.01No (cloud API)
  • Gemini-3.8-Flash assigns only 46.11% to its own zero-shot outputs and agrees with Qwen3.8-Flash on 92.70% of 6,480 judgments (κ=0.824\kappa = 0.824; human κ=0.849\kappa = 0.849, App. B.5), indicating no evident self-preference.
  • On a single local GPU, 8B SHUTTLE matches GPT-6-Luna in accuracy at 13.6×13.6\times lower median latency.

6. Decoupled Training, Single-Module Hierarchy & Closed-Loop Analysis (Q1 & Q3)

  • Design considerations for modular training: Training the proposer, decision module, and executor separately rests on two considerations.

  • Sample efficiency: the proposer learns state boundaries from paired on/off images, the decision module trains on text decisions, and the executor is reused frozen, so no robot trajectories are collected.

  • Isolating interference and enabling replacement: separate adapters on one backbone let the proposer fine-tune vision while the decision module keeps vision frozen, avoiding conflicting updates; upgrading either the proposer or the executor needs no retraining of SHUTTLE.

  • Comparison with hierarchical VLAs: Hierarchical models such as Hi Robot are trained only on human instructions and have no separate scene-proposal channel. We therefore fine-tuned a single-module commander without proposal input on the same backbone and data and drove frozen GR00T with it: it reaches 88.89% decision accuracy and 50.00% completion on matched requests, but drops to 28.89% and 0.00% when an absent target co-occurs with a visible hazard, against 82.22% and 42.22% for TWILL (App. C.3, Table 30).

  • Per-component closed-loop attribution (App. C.7, Table 30):

  • Request-free and Cancel: These probe autonomous monitoring and the shared stop mechanism rather than arbitration. With no request the decision module is not called and the proposer drives the executor directly, so SHUTTLE and Direct both reach 81.48%; mid-episode cancellation tests the shared safety stop.

  • Matched-request gap: Table 30 decomposes the failures behind SHUTTLE's slightly lower matched-request completion. The gap comes from stochastic frozen-executor timeouts and from proposer misreads in two cropped or shadowed views: given a false conflicting proposal, SHUTTLE weighs both inputs, whereas Direct repeats the request regardless.

  • Closed-loop conflict and broader arbitration: To locate the completion bottleneck under conflict, Table 30 decomposes the absent-object condition. Direct's habit of repeating the request misses the on-site hazard, whereas SHUTTLE keeps the valid proposal and nearly doubles completion (42.22% vs. 21.48%); the remaining failures after a valid decision are all frozen-executor timeouts rather than arbitration errors. Fig. 5 verifies closed-loop conditional deferral, and for forbidden actions and action ordering beyond the executor's three atomic skills, 7-seed TWINE shows gains of 17.62 pp and 5.24 pp over Direct (Table 25b).

7. External Benchmarks & Unseen-Scene Generalization (Evaluation Scope)

Standard manipulation benchmarks (RoboCasa365, BEHAVIOR, ALFRED) contain no robot proposal, while proactive benchmarks test when to initiate help rather than arbitrating between proposal and request (App. C.5). To test generalization, we built a benchmark whose train, development, and test kitchen families are disjoint with low request overlap (mean Jaccard 0.279; Fig. 6), and evaluated the 7-seed models on 156 cases across 69 further unseen RoboCasa365 families (Table 26): SHUTTLE reaches 69.23%, ahead of Direct (+5.22 pp) and Merged-text (+2.66 pp).

Reviewer_RkMAOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TWILL, a proactive robotic system that combines an image-conditioned proposer, a language decision module, and a frozen GR00T N1.6 executor. Both the proposer and the decision module use Qwen3-VL-8B-Instruct with task-specific adaptation. The decision module adds SHUTTLE, a residual attention module at decoder layer 18 that separately reads the scene context, robot proposal, and human request. The paper also introduces TWINE, which evaluates proposal accuracy, paired language decisions, and closed-loop behavior in simulated kitchens.

Critical strengths
  1. Relevant problem formulation. Coordinating scene-driven proposals with human requests is useful for proactive robots, particularly when requests introduce conditions, additional goals, or invalid targets.
  2. Improved restraint in proposal generation. Within the evaluated kitchen scenes, the trained proposer increases correct holding on no-action observations from 23.16% to 96.84%, reducing false proposals from 88 to 3. Avoiding unnecessary interventions is a useful capability for a proactive robot.
Critical weaknesses
  1. The motivation for the added attention module needs stronger support. The decision task is learned through supervised image-and-text-to-decision training. Although SHUTTLE improves over Direct, the paper provides limited evidence that competition between input sources in native attention is the bottleneck responsible for the observed errors.
  2. A matched fine-tuned VLM baseline is missing. A useful baseline would fine-tune a VLM on the same task data to take the scene image and human request as input and directly generate the executor instruction. The reported VLM commander is untuned.
  3. Limited evaluation scope. The proposer covers six predefined problems involving closing or turning off fixtures, and closed-loop evaluation covers only faucets, cabinet doors, and drawers in simulation. The reported results therefore establish performance within this limited task scope; transfer to other household tasks or physical robots has not been demonstrated.
Questions for authors

See weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer RkMA (W1: Mechanism, W2: Tuned Commander, W3: Generalization & Scope)

We thank Reviewer RkMA for the clear and constructive comments.

Response to W1: Input-Source Competition and What SHUTTLE Changes

Across all 360 TWINE cases, 57.0% of Direct's failures are arbitration-direction errors—wrong act/wait/hold mode or wrong governing source (20.28% of runs; Sec. 6.5, App. C.4). To test whether one softmax over context, ℓp\ell^p, and ℓh\ell^h drives them, we trained four variants on identical data and schedules across 7 seeds (Table 25):

Method (7-seed mean, Qwen3.8-Flash)Pass (%)Dir. err. (%)Dir. err. Δ\Delta vs. Direct [95% CI]pp
SHUTTLE (separate softmax)69.1316.79−3.49-3.49 [−4.68-4.68, −2.34-2.34]<0.0002< 0.0002
Matched adapter (+5.78M)66.7119.72−0.56-0.56 [−1.55-1.55, +0.40+0.40]0.27
Merged-text (shared [ℓp;ℓh][\ell^p; \ell^h] softmax)65.5218.85−1.43-1.43 [−2.62-2.62, −0.28-0.28]0.011
Direct (native self-attention)64.2920.28——
  • Mechanism analysis of per-source attention: Holding the insertion layer and added parameters fixed, generic capacity leaves direction errors unchanged, whereas splitting Merged-text's shared softmax into per-source softmaxes cuts them by 2.06 pp, accounting for 72% of SHUTTLE's gain (3.49/4.84; Table 25a). Gains concentrate on the 150 conflict and revision cases: SHUTTLE reaches 70.10%, 7.62 pp above Direct and 8.48 pp above Merged-text (Table 25b).
  • Role of the residual branch: With all weights fixed, zeroing the residual Δdj\Delta d_j (Eq. 7) at test time lowers pass rate on all seven seeds, by 3.13 pp on average (App. C.4); the residual therefore carries most of the gain at inference.

Response to W2: Matched Fine-Tuned Single-Module VLM Commander

To test whether task-specific fine-tuning removes the need for an explicit proposal channel ℓp\ell^p, we fine-tuned Qwen3-VL-8B on the same 6,268 demonstrations and schedule using only image, context, and request to generate commands (App. C.3, Tables 23–24):

Closed-loop systemMatched: Decision / End-to-End (%)Absent obj. + hazard: Decision / End-to-End (%)
TWILL + SHUTTLE (ours)95.56 / 88.1582.22 / 42.22
VLM commander, fine-tuned (no ℓp\ell^p)88.89 / 50.0028.89 / 0.00
VLM commander, untuned (no ℓp\ell^p)60.00 / 60.0033.33 / 0.00

Fine-tuning raises initial decision accuracy from 60.00% to 88.89%, though mid-episode replanning triggers early holds in 7 of 20 valid runs, giving 50.00% completion. On Absent object it decides validly in 6 of 22 episodes, completes none, and passes 3.61% offline vs. 66.11% for same-schedule Direct. The untuned backbone rarely declines to act, so it occasionally hits the hazard by chance, whereas fine-tuning teaches the model to reject nonexistent targets; without ℓp\ell^p, that rejection resolves to holding instead of addressing the unmentioned hazard, making its rate slightly lower (App. C.3, Table 7).

Response to W3: Scope, Unseen Generalization & Sim-to-Real

To test generalization beyond TWINE's 60 test scene families, we evaluated the 7-seed models on 156 cases across 69 additional unseen RoboCasa365 kitchen families (App. C.5, Table 26):

Evaluation set (7 seeds)CasesSHUTTLEDirectMerged-textSHUTTLE −- Direct [95% CI] (pp)SHUTTLE −- Merged-text [95% CI] (pp)
TWINE (60 families)36069.1364.2965.52+4.84 [+3.45, +6.27] (<0.0002< 0.0002)+3.61 [+2.18, +5.08] (<0.0002< 0.0002)
Unseen (69 families)15669.2364.0166.58+5.22 [+3.14, +7.51] (<0.0002< 0.0002)+2.66 [+0.99, +4.41] (0.00160.0016)
  • Task scope and unseen-scene generalization: To decouple arbitration from motor execution, closed-loop runs use frozen GR00T N1.6 on its 3 reliable fixture skills (88.89% oracle), where SHUTTLE nearly doubles Direct's completion under conflict (42.22% vs. 21.48%; App. C.7). TWINE further tests placement, removal, prerequisites, and prohibitions via open-text ℓp\ell^p over 127 training, 60 test, and 69 unseen families. As a first check of visual-domain transfer, the same 7-seed models evaluated zero-shot on 82 paired AI2-THOR cases (Kolve et al., 2017) still rank SHUTTLE first; end-to-end transfer would require adapting the executor:
AI2-THOR offline set (7 seeds, Qwen3.8-Flash)SHUTTLEDirectMerged-textAdapterSHUTTLE −- Direct [95% CI]
XSIM_THOR-82 (28 kitchens, 3 fixtures)65.5158.5462.7262.72+6.97 [+3.93, +10.00]
  • Transferability analysis: Because gϕg_\phi and pθp_\theta take only standard RGB images and text with no privileged simulator state (App. C.1), physical deployment requires only replacing frozen eψe_\psi with a real-robot VLA and adapting gϕg_\phi on paired on/off images, without trajectory collection or changes to SHUTTLE. Real-world clutter, occlusion, and lighting remain important limitations awaiting physical validation (Sec. 7).

Comments on the paper

3 comments
AuthorsOpenReview ↗

Global Response: Summary of 5 New Experiment Groups & Method Clarifications

We thank the Area Chair and all reviewers for their constructive feedback. The revised manuscript adds five experiment groups (Section 6, Table 3, Appendix C) and provides verbatim prompts and executor-grounding rules (Appendices C.1–C.2).

New experimentLocationMain result (TWILL/SHUTTLE vs. baselines)
1. Fine-tuned 8B single-module commander (no ℓp\ell^p)App. C.3, Tables 23–24Absent object + hazard (closed loop): TWILL 82.22% decision / 42.22% completion vs. 28.89% / 0.00%
2. 7-seed TWINE evaluation (360 cases)Table 3a, App. C.4Pass rate: 69.13%, ahead of Direct (+4.84 pp), Merged-text (+3.61 pp), and adapter (+2.42 pp); all p≤0.001p \le 0.001
3. Failure attribution & mechanismTable 3a, App. C.4Direction error: 16.79% vs. Direct 20.28% (−3.49-3.49 pp, p<0.0002p < 0.0002; 72% of gain); adapter 19.72% (p=0.27p = 0.27)
4. Unseen families & closed-loop conflictTable 3b, Apps. C.5, C.769 unseen families: 69.23% vs. Direct 64.01% (+5.22 pp, p<0.0002p < 0.0002); conflict completion: 42.22% vs. 21.48% (p=0.0056p = 0.0056)
5. Proposer baselines, λ\lambda ablation & cloud VLMsApp. C.6, Tables 27–29Proposer: 96.84% restraint, 3 FPs (λ=0.1\lambda = 0.1 cuts FPs 34%); SHUTTLE (71.85%, 1.96 s) matches GPT-6-Luna (71.11%, 26.65 s)
  1. Role of an explicit proposal channel (RkMA W2, yVxN Q1, 7Myw W5): To test whether a single task-adapted VLM suffices without ℓp\ell^p, we fine-tuned Qwen3-VL-8B on the same demonstrations using only the image and request. It executes valid requests (88.89% initial decision accuracy) but completes 0/22 closed-loop episodes when an invalid request co-occurs with a visible hazard, compared with 42.22% for TWILL.
  2. Mechanism of per-source attention normalization (RkMA W1, yVxN Motivation, 7Myw W1): To test whether arbitration errors arise from cross-source attention competition rather than limited capacity, we compared parameter-matched variants across 7 seeds. Adding 5.78M generic parameters leaves arbitration-direction errors unchanged (p=0.27p = 0.27), whereas replacing a shared [ℓp;ℓh][\ell^p; \ell^h] softmax with per-source softmaxes reduces them by 3.49 pp over Direct (p<0.0002p < 0.0002, accounting for 72% of the gain), improving pass rates on TWINE, 69 unseen scene families, and conflicting closed-loop episodes.
  3. Proposer restraint and local deployment (RkMA W3, yVxN Q1, 7Myw W3): To evaluate our perception and deployment trade-offs, we benchmarked the proposer against supervised classifiers and detectors, ablated its ranking loss, and compared SHUTTLE with cloud VLMs. Adapting the shared 8B backbone with pairwise state ranking yields the highest restraint (96.84%) and cuts false proposals by 34%, while local SHUTTLE matches zero-shot GPT-6-Luna at 13.6×13.6\times lower median latency.
AuthorsOpenReview ↗

Global Response: Table numbering in the revised manuscript, and where to find it

The revised manuscript has been uploaded as supplementary material, with every post-submission addition and refinement highlighted in blue. Section, figure, and appendix numbers are unchanged. Tables 1–25 keep their numbers. The revision adds two tables:

  • Table 26 — seven-seed replication under the original Gemini-3.8-Flash judge (App. C.4).
  • Table 31 — absent-object episodes by fixture (App. C.7).

Accordingly, former Tables 26–29 are now Tables 27–30, and the former component-breakdown table is now Table 32.

Cited in our Round-1 replies asIn the revised manuscript
Table 26 — generalization on 69 unseen familiesTable 27
Table 27 — proposer front ends / baselinesTable 28
Table 28 — pairwise-ranking ablationTable 29
Table 29 — frontier VLM and on-robot deployment referenceTable 30
Table 30 — closed-loop component breakdownTable 32
—Table 26 (new: judge replication)
—Table 31 (new: by fixture)

Our Round-1 replies predate this revision and use the earlier numbering, so Tables 26–30 cited there should be read with the right-hand column. Our Round-2 replies already use the revised numbering and need no adjustment.

Area_Chair_iGnjOpenReview ↗

Please Review the Author Rebuttal and Participate in the Discussion

Dear Reviewers,

Thank you for your efforts in reviewing this submission. As the discussion period is approaching its end, I would like to kindly encourage everyone to actively participate in the discussion and carefully consider the authors' rebuttal.

In particular, if you have not yet responded, please share your thoughts on whether the authors have adequately addressed your concerns. Please also engage with fellow reviewers where appropriate, and provide your final assessment, including any score updates or a brief justification for maintaining your original rating.

Your timely input would be greatly appreciated and will help us reach a fair and well-informed decision.

Thank you again for your time and valuable contributions!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.