acceptodds
Under review as a conference paper at ICLR 2027

Utility-Guided World Feedback for Visuotactile Manipulation

Abstract

World–action models jointly predict future observations and actions, but predictive accuracy alone does not determine which visual and tactile forecasts are useful for control. This gap is particularly important in contact-rich manipulation, where the utility of predicted evidence varies across sensors and time. We propose Utility-Guided World Feedback (UWF), a framework that learns to assess and selectively incorporate predicted evidence into action generation. UWF estimates the marginal utility of sensor–time evidence groups against a shared supervised action objective, converts utility-weighted evidence into action-chunk corrections, and regulates their adoption through sequence-level risk estimates. A shared action reference supports consistent evidence assessment, while joint world–action denoising updates feedback as predictions evolve. The framework learns from demonstrations without additional reward annotations or simulator rollouts for utility supervision. Across eight UniVTAC simulation tasks, UWF achieves a mean success rate of 90.1%. On eight real-robot tasks evaluated against four baselines with 20 trials per task and method, UWF achieves 91.3% mean success, exceeding the strongest baseline by 11.9 percentage points and obtaining six task wins and two ties. These results support utility-guided selection of predicted visuotactile evidence as an effective approach to contact-rich manipulation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
World–Action Models, Visuotactile Manipulation, Utility-Guided Feedback

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PEGmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces UWF, a framework that selectively uses predicted visual and tactile information to improve action generation. It estimates the usefulness of different sensor predictions through their contribution to matching demonstrated actions, then converts the selected information into action corrections. Experiments in simulation and on real robots demonstrate improved manipulation success over the compared baselines.

Critical strengths
  • The paper identifies a meaningful distinction between predicting future observations accurately and predicting information that is useful for control. This is particularly relevant to contact rich manipulation.

  • The method derives its utility supervision from existing demonstrations without requiring additional reward annotations or simulator rollouts.

Critical weaknesses
  • My biggest concern is that the utility measure rewards corrections that reduce action denoising error relative to demonstrations, but this does not necessarily establish their usefulness for completing the task. Different actions may achieve the same goal, and recovering from an unexpected contact may require behavior that differs from the demonstrated motion.

  • Then, the gate’s ability to recognize unreliable corrections is not sufficiently established. If i understand correctly, they are learned from demonstrations, so an unfamiliar contact situation could make both the predicted sensory information and the acceptance decision inaccurate.

Questions for authors

In the real robot protocol, why are visual and tactile keyframes collected every three action slots but incorporated into the history only after the complete chunk, and what mechanism handles unexpected contact changes during that interval?

Given that zero predicted visual and tactile change forces a zero correction, how does UWF handle cases where the absence of an expected change is itself a reason to revise the action?

Can author clarify why using one correction scale for the entire action chunk, given that an unfavorable estimate in any temporal block rejects the whole correction?

3 comments

Response to Reviewer PEGm (Part 1/2)

Thank you for the thoughtful review. Several of your points are real limitations, and we say so directly. Where we have new evidence, it comes from 27 models retrained under matched conditions, 9 of them also evaluated in the closed-loop simulator. New results are marked (rebuttal).

W1: reducing denoising error against demonstrations is not the same as task usefulness

Conclusion: we agree that the utility target is a proxy. The external check is closed-loop task success. On the task we tested, removing utility weighting lowers success by 11 points. This supports the proxy there, but it does not show that the proxy and usefulness coincide in general.

The utility target rewards evidence that reduces the action flow-matching loss against demonstrations. Two consequences you raise are correct:

  • Behaviour that is absent from the demonstrations gets no credit, even if it would succeed. Recovering from an unexpected contact is the clearest case.
  • The proxy can, in principle, favour evidence that helps imitate the demonstration but not the task.

One clarification on multiple valid actions. The target is the flow-matching velocity, whose minimiser represents the whole demonstrated action distribution rather than a single trajectory. So several demonstrated strategies are not penalised against each other. Strategies that no demonstration contains are, as you note, not covered.

New evidence (rebuttal). We retrained UWF with every evidence group weighted equally, which turns the utility-guided feedback into ordinary attention. All other settings are unchanged. Results on Insert HDMI, closed loop, 100 evaluation episodes:

modelsuccess /100
UWF90
uniform evidence weights79 (Fisher p = 0.049)

Over 3 training seeds the uniform model also has a higher action loss on every seed (mean +7.3%). So on this task, training with the demonstration-derived utility improves task success, not only the proxy. This is one task and one evaluated training seed. We present it as evidence that the proxy is useful here, not as proof that it measures task usefulness.

Revision: the limitations section will state that the utility target is a demonstration-relative proxy and cannot credit successful behaviour outside the demonstrations. The uniform-weight ablation will be added as the task-level check.

W2: the gate is learned from demonstrations, so unfamiliar contact can make it unreliable

Conclusion: conceded. The gate is not a safety certificate and gives no guarantee under unfamiliar contact. What it does provide is a structural fallback to the base policy.

  • The risk head can be wrong out of distribution. It is a quantile estimate fitted on demonstration data, so it can be wrong exactly where the predicted evidence is wrong.
  • Fallback to the base policy. When λ = 0 the applied velocity is exactly the base velocity v₀. We verified this in the recorded data: the empty-evidence residual is exactly 0.0 and reproduces the base loss to float64 precision.

In-distribution evidence (rebuttal). We overrode the gate's decision on all three insertion tasks in closed loop.

  • With a standing correction λ = 0.5, success is 263/300, against 281/300 with the gate (stratified p = 0.017).
  • With the correction always on (λ = 1), success on Insert HDMI is 84, against 90.

On these tasks the certified λ is 0 on the training episodes. So the gate is declining the correction rather than choosing among corrections, and declining it is the better decision. This is in-distribution evidence only. It does not show that the gate recognises unreliable corrections under unfamiliar contact, and we do not claim it does.

Revision: the method and limitations sections will describe the gate as a demonstration-calibrated surrogate whose failure mode is falling back to the base policy, not as a safety mechanism.

Response to Reviewer PEGm (Part 2/2)

Q1: keyframes every 3 slots, but history committed only after the full chunk

Conclusion: this is the deployed execution protocol, which UWF inherits from N0-TWAM. Nothing revises the action within a chunk. This is a limitation of the protocol, and the gate does not compensate for it.

What the protocol does:

  • Observations. Real visual and tactile observations are captured every 3 action slots, i.e. 4 keyframes per 12-slot frame.
  • Commitment. When a chunk completes, its observed keyframes are committed to the model's history. An aborted chunk commits nothing.
  • Chunk length. After the cold-start chunk, each chunk executes 24 slots.

What it does not do:

  • Inside a chunk, the captured observations do not yet condition the actions being executed. They condition the next chunk.

So an unexpected contact change mid-chunk is reflected only from the next chunk on. Within the current chunk the only safeguard is abort. We do not claim any mechanism beyond this.

Revision: the appendix on deployment will state the commit rule and this limitation explicitly.

Q2: zero predicted change forces zero correction. What if the absence of an expected change should revise the action?

Conclusion: you are right. UWF cannot act on the absence of an expected change. This is a genuine architectural limit.

The residual decoder is zero-initialised and has no bias, so zero predicted evidence yields exactly zero residual. We confirmed this numerically: the empty-evidence residual has a maximum absolute value of exactly 0.0.

Acting on a missing change, for example "contact should have occurred by now", would need a conditioning signal on the discrepancy between expected and observed sensory change. That is a different design from conditioning on the predicted change itself.

Revision: this case will be stated as a limitation, with the discrepancy signal named as the required extension.

Q3: why one correction scale for the whole chunk, when one unfavourable block rejects everything?

Conclusion: the shared scale is a conservative design choice. A single unfavourable temporal block does cap the whole chunk, as you describe. We measured the cost: it is real but small.

The deployed rule is

λ = clip[0,1]( min( aggregate term, min_b q_b / E_b ) ).

Here q_b is the predicted lower-bound benefit of block b and E_b is its residual energy, with 4 blocks per chunk. The second term is a worst-block constraint.

Exact recomputation (rebuttal). We used the recorded residuals for all 256 evidence subsets at three training windows. The gate identity err(λ) − err(0) = λ²E − λB holds to 2.3e-18, so these are exact recomputations.

  • In every window only 1–2 of the 4 blocks have a positive benefit.
  • A per-block scale beats the shared scale in every window, by 0.0008%, 0.016% and 0.12% of the action loss.

Retrained per-block gate (rebuttal). We retrained UWF with one scale per block.

  • Closed loop: 88/100, against 90/100 with the shared scale (p = 0.82).
  • Over 3 training seeds: action loss +1.76% (p = 0.54).

The per-block rule recovers what the shared rule rejects, but at this training length that amount is too small to change task success.

Revision: the shared-scale restriction and its measured cost will be stated as a limitation in the method section.

Ethics flag

The ethics statement records that no experiments were conducted with humans as research subjects. It also states that the gate is not a safety guarantee and that deployment near people requires independent safeguards. If a specific concern prompted the flag, we would be glad to address it directly.

Thank you to the authors for the detailed rebuttal, which addressed most of my concerns.

However, my one remain concern is the limited scope of evaluation, particularly regarding robustness to unexpected contacts and out-of-distribution scenarios, epseically for real-robot eval. So, although the rebuttal has strengthened the paper, I will maintain my original rating.

Reviewer_M62DOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes UWF, which adapts a pretrained visuotactile world-action model to selectively use predicted sensory changes for action generation. Demonstration-derived marginal contributions supervise a utility head that weights sensor-time evidence groups. A feedback head converts this evidence into action-velocity corrections, while a lower-quantile benefit estimate determines a shared correction scale for the action chunk. The paper reports results on eight UniVTAC tasks and eight real-robot tasks, together with reconstruction and action-conditioning diagnostics.

Critical strengths
  1. The paper investigates which predicted visual and tactile information actually helps action generation, a useful research question. The demonstration-based utility targets and explicit separation between evidence weighting and correction adoption offer a concrete approach.
  2. The full system is evaluated in simulation and on real hardware, with reported improvements over N0-TWAM. These results provide evidence of gains across multiple tasks.
Critical weaknesses
  1. The method is relatively complex, but lacks component ablations to justify this complexity. UWF combines marginal-utility estimation, weighted world-model training and feedback attention, a quantile-based commitment gate, read-only action conditioning, and multiple auxiliary losses. The paper does not isolate their contributions or demonstrate why this combination is preferable to a simpler feedback mechanism. The reconstruction diagnostics and action-prefix interventions are useful, but do not establish the necessity of these individual design choices for closed-loop control. Controlled task-level ablations are needed to show which components account for the gains and whether the added complexity is justified.
  2. The real-robot task selection only partly targets the claimed contact-rich setting. Three of the eight tasks are primarily pick-and-place tasks. These involve picking and placing a small object, transferring a pouch into a box, and placing objects in a basket.
  3. The real-robot evaluation setup is insufficiently specified. I could not find explicit robot-arm, gripper, or tactile-sensor models in the manuscript or appendices. Clearer hardware documentation is needed.
Questions for authors
  1. Which components are necessary to obtain the reported gains, and how does UWF compare with a simpler feedback design? Please ablate utility weighting, the commitment gate, and the full/subset auxiliary losses under matched training and execution conditions. Comparisons with uniform weights or ordinary attention, and a fixed correction scale, would help justify the method's complexity.
  2. Vision-based baselines without tactile input also achieve high success rates on the three pick-and-place tasks. I suggest replacing these tasks with more contact-rich tasks involving constrained contact or force regulation.
  3. What are the robot, gripper, and tactile-sensor models and sensor mounting locations? Please document the tactile acquisition rate, input resolution, preprocessing, and synchronization with cameras and actions.
4 comments

Response to Reviewer M62D (Part 1/3)

Thank you for the detailed review. Your main concern is the lack of controlled task-level ablations, which is also why you gave Soundness 1. To address it we retrained 27 models under matched conditions. We evaluated 9 of them in the closed-loop simulator, and we trained every ablation with 3 seeds. New results are marked (rebuttal); results already in the paper are marked (paper).

W1 / Q1: which components are necessary, and how does UWF compare with simpler feedback?

Conclusion: each component you asked about lowers task success when removed. Uniform weights, which make the feedback attention ordinary attention, cost the most: 11 points. A fixed correction scale is the strongest simpler design. It nearly matches UWF on one task but loses on all three insertion tasks.

Matched conditions.

  • Training. Every model is retrained from the same pretrained N0-TWAM bundle with the paper's recipe: same data cache, schedule, optimizer and batch size, 1000 steps. Each ablated model differs from the full model only in its named switch, which we checked by diffing the flattened configs. All runs record one identical source-code fingerprint.
  • Execution. All models use the same sampler and action horizon. Closed-loop success follows the paper's Table 2 protocol: 100 fixed evaluation seeds (200–299, disjoint from training data), with crashes and timeouts counted as failures.
  • Reproduction check. The retrained full model reproduces Table 2 exactly: 90 / 96 / 95 on Insert HDMI / Hole / Tube.

Closed-loop task success on Insert HDMI (rebuttal; training seed 1; 100 episodes each):

modelwhat is removed or replacedsuccess /100ΔFisher p
UWF (full)—90——
uniform weightsutility weighting → weight 1 for every evidence group, i.e. ordinary feedback attention and an unweighted world loss79−110.049
no gatecommitment gate → λ ≡ 1 (correction always applied)84−60.29
fixed correction scalecommitment gate → λ ≡ 0.587−30.66
per-block gateshared scale → one scale per temporal block88−20.82
no auxiliary lossesfull and subset auxiliary losses → 0cannot train with the feedback loop intact (Part 2)

The fixed scale across the insertion family (rebuttal; closed loop):

taskUWFfixed λ = 0.5Δ
Insert HDMI9087−3
Insert Hole9690−6
Insert Tube9586−9
pooled, stratified by task281/300263/300−18, p = 0.017

The closed-loop p-values reflect only evaluation noise over 100 episodes, not training-seed variance. The next part therefore adds three training seeds per model.

Response to Reviewer M62D (Part 2/3)

Three training seeds per model (rebuttal). All five Insert HDMI models were trained with 3 seeds. Seed k shares initialization and noise streams across models, so differences are paired by seed. The metric is the teacher-forced action-flow loss on the 100 training episodes; lower is better. This is an offline loss, not success.

modelΔ action loss vs full (seeds 1, 2, 3)meanseed-level p
uniform weights+6.79%, +12.93%, +2.27%+7.33%0.14
no gate+0.05%, +2.21%, +3.42%+1.89%0.20
fixed scale−1.50%, −1.68%, +3.80%+0.21%0.93
per-block gate+2.05%, +5.86%, −2.62%+1.76%0.54

Uniform weights and no gate are worse on all three seeds, consistent with the closed-loop ordering. Tactile prediction loss is also worse on all three seeds for uniform weights (+2.08%). With n = 3 the seed-level tests are not significant. We therefore claim consistent direction across two independent measurements, not established significance.

Auxiliary losses: a structural answer. With both the full and the subset auxiliary losses removed, training stops at step 3. A fail-closed check finds that no action-feedback gradient reaches the world model's predicted outputs. We traced the gradient paths to find out why:

gradient path into the residualmeasured norm
full-evidence auxiliary loss only4.95e-2, identical to the full model
subset auxiliary loss only0
both removed0

The world-model gradient from the action objective is also identical across trained models to 4 significant figures (4.60e-8), whether the applied scale is 0 (full model) or 0.46 (fixed-scale model). So this gradient flows through the auxiliary loss, not through λ·r. With the auxiliary losses off and the certified gate at 0, the world model receives no action feedback at all. The auxiliary losses are therefore the only part of the feedback loop that bypasses the gate, not optional regularizers.

For completeness we also trained this model with the check waived. It reaches action loss +2.31%. Waiving the check changes the code, so this run has a different fingerprint and we report it as a trend only.

What we conclude and what we do not.

  • Supported. Utility-weighted training improves the policy: −11 points without it, and worse on all 3 seeds offline. The gate improves on both "always correct" and "constant scale" across the insertion family.
  • Not supported. At the paper's audited 1000-step checkpoint the learned utility weights are within 1e-4 of uniform (weight spread ≤ 8.6e-5). Their ranking agrees with exact Shapley values in one of three audited windows (rank correlation +0.95) but not in the other two (−0.55 and −0.60). So we do not claim that the utility head is a reliable evidence ranker. The benefit we measure belongs to utility-weighted training as a whole.
  • Not supported. A per-block gate brings no measurable gain (88 vs 90).

Revision: a new ablation section with both tables, and the near-uniform weighting finding stated in the limitations.

Response to Reviewer M62D (Part 3/3)

W2 / Q2: three of the eight real-robot tasks are pick-and-place

Conclusion: we agree, and we will not present those three tasks as evidence for the contact-rich claim. We cannot run new real-robot tasks during the discussion period.

The three tasks are Pick & place, Transfer pouch and Fill basket. Here is the real-robot table (paper) split by task type; each task has 20 trials per method:

task groupUWFN0-TWAMbest other baseline
5 contact-rich tasks (remove cap, turn vise, close box, reinsert dropper, erase board)86/10074/10068/100 (G0.5)
3 pick-and-place tasks60/6053/6056/60 (G0.5)

On the five contact-rich tasks alone, UWF still leads N0-TWAM by 12 trials. A Cochran–Mantel–Haenszel test stratified by task gives p = 0.038; this is our computation on the published counts. As you note, the vision-only baselines are close to ceiling on the three pick-and-place tasks, so those tasks add little discriminative evidence.

Within the discussion period we can strengthen the contact-rich evidence only in simulation, where all three insertion tasks are now covered by the ablations above.

Revision:

  • Report the contact-rich subset separately in the real-robot table.
  • Narrow the contact-rich claim to those five tasks.
  • Label the three pick-and-place tasks as such.

W3 / Q3: hardware documentation

Conclusion: we will add a hardware subsection. The simulation sensor configuration is below; the real-robot specification will be in the revision.

  • Simulation (UniVTAC), checked from the data files.
    • Sensors: 2 RGB cameras (head and wrist) and 2 GelSight Mini tactile sensors (gsmini).
    • Timing: 24 action slots per chunk, with visual and tactile keyframes every 3 slots.
  • Real robot, execution protocol.
    • UWF uses the same deployment configuration as N0-TWAM.
    • Chunks have 12 action slots per frame; after the first, cold-start chunk, each chunk executes 24 slots. The model uses 3 world and 4 action denoising steps.
    • Real visual and tactile observations are captured every 3 action slots, i.e. 4 keyframes per 12-slot frame.
    • When a chunk completes, its observed keyframes are committed to the model's history; an aborted chunk commits nothing.

The revision will add a hardware table covering:

  • the arm and gripper models;
  • the tactile sensor model and mounting location;
  • the camera models and placement;
  • the tactile acquisition rate, input resolution and preprocessing;
  • how cameras, tactile sensors and actions are synchronized.

Revision: a new appendix subsection on hardware and data acquisition.

Reviewer_M62DOpenReview ↗

Concerns Remain

Thank you for the additional experiments and clarifications. My main concerns remain. The component ablation is limited to one task, and the gains over simpler variants are marginal. I do not find this evidence sufficient to justify the method’s added complexity. No new real-robot tasks were added either. I encourage the authors to report more real contact-rich experiments and a more comprehensive set of ablation results to strengthen the empirical support for the method. I will therefore maintain my rating.

Reviewer_EiB8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

UWF is a world-action model (WAM) that aims to solve the disjoint predictions between world forecasting accuracy and predicted action optimality. The authors propose a framework that evaluates predicted visuo-tactile changes against a shared supervised action target. During training, it measures whether specific visual or tactile changes reduce action error against ground-truth values. It regulates their adoption through risk estimates applied to corrections on the action chunk. Results shown in sim and real.

Critical strengths
  1. Grounded in demonstrations, without reward engineering or annotation

  2. Selective spatiotemporal modulation: Rather than treating visuotactile info as a single stream, UWF evaluates the marginal utility of 8 sensor-time evidence groups (2 cameras, 2 tactile sensors, 2 timesteps)

Critical weaknesses
  1. Analysis does not apply to core components of UWF. It’s missing some ablations isolating individual performance impact of each component (e.g. risk gating, utility weighting)

  2. Baseline comparisons are made with published numbers from the sources. Does this mean that the authors have not run the baseline themselves? See table 2. Also I see text that says “Published scores are not our matched reruns”. Hope the authors can clarify this.

Questions for authors

1 It would seem like all tasks in a family of tasks (e.g. insertion) should exhibit similar results. Why do some insertion tasks show improvement over N0-TWAM whereas some do not? Is the risk gating factor λ\lambda too conservative? Does it prematurely reject valid residual corrections?

  1. gating factor λ\lambda operates as an all-or-nothing scaling factor for the whole chunk, is that correct?
3 comments

Response to Reviewer EiB8 (Part 1/3)

Thank you for the careful review. You asked four things: component ablations, why the insertion tasks differ, whether λ is all-or-nothing, and where the baseline numbers come from. For the first three we ran new matched-condition experiments: 27 retrained models, 9 of which we also evaluated in the closed-loop simulator. New results are marked (rebuttal); results already in the paper are marked (paper).

How the new experiments were run.

  • Every model is retrained from the same pretrained N0-TWAM bundle with the paper's recipe: same data cache, schedule and optimizer, 1000 steps.
  • Each ablated model differs from the full model only in its named switch. We checked this by diffing the flattened configs. All runs record one identical source-code fingerprint.
  • Closed-loop success follows the paper's Table 2 protocol: 100 fixed evaluation seeds (200–299, disjoint from training data), with crashes and timeouts counted as failures.
  • The retrained full model reproduces the paper's Table 2 scores exactly: 90 / 96 / 95 on Insert HDMI / Hole / Tube.

W1: ablations isolating risk gating and utility weighting

Conclusion: removing either component lowers closed-loop success. Removing utility weighting costs the most.

Closed-loop success on Insert HDMI, training seed 1, 100 evaluation episodes per model (rebuttal):

modelwhat changessuccess /100ΔFisher p
UWF (full)—90——
uniform weightsall utility weights set to 1 (ordinary feedback attention, unweighted world loss)79−110.049
no gateλ ≡ 1: correction always applied in full84−60.29
fixed scaleλ ≡ 0.5: risk estimate ignored87−30.66
per-block gateone λ per temporal block88−20.82

These closed-loop p-values reflect only evaluation noise over 100 episodes, not training-seed variance. To check that the ordering is not specific to one training seed, we trained every model with 3 seeds. For each run we measured the teacher-forced action-flow loss on the 100 training episodes, and paired the runs by seed (lower is better) (rebuttal):

modelΔ action loss vs full (seed 1, 2, 3)meanseed-level p
uniform weights+6.79%, +12.93%, +2.27%+7.33%0.14
no gate+0.05%, +2.21%, +3.42%+1.89%0.20
fixed scale−1.50%, −1.68%, +3.80%+0.21%0.93
per-block gate+2.05%, +5.86%, −2.62%+1.76%0.54

Removing utility weighting or the gate makes the loss worse on all three seeds, which matches the closed-loop ordering. With three seeds the seed-level test is not significant, so we present these as consistent effects in two independent measurements, not as established ones.

The gate also beats a constant scale across the insertion family. Forcing λ = 0.5 lowers success on all three insertion tasks: 263/300 vs 281/300 (stratified p = 0.017). The details are in Q1.

Revision: a component-ablation table (closed loop plus 3-seed offline loss) in the experiments section.

Response to Reviewer EiB8 (Part 2/3)

Q1: why do some insertion tasks improve over N0-TWAM and others not? Is λ too conservative?

Conclusion: λ is not prematurely rejecting valid corrections: forcing a correction in lowers success on every insertion task. The two tasks that do not improve are the ones N0-TWAM already solves at 99% and 98%, and their −3 is within evaluation noise.

1. The Hole and Tube differences are within evaluation noise. These are Table 2 scores (paper); the test is ours. Each task has 100 evaluation episodes:

taskN0-TWAMUWFΔFisher p
Insert HDMI6890+220.0002
Insert Hole9996−30.37
Insert Tube9895−30.45

On Hole and Tube the base policy leaves only 1–2 points of headroom, and the −3 cannot be distinguished from zero. HDMI is the task where the base policy fails often, and there UWF's gain is large and significant. We do not claim that UWF improves Hole or Tube.

2. The gate is not too conservative (rebuttal). On each insertion task we trained two models: the full model, and a fixed-scale model that applies λ = 0.5 on every valid block regardless of the risk estimate. We evaluated both in closed loop (training seed 1):

taskcertified gatefixed λ = 0.5ΔFisher p
Insert HDMI9087−30.66
Insert Hole9690−60.16
Insert Tube9586−90.051
pooled, stratified by task281/300263/300−180.017

If the gate were rejecting valid corrections, applying them should help. Instead it lowers success on every task, and most on the two near-ceiling tasks. Removing the gate entirely (λ ≡ 1) also lowers HDMI success, from 90 to 84 (Part 1).

3. What the gate actually does.

  • λ is always 0. On all nine retrained full models (3 tasks × 3 seeds) the certified λ is 0 on the training episodes. This matches the paper's gate audit, which recorded 18 gate events, all with λ = 0.
  • This is not a stuck head. The risk head's lower bound is calibrated: its coverage is 0.895–0.917 against the 0.9 target on all nine runs.
  • The residual's benefit straddles zero. The full residual reduces action error on only 39–53% of training episodes. A calibrated 10th-percentile bound on such a distribution is negative, so the worst-block term clips λ to 0.
  • Negative control. Fitting the same pinball loss to a reliably positive benefit gives a positive bound (coverage 0.902), and the gate then admits the correction.

So on these tasks the gate's measurable role is to block a correction that does not reliably help. The fixed-scale and no-gate experiments confirm that role in closed loop.

4. Where the HDMI gain comes from, given λ = 0. With λ = 0 the executed action is the base velocity v₀ of the UWF-trained model, not of N0-TWAM. The shared backbone is trained jointly with the utility-weighted world objective and the auxiliary losses. The uniform-weight ablation (90 → 79, Part 1) shows that this training matters.

The gate behaves the same on the improving and the non-improving tasks, so it does not explain the per-task pattern. Beyond the ceiling effect above, we cannot identify a further cause from these experiments, and we do not offer one.

Revision: a per-task analysis paragraph that reports the noise levels and the λ = 0 finding.

Response to Reviewer EiB8 (Part 3/3)

Q2: does λ act as an all-or-nothing scale for the whole chunk?

Conclusion: yes. One λ is computed per action chunk, and a single unfavorable temporal block caps it. We measured what this costs: the cost is real but small.

The deployed rule is

λ = clip[0,1]( min( Σ_b n_b q_b / (2 Σ_b n_b E_b), min_b q_b / E_b ) ),

where q_b is the predicted lower-bound benefit of block b and E_b its residual energy, with 4 blocks per chunk. The second term is a worst-block constraint, so one block with a negative predicted benefit sets λ to 0 for the whole chunk. λ itself is continuous in [0, 1], not binary. It is all-or-nothing in the sense that one value applies uniformly across the chunk.

Exact recomputation (rebuttal). We used the recorded residuals for all 256 evidence subsets at three training windows. The gate identity err(λ) − err(0) = λ²E − λB holds to 2.3e-18, so these are exact recomputations. In each window only 1–2 of the 4 blocks have positive benefit. A per-block λ beats the shared scale in every window, by 0.0008%, 0.016% and 0.12% of action loss.

Retrained per-block gate (rebuttal). We retrained UWF with one λ per block.

  • Closed loop: 88/100, vs 90/100 with the shared scale (p = 0.82).
  • Over 3 training seeds: action loss +1.76% (p = 0.54).

The per-block rule recovers what the shared rule rejects, but at this training length that amount is too small to change task success.

Revision: the shared-scale restriction and its measured cost are stated as a limitation in the method section, with the recomputation in the appendix.

W2: did the authors run the baselines themselves?

Conclusion: Table 2 mixes published scores with our own evaluations. We will mark every row.

  • 12 published rows. Of the 15 baseline rows in Table 2, 12 are scores published by the original authors, copied with their rounding. N0-TWAM is one of them. The sentence you quote ("Published scores are not our matched reruns") refers to these 12 rows: their evaluation protocols are not verified to be identical to ours, which is why we do not pool them.
  • 3 rows evaluated by us: OpenWAM-α, Xiaomi-Robotics-1 and GalaxeaVLA G0.5.
  • Appendix tables follow the same rule, with a letter marking the source of each score.

Every new number in this rebuttal comes from our own matched runs under one protocol. The retrained full model reproduces Table 2's UWF scores on all three insertion tasks (90 / 96 / 95).

Revision: a per-row provenance marker in Table 2 (published / evaluated by us), with the protocol caveat in the caption.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.