- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
WAM with high LIBERO success often fail to recover after a grasped object is forcibly dropped, and that an auxiliary box-regression head keeps predicting the object near the gripper after the drop. It proposes Grounding Attention Distillation, a thresholded loss that forces action queries in selected heads to put a minimum attention mass on simulator-derived target patches and instruction tokens, and reports gains on LIBERO, LIBERO-Plus, and a custom object-release benchmark.
Critical strengths
Asking whether high nominal success hides poor recovery is a useful question. The object-release probe cleanly separates the object–gripper relationship from task identity.
The release protocol is specified in unusual detail, including trigger geometry, drop and regrasp criteria, denominators, and paired initial states and seeds.
The method adds no inference-time cost or dependencies. Annotations are used only during training, and the deployed policy keeps the standard observation-to-action interface.
Critical weaknesses
Author need to test on a real-world machine to test the performance.
It have shortcut learning problem. Both signals come from the same simulator boxes on the same correlated demonstration distribution, so nothing principled prevents learned attention from collapsing toward the gripper after a drop, just as the box head does. The most direct test of the hypothesis is missing: the full model has a box head, yet its post-drop IoU is never reported, and attention maps are shown for only two hand-picked episodes.
The evidence that GAD improves “grounding” in general is weak. The large gain appears only on Spatial (48.5 to 90.3), while Object improves just 39.5 to 44.0, which points to a suite-specific effect rather than better relocalization. On LIBERO-Plus the full-benchmark gain over GAM is 1.6 points, Camera is worse than ImageWAM, and the ablation differences (87.45 vs 88.80) come from single-seed, single-rollout runs with no confidence intervals. Baseline numbers also mix published results with reproduced ones under each model’s own inference settings.
Supervising attention toward annotated evidence is well established (“Right for the Right Reasons”, GuidedVLA, OREO), and the paper adds a hinge threshold applied to arbitrarily chosen layers and heads (5×6, τ = 0.4/0.1) with no sensitivity analysis.
Questions for authors
See Weakness
2 comments
Response to Reviewer DUcT (2)
A4 (Weakness 3): Recovery Gains across Suites
Thank you for highlighting the differences across suites. We further decomposed recovery into regrasp and subsequent task completion. The following rates condition on an eligible confirmed drop, excluding episodes that completed the task during the forced-open interval.
| Suite | ImageWAM regrasp rate | GroundingWAM regrasp rate | Gain | ImageWAM regrasp-and-completion rate | GroundingWAM regrasp-and-completion rate | Gain |
|---|---|---|---|---|---|---|
| Spatial | 61.00% | 88.77% | +27.76 pp | 40.39% | 83.29% | +42.90 pp |
| Object | 39.89% | 43.67% | +3.77 pp | 27.13% | 32.08% | +4.95 pp |
| Goal | 24.22% | 38.13% | +13.90 pp | 20.50% | 33.75% | +13.25 pp |
| Long | 50.18% | 56.60% | +6.42 pp | 31.79% | 47.92% | +16.13 pp |
| Overall | 47.32% | 59.35% | +12.03 pp | 31.46% | 51.21% | +19.75 pp |
Regrasp improved in all four suites, and Spatial exhibits the largest gain.
A drop changes both the object's location and its grasping pose. Grounding supervision may help preserve the use of target-object information when its spatial relationship to the gripper and background becomes unfamiliar. All Spatial tasks involve a black bowl, and the demonstrations include grasps from multiple sides. When the bowl lands upright, its approximately rotationally symmetric rim supports similar grasps across different yaw angles. For a rectangular object, falling onto a different face can require a wrist orientation and finger placement that are poorly represented in training. Accurate localization may therefore be insufficient to recover a successful grasp. This provides a possible explanation for the uneven gains across suites.
A5 (Weakness 3): LIBERO-Plus Results and Comparison Protocols
Thank you for raising these evaluation questions. On all 10,030 LIBERO-Plus scenarios, GroundingWAM achieved 87.08%, compared with published results of 85.5% for GAM and 83.1% for ImageWAM. We acknowledge that Camera performance was lower than ImageWAM's published result, at 76.99% versus 80.8%.
On the shared 2,000-scenario component-evaluation subset, ImageWAM and GroundingWAM achieved 83.80% and 88.80%, respectively. The largest improvements occurred under changes to the robot's initial state, from 52.10% to 86.01%, and object layout, from 77.89% to 86.67%. Sensor-noise performance decreased from 96.49% to 87.72%, and background performance decreased from 91.96% to 90.56%. These results identify the conditions under which the method provides its strongest benefits. The gains under robot-state and object-layout changes are consistent with the motivation behind our action-to-vision attention guidance, which encourages action queries to attend to task-relevant visual evidence when familiar spatial relationships change.
Regarding result provenance, all external baseline scores in Table 1 come from the respective publications. For the release benchmark in Tables 2–3, we evaluated the policies under the same release and scoring protocol while retaining their released inference settings.
A6 (Weakness 4): Relationship to Prior Attention-Guidance Methods
Thank you for highlighting the related work. Although prior methods employ similar attention-guidance designs, our work has a distinct motivation and evaluation focus. Our central question is whether grounding supervision improves recovery when a WAM departs from the demonstrated trajectory.
Our investigation began with post-drop recovery failures, which suggested that the representations used for action prediction might inadequately encode task-relevant evidence. We first introduced auxiliary box supervision to encourage these representations to encode object location. This improved recovery, but visualizations showed that box predictions could still remain near the gripper after the object had fallen. This observation motivated GAD, which directly supervises intermediate attention to task-relevant image regions and instruction tokens.
The resulting supervision provides benefits complementary to box prediction without adding recovery demonstrations. On the Spatial release benchmark, task success was 48.50% for ImageWAM, 73.00% with box supervision, 81.00% with guidance alone, and 90.25% with both.
Our contribution centers on the recovery problem, the localization diagnostics motivating the design, and the demonstrated benefits of grounding supervision in WAMs. The thresholded attention-mass objective is the implementation used to encourage sufficient attention to annotated evidence; we do not claim the mass objective itself as a standalone contribution.
Response to Reviewer DUcT (1)
A1 (Weakness 1): Real-Robot Experiments
Thank you for suggesting real-robot experiments. We conducted experiments using a Franka R3 arm with two cameras and demonstrations of a two-object pick-and-place setting. Grounding DINO and SAM 2.1 provided front-camera boxes and masks. We screened the target-object and basket annotations and trained both policies on the same retained demonstrations. The deployed policies required no detector.
ImageWAM and GroundingWAM were trained for 5,000 steps from the same pretrained FLUX weights. After the initial grasp and a further 48 control steps (2.4 seconds), we paused inference, removed and relocated the target object, and resumed the policy. Success required regrasping the relocated target and placing it in the basket.
The second protocol used a fixed starting arrangement from which both policies could grasp the target. Both policies were evaluated at the same nine relocation positions, with three trials per position. In four of its 23 failed randomized-placement trials, ImageWAM attempted to grasp the distractor. These experiments demonstrate recovery improvements on physical hardware with externally generated grounding annotations.
A2 (Weakness 2): Post-Drop Localization of the Full Model
Thank you for pointing out the missing localization comparison. We evaluated the built-in box heads of Box-only and full GroundingWAM on the same post-drop observations across all ten LIBERO-Spatial tasks. Both models received identical camera views, robot states, and task instructions, controlling for differences in their rollout trajectories.
GroundingWAM improved task-averaged target-box IoU from 0.566 to 0.724, with improvements on nine of ten tasks. In the 29 frames where the projected gripper-to-object distance was at least one ground-truth box diagonal, frame-averaged IoU improved from 0.071 to 0.569. These results support improved localization when the gripper's position becomes an unreliable cue to the object's location.
We also quantified attention across 37 perturbation tasks. In the external-camera view, the dropped object occupied an average of 1.06% of the image while receiving 16.20% of the attention in the supervised heads. The median frame-level attention enrichment was 17.28× relative to uniform allocation. These measurements extend the evidence beyond the selected visualizations and support increased attention to the dropped target.
A3 (Weakness 2): Interpretation of Grounding Attention and Shortcut Learning
Thank you for raising this point. We agree that supervised attention maps and improved task success alone do not establish that a policy has stopped exploiting object–gripper correlations. We view GAD as a representation-learning objective that shapes the intermediate representations used for action prediction through grounding supervision.
We additionally evaluated both policies from identical post-drop states. For each state, we replayed the same control prefix and verified that the simulator state, proprioception, and both camera images matched before handing control to each policy. Without attention intervention, ImageWAM completed the task in 31/109 cases (28.4%), compared with 60/109 (55.0%) for GroundingWAM, a gain of 26.6 percentage points. This comparison controls for differences in the states reached before recovery.
We also transferred 90% of the attention mass assigned to the dropped object to background patches in the six supervised heads of each supervised layer. The modified attention probabilities were used to weight the value vectors during action computation. GroundingWAM's task success changed from 55.0% to 56.0%, showing little sensitivity to this intervention.
Training-time gradients can improve the representations used for action prediction even when perturbing the supervised attention heads at inference produces little change in task success. The experiments support improved post-drop localization and recovery, while the precise mechanism through which the learned representations improve recovery remains open.