acceptodds
Under review as a conference paper at ICLR 2027

GF-VLA: Dual Geometry Forcing for Vision-Language-Action Models

Abstract

Vision-language-action (VLA) models generate robot actions from visual observations and language instructions, but recognizing a manipulation goal does not determine how the end effector should move through the surrounding scene. We introduce **Geometry Forcing for VLA (GF-VLA)**, which brings both motion and scene geometry together on a shared action-conditioning pathway. Motion geometry forcing supervises time-indexed motion latents with positions and successive displacements from a sparse, temporally ordered 3D end-effector (EEF) trace within an action chunk. Scene geometry forcing reconstructs future depth at the chunk endpoint, guiding the scene latents read by motion queries. These motion latents, together with native vision-language features, condition a flow-matching Action Expert, supporting embodied reasoning that jointly considers 3D motion and future scene geometry during action generation. GF-VLA achieves 98.40% mean success on LIBERO and 83.8% overall zero-shot success on LIBERO-Plus; on RoboCasa-GR1, it improves success by 6.3 pp over a strong VLA baseline. Across three real-world tasks, GF-VLA achieves 73.3% average success under standard conditions, exceeding the baseline by 12.2 pp. These results support the benefit of coupling 3D motion trajectories with scene geometry at the action-chunk endpoint for action generation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Embodied Reasoning, 3D Motion Geometry, Future Scene Geometry

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ruZMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GF-VLA learns scene latents supervised by future depth and motion latents supervised by sparse, temporally ordered UVD end-effector traces. Motion queries attend to scene latents, and the resulting features condition a flow-matching action expert. Geometry decoders are training-only. The evaluation covers LIBERO, LIBERO-Plus, RoboCasa-GR1 and real manipulation on two platforms, with controls for geometry dimensionality and supervision targets.

Critical strengths
  1. Combining endpoint scene geometry with within-chunk motion geometry is a clear spatial-control motivation. Training-only decoders preserve action-only deployment, and applying the representation to both grippers and dexterous hands gives the method practical breadth.
  2. The evaluation includes meaningful 2D/3D motion, current/future depth, and single-branch controls. On RoboCasa the joint model reaches 71.9% versus the base VLA's 65.6%; real-robot easy/hard improvements also reproduce from the stated counts. The appendix retains task-level regressions, making the empirical picture more informative than aggregate gains alone.
Critical weaknesses
  1. The evidence for dual coupling specifically is limited. Table 5 reports 70.8% with scene supervision alone and 71.9% jointly, so the marginal joint gain is 1.1 points out of the 6.3-point total. Without independent-training uncertainty or an attention control retaining both losses, most of the evidence supports future-scene supervision rather than the necessity of the coupling mechanism.
  2. The geometric diagnostics measure agreement between predicted and executed motion, not accuracy against desired motion or successful task completion; a consistently wrong policy can still agree with its own predictions. The calibrated UVD/depth target-generation pipeline is also insufficiently specified for direct reproduction from the supplied materials.
Questions for authors
  1. Decision-relevant mechanism check: is the 1.1-point joint-over-scene gain stable across available runs, and what happens when scene-to-motion attention is removed while retaining both geometry losses and a comparable parameter budget? If this control is unavailable, please narrow the dual-coupling claim.
  2. Diagnostic/reproducibility clarification: please distinguish prediction-execution consistency from goal- or demonstration-referenced accuracy, including on failed rollouts, and specify the calibration/target-generation pipeline. Which LIBERO and LIBERO-Plus comparisons use the same underlying base VLA and training budget?
5 comments
AuthorsOpenReview ↗

Response (Part1)

We thank the reviewer for recognizing the spatial-control motivation for combining endpoint scene geometry with within-chunk motion geometry, the action-only deployment and applicability to both grippers and dexterous hands, and the informative controls and transparent reporting of task-level results. Below, we address these points, clarify the purpose of the geometric diagnostics, and specify the target-generation and benchmark-comparison settings.

R1. Response to W1 and Q1: Incremental benefits of joint supervision and evidence for scene-to-motion coupling

[Method motivation]

Motion forcing explicitly supervises the EEF's 3D trajectory throughout an action chunk, encouraging the policy to capture where the EEF should move and how its movement unfolds over time. The resulting motion latents condition the Action Expert. Scene forcing supplies complementary spatial context by supervising scene depth at the endpoint of the same chunk, and motion queries read these scene latents before conditioning action generation.

We examine the contribution of motion forcing across evaluation seeds and task conditions, and separately test the directed pathway while retaining both geometry losses.

[Additional evaluations and task-dependent contributions]

  • Four-way ablation across evaluation seeds

Table 5 evaluates each trained model on 24 RoboCasa-GR1 tasks with 50 rollouts per task, totaling 1,200 episodes per model. We extend this comparison to three evaluation seeds. The variants use the same backbone, demonstration data, observation inputs, and training setup. Each variant keeps its trained checkpoint fixed across evaluations.

ModelSeed 7Seed 27Seed 1137Mean ± Std
Base65.665.667.666.3 ± 1.2
Motion-only66.866.470.267.8 ± 2.1
Scene-only70.870.373.371.4 ± 1.7
Full71.972.675.273.2 ± 1.7

Results across three evaluation seeds.

Full exceeds Scene-only in all three evaluations, with a reported mean gain of 1.8 pp, and achieves the highest success rate among the four variants in each evaluation. These results show that the joint-over-scene gain recurs beyond the original evaluation.

The branch ablations remove the corresponding branch rather than only disabling its loss. Scene-only removes the motion queries, UVD prediction head, motion loss, and motion-to-Action-Expert connections. Its Action Expert uses native vision-language features without directly reading scene latents. Motion-only removes the scene queries and depth supervision while retaining motion supervision and motion-to-Action-Expert conditioning. The attention control below addresses the pathway separately.

  • RoboCasa task categories

Scene supervision already provides useful geometry for action learning. We believe that the incremental benefit of motion forcing becomes more apparent when tasks require more precise movement and stronger spatial coordination. We therefore separate RoboCasa's two task categories. In-container placement and closure requires placing an object within a constrained space and subsequently closing an articulated container, whereas the other category involves object pick-and-place.

ModelPlacement + closurePick-and-place
Base54.6 ± 2.470.1 ± 1.1
Motion-only54.6 ± 4.172.2 ± 1.5
Scene-only60.8 ± 4.275.0 ± 0.9
Full66.0 ± 2.675.6 ± 1.5
Full − Base+11.4 pp+5.5 pp
Full − Scene-only+5.2 pp+0.6 pp

Results across three evaluation seeds. Placement + closure denotes in-container placement followed by closure.

Adding motion forcing to Scene-only yields 5.2 pp on placement and closure, compared with 0.6 pp on pick-and-place. This supports its usefulness when successive EEF movements must remain coordinated with changing relationships among the hand, object, and container.

AuthorsOpenReview ↗

Response (Part2)

  • Real-world branch ablations

We further examine Bricks Disassembly and Hanger Placement, evaluating each variant in 30 trials per task and condition. Easy conditions resemble demonstration collection. Hard conditions introduce distractor bricks and unseen hook positions, respectively, increasing the demands on robust execution and spatial generalization.

TaskConditionBaseScene-onlyMotion-onlyFull
Bricks DisassemblyEasy53.360.056.763.3
Bricks DisassemblyHard36.736.746.750.0
Hanger PlacementEasy50.056.756.763.3
Hanger PlacementHard23.326.730.033.3
Two-task averageEasy51.758.356.763.3
Two-task averageHard30.031.738.341.7

The averages weight the two tasks equally.

Removing the motion branch from Full reduces success by 10.0 pp under Hard conditions, compared with 5.0 pp under Easy conditions. Under Hard conditions, Motion-only exceeds Base by 8.3 pp, while Scene-only exceeds Base by 1.7 pp. Full remains best, with a further 3.3 pp gain over Motion-only. These comparisons support complementary contributions from the two branches, with a larger incremental motion benefit under the evaluated conditions that place greater demands on execution and generalization.

[Attention control retaining both geometry losses]

To isolate the coupling mechanism, we trained a control that retains both query sets, both geometric prediction heads and losses, and motion-to-Action-Expert conditioning, while removing only motion queries' access to scene latents. Both branches continue to read vision-language features. This preserves the parameterized modules and query counts, changing the attention access rather than removing a branch or adding parameters. We evaluate this control on RoboCasa's in-container placement and closure tasks.

ModelSeed 7Seed 27Seed 1137Mean ± Std
Full64.764.369.066.0 ± 2.6
Without scene-to-motion access62.760.765.062.8 ± 2.2

Results across three evaluation seeds on in-container placement and closure tasks.

Removing the pathway lowers success in every reported evaluation, with a mean decrease of 3.2 pp. This provides evidence for the directed scene-to-motion connection beyond the presence of both supervision losses.

R2. Response to W2 and Q2: Interpreting geometric consistency together with task success

SR measures goal completion, while UV and depth errors measure prediction–execution consistency. Motion latents represent future EEF movement and condition the Action Expert. The consistency diagnostic examines whether the decoded motion agrees with the movement actually produced by the resulting actions. We agree that an incorrect motion can still be executed consistently, so low geometric error alone does not establish task success.

  • Why not directly compare with a demonstration?

Demonstrations provide training supervision, but online rollouts can visit different states, and multiple trajectories can complete the same task. Direct comparison with one demonstration can therefore mix motion error with differences in state, execution timing, and valid trajectory choice. We instead compare predicted and executed EEF reference points at matched time offsets in the same camera frame. UV error measures their mean absolute image-coordinate difference in pixels, and depth error measures their absolute optical-axis difference in millimeters.

  • Overall results and successful versus failed rollouts

We report SR alongside UV and depth consistency, followed by separate errors for successful and failed rollouts. All evaluations use the same trained GF-VLA checkpoint.

Evaluation seedOverall SR (%) ↑UV error (px) ↓Depth error (mm) ↓
771.93.368.09
2772.63.297.84
113775.23.257.74
Evaluation seedSuccess UV (px)Failure UV (px)Success depth (mm)Failure depth (mm)
73.303.387.718.65
273.273.267.658.17
11373.243.237.598.11

Successful and failed rollouts show similar UV errors, while successful rollouts have lower depth errors across all three evaluation seeds. These results illustrate that consistency and task outcome capture different aspects of policy behavior. We therefore analyze SR together with UV and depth consistency to assess both task completion and the agreement between predicted and executed motion. This joint interpretation provides a more complete view of policy performance than either success rate or geometric consistency alone.

AuthorsOpenReview ↗

Response (Part3)

R3. Response to W2 and Q2: Calibration coordinates and construction of UVD and depth targets

Scene targets use future depth observations at the action-chunk endpoint, while motion targets express temporally ordered EEF reference points within the same chunk in the designated camera's UVD coordinates. Appendix A.1, Geometric Target Preparation, specifies the simulation replay, reference-point definitions, projection convention, and real-world depth sources. We summarize the construction here.

  • Simulation

The released LIBERO and RoboCasa-GR1 training data do not directly provide all depth maps and camera parameters needed for our geometry targets. We restore demonstration episodes from the corresponding HDF5 simulator records, replay the recorded states, and rerender camera views while retaining the original actions. This provides frame-aligned metric depth, camera parameters, and EEF reference-point positions. LIBERO uses the gripper center; RoboCasa-GR1 uses the midpoint between the thumb and index-finger positions of each hand.

  • Projection and temporal alignment

The reference point is transformed from its recorded world or robot frame into the designated RGB camera frame using the calibrated extrinsics, then projected using intrinsics adjusted for image preprocessing. U and V are image coordinates; D is camera-frame depth along the optical axis. For training, U and V are divided by image width minus one and image height minus one, respectively, and D remains in meters. Motion traces sample temporally ordered positions over the action chunk, including its start and endpoint. Scene targets use depth at that same endpoint. LIBERO supplies scene-depth supervision from the agent and wrist views; RoboCasa-GR1 uses the main view.

  • ARX-Lift2S

RealSense D405 cameras provide depth with depth-to-RGB alignment enabled. Motion targets are constructed from recorded EEF poses and the center of each gripper. Both gripper trajectories are projected into the central primary RGB camera, which defines their UVD coordinates. Scene supervision uses measured depth from that camera at the chunk endpoint.

  • RM75

The external camera is a RealSense L515 and the wrist camera is a D435, also with depth-to-RGB alignment enabled. Motion targets use the recorded EEF poses and gripper-center reference point, projected into the fixed external RGB camera. Scene supervision uses measured endpoint depth from both external and wrist cameras. The wrist view adds scene supervision without changing the motion-coordinate frame.

The motion and scene targets are associated with the same demonstration action interval and serve as training supervision. Future depth and recorded EEF poses are not supplied as inputs to the deployed policy, and the geometry decoders are training-only.

R4. Response to Q2: LIBERO-to-LIBERO-Plus transfer and consistent score aggregation

For GF-VLA across the two benchmarks, the model and training budget are unchanged. We train on the original LIBERO demonstrations and directly evaluate the same checkpoint on all 10,030 LIBERO-Plus instances, without further training or adaptation. GF-VLA uses the Qwen3.5-4B backbone and flow-based Action Expert. LIBERO-Plus introduces different evaluation conditions without an additional training stage.

All methods in our LIBERO-Plus comparison follow the same zero-shot transfer protocol. Their policies are trained on LIBERO data and then evaluated on LIBERO-Plus without additional training on LIBERO-Plus. This evaluates how LIBERO-trained policies generalize to the benchmark's perturbed conditions.

We have also unified the aggregation of LIBERO-Plus scores. Because its seven evaluation categories contain different numbers of instances, we compute Overall SR by weighting each category's success rate by its instance count. The source-reported Overall scores for VLA-JEPA and PILOT correspond to the simple, unweighted average of their seven category scores. We retain their published category scores and recompute Overall using the same instance-weighted rule, updating VLA-JEPA from 79.5% to 78.0% and PILOT from 81.0% to 80.3%. These updates reaggregate the published results and do not involve new model evaluations. Appendix B now provides the result sources, evaluation settings, category instance counts, and details of these aggregation updates.

Reviewer_mgK1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GF-VLA, which augments a VLM-based VLA with two training-only geometric auxiliary supervisions: (i) motion geometry forcing, where time-indexed motion latents are supervised to decode a sparse, temporally ordered 3D end-effector trace (image coordinates plus camera depth) spanning the action chunk; and (ii) scene geometry forcing, where scene latents are supervised to reconstruct depth at the endpoint of the same action chunk. Motion queries causally attend to the depth-supervised scene latents, and the resulting latents condition a flow-matching action expert; both geometric decoders are discarded at inference. The method is evaluated on three simulation benchmarks and three real-world tasks, with ablations isolating each geometric target.

Critical strengths
  • Chunk-aligned dual geometric targets with zero inference overhead. Both auxiliary targets are aligned to the same action chunk (trace over the chunk, depth at its endpoint) and coupled by letting time-indexed motion queries read depth-supervised scene latents, while the decoders are used only in training, so deployment cost is essentially unchanged. This is a clean way to shape representations without extra inference-time modules, and the qualitative visualizations support that the latents track scene and motion changes.

  • Broad, largely well-controlled evaluation with honest reporting. The evaluation spans three simulators and two real robot platforms, with the strongest real-world comparison made against a backbone-matched baseline trained on the same demonstrations. The appendix reports per-task results including cases where the full model regresses, which increases the credibility of the aggregate numbers.

Critical weaknesses
  • Margins are within evaluation noise, and the "motion" half of the contribution lacks statistical support. All simulation evaluations use a single seed with no confidence intervals, while the headline margins over external SOTA (and the joint-over-scene-only ablation gap of ~1 pp on ~1,200 trials) are on the order of one standard deviation of binomial noise. Motion forcing alone—the paper's novel component—contributes only ~1 pp over the base model, so the central "dual geometry" claim rests on differences that multi-seed evaluation could plausibly erase.

  • Unclear baseline comparability and unexplained anomalies. It is not stated whether the external baseline numbers were reproduced under matched data, backbones, and training budgets, and the paper's own base model is well below the strongest published baselines, so the improvement over it may partly reflect a weak starting point rather than the proposed mechanism. There are also anomalies that contradict the narrative: future-depth-only clearly beats current+future depth in the ablation, and the model underperforms several non-geometry baselines under the camera perturbation, yet the text presents both favorably without analysis.

Questions for authors
  1. Were the external baseline results in the main tables reproduced under matched data/protocols, or cited from the original papers?
  2. How are the real-world scene-depth supervision targets acquired (depth sensor, stereo, or calibrated reprojection)? Appendix A.1 only details the simulation replay procedure.
  3. Can you report mean ± std over multiple evaluation seeds for the key ablation (scene-only vs. joint), which directly determines whether W1 holds?
  4. Why does adding current-depth supervision hurt relative to future-only, and why does the full model regress on several individual tasks relative to the base model?
  5. Do the baseline and GF-VLA both receive proprioceptive/EEF state as input? If so, what does latent trace supervision add beyond explicit state conditioning?
6 comments
AuthorsOpenReview ↗

Response (Part1)

We thank the reviewer for recognizing the chunk-aligned dual-geometric design, the use of geometric decoders only during training, and the breadth of our evaluation and controlled real-world comparisons. We also appreciate the positive assessment of our transparent reporting of task-level regressions. Below, we respond to the concerns and questions in detail.

Response to W1, Q3: Reliability of the gains and contribution of motion forcing

[Method motivation and evaluation setting]

Motion forcing explicitly supervises the EEF’s 3D trajectory throughout an action chunk, encouraging the policy to capture where the EEF should move and how its movement unfolds over time. The resulting motion latents condition the Action Expert, providing geometric guidance for action generation beyond action supervision alone. Scene forcing supports this process by supplying scene geometry that helps the motion branch learn future movement in its spatial context.

Table 5 evaluates each trained model over 24 RoboCasa-GR1 tasks with 50 rollouts per task, totaling 1,200 episodes per model. This follows the evaluation protocol adopted by most established studies. For example, ABot-M0 jointly trains one model on all 24 tasks and reports results over 50 rollouts per task, matching our task coverage and rollout count. Our internal comparisons additionally use the same backbone, demonstration data, observation inputs, and training setup. To address the concern about evaluation noise, we supplement this evaluation with three evaluation seeds and additional real-world branch ablations. ABot-M0

[Additional experimental evidence]

  • Additional evaluation.

We report the complete four-way ablation across three evaluation seeds. Full achieves the highest success rate in all three evaluations, exceeding both Base and Scene-only in each case.

ModelSeed 7Seed 27Seed 1137Mean ± Std
Base65.665.667.666.3 ± 1.2
Motion-only66.866.470.267.8 ± 2.1
Scene-only70.870.373.371.4 ± 1.7
Full71.972.675.273.2 ± 1.7

Results are reported as Mean ± Std across three evaluation seeds.

The reported mean gains are +7.0 pp over Base, +5.4 pp over Motion-only, and +1.8 pp over Scene-only. The ordering Full > Scene-only > Motion-only > Base is maintained across all three evaluation seeds. The positive gains therefore persist beyond the original evaluation, providing further support for the reliability of the improvement.

Scene supervision already supplies useful geometric information for action learning. The following simulation and real-world comparisons further show that the incremental contribution of motion forcing is larger in the evaluated settings requiring stronger spatial coordination and generalization.

  • Performance across the two RoboCasa task categories.

The 24 tasks in our RoboCasa-GR1 evaluation fall into two categories, in-container placement and closure, and object pick-and-place. The former requires placing an object within a constrained space and subsequently closing an articulated container, imposing greater demands on spatial coordination. We report the two categories separately to examine how the contribution of motion forcing varies with these demands.

ModelPlacement + closurePick-and-place
Base54.6 ± 2.470.1 ± 1.1
Motion-only54.6 ± 4.172.2 ± 1.5
Scene-only60.8 ± 4.275.0 ± 0.9
Full66.0 ± 2.675.6 ± 1.5
v.s. Base+11.4 pp+5.5 pp
v.s. Scene-only+5.2 pp+0.6 pp

Results are reported as Mean ± Std across three evaluation seeds.

Adding motion forcing to Scene-only yields +5.2 pp on in-container placement and closure, compared with +0.6 pp on object pick-and-place. The larger gain on the more demanding category supports the usefulness of motion forcing when successive EEF movements must remain coordinated with changing relationships among the hand, object, and container.

AuthorsOpenReview ↗

Response (Part2)

  • Real-world ablations.

We supplement the rebuttal with real-world branch ablations on two relatively challenging tasks, Bricks Disassembly and Hanger Placement, comparing GF-VLA with our Base, Scene-only, and Motion-only variants. Easy conditions resemble demonstration collection. Hard conditions introduce distractor bricks and unseen hook positions, respectively, increasing the demands on robust execution and spatial generalization.

The task-level success rates show that Full improves over Base in both conditions.

TaskEasy · BaseEasy · FullHard · BaseHard · Full
Bricks Disassembly53.363.3 (+10.0 pp)36.750.0 (+13.3 pp)
Hanger Placement50.063.3 (+13.3 pp)23.333.3 (+10.0 pp)

Parentheses show gains over Base.

The four-way ablation, averaged equally across the two tasks, further identifies the contribution of motion forcing. We use Full as the reference in the following table.

ModelEasy SR (%)Hard SR (%)
Base51.7 (−11.7 pp)30.0 (−11.7 pp)
Scene-only58.3 (−5.0 pp)31.7 (−10.0 pp)
Motion-only56.7 (−6.7 pp)38.3 (−3.3 pp)
Full63.341.7

Parentheses indicate differences relative to Full.

Removing the motion branch from Full reduces success by 10.0 pp under Hard conditions, compared with 5.0 pp under Easy conditions. Under Hard conditions, Motion-only also exceeds Base by 8.3 pp, whereas Scene-only exceeds Base by 1.7 pp. Full remains the best-performing variant, and adding scene forcing to Motion-only provides a further 3.3 pp. The larger incremental motion gain under Hard conditions provides further evidence of its contribution when execution and generalization demands increase.

Together, the multi-seed evaluations, RoboCasa category comparisons, and real-world ablations support a consistent conclusion. Motion forcing complements scene supervision, with larger benefits in the evaluated settings that demand stronger spatial coordination and generalization.

Response to W2, Q1: External baseline results and comparison settings

The compared methods generally follow the standard training and evaluation setup of each benchmark, using the corresponding benchmark demonstrations for task-specific training or fine-tuning. Our experiments follow these benchmark protocols. The external scores in the main tables are taken from the original papers, evaluations reported in third-party papers, or evaluation reports. We will clarify these sources in the appendix.

For LIBERO, our evaluation covers four suites with ten tasks per suite and 50 rollouts per task, totaling 2,000 episodes. For LIBERO-Plus, the LIBERO-trained policy is evaluated directly on 10,030 instances across seven perturbation categories without further training. For RoboCasa-GR1, we evaluate the standard 24 tasks with 50 rollouts per task, consistent with the task coverage and rollout count used by methods such as ABot-M0 and DIAL. ABot-M0; DIAL

Some methods additionally use a separate pretraining stage. For example, ZR-0 uses a Qwen3-VL-2B-Instruct backbone pretrained on ProcCorpus-60M, and ABot-M0 uses Qwen3-VL-4B with UniACT pretraining. PILOT builds on the pretrained Wan2.2-TI2V-5B video backbone. GF-VLA uses Qwen3.5-4B and learns from target-benchmark demonstrations without an additional large-scale robot-data pretraining stage. ZR-0; ABot-M0; PILOT

We will supplement Appendix B with the exact source of each external score and describe paper-specific settings, including additional pretraining, training budgets, and evaluation-budget exceptions. This will make the provenance and comparison settings explicit. Our internal ablations use the same backbone, demonstrations, observations, and training budget to examine the contribution of geometric learning.

Response to W2: Strength of our base model

Our Base is already a competitive VLA model. It uses the same Qwen3.5-4B backbone and flow-based Action Expert as GF-VLA, trained on the same demonstrations with the same observation inputs and training setup. On RoboCasa-GR1, Base achieves 65.6%, compared with 63.2% for JoyAI-RA, 62.6% for PILOT, and 58.3% for ABot-M0 in Table 3.

It therefore provides a competitive starting point for the controlled ablation. Adding dual-geometric learning increases success from 65.6% to 71.9% in the original evaluation. The additional evaluations preserve this advantage, with 66.3% for Base and 73.2% for Full on average, supporting a contribution beyond the choice of backbone alone.

AuthorsOpenReview ↗

Response (Part3)

Response to W2, Q4: Why future-only depth supervision outperforms current + future

We choose future-depth supervision to align scene learning with the future movement that the policy must execute. Future-depth prediction and future-motion prediction both infer geometry from the current observation and instruction over the same action horizon. The motion target describes the EEF trajectory throughout the action chunk, while the depth target describes the scene at its endpoint. This makes future scene geometry directly relevant to the motion representation used for action generation.

Current-depth reconstruction focuses on the already observed scene and may introduce partially redundant information. Combining current and future reconstruction also distributes the shared representation’s learning capacity across current-scene reconstruction and future prediction. Future-only supervision keeps the scene objective focused on the geometry associated with the future motion.

We compare current-only, current-plus-future, and future-only supervision in Table 4. Future-only achieves 71.9%, compared with 67.6% for current-only and 67.9% for current-plus-future, and also yields the lowest prediction–execution UV and depth errors. These comparisons support the choice of temporally aligned future-depth supervision.

Response to W2: Performance under camera perturbations

We agree that GF-VLA has room to improve under camera perturbations. Its 68.9% Camera success rate is below the stronger-performing models in this category in Table 2. Our original discussion did not make this limitation sufficiently explicit.

Both UVD motion targets and scene-depth targets are expressed in camera coordinates. They can provide useful spatial supervision without guaranteeing viewpoint-invariant representations, because changing the camera also changes the image coordinates and depth of the same physical scene. This offers a plausible explanation for the remaining viewpoint sensitivity.

GF-VLA’s 83.8% overall LIBERO-Plus score supports strong aggregate zero-shot transfer across the evaluated perturbations, but does not establish superiority under camera changes. We will revise the discussion accordingly. Camera augmentation and more viewpoint-robust geometric representations are directions for further improvement.

Response to Q2: Real-world depth targets and reproducibility

Scene-depth targets are measured directly by depth cameras. ARX-Lift2S uses RealSense D405 cameras; RM75 uses an external RealSense L515 and a wrist-mounted D435. Depth-to-RGB alignment is enabled on both platforms. For each demonstration chunk, we use the aligned depth frame at its endpoint as scene supervision. The supervised views are the central primary camera for ARX-Lift2S and the external and wrist cameras for RM75.

Motion targets are constructed from recorded EEF poses and the defined gripper reference points using calibrated projection. Both ARX gripper trajectories are projected into the central primary RGB camera; the RM75 trajectory is projected into the fixed external RGB camera. U and V are projected image coordinates, and D is depth along that camera’s optical axis. RM75’s wrist view supplies additional scene-depth supervision without changing the motion-coordinate frame.

Reproducing these targets requires associating the recorded EEF trajectory and aligned depth frames with the same action chunk, projecting the gripper reference points into the designated RGB camera, and selecting scene depth at the chunk endpoint. Both targets are used for training supervision; future depth is not supplied to the deployed policy. We will expand Appendix A.1 with this real-world acquisition and target-construction procedure alongside the simulation replay procedure.

Reviewer_VM6QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper GF-VLA augments a VLA with motion and scene supervision. The proposed time-indexed motion latents are supervised by end-effector coordinates and metric camera depth. The scene latents reconstruct depth at the chunk endpoint, and motion queries attend to those scene latents. The otion / scene queries are used at inference. GF-VLA reports 98.4% on LIBERO, 83.8% zero-shot on LIBERO-Plus, and 71.9% on RoboCasa-GR1. Ablations compare 2D and 3D traces, scene-supervision targets, and the two geometry objectives.

Critical strengths
  1. Idea is novel. A sparse 3D end-effector trace supplies motion information, while depth at the same chunk endpoint supplies temporally aligned scene geometry. The architecture gives scene supervision a path into the motion latents used by the action expert.

  2. The empirical study spans standard manipulation, zero-shot perturbations, dexterous bimanual simulation, and three real-world tasks.

Critical weaknesses

My main issue wit the paper is that incremental contribution of motion forcing needs stronger empirical support. In Table 5, scene forcing alone achieves 70.8% success, compared with 71.9% for full GF-VLA and 65.6% for the baseline. Thus, most of the improvement is already obtained through scene supervision, while motion forcing adds 1.1 percentage points. Independent training-run uncertainty is needed to establish the reliability of the incremental gain. The task-level regressions in Table 13 also suggest that motion forcing is not consistently beneficial.

Then this raises the contribution question. Related work already explores depth reconstruction with 3D hand trajectories, such as 3D HAMSTER, and latent 3D motion conditioning of an Action Expert, such as LaMP. GF-VLA’s distinctive contribution therefore depends on its particular combination of temporally aligned geometric targets and directed scene-to-motion attention. However, Table 5 varies the supervision objectives without isolating the attention pathway. The novelty is there but fails to be justified.

Questions for authors
  1. In the scene-only variant, is only the motion-supervision loss removed, or are motion queries and their attention connections also changed?

  2. With both geometric targets, query counts, and training budgets fixed, how does performance change when motion queries cannot attend to scene latents? How does the proposed pathway compare with allowing the Action Expert to read scene latents directly?

  3. Please quantify uncertainty in the 1.1-point improvement. I wonder if this is pure noise.

4 comments
AuthorsOpenReview ↗

Response (Part1)

We thank the reviewer for recognizing the novelty of GF-VLA’s temporally aligned motion and scene supervision, the scene-to-motion pathway, and the breadth of our evaluation across simulation and real-world tasks. Below, we respond to the concerns and questions in detail.

Response to W1, Q3: Contribution of motion forcing

[Method motivation]

Motion forcing explicitly supervises the EEF’s 3D trajectory throughout an action chunk, encouraging the policy to capture where the EEF should move and how its movement unfolds over time. The resulting motion latents condition the Action Expert, providing geometric guidance for action generation beyond action supervision alone. Scene forcing supports this process by supplying scene geometry that helps the motion branch learn future movement in its spatial context.

Table 5 evaluates each trained model over 24 RoboCasa-GR1 tasks with 50 rollouts per task, totaling 1,200 episodes per model. This setup is consistent with existing work that evaluates trained policies across multiple tasks and repeated rollouts. For example, StarVLA’s official LIBERO evaluation evaluates one policy across four suites, with 500 trials per suite. On RoboCasa-GR1, ABot-M0 jointly trains one model on all 24 tasks and reports results over 50 rollouts per task, matching our task coverage and rollout count. This broad task coverage and repeated evaluation provide a substantial empirical basis for the comparisons in Table 5.

[Additional experimental results]

  • Additional evaluation. To further examine the reliability of these improvements, we report the complete four-way ablation across three evaluation seeds. Full achieves the highest success rate in all three reported evaluations, exceeding both Base and Scene-only in each case.
ModelSeed 7Seed 27Seed 1137Mean ± Std
Base65.665.667.666.3 ± 1.2
Motion-only66.866.470.267.8 ± 2.1
Scene-only70.870.373.371.4 ± 1.7
Full71.972.675.273.2 ± 1.7

Results across three evaluation seeds.

The reported mean gains are +7.0 pp over Base, +5.4 pp over Motion-only, and +1.8 pp over Scene-only. The positive gains recur across all three reported evaluation seeds, providing further support for the reliability of the improvement beyond a single evaluation.

Besides, we believe that motion forcing can provide greater benefits for action generation when tasks demand more precise movement, stronger spatial understanding, and greater generalization. Scene supervision already provides some useful geometric information for action learning, while the incremental contribution of motion forcing can become more apparent as these demands increase. We further analyze this interpretation through comparisons in simulation and the real world.

  • Performance across the two RoboCasa task categories. The 24 tasks in our RoboCasa-GR1 evaluation fall into two categories: in-container placement and closure, and object pick-and-place. The former requires placing an object within a constrained space and subsequently closing an articulated container, imposing greater demands on spatial coordination than the latter. We report the two categories separately to examine how the contribution of motion forcing varies with these demands.
ModelPlacement + closurePick-and-place
Base54.6 ± 2.470.1 ± 1.1
Motion-only54.6 ± 4.172.2 ± 1.5
Scene-only60.8 ± 4.275.0 ± 0.9
Full66.0 ± 2.675.6 ± 1.5
Full − Base+11.4 pp+5.5 pp
Full − Scene-only+5.2 pp+0.6 pp

Placement + closure denotes in-container placement followed by closure; pick-and-place denotes object pick-and-place. Results are reported as Mean ± Std across three evaluation seeds.

Adding motion forcing to Scene-only yields +5.2 pp on in-container placement and closure, compared with +0.6 pp on object pick-and-place. The larger gain on the more demanding category supports our view that motion forcing is particularly useful when successive EEF movements must remain coordinated with changing relationships among the hand, object, and container. Although individual tasks exhibit some regressions, the reported aggregate results remain positive, with more substantial gains on container placement and closure.

AuthorsOpenReview ↗

Response (part2)

  • Real-world ablations. We have supplemented additional real-device experiments in this rebuttal. We further examine this pattern on two relatively challenging real-world tasks, Bricks Disassembly and Hanger Placement, comparing GF-VLA with our own Base, Scene-only, and Motion-only variants. Each variant is evaluated in 30 trials per task and condition. Easy conditions resemble demonstration collection. Hard conditions introduce distractor bricks and unseen hook positions, respectively, increasing the demands on robust execution and spatial generalization.

The task-level success rates show that Full improves over Base in both conditions.

TaskEasy · BaseEasy · FullHard · BaseHard · Full
Bricks Disassembly53.363.3 (+10.0 pp)36.750.0 (+13.3 pp)
Hanger Placement50.063.3 (+13.3 pp)23.333.3 (+10.0 pp)

Parentheses show gains over Base.

The four-way ablation, averaged equally across the two tasks, further identifies the contribution of motion forcing. We use Full as the reference in the following table.

ModelEasy SR (%)Hard SR (%)
Base51.7 (−11.7 pp)30.0 (−11.7 pp)
Scene-only58.3 (−5.0 pp)31.7 (−10.0 pp)
Motion-only56.7 (−6.7 pp)38.3 (−3.3 pp)
Full63.341.7

Parentheses indicate differences relative to Full.

Removing the motion branch from Full reduces success by 10.0 pp under Hard conditions, compared with 5.0 pp under Easy conditions. Under Hard conditions, Motion-only also exceeds Base by +8.3 pp, whereas Scene-only exceeds Base by +1.7 pp. Full remains the best-performing variant, and adding scene forcing to Motion-only provides a further +3.3 pp. The larger incremental motion gain under Hard conditions provides further evidence that motion forcing becomes more useful as execution and generalization demands increase.

Together, the reported multi-seed evaluations, RoboCasa category comparisons, and real-world ablations support a consistent conclusion: motion forcing complements scene supervision, with larger benefits in the evaluated settings that demand stronger spatial coordination and generalization.

Response to W2: Clarifying the novelty of temporally aligned geometric targets and directed scene-to-motion attention

GF-VLA differs in emphasis from the two related works highlighted by the reviewer. 3D HAMSTER requires the input to contain additional depth information. It mainly focuses on the interface between hierarchical planning and control, which then guides the low-level point-cloud-based policy. LaMP learns a motion prior from dense 3D scene flow and conditions an Action Expert on Motion Expert features. We will revise the Related Work section to clarify these distinctions.

As the reviewer notes, our distinctive design combines temporally aligned geometric targets and directed scene-to-motion attention. Specifically, we jointly supervise the EEF’s motion throughout an action chunk and scene depth at the endpoint of that same chunk. Motion queries read the depth-supervised scene latents, and the resulting motion representations condition the Action Expert. Both geometric decoders are used only during training. We provide evidence for these two design choices below.

[Temporally aligned geometric targets]

Table 4 compares current-only, current-plus-future, and future-only scene-depth supervision while retaining motion forcing.

Scene-depth supervisionSR (%) ↑UV error (px) ↓Depth error (mm) ↓
None66.83.468.67
Current only67.63.749.34
Current + future67.93.478.47
Future only71.93.368.09

Future-only supervision improves success by +4.3 pp over current-only and +4.0 pp over current-plus-future, while also achieving the lowest prediction–execution UV and depth errors. These results support aligning scene supervision with the future motion being learned.

We choose future depth because future-depth prediction and future-motion prediction share a closely related predictive objective. Both infer future geometry from the current observation and instruction over the same action horizon. The motion target describes how the EEF moves through the action chunk, while the depth target describes the scene at its endpoint. Future scene supervision therefore provides spatial information relevant to the movement being learned.

AuthorsOpenReview ↗

Response (Part3)

[Directed scene-to-motion attention]

To address the reviewer’s concern that Table 5 does not isolate the attention pathway, we trained a control variant that retains both query sets, both geometric losses, and motion-to-Action-Expert conditioning, while removing only motion queries’ access to scene latents. Both branches continue to read vision-language features. Within the available rebuttal time and resources, we prioritized examining its effect on RoboCasa’s relatively challenging in-container placement and closure tasks.

ModelSeed 7Seed 27Seed 1137Mean ± Std
Full64.764.369.066.0 ± 2.6
Without scene-to-motion access62.760.765.062.8 ± 2.2
Gain from retaining the pathway+2.0 pp+3.7 pp+4.0 pp+3.2 pp

Results across three evaluation seeds.

Removing the connection reduces success under all three reported evaluation seeds, with a mean decrease of 3.2 pp. These results support the effectiveness of the directed pathway.

We retain scene-to-motion attention because learning future EEF movement requires relating the trajectory to surrounding objects and free space. Allowing motion queries to read scene latents gives the motion representation access to this geometric context before it conditions the Action Expert. The attention control supports this design, while the timing ablation supports the choice of future scene supervision. Together, these experiments provide empirical support for the two components of our distinctive scene–motion–action formulation.

Response to Q1: Scene-only and Motion-only definitions

The ablations remove the corresponding branch, rather than only disabling its loss.

Scene-only removes the motion queries, UVD prediction head, motion loss, and motion-to-Action-Expert connections. Scene queries and depth supervision remain. The Action Expert uses only native vision-language features and does not directly read scene latents.

Motion-only removes the scene queries and depth supervision. Motion queries read vision-language features, receive UVD trajectory supervision, and continue to condition the Action Expert.

Response to Q2: Attention-pathway experiments

[Removing scene-to-motion access]

As reported in our response to W2, removing motion queries’ access to scene latents reduces success on RoboCasa’s in-container placement and closure tasks under all three reported evaluation seeds, with a mean decrease of 3.2 pp. Please refer to our response to W2 for the control settings and detailed results.

[Adding direct scene access to the Action Expert]

Building on the scene–motion–action design discussed in our response to W2, we also plan to examine whether allowing the Action Expert to additionally read scene latents can further improve performance. Given the time and computational constraints of the rebuttal period, we will make every effort to include this experiment in the revised version in two days, keeping the geometric targets, query counts, training budget, and existing motion pathway unchanged, while adding direct access from the Action Expert to scene latents.

AuthorsOpenReview ↗

Response (Part4)

Thank you for suggesting this comparison, which has helped us make the experimental analysis more complete. Following up on Q2 and our earlier response, we have completed the additional experiment allowing the Action Expert to directly read scene latents. We retain the geometric targets, query counts, training budget, and original motion pathway, adding only direct scene access to the Action Expert.

Results below report mean ± sample SD across three evaluation seeds on RoboCasa-GR1.

ModelSuccess rate (%) ↑UV error (px) ↓Depth error (mm) ↓
Full73.2 ± 1.73.30 ± 0.067.89 ± 0.18
Full + direct scene access72.7 ± 0.73.16 ± 0.027.75 ± 0.19

The additional connection reduces prediction–execution errors, while the mean success rate decreases from 73.2% to 72.7%. Direct scene access improves geometric consistency a little in this comparison but does not improve task success.

In future work, we will explore more flexible ways of integrating scene and motion information into the Action Expert to better support task execution.

We would greatly appreciate your feedback on these additional results and our earlier responses, particularly whether any concerns remain unresolved. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.