- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
GeoAct introduces geometric feedback into the sampling process of a joint video–action model. At selected denoising steps, a frozen VGGT backbone and a trained joint readout estimate robot configurations from the predicted clean video. A bounded proximal update moves the predicted action toward these configurations. The corrected action is then reinjected at the current noise level, and joint denoising continues.
The paper reports improvements over the native sampler and a post-hoc correction baseline on LIBERO and RoboTwin 2.0. Ablations examine readout refresh, correction constraints, and additional model evaluations. The default configuration adds 22% latency relative to the native sampler.
Critical strengths
-
The paper gives generated video a concrete role in action selection. The feedback mechanism is clearly described, including how corrections are mapped back to the noisy state and how the solver handles the intervention. Keeping the world action model frozen makes this a potentially useful addition to existing joint models.
-
The shared-checkpoint comparison with post-hoc correction is informative. The reported improvements occur across both benchmarks, and the matched model-evaluation baseline suggests that additional evaluations alone do not explain the gain. The finding that unconstrained correction improves agreement while reducing task success is also useful: it exposes an important limitation of agreement as an optimization target.
Critical weaknesses
-
My main concern is that it is unclear when the video should be trusted over the predicted action. The action may already be correct while the video is inaccurate. Moving the action toward that video could make it worse. Repeating this process could make video and action increasingly consistent while both move away from the task goal. The update bounds limit how much the action changes, but do not tell us whether the change is helpful. The reported success rates are encouraging, yet the paper does not directly examine when corrections help or hurt. The DROID readout results also do not establish accuracy on generated videos at the intervention steps.
-
Important experimental details are missing. The paper does not clearly identify the checkpoints used for the main benchmarks, their adaptation data, or the training data and accuracy of the benchmark-specific readouts. The conversion between LIBERO end-effector actions and joint targets is also underspecified. Since these components determine the correction target, the omissions affect both reproducibility and confidence in the results.
Questions for authors
-
What evidence supports trusting the video when it disagrees with the action? As discussed in the first weakness, the original action may be correct while the video is inaccurate. Correcting the action could then influence the next video prediction, making both outputs more consistent but less useful for the task. How do you assess this risk? A simple starting point would be to report how often GeoAct turns a successful Base rollout into a failure, alongside the opposite transition, and examine whether agreement improves in those failed cases.
-
How accurate are the embodiment-specific readouts on benchmark observations and generated clean-video predictions at the intervention steps? How does this accuracy change with the noise level?
-
Which checkpoints and training data are used for each benchmark? Please also specify the LIBERO action-to-joint conversion, intervention schedule, and correction settings.
Further clarification and additional key experiments could substantially influence my score.
5 comments
Response to Reviewer 9VaF [2/2]
W2 & Q3. "Important experimental details are missing. The paper does not clearly identify the checkpoints used for the main benchmarks, their adaptation data, or the training data and accuracy of the benchmark-specific readouts. The conversion between LIBERO end-effector actions and joint targets is also underspecified. Since these components determine the correction target, the omissions affect both reproducibility and confidence in the results." / "Which checkpoints and training data are used for each benchmark? Please also specify the LIBERO action-to-joint conversion, intervention schedule, and correction settings."
A3. Within each benchmark, Base, Post-hoc, and GeoAct share one checkpoint, one training set, seeds, and initial noise (Sec. 4.1), so the sampler is the only variable. We will add all details below to the appendix.
Checkpoint and freezing scope: DreamZero-DROID (14B), adapted separately on public LIBERO demonstrations (Liu et al., 2023; 1,800 training episodes across 40 tasks, 45 per task; 20,000 optimizer updates) and public RoboTwin 2.0 Aloha-AgileX demonstrations clean_50 and randomized_500 (Chen et al., 2025; 24,750 training episodes across 50 tasks, 45 clean and 450 randomized per task; 40,000 optimizer updates). The remaining 200 LIBERO (5 per task) and 2,750 RoboTwin (5 clean and 50 randomized per task) demonstrations are held out from both checkpoint adaptation and readout training; the 50 LIBERO and 100 RoboTwin trials per task per setting in Sec. 4.1 are separate evaluation rollouts. Once adapted, the WAM checkpoint is held frozen throughout readout training and all sampler comparisons, and VGGT remains frozen throughout.
Readouts: the recipe of Sec. 3.2 (used there for the DROID reader on real video) and loss of Eq. 5, applied per embodiment as in Sec. 4.1 and trained on 288,000 timestamp draws from the same 1,800 training demonstrations for LIBERO and 3,960,000 timestamp draws from the same 24,750 training demonstrations for RoboTwin (160 draws per episode, sampled uniformly with replacement over valid timestamps, with no extra rollouts). Each draw contains synchronized RGB views and joint labels, with one set of joint queries and one kinematic chain per arm on RoboTwin. Accuracy is in A2.
LIBERO conversion: 6-DoF end-effector pose deltas are mapped to joint targets by differentiable damped least-squares IK with 20 iterations and damping , initialized at the measured joint state to keep the solution on the branch of the current arm configuration. Full 6-DoF pose forward kinematics maps the corrected joints back to end-effector pose deltas in the same convention.
Schedule and settings: all three samplers use the native 16-step FlowUniPC scheduler (sigma_shift=5.0) with coupled video/action noise (decouple_inference_noise=false) and all 16 solver steps evaluated without step-skipping. With , GeoAct intervenes at zero-based indices 13 and 14 (after field prediction and before the solver update), which are the last two steps preceding the final denoising step 15 (as required by Secs. 3.5 and 4.4), at 0.5354 and 0.4164 ( in Fig. 2a uses indices 11–14; applies all 8 gradient steps at index 14). Each accepted edit discards stale predictions, reevaluates the joint field at the same noise level (including the conditional/unconditional passes for CFG scale 5.0), and restarts the multistep solver, adding up to field evaluations beyond Base's 16 (matched at 18 in Table 3, row 2). Hyperparameters: = 0.08, = 0.10 in normalized action units, = 1.0, = 0.10 m, 4 projected gradient steps per intervention (8 total; Post-hoc uses the same 8-step budget after generation); readout = 1.0 after a 2,000-update joint-only warmup.
Dear Reviewer, Thank you once again for your thoughtful comments and constructive suggestions. We submitted our detailed response a few days ago and would greatly appreciate it if you could review it at your convenience. We would be happy to provide any further clarification you may need.
Response to Reviewer 9VaF [1/2]
W1 & Q1. "My main concern is that it is unclear when the video should be trusted over the predicted action. The action may already be correct while the video is inaccurate. Moving the action toward that video could make it worse. Repeating this process could make video and action increasingly consistent while both move away from the task goal. The update bounds limit how much the action changes, but do not tell us whether the change is helpful. The reported success rates are encouraging, yet the paper does not directly examine when corrections help or hurt. The DROID readout results also do not establish accuracy on generated videos at the intervention steps." / "What evidence supports trusting the video when it disagrees with the action? As discussed in the first weakness, the original action may be correct while the video is inaccurate. Correcting the action could then influence the next video prediction, making both outputs more consistent but less useful for the task. How do you assess this risk? A simple starting point would be to report how often GeoAct turns a successful Base rollout into a failure, alongside the opposite transition, and examine whether agreement improves in those failed cases."
A1. The corrections move the executed motion toward the task goal, not only toward the video. The paper measures this with EE error, which compares simulator motion with task-defined targets and never uses the readout (Sec. 4.1). In Table 3, every correction variant lowers EE error relative to Base (5.8 cm), including the unbounded one (5.2 cm, 90.8% success vs. 89.2% for Base), and GeoAct reaches 3.5 cm with 95.6% success.
Repeated feedback does not drift away from the goal. Success rises with the number of interventions per chunk: 89.2% (none), 94.0% (one), 95.6% (two), 95.8% (four) on LIBERO-Long (Fig. 2a). Reading the updated video at each intervention is worth 3.2 points over reusing the first reading (no-refresh row, Table 3), so the later video, after it has responded to the corrected action, gives a better target than the earlier one.
Paired rollouts (same scene, seed, and initial noise), as the reviewer suggested. Base has 1,890 successes and 110 failures on LIBERO, and 8,180 successes and 1,820 failures on RoboTwin; RoboTwin pools 5,000 clean and 5,000 randomized episodes with equal weight:
GeoAct recovers 66.4% / 49.0% of Base failures and turns 0.7% / 2.1% of Base successes into failures on LIBERO / RoboTwin, respectively; the gap shrinks in both groups, so the outcome is decided by where the corrected motion goes, which the readout-free EE error in Table 3 measures.
Q2. "How accurate are the embodiment-specific readouts on benchmark observations and generated clean-video predictions at the intervention steps? How does this accuracy change with the noise level?"
A2. At the intervention noise levels, median EE error on generated clean predictions is within 0.53 cm of the error on held-out observations across all three settings. We noise held-out benchmark frames to the video noise level σV, take the model's one-step clean prediction V^0, and compare the readout on V^0 with the simulator joints. This reflects the combined deviation of the one-step video prediction V^0 and the readout at noise level σV.
GeoAct intervenes at action noise levels σa≈ 0.5354 and 0.4164, with equal video noise levels in the coupled schedule specified in A3. These late steps also give a larger state shift than earlier steps for the same clean-endpoint edit, since the shift scales with 1−σa (Eq. 9). The open-loop readout error is not a floor on closed-loop executed error: GeoAct takes a bounded step from the model's action rather than executing qV directly, and each chunk is anchored at the measured robot state (Eq. 1).
Sec. 4.5 evaluates one DROID reader across diverse real-world scenes and camera placements (10.55 cm median error on 850 validation samples from 30 held-out episodes), whereas each simulation reader is trained and evaluated within one benchmark embodiment and camera setup.