acceptodds
Under review as a conference paper at ICLR 2027

GeoAct: Reconciling Video and Action with Robot Geometry

Abstract

World action models generate a future video and an action chunk for a robot in one sampling process. The training loss does not tie the two outputs to the robot’s kinematics. As a result, the video and the action can describe different arm motions. We propose GEOACT, a sampler that uses robot geometry to reconcile the two outputs during generation. At selected denoising steps, a frozen geometric backbone and a trained joint readout estimate the arm configurations shown in the clean video prediction. Forward kinematics places this estimate and the predicted action in the robot base frame. A bounded proximal step moves the clean action toward the visual estimate. GEOACT reinjects the corrected action at the current noise level and evaluates the world action model again. The remaining video generation can then respond to the correction. On a shared backbone, GEOACT raises task success from 94.5% to 97.5% on LIBERO and from 81.8% to 89.0% on RoboTwin 2.0. Post-hoc correction with the same readout and the same budget reaches 95.6% and 83.9%. Ablations show that the gain requires a fresh reading of the updated video and a bound on each update. Two interventions per action chunk cost 1.22× the base latency.

Primary area:
generative models
Keywords:
world model, diffusion

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9VaFOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

GeoAct introduces geometric feedback into the sampling process of a joint video–action model. At selected denoising steps, a frozen VGGT backbone and a trained joint readout estimate robot configurations from the predicted clean video. A bounded proximal update moves the predicted action toward these configurations. The corrected action is then reinjected at the current noise level, and joint denoising continues.

The paper reports improvements over the native sampler and a post-hoc correction baseline on LIBERO and RoboTwin 2.0. Ablations examine readout refresh, correction constraints, and additional model evaluations. The default configuration adds 22% latency relative to the native sampler.

Critical strengths
  1. The paper gives generated video a concrete role in action selection. The feedback mechanism is clearly described, including how corrections are mapped back to the noisy state and how the solver handles the intervention. Keeping the world action model frozen makes this a potentially useful addition to existing joint models.

  2. The shared-checkpoint comparison with post-hoc correction is informative. The reported improvements occur across both benchmarks, and the matched model-evaluation baseline suggests that additional evaluations alone do not explain the gain. The finding that unconstrained correction improves agreement while reducing task success is also useful: it exposes an important limitation of agreement as an optimization target.

Critical weaknesses
  1. My main concern is that it is unclear when the video should be trusted over the predicted action. The action may already be correct while the video is inaccurate. Moving the action toward that video could make it worse. Repeating this process could make video and action increasingly consistent while both move away from the task goal. The update bounds limit how much the action changes, but do not tell us whether the change is helpful. The reported success rates are encouraging, yet the paper does not directly examine when corrections help or hurt. The DROID readout results also do not establish accuracy on generated videos at the intervention steps.

  2. Important experimental details are missing. The paper does not clearly identify the checkpoints used for the main benchmarks, their adaptation data, or the training data and accuracy of the benchmark-specific readouts. The conversion between LIBERO end-effector actions and joint targets is also underspecified. Since these components determine the correction target, the omissions affect both reproducibility and confidence in the results.

Questions for authors
  1. What evidence supports trusting the video when it disagrees with the action? As discussed in the first weakness, the original action may be correct while the video is inaccurate. Correcting the action could then influence the next video prediction, making both outputs more consistent but less useful for the task. How do you assess this risk? A simple starting point would be to report how often GeoAct turns a successful Base rollout into a failure, alongside the opposite transition, and examine whether agreement improves in those failed cases.

  2. How accurate are the embodiment-specific readouts on benchmark observations and generated clean-video predictions at the intervention steps? How does this accuracy change with the noise level?

  3. Which checkpoints and training data are used for each benchmark? Please also specify the LIBERO action-to-joint conversion, intervention schedule, and correction settings.

Further clarification and additional key experiments could substantially influence my score.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 9VaF [1/2]

W1 & Q1. "My main concern is that it is unclear when the video should be trusted over the predicted action. The action may already be correct while the video is inaccurate. Moving the action toward that video could make it worse. Repeating this process could make video and action increasingly consistent while both move away from the task goal. The update bounds limit how much the action changes, but do not tell us whether the change is helpful. The reported success rates are encouraging, yet the paper does not directly examine when corrections help or hurt. The DROID readout results also do not establish accuracy on generated videos at the intervention steps." / "What evidence supports trusting the video when it disagrees with the action? As discussed in the first weakness, the original action may be correct while the video is inaccurate. Correcting the action could then influence the next video prediction, making both outputs more consistent but less useful for the task. How do you assess this risk? A simple starting point would be to report how often GeoAct turns a successful Base rollout into a failure, alongside the opposite transition, and examine whether agreement improves in those failed cases."

A1. The corrections move the executed motion toward the task goal, not only toward the video. The paper measures this with EE error, which compares simulator motion with task-defined targets and never uses the readout (Sec. 4.1). In Table 3, every correction variant lowers EE error relative to Base (5.8 cm), including the unbounded one (5.2 cm, 90.8% success vs. 89.2% for Base), and GeoAct reaches 3.5 cm with 95.6% success.

Repeated feedback does not drift away from the goal. Success rises with the number of interventions per chunk: 89.2% (none), 94.0% (one), 95.6% (two), 95.8% (four) on LIBERO-Long (Fig. 2a). Reading the updated video at each intervention is worth 3.2 points over reusing the first reading (no-refresh row, Table 3), so the later video, after it has responded to the corrected action, gives a better target than the earlier one.

Paired rollouts (same scene, seed, and initial noise), as the reviewer suggested. Base has 1,890 successes and 110 failures on LIBERO, and 8,180 successes and 1,820 failures on RoboTwin; RoboTwin pools 5,000 clean and 5,000 randomized episodes with equal weight:

LIBERO (2,000 ep.)RoboTwin (10,000 ep.; clean + rand.)
Base fail → GeoAct success (% of Base failures)73 (66.4%)891 (49.0%)
Base success → GeoAct fail (% of Base successes)13 (0.7%)171 (2.1%)
Agreement gap, Base → GeoAct, in fail→success / success→fail (cm)12.6→4.1 / 10.9→4.715.2→5.4 / 13.8→6.3

GeoAct recovers 66.4% / 49.0% of Base failures and turns 0.7% / 2.1% of Base successes into failures on LIBERO / RoboTwin, respectively; the gap shrinks in both groups, so the outcome is decided by where the corrected motion goes, which the readout-free EE error in Table 3 measures.


Q2. "How accurate are the embodiment-specific readouts on benchmark observations and generated clean-video predictions at the intervention steps? How does this accuracy change with the noise level?"

A2. At the intervention noise levels, median EE error on generated clean predictions is within 0.53 cm of the error on held-out observations across all three settings. We noise held-out benchmark frames to the video noise level σV\sigma^V, take the model's one-step clean prediction V^0\hat V_0, and compare the readout on V^0\hat V_0 with the simulator joints. This reflects the combined deviation of the one-step video prediction V^0\hat V_0 and the readout at noise level σV\sigma^V.

Median EE (cm) / normalized joint MAE (Eq. 5)Held-out obs.V^0\hat V_0, σV\sigma^V=0.4164V^0\hat V_0, σV\sigma^V=0.5354V^0\hat V_0, σV\sigma^V=0.7497
LIBERO1.82 / 0.0141.94 / 0.0152.17 / 0.0173.86 / 0.031
RoboTwin (clean) / (rand.)(2.34 / 0.018) / (2.78 / 0.022)(2.49 / 0.019) / (2.96 / 0.024)(2.78 / 0.022) / (3.31 / 0.027)(4.72 / 0.039) / (5.64 / 0.047)

GeoAct intervenes at action noise levels σa≈\sigma^a\approx 0.5354 and 0.4164, with equal video noise levels in the coupled schedule specified in A3. These late steps also give a larger state shift than earlier steps for the same clean-endpoint edit, since the shift scales with 1−σa1-\sigma^a (Eq. 9). The open-loop readout error is not a floor on closed-loop executed error: GeoAct takes a bounded step from the model's action rather than executing qVq^V directly, and each chunk is anchored at the measured robot state (Eq. 1).

Sec. 4.5 evaluates one DROID reader across diverse real-world scenes and camera placements (10.55 cm median error on 850 validation samples from 30 held-out episodes), whereas each simulation reader is trained and evaluated within one benchmark embodiment and camera setup.


AuthorsOpenReview ↗

Response to Reviewer 9VaF [2/2]

W2 & Q3. "Important experimental details are missing. The paper does not clearly identify the checkpoints used for the main benchmarks, their adaptation data, or the training data and accuracy of the benchmark-specific readouts. The conversion between LIBERO end-effector actions and joint targets is also underspecified. Since these components determine the correction target, the omissions affect both reproducibility and confidence in the results." / "Which checkpoints and training data are used for each benchmark? Please also specify the LIBERO action-to-joint conversion, intervention schedule, and correction settings."

A3. Within each benchmark, Base, Post-hoc, and GeoAct share one checkpoint, one training set, seeds, and initial noise (Sec. 4.1), so the sampler is the only variable. We will add all details below to the appendix.

Checkpoint and freezing scope: DreamZero-DROID (14B), adapted separately on public LIBERO demonstrations (Liu et al., 2023; 1,800 training episodes across 40 tasks, 45 per task; 20,000 optimizer updates) and public RoboTwin 2.0 Aloha-AgileX demonstrations clean_50 and randomized_500 (Chen et al., 2025; 24,750 training episodes across 50 tasks, 45 clean and 450 randomized per task; 40,000 optimizer updates). The remaining 200 LIBERO (5 per task) and 2,750 RoboTwin (5 clean and 50 randomized per task) demonstrations are held out from both checkpoint adaptation and readout training; the 50 LIBERO and 100 RoboTwin trials per task per setting in Sec. 4.1 are separate evaluation rollouts. Once adapted, the WAM checkpoint is held frozen throughout readout training and all sampler comparisons, and VGGT remains frozen throughout.

Readouts: the recipe of Sec. 3.2 (used there for the DROID reader on real video) and loss of Eq. 5, applied per embodiment as in Sec. 4.1 and trained on 288,000 timestamp draws from the same 1,800 training demonstrations for LIBERO and 3,960,000 timestamp draws from the same 24,750 training demonstrations for RoboTwin (160 draws per episode, sampled uniformly with replacement over valid timestamps, with no extra rollouts). Each draw contains synchronized RGB views and joint labels, with one set of joint queries and one kinematic chain per arm on RoboTwin. Accuracy is in A2.

LIBERO conversion: 6-DoF end-effector pose deltas are mapped to joint targets by differentiable damped least-squares IK with 20 iterations and damping λ=10−3\lambda=10^{-3}, initialized at the measured joint state to keep the solution on the branch of the current arm configuration. Full 6-DoF pose forward kinematics maps the corrected joints back to end-effector pose deltas in the same convention.

Schedule and settings: all three samplers use the native 16-step FlowUniPC scheduler (sigma_shift=5.0) with coupled video/action noise (decouple_inference_noise=false) and all 16 solver steps evaluated without step-skipping. With m=2m=2, GeoAct intervenes at zero-based indices 13 and 14 (after field prediction and before the solver update), which are the last two steps preceding the final denoising step 15 (as required by Secs. 3.5 and 4.4), at σa=σV≈\sigma^a=\sigma^V\approx 0.5354 and 0.4164 (m=4m=4 in Fig. 2a uses indices 11–14; m=1m=1 applies all 8 gradient steps at index 14). Each accepted edit discards stale predictions, reevaluates the joint field at the same noise level (including the conditional/unconditional passes for CFG scale 5.0), and restarts the multistep solver, adding up to mm field evaluations beyond Base's 16 (matched at 18 in Table 3, row 2). Hyperparameters: η\eta = 0.08, δ\delta = 0.10 in normalized action units, β\beta = 1.0, ℓ\ell = 0.10 m, 4 projected gradient steps per intervention (8 total; Post-hoc uses the same 8-step budget after generation); readout α\alpha = 1.0 after a 2,000-update joint-only warmup.

Dear Reviewer, Thank you once again for your thoughtful comments and constructive suggestions. We submitted our detailed response a few days ago and would greatly appreciate it if you could review it at your convenience. We would be happy to provide any further clarification you may need.

Reviewer_nwXLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GeoAct uses a robot model to reduce disagreement between the future video and the action sequence produced by a joint world action model. At selected late sampling steps, a frozen VGGT backbone and a trained readout estimate the robot joints shown in the current clean video prediction. A bounded correction moves the predicted action toward this estimate, using both joint differences and end-effector positions. The corrected action is returned to the current noise level, and the joint model is evaluated again so that the video can respond. On the same backbone, average success increases from 94.5% to 97.5% on LIBERO and from 81.8% to 89.0% on RoboTwin 2.0. The authors also compare against correction after generation and study readout refresh, update bounds, and compute. Separate DROID experiments evaluate the visual readout; closed-loop task results are all in simulation.

Critical strengths

1: The method compares video and action in meaningful robot coordinates. It also converts both the arm motion shown in the video and the predicted actions into joint configurations and end-effector positions. This gives their disagreement a clear physical meaning, rather than relying on similarity between visual features.

2: The feedback happens before generation finishes. After correcting the action, GeoAct runs the joint model again so the video can respond. Later corrections use this updated video. This creates a useful feedback loop between the two outputs, instead of editing the action after the video is already fixed.

3: The correction avoids copying the video prediction. Each update stays close to the original action, has a limited size, and respects joint limits. These restrictions are sensible because the visual estimate can be wrong. The method also accounts for the current noise level when returning the correction to sampling.

Critical weaknesses

1: The reader is evaluated on real DROID videos, but GeoAct takes feedback from generated video estimates during sampling. Its accuracy on these images in LIBERO and RoboTwin is not reported.

2: Video–action agreement does not establish that the plan is correct. Both outputs come from the same generator, so they may share the same mistake. Moving the action toward the video could reinforce an incorrect plan. The authors recognize that agreement is not task success, but cases with incorrect imagined futures are needed to clarify when this feedback helps or hurts.

3: The correction covers only part of physical execution. It matches joint configurations and end-effector positions, while keeping the gripper command unchanged. It does not explicitly check collisions or whether a grasp remains valid after the arm motion changes. An analysis of collision, contact, and grasp failures would help show the limits of these corrections.

4: MINOR: The benchmark checkpoints, readout training data, correction settings, and LIBERO action-to-joint conversion are not fully specified. These choices directly affect the feedback signal and the resulting actions. More detail is needed to reproduce the results.

Questions for authors

Please refer to the weakness sections.

5 comments
AuthorsOpenReview ↗

Response to Reviewer nwXL [1/2]

W1. "The reader is evaluated on real DROID videos, but GeoAct takes feedback from generated video estimates during sampling. Its accuracy on these images in LIBERO and RoboTwin is not reported."

A1. At the intervention noise levels, median EE error on generated clean predictions is within 0.53 cm of the error on held-out observations across all three settings. DROID (Sec. 4.5) selects the readout design on real video. The benchmark readouts use the same design and are trained per embodiment (Sec. 4.1) on the same 1,800 LIBERO and 24,750 RoboTwin training demonstrations used to adapt the checkpoint, with 200 and 2,750 demonstrations held out for evaluation. We noise held-out benchmark frames to the video noise level σV\sigma^V, take the model's one-step clean prediction V^0\hat V_0, and compare the readout with the simulator joints; this reflects the combined deviation of V^0\hat V_0 and the readout at σV\sigma^V.

Median EE (cm) / normalized joint MAE (Eq. 5)Held-out obs.V^0\hat V_0, σV\sigma^V=0.4164V^0\hat V_0, σV\sigma^V=0.5354V^0\hat V_0, σV\sigma^V=0.7497
LIBERO1.82 / 0.0141.94 / 0.0152.17 / 0.0173.86 / 0.031
RoboTwin (clean) / (rand.)(2.34 / 0.018) / (2.78 / 0.022)(2.49 / 0.019) / (2.96 / 0.024)(2.78 / 0.022) / (3.31 / 0.027)(4.72 / 0.039) / (5.64 / 0.047)

GeoAct intervenes at σa=σV≈\sigma^a=\sigma^V\approx 0.5354 and 0.4164 (the last two steps before the final denoising step under the 16-step sigma_shift=5.0 scheduler). Whereas DROID (Sec. 4.5, 10.55 cm) evaluates one reader across diverse real-world camera placements, each simulation reader is trained and evaluated within one benchmark camera setup.


W2. "Video–action agreement does not establish that the plan is correct. Both outputs come from the same generator, so they may share the same mistake. Moving the action toward the video could reinforce an incorrect plan. The authors recognize that agreement is not task success, but cases with incorrect imagined futures are needed to clarify when this feedback helps or hurts."

A2. GeoAct uses agreement as a bounded local correction signal, and the paper evaluates it with task success and a simulator EE error that never uses the video. Table 3 separates internal agreement from task execution: the unbounded variant reaches the smallest agreement gap (2.7 cm), while bounded GeoAct reaches the best task EE error (3.5 cm) and success (95.6%). When video and action share the same mistake, the gap between them is already small and the update is small, leaving the trajectory close to the model's prediction rather than destabilizing valid rollouts. Paired rollouts show how often the feedback changes the outcome:

LIBERO (2,000 ep.)RoboTwin (10,000 ep.; clean + rand.)
Base fail → GeoAct success (% of Base failures)73 (66.4%)891 (49.0%)
Base success → GeoAct fail (% of Base successes)13 (0.7%)171 (2.1%)

The net gains are 60 of 2,000 episodes (+3.0 points) on LIBERO and 720 of 10,000 episodes (+7.2 points) on RoboTwin, with regressions affecting only 0.7% and 2.1% of Base successes, respectively. The gain is also largest in the harder setting: under RoboTwin scene randomization, Base drops 6.8 points whereas GeoAct drops 2.4 points (Table 2).


W3. "The correction covers only part of physical execution. It matches joint configurations and end-effector positions, while keeping the gripper command unchanged. It does not explicitly check collisions or whether a grasp remains valid after the arm motion changes. An analysis of collision, contact, and grasp failures would help show the limits of these corrections."

A3. The gripper is held fixed only inside the proximal step; the denoising that follows updates every action coordinate, including the gripper, together with the video (Alg. 1). After a correction, the joint model is evaluated again on (Zk,ak+)(Z_k, a_k^+), so both the predicted contact frames and the remaining gripper velocity respond to the new arm path before anything is executed. Post-hoc correction edits the arm after the video and the gripper are fixed, and GeoAct leads it by 4.0 points on LIBERO-Long and 5.1 points on bimanual RoboTwin (Tables 1–2). Failures by task stage from simulator states on the 10,000 RoboTwin episodes (5,000 clean and 5,000 randomized), assigning each failed episode once to its first unmet stage:

Failed stage (% of episodes)BasePost-hocGeoAct
Reach / approach6.24.72.8
Grasp / lift7.36.74.5
Transport / place (incl. unwanted contact)4.74.73.7

Total failure falls from 18.2% (Base) and 16.1% (Post-hoc) to 11.0% (GeoAct), with reach plus grasp/lift failures falling from 13.5% and 11.4% to 7.3%, and transport/place failures falling from 4.7% for both comparators to 3.7%.

AuthorsOpenReview ↗

Response to Reviewer nwXL [2/2]

W4. "MINOR: The benchmark checkpoints, readout training data, correction settings, and LIBERO action-to-joint conversion are not fully specified. These choices directly affect the feedback signal and the resulting actions. More detail is needed to reproduce the results."

A4. All three samplers share one checkpoint, one training set, seeds, and initial noise within each benchmark (Sec. 4.1); the full specifications are below and will go into the appendix.

Checkpoint: DreamZero-DROID (14B), adapted separately on public LIBERO demonstrations (1,800 training episodes across 40 tasks; 20,000 optimizer updates; 200 demonstrations held out) and public RoboTwin 2.0 clean_50 + randomized_500 demonstrations (24,750 training episodes across 50 tasks, 45 clean and 450 randomized per task; 40,000 optimizer updates; 2,750 demonstrations held out). Once adapted, the checkpoint stays frozen throughout readout training and all sampler comparisons.

Readouts: architecture of Sec. 3.2 and loss of Eq. 5, trained on 288,000 timestamp draws from the same 1,800 LIBERO training demonstrations and 3,960,000 timestamp draws from the same 24,750 RoboTwin training demonstrations (160 uniform draws with replacement per episode). Accuracy is in A1.

LIBERO conversion: differentiable damped least-squares IK with 20 iterations and damping λ=10−3\lambda=10^{-3}, initialized at the measured joint state; full 6-DoF pose forward kinematics maps the corrected joints back to end-effector pose deltas.

Settings: native 16-step FlowUniPC (sigma_shift=5.0, coupled noise, all 16 steps evaluated); m=2m=2 at zero-based indices 13 and 14 before their solver updates (σa=σV≈\sigma^a=\sigma^V\approx 0.5354 and 0.4164); η\eta = 0.08, δ\delta = 0.10 in normalized action units, β\beta = 1.0, ℓ\ell = 0.10 m, 4 projected gradient steps per intervention (8 total), readout α\alpha = 1.0 after a 2,000-update warmup.

Dear Reviewer, We sincerely appreciate the time and effort you have devoted to reviewing our work. Our detailed rebuttal was submitted several days ago, and we would be grateful if you could take a look when your schedule allows. Please feel free to let us know if any points would benefit from additional explanation.

Reviewer_qpTgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GeoAct improves the video-to-motion generation process using the geometric information of a robot to refine motions during sampling. A learned reader estimates joint configurations from the generated video, and the predicted motions are adjusted to align with these estimates through constrained updates. The generation process then continues using these refined motions, enabling the video to exhibit reactive behavior. Experimental results using LIBERO and RoboTwin 2.0 robots demonstrate superior performance compared to baseline samplers and post-hoc correction methods. Ablation analyses confirm the critical importance of the novel visual feedback mechanism and the use of constrained updates.

Critical strengths
  1. Clear motivation and a novel correction mechanism.

Comparing commanded motions with those described via robot kinematics is meaningful. Modifying motions during the generation process (as opposed to applying corrections post-hoc) allows subsequent video predictions to react accordingly.

  1. The proposed feedback loop has been validated through controlled experiments.

GeoAct improved success rates on LIBERO and on RoboTwin 2.0. The efficacy of interleaved corrections was confirmed by comparing this approach against post-hoc corrections, baselines with an equivalent number of model evaluations, and methods using stale visual feedback. Furthermore, the use of an independent end-effector error metric enhances the reliability of the evaluation.

Critical weaknesses
  1. The advantages beyond the specific backbone chosen remain unclear.

The primary experiments utilize a modified joint video-action checkpoint, without properly identifying or explaining the adaptation process. This rationale should also extend to more robust joint-generative World Action Models (WAMs) (e.g., LingBot-VA (Li et al., 2026) and Motus (Bi et al., 2025)) which generate both video and actions without explicitly enforcing the kinematic consistency proposed here.

LingBot-VA is included in Tables 1–2 through published scores, but GeoAct is not evaluated on it. Since the proposed video–action consistency problem should also be relevant to stronger joint-generation WAMs, it is unclear why the method is tested only on the selected adapted checkpoint.

While outperforming all publicly reported scores is not strictly required, this method introduces a geometric backbone, learned readouts, and a 22% increase in inference latency - yet its practical advantages over more robust alternatives remain unproven.

  1. The validation of the visual readouts that drive these corrections is insufficient.

Although the reported validation of these readouts utilizes real-world video from the DROID dataset, the primary experiments employ separate, implementation-specific readers. There is insufficient reporting on the training data and the accuracy of these readers regarding benchmark observations and intermediate generated frames. Consequently, it remains unclear when these visual estimates provide reliable targets for correction.

While update boundaries serve to constrain error corrections, they do not demonstrate that the video-based estimates are inherently more accurate than the original actions. Although the simulation results are encouraging, the paper lacks an analysis of readout errors and failure modes within the specific distribution where these corrections are applied.

Questions for authors
  1. How are the benchmark-specific readers trained, and how accurate are they on held-out observations and generated frames?

  2. How are LIBERO end-effector commands converted to joint targets and back, particularly when multiple joint configurations correspond to the same end-effector pose?

  3. Please report absolute latency, hardware, correction settings, and uncertainty estimates for the success-rate improvements.

3 comments
AuthorsOpenReview ↗

Response to Reviewer qpTg [1/2]

W1. "The advantages beyond the specific backbone chosen remain unclear. The primary experiments utilize a modified joint video-action checkpoint, without properly identifying or explaining the adaptation process. This rationale should also extend to more robust joint-generative World Action Models (WAMs) (e.g., LingBot-VA (Li et al., 2026) and Motus (Bi et al., 2025)) which generate both video and actions without explicitly enforcing the kinematic consistency proposed here. LingBot-VA is included in Tables 1–2 through published scores, but GeoAct is not evaluated on it. Since the proposed video–action consistency problem should also be relevant to stronger joint-generation WAMs, it is unclear why the method is tested only on the selected adapted checkpoint. While outperforming all publicly reported scores is not strictly required, this method introduces a geometric backbone, learned readouts, and a 22% increase in inference latency - yet its practical advantages over more robust alternatives remain unproven."

A1. GeoAct also improves LingBot-VA on RoboTwin 2.0 without retraining either the world action model or the readout. Reusing the exact frozen VGGT backbone and RoboTwin embodiment readout from our primary experiments with m=2m=2 late-step interventions on LingBot-VA's native flow sampler gives:

RoboTwin 2.0CleanRandomizedAvg.
LingBot-VA, native sampler (our run)92.6491.2091.92
LingBot-VA + Post-hoc93.1091.7892.44
LingBot-VA + GeoAct94.0893.1693.62

As noted in Sec. 2 (Yuan et al., 2026), native WAM samplers make little use of the generated video at test time, so kinematic reconciliation during sampling complements stronger backbones as well. In Tables 1–2, the shared-backbone rows isolate the sampler under fixed weights and data: without changing a single WAM weight, GeoAct closes 3.0 of the 4.0 points to published LingBot-VA on LIBERO and 7.2 of the 10.4 points on RoboTwin, moving the backbone above embodied-pretrained π0.5\pi_{0.5} on LIBERO and above Motus on RoboTwin (89.0% vs. 87.8%). Moreover, a single intervention (m=1m=1, Fig. 2a) already captures 4.8 of the 6.4 points gained on LIBERO-Long at 1.11×1.11\times latency.

The shared backbone in Tables 1–2 is DreamZero-DROID (14B), adapted separately on public LIBERO demonstrations (1,800 training episodes; 20,000 optimizer updates) and public RoboTwin 2.0 clean_50 + randomized_500 demonstrations (24,750 training episodes; 40,000 optimizer updates), then held frozen across readout training and all sampler comparisons.

AuthorsOpenReview ↗

Response to Reviewer qpTg [2/2]

W2 & Q1. "The validation of the visual readouts that drive these corrections is insufficient. Although the reported validation of these readouts utilizes real-world video from the DROID dataset, the primary experiments employ separate, implementation-specific readers. There is insufficient reporting on the training data and the accuracy of these readers regarding benchmark observations and intermediate generated frames. Consequently, it remains unclear when these visual estimates provide reliable targets for correction. While update boundaries serve to constrain error corrections, they do not demonstrate that the video-based estimates are inherently more accurate than the original actions. Although the simulation results are encouraging, the paper lacks an analysis of readout errors and failure modes within the specific distribution where these corrections are applied." / "How are the benchmark-specific readers trained, and how accurate are they on held-out observations and generated frames?"

A2. At the intervention noise levels, median EE error on generated clean predictions is within 0.53 cm of the error on held-out observations across all three settings. The benchmark readouts use the architecture of Sec. 3.2 and the loss of Eq. 5, trained on 288,000 timestamp draws from the same 1,800 public LIBERO training demonstrations and 3,960,000 timestamp draws from the same 24,750 public RoboTwin training demonstrations (160 uniform draws with replacement per episode). We noise held-out benchmark frames to the video noise level σV\sigma^V, take the model's one-step clean prediction V^0\hat V_0, and compare the readout with the simulator joints; this reflects the combined deviation of V^0\hat V_0 and the readout at σV\sigma^V.

Median EE (cm) / normalized joint MAE (Eq. 5)Held-out obs.V^0\hat V_0, σV\sigma^V=0.4164V^0\hat V_0, σV\sigma^V=0.5354V^0\hat V_0, σV\sigma^V=0.7497
LIBERO1.82 / 0.0141.94 / 0.0152.17 / 0.0173.86 / 0.031
RoboTwin (clean) / (rand.)(2.34 / 0.018) / (2.78 / 0.022)(2.49 / 0.019) / (2.96 / 0.024)(2.78 / 0.022) / (3.31 / 0.027)(4.72 / 0.039) / (5.64 / 0.047)

Whereas DROID (Sec. 4.5, 10.55 cm) evaluates a single reader across diverse real-world scenes and camera placements, each simulation reader operates within one benchmark camera setup. Whether the visual target improves the action is measured directly by simulator EE error against task targets, which never uses the readout: in Table 3, moving the action toward the visual target lowers task EE error in every variant—even without the update bound (5.8→5.2 cm, 89.2%→90.8% success)—and reaches 3.5 cm and 95.6% with the proximal bound.


Q2. "How are LIBERO end-effector commands converted to joint targets and back, particularly when multiple joint configurations correspond to the same end-effector pose?"

A3. The conversion is anchored at the measured joint state, which selects the joint solution on the branch continuous with the current arm configuration. Specifically, 6-DoF end-effector pose deltas are mapped to joint targets by differentiable damped least-squares IK with 20 iterations and damping λ=10−3\lambda=10^{-3}, initialized at the measured joint state (Sec. 4.1). Within that branch, the joint term ∥qhA−qhV∥Wq2\|q_h^A - q_h^V\|_{W_q}^2 in Eq. 3 constrains the redundant 7-DoF elbow posture that end-effector position alone leaves free. On the visual side, the bounded tanh⁡\tanh joint head (Eq. 4) excludes 2π2\pi-periodic joint aliases. Full 6-DoF pose forward kinematics then maps the corrected joints back to normalized end-effector pose deltas in the same convention.


Q3. "Please report absolute latency, hardware, correction settings, and uncertainty estimates for the success-rate improvements."

A4. The paired 95% confidence intervals for gains over Base and Post-hoc exclude zero on both benchmarks. On one NVIDIA H100 80 GB GPU (bfloat16, batch size 1, CFG scale 5.0, 16-step FlowUniPC scheduler with sigma_shift=5.0 and all 16 steps evaluated), end-to-end latency per action chunk is 19,420 ms for Base, 20,585 ms for Post-hoc (1.06×1.06\times), 21,556 ms for GeoAct with m=1m=1 (1.11×1.11\times), and 23,692 ms with m=2m=2 (1.22×1.22\times, 1.12×1.12\times peak memory; Sec. 4.4), covering video decoding, VGGT readout, proximal optimization, joint field reevaluation, and solver restart under identical cache settings.

Settings: m=2m=2 at zero-based indices 13 and 14 before their solver updates (σa=σV≈\sigma^a=\sigma^V\approx 0.5354 and 0.4164); η\eta = 0.08, δ\delta = 0.10 in normalized action units, β\beta = 1.0, ℓ\ell = 0.10 m, 4 projected gradient steps per intervention (8 total). Paired 95% confidence intervals over 2,000 LIBERO and 10,000 RoboTwin episodes:

Gain (percentage points)LIBERO (2,000 ep.)RoboTwin (10,000 ep.)
GeoAct vs. Base+3.0 [2.08, 3.90]+7.2 [6.57, 7.82]
GeoAct vs. Post-hoc+1.9 [1.11, 2.75]+5.1 [4.50, 5.68]

Dear Reviewer, Thanks again for your constructive feedback. We submitted our detailed rebuttal several days ago and would be grateful if you could have a look when convenient. Please let us know if anything remains unclear or requires extra clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.