- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ActForge is a multi-view robot video world model trained across four datasets and five arm types. It first refines camera geometry from existing recordings and represents actions using calibrated robot renderings and camera-ray features. It then post-trains on recorded and counterfactual action conditions using a video reward model learned from human annotations of embodiment, object, and interaction defects. On AgiBot, action following reaches an nDTW of 0.8772, while post-training reduces human-assessed interaction defects from 48.12% to 6.25%. An additional RoboTwin study reports improved prediction of policy outcomes. The work addresses useful problems in action conditioning and interaction modeling, but its strongest evidence concerns visual consistency rather than independently verified counterfactual dynamics.
Critical strengths
- The geometric calibration component addresses a concrete obstacle to learning from heterogeneous robot data. Misalignment between rendered action conditions and recorded observations can directly undermine action following. The paper evaluates calibration at substantial scale and separates inference-only refinement from refinement during both training and inference. The resulting improvements in trajectory alignment and reconstruction support the practical value of this component, and the proposed release of calibration parameters could benefit subsequent work.
- The human evaluation provides credible evidence of reduced visible defects. All 960 counterfactual test videos receive three independent, blinded assessments, with agreement statistics and confidence intervals reported in the appendix. The reduction from 48.12% to 6.25% in interaction defects relative to Stage I is substantial. The study also includes recorded-action fidelity, a separate simulated outcome-prediction evaluation, and a large annotation resource, giving the contribution value beyond selected qualitative examples.
Critical weaknesses
- The evaluation does not sufficiently establish physical fidelity under counterfactual controls. The reward model receives only the generated video and a fixed prompt, without the commanded action condition (Appendix B.1). It can therefore encourage visibly coherent interactions without directly verifying that they are the outcomes of the supplied actions. The AgiBot counterfactual benchmark likewise lacks paired physical futures, and nDTW evaluates gripper-trajectory agreement rather than object-state correctness. The RoboTwin policy-outcome study is useful, but covers three tasks and replays complete action sequences obtained from simulator executions; it does not test closed-loop policies acting on generated observations. The real-world OOD examples are predictions from recordings rather than controlled hardware interventions. Consequently, the evidence supports improved visual plausibility more strongly than the broader use of the model as a reliable physical simulator.
- The experiments do not isolate the claimed contribution of counterfactual feedback. Stage II jointly introduces counterfactual conditions, multiple defect rewards, and recorded-action PSNR rewards, but no matched recorded-action-only post-training control or reward-channel ablation establishes which changes produce the gains. The policy-outcome study compares the full model with Ctrl-World without including Stage I, so it also cannot attribute the downstream improvement to counterfactual post-training. Furthermore, the strongest external baseline already has a 7.50% interaction-defect rate versus 6.25% for ActForge, with overlapping reported confidence intervals; this requires the paired comparison rather than a claim based on the large Stage-I reduction. These gaps leave the distinctive benefit of the proposed training strategy insufficiently established, leading me to weak rejection.
Questions for authors
- Can you validate predicted object outcomes against actual executions of matched counterfactual actions, including near-miss grasps and partial contacts? A controlled hardware subset would clarify the relevance of the visual defect results to real interactions.
- Under matched data and optimization budgets, how do recorded-action-only post-training, recorded-plus-counterfactual post-training, and removal of individual reward channels compare?
- Can you include Stage I in the policy-outcome evaluation and report the paired statistical comparison with GE-Sim 2.0 on interaction defects? These controls would help separate the value of calibration, general post-training, and counterfactual feedback.
2 comments
Response to Reviewer qUPG (Part 2)
Response to Reviewer qUPG (Part 2)
Q3: Stage-I policy-outcome evaluation and paired comparison with GE-Sim 2.0.
Can you include Stage I in the policy-outcome evaluation and report the paired statistical comparison with GE-Sim 2.0 on interaction defects? These controls would help separate the value of calibration, general post-training, and counterfactual feedback.
We have added Stage I to the policy-outcome evaluation and expanded the interaction-defect comparison with GE-Sim 2.0. The Stage-I comparison supports improved outcome prediction and success-rate calibration after post-training; the paired analysis reports both the defect differences and their statistical strength.
(1) Policy-outcome evaluation with Stage I.
Compared with Stage I, post-training brings predicted outcomes into closer agreement with actual policy executions and reduces overestimation of policy success. We evaluate Stage I and the full model on the same 240 policy trajectories across 24 task-policy-setting combinations in RoboTwin. Both models share the same calibrated Stage-I starting point, so this comparison measures the incremental benefit of the complete Stage-II training recipe (Figure 8 and Table 12).
| Method | Success-rate MAE (pp) ↓ | Signed bias (pp) | Outcome accuracy (%) ↑ | F1 ↑ | Pearson r ↑ |
|---|---|---|---|---|---|
| Ours (w/o RL) | 9.17 | +2.50 | 87.50 | 0.848 | 0.936 |
| Ours (full) | 7.08 | +0.42 | 90.42 | 0.881 | 0.964 |
Post-training reduces success-rate MAE by 22.8% and brings the aggregate prediction bias closer to zero, indicating more accurate estimates and less overestimation of policy success. Outcome accuracy and F1 also improve. These results assess agreement with actual simulator outcomes under fixed action trajectories.
(2) Paired interaction-defect comparison with GE-Sim 2.0.
ActForge has fewer interaction defects than GE-Sim 2.0 under both annotation criteria, including half as many unanimously identified defects. We report two-sided exact McNemar tests on 160 matched conditions, each evaluated independently by three annotators. We compare both majority-vote and unanimous defect labels. Discordant pairs are conditions where only one model is labeled defective.
| Defect criterion | ActForge | GE-Sim 2.0 | Discordant pairs favoring ActForge / GE-Sim 2.0 | Exact p-value |
|---|---|---|---|---|
| Majority vote (≥2/3) | 10/160 (6.25%) | 12/160 (7.50%) | 9 / 7 | 0.804 |
| Unanimous (3/3) | 4/160 (2.50%) | 8/160 (5.00%) | 7 / 3 | 0.344 |
The paired tests do not establish statistically significant superiority on these 160 conditions.
We thank you again for the helpful suggestions and hope that these additional results and clarifications address your concerns.
Response to Reviewer qUPG (Part 1)
Response to Reviewer qUPG (Part 1)
We thank you for the constructive suggestions on validating predicted outcomes and separating the effects of post-training. We added controlled real-world validation, training ablations, and a Stage-I policy-outcome comparison, and report the paired defect analysis with GE-Sim 2.0. We address Q1-Q2 below and Q3 in Part 2.
Q1: Real-world validation under matched counterfactual actions.
We added a controlled hardware evaluation to compare predicted lift outcomes with actual executions under the same perturbed actions (Table 4). The results support more accurate prediction of action-dependent grasp outcomes after post-training, particularly when the robot fails to lift the object. We evaluate three tasks, each containing 10 lift and 10 non-lift cases, for a total of 60 cases. The cases cover failure modes including missed grasps and partial contacts. Different action variants are executed from matched initial states, and both models are evaluated on exactly the same cases.
The model without RL incorrectly predicts lifting in 12 of the 30 actual non-lift cases. The full model correctly predicts all 30 non-lift cases and 29 of the 30 lift cases. We also provide qualitative examples of contact-induced object displacement when the gripper contacts and pushes objects during closure (Figures 17 and 18). The controlled outcome comparison and contact examples provide execution-based evidence supporting the physical relevance of the visual-defect improvements.
Q2: Recorded-only training, counterfactual training, and reward ablations.
Thank you for suggesting these controls. We added two recorded-action-only variants to clarify the roles of reward supervision and counterfactual action coverage (Table 8). Embodied feedback improves interaction quality on recorded actions, with substantially greater gains when this feedback is extended to counterfactual actions, while preserving gripper action following.
All RL variants use the same source recordings, Stage-I checkpoint, training steps, batch size, and other configurations, differing only in action sampling and rewards. All models are evaluated on the same 160 counterfactual conditions. The embodied reward combines robot-embodiment, object-consistency, and interaction-plausibility feedback; PSNR applies only to recorded actions with paired ground-truth futures. These controls isolate the embodied reward as a whole, rather than the individual effects of its three channels.
Embodiment defects are 0.00% throughout.
(1) Embodied feedback. On recorded actions, adding embodied feedback to PSNR reduces object defects from 30.00% to 24.38% and interaction defects from 45.00% to 42.50%, supporting its benefit under fixed action coverage.
(2) Counterfactual training. With the reward definitions fixed, extending training to counterfactual actions further reduces object defects from 24.38% to 3.12% and interaction defects from 42.50% to 6.25%, while maintaining gripper-trajectory nDTW (0.8792 → 0.8831). This supports the value of applying embodied feedback to a broader range of actions and contact configurations.