- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ForeAct3D augments a vision-language-action policy with auxiliary prediction of current and future depth, three-class semantic masks, and wrist-camera pose, with future queries conditioned on the policy-generated action chunk. Background staticity, instance-level rigid alignment, and kinematic pose supervision regularize the shared policy representation; future predictions are not required at inference. Without robot pretraining, the method reports 98.3% average success on LIBERO, 3.73 average task length on CALVIN ABC→D, and 37.8% average success across three real-world tasks.
Critical strengths
-
Tables 1–2 report 96.6% → 98.3% on LIBERO and 2.94 → 3.73 average task length on CALVIN, with matched training configurations stated in Sections 4.1 and 4.3. Evaluation covers 2,000 LIBERO episodes and 1,000 CALVIN sequences, with CALVIN testing on unseen environment D (Appendix A.3), providing useful evidence across both task diversity and environmental shift.
-
The real-world experiments corroborate the overall benefit with explicit outcome definitions, where with 50 demonstrations per task, total successes increase from 3/45 to 17/45 across placement, insertion, and sequential manipulation. Distinguishing target entry from final retention, and intermediate completion from full-task success, makes the practical improvement more informative than a single aggregate success rate.
Critical weaknesses
-
Sections 3.1 and 3.4 condition forecasting on policy-generated actions. But they do not establish whether future targets come from executing those actions or from the recorded demonstration, nor whether forecasting gradients pass through the action output. If a demonstration endpoint supervises a different generated chunk, the target need not represent that chunk’s consequence; Tables 4(b) and 9 demonstrate the usefulness of the action-feature pathway without resolving this issue or testing different executed actions from the same state. This is a substantial, potentially fixable gap in the central scientific interpretation.
-
Equation (3) and Appendix A.1 fit Kabsch transforms between identical pixel coordinates sharing an instance ID, although these generally need not observe the same material point after motion. Instances with fewer than 32 overlapping pixels are excluded. The authors acknowledge this approximation, but provide no moving-instance coverage or correspondence-sensitive validation showing that the loss preserves rigid geometry under the evaluated motions.
-
Appendix A.4 selects predicted and target points at ground-truth future instance pixels, thereby supplying the object’s future image support, and excludes instances with insufficient valid predicted depth from recall without penalty. The predicted-mask variant still retains ground-truth instance IDs and lacks established identical filtering across conditions (Appendix A.5). On CALVIN, its reported error rises from 1.25 to 3.49 cm and recall falls from 92.8% to 65.9% (Table 8). These transparently disclosed qualifications substantially limit how strongly the localization scores validate the proposed future-state representatio.
-
The contribution beyond established geometric auxiliary learning is insufficient. DreamVLA already uses query-based future depth and semantic forecasting with training-only decoders, while GeoPredict combines predictive robot kinematics and 3D geometry with training-only geometric decoding.
-
No training-seed variation is reported, although the LIBERO closure and action-conditioning increments are only 0.6 and 0.4 percentage points; each real-world cell contains 15 trials, and the long-task action-conditioning gain is 1/15 → 3/15.
Questions for authors
-
What actions generate each future training target, and how are discrepancies between recorded and policy-generated chunks handled?
-
What fraction of moving instances receives the rigidity penalty, and does it improve geometric consistency when measured using valid temporal correspondences? Does the kinematic term impose any constraint beyond increasing the coefficient of the existing labeled pose loss?
-
Are forecasting comparisons evaluated on identical held-out observation/action/target windows? Can the main action-conditioning comparison report fixed instance populations, excluded-instance coverage, and performance relative to copying the current scene, especially for objects that move?
-
How does a matched current-only depth/semantic/pose auxiliary baseline compare with current-plus-future supervision?
-
What are the exact backbone checkpoint and action-head configuration, and how stable are the principal policy and component gains across training seeds?