- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Act4R addresses visual action-to-motion calibration for robots without manually specified kinematic models. Given a current image and an control action, the framework predicts current and future 3D pointmaps, from which it derives action-conditioned scene flow. Inverse control is then performed by optimizing the action through the trained model to match a desired 2D or 3D motion. The evaluation covers four real-world robot datasets and three simulated embodiments. The paper reports improvements over Neural Jacobian Fields in scene-flow prediction and offline action recovery, together with reference-image-guided closed-loop control on three simulated robots.
Critical strengths
S1. The framework connects a pretrained 4D reconstruction model to actuator commands, producing a dense spatial prediction that can be differentiated for inverse control. This is an appropriate formulation for robots whose analytical kinematics are difficult to specify, including compliant and unconventional platforms. Conditioning on actions also avoids requiring robot point-flow actions to be constructed from known forward kinematics.
S2. Act4R improves the reported 3D flow error over NJF on all seven platforms, including all four real-world datasets, rather than obtaining its aggregate gain from only one robot. The VGGT-depth variant further provides encouraging evidence for reducing measured-depth requirements for 2D-target inversion.
Critical weaknesses
W1.The closed-loop evaluation does not sufficiently establish the claimed control advantage. Section 4.3 repeatedly applies corrective actions until the observation is sufficiently close to each reference waypoint, while Eq. (11) measures joint error upon reaching that waypoint. However, the maximum correction budget and action-selection latency are unspecified. Consequently, the reported CJPE cannot distinguish accurate, efficient control from endpoint accuracy obtained through extensive feedback correction.
W2.The existing ablations primarily examine spatial conditioning, feature alignment, and geometric supervision, leaving two architectural aspects underexplored: the independent contribution of gated value fusion within the two-stage cross-attention structure, and the overall advantage of the custom SAE over conventional action-conditioned future-feature predictors. In particular, the Gr00t-style baseline simultaneously changes spatial conditioning and the fusion mechanism within the action encoder. Consequently, the reported results do not clearly distinguish the general benefits of spatial–action interaction from the incremental gains attributable to SAE’s specific architectural choices.
Questions for authors
Q1. What is the independent contribution of SAE’s gated value fusion when the two-stage spatial–action cross-attention structure is preserved? The Gr00t-style baseline simultaneously changes spatial conditioning and the fusion mechanism within the action encoder. What evidence distinguishes the general benefits of spatial–action interaction from the incremental gains attributable specifically to gated value fusion?
Q2. How does Act4R advance beyond a conventional action-conditioned latent world model equipped with the same geometric decoder? Given that Act4R predicts future spatial features from the current observation and actuator commands, what evidence establishes the advantage of its custom SAE over a standard future-feature predictor under matched initialization and supervision? How can the benefits of the geometric representation be distinguished from those of SAE’s specific architecture?
Q3. What additional benefit does explicit 3D prediction provide for the paper’s 2D-goal inverse-control tasks? Under matched observations, action data, visual priors, and motion targets, how does Act4R compare with an image-space Jacobian controller such as VERA’s J-IDM[1] in action recovery, closed-loop performance, and computational cost? How do the existing ablations distinguish the benefit of 3D supervision from that of the 3D representation itself?
[1]Turning Video Models into Generalist Robot Policies(https://arxiv.org/pdf/2605.27817)
7 comments
Response to W2/Q1: Contribution of SAE Architecture
It is unclear to us if specific literature is referred to by “conventional action-conditioned future-feature predictor” - as far as we know, NJF is the only peer-reviewed work on predicting future robotic motion explicitly in 3D. We note that our SAE is specifically designed for recovering unknown kinematics. We also want to clarify that the design of SAE is very lightweight and minimum, consisting of only two cross-attention blocks. While the gated fusion design is explained in the architecture, the general idea of gated fusion is not unique to SAE; e.g., Gated Multimodal Units (Arevalo et al. ) is a predecessor. Our main contribution is to propose a mapping that converts current spatial features + actuator commands to predicted future spatial features compatible with the pretrained geometric decoder. As such, we do not make a distinction between the novelty of the SAE formulation and the gated value fusion design, nor do we ablate on gated fusion design. We did demonstrate that simpler designs, e.g., a Gr00t-like design, which does not integrate image information as an input, leads to worse performance, as shown in Tables 1–4 in the manuscript. We will add clarification and appropriate citation to clarify that our claimed novelty is the mapping formulation and not the exact gated fusion mechanism.
Response to Q2: How does Act4R advance beyond a conventional action-conditioned latent world model equipped with the same geometric decoder?
We appreciate the reviewer's suggestion. However, we are not aware of a “standard action-conditioned future-feature predictor” that directly addresses our setting of mapping raw robot actions into spatial embeddings compatible with a pretrained 3D reconstruction decoder. Could the reviewer kindly clarify which specific architecture or prior work they are referring to? In general, many action-conditioned world models provide plausible visual states, but do not provide accurate spatial predictions, e.g., 4D tracking.
Response to W1: maximum correction budget and action-selection latency are unspecified
We thank the reviewer for recognizing Act4R's formulation and focus for robots whose analytical kinematics are difficult to specify, Act4R's improvement over NJF on all seven platforms, and the discovery that VGGT-based depth is often useful enough for real-world inverse kinematics mapping. We seek to address the reviewer's remaining concerns here, and also seek to get some clarification on the questions so we can better respond.
W1. Maximum Correction Budget and Action-Selection Latency are Unspecified We thank the reviewer for raising this concern. We first provide additional details on computing action latency and then the correction budget based on action latency:
Our work currently focuses on validating the feasibility of using vision-based approach for accurate and high degree-of-freedom inverse control; action latency is a necessary step in the future, but is not the focus. Since Act4R relies on inverse optimizing through a large backbone, the practical speed is about 10X slower than NJF, which extracts the linear Jacobian approximation and optimizes the linear equations only. The exact computational speed is ~16.93ms for Act4R v ~1.81ms for NJF for each optimization iteration, measured on an RTX Pro 6000 Blackwell GPU. While NJF is faster, their first order Jacobian approximation is insufficient to describe complex robotic movement.
To this end, we designed our correction budget based on each method's latency to be fair and consistent. To avoid extremely long optimization, we set up an empirical threshold with the following mechanism: if Act4R/NJF reach the target threshold within fair correction budget, then we will skip the remaining optimization step; if Act4R/NJF does not reach the target threshold within this budget, we will simply move towards the next waypoint. CJPE records the cumulative error regardless of whether the threshold is reached. In practice, most of the correct budget is spent as we manually tuned it to be efficient; however, we understand that this can provide confusion in terms of consistent comparison.
To this end, we performed an exact experiment where Act4R uses exactly 10 optimization step and NJF uses 100 optimization steps for close-loop control, eliminating the variation caused by a manually defined threshold. As shown in following table, even when NJF optimizes for 10X more iterations, the cumulative joint error for closed-loop tasks is still significantly worse than Act4R (and we do not find the error to reduce even with more iterations). The results are reported using CJPE (↓), which is described in the paper. This lack of accuracy, even with many iterations, is the main issue we want to tackle.
There are various engineering improvements we can add on top of Act4R to significantly speed up inverse control and reach real-time control. One obvious direction is to avoid computation of 3D point-clouds over the entire frame, and only estimate 3D movements over user-provided regions. In NJF, only 200 selected points are used for inverse control, while Act4R estimates thousands of points uniformly sampled from the entire image. We look to build on Act4R in the future to address the real time control better.