acceptodds
Under review as a conference paper at ICLR 2027

Act4R: Action-Conditioned 4D Reconstruction for Kinematics-Free Control

Abstract

Precise control of movement across various robotic embodiments is a longstanding challenge. Modeling complex robotic structures requires significant time, dedicated sensors, and manual labeling. Recent advancements in 4D representation learning offer an interesting alternative: can we leverage 4D features lifted from 2D observations to densely map a robot's spatial movement based on the action signal? To this end, we propose Act4R, a framework that builds on 3D foundation model to predict a robot's 3D scene flow with any given action. Act4R introduces a Spatial-Action Encoder, which encodes the current 2D robot state and its action signals into embeddings that represent future robot state in 3D. Then, both the current and future spatial embeddings are decoded into point maps to estimate action-conditioned scene flow. To effectively learn the mapping between action and future state, Act4R introduces spatial-action fusion in its transformer-based encoder, and a spatial alignment loss that guides the encoder for future flow prediction. After training on some samples of a given embodiment, Act4R can perform inverse control by optimizing the signal to achieve the desired scene flow. We evaluate Act4R on a public real-world dataset and synthetic data collected in simulator. Experiments on various robotic platforms show that Act4R outperforms the previous SoTA in scene flow prediction. Our method also performs better in inverse control, both on the action accuracy metric and in closed-loop control in simulation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Robotic Manipulation, Vision Foundation Model, Computer Vision

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_tXywOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Act4R addresses visual action-to-motion calibration for robots without manually specified kinematic models. Given a current image and an control action, the framework predicts current and future 3D pointmaps, from which it derives action-conditioned scene flow. Inverse control is then performed by optimizing the action through the trained model to match a desired 2D or 3D motion. The evaluation covers four real-world robot datasets and three simulated embodiments. The paper reports improvements over Neural Jacobian Fields in scene-flow prediction and offline action recovery, together with reference-image-guided closed-loop control on three simulated robots.

Critical strengths

S1. The framework connects a pretrained 4D reconstruction model to actuator commands, producing a dense spatial prediction that can be differentiated for inverse control. This is an appropriate formulation for robots whose analytical kinematics are difficult to specify, including compliant and unconventional platforms. Conditioning on actions also avoids requiring robot point-flow actions to be constructed from known forward kinematics.

S2. Act4R improves the reported 3D flow error over NJF on all seven platforms, including all four real-world datasets, rather than obtaining its aggregate gain from only one robot. The VGGT-depth variant further provides encouraging evidence for reducing measured-depth requirements for 2D-target inversion.

Critical weaknesses

W1.The closed-loop evaluation does not sufficiently establish the claimed control advantage. Section 4.3 repeatedly applies corrective actions until the observation is sufficiently close to each reference waypoint, while Eq. (11) measures joint error upon reaching that waypoint. However, the maximum correction budget and action-selection latency are unspecified. Consequently, the reported CJPE cannot distinguish accurate, efficient control from endpoint accuracy obtained through extensive feedback correction.

W2.The existing ablations primarily examine spatial conditioning, feature alignment, and geometric supervision, leaving two architectural aspects underexplored: the independent contribution of gated value fusion within the two-stage cross-attention structure, and the overall advantage of the custom SAE over conventional action-conditioned future-feature predictors. In particular, the Gr00t-style baseline simultaneously changes spatial conditioning and the fusion mechanism within the action encoder. Consequently, the reported results do not clearly distinguish the general benefits of spatial–action interaction from the incremental gains attributable to SAE’s specific architectural choices.

Questions for authors

Q1. What is the independent contribution of SAE’s gated value fusion when the two-stage spatial–action cross-attention structure is preserved? The Gr00t-style baseline simultaneously changes spatial conditioning and the fusion mechanism within the action encoder. What evidence distinguishes the general benefits of spatial–action interaction from the incremental gains attributable specifically to gated value fusion?

Q2. How does Act4R advance beyond a conventional action-conditioned latent world model equipped with the same geometric decoder? Given that Act4R predicts future spatial features from the current observation and actuator commands, what evidence establishes the advantage of its custom SAE over a standard future-feature predictor under matched initialization and supervision? How can the benefits of the geometric representation be distinguished from those of SAE’s specific architecture?

Q3. What additional benefit does explicit 3D prediction provide for the paper’s 2D-goal inverse-control tasks? Under matched observations, action data, visual priors, and motion targets, how does Act4R compare with an image-space Jacobian controller such as VERA’s J-IDM[1] in action recovery, closed-loop performance, and computational cost? How do the existing ablations distinguish the benefit of 3D supervision from that of the 3D representation itself?

[1]Turning Video Models into Generalist Robot Policies(https://arxiv.org/pdf/2605.27817)

7 comments
AuthorsOpenReview ↗

Response to W1: maximum correction budget and action-selection latency are unspecified

We thank the reviewer for recognizing Act4R's formulation and focus for robots whose analytical kinematics are difficult to specify, Act4R's improvement over NJF on all seven platforms, and the discovery that VGGT-based depth is often useful enough for real-world inverse kinematics mapping. We seek to address the reviewer's remaining concerns here, and also seek to get some clarification on the questions so we can better respond.

W1. Maximum Correction Budget and Action-Selection Latency are Unspecified We thank the reviewer for raising this concern. We first provide additional details on computing action latency and then the correction budget based on action latency:

Our work currently focuses on validating the feasibility of using vision-based approach for accurate and high degree-of-freedom inverse control; action latency is a necessary step in the future, but is not the focus. Since Act4R relies on inverse optimizing through a large backbone, the practical speed is about 10X slower than NJF, which extracts the linear Jacobian approximation and optimizes the linear equations only. The exact computational speed is ~16.93ms for Act4R v ~1.81ms for NJF for each optimization iteration, measured on an RTX Pro 6000 Blackwell GPU. While NJF is faster, their first order Jacobian approximation is insufficient to describe complex robotic movement.

To this end, we designed our correction budget based on each method's latency to be fair and consistent. To avoid extremely long optimization, we set up an empirical threshold with the following mechanism: if Act4R/NJF reach the target threshold within fair correction budget, then we will skip the remaining optimization step; if Act4R/NJF does not reach the target threshold within this budget, we will simply move towards the next waypoint. CJPE records the cumulative error regardless of whether the threshold is reached. In practice, most of the correct budget is spent as we manually tuned it to be efficient; however, we understand that this can provide confusion in terms of consistent comparison.

To this end, we performed an exact experiment where Act4R uses exactly 10 optimization step and NJF uses 100 optimization steps for close-loop control, eliminating the variation caused by a manually defined threshold. As shown in following table, even when NJF optimizes for 10X more iterations, the cumulative joint error for closed-loop tasks is still significantly worse than Act4R (and we do not find the error to reduce even with more iterations). The results are reported using CJPE (↓), which is described in the paper. This lack of accuracy, even with many iterations, is the main issue we want to tackle.

DatasetNJF 100 iterations (181 ms)Act4R 10 iterations (169 ms)
Franka10.8626.872
Allegro-sim28.6869.574
UR5e10.2575.857

There are various engineering improvements we can add on top of Act4R to significantly speed up inverse control and reach real-time control. One obvious direction is to avoid computation of 3D point-clouds over the entire frame, and only estimate 3D movements over user-provided regions. In NJF, only 200 selected points are used for inverse control, while Act4R estimates thousands of points uniformly sampled from the entire image. We look to build on Act4R in the future to address the real time control better.

AuthorsOpenReview ↗

Response to W2/Q1: Contribution of SAE Architecture

It is unclear to us if specific literature is referred to by “conventional action-conditioned future-feature predictor” - as far as we know, NJF is the only peer-reviewed work on predicting future robotic motion explicitly in 3D. We note that our SAE is specifically designed for recovering unknown kinematics. We also want to clarify that the design of SAE is very lightweight and minimum, consisting of only two cross-attention blocks. While the gated fusion design is explained in the architecture, the general idea of gated fusion is not unique to SAE; e.g., Gated Multimodal Units (Arevalo et al. ) is a predecessor. Our main contribution is to propose a mapping that converts current spatial features + actuator commands to predicted future spatial features compatible with the pretrained geometric decoder. As such, we do not make a distinction between the novelty of the SAE formulation and the gated value fusion design, nor do we ablate on gated fusion design. We did demonstrate that simpler designs, e.g., a Gr00t-like design, which does not integrate image information as an input, leads to worse performance, as shown in Tables 1–4 in the manuscript. We will add clarification and appropriate citation to clarify that our claimed novelty is the mapping formulation and not the exact gated fusion mechanism.

AuthorsOpenReview ↗

Response to Q2: How does Act4R advance beyond a conventional action-conditioned latent world model equipped with the same geometric decoder?

We appreciate the reviewer's suggestion. However, we are not aware of a “standard action-conditioned future-feature predictor” that directly addresses our setting of mapping raw robot actions into spatial embeddings compatible with a pretrained 3D reconstruction decoder. Could the reviewer kindly clarify which specific architecture or prior work they are referring to? In general, many action-conditioned world models provide plausible visual states, but do not provide accurate spatial predictions, e.g., 4D tracking.

Reviewer_roiMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  1. 3D Motion Prediction: Leveraging a pretrained 3D vision model, a Spatial-Action Encoder (SAE) is designed to integrate current image data with action signals, enabling the prediction of a robot's 3D motion.
  2. Spatial Feature Alignment: Predicted future features are aligned with the actual features of the subsequent frame, utilizing the geometric priors embedded in the pretrained model.
  3. Simulation Benchmark: Simulation data for three distinct robot morphologies is collected to establish a unified evaluation protocol for control tasks involving unknown kinematics.
Critical strengths
  1. Effective utilization of pretrained geometric priors: Leveraging 3D vision models for action conditioned motion prediction through SAE (Spatial-Action Encoder) and the alignment of future spatial features.
  2. Support for dense motion prediction and inverse control: Predicting the 3D motion of points across a robot’s surface, thereby capturing local deformations, and deriving the necessary actions to achieve a target motion through inverse optimization.
Critical weaknesses
  1. Real time control capability is not yet clear: since action inference requires iterative optimization, the lack of a systematic analysis regarding the number of optimization iterations, control latency, and convergence stability leaves the framework’s real time control capability uncertain.
Questions for authors

The method relies on iterative optimization to determine control actions, but the paper does not explicitly report the number of optimization iterations, control latency, or sensitivity to initialization. The following experiments are recommended:

  1. Control efficiency experiment: Compare control latency and trajectory error across different numbers of optimization iterations to analyze the trade-off between accuracy and efficiency.
  2. Convergence stability experiment: Perform repeated optimizations using different initializations for the control actions, and report the mean and standard deviation of the final error to evaluate the stability of the solution process.
4 comments
AuthorsOpenReview ↗

Response to Weakness: Control Efficiency and Real-Time Capability

We thank the reviewer for pointing out the real time control capability of our method. Our work currently, similar to NJF, focuses on validating the feasibility of using vision-based approach for accurate and high degree-of-freedom inverse control; real time control capability is a necessary step in the future, but is not the focus. We elaborate on the reasoning below:

Since Act4R relies on inverse optimizing through a large backbone, the practical speed is about 10X slower than NJF, which extracts the linear Jacobian approximation and optimizes the linear equations only. The computational speed is ~16.93ms for Act4R v ~1.81ms for NJF for each optimization iteration. While NJF is faster, their first order Jacobian approximation is insufficient to describe complex robotic movement.

As shown in following table, even if NJF optimizes for 10X more iterations, the cumulative joint error for closed-loop tasks is still significantly worse than Act4R. The results are reported using CJPE (↓), which is described in the paper.

DatasetNJF 100 iterations (181 ms)Act4R 10 iterations (169 ms)
Franka10.8626.872
Allegro-sim28.6869.574
UR5e10.2575.857

There are various engineering improvements we can add on top of Act4R to significantly speed up inverse control. One obvious direction is to avoid computation of 3D point-clouds over the entire frame, and only estimate 3D movements over user-provided regions. In NJF, only 200 selected points are used for inverse control, while Act4R estimates thousands of points uniformly sampled from the entire image. We look to build on Act4R in the future to address the real time control better; for now, our work seeks to validate accurate control first.

AuthorsOpenReview ↗

Q1. Control Accuracy-Efficiency Trade-off

Q1. Control accuracy-efficiency tradeoff experiment

In additional to speed, the ablation study results on the relationship between optimization iterations and control accuracy are shown below. We report cosine similarity (↑) as the evaluation metric, where higher values indicate better performance.

Optimization iterations510203550
Allegro0.6070.6790.7230.7390.727
Franka0.4090.3990.4110.3540.308
HSA0.4920.4930.4890.4460.387
Toy Arm0.5140.5360.6240.6130.525
Pneumatic0.3420.3760.3760.3530.325
Allegro-sim0.4970.4750.4250.4380.423
UR5e0.6630.6230.6200.6060.541

Overall, different robot embodiment require different numbers of optimization iterations to achieve their best performance. We choose 10 iterations as a practical trade-off between control accuracy and computational efficiency and use this setting throughout our experiments.

AuthorsOpenReview ↗

Q2. Convergence Stability under Different Initializations

Q2. Convergence stability experiment

We evaluate the effect of different initializations on control accuracy. We report cosine similarity (↑) as the evaluation metric, where higher values indicate better performance. The results for random initialization are averaged over 10 different initialization values.

DatasetZero initRandom initRandom init std
Allegro0.6790.671±0.0116
Franka0.3990.426±0.0160
HSA0.4930.502±0.0087
Toy Arm0.5360.579±0.0082
Pneumatic0.3760.367±0.0086
Allegro-sim0.4750.483±0.0140
UR5e0.6230.548±0.0280

The results show that random initialization achieves overall comparable performance to zero initialization, suggesting that Act4R is relatively robust to different initialization choices. These variations do not change our main conclusions.

Dear Reviewer roiM,

Thank you for taking the time to review our work. As the discussion period is coming to an end, we hope our rebuttal and additional experiments have helped address your concerns. If you have any remaining questions or concerns, we would be happy to provide further clarification. If you find our responses convincing and have no remaining concerns, we would sincerely appreciate it if you could consider revisiting your initial score.

Thanks again for your time and consideration!

Reviewer_rZZPOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Act4R, an action-conditioned 4D reconstruction framework for kinematics-free robotic control. The method leverages a pretrained 3D foundation model and introduces a Spatial-Action Encoder to predict future 3D scene flow from the current image and an action command. The predicted flow is then used for inverse control through action optimization. Experiments on four real-world and three simulated robotic platforms demonstrate improvements over Neural Jacobian Fields (NJF) in scene-flow prediction and inverse-action estimation.

Critical strengths

The paper addresses an important problem: controlling robots without known kinematic models. This is especially useful for soft robots, where accurate modeling is difficult.

Act4R consistently outperforms NJF in 3D scene-flow prediction and inverse-action estimation across different robotic platforms, demonstrating the effectiveness of the proposed method.

Critical weaknesses

W1. Act4R learns the action-to-motion mapping from robot-specific trajectories. This is particularly concerning for Pneumatic, where the deformation space can be highly complex. However, the paper provides limited details about how the training action sequences are generated and whether they sufficiently cover the robot's motion space. More importantly, the paper does not evaluate how well the model generalizes to OOD trajectories, such as unseen robot configurations, action combinations, or deformation patterns, which could be particularly problematic

W2. Although the paper evaluates scene-flow prediction and inverse-action estimation on real-world datasets, closed-loop control is demonstrated only in simulation. This limits the evidence supporting the practical effectiveness of Act4R, especially for robots that mentioned in the paper with complex deformation and actuator uncertainty.

W3. Act4R relies on CoTracker-generated correspondences to construct supervision. However, these correspondences may be inaccurate under occlusion, large deformation, or significant viewpoint changes. The paper does not sufficiently investigate how tracking errors or missing correspondences affect the quality of the training data and the robustness of the learned action-to-motion mapping.

Questions for authors

I would appreciate it if the authors could address the concerns and questions raised in the Weaknesses section above.

4 comments
AuthorsOpenReview ↗

Response to W1: Motion Space Coverage and Generalization

We thank the reviewer for recognizing the importance of inverse soft robotic control and novelties introduced by Act4R. We address the concerns and questions listed by the reviewer below:

W1.1. Details about how the training action sequences are generated and whether they sufficiently cover the robot's motion space.

We use two datasets in our paper. The real-world dataset from NJF has been demonstrated to generalize to novel views and novel configurations by NJF’s implementation. In terms of motion space coverage: as shown in the following table, NJF’s dataset covers almost 100% of all possible robotic action ranges (configured by a safety threshold) for each range. We also provide additional metrics such as the median and 95th-percentile joint displacements between two actions as the percentage of maximum possible displacement of the joints, as a metric of speed. As shown in the table, there is a wild range of joint displacement between actions. Our Omniverse simulation dataset follows the protocol from NJF, and exhibits a similar range of movement, with a slightly smaller motion range.

We also note that every robot and robotic joint have different patterns of movement. As such, the definition of sufficiency is not well defined. The general acquisition pattern is that several joints move together randomly at one time, and sufficient initialization of joints are collected over the safe action space. We are happy to follow up with additional experiments/metrics if reviewer sees fit.

DatasetActive dimensionsJoint motion coverageMedian absolute adjacent step95th-percentile absolute adjacent step
Allegro899.997%0.02-1.57%9.81-12.54%
HSA4100.00%0.00-14.26%0.09-14.35%
Toy Arm6~100.00%0.85-1.85%15.39-17.11%
Pneumatic hand, low + high6 paired controls97.61%0.85-3.76%12.01-27.76%
Franka7~100.00%0.85-1.27%3.42-5.61%
Allegro-sim8~100.00%1.05-1.39%4.34-5.31%
UR5e6~100.00%1.14-1.18%4.47-4.77%

W1.2. The paper does not evaluate how well the model generalizes to OOD trajectories, such as unseen robot configurations, action combinations, or deformation patterns

We thank the reviewer for pointing out the OOD capability question of our work. We break down OOD capability in terms of unseen action combinations/deformation patterns and unseen/novel views:

  • Unseen action combinations/deformation patterns: The action combinations in the test sequences are not observed during training. Therefore, our evaluation already shows that Act4R has the ability to generalize to novel action combinations.

  • Generalization to unseen viewpoints: We conduct additional experiments to evaluate the generalization of Act4R to unseen viewpoints. Specifically, we collect new evaluation data from camera viewpoints not observed during training and compare Act4R with NJF. The results are reported below using CJPE (↓).

    DatasetTraining view (Act4R)Training view (NJF)Novel view (Act4R)Novel view (NJF)
    Franka6.87210.3456.69918.695
    Allegro-sim9.57427.5129.94435.481
    UR5e5.8579.0035.96421.938

    Act4R consistently outperforms NJF on unseen viewpoints and maintains performance comparable to that on training viewpoints. These results demonstrate that Act4R generalizes well to unseen camera viewpoints, that the learned action-to-motion mapping is not overfitting on specific viewpoints observed during training. We are happy to provide additional experiments if requested.

AuthorsOpenReview ↗

Response to W2: Simulation-Based Close-loop Control

W2. closed-loop control is demonstrated only in simulation

Since NJF has proven demonstration of inverse robotic manipulation on real-world demonstration, our work mainly focuses on improving the underlying spatial representation and related metrics. A simulation dataset provides an easy way for researchers to work on inverse kinematics algorithms without expensive physical equipment. Since Act4R is a self-optimization based algorithm fitted to individual embodiment rather than a generalized inverse control algorithm, the risk of sim-to-real gaps is lower.

Simulation benchmarks provide real groundtruth in depth and in action, whereas real world measurements are noisy. In our experiments on the NJF datasets, we find that the device measured depth on a real-world dataset is similarly performant compared to VGGT-dervied depth (Table 1 and Table 2 in the manuscript), whereas groundtruth depth leads to better performance compared to VGGT-dervied depth on simulated environments. This gives context on real-world noise effects.

In summary, we believe our simulation datasets provide unique values compared to real-world datasets, while also recognize that real-world demonstration would be ideal. Since Act4R is a self-optimization algorithm fitted to individual embodiment, and has been tested on real-world datasets for standard metrics, we find the risk that it does not generalize to real-world close-loop tasks to be relatively low.

AuthorsOpenReview ↗

Response to W3: Tracking Errors, Occlusion, and Robustness

W3. How do tracking errors or missing correspondences affect the quality of the training data and the robustness of the learned action-to-motion mapping.

We agree with the reviewer that Cotracker/2D groundtruth track can be unreliable due to occlusion. We note that Act4R’s formulation can handle occlusion through spatial computation. In Act4R, the input image is projected to an explicit 3D pointcloud, and the legitimate correspondences are only computed over visible pointcloud overlaps. This means occluded pointclouds and their subsequent forward-projected 2D tracks will not be involved in loss computation, eliminating the noisy 2D tracks over occluded regions.

Practically, in both real and simulated datasets, occlusion is not prominent. Our work mainly focuses on establishing the feasibility of 3D/4D foundation model as the spatial representation for inverse kinematics. We thank the reviewer for pointing this out and will analyzing occlusion more closely as a future work in more challenging, in the wild scenarios.

Dear Reviewer rZZP,

Thank you for taking the time to review our work. As the discussion period is coming to an end, we hope our rebuttal and additional experiments have helped address your concerns. If you have any remaining questions or concerns, we would be happy to provide further clarification. If you find our responses convincing and have no remaining concerns, we would sincerely appreciate it if you could consider revisiting your initial score.

Thanks again for your time and consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.