acceptodds
Withdrawn from ICLR 2027

RoPE-Flow: Monocular Vision-based Robot Pose Estimation via 2D Graph-conditioned 3D Flow Matching

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Camera-to-robot pose estimation from a monocular image is significant for vision-based robot navigation and control. However, recovering 3D structural features from 2D observations remains a challenging task. Existing camera-to-robot pose estimation methods either rely on time-consuming iterative rendering or involve sophisticated 2D–3D registration which suffers from projection ambiguity, accumulated errors for distal joints and imbalanced learning for different joints. In this work, we propose **RoPE-Flow**, an end-to-end flow matching-based framework for camera-to-robot pose estimation. From the predicted 2D heatmaps, we propose a Keypoint-centered Feature Extractor to obtain positional features along with contextual representations from keypoints-centered multi-scale patches. Moreover, we introduce Graph-guided Condition Generator, which embeds a learnable adjacency map to generate condition for flow matching based on correlation among keypoints. Besides, we introduce Bone Length Loss and Ray-Depth Loss for topological consistency and scale stability, with joint-wise weighting to mitigate varied prediction difficulty across joints. The proposed method outperforms state-of-the-art methods for pose estimation across different types of robots with mutiple cameras. Besides, we establish a novel cross-scene benchmark to demonstrate the outstanding performance of RoPE-Flow on zero-shot cross-scene robot pose estimation. Meanwhile, animated visualizations are shown on the website: https://ropeflow.netlify.app/.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Robot Pose Estimation, Flow Matching, Generative Model, 3D Reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Sdk5OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Camera-to-robot pose estimation from a single RGB image is important for vision-based robot navigation and control. This paper presents RoPE-Flow, an end-to-end framework based on conditional flow matching. First, the method employs a frozen DINOv3 backbone and LoRA adapters to generate multi-scale keypoint center features. Then, the proposed Keypoint-centered Feature Extractor combines positional information with contextual representations extracted from multi-scale heatmap patches centered on predicted 2D keypoints. Moreover, a Graph-guided Condition Generator then captures relationships among keypoints through a learnable adjacency matrix to construct conditioning representations for 3D keypoint generation. Finally, the Bone Length Loss and Ray-Depth Loss further encourage structural consistency and scale stability, while joint-wise loss weighting accounts for differences in prediction difficulty across keypoints. Experiments on the DREAM dataset show improved performance over prior methods across multiple robot types and camera settings. The authors also introduce LabRoPE, a real-world benchmark for evaluating zero-shot cross-scene generalization.

Critical strengths

Strengths:

  1. The paper evaluates the method across multiple robot types and camera settings on DREAM, with additional experiments on cross-scene generalization. The reported results show clear improvements over competitive baselines.

  2. The method addresses relevant challenges in monocular robot pose estimation, including inter-keypoint dependencies, depth ambiguity, and differences in prediction difficulty across joints. The combination of keypoint-centered feature extraction, graph-based conditioning, and conditional flow matching is logically motivated.

  3. The bone-length and ray-depth losses incorporate useful geometric priors into training, particularly during adaptation to real images without 3D annotations.

Critical weaknesses

Weaknesses:

  1. The paper only includes component-removal experiments. However, these experiments mainly establish the contribution of individual design choices within the complete system; they provide less insight into whether the proposed components offer complementary benefits.

  2. The specific contribution of conditional flow matching is not sufficiently isolated. Table 5 reports a substantial performance decrease when CFM is removed. However, the corresponding experiment replaces the 3D prediction pathway with the FK-and-PnP pipeline from RoboPEPP. This changes both the prediction formulation and the inference procedure, making it difficult to attribute the improvement specifically to flow matching.

  3. In Table 5, removing LboneL_{\mathrm{bone}} or LFKL_{\mathrm{FK}} reduces average ADD AUC from 84.6 to 82.2 and 83.3, respectively, whereas removing the joint-wise weights wkw_k reduces it to 44.2. These results suggest that joint-wise weighting is critical to the reported performance, but its unusually large effect deserves further analysis. How were the weights selected?

  4. Inconsistent and overloaded notation makes the method unnecessarily difficult to follow. Several notation issues should be corrected or explicitly clarified:

For example:

(1) Eq.4 updates xs3Dx_s^{3D} but evaluates the velocity network at ysy_s, whose relationship to xs3Dx_s^{3D} is not specified.

(2) Eq.6 introduces y0y_0, yty_t, and y1y_1 without explicitly relating them to the x03Dx_0^{3D}, xt3Dx_t^{3D}, and x13Dx_1^{3D} notation used earlier.

(3) FF denotes different feature representations in different sections.

(4) In Fout=softmax⁡(A)FinWF_{\mathrm{out}}=\operatorname{softmax}(A)F_{\mathrm{in}}W, W should be explicitly identified。

(5) Sec.3.3 introduces ckpc_{kp}, whereas Eq.2 uses cgcnc_{\mathrm{gcn}}. Please state whether these denote the same representation.

(6) The interpolation expression in Fig. 3 is inconsistent with Eq.3. Please reconcile the time direction and interpolation coefficients.

  1. Some technical details are not described clearly.

(1) The role of cimgc_{\mathrm{img}}. Sec.3.1 states that the pooled image representation conditions “heatmap flow matching during training,” but no corresponding heatmap flow-matching formulation is subsequently provided. Please clarify whether this is an additional component, an omitted description, or a wording error.

(2) Although Sec. 3.6.2 refers to adjacent supervised links, the exact construction of E\mathcal E is not specified. Please define the selected keypoint pairs for each robot.

(3) The paper does not clearly explain how the predicted 3D keypoints X^\hat X used to compute the geometric losses are generated during training.

(4) The best results obtained with 10 sampling steps in Table 12 differ from those reported in Table 2. Please clarify whether these results use different model checkpoints or evaluation settings.

(5) The paper does not clearly describe how occluded keypoints, keypoints outside the image boundaries, and unreliable heatmap peaks are handled.

Questions for authors

Please address the clarification questions raised in the Weaknesses section.

Reviewer_8fYeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RoPE-Flow, an end-to-end conditional flow matching framework for monocular camera-to-robot pose estimation. From predicted 2D keypoint heatmaps, it extracts keypoint-centered multi-scale features (KFE), builds a graph-guided condition (GCG) from keypoint correlations and camera intrinsics, and generates camera-frame 3D keypoints via 3D conditioned flow matching, followed by iterative residual MLP joint-angle prediction. Training combines a flow-matching velocity loss with bone-length, FK-alignment, ray-depth, and joint-wise weighted losses, plus self-supervised sim-to-real fine-tuning. On DREAM, RoPE-Flow achieves the best ADD AUC on all 9 splits and the best joint-angle MAE on Panda/Kuka, without requiring bounding boxes or joint-state priors. The paper also introduces LabRoPE, a small zero-shot cross-scene benchmark, on which it reports higher mask mIoU/Dice than RoboPEPP and HPE.

Critical strengths
  1. The authors report state-of-the-art results: RoPE-Flow achieves the best ADD AUC on all 9 DREAM splits and the best joint-angle MAE on Panda/Kuka, without requiring bounding boxes or joint-state priors, outperforming strong baselines including RoboPose, HPE and RoboPEPP.

  2. The ablation study is well designed and substantial: it covers keypoint feature extraction (patch scales and position embeddings), graph conditioning, the flow matching generator (vs. PnP-FK), each loss term, sim-to-real losses, sampling steps, and zero-shot evaluation, giving clear attribution for the gains.

Critical weaknesses
  1. The closest generative predecessors are absent from every main comparison table. The central "outperforms SOTA" claim is therefore not tested against the most relevant methods, and the claimed "first flow matching-based" novelty is not clearly differentiated from the diffusion-based line.

  2. The proposed zero-shot benchmark LabRoPE does not yet substantiate the cross-scene generalization claim: ground-truth masks are auto-generated by GroundingDINO + SAM2 with no reported human validation and no 3D keypoint/joint-angle ground truth; only mask mIoU/Dice are reported; the dataset contains only 50 images from a single robot and camera; baseline training protocols and ROI conditions are unspecified; and the large variance of baseline results raises statistical significance concerns.

Questions for authors
  1. In Sec. 3.4,x03Dis defined both as the 2D-anchored data point (Eq. 3) and as Gaussian noise at sampling. Given the velocity target (y1 − y0) in Eq. 6, does sampling integrate backward from t = 1 to t = 0, or are the notation/signs inconsistent? What is the third dimension of X2D?

  2. Why are RoboKeyGen, MonoSE(3)-Diffusion, and PoseDiff absent from all main comparison tables? Can their official numbers be included?

  3. For LabRoPE, what fraction of the auto-generated masks was manually verified? Can 3D pose or joint-angle ground truth be added? Were the baselines fine-tuned, were they given ROIs, and is the mIoU/Dice difference statistically significant?

  4. In the "w/o Position" ablation (Table 5), several splits drop to 0.0. Does this setting converge, and what remains as the input after removing position embeddings?

Reviewer_zC7XOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies monocular camera-to-robot pose estimation from a single full-frame RGB image without external bounding-box crops. Built on a frozen DINOv3 ViT-L/16 backbone with LoRA adapters, RoPE-Flow uses a 2D heatmap decoder and a Keypoint-centered Feature Extractor (KFE) to crop multi-scale local features around predicted 2D joints, followed by a Graph-guided Condition Generator (GCG) that encodes structural connectivity and camera intrinsics. Conditioned on these features, a 4-layer MLP trained with Conditional Flow Matching (CFM) integrates a 10-step Euler ODE from Gaussian noise to predict camera-frame 3D keypoints, supervised by velocity matching, direct L1 3D keypoint loss, bone-length regularization, and ground-truth forward-kinematics (FK) alignment. A separate parallel branch with its own LoRA adapters, heatmap decoder, KFE, GCG, and a 4-step residual MLP regresses robot joint angles. Models are trained per robot in three supervised stages on synthetic DREAM data, adapted to real splits via self-supervised losses, and evaluated on DREAM and a new 50-image real-world Panda dataset (LabRoPE).

Critical strengths
  1. Strong DREAM accuracy with full-frame inference via local and structural feature design (KFE + GCG). RoPE-Flow reaches 84.6 average ADD AUC on DREAM (Table 1), ahead of RoboTAG (76.9) and RoboPEPP (74.0), while running directly on full-frame RGB images without an external object detector at test time. Cropping multi-scale features around predicted 2D keypoints (KFE) and passing them through a kinematics-aware graph module with camera intrinsics (GCG) is well suited to 2D-to-3D keypoint lifting. Why consequential: This demonstrates that combining a strong vision foundation backbone with keypoint-centered local cropping and structural graph conditioning can eliminate reliance on external bounding-box detectors while achieving high 2D-to-3D lifting accuracy on standard benchmarks.

  2. Detailed architecture, training, and computational cost reporting. Network specifications (Table 7), stage-wise training hyperparameters (Tables 8–10), and compute requirements (Table 13: 813.4M parameters, 4.33 GB VRAM, 36.5 ms latency on an H100 GPU) are clearly tabulated. Why consequential: This transparency regarding model scale, multi-stage training schedules, and hardware cost is valuable for reproducibility and makes it possible to rigorously assess the trade-offs between accuracy and the 813.4M-parameter footprint.

Critical weaknesses
  1. Unsubstantiated Flow Matching formulation and kinematically inconsistent dual-branch architecture. • Flow Matching behaves as a fixed 10-step deterministic regressor: The paper motivates CFM through depth ambiguity and uncertainty, yet test-time inference (Eq. 4) draws a single noise sample to produce one deterministic point estimate. In Table 5, dropping the direct L1 3D loss (w/o L_3D) causes average ADD AUC to drop from 84.6 to 47.2 (and to 4.5 on Panda-RS). In Table 12, increasing Euler integration steps from 10 → 15 → 20 steadily hurts accuracy (e.g., 85.29 → 76.85 on Panda-RS), which is inconsistent with learning a well-behaved velocity field. Also, the w/o CFM row in Table 5 replaces the head with PnP + FK rather than testing a standard MLP regression head on the same DINOv3 + KFE + GCG features. • Inconsistent 3D keypoint and joint-angle branches, and missing 6D base pose recovery: The architecture duplicates the LoRA adapters, heatmap decoder, KFE, and GCG into two separate branches for 3D keypoints X̂ and joint angles q̂. At test time, X̂ is predicted as a free 3D point cloud that violates rigid link lengths and does not match FK(q̂), making direct ADD comparisons against kinematically constrained baselines inequitable. Training (Eq. 9) relies on ground-truth extrinsics T_(c←r)^gt, and the paper never explains how the 6D base pose T̂_(c←r) is estimated at test time to render the URDF meshes shown in Figs. 3–5 and Table 4 (in addition to minor notation errors in Fig. 3 and Eqs. 4 and 6). • Why consequential to rejection: These issues invalidate the paper’s core methodological claims. Because the CFM module fails without direct L1 supervision, degrades with finer ODE integration, and lacks a controlled comparison against a direct MLP regression head on the same features—while predicting unconstrained 3D points that violate rigid-body kinematics and leaving 6D base pose recovery undefined—the core technical contribution is not sound.

  2. Uncontrolled 813M backbone, flawed Sim-to-Real protocol, and insufficient real-world validation and deployment justification. • Uncontrolled 813M backbone and flawed Sim-to-Real protocol: RoPE-Flow relies on an 813.4M-parameter DINOv3 ViT-L/16 backbone (2.4× RoboPEPP and >10× HPE in Table 13) without backbone-matched comparisons. Even the w/o CFM (PnP + FK) variant already outperforms RoboPEPP† (75.4 vs. 74.0 average, and 54.1 vs. 34.4 on Baxter-DR), indicating that much of the gain comes from the backbone and 2D keypoint front-end. For Sim-to-Real adaptation, the first term of Eq. 14 (SmoothL1(X̂, X̃)) provides zero gradient along depth because X̃_(k,z) ≡ X̂_(k,z) by construction, and Appendix B states that checkpoints are selected every epoch using validation metrics on the target splits. • Small real-world test set and unexplained baseline gaps: LabRoPE consists of only 50 images of a single robot (Panda) captured by one camera (RealSense) in one room, and omits both RoboTAG (the strongest DREAM baseline at 76.9 average) and RoboPose. In Table 1, the large gap between predicted-box (†) and ground-truth-box (‡) baselines (such as HPE† 40.7 vs. HPE‡ 82.0 on Panda-Photo) and the size of the synthetic training set are not explained. • Unclear practical deployment setting: RoPE-Flow requires known camera intrinsics (κ in Eq. 2) and trains a separate 813M-parameter checkpoint per robot (§4.1), consuming 4.33 GB VRAM and 36.5 ms per frame. The paper does not identify a concrete robotic task where running an 813M vision model per robot is preferable to standard one-off hand-eye calibration combined with internal joint encoders. • Why consequential to rejection: Although the headline numbers on DREAM are high, the empirical improvements are confounded by an unablated 813M-parameter DINOv3 backbone and by selecting sim-to-real checkpoints on target splits. Because fixing these issues requires redesigning the head/kinematic formulation and re-running controlled backbone and sim-to-real baselines, the paper cannot be fixed within a rebuttal cycle.

Questions for authors
  1. Why does performance degrade when increasing Euler steps from 10 → 20 (Table 12) and drop sharply without L_3D (Table 5), and how does a direct MLP regression head on the same DINOv3 + KFE + GCG features perform?
  2. How is the 6D base pose T̂_(c←r) recovered at test time for URDF rendering, and what is the ADD AUC when evaluating the kinematically valid keypoints T̂_(c←r) ∘ FK(q̂)?
  3. In Appendix B, were target test-split metrics used to pick the best sim-to-real checkpoint at each epoch, and doesn’t X̃_(k,z) ≡ X̂_(k,z) make the Z-component of the first term in Eq. 14 identically zero?
  4. What causes the large gap between † and ‡ baselines in Table 1 (e.g., HPE† 40.7 vs. HPE‡ 82.0 on Panda-Photo), why was RoboTAG omitted on LabRoPE, and in what practical setup is an 813M per-robot model preferable to one-off calibration with joint encoders?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.