acceptodds
Under review as a conference paper at ICLR 2027

ObjAligner: Feed-Forward Object Pose Estimation and Reconstruction via Shared Geometry

Abstract

Feed-forward geometry models recover dense 3D structure in camera-centric coordinates, but camera motion alone cannot align an independently moving object across views. We address joint 6D pose estimation and reconstruction with a simple insight: represent the same rigid object through two geometries that play complementary yet asymmetric roles. Dense per-view geometry in camera coordinates preserves the surface detail required for reconstruction, while sparse sequence-shared geometry provides the rigid scaffold needed to determine motion. Their pixel-aligned correspondences turn pose estimation into differentiable rigid alignment, and the same transform maps dense local geometry into the shared frame for reconstruction. The shared frame need not be canonical: it is learned only up to a sequence-common rigid transform, avoiding supervision of absolute object coordinates. We present ObjAligner, a feed-forward framework that takes an RGB sequence and a first-frame target box, without CAD/NOCS models, later-frame annotations, or test-time optimization. Experiments across camera-only, object-only, and joint camera-object motion validate the shared geometric relation for both pose and reconstruction; ObjAligner outperforms matched direct pose regression and transfers to unseen objects and external domains without target-domain adaptation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
object-centric 3D reconstruction, object pose estimation, multi-view geometry, dynamic scenes

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vcdBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a feed-forward framework for joint object pose estimation and reconstruction from RGB sequences. The core idea is to combine dense camera-local geometry with sparse sequence-shared geometry, recovering object poses through differentiable rigid alignment and using the same transformations for multi-view reconstruction. Experiments across different motion settings and datasets demonstrate strong performance.

Critical strengths
  • The idea of connecting object pose estimation and reconstruction through a shared representation is well-motivated and intuitive.
  • The framework is reasonably designed, avoiding predefined canonical object coordinates and test-time optimization while maintaining a unified geometric relationship between the two tasks.
  • The experiment is relatively extensive, covering different motion conditions, external datasets, and component-level analyses.
Critical weaknesses
  • The direct-regression baseline uses a frozen Stage-I representation, whereas the full method benefits from additional Stage-II training and geometric supervision. A comparison under matched training and supervision conditions would better isolate the contribution.
  • The reported translation-direction metric is insensitive to both translation magnitude and direction reversal. Additional sign-sensitive direction and metric translation evaluations would provide a more complete assessment of pose accuracy.
  • The AUC@15 values for ObjAligner differ between the main tables and Appendix Table 8. Similar discrepancies appear in several reconstruction metrics.
  • Although the method avoids test-time optimization, it introduces additional shared-geometry prediction and rigid alignment on top of a pretrained multi-view backbone. End-to-end runtime and memory comparisons would better support the claimed efficiency.
Questions for authors

Please refer to the Weaknesses.

2 comments

Thank you for your careful review of our paper. In response to your questions about training ablations, translation evaluation, result consistency, and runtime overhead, we provide the following additional experiments and clarifications.

1. Additional training for Direct (W1)

Following your suggestion regarding direct regression training, we used the same Stage-I weights to conduct matched Stage-II training with direct pose regression, retaining the original relative-pose supervision. The results are as follows:

DatasetDirect: AUC@15 / F@2% (%)ObjAligner: AUC@15 / F@2% (%)
COPE1194.99 / 27.1127.84 / 51.35
UOPE564.52 / 30.3521.18 / 52.01
HO3D1.43 / 33.7311.01 / 47.93

After this training, Direct still exhibits a clear gap relative to the full framework. These results will be added to the ablation table in the main paper.

2. Evaluation of signed translation direction and translation magnitude (W2)

Following your suggestion regarding translation evaluation, we additionally recomputed pose AUC using a direction error that preserves the translation sign, namely the angle between the predicted and ground-truth translation vectors (0°–180°). AUC@15 on COPE119 / UOPE56 / HO3D remains 27.84 / 21.18 / 11.01, indicating that the original AUC advantage does not rely on forgiving scoring of direction reversals.

Without GT calibration, the model can recover inter-frame relative rotation, translation direction, and relative translation at a common sequence scale. Translation magnitude is determined up to a single scale factor shared across the sequence.

Building on this, to evaluate the accuracy of translation magnitude, we further aligned the predicted and ground-truth translations using one nonnegative scale per sequence and computed scale-normalized translation NRMSE. The results are as follows:

MethodCOPE119: NRMSE↓UOPE56: NRMSE↓HO3D: NRMSE↓
π³ + black-bbox0.5660.6310.590
ObjAligner0.3130.3590.518

3. Consistency of results between the main paper and appendix (W3)

Thank you for pointing out this discrepancy. We reported ablations of the inference settings in the appendix: a fusion threshold of 0.75 achieves the highest F in Table 14, and 32×32 queries achieve the highest AUC in Table 15. We therefore adopted this setting for the corresponding comparisons in the appendix, as explicitly stated at the beginning of Appendix D and in the caption of Table 8. Under the main-table setting of 12×12 queries and a 0.5 fusion threshold, AUC@15 on COPE119 / UOPE56 / HO3D is 27.84 / 21.18 / 11.01, still higher than all comparison methods in their respective main tables.

However, recognizing that this can indeed cause ambiguity, we will consistently use the main-table setting of 12×12 queries and a 0.5 fusion threshold and revise B.1, Appendix D, and the relevant table captions accordingly, so that the numerical results in the main paper and appendix correspond directly.

4. Runtime efficiency and GPU memory usage (W4)

Regarding your concern about runtime overhead, on the same RTX 4090 with a fixed 8-frame input, ObjAligner takes 0.57 seconds from RGB to poses and a fused point cloud, compared with 0.85 seconds for SAM2 + π³ + black-bbox, reducing runtime by approximately 33%.

MethodRuntime per fixed 8-frame clip (seconds)Peak allocated GPU memory (GiB)
ObjAligner0.575.89
SAM2 + π³ + black-bbox0.855.11

These results will be added to the paper.

Thank you again for your helpful suggestions. We have added Direct training, signed translation and NRMSE evaluations, and runtime and GPU memory measurements, and unified the inference settings and corresponding results in the main and appendix tables. Signed evaluation preserves the reported AUC, while ObjAligner reduces runtime by approximately 33% compared with SAM2 + π³ + black-bbox. We would appreciate your reassessment in light of these results and revisions when you have time.

Reviewer_bHJHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents a feed-forward framework for joint 6D object pose estimation and 3D reconstruction from RGB sequences. The key idea is to predict dense camera-centric point maps and an additional sequence-shared geometry representation to establish rigid alignment across frames. By leveraging pixel-aligned correspondences between the per-view point maps and the shared geometry, the method estimates object poses through rigid alignment and transforms the dense geometry into a shared coordinate frame for reconstruction.

Critical strengths
  1. The research direction of jointly estimating object poses and reconstructing unseen objects without requiring pre-existing CAD models is interesting and potentially useful.

  2. The experiments demonstrate improvements in object pose estimation over several existing methods.

Critical weaknesses
  1. Concerns about the problem formulation and the necessity of the proposed shared geometry. The method is motivated by the difficulty of separating camera motion from independently moving object motion using existing multi-view reconstruction networks. To address this, the authors introduce an additional shared geometry representation for object pose estimation. However, I am not fully convinced that learning this additional representation is necessary. Recent multi-view reconstruction methods, including π³ used in this work, can predict per-view 3D geometry from RGB sequences. Based on these predictions, there seem to be two more straightforward alternatives: 1) Point cloud registration: Extract the target object's point clouds from individual views and estimate their relative transformations using feature correspondences and rigid registration methods such as ICP; 2) Object-centric reconstruction: Use SAM or another off-the-shelf segmentation model to isolate the target object in each frame, and feed the foreground observations into a multi-view reconstruction model. By treating the object as stationary and interpreting its motion as virtual camera motion, the resulting relative camera poses could potentially be used to recover object motion.

Although the paper includes a "π³ + black-bbox" baseline corresponding to the second alternative, its inferior performance could also be attributed to the lack of object-specific fine-tuning, which ObjAligner benefits from, rather than to an inherent limitation of this simpler approach. Without a more convincing comparison against these straightforward alternatives, I find the motivation for introducing an additional learned shared geometry representation insufficient.

  1. Poor presentation and unclear technical descriptions. The paper is difficult to follow, mainly due to issues with its organization, technical explanations, and positioning relative to existing work. 1) Introduction and Related Works: The discussion in Lines 62–67 is particularly unclear. The authors first state that existing methods do not eliminate the need for an object reference, but then introduce D4RT as a method that does not define a rigid reference. The distinction between lacking an explicit object reference and requiring one for pose estimation is not adequately explained. The Related Works section has a similar issue: it largely lists existing papers without sufficiently analyzing their differences or clearly establishing the novelty of the proposed method. As a result, it is difficult to understand the precise research gap that this work aims to address. 2) Terminology and notation: Several terms, such as "sequence-common rigid transform", are introduced without sufficient intuitive explanation. The notation in Eq. (1) is also rather compressed, and the roles of the transformation symbols and point index (k) could be explained more explicitly before introducing the formulation. 3) Architecture description: The network architecture is not explained in sufficient detail. In particular, the interactions between the image branch, object tokens, and shared geometry decoder remain unclear. A more explicit description of how these components exchange information would improve the technical clarity.
Questions for authors
  1. Choice of baselines: As the paper focuses on object pose estimation, why do the main quantitative comparisons primarily involve general-purpose scene reconstruction methods? Although several object pose estimation and tracking methods are mentioned in Section 4.1, their results are not presented in the main comparison tables. More direct comparisons with relevant object pose estimation approaches would help establish the effectiveness of the proposed method.

  2. Missing related work: Several recent methods have also explored multi-view reasoning for unseen object pose estimation, such as RayPose (ICCV 2025) and PoseGAM (CVPR 2026). Although their input assumptions are not identical to those of ObjAligner, they are relevant to the broader research direction. It would be helpful for the authors to discuss these methods and clarify the conceptual differences.

  3. Unclear fallback mechanism: In Line 202, what exactly does "predictions fall back to the prompt region" mean? Since the initial bounding box is only provided for the first frame, how is this fallback region determined for subsequent frames, particularly when the target object undergoes significant motion?

5 comments

Thank you for your careful review and suggestions. In response to your questions about shared geometry, comparison methods, and technical presentation, we provide the following additional experiments and clarifications.

1. Shared geometry and two alternative approaches (W1)

Regarding the necessity of shared geometry, we additionally evaluated the two alternative approaches you proposed.

For point cloud registration, we performed SIFT + RANSAC + ICP registration on the same local point clouds predicted by ObjAligner on COPE119, UOPE56, and HO3D, without CAD, GT initialization, or GT scale. The results are as follows:

MethodCOPE119: AUC@15 / F@2% (%)UOPE56: AUC@15 / F@2% (%)HO3D: AUC@15 / F@2% (%)
SIFT + RANSAC + ICP11.03 / 47.2610.29 / 42.488.15 / 47.89
ObjAligner27.84 / 51.3521.18 / 52.0111.01 / 47.93

When both the object and camera move, the partial object surfaces observed from different viewpoints may have little or even no overlap, which limits conventional registration methods in establishing reliable correspondences and accurately estimating relative object poses.

For the π³ + black-bbox approach, which interprets object motion as virtual camera motion, we added the object-local geometry supervision used in the submitted model to the original training architecture and further trained π³ on the same datasets to adapt it to the new datasets. The original and adapted π³ use the same black-bbox inputs and scoring protocol. The results are as follows:

MethodEvaluation inputCOPE119: AUC@15 / F@2% (%)UOPE56: AUC@15 / F@2% (%)HO3D: AUC@15 / F@2% (%)
Original π³black-bbox4.78 / 29.347.38 / 39.996.81 / 38.61
Adapted π³black-bbox1.31 / 36.523.20 / 43.694.14 / 42.15
ObjAlignerFull RGB27.84 / 51.3521.18 / 52.0111.01 / 47.93

After adaptation, reconstruction F improves on all three moving-object test sets, indicating that object-local geometry supervision is effective for reconstruction. However, object relative-pose AUC decreases rather than improves, suggesting that adaptation on data containing moving objects may affect π³'s original pose prediction prior to some extent. We will add these experiments to the paper.

To address differences in background and segmentation, we fixed the model weights, test frames, and evaluation protocol, replaced the model's own segmentation support under full RGB with SAM2 (A), and then changed the input to the same black-bbox input used by π³ (B) to further match the baseline's RGB context. Both settings use the object pose branch.

ObjAligner settingInput / Segmentation supportCOPE119: AUC@15 / F@2% (%)UOPE56: AUC@15 / F@2% (%)HO3D: AUC@15 / F@2% (%)
Original settingFull RGB / Own predictions27.84 / 51.3521.18 / 52.0111.01 / 47.93
AFull RGB / SAM228.08 / 52.2721.42 / 55.2512.80 / 47.89
Bblack-bbox / SAM220.34 / 50.9916.39 / 57.2111.58 / 52.30

Keeping SAM2 support unchanged while replacing full RGB with black-bbox input (A→B) reduces ObjAligner's pose performance. Nevertheless, under input and segmentation support matched to π³ (B), both AUC and F remain higher than those of the original and adapted π³ on all three datasets. This indicates that background information helps pose estimation, while ObjAligner retains its advantages in pose and reconstruction when input and segmentation support are matched.

2. Specialized object pose methods and related work (Q1, Q2; W2)

We thank the reviewer for suggesting RayPose and PoseGAM. Both use multi-view reference images and their known object-to-camera poses to provide an object reference, whereas our method learns a shared reference from the test sequence itself. For specialized object pose methods, as we have not yet obtained publicly available RayPose weights, we added a comparison with PoseGAM on COPE119 / UOPE56 / HO3D and measured runtime and peak allocated GPU memory for each 8-frame clip on the same RTX 4090 GPU model. The results are as follows. This comparison will be added to the paper.

MethodInputs and object referenceCOPE119: AUC@15 (%) / Runtime (seconds) / GPU memory (GiB)UOPE56: AUC@15 (%) / Runtime (seconds) / GPU memory (GiB)HO3D: AUC@15 (%) / Runtime (seconds) / GPU memory (GiB)
PoseGAMRGB, SAM2 masks, camera intrinsics, object CAD, and reference images rendered from CAD with known object-to-camera poses29.34 / 26.36 / 10.7034.97 / 26.26 / 10.7835.22 / 23.48 / 10.71
ObjAlignerRGB and a first-frame target box; no CAD or reference-view poses required27.84 / 0.61 / 5.8921.18 / 0.61 / 5.8911.01 / 0.62 / 5.89

The above PoseGAM comparison relies on object CAD and reference-view poses, so its input protocol differs substantially from ours, and its runtime is approximately 38–43 times that of ObjAligner. ObjAligner requires no object CAD in advance and jointly predicts poses and visible surfaces with lower runtime and GPU memory usage, making it better suited to rapidly processing unseen objects in potential scenarios where CAD or reference views with known poses cannot be obtained in advance, such as embodied intelligence.

3. Technical presentation and fallback mechanism (W2; Q3)

To address potential ambiguity, we will revise the transitions in the related work discussion to clarify the respective roles of D4RT's point-level dynamic geometry, FMOV's per-scene optimization, and KV-Tracker's online mapping, highlighting how our method differs by establishing a shared object reference from the sequence itself and performing feed-forward joint prediction.

We thank the reviewer for the suggestions on technical presentation. Lines 198–201 in §3.2 of the original paper already explain that the first-frame box initializes the object tokens, which cross-attend to image features and provide context for segmentation. Lines 211–215 in §3.3 already explain that the shared-geometry decoder combines image features at the query locations with sequence-wide object context to predict shared points. To further clarify the notation, we will specify the mapping direction of QiQ_i and the meaning of kk as a within-frame query index before Eq. (1), and explain “sequence-common rigid transform” beside Eq. (2) as “the same coordinate transform shared by all frames.” For the fallback in line 202, we will explicitly add that “when there are fewer than 16 foreground pixels, the query region reuses the normalized image position of the first-frame box,” rather than a tracking box that follows the object's motion.

Thanks for the response and the additional experiments. However, my main concerns remain.

  1. The adapted π³ baseline does not test the alternative I raised in W1. My suggestion was to isolate the object and fine-tune a reconstruction model to treat object motion as virtual camera motion. The revision describes the adapted π³ only as trained with "additional object-local geometry supervision", without specifying whether it used black-bbox or full-RGB inputs, or whether its camera head was supervised with object-relative poses. After adaptation, its pose AUC falls below that of the un-tuned model on all three datasets (e.g., 4.78 → 1.31 on COPE119). The rebuttal attributes this to fine-tuning damaging π³'s pose prior, yet ObjAligner is fine-tuned from the same π³ on the same data and achieves far higher pose accuracy. The drop therefore more likely reflects how the baseline was supervised than an inherent limitation of the alternative, and the necessity of the learned shared geometry remains unestablished.

  2. The presentation concerns (W2) are addressed only by local wording changes; the organization, the Related Work, and the architecture description are not substantively revised.

The revised paper also contains clear inconsistencies. For example, Sec. 4.1 lists FoundationPose, Point2Pose, FMOV, KV-Tracker, D4RT, OnePose++, and Any6D as baselines, yet none of them appears in any table or in Figure 4. Consequently, Q1 also remains unaddressed in the main paper.

I therefore maintain my rating.

Reviewer_agV3OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents ObjAligner, a feed-forward framework for joint 6D pose estimation and visible-surface reconstruction of unseen rigid objects from multi-view RGB images. The method addresses the challenge of aligning independently moving objects across views without requiring canonical object coordinates, CAD models, or test-time optimization.

The key idea is to represent an object using two complementary geometric representations: dense camera-local geometry and sparse sequence-shared geometry. Building upon a pretrained π3 model, ObjAligner predicts dense local point maps while learning sparse object points in a shared coordinate frame through target-conditioned multi-view encoding. Pixel-aligned correspondences between these representations enable pose estimation through differentiable weighted rigid alignment, and the recovered transformations are subsequently used to fuse local geometry for reconstruction. A two-stage training strategy further enforces relative-pose and cross-view geometric consistency without requiring predefined canonical coordinates.

Experiments on COPE119, UOPE56, HO3D, NAVI, and OnePose demonstrate substantial improvements in relative pose estimation, particularly under joint camera-object motion. The method also achieves competitive reconstruction performance and shows promising generalization to unseen objects and external datasets without target-domain adaptation.

Critical strengths

(1) The paper addresses the challenging problem of jointly recovering object motion and geometry without predefined canonical coordinates or object models. The proposed combination of dense camera-local and sparse sequence-shared geometry provides a coherent formulation for connecting pose estimation with reconstruction. (2) ObjAligner achieves substantial improvements over existing feed-forward baselines across multiple datasets and motion settings. The consistent gains on external benchmarks, particularly UOPE56 and HO3D, provide encouraging evidence of generalization to unseen objects and challenging motion patterns. (3) The paper evaluates both pose estimation and reconstruction across diverse datasets and includes comparisons with a direct pose regression baseline. The additional theoretical analysis and sensitivity experiments also help explain the proposed shared-frame representation and its practical behavior.

Critical weaknesses

(1) The compared methods receive different input representations: some baselines use SAM2-propagated foreground masks or blacked-out backgrounds, whereas ObjAligner processes full RGB images with its own target-conditioned segmentation module. These differences may affect the amount of contextual information available to each method, making it difficult to attribute the performance improvements exclusively to the proposed shared-geometry representation. Furthermore, although the paper discusses several specialized object pose estimation and tracking approaches, the main quantitative comparisons primarily focus on feed-forward geometry models. More carefully controlled comparisons would strengthen the conclusions. (2) The main pose evaluation primarily reports relative rotation and translation-direction errors rather than full translation accuracy. In particular, Appendix C.1 defines a sign-insensitive translation-axis metric, which does not penalize reversed translation directions. Moreover, the reported relative object-to-camera pose metrics do not directly establish the accuracy of object motion in a world coordinate frame, while physical trajectory evaluations are limited to training-overlapping synthetic scenes. Although the shared-frame ambiguity is theoretically justified, additional evaluation is needed to establish how accurately the method recovers physically meaningful object motion beyond angular agreement. (3) Table 4 compares the full framework with direct pose regression based on a frozen Stage-I representation, but it is unclear whether the two approaches are matched in optimization budget and trainable capacity. Therefore, the observed improvements may not exclusively reflect the advantage of predicting shared geometry over direct pose regression. Additionally, the local-support ablation removes the corresponding features only at inference time rather than retraining the architecture without them. The individual effects of the relative-pose and shared-geometry consistency objectives are also not separately evaluated, making their respective contributions difficult to assess.

Questions for authors

(1) Could the authors clarify how much of the performance improvement comes from the proposed shared-geometry representation rather than differences in input preprocessing and target segmentation? In particular, can representative baselines be evaluated under more closely matched RGB context and target-mask conditions? (2) Given that the translation-direction metric is sign-insensitive and the learned object frame has a sequence-dependent coordinate ambiguity, could the authors clarify which components of the predicted 6D motion are directly recoverable without ground-truth calibration? Can additional results using signed translation-direction errors or scale-normalized translation errors be provided on external benchmarks? (3) Is the direct pose regression baseline trained with the same optimization budget, trainable backbone components, and supervision as the full shared-geometry model? Could the authors also clarify the individual contributions of the relative-pose and geometric consistency losses, and whether the local-support branch remains beneficial when the corresponding ablated model is retrained?

3 comments

Thank you for your positive assessment of our paper. In response to your questions about comparison fairness, motion evaluation, and training ablations, we provide the following additional experiments and clarifications.

1. The role of shared geometry and input fairness (W1; Q1)

To address your questions about the sources of the performance gains and input fairness, we conducted additional π³ adaptation training and controlled-input experiments.

Specifically, we added the object-local geometry supervision used in the submitted model to the original training architecture and further trained π³ on the same datasets to adapt it to the new datasets. The original and adapted π³ use the same black-bbox inputs and scoring protocol. The results are as follows:

MethodEvaluation inputCOPE119: AUC@15 / F@2% (%)UOPE56: AUC@15 / F@2% (%)HO3D: AUC@15 / F@2% (%)
Original π³black-bbox4.78 / 29.347.38 / 39.996.81 / 38.61
Adapted π³black-bbox1.31 / 36.523.20 / 43.694.14 / 42.15
ObjAlignerFull RGB27.84 / 51.3521.18 / 52.0111.01 / 47.93

After adaptation, reconstruction F improves on all three moving-object test sets, indicating that object-local geometry supervision is effective for reconstruction. However, object relative-pose AUC decreases rather than improves, suggesting that adaptation on data containing moving objects may affect π³'s original pose prediction prior to some extent.

To address differences in background and segmentation, we fixed the model weights, test frames, and evaluation protocol, replaced the model's own segmentation support under full RGB with SAM2 (A), and then changed the input to the same black-bbox input used by π³ (B) to further match the baseline's RGB context. Both settings use the object pose branch.

ObjAligner settingInput / Segmentation supportCOPE119: AUC@15 / F@2% (%)UOPE56: AUC@15 / F@2% (%)HO3D: AUC@15 / F@2% (%)
Original settingFull RGB / Own predictions27.84 / 51.3521.18 / 52.0111.01 / 47.93
AFull RGB / SAM228.08 / 52.2721.42 / 55.2512.80 / 47.89
Bblack-bbox / SAM220.34 / 50.9916.39 / 57.2111.58 / 52.30

Keeping SAM2 support unchanged while replacing full RGB with black-bbox input (A→B) reduces ObjAligner's pose performance. Nevertheless, under input and segmentation support matched to π³ (B), both AUC and F remain higher than those of the original and adapted π³ on all three datasets. This indicates that background information helps pose estimation, while ObjAligner retains its advantages in pose and reconstruction when input and segmentation support are matched.

2. Translation metrics and the meaning of physical motion (W2; Q2)

We additionally recomputed pose AUC using a direction error that preserves the translation sign, namely the angle between the predicted and ground-truth translation vectors (0°–180°). AUC@15 on COPE119 / UOPE56 / HO3D remains 27.84 / 21.18 / 11.01, indicating that the original AUC advantage does not rely on forgiving scoring of direction reversals.

Without GT calibration, the model can recover inter-frame relative rotation, translation direction, and relative translation at a common sequence scale. A change in the shared coordinate frame applies the same rigid transform BB to all frames, and (BQi)−1(BQj)=Qi−1Qj(BQ_i)^{-1}(BQ_j)=Q_i^{-1}Q_j, so these relative motion quantities are unaffected. Translation magnitude is determined up to a single scale factor shared across the sequence.

For object-to-camera pose specifically, Qi−1Q_i^{-1} is obtained by rigidly aligning local and shared points at the same pixels and represents the transform from the learned object frame to the camera frame, without GT calibration. A sequence-fixed transform BB is needed only when expressing it in the dataset's predefined object frame. In practical applications, the object reference frame can be defined freely, for example by making it coincide with the camera frame in the first frame.

Building on this, to evaluate the accuracy of translation magnitude, we further aligned the predicted and ground-truth translations using one nonnegative scale per sequence and computed scale-normalized translation NRMSE. The results are as follows:

MethodCOPE119: NRMSE↓UOPE56: NRMSE↓HO3D: NRMSE↓
π³ + black-bbox0.5660.6310.590
ObjAligner0.3130.3590.518

3. Retraining ablations for Direct and Local Skip (W3; Q3)

To address the questions about training settings and retraining ablations, we used the same Stage-I weights to conduct matched Stage-II training with direct pose regression and, separately, performed full Stage-II retraining of ObjAligner with only the Local Skip branch disabled. The results are as follows:

DatasetDirect: AUC@15 / F@2% (%)Without Local Skip: AUC@15 / F@2% (%)ObjAligner: AUC@15 / F@2% (%)
COPE1194.99 / 27.1126.46 / 49.8527.84 / 51.35
UOPE564.52 / 30.3520.63 / 49.8621.18 / 52.01
HO3D1.43 / 33.7313.04 / 45.4511.01 / 47.93

After this training, Direct still exhibits a clear gap relative to the full framework. Removing Local Skip reduces reconstruction F on all three datasets and also reduces pose AUC on COPE119 and UOPE56. On HO3D, where hand–object occlusion is present, removing Local Skip reduces interference from unreliable local features to some extent, leading to an increase in AUC. Overall, however, Local Skip has a positive effect and is particularly effective for reconstruction. These results will be added to the ablation table in the main paper.

Regarding the two losses, Eqs. (8)–(9) and lines 320–323 in §3.5 of the main paper already explain that the relative-pose loss constrains the recovered rigid transforms, while the consistency loss maps shared points from each frame into the same reference frame to enforce cross-frame geometric consistency.

Thank you again for your constructive feedback. Our response adds controlled RGB/mask comparisons, signed translation and NRMSE evaluations, additional Direct training, and retraining without Local Skip. ObjAligner retains its pose and reconstruction advantages over π³ under matched inputs and segmentation support, and its AUC remains unchanged under signed evaluation. If you have a chance to review these additions, we would appreciate your updated assessment. Thank you for your time.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their careful reading of ObjAligner. The central finding of our revision is that learning an object frame from the sequence improves relative pose estimation when the object and camera move independently. This advantage persists when competing methods use the same local point clouds or the same image inputs. New controlled experiments provide direct evidence for this design. We also evaluate translation direction and displacement magnitude and measure the end-to-end cost of recovering poses and a fused surface.We have also updated the supplementary code with entry points for the controlled-input experiments and end-to-end runtime and GPU-memory measurements, together with usage instructions.

One alignment for pose and surface reconstruction

To separate the contribution of the shared frame from the quality of the local geometry, we gave a conventional registration pipeline the point clouds predicted by ObjAligner itself. UOPE56 is particularly informative here because both the camera and the object move, and the dataset was held out from training and checkpoint selection. Registering the local clouds with SIFT, RANSAC and ICP yields 10.29 pose AUC@15. ObjAligner uses a different source of correspondences. At each queried pixel, it predicts where the same 3D point lies in the camera frame and in the shared object frame, so each view provides its own correspondences for rigid alignment. Using this alignment raises AUC@15 to 21.18. Applying the recovered transforms to the same local clouds also raises the visible-surface F-score from 42.48 to 52.01.

A complementary control tests whether pose can be learned directly from the same Stage-I initialization. We conducted matched Stage-II training for direct pose regression, retaining its original relative-pose supervision. Direct regression reaches 4.52 AUC@15 on UOPE56. The advantage of recovering pose through shared geometry extends to COPE119 and HO3D, where ObjAligner also outperforms both registration and direct regression in pose (Tables 4–5).

The pose advantage persists with matched image inputs

Because ObjAligner normally sees the full scene, we also measured how much information outside the target box contributes to its pose estimates. On UOPE56, holding the SAM2 masks fixed and blacking out pixels outside the box lowers ObjAligner’s AUC@15 from 21.42 to 16.39. Original π³ scores 7.38 on exactly those box-masked images. Scene context therefore contributes to pose accuracy, and a substantial gap remains when both models see the same pixels. Under this common input protocol, ObjAligner achieves higher pose AUC and reconstruction F-score on all three moving-object benchmarks. The adaptation experiment further distinguishes these two outcomes. Training π³ on the same datasets with additional object-local geometry supervision improves its reconstruction F-score on all three benchmarks, while its pose AUC decreases (Table 5).

Relative motion accuracy at a common sequence scale

To assess the motion underlying these pose gains, we compare how the object’s pose relative to the camera changes between views. Changing the origin or orientation of the shared frame affects every view in the same way and therefore cancels from each pairwise comparison. Monocular translation retains one unknown scale for the sequence, so we assess direction directly and displacement magnitude at that shared scale.

The main paper and appendix now use the same inference setting for their headline pose and reconstruction results. Under this setting, scoring translation direction over the full 0–180° range penalizes reversals and gives the same AUC@15 to the reported precision on all three benchmarks. To assess displacement magnitude, we fit one nonnegative scale per sequence before comparing the predicted and ground-truth translation vectors. Because every displacement shares that scale, overestimating one displacement relative to another still incurs an error. On UOPE56, normalized translation error falls from 0.631 for π³ to 0.359 for ObjAligner, with lower error on COPE119 and HO3D as well (Table 9).

Joint inference from the test sequence alone

PoseGAM achieves higher pose AUC on all three benchmarks when supplied with CAD-rendered reference views and their known poses (Table 17). ObjAligner obtains its reference from the test sequence itself, using a single box in the first frame to identify the target. We therefore measured the cost of the full path from RGB to both outputs. On an RTX 4090, ObjAligner returns poses and a fused surface for eight frames in 0.57 s, compared with 0.85 s for the SAM2 + π³ pipeline, using 0.78 GiB more peak allocated GPU memory (Table 7). For an unfamiliar object encountered in motion, a single RGB sequence thus supplies both the visible geometry and the reference needed to align it across views.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.