acceptodds
Under review as a conference paper at ICLR 2027

Every9D-22M: Large-Scale Real-World 9D Canonicalization of Everyday Objects

Abstract

Estimating the 9D pose of everyday objects from a single real-world image remains challenging. This is largely due to the lack of large-scale supervision. Most existing datasets either rely heavily on synthetic renderings or provide limited coverage of real-world objects. We address this gap with Every9D-22M, a dataset of 9D pose annotations for 22.8M real-world images from 114K object-centric videos spanning 630 everyday object categories, making it nearly two orders of magnitude larger than any prior real-world 9D pose dataset in at least one of category, object, or image count. To achieve this scale, we leverage object-centric videos by reconstructing object-level point clouds via multi-view geometry and aligning similar instances into a shared canonical coordinate frame. Canonical poses are manually annotated for only a small set of reference objects (fewer than 0.01% of all images) and propagated to the remaining instances via cross-instance alignment. All propagated canonical poses are then verified from multiple viewpoints. We further introduce cross-category orientation rules that induce category-level symmetries, enabling symmetry-aware evaluation. Beyond establishing dedicated training and evaluation splits as a benchmark for 9D pose foundation models, we show that training on Every9D-22M generalizes better to other datasets than training on existing datasets, and outperforms prior foundation models in rotation accuracy on all benchmarks except the viewpoint-biased ImageNet3D, where joint training with ImageNet3D surpasses them as well. Data and code will be made publicly available.

Primary area:
datasets and benchmarks
Keywords:
3D representation learning, 3D canonicalization, 9D pose, 3D pose

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_BM49OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a large-scale real-world dataset containing 9D pose annotations for 22.8M images from 114K object-centric videos. The authors propose a scalable canonicalization pipeline that combines multi-view reconstruction, cross-instance alignment, reference-based pose propagation, and human verification. Experiments demonstrate promising cross-dataset generalization and improved pose estimation performance over existing foundation models.

Critical strengths
  • The paper addresses an important limitation in existing 9D pose datasets.
  • The proposed annotation pipeline is well motivated and practical.
  • The dataset offers impressive scale and category diversity. The cross-dataset experiments demonstrate strong generalization performance.
Critical weaknesses
  • The current annotation quality analysis mainly focuses on rotation accuracy using 125 sampled objects. However, translation and object extent are also essential components of 9D pose annotations and may be affected by reconstruction errors, incomplete geometry, and scale ambiguity.
  • The proposed pipeline relies on clustering and aligning different object instances to shared references. While reasonable, the main alignment validation is conducted on a relatively small subset.
  • While the cross-dataset results are encouraging, it remains unclear how much of the improvement comes from the number of training instances, category diversity, or viewpoint coverage.
Questions for authors

Please refer to the Weaknesses.

3 comments

Response to Reviewer BM49 (R3)

We thank the reviewer for their positive assessment of the pipeline, dataset scale, and cross-dataset results. We have added direct annotation-quality measurements and controlled ablations in response to the concerns below; the shared questions are also discussed in the General Response (G2, G3).

W1. Quality of translation and extent

We agree that rotation accuracy alone is not sufficient to validate 9D pose annotations. We therefore manually annotated oriented 3D bounding boxes, including rotation, center, and extent, for 125 training objects from 125 categories, and compared them directly with our pipeline's boxes:

  • Translation and extent: Relative to the manually annotated box diagonal, center error is below 10% for 100% of objects and below 5% for 92.0%. Extent error (box diagonal) is below 10% for 96.0%.
  • 3D IoU and rotation: Pipeline boxes achieve 74.9 mean 3D IoU against manual annotations, with 96.8% exceeding 50 IoU. Rotation accuracy is 92.8% within 30° and 85.6% within 15° (Sec. 4).
  • Downstream transfer: Trained only on Every9D, our model achieves 41.8 3D IoU on Omni6DPose-ROPE with real sensor depth (45.2 in-domain, SAM3D: 39.5), providing complementary evidence of full 9D pose transfer.
  • Reconstruction quality: Every sequence is visually verified, and unreliable geometry is permanently filtered (18.1%, Tab. S3).

Regarding metric scale ambiguity, please see General Response G3. Our model handles arbitrary depth scales through point-cloud normalization, although the annotations themselves are not metric.

W2. Size of the alignment validation

We agree that the alignment validation is based on a relatively small subset. It was the largest we could identify with ground-truth poses for real-world object-centric videos. Importantly, though, the alignment ablation evaluates an intermediate step, not the quality of the final accepted labels:

  1. Exhaustive verification: The alignment ablation (Tab. 2) evaluates predictions before verification. In our full pipeline, every propagated pose is visually checked, requiring 132 hours of human verification. Incorrect alignments are rejected, reprocessed, or filtered. The quality of accepted annotations is assessed separately in Sec. 4 and W1 above.
  2. Downstream evidence: Consistent alignment is important for learning transferable canonical orientations. Trained on Every9D, our model substantially outperforms other real-world training sources on held-out benchmarks (e.g., Omni6DPose Acc@30°: 54.1 vs. 25.1 for ImageNet3D and 17.1 for HANDAL; Tab. 3).

The direct box evaluation in W1, together with exhaustive verification and downstream transfer, therefore provides additional evidence about the final annotations beyond the small alignment ablation.

W3. Instances vs. category diversity vs. viewpoint coverage

Please see General Response G2. We conducted controlled single-factor ablations and training-scale experiments (Tab. 6). Reducing viewpoint coverage or category diversity causes drops of up to 7.8 and 9.4 points, respectively, while reducing the instance count to 38% with all categories retained causes at most a 2.0-point drop. These results show that dataset breadth contributes substantially to cross-dataset generalization beyond instance count.

I appreciate the authors' response. My score remains unchanged.

We sincerely appreciate your time, positive assessment, and constructive suggestions, which have helped us further refine and strengthen the manuscript.

Reviewer_b2NLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Every9D-22M, a large-scale real-world dataset comprising 9D bounding box annotations across 22.8M frames from 114K object-centric videos and 630 categories derived from uCO3D. To scale annotations efficiently, the authors develop a semi-automated pipeline that clusters visual embeddings, annotates representative medoid instances manually, aligns remaining instances using geometry and appearance correspondence, and verifies quality via multi-view human inspection. They standardize category-level canonical coordinate frames and symmetry definitions, and evaluate a monocular baseline to demonstrate cross-domain transferability while addressing the viewpoint biases prevalent in existing benchmarks.

Critical strengths
  1. Efficient and scalable canonicalization pipeline: The combination of feature clustering, medoid selection, cross-instance alignment, and multi-view verification scales 9D bounding box annotations to 114K real-world videos with only 199 hours of human supervision, striking a practical balance between scalability and quality control.
  2. Viewpoint diversity and downstream transfer utility: The dataset effectively counters the equatorial viewpoint bias typical of single-image benchmarks like ImageNet3D. Cross-dataset evaluation demonstrates that pre-training on Every9D yields consistent gains for downstream 9D/6D pose estimation tasks.
Critical weaknesses
  1. Overly coarse rotation evaluation metric for downstream models: While the authors analyze label quality at 15 degrees in Section 4, the main model benchmarking in Table 3 relies exclusively on Acc@30 degrees for rotation accuracy. This is an excessively forgiving threshold that fails to reflect the fine-grained rotational accuracy required for practical robotics and AR applications. Standard stricter thresholds (such as Acc@5 degrees, Acc@10 degrees, or median angular error) are omitted from the model comparisons, making it difficult to assess how well foundation models and the proposed baseline perform on high-precision pose estimation.
  2. Metric scale ambiguity and pseudo-depth reliance: Because uCO3D consists of uncalibrated monocular smartphone videos, the structure-from-motion reconstructions lack true physical metric scale. Aligning DepthAnythingV2 predictions with sparse SfM point clouds provides only an up-to-scale relative depth rather than physical sensor ground truth. The paper does not adequately clarify how metric scale ambiguity is resolved across different datasets, which compromises the physical fidelity and reliability of the 3D translation and extent evaluation in full 9D pose estimation.
Questions for authors
  1. Could you report rotational accuracy under stricter thresholds (e.g., Acc@5 degrees, Acc@10 degrees, and median angular error) in Table 3 across both Every9D-test and the external benchmarks?
3 comments

Response to Reviewer b2NL (R2)

We thank the reviewer for highlighting the scalability of our pipeline and the value of Every9D's viewpoint diversity and cross-dataset results. We have addressed both concerns in the revised paper and discuss the full results in the General Response (G1, G3).

W1 / Q1. Stricter rotation thresholds

As requested, we now report symmetry-aware Acc@30° / 10° / 5° and median rotation error across all benchmarks (Tab. 3, 4; General Response G1). Our model trained on Every9D outperforms all evaluated foundation models at every threshold on HANDAL, Omni6DPose, and Every9D-test, while roughly halving median error. It also runs 240× faster than SAM3D (36 vs. 0.15 FPS).

We retain Acc@30° as the primary metric because category-level semantic orientations are not always defined within a few degrees. We agree that our model alone is not yet precise enough for applications requiring few-degree accuracy; in these settings, it could serve as initialization for instance-level refinement.

W2. Metric scale ambiguity and pseudo-depth

The reviewer is correct that our depth is estimated with DepthAnythingV2 and aligned to SfM, rather than measured by a metric sensor. Consequently, Every9D's translation and extent annotations are not metric. We now state this limitation explicitly (Sec. 3.3, 3.7, Limitations).

Our model handles arbitrary input scales through point-cloud normalization, with predictions expressed in the scale of the input depth (General Response G3). HANDAL and Omni6DPose use metric sensor depth, while ImageNet3D lacks scale-aligned depth and annotations, so we train only with the orientation loss on that dataset.

Despite training on non-metric Every9D annotations, our model achieves 41.8 3D IoU on Omni6DPose-ROPE with metric sensor depth, compared with 45.2 for in-domain training and 39.5 for SAM3D. This demonstrates effective transfer to full 9D pose estimation using real sensor depth.

The concerns are largely addressed so I will maintain my positive rating.

We sincerely appreciate your time, positive feedback, and constructive suggestions, which have helped us further sharpen and improve the manuscript.

Reviewer_ui1HOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a large-scale 9D object pose estimation dataset constructed from object-centric videos. The authors propose a canonicalization pipeline that clusters videos using DINOv3 features, manually annotates canonical poses for a small set of reference objects, and propagates these annotations through cross-instance geometric and appearance-based alignment. The resulting poses are then manually verified, with incorrectly aligned instances reprocessed or filtered out.

Critical strengths
  1. The paper introduces a large-scale object pose estimation dataset that could be valuable to the research community if publicly released.

  2. The proposed dataset construction pipeline is generally reasonable and appears reproducible.

Critical weaknesses

For a paper whose primary contribution is a new dataset, I believe three aspects are particularly important: (1) the significance and usefulness of the dataset, (2) the soundness of the dataset construction pipeline, and (3) comprehensive experimental validation of the dataset's utility from multiple perspectives.

Although the paper presents some promising results, several important concerns remain insufficiently addressed. Due to the two-point limit for this section, I elaborate on these concerns in the Questions section below.

Questions for authors

Concerns about the dataset construction pipeline:

  1. Scale consistency. The authors use VGGSfM to reconstruct point clouds for individual objects. However, VGGSfM reconstructions are subject to normalization and scale ambiguity, potentially leading to substantial scale variations for the same object under different backgrounds or capture conditions. This raises concerns about the consistency and reliability of the scale annotations in the proposed dataset. I could not find a clear explanation of how the authors address this issue.

  2. Reconstruction reliability. Based on my experience, reconstruction methods such as VGGSfM, which are primarily trained on scene-level data, can produce unstable predictions on object-centric data. Inaccurate reconstructed point clouds could introduce substantial pose errors during the subsequent alignment optimization. Although the proposed pipeline includes manual verification and correction of problematic samples, I suspect this issue may be systematic rather than limited to a small number of outliers, potentially affecting the overall annotation quality.

Concerns about the experimental evaluation:

  1. Comparability across datasets. Table 3 compares different foundation models and training configurations across multiple datasets. However, different datasets may follow different canonicalization conventions, meaning that models trained on these datasets could predict poses in inconsistent canonical frames. How do the authors ensure that the results across different models and training configurations are directly comparable?

  2. Evaluation metrics. The authors primarily report Acc@30°, which uses a relatively loose angular error threshold of 30°. High accuracy under this threshold may not adequately reflect the precision required for practical applications. I suggest reporting results under stricter thresholds, such as Acc@5° and Acc@10°, to provide a more comprehensive evaluation of pose estimation accuracy.

  3. Generalization to unseen categories. How does the proposed dataset improve model generalization? The current training and evaluation settings appear to share the same object categories, making it difficult to assess generalization to unseen categories. I suggest holding out a subset of object categories during training and evaluating performance on these unseen categories to better demonstrate the generalization benefits of the dataset.

  4. Impact of dataset scale and diversity. Is the observed performance improvement primarily driven by the larger number of object categories or by the increased number of training images per category? The current experiments do not sufficiently disentangle these factors. I suggest conducting controlled ablations using different fractions of the proposed dataset (e.g., 1%, 10%, and 100%), ideally varying category diversity and per-category sample counts separately, to better understand how dataset scale and diversity contribute to pose estimation performance.

5 comments

Response to Reviewer ui1H (R1), part 1/2

We thank the reviewer for the detailed and constructive questions. They motivated several additional experiments, now included in the revised paper (Tab. 3–6, S4, S5, Fig. S5; changes in blue). We address the concerns shared with other reviewers in the General Response (G1–G3) and focus here on the remaining questions.

Q1. Scale consistency

We agree that SfM does not provide consistent metric scale across videos. Within each video, however, cameras, depth, and 9D boxes share one scale. Our model handles the ambiguity through point-cloud normalization (General Response G3). Despite training on non-metric depth, it achieves 41.8 3D IoU on Omni6DPose-ROPE with real sensor depth (45.2 in-domain). We now clarify this limitation in Sec. 3.3, 3.7, and the Limitations.

Q2. Reconstruction reliability

We share the concern that reconstruction failures can be systematic rather than isolated outliers. Our pipeline therefore includes an explicit quality-control stage: every sequence is visually checked from three canonical views, and those with unreliable geometry (noise, missing parts, moving objects) are permanently filtered (18.1%, Tab. S3).

For the accepted sequences, we find several indications that systematic annotation errors are limited:

  1. Annotation quality: Only 2 of 125 audited objects (1.6%) have genuine rotation annotation errors (Sec. 4). The rest are intrinsic ambiguities (e.g., a switched-off smartphone, whose top and bottom look alike). For 125 manually annotated boxes, the pipeline boxes reach a mean 3D IoU of 74.9 (see our reply to R3, W1).
  2. Generalization: Every9D is the strongest real-world training source at every rotation threshold on held-out benchmarks (Tab. 3), with additional results on unseen categories (Tab. 5). Such transfer would be difficult to achieve with significant systematic annotation errors.
  3. Canonical consistency: Our model predicts a consistent convention for 85.5% of frames within a video and 70.2% within an orientation category, outperforming all evaluated foundation models (Tab. S5).

Together, the verification, annotation audit, and downstream results provide evidence that our pipeline effectively limits reconstruction-related annotation errors.

Q3. Comparability across datasets with different conventions

The reviewer raises an important point: a model can predict a consistent orientation and still score poorly if it follows a different canonical convention. Our original submission did not explicitly test this possibility.

We use orientation categories (Sec. 3.3) to define shared semantic axes across datasets. For example, "contain upward" defines a bottle's top, while "along grab two hands left right" defines an axis for trays, inducing a symmetry because its direction is underdetermined. All evaluated categories are mapped to these rules, including when evaluating foundation models.

To test whether lower foundation-model accuracy simply reflects different conventions, we added a convention-invariant consistency analysis (Tab. S5, Fig. S5 left). For predictions correct up to one of 24 possible axis permutations and sign flips, we measure the fraction of frames in each group explained by a single shared transformation. A consistently different convention would score 100%.

Every9D-test (Tab. S5)VideoCategoryOrientation category
Ours (Every9D)85.579.970.2
SAM3D77.869.653.4
OrientAnythingV276.768.850.7
WildDet3D72.663.649.5
OrientAnythingV166.958.841.6

Share (%) of the frames that one single axis flip explains.

Our model shows the highest consistency at all three levels. We observe the same advantage on Omni6DPose (+11.0 points within orientation categories) and, for the jointly trained model, on ImageNet3D (+22.6 points).

SAM3D illustrates the issue clearly: its boxes often fit the objects, but their axis assignments vary even within a video (Fig. S5: a pitcher's handle switches sides, and two wrenches differ by 90°).

Such inconsistencies cannot be resolved by a fixed convention remapping. Thus, convention differences alone cannot explain the gap in Tab. 4. Our model is not perfectly consistent either (70–86%), but training on Every9D substantially improves consistency.

Response to Reviewer ui1H (R1), part 2/2

Q4. Stricter metrics

Please see General Response G1. We now report Acc@30° / 10° / 5° and median rotation error on all benchmarks (Tab. 3, 4). Trained on Every9D, our model outperforms all evaluated foundation models at every threshold on three of four benchmarks, while roughly halving median error.

Q5. Generalization to unseen categories

To test generalization beyond the training categories, we evaluated all 65 Every9D-test categories absent from Every9D-train (Tab. 5, Fig. S5 right). They include beepers, lab coats, newsstands, tambourines, and tacos. Nine categories (e.g., cabinets, projectors, sinks) also appear in ImageNet3D or Omni6DPose, potentially favoring those training sets.

Trained on (Tab. 5)Acc@30°Acc@10°3D IoU
Every9D49.218.647.4
Omni6DPose-SOPE (synthetic)41.913.637.8
ImageNet3D25.64.8–
HANDAL15.75.426.9
OrientAnythingV2 renders (synthetic)13.53.7–

Every9D performs best at both rotation thresholds and in 3D IoU, improving over the strongest alternative, synthetic Omni6DPose-SOPE, by 7.3 points at 30° and 9.6 points in 3D IoU (better in 56 of 65 categories). Together with the cross-dataset results, this demonstrates generalization to unseen object categories.

Q6. Impact of dataset scale and diversity

Please see General Response G2. We conducted the requested 1% / 10% / 100% scale experiments and controlled ablations separating instance count, category diversity, and viewpoint coverage (Tab. 6). The results show that while scale matters, broader category and viewpoint coverage contribute substantially to cross-dataset generalization.

Informativeness

We acknowledge that the submission could have better highlighted its findings beyond the dataset itself. We now summarize them under Insights beyond the dataset in General Response (2/2), including several findings motivated directly by the reviewer's questions.

We thank the reviewer again. The convention analysis (Q3), unseen-category evaluation (Q5), and scale and diversity ablations (Q6) have substantially strengthened the paper. We hope these results address the concerns and would appreciate reconsideration of the rating in light of the new evidence.

Dear Reviewer, As we are nearing the end of the discussion period, we hope you have had a chance to look at our rebuttal and the revised manuscript. If they address your concerns, we would appreciate it if you could consider updating your rating. If not, please let us know what remains. Thank you for your time and feedback.

Comments on the paper

2 comments

General Response (2/2)

G2. Instances vs. category diversity vs. viewpoint coverage (R1, R3)

To understand where Every9D's gains come from, we conducted controlled ablations (Tab. 6) with the model, training schedule, and test sets fixed. We reduced Every9D-train along one dimension at a time to match ImageNet3D in viewpoints (matching its viewpoint histogram), categories (189 instead of 895), or instances (41.7K instead of 110K videos, retaining all categories). We also trained on random 10% and 1% subsets; unlike the single-factor ablations, these also contain fewer categories.

Every9D-train reduced to …ImageNet3DHANDALOmni6DPoseEvery9D-test
ImageNet3D's viewpoints+5.5−3.7−5.3−7.8
ImageNet3D's category count (189)−2.4+2.1−2.6−9.4
ImageNet3D's instance count (38% of the videos)−1.4−0.5−0.9−2.0
a random 10% of the videos−9.0−1.5−6.3−6.4
a random 1% of the videos−16.1−5.8−17.8−21.0

Changes in Acc@30° (percentage points) relative to training on all of Every9D-train. Negative values indicate a performance drop.

The results show:

  • Viewpoints: Matching ImageNet3D's distribution reduces accuracy by 3.7–7.8 points on the other three benchmarks, but improves ImageNet3D itself (+5.5), highlighting its viewpoint bias.
  • Categories: Reducing category diversity causes the largest drop on Every9D-test (−9.4), while slightly improving HANDAL (+2.1).
  • Instances: Reducing videos to 38% while retaining all categories has a small effect (at most −2.0). Random 10% / 1% subsets cause larger drops (up to 9 / 21 points), but also retain fewer categories (596 / 317).
  • Scale: Increasing training data from 1% to 10% improves accuracy by 4–15 points, with a further 2–9 point gain at full scale.

Together, these ablations show that while training scale matters, Every9D's benefits extend beyond instance count: its broad viewpoint and category coverage play a particularly important role in cross-dataset generalization.

G3. Scale ambiguity and pseudo-depth (R1, R2, R3)

The reviewers are right that SfM methods such as VGGSfM cannot recover metric scale. Translation and extent in Every9D are consistent within each video, but are not metric, and we now state this explicitly as a limitation. The model does not require a fixed absolute input scale, which allows us to train on these annotations and transfer to other depth sources:

  1. Scale-agnostic input: Before LitePT, we center each point cloud at its median and normalize it by the 0.90 quantile of its absolute coordinates. We reverse this normalization after prediction, expressing the 9D pose in the scale of the input depth.
  2. Transfer to metric depth: Trained only on Every9D, our model achieves 41.8 3D IoU on Omni6DPose-ROPE with real sensor depth, approaching in-domain training (45.2) and outperforming SAM3D (39.5). Since 3D IoU jointly evaluates rotation, translation, and extent, this demonstrates effective transfer to full 9D pose estimation despite non-metric training annotations.
  3. Depth-domain flexibility: Our model design is flexible and operates on foundation-model-estimated depth (Every9D), synthetic rendered depth (Omni6DPose-SOPE), and real sensor depth (HANDAL, Omni6DPose-ROPE).

We clarify these points in Sec. 3.3, Sec. 3.7, and the Limitations.

Release. We will publicly release all 9D annotations, splits, orientation categories, symmetries, canonicalization code, and trained models. Annotations will be released under CC BY 4.0 (following uCO3D), and code and models under MIT (following LitePT).

Insights beyond the dataset

Beyond introducing the largest real-world 9D pose dataset by a wide margin (22.8M frames, 114K objects, 630 categories), our work provides several findings:

  1. Scalable annotation: 9D poses for 22.8M frames can be obtained with manual pose annotations for fewer than 0.01% of them.
  2. Cross-dataset generalization: Every9D provides the strongest real-world training source for coarse rotation accuracy on held-out benchmarks (Tab. 3).
  3. Foundation-model performance: Training a simple model on Every9D surpasses prior foundation models in rotation accuracy on three of four benchmarks (Tab. 4).
  4. Viewpoint bias: Existing pose benchmarks and foundation models exhibit substantial viewpoint bias (Tab. 3, 6).
  5. Canonical consistency: Foundation models do not consistently follow a single orientation convention, even within individual videos (Tab. S5, Fig. S5).
  6. What drives generalization: Beyond training scale, category diversity and viewpoint coverage contribute substantially to cross-dataset generalization (Tab. 6).
  7. Scalable canonicalization: Cross-category orientation rules with automatically derived symmetries enable consistent canonicalization at this scale.

General Response (1/2)

We thank all reviewers (R1: ui1H, R2: b2NL, R3: BM49) for their constructive feedback. They highlight Every9D-22M's "impressive scale and category diversity" (R3), the "efficient and scalable" pipeline (R2), and the "strong" cross-dataset results (R3).

We have revised the paper (changes in blue). New results appear in Tab. 3–6, S4, S5, Fig. S5, and Sec. 5.2, 5.3, with further revisions in Sec. 3.3, 3.7, 4, and 6. We address three shared concerns below (G1–G3) and reviewer-specific questions in the individual responses.

G1. Stricter rotation metrics (R1, R2)

We agree that Acc@30° alone does not capture fine-grained rotation accuracy. We therefore expanded Tab. 3 and 4 to report symmetry-aware Acc@30° / 10° / 5°, median rotation error, and 3D IoU for all models and benchmarks.

We retain Acc@30° as the primary metric because category-level orientation defines a semantic frame shared across instances, which is not always meaningful to within a few degrees. For ambiguous objects, such as curled animals, even human annotators disagree by more than 15° (Sec. 4). Even in-domain, Acc@5° is only 11–17% on the three category-diverse benchmarks (only HANDAL, 17 categories with well-defined tool axes, is higher). Importantly, our main cross-dataset findings also hold at these thresholds.

Each cell: Acc@30° / 10° / 5° (%, ↑) · median rotation error (↓) · 3D IoU (↑, "–" for rotation-only models, none for ImageNet3D). Best, second best‾\underline{\text{second best}}.

Our model trained on different datasets (Tab. 3). Gray: in-distribution, excluded from ranking. † Synthetic.

Ours, trained onImageNet3DHANDALOmni6DPose-ROPEEvery9D-test
OrientAnythingV2†24.6 / 8.5‾\underline{8.5} / 3.5 · 64.1°2.6 / 0.4 / 0.1 · 102.9° · –13.9 / 3.8 / 1.9 · 77.1° · –13.7 / 4.9 / 3.0 · 77.7° · –
ImageNet3D64.2 / 36.4 / 17.3 ⋅ 15.2°\color{gray}{\text{64.2 / 36.4 / 17.3 · 15.2°}}10.0 / 2.4 / 0.6 · 68.3° · –25.1‾\underline{25.1} / 6.1 / 2.6 · 56.9°‾\underline{56.9°} · –28.0‾\underline{28.0} / 9.0‾\underline{9.0} / 4.3‾\underline{4.3} · 56.3°‾\underline{56.3°} · –
HANDAL7.1 / 1.5 / 0.4 · 105.0°77.7 / 71.8 / 43.0 ⋅ 5.9° ⋅ 46.4\color{gray}{\text{77.7 / 71.8 / 43.0 · 5.9° · 46.4}}17.1 / 6.9‾\underline{6.9} / 3.0‾\underline{3.0} · 87.8° · 15.9‾\underline{15.9}17.1 / 7.3 / 3.8 · 90.3° · 22.3‾\underline{22.3}
Omni6DPose-SOPE†25.3‾\underline{25.3} / 4.7 / 1.3 · 58.2°‾\underline{58.2°}26.5‾\underline{26.5} / 9.8 / 3.9 · 65.1°‾\underline{65.1°} · 27.561.1 / 30.3 / 12.1 ⋅ 19.1° ⋅ 45.2\color{gray}{\text{61.1 / 30.3 / 12.1 · 19.1° · 45.2}}46.9 / 19.6 / 8.0 · 35.1° · 40.7
Every9D39.2 / 8.6 / 2.4‾\underline{2.4} · 38.5°29.8 / 7.4‾\underline{7.4} / 2.1‾\underline{2.1} · 50.9° · 26.9‾\underline{26.9}54.1 / 21.9 / 7.3 · 26.0° · 41.861.4 / 28.2 / 10.7 ⋅ 18.8° ⋅ 50.6\color{gray}{\text{61.4 / 28.2 / 10.7 · 18.8° · 50.6}}

Every9D is the strongest real-world training source at every threshold on every held-out benchmark. Synthetic training sets are ahead only in a few cases at strict thresholds: Omni6DPose-SOPE on HANDAL at 10° / 5°, and OrientAnythingV2 renders on ImageNet3D at 5°.

Comparison with foundation models (Tab. 4). ‡ Also trained on ImageNet3D.

ModelImageNet3DHANDALOmni6DPose-ROPEEvery9D-test
OrientAnythingV148.4 / 24.7 / 11.0 · 30.5°7.4 / 1.0 / 0.3 · 102.1° · –29.8 / 12.5 / 4.3 · 75.6° · –37.5 / 15.5 / 6.2 · 71.0° · –
OrientAnythingV2‡62.7‾\underline{62.7} / 31.4‾\underline{31.4} / 13.9‾\underline{13.9} · 18.7°‾\underline{18.7°}3.8 / 1.0 / 0.3 · 98.8° · –33.4 / 11.1 / 3.8 · 52.5° · –46.1 / 16.2 / 6.1 · 37.8° · –
WildDet3D (RGB-D)37.5 / 15.7 / 6.9 · 67.9°2.7 / 1.3 / 0.2 · 102.5° · 7.026.9 / 7.4 / 2.8 · 84.0° · 4.333.5 / 13.7 / 5.9 · 85.4° · 22.6
SAM3D (RGB-D)55.0 / 25.6 / 10.2 · 26.0°8.8 / 6.3‾\underline{6.3} / 1.8‾\underline{1.8} · 103.4° · 29.335.3 / 18.7‾\underline{18.7} / 6.0‾\underline{6.0} · 69.7° · 39.547.4 / 24.6‾\underline{24.6} / 10.2‾\underline{10.2} · 53.3° · 40.8
Ours (Every9D)39.2 / 8.6 / 2.4 · 38.5°29.8 / 7.4 / 2.1 · 50.9° · 26.9‾\underline{26.9}54.1 / 21.9 / 7.3 · 26.0° · 41.861.4 / 28.2 / 10.7 · 18.8° · 50.6
Ours (ImageNet3D + Every9D)65.4 / 33.7 / 14.7 · 16.2°23.2‾\underline{23.2} / 6.2 / 1.3 · 52.2°‾\underline{52.2°} · 23.051.4‾\underline{51.4} / 17.7 / 5.9 · 29.7°‾\underline{29.7°} · 39.6‾\underline{39.6}57.7‾\underline{57.7} / 24.5 / 9.3 · 21.5°‾\underline{21.5°} · 48.0‾\underline{48.0}

Trained on Every9D alone, our model outperforms all foundation models at every rotation threshold on HANDAL, Omni6DPose, and Every9D-test, while roughly halving median rotation error (e.g., 26.0° vs. 52.5° on Omni6DPose). Joint training with ImageNet3D also achieves the best results on ImageNet3D. Our model runs 240× faster than SAM3D (36 vs. 0.15 FPS), whose training code and data are not publicly available.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.