acceptodds
Under review as a conference paper at ICLR 2027

Beyond Recognition: Benchmarking and Advancing Fine-Grained 3D Object Understanding

Abstract

Object-centric understanding is fundamental to how intelligent systems perceive, reason about, and interact with the 3D world. Yet 3D vision-language models (VLMs) have largely been evaluated on category recognition, captioning, and broad question answering, which reveals limited insight about whether they understand the fine-grained properties of individual objects. General-purpose VLMs offer a promising alternative by reasoning over multi-view renderings, but it remains unclear whether they can answer fine-grained, object-specific questions reliably, and whether that reliability persists when an asset’s appearance or orientation changes. We introduce ObjectArena3D, a benchmark of 16,319 questions over 12,431 assets spanning 20 tasks across semantic attributes, camera awareness, part structure, and surface properties. Beyond standard evaluation, it tests robustness by randomizing object orientations and replacing textures with random solid colors. Our evaluation reveals weak viewpoint and orientation reasoning, with category recognition further degraded under randomized orientations. To narrow these gaps, we propose Objective3D, an framework that fine-tunes a pretrained VLM using view augmentation and intra-view attention across aligned rendering modalities. Compared to baseline Qwen3-VL-8B, Objective3D raises the average benchmark score from 42.95 to 70.26, surpassing the strongest proprietary baseline GPT-5.6-Sol by 4.34 points.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Object Understanding, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_taqzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  1. The paper argues that 3D VLMs are evaluated mainly on category recognition and captioning, leaving fine-grained object understanding unmeasured. It introduces ObjectArena3D: 16,319 questions over 12,431 assets, covering 20 tasks in four families (semantic, camera, part structure, surface), plus random-orientation and random-color robustness splits.
  2. It proposes Objective3D, which fine-tunes Qwen3-VL-8B with multi-view augmentation and intra-view attention across aligned rendering modalities (RGB, normal, wireframe, component), trained on a two-stage multitask mixture.
  3. Benchmark results show that camera awareness is the weakest capability, part grounding relies heavily on appearance, and specialist 3D models trail general VLMs. Objective3D raises the average from 42.95 to 70.26, surpassing GPT-5.6 Sol, with gains concentrated on camera tasks.
Critical strengths
  1. The motivation is strong and sound. The benchmark idea is original, large-scale and diverse.
  2. The paper provides insightful analysis that I believe will benefit the community.
  3. The paper is well written, detailed, and easy to follow, with precise task definitions and deterministic scoring.
Critical weaknesses

[W1] Dataset curation unclear. The collection, filtering, and quality-control process is not clearly described, which limits reproducibility.

[W2] Limited novelty in Method development. The method mainly appears to add more modalities to existing approaches, without a clear technical contribution beyond that.

[W3] Annotation details missing. The paper does not report the number of annotators, annotation time, or any cross-checking or agreement measures used to ensure question quality.

[W4] Limited method effectiveness. In the Sec. 4.3 ablation, plain fine-tuning already reaches 58.24 while the proposed components add only +1.39, so most gains seem to come from training data that shares the benchmark’s pipeline and formats and may be biased toward it.

[W5] Limited Generalization. The improvement over Qwen3-VL-8B is only 0.0023, which doesn’t convincingly show the method’s benefit or generalization.

Minor Weakness:

  • Abstract, L41: “an framework” should be “a framework.”
Questions for authors

[Q1] The paper describes only part of its data sources. Could you specify where the 12,431 assets come from, how many annotators were involved, and how label quality was cross-checked for the questions?

[Q2] Regarding the motivation of Objective3D: in the introduction, you show that specialized 3D object models have access to full geometric information, yet they underperform general VLMs. How does this fit your hypothesis that models need to reason explicitly over geometry?

[Q3] Why do you expect surface normals, wireframes, and components to help with question types such as view correspondence and orientation?

[Q4] How do other models perform when fine-tuned on the same training data used for Objective3D? For example, Qwen3-VL-8B trained on your data without architectural modifications, or a specialist model such as EVA01.

[Q5] Where does the training data come from, and how does it relate to the test set? If both are generated by the same pipeline, could this introduce bias that inflates the results?

4 comments

W5. Generalization of Objective3D.

We respectfully disagree that the MMBench result alone supports the conclusion of limited generalization. MMBench assesses broad multimodal understanding and general VQA ability. Our fine-tuning data focus on fine-grained 3D object understanding and do not include a dedicated general-knowledge training component; improvement on MMBench was therefore not the objective of this experiment. Instead, we used it to assess whether object-centric fine-tuning preserves the backbone’s existing capabilities. MMBench accuracy changed from 88.58% to 88.35%, a decrease of only 0.23 percentage points, while ObjectArena3D performance increased from 42.95% to 70.26% (+27.31 points in Table 2). This suggests that Objective3D substantially improves the targeted capabilities while largely retaining general VQA performance.

Beyond this retention check, we further investigate positive transfer to external tasks by evaluating a Qwen3.5-9B-based Objective3D against Qwen3.5-9B on MMLU-Pro, SuperGPQA, MMBench, ERQA, and CountBench. These additional experiments examine both the preservation of general knowledge and VQA capabilities and transfer to related spatial-reasoning and counting tasks, as reported below.

ModelMMLU-Pro [1]SuperGPQA [2]MMBench (Circular) [3]ERQA [4]CountBench [5]
CapabilityKnowledgeKnowledgeGeneral VQASpatial/embodied reasoningCounting
Qwen3.5-9B75.7447.2084.2734.7584.36
Objective3D74.9648.8384.9542.2586.34
Change−0.78+1.63+0.68+7.50+1.98

Objective3D improves on four of the five benchmarks, with the largest gain on ERQA (+7.50 points) and an additional improvement on CountBench (+1.98 points). General VQA and knowledge performance are broadly preserved: MMBench and SuperGPQA improve, although MMLU-Pro decreases by 0.78 points. These results provide evidence that our object-centric training transfers beyond ObjectArena3D, particularly to spatial reasoning and counting, while largely maintaining general-purpose capabilities.

[1] Y. Wang et al., “MMLU-Pro: A more robust and challenging multi-task language understanding benchmark,” NeurIPS, 2024.

[2] X. Du et al., “SuperGPQA: Scaling LLM evaluation across 285 graduate disciplines,” NeurIPS, 2025.

[3] Y. Liu et al., “MMBench: Is your multi-modal model an all-around player?” ECCV, 2024.

[4] Gemini Robotics Team et al., “Gemini Robotics: Bringing AI into the physical world,” arXiv:2503.20020, 2025.

[5] R. Paiss et al., “Teaching CLIP to count to ten,” ICCV, 2023

AuthorsOpenReview ↗

Q2, Q3, Q4, and minor problems

Q2. Why emphasize geometry when specialist 3D models underperform?

We clarify that Objective3D uses rendered images and text at both training and inference, without point-cloud, voxel, or mesh-token inputs (Line 273-274). By “geometry explicitly aligned with RGB,” we mean geometric cues—such as surface normals—rendered in the same image coordinates as the RGB observations.

Why specialist models’ results do not contradict this motivation. Direct access to a 3D representation does not guarantee effective geometric understanding. Sampling and token compression may discard fine geometric or appearance details, while a separate 3D input interface does not automatically inherit the visual representations and image–language alignment of pretrained VLMs. Differences in backbones and supervision also mean that the specialist models’ lower scores cannot be attributed solely to their representations.

Our approach exposes geometric evidence through the pretrained visual interface: multi-view RGB images capture complementary observations, normal maps expose surface orientation, and supplementary component and wireframe renderings highlight structural cues. Intra-view attention connects corresponding patches across modalities, while synchronized view augmentation varies the observations without breaking their alignment.

In the revised results, the complete framework improves Qwen3-VL-8B from 44.73% to 71.03% overall, with 2D–3D projection increasing from 19.64% to 79.96%. This demonstrates stronger spatial capabilities without native 3D inputs. The direction is consistent with VLM3 [1], which shows that standard image-based VLMs can learn diverse 3D tasks through appropriate supervision. We will clarify that our motivation concerns how geometric evidence is represented and used, rather than requiring native 3D inputs.

[1] Cai Z, Liu Z, Xiong Y, et al. VLM3: Vision Language Models Are Native 3D Learners. arXiv preprint arXiv:2605.30561, 2026.

Q3. Why might normals, wireframes, and component renders help correspondence or orientation?

These renderings expose complementary geometric cues through an image-based interface. Surface normals encode local surface orientation even when surfaces have similar colors. Supplementary wireframe and component renderings highlight mesh edges and part boundaries, providing additional structural cues during training.

For view correspondence, these cues may help distinguish regions with similar appearance through their local geometry and surrounding part configuration. For orientation, surface directions and the arrangement of visible parts provide evidence for relating an observation to the annotated object orientation.

Q4. How do plain Qwen and EVA01 perform when trained on the same data?

As shown in Global Response, we include a matched-data baseline using Qwen3.5-9B in our new single-stage ablations. With the same 210,283 training examples and training settings, plain fine-tuning achieves 68.86%, compared with 45.17% before fine-tuning and 69.43% with both proposed components. This demonstrates substantial gains from our training data without architectural modifications, with the components adding 0.57 percentage points.

For EVA01, its official repository lists the release of training code and recipes as future work. We therefore do not currently provide a matched-data fine-tuning comparison for this model.

Minor comment. Wording in the abstract

We will fix typo error in revised version.

W1/W3/Q1/Q5. Dataset construction, annotation effort and quality control, and training–test bias

Our Global Response details the asset sources and task-specific construction procedures. Here, we clarify the annotation workforce, acceptance protocol, and relationship between benchmark and training data.

Curation and human verification

Candidates undergo graphics-based and VLM-assisted screening, followed by category-aware sampling and task-specific validation. Every benchmark question underwent human inspection and correction where necessary. Each task involved 10 annotators with non-overlapping sample assignments. Annotation managers provided task-specific training and conducted sampled checks during a ramp-up phase; annotators proceeded to full-scale work only after their performance stabilized.

Acceptance review was an additional quality-control stage. In each round, two annotation managers and the task designer each independently sampled 10% of the submitted annotations. These samples could overlap. Submissions failing acceptance were returned for correction and rechecked until sampled accuracy exceeded 90%, with final acceptance determined by the task designer. We did not conduct independent double annotation and therefore do not report an inter-annotator agreement score. The 90% threshold is an acceptance criterion, rather than a measured accuracy for the entire benchmark. The annotation phase took approximately one week, with 10 annotators assigned to each task.

Training–test separation

Benchmark assets and their detected duplicates are excluded from training using file MD5, DINOv2 similarity between rendered images, and voxel-based geometry checks. Separation is defined at the underlying-asset level, with rendered and transformed variants inheriting their source asset’s split.

Shared generation procedures and potential bias. Programmatically constructed benchmark and training questions share a common generation framework. Within this framework, we introduce diversity in question formulations and rendering configurations, while distinguishing benchmark references from automated training supervision:

  • Diverse question formulations. Part-understanding questions can identify a target through its part name, an affordance-based expression, or a caption describing its appearance or function. We augment question wording and captions and, where applicable, vary distractors and option order while updating the corresponding reference answers.
  • Object and viewpoint augmentation. Camera questions incorporate object rotation, scaling, and translation, together with rendering-viewpoint augmentation. Reference answers are computed from the resulting transformations, geometry, and camera parameters, keeping supervision consistent with the rendered observations.
  • Human-established surface references. Surface benchmark labels are annotated from scratch and verified by humans. Training labels are selected through agreement between a VLM and feature-based classifiers trained on human-labeled data, using descriptors such as contrast and curvature variation. Surface evaluation therefore uses human-established references rather than automatically accepting the predictions used to construct training supervision.

These measures are designed to reduce dependence on fixed wording and canonical rendering configurations. Together with asset-level separation, they support evaluation on unseen assets under varied observations. However, they do not establish disjoint template distributions or eliminate all potential

W2/W4. Methodological contribution and the roles of training data and architectural adaptation

Our contributions include ObjectArena3D, the accompanying object-centric training data, and the Objective3D adaptation framework. We agree that fine-tuning accounts for the largest share of the performance improvement. The task-specific data construction pipeline and training-data recipe that enable these gains are themselves key contributions of our work.

What Objective3D adds beyond additional modalities. Objective3D introduces an explicit mechanism for exchanging information between spatially corresponding observations. After the visual merger, intra-view attention operates across modalities at the same view and image-patch location, connecting RGB appearance with geometric cues from aligned renderings before language-model processing. A learnable gate controls this interaction, and the fusion module preserves the existing visual-token count and order. Synchronized view augmentation further varies the observations while maintaining alignment across modalities. These components specify how complementary evidence is selected and combined, beyond simply supplying additional images.

How we interpret the gains. We acknowledge that the reported difference between plain fine-tuning (58.24%) and the configuration with view augmentation and intra-view attention (59.63%) is modest. These results do not justify attributing the overall improvement to the added modules. The accompanying data construction is itself a substantive contribution: it converts rendering geometry, part annotations, and verified semantic and surface labels into supervision for fine-grained object understanding. We therefore present Objective3D as a combined data-and-adaptation framework and will distinguish its overall effectiveness from the incremental contribution of individual components.

Could benchmark familiarity explain the improvement? Shared task definitions and question-generation procedures may account for part of the gain, even when training and benchmark assets are disjoint. As described in our Global Response, we exclude benchmark assets and detected duplicates from training and diversify question formulations, referring expressions, object transformations, and viewpoints. These measures reduce reliance on fixed examples and formulations but do not eliminate all pipeline-related bias.

We therefore evaluate transfer beyond ObjectArena3D. In the additional Qwen3.5-9B-based comparison against its own backbone, Objective3D improves ERQA by 7.50 percentage points and CountBench by 1.98 points. These external gains provide evidence that the learned capabilities extend beyond familiarity with ObjectArena3D’s formats. They support the complete training approach, while leaving the causal contribution of each architectural component to be established separately.

Reviewer_bi6ROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ObjectArena3D, a benchmark with 20 tasks that test object attributes, camera reasoning, part structure, and surface properties. It also presents Objective3D, a fine-tuned Qwen3-VL model with view augmentation and cross-modal attention, and evaluates models under both standard and altered viewing conditions.

Critical strengths
  1. The benchmark tests more than object recognition. It brings together tasks such as part localization, orientation estimation, and surface-quality assessment, providing a broader picture of what models can understand about 3D objects.

  2. The paper describes its tasks and scoring rules in detail and uses human checks and rendering geometry to verify answers. Testing multiple models under changes in orientation and appearance also helps reveal weaknesses that standard evaluation may miss.

Critical weaknesses
  1. The paper does not clearly explain what is new compared with PartNeXt, 3DCoMPaT200, and Objaverse++. It is unclear which assets and annotations are reused, which are newly created, and whether training and test sets contain different assets. These details are important for judging both the benchmark’s contribution and the reported gains.

  2. The proposed additions improve the controlled fine-tuning baseline from 58.24 to 59.63, a modest gain of 1.39 points. Since view augmentation and cross-modal attention are tested together and no variation across runs is reported, it is unclear which component helps and whether the gain is consistent.

Questions for authors
  1. Could the authors provide a breakdown of asset sources, reused versus new annotations, and how assets and their variants are separated between training and testing?

  2. Are separate results available for view augmentation and cross-modal attention, and does the improvement hold across repeated training runs?

1 comment

W1 and Q1. What does ObjectArena3D contribute, which annotations are reused, and how are assets split?

Our Global Response provides the asset-source breakdown and the contributions of each source to the task families. Here, we clarify the benchmark’s contribution, annotation reuse, and training–test separation.

What ObjectArena3D contributes.

ObjectArena3D provides a unified, human-verified evaluation of fine-grained 3D object understanding across 20 tasks and four capability families: semantics and attributes, camera awareness, part structure, and surface understanding. PartNext emphasizes hierarchical part understanding, including part-centric QA; 3DCoMPoT200 emphasizes compositional understanding of parts and materials; and Objaverse++ provides asset-quality and attribute annotations. ObjectArena3D brings these complementary aspects together with explicit orientation, cross-view correspondence, and 2D–3D projection tasks under a common evaluation protocol (as shown in Table 1). Its contribution includes task-specific questions and reference answers, deterministic scoring, and controlled orientation and appearance perturbations. These evaluations reveal whether models can interpret an object’s geometry, structure, and surface properties, and whether their performance persists when familiar viewing or appearance cues change.

Reused versus newly constructed annotations.

Across benchmark and training-data construction, the reused annotations are limited to part-level annotations from PartNeXt and 3DCoMPaT200. For example, PartNeXt supplies mesh-face part segmentations and hierarchical semantic part names. We use this inherited supervision to derive counting and rendered localization targets, and construct task-specific questions and referring expressions based on part names, appearance descriptions, and affordances. We explicitly attribute the underlying assets and inherited part annotations to their original sources.

We do not use Objaverse++ annotations or 3DCoMPaT200 material annotations. Surface benchmark labels are annotated from scratch under our task-specific criteria, with controlled synthesis supplementing a subset of texture-abnormality examples. Every benchmark question undergoes human inspection and correction where necessary.

Training–benchmark separation.

Training and benchmark sets may draw from the same source datasets, but benchmark assets and their detected duplicates are excluded from our training data. We perform cross-source deduplication using file MD5, DINOv2 similarity between rendered images, and voxel-based geometry comparison. Separation is enforced at the underlying-asset level: all rendered views, rotated/scaled/translated versions, recolorings, and synthetic texture variants inherit their source asset’s split. The exclusion applies across tasks, so a benchmark asset cannot enter training through another task or a transformed version.

W2 and Q2. Which component helps, and does the gain repeat across runs?

We provide separate component ablations on Qwen3.5-9B under the same single-stage training setup Global Response. Relative to plain fine-tuning (68.86%), view augmentation achieves 68.98% (+0.12), intra-view attention 69.24% (+0.38), and their combination 69.43% (+0.57). Intra-view attention primarily benefits camera and structure tasks, while view augmentation provides more consistent improvements across semantic tasks. We observe consistent improvements across different base models, providing evidence that the benefits of our training and adaptation framework extend beyond a single backbone. These results support the generality of the overall approach, while variability across repeated training runs remains to be evaluated.

Reviewer_rdDwOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ObjectArena3D, a benchmark for fine-grained understanding of rendered 3D assets beyond category recognition. Its 20 tasks cover semantic attributes, camera awareness, part structure, and surface properties, with deterministic scoring and randomized orientation and appearance conditions. It also proposes Objective3D, a two-stage fine-tuning framework for Qwen3-VL-8B that combines view augmentation, intra-view attention across aligned rendering modalities, and fixed multitask data mixtures.

Evaluation covers general-purpose VLMs and specialized 3D models, revealing substantial differences across tasks and sensitivity to altered observations. Objective3D raises the reported average score from 42.95 to 70.26, with particularly large gains in relative camera rotation and inverse projection, while largely retaining its backbone’s MMBench performance.

Critical strengths
  • The task design covers distinct aspects of object understanding, including cross-view correspondence, part counting, language-conditioned part grounding, and material assessment. This provides a more informative diagnostic profile than category recognition alone and addresses concrete needs in inspecting digital 3D assets.

  • Camera targets are derived from known rendering geometry, with human checks for interpretability and reference consistency. Explicit coordinate conventions and deterministic scoring make the geometric tasks comparatively well defined, without relying on an LLM judge.

  • The evaluation includes several model families and detailed diagnostics beyond the headline average. Joint pose accuracy, relaxed localization thresholds, and all-view versus view-averaged grounding scores reveal meaningful differences that the grouped results would otherwise conceal.

  • Intra-view attention uses the known alignment between rendering modalities to exchange information at corresponding image regions while preserving image-token count and order. The reported improvements on several geometric tasks, together with nearly unchanged MMBench performance, provide encouraging evidence for targeted adaptation within the evaluated setting.

Critical weaknesses
  • The evaluation does not establish that ObjectArena3D assets are excluded from both stages of Objective3D training. The reported 1% validation holdout does not resolve whether different questions or renderings of the same asset can occur in training and evaluation. Consequently, the large score improvement does not establish generalization to unseen objects.

  • The randomized-color grounding results do not isolate the claimed dependence on appearance, because the intervention can also change the selected views. Matching question identities and recomputing bounding boxes do not remove this confound. The results demonstrate vulnerability to the combined intervention, but do not establish how much of the decline reflects appearance-dependent structural reasoning.

  • The headline improvement is not adequately attributed to view augmentation and intra-view attention. The controlled 120K-example experiment shows a 1.39-point joint gain over fine-tuning alone, whereas the final result uses a substantially different two-stage training regime. This evidence does not establish the individual mechanisms’ contributions or how much they explain beyond ordinary task-specific fine-tuning.

  • The evaluation does not show how category recognition and property understanding relate on the same assets, despite this being a central motivation for ObjectArena3D. The single-view and multi-view grounding sets are paired, but cross-family asset coverage and property accuracy on correctly recognized objects are not reported. Separate task averages therefore provide limited evidence for the claimed ability to diagnose coherent understanding of individual objects.

Questions for authors

Please see my "weaknesses" comments.

1 comment

W1. Are the benchmark assets excluded from both training stages?

Yes, all assets in benchmark are excluded from training sets based on cross-source duplication including file MD5, DINOv2 similarity between rendered images, and voxel-based geometry comparison. There is no any assets leaky between our ObjectArena3D and benchmark. The reported 1% validation is splited from training set (Appendix A.1). Specifically, our training set involves 234,983 assets which is different with benchmark。

W2. Does randomized-color grounding isolate the effect of appearance?

In randomized-color grounding task, we used Gemini-3.1pro to exclude grounding questions that mention color or texture. The 818 questions retained per grounding task therefore still identify their targets after recoloring (Table A14).

W3. How much of the headline gain comes from view augmentation and intra-view attention?

As shown in Global Response, We provide new controlled, single-stage ablations on Qwen3.5-9B yield 68.86% with plain fine-tuning, 68.98% with view augmentation (+0.12), 69.24% with intra-view attention (+0.38), and 69.43% with both (+0.57). These results show that task-specific fine-tuning contributes most of the improvement, while the components provide smaller, task-dependent gains. The data-construction pipeline and training recipe are also contributions of our work. We will distinguish these incremental effects from the headline improvement, which the single-stage ablations cannot directly decompose.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Global response for benchmark construction

We thank the reviewers for requesting clarification on benchmark construction and training-data provenance, and more ablation studies. We will add details about asset sources, inherited annotations, constructed supervision in the revised version.

Contribution and annotation provenance.

ObjectArena3D contributes a unified evaluation protocol spanning 20 tasks across semantic understanding, camera awareness, part structure, and surface understanding. Its contribution is a new task-specific annotation and evaluation layer built on renderings from existing 3D assets, with human verification and controlled robustness evaluations. Across benchmark and training-data construction, the reused annotations are limited to part-level annotations from PartNeXt and 3DCoMPaT200. In particular, PartNeXt provides mesh-face part segmentations and hierarchical semantic part names, which support the derivation of part counts and rendered localization targets. We additionally construct task-specific questions, referring expressions, and evaluation examples. We do not use Objaverse++ annotations or the material annotations provided by 3DCoMPaT200. Our surface-task labels are obtained through our own annotation process, with controlled synthesis supplementing a subset of texture-abnormality examples.

Asset Sources and Overlap with Public Datasets

Our asset pool combines public datasets, online repositories, and additional purchased assets. The table below shows the proportional allocation for benchmark and training set. No any assets overlap between benchmark and training set.

Asset sourceBenchmark assetsBenchmark percentageBenchmark task QA contribution (%)Training assetsTraining percentageTraining task QA contribution (%)
Sketchfab4,84538.98%32.14 / 50.47 / 39.57 / 43.4386,43636.79%20.94 / 47.70 / 6.31 / 44.29
Objaverse1,2179.79%8.17 / 6.48 / 17.30 / 9.1319,0098.09%3.30 / 6.36 / 36.93 / 7.36
Objaverse-XL1,1549.28%11.55 / 12.85 / 4.93 / 7.8519,3018.22%11.92 / 11.65 / 0.86 / 6.88
PartNeXt6124.92%0.00 / 0.00 / 24.68 / 0.0012,1625.18%0.00 / 0.00 / 44.75 / 0.00
3DCoMPaT20000.00%03,0081.28%0.00 / 0.00 / 9.16 / 0.00
Other public datasets680.55%0.96 / 0.21 / 0.00 / 0.351,2600.54%0.79 / 0.08 / 0.00 / 0.80
Other online repositories2191.76%3.36 / 0.76 / 0.46 / 1.633,6771.57%3.00 / 0.67 / 0.04 / 1.71
Additional collected assets4,31634.72%43.83 / 29.23 / 13.06 / 37.6290,07738.34%60.03 / 33.54 / 1.95 / 38.96
Total12,431100%100 / 100 / 100 / 100234,930100%100 / 100 / 100 / 100

"Task QA contribution" reports each source’s percentage of QA samples within semantic understanding, camera awareness, part structure, and surface understanding, respectively. "Other public datasets" includes Project Aria, 3D-FUTURE, ABO, Smithsonian, Human4DiT, 2K2K, ShapeNet, Toys4K, THuman, CustomHumans, and 3DCoMPaT++.

Training Set and Benchmark Duplication.

We exclude benchmark assets and their detected duplicates from our training data using cross-source checks based on file MD5, DINOv2 similarity between rendered images, and voxel-based geometry comparison. These checks address both exact copies and similar assets that may appear under different identifiers or sources. The exclusion unit is the underlying asset: different rendered views, rotations, scalings, recolorings, and synthetic texture variants must inherit the source asset’s split. The final split documentation should therefore report overlap at the base-asset level across tasks and training stages, rather than only comparing image filenames or QA records.

AuthorsOpenReview ↗

Global Response for training-data provenance

Benchmark Construction

Candidate collection and filtering. We screen collected assets using graphics-derived features and VLM-assisted assessment to remove assets that are unusable, semantically uninterpretable, or unsuitable for the intended task. We then sample across semantic categories derived from Gemma4-31B to reduce concentration in dominant classes and apply task-specific quality checks. Filtering is task-dependent: an asset with an identifiable texture defect can remain a valid example for surface assessment, even though it would be unsuitable as a clean reference for another task.

Annotation construction and task-specific checks. Every benchmark question underwent human inspection. The following summarizes how reference answers were constructed and what annotators checked for each task:

  • Camera awareness. Annotators establish the canonical front and upright direction for orientation tasks using category-specific semantic definitions. We vary object transformations and viewpoints and recompute the answers from the applied transformations and camera parameters. Point correspondence, relative camera rotation, and 2D–3D projection targets are derived from rendering geometry, including visibility and projected coordinates. Human reviewers check interpretability and consistency between the observations and reference answers. This screening retained 4,351 of 5,000 candidates: 1,991/2,500 orientation questions and 2,360/2,500 correspondence/projection questions.
  • Part structure. Existing part annotations provide the basis for counting and localization. For open-vocabulary annotation, Gemini receives highlighted full-object views and local crops of the same part and proposes names, visual descriptions, functional roles, and affordances. The annotation implementation selects up to three views with the largest segmentation-derived bounding boxes and encourages consistent naming of semantically equivalent parts. Human reviewers verify distinct part-instance counts, target visibility, referring-expression clarity, and bounding-box correctness. Examples with incorrect localization or ambiguous references are discarded.
  • Semantics and attributes. Gemma-4-31B and Gemini-3.1 Pro independently labeled 10,000 semantic-category candidates. We selected 3,000 for human review, comprising 900 agreement cases and 2,100 disagreement cases. Human annotators correct labels and identify ambiguous categories or examples. Posture follows a similar agreement/disagreement-based review procedure. Object counts, symmetry and modeling style are reviewed alongside posture, with count-balanced sampling based on preliminary model predictions and symmetry proposals computed after canonical alignment.
  • Surface understanding. Surface benchmark labels were annotated from scratch using task-specific criteria. Controlled texture removal, texture corruption, and brightness alterations supplemented a subset of texture-abnormality examples; annotators checked whether the rendered observations clearly exhibited the intended defects. Surface training labels were selected through agreement between a VLM and feature-based classifiers trained on human-labeled data, using descriptors such as contrast and curvature variation.

Annotation Effort and Quality Control

Annotation workforce and acceptance protocol. Each task involved 10 annotators with non-overlapping sample assignments. Annotation managers provided task-specific training and conducted sampled checks during an initial ramp-up phase. Annotators proceeded to full-scale work only after their performance stabilized.

Following annotation and verification, three reviewers—two annotation managers and one task designer—each independently sampled 10% of the submitted annotations in every acceptance round. These independently drawn samples could overlap. Submissions failing the acceptance criterion were returned for correction and subjected to further checks until sampled accuracy exceeded 90%, with final acceptance determined by the task designer. We did not conduct independent double annotation. The 90% threshold is an acceptance criterion, not an inter-annotator agreement statistic or a measured accuracy for the entire benchmark.

AuthorsOpenReview ↗

Global Response for Ablations

Training-Data Contribution, Component Ablations, and Reproducibility

We agree that fine-tuning accounts for the largest share of the improvement. The task-specific data-construction pipeline and training-data recipe enabling these gains are themselves contributions of our work, alongside the proposed model adaptation.

We conduct additional ablations on Qwen3.5-9B, evaluating view augmentation and intra-view attention separately against an fine-tuning baseline. To control for differences in stage-specific training schedules and enable efficient comparisons, all configurations use a common single-stage protocol. We sample 202,913 samples from the pooled mid-training and supervised fine-tuning data through task-stratified sampling:

Training dataExamplesPercentage
Camera awareness80,05439.5%
Surface properties71,60435.3%
Part structure29,28814.4%
Semantics and attributes21,96710.8%
Total202,913100.0%

Contribution of training data and individual components. Intra-view attention mainly benefits camera and structure tasks, improving correspondence, orientation, and projection by 1.90, 0.82, and 2.62 points, respectively, alongside gains on all structure metrics. This is consistent with better use of geometric cues across views. View augmentation gives more consistent gains on semantic tasks. Combining both achieves 69.43%, up 0.57 points over plain fine-tuning.

MethodsAvg.Category L1Category L2StyleAttr. AnalysisView Corr.Orient. Analysis2D–3D Proj.Part Ground (SV)Part Ground (MV)Part CountingTextureMR QualityMat. Pred.
Proprietary Models
Gemini-3.1 Pro65.9083.8668.7673.1764.6732.3352.3748.0482.9674.0273.6766.1772.9263.72
GPT-5.6 Sol65.9279.1362.9365.4960.4752.7447.2160.6285.4676.7673.2652.7478.6561.44
Claude-5.0 Opus65.2883.2465.8569.3162.6433.7244.7555.3983.3170.9270.0369.3275.9364.25
Ablations (Qwen3.5-9B)
Qwen3.5-9B45.1775.8548.4653.4450.3415.9526.2524.1758.164.7762.9655.5854.4456.79
Qwen3.5 FT68.8681.3761.9577.1063.1764.6757.8668.6878.7866.0364.7263.0186.1061.80
Qwen3.5 FT + View augmentation68.9881.9962.4177.2563.9265.4758.3365.8678.3166.1565.1463.7485.9662.26
Qwen3.5 FT + Intra-view attention69.2481.3761.9577.1063.1766.5758.6871.3079.1466.2765.4562.6385.2461.26
Objective3D69.4382.5362.9977.7164.1466.5758.4166.9779.6266.8766.1862.6686.6861.26

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.