acceptodds
Under review as a conference paper at ICLR 2027

Dual-Role Geometry, Contextual RGB: Representing Point Clouds for 3D Robot Policies

Abstract

Colored point clouds provide robot policies with spatial geometry and appearance cues, whose relative importance varies across manipulation tasks. Jointly encoding XYZ and RGB at each point can reduce performance on tasks requiring precise spatial execution, while separate pointwise encoding still lacks explicit interactions among neighboring appearance features. We propose GeoRGB, a 3D robot policy that preserves pointwise spatial references while organizing appearance features through geometry-guided interactions. Its asymmetric dual-tower encoding (ADTE) pairs a dedicated pointwise XYZ pathway with a multiscale RGB hierarchy, separating spatial reference encoding from appearance contextualization before fusion. Within the RGB hierarchy, geometry-guided RGB contextualization (GRC) aggregates neighboring appearance features within and across scales, using XYZ-defined neighborhoods and directional weights derived from relative positions. Across 50 RoboTwin 2.0 tasks, GeoRGB achieves 77.5% average success, improving over the strongest evaluated baseline by 14.3 percentage points. It further achieves 93.8% success across four real-world tasks. On MetaWorld, a deeper variant reaches 75.0% success, matching the best-performing baseline with fewer than half the parameters and less than one-quarter the training time.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robotic Manipulation, Point Cloud Representation Learning, Imitation Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sYxNOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Colored point clouds contain both geometric information XYZ required for precise manipulation and appearance cues RGB needed for target identification. However, jointly encoding XYZ and RGB may impair manipulation, while separate pointwise encoding lacks contextual interactions. To address this issue, the paper proposes GeoRGB, which assigns geometry two complementary roles: preserving explicit spatial references and guiding appearance contextualization. Specifically, GeoRGB employs asymmetric dual-tower encoding (ADTE) that independently encodes XYZ features through a pointwise network while aggregating RGB features across multiple scales using geometry-guided RGB contextualization (GRC). The fused features are utilize to condition a flow-matching policy to generate actions. Experiments on two simulation benchmarks, RoboTwin 2.0 and MetaWorld, and four real-world manipulation tasks demonstrate strong performance compared with existing baselines, supporting the effectiveness of the proposed representation across diverse manipulation settings.

Critical strengths

(1) Method. The paper addresses the practical challenge of preserving geometric information for precise manipulation while leveraging color cues for target identification in point-cloud-based robot policies. The authors propose a simple yet effective approach, GeoRGB, which maintains explicit spatial references through an independent pointwise XYZ branch while constructing local RGB context through geometry-guided interactions. The proposed asymmetric dual-tower encoding further enables the integration of complementary spatial and appearance features for action prediction.

(2) Experiments. The authors conduct extensive evaluations on two simulation benchmarks, RoboTwin 2.0 and MetaWorld, and four real-world manipulation tasks, providing comprehensive evidence of the method's effectiveness. GeoRGB achieves substantial performance improvements over existing baselines on RoboTwin 2.0 and real-world tasks. These results, together with the ablation studies, support the practical effectiveness of the proposed encoding design.

Critical weaknesses

(1) Method Clarity. Fixed kernel points are central to GRC: their positions determine the nearest-kernel assignments πij\pi_{ij} and distance-based influence weights αij\alpha_{ij}, which directly affect the aggregation of neighboring features. Although Eq. (3) describes how these kernel points are used, the paper does not specify their number, spatial arrangement, or how their positions are generated or initialized. The authors should provide the kernel-point definitions and configurations to facilitate reproducibility. It would also be helpful to discuss how different kernel-point counts, layouts, and generation procedures affect GeoRGB's performance.

(2) Novelty compared to Existing Methods. The proposed encoding design is closely related to prior approaches: PolarNet [1] combines hierarchical point-cloud features with an explicit XYZ positional embedding. RISE-2 [2] separately encodes 3D geometry and 2D appearance before spatially aligning and fusing them. The authors should clarify the key methodological differences and the additional contribution of GeoRGB's asymmetric encoding and GRC. It would be helpful to compare GeoRGB with the aforementioned methods while keeping the downstream action policy fixed, to further validate its effectiveness.

[1] Chen et al. PolarNet: 3D Point Clouds for Language-Guided Robotic Manipulation.

[2] Fang et al. AirExo-2: Scaling up Generalizable Robotic Imitation Learning with Low-Cost Exoskeletons

Questions for authors

Please refer to the critical weakness.

4 comments
AuthorsOpenReview ↗

Response to reviewer sYxN (1/2)

Thank you for your review and helpful suggestions. We provide further details and discussion below.

W1. The authors should provide the kernel-point definitions and configurations to facilitate reproducibility and discuss how different kernel-point counts, layouts, and generation procedures affect GeoRGB's performance.

A1.

Kernel-point definitions and configurations.

We use 15 fixed kernel points, which include the center, six axis-aligned points at radius 0.5, and eight cube vertices at radius 1. Their coordinates are constructed deterministically and the two radii were chosen to support aggregation at multiple radial scales.

Effects of kernel-point counts, layouts, and generation procedures.

All additional experiments use training seed 0 and evaluation seed 0, with 100 evaluation episodes per task.

To examine how kernel count and layout affect robotic manipulation, we evaluate four alternative configurations:

  1. More kernel points: Starting from GeoRGB's default layout, we add a point halfway between the center and each of the 14 noncentral kernel points, yielding 29 points in total.
  2. Spherical layout: We use the 12 vertices of a regular icosahedron at radius 1, together with the center, yielding 13 points.
  3. Dual-shell spherical layout: We add an inner copy of the icosahedral shell at radius 0.5, yielding 25 points in total.
  4. Random kernel layout: We randomly sample 14 points within the unit ball and add the center, yielding 15 points in total.
TaskSpherical layoutDual-shell spherical layoutRandom kernel pointsMore kernel points (GeoRGB)GeoRGB
Stamp Seal7567767677
Stack Blocks Two9089889191
Put Object Cabinet5654565062
Blocks Ranking RGB8785868488
Open Laptop8687848989
Turn Switch6466646465
Mean Success Rate(%)76.374.775.775.778.7

We interpret these results as follows:

  1. Kernel layouts should match scene complexity. An appropriate kernel-point density helps capture spatial details at different scales, while an overly dense layout may make aggregation responses more dependent on specific geometric details. A further hypothesis is that kernel layouts need regularization to improve policy generalization.
  2. Effective kernel layouts may not be unique. Although the alternative layouts generally achieve lower success rates than GeoRGB across these tasks, most still perform well. This suggests that other layouts may also be viable when matched to scene complexity. At the same time, compatibility with multiple layouts further supports GeoRGB's colored point cloud representation design, which aggregates RGB features under geometric guidance. The default kernel layout is one efficient way to implement this process.
AuthorsOpenReview ↗

Response to reviewer sYxN (2/2)

W2. It would be helpful to compare GeoRGB with the aforementioned methods while keeping the downstream action policy fixed, to further validate its effectiveness.

A2.

PolarNet studies how to construct point cloud representations that can be combined with language instructions to support robotic manipulation. Introduced in AirExo-2, RISE-2 fuses 3D geometry with 2D visual semantics to improve policy generalization across scenes. The table below summarizes how their encoders differ from GeoRGB.

Comparison dimensionPolarNetRISE-2GeoRGB
Visual inputXYZRGB point clouds + normals and heightRGB images + voxelized XYZ point cloudsXYZRGB point clouds
Feature extractionPointNeXt jointly encodes geometric and appearance attributesA sparse 3D ResNet extracts geometric features; DINOv2 extracts semantic features from imagesSeparate pointwise MLPs extract XYZ spatial features and initial RGB features from the point cloud
Contextual interactionJoint features: hierarchical neighborhood aggregationImages: self-attention; voxels: 3D convolutional aggregationXYZ features: independently encoded per point; RGB features: geometry-guided multiscale aggregation
Feature fusionJoint features with additive XYZ positional embeddingsSpatial alignment + concatenation + convolutional aggregationPointwise concatenation

We also keep the downstream action policy fixed and replace only GeoRGB's encoder with the PolarNet or RISE-2 encoder to compare their performance on robotic manipulation tasks. The results are reported below.

TaskPolarNet EncoderRISE-2 EncoderGeoRGB
Stamp Seal92277
Stack Blocks Two546591
Turn Switch533165
Mean38.739.377.7

We interpret these results as follows:

  1. One possible reason for the lower performance of the PolarNet encoder is the early mixing of geometry and color information. As illustrated in Figure 1, geometry and appearance need to be organized appropriately so that a policy can use them effectively.
  2. RISE-2 provides effective geometry–semantics alignment, but the low spatial resolution of its final fused features may make it harder for the action network to exploit fine-grained geometric information.

DP3 [1] demonstrates the effectiveness of simple 3D representations for learning robotic manipulation from limited demonstrations. Following this line of thinking, GeoRGB focuses on how to organize geometry and color information to better support action prediction, using only colored point clouds from a single head camera. Combining this encoding design with multiple views, image features, or language conditioning is a valuable direction for future work.

[1] Ze et al. 3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations. RSS 2024.

Reviewer_sYxNOpenReview ↗

Reply to Authors Rebuttal

Thank you for the detailed rebuttal and additional experimental evidence.

  1. Method Clarity. The clarification of kernel-point configurations and additional experiments on different kernel counts and layouts address my concern regarding method clarity and reproducibility.

  2. Novelty and Comparison with Prior Work. The clarification of methodological differences and the additional comparisons with PolarNet and RISE-2 under the same downstream action policy provide stronger evidence of GeoRGB's effectiveness and address my concern.

Considering the motivation, proposed method, and comprehensive experimental results, I maintain my original rating.

Best,
Reviewer sYxN

AuthorsOpenReview ↗

Thanks for Your Feedback

Thank you for reviewing our clarifications and additional experiments. We are glad they address your concerns!

The core encoder code is included in the supplementary material, and we welcome the community to try our design when building 3D robot policies for real-world or simulated environments!

Reviewer_7PwCOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This papers introduces GeoRGB a 3D visuomotor policy that separately performs point cloud (xyz) encoding from the geometry guided appearance (colored point cloud, xyzrgb) encoding. The proposed method is tested on RoboTwin 2.0, MetaWorld, and real-world manipulation tasks. GeoRGB achieves 77.5% average success across 50 RoboTwin tasks (which is only shown in the appendix section), outperforming the strongest reported baseline by 14.3 percentage points. It also achieves 93.8% average success on real-world tasks .

Critical strengths
  1. The paper has clear motivation and meaningful representation design.
  2. The figures helps understanding the proposed methods.
  3. Strong empirical results across multiple benchmarks.
Critical weaknesses
  1. Limited novelty and insufficient discussion of prior work. RGB–point cloud fusion has been extensively studied in 3D dense prediction tasks such as detection and segmentation (e.g., ImVoteNet[1], BPNet[2]), as well as robotic manipulation (e.g., CLIPort[3], SGR[4]). The paper does not sufficiently distinguish its approach from these methods.

  2. Limited generalization across benchmarks. Despite strong RoboTwin results, GeoRGB does not demonstrate consistent improvements on MetaWorld. The default model achieves 73.5%, below ManiFlow's 74.9%, while the larger 12-block variant only matches it (75.0%). This discrepancy raises questions about the generalizability of the proposed representation across different tasks and learning settings.

  3. Insufficiently explained ablations. The ablations lack clear interpretation of what each variant isolates, such as "w/o pointwise XYZ branch" and "Constant RGB input," since geometric information remains available in both settings. Notably, Constant RGB outperforms GeoRGB on Stamp Seal (89.0% vs. 80.7%). Furthermore, removing same-scale interaction barely affects average performance (85.8% vs. 86.5%), questioning its necessity. These findings deserve more analysis rather than simply reporting results.

  4. GeoRGB relies on associating RGB features with 3D points, but real RGB-D sensors suffer from calibration errors, depth noise, and imperfect alignment. Although the paper includes four real-world tasks, it does not evaluate robustness to such errors. Controlled perturbation experiments would help establish whether geometry-guided RGB contextualization is reliable under imperfect real-world sensing.

Reference:

  1. Qi et al. ImVoteNet: Boosting 3D Object Detection in Point Clouds with Image Votes. CVPR 2020
  2. Hu et al. Bidirectional Projection Network for Cross Dimensional Scene Understanding. CVPR 2021
  3. Shridhar et al. CLIPort: What and Where Pathways for Robotic Manipulation. CoRL 2021
  4. Zhang et al. A Universal Semantic-Geometric Representation for Robotic Manipulation. CoRL 2023
Questions for authors

See weakness

5 comments
AuthorsOpenReview ↗

Response to reviewer 7PwC (1/4)

We appreciate your expert feedback, which has been very helpful to us.

W1. The paper does not sufficiently distinguish its approach from these methods.

A1.

About our research

We study colored point cloud representations for 3D visuomotor policies, focusing on the effective use of color. To this end, we propose asymmetric encoding based on dual-role geometry and contextual RGB: XYZ coordinates are independently encoded at each point to provide spatial references, while XYZ-defined neighborhoods and relative positions guide contextual RGB aggregation.

Comparison with ImVoteNet, BPNet, CLIPort, and SGR

ImVoteNet fuses voting cues from images and point clouds to improve 3D object detection. BPNet combines 2D and 3D representations through multilevel bidirectional interaction to improve scene understanding in both domains. CLIPort combines a CLIP semantic pathway with a Transporter spatial pathway to support both semantic understanding and precise spatial reasoning. SGR integrates pretrained image semantics with explicit 3D geometry to construct a general representation for robotic manipulation. The table below provides a more detailed comparison of their encoders with GeoRGB:

Comparison dimensionImVoteNetBPNetCLIPortSGRGeoRGB (ours)
Visual inputRGB images + XYZ point cloudsRGB images + voxelized XYZRGB point cloudsOrthographic top-down RGB-D mapsRGB images + XYZ point cloudsXYZRGB point clouds only
Feature extractionPointNet++ extracts geometric features from point clouds; ResNet-50 extracts image featuresA 3D U-Net jointly encodes voxel geometry and color; a 2D U-Net extracts image featuresA Transporter ResNet jointly encodes color and height; CLIP extracts semantic features from imagesPointNeXt extracts geometric features from point clouds; CLIP extracts semantic features from imagesSeparate pointwise MLPs extract XYZ spatial features and initial RGB features
Contextual interactionImages: 2D convolutional aggregation; point clouds: hierarchical neighborhood aggregationImages: 2D convolutional aggregation; voxels: 3D convolutional aggregationRGB semantic pathway: 2D convolutional aggregation; RGB-D spatial pathway: 2D convolutional aggregationImages: 2D convolutional aggregation; point clouds: hierarchical neighborhood aggregationXYZ features: independently encoded per point; RGB features: geometry-guided multiscale aggregation
Feature fusionConcatenation + vote aggregationBidirectional projection + concatenation + convolutional fusionConcatenation + convolutional fusionConcatenation + hierarchical set abstractionPointwise concatenation

We provide a more in-depth discussion below:

  1. These methods generally obtain appearance or semantic features from images and geometric features from point clouds, voxels, or depth, while GeoRGB constructs spatial and appearance representations directly from the same colored point cloud, avoiding separate image feature extraction and the additional alignment of image features with 3D features.
  2. DP3 [1] demonstrates the effectiveness of simple 3D representations for learning robotic manipulation from limited demonstrations. We follow this viewpoint and explore how to use color effectively with a lightweight encoder. Our results show that GeoRGB achieves strong manipulation performance while keeping training costs low.
  3. Our experiments highlight the benefit of preserving explicit spatial information while enriching RGB features through geometric guidance. This allows a single colored point cloud representation to support both color-based target selection and action execution in 3D space. We believe this line of research has practical value for robotic manipulation and may offer a new perspective on future integration with image semantics or language conditioning.

[1] Ze et al. 3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations. RSS 2024.

AuthorsOpenReview ↗

Response to reviewer 7PwC (2/4)

W2. This discrepancy raises questions about the generalizability of the proposed representation across different tasks and learning settings.

A2.

We use MetaWorld as a benchmark for joint training across tasks and provide the following explanations for the observed performance differences:

  1. Multitask data contain not only different visual scenes but also more diverse action trajectories. We hypothesize that a deeper flow-matching action network may therefore be needed to predict future actions from complex joint visual–state representations.
  2. Our “deeper variants” increase only the depth of the flow-matching action network, with other training settings unchanged. This improves the overall mean success rate from 73.5% to 75.0%. More importantly, success rates in the Very Hard group increase from 57.7%→62.3%→65.0% as the action head grows from 4 to 8 to 12 layers, exceeding ManiFlow [1]'s 56.7%. These results show that the same lightweight colored point cloud representation supports joint learning across 48 tasks, enabling GeoRGB to match a strong baseline with fewer parameters and lower training costs.
  3. We report that GeoRGB achieves comparable performance with a much lower overall training cost. The 12-layer GeoRGB model has 81.8M parameters and takes 17.9 hours for a complete training run, whereas ManiFlow has 181M parameters and takes 79.3 hours.
  4. Compared with RoboTwin 2.0, we observe larger standard deviations in success rates on some MetaWorld tasks for DP3, ManiFlow, and GeoRGB. Examples include Dial Turn, Reach Wall, and Stick Pull (e.g., ±15.0 percentage points). We are considering how insights from community discussions of MetaWorld evaluation variance could help make these comparisons more stable.

[1] Yan et al. ManiFlow: A General Robot Manipulation Policy via Consistency Flow Training. CoRL 2025.

AuthorsOpenReview ↗

Response to reviewer 7PwC (3/4)

W3. The ablations lack clear interpretation of what each variant isolates.

A3.

We acknowledge that our ablation analysis in the main paper was not sufficiently thorough and some variant names may be ambiguous. We therefore provide a more systematic breakdown and analysis of their results below.

  1. These ablations follow the same training and evaluation settings described in the main paper. We extend the evaluation to more tasks.
  2. Variants marked with an asterisk (*) correspond to ablation categories already introduced in the main paper, such as “w/o XYZ tower”.

Ablations of geometric information

VariantStamp SealStack Blocks TwoPut Object CabinetBlocks Ranking RGBOpen LaptopAvg.
GeoRGB779162888981.4
w/o XYZ tower*21183578232.6
Kernel aggregation → global attention528852688268.4
w/o XYZ tower + Global RGB attention01200259.2

Here, we examine the effects of removing the XYZ tower, removing geometric guidance from the RGB tower by replacing RGB interactions with global RGB attention, and weakening both geometric roles simultaneously:

  1. The results show that robotic manipulation depends strongly on geometric information. Weakening both geometric roles substantially reduces the reliability of task completion.
  2. The results support the need for an explicit XYZ tower, which maps raw coordinates to spatial representations suited to action prediction. The geometric information remaining in the contextual RGB tower alone is insufficient to support the spatial understanding needed for robotic manipulation.
  3. Removing geometric guidance from the RGB tower impairs precise manipulation. Without local spatial constraints, RGB-based attention can mix spatially distant points with similar appearances. Even so, this variant generally outperforms baselines such as joint XYZRGB encoding in Fig. 1, despite the absence of geometric guidance in the RGB tower. We view this as reflecting the combined benefits of preserving independent spatial information and allowing contextual interaction among color features.

Ablations of RGB information

VariantStamp SealStack Blocks TwoPut Object CabinetBlocks Ranking RGBOpen LaptopAvg.
GeoRGB779162888981.4
w/o contextual RGB tower*583767128952.6
Constant RGB input*894964128860.4
  1. Removing color causes substantial performance drops on some tasks that depend on color and limits practical applicability. More broadly, ignoring color remains a common limitation of current 3D robot policies that use only point cloud inputs.
  2. “Constant RGB input” sets each point’s RGB to (0,0,0) while retaining the contextual tower. Interestingly, the contextual tower's output is still nonzero: the RGB encoder can map zero input to a nonzero constant vector, and kernel-point assignments and distance weights allow this constant vector to produce aggregation results that vary with local geometry (see the formula below). We attribute this variant's unexpectedly strong performance on Stamp Seal to two factors. The tower retains valuable geometric context. At the same time, because this task depends little on color, zeroing RGB can implicitly regularize learning by suppressing appearance variation when demonstrations are limited.
=Proj(h⋆)⊙∑j∈NX(i)αijwπij.=Proj(\mathbf h_\star)\odot \sum_{j\in\mathcal N_{\mathbf X}(i)} \alpha_{ij}\mathbf w_{\pi_{ij}}.

Here, h⋆=frgb(0)\mathbf h_\star=f_{\mathrm{rgb}}(\mathbf 0) is the initial feature (a nonzero constant vector); NX(i)\mathcal N_{\mathbf X}(i) is the XYZ neighborhood; αij\alpha_{ij} is a distance-based weight; πij\pi_{ij} is the kernel-point index; and wπij\mathbf w_{\pi_{ij}} is the corresponding learnable channel-wise weight.

Ablations of scale interaction in the RGB tower

VariantStamp SealStack Blocks TwoPut Object CabinetBlocks Ranking RGBOpen LaptopAvg.
GeoRGB779162888981.4
w/o cross-scale blocks749255828677.8
w/o same-scale blocks*769259808778.8
  1. Cross-scale blocks aggregate features from neighbors in the preceding, denser point set, capturing local details that support precise manipulation.
  2. Same-scale blocks aggregate features within the current point set, further expanding the effective receptive field and providing broader context.
  3. As evaluation expands to more tasks and scenes of varying complexity, the performance drops caused by removing these blocks become more apparent.
Reviewer_ukxEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GeoRGB, a point-cloud policy that separates pointwise XYZ encoding from geometry-guided contextual RGB encoding. The RGB branch uses a multiscale hierarchy with depthwise kernel-point aggregation, while an independent XYZ branch preserves spatial references. Their fused pointwise features condition a flow-matching action generator.

Critical strengths

The authors conduct comprehensive experiments with 50 RoboTwin tasks, 48-task MetaWorld learning, and four real-world tasks. Other ablation studies such as modality, choices of point cloud backbones, task-dependent color requirements also helps to understand the paper.

Critical weaknesses

The author claims that remove pointwise XYZ branch will drop the average success rate from 86% to 18%, which is a strong evidence to prove the effectiveness of spatial clues. But maybe directly concate xyz can achieve similar results, the authors should do some ablation to explain the value of xyz encoding tower. How does the learned XYZ tower compare with directly appending raw XYZ or positional embeddings to contextual point features? Most component ablations use Stamp Seal and Stack Blocks Two. These tasks are informative, but they also motivate the architecture, so broader validation would strengthen the results.

Questions for authors

see the weakness part and provide more evidence to prove the effectiveness of the paper

1 comment
AuthorsOpenReview ↗

Response to reviewer ukxE

Thank you for your review and helpful suggestions.

W. How does the learned XYZ tower compare with directly appending raw XYZ or positional embeddings to contextual point features?

A. As shown in Figure 2(a), GeoRGB concatenates learned explicit XYZ features with geometry-guided contextual RGB features. To explain the value of the XYZ encoding tower, we keep the downstream action policy fixed and compare three alternatives: directly concatenating raw XYZ, or adding XYZ positional embeddings (with linear-based or Fourier-based variants) to contextual RGB features. The controlled comparison across six tasks is reported below.

Concatenate raw XYZ: fi=[pi;ci]f_i=[p_i;c_i], where pip_i, cic_i, and fif_i denote the XYZ coordinates, contextual RGB feature, and fused feature of point ii, respectively.

Linear-based PE: fi=ci+(WLpi+bL)f_i=c_i+(W_Lp_i+b_L), using a learned linear projection of XYZ.

Fourier-based PE: fi=ci+(WFγ(pi)+bF)f_i=c_i+(W_F\gamma(p_i)+b_F), where γ\gamma is a fixed multi-frequency sine/cosine encoding and (WF,bF)(W_F,b_F) define a learned linear projection.

TaskConcatenate Raw XYZLinear-based PEFourier-based PELearned XYZ tower (GeoRGB)
Stamp Seal30485977
Stack Blocks Two57698691
Put Object Cabinet39434862
Blocks Ranking RGB21525688
Open Laptop89888489
Pick Diverse Bottles66867887
Mean50.364.368.582.3

The results show that directly concatenating raw XYZ or adding positional embeddings reduces performance on most evaluated tasks. We offer two explanations for this observation:

  1. The XYZ tower maps raw coordinates to spatial representations suited to action prediction. Recent encoder-free architectures aim to remove heavy visual encoders from multimodal models and have shown promising scaling behavior [1]. However, these approaches often rely on billion-parameter pretrained backbones [2] and large training datasets. Besides, encoder-free methods often perform worse than encoder-based methods at smaller training scales. Our lightweight encoder has only 1.7M parameters and its XYZ tower reorganizes raw XYZ information to make it easier for the downstream policy to use, enabling nonlinear responses to different spatial regions, directions, and height ranges.

  2. Pointwise concatenation preserves the correspondence between each point’s spatial representation and its contextual RGB features. GeoRGB maintains separate sets of feature channels for spatial information and color context, allowing the two representations to be transformed independently. These representations are then concatenated pointwise, enabling the downstream network to reason jointly about spatial and color information. Furthermore, in the contextual RGB tower, the geometry-guided aggregation at each point depends on its XYZ coordinates. In contrast, directly adding positional encodings mixes spatial and color information within the same channels before downstream reasoning. Interestingly, from another perspective, the learnable spatial representations serve as a simple and effective form of 3D positional encoding when concatenated pointwise with contextual RGB features.

[1] Chen et al. How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining, 2026

[2] Tang et al. Exploring the Potential of Encoder-free Architectures in 3D LMMs, ICLR 2026

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.