- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Colored point clouds contain both geometric information XYZ required for precise manipulation and appearance cues RGB needed for target identification. However, jointly encoding XYZ and RGB may impair manipulation, while separate pointwise encoding lacks contextual interactions. To address this issue, the paper proposes GeoRGB, which assigns geometry two complementary roles: preserving explicit spatial references and guiding appearance contextualization. Specifically, GeoRGB employs asymmetric dual-tower encoding (ADTE) that independently encodes XYZ features through a pointwise network while aggregating RGB features across multiple scales using geometry-guided RGB contextualization (GRC). The fused features are utilize to condition a flow-matching policy to generate actions. Experiments on two simulation benchmarks, RoboTwin 2.0 and MetaWorld, and four real-world manipulation tasks demonstrate strong performance compared with existing baselines, supporting the effectiveness of the proposed representation across diverse manipulation settings.
Critical strengths
(1) Method. The paper addresses the practical challenge of preserving geometric information for precise manipulation while leveraging color cues for target identification in point-cloud-based robot policies. The authors propose a simple yet effective approach, GeoRGB, which maintains explicit spatial references through an independent pointwise XYZ branch while constructing local RGB context through geometry-guided interactions. The proposed asymmetric dual-tower encoding further enables the integration of complementary spatial and appearance features for action prediction.
(2) Experiments. The authors conduct extensive evaluations on two simulation benchmarks, RoboTwin 2.0 and MetaWorld, and four real-world manipulation tasks, providing comprehensive evidence of the method's effectiveness. GeoRGB achieves substantial performance improvements over existing baselines on RoboTwin 2.0 and real-world tasks. These results, together with the ablation studies, support the practical effectiveness of the proposed encoding design.
Critical weaknesses
(1) Method Clarity. Fixed kernel points are central to GRC: their positions determine the nearest-kernel assignments and distance-based influence weights , which directly affect the aggregation of neighboring features. Although Eq. (3) describes how these kernel points are used, the paper does not specify their number, spatial arrangement, or how their positions are generated or initialized. The authors should provide the kernel-point definitions and configurations to facilitate reproducibility. It would also be helpful to discuss how different kernel-point counts, layouts, and generation procedures affect GeoRGB's performance.
(2) Novelty compared to Existing Methods. The proposed encoding design is closely related to prior approaches: PolarNet [1] combines hierarchical point-cloud features with an explicit XYZ positional embedding. RISE-2 [2] separately encodes 3D geometry and 2D appearance before spatially aligning and fusing them. The authors should clarify the key methodological differences and the additional contribution of GeoRGB's asymmetric encoding and GRC. It would be helpful to compare GeoRGB with the aforementioned methods while keeping the downstream action policy fixed, to further validate its effectiveness.
[1] Chen et al. PolarNet: 3D Point Clouds for Language-Guided Robotic Manipulation.
[2] Fang et al. AirExo-2: Scaling up Generalizable Robotic Imitation Learning with Low-Cost Exoskeletons
Questions for authors
Please refer to the critical weakness.
4 comments
Response to reviewer sYxN (2/2)
W2. It would be helpful to compare GeoRGB with the aforementioned methods while keeping the downstream action policy fixed, to further validate its effectiveness.
A2.
PolarNet studies how to construct point cloud representations that can be combined with language instructions to support robotic manipulation. Introduced in AirExo-2, RISE-2 fuses 3D geometry with 2D visual semantics to improve policy generalization across scenes. The table below summarizes how their encoders differ from GeoRGB.
| Comparison dimension | PolarNet | RISE-2 | GeoRGB |
|---|---|---|---|
| Visual input | XYZRGB point clouds + normals and height | RGB images + voxelized XYZ point clouds | XYZRGB point clouds |
| Feature extraction | PointNeXt jointly encodes geometric and appearance attributes | A sparse 3D ResNet extracts geometric features; DINOv2 extracts semantic features from images | Separate pointwise MLPs extract XYZ spatial features and initial RGB features from the point cloud |
| Contextual interaction | Joint features: hierarchical neighborhood aggregation | Images: self-attention; voxels: 3D convolutional aggregation | XYZ features: independently encoded per point; RGB features: geometry-guided multiscale aggregation |
| Feature fusion | Joint features with additive XYZ positional embeddings | Spatial alignment + concatenation + convolutional aggregation | Pointwise concatenation |
We also keep the downstream action policy fixed and replace only GeoRGB's encoder with the PolarNet or RISE-2 encoder to compare their performance on robotic manipulation tasks. The results are reported below.
| Task | PolarNet Encoder | RISE-2 Encoder | GeoRGB |
|---|---|---|---|
| Stamp Seal | 9 | 22 | 77 |
| Stack Blocks Two | 54 | 65 | 91 |
| Turn Switch | 53 | 31 | 65 |
| Mean | 38.7 | 39.3 | 77.7 |
We interpret these results as follows:
- One possible reason for the lower performance of the PolarNet encoder is the early mixing of geometry and color information. As illustrated in Figure 1, geometry and appearance need to be organized appropriately so that a policy can use them effectively.
- RISE-2 provides effective geometry–semantics alignment, but the low spatial resolution of its final fused features may make it harder for the action network to exploit fine-grained geometric information.
DP3 [1] demonstrates the effectiveness of simple 3D representations for learning robotic manipulation from limited demonstrations. Following this line of thinking, GeoRGB focuses on how to organize geometry and color information to better support action prediction, using only colored point clouds from a single head camera. Combining this encoding design with multiple views, image features, or language conditioning is a valuable direction for future work.
[1] Ze et al. 3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations. RSS 2024.
Reply to Authors Rebuttal
Thank you for the detailed rebuttal and additional experimental evidence.
-
Method Clarity. The clarification of kernel-point configurations and additional experiments on different kernel counts and layouts address my concern regarding method clarity and reproducibility.
-
Novelty and Comparison with Prior Work. The clarification of methodological differences and the additional comparisons with PolarNet and RISE-2 under the same downstream action policy provide stronger evidence of GeoRGB's effectiveness and address my concern.
Considering the motivation, proposed method, and comprehensive experimental results, I maintain my original rating.
Best,
Reviewer sYxN
Thanks for Your Feedback
Thank you for reviewing our clarifications and additional experiments. We are glad they address your concerns!
The core encoder code is included in the supplementary material, and we welcome the community to try our design when building 3D robot policies for real-world or simulated environments!
Response to reviewer sYxN (1/2)
Thank you for your review and helpful suggestions. We provide further details and discussion below.
W1. The authors should provide the kernel-point definitions and configurations to facilitate reproducibility and discuss how different kernel-point counts, layouts, and generation procedures affect GeoRGB's performance.
A1.
Kernel-point definitions and configurations.
We use 15 fixed kernel points, which include the center, six axis-aligned points at radius 0.5, and eight cube vertices at radius 1. Their coordinates are constructed deterministically and the two radii were chosen to support aggregation at multiple radial scales.
Effects of kernel-point counts, layouts, and generation procedures.
To examine how kernel count and layout affect robotic manipulation, we evaluate four alternative configurations:
We interpret these results as follows: