Domain3D-CIL: Evaluating and Improving Heterogeneous 3D Class-Incremental Learning

3D perception plays a crucial role in real-world applications such as autonomous driving, robotics, and AR/VR. In practical scenarios, 3D perception models need to continually adapt to newly emerging 3D object categories, making class-incremental learning (CIL) particularly important. However, 3D point clouds are highly heterogeneous: objects from the same class may come not only from the clean CAD domain, but also from RGB-D camera scans of varying quality, video reconstructions, or even corrupted observations. This heterogeneity increases the complexity of intra-class feature distributions, posing challenges for effective representation and memory in incremental learning. To investigate this problem, we establish the Domain3D-CIL training and evaluation protocol, which includes point cloud data from heterogeneous domains, and adapt a wide range of mainstream CIL methods to the 3D modality for systematic evaluation. Building on this, we introduce PolyMem, an exemplar-free approach that implicitly models rich higher-order statistics of feature distributions to enhance cross-domain continual learning. Experiments demonstrate that our method improves average performance across domains in heterogeneous 3D class-incremental learning, with particularly notable accuracy gains on real-world domains.