- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies heterogeneous 3D class-incremental learning, where samples from the same semantic class may originate from substantially different sources, including clean CAD models, real-world RGB-D scans, video-reconstructed point clouds, and corrupted observations. To systematically evaluate this setting, the authors introduce Domain3D-CIL, a replay-free benchmark protocol that mixes heterogeneous 3D domains within shared semantic classes while withholding domain labels during training. Building on analytic statistical-memory-based continual learning, the paper further proposes PolyMem, which applies an RBF-derived structured polynomial mapping to frozen pretrained 3D features in order to explicitly encode multi-order feature interactions, followed by incremental accumulation of class-wise and Gram statistics and a closed-form ridge-regression classifier. Extensive experiments on four heterogeneous dataset combinations show that PolyMem consistently improves domain-averaged performance over a broad set of PTM-based CIL baselines, with particularly notable gains on real-world domains. Additional evaluations across different backbones, domain numbers, task splits, interaction orders, and mapping dimensions further support the robustness of the proposed approach.
Critical strengths
The paper identifies a practically relevant yet underexplored problem of heterogeneous 3D class-incremental learning. The proposed Domain3D-CIL protocol explicitly considers mixed 3D sources such as CAD models, real-world scans, video reconstructions, and corrupted point clouds, while withholding domain labels during training. This setting better reflects realistic continual-learning scenarios in which samples from the same semantic class may exhibit substantial source-dependent distribution shifts.
PolyMem provides a simple and efficient exemplar-free solution by enriching statistical memory with structured higher-order feature interactions. Instead of storing historical samples or repeatedly fine-tuning the backbone, the method uses a fixed polynomial feature mapping and accumulates compact sufficient statistics for closed-form ridge classification. This design preserves the efficiency and replay-free advantages of analytic CIL while improving the representational capacity of the statistical memory.
The experimental evaluation is extensive and reasonably comprehensive. The authors evaluate on four heterogeneous dataset combinations, compare against a broad range of PTM-based CIL baselines, and further verify robustness across different backbones, numbers of domains, task splits, polynomial orders, and mapping dimensions. The consistent gains, particularly on real-world domains, provide solid empirical support for the proposed method.
Critical weaknesses
PolyMem largely follows the same analytic statistical-memory framework as RanPAC, including frozen pretrained features, accumulated feature statistics, and closed-form ridge regression. The main methodological difference appears to be replacing RanPAC’s random nonlinear mapping with an RBF-derived structured polynomial mapping, and the current experiments do not fully isolate whether the gain comes from modeling heterogeneous high-order statistics or simply from using a stronger nonlinear feature transformation.
The paper motivates PolyMem by showing substantial statistical differences between CAD and real-world features, but it does not directly analyze how the proposed mapping changes these heterogeneous feature distributions. As a result, it remains unclear whether PolyMem actually improves cross-domain intra-class structure or mainly enhances separability in the transformed space.
Since PolyMem directly accumulates and over all observed samples, its statistical memory may be sensitive to the relative numbers of samples from different domains. However, the experiments mainly use the original dataset sizes and do not systematically evaluate different CAD/real mixing ratios or severe domain/class imbalance.
Questions for authors
Since PolyMem and RanPAC differ primarily in the feature mapping used to construct statistical memory, could the authors provide additional controlled comparisons with alternative nonlinear or structured mappings under the same dimensionality and storage budget? This would help clarify whether the improvement is specifically due to higher-order heterogeneous statistics rather than stronger nonlinear feature expansion.
Could the authors provide a direct analysis of the feature distributions before and after the PolyMem mapping, for example through cross-domain intra-class distances, class separability, distribution discrepancy metrics, or feature visualizations? Such evidence would better support the claimed mechanism of higher-order statistical memory.
How robust is PolyMem to different domain-mixing ratios and sample imbalances? In particular, could the authors evaluate settings where CAD and real-world samples appear in substantially different proportions, since the accumulated statistics may otherwise be dominated by the larger domain?
6 comments
Q1 / W1: mapping comparisons
Thank you for your suggestion. At the same dimensionality () and storage budget, we fix and the classifier and change only the mapping. We add two comparisons, listed alongside RanPAC's ReLU random projection in Table R1. The first is RFF for the same RBF kernel: it approximates the full RBF kernel, superposing all orders, including orders 5 and above, in the same coordinates, and thus provides a more complete nonlinear expansion. In contrast, PolyMem (Eq. (4)) projects interactions of orders 1–4 into separate blocks. The second is a PolyMem variant that is also truncated at order 4 but replaces the RBF Taylor coefficients with those of the quartic polynomial kernel , without involving RBF.
PolyMem outperforms RFF on every evaluated domain, with an average gain of about 3 percentage points in average LA across the four benchmarks. With the quartic-kernel coefficients, the variant and the original each perform better on some benchmarks, with an average difference of about 0.7 points. Replacing RanPAC's ReLU random projection with RFF also improves performance, but by only about 2 points on average, less than half of PolyMem's improvement over RanPAC (about 5 points). Thus, a more complete nonlinear expansion explains less than half of the improvement, and the specific coefficients have little effect; the main benefit comes from interactions of orders 1–4 themselves. These interactions are stored in statistical memory as per-class statistics of orders 1–4 (Sec. 4.3). The first four moments correspond to the mean, variance, skewness, and kurtosis compared in Fig. 3 (Appendix F.1). Please see our response to Q2 for a direct analysis of the feature distributions before and after mapping.
Table R1. Mapping comparisons at the same dimensionality () and storage budget, LA (%). Entries: D1 / D2 / Avg. RanPAC results are from Table 1 of the paper. Best values in bold.
| Mapping | MN+SONN | MN+CO3D | SN+Omni | SN+SN-C |
|---|---|---|---|---|
| ReLU random projection (RanPAC) | 89.30 / 80.52 / 84.91 | 90.34 / 81.32 / 85.83 | 83.26 / 74.69 / 78.98 | 76.33 / 69.70 / 73.02 |
| RFF | 90.36 / 83.42 / 86.89 | 89.91 / 82.06 / 85.99 | 85.06 / 78.38 / 81.72 | 80.44 / 74.56 / 77.50 |
| PolyMem (quartic-kernel coefficients) | 90.27 / 82.37 / 86.32 | 91.97 / 83.82 / 87.89 | 87.02 / 82.17 / 84.60 | 86.29 / 78.19 / 82.24 |
| PolyMem (RBF coefficients, main paper) | 92.50 / 87.90 / 90.20 | 90.67 / 84.35 / 87.51 | 87.15 / 83.52 / 85.34 | 83.72 / 77.85 / 80.79 |
Q2 / W2: How the mapping acts on heterogeneous distributions
Thank you for your question. We first clarify how PolyMem works. Its mapping does not align CAD and real-world features or change their within-class structure; it changes what the statistical memory records. After mapping, the statistics for each class include moments of orders 1–4 (Sec. 4.3), allowing classes that are difficult to distinguish using only first- and second-order statistics to be separated. Of the two possibilities you raised, PolyMem therefore corresponds to the latter: the improvement comes from separability in the mapped space. We test this directly at the class level: if the improvement comes from higher-order heterogeneous statistics, classes with larger differences in higher-order statistics between CAD and real-world data should benefit more from adding third- and fourth-order interactions.
On ShapeNet + OmniObject3D, we compute the CAD–real differences in the statistics shown in Fig. 3 for each class and measure their Spearman correlation with the class's real-domain accuracy gain from to (Table R2). Differences in skewness and kurtosis are significantly positively correlated with the gain (), and the correlation remains 0.47 after controlling for accuracy at . In contrast, differences in mean and variance have almost no correlation with the gain (). In other words, the gains from higher-order interactions are concentrated in classes with heterogeneous higher-order statistics.
Table R2. Spearman correlation between per-class CAD–real statistical differences (absolute coordinate-wise differences averaged over 1,024 dimensions) and real-domain accuracy gains from to on SN+Omni.
| Statistical differences | Spearman |
|---|---|
| Mean and variance | 0.08 |
| Skewness and kurtosis | 0.52 |
| Skewness and kurtosis (controlling for accuracy at ) | 0.47 |
Thank you for recognizing the practical relevance of the setting, the efficiency of PolyMem, and the breadth of the experiments. Below, we address your questions on the mapping design, statistical analysis, and sample imbalance.