acceptodds
Under review as a conference paper at ICLR 2027

Domain3D-CIL: Evaluating and Improving Heterogeneous 3D Class-Incremental Learning

Abstract

3D perception plays a crucial role in real-world applications such as autonomous driving, robotics, and AR/VR. In practical scenarios, 3D perception models need to continually adapt to newly emerging 3D object categories, making class-incremental learning (CIL) particularly important. However, 3D point clouds are highly heterogeneous: objects from the same class may come not only from the clean CAD domain, but also from RGB-D camera scans of varying quality, video reconstructions, or even corrupted observations. This heterogeneity increases the complexity of intra-class feature distributions, posing challenges for effective representation and memory in incremental learning. To investigate this problem, we establish the Domain3D-CIL training and evaluation protocol, which includes point cloud data from heterogeneous domains, and adapt a wide range of mainstream CIL methods to the 3D modality for systematic evaluation. Building on this, we introduce PolyMem, an exemplar-free approach that implicitly models rich higher-order statistics of feature distributions to enhance cross-domain continual learning. Experiments demonstrate that our method improves average performance across domains in heterogeneous 3D class-incremental learning, with particularly notable accuracy gains on real-world domains.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Class-incremental learning, 3D point cloud, sim-to-real domain gap

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_mNhROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies heterogeneous 3D class-incremental learning, where samples from the same semantic class may originate from substantially different sources, including clean CAD models, real-world RGB-D scans, video-reconstructed point clouds, and corrupted observations. To systematically evaluate this setting, the authors introduce Domain3D-CIL, a replay-free benchmark protocol that mixes heterogeneous 3D domains within shared semantic classes while withholding domain labels during training. Building on analytic statistical-memory-based continual learning, the paper further proposes PolyMem, which applies an RBF-derived structured polynomial mapping to frozen pretrained 3D features in order to explicitly encode multi-order feature interactions, followed by incremental accumulation of class-wise and Gram statistics and a closed-form ridge-regression classifier. Extensive experiments on four heterogeneous dataset combinations show that PolyMem consistently improves domain-averaged performance over a broad set of PTM-based CIL baselines, with particularly notable gains on real-world domains. Additional evaluations across different backbones, domain numbers, task splits, interaction orders, and mapping dimensions further support the robustness of the proposed approach.

Critical strengths

The paper identifies a practically relevant yet underexplored problem of heterogeneous 3D class-incremental learning. The proposed Domain3D-CIL protocol explicitly considers mixed 3D sources such as CAD models, real-world scans, video reconstructions, and corrupted point clouds, while withholding domain labels during training. This setting better reflects realistic continual-learning scenarios in which samples from the same semantic class may exhibit substantial source-dependent distribution shifts.

PolyMem provides a simple and efficient exemplar-free solution by enriching statistical memory with structured higher-order feature interactions. Instead of storing historical samples or repeatedly fine-tuning the backbone, the method uses a fixed polynomial feature mapping and accumulates compact sufficient statistics for closed-form ridge classification. This design preserves the efficiency and replay-free advantages of analytic CIL while improving the representational capacity of the statistical memory.

The experimental evaluation is extensive and reasonably comprehensive. The authors evaluate on four heterogeneous dataset combinations, compare against a broad range of PTM-based CIL baselines, and further verify robustness across different backbones, numbers of domains, task splits, polynomial orders, and mapping dimensions. The consistent gains, particularly on real-world domains, provide solid empirical support for the proposed method.

Critical weaknesses

PolyMem largely follows the same analytic statistical-memory framework as RanPAC, including frozen pretrained features, accumulated feature statistics, and closed-form ridge regression. The main methodological difference appears to be replacing RanPAC’s random nonlinear mapping with an RBF-derived structured polynomial mapping, and the current experiments do not fully isolate whether the gain comes from modeling heterogeneous high-order statistics or simply from using a stronger nonlinear feature transformation.

The paper motivates PolyMem by showing substantial statistical differences between CAD and real-world features, but it does not directly analyze how the proposed mapping changes these heterogeneous feature distributions. As a result, it remains unclear whether PolyMem actually improves cross-domain intra-class structure or mainly enhances separability in the transformed space.

Since PolyMem directly accumulates QQ and GG over all observed samples, its statistical memory may be sensitive to the relative numbers of samples from different domains. However, the experiments mainly use the original dataset sizes and do not systematically evaluate different CAD/real mixing ratios or severe domain/class imbalance.

Questions for authors

Since PolyMem and RanPAC differ primarily in the feature mapping used to construct statistical memory, could the authors provide additional controlled comparisons with alternative nonlinear or structured mappings under the same dimensionality and storage budget? This would help clarify whether the improvement is specifically due to higher-order heterogeneous statistics rather than stronger nonlinear feature expansion.

Could the authors provide a direct analysis of the feature distributions before and after the PolyMem mapping, for example through cross-domain intra-class distances, class separability, distribution discrepancy metrics, or feature visualizations? Such evidence would better support the claimed mechanism of higher-order statistical memory.

How robust is PolyMem to different domain-mixing ratios and sample imbalances? In particular, could the authors evaluate settings where CAD and real-world samples appear in substantially different proportions, since the accumulated statistics may otherwise be dominated by the larger domain?

6 comments

Thank you for recognizing the practical relevance of the setting, the efficiency of PolyMem, and the breadth of the experiments. Below, we address your questions on the mapping design, statistical analysis, and sample imbalance.

AuthorsOpenReview ↗

Q1 / W1: mapping comparisons

Thank you for your suggestion. At the same dimensionality (M=10,000M=10{,}000) and storage budget, we fix γ\gamma and the classifier and change only the mapping. We add two comparisons, listed alongside RanPAC's ReLU random projection in Table R1. The first is RFF for the same RBF kernel: it approximates the full RBF kernel, superposing all orders, including orders 5 and above, in the same coordinates, and thus provides a more complete nonlinear expansion. In contrast, PolyMem (Eq. (4)) projects interactions of orders 1–4 into separate blocks. The second is a PolyMem variant that is also truncated at order 4 but replaces the RBF Taylor coefficients with those of the quartic polynomial kernel (F⊤F′+1)4(F^\top F'+1)^4, without involving RBF.

PolyMem outperforms RFF on every evaluated domain, with an average gain of about 3 percentage points in average LA across the four benchmarks. With the quartic-kernel coefficients, the variant and the original each perform better on some benchmarks, with an average difference of about 0.7 points. Replacing RanPAC's ReLU random projection with RFF also improves performance, but by only about 2 points on average, less than half of PolyMem's improvement over RanPAC (about 5 points). Thus, a more complete nonlinear expansion explains less than half of the improvement, and the specific coefficients have little effect; the main benefit comes from interactions of orders 1–4 themselves. These interactions are stored in statistical memory as per-class statistics of orders 1–4 (Sec. 4.3). The first four moments correspond to the mean, variance, skewness, and kurtosis compared in Fig. 3 (Appendix F.1). Please see our response to Q2 for a direct analysis of the feature distributions before and after mapping.

Table R1. Mapping comparisons at the same dimensionality (M=10,000M=10{,}000) and storage budget, LA (%). Entries: D1 / D2 / Avg. RanPAC results are from Table 1 of the paper. Best values in bold.

MappingMN+SONNMN+CO3DSN+OmniSN+SN-C
ReLU random projection (RanPAC)89.30 / 80.52 / 84.9190.34 / 81.32 / 85.8383.26 / 74.69 / 78.9876.33 / 69.70 / 73.02
RFF90.36 / 83.42 / 86.8989.91 / 82.06 / 85.9985.06 / 78.38 / 81.7280.44 / 74.56 / 77.50
PolyMem (quartic-kernel coefficients)90.27 / 82.37 / 86.3291.97 / 83.82 / 87.8987.02 / 82.17 / 84.6086.29 / 78.19 / 82.24
PolyMem (RBF coefficients, main paper)92.50 / 87.90 / 90.2090.67 / 84.35 / 87.5187.15 / 83.52 / 85.3483.72 / 77.85 / 80.79
AuthorsOpenReview ↗

Q2 / W2: How the mapping acts on heterogeneous distributions

Thank you for your question. We first clarify how PolyMem works. Its mapping does not align CAD and real-world features or change their within-class structure; it changes what the statistical memory records. After mapping, the statistics for each class include moments of orders 1–4 (Sec. 4.3), allowing classes that are difficult to distinguish using only first- and second-order statistics to be separated. Of the two possibilities you raised, PolyMem therefore corresponds to the latter: the improvement comes from separability in the mapped space. We test this directly at the class level: if the improvement comes from higher-order heterogeneous statistics, classes with larger differences in higher-order statistics between CAD and real-world data should benefit more from adding third- and fourth-order interactions.

On ShapeNet + OmniObject3D, we compute the CAD–real differences in the statistics shown in Fig. 3 for each class and measure their Spearman correlation with the class's real-domain accuracy gain from L=2L=2 to L=4L=4 (Table R2). Differences in skewness and kurtosis are significantly positively correlated with the gain (ρ=0.52\rho=0.52), and the correlation remains 0.47 after controlling for accuracy at L=2L=2. In contrast, differences in mean and variance have almost no correlation with the gain (ρ=0.08\rho=0.08). In other words, the gains from higher-order interactions are concentrated in classes with heterogeneous higher-order statistics.

Table R2. Spearman correlation between per-class CAD–real statistical differences (absolute coordinate-wise differences averaged over 1,024 dimensions) and real-domain accuracy gains from L=2L=2 to L=4L=4 on SN+Omni.

Statistical differencesSpearman ρ\rho
Mean and variance0.08
Skewness and kurtosis0.52
Skewness and kurtosis (controlling for accuracy at L=2L=2)0.47
Reviewer_QLErOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents CoTL, which learns a compact representation of a scene from unposed multi-view images. A VGGT-Ω encoder stores only 16 register tokens per view (with Plücker-ray embeddings) instead of the dense patch grid. A decoder with learnable queries independent of the scene only reads these tokens and predicts 32K 3D Gaussians, trained end-to-end with a rendering loss. The paper introduces novel-view synthesis on RealEstate10K (6-36 views), zero-shot transfer to ACID, 15× fewer tokens than a dense grid, and qualitative scene generation with a flow-matching model in token space.

Critical strengths
  1. Ten CIL methods based on PTM are run with the same backbone and protocol, and results are reported per domain with standard deviations over 3 runs. The PolyMem gains hold across all four pairs of datasets, both alternative backbones, and both extra task splits. This finding is relevant for my assessment in that since the setting is new, a careful baseline study is valuable even outside the method. The code for the baselines is included.
  2. The method is cheap and simple. PolyMem has no backpropagation, a closed-form classifier, and similar storage and runtime to RanPAC (Appendix C). I don’t consider the simplicity to be a weakness in and of itself, as a drop-in change that improves the strongest statistical-memory baseline at no additional cost is practical.
Critical weaknesses
  1. The link between the method and the problem under study is not shown, and the main baseline is absent. The motivation for the paper is intra-class heterogeneity (Fig. 3 shows per-dimension skewness and kurtosis differing between CAD and real data), but nothing in PolyMem is specific to heterogeneity or to 3D. It is another random feature map for kernel ridge regression. The single-domain results in Tab. 3 (K = 1) improve just as well, suggesting that the gain comes from a better kernel approximation in general. Nor is there any evidence that PolyMem actually captures the higher-order statistics shown in Figure 3. The obvious comparison is missing: random Fourier features or Nyström for the same RBF kernel at the same M, and exact kernel ridge regression as an upper bound. Without it, I don't know if the improvement comes from the proposed order-wise polynomial structure or if it's just from the RBF kernel instead of RanPAC's ReLU projection. The L = 2 result supports this claim, being almost identical to RanPAC (Sec. 5.3). This result is the heart of the contribution claim, so it has a big impact on my rating.
  2. The hyperparameters seem to be based on test results. The benchmark is rather thin. From Fig. 5 and Fig. 6, which report the last accuracy, apparently on the test split of two of the four benchmarks, we pick L = 4 and M = 10000. The margins on some pairs are small (e.g., ~1.5 points average LA over TUNA on ShapeNet + ShapeNet-C), so this choice can matter. Domain3D-CIL is a benchmark that concatenates existing datasets in their natural size. The non-CAD domain is just a subset of classes (13 out of 40 for CO3D),, and the mixing ratio is not controlled (acknowledged in the limitations), and the equal-weight average hides these differences. This limits the amount the protocol can count as its own contribution.
Questions for authors
  1. Clarification that could change my score: can you add random Fourier features (and ideally Nyström or exact kernel ridge) for the same RBF kernel and γ at M = 10000? If PolyMem clearly beats RFF, the structured polynomial design is justified.
  2. Clarification: were L and M (and the median-heuristic γ) chosen using test accuracy? If so, how do the results look when these are selected on a held-out validation split?
  3. Clarification: Is there any direct evidence that PolyMem helps because of heterogeneity? For example, does the D1–D2 gap shrink relative to RanPAC, or do the gains on K = 2 exceed those on K = 1 for the same CAD data?
  4. Suggestion (future work, not needed for my assessment): results with controlled mixing ratios and with a first-task-adapted backbone (as in APER) combined with PolyMem.
7 comments

Thank you for recognizing the thorough baseline evaluation, the consistent gains across settings, and the simplicity and practical value of PolyMem. We address your questions below.

AuthorsOpenReview ↗

Q1 / W1: Comparisons with RFF and other kernel approximations

Thank you for suggesting this key comparison. We added the following comparisons as requested; the results are in Table R1.

Additional comparisons. We fix γ\gamma, M=10,000M=10{,}000 and the classifier, changing only the mapping:

  • RFF and Nyström for the same RBF kernel. To keep the mapping fixed after the first task, Nyström landmarks must be drawn from the first task's training data. Its actual dimension is therefore limited by the number of samples in that task and is below M.
  • A PolyMem variant that is likewise truncated at order 4 but replaces the RBF Taylor coefficients with those of the quartic polynomial kernel (F⊤F′+1)4(F^\top F'+1)^4, without involving the RBF kernel.
  • Exact RBF kernel ridge regression, as the theoretical upper limit of kernel approximation. It requires retaining all training samples and cannot be used in practical incremental learning.

Expanding the feature by order as [1,F,vec(F⊗2),… ][1,F,\mathrm{vec}(F^{\otimes 2}),\dots], PolyMem and RFF can be written as

PolyMem: ϕ(F)=[c0,c1P1F,c2P2,vec(F⊗2),…,cLPL,vec(F⊗L)]\text{PolyMem: }\phi(F)=\big[\sqrt{c_0},\\ \sqrt{c_1}P_1F,\\ \sqrt{c_2}P_2\\,\mathrm{vec}(F^{\otimes 2}),\\ \dots,\\ \sqrt{c_L}P_L\\,\mathrm{vec}(F^{\otimes L})\big]

RFF: z(F)=a0+A1F+A2,vec(F⊗2)+A3,vec(F⊗3)+⋯\text{RFF: }z(F)=a_0+A_1F+A_2\\,\mathrm{vec}(F^{\otimes 2})+A_3\\,\mathrm{vec}(F^{\otimes 3})+\cdots

where PnP_n is the TensorSketch random projection and AnA_n comes from the Taylor expansion of cos⁡(ωj⊤F+bj)\cos(\omega_j^\top F+b_j). PolyMem projects each order into its own block and truncates at L=4L=4; RFF approximates the full RBF kernel, and every coordinate superposes all orders, including order 5 and above.

Results. As shown in Table R1:

  • PolyMem outperforms RFF on every evaluated domain, with an average gain of about 3 percentage points in domain-averaged LA across the four benchmarks. Its average LA also exceeds Nyström on all four benchmarks.
  • With the quartic polynomial kernel coefficients, the variant and the original each perform better on some benchmarks, and their average LA differs by about 0.7 percentage points on average across the four benchmarks.
  • Exact RBF kernel ridge regression achieves the highest average LA on all four benchmarks. At the same mapping dimension, PolyMem and RFF trail this reference by about 1.5 and 4.5 percentage points, respectively, averaged across the four benchmarks.

Analysis. Increasing L from 2 to 4 (Sec. 5.3) both introduces third- and fourth-order interactions and brings the mapping closer to the RBF kernel. The results above separate these two possibilities: removing the RBF coefficients yields results close to the original, while RFF and Nyström, which approximate the full RBF kernel, both underperform PolyMem. Thus, under the same storage budget, PolyMem's advantage comes from the order-wise construction of interactions through order 4 and does not depend on the RBF kernel.

In addition, the order-wise blocks allow each order's contribution to be tested separately. The n-th block of Q for class c is

Qc(n)=cn,Pn∑i:,yi=cvec(Fi⊗n),Q_c^{(n)}=\sqrt{c_n}\\,P_n\sum_{i:\\,y_i=c}\mathrm{vec}(F_i^{\otimes n}),

which is a projection of that class's n-th raw moment. The first four moments determine the mean, variance, skewness and kurtosis compared in Fig. 3. This enables order-wise ablations as in Fig. 5. We also considered RFF when designing the method, but because all orders are mixed in every projected coordinate, it does not support this type of ablation.

Table R1. Comparison of mappings on the Uni3D backbone, LA (%). Each cell reports D1 / D2 / Avg.

MethodMN+SONNMN+CO3DSN+OmniSN+SN-C
RFF90.36 / 83.42 / 86.8989.91 / 82.06 / 85.9985.06 / 78.38 / 81.7280.44 / 74.56 / 77.50
Nyström89.72 / 84.22 / 86.9789.96 / 80.37 / 85.1786.34 / 67.13 / 76.7383.27 / 75.32 / 79.30
PolyMem (quartic-kernel coefficients)90.27 / 82.37 / 86.3291.97 / 83.82 / 87.8987.02 / 82.17 / 84.6086.29 / 78.19 / 82.24
PolyMem (RBF coefficients, main paper)92.50 / 87.90 / 90.2090.67 / 84.35 / 87.5187.15 / 83.52 / 85.3483.72 / 77.85 / 80.79
Exact RBF (upper bound)92.73‾\overline{92.73} / 90.06‾\overline{90.06} / 91.40‾\overline{91.40}92.60‾\overline{92.60} / 86.32‾\overline{86.32} / 89.46‾\overline{89.46}87.25‾\overline{87.25} / 84.49‾\overline{84.49} / 85.87‾\overline{85.87}86.83‾\overline{86.83} / 79.71‾\overline{79.71} / 83.27‾\overline{83.27}
AuthorsOpenReview ↗

Q2 / W2: Hyperparameter selection

Thank you for raising this point. The bandwidth γ\gamma does not use the test set: it is computed using only the first task's training set through the median heuristic in Sec. 4.2 (Algorithm 1, line 5). Moreover, Eqs. (1)–(3) show that γ\gamma only determines the coefficients of each order. The variant in Table R1 replaces these coefficients with the binomial coefficients of the quartic polynomial kernel, which do not correspond to the RBF coefficients for any γ\gamma, yet achieves results close to the original. This indicates that PolyMem is not sensitive to γ\gamma.

We use M=10,000M=10{,}000 to match RanPAC's mapping dimension (Sec. 5.3) and compare under the same storage budget. Figure 6 also shows that performance is largely stable above 10,000 dimensions. The choice of L=4L=4 has two motivations: the first four moments correspond to the mean, variance, skewness and kurtosis (Appendix F.1), which are the statistics compared in Fig. 3; and Kernel Pooling (Cui et al., 2017) also shows that order 4 suffices to approximate the RBF kernel. As discussed in Sec. 5.3, when L>4L>4, fewer dimensions are allocated to each order and real-domain accuracy decreases somewhat. However, all evaluated settings with L≥3L\geq3 outperform RanPAC, so the conclusion does not depend on choosing exactly L=4L=4.

Reviewer_39MfOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2(+1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies class-incremental learning (CIL) for 3D point clouds from different domains. It introduces Domain3D-CIL, a protocol that combines existing datasets to evaluate learning new classes with mixed-domain data. The proposed method, PolyMem, applies a polynomial mapping to frozen pretrained features and accumulates statistics to update a ridge regression classifier without storing past training samples. Results on four dataset combinations show higher average accuracy than the evaluated baselines.

Critical strengths

PolyMem performs well across the four dataset combinations, consistently improving the average accuracy over the compared baselines. The gains are particularly clear on OmniObject3D. The results with different backbones and the ablation on polynomial order also support the effectiveness of the method.

Critical weaknesses

1 Additional evidence for the method’s contribution: Are comparisons with KLDA [1] and FeCAM [2] available using the same backbone and comparable memory budgets? Could you also clarify whether CaRE [3] can be fairly evaluated under this protocol and whether comparison results are available?

2 Clarification of the protocol’s contribution: What do the existing results show about how domain mixing affects old-class and new-class accuracy separately? Can these effects be distinguished from differences in class coverage and sample counts, to clarify whether the setting introduces additional forgetting or mainly increases classification difficulty?

[1] Continual Learning Using a Kernel-Based Method Over Foundation Models. AAAI, 2025.

[2] FeCAM: Exploiting the Heterogeneity of Class Distributions in Exemplar-Free Continual Learning. NeurIPS, 2023.

[3] Scaling Continual Learning to 300+ Tasks with Bi-Level Routing Mixture-of-Experts. ICML, 2026.

Questions for authors
  1. How does PolyMem compare with KLDA using the same frozen backbone and comparable memory budgets? Are there results comparing RFF and TensorSketch with the same ridge classifier that could help isolate the benefit of the proposed mapping?

  2. What do the existing results show about how domain mixing affects old-class and new-class accuracy separately? Can these effects be distinguished from differences in class coverage and sample counts, to clarify whether the setting introduces additional forgetting or mainly increases classification difficulty?

6 comments

Thank you for your review and for recognizing the consistent gains of PolyMem. We address your two questions, on further baselines and mappings and on the protocol, in the responses below.

AuthorsOpenReview ↗

Q1 / W1: baselines and mappings comparisons

Thank you for your suggestions. We added two groups of comparisons; the results are in Table R1.

Additional baselines. We added KLDA, FeCAM and CaRE. For KLDA, the RFF dimension is also set to M=10,000M=10{,}000. The official CaRE code is built on the same framework as ours, so it can be adapted directly to 3D CIL.

Additional mapping comparisons. We keep all other hyperparameters fixed and change only the feature mapping. Expanding the feature by order as [1,F,vec(F⊗2),… ][1,F,\mathrm{vec}(F^{\otimes 2}),\dots], PolyMem and RFF can be written as

PolyMem: ϕ(F)=[c0,c1P1F,c2P2,vec(F⊗2),…,cLPL,vec(F⊗L)]\text{PolyMem: }\phi(F)=\big[\sqrt{c_0},\\ \sqrt{c_1}P_1F,\\ \sqrt{c_2}P_2\\,\mathrm{vec}(F^{\otimes 2}),\\ \dots,\\ \sqrt{c_L}P_L\\,\mathrm{vec}(F^{\otimes L})\big]

RFF: z(F)=a0+A1F+A2,vec(F⊗2)+A3,vec(F⊗3)+⋯\text{RFF: }z(F)=a_0+A_1F+A_2\\,\mathrm{vec}(F^{\otimes 2})+A_3\\,\mathrm{vec}(F^{\otimes 3})+\cdots

where PnP_n is the TensorSketch random projection and AnA_n comes from the Taylor expansion of cos⁡(ωj⊤F+bj)\cos(\omega_j^\top F+b_j). PolyMem projects each order into its own block and truncates at L=4L=4; RFF approximates the full RBF kernel, and every coordinate superposes all orders, including order 5 and above. We also tested a PolyMem variant that is likewise truncated at order 4 but replaces the RBF Taylor coefficients with those of the quartic polynomial kernel (F⊤F′+1)4(F^\top F'+1)^4, so it does not involve the RBF kernel at all.

Results. As shown in Table R1:

  • PolyMem achieves a higher average LA than all added baselines on all four benchmarks, with larger gains on the real domain.
  • PolyMem outperforms RFF on every evaluated domain, with an average gain of about 3 percentage points in domain-averaged LA across the four benchmarks.
  • With the quartic polynomial kernel coefficients, the variant and the original each perform better on some benchmarks, and their average LA differs by about 0.7 percentage points on average across the four benchmarks.

Analysis. The gains thus come from the order 1–4 interactions themselves, rather than the specific RBF coefficients or interactions of order 5 and above. The variant with quartic polynomial kernel coefficients involves no RBF kernel, yet performs close to the original, indicating that the benefits of higher-order interactions do not depend on the RBF coefficients. We truncate at order 4 for three reasons: Fig. 5 shows that, at a fixed dimensionality, adding orders above 4 lowers real-domain accuracy; Kernel Pooling (Cui et al., CVPR 2017) shows that order 4 suffices to approximate the RBF kernel; and the first through fourth moments correspond to the mean, variance, skewness and kurtosis compared in Fig. 3, whereas moments of order 5 and above have no widely accepted statistical interpretation and are less stable for characterizing distributions.

In addition, the order-wise blocks allow the contribution of each order to be tested separately: the n-th block of Q is a projection of the per-class n-th order moments, enabling order-wise ablations as in Fig. 5. We also considered RFF when designing the method, but because all orders are mixed in every projected coordinate, it does not support this type of ablation.

Table R1. Feature mappings and CIL methods on the Uni3D backbone, LA (%). Each cell reports D1 / D2 / Avg. Best results in bold.

MethodMN+SONNMN+CO3DSN+OmniSN+SN-C
RFF90.36 / 83.42 / 86.8989.91 / 82.06 / 85.9985.06 / 78.38 / 81.7280.44 / 74.56 / 77.50
PolyMem (quartic-kernel coefficients)90.27 / 82.37 / 86.3291.97 / 83.82 / 87.8987.02 / 82.17 / 84.6086.29 / 78.19 / 82.24
PolyMem (RBF coefficients, main paper)92.50 / 87.90 / 90.2090.67 / 84.35 / 87.5187.15 / 83.52 / 85.3483.72 / 77.85 / 80.79
KLDA (AAAI'25)90.56 / 84.62 / 87.5989.30 / 80.62 / 84.9685.65 / 60.42 / 73.0479.85 / 68.45 / 74.15
FeCAM (NeurIPS'23)91.49 / 86.28 / 88.8991.00 / 80.00 / 85.5085.40 / 76.39 / 80.8981.20 / 73.65 / 77.43
CaRE (ICML'26)90.80 / 84.69 / 87.7589.50 / 78.56 / 84.0384.98 / 75.80 / 80.3979.75 / 72.80 / 76.28
AuthorsOpenReview ↗

Q2 / W2: Forgetting or classification difficulty

Thank you for your question. We combine the results in the paper with additional analyses to explain the challenges different methods face under the mixed-domain protocol.

Class coverage and sample counts. Joint training in Appendix Tables 4–7 uses the same training data and test sets, fine-tuning Uni3D and the classifier jointly on all training data as a performance reference. For every incremental method, the loss relative to this reference is larger on D2D_2 than on D1D_1. This also holds on ShapeNet + ShapeNet-C, where both domains have identical class coverage and a 1:1 training-sample ratio (Appendix H.3). Thus, the difficulty on non-CAD domains is not solely due to differences in class coverage or sample counts.

Old versus new classes. Table R2 reports the final accuracies on old and new classes for ShapeNet + OmniObject3D. For each method, we subtract its new-class loss relative to joint training from its old-class loss, calling this the “additional old-class loss” to quantify the additional impact on old classes. This metric is below 1 percentage point for RanPAC and PolyMem in both domains; for MOS and TUNA, it is approximately 2 points on D1D_1 and 10 points on D2D_2. This indicates that the real domain is particularly affected for methods that update their parameters.

What PolyMem improves. The incremental solutions of RanPAC and PolyMem both equal their respective batch solutions on all observed data (Sec. 4.3); they differ in separability after feature mapping. In Table R2, PolyMem improves both old- and new-class accuracy on the real domain. This shows that the challenges of mixed domains involve not only retaining old classes but also separating heterogeneous classes, which is what PolyMem improves. We will add Table R2 to the appendix.

Table R2. Results on ShapeNet + OmniObject3D: old-class accuracy / new-class accuracy (%) / additional old-class loss (points).

MethodD1 (CAD)D2 (Real)
Joint training88.31 / 89.20 / —85.89 / 87.12 / —
RanPAC82.99 / 84.51 / 0.6374.31 / 76.41 / 0.87
MOS81.36 / 84.63 / 2.3868.18 / 79.05 / 9.64
TUNA83.94 / 86.95 / 2.1263.03 / 75.53 / 11.27
PolyMem86.92 / 88.22 / 0.4183.18 / 85.09 / 0.68

Comments on the paper

1 comment

We thank all reviewers for their detailed comments and for recognizing the relevance of the problem, the simplicity of the method and its consistent gains. We focus on three aspects: (1) the advantage of PolyMem over other methods and feature mappings: it outperforms RFF of the same dimension on every domain and outperforms KLDA, FeCAM and CaRE on all four benchmarks; (2) the role of high-order statistics: a variant without RBF coefficients performs close to the original, showing that the gain comes from the order 1–4 interactions; (3) the practical motivation of the mixed-domain protocol and sample imbalance: the joint-training results in the appendix show that the difficulty on the non-CAD domain is not explained by class coverage or sample counts alone. Details are in the individual responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.