acceptodds
Under review as a conference paper at ICLR 2027

Paired-Control Contribution Audits for Diffusion-Assisted Incremental Recognition

Abstract

Class-incremental learning (CIL) and its few-shot variant (FSCIL) train a classifier on classes that arrive over time while it must keep recognizing earlier ones. Recent methods insert diffusion models into this pipeline, for example to generate replay images of earlier classes, to select among generated samples, or to supply frozen features. When such a method gains accuracy, the gain may come from the generated samples, from simply training on more data, from the selection step, from a stronger representation, or from extra training steps, and final accuracy alone cannot tell these apart. We propose a paired-control contribution audit, an evaluation protocol that tests which component supports such a gain. For each way a method uses diffusion, we rerun the same method with only that use replaced by a simple substitute, for example random instead of reward-based selection, while keeping the data, seeds, and training steps fixed, and we report the accuracy difference over repeated matched runs. We apply the audit to six diffusion-assisted CIL and FSCIL image-classification methods and check each conclusion against alternative substitutes. Each comparison shows what supports a gain only in the tested setting, and the answers differ across methods. Diffusion-generated image replay in one method raises average incremental accuracy by up to 10.1 points, and the gain is not explained by extra training steps, whereas in three other methods the diffusion component shows no reliable gain over a simpler substitute, such as an unselected pool of generated images or replay of stored class statistics. We therefore recommend reporting, alongside final accuracy, how each diffusion component compares with its simplest substitute and the settings in which each claim was tested. Code is available at https://anonymous.4open.science/r/dca-iclr27-F078.

Primary area:
datasets and benchmarks
Keywords:
continual learning, class-incremental learning, diffusion models, generative replay, evaluation methodology

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_nN6QOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper focuses on diffusion-assisted class-incremental learning (e.g. synthetic images for CIL, or other signal from the diffusion model). It highlights that the current state of the field is that methods do not compare to a control when measuring their value, making it impossible to isolate where the improvements came from. This paper proposes an auditing framework that replaces the diffusion-assisted portion of the system with a control (this work proposing a series of controls which could be fitting for different diffusion-model uses), and reports the difference between the final methods and their controls.

They then use this framework to evaluate CIL methods, providing a new perspective on the methods than what was originally reported. They additionally compare the primary controls with other alternatives, to test that the findings are consistent across controls.

Critical strengths

S1. Important audit and interesting insights. The work reveals a significant problem in the evaluation of diffusion-assisted CIL methods. The insights given in the from the audit of existing methods highlight that this failure to properly measure the impact of methods results in the previous work overlooking problems and incorrectly assigning credit.

S2. Well-organized, generalizable method. The paper presents a very organized solution in the form of an auditing method that can apply to many diffusion-assisted CIL methods. The additionally cross-check the results with alternative controls, decreasing the impacts of fluctuations due to single, over-fitted controls (but see W1 for caveat to interpreting the results).

Critical weaknesses

W1. Unclear presentation of alternative control results. While the primary results in Table 3 are clear, the results of the alternative controls are challenging to evaluate due to lack of inclusion / organization. Table 5 includes only summary information with no direct results, while the indicated Table 16 'full results' also provides only summary statistics. It would be important for this to be clearer (and present), to evaluate if the alternative controls are sound.

W2. No clear direction forward. At the end of this paper, it is clear that many of the existing methods are not working as previously thought, but it is not clear the way forward (e.g. certain methods that work much better than others). If the authors could translate their results into a forward-looking direction, it would strengthen the work.

An additional note. While the diffusion-assisted CIL methods seem well-covered, the work also seems to be missing citations for the bodies of work which create the foundations for studying how diffusion models may assist CIL. It may be fair to include these as well in related works, as the application to CIL would be impossible without them. Some examples include that work that studied how information can be extracted from diffusion models [1,2], and other key work in synthetic training data [3,4,5,6,7].

[1] Open-Vocabulary Attention Maps with Token Optimization for Semantic Segmentation in Diffusion Models, Manchon et al, CVPR'24.

[2] Your Diffusion Model is Secretly a Zero-Shot Classifier, Li et al, ICCV'23.

[3] Diversify Your Vision Datasets with Automatic Diffusion-Based Augmentation, Dunlap et al, NeurIPS'23.

[4] DataDream: Few-shot Guided Dataset Generation, Kim et al, ECCV'24.

[5] Training on Thin Air: Improve Image Classification with Generated Data, Zhou et al, DMLR workshop at ICML'23.

[6] SynthCLIP: Are We Ready for a Fully Synthetic CLIP Training? Hammoud et al, SynData4CV at CVPR'24.

[7] Diversified in-domain synthesis with efficient fine-tuning for few-shot classification, Turrisi da Costa, ICIAP'25.

Questions for authors

N/A

Reviewer_sjwJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a paired-control contribution audit protocol to investigate whether diffusion models genuinely contribute to performance improvements in class-incremental learning (CIL) and few-shot class-incremental learning (FSCIL). The protocol replaces one diffusion-related component at a time with a simple alternative while controlling for factors such as data splits, random seeds, training steps, and replay budgets, and evaluates paired accuracy differences with 95% confidence intervals. Experiments on six diffusion-assisted methods show that the benefits of diffusion vary substantially across methods and settings: DiffClass achieves up to a 10.13 percentage-point improvement in average incremental accuracy through Stable Diffusion image replay, while reward-based selection in DCS, diffusion feature replay in DiffFR, and additional diffusion feature views in Beyond show limited or no reliable gains over simpler alternatives. The paper highlights the importance of controlled comparisons, uncertainty reporting, and clearly scoped claims when evaluating diffusion-assisted incremental learning methods.

Critical strengths
  1. The paper addresses a fundamental attribution problem in diffusion-assisted incremental learning: whether observed improvements arise from diffusion itself or from confounding factors such as additional training steps, synthetic data exposure, sample selection, or stronger representations. This is important for fair evaluation and reliable scientific conclusions.
  2. The proposed paired-control audit provides a clear procedure for identifying diffusion-related functional roles, constructing appropriate counterfactual baselines, and reporting paired differences with confidence intervals. The use of alternative controls further strengthens the credibility of the findings.
  3. The study examines six method families across multiple datasets, task configurations, generators, and replay budgets. Results such as the persistent gains from DiffClass replay and the limited contribution of certain selection and feature-replay components challenge simplistic assumptions about diffusion's effectiveness and offer actionable guidance for future research.
Critical weaknesses
  1. The paper evaluates the contributions of individual diffusion-related components by replacing them with simpler alternatives. However, in methods involving multiple components, such as synthetic data generation, sample selection, feature augmentation, and distillation, their effects may be interdependent. Replacing one component at a time measures its conditional contribution under the specific configuration of the remaining components, but does not fully reveal its independent effect or potential synergy with other components. Consequently, the current protocol may not fully explain how individual components contribute to the overall performance of complex diffusion-assisted methods. The authors could strengthen the analysis by conducting factorial ablation experiments on at least one method with multiple diffusion-related components, examining interaction effects and distinguishing conditional component contributions from gains arising from component combinations.
  2. Although the paper attempts to keep factors such as data splits, training steps, and sample counts consistent, differences in data quality, feature distributions, generation costs, and other training conditions may remain across control settings. For example, the compute-matched experiments for DiffClass primarily address whether the observed gains arise from additional optimization steps, but do not establish that diffusion-based generation is superior to alternative data-generation strategies under an equivalent end-to-end computational budget. Similarly, the computational costs of reward models in methods such as DCS are not necessarily fully matched. Therefore, some experiments establish that a particular component outperforms the selected substitute, but do not fully identify the source of its advantage over other viable alternatives. The authors could improve the reliability of their attribution claims by introducing stricter budget-matched controls, reporting the complete computational costs of generation, selection, cache construction, and downstream training, and, where feasible, incorporating alternative controls with more comparable data quality or feature distributions.
  3. The paper assesses uncertainty primarily through paired repeated runs and confidence intervals. However, most experiments involve six repeated runs, while the primary comparisons for DiffFR use only three. Moreover, the paper acknowledges that most confidence intervals are computed with fixed generation caches and therefore primarily capture variability associated with downstream training and class ordering, rather than the full uncertainty introduced by regenerating the caches. This is particularly relevant because cache generation is itself a central component of the methods being audited. Stable results with fixed caches do not necessarily imply comparable stability across the entire generation-and-training pipeline. The authors could strengthen their conclusions by independently repeating both cache generation and downstream training for key experiments, explicitly distinguishing cache-generation uncertainty from downstream-training uncertainty, and increasing the number of repetitions to assess whether the main findings remain robust to randomness throughout the full pipeline.
Questions for authors

Please refer to the Weaknesses section.

Reviewer_Z4wAOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work examines the contribution of diffusion components to class-incremental learning. It introduces a paired-control evaluation protocol that removes or replaces individual components while keeping data, random seeds, and training budgets matched wherever feasible.

Critical strengths

This work proposes a unified protocol, Paired-Control Contribution Audits, to assess the role of diffusion in incremental recognition.

Critical weaknesses

The work primarily standardizes an evaluation procedure, while its contribution to learning methodology or our understanding of learning remains unclear. As presented, I am not convinced that it meets the contribution bar for ICLR.

Questions for authors

(1) Exactly how it differs from conventional ablation studies remains unclear. Sometimes, it appears to be no different from conventional ablation studies. (2) Beyond standardizing evaluation practices, what new insights does this work provide into incremental learning or the mechanisms through which diffusion helps prevent forgetting? (3) How are the simplest substitutes selected, and how sensitive are the conclusions to these choices?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.