- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper focuses on diffusion-assisted class-incremental learning (e.g. synthetic images for CIL, or other signal from the diffusion model). It highlights that the current state of the field is that methods do not compare to a control when measuring their value, making it impossible to isolate where the improvements came from. This paper proposes an auditing framework that replaces the diffusion-assisted portion of the system with a control (this work proposing a series of controls which could be fitting for different diffusion-model uses), and reports the difference between the final methods and their controls.
They then use this framework to evaluate CIL methods, providing a new perspective on the methods than what was originally reported. They additionally compare the primary controls with other alternatives, to test that the findings are consistent across controls.
Critical strengths
S1. Important audit and interesting insights. The work reveals a significant problem in the evaluation of diffusion-assisted CIL methods. The insights given in the from the audit of existing methods highlight that this failure to properly measure the impact of methods results in the previous work overlooking problems and incorrectly assigning credit.
S2. Well-organized, generalizable method. The paper presents a very organized solution in the form of an auditing method that can apply to many diffusion-assisted CIL methods. The additionally cross-check the results with alternative controls, decreasing the impacts of fluctuations due to single, over-fitted controls (but see W1 for caveat to interpreting the results).
Critical weaknesses
W1. Unclear presentation of alternative control results. While the primary results in Table 3 are clear, the results of the alternative controls are challenging to evaluate due to lack of inclusion / organization. Table 5 includes only summary information with no direct results, while the indicated Table 16 'full results' also provides only summary statistics. It would be important for this to be clearer (and present), to evaluate if the alternative controls are sound.
W2. No clear direction forward. At the end of this paper, it is clear that many of the existing methods are not working as previously thought, but it is not clear the way forward (e.g. certain methods that work much better than others). If the authors could translate their results into a forward-looking direction, it would strengthen the work.
An additional note. While the diffusion-assisted CIL methods seem well-covered, the work also seems to be missing citations for the bodies of work which create the foundations for studying how diffusion models may assist CIL. It may be fair to include these as well in related works, as the application to CIL would be impossible without them. Some examples include that work that studied how information can be extracted from diffusion models [1,2], and other key work in synthetic training data [3,4,5,6,7].
[1] Open-Vocabulary Attention Maps with Token Optimization for Semantic Segmentation in Diffusion Models, Manchon et al, CVPR'24.
[2] Your Diffusion Model is Secretly a Zero-Shot Classifier, Li et al, ICCV'23.
[3] Diversify Your Vision Datasets with Automatic Diffusion-Based Augmentation, Dunlap et al, NeurIPS'23.
[4] DataDream: Few-shot Guided Dataset Generation, Kim et al, ECCV'24.
[5] Training on Thin Air: Improve Image Classification with Generated Data, Zhou et al, DMLR workshop at ICML'23.
[6] SynthCLIP: Are We Ready for a Fully Synthetic CLIP Training? Hammoud et al, SynData4CV at CVPR'24.
[7] Diversified in-domain synthesis with efficient fine-tuning for few-shot classification, Turrisi da Costa, ICIAP'25.
Questions for authors
N/A