- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper explores the use of patch level embeddings and broad wordlist for dataset diffing and argues that broad wordlist is more efficient approach than using MLLM proposals. It also argues that the existing benchmarks are saturated and proposes a new benchmark to address the problems (single concept difference and less noisy) in the existing benchmarks. It explores applications of dataset diffing in finding biases in datasets and models.
Critical strengths
- The paper identifies and tries to solve/ explore several problems related to dataset diffing like efficiency, diffing capability, and benchmarks. The new benchmark is an useful contribution.
Critical weaknesses
1)Presentation and Experiment Design: The paper is not well presented, the writing can significantly improve the paper. The motivation, contributions, and claims could to be presented more clearly and with a better overall flow throughout the paper.
There are many important claims which are overstated as the experiments do not fully support them:
a) Patches vs Image embeddings: There is no comparison between these with the backbone held constant. In Table 2, FG-CLIP underperforms SigLIP and only does better with masking. It is not clear if the gain is coming from masking or the patch level features as results of SigLIP with masking are not shown.
b) The paper claims using an MLLM is expensive as compared to the proposed method, but there is no comparison w.r.t compute or time. MLLMs are used to get competitive performance for the papers own method without quantitative evaluation of efficiency gains.
c) The empirical evidence for saturation of current benchmarks rests mainly on and as paper states since target concept is present in almost all images it is easy for to get saturated while embedding similarity still shows room for improvement.
d) As spurious correlation application shown doesn't involve a classifier, I believe it should be framed differently as without a classifier it is unproven if it is spurious or not.
2)Prior Work and Significance: Some of the problems and techniques presented in this paper have parallels in the prior work and several of those which impact the significance of this paper are missing from prior work and/or baselines or not cited.
a)Works like SpLiCE [1] have used vocabulary set to get dataset level concept summaries.
b)OMP has been used to attribute neurons to text by [2]. Closely related work on dataset comparison [4], redundancy awareness and cost argument of using LLMs for proposals have not been cited [3] .
[1] Bhalla, Usha, et al. "Interpreting clip with sparse linear concept embeddings (splice)." Advances in Neural Information Processing Systems 37 (2024): 84298-84328.
[2] Gandelsman, Yossi, Alexei A. Efros, and Jacob Steinhardt. "Interpreting the second-order effects of neurons in clip." arXiv preprint arXiv:2406.04341 (2024).
[3]Movva, Rajiv, et al. "Sparse autoencoders for hypothesis generation." arXiv preprint arXiv:2502.04382 (2025).
[4] Flora, James, et al. "LatentDiff: Scaling Semantic Dataset Comparison to Millions of Images." arXiv preprint arXiv:2605.00899 (2026).
Questions for authors
1)How does the max-over-patches scoring extend to cases where concepts might not be restricted to a patch as there can be concepts which cover multiple patches in the image? Why was combination of SigLIP and FG-CLIP not considered for Table 2 evaluations.
2)Do you have any results on efficiency gains (compute/time) from the proposed method w.r.t the baselines?
3)What are the limitations of the proposed method and the new benchmark?