acceptodds
Under review as a conference paper at ICLR 2027

Detecting Fine-Grained Differences Between Image Datasets

Abstract

Detecting whether and in particular _how_ two data distributions differ is a fundamental statistical problem. For example, we might be interested in which concepts (expressed in natural language) differ between two classes, the outputs of different generative models, or the same model with different prompt specifications. When working with large and complex image datasets, we intuitively think that first selecting concepts from a broad list using fine-grained patch embeddings, and then optimizing the concepts to better fit the data is an effective workflow. However, prior work instead uses MLLMs to propose concepts and image-level embeddings to select them. Therefore, we systematically compare these techniques and find that selecting concepts from broad lists can replace MLLM-proposals at obviously lower cost. MLLM inference can then be spent on optimizing selected concepts instead to achieve substantial improvements. Additionally, by designing a new benchmark that focuses on low-prevalence and multi-concept distinctions, we find that selecting nonredundant concepts instead of only the most distinguishing concepts matters, and we introduce techniques for how to effectively do this. Finally, we show how our methods address important problems like detecting spurious correlations and biases. Overall, our work presents a fundamental perspective shift on how we analyze interpretable distributional differences between datasets.

Primary area:
interpretability and explainable AI
Keywords:
Dataset Diffing, Multimodal Large Language Models, Fine-Grained Vision, Distribution Shift, Concept Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_w6B6OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper explores the use of patch level embeddings and broad wordlist for dataset diffing and argues that broad wordlist is more efficient approach than using MLLM proposals. It also argues that the existing benchmarks are saturated and proposes a new benchmark to address the problems (single concept difference and less noisy) in the existing benchmarks. It explores applications of dataset diffing in finding biases in datasets and models.

Critical strengths
  1. The paper identifies and tries to solve/ explore several problems related to dataset diffing like efficiency, diffing capability, and benchmarks. The new benchmark is an useful contribution.
Critical weaknesses

1)Presentation and Experiment Design: The paper is not well presented, the writing can significantly improve the paper. The motivation, contributions, and claims could to be presented more clearly and with a better overall flow throughout the paper.

There are many important claims which are overstated as the experiments do not fully support them:

a) Patches vs Image embeddings: There is no comparison between these with the backbone held constant. In Table 2, FG-CLIP underperforms SigLIP and only does better with masking. It is not clear if the gain is coming from masking or the patch level features as results of SigLIP with masking are not shown.

b) The paper claims using an MLLM is expensive as compared to the proposed method, but there is no comparison w.r.t compute or time. MLLMs are used to get competitive performance for the papers own method without quantitative evaluation of efficiency gains.

c) The empirical evidence for saturation of current benchmarks rests mainly on DD and as paper states since target concept is present in almost all images it is easy for DD to get saturated while embedding similarity still shows room for improvement.

d) As spurious correlation application shown doesn't involve a classifier, I believe it should be framed differently as without a classifier it is unproven if it is spurious or not.

2)Prior Work and Significance: Some of the problems and techniques presented in this paper have parallels in the prior work and several of those which impact the significance of this paper are missing from prior work and/or baselines or not cited.

a)Works like SpLiCE [1] have used vocabulary set to get dataset level concept summaries.

b)OMP has been used to attribute neurons to text by [2]. Closely related work on dataset comparison [4], redundancy awareness and cost argument of using LLMs for proposals have not been cited [3] .

[1] Bhalla, Usha, et al. "Interpreting clip with sparse linear concept embeddings (splice)." Advances in Neural Information Processing Systems 37 (2024): 84298-84328.

[2] Gandelsman, Yossi, Alexei A. Efros, and Jacob Steinhardt. "Interpreting the second-order effects of neurons in clip." arXiv preprint arXiv:2406.04341 (2024).

[3]Movva, Rajiv, et al. "Sparse autoencoders for hypothesis generation." arXiv preprint arXiv:2502.04382 (2025).

[4] Flora, James, et al. "LatentDiff: Scaling Semantic Dataset Comparison to Millions of Images." arXiv preprint arXiv:2605.00899 (2026).

Questions for authors

1)How does the max-over-patches scoring extend to cases where concepts might not be restricted to a patch as there can be concepts which cover multiple patches in the image? Why was combination of SigLIP and FG-CLIP not considered for Table 2 evaluations.

2)Do you have any results on efficiency gains (compute/time) from the proposed method w.r.t the baselines?

3)What are the limitations of the proposed method and the new benchmark?

Reviewer_GGZJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies fine-grained differences between image datasets by selecting concepts from a general vocabulary and subsequently refining them with an MLLM. They propose combining patch-level representations, multi-level masking, and redundancy-aware selection, and evaluate the pipeline on existing benchmarks and practical applications.

Critical strengths
  1. The paper focuses on multiple, low-prevalence differences and addresses a meaningful limitation of simpler evaluation settings, which has practical relevance for dataset analysis.

  2. The experiments include component ablations, alternative selection algorithms, sensitivity analyses, and applications. They support the competitiveness of vocabulary-based candidates and demonstrate the benefits of masking, redundancy reduction, and concept refinement.

Critical weaknesses
  1. The paper is not well-organized. The Introduction does not clearly position the contribution relative to earlier candidate-selection approaches. The claim of a “fundamental perspective shift” appears overstated. Key components are introduced in the experimental section rather than incorporated into a complete method description.

  2. Table 2 progressively enhances FG-CLIP but omits corresponding SigLIP configurations with selection and MLLM optimization. Comparing different encoders also makes it difficult to attribute improvements specifically to representation granularity.

  3. The paper claims to reduce MLLM cost; however, the paper does not report end-to-end computational costs or compare performance under matched budgets.

Questions for authors

Please see the above Weaknesses

Reviewer_o1RLOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses finding fine grained differences in image datasets. The goal is ascribed as finding natural language concepts whose prevalence differs between image datasets using a perception function coupled with a search over concepts. Compared to prior works which have MLLMs propose candidate concepts from small subsets and then rank them with image-level embeddings (PTR), the paper instead leverages a large static wordlist (filtered COCA) against every image. It then uses SigLIP/FG-CLIP2 similarities and selects the concepts with the largest score divergence. The primary contributions claimed are : (1) patch level scoring outperforms global embeddings for localized concepts, (2) a static wordlist can replace MLLM proposals, thus reducing overall cost, (3) a synthetic benchmark built from existing datasets, and (4) several applications.

Critical strengths
  1. The problem motivation is very clear, and using a cheap vocabulary instead of expensive MLLM proposals will be a material improvement at scale.

  2. The paper also presents a redundancy aware selection which shows consistent improvement over alternative methods.

Critical weaknesses
  1. The paper does not provide any quantitative measurements on the token savings as compared to PTR. This would be a useful metric to measure.

  2. The MLLM optimizer sees both images and concepts, so it is unclear to me if it is refinining or using the given concepts at all. The evidence is mixed as COCA and MLLM both converge to similar recall, and showing images that the concepts fails to explain has no effect. On the other hand, SigLIP initialized optimization underperforms PTR.

  3. The verification is not independent. The same family of MLLMs proposes, optimizes and verifies concepts. These MLLMs share pretraining distributions with the embedding models, and more likely than not with COCO, and Imagenet. Biases in embedding therefore can be confirmed by a verified. Swapping judges does not measure this as well. I believe a small human evaluation on a random sample of verified and rejected candidates with reported human MLLM agreement would confirm these results.

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.