- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies dataset inference for CLIP-style dual encoders when the set-level score is the batch-mean InfoNCE loss. The authors argue that the usual scaling of separability breaks down in this setting, because a larger batch also gives each pair more negatives. On a 400M-pair CLIP model, the effect size of the in-batch loss grows as , while scoring the same pairs against a fixed bank of negatives gives . They explain this with a design-effect decomposition and attribute the extra growth to pairs whose positive similarity barely exceeds that of their hardest negative. Building on this analysis, they propose per-pair detectors that need only embeddings, study the sample size and failure modes of set-level audits, and test several predictions on additional checkpoints, including one pre-registered test.
Critical strengths
- The paper targets a relevant problem. Data-provenance audits for multimodal foundation models are of growing practical interest, and the observation that in-batch contrastive scores depend on what else is in the batch is a valid caution for those who wants to use a contrastive loss as a membership score.
- The experiments of this work are quite thorough. The paper uses exchangeable member/non-member splits, includes a blind-model control, and replicates the main measurements on several checkpoints. This kind of falsifiable setup is rare in this area, and the paper reports the predictions that failed as well as those that passed.
Critical weaknesses
-
The paper's main negative claim—that the square-root law fails—seems to target a position that previous work does not actually hold. In the introduction, the authors state that "the standard justification is a central-limit argument" and derive , yet this sentence carries no citation. The dataset-inference studies cited in the same paragraph (Maini et al., 2021; 2024; Dziedzic et al., 2022; Puerto et al., 2025) aggregate per-example scores with a t-test and report required sample sizes empirically. However, none of them derives a scaling or refers to a square-root law. The "approximately square root function" in Puerto et al. relates paragraph-level AUROC to collection-level AUROC, which is a different quantity from the number of aggregated examples. Nor is the reading that the law, "taken at face value", implies that "any score, however weak, suffices given enough examples" attributed to any prior work. More importantly, the related-work section also states that existing methods "all aggregate a per-example statistic that does not depend on what else is aggregated, and so inherit the square-root law". The reported failure therefore concerns only the batch-mean InfoNCE loss, and the paper cites no audit that uses this statistic. The detectors recommended in Sec. 5 are again per-example statistics that "obey the square-root law by construction", which is the kind of statistic prior dataset-inference work already uses. I therefore find it difficult to identify an existing practice that the main finding corrects, and the paper’s practical contribution appears limited.
-
I am also not convinced that the best-performing detectors have much to do with the near-tie mechanism. The lower-tail mean is essentially the cosine-similarity attack of Ko et al. (2023), which is the mean-alignment baseline in Sec. 5, combined with a Min-K%-style aggregation over the lower tail (Shi et al., 2024). The related work section already describes the two as parallel. The expected crossing score is a monotone transform of where falls in the reference distribution, so in practice it also puts more weight on the lower tail. The paper's own experiments point the same way. The permutation test in Table 29 concludes that the advantage of the crossing score "is a property of where the shift sits in the distribution of , not of any pair's own rivals", and Sec. 5 states that "which quantiles of a score reads matters more than its form". In other words, what helps is reading the right quantile, and the near-tie analysis plays little role in choosing it. On checkpoint B, where the shift is uniform, the advantage disappears and no score does significantly better than mean alignment. Overall, the main contribution seems to be an explanation of why a suboptimal statistic is suboptimal, even though no cited audit uses that statistic (see point 1). The paper also proposes a tail detector that builds on existing ideas and only helps if the auditor already knows the shape of the shift.
-
More broadly, I do not see how the near-tie analysis translates into practical guidance for an auditor, and several claims in the main text are stronger than the evidence in the appendices. The 1.53 ratio that the near-tie model "reproduces" in Sec. 4 is obtained by feeding in checkpoint A's own measured shift. When A's shape is used to predict a new model, the near-tie gain misses (see point 2), and the authors attribute this to a "residual shape difference". In practice, this means the shift has to be measured on every new model before the mechanism can say which detector wins. The claim in Sec. 3.3 that "the largest available batch is not the best one" () also rests only on the per-pair estimator. In Table 18, the direct estimator gives local slopes of 0.56, 0.56 and 0.57 between 4k and 32k, which never drop below 1/2, although its intervals are too wide to rule the claim out. Finally, the paper acknowledges that the best operating point "sits where the shift is, which an auditor does not know in advance". On checkpoint B, a bank sized for keeps only 0.59 of the effect size of mean alignment, against 0.91 at its own optimum. Appendix N also reports that selecting the bank by near-tie density "is never better than random selection". The final recommendation, reporting the test at two operating points with a Bonferroni correction, is sensible. However, it does not visibly rely on the near-tie analysis.
-
I also struggled with the writing, and I suspect many readers will. To be honest, I directly work in this area and am quite familiar with MIA, but this paper is still extremely difficult to read for me. The ideas are not especially hard, but getting to them took me far longer than it should have. The sentences are also heavily compressed. A single sentence often strings three or four claims together with colons and semicolons, attaches a number and an interval to each of them, and then moves on with almost no transition.
-
The paper coins too many non-standard new terms. The standard term "negatives" becomes "rivals", and the reader then meets "near-ties", "the near-tie gain", "the negative swap", "the symmetrization gain", "neutral banks", "the confound floor", "exposure-balanced controls", "admissible targets", "signatures" and a "strong pass". The smaller checkpoints are called "rungs", a word that appears 35 times and first shows up as a row label in Table 1 without explanation, presumably because they sit on "the ladder" of Appendix P. Several of these terms are never properly introduced in the main text. The "near-tie gain", for example, makes its first appearance in Sec. 6 only to report that it "misses (1.32 against 1.60)". The "signatures" appear in the contributions list long before the reader can work out what they are, and the "symmetrization gain" shows up only as a row label in an appendix table. Reading the paper often feels like joining a long-running project halfway through, where everyone except the reader already knows what a "symmetrization gain" is.
-
On top of this, the same concept often goes by several names. Separability, effect size, and "signal" refer to roughly the same quantity. The same goes for "negatives", "rivals", "the fixed bank" and "the reference set", for "temperature" and "logit scale", for "cosine similarity" and "alignment", and for "collision rate" and "crossing rate". The recommended detector is called the "expected crossing score" in the abstract and a "rival-free lower-tail statistic" in the contributions. The main text then splits it into a "bank crossing score" and a "lower-tail mean". To be honest, matching these names up felt more like detective work than reading. I had to piece together clues from figure captions, table headers and appendices, and sometimes simply guess, before I could be reasonably sure which terms referred to the same thing. I am still not confident that I solved every case. For instance, I never worked out why the "rival-free" contains the number of rivals in its formula.
-
The image–text pairs in this paper also seem to have very colorful lives. They have a "true partner" and a "strongest rival", some "win comfortably", others are "overtaken", and in the end they face an audit whose "verdict" can be "manufactured". Meanwhile, the square-root law makes a "promise", the signal "lives in" the similarity, and the loss "outgrows" the law. Each metaphor is easy enough on its own. Together, however, they mean that almost every sentence has to be translated back into technical terms before it can be followed.
-
Finally, the information density is too high. The paper appears to compress a 42-page manuscript into nine pages of main text with little selection, as if every result had to be reported there. The abstract alone contains more than a dozen numbers, and a single paragraph in Sec. 3.2 reports seven confidence intervals. The pair of slopes 0.50 and 0.64 appears four times before Sec. 2, and the slogan "batch size is not sample size" appears in the title, the abstract, the introduction and the heading of Sec. 3. Even section headings are written as conclusions, such as "A marginal shift, not relational structure". The whole paper is also flooded with numbers, equations and confidence intervals. To be honest, I do not know why this paper needs an interval on nearly every number it reports. Some of the CIs, in my opinion, can completely be OK to be placed into the Appendix. With this much quantitative detail on every page, I found it very hard to grasp the core idea, or to tell which of the many numbers actually carry the paper's message.
-
Overall, the reader has to get through a layer of notation, a layer of new names and a layer of metaphors before reaching the actual claims. I would suggest that the authors keep one name per concept (using the standard term "negatives"), add a notation table, and move most of the interval estimates in the main text to the appendix.
Questions for authors
- Is there any published dataset inference or audit method that uses the batch-mean InfoNCE loss as its set statistic? If not, which existing practice does the reported failure of the square-root law correct?
- Since holds by construction for fixed-bank scores, which result in Sec. 3 could have contradicted the near-tie mechanism? Relatedly, given that the near-tie gain missed in the pre-registered test (1.60 against 1.32), can the model predict which detector wins on a new checkpoint without first measuring that checkpoint's shift?
- Beyond guiding the choice of the quantile , what does the near-tie analysis add over the similarity score of Ko et al. (2023) combined with Min-K%-style tail aggregation? When the shape of the shift is unknown in advance, how should an auditor choose between these statistics and their operating points?