acceptodds
Under review as a conference paper at ICLR 2027

Just Read the Caption: A Black-Box Privacy Attack on Knowledge Inference

Abstract

In this work, we demonstrate that vision classifiers leak private and potentially sensitive information about their semantic domain and class labels even under restricted black-box access. We show that softmax confidence acts as a natural in-distribution discriminator that an attacker can exploit to isolate relevant images from noisy web collections, using their captions to infer the meaning of otherwise opaque predictions. We introduce KEEP, a black-box knowledge inference attack that requires only the top-1 class identifier and its confidence, without prior knowledge of the victim's domain, training data, or class names. KEEP probes the victim with general-purpose image-caption pairs and uses a language model to infer its domain from the highest-confidence evidence. This domain then guides the construction of targeted probing pools from which to recover individual class labels. Across fourteen classifiers spanning four architectures and diverse domains, KEEP recovers exact class names with high accuracy, including copyrighted characters, trademarked brands, and personal identities. These results reveal a largely unexplored source of information leakage, with significant implications for the privacy and confidentiality of models trained on sensitive or proprietary data. Our code is available at https://anonymous.4open.science/r/black-box-knowledge-inference-C195/.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
privacy, AI safety, information leakage, black-box attacks, knowledge inference, image classifiers, label space recovery, model inversion

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RQbWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces an attack, KEEP, that recovers an image classifier’s domain and class names using only its top-1 label and confidence score. KEEP first probes the model with about one million captioned LAION and WIT images, then uses high-confidence captions to infer the domain. It builds a domain-specific probe set and names each class from its 20 highest-confidence captions.

On ImageNet-1K, KEEP reaches 0.49 exact-label accuracy and about 0.9 lexname accuracy. Across 14 fine-tuned classifiers, it outperforms CLIP-Dissect on 10, despite CLIP-Dissect receiving the true vocabulary. Hiding confidence scores largely prevents domain inference, while rounding or adding noise does not.

Critical strengths
  1. It's novel to treat a classifier's label space as the leakage target.
  2. The proposed method requires no shadow models, gradients, or optimization.
  3. The evaluation is broad. The fourteen victims cover four architectures and diverse domains. The ablations over pool size, number of retained probes, probe selection, and namer size show that confidence-based selection drives recovery .
  4. The finding that rejecting low-confidence inputs strengthens the attack rather than weakening it is counterintuitive, but valuable.
Critical weaknesses
  1. All victims are fine-tuned on public benchmarks, and GPT-5.6 explicitly identifies "Food-101-style," "Places365," and "NWPU-RESISC45" as domains. So it is possible that the lexical roster benefits from the LLM's memory of these taxonomies. And LAION and WIT may contain the benchmark images used to train the victims, which would increase confidence on memorized images.
  2. The celebrity victim recognizes famous people whose names appear the most in web captions, while a face recognition model covering private individuals offer no captioned evidence. In the intellectual-property framing, KEEP recovers which concepts a model recognizes, not which data it was trained on, and no training samples are identified or reconstructed.
  3. In the lexical setting, KEEP first asks an LLM for possible class names and then searches for captions containing those names. The paper does not compare against simple alternatives, such as selecting frequent phrases from the top captions or using a vision-language model to describe the top images. The comparison with CLIP-Dissect is also unclear because KEEP uses a targeted image pool while CLIP-Dissect uses the original pool.
Questions for authors
  1. How do you rule out benchmark memorization by the LLM or overlap between the victim’s training data and the LAION/WIT probe pools? Could you test KEEP on a private taxonomy and a deduplicated probe set?
  2. Does KEEP reveal training-data information or only the concepts a model recognizes? How would it perform on private identities or classes with little captioned web data?
  3. Can you compare KEEP with simple caption-frequency and VLM-captioning baselines under the same probing pool?
4 comments

We thank the reviewer for the constructive assessment and for the concrete experiments suggested, all of which we have run. We address each point below. All new experiments use the pipeline of the paper, with the same prompts and settings.

W1 / Q1 (a). The language model recalls the class list of two victims only, and KEEP recovers class lists that no model could have memorized.

We agree that GPT-5.6 recognizes the benchmark behind three victims (Table 5), and we measured how much this contributes. If the roster R were recalled from memory, its entries would be the victim's classes. This is the case for aerial scenes, where all 45 entries are class names, and largely for places (61%). In contrast, only 7% of the 1,461 entries generated for the food victim are Food-101 classes, despite the domain being described as "Food-101-style", and the share is at most 18% for nine of the remaining victims (37% for celebrities and 62% for Pokémon, whose classes are most of the existing species). The roster is therefore an enumeration of the inferred domain, from which the victim's confidence selects the relevant entries. We also repeated the lexical attack on the three victims with the domain and roster produced by a smaller language model (GPT-5.4-mini) that does not recognize any benchmark: recovery is 80% for food, 57% for places, and 33% for aerial scenes, against 98%, 69%, and 36% in the paper. We note that this run also used a smaller retrieval corpus, so it is a lower bound. Finally, the semantic pool does not use the roster at all, and still recovers 89% of the food classes, 90% of the dog breeds, 70% of the logos, and 62% of the Pokémon (Table 6).

Private taxonomy. Training a classifier on a private taxonomy was not feasible within the discussion period, so we constructed victims whose label set is private. We take a public ImageNet-21K classifier (ViT-B/16) and discard all outputs except a random subset of the ImageNet-21K classes of one domain that are not ImageNet-1K classes. To the attacker this is a classifier with opaque identifiers, and its class list is a random draw that appears in no benchmark. We ran the full attack on three such victims, namely the three domains with the densest caption evidence in a pilot over ten candidate domains.

FurnitureMusical instrumentsGarments
Classes10075100
Inferred domainfurniture types and household furnishingsmusical instruments and instrument typesclothing, accessories, and garment types
Roster entries that are class names5% of 1,2073% of 2,8515% of 978
Recovery, unfiltered pool38%21%19%
Recovery, lexical pool53%55%31%

The domain is inferred correctly for the three victims, and KEEP recovers between a third and a half of class lists that the language model cannot have memorized. The remaining classes are mostly rare WordNet names (e.g., "cathedra", "box coat"), for which the predicted name is a broader category such as "chair" or "jacket". We acknowledge that only the label set of these victims is private, as the underlying model was trained on public data.

W2 / Q2 (a). KEEP reveals that a class was part of the training data, and not individual training samples.

We agree that KEEP recovers the concepts a model was trained to recognize, and that it does not identify or reconstruct training samples. Sample-level attacks such as membership inference and model inversion address that question, and they rely on the domain or class knowledge that KEEP recovers (Section 2). A classifier has an output class only because it was trained on examples of it, so the recovered vocabulary is information about the training data at the level of the class. As argued in Section 1, this is the question at stake for a rights holder who can only query the deployed model.

W2 / Q2 (b). Recovery decreases with the amount of captioned web data, and a pool collected by the attacker restores it.

We agree that KEEP depends on the availability of captioned evidence, and we measured how recovery varies with it. For each of the 3,415 classes of the fourteen victims, we counted the captions that name it among the 106.7M captions of our retrieval corpus. Recovery is 40% for the classes named in fewer than ten captions, 75% for those named in ten to 99, and 70% above. For the celebrity victim, it is 7% for the identities named in fewer than 100 captions, 53% between 100 and 999, and 80% above 10,000. We therefore acknowledge that identities with little web presence, and private individuals in particular, are not recovered from web pools, in which case KEEP abstains (Section 3).

However, once the domain is inferred, the pool does not need to come from the web (Table 3). To test this for identities, we replaced the web pool with a gallery of named face photographs (VGGFace2, 20 images for each of its 9,131 identities), in which the only text attached to an image is a name. The gallery is not built from the victim's classes, and contains 223 of its 997 identities. From this gallery, KEEP names 214 of the 223 (96%), against 134 (60%) from the lexical web pool, including 85 of the 89 that the web pool misses. The identities absent from the gallery receive a wrong name, but the two cases are separable from the evidence itself: retaining only the classes for which one name covers at least half of the selected probes gives 212 predictions with a precision of 90%. These identities are public figures and serve as a stand-in; we did not collect data of private individuals. The experiment nevertheless suggests that, for a face-recognition model, the limiting factor is whether the attacker can obtain named photographs of the candidates (e.g., from professional or social-network profiles), rather than the captions available on the web.

W3 / Q3. KEEP outperforms a caption-frequency rule and a vision-language model given the same probes.

We thank the reviewer for suggesting these baselines. In both, every method names a class from exactly the same evidence, namely its k = 20 most confident probes, on the unfiltered pool and on the lexical pool. The caption-frequency baseline returns a phrase from the captions of these probes without a language model; we report, for each victim, the best of three rules (the most frequent phrase, the most distinctive phrase under class-based TF-IDF, and its longest variant). The vision-language baseline (Qwen2.5-VL-7B, of the same family as our namer) is shown the 20 images with the prompt of KEEP and never sees their captions. All rows are scored as in Figure 6.

Recovery (%)FoodDogsFlowersPetsSneakersAircraftBirdsPlacesLogosCarsPokémonCeleb.AerialSkin
Unfiltered pool: KEEP71684778543515434141712273
Caption frequency36182427161321291301110
Vision-language model64615573462310485438215386
Lexical pool: KEEP9893858682797869636869553629
Caption frequency8278757068474651415151423810
Vision-language model9176758468344348505743433116

Caption frequency. KEEP outperforms the frequency rule on all fourteen victims on the unfiltered pool, where the rule recovers at most 36% of the classes, and on thirteen on the lexical pool (aerial scenes being the exception, 38% against 36%). The difference is smaller on the lexical pool because every caption there was retrieved by a roster name, which is consistent with our response to Reviewer LNYR: a large part of the recovery comes from the targeted pool, which is itself a product of the domain inferred by KEEP.

Vision-language model. On the lexical pool KEEP outperforms the vision-language model on all fourteen victims, with the largest differences where the class is a specific named entity (79% against 34% on aircraft, 78% against 43% on birds, and 69% against 43% on Pokémon). On the unfiltered pool KEEP is ahead on eight victims and the vision-language model on six (flowers, places, logos, celebrities, aerial scenes, and skin lesions), i.e., where the web captions of the probes do not name the class but the images depict it. We see this as complementary to KEEP, in line with the extension discussed in Section 5, and note that on every victim except aerial scenes the vision-language model on the unfiltered pool remains below KEEP on the lexical pool. As a further control, we gave our namer the captions written by the vision-language model for the five most confident probes of each class instead of their web captions: on the twelve victims we could run, the web captions give higher recovery on ten and the same on two.

CLIP-Dissect. We agree that Figure 6 does not separate the pool from the naming procedure. The comparison on the same lexical pool is reported in our response to Reviewer LNYR (Q3 (c)): with the same evidence, KEEP outperforms CLIP-Dissect with the ground-truth vocabulary on seven of the fourteen victims and matches it on one, and outperforms it on eleven when CLIP-Dissect is given the roster R, the only vocabulary available to the attacker.

Thanks for your response, which has addressed my concerns. However, considering the overall novelty and quality of the manuscript, I maintain my current score.

Reviewer_LNYROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates knowledge inference attacks against black-box image classifiers, aiming to recover their semantic domains and class names without prior knowledge of the training data or label space. The authors propose KEEP, a two-stage framework that first probes an unknown classifier using general-purpose image–caption pairs and selects informative evidence based on prediction confidence. An LLM then infers the classifier's domain and guides targeted probing through semantic or lexical retrieval. Finally, individual class names are recovered from the captions associated with high-confidence predictions. Experiments on 14 classifiers across different architectures and domains demonstrate that KEEP can recover a substantial proportion of class names using only top-1 class identifiers and confidence scores. The authors argue that these results reveal an underexplored privacy risk associated with the semantic knowledge encoded in trained models.

Critical strengths
  1. The paper investigates semantic information leakage from black-box classifiers, extending the discussion beyond conventional sample-level privacy attacks. Recovering an unknown classifier's class vocabulary is an interesting problem with potential implications for model auditing and information security.

  2. KEEP presents an intuitive two-stage approach combining confidence-based evidence selection with domain-guided retrieval. The experiments cover 14 classifiers, and the ablation studies provide useful evidence supporting the effectiveness of confidence-based probing.

Critical weaknesses
  1. KEEP primarily succeeds on publicly documented categories with abundant external image–caption data. While this demonstrates class-name recovery, the paper does not convincingly establish the resulting privacy harm or evaluate genuinely confidential settings, such as non-public identities or unreleased proprietary products. More importantly, the method relies heavily on external semantic knowledge, which may be unavailable precisely in these privacy-sensitive scenarios. This creates a fundamental gap between the demonstrated capability and the paper's central privacy claims.

  2. Related work has already explored black-box domain inference (Gu and Chen, AAAI 2023; Gu et al., KDD 2025) and class-level information leakage (Struppek et al., 2023). Although KEEP extends these directions toward class-name recovery, its methodology mainly combines existing confidence-based selection, LLM reasoning, and retrieval techniques. Furthermore, the high and domain-dependent query costs and insufficiently controlled baseline comparisons weaken the evidence for its practical advantage.

Questions for authors

1.What genuinely sensitive information is disclosed by recovering publicly documented categories? Can the authors clarify how KEEP would recover non-public identities or proprietary product names when their corresponding images and textual descriptions are unavailable externally?

  1. Does KEEP require the general-purpose probing pool to contain semantically relevant samples from the victim's domain? What information can still be recovered if this assumption fails, and how do the existing results characterize this limitation?

  2. What is the essential distinction between KEEP and existing domain- or class-level inference attacks? Could the authors also clarify the practical end-to-end attack cost and how much of KEEP's improvement over CLIP-Dissect comes from targeted retrieval rather than the naming procedure?

3 comments

We thank the reviewer for the careful reading and the constructive questions. We address each point below, and report an additional controlled comparison against CLIP-Dissect in response to Q3.

W1 / Q1. The sensitive information is the presence of a category in a model's training data, which remains undisclosed even when the category itself is public.

We agree that the categories KEEP recovers are publicly documented, and we emphasize that this is the setting our work was designed for. Our main motivation is the increasing number of intellectual-property disputes over the alleged use of copyrighted material to train models (Section 1). In these cases the protected material is, by its nature, abundantly available online, for example as film and television stills, character artwork, book excerpts, or product photographs, whereas the fact that a particular model was trained on it is not disclosed by its developer. What KEEP reveals is therefore not the existence of a character, brand, or person, but that the victim was trained to recognize it, which is the question at stake for a rights holder who can only query the deployed model. For the same reason, the external image–text evidence that KEEP relies on is readily available precisely in these scenarios, as reflected by the recovery of 69% of Pokémon characters, 70% of brand logos, and 82% of sneaker models (Section 4.2).

We acknowledge that KEEP cannot name a class for which no external image–text evidence exists, such as a non-public identity or an unreleased product. In this case KEEP is designed to abstain rather than to predict from insufficient evidence (Section 3). We note, however, that the probing pool does not need to come from the web: once the domain is inferred, the attacker can replace it with data of that domain, as we show for skin lesions and aerial scenes (Table 3), which may include data held by the interested party itself.

Q2. KEEP requires the pool to contain the victim's broad domain, but not its individual classes, and coarser semantic information is still recovered when the classes are absent.

The hierarchical victims of Section 4.1 characterize this limitation. The most direct case is the ImageNet-20K pool of Table 1, which by construction contains none of the victim's classes: the predictions still fall in the correct WordNet category for 87% of the classes (43% at chance), with an IC-recovery of 0.570.57 (0.180.18 at chance). Figure 3 shows the same behaviour on finer domains, where accuracy increases from 0.320.32 at the species level to 0.530.53 at the family level on iNaturalist, and from 0.730.73 for the variant to 0.880.88 for the family on FGVC-Aircraft. Therefore, even when an individual class cannot be named exactly, for instance because it is absent from the pool, KEEP still recovers related entities in the hierarchy and the general domain of the victim. The latter is what enables targeted recovery.

W2 / Q3 (a). Existing domain- and class-level attacks select among candidates held by the attacker, whereas KEEP assigns a free-text name to every output class.

The essential distinction is that existing domain- and class-level inference attacks (Gu & Chen, 2023; Gu et al., 2025; Struppek et al., 2023) select among a set of candidates that the attacker defines in advance, such as candidate datasets, concepts, or attribute values, and none of them recovers the class vocabulary of the victim. In contrast, KEEP assigns a free-text name to every individual output class without a predefined candidate set, while requiring only the top-1 identifier and its confidence rather than the full output vector. We agree that the components of KEEP are simple. Our contribution is to show that softmax confidence alone suffices to isolate the relevant evidence from general web data (Figures 2 and 4), and we believe that the simplicity of the attack makes this leakage more concerning.

Q3 (b). The full attack needs 15k to 90k queries for most victims.

The total number of queries is dominated by the first stage, since the targeted pools are comparatively small (Table 2). In Appendix A.2 we therefore repeat the full attack with a first stage of only 10k general probes (Table 8): recovery stays within four points of the 1M first stage for nine of the twelve victims whose domain is inferred, with 15k to 90k queries in total, targeted pool included. For example, 67% of the Pokémon classes are recovered with 20k queries, against 7% when probing blindly with 1M images. The cost does depend on the domain, as rarer domains need more general probes before the domain is inferred (Table 7).

Q3 (c). Most of the improvement over CLIP-Dissect in Figure 6 comes from the targeted pool, and KEEP's naming is more accurate once the baseline is restricted to the attacker's knowledge.

We thank the reviewer for this question. In Figure 6, CLIP-Dissect is evaluated on the unfiltered pool only, while KEEP is also evaluated on the targeted pools, so the comparison does not separate the contribution of targeted retrieval from that of the naming procedure. To isolate the two, we ran an additional experiment in which both methods name each class from exactly the same evidence, namely the k=20k = 20 most confident probes of the class in the lexical pool. The methods then differ only in how the class is named: KEEP gives the captions of these probes to a language model, whereas CLIP-Dissect matches their images against a fixed vocabulary with CLIP. We consider two vocabularies for CLIP-Dissect: the ground-truth class names of the victim (oracle), as in the paper, and the roster R\mathcal{R} that KEEP generates from the general evidence (Section 3), which is the only vocabulary that an attacker without access to the class names could provide.

Recovery (%), lexical poolKEEPCLIP-Dissect (roster)CLIP-Dissect (oracle)
Food9895100
Dog breeds937975
Flowers857274
Pets8676100
Sneakers825282
Aircraft793046
Birds784468
Places697785
Brand logos635470
Car models686063
Pokémon694848
Celebrities555766
Aerial scenes364949
Skin lesions291316

These results lead to two conclusions. First, most of the improvement over CLIP-Dissect in Figure 6 comes from targeted retrieval. Once it is given the same lexical pool, oracle CLIP-Dissect improves on most victims with respect to the unfiltered pool (e.g., from 6% to 48% on Pokémon and from 19% to 66% on celebrities), and KEEP outperforms it on seven of the fourteen victims and matches it on one, rather than on ten. The targeted pool is, however, itself a product of KEEP, since it is built from the domain inferred in the first probing round, and it benefits either naming procedure. Second, the naming procedure of KEEP is more accurate once both methods are restricted to the knowledge of the attacker. The oracle variant is given the class names that the attack aims to recover, and is therefore an upper reference rather than a feasible attack. When CLIP-Dissect is instead given the roster, its recovery decreases on eleven victims, by up to 30 points, and KEEP outperforms it on eleven of the fourteen, with the largest differences on fine-grained domains (79% against 30% on aircraft, 78% against 44% on birds, and 82% against 52% on sneakers). Of the remaining three, the difference on celebrities is two points, and for places and aerial scenes the roster is close to the oracle vocabulary, as their training dataset is recognized during domain inference (Table 5).

Reviewer_LNYROpenReview ↗

Responses to Authors

Thank you for the clarifications and additional experiments. The controlled comparisons address my concerns about the baseline evaluation. However, the practical scope of the privacy claims remains insufficiently established, particularly when external evidence is limited. The reliability of abstention and the risk of incorrect naming in such cases also remain unclear.

I therefore maintain my weak reject recommendation.

Reviewer_kK6QOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a new privacy attack called KEEP (Knowledge Extraction via External Probing). The goal of the attack is to infer the class labels of a given image classifier model. In contrast to existing membership and model inversion attacks, KEEP does not require knowledge or possession of training images.

Critical strengths
  • The paper is well written and can be easily followed
  • The experimental evaluation is rather extensive. Even though, it is not clear how the OOD data was selected (see weaknesses)
Critical weaknesses
  • I am not sure how this attack leaks any private information. As far as I understand the class labels are already given.
  • It was shown by previous work [1] that models (in this case ReLU networks in particular, but this also holds for other architectures) yield overconfident predictions when queried with OOD data. So while experiments were conducted on OOD data, it is not clear to me how OOD was defined in this case. Just choosing a dataset from a different domain seems insufficient to me.

[1] Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problem, Hein et al., CVPR 2019

Questions for authors

Q1: I am not sure what the underlying scenario for this paper is in which private information is leaked only based on the class label. Could you provide a scenario in which this is a serious problem? Why would an attacker have access to a model without having access to the labels?
Q2: How was OOD defined? How were the datasets chosen? Q3: Am I correct in my assumption that KEEP only works when images of the same domain as the true class labels are within the probing pool?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.