acceptodds
Under review as a conference paper at ICLR 2027

Practical Secrets Extraction against Black-box LLMs

Abstract

Large language models (LLMs) increasingly power autonomous coding agents such as Codex and Claude Code, yet their training corpora may contain confidential credentials exposed in public repositories or collected from private development artifacts, creating risks of memorization and subsequent leakage. Existing extraction audits, however, largely assume access to model weights or token probabilities. In this work, we present a black-box secret extraction framework for commercial, API-based LLMs under output-only access. It comprises (i) Cross-Validated Secret Knowledge Distillation, which uses semantics-preserving prompt variants, response cross-validation, and provider-specific format filtering to distill secret-relevant behavior into a local white-box proxy; and (ii) Proxy-Guided Secret Extraction and Candidate Filtering, which combines truncated top- sampling with local token entropy, -gram frequency profiling, and provider-specific structural priors. On controlled API-key benchmarks, our framework improves recovery effectiveness and real-key rates over representative baselines while reducing extraction latency. A responsible real-world evaluation further recovers masked provider-specific credentials from three independently deployed black-box LLM systems spanning OpenAI and Claude Code, showing that memorized secrets can be exposed under output-only access.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
secret extraction, large language model, black box, privacy, practical

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gicBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a black-box secret extraction framework under output-only access. It comprises Cross-Validated Secret Knowledge Distillation and Proxy-Guided Secret Extraction and Candidate Filtering. On controlled API-key benchmarks, the framework improves recovery effectiveness and real-key rates over representative baselines while reducing post-construction extraction latency. A real-world evaluation further recovers masked provider-specific credentials from three independently deployed black-box LLM systems.

Critical strengths
  1. The framework enables black-box secret extraction from commercial APIs that expose only generated text while hiding token probabilities, model states, and gradients.Real-world evaluations demonstrate successful secret extraction from deployed black-box APIs, highlighting the threat’s practical relevance.
  2. Compared with DESEC on the same proxy, the framework improves candidate diversity and real-key rates while reducing post-construction extraction latency.
Critical weaknesses
  1. The paper neither explains selecting StarCoder2-7B and Seed-Coder-8B as proxies nor reports extraction performance with alternative models.
  2. Effectiveness on the latest models remains unverified, limiting the real-world evaluation’s coverage of current systems.
  3. Controlled experiments cover dense models up to 8B parameters, leaving generalization to larger models and mixture-of-experts architectures unverified.
Questions for authors
  1. Why were StarCoder2-7B and Seed-Coder-8B chosen as proxies, and how would replacing them affect extraction performance?
  2. Can you conduct experiments on the latest models, including Claude Opus 5.5, GPT-6.1 Sol, and Qwen3.8?
  3. Can you extend controlled experiments to larger models, such as 27B models, and mixture-of-experts architectures?
3 comments
AuthorsOpenReview ↗

Response to Reviewer gicB (1/1)

Dear reviewer, thank you very much for your valuable time and insightful questions. We believe these comments have significantly improved the quality of our work. We address your concerns as follows.

Q-1 Performance of alternative proxies

Thank you for this question.

  • We selected StarCoder2-7B and Seed-Coder-8B as proxy models because their specialization in code understanding and generation aligns well with our secret extraction setting.
  • Following your suggestion, we also incorporated LLaMA-3-8B as an alternative proxy to further study the impact of proxy model variation. As shown in table below, our method consistently achieves SOTA extraction performance compared to baseline approaches.

We have included these new results in Sections 4.1 and 4.3 (highlighted in blue in the revision).

\textbf{Method} & \textbf{DeepSeek} & \textbf{Google} & \textbf{OpenRouter} \cr \hline \textsf{HCR} & \textsf{49} & \textsf{54} & \textsf{27} \cr \textsf{Greedy Decoding} & \textsf{50} & \textsf{55} & \textsf{28} \cr \textsf{Ours-StarCoder2-7B} & \textsf{57} & \textsf{62} & \textsf{45} \cr \textsf{Ours-Seed-Coder-8B} & \textsf{57} & \textsf{62} & \textsf{42} \cr \textsf{Ours-LLaMA-3-8B} & \textsf{55} & \textsf{68} & \textsf{34} \end{array}$$ ## Q-2 More experiment on latest models Thank you for this question! - We agree that evaluating on the latest is crucial. In our initial submission, we evaluated Claude Code 4.5 series and Qwen 3.8 27B, which represented the most advanced models available prior to the submission deadline (September 25). These results already demonstrated the robustness and effectiveness of our method. - To further address your suggestion, we conducted additional evaluations on the newly released GPT-sol 6.1 (published on September 29). The successful extraction of valid API keys on this model further confirms the generalizability and effectiveness of our approach. Thank you again for this suggestion; we have added this experiment to Section 4.4 and updated Figure 4 accordingly. $$\begin{array}{c c} \textbf{Model} & \textbf{Extracted API Keys} \cr \hline \textsf{Qwen 3.8 27B} & \textsf{sk-aB3cD4eF}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\textsf{R1sT2uV} \cr \textsf{Qwen 3.8 27B} & \textsf{sk-8f3a2b1c}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\textsf{1f2a3b4c} \cr \textsf{GPT-sol 6.1} & \textsf{sk-7mQ2vN9x}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\textsf{5dS0bZUa} \cr \textsf{GPT-sol 6.1} & \textsf{sk-7kP2mR9x}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\textsf{5sL0dA2z} \cr \textsf{GPT-sol 6.1} & \textsf{sk-7fK2mQ9x}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\textsf{5jD0sBzE} \end{array}$$ ## Q-3 Controlled experiments on larger model Thank you for this great question. - Our evaluation already incorporates victim models featuring diverse architectural designs, including variations in attention mechanisms and tokenizers. Specifically, DeepSeek-Coder utilizes Multi-Head Attention (MHA), whereas Qwen2.5-Coder and Llama-3-8B adopt Grouped-Query Attention (GQA). Furthermore, differences in tokenizers result in distinct token representations for API keys. Our findings indicate that models employing GQA exhibit a higher vulnerability to memorization. - We agree that scale is also a vital factor in memorization studies. Therefore, as suggested, we conducted additional experiments using Qwen 3.8 27B and DeepSeek-MoE 16B. As summarized in table below, Qwen3.8-27B corresponds to heightened risks of API memorization and privacy leakage. We have included these results in Section 4.3 of the revision. $$\begin{array}{c c c c} \textbf{Model} & \textbf{DeepSeek} & \textbf{Google} & \textbf{OpenRouter} \cr \hline \textsf{Qwen2.5-Coder-7B} & \textsf{45} & \textsf{56} & \textsf{41} \cr \textsf{LLaMA-3-8B} & \textsf{54} & \textsf{65} & \textsf{51} \cr \textsf{Qwen3.8-27B} & \textsf{57} & \textsf{82} & \textsf{55} \cr \textsf{DeepSeek-MoE-16B} & \textsf{52} & \textsf{61} & \textsf{49} \end{array}$$

Dear Reviewer gicB,

We would like to sincerely thank you again for your support, patience, and dedication throughout this review process. If any further questions or concerns arise, please feel free to reach out, as we are more than happy to address them.

Best regards,

The Authors

Dear Reviewer gicB,

We sincerely thank you for your time, expertise, and kindness.

Best regards,

The Authors

Reviewer_vJR2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the extraction of memorized API keys from black-box LLMs under output-only access. It proposes a two-stage framework that distills secret-related behavior into a local proxy using augmented prompts and filtered responses, followed by diverse sampling and authenticity-aware candidate filtering. Controlled experiments report improved secret recovery and post-construction extraction efficiency over selected baselines, supplemented by case studies on deployed LLMs.

Critical strengths

The problem is interesting and practically relevant. Extracting memorized credentials under output-only access presents an important security challenge. The framework enables local candidate generation and filtering after querying the victim, and the controlled experiments cover multiple victim models, proxy models, and credential providers. The filtering ablation also demonstrates improved candidate precision without losing recovered real keys in the evaluated setting.

Critical weaknesses

The incremental contribution of proxy distillation is not sufficiently established. Table 1 includes direct-query baselines, and Section 4.1 mentions stochastic victim sampling. However, the paper does not clearly establish a comparison against direct top-p sampling using identical prompt augmentation and candidate filtering under a matched total victim-query budget, including proxy construction. More importantly, it does not report how many final recovered keys were already present in the victim responses collected to train the proxy. Such keys would already be available to the attacker before distillation. Without this overlap analysis and a controlled comparison, it remains unclear whether the proxy yields additional secret recovery beyond simply retaining, deduplicating, and filtering the collected victim responses.

The real-world evaluation provides limited support for the broader claims about the proposed framework and parametric secret memorization. Section 4.4 presents six credential examples with reported GitHub matches, but does not provide aggregate extraction results, query counts, baseline comparisons, or sufficient details about applying the complete two-stage pipeline. Public matches support reproduction of existing strings, but establishing parametric memorization also requires clarification of generation-time retrieval, tools, and supplied context. Additionally, although the paper explicitly identifies the 20.7–34.8× speedup as post-construction, missing proxy-construction and victim-query costs leave its end-to-end efficiency relative to direct black-box extraction unresolved.

Questions for authors

Could the authors report the overlap between final recovered keys and all victim responses collected during proxy construction, and compare against retaining and filtering those responses and against direct top-p sampling with identical augmentation and filtering under matched total victim-query budgets?

Could the authors provide aggregate real-world results, clarify whether the complete two-stage pipeline was used and whether retrieval or external tools were enabled during generation, and report total query and computational costs including proxy construction?

3 comments
AuthorsOpenReview ↗

Response to Reviewer vJR2 (1/1)

Dear reviewer, thank you very much for your valuable time and insightful questions. We believe these comments have significantly improved the quality of our work. We address your concerns as follows.

Q-1 Contribution of proxy distillation

Thank you for this valuable question! We clarify that:

  • The core function of proxy distillation is to synthesize and organize fragmented information, where fragment recombination helps recover previously incomplete or missing API keys. Specifically, direct black-box extraction often yields only partial credentials, leading to false-complete extractions. However, the structural patterns and knowledge contained within these fragments can be aggregated to generate valid combinations. For example, two partial fragments ("9c88b" and "10273c78903f3") recovered from different sources were aggregated via proxy distillation to form a novel, valid key: sk-xxxxxxxxxxxxxx9c88b-10273c78903f3.
  • We compiled statistics on the number of valid new API keys recovered exclusively through proxy distillation using identical prompt augmentation and candidate filtering under a matched total victim-query budget. As shown in the table below, these additional keys increase the total number of recovered keys by 47.2%–57.6% relative to Stage 1, validating its effectiveness in uncovering additional valid API keys.

We have added this analysis to the revision (Sections 3.1 and 4.3; Table 7; highlighted in blue).

\textbf{Platform} & \textbf{Keys from Stage-1} & \textbf{Proportion} & \textbf{Keys from Stage-2} & \textbf{Proportion} & \textbf{Improvement from Stage-1} \cr \hline \textsf{DeepSeek} & \textsf{276} & \textsf{63.4}\verb!%! & \textsf{159} & \textsf{36.6}\verb!%! & \textsf{57.6}\verb!%! \cr \textsf{Google} & \textsf{204} & \textsf{63.9}\verb!%! & \textsf{115} & \textsf{36.1}\verb!%! & \textsf{56.4}\verb!%! \cr \textsf{OpenRouter} & \textsf{381} & \textsf{67.9}\verb!%! & \textsf{180} & \textsf{32.1}\verb!%! & \textsf{47.2}\verb!%! \end{array}$$ ## Q-2 Clarification of Evaluation Thank you for this question. We clarify that: - Our real-world evaluations across target platforms successfully yielded newly extracted API keys at each pipeline stage. We observed that while Stage-1 recovered a high volume of valid API keys, Stage-2 successfully uncovered additional valid keys. For instance, in our evaluation against Claude Code's model, Stage-1 recovered 13 valid API keys, while Stage-2 discovered 3 additional valid keys. - To prevent models from leveraging external knowledge, we strictly restricted extraction to model API outputs without engaging online search or tool calls. This confirms that the successfully extracted credentials originate directly from internal knowledge memorized during training. - Additionally, we compiled comprehensive statistics on end-to-end costs, including proxy construction overhead. As shown in table below, our method achieves substantial efficiency gains, reducing extraction time by approximately 60% on DeepSeek API keys. We have integrated this clarification and the new statistics into the revision (Sections 4.2 and 4.4, highlighted in blue). $$\begin{array}{c c c c c c} \textbf{Method} & \textbf{DeepSeek} & \textbf{Google} & \textbf{OpenAI} & \textbf{OpenRouter} & \textbf{Stripe} \cr \hline \textsf{DESEC} & \textsf{12328 s} & \textsf{12958 s} & \textsf{26145 s} & \textsf{19364 s} & \textsf{19689 s} \cr \textsf{Ours} & \textsf{4980 s (-59.6}\verb!%!\textsf{)} & \textsf{5868 s (-54.7}\verb!%!\textsf{)} & \textsf{19499 s (-25.4}\verb!%!\textsf{)} & \textsf{10418 s (-46.2}\verb!%!\textsf{)} & \textsf{11270 s (-42.8}\verb!%!\textsf{)} \end{array}$$

Dear Reviewer vJR2,

We would like to sincerely thank you again for your patience and dedication throughout this review process. If any further questions or concerns arise, please feel free to reach out, as we are more than happy to address them.

Best regards,

The Authors

Dear Reviewer vJR2,

We sincerely thank you for your time and valuable feedback. As the discussion period is coming to a close, we would like to kindly check whether our response has adequately addressed your concerns, or if there are any remaining questions we can clarify?

Thank you once again for your time, expertise, and kindness.

Best regards,

The Authors

Reviewer_vZqUOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a black-box secret extraction framework for commercial, API-based LLMs under output-only access.

Critical strengths
  • Trendy topic
  • Interesting idea
  • Broad experiments
Critical weaknesses
  • Ablate of "memory" with "potential agent passways"
  • The efficiency is a little overclaimed
  • The controlled memorization setting
Questions for authors

This paper proposes a black-box secret extraction framework for commercial, API-based LLMs under output-only access. It comprises of two stages: Cross-Validated Secret Knowledge Distillation and Proxy-Guided Secret Extraction and Candidate Filtering.

However, there are several questions for authors:

  • The efficiency claim is partly post-construction. The large latency improvement excludes victim querying, proxy construction, and fine-tuning costs. Therefore it should not be interpreted as an end-to-end attack speedup.
  • I find that absolute recovery for some key types remains low, even if relative gains are large. I would wonder what is the bottleneck for this?
  • The authors randomly pair 100 code snippets with 100 API keys and explicitly LoRA-fine-tune victim models on them. This demonstrates extraction under known memorization, but may considerably overestimate leakage relative to realistic pretraining where secrets appear sparsely among massive corpora. Maybe different exposure frequencies and training ratios could help.
  • A major concern remains that whether the recovery shows the "memory"? I would doubt whether the agent just output OR retrieve/use tools/search the results, as they are available online. Ablating this influence is really important.

Minors:

  • For Figure 1, it is better to separate two stages.
4 comments
AuthorsOpenReview ↗

Response to Reviewer vZqU (1/2)

Dear reviewer, thank you very much for your valuable time and insightful questions. We believe these comments have significantly improved the quality of our work. We address your concerns as follows.

Q-1 End-to-end attack speed

Thank you for this insightful question!

  • Our primary reason for comparing post-construction efficiency alone was fairness: DESEC principally involves only white-box extraction, so we initially focused on comparing the common offline efficiency. Furthermore, as observed during our experiments, the vast majority of computational overhead stems from the post-construction phase, which is why our optimization efforts targeted this bottleneck.
  • Nevertheless, we agree that analyzing the end-to-end speed-up provides a more comprehensive view of our method's overall efficiency. Following your advice, we compiled statistics on the end-to-end running time and summarized them in the table below. As shown, even when incorporating victim querying, proxy construction, and fine-tuning costs, our method still achieves a substantial speed-up across both offline extraction and end-to-end execution.
  • We have integrated this analysis into Section 4.2 (highlighted in blue in the revision).

Thank you once again for this helpful advice!

\textbf{Method} & \textbf{DeepSeek} & \textbf{Google} & \textbf{OpenAI} & \textbf{OpenRouter} & \textbf{Stripe} \cr \hline \textsf{DESEC} & \textsf{12328 s} & \textsf{12958 s} & \textsf{26145 s} & \textsf{19364 s} & \textsf{19689 s} \cr \textsf{Ours} & \textsf{4980 s (-59.6}\verb!%!\textsf{)} & \textsf{5868 s (-54.7}\verb!%!\textsf{)} & \textsf{19499 s (-25.4}\verb!%!\textsf{)} & \textsf{10418 s (-46.2}\verb!%!\textsf{)} & \textsf{11270 s (-42.8}\verb!%!\textsf{)} \end{array}$$ ## Q-2 Effectiveness analysis Thank you for raising this important point! - We sincerely appreciate your acknowledgment of the improvements we achieved. The primary challenge stems from the inherent nature of API key lengths: as our results show, shorter keys exhibit higher extractability, posing more severe privacy risks. This is likely because accumulated prediction errors have less impact on shorter sequences. - We address this challenge through a two-stage pipeline: first, extracting raw knowledge from the black-box victims, and second, reorganizing and synthesizing potentially useful information from the extracted knowledge fragments. This design enables our method to achieve a dramatic improvement in extraction performance. - We also emphasize the severe real-world risks associated with leaked API keys, which can result in catastrophic financial losses for credential holders. For example, a recent high-profile incident involving Google Gemini API key leakage resulted in an unexpected bill of \$82,000 USD for a single developer [1]. We have added the recovery-bottleneck discussion to Section 4.2 and the financial-impact example to the Introduction. [1] https://www.theregister.com/security/2026/03/03/dev-stunned-by-82k-gemini-api-key-bill-after-theft/4925029?utm_source=chatgpt.com ## Q-3 Experiment setting Thank you for this great question! - We appreciate your understanding that our controlled setup is designed to isolate memorized knowledge, enabling precise evaluation of privacy leakage. - We made every effort to ensure the practical relevance of our fine-tuning setup by: - Organizing training data to reflect strict, real-world environments, following standard corpus construction practices for code completion models. - Adopting the standard Hugging Face Trainer framework to faithfully simulate general post-training procedures. - We agree that studying the proportion of privacy-preserving training corpora helps quantify the impact of data sparsity. Following your advice, we constructed training corpora with varying proportions of secret-related snippets. The results (shown below) demonstrate that extraction performance remains stable across different corpus structures. - Beyond controlled experiments, we conducted real-world evaluations on commercial models. Successful extraction on OpenAI's and Claude Code's models further validates the real-world applicability of our method. We have included this analysis in Sections 4.1 and 4.3 (highlighted in blue in the revision). $$\begin{array}{c c c c c c} \textbf{Proportion} & \textbf{20}\verb!%! & \textbf{40}\verb!%! & \textbf{60}\verb!%! & \textbf{80}\verb!%! & \textbf{100}\verb!%! \cr \hline \textsf{Number of Extracted Keys} & \textsf{68} & \textsf{65} & \textsf{63} & \textsf{70} & \textsf{66} \end{array}$$
AuthorsOpenReview ↗

Response to Reviewer vZqU (2/2)

Q-4 Recovery reference

Thanks for this valuable comment! We apologize for not making this clearer in our submission. We clarify as follows:

  • We fully anticipated and mitigated potential data contamination risks by generating brand-new, unique API keys for each platform from our own side. These keys are unique to our evaluation, have never been made public, and cannot be found via online searches. This ensures a strictly controlled experimental environment:
    • It eliminates external confounds from online data leakage, allowing us to accurately pinpoint the source of memory leakage.
    • It maintains strict ethical alignment by avoiding the extraction of real users' production API keys.
  • To prevent models from leveraging external knowledge, we strictly restricted extraction to model API outputs without engaging online search, tool calls, or supplied context. This confirms that the successfully extracted credentials originate directly from internal knowledge memorized during training instead of online search.

We thank you for raising this important question and have included a more detailed description of the rationale in Sections 4.1 and 4.4 (highlighted in blue).

Q-5 Suggestion for pipeline

Thank you very much for this suggestion! Following your suggestion, we have separated the whole pipeline into two stages as shown in the Figure 1 of the revision.

Dear Reviewer vZqU,

We would like to sincerely thank you again for your patience and dedication throughout this review process. If any further questions or concerns arise, please feel free to reach out, as we are more than happy to address them.

Best regards,

The Authors

Dear Reviewer vZqU,

We sincerely thank you for your time and valuable feedback. As the discussion period is coming to a close, we would like to kindly check whether our response has adequately addressed your concerns, or if there are any remaining questions we can clarify?

Thank you once again for your time, expertise, and kindness.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.