- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents a black-box secret extraction framework under output-only access. It comprises Cross-Validated Secret Knowledge Distillation and Proxy-Guided Secret Extraction and Candidate Filtering. On controlled API-key benchmarks, the framework improves recovery effectiveness and real-key rates over representative baselines while reducing post-construction extraction latency. A real-world evaluation further recovers masked provider-specific credentials from three independently deployed black-box LLM systems.
Critical strengths
- The framework enables black-box secret extraction from commercial APIs that expose only generated text while hiding token probabilities, model states, and gradients.Real-world evaluations demonstrate successful secret extraction from deployed black-box APIs, highlighting the threat’s practical relevance.
- Compared with DESEC on the same proxy, the framework improves candidate diversity and real-key rates while reducing post-construction extraction latency.
Critical weaknesses
- The paper neither explains selecting StarCoder2-7B and Seed-Coder-8B as proxies nor reports extraction performance with alternative models.
- Effectiveness on the latest models remains unverified, limiting the real-world evaluation’s coverage of current systems.
- Controlled experiments cover dense models up to 8B parameters, leaving generalization to larger models and mixture-of-experts architectures unverified.
Questions for authors
- Why were StarCoder2-7B and Seed-Coder-8B chosen as proxies, and how would replacing them affect extraction performance?
- Can you conduct experiments on the latest models, including Claude Opus 5.5, GPT-6.1 Sol, and Qwen3.8?
- Can you extend controlled experiments to larger models, such as 27B models, and mixture-of-experts architectures?
3 comments
Dear Reviewer gicB,
We would like to sincerely thank you again for your support, patience, and dedication throughout this review process. If any further questions or concerns arise, please feel free to reach out, as we are more than happy to address them.
Best regards,
The Authors
Dear Reviewer gicB,
We sincerely thank you for your time, expertise, and kindness.
Best regards,
The Authors
Response to Reviewer gicB (1/1)
Dear reviewer, thank you very much for your valuable time and insightful questions. We believe these comments have significantly improved the quality of our work. We address your concerns as follows.
Q-1 Performance of alternative proxies
Thank you for this question.
We have included these new results in Sections 4.1 and 4.3 (highlighted in blue in the revision).
\textbf{Method} & \textbf{DeepSeek} & \textbf{Google} & \textbf{OpenRouter} \cr \hline \textsf{HCR} & \textsf{49} & \textsf{54} & \textsf{27} \cr \textsf{Greedy Decoding} & \textsf{50} & \textsf{55} & \textsf{28} \cr \textsf{Ours-StarCoder2-7B} & \textsf{57} & \textsf{62} & \textsf{45} \cr \textsf{Ours-Seed-Coder-8B} & \textsf{57} & \textsf{62} & \textsf{42} \cr \textsf{Ours-LLaMA-3-8B} & \textsf{55} & \textsf{68} & \textsf{34} \end{array}$$ ## Q-2 More experiment on latest models Thank you for this question! - We agree that evaluating on the latest is crucial. In our initial submission, we evaluated Claude Code 4.5 series and Qwen 3.8 27B, which represented the most advanced models available prior to the submission deadline (September 25). These results already demonstrated the robustness and effectiveness of our method. - To further address your suggestion, we conducted additional evaluations on the newly released GPT-sol 6.1 (published on September 29). The successful extraction of valid API keys on this model further confirms the generalizability and effectiveness of our approach. Thank you again for this suggestion; we have added this experiment to Section 4.4 and updated Figure 4 accordingly. $$\begin{array}{c c} \textbf{Model} & \textbf{Extracted API Keys} \cr \hline \textsf{Qwen 3.8 27B} & \textsf{sk-aB3cD4eF}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\textsf{R1sT2uV} \cr \textsf{Qwen 3.8 27B} & \textsf{sk-8f3a2b1c}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\textsf{1f2a3b4c} \cr \textsf{GPT-sol 6.1} & \textsf{sk-7mQ2vN9x}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\textsf{5dS0bZUa} \cr \textsf{GPT-sol 6.1} & \textsf{sk-7kP2mR9x}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\textsf{5sL0dA2z} \cr \textsf{GPT-sol 6.1} & \textsf{sk-7fK2mQ9x}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\unicode{42}\textsf{5jD0sBzE} \end{array}$$ ## Q-3 Controlled experiments on larger model Thank you for this great question. - Our evaluation already incorporates victim models featuring diverse architectural designs, including variations in attention mechanisms and tokenizers. Specifically, DeepSeek-Coder utilizes Multi-Head Attention (MHA), whereas Qwen2.5-Coder and Llama-3-8B adopt Grouped-Query Attention (GQA). Furthermore, differences in tokenizers result in distinct token representations for API keys. Our findings indicate that models employing GQA exhibit a higher vulnerability to memorization. - We agree that scale is also a vital factor in memorization studies. Therefore, as suggested, we conducted additional experiments using Qwen 3.8 27B and DeepSeek-MoE 16B. As summarized in table below, Qwen3.8-27B corresponds to heightened risks of API memorization and privacy leakage. We have included these results in Section 4.3 of the revision. $$\begin{array}{c c c c} \textbf{Model} & \textbf{DeepSeek} & \textbf{Google} & \textbf{OpenRouter} \cr \hline \textsf{Qwen2.5-Coder-7B} & \textsf{45} & \textsf{56} & \textsf{41} \cr \textsf{LLaMA-3-8B} & \textsf{54} & \textsf{65} & \textsf{51} \cr \textsf{Qwen3.8-27B} & \textsf{57} & \textsf{82} & \textsf{55} \cr \textsf{DeepSeek-MoE-16B} & \textsf{52} & \textsf{61} & \textsf{49} \end{array}$$