- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents CryptHarness, an agent-based cryptanalytic research framework where a human expert can interact with an LLM agent (e.g., ChatGPT or DeepSeek), supporting mathematical formalization and modeling of the cryptanalysis problem, proof-of-concept attack implementation, and machine-assisted proofs (based on lean) of mathematical claims for establishing the attack. To show the effectiveness of CryptHarness, the authors analyzed two NIST-standard digital signatures schemes, ML-DSA and Falcon. For the former, CryptHarness discovered a new result on decompressing the public key using only almost one third of the signatures required from the previous SOTA. For the latter, CryptHarness devises a new secret key recovery attack under the randomness leakage uisong only a half of the signatures required from the previous SOTA.
Critical strengths
-
The proposed CryptHarness provides a clean framework where human expert can interact with an Agent in an end-to-end cryptanalysis research with machine-assisted verification, though the authors provide none of prompt inputs and produced artifacts (e.g., lean verification script or attack code).
-
The paper presents new cryptanalysis results of NIST-standard digital signature schemes (ML-DSA, Falcon) by reducing the number of valid sigatures collected by the adversary. One caveat is that the paper did not claim CryptHarness fully break these scheme; instead, it accelerates finding some necessary parts for side-channel attacks, e.g., public key decompression in ML-DSA, or leverage the attack results for secret key recovery (in the case of Falcon).
Critical weaknesses
- I'm not fully convinced whether the paper's core results are derived solely from CryptHarness. It is already known that ChatGPT 5.6 Sol--one of the backend LLMs used by the authors--has enough capability to solve research-grade cryptographic problems with human expert's feedback. Right after the release of ChatGPT 5.6 Sol, many cryptographers solved several open problems in theoretical cryptography with its assistance [Ragavan26, Hhan26, CLX26]. I have a doubt that the claimed results could be obtained without CryptHarness.
References
[Ragavan26] Ragavan, Seyoon. "Efficient Unclonable Encryption from Pauli Eigenstates." Cryptology ePrint Archive (2026).
[Hhan26] Hhan, Minki. "Finding a Shortest Vector and More in $2^{n/2+ o (n)} $ Time using $ q $-ary Coset Difference Tree." arXiv preprint arXiv:2609.02764 (2026).
[CLX26] Coladangelo, Andrea, Liu, Qipeng, and Xie, Ziyi. "Unclonable encryption from BB84 states: a simultaneous Goldreich-Levin reduction." arXiv preprint arXiv:2608.17629 (2026).
Questions for authors
-
What happens if you directly provide the input specified in Section 3.2 and case studies to ChatGPT 5.6 Sol? As you mentioned in Section 6, its safety guardrail would not be triggered if you provide a mathematically equivalent form of the cryptanalytic question.
-
If ChatGPT 5.6 Sol alone can solve the same problem without CryptHarness as I concerned, it seems that the key advantage of CryptHarness then becomes reducing the human expert's labor in a "verifiable" way. Could you comment on assessing such a kind of advantage, or at least could you provide the exact input/output of CryptHarness in case studies? I could not assess how CryptHarness is useful for reducing the human expert's burden from the current form of the manuscript.
-
From the case study about ML-DSA, I feel that the lower bound claim of the required number of signatures for recovering should be more carefully stated. Are the underling hypotheses (H1--H3) stated in Appendix C well-justified? I couldn't decide whether these assumptions are reasonble or not. In addition, does such a bound apply to other possible types of attacks? From my reading, the claimed bound is derived from Oliveira et al.'s formulation based on a system of linear inequalities, and perhaps a better attack may break this bound in the future (I'm not fully certain about this though). Current claims in the main text seem easy to misinterpret, when answers for both questions are negative, though the author acknowledged the adoption of such hypotheses as limitation.
-
The case studies give me an impression that CryptHarness merely optimizes the SOTA attack framework via human expert's initial hints, instead of developing a completely new method. I don't think this is the inherit limitation of CryptHarness because finding such a solution would be the shortest path in current experimental design. What happens if you does not provide additional contexts beyond the problem description, such as previous SOTA papers and human's hint? Can CryptHarness can discover novel attack framewoks in such a setting?
2 comments
Thank you for your detailed response.
The supplementary material was especially useful as it shows how simple the initial prompts can be, and the author's revision looks appropriate for me to resolve my concerns.
All of my concerns are resolved, but it still seems that the expert's guidance played a dominant role in obtaining new results (as shown in the author's response), and it would be nice if the authors could clearly present the human experts' interaction/guidance during the whole process of the cryptanalysis.
Nevertheless, I believe human intervention can be reduced with more advances in AI agents in the future, and I think this work has merit for demonstrating the feasibility of cryptanalysis research with AI agents.
I mildly support the acceptance of this work, raising my score to weak acceptance.
We thank the reviewer for the thoughtful questions and for the positive assessment of the framework and cryptanalytic results. We agree that the original evaluation did not sufficiently isolate the contribution of CryptHarness from that of the underlying model. To address this we conducted additional experiments for the revised manuscript.
Comparison with ChatGPT 5.6 Sol in Codex. We agree that frontier models can contribute to research-level cryptography and have added the three works highlighted by the reviewer to our discussion of concurrent work. We added a matched generic Codex baseline using ChatGPT 5.6 Sol with high reasoning and the same problem materials, literature, expert direction, and tool access (cf. Section 6). Generic Codex recovered 3/3 ML-DSA-44 vectors at 100,000 signatures, compared with 10/10 at a mean of 60,112 for CryptHarness. For Falcon-512, generic Codex recovered 4/5 complete keys at 100 signatures, compared with 12/12 at 50 signatures for CryptHarness. Thus, the recovery techniques developed by generic Codex still succeeded in improving on the state-of-the-art but required significantly more signatures for recovery than CryptHarness.
Removing human hints. CryptHarness also succeeded without the expert attack direction (Section 6): 12/12 ML-DSA-44 vectors at a mean of 95,658 signatures and 19/20 Falcon-512 keys at a mean of 81.1 signatures among successes. In these runs, CryptHarness found methods different from those used in the original case studies: staged LP localization followed by active-set min-conflicts for ML-DSA, and scale-ray enumeration with bounded public-key-guided repair for Falcon. Their weaker performance indicates that expert guidance remained useful in our case studies.
Case-study inputs and outputs. We added the problem statements given to CryptHarness to Appendix F and provide the resulting attack code and standalone proofs of concept, in the updated supplementary material. For ML-DSA, success means recovering the entire omitted vector t0. For Falcon, it means returning a complete (f,g,F,G) key that satisfies the public algebraic relations and produces a fresh valid signature.
Theoretical scope. We have clarified wording that could have suggested a stronger result and now describe it throughout as a conditional first-moment threshold. H1 reflects ML-DSA's fixed-weight challenge sampler, H2 uses the standard uniform-residual approximation, and H3 models independently generated signatures. The analysis was reviewed by cryptanalysts, and the predicted thresholds are within approximately 5--11% of the empirical mean recovery counts across all three parameter sets. This agreement gives us confidence that the approximation captures the main behavior observed in these experiments. We have revised the paper to state its conditional scope explicitly and avoid implying a universal lower bound.
We appreciate the reviewer's feedback, which led to a stronger evaluation and a more precise statement of the theoretical result.