- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CONSTRAINTDIAL, an automated framework for continuously constructing smart contract security tasks through protocol mutation. The framework addresses the problem that LLM agents may solve benchmark tasks by memorizing or copying publicly available answers. It changes the function calls or inputs needed to trigger an attack while preserving the underlying vulnerability. It does this by changing the state or input preparation required by the protocol. Using CONSTRAINTDIAL, the authors construct the ReCast benchmark from 203 public cases. The benchmark contains 3,045 task inputs covering vulnerability detection, exploitation, and patching. Compared with reconstruction alone, protocol mutation reduces the success rate when models are given public answers by an average of 10.9 percentage points. It also reduces the proportion of tasks solved by copying online answers by 9.3 percentage points.
Critical strengths
- The paper addresses an important problem in smart contract security evaluation. Public attack scripts and reports can allow models to solve tasks by copying existing answers. CONSTRAINTDIAL changes the calls or inputs needed to carry out an attack while keeping the underlying vulnerability and security objective. This provides a practical way to reduce direct answer transfer.
- The framework automates local contract reconstruction and protocol mutation. It also uses failed execution checks to guide later revisions. The ablation study shows that using test feedback during construction produces more accepted task versions than applying the same checks only at the end.
- The framework uses several checks to validate each mutated task. The original attack must fail, an adapted attack must succeed on the vulnerable protocol, the adapted attack must fail on the repaired protocol, and normal operations must still work. These checks help ensure that the mutation changes the attack conditions without removing the vulnerability or breaking the protocol.
- The paper evaluates six LLM-based agents and compares settings with and without supplied public answers. It also allows network access in both settings. The results provide evidence that protocol mutation reduces the usefulness of copied answers, especially for exploit tasks, while still allowing models to solve tasks through adaptation.
Critical weaknesses
- Increasing the mutation complexity does not consistently reduce success rates across all models. For exploit tasks, the success rate on Combined changes increases for four of the six models. This suggests that the current mutations do not always make the tasks harder and may mainly change the solution path.
- Some successful attacks require only simple preparation calls. The paper reports that 28 of 88 successful attacks under protocol mutation only added preparation calls, such as registration or opening a window. This suggests that some mutations may not test deeper changes in data flow or control flow, which limits how well the benchmark measures advanced vulnerability reconstruction.
- The statistical evidence for the overall Base-to-Combined changes differences is limited. All reported 95% confidence intervals for these comparisons include zero. Therefore, the observed changes do not provide strong evidence of a consistent increase in difficulty at the current sample size.
- The cost analysis reports average construction costs, including rejected proposals and revision attempts, but gives limited information about the cost distribution. For example, it does not show how construction cost changes with mutation type or how much extra cost is needed for versions that require more revisions. It would also be useful to relate construction cost to the number and quality of the accepted tasks.
Questions for authors
- Why does the exploit success rate increase for some models on Combined changes? Could some mutations unintentionally introduce patterns or protocol structures that are easier for LLMs to recognize?
- The paper reports that some successful attacks only add preparation calls. Do the authors plan to include more mutations involving deeper state dependencies, input relationships, or control flow changes?
- Since all 95% confidence intervals for the Base-to-Combined changes success differences include zero, could the authors provide more samples or additional paired analysis to determine whether the observed changes are reliable?
3 comments
Response to Reviewer gLRc (2/2)
W4. How does construction cost relate to revisions and accepted output?
Simple preparation produces the most accepted versions at the lowest cost. The full method produces more accepted versions than direct rewriting at a lower cost per accepted version, but with a higher total cost. Appendix C.5 reports all costs on the same 61 cases and 122 proposals per method, including rejected proposals and revisions.
| Method | Fully accepted versions | Mean revisions | Total cost (USD) | Cost per proposal (USD) | Cost per accepted version (USD) |
|---|---|---|---|---|---|
| Simple preparation | 105 | 0.7 | 47.58 | 0.39 | 0.45 |
| Direct rewriting | 65 | 1.5 | 112.24 | 0.92 | 1.73 |
| Full method | 100 | 1.4 | 150.06 | 1.23 | 1.50 |
| Without execution traces | 73 | 1.9 | 142.74 | 1.17 | 1.96 |
| Without paired checks | 75 | 1.1 | 122.00 | 1.00 | 1.63 |
| Without test feedback | 61 | 0.0 | 95.16 | 0.78 | 1.56 |
Revision burden and rejected proposals both contribute to total cost, while cost per accepted version connects that cost with output. The first table aggregates results by method; the next two break them down further.
For the full method, the following tables add the cost distribution by mutation type and by revision count, including rejected proposals and revision costs.
| Mutation type | Proposals | Mean revisions | Accepted | Total cost (USD) | Cost per accepted version (USD) |
|---|---|---|---|---|---|
| Preparation only | 30 | 0.6 | 27 | 18.90 | 0.70 |
| State dependency | 34 | 1.2 | 29 | 38.76 | 1.34 |
| Input relationship | 32 | 1.5 | 25 | 43.20 | 1.73 |
| Control-flow change | 26 | 2.6 | 19 | 49.20 | 2.59 |
| Revision count | Proposals | Accepted | Total cost (USD) | Cost per accepted version (USD) |
|---|---|---|---|---|
| 0 | 34 | 32 | 17.34 | 0.54 |
| 1 | 42 | 38 | 42.84 | 1.13 |
| 2 | 28 | 22 | 41.16 | 1.87 |
| 3 or more | 18 | 8 | 48.72 | 6.09 |
Thank you again for you thoughtful reviews and time and effor to our community!
Best,
Authors
Dear Reviewer gLRc,
Thank you again for your thoughtful feedback! We hope our response has addressed your concerns. Please feel free to let us know if there are any remaining questions or points that you would like us to clarify. We would be happy to discuss them further.
Best regards,
The Authors
Response to Reviewer gLRc (1/2)
Dear Reviewer gLRc,
Thank you for your careful review. We have uploaded the latest revised PDF. References below use its numbering.
W1 / Q1. Why does exploit success increase for some models?
The mutations change the required attack sequence rather than imposing a fixed difficulty scale. A registration or window-opening requirement can make that sequence more explicit and easier for an agent to follow. Changes to accounts, asset sources, callback senders, or input dependencies can require a different adaptation.
The case transitions show both gains and losses. Qwen3.8-Max gains ten cases and loses three from Base to Combined; DeepSeek-V4.1-Flash gains twelve and loses six. An aggregate increase can therefore coexist with failures on previously solved cases (Section 4.2.3 and Table 9). We report these transitions without assigning a single cause to the increase or claiming that mutation always makes tasks harder.
W2 / Q2. Do mutations require more than preparation calls?
Yes. The construction supports changes to required state, participant and asset relationships, call order, callback senders, and inputs derived from earlier operations (Section 3.2.2). Adding a preparation call is only one possible adaptation.
With a public answer supplied, 60 of 88 successful attacks on mutated protocols change calls or inputs beyond preparation, while 28 succeed with preparation alone. The corresponding simple-preparation result is 9 of 100 (Table 10). These counts describe successful submissions.
The SMOOFSStaking replay shows why preparation alone may fail. Adding registration and withdrawal-window opening reaches NFT authorization but fails there. The NFT deposit reverts because the vault is not approved to transfer the helper's NFT, 45,000 reward tokens remain in the vault, and 0 of 2 extraction checks pass. The agent succeeds only after also authorizing the vault and correcting the callback sender (Appendix C.2).
We also construct deeper variants that add dependencies between state updates, input values, and operation order rather than only adding preparation calls. These changes are controllable but can increase task difficulty and construction cost. The deeper set contains 36 variants, approximately 0.34 times the 105 preparation-only versions. It is evaluated on all 61 families with three runs per condition. Across these runs, mean success is 45.8% for the deeper variants and 54.6% for the preparation-only variants. Direct copying succeeds in 9.0% of deeper-variant sessions, compared with 23.2% on Base.
W3 / Q3. What do the paired analyses show?
All Base-to-Combined intervals for success with usual materials include zero (Table 9), so these results do not establish a consistent change in overall success. The primary single-mutation contrast is Change 1 versus Equivalent control, rather than Change 1 versus Base.
For the public-answer analyses, the same 61 families give:
Each resample retains both models, all three task types, and both versions of a family (Appendix B.5). The full 61-case evaluation includes three runs per task, including the initial run.
The full 61-case evaluation includes three runs per task, including the initial run. We report per-run outcomes for all 61 families. Run 1 gives Base 52.7%, Equivalent control 55.5%, and Combined changes 55.7%. Run 2 gives Base 55.7%, Equivalent control 54.1%, and Combined changes 52.7%. Run 3 gives Base 54.1%, Equivalent control 55.5%, and Combined changes 55.5%. The mean Combined-minus-Base difference is +0.5 percentage points.