acceptodds
Under review as a conference paper at ICLR 2027

Beyond Copying: Lifelong Smart Contract Security Task Construction via Protocol Mutation

Abstract

AI agents can inspect code and run tests to support smart contract security. SlowMist reports about $38.6 billion in losses across the broader blockchain ecosystem. Existing evaluations can reward memorized or copied answers, including attack code recovered from public blockchain transactions. Reconstructing contracts and validating tasks also require costly manual effort. To address these issues, we propose ConstraintDial, a framework for lifelong construction of smart contract security tasks via protocol mutation. It automates local contract reconstruction and uses attack execution to select changes to the required calls or inputs. Execution checks guide revisions that make the original attack fail while preserving exploitability, the security objective, and normal functionality. New incidents and audit findings enter the same procedure. With ConstraintDial, we construct 3,045 task inputs from 203 public cases, covering detect, exploit, and patch. Compared with reconstruction alone, protocol mutation reduces success with supplied public answers by 10.9 percentage points on average across detect, exploit, and patch. The fraction of tasks solved by copying online answers falls by 9.3 points.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM agents, smart contract, vulnerability, mutation, task synthesis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_gLRcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CONSTRAINTDIAL, an automated framework for continuously constructing smart contract security tasks through protocol mutation. The framework addresses the problem that LLM agents may solve benchmark tasks by memorizing or copying publicly available answers. It changes the function calls or inputs needed to trigger an attack while preserving the underlying vulnerability. It does this by changing the state or input preparation required by the protocol. Using CONSTRAINTDIAL, the authors construct the ReCast benchmark from 203 public cases. The benchmark contains 3,045 task inputs covering vulnerability detection, exploitation, and patching. Compared with reconstruction alone, protocol mutation reduces the success rate when models are given public answers by an average of 10.9 percentage points. It also reduces the proportion of tasks solved by copying online answers by 9.3 percentage points.

Critical strengths
  1. The paper addresses an important problem in smart contract security evaluation. Public attack scripts and reports can allow models to solve tasks by copying existing answers. CONSTRAINTDIAL changes the calls or inputs needed to carry out an attack while keeping the underlying vulnerability and security objective. This provides a practical way to reduce direct answer transfer.
  2. The framework automates local contract reconstruction and protocol mutation. It also uses failed execution checks to guide later revisions. The ablation study shows that using test feedback during construction produces more accepted task versions than applying the same checks only at the end.
  3. The framework uses several checks to validate each mutated task. The original attack must fail, an adapted attack must succeed on the vulnerable protocol, the adapted attack must fail on the repaired protocol, and normal operations must still work. These checks help ensure that the mutation changes the attack conditions without removing the vulnerability or breaking the protocol.
  4. The paper evaluates six LLM-based agents and compares settings with and without supplied public answers. It also allows network access in both settings. The results provide evidence that protocol mutation reduces the usefulness of copied answers, especially for exploit tasks, while still allowing models to solve tasks through adaptation.
Critical weaknesses
  1. Increasing the mutation complexity does not consistently reduce success rates across all models. For exploit tasks, the success rate on Combined changes increases for four of the six models. This suggests that the current mutations do not always make the tasks harder and may mainly change the solution path.
  2. Some successful attacks require only simple preparation calls. The paper reports that 28 of 88 successful attacks under protocol mutation only added preparation calls, such as registration or opening a window. This suggests that some mutations may not test deeper changes in data flow or control flow, which limits how well the benchmark measures advanced vulnerability reconstruction.
  3. The statistical evidence for the overall Base-to-Combined changes differences is limited. All reported 95% confidence intervals for these comparisons include zero. Therefore, the observed changes do not provide strong evidence of a consistent increase in difficulty at the current sample size.
  4. The cost analysis reports average construction costs, including rejected proposals and revision attempts, but gives limited information about the cost distribution. For example, it does not show how construction cost changes with mutation type or how much extra cost is needed for versions that require more revisions. It would also be useful to relate construction cost to the number and quality of the accepted tasks.
Questions for authors
  1. Why does the exploit success rate increase for some models on Combined changes? Could some mutations unintentionally introduce patterns or protocol structures that are easier for LLMs to recognize?
  2. The paper reports that some successful attacks only add preparation calls. Do the authors plan to include more mutations involving deeper state dependencies, input relationships, or control flow changes?
  3. Since all 95% confidence intervals for the Base-to-Combined changes success differences include zero, could the authors provide more samples or additional paired analysis to determine whether the observed changes are reliable?
3 comments
AuthorsOpenReview ↗

Response to Reviewer gLRc (1/2)

Dear Reviewer gLRc,

Thank you for your careful review. We have uploaded the latest revised PDF. References below use its numbering.

W1 / Q1. Why does exploit success increase for some models?

The mutations change the required attack sequence rather than imposing a fixed difficulty scale. A registration or window-opening requirement can make that sequence more explicit and easier for an agent to follow. Changes to accounts, asset sources, callback senders, or input dependencies can require a different adaptation.

The case transitions show both gains and losses. Qwen3.8-Max gains ten cases and loses three from Base to Combined; DeepSeek-V4.1-Flash gains twelve and loses six. An aggregate increase can therefore coexist with failures on previously solved cases (Section 4.2.3 and Table 9). We report these transitions without assigning a single cause to the increase or claiming that mutation always makes tasks harder.

W2 / Q2. Do mutations require more than preparation calls?

Yes. The construction supports changes to required state, participant and asset relationships, call order, callback senders, and inputs derived from earlier operations (Section 3.2.2). Adding a preparation call is only one possible adaptation.

With a public answer supplied, 60 of 88 successful attacks on mutated protocols change calls or inputs beyond preparation, while 28 succeed with preparation alone. The corresponding simple-preparation result is 9 of 100 (Table 10). These counts describe successful submissions.

The SMOOFSStaking replay shows why preparation alone may fail. Adding registration and withdrawal-window opening reaches NFT authorization but fails there. The NFT deposit reverts because the vault is not approved to transfer the helper's NFT, 45,000 reward tokens remain in the vault, and 0 of 2 extraction checks pass. The agent succeeds only after also authorizing the vault and correcting the callback sender (Appendix C.2).

We also construct deeper variants that add dependencies between state updates, input values, and operation order rather than only adding preparation calls. These changes are controllable but can increase task difficulty and construction cost. The deeper set contains 36 variants, approximately 0.34 times the 105 preparation-only versions. It is evaluated on all 61 families with three runs per condition. Across these runs, mean success is 45.8% for the deeper variants and 54.6% for the preparation-only variants. Direct copying succeeds in 9.0% of deeper-variant sessions, compared with 23.2% on Base.

W3 / Q3. What do the paired analyses show?

All Base-to-Combined intervals for success with usual materials include zero (Table 9), so these results do not establish a consistent change in overall success. The primary single-mutation contrast is Change 1 versus Equivalent control, rather than Change 1 versus Base.

For the public-answer analyses, the same 61 families give:

ResultReduction
Supplied-answer success10.9 pp
Successful direct copying9.3 pp

Each resample retains both models, all three task types, and both versions of a family (Appendix B.5). The full 61-case evaluation includes three runs per task, including the initial run.

The full 61-case evaluation includes three runs per task, including the initial run. We report per-run outcomes for all 61 families. Run 1 gives Base 52.7%, Equivalent control 55.5%, and Combined changes 55.7%. Run 2 gives Base 55.7%, Equivalent control 54.1%, and Combined changes 52.7%. Run 3 gives Base 54.1%, Equivalent control 55.5%, and Combined changes 55.5%. The mean Combined-minus-Base difference is +0.5 percentage points.

AuthorsOpenReview ↗

Response to Reviewer gLRc (2/2)

W4. How does construction cost relate to revisions and accepted output?

Simple preparation produces the most accepted versions at the lowest cost. The full method produces more accepted versions than direct rewriting at a lower cost per accepted version, but with a higher total cost. Appendix C.5 reports all costs on the same 61 cases and 122 proposals per method, including rejected proposals and revisions.

MethodFully accepted versionsMean revisionsTotal cost (USD)Cost per proposal (USD)Cost per accepted version (USD)
Simple preparation1050.747.580.390.45
Direct rewriting651.5112.240.921.73
Full method1001.4150.061.231.50
Without execution traces731.9142.741.171.96
Without paired checks751.1122.001.001.63
Without test feedback610.095.160.781.56

Revision burden and rejected proposals both contribute to total cost, while cost per accepted version connects that cost with output. The first table aggregates results by method; the next two break them down further.

For the full method, the following tables add the cost distribution by mutation type and by revision count, including rejected proposals and revision costs.

Mutation typeProposalsMean revisionsAcceptedTotal cost (USD)Cost per accepted version (USD)
Preparation only300.62718.900.70
State dependency341.22938.761.34
Input relationship321.52543.201.73
Control-flow change262.61949.202.59
Revision countProposalsAcceptedTotal cost (USD)Cost per accepted version (USD)
0343217.340.54
1423842.841.13
2282241.161.87
3 or more18848.726.09

Thank you again for you thoughtful reviews and time and effor to our community!

Best,
Authors

Dear Reviewer gLRc,

Thank you again for your thoughtful feedback! We hope our response has addressed your concerns. Please feel free to let us know if there are any remaining questions or points that you would like us to clarify. We would be happy to discuss them further.

Best regards,
The Authors

Reviewer_r7prOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to build smart contract security tasks that cannot be solved simply by copying public exploits. The proposed framework, ConstraintDial, reconstructs a public incident or audit finding as a local vulnerable protocol P and a repaired protocol Q. It then changes a precondition on the reference attack path (e.g., a required state or a call dependency) in both protocols. A mutated version is accepted if the ported original attack fails, an adapted attack succeeds on P but fails on Q, rolling back the mutation restores the original attack, and normal-operation tests pass on both protocols. Failed checks are fed back to the construction model (GLM-5.3) for up to 10 revisions.

The authors use this pipeline to build ReCast from 203 public cases, with five versions per case (Base, Equivalent control, Change 1, Change 2, Combined) and three tasks per version (detect, exploit, patch), for 3,045 task inputs. Six models are evaluated with Claude Code on 61 sampled cases (915 tasks per model), with one one-hour run per task and network access enabled. The paper reports three main findings. Mutation lowers success with a supplied public answer by 10.9 pp on average. It reduces the share of tasks solved by copying online answers from 23.2% to 13.9%. It changes which cases a model solves, without uniformly lowering success rates. Ablations suggest that execution traces, paired checks, and test feedback all improve construction yield.

Critical strengths

S1. The paired vulnerable/repaired protocol design is the most convincing part of the paper. The adapted attack must fail on Q, and rolling back the mutation must restore the original attack. Together these give reasonable evidence that the mutation changes the attack precondition and leaves the vulnerability intact.

S2. The problem is relevant, and the approach does not rely on restricting network access. Changing whether a public answer still applies is a cleaner way to study answer reuse than blocking retrieval.

S3. Grading is designed with care. Diagnoses and patches that remain valid after mutation still receive credit, and trigger coverage is scored separately. The patch grader is also checked against unchanged code and against repairs that over-block.

S4. The results section reports mixed and null results honestly (e.g., exploit success rising after mutation, and all Base vs. Combined intervals in Table 8 including zero). The construction ablations apply the same final checks to every variant.

Critical weaknesses

W1. It is unclear which tasks in the released benchmark passed full validation. C.3.1 says the complete checks (repaired protocol, rollback, task checks) were run only on the 305 sampled versions, and its execution counts do not add up. Each task type also has exactly 1,015 inputs, although Tables 13–14 show that acceptance differs by task type.

W2. Fidelity to the source cases is not established. All artifacts come from the same LLM pipeline, so they can agree with each other while misrepresenting the original case. Table 12 shows that ported historical attacks succeed on only 5 of 10 Base reconstructions. Several checks also read as manual (§3.2.1, §3.2.2, B.4.2, B.4.3), which conflicts with the "full automation" claim in Table 1.

W3. The reduction in answer benefit is uneven across tasks. Pooling the two models, the gain from supplied answers drops for exploit (33.6 → 9.8 pp) and detect (38.5 → 27.0 pp), but remains 21.3 pp for patch. For exploit, about half of this reduction comes from higher success without supplied answers (51.6% → 62.3%). The paper should better distinguish changes in task solvability from changes in answer transferability.

W4. The robustness of the reported effects needs further assessment. Each task is run only once. For DeepSeek on exploit, the only Table 8 interval excluding zero compares Change 1 against an Equivalent control 13.1 pp above Base, while Change 1 is only 1.6 pp below Base. The headline 10.9 and 9.3 pp reductions lack uncertainty estimates. Repeated runs on a subset would help assess stability.

W5. The difference from the simple-preparation baseline is unclear. Every example in the paper adds registration or window-opening calls, which is how that baseline is defined (§4.2.3). The baseline also accepts more versions at lower cost. The case for the full method rests on Table 9, which uses only two models and single runs.

Questions for authors
  1. Appendix C.3.1 reports complete checks for the 305 sampled versions. Did the remaining versions also pass all applicable protocol and task checks? Please clarify the filtering and acceptance counts leading to the 203 retained families.

  2. Which checks in §3.2.1, §3.2.2, B.4.2, and B.4.3 are performed by humans, LLMs, or scripts? Please distinguish task construction and validation from agent-output grading and clarify the scope of the “full automation” claim.

  3. What causes the five failed transfers of mechanically ported historical attacks to Base in Table 12, and what evidence confirms that these reconstructions preserve the source vulnerability and intended normal behavior?

  4. Could you repeat the Base, Equivalent, and Combined conditions on a representative subset to assess within-task variability and the stability of their differences? Please also report paired uncertainty for the supplied-answer and direct-copy results.

  5. Table 9 classifies changes in successful agent submissions. What fraction of the constructed mutations requires changes beyond adding preparation calls? Could you provide representative examples explaining why adding preparation calls to the validated Base attack is insufficient?

3 comments
AuthorsOpenReview ↗

Response to Reviewer r7pr (1/2)

Dear Reviewer r7pr,

Thank you for carefully examining our validation and grading procedures. We have uploaded the latest revised PDF. References below use its numbering.

W1 / Q1. Which inputs passed full validation, and how are checks counted?

The 3,045 figure counts prepared inputs from 203 families, with five versions and three task types per family, or 1,015 inputs per type. Complete protocol and task checks are reported for 305 sampled versions. The other 710 prepared versions have the separate reference execution screen (Section 4.1 and Appendices B.1 and C.3.1). Tables 13–14 report acceptance by task type.

Appendix C.3.1 now reconciles the reference execution counts. Each family has 18 outcomes:

  • 2 Base self-checks, one per initialization;
  • 2 Equivalent-control mechanical attack checks, one per initialization;
  • 8 normal-operation checks for Equivalent control and three changed versions on two initializations; and
  • 6 changed-version attack checks for those three versions on two initializations.

Thus, 2 + 2 + 8 + 6 = 18 per family and 203 × 18 = 3,654 recorded outcomes. The total already includes both initializations. These outcomes are reference execution results, while the 305 sampled versions have complete protocol and task validation, including repaired protocols and rollback.

W2 / Q3. What supports source fidelity, and what caused the five failed transfers?

Source fidelity comes from the local reference attack and the paired protocol checks. The local reference attack must achieve the fixed objective on vulnerable Base and fail on its repaired counterpart, while normal-operation tests pass on both (Section 3.2.1, Eq. 1). The vulnerability and repair must also match the source evidence. Appendix A.1 gives a concrete account-mismatch example, and Appendix A.2 describes the paired-code, execution, and rollback checks.

Table 13 tests a different artifact: whether a historical public attack works after only identifier and interface substitutions. Five of ten transfers succeed on Base. The current records do not break down the other five into case-specific causes, and the revision states this directly. A failed historical transfer does not imply that the validated local reference attack fails.

Q2. Which checks are performed by humans, LLMs, or scripts?

Construction uses an LLM for source correspondence, and execution uses scripts for protocol validation. The construction LLM checks the source evidence and paired code. Scripts compile and initialize the protocols, replay attacks, perform rollback, and run normal-operation tests.

Output grading is separate. Scripts check locations and determine exploit success from replayed transactions, balances, receipts, and protocol state. Blinded reviewers assess explanation rubrics and review paths attributed to another vulnerability or an unintended shortcut (Appendices A.2 and B.4). The current records do not identify whether those reviewers are humans or LLMs, so “full automation” refers only to task construction and execution-based protocol validation (Section 2).

W3. How do answer benefits and task solvability differ?

The three tasks show different changes in success with usual materials and with a supplied answer.

TaskUsual Base → CombinedSupplied Base → CombinedAnswer benefit Base → Combined
Detect44.3% → 45.9%82.8% → 73.0%38.5 → 27.0 pp
Exploit51.6% → 62.3%85.2% → 72.1%33.6 → 9.8 pp
Patch64.8% → 54.9%86.1% → 76.2%21.3 → 21.3 pp

For exploit, the answer benefit decreases through both a 13.1-point reduction in supplied-answer success and a 10.7-point increase in success with usual materials. For patch, both rates fall by the same amount, leaving the benefit unchanged. Correct diagnoses and repairs still receive credit when they remain valid after mutation.

AuthorsOpenReview ↗

Response to Reviewer r7pr (2/2)

W4 / Q4. How stable are the effects?

The primary single-mutation comparison is Change 1 versus Equivalent control. For DeepSeek-V4.1-Flash on exploit, Base, Equivalent, and Change 1 success rates are 59.0%, 72.1%, and 57.4%. The −14.8-point difference has interval [−26.2, −3.3]. Equivalent is 13.1 points above Base, while Change 1 is 1.6 points below Base. All Base-to-Combined intervals include zero (Table 9 and Appendix B.5).

For the two answer-reuse results, the same 61 families give:

ResultReduction
Supplied-answer success10.9 pp
Successful direct copying9.3 pp

Each resample keeps both models, all three task types, and both versions within each family. The full 61-case evaluation includes three runs per task, including the initial run.

The full 61-case evaluation includes three runs per task, including the initial run. We report per-run outcomes for all 61 families. Run 1 gives Base 52.7%, Equivalent control 55.5%, and Combined changes 55.7%. Run 2 gives Base 55.7%, Equivalent control 54.1%, and Combined changes 52.7%. Run 3 gives Base 54.1%, Equivalent control 55.5%, and Combined changes 55.5%. The mean Combined-minus-Base difference is +0.5 percentage points.

W5 / Q5. What distinguishes mutation from simple preparation?

Most successful attacks with a supplied answer require changes beyond preparation. The count is 60 of 88 on mutated protocols, compared with 9 of 100 under simple preparation. By model, the counts are 29/40 versus 5/47 for Qwen3.8-Max and 31/48 versus 4/53 for DeepSeek-V4.1-Flash. Another 28 successful attacks on mutated protocols use preparation calls alone. These counts describe successful submissions rather than every constructed mutation (Table 10).

Appendix C.2 gives a concrete contrast. Audius works with preparation alone. On SMOOFSStaking, the preparation-only replay adds registration and withdrawal-window opening but fails at NFT authorization. The first decisive failure is the NFT deposit: the vault is not approved to transfer the helper's NFT, 45,000 reward tokens remain in the vault, and 0 of 2 extraction checks pass. The agent succeeds only after also authorizing the vault and correcting the callback sender.

We also construct deeper variants that add dependencies between state updates, input values, and operation order rather than only adding preparation calls. These changes are controllable but can increase task difficulty and construction cost. The deeper set contains 36 variants, approximately 0.34 times the 105 preparation-only versions. It is evaluated on all 61 families with three runs per condition. Across these runs, mean success is 45.8% for the deeper variants and 54.6% for the preparation-only variants. Direct copying succeeds in 9.0% of deeper-variant sessions, compared with 23.2% on Base.

Simple preparation remains more productive and cheaper: 105 fully accepted versions at USD 0.39 per proposal and USD 0.45 per accepted version, compared with 100 at USD 1.23 and USD 1.50 for the full method (Appendix C.5). The revision presents this as a construction trade-off rather than a uniform advantage.


Thank you again for you thoughtful reviews and time and effor to our community!

Best,
Authors

Dear Reviewer r7pr,

Thank you again for your thoughtful feedback! We hope our response has addressed your concerns. Please feel free to let us know if there are any remaining questions or points that you would like us to clarify. We would be happy to discuss them further.

Best regards,
The Authors

Reviewer_Qw6kOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ConstraintDial, an automated framework for constructing smart contract security tasks through protocol mutation, aiming to reduce answer copying and manual construction effort in existing benchmarks. The framework first reconstructs vulnerable and repaired protocols from public security cases and analyzes execution traces to identify vulnerability trigger conditions. It then applies protocol mutations guided by execution feedback to invalidate original attacks while preserving exploitability and normal functionality. Using this framework, the authors construct ReCast, a benchmark containing 3,045 task inputs from 203 public cases, covering vulnerability detection, exploitation, and patching. Experiments with six LLM-based agents show that the approach can reduce direct reuse of public answers and improve the generation of valid tasks.

Critical strengths
  1. The paper addresses a practically important problem. The paper highlights how public vulnerability reports and historical attack code can affect the reliability of smart contract security benchmarks. Rather than restricting network access, it uses protocol mutation to reduce agents' reliance on existing answers while allowing online information retrieval. This setting better reflects real-world scenarios where agents can search for information and use external tools.
  2. An automated pipeline with useful empirical results. The framework combines paired vulnerable and repaired protocols, execution-based validation, and iterative feedback to automate task construction. The experiments evaluate agents under different information conditions, including access to historical public answers and validated Base answers, providing useful insights into how agents adapt existing solutions. Multi-model evaluations and ablation studies further support the practical value of the framework and its components.
Critical weaknesses
  1. Limited methodological novelty. The proposed framework mainly combines existing techniques, including program mutation, execution trace analysis, and test feedback, with similar ideas already explored in prior benchmark construction research. The method primarily changes vulnerability reachability and trigger conditions rather than introducing substantially new vulnerability interactions or richer protocol semantics. As a result, the technical novelty appears incremental, and the paper does not clearly establish a sufficiently distinct methodological contribution.
  2. Insufficient experimental support for the claimed benefits. Although the paper reports a reduction in direct answer-copying success from 23.2% to 13.9%, the relevant experiments involve only two models and 61 cases, excluding prominent GPT and Claude models with strong coding capabilities, which limits the generalizability of the findings. Moreover, protocol mutation has inconsistent effects across detection, exploitation, and patching tasks, and some historical diagnoses and patches remain valid after mutation. In Table 8, none of the success-rate changes from Base to Combined changes are statistically significant at the 95% confidence level, leaving the broader impact of protocol mutation on agent performance uncertain.
Questions for authors

The main questions are outlined in Critical Weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer Qw6k

Dear Reviewer Qw6k,

Thank you for recognizing the value of studying public-answer reuse with network access. We have uploaded the latest revised PDF. References below use its numbering.

W1. What paired protocol mutation contributes

ConstraintDial constructs new attack requirements and checks them on both the vulnerable and repaired protocols. It selects a condition from the reference attack's execution trace and changes that condition in both protocols while keeping the vulnerability and security objective (Section 3.2, Figure 1).

An accepted version must satisfy four checks:

  1. the mechanically ported attack fails at the selected condition;
  2. an adapted attack succeeds on the vulnerable protocol and fails on the repaired one;
  3. rolling back the change restores the original attack; and
  4. normal-operation tests pass on both protocols.

BenchEvolver rewrites reference solutions and selects variants by measured model difficulty (Section 2). Our construction creates new requirements involving state, participants, assets, call dependencies, and inputs, and verifies them through execution.

The revised text also separates task construction from agent adaptation. With a public answer supplied, 60 of 88 successful attacks on mutated protocols change calls or inputs beyond preparation, compared with 9 of 100 under simple preparation (Section 4.2.3, Table 9). For construction, the full method produces 100 fully accepted versions out of 122, compared with 65 for direct rewriting (Section 4.2.4, Table 4).

W2. Evidence across models and tasks

The answer-reuse analysis covers Qwen3.8-Max and DeepSeek-V4.1-Flash on 61 cases per task, with network access in both conditions. GPT and Claude were excluded after setup refusals, so the conclusions are limited to these two models. The separate six-model comparison evaluates all five versions with usual materials (Table 2).

With a supplied public answer, success falls by 10.9 percentage points from Base to Combined, with a case-paired 95% CI of [6.0, 16.1]. Successful direct copying with usual materials falls from 23.2% to 13.9%, a reduction of 9.3 points, with a case-paired 95% CI of [4.6, 13.9]. The corresponding reductions are 4.1 points for detect, 12.3 for exploit, and 11.5 for patch (Table 12 and Appendix B.5).

The three tasks show different effects:

  • Exploit: answer benefit falls from 33.6 to 9.8 points.
  • Detect: answer benefit falls from 38.5 to 27.0 points.
  • Patch: answer benefit stays at 21.3 points because both success rates fall by the same amount.

A diagnosis or repair that remains valid after mutation still receives credit (Section 3.2.3 and Appendix C.3.5). This separates whether a public answer still works from which cases an agent can solve.

Finally, all Base-to-Combined intervals for success with usual materials include zero (Table 9). We therefore report case transitions alongside the aggregate rates and describe the effect as a change in answer reuse and in which cases are solved, rather than a uniform increase in difficulty. The full 61-case evaluation includes three runs per task, including the initial run. The paired intervals above describe variation across case families.

The full 61-case evaluation includes three runs per task, including the initial run. We report per-run outcomes for all 61 families. Run 1 gives Base 52.7%, Equivalent control 55.5%, and Combined changes 55.7%. Run 2 gives Base 55.7%, Equivalent control 54.1%, and Combined changes 52.7%. Run 3 gives Base 54.1%, Equivalent control 55.5%, and Combined changes 55.5%. The mean Combined-minus-Base difference is +0.5 percentage points, with a run-paired 95% interval of [-7.3, 8.2]. These runs supplement the family-resampled intervals above.

We also construct deeper variants that add dependencies between state updates, input values, and operation order rather than only adding preparation calls. These changes are controllable but can increase task difficulty and construction cost. The deeper set contains 36 variants, approximately 0.34 times the 105 preparation-only versions. It is evaluated on all 61 families with three runs per condition. Across these runs, mean success is 45.8% for the deeper variants and 54.6% for the preparation-only variants. Direct copying succeeds in 9.0% of deeper-variant sessions, compared with 23.2% on Base.


Thank you again for you thoughtful reviews and time and effor to our community!

Best,
Authors

Dear Reviewer Qw6k,

Thank you again for your thoughtful feedback! We hope our response has addressed your concerns. Please feel free to let us know if there are any remaining questions or points that you would like us to clarify. We would be happy to discuss them further.

Best regards,
The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Reviewers, Area Chair, Senior Area Chair, and Program Chairs,

Thank you for the careful reviews and for your time and effort on behalf of the community! This response briefly summarizes the reviewers' positive assessments and how the revision addresses their main concerns.

We are grateful to the three reviewers for recognizing the following aspects of our work:

  • Qw6k: “The paper addresses a practically important problem.” They also highlighted our “automated pipeline with useful empirical results.”
  • r7pr: “The paired vulnerable/repaired protocol design is the most convincing part of the paper.” They further noted that “grading is designed with care.”
  • gLRc: “The paper addresses an important problem in smart contract security evaluation.” They also wrote that “the framework automates local contract reconstruction and protocol mutation.”

These comments identify our central contribution: paired, execution-checked construction of security tasks that changes what an attack requires while preserving the vulnerability and normal operations.

2. Responses to shared concerns

  1. Validation and counting: We clarified the validation scope and counting: 3,045 prepared inputs, 305 fully validated versions, and a separate reference execution screen for the remaining versions.
  2. Method and answer reuse: We explained the method through paired execution checks and reported answer-reuse reductions of 10.9 points for supplied-answer success and 9.3 points for direct copying.
  3. Robustness and deeper variants: We added a full 61-case, three-run evaluation with a run-paired 95% interval of [−7.3, 8.2], and added 36 variants with added state, input, and control-flow dependencies.
  4. Construction cost: We reported costs by mutation type and revision count, while keeping the cost advantage of simple preparation explicit.

3. Responses to individual reviewers

Reviewer Qw6k

  1. Methodological novelty: We explain the contribution as new attack requirements verified by paired execution checks, and distinguish task construction from agent adaptation.
  2. Experimental support and scope: The answer-reuse analysis covers Qwen3.8-Max and DeepSeek-V4.1-Flash on 61 cases per task, with paired intervals for the two headline reductions of 10.9 and 9.3 points. We also report different effects by task and do not claim a uniform increase in difficulty.

Reviewer r7pr

  1. Validation counts: We distinguish 3,045 prepared inputs from 305 fully validated versions and the separate reference execution screen.
  2. Source fidelity and automation: We explain the local reference attack, paired checks, and the separate roles of the construction LLM, execution scripts, and blinded reviewers.
  3. Answer benefits and solvability: We separate usual-material success from supplied-answer success and explain the different changes for detect, exploit, and patch.
  4. Stability: We added a full 61-case, three-run evaluation with a run-paired 95% interval of [−7.3, 8.2].
  5. Simple preparation: We compare preparation-only adaptation with deeper mutation and include the SMOOFSStaking replay, which fails at NFT authorization.

Reviewer gLRc

  1. Higher exploit success on Combined changes: We explain that mutation can make the required sequence more explicit and report both gained and lost cases.
  2. Preparation-only attacks: We add 36 variants with added state, input, and control-flow dependencies and compare them with preparation-only variants.
  3. Uncertainty: We report paired intervals for supplied-answer success and direct copying, and add the repeated-run analysis.
  4. Construction cost: We report costs by mutation type and revision count, including rejected proposals and revision effort, while keeping simple preparation's cost and output advantage explicit.

Thank you again for your time and constructive feedback.

Best regards,
The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.