acceptodds
Under review as a conference paper at ICLR 2027

CoEvo-SQLi: Co-Evolving SQL Injection Detectors via Schema-Grounded Attack Synthesis

Abstract

Pre-execution SQL injection (SQLi) detection must block malicious queries without disrupting benign workloads and keep sensitive queries and schemas within a local trust boundary, requirements that existing defenses only partially meet. Rule- and signature-based systems are lightweight but brittle under obfuscation and semantics-preserving rewrites; learned and LLM-based detectors capture richer query patterns but are typically trained once on a fixed corpus. For small local detectors, such static training raises two problems: variants outside the fixed corpus cannot be absorbed without full retraining, and an undifferentiated budget is spent largely on already-learned, conspicuous patterns while underrepresented variants with unresolved errors receive little. We address both problems with CoEvo-SQLi, a feedback-directed co-evolutionary training framework coupling attack synthesis with multi-round detector updates. In each round, an Attacker synthesizes SQLi examples, guided by Verifier signals indicating which injection clusters should receive more training budget and where to explore further variants; the Defender fine-tunes on the resulting training slice and runs inference on a fixed validation set, whose errors the Verifier converts into next-round feedback. These coupled updates let the Attacker target the Defender's current weaknesses while the Defender learns from the resulting variants. We also introduce SQLi-bench, a schema-grounded benchmark of 48 SQLi clusters for controlled training and evaluation across diverse attack forms. Under a matched training-example budget, CoEvo-SQLi achieves a Matthews correlation coefficient (MCC, ×100) of 92.5 on SQLi-bench, versus 78.0 for an identically configured static fine-tuning baseline; removing feedback-directed synthesis or payload mutation costs 10.5 and 9.2 points, respectively. Its query-only version attains the highest mean MCC on SQLi-bench, rbsqli, and WAF-A-MoLe.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
SQL injection detection, cybersecurity, adaptive data synthesis, schema-grounded learning, large language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_p1eFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CoEvo-SQLi, a framework for improving SQL injection detection through iterative training and feedback-guided attack generation. It combines attack synthesis, LLM-based payload mutation, and adaptive sampling based on the detector's validation errors. The authors also introduce SQLI-BENCH, a schema-grounded benchmark covering 48 injection clusters. The proposed method achieves better results than static fine-tuning, and the experiments include ablation studies and evaluations on external datasets. The work is practically relevant and generally well presented, although some aspects of the method and evaluation could be strengthened.

Critical strengths

The paper addresses the challenge of detecting evolving SQL injection attacks. The idea of using detection errors to guide further training is reasonable and well aligned with the problem.

SQLI-BENCH provides a structured evaluation of different attack types. Separating databases, queries, and payload templates across data splits is a strength of the experimental design.

The ablation studies help explain the contributions of feedback and mutation, while the external evaluations and efficiency measurements provide additional evidence of its practical potential.

Critical weaknesses

The framework combines several established techniques, including adaptive sampling, multiplicative weight updates, LLM-based data augmentation, and iterative fine-tuning. While the combination is useful, it is not entirely clear what new learning capability it offers beyond existing approaches. Comparisons with stronger adaptive sampling and adversarial training baselines would help clarify its contribution.

Attack generation relies on predefined templates and constrained LLM mutation. These constraints help maintain consistency but may limit the diversity of generated attacks. The paper does not fully establish whether the generated SQL queries represent effective attacks in real database environments. A comparison with direct LLM-based attack generation would also help justify the proposed design.

The method repeatedly uses the same validation set to adjust sampling weights, which may introduce bias toward particular attack patterns. Although the authors evaluate external datasets, these experiments only cover the query-only variant, and some improvements over Static SFT are not statistically significant. Further analysis of training stability and performance under different validation splits would be helpful.

There are a few minor inconsistencies between the main tables and appendix. For example, the TPR of deepseek-v4-flash is 60.0% in Figure 4 but 60.1% in Table 6. Similar differences appear between Tables 1 and 7. Check these values and ensure consistent reporting throughout the paper.

Questions for authors

How much does continuously updating the attack-cluster weights improve detection performance compared with calculating the weights once after the first training round and keeping them fixed for subsequent rounds, while maintaining the same training budget and other settings?

What are the main advantages of the proposed template-based, schema-grounded attack generation approach over direct LLM-based attack generation, particularly in terms of attack validity, diversity, and downstream detection performance?

3 comments
AuthorsOpenReview ↗

Response to Q1 (Part 1 of 3)

Thank you for your careful reading and constructive comments. We first answer the two questions you raised and then summarize the other revisions.

Q1. How much does continuously updating the attack-cluster weights improve detection performance compared with calculating the weights once after the first training round and keeping them fixed for subsequent rounds, while maintaining the same training budget and other settings?

We added a frozen-controller experiment in Appendix G. The backbone, fine-tuning configuration, eight-round training schedule, total training budget, validation set, and benign-ratio controller remain unchanged. The only change is that the cluster weights are computed from the first-round validation errors and then held fixed in rounds 2–8.

Setting: schema-grounded; frozen-controller comparison

ConditionTPR (%)FPR (%)MCC
Uniform sampling with mutation92.24.182.0
Frozen cluster weights after round 189.82.279.4
Full CoEvo-SQLi97.53.292.5

Freezing the cluster weights after round 1 makes MCC 13.1 points lower than the full system and 2.6 points lower than uniform sampling with mutation. The first-round error distribution can only reflect a local state of detector difficulty. Once the initially difficult clusters become easier through training, other clusters may become the main source of later validation errors. Therefore, fixing the first-round weights allocates the subsequent training budget according to an outdated error profile.

To distinguish accumulated credit assignment from simple hard-example selection, we also replace the accumulated update with instantaneous error-proportional resampling.

Setting: schema-grounded; cluster-selection comparison

Cluster-selection strategyTPR (%)FPR (%)MCC
Instantaneous error-proportional resampling95.02.987.9
Full accumulated controller97.53.292.5

The instantaneous rule uses only the validation errors from the current round, whereas the accumulated controller retains information across rounds. The 4.6-point MCC difference indicates that, beyond selecting currently difficult clusters, the contribution of CoEvo-SQLi is also related to cross-round credit assignment.

(This response continues in Part 2 of 3, supplied as a separate file: Q2 on template-based versus direct LLM-based attack generation, the other revisions, and the adaptive-sampling comparison.)

AuthorsOpenReview ↗

Response to Q2 (Part 2 of 3)

Q2. What are the main advantages of the proposed template-based, schema-grounded attack generation approach over direct LLM-based attack generation, particularly in terms of attack validity, diversity, and downstream detection performance?

We agree that a complete end-to-end comparison with unconstrained LLM generation would be valuable. However, we did not add such a baseline in this revision because we tried it early and eventually abandoned it. We therefore do not claim that template-based generation has been shown to outperform unconstrained generation in downstream detection. The specific reasons are as follows: First, unconstrained generation often produces samples with syntax errors, missing placeholders, or references to nonexistent tables or columns. Such samples cannot pass the acceptance checks; retrying them is substantially more costly than mutation under template constraints, and they may also teach incorrect decision cues. Second, unconstrained generation has a diversity problem: its outputs may differ substantially at the surface level while repeating the same underlying patterns, and most candidates are discarded after applying the requirement that they differ from previously accepted mutations. Finally, in downstream detection, schema-bound generation produces TSR payloads. This is the point rather than a defect. The samples refer to target tables, columns, and aliases, and the text alone cannot determine whether these objects actually exist; the label must be determined using the schema. This mode itself provides evidence of schema awareness.

We therefore added two new data-augmentation baselines that share the budget with CoEvo. The first is WAF-A-MoLE augmentation, which uses the same Qwen2.5-Coder-0.5B-Instruct backbone, fine-tuning configuration, and training slots as the CoEvo-SQLi query-only variant, allowing a comparison of different data-generation strategies. The second is GAN+RL, which jointly optimizes a generator and detector during training and is a generative baseline closer to a jointly trained attacker. Because this method uses its native generator-detector architecture, we treat it as a broader generative-training comparison rather than a same-backbone comparison. The query-only comparison on SQLI-BENCH is as follows:

Setting: query-only; SQLI-BENCH

MethodMCCTPR (%)FPR (%)Worst-cluster recall (%)
Static SFT77.187.81.025.6
WAF-A-MoLE augmentation71.983.91.216.8
GAN+RL79.394.613.757.4
CoEvo-SQLi81.591.02.046.9

WAF-A-MoLE augmentation uses the same Qwen2.5-Coder-0.5B-Instruct backbone, fine-tuning configuration, query-only input, and training-slot budget as the CoEvo-SQLi query-only variant. CoEvo-SQLi improves MCC by 9.6 points and worst-cluster recall by 30.1 points. GAN+RL has higher TPR and worst-cluster recall, but its FPR reaches 13.7%. In addition, CoEvo-SQLi achieves better performance on the other three public datasets; detailed results are provided in Appendix F.

Other revisions

We added stronger adaptive-sampling and adversarial-training baselines. For adaptive sampling, instantaneous error-proportional resampling replaces the original accumulated, centered multiplicative-weights update with sampling proportional to the current-round validation error, while keeping the backbone, mutation procedure, benign-ratio controller, number of training rounds, training budget, and available validation information unchanged. The schema-grounded results are:

Setting: schema-grounded; adaptive-sampling comparison

MethodMCCTPR (%)FPR (%)
Instantaneous error-proportional resampling87.995.02.9
Full CoEvo-SQLi92.597.53.2

(This response continues in Part 3 of 3, supplied as a separate file: the adversarial-training comparison, the training-time mutation audit, the validation-set analyses, the external-transfer results, and the numerical corrections.)

AuthorsOpenReview ↗

Response to other weakness; Revisions (Part 3 of 3)

For adversarial training, we reimplemented ModSec-AdvLearn in its native rule-derived feature space and classifier. The query-only results are:

Setting: query-only; adversarial-training comparison

MethodMCCTPR (%)FPR (%)
L1-SVM + adversarial training43.071.520.2
SecSVM + adversarial training41.882.436.9
CoEvo-SQLi (query-only)81.591.02.0

The MCC of instantaneous error-proportional resampling is 87.9, whereas the MCC of the full system is 92.5, a decrease of 4.6 points. This shows that the contribution of CoEvo-SQLi is not limited to selecting the currently hardest attack clusters; as the detector state changes, accumulated validation errors across rounds provide additional information. The L1-SVM and SecSVM variants of ModSec-AdvLearn achieve MCC values of 43.0 and 41.8, respectively, under the query-only contract, substantially below the MCC of 81.5 for the CoEvo-SQLi query-only variant.

We also audited the 240 accepted mutations actually used during training. These samples cover three seeds and eight rounds. Two annotators with SQL-security backgrounds judged the samples independently, a third annotator resolved disagreements, and every query was executed on the target MySQL database.

Setting: training-time mutation audit

Audit criterionResult
Intended attack technique preserved227/240 (94.6%)
Full semantic validity226/240 (94.2%)
Schema-referencing examples consistent with the schema153/155 (98.7%)
Clear malicious/benign judgement with SQL and schema238/240 (99.2%)
Structural breakage or unparseable9/240 (3.8%)

These results indicate that most of the generated SQL queries can constitute valid attacks in a real database environment, and they also support the validity and controllability of the constrained pipeline.

We also analyzed the risk that repeatedly using the same validation set could cause the model to overemphasize attack patterns that are particularly visible in that set. We reduce this risk in three ways:

  • feedback-driven sampling is mixed with uniform exploration, so every attack cluster retains a nonzero sampling probability in every round;
  • the training, validation, and test partitions are disjoint in databases, carrier queries, payload templates, and comment resources;
  • mutation changes payload forms or embedded SQL structures rather than simply repeating validation queries.

We also tested sensitivity to validation-set size. The validation-set-size analysis is as follows:

Setting: schema-grounded; validation-set size

Validation sizeFPR (%)FPR excess over Static SFT (percentage points)TPR (%)MCC
40 examples per cluster + 40 benign3.21.997.592.5
80 examples per cluster + 80 benign3.01.797.091.5
160 examples per cluster + 160 benign2.61.397.292.2

After enlarging the validation set, the main detection gain remains, but we do not claim that this analysis establishes robustness to all possible validation splits.

Finally, the query-only detector performs comparatively well when transferred to three external datasets without additional adaptation. We therefore interpret these external results as evidence of positive transfer on independently collected datasets:

Setting: query-only; external transfer

DatasetStatic SFT MCCCoEvo-SQLi MCC
rbsqli92.593.1
SQLiV365.877.1
WAF-A-MoLE84.487.9

The external experiments use query-only inputs because the comparison methods and public benchmarks generally do not provide schema inputs. We quantify the contribution of schema context through paired schema-grounded and query-only experiments on SQLI-BENCH.

The numerical inconsistencies identified by the reviewer were our mistake. In the revision, all tables were regenerated from the same artifacts and checked cell by cell. The two differences identified by the reviewer, as well as other last-digit differences of the same type, have been corrected. We also added discussion of the non-significant results.

The revisions include the frozen-controller experiment in Appendix G; the mutation audit in Appendix H; the validation-size analysis in Appendix I; the low-FPR operating-point analysis in Appendix J; the complete comparator protocols in Appendices K and E.4; and the corresponding explanations in Sections 5.1, 5.3, 5.4, and 7.

Reviewer_fjVxOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CoEvo-SQLi, a framework for training a small, locally deployed SQL injection detector. Over eight rounds, it uses validation errors to adjust which attack clusters receive more synthesized training examples and how much benign data is included. A fixed auxiliary language model generates additional attack variants.

The authors also introduce SQLi-Bench, which organizes attacks into 48 clusters and supports both schema-grounded and query-only evaluation. Under a matched 2,400-example training budget, schema-grounded MCC increases from 78.0 for Static SFT to 92.5 for CoEvo-SQLi. Attack recall improves from 88.5% to 97.5%, while the false-positive rate increases from 1.3% to 3.2%.

Critical strengths

the pipeline design for schema-grounded SQLi is relatively thorough, and the comparative experimental results are generally credible.

Critical weaknesses
  1. The semantic quality of dynamic mutations and the source of schema gains need more direct evidence. Mutation acceptance is mainly described through format, placeholder, category, and structural checks. The completeness audit explicitly does not guarantee successful execution. The expert audit examines a test subset without schema information and excludes ambiguous or invalid judgments when calculating label agreement. These checks are useful, but they do not directly establish that dynamically generated training examples preserve the intended attack mechanism. The schema example also relies on an incompatible UNION structure. Since malformed structure alone does not establish malicious intent, the evaluation would benefit from benign queries that look attack-like, legitimate metadata queries, and non-malicious structural errors. The goal should be to validate the intended attack behavior, without requiring every malicious attempt to succeed.
  2. The feedback components and false-positive trade-off are not fully separated. Feedback changes both attack-cluster weights and the benign sampling ratio, but the ablation does not clearly explain which controls remain active in the uniform condition. The results therefore establish the value of the combined feedback mechanism more clearly than the separate value of cluster-directed synthesis. The validation set also contains only 40 benign queries. One additional false positive changes its smoothed error rate by roughly 2.38 percentage points, which could affect the controller. Since the main result raises FPR from 1.3% to 3.2%, a larger benign evaluation and comparisons at the same low-FPR operating point would make the deployment claim more convincing.
Questions for authors
  1. Could you report category-level semantic validation of the generated mutations, including whether they preserve the intended attack mechanism? Could you also audit labels with schema information available?
  2. Does the uniform ablation retain benign-ratio control? How do fixed-ratio, ratio-only, and cluster-weight-only variants compare? How does the method compare with simple error-based resampling?
  3. How sensitive are the controller and final FPR to the small benign validation set? At the same low FPR, how much recall does CoEvo-SQLi gain over Static SFT?
  4. Could you complete the API, CRS, and learned-baseline configurations and add more seeds for the mutation and transfer comparisons that are currently inconclusive?
5 comments
AuthorsOpenReview ↗

Response to Q1,Q2 (Part 1 of 3)

Thank you for your careful reading and constructive comments. We respond point by point below.

Q1. Could you report category-level semantic validation of the generated mutations, including whether they preserve the intended attack mechanism? Could you also audit labels with schema information available?

The mutation acceptance checks in the original manuscript were mainly syntactic checks. A candidate mutation had to parse, preserve placeholders, match the specified category, and differ from existing mutations, but these conditions alone could not establish that the final query still expressed the intended attack. We therefore added an independent audit of 240 dynamic mutations actually used during training. These samples cover three seeds and eight rounds, with ten samples drawn from each seed-by-round stratum. Two annotators with SQL-security backgrounds judged each sample independently, and a third annotator resolved disagreements. The audit provided the complete SQL and target schema but hid the original binary label; each sample was also executed on the target MySQL database.

Setting: training-time mutation audit

Audit criterionResult
Intended attack technique preserved227/240 (94.6%)
Full semantic validity226/240 (94.2%)
Schema-referencing examples consistent with the schema153/155 (98.7%)
Clear label judgement with SQL and schema238/240 (99.2%)
Structurally invalid or unparseable9/240 (3.8%)

These results indicate that the dynamic training corpus is not dominated by syntactic or semantic label noise. They also indicate that most audited mutations preserve the intended attack mechanism and are consistent with the target schema.

Q2. Does the uniform ablation retain benign-ratio control? How do fixed-ratio, ratio-only, and cluster-weight-only variants compare? How does the method compare with simple error-based resampling?

The original 2×2 ablation switched the two feedback signals on or off together. We therefore added controller-separated experiments in Appendix G.

Setting: schema-grounded; controller comparison

ConditionMCCFPR (%)
Uniform / mutation82.04.1
Ratio control only84.13.1
Cluster control only88.14.8
Full system92.53.2

The cluster controller provides the larger recall-oriented gain, but it also increases FPR without benign-ratio control. The ratio controller alone improves MCC by 2.1 points and reduces FPR by 1.0 percentage point. When both controllers are active, the system reaches MCC 92.5 while keeping FPR at 3.2%.

We also added two control experiments to compare the cluster update rule and schedule.

Setting: schema-grounded; update-rule comparison

ConditionMCCFPR (%)
Instantaneous error-proportional resampling87.92.9
Frozen cluster weights after round 179.42.2
Full system92.53.2

We also added two control experiments to compare the update rule and schedule. Instantaneous error-proportional resampling uses only the validation errors from the current round and reaches MCC 87.9 with FPR 2.9%. The full accumulated controller reaches MCC 92.5 with FPR 3.2%. When the cluster weights are frozen after round 1 while the benign ratio continues to update, MCC falls to 79.4, which is 13.1 points below the full system and 2.6 points below uniform mutation. These results indicate that the controller needs to be updated as the detector state changes. A cluster may be difficult early on, become temporarily easier after one update, and become difficult again after other clusters have been learned. A purely instantaneous rule responds only to the current error surface, whereas the accumulated update retains information from earlier rounds.

(This response continues in Part 2 of 3, supplied as a separate file: Q3 on validation-set size and low-FPR operating points, and Q4 on the API, CRS, and learned-baseline configurations.)

AuthorsOpenReview ↗

Response to Q3, Q4 (Part 2 of 3)

Q3. How sensitive are the controller and final FPR to the small benign validation set? At the same low FPR, how much recall does CoEvo-SQLi gain over Static SFT?

We agree that the original validation set was small. While keeping the other settings unchanged, we therefore enlarged both the malicious and benign parts of the validation set. As the validation set grows, the schema-grounded results are:

Setting: schema-grounded; validation-set size

Validation sizeFPR (%)FPR excess(percentage points)TPR (%)MCC
40 examples per cluster + 40 benign3.21.997.592.5
80 examples per cluster + 80 benign3.01.797.091.5
160 examples per cluster + 160 benign2.61.397.292.2

As the validation set grows, FPR excess decreases while TPR and MCC remain close to the main results. Therefore, enlarging the validation set does not eliminate the main detection gain.

We also replaced the original fixed argmax decision with a continuous label-token log-odds score. The threshold is selected on a calibration split and fixed before testing.

Setting: schema-grounded; operating points

Calibration-side FPR targetStatic SFT test TPR (%)CoEvo-SQLi test TPR (%)Static SFT test FPR (%)CoEvo-SQLi test FPR (%)
0.5%79.389.70.100.15
1%88.495.61.161.55
2%92.498.03.202.86
3%93.898.58.0110.63

At every calibration-side target, CoEvo-SQLi has higher test recall. Because the calibration and test benign pools come from different held-out databases, the realized test FPR does not exactly equal the calibration target. At the 2% target, CoEvo-SQLi achieves both higher TPR and lower realized test FPR.

Q4. Could you complete the API, CRS, and learned-baseline configurations and add more seeds for the mutation and transfer comparisons that are currently inconclusive?

We have added complete configurations for the API, CRS, and learned baselines. Appendix K gives the reproduction protocols for ModSecurity CRS, Trident, EP-CNN, ModSec-AdvLearn, WAF-A-MoLE augmentation, and GAN+RL. Appendix E.4 gives the exact model identifiers, request strings, access route, concurrency, timeout, sampling settings, and thinking-mode settings for the API baselines.

The locally trained comparisons on the current query-only benchmark are:

Setting: query-only; locally trained baselines

MethodMCCTPR (%)FPR (%)Worst-cluster recall (%)
Static SFT77.187.81.025.6
WAF-A-MoLE augmentation71.983.91.216.8
GAN+RL79.394.613.757.4
CoEvo-SQLi81.591.02.046.9
L1-SVM with adversarial training43.071.520.213.9
SecSVM with adversarial training41.882.436.925.6

GAN+RL has the highest TPR, but its FPR is also substantially higher. CoEvo-SQLi has the highest MCC in this comparison; WAF-A-MoLE augmentation has a lower FPR but lower aggregate performance and worst-cluster recall.

We did not add more seeds for all transfer and mutation comparisons. The additional computation was used for the mechanism-separation experiments and the validation-size analysis. We therefore explicitly mark the non-significant comparisons and treat the corresponding margins as descriptive rather than conclusive. In particular, the transfer gains on rbsqli and WAF-A-MoLE cannot be statistically distinguished with three seeds.

(This response continues in Part 3 of 3, supplied as a separate file: our responses on the incompatible UNION example and the hard-negative set, and the closing list of revisions.)

AuthorsOpenReview ↗

Responses on the UNION Example and Hard Negatives; List of Revisions (Part 3 of 3)

We now turn to the issues raised in the weaknesses section:

You also pointed out the incompatible UNION structure in the motivating example. Your observation is well taken. In general, we consider a query that can run on the backend to be syntactically valid and consistent with the business logic. If a developer writes an incorrect number of UNION columns, the query will directly produce an error and will not enter the production path. Therefore, an incompatible projection count would rarely arise naturally in a normal business database. In an SQL injection workflow, however, an incompatible UNION probe can be an intermediate step for discovering the column count: an attacker first submits a query with mismatched projections, observes the resulting error, and then adjusts the next query. We therefore treat this example as supporting evidence of injection-oriented probing, rather than as sufficient evidence of maliciousness on its own.

We agree with the hard-negative set you suggested and confirm that it is a real gap in the current evaluation. All benign samples are taken from open-source datasets, and their SQL structures are left unchanged. The only perturbation is comment-layer prompt injection, applied symmetrically to both positive and negative classes. Consequently, the benign side does not contain deliberately constructed attack-like examples, and we cannot test whether the detector is misled by structural anomalies. Constructing a structure-aware hard-negative set will be left for future work.

In addition, we added explanations of the incompatible UNION example, the mutation audit, controller separation, instantaneous resampling, the frozen-controller experiment, validation-size analysis, and low-FPR analyses in Appendices G–J. Appendix K and Appendix E.4 provide the complete comparator protocols, and Sections 5.1, 5.3, 5.4, and 7 include the corresponding explanations.

Reviewer_AyZsOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies pre-execution SQL injection detection from complete SQL queries and optional database schemas. It proposes a multi-round training procedure in which a fixed language model generates taxonomy-constrained attack variants, validation errors from a small detector control subsequent attack-cluster sampling and the benign-example ratio, and the detector is fine-tuned on each generated training slice. The paper also introduces SQLI-BENCH, organized into 48 attack clusters, and evaluates the approach against static SFT, rule-based detectors, conventional learned detectors, and zero-shot API models under matched nominal training-example budgets.

Critical strengths

The separation of databases, carrier queries, payload templates, and comment resources across training, validation, and test sets is a substantive strength. The splits use four, two, and five databases, respectively, with 63 training templates separated from 180 test templates. This is more convincing than randomly splitting closely related variants. Pairing complete queries with schemas also introduces structural information generally absent from query-only evaluations.

On schema-grounded SQLI-BENCH, MCC improves from 78.0 for Static SFT to 92.5, with the appendix reporting p=.037p=.037 across three seeds. The 2×2 ablation gives MCC values of 73.3, 82.0, 83.3, and 92.5 for uniform/no mutation, uniform/mutation, feedback/no mutation, and the complete method, respectively. The improvement over uniform/mutation is reported at p=.010p=.010.

Recall on the hardest error-based/literal-only/no-comment cluster improves from 34% to 81%, and worst-cluster recall improves from 33.6 to 80.1. These are substantial gains that support the effectiveness of the recipe on this benchmark.

Critical weaknesses

1. The methodological contributions are primarily task-specific training recipes and benchmarks, with a lack of comparison against nearest-neighbor methods.

My first question is what remains methodologically new once the ATTACKER, DEFENDER, and VERIFIER terminology is removed. WAF-A-MoLE already studies feedback-guided SQLi mutation. More directly, ModSec-AdvLearn generates adversarial examples against the current detector, adds them to training, and retrains the defense. LLM-assisted attack generation and feedback loops have also been explored by SqliGPT and GenSQLi. The broad idea of generating attacks, obtaining feedback, and strengthening the defense is therefore not new here. WAF-A-MoLE, ModSec-AdvLearn, SqliGPT, GenSQLi

The clearest incremental contribution I can identify is the instantiation of these mechanisms for small-model detection from complete SQL queries and schemas: organizing a 48-cluster taxonomy, using validation errors to control multi-round budgets and the benign-example ratio, and providing the accompanying benchmark. This can be useful, but currently amounts to a specific training and data-construction recipe. The attack model is not trained alongside the defender, and the Verifier is not a learned policy; “co-evolution” primarily refers to feedback between the sampling policy and detector training.

The experiments do not establish the importance of this increment. The manuscript cites ModSec-AdvLearn, which has a public implementation, but Table 1 compares against ordinary ModSecurity CRS rather than that adversarial-training approach. WAF-A-MoLE appears as test data, not as a generation strategy used to train a comparable detector. Zero-shot API models, Trident, and EP-CNN provide system-level reference points, but do not replace comparisons with the closest training methods.

I do not expect native WAF numbers to be inserted directly into a schema-grounded table. A fairer comparison would fix the small backbone, complete SQL/schema inputs, generator, and budget, then compare ordinary adversarial augmentation, simple hard-example selection, and the proposed controller. Without such comparisons, the need for this particular design remains unclear.

2. Existing ablation studies do not sufficiently isolate the sources of performance gains.

Even if the proposed design is accepted as novel, the paper still needs to establish why it works. Data content, class proportions, the benign/malicious ratio, presentation order, and the multi-round optimization schedule change together. The 2×2 ablation is informative about the broad feedback and mutation switches, but does not distinguish generating a better training corpus from requiring dynamic feedback during training.

An important missing control is ordinary static SFT on the same data actually generated by CoEvo. Another is simple hard-cluster sampling with access to the same validation information. Without these controls, the gains cannot be attributed specifically to the proposed controller rather than to more suitable data and training schedules.

The feedback ablation is also underspecified. Figure 5 labels the axis as cluster selection, while the Verifier adjusts both cluster weights and the benign-example ratio. It is unclear whether the uniform conditions retain the benign-ratio controller. Although removing mutation reduces mean MCC by 9.2, the three-seed comparison gives p=.163p=.163, leaving uncertainty about the stability of that component’s contribution.

Matching 2,400 training-example slots is useful, but does not mean using the same 2,400 independent examples: training slices are not globally deduplicated, and benign examples can recur across rounds. CoEvo also repeatedly uses 1,960 validation examples for feedback. This is legitimate development supervision, and there is no evidence that test data enter the feedback loop; the issue is that alternative training strategies should receive the same information. Total offline time is 35.4 minutes versus 5.9 minutes for Static SFT, so matched example budgets should not be interpreted as comparable total resource costs.

3. The paper fails to demonstrate that it meets its core operational requirement: improving attack detection while keeping false positives under control.

The application motivation emphasizes avoiding disruption to benign workloads. Yet in the schema-grounded comparison, TPR increases from 88.5% to 97.5% while FPR rises from 1.3% to 3.2%. This is an operating-point trade-off whose costs need to be evaluated; higher MCC alone does not establish a better operational outcome.

Approximately 78.6% of test examples are malicious, while attack prevalence and false-blocking costs can differ in deployment. The reported precision or MCC does not directly establish whether the trade-off remains acceptable on workloads dominated by legitimate queries. Recall should be compared at the same strict FPR constraint.

The validation set used to control the benign ratio contains only 40 benign queries. A single false positive changes the raw FPR by 2.5 percentage points. This is insufficient evidence that the controller can reliably manage low false-positive rates. Benign-workload coverage is also limited, particularly for legitimate complex queries, administrative or metadata queries, and normal requests under distribution shift.

Questions for authors
  1. What is the central additional training mechanism relative to approaches such as ModSec-AdvLearn that already generate adversarial examples using detector feedback and retrain the detector? Can it be compared with ordinary adversarial augmentation or simple hard-example selection using the same backbone, complete SQL/schema inputs, and budgets? If not, what specifically prevents this?

  2. What happens if the data actually generated across CoEvo rounds are collected and used for static SFT at the same total example-presentation budget? This would directly distinguish a better-corpus explanation from benefits requiring dynamic feedback during training.

  3. What exactly changes when feedback is disabled in Figure 5? Do uniform conditions retain the benign-ratio controller? Are the optimization schedule and available validation information otherwise held constant?

  4. At the same strict low-FPR operating point, how much recall improvement remains? Are 40 benign validation examples sufficient to set and reliably control that operating point?

  5. Does the adaptation claim concern new templates within the known 48 clusters, or also attack forms outside that taxonomy? Is there evidence independent of the current generation pipeline that distinguishes template generalization from broader adaptation?

  6. How much ambiguity remains in the 16% of examples judged ambiguous once experts receive the correct schema? How much of the detection benefit can be explained by correct versus incorrect or irrelevant-schema controls and simple structural checks?

  7. What fraction of dynamic mutations receives independent validation of category consistency, attack semantics, or execution? Are there separate audits or results for ambiguous examples, invalid examples, and legitimate but complex or malformed queries?

6 comments
AuthorsOpenReview ↗

Response to Q1,Q2 (Part 1 of 3)

Thank you for your careful and constructive evaluation. We respond point by point below.

Q1. What is the central additional training mechanism relative to approaches such as ModSec-AdvLearn that already generate adversarial examples using detector feedback and retrain the detector? Can it be compared with ordinary adversarial augmentation or simple hard-example selection using the same backbone, complete SQL/schema inputs, and budgets? If not, what specifically prevents this?

The core mechanism of this paper is not the use of feedback alone. Instead, the detector's validation errors are converted into two continuously updated control signals: accumulated attack-cluster weights and a benign sampling ratio. In each round, the attack-sample distribution and the benign/malicious ratio are adjusted according to the current detector errors; the detector is then fine-tuned on the newly generated training slice and evaluated again on the validation set. Co-evolution therefore reflects a multi-round closed loop between the attack-data distribution and the detector state.

To compare with nearest-neighbour training strategies, we added four query-only baselines under a matched 2,400-slot training budget. Because WAF-A-MoLE, ModSec-AdvLearn, and GAN+RL do not define a schema-grounded input protocol, these comparisons uniformly use the query-only setting.

Setting: query-only; benchmark: SQLI-BENCH

MethodMCCTPR (%)FPR (%)Worst-cluster recall (%)
Static SFT77.187.81.025.6
WAF-A-MoLE augmentation71.983.91.216.8
GAN+RL79.394.613.757.4
L1-SVM + adversarial training43.071.520.213.9
SecSVM + adversarial training41.882.436.925.6
CoEvo-SQLi81.591.02.046.9

The most direct controlled comparison is WAF-A-MoLE augmentation. It uses the same Qwen2.5-Coder-0.5B-Instruct backbone, fine-tuning configuration, query-only input, and training-slot budget as CoEvo-SQLi. CoEvo-SQLi improves MCC by 9.6 points and worst-cluster recall by 30.1 points. ModSec-AdvLearn and GAN+RL use their respective detector architectures and therefore serve as broader adversarial-training references rather than strict same-backbone comparisons.

Q2. What happens if the data actually generated across CoEvo rounds are collected and used for static SFT at the same total example-presentation budget? This would directly distinguish a better-corpus explanation from benefits requiring dynamic feedback during training.

We added a corpus-replay control in Appendix G. The eight training slices generated during a complete CoEvo-SQLi run are pooled and used in one training pass. Cross-round duplicates are retained. The replay condition uses the same 2,400 training slots and is trained for two epochs, corresponding to 4,800 example presentations in total. The backbone, fine-tuning configuration, validation set, and test set remain unchanged.

Setting: schema-grounded; corpus replay

ConditionMCC
Static SFT78.0
Uniform synthesis with mutation, no feedback82.0
Corpus replay using the complete CoEvo-generated corpus90.4
Full CoEvo-SQLi92.5

The 8.4-point MCC difference between uniform synthesis and corpus replay reflects the effect of feedback on the content of the generated data. The additional 2.1 points from corpus replay to full CoEvo-SQLi are not statistically significant with three seeds (Welch test, p=.471p=.471). We therefore do not claim that the main performance gain necessarily depends on repeatedly training the detector inside the loop. The more direct conclusion supported by this experiment is that feedback changes the distribution of subsequently generated data, and that this change in corpus composition explains most of the performance improvement. Without running the feedback loop, the exact same corpus cannot be obtained in advance in a single synthesis pass because subsequent sampling decisions depend on the detector state after earlier training rounds. Once the detector has learned the more conspicuous attack clusters, the validation-error distribution changes and subsequent training slices are redirected toward clusters that remain difficult. Thus, the corpus-replay experiment separates “what data are generated” from “whether training occurs inside the loop.”

(This response continues in Part 2 of 3, supplied as a separate file: Q3 on what changes when feedback is disabled, Q4 on the low-FPR operating point and validation-set size, and Q5 on the scope of the adaptation claim.)

AuthorsOpenReview ↗

Response to Q3, Q4 (Part 2 of 3)

Q3. What exactly changes when feedback is disabled in Figure 5? Do uniform conditions retain the benign-ratio controller? Are the optimization schedule and available validation information otherwise held constant?

The original 2×2 ablation simultaneously switched the two feedback signals on or off. We therefore added controller-separated experiments in Appendix G.

Setting: schema-grounded; controller comparison

ConditionTPR (%)FPR (%)MCC
Uniform / mutation92.24.182.0
Ratio control only93.03.184.1
Cluster control only95.84.888.1
Instantaneous error resampling95.02.987.9
Frozen cluster weights after round 189.82.279.4
Corpus replay96.11.990.4
Full CoEvo-SQLi97.53.292.5

The cluster controller provides the larger recall-oriented gain, but it also increases FPR without benign-ratio control. The ratio controller alone improves MCC by 2.1 points and reduces FPR by 1.0 percentage point.

We also compare the accumulated controller with two simpler alternatives. Instantaneous error-proportional resampling uses only the current round's validation errors to select attack clusters and reaches MCC 87.9 and FPR 2.9%. Its MCC difference from the cluster-only condition is not statistically significant (p=.965p=.965). This suggests that the difference between the full system and simple hard-example selection is more likely due to cross-round accumulated credit assignment than merely selecting the currently hardest attack clusters. When the cluster weights are computed only from the first-round errors and then kept fixed, MCC drops to 79.4, which is 13.1 points below the full system and 2.6 points below uniform mutation. This indicates that cluster weights need to be continuously re-estimated as the detector state changes rather than computed only once in the first round.

Q4. At the same strict low-FPR operating point, how much recall improvement remains? Are 40 benign validation examples sufficient to set and reliably control that operating point?

We replaced the original fixed argmax decision with a continuous label-token log-odds score. The threshold is selected on a calibration split and fixed before testing.

Setting: schema-grounded; operating points

Calibration-side FPR targetStatic SFT test TPR (%)CoEvo-SQLi test TPR (%)Static SFT test FPR (%)CoEvo-SQLi test FPR (%)
0.50%79.389.70.100.15
1%88.495.61.161.55
2%92.498.03.22.86
3%93.898.58.0110.63

At every calibration-side target, CoEvo-SQLi has higher test recall. Because the calibration and test benign pools come from different held-out databases, the realized test FPR does not exactly equal the calibration target. At the 2% target, CoEvo-SQLi achieves both higher TPR and lower realized test FPR.

We also tested sensitivity to validation-set size.

Setting: schema-grounded; validation-set size

Validation sizeFPR (%)FPR excess over Static SFT (percentage points)TPR (%)MCC
40 examples per cluster + 40 benign3.21.997.592.5
80 examples per cluster + 80 benign3.01.797.091.5
160 examples per cluster + 160 benign2.61.397.292.2

After enlarging the validation set, the controller becomes less sensitive to the coarseness of the benign-error estimate, while the main detection gain remains. At a 1% attack prevalence, the 1% calibration target corresponds to approximately 7.2 additional attacks detected and 38.6 additional false blocks per 10,000 queries. We report this trade-off without claiming that one operating point is universally preferable across deployments.

(This response continues in Part 3 of 3, supplied as a separate file: Q6 on the remaining ambiguity, Q7 on the mutation audits, and the closing list of revisions.)

AuthorsOpenReview ↗

Response to Q5, Q6, Q7 and revisions (Part 3 of 3)

Q5. Does the adaptation claim concern new templates within the known 48 clusters, or also attack forms outside that taxonomy? Is there evidence independent of the current generation pipeline that distinguishes template generalization from broader adaptation?

Our adaptation claim is limited to new templates and new surface forms within the existing 48 attack clusters. Training uses 63 templates across four databases, while testing uses 180 held-out templates across five unseen databases. Thus, the test primarily examines generalization to new templates, carrier queries, schemas, and comment resources within the declared taxonomy.

We also evaluate on the external WAF-A-MoLE benchmark in the query-only setting. The attacks in this benchmark are generated by an external fuzzer rather than by our generation pipeline. CoEvo-SQLi reaches an MCC of 87.9 on this dataset, the highest result among the locally trained query-only models. However, new attack families outside the 48 clusters are not covered by the current controller, and cluster-level feedback cannot directly target them. We have narrowed the adaptation claim accordingly.

Q6. How much ambiguity remains in the 16% of examples judged ambiguous once experts receive the correct schema? How much of the detection benefit can be explained by correct versus incorrect or irrelevant-schema controls and simple structural checks?

Appendix H audits the 240 accepted mutations actually used during training. These samples cover three seeds and eight rounds, with ten samples drawn from each seed-by-round stratum. The original binary label was hidden during the audit.

Setting: training-time mutation audit

Audit criterionResult
Intended attack technique preserved227/240 (94.6%)
Full semantic validity226/240 (94.2%)
Schema-referencing examples consistent with the schema153/155 (98.7%)
Clear label judgement with SQL and schema238/240 (99.2%)
Structurally invalid or unparseable9/240 (3.8%)

The 99.2% judgeability result indicates that schema information improves the interpretability of these training-time audited examples. However, this audit is not a paired comparison with the query-only ambiguity audit on the same samples and therefore cannot quantitatively decompose the full schema gain.

Q7. What fraction of dynamic mutations receives independent validation of category consistency, attack semantics, or execution? Are there separate audits or results for ambiguous examples, invalid examples, and legitimate but complex or malformed queries?

The table above audits the dynamic mutations actually used for training. Two annotators with SQL-security backgrounds judged each sample independently, and a third annotator resolved disagreements; the semantic-validity assessment also included execution against the target MySQL database. These results indicate that the dynamic corpus is not dominated by syntactic or semantic label noise.

We added the nearest-neighbour baselines and their protocols in Appendix K; controller separation, corpus replay, instantaneous resampling, and frozen-controller experiments in Appendix G; the mutation audit in Appendix H; validation-size analysis in Appendix I; matched operating-point analysis in Appendix J; and the corresponding explanations in Sections 1, 5.3–5.5, 6.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.