acceptodds
Under review as a conference paper at ICLR 2027

MetaVul: Learning Transferable Agent Harnesses for Vulnerability Detection

10 views

Abstract

Large Language Model (LLM) agents are increasingly used to automate vulnerability detection, but reliable detection remains a challenging problem because agents must distinguish subtle security-relevant differences from benign code variation. While underlying LLM capability has improved significantly, agent accuracy also depends heavily on the harness: the code that defines the scaffolding around the LLM. Harnesses are typically fixed and largely hand-written, limiting adaptation to evaluator feedback. Additionally, harness improvements for one dataset or model may fail to transfer to others. To address these challenges, we introduce MetaVul, a self-improving framework for harness discovery in vulnerability detection. Starting from a seed structured reasoning scaffold, MetaVul uses a workflow-based optimization loop to iteratively refine the harness using feedback from paired vulnerable and patched functions, all while preserving the structured reasoning and verdict format. At each iteration, it searches over the full history of prior candidate harnesses, with access to their trajectories, logs, and metrics, and it returns a set of Pareto-optimal harnesses rather than a single best candidate. We apply MetaVul to discover harnesses for architectures including Codex CLI and Claude Code. A validation set (202 pairs) from TitanVul and CWE-Eval is used for optimization, while the unseen PrimeVul dataset (430 pairs) is kept as the held-out set. We later test whether harnesses learned from one model transfer to others not seen during optimization. Results on both frontier and small LLMs show that the optimized harnesses improve pairwise vulnerability detection up to 23.3 percentage points on the validation set (Opus 4.6) and up to 25.8 points on the held-out test set (DeepSeek V4 Flash). Harnesses learned with one model also transfer to others. We show that harnesses optimized with Opus 4.6 transfer to Opus 5, improving held-out pairwise accuracy by 14.7 points without the need for further optimization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vulnerability detection, llm agents, self-improving systems, meta optimization, harness discovery, cross-model transfer

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NCc7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MetaVul automatically refines vulnerability-detection harnesses using labeled vulnerable–patched function pairs. An optimizer revises a Python wrapper containing analysis instructions and verdict parsing. Candidates are evaluated on 202 development pairs, and a Pareto frontier is retained over pairwise accuracy and joint CWE/localization performance. Evaluation uses PrimeVul and several models not involved in optimization. The paper reports improved pairwise discrimination, alongside trade-offs in recall and localization.

Critical strengths
  1. The paper evaluates transfer across both datasets and models. This provides useful evidence that some optimized instructions remain effective beyond the optimization setting.

  2. The paired evaluation checks whether the detector distinguishes vulnerable functions from their fixes. Reporting recall, F1, and localization alongside pairwise accuracy makes the trade-offs visible.

Critical weaknesses
  1. The contribution over existing optimization methods is unclear. The demonstrated setup mainly optimizes prompts and output handling for single-function classification. There is no comparison with GEPA, Meta-Harness, or a simple iterative rewriting baseline under matched budgets. Appendix J also does not establish a benefit from the full feedback history. The paper offers limited insight into what the optimized instructions learn beyond making more conservative predictions.

  2. Some headline results use test performance to choose the reported candidate. Section 6 highlights iteration 12 because it has the highest PrimeVul accuracy. Table 13 instead marks iteration 11 as validation-selected. These candidates correctly classify 100/430 and 89/430 pairs, respectively. The gain over the seed remains, but the stronger comparison uses test-based selection. Reporting the best test result within a validation-selected frontier raises the same concern. A consistent selection rule that does not use test labels is needed.

Questions for authors
  1. Which candidate would be selected for each evaluation without seeing any PrimeVul results? Can you report those candidates consistently across RQ1 and RQ2?

  2. What distinguishes the optimization procedure from GEPA or Meta-Harness beyond the task-specific objectives? Are any matched-budget optimizer comparisons available?

  3. What concrete changes recur across successful harnesses? Could you give examples showing improved vulnerability reasoning, beyond a more conservative verdict policy?

12 comments

Dear Reviewer NCc7,

Thank you for your questions and suggestions. We are happy to answer any questions you may have below and integrate your feedback to revise and improve the manuscript.

Regarding Question 1 and Weakness 2:

Which candidate would be selected for each evaluation without seeing any PrimeVul results? Can you report those candidates consistently across RQ1 and RQ2?

Some headline results use test performance to choose the reported candidate. Section 6 highlights iteration 12 because it has the highest PrimeVul accuracy. Table 13 instead marks iteration 11 as validation-selected. These candidates correctly classify 100/430 and 89/430 pairs, respectively. The gain over the seed remains, but the stronger comparison uses test-based selection. Reporting the best test result within a validation-selected frontier raises the same concern. A consistent selection rule that does not use test labels is needed.

Please note, the Pareto frontier from the paper is always selected based on validation-data and never on test-data.

The reason only iter 12 was included in Table 2 was to illustrate an upper bound, however, we agree that this presentation may have given the impression of selecting harnesses based on the test results. In fact, the validation-selected harness (which corresponds to the iteration 11 harness, as you noted) also outperforms the other llm and agent baselines. Iteration 11 is selected on validation, and it correctly classifies 89 of the pairs, compared to 81 by the next-best approach.

This does not change any of the conclusions. We have updated the table to improve reporting presentation.

In summary, in RQ1, all reported harnesses come from the validation Pareto frontier, so none were chosen using PrimeVul. In RQ2, the validation-selected candidate is iteration 11, which we now report alongside iteration 12.

(This concludes author comments, Part 1. Author comments Part 2 to be continued below)

Dear Reviewer NCc7,

Below, we address Question 2.

What distinguishes the optimization procedure from GEPA or Meta-Harness beyond the task-specific objectives? Are any matched-budget optimizer comparisons available?

Yes. MetaVul significantly outperforms simpler rewriting on a matched-budget comparison.

We outline the details of this claim below:

To test the benefit of the full history optimization on a matched budget, we try 3 different settings:

  • full history, which is how MetaVul is configured by default. In this setting, the optimizer can see every past harness, all past metrics, and all labeled validation predictions with reasoning.
  • In the no history setting, only the latest harness and metrics are visible.
  • In the harness only setting, only the latest harness is visible. Everything else stays fixed; that includes the same seed harness, the same validation set, 7 iterations, 404 validation functions per candidate harness, and hence the same number of detector calls. We use Deepseek V4 Flash.
SettingVal seedVal pairwiseVal gainPrimeVul seedPrimeVul pairwisePrimeVul gain (95% CI)p vs seed
Full history (MetaVul)14.637.6+23.013.329.0+15.7 [+11.5, +19.9]< 0.001
No history15.820.8+5.013.317.2+4.0 [+0.3, +7.4]0.03
Harness only16.625.0+8.413.321.7+8.5 [+4.7, +12.2]< 0.001

Unlike the ablation in Appendix J, these use matched budget, repeated runs (we run twice each setting), and also include held out evaluation. Please note, the val seeds are different only because each setting starts from scratch.

As we see, the full history nearly doubles (over harness only) and quadruples (over no history) the held-out gains. The difference is statistically significant: every setting beats the seed, and the full history setting beats the simpler rewriting (full history vs. harness only: +7.2, p < 0.001; vs. no history: +11.7, p < 0.001). The full history and feedback mechanism are necessary.

Yes, our approach does outperform simpler rewriting on held-out data. And yes, we can show that the full history beats simpler rewriting.

(This concludes author comments, Part 2. Author comments Part 3 to be continued below)

The previous findings also relate to Weakness 1:

The contribution over existing optimization methods is unclear. The demonstrated setup mainly optimizes prompts and output handling for single-function classification. There is no comparison with GEPA, Meta-Harness, or a simple iterative rewriting baseline under matched budgets. Appendix J also does not establish a benefit from the full feedback history. The paper offers limited insight into what the optimized instructions learn beyond making more conservative predictions.

MetaVul outperforms GEPA under a matched budget experiment, as well.

To compare MetaVul against GEPA,

we run both methods from the same seed harness. We use the same detector (DeepSeek V4 Flash), same validation set, and the same number of detector calls (7 iters, 2,828 calls). We run each method twice. Each method picks the final harness according to its algorithmic rule, and only that harness is evaluated on the held-out (PrimeVul).

MethodVal seedVal pairwiseVal gainPrimeVul seedPrimeVul pairwisePrimeVul gain (95% CI)p vs seed
MetaVul14.637.6+23.013.329.0+15.7 [+11.5, +19.9]< 0.001
GEPA15.836.4+20.513.324.8+11.5 [+7.4, +15.6]< 0.001

MetaVul beats GEPA by 4.2 points on held-out. This difference is statistically significant.

A major difference is GEPA proposes each new candidate from a single LLM reflection call over a small minibatch (a few pairs) and uses that minibatch to filter candidates. In contrast, MetaVul's optimizer is an agent that has access to the full history, all labeled validation predictions, and every candidate it scored on the full validation set.

In summary, we show that the full history and feedback have critical value. We also compare it to GEPA, which is a different algorithm. The results support that (1) the full history provides signals that drive MetaVul's gains and (2) MetaVul outperforms GEPA under a matched budget.

We thank the reviewer for their very useful feedback. We have updated the details of the GEPA comparison and the ablation in the appendix, which is now conclusive.

(This concludes author comments, Part 3. Author comments Part 4 to be continued below)

Reviewer_ekbqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MetaVul optimizes vulnerability-detection harnesses while keeping language-model weights fixed. An optimizer rewrites the harness using development histories, scores and labeled predictions. A validation Pareto frontier balances vulnerable-versus-patched discrimination against a joint CWE and evidence-line hit score. Frozen harnesses are then evaluated on PrimeVul and models not used during discovery.

Critical strengths
  1. I find the paired criterion in Section 4.2 appropriate for this task. It requires both the vulnerable function and its patch to be classified correctly while the detector sees each function separately. This directly tests sensitivity to a security-relevant change and penalizes flagging both versions as vulnerable.

  2. Table 13 gives a useful transfer result even under validation selection: the selected GPT-4o harness reaches 20.70% pairwise accuracy, versus 16.51% for the seed. That makes automated harness adaptation worth studying. I also value the tradeoffs in Table 6, which show how rewriting can improve discrimination while weakening other detection capabilities.

Critical weaknesses
  1. I see limited methodological novelty: code-space search, full-history feedback and Pareto retention already appear in Meta-Harness, cited in Section 2. The vulnerability-specific objectives are worthwhile, but Table 15 leaves the advantage over simpler rewriting inconclusive. I would need a budget-matched held-out comparison before crediting the richer feedback with the gains.

  2. Section 6 chooses the transferred harness with the highest test accuracy, whereas Table 13 marks a different candidate as validation-selected. I cannot read these headline gains as the performance of a harness chosen before seeing test outcomes. The complete frontier is useful exploratory evidence, but the main comparisons need one consistent validation-only rule.

  3. The joint score in Section 4.2 only counts correctly detected vulnerable functions with annotations, so missed vulnerabilities disappear from its denominator. The recall decline in Table 2 is already reported. I would judge overall coverage using success across all eligible vulnerable functions. The conditional score alone leaves the practical security benefit unclear.

Questions for authors
  1. Under one fixed validation-only selection rule, what are the held-out and cross-model gains, with paired uncertainty against the seed?

  2. In one main configuration with matched search and detector inference budgets, does full-history optimization outperform simpler rewriting on held-out data?

  3. How do candidate rankings change when joint CWE and line hits are normalized over all eligible vulnerable functions, including missed cases?

9 comments

Dear Reviewer ekbq,

We thank you for your helpful comments. We are happy to address them and revise the manuscript accordingly.

With respect to Q1:

Question 1.

Under one fixed validation-only selection rule, what are the held-out and cross-model gains, with paired uncertainty against the seed?

We still see strong held-out and cross-model gains under one fixed validation-only selection rule.

We can use the joint metric as the validation-only, fixed selection rule. Under this rule, the selected harness beats the seed in all 8 of the held-out and cross-model evaluations. Every gain is significant.

RunEvaluationSeed pAccpAcc gain over seed (Paper's frontier, mean)pAcc gain over seed (Joint rule)p-value (Joint rule vs. seed)95% lower bound (Joint rule gain)
Opus 4.6held-out10.5+7.8+8.4<0.001+4.7
cross-transfer (Opus 5)22.6+12.1+9.5<0.001+6.0
Inkling-Smallheld-out8.8+9.5+6.70.001+3.3
cross-transfer (Nemotron)12.8+5.6+6.30.004+2.3
DeepSeek V4 Flashheld-out13.3+13.8+11.6<0.001+7.7
cross-transfer (GLM-4.7 Flash)15.8+2.3+7.90.003+3.5
GLM-5.3held-out18.4+5.6+5.60.009+2.1
cross-transfer (GPT-OSS-120B)14.4+4.7+4.70.034+0.7
Beats seed8/88/88/8 significant
Mean, held-out+9.2+8.1
Mean, cross-transfer+6.2+7.1

To summarize, we're showing the results based on a rule which meets the following conditions: (1) a single rule (2) selected using only validation data (3) selects a single harness (4) which still shows the gains on both held out and cross transfer, and (5) for which every gain is significant against the seed (all p < 0.05, with all lower bounds above zero).

We have revised the PDF to add this table, along with the significance test, to the Appendix.

Please note that, by default, the paper’s Pareto frontier is built from validation only. The rules are selected using validation by default, so it is not based on test performance, i.e., the rules by default are test-independent. But even if we were to pick Joint as the single rule, we would still see statistically significant gains, as shown above.

Weakness 2:

Section 6 chooses the transferred harness with the highest test accuracy, whereas Table 13 marks a different candidate as validation-selected. I cannot read these headline gains as the performance of a harness chosen before seeing test outcomes. The complete frontier is useful exploratory evidence, but the main comparisons need one consistent validation-only rule.

We agree that this presentation may have given the impression of selecting harnesses based on the test results.

However, although it was not included in the original Table 2, the validation-selected harness (iter 11) beats the other llm-based and agentic methods, as well. It detects 89 pairwise samples correctly, while the next-best method detects 81. So iteration 11 outperforms VulTrial, a multi-agent approach.

In the first version of the paper, we included only iteration 12 to illustrate an upper bound. We have revised the paper to include both iter 11 and 12 in the table now.

(This concludes author comments, Part 1. Author comments Part 2 to be continued below)

Regarding Weakness 1 and Question 2:

The vulnerability-specific objectives are worthwhile, but Table 15 leaves the advantage over simpler rewriting inconclusive. I would need a budget-matched held-out comparison before crediting the richer feedback with the gains.

In one main configuration with matched search and detector inference budgets, does full-history optimization outperform simpler rewriting on held-out data?

Yes, under a matched search and detector inference budgets, full history optimization outperforms simpler rewriting on held out data.

To test this under one main configuration:

We try 3 different settings:

  • full history, which is how MetaVul is configured by default. In this setting, the optimizer can see every past harness, all past metrics, and all labeled validation predictions with reasoning.
  • In the no history setting, only the latest harness and metrics are visible.
  • In the harness only setting, only the latest harness is visible.

Everything else stays fixed; that includes the same seed harness, the same validation set, 7 iterations, 404 validation functions per candidate harness, and hence the same number of detector calls. We use Deepseek V4 Flash.

SettingVal seedVal pairwiseVal gainPrimeVul seedPrimeVul pairwisePrimeVul gain (95% CI)p vs seed
Full history (MetaVul)14.637.6+23.013.329.0+15.7 [+11.5, +19.9]< 0.001
No history15.820.8+5.013.317.2+4.0 [+0.3, +7.4]0.03
Harness only16.625.0+8.413.321.7+8.5 [+4.7, +12.2]< 0.001

Unlike the ablation in Table 15, these use matched budget, repeated runs (we run twice each setting), and also include held out evaluation. Please note, the val seeds are different only because each setting starts from scratch.

As we see, the full history nearly doubles (over harness only) and quadruples (over no history) the held-out gains. The difference is statistically significant: every setting beats the seed, and the full history setting beats the simpler rewriting (full history vs. harness only: +7.2, p < 0.001; vs. no history: +11.7, p < 0.001). The full history and feedback mechanism are necessary.

Yes, our approach does outperform simpler rewriting on held-out data. And yes, we can show that the full history beats simpler rewriting.

In summary, we show that the full history and feedback have critical value. The results support that the full history optimization provides signals that drive MetaVul’s gains.

We thank the reviewer for this useful feedback. We have updated the details of the experiment and the ablation in Table 15, which is now conclusive.

(This concludes author comments, Part 2. Author comments Part 3 to be continued below)

Dear Reviewer ekbq,

We address the final question, relating also to Weakness 3, below:

Weakness 3:

The joint score in Section 4.2 only counts correctly detected vulnerable functions with annotations, so missed vulnerabilities disappear from its denominator. The recall decline in Table 2 is already reported. I would judge overall coverage using success across all eligible vulnerable functions. The conditional score alone leaves the practical security benefit unclear.

Question 3:

  1. How do candidate rankings change when joint CWE and line hits are normalized over all eligible vulnerable functions, including missed cases?

We update the joint metric to use all eligible vulnerable functions as the denominator, as defined in your Question 3.

We call this metric “Eligible Joint”.

As we illustrate below, computing the frontier over the eligible joint does not change our conclusions. Gains on the validation, held out, and transfer still hold.

The candidate rankings do not change for Claude + Opus, nor for Codex CLI + Inkling. The harness choice remains the same. The Pareto frontier shrinks to iterations 11 and 12 for Codex CLI + Deepseek V4 Flash, and the selected harness improves the held-out results (+12.1 vs +11.6 held-out). For GLM 5.3, the harness switches to candidate 4 (which had the same performance on validation as the original metric), but still this does not change the conclusions of the RQs. The gains still hold.

For each setting, we compare picking a single harness with Original Joint versus with Eligible Joint. With Original Joint, we build the validation Pareto frontier over pairwise accuracy and Original Joint, and select the frontier harness with the highest Original Joint. With Eligible Joint, we do the same over pairwise accuracy and Eligible Joint. Only the selected harness is evaluated on PrimeVul (that's what the "Selected" column refers to).

Agent + optimizerRuleFrontierSelectedVal pairwisePrimeVul (same model)PrimeVul (cross transfer)
Claude Code + Opus 4.6Seed010.410.5Opus 5: 22.6
Original Joint2, 3231.2 (+20.8)18.8 (+8.4)Opus 5: 32.1 (+9.5)
Eligible Joint2, 3231.2 (+20.8)18.8 (+8.4)Opus 5: 32.1 (+9.5)
Codex CLI + Inkling-SmallSeed06.48.8Nemotron 3.5 Lightning: 12.8
Original Joint6, 8, 12620.8 (+14.4)15.6 (+6.7)Nemotron 3.5 Lightning: 19.1 (+6.3)
Eligible Joint6, 8, 12620.8 (+14.4)15.6 (+6.7)Nemotron 3.5 Lightning: 19.1 (+6.3)
Codex CLI + DeepSeek V4 FlashSeed014.413.3GPT-4o: 16.5; GLM-4.7 Flash: 15.8
Original Joint1, 5, 9, 11, 14124.3 (+9.9)24.9 (+11.6)GPT-4o: 19.7 (+3.1); GLM-4.7 Flash: 23.7 (+7.9)
Eligible Joint11, 121234.6 (+20.2)25.4 (+12.1)GPT-4o: 23.3 (+6.8); GLM-4.7 Flash: 19.1 (+3.3)
Claude Code + GLM-5.3Seed019.818.4GPT-OSS-120B: 14.4
Original Joint6631.2 (+11.4)23.9 (+5.6)GPT-OSS-120B: 19.1 (+4.7)
Eligible Joint4431.2 (+11.4)21.2 (+2.8)GPT-OSS-120B: 15.3 (+0.9)

All values are pairwise accuracy (in %); the gains over the seed are in parentheses.

Thus, defining the metric by Eligible joint does not change the candidate harness choice for 2 settings, and even improves held out results on one setting. Overall, selecting harnesses with Eligible Joint offers about the same discrimination gain as the original, so the choice of metric does not change our conclusions.

With this, we conclude the author response.

We have shown: (1) there exists a single rule that selects a single harness that can show held out and transfer gains, (2) the full history provides signals that drive MetaVul’s gains, and (3) our conclusions still hold under the “eligible joint” rule.

We are extremely grateful to reviewer ekbq for their feedback, which has significantly helped to improve the paper.

We really look forward to your response and to engaging further with you. We hope our answers have addressed your questions and that you will consider them in your final evaluation.

Sincerely,

Authors of MetaVul

Reviewer_wYtdOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MetaVul applies feedback-driven optimization to the prompts and code scaffolding used by LLM vulnerability detectors. It refines a structured-reasoning seed on 202 vulnerable/patched function pairs and retains a validation Pareto frontier over pairwise accuracy and joint CWE identification and localization. Experiments on 430 held-out PrimeVul pairs and additional models show improvements over the seed, alongside trade-offs in recall and localization.

Critical strengths
  1. The paired evaluation targets the practically important ability to distinguish vulnerable functions from their fixes. Testing frozen harnesses on a separate dataset and on models not used during discovery provides useful evidence of transfer.

  2. The paper provides substantial supporting detail, including per-iteration results, overlap removal, agent isolation, harness audits, and code artifacts. It also clearly acknowledges the recall/localization trade-offs and the limits of its preliminary ablation.

Critical weaknesses
  1. The selection protocol for the headline transfer results is insufficiently clear. Section 5 reports the best validation-frontier member for each evaluation without specifying a test-independent rule for choosing among frontier members, while Table 2 uses GPT-4o iteration 12 selected by its accuracy on the transfer evaluation. Table 13 instead marks iteration 11 as validation-selected, with 89 rather than 100 correct pairs out of 430. Reporting a fixed validation-only choice separately from the best observed test result is necessary to assess the headline transfer gains.

  2. The benefit of the proposed search mechanism over simpler refinement is insufficiently established. There is no budget-matched comparison with relevant prompt/harness optimizers such as GEPA or Meta-Harness, and the preliminary workspace ablation in Appendix J is explicitly inconclusive. The evidence therefore supports the usefulness of optimized harnesses more strongly than the added value of the full history and feedback mechanism.

Questions for authors
  1. What test-independent rule selects a single harness from the validation frontier? Please report the main held-out and transfer results under that rule, and distinguish them from the best results observed across evaluated candidates.

  2. Can the authors compare MetaVul with a standard prompt/harness optimizer and simpler refinement using the same seed, backbone, search budget, and inference budget, with repeated discovery runs? This would help identify which components account for the gains.

  3. Since Joint is conditioned on true positives and recall changes across harnesses, how do the conclusions change when successful detection, CWE identification, and localization are jointly measured over all vulnerable functions with the required labels?

6 comments

Dear Reviewer wYtd,

Thank you very much for your feedback! We are happy to directly address your questions below:

Question 1:

Question 1. What test-independent rule selects a single harness from the validation frontier? Please report the main held-out and transfer results under that rule, and distinguish them from the best results observed across evaluated candidates.

Yes, there is a test-independent rule that still shows the gains on the held out and cross transfer.

By default, our rules are already test-independent (as illustrated in the Pareto selection algorithm). All results in Fig. 3 are based on the test-independent selection.

However, let’s say we want to fix a single rule that selects a single harness from the validation frontier as you suggested.

We can still pick a rule that would still see gains over both held-out and the cross-transfer.

For example, if we use the joint metric as the rule:

RunEvaluationSeed pAccpAcc gain over seed (Paper's frontier, mean)pAcc gain over seed (Joint rule)
Opus 4.6held-out10.5+7.8+8.4
cross-transfer (Opus 5)22.6+12.1+9.5
Inkling-Smallheld-out8.8+9.5+6.7
cross-transfer (Nemotron)12.8+5.6+6.3
DeepSeek V4 Flashheld-out13.3+13.8+11.6
cross-transfer (GLM-4.7 Flash)15.8+2.3+7.9
GLM-5.3held-out18.4+5.6+5.6
cross-transfer (GPT-OSS-120B)14.4+4.7+4.7
Beats seed8/88/8
Mean, held-out+9.2+8.1
Mean, cross-transfer+6.2+7.1

This is (1) a single rule (2) test-independent, (3) selects a single harness, and (4) still shows the gains on both (4.1) held out and (4.2) cross transfer.

Weakness 1:

Weakness 1. The selection protocol for the headline transfer results is insufficiently clear. Section 5 reports the best validation-frontier member for each evaluation without specifying a test-independent rule for choosing among frontier members, while Table 2 uses GPT-4o iteration 12 selected by its accuracy on the transfer evaluation. Table 13 instead marks iteration 11 as validation-selected, with 89 rather than 100 correct pairs out of 430. Reporting a fixed validation-only choice separately from the best observed test result is necessary to assess the headline transfer gains.

Even though our RQ2 Findings would still hold, we agree that the selection should be clearer. In Table 2, iteration 12 is meant to note that there could be an even better upper bound, but our test-independent selected harness (iteration 11, 89/430 correct pairs) still outperforms the other methods (the next best, a mulit-agent method VulTrial, gets only 81).

We added iteration 11 as a row in Table 2. Please note, the inclusion of iteration 11 does not change our RQ2 Conclusions whatsoever.

(This concludes author comments, Part 1. Author comments Part 2 to be continued below)

Regarding Weakness 2 and Question 2:

The benefit of the proposed search mechanism over simpler refinement is insufficiently established. There is no budget-matched comparison with relevant prompt/harness optimizers such as GEPA or Meta-Harness, and the preliminary workspace ablation in Appendix J is explicitly inconclusive. The evidence therefore supports the usefulness of optimized harnesses more strongly than the added value of the full history and feedback mechanism.

Can the authors compare MetaVul with a standard prompt/harness optimizer and simpler refinement using the same seed, backbone, search budget, and inference budget, with repeated discovery runs? This would help identify which components account for the gains.

Yes, under a matched seed, backbone, search/inference budget, our approach MetaVul does in fact outperform GEPA on held-out data.

MetaVul vs. GEPA.

We run both methods from the same seed harness. We use the same detector (DeepSeek V4 Flash), same validation set, and the same number of detector calls (7 iters, 2,828 calls). We run each method twice. Each method picks the final harness according to its algorithmic rule, and only that harness is evaluated on the held-out (PrimeVul).

MethodVal seedVal pairwiseVal gainPrimeVul seedPrimeVul pairwisePrimeVul gain (95% CI)p vs seed
MetaVul14.637.6+23.013.329.0+15.7 [+11.5, +19.9]< 0.001
GEPA15.836.4+20.513.324.8+11.5 [+7.4, +15.6]< 0.001

MetaVul beats GEPA by 4.2 points on held-out. This difference is statistically significant.

Yes, our approach does outperform simpler rewriting on held-out data. And yes, we can show that the full history has critical value.

We test 3 different settings: full history, which is how MetaVul is configured by default. In this setting, the optimizer can see every past harness, all past metrics, and all labeled validation predictions with reasoning. In the no history setting, only the latest harness and metrics are visible. In the harness only setting, only the latest harness is visible.

SettingVal seedVal pairwiseVal gainPrimeVul seedPrimeVul pairwisePrimeVul gain (95% CI)p vs seed
Full history (MetaVul)14.637.6+23.013.329.0+15.7 [+11.5, +19.9]< 0.001
No history15.820.8+5.013.317.2+4.0 [+0.3, +7.4]0.03
Harness only16.625.0+8.413.321.7+8.5 [+4.7, +12.2]< 0.001

Unlike the ablation in appendix J, these use matched budget, repeated runs (we run twice each setting), and also include held out evaluation. As we see, the full history nearly doubles (over harness only) and quadruples (over no history) the held-out gains. The difference is statistically significant. The full history and feedback mechanism are necessary.

In summary, we show that the full history and feedback have critical value. We also compare it to GEPA. The results support that (1) the full history provides signals that drive MetaVul’s gains and (2) MetaVul clearly outperforms GEPA under a matched budget.

We thank the reviewer for this useful feedback and we have updated the details of the experiment and the workspace ablation in Appendix J, which is now conclusive.

(This concludes author comments, Part 2. Author comments Part 3 to be continued below)

Dear Reviewer wYtd,

We address the final question below:

  1. Since Joint is conditioned on true positives and recall changes across harnesses, how do the conclusions change when successful detection, CWE identification, and localization are jointly measured over all vulnerable functions with the required labels?

We update the joint metric to use all eligible vulnerable functions as the denominator, as defined in your Question 3.

We call this metric “Eligible Joint”.

As we illustrate below, computing the frontier over the eligible joint does not change our conclusions. Gains on the validation, held out, and transfer still hold.

The candidate rankings do not change for Claude + Opus, nor for Codex CLI + Inkling. The harness choice remains the same. The Pareto frontier shrinks to iterations 11 and 12 for Codex CLI + Deepseek V4 Flash, and the selected harness improves the held-out results (+12.1 vs +11.6 held-out). For GLM 5.3, the harness switches to candidate 4 (which had the same performance on validation as the original metric), but still this does not change the conclusions of the RQs. The gains still hold.

For each setting, we compare picking a single harness with Original Joint versus with Eligible Joint. With Original Joint, we build the validation Pareto frontier over pairwise accuracy and Original Joint, and select the frontier harness with the highest Original Joint. With Eligible Joint, we do the same over pairwise accuracy and Eligible Joint. Only the selected harness is evaluated on PrimeVul (that's what the "Selected" column refers to).

Agent + optimizerRuleFrontierSelectedVal pairwisePrimeVul (same model)PrimeVul (cross transfer)
Claude Code + Opus 4.6Seed010.410.5Opus 5: 22.6
Original Joint2, 3231.2 (+20.8)18.8 (+8.4)Opus 5: 32.1 (+9.5)
Eligible Joint2, 3231.2 (+20.8)18.8 (+8.4)Opus 5: 32.1 (+9.5)
Codex CLI + Inkling-SmallSeed06.48.8Nemotron 3.5 Lightning: 12.8
Original Joint6, 8, 12620.8 (+14.4)15.6 (+6.7)Nemotron 3.5 Lightning: 19.1 (+6.3)
Eligible Joint6, 8, 12620.8 (+14.4)15.6 (+6.7)Nemotron 3.5 Lightning: 19.1 (+6.3)
Codex CLI + DeepSeek V4 FlashSeed014.413.3GPT-4o: 16.5; GLM-4.7 Flash: 15.8
Original Joint1, 5, 9, 11, 14124.3 (+9.9)24.9 (+11.6)GPT-4o: 19.7 (+3.1); GLM-4.7 Flash: 23.7 (+7.9)
Eligible Joint11, 121234.6 (+20.2)25.4 (+12.1)GPT-4o: 23.3 (+6.8); GLM-4.7 Flash: 19.1 (+3.3)
Claude Code + GLM-5.3Seed019.818.4GPT-OSS-120B: 14.4
Original Joint6631.2 (+11.4)23.9 (+5.6)GPT-OSS-120B: 19.1 (+4.7)
Eligible Joint4431.2 (+11.4)21.2 (+2.8)GPT-OSS-120B: 15.3 (+0.9)

All values are pairwise accuracy (in %); the gains over the seed are in parentheses.

Thus, defining the metric by Eligible joint does not change the candidate harness choice for 2 settings, and even improves held out results on one setting.

With this, we conclude the author response.

We have shown: (1) there exists a single rule that selects a single harness that can show held out and transfer gains, (2) the full history provides signals that drive MetaVul’s gains, (3) MetaVul outperforms GEPA under a matched budget, and (4) our conclusions hold under the eligible joint rule.

We are extremely grateful to reviewer wYtd for their feedback, which has substantially strengthened the paper.

We hope our responses address your concerns, and we would be very grateful if you could consider the answers to the questions in your final evaluation. We look forward to your response.

Sincerely,

Authors of MetaVul

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.