- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MetaVul automatically refines vulnerability-detection harnesses using labeled vulnerable–patched function pairs. An optimizer revises a Python wrapper containing analysis instructions and verdict parsing. Candidates are evaluated on 202 development pairs, and a Pareto frontier is retained over pairwise accuracy and joint CWE/localization performance. Evaluation uses PrimeVul and several models not involved in optimization. The paper reports improved pairwise discrimination, alongside trade-offs in recall and localization.
Critical strengths
-
The paper evaluates transfer across both datasets and models. This provides useful evidence that some optimized instructions remain effective beyond the optimization setting.
-
The paired evaluation checks whether the detector distinguishes vulnerable functions from their fixes. Reporting recall, F1, and localization alongside pairwise accuracy makes the trade-offs visible.
Critical weaknesses
-
The contribution over existing optimization methods is unclear. The demonstrated setup mainly optimizes prompts and output handling for single-function classification. There is no comparison with GEPA, Meta-Harness, or a simple iterative rewriting baseline under matched budgets. Appendix J also does not establish a benefit from the full feedback history. The paper offers limited insight into what the optimized instructions learn beyond making more conservative predictions.
-
Some headline results use test performance to choose the reported candidate. Section 6 highlights iteration 12 because it has the highest PrimeVul accuracy. Table 13 instead marks iteration 11 as validation-selected. These candidates correctly classify 100/430 and 89/430 pairs, respectively. The gain over the seed remains, but the stronger comparison uses test-based selection. Reporting the best test result within a validation-selected frontier raises the same concern. A consistent selection rule that does not use test labels is needed.
Questions for authors
-
Which candidate would be selected for each evaluation without seeing any PrimeVul results? Can you report those candidates consistently across RQ1 and RQ2?
-
What distinguishes the optimization procedure from GEPA or Meta-Harness beyond the task-specific objectives? Are any matched-budget optimizer comparisons available?
-
What concrete changes recur across successful harnesses? Could you give examples showing improved vulnerability reasoning, beyond a more conservative verdict policy?
12 comments
Dear Reviewer NCc7,
Below, we address Question 2.
What distinguishes the optimization procedure from GEPA or Meta-Harness beyond the task-specific objectives? Are any matched-budget optimizer comparisons available?
Yes. MetaVul significantly outperforms simpler rewriting on a matched-budget comparison.
We outline the details of this claim below:
To test the benefit of the full history optimization on a matched budget, we try 3 different settings:
- full history, which is how MetaVul is configured by default. In this setting, the optimizer can see every past harness, all past metrics, and all labeled validation predictions with reasoning.
- In the no history setting, only the latest harness and metrics are visible.
- In the harness only setting, only the latest harness is visible. Everything else stays fixed; that includes the same seed harness, the same validation set, 7 iterations, 404 validation functions per candidate harness, and hence the same number of detector calls. We use Deepseek V4 Flash.
| Setting | Val seed | Val pairwise | Val gain | PrimeVul seed | PrimeVul pairwise | PrimeVul gain (95% CI) | p vs seed |
|---|---|---|---|---|---|---|---|
| Full history (MetaVul) | 14.6 | 37.6 | +23.0 | 13.3 | 29.0 | +15.7 [+11.5, +19.9] | < 0.001 |
| No history | 15.8 | 20.8 | +5.0 | 13.3 | 17.2 | +4.0 [+0.3, +7.4] | 0.03 |
| Harness only | 16.6 | 25.0 | +8.4 | 13.3 | 21.7 | +8.5 [+4.7, +12.2] | < 0.001 |
Unlike the ablation in Appendix J, these use matched budget, repeated runs (we run twice each setting), and also include held out evaluation. Please note, the val seeds are different only because each setting starts from scratch.
As we see, the full history nearly doubles (over harness only) and quadruples (over no history) the held-out gains. The difference is statistically significant: every setting beats the seed, and the full history setting beats the simpler rewriting (full history vs. harness only: +7.2, p < 0.001; vs. no history: +11.7, p < 0.001). The full history and feedback mechanism are necessary.
Yes, our approach does outperform simpler rewriting on held-out data. And yes, we can show that the full history beats simpler rewriting.
(This concludes author comments, Part 2. Author comments Part 3 to be continued below)
The previous findings also relate to Weakness 1:
The contribution over existing optimization methods is unclear. The demonstrated setup mainly optimizes prompts and output handling for single-function classification. There is no comparison with GEPA, Meta-Harness, or a simple iterative rewriting baseline under matched budgets. Appendix J also does not establish a benefit from the full feedback history. The paper offers limited insight into what the optimized instructions learn beyond making more conservative predictions.
MetaVul outperforms GEPA under a matched budget experiment, as well.
To compare MetaVul against GEPA,
we run both methods from the same seed harness. We use the same detector (DeepSeek V4 Flash), same validation set, and the same number of detector calls (7 iters, 2,828 calls). We run each method twice. Each method picks the final harness according to its algorithmic rule, and only that harness is evaluated on the held-out (PrimeVul).
| Method | Val seed | Val pairwise | Val gain | PrimeVul seed | PrimeVul pairwise | PrimeVul gain (95% CI) | p vs seed |
|---|---|---|---|---|---|---|---|
| MetaVul | 14.6 | 37.6 | +23.0 | 13.3 | 29.0 | +15.7 [+11.5, +19.9] | < 0.001 |
| GEPA | 15.8 | 36.4 | +20.5 | 13.3 | 24.8 | +11.5 [+7.4, +15.6] | < 0.001 |
MetaVul beats GEPA by 4.2 points on held-out. This difference is statistically significant.
A major difference is GEPA proposes each new candidate from a single LLM reflection call over a small minibatch (a few pairs) and uses that minibatch to filter candidates. In contrast, MetaVul's optimizer is an agent that has access to the full history, all labeled validation predictions, and every candidate it scored on the full validation set.
In summary, we show that the full history and feedback have critical value. We also compare it to GEPA, which is a different algorithm. The results support that (1) the full history provides signals that drive MetaVul's gains and (2) MetaVul outperforms GEPA under a matched budget.
We thank the reviewer for their very useful feedback. We have updated the details of the GEPA comparison and the ablation in the appendix, which is now conclusive.
(This concludes author comments, Part 3. Author comments Part 4 to be continued below)
Dear Reviewer NCc7,
Thank you for your questions and suggestions. We are happy to answer any questions you may have below and integrate your feedback to revise and improve the manuscript.
Regarding Question 1 and Weakness 2:
Please note, the Pareto frontier from the paper is always selected based on validation-data and never on test-data.
The reason only iter 12 was included in Table 2 was to illustrate an upper bound, however, we agree that this presentation may have given the impression of selecting harnesses based on the test results. In fact, the validation-selected harness (which corresponds to the iteration 11 harness, as you noted) also outperforms the other llm and agent baselines. Iteration 11 is selected on validation, and it correctly classifies 89 of the pairs, compared to 81 by the next-best approach.
This does not change any of the conclusions. We have updated the table to improve reporting presentation.
In summary, in RQ1, all reported harnesses come from the validation Pareto frontier, so none were chosen using PrimeVul. In RQ2, the validation-selected candidate is iteration 11, which we now report alongside iteration 12.
(This concludes author comments, Part 1. Author comments Part 2 to be continued below)