- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces SecProbe, an adaptive framework to evaluate coding agents' ability to identify and patch code vulnerabilities. The key idea of SecProbe is to adaptively select or synthesize evaluation tasks in order to provide more informative assessments. The SecProbe adaptive evaluation is achieved through two feedback loops:
- The outer loop decides which task to evaluate next using an Item Response Theory (IRT) model, updated after each evaluation batch.
- The inner loop synthesizes new tasks through five specialized agents.
The authors construct 353 tasks across five programming languages and 149 CWE types. They evaluate nine frontier coding LLMs with two agent harnesses, showing substantial gaps in vulnerability detection and patching.
Critical strengths
- Novel adaptive evaluation framework that combines task selection with on-demand task synthesis.
- The IRT model and information gap metric are clearly defined and effectively integrated into the adaptive evaluation workflow.
- Significantly broader CWE coverage compared to existing cybersecurity benchmarks.
Critical weaknesses
- It is unclear whether adaptive test selection helps improve the overall quality of assessment beyond reducing the number of required tasks. In other words, the IRT-based ability estimation may not accurately reflect the coding agents' cybersecurity capabilities.
- Although the paper evaluates the quality and diversity of synthesized tasks, it could be hard to maintain consistent quality and diversity throughout the adaptive evaluation process.
- On-demand task synthesis does not necessarily eliminate the risk of training-data contamination, as LLM-generated tasks may resemble or reproduce examples seen during model training.
Questions for authors
- Beyond reducing the number of required tasks, how does adaptive evaluation help to provide more accurate or reliable assessments? Could the authors validate this using metrics other than the IRT-based information gap?
- When evaluating a new agent with substantially different capabilities, how reliable is the pre-calibrated task pool? For example, if the new agent's ability is substantially better than the range of the agents used for calibration, can SecProbe still provide accurate ability estimates?
- How reliable is the mapping between the IRT parameters and the task generation configuration ? It is unclear whether the generated tasks consistently achieve the target difficulty and discrimination predicted by the heuristic mappings.
- How sensitive is SecProbe's task synthesis to the choice of backbone LLM? Have the authors used different models, rather than relying solely on GPT-5.6 Luna?
8 comments
Response to Weakness 2 and Question 2
W2 / Q2. Quality as the task pool grows and reliability for new agents
-
We understand the reviewer’s two concerns about whether SecProbe remains effective as the evaluation grows (later rounds may produce lower-quality or more repetitive tasks), and a pool calibrated on existing agents may provide too little information about a stronger one. We addressed these concerns by extended experimental runs, from where we see that: 1) expert ratings and code diversity remain stable through 500 tasks; and 2) the fixed pool also distinguishes a previously unseen model at the top of the observed performance range, and the expanded agent ranking remains closely aligned with two established cybersecurity benchmarks.
-
1) Maintaining quality and diversity through 500 tasks. We extend the original run from 353 to 500 accepted tasks to test whether later rounds maintain the quality and variety of earlier ones. Three security experts independently review all 500 tasks using the criteria in Table 5 of the original manuscript. Mean overall quality is 4.43, 4.41, and 4.39 out of 5 for the initial 150 tasks, the next 203, and the final 147, respectively. All 500 tasks receive majority acceptance. This shows that the aggregate quality score is sustained in the later batches as well.
We also check whether the later batches repeat earlier implementations. After excluding shared scaffolding, the median similarity to each task’s closest earlier repository in the same language is 0.18, 0.20, and 0.19 across the three stages. The final 147 tasks add five CWE types and still span all six programming languages and three deployment levels. Together, these results show that the extension broadens coverage without a corresponding increase in code similarity or a marked decline in expert ratings. -
2) Transferring the fixed pool to a stronger agent. We accounted for this concern when designing the original evaluation. We selected calibration backbones that span a range of repair performance and evaluated each with two harnesses, so the item parameters are informed by responses from agents at different ability levels. To test whether that calibration remains useful for a new model near the upper end, we evaluate Gemini 4 Carbon on all 353 tasks with the original item parameters held fixed.
Gemini ranks first by repair pass rate under both harnesses, completing 101/353 repairs (28.61%) with Mini-SWE-Agent and 104/353 (29.46%) with Terminus-2, compared with the previous best rates of 26.91% and 28.33%, respectively. With both Gemini configurations added to the held-out comparison at 280 tasks, SecProbe’s IRT ranking reaches Spearman correlations of 0.90 with CyberGym and 0.95 with CyberGym E2E. The fixed pool thus distinguishes a previously unseen model at the top of the observed performance range, and the expanded ranking remains closely aligned with both reference benchmarks. If a future agent solves nearly all calibrated tasks, those tasks provide less information about its ability. SecProbe can then synthesize harder tasks and use the new responses to refine the estimate.
Appendix A.5 (Held-out-Agent Adaptive Evaluation) of the revised manuscript adds the Gemini transfer experiment.
-
Response to Weakness 3
W3. Training-data contamination
-
Regarding the concern that on-demand synthesis may still produce tasks that resemble examples seen during model training, we accounted for it in SecProbe’s design by grounding task generation in retrieved documentation and a specific software context, and by allowing evaluators to create fresh, private task pools.
During generation, the research agent retrieves public documentation about the target weakness. The design agent uses it to specify requirements for a particular software context, and the implementation and review agents build and check the resulting task. Each specialist operates in a separate context window with its own working memory. Agents pass designated intermediate artifacts between stages without sharing their full conversation histories. By grounding the task in retrieved material and decisions made across these stages, SecProbe relies less on any single agent recalling a complete example from training. Generated tasks can still overlap with examples seen during training.
We clarify in the Introduction of the revised manuscript that the main benefit of on-demand synthesis is reducing repeated exposure to fixed benchmark instances. Evaluators can generate a fresh pool after the model under evaluation is fixed, keep the tasks and reference repairs private, and refresh the pool for subsequent evaluations. Keeping those specific instances private reduces the chance that they enter future training corpora.
Response to Weakness 1 and Question 1
Thank you for the thoughtful feedback and constructive review.
W1 / Q1. Assessment quality beyond task efficiency
We understand the reviewer’s concern that reducing the information gap or the number of tasks only partially validates SecProbe's advantage. We address this concern by clarifying SecProbe's effectiveness via three aspects: 1. at a matched task budget, adaptive selection of SecProbe estimates abilities more accurately within the full pool, 2. adaptive synthesis of SecProbe produces pools that better recover rankings on well-established cybersecurity benchmarks, and 3. reveal more confirmed weaknesses. More details of each aspect are provided below:
1. More accurate ability estimates at the same budget. Appendix A.5 of the original manuscript tests whether adaptive selection can recover the ability estimates from all 353 tasks more accurately when every policy evaluates the same number of tasks. We measure error relative to the full-pool estimates for the same held-out agents. At 100 tasks, adaptive selection achieves an RMSE of 0.180, compared with 0.240 for random selection and 0.260 for fixed-order selection. One unit on the IRT ability scale corresponds to one standard deviation of the ability prior, so these errors are 18%, 24%, and 26% of that reference scale. With the same 100 tasks, adaptive selection reduces estimation error by 25.0% relative to random selection and 30.8% relative to fixed-order selection.
To test whether this advantage depends on which tasks the assessment starts with, we pair the policies across 20 shared initialization seeds and compare their RMSE at the 100-task checkpoint. Both reductions are statistically significant in two-sided paired permutation tests (after Holm correction) (adjusted p=0.0037 versus random selection and p=0.0015 versus fixed-order selection). This provides evidence that the lower estimation error persists across variations in the initial task set.
2. Better agreement with established cybersecurity benchmarks. To test whether the ability estimates reflect security performance beyond SecProbe’s own tasks, we compare the same held-out model and harness configurations on CyberGym Level 1 and CyberGym E2E. Both benchmarks use real software vulnerabilities and verify outcomes through execution. We use Spearman correlation because it measures whether the assessments agree on the relative strengths of the agents across different tasks and scoring scales. At 280 accepted tasks, adaptive synthesis achieves correlations of 0.89 with CyberGym and 0.94 with CyberGym E2E, compared with 0.71 and 0.66 for random synthesis, 0.77 and 0.71 for one-shot synthesis, and 0.83 and 0.77 for balanced difficulty synthesis. The stronger agreement on both benchmarks shows that, at the same budget, the adaptively generated pool better reflects the relative strengths observed in established cybersecurity evaluations.
3. More confirmed agents' weaknesses at the same budget. An overall ability ranking tells us which agents perform better, while weakness discovery identifies where individual agents struggle. We count one weakness per agent and security category when low repair success is confirmed on a separate set of tasks. At 280 accepted tasks, adaptive synthesis identifies 26 confirmed weaknesses, compared with 13 for random synthesis, 15 for one-shot synthesis, and 16 for balanced difficulty synthesis. That is 10 more confirmed weaknesses than the strongest baseline at the same budget. The additional findings show that adaptive synthesis improves the diagnostic value of the assessment alongside the quality of its overall ranking.
Section 5 (Discussion) of the revised manuscript reports the ranking and weakness discovery comparisons for adaptive, random, and one-shot synthesis.