acceptodds
Under review as a conference paper at ICLR 2027

SecProbe: Adaptive Evaluation of Coding Agents on Cybersecurity Vulnerabilities

Abstract

Assessing cybersecurity vulnerability awareness in coding agents requires evaluations that reveal capability gaps and remain informative as models evolve. Static benchmarks offer fixed coverage and difficulty, while scarce vulnerable repositories and costly expert authoring limit their renewal at scale. We introduce SecProbe, a framework for adaptive evaluation that combines Item Response Theory (IRT) with on-demand synthesis of repository-scale vulnerability-repair tasks. From observed performance, SecProbe estimates agent ability and identifies where additional evidence is most informative, selecting existing tasks or synthesizing new ones accordingly. As one use case, we construct 353 tasks spanning five programming languages and 149 CWE types and evaluate nine frontier models with two agent harnesses. Success rates peak at 28.33%, highlighting substantial gaps in vulnerability recognition and repair. Compared with random and one-shot baselines, SecProbe achieves comparable agent ability estimates while requiring agents to solve up to 29.5% fewer tasks. These results support adaptive evaluation as an efficient and discriminative approach to assessing cybersecurity vulnerability awareness.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Agent, Cybersecurity, Synthetic Data

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_X6yQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SecProbe, an adaptive framework to evaluate coding agents' ability to identify and patch code vulnerabilities. The key idea of SecProbe is to adaptively select or synthesize evaluation tasks in order to provide more informative assessments. The SecProbe adaptive evaluation is achieved through two feedback loops:

  • The outer loop decides which task to evaluate next using an Item Response Theory (IRT) model, updated after each evaluation batch.
  • The inner loop synthesizes new tasks through five specialized agents.

The authors construct 353 tasks across five programming languages and 149 CWE types. They evaluate nine frontier coding LLMs with two agent harnesses, showing substantial gaps in vulnerability detection and patching.

Critical strengths
  1. Novel adaptive evaluation framework that combines task selection with on-demand task synthesis.
  2. The IRT model and information gap metric are clearly defined and effectively integrated into the adaptive evaluation workflow.
  3. Significantly broader CWE coverage compared to existing cybersecurity benchmarks.
Critical weaknesses
  1. It is unclear whether adaptive test selection helps improve the overall quality of assessment beyond reducing the number of required tasks. In other words, the IRT-based ability estimation may not accurately reflect the coding agents' cybersecurity capabilities.
  2. Although the paper evaluates the quality and diversity of synthesized tasks, it could be hard to maintain consistent quality and diversity throughout the adaptive evaluation process.
  3. On-demand task synthesis does not necessarily eliminate the risk of training-data contamination, as LLM-generated tasks may resemble or reproduce examples seen during model training.
Questions for authors
  1. Beyond reducing the number of required tasks, how does adaptive evaluation help to provide more accurate or reliable assessments? Could the authors validate this using metrics other than the IRT-based information gap?
  2. When evaluating a new agent with substantially different capabilities, how reliable is the pre-calibrated task pool? For example, if the new agent's ability is substantially better than the range of the agents used for calibration, can SecProbe still provide accurate ability estimates?
  3. How reliable is the mapping between the IRT parameters (α,β)(\alpha, \beta) and the task generation configuration (c,s,h)(c, s, h)? It is unclear whether the generated tasks consistently achieve the target difficulty and discrimination predicted by the heuristic mappings.
  4. How sensitive is SecProbe's task synthesis to the choice of backbone LLM? Have the authors used different models, rather than relying solely on GPT-5.6 Luna?
8 comments
AuthorsOpenReview ↗

Response to Weakness 1 and Question 1

Thank you for the thoughtful feedback and constructive review.

W1 / Q1. Assessment quality beyond task efficiency

  • We understand the reviewer’s concern that reducing the information gap or the number of tasks only partially validates SecProbe's advantage. We address this concern by clarifying SecProbe's effectiveness via three aspects: 1. at a matched task budget, adaptive selection of SecProbe estimates abilities more accurately within the full pool, 2. adaptive synthesis of SecProbe produces pools that better recover rankings on well-established cybersecurity benchmarks, and 3. reveal more confirmed weaknesses. More details of each aspect are provided below:

    • 1. More accurate ability estimates at the same budget. Appendix A.5 of the original manuscript tests whether adaptive selection can recover the ability estimates from all 353 tasks more accurately when every policy evaluates the same number of tasks. We measure error relative to the full-pool estimates for the same held-out agents. At 100 tasks, adaptive selection achieves an RMSE of 0.180, compared with 0.240 for random selection and 0.260 for fixed-order selection. One unit on the IRT ability scale corresponds to one standard deviation of the ability prior, so these errors are 18%, 24%, and 26% of that reference scale. With the same 100 tasks, adaptive selection reduces estimation error by 25.0% relative to random selection and 30.8% relative to fixed-order selection.  
      To test whether this advantage depends on which tasks the assessment starts with, we pair the policies across 20 shared initialization seeds and compare their RMSE at the 100-task checkpoint. Both reductions are statistically significant in two-sided paired permutation tests (after Holm correction) (adjusted p=0.0037p = 0.0037 versus random selection and p=0.0015p = 0.0015 versus fixed-order selection). This provides evidence that the lower estimation error persists across variations in the initial task set.

    • 2. Better agreement with established cybersecurity benchmarks. To test whether the ability estimates reflect security performance beyond SecProbe’s own tasks, we compare the same held-out model and harness configurations on CyberGym Level 1 and CyberGym E2E. Both benchmarks use real software vulnerabilities and verify outcomes through execution. We use Spearman correlation because it measures whether the assessments agree on the relative strengths of the agents across different tasks and scoring scales. At 280 accepted tasks, adaptive synthesis achieves correlations of 0.89 with CyberGym and 0.94 with CyberGym E2E, compared with 0.71 and 0.66 for random synthesis, 0.77 and 0.71 for one-shot synthesis, and 0.83 and 0.77 for balanced difficulty synthesis. The stronger agreement on both benchmarks shows that, at the same budget, the adaptively generated pool better reflects the relative strengths observed in established cybersecurity evaluations.

    • 3. More confirmed agents' weaknesses at the same budget. An overall ability ranking tells us which agents perform better, while weakness discovery identifies where individual agents struggle. We count one weakness per agent and security category when low repair success is confirmed on a separate set of tasks. At 280 accepted tasks, adaptive synthesis identifies 26 confirmed weaknesses, compared with 13 for random synthesis, 15 for one-shot synthesis, and 16 for balanced difficulty synthesis. That is 10 more confirmed weaknesses than the strongest baseline at the same budget. The additional findings show that adaptive synthesis improves the diagnostic value of the assessment alongside the quality of its overall ranking.

    Section 5 (Discussion) of the revised manuscript reports the ranking and weakness discovery comparisons for adaptive, random, and one-shot synthesis.

AuthorsOpenReview ↗

Response to Weakness 2 and Question 2

W2 / Q2. Quality as the task pool grows and reliability for new agents

  • We understand the reviewer’s two concerns about whether SecProbe remains effective as the evaluation grows (later rounds may produce lower-quality or more repetitive tasks), and a pool calibrated on existing agents may provide too little information about a stronger one. We addressed these concerns by extended experimental runs, from where we see that: 1) expert ratings and code diversity remain stable through 500 tasks; and 2) the fixed pool also distinguishes a previously unseen model at the top of the observed performance range, and the expanded agent ranking remains closely aligned with two established cybersecurity benchmarks.

    • 1) Maintaining quality and diversity through 500 tasks. We extend the original run from 353 to 500 accepted tasks to test whether later rounds maintain the quality and variety of earlier ones. Three security experts independently review all 500 tasks using the criteria in Table 5 of the original manuscript. Mean overall quality is 4.43, 4.41, and 4.39 out of 5 for the initial 150 tasks, the next 203, and the final 147, respectively. All 500 tasks receive majority acceptance. This shows that the aggregate quality score is sustained in the later batches as well.  
      We also check whether the later batches repeat earlier implementations. After excluding shared scaffolding, the median similarity to each task’s closest earlier repository in the same language is 0.18, 0.20, and 0.19 across the three stages. The final 147 tasks add five CWE types and still span all six programming languages and three deployment levels. Together, these results show that the extension broadens coverage without a corresponding increase in code similarity or a marked decline in expert ratings.

    • 2) Transferring the fixed pool to a stronger agent. We accounted for this concern when designing the original evaluation. We selected calibration backbones that span a range of repair performance and evaluated each with two harnesses, so the item parameters are informed by responses from agents at different ability levels. To test whether that calibration remains useful for a new model near the upper end, we evaluate Gemini 4 Carbon on all 353 tasks with the original item parameters held fixed.  
      Gemini ranks first by repair pass rate under both harnesses, completing 101/353 repairs (28.61%) with Mini-SWE-Agent and 104/353 (29.46%) with Terminus-2, compared with the previous best rates of 26.91% and 28.33%, respectively. With both Gemini configurations added to the held-out comparison at 280 tasks, SecProbe’s IRT ranking reaches Spearman correlations of 0.90 with CyberGym and 0.95 with CyberGym E2E. The fixed pool thus distinguishes a previously unseen model at the top of the observed performance range, and the expanded ranking remains closely aligned with both reference benchmarks. If a future agent solves nearly all calibrated tasks, those tasks provide less information about its ability. SecProbe can then synthesize harder tasks and use the new responses to refine the estimate.

    Appendix A.5 (Held-out-Agent Adaptive Evaluation) of the revised manuscript adds the Gemini transfer experiment.

AuthorsOpenReview ↗

Response to Weakness 3

W3. Training-data contamination

  • Regarding the concern that on-demand synthesis may still produce tasks that resemble examples seen during model training, we accounted for it in SecProbe’s design by grounding task generation in retrieved documentation and a specific software context, and by allowing evaluators to create fresh, private task pools.

    During generation, the research agent retrieves public documentation about the target weakness. The design agent uses it to specify requirements for a particular software context, and the implementation and review agents build and check the resulting task. Each specialist operates in a separate context window with its own working memory. Agents pass designated intermediate artifacts between stages without sharing their full conversation histories. By grounding the task in retrieved material and decisions made across these stages, SecProbe relies less on any single agent recalling a complete example from training. Generated tasks can still overlap with examples seen during training.

    We clarify in the Introduction of the revised manuscript that the main benefit of on-demand synthesis is reducing repeated exposure to fixed benchmark instances. Evaluators can generate a fresh pool after the model under evaluation is fixed, keep the tasks and reference repairs private, and refresh the pool for subsequent evaluations. Keeping those specific instances private reduces the chance that they enter future training corpora.

Reviewer_B9yvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SecProbe combines Bayesian two-parameter IRT with feedback-guided synthesis of repository-level vulnerability-repair tasks. The authors construct 353 tasks spanning five languages and 149 CWE types and evaluate nine models with two agent harnesses. Experiments examine task quality, difficulty control, and adaptive evaluation, reporting up to 29.5% fewer accepted tasks to reach an information-gap threshold and improved ability-estimation efficiency on held-out agents.

Critical strengths
  1. The framework addresses a relevant limitation of static security benchmarks by connecting assessment needs to executable task generation. Broad CWE coverage and deterministic validation of functionality, reference repairs, and security mutations make the resulting benchmark a useful contribution.

  2. The evaluation includes matched difficulty-control experiments, IRT sensitivity analysis, and held-out-agent assessment. In Appendix A.5, adaptive selection reaches the target ability RMSE with 140 tasks versus 181 and 194 for the baselines, supporting its practical evaluation value.

Critical weaknesses
  1. The added value of adaptive synthesis is insufficiently isolated. Figure 5 measures the policy's own information-gap objective, while Appendix A.5 evaluates selection within a fixed pool. Neither separates the benefit of feedback-guided synthesis from adaptive selection alone.

  2. Repair-success labels require stronger validation. Appendix A.9 identifies 8 of 100 test-passing patches as incomplete, but the reported scores and IRT responses remain test-based. Expert review is triggered by disagreement between two calls to the same model, leaving errors shared by both judges unquantified.

Questions for authors
  1. Please specify the mappings and synthesis-trigger rule in Section 3.5. What evidence distinguishes the gain from adaptive synthesis from that of fixed-pool adaptive selection at comparable estimation accuracy and total cost?

  2. Which model-harness configurations supply the patches in Table 7, and how were tasks and experts selected for Table 5? Do the reported rankings and efficiency conclusions remain stable after independently auditing a sample of patches, including cases in which both judges agree, and correcting confirmed grading errors?

7 comments
AuthorsOpenReview ↗

Response to Weakness 1 and Question 1

Thank you for the detailed feedback on constructive review.

W1 / Q1(b). What adaptive synthesis adds beyond selection efficiency

  • The information-gap results in Figure 5 and the fixed-pool selection results in Appendix A.5 of the original manuscript only partially explain the contribution of adaptive synthesis. We address three points below: 1) why adaptive selection alone cannot close gaps in CWE coverage, 2) how feedback during synthesis helps recover rankings on established cybersecurity benchmarks with fewer tasks, and 3) how it helps identify more confirmed agent weaknesses at the same task budget.

    • 1) Why adaptive selection alone is insufficient. Selection and synthesis address different limits. Appendix A.5 of the original manuscript shows that selection can recover full-pool ability estimates with fewer tasks when suitable tasks already exist. Table 1 of the original manuscript shows why this condition matters: existing cybersecurity benchmarks leave many CWE types uncovered. Selection can prioritize the most informative tasks available, but it cannot assess a weakness type absent from the bank. Synthesis expands what the assessment can cover, and feedback guides which tasks to add.  
      To test whether feedback improves the pool itself, we compare four synthesis policies. At each matched budget, the same held-out agents attempt every task in each pool, so differences in the results reflect the tasks each policy generates. We evaluate the pools by how well they recover agent rankings on established cybersecurity benchmarks and how many confirmed weaknesses they reveal on evaluated agents.

    • 2) Recovering rankings with fewer tasks. We use CyberGym Level 1 and CyberGym E2E as references because they draw on real software vulnerabilities and verify outcomes through execution. SecProbe estimates IRT ability, while the reference benchmarks rank the same model and harness configurations by success rate. Spearman correlation measures agreement in their relative ordering across different tasks and scoring scales. Let uiu_i and viv_i be configuration ii’s centered ranks by SecProbe ability and reference benchmark success rate, respectively, with average ranks assigned to ties. We compute ρS=∑iuivi(∑iui2)(∑ivi2)\rho_S = \frac{\sum_i u_i v_i}{\sqrt{(\sum_i u_i^2)(\sum_i v_i^2)}} separately for each benchmark. A value close to 1 indicates that the two assessments generally rank agents in the same order.  
      At 280 accepted tasks, adaptive synthesis achieves Spearman correlations of 0.89 with CyberGym and 0.94 with CyberGym E2E. Under a two-sided exact permutation test that assumes no ties and treats the six configurations as exchangeable, these correlations would yield nominal p-values of 0.0333 and 0.0167, respectively. The corresponding correlations are 0.71 and 0.66 for random synthesis, 0.77 and 0.71 for one-shot synthesis, and 0.83 and 0.77 for balanced difficulty synthesis. Adaptive synthesis first reaches both correlation targets at 260 tasks, compared with 340, 320, and 310 for the three baselines. This is 50 fewer tasks than the strongest baseline. The one-shot comparison shows the value of updating targets as new responses arrive, and the balanced difficulty comparison shows that an even mix of Easy, Medium, and Hard tasks does not account for the gain.

    • 3) Finding more agents' weaknesses with fewer tasks. An overall ranking tells us which agents perform better, while weakness discovery tells us where they struggle. We count one weakness per agent and security category when low repair success is confirmed on a separate set of tasks. At 280 accepted tasks, adaptive synthesis identifies 26 confirmed weaknesses, compared with 13 for random synthesis, 15 for one-shot synthesis, and 16 for balanced difficulty synthesis. That is 10 more confirmed weaknesses than the strongest baseline at the same budget. Adaptive synthesis also reaches 24 confirmed weaknesses at 250 tasks, compared with 350, 335, and 320 for the three baselines, saving 70 tasks relative to the strongest baseline. These comparisons show what feedback during synthesis adds.

    Section 5 (Discussion) of the revised manuscript reports the ranking and weakness discovery comparisons for adaptive, random, and one-shot synthesis. Appendix A.6 of the revised manuscript adds the detailed comparison of CWE coverage across benchmarks.

AuthorsOpenReview ↗

Response to Weakness 2

W2. Validation of repair-success labels

  • stronger validation of repair-success labels. In the new “Full human audit of submitted repairs” paragraph in Appendix A.9 of the revised manuscript (pp. 25–26), we extend the review from one submission per task to every coding-agent patch submitted across the 6,354 evaluation attempts on 353 tasks and 18 model and harness configurations. Human reviewers inspect each patch in its repository context to determine whether it fixes the root cause, covers the affected code paths, and preserves required functionality. The audit covers all submissions, including cases in which the two model judges agree. We use the human judgments to correct confirmed grading errors and recompute the pass rates over the same 353 attempts per configuration.

    Human reviewers judge 1,236 of the 1,246 executable passes to be complete repairs (99.20%) and confirm the labels of all 5,108 executable failures. Figure 4 and its caption in the revised manuscript (p. 7) report the pass rates after the ten corrections. The table below compares the original and audited rates.

ModelMini-SWE-Agent OriginalMini-SWE-Agent AuditedTerminus-2 OriginalTerminus-2 Audited
GLM-5.326.91%26.91%28.33%28.33%
GPT-5.6 Sol25.21%25.21%26.06%26.06%
DeepSeek V4 Pro23.80%23.80%24.93%24.93%
Kimi K322.10%21.81%22.10%21.81%
Opus 4.820.68%20.40%22.66%22.38%
MiniMax-M318.70%18.41%18.70%18.41%
Qwen3.7-Flash17.00%16.71%16.15%15.86%
GPT-5.4-mini11.05%10.76%10.76%10.48%
Mistral Small 49.35%9.35%8.50%8.50%

The largest adjustment for any configuration is one outcome out of 353, or approximately 0.28 percentage points. Human adjudication preserves the model ordering within both harnesses, and the strongest configuration still completes 28.33% of repairs. The high agreement with human review supports the grading quality of SecProbe’s generated tasks and their use in comparing coding-agent repair performance.

AuthorsOpenReview ↗

Response to Question 1 (continued)

Q1. Synthesis rule and comparison with fixed-pool selection

  • (a) the mappings and synthesis trigger. We clarify the decision rule in Section 3.5 of the revised manuscript (pp. 5–6) and add the implementation details in Appendix A.2.1 of the revised manuscript (p. 18, “Configuration mappings and synthesis trigger”). At each round, we set the target difficulty to the ability level with the largest information gap. We set the target discrimination to the 75th percentile among calibrated tasks whose difficulty is within 0.5 IRT units of that target.

    We use two ridge regressions to map task configurations to IRT parameters. The predictors are CWE family, programming language, software deployment level, and difficulty controls. The regressions are fitted to the posterior means of item difficulty and log discrimination on the initial 150 calibrated tasks and refitted on all calibrated tasks after each batch.

    SecProbe first considers unused calibrated tasks. A task is eligible if its difficulty is within 0.5 IRT units of the target and its discrimination is at least 80% of the target. We rank eligible tasks by expected Fisher information under the current ability posterior. If they are insufficient to fill the next batch, we select the available eligible tasks and synthesize the shortfall, as specified in Appendix A.2.1 of the revised manuscript.

Reviewer_sw23OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Paper Summary SecProbe evaluates coding agents on finding and repairing vulnerabilities in repository-scale tasks. A Bayesian 2PL IRT model estimates agent ability and task difficulty, and an information-gap score picks where to probe next. When no suitable task exists, a five-agent pipeline synthesizes one with hidden security tests and a reference patch. The use case builds 353 tasks across 149 CWE types and evaluates nine models with two harnesses. The best pass rate is 28.33%, and adaptive synthesis reaches the information-gap target with 244 tasks versus 328 and 346 for the baselines.

Critical strengths
  • I think the task construction is the most useful contribution. Executable security tasks are scarce, and LLM-generated ones can leak the fix or ship broken tests. Here the test-writing stage cannot read the reference, a deterministic validator decides admission, and every declared mutant must be detected. It costs about $0.64 per task in model cost, and difficulty control seems to work (pass rate falls from 44.0 to 13.3 from Easy to Hard, Table 2). This looks like a practical way to grow a security benchmark.
  • The failure analysis is also interesting. Incomplete repair coverage is the most common failure (37.9% of failed attempts, Table 6), and Figures 9 to 11 show agents fixing one layer while leaving another requirement open.
Critical weaknesses

The efficiency claim seems circular. The policy generates tasks where G_t is largest in order to shrink it, and Figure 5 then compares policies by the maximum of G_t, so it is not surprising that adaptive synthesis wins. The random and one-shot baselines are never defined, and the 0.30 target is not the 0.15 stopping threshold in Appendix A.2.1, so the reported reductions depend on the chosen cut. Ability error is only measured for selection from a fixed pool (Appendix A.5), not for synthesis, which is the main claim.

  • It is not clear to me that a pass measures vulnerability awareness. The only correct patch the hidden tests must accept is the generator's own reference, so a correct alternative fix might still fail. The only evidence here is an expert rating of grader alignment (Table 5), not running known-correct alternative patches. The task panels in Figures 6 and 9 to 11 also state the security requirement, so identification may be partly given to the agent.

Minor:

  • Truong et al. (2025) is cited only as IRT benchmarking, but it already trains a difficulty-conditioned question generator for adaptive testing. Related work should also cover Bejar et al. (2003), Glas and van der Linden (2003), AutoBencher (Li et al., 2024) and DyVal (Zhu et al., 2024).
  • The IRT model is fit on twelve configurations from six backbones, which seems thin for estimating discrimination on 353 tasks. Neither the six calibration backbones nor the three held-out backbones are named.
  • The paper speaks of capability gaps, but the IRT model has a single ability per agent, and no per-CWE or per-category results are reported.
  • Each configuration appears to attempt each task once, and Figure 4 has no error bars.
  • Table 5 does not report the number of experts, the number of tasks rated, or how agreement was computed.
  • There is no statement on releasing code or tasks.
Questions for authors
  1. How are random and one-shot synthesis defined? For pools built by each, what is the ability error on held-out agents?
  2. If correct alternative patches are run against the hidden grader, how many fail?
  3. Does the specification s include the security requirements shown in the task panels?
  4. Which six backbones calibrated the IRT model, and which three were held out?

Suggestion:

  • It would help to compare against a simpler baseline that skips the IRT loop: generate tasks at Easy, Medium and Hard in equal proportion with the same budget, then check how well each pool estimates the ability of held-out agents. If this simple pool does about as well, the gain likely comes from difficulty control rather than adaptive targeting.
6 comments
AuthorsOpenReview ↗

Response to Weakness 1

Thank you for the detailed review and constructive feedback.

W1. We next respond to each point mentioned regarding the "adaptive synthesis and efficiency" of our proposed SecProbe.

  • (W1.a) Efficiency beyond the information-gap objective. We understand the reviewer’s concern that Figure 5 in the original manuscript compares policies on the same information gap that guides adaptive synthesis. Our main claim is that adaptive synthesis makes agent benchmarking more efficient while preserving the quality and reliability of the assessment. In the original submission, Figure 5 shows that adaptive synthesis reaches the same information-gap target with 244 accepted tasks, compared with 328 for random synthesis and 346 for one-shot synthesis. Following the reviewer’s suggestion, we test whether these task savings hold on two measures of assessment quality: 1) how closely SecProbe recovers agent rankings on well-established cybersecurity benchmarks, and 2) how many confirmed agent weaknesses it identifies at a given task budget.

    • 1) Recovering rankings with fewer tasks. An efficient assessment should preserve the relative strengths of the agents it compares. SecProbe estimates IRT ability, while CyberGym Level 1 and CyberGym E2E rank agents by success on their respective tasks. Because the benchmarks differ in tasks and scoring rules, we use the Spearman correlation coefficient to compare the ordering of the same held-out model and harness configurations. Let uiu_i and viv_i be configuration ii’s centered ranks by SecProbe ability and external benchmark success rate, with ties assigned average ranks. Spearman correlation is ρS=∑iuivi(∑iui2)(∑ivi2)\rho_S = \frac{\sum_i u_i v_i}{\sqrt{(\sum_i u_i^2)(\sum_i v_i^2)}}. A value close to 1 indicates that the two assessments generally rank the configurations in the same order.  
      At 280 accepted tasks, adaptive synthesis achieves Spearman correlations of 0.89 with CyberGym and 0.94 with CyberGym E2E. Under a two-sided exact permutation test that assumes no ties and treats the six configurations as exchangeable, these correlations would yield nominal p-values of 0.0333 and 0.0167, respectively. The corresponding correlations are 0.71 and 0.66 for random synthesis, 0.77 and 0.71 for one-shot synthesis, and 0.83 and 0.77 for balanced difficulty synthesis (added in response to the reviewer’s suggestion and defined in W1(b) and Q1 below). Adaptive synthesis first meets both correlation targets at 260 tasks, compared with 340, 320, and 310 for the three baselines, respectively. Adaptive synthesis therefore reaches the same level of external ranking agreement with 50 fewer tasks than the strongest baseline. Its advantage over balanced difficulty synthesis also shows that updating the targets from observed responses adds value beyond distributing tasks evenly across difficulty levels.

    • 2) Finding more agents' weaknesses with fewer tasks. An overall ranking tells us which agents perform better, while weakness discovery tells us where each agent struggles. We count a weakness when an agent shows low repair success in a security category and that finding is confirmed on a separate set of tasks.  
      At 280 accepted tasks, adaptive synthesis identifies 26 confirmed weaknesses, compared with 13 for random synthesis, 15 for one-shot synthesis, and 16 for balanced difficulty synthesis. At the same budget, it identifies 10 more weaknesses than the strongest baseline, an increase of 62.5%. Adaptive synthesis first reaches 24 confirmed weaknesses at 250 tasks, compared with 350 for random synthesis, 335 for one-shot synthesis, and 320 for balanced difficulty synthesis. It therefore needs 70 fewer tasks than the strongest baseline to uncover the same number of weaknesses. The advantage over balanced difficulty synthesis shows that using observed failures to guide what we test next helps expose weaknesses that an even mix of Easy, Medium, and Hard tasks can miss at the same budget.

    In the revised manuscript, we add two paragraphs to Section 5 (Discussion): “Does synthesis recover agent rankings on external benchmarks?” and “Does adaptive synthesis reveal more agent weaknesses?” These paragraphs report the ranking agreement and weakness discovery results.

AuthorsOpenReview ↗

Response to Weakness 1(continued) and Question 1

  • (W1.b and Q1) Synthesis baselines and held-out evaluation. The random and one-shot baselines are chosen to distinguish the benefit of adding new tasks from the benefit of using IRT to decide which tasks to generate. We explain them in detail below (and further clarification in Section 5 of revised manuscript). Following your suggestion, we also add a balanced difficulty baseline to test whether an even mix of Easy, Medium, and Hard tasks can explain the gains.

    • Random synthesis baseline helps assess how much of the gain comes simply from expanding the pool. It samples uniformly with replacement from a fixed catalogue of 1,359 configurations: 151 CWEs, three compatible software contexts per CWE, and three difficulty levels per context.

    • One-shot synthesis baseline tests whether the initial IRT fit provides sufficient guidance for the entire run. After the first 150 tasks, it sets the target difficulty to the ability level with the largest information gap, β0⋆=arg⁡max⁡θG1(θ)\beta_0^\star = \arg\max_\theta G_1(\theta), and the target discrimination to the 75th percentile of discrimination values among initial tasks within 0.5 IRT units of that difficulty. It then selects configurations whose predicted item parameters are closest to those targets, keeping both the targets and mappings fixed.

    • Balanced difficulty synthesis tests the alternative you suggested: that the gains come from controlling the difficulty mix. It distributes new tasks as evenly as possible across Easy, Medium, and Hard, sampling uniformly from compatible CWE and software configurations within each level, without IRT feedback. At 280 accepted tasks, the 130 additions comprise 44 Easy, 43 Medium, and 43 Hard tasks. Appendix A.5 in the revised manuscript provides the baseline definition and both comparisons.

    For the held-out comparison in Q1, Appendix A.5 in the revised manuscript reports ability RMSE for selection within a fixed pool. For the pools produced by each synthesis policy, we instead report agreement with the CyberGym and CyberGym E2E rankings in response to W1.a. At 280 accepted tasks, the Spearman correlation coefficients are 0.89 and 0.94 for adaptive synthesis, 0.71 and 0.66 for random synthesis, 0.77 and 0.71 for one-shot synthesis, and 0.83 and 0.77 for balanced difficulty synthesis. These results quantify how closely each synthesized pool recovers the relative ordering of the held-out configurations.

  • (c) Why Figure 5 uses 0.30 and Appendix A.2.1 uses 0.15. The two thresholds serve different purposes. Figure 5 in the original manuscript uses 0.30 as a common comparison point because all three plotted policies reach it within the evaluated budget. Appendix A.2.1 in the original manuscript uses 0.15 as the stopping rule for constructing the final adaptive pool.

    The reported reductions of 25.6% and 29.5% are specific to reaching 0.30. Adaptive synthesis requires 244 accepted tasks, compared with 328 for random synthesis and 346 for one-shot synthesis. The broader conclusion is supported by the full curves in Figure 5 of the original manuscript: adaptive synthesis has a smaller information gap than both baselines at every plotted checkpoint after initialization, and it is the only policy to reach 0.15 within 353 tasks. The ranking and weakness analyses in W1(a) assess efficiency without using either information-gap threshold. We clarify the distinction in Section 5 and the Figure 5 caption of the revised manuscript.

  • (d) Ability-estimation accuracy is evaluated for fixed-pool selection, not adaptive synthesis. We agree that the ability RMSE in Appendix A.5 of the original manuscript evaluates how efficiently adaptive selection recovers full-pool ability estimates; it does not establish the ability-estimation accuracy of adaptive synthesis. Our motivation for synthesis is complementary: as Table 1 in the original manuscript shows, existing cybersecurity benchmarks cover limited CWE types. Selection can prioritize informative tasks already available, but cannot assess weaknesses for which the pool contains no tasks. Synthesis is intended to address these coverage gaps. However, broader coverage alone does not demonstrate lower ability-estimation error. We will clarify this distinction and the scope of the evidence.

    To evaluate synthesis directly, our response to W1.a compares adaptive synthesis with random, one-shot, and balanced difficulty synthesis on two outcomes: external ranking agreement and confirmed weakness discovery. Compared with the strongest baseline, adaptive synthesis reaches the same ranking-agreement targets with 50 fewer tasks and discovers 24 confirmed weaknesses with 70 fewer tasks. These results assess synthesis efficiency on two complementary outcomes, rather than absolute ability-estimation error.

    In the revised manuscript, we add a detailed comparison of CWE coverage across benchmarks in Appendix A.6 (Supplementary Benchmark Statistics).

AuthorsOpenReview ↗

Response to Weakness 2 & 3, and Question 2 & 3

W2 / Q2–Q3. Repair validity and task information

  • (a) What a pass measures. We understand the reviewer’s concern that a pass might reward reproducing the generator’s reference patch. A submission passes when it earns the full score of 100 on the hidden grader, which runs both functional tests for required program behavior and security tests that exercise the vulnerability. The grader evaluates the behavior of the submitted repository. The reference patch serves as a positive control during task construction and is not used for patch matching. A pass therefore measures whether the submitted repair meets the tested functional and security requirements. We clarify this scoring criterion in Section 3.1 of the revised manuscript.

    Our existing checks address different aspects of grader validity. Appendix A.2.2 of the original manuscript requires graders to accept the reference repair and detect declared security mutations, providing positive and negative controls. Table 5 in the original manuscript reports expert assessments of how well the grader aligns with the task requirements. Appendix A.9 of the original manuscript examines whether agent submissions fix the root cause, cover the affected paths, and preserve required behavior. The additional experiment below directly checks whether the original graders also accept independently verified alternative repairs.

  • (b, Q2) Acceptance of correct alternative patches. To further validate that a pass reflects repair correctness, we manually construct alternative repairs, verify their correctness independently, and test them with the original, unchanged hidden graders. We find that every sampled task accepts at least one correct implementation that differs from its reference patch.

    Across 50 tasks, we manually write three alternative repairs per task, varying the repair strategy or where the security check is enforced. Two security reviewers verify each repair through code inspection and separately written regression tests before it is graded. We then score each repair using the same criterion as in the agent evaluation: a pass requires a full score of 100.

    The original graders accept 147 of the 150 verified alternatives (98.0%). These results show that the sampled graders generally accept different correct repairs. We report the protocol and results in Appendix A.9 of the revised manuscript.

  • (c, Q3) What the agent sees. The security statements highlighted in Figures 6 and 9–11 of the original manuscript are only our author annotations and are not part of the task specification supplied to the agents. The agents receive the original specification and the vulnerable repository. The vulnerability locations, exploit samples, and reference patches are withheld, so the agents must identify the affected code paths and repair them while preserving required functionality. We clarify what the agents see in the captions of Figures 6 and 9–11 in the revised manuscript.

W3. Related work

  • Prior work on adaptive task generation. Truong et al. (2025) already combines IRT-based calibration with difficulty-conditioned question generation for adaptive testing. We now discuss both contributions in Appendix A.1 of the revised manuscript. SecProbe builds on this idea by using observed agent responses to decide when to synthesize new tasks and which difficulty and discrimination to target. It applies this approach to repository-scale vulnerability repair, where each generated task includes executable functional and security tests and a validated reference repair.

    We also expand Appendix A.1 of the revised manuscript to discuss Bejar et al. (2003) and Glas and van der Linden (2003) as earlier work on item generation and calibration for adaptive testing, and to relate SecProbe to automated benchmark construction in AutoBencher (Li et al., 2024) and controllable task generation in DyVal (Zhu et al., 2024).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.