acceptodds
Under review as a conference paper at ICLR 2027

Atoms to Agents: Efficient Proxies for Cybersecurity Agentic Evaluation

Abstract

End-to-end benchmarks for cybersecurity large language model (LLM) agents are costly to run: they require interactive environments, tool execution, and multi-turn reasoning, yet ultimately provide only a coarse measure of overall task success. This makes them expensive for repeated model comparison and provides limited visibility into the underlying operations that contribute to success or failure. We introduce ATOMS TO AGENTS, a framework that decomposes agent trajectories into recurring atomic tasks, organizes them into a trajectory-derived capability taxonomy, and converts each into an isolated evaluation with an independently verifiable answer. By posing each task from a fixed trajectory state, the framework prevents upstream errors from propagating and avoids full multi-turn rollouts. The resulting tasks assess both operation execution and, where applicable, action selection in context, localizing model weaknesses to specific capabilities. On a panel of 13 models evaluated on ExCyTIn and CTI-REALM—the two benchmarks represented in bank construction—atomic-task scores correlate with full agentic performance at Spearman ρ ≈ 0.9. Compared with uniformly sampled benchmark subsets that achieve the same Spearman correlation with the full-benchmark model ranking, our best observed 25-item proxy uses 76× fewer tokens on ExCyTIn and 96× fewer on CTI-REALM. Transfer to a new workflow is weaker when relevant capabilities are missing from the evaluation set, but the framework’s taxonomy helps identify these gaps and guide targeted expansion, hence improving ranking agreement. These findings support efficient model comparison and capability diagnosis while highlighting the need for coverage-aware maintenance and end-to-end validation.

Primary area:
datasets and benchmarks
Keywords:
Cybersecurity agents, LLM evaluation, proxy benchmarks

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_y5aeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Atoms to Agents, which decomposes recorded cybersecurity agent trajectories into recurring atomic operations, organizes them into a capability taxonomy, and turns them into single-turn items with verified answers and five question forms. On 13 models, scores on the resulting 2094 item bank track rankings on ExCyTIn and CTI-REALM with Spearman correlations near 0.9, and the best observed 25-item proxy uses 76 to 96 times fewer tokens than benchmark subsets of matched quality. Transfer to the unseen SOCBench workflow is weaker, but taxonomy-guided additions to the bank improve it.

Critical strengths
  1. The paper tackles a real cost problem. Harness based agentic benchmarks consume many tokens per evaluation, and common remedies either evaluate a benchmark subset or rely on partial evaluation. Predicting end to end rankings from cheap atomic tasks mined from agent trajectories is a new and complementary direction.
  2. Items are verifiable and can be regenerated from new trajectories, so the pipeline offers a way to refresh the bank and partly reduce the contamination risk that affects static public benchmarks.
  3. Gold answers are almost always grounded in execution or recorded environment output, checkers are deterministic, and each admission guard in the appendix is tied to a real defect. This makes the bank easy to audit.
  4. In domain ranking agreement is strong and stable across estimators. The whole bank average alone, with no fitting or target labels, reaches about 0.85 on ExCyTIn and 0.92 on CTI-REALM, and the signal holds under target blind selection, PACE, Ridge, and family holdout.
Critical weaknesses
  1. I would appreciate a clearer distinction between the prespecified and exploratory efficiency results. The reported 76 and 96x savings come from the 25 item budget, which performed best among 14 budgets after examining the target results. The prespecified 100 item budget gives savings of about 12 and 10x. I would suggest reporting the prespecified results in the abstract and Figure 1, while presenting the larger savings as an exploratory finding.
  2. The cost comparison would be more convincing if both sides used the same summary statistic. Currently, the proxy uses its single best correlation, while the benchmark uses the average correlation over 10000 random subsets. This difference may favor the proxy, and using a consistent statistic would help readers assess the efficiency gains more fairly.
  3. I would like to better understand how the taxonomy contributes to prediction. The best selector does not use it, and the 40 aggregation leaves are separate from the 172 leaf taxonomy. An ablation without the taxonomy would help clarify its contribution. Human validation, quantitative stability results, and details of the models used to build it would also make this part of the work easier to assess.
  4. As mentioned in S4, I appreciate that the pipeline can generate new items and help reduce data contamination based on my understanding. However, the proposed bank itself is fixed and public, so contamination remains possible. With about a quarter of its items already solved by every model, it would be useful to discuss the risk of early saturation. I would also appreciate clarification on how refusals were scored and whether the frontier model that refused 65% of the bank was included in the evaluation panel, since its score may partly reflect refusal behavior rather than capability.
Questions for authors
  1. If the budget were chosen without using target scores, would the 25 item budget still be selected?
  2. How large are the token savings when both sides use the same statistic, such as the average correlation across resamples?
  3. How does prediction change if items are aggregated without the taxonomy?
  4. How were refusals scored, and does the ranking change if the refusing model is excluded?
Reviewer_tPyPOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors introduce Atoms To Agents, which takes AI agent trajectories recorded from existing cybersecurity eval suites and turns them into atomic tasks or standalone questions to try and reconstruct agentic capabilities through more atomic tasks. The authors find that small tests with atomic questions can pretty reliably reconstruct model rankings on existing benchmarks, while operating at a significantly lower token cost.

Critical strengths

While proxy metrics can sometimes lack fidelity, they are undoubtedly important in the evaluation of AI agents. It is indeed the case that it is often incredibly expensive to run long-horizon agentic benchmarks, particularly on cybersecurity tasks, which makes them difficult to run either repeatedly or at scale. By breaking down those tasks into atomic questions, the authors contribute quite a lot to the ecosystem. For starters, their reported efficiency gains are quite telling. Further, having a more fine-grained understanding of what and where agents fail instead of binary yes/no grading (like in most CTF or other verifiable cyber benchmarks) is certainly informative. Finally, and perhaps this is not something the authors have thought about, but having question-answer proxies is also useful for assessing where the split between specific cyber knowledge and general agentic capabilities occurs during the training regimes of LLMs.

Critical weaknesses

I believe that the authors have over-indexed on their Spearman correlation metrics and cost-efficiency metrics, instead of focusing on finegrained explanations for why specific agents fail at certain benchmarks. While being able to reliably reconstruct proxy metrics for agentic performance on long-horizon tasks is useful at smaller scales, I believe the single largest contribution is showing that, by using this pipeline, the authors were able to roughly recreate long-horizon benchmarks with much better atomicity. I would have liked to see more specific assessments or examples of atomic tasks where failure causes specific models to perform poorly at specific evals, as I think that could be one of the largest strengths of the paper.

Questions for authors
  1. Any evidence towards my identified weakness would be great.
  2. Why not use Cybench or CyberGym as evaluation targets, given that they are both more cited / arguably more relevant than ExCyTIn and CTI-REALM?
Reviewer_toknOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether inexpensive static evaluations can approximate the end-to-end performance of cybersecurity agents. The authors extract local operations from 1,700 agent trajectories across five benchmarks, construct a capability taxonomy using LLM-FINCH, and create atomic tasks with fixed contexts and checkers. The resulting evaluation bank contains 2,094 items across 40 evaluation leaves and five question forms, evaluated on 13 models.

On ExCyTIn and CTI-REALM, the two in-domain benchmarks represented in bank construction, the prespecified target-blind, 100-item, leave-one-model-out setting achieves Spearman correlations of 0.885 and 0.890, respectively. The prominently reported 25-item setting reaches 0.912 and 0.956, using approximately 76× and 96× fewer tokens than random end-to-end benchmark subsets at matched ranking quality. However, 25 items is the best observed point in a budget sweep, rather than a budget fixed in advance. For SOCBench, which did not contribute to the original bank, the authors use capability-coverage diagnosis to add 315 items, increasing target-blind correlation from 0.552 to 0.783 on the same 12-model panel. The main contribution is a workflow-grounded model-screening instrument, rather than a replacement for evaluating complete agent behavior.

Critical strengths

Under the prespecified 100-item target-blind setting, the paper reports model-ranking correlations of 0.885/0.890 without using target-benchmark scores for item selection. Whole-bank aggregation also provides positive evidence of ranking recovery. Appendix C.1 additionally reports holdout results across eight model families, so the evaluation should not be characterized as lacking family-level generalization checks altogether. Although the bank-quality concerns remain unresolved, these results suggest practical potential for inexpensive candidate-model screening on established benchmarks.

Critical weaknesses

Appendix B.5 explicitly states that a post-freeze provenance audit of all 587 ExCyTIn items recommended dropping 182 items, but those recommendations were not applied to the fixed bank evaluated in the paper. This represents 31.0% of the ExCyTIn-derived items and 8.7% of the entire bank. These percentages are not established error rates, and the audit recommendations may themselves require human adjudication. Nevertheless, without the reasons for exclusion, an independent adjudication, and a cleaned-bank sensitivity analysis, it is unclear whether the main correlations depend on questionable items or whether the 25-/100-item selection procedures amplify their influence. [p. 20, Appendix B.5, lines 1059–1061]

In addition, the AI Use Statement on page 10 says that every gold label comes from execution or recorded environment output, never from an LLM's assertion. Yet Appendix B.3 and Table 6 explicitly retain 33 items labeled with decomposition-derived subgoals. This discrepancy requires an explanation or correction. A deterministic checker does not, by itself, establish that the reference label was independently validated. [p. 10, lines 488–493; pp. 19–20, Appendix B.3 and Table 6]

More fundamentally, being able to verify an environment's recorded output does not imply that the output is uniquely inferable from the item prompt. For example, the ANTICIPATE example on page 18 shows a list of available tags and asks whether a rule-library search will return no results. The displayed context does not establish whether the library contains matching rules. Similarly, a recorded next action is not automatically the uniquely correct action in the supplied state. The authors need to establish whether these items measure identifiable local capabilities or require guessing hidden environment state or a reference trajectory. This issue affects the core construction method and capability-diagnosis claim, not merely the wording of a few examples.

Questions for authors
  1. Clarification and essential sensitivity analysis: What specific issues led to the recommendation to drop 182 items? What criteria did the audit use, and was there independent human adjudication? How many of these items appear in each fold's 25-/100-item selection? After cleaning under explicitly defined quality criteria and rerunning item selection and fitting, do the results in Tables 1 and 2 and the main cost comparisons remain stable? Updating only the final aggregate scores would not answer this question.

  2. Methodological consistency: Were the final labels for the 33 decomposition-derived subgoal items independently verified by another mechanism? If so, please describe it. For the ANTICIPATE example on page 18, what evidence in the prompt determines that the search returns none, or has necessary context been omitted from the example?

  3. Existing evidence or minimal additional analysis: Is there already an evaluation on new models with the budget, bank, and all selection rules frozen beforehand? If not, please consistently identify the 25-item result as exploratory and prioritize the prespecified 100-item result when discussing deployment benefits. The cost discussion should also maintain a consistent comparator: the appendix's break-even points of 2/15 additional models are relative to full end-to-end evaluation, not the matched-quality subsets used for the 76×/96× comparisons.

  4. Controlled evidence for the taxonomy's contribution: Are there results for a random expansion matched to the 315 added items in number, source, and question form, or validation on incidents/workflows not used in coverage diagnosis?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.