- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Atoms to Agents, which decomposes recorded cybersecurity agent trajectories into recurring atomic operations, organizes them into a capability taxonomy, and turns them into single-turn items with verified answers and five question forms. On 13 models, scores on the resulting 2094 item bank track rankings on ExCyTIn and CTI-REALM with Spearman correlations near 0.9, and the best observed 25-item proxy uses 76 to 96 times fewer tokens than benchmark subsets of matched quality. Transfer to the unseen SOCBench workflow is weaker, but taxonomy-guided additions to the bank improve it.
Critical strengths
- The paper tackles a real cost problem. Harness based agentic benchmarks consume many tokens per evaluation, and common remedies either evaluate a benchmark subset or rely on partial evaluation. Predicting end to end rankings from cheap atomic tasks mined from agent trajectories is a new and complementary direction.
- Items are verifiable and can be regenerated from new trajectories, so the pipeline offers a way to refresh the bank and partly reduce the contamination risk that affects static public benchmarks.
- Gold answers are almost always grounded in execution or recorded environment output, checkers are deterministic, and each admission guard in the appendix is tied to a real defect. This makes the bank easy to audit.
- In domain ranking agreement is strong and stable across estimators. The whole bank average alone, with no fitting or target labels, reaches about 0.85 on ExCyTIn and 0.92 on CTI-REALM, and the signal holds under target blind selection, PACE, Ridge, and family holdout.
Critical weaknesses
- I would appreciate a clearer distinction between the prespecified and exploratory efficiency results. The reported 76 and 96x savings come from the 25 item budget, which performed best among 14 budgets after examining the target results. The prespecified 100 item budget gives savings of about 12 and 10x. I would suggest reporting the prespecified results in the abstract and Figure 1, while presenting the larger savings as an exploratory finding.
- The cost comparison would be more convincing if both sides used the same summary statistic. Currently, the proxy uses its single best correlation, while the benchmark uses the average correlation over 10000 random subsets. This difference may favor the proxy, and using a consistent statistic would help readers assess the efficiency gains more fairly.
- I would like to better understand how the taxonomy contributes to prediction. The best selector does not use it, and the 40 aggregation leaves are separate from the 172 leaf taxonomy. An ablation without the taxonomy would help clarify its contribution. Human validation, quantitative stability results, and details of the models used to build it would also make this part of the work easier to assess.
- As mentioned in S4, I appreciate that the pipeline can generate new items and help reduce data contamination based on my understanding. However, the proposed bank itself is fixed and public, so contamination remains possible. With about a quarter of its items already solved by every model, it would be useful to discuss the risk of early saturation. I would also appreciate clarification on how refusals were scored and whether the frontier model that refused 65% of the bank was included in the evaluation panel, since its score may partly reflect refusal behavior rather than capability.
Questions for authors
- If the budget were chosen without using target scores, would the 25 item budget still be selected?
- How large are the token savings when both sides use the same statistic, such as the average correlation across resamples?
- How does prediction change if items are aggregated without the taxonomy?
- How were refusals scored, and does the ranking change if the refusing model is excluded?