- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents SandboxLive, a live benchmark and open harness for evaluating LLM agents on end-to-end dynamic malware analysis. The author designed a standard for filtering samples from MalwareBazaar, and designed a framework for evauations. In this paper, the evaluated model plays four roles that communicate through a JSON specification. Meanwhile, a deterministic controller locks the sample, downgrades the network route to drop, validates the submission, verifies that the retrieved report matches the sample hash and shows execution, and withholds delivery otherwise. The sandbox is a QEMU emulator with a Windows and an ubuntu, exposed through a CAPE-style client interface. Delivered reports are scored by 4 of 31 rules against the evidence of their own run: category compatibility with MalwareBazaar metadata, ATT&CK technique precision, technique recall, and IOC grounding via normalized substring match.
The experiment cover 8 llm agents across 3 sample collection seasons and answer 3 RQs which includes: 1. The performance of LLM Agent analyze unseen samples, 2. The failure analysis, 3. The analyze of harness performance and cost of violations.
Critical strengths
-
The evaluation addresses a real limitation of prior work. Existing malware benchmarks either directly provide reports to the agent or do not actually execute the samples at all; in contrast, this paper requires the agent to submit samples for execution, using samples that postdate the model's release.
-
The failure attribution and enforcement analysis produces a concrete, transferable conclusion. The failure analysis gives an overview of how these 8 LLMs behave as agents from another perspective, and the recommendation to enforce conventions in the harness goes beyond this paper, I think.
Critical weaknesses
-
The F1 rests on too few dimensions, and the rules were fitted to the same runs. In my opinion, for many reports only one or two of the three dimensions can be judged, so a single F1 often comes from one number. And the predicates and behavior classes were tuned on these same runs. This is a separate issue from the ranking instability the authors already admit.
-
The agent decides less than "end to end" suggests. The controller fixes the route and the backend picks the guest, so the LLM seem like just sets the window and the DLL export. The paper's own number shows a file-format rule gets the package right in 652 of 654 runs. What is really being tested is reading the telemetry and writing the report, which is a smaller task than the title claims. In another word, how the author define 'end to end'.
Questions for authors
Are all four llm roles needed? Seems like that the Scout and Architect never disagree on the OS or architecture, and the package is fixed by a file-format rule in 652 of 654 runs, so those two stages look replaceable by code.
How many of the ATT&CK techniques that the models claim have no predicate at all, and so never enter technique precision? Does this differ by model? If one model likes techniques outside your 23 predicates, its precision is computed on a different slice of its claims than the others.
The Analyst section says the backend report contains "four heuristic signatures". What exactly are they, and how many of the 12 R2 detection rules depend on them versus the raw Sysmon/strace fields?
Did you check whether any public vendor write-up for the sampled hashes appeared between the first-seen date and the date you ran the evaluation, and were the model endpoints you used snapshots fixed before each season's window?
2 comments
W2 and Q1: End-to-end analysis and agent roles
W2: The agent decides less than "end to end" suggests. The controller fixes the route and the backend picks the guest, so the LLM seem like just sets the window and the DLL export. The paper's own number shows a file-format rule gets the package right in 652 of 654 runs. What is really being tested is reading the telemetry and writing the report, which is a smaller task than the title claims. In another word, how the author define 'end to end'.
and
Q1: Are all four llm roles needed? Seems like that the Scout and Architect never disagree on the OS or architecture, and the package is fixed by a file-format rule in 652 of 654 runs, so those two stages look replaceable by code.
TL;DR: The agents analyze the malware sample end-to-end without human intervention. At each stage, agents' decisions are still important and evaluated via verifiable rules. The static analysis of Scout is very critical to the report quality.
By end-to-end, we mean that, given only the sample, the agents carry out the entire workflow (static triage, environment setup, detonation, evidence interpretation, and reporting) without human intervention. Constraining some risky decisions does not make the pipeline less end-to-end.
- We limit agent decisions for safety concern. The key difference from related work is that we execute real malware samples, so a code-based controller must gate every risky decision. Even with such a strong harness, unsafe actions could still happen (e.g., GPT-5.6 Luna case in our case studies).
- Agents' actions and choices are still evaluated at each stage. The controller prevents unsafe operations, but agents still decide which environment, system, and sandbox configuration each sample needs. These decisions are evaluated by the rules in Scout and Architect stages though some are not adopted due to safety concern. These records are analyzed in our case studies and appendix. These rule-based evaluations support end-to-end benchmarking beyond the main F1.
- Agents do more than just "Reading the telemetry and writing the report". Modern malware analysis consists of two main steps: static analysis and dynamic analysis (Figure 1). Without the Scout's static analysis, the agents can only analyze the sample based on dynamic evidence.
As an ablation, we tested replacing Scout, Architect, and Executor with scripts. The experiments are conducted with DeepSeek on 39 S4 samples with three repetitions per arm and same scoring.
| Arm | Delivered runs | Mean F1 |
|---|---|---|
| Four-role pipeline | 92/117 | 0.729 |
| Scripted setup + same Analyst | 100/117 | 0.624 |
From the experiments, we found that script avoids setup errors caused by agents (e.g., tool calling errors of DeepSeek) so the delivered reports increase. However, without static analysis in Scout stage, the Analyst becomes so conservative that it labels 43% of delivered reports as “unknown”, compared with 4% in the four-role pipeline. Thus the F1 scores drop sharply due to low recall.
We do not claim that all four roles are necessary for malware analysis in general; rather, they form a safe, standardized harness for benchmarking agents’ malware-analysis capabilities.
Rebuttal: We are grateful for the encouraging review and constructive feedback! With our best efforts, we conducted additional experiments and analysis to address your concerns. Additional results and clarifications will be added to the manuscript accordingly. We will carefully proofread and revise unclear parts in the updated version as well. Below we provide the point-by-point responses. For your convenience, we provided TL;DR summaries before lengthy detailed explanations to help you quickly grasp our main points.
W1: F1 design and same runs fitting
TL;DR: Most reports use at least two dimensions. We further clarify why we designed rules on the same runs and present experiments on a new season.
Response:
We examined dimension availability across the 612 delivered reports. This shows that most of reports are using at least two dimensions in precision or recall.
The rules were developed using S1–S3 and we explain in Appendix C.3 why these rules are based on these runs. And we already acknowledge that these rules may inherit bias in our runs thus may not remain valid for other models or samples (line 313-315).
To further clarify,
To address your concern, we have applied the same evaluation to a new season S4 (the first half of September), covering six models, 39 samples, and three repetitions per model.
The table shows that Claude and Grok still have better performance than DeepSeek and GPT models on new samples (ranking in S3). And the rankings of three independent runs remain stable based our main F1.