acceptodds
Under review as a conference paper at ICLR 2027

SandboxLive: Benchmarking LLM Agents on End-to-End Dynamic Malware Analysis

Abstract

LLM agents are increasingly used as security analysts, yet it remains unclear whether they can analyze an unseen malware sample end to end, i.e., triage it, plan and run its detonation in a sandbox, and report only what the run supports. Existing benchmarks evaluate agents either inside environments that are prepared for them or on static and question-answering tasks, leaving untested whether an agent can run the detonation of a real sample itself and ground its report in what the run observed. In this paper, we present SandboxLive, a live benchmark and an open harness for dynamic malware analysis by LLM agents. Each season draws Windows and Linux samples first seen after the release of every evaluated model, and the evaluated model plays four agent roles that operate a real sandbox under a deterministic controller. After the experiments, each analysis report is scored against the evidence of its own run by deterministic verification rules, without an LLM judge. Over three seasons (119 samples, eight models, and 675 runs), our results reveal three key findings. (1) The leading models are close, and they differ in how they interpret the evidence rather than in how they set up the detonation. (2) Only 26 of the 63 failed runs are attributable to the model. (3) How the harness enforces a constraint, rather than how often a model violates it, determines the cost of a violation. Code and season lists are provided in the supplementary material.

Primary area:
datasets and benchmarks
Keywords:
LLM Agents, Agent Evaluation, Sandbox Environment Synthesis, Long-Horizon Tasks, Dynamic Malware Analysis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TfNMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents SandboxLive, a live benchmark and open harness for evaluating LLM agents on end-to-end dynamic malware analysis. The author designed a standard for filtering samples from MalwareBazaar, and designed a framework for evauations. In this paper, the evaluated model plays four roles that communicate through a JSON specification. Meanwhile, a deterministic controller locks the sample, downgrades the network route to drop, validates the submission, verifies that the retrieved report matches the sample hash and shows execution, and withholds delivery otherwise. The sandbox is a QEMU emulator with a Windows and an ubuntu, exposed through a CAPE-style client interface. Delivered reports are scored by 4 of 31 rules against the evidence of their own run: category compatibility with MalwareBazaar metadata, ATT&CK technique precision, technique recall, and IOC grounding via normalized substring match.

The experiment cover 8 llm agents across 3 sample collection seasons and answer 3 RQs which includes: 1. The performance of LLM Agent analyze unseen samples, 2. The failure analysis, 3. The analyze of harness performance and cost of violations.

Critical strengths
  1. The evaluation addresses a real limitation of prior work. Existing malware benchmarks either directly provide reports to the agent or do not actually execute the samples at all; in contrast, this paper requires the agent to submit samples for execution, using samples that postdate the model's release.

  2. The failure attribution and enforcement analysis produces a concrete, transferable conclusion. The failure analysis gives an overview of how these 8 LLMs behave as agents from another perspective, and the recommendation to enforce conventions in the harness goes beyond this paper, I think.

Critical weaknesses
  1. The F1 rests on too few dimensions, and the rules were fitted to the same runs. In my opinion, for many reports only one or two of the three dimensions can be judged, so a single F1 often comes from one number. And the predicates and behavior classes were tuned on these same runs. This is a separate issue from the ranking instability the authors already admit.

  2. The agent decides less than "end to end" suggests. The controller fixes the route and the backend picks the guest, so the LLM seem like just sets the window and the DLL export. The paper's own number shows a file-format rule gets the package right in 652 of 654 runs. What is really being tested is reading the telemetry and writing the report, which is a smaller task than the title claims. In another word, how the author define 'end to end'.

Questions for authors

Are all four llm roles needed? Seems like that the Scout and Architect never disagree on the OS or architecture, and the package is fixed by a file-format rule in 652 of 654 runs, so those two stages look replaceable by code.

How many of the ATT&CK techniques that the models claim have no predicate at all, and so never enter technique precision? Does this differ by model? If one model likes techniques outside your 23 predicates, its precision is computed on a different slice of its claims than the others.

The Analyst section says the backend report contains "four heuristic signatures". What exactly are they, and how many of the 12 R2 detection rules depend on them versus the raw Sysmon/strace fields?

Did you check whether any public vendor write-up for the sampled hashes appeared between the first-seen date and the date you ran the evaluation, and were the model endpoints you used snapshots fixed before each season's window?

2 comments

Rebuttal: We are grateful for the encouraging review and constructive feedback! With our best efforts, we conducted additional experiments and analysis to address your concerns. Additional results and clarifications will be added to the manuscript accordingly. We will carefully proofread and revise unclear parts in the updated version as well. Below we provide the point-by-point responses. For your convenience, we provided TL;DR summaries before lengthy detailed explanations to help you quickly grasp our main points.

W1: F1 design and same runs fitting

W1: The F1 rests on too few dimensions, and the rules were fitted to the same runs. In my opinion, for many reports only one or two of the three dimensions can be judged, so a single F1 often comes from one number. And the predicates and behavior classes were tuned on these same runs. This is a separate issue from the ranking instability the authors already admit.

TL;DR: Most reports use at least two dimensions. We further clarify why we designed rules on the same runs and present experiments on a new season.

Response:

We examined dimension availability across the 612 delivered reports. This shows that most of reports are using at least two dimensions in precision or recall.

Judgeable dimensions per report0123
Precision159233319
Recall16216380Not applicable

The rules were developed using S1–S3 and we explain in Appendix C.3 why these rules are based on these runs. And we already acknowledge that these rules may inherit bias in our runs thus may not remain valid for other models or samples (line 313-315).

To further clarify,

  1. The most vital reason is that the behaviours mainly depend on the sample under specific environments. Thus the rules should be designed based on the experiments of these specific samples under our experiments. The reruns via different models also capture the variability.
  2. We considered a stronger design when building the benchmark: a group of experienced analysts would detonate and analyze a broader set of samples, discuss and ensemble their findings into a ground truth, and set and validate the rules against it. We did not pursue it due to budget and time constraints. And the samples are new and span many families, so a group of analysts familiar with most of them could be hard to organize.

To address your concern, we have applied the same evaluation to a new season S4 (the first half of September), covering six models, 39 samples, and three repetitions per model.

ModelMean F1RankingsSDReport Deliveries
Grok 4.60.8451/1/10.04129/28/31
Claude Opus 50.8322/2/20.03035/36/35
Claude Sonnet 50.8073/3/30.03231/28/33
DeepSeek-V4-Pro0.7294/5/40.06228/34/30
GPT-5.6 Sol0.6765/4/60.08837/36/37
GPT-5.6 Luna0.6546/6/50.02936/37/35

The table shows that Claude and Grok still have better performance than DeepSeek and GPT models on new samples (ranking in S3). And the rankings of three independent runs remain stable based our main F1.

W2 and Q1: End-to-end analysis and agent roles

W2: The agent decides less than "end to end" suggests. The controller fixes the route and the backend picks the guest, so the LLM seem like just sets the window and the DLL export. The paper's own number shows a file-format rule gets the package right in 652 of 654 runs. What is really being tested is reading the telemetry and writing the report, which is a smaller task than the title claims. In another word, how the author define 'end to end'.

and

Q1: Are all four llm roles needed? Seems like that the Scout and Architect never disagree on the OS or architecture, and the package is fixed by a file-format rule in 652 of 654 runs, so those two stages look replaceable by code.

TL;DR: The agents analyze the malware sample end-to-end without human intervention. At each stage, agents' decisions are still important and evaluated via verifiable rules. The static analysis of Scout is very critical to the report quality.

By end-to-end, we mean that, given only the sample, the agents carry out the entire workflow (static triage, environment setup, detonation, evidence interpretation, and reporting) without human intervention. Constraining some risky decisions does not make the pipeline less end-to-end.

  1. We limit agent decisions for safety concern. The key difference from related work is that we execute real malware samples, so a code-based controller must gate every risky decision. Even with such a strong harness, unsafe actions could still happen (e.g., GPT-5.6 Luna case in our case studies).
  2. Agents' actions and choices are still evaluated at each stage. The controller prevents unsafe operations, but agents still decide which environment, system, and sandbox configuration each sample needs. These decisions are evaluated by the rules in Scout and Architect stages though some are not adopted due to safety concern. These records are analyzed in our case studies and appendix. These rule-based evaluations support end-to-end benchmarking beyond the main F1.
  3. Agents do more than just "Reading the telemetry and writing the report". Modern malware analysis consists of two main steps: static analysis and dynamic analysis (Figure 1). Without the Scout's static analysis, the agents can only analyze the sample based on dynamic evidence.

As an ablation, we tested replacing Scout, Architect, and Executor with scripts. The experiments are conducted with DeepSeek on 39 S4 samples with three repetitions per arm and same scoring.

ArmDelivered runsMean F1
Four-role pipeline92/1170.729
Scripted setup + same Analyst100/1170.624

From the experiments, we found that script avoids setup errors caused by agents (e.g., tool calling errors of DeepSeek) so the delivered reports increase. However, without static analysis in Scout stage, the Analyst becomes so conservative that it labels 43% of delivered reports as “unknown”, compared with 4% in the four-role pipeline. Thus the F1 scores drop sharply due to low recall.

We do not claim that all four roles are necessary for malware analysis in general; rather, they form a safe, standardized harness for benchmarking agents’ malware-analysis capabilities.

Reviewer_J7CMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces SandboxLive, a benchmark for evaluating the end-to-end capabilities of LLM agents, from sample triage and sandbox execution to evidence interpretation and report generation. Organized into half-month seasons, the benchmark selects Windows and Linux malware samples first recorded after the evaluated models were released. Four agent roles operate a real sandbox under a deterministic controller, while rule-based checks verify malware categories, ATT&CK techniques, and indicators of compromise against execution evidence. Across three seasons comprising 119 samples, eight models, and 675 runs, the results show limited differences among leading models, with failures arising from models, interfaces, samples, and infrastructure. The work’s value lies in connecting real execution with auditable evaluation. However, the scoring rules were developed on the same experimental results, the main metric does not adequately capture behavioral evidence collection, and substantial interface and environmental constraints remain. Consequently, the findings primarily characterize reporting capabilities and workflow reliability within the specific harness.

Critical strengths
  1. The task has practical significance, and the benchmark offers a clearly differentiated contribution.

    Agents must actually perform sample triage, submit samples for execution, retrieve results, and generate reports. This introduces genuine requirements for tool operation and workflow management. Coverage of both Windows and Linux, together with stratified sampling by operating system and malware-family prevalence, also provides some breadth.

    Organizing newly collected samples around model release dates helps reduce the influence of directly memorizing analysis reports for specific samples. The main innovation lies in the combination of real execution, verification against run-specific evidence, and rolling data collection, which constitutes a potentially valuable benchmark contribution. (pp. 2, 4–6)

  2. The evaluation is auditable and imposes meaningful constraints on the provenance of report evidence.

    The controller checks the sample hash, task identity, actual execution, and telemetry signals, preventing an agent's claim of completion from being accepted as sufficient evidence of success. Report scoring relies on actual tool returns and task logs, while excluding files written by agents and subsequently read back. This reduces the risk of agents using their own assertions as evidence.

    Rule-based scoring avoids reliance on an LLM judge and provides fixed rules, counterexample tests, and byte-for-byte reproducibility checks. These design choices improve the traceability of the results and make rescoring more straightforward. (pp. 5–6, 18–19)

  3. The diagnostic analysis is informative, and the authors carefully disclose sensitivity to evaluation choices.

    The paper separately records delivery rates, report quality, formatting errors, fabricated task identifiers, unauthorized calls, and fabricated tool returns. This helps identify where failures occur within the workflow.

    The authors also compare variants that assign zero scores to failed runs, restrict evaluation to a common delivered subset, and use alternative category rules. They disclose changes in rankings, interface incompatibilities, and framework-version effects. This transparency supports the cautious conclusion that leading models are close and strict rankings are unstable, giving the paper value beyond a leaderboard. (pp. 7–9, 20–25)

Critical weaknesses
  1. The scoring rules were developed on the same evaluation results, without independent validation of their semantic validity.

    The authors explicitly state that the scorer was revised through multiple rounds of manual analysis of these three seasons, and acknowledge that the rules may not apply to other models or new samples. Code reproducibility, counterexample tests, and implementation reviews can establish that the rules are executed correctly, but do not sufficiently establish that they measure the intended analytical capabilities.

    One concrete example is the rule that accepts a dropper classification whenever a dropped executable is observed. This choice raises category precision by an average of 0.30 and recall by 0.26; removing it also changes the ordering of leading models. This demonstrates that the semantics of the scoring rules can substantially affect the results.

    The rules should be frozen before evaluating a new season, and their accuracy and coverage should be assessed against consensus annotations from independent security experts. (pp. 6–7, 19–20, 24)

  2. The main score more closely measures reporting faithfulness to available evidence than the completeness of end-to-end evidence collection.

    The denominator of technique recall consists of behaviors observed in the particular run. When no relevant behavior is observed, the dimension may be omitted. Delivery also requires only that the sample starts and produces telemetry, rather than that sample-specific malicious behavior is actually activated.

    Consequently, a run that observes less may have a smaller recall denominator, and a high-scoring report does not necessarily correspond to comprehensive behavioral discovery. Although the paper includes a behavioral-capture metric, E2, it is only diagnostic and does not enter the main F1 score.

    Furthermore, technique scoring covers only 23 predicates and 12 behavior classes. Among the 612 delivered reports, technique precision cannot be evaluated for 224 reports, and technique recall for 190. Omitting these dimensions can also give different models different effective scoring weights. Behavioral activation, evidence coverage, and report quality should be reported jointly, with an explicit policy for handling missing dimensions. (pp. 6, 14, 17–21)

  3. The harness strongly restricts the action space, while the tool protocol introduces confounding in model comparisons.

    The infrastructure and guest environments are prepared in advance, the network route is fixed to drop traffic, and agents mainly control execution duration and DLL export functions. A simple file-format rule reproduces 652 of 654 analysis-package selections. The observation that models differ mainly in interpretation rather than configuration therefore partly reflects the benchmark's restricted configuration space.

    At the same time, imposing a common text-based tool protocol does not automatically ensure fairness. On the same 29 samples, switching Kimi to native tool calling increases delivered runs from 15 to 24, demonstrating that interface adaptation can substantially change performance.

    Baselines combining a deterministic execution script with the same Analyst, comparisons between a single agent and the four-role pipeline, and text-versus-native tool-calling ablations are needed to isolate the contributions of orchestration, interface compatibility, and analytical capability. (pp. 2, 6–7, 14, 25)

  4. Single-run evaluations and unequal season coverage limit the statistical reliability of model comparisons.

    Each model runs on each sample only once, preventing estimation of variability under a fixed configuration. Models also participate in different seasons, so pooled scores reflect different sample distributions and cannot directly support fully comparable capability rankings.

    The repeated S2 experiments in the appendix simultaneously change prompt examples, write permissions, controller fields, and execution code. The resulting mean absolute per-report F1 differences of approximately 0.11–0.14 on the same samples demonstrate sensitivity to the harness, but cannot estimate variability due solely to stochastic reruns.

    Although the authors appropriately avoid asserting a strict ranking, the evaluation should still include repeated runs under a frozen harness, paired comparisons on common samples, and confidence intervals. (pp. 4, 6–7, 20–22)

  5. Sample selection and execution conditions limit representativeness, and newly recorded samples do not fully eliminate contamination concerns.

    The benchmark retains only samples with Triage static scores of 1–3 and dynamic scores of at least 7. It therefore favors a subset for which another sandbox has already observed substantial behavior. This is suitable for studying dynamic-evidence interpretation, but provides limited coverage of sandbox-evasive, interaction-dependent, command-and-control-dependent, and dormant samples.

    The local setup also uses network isolation, TCG emulation, and a limited set of guest environments. It cannot observe complete HTTP exchanges, command-and-control content, or subsequent payloads, further restricting the behaviors being evaluated.

    The temporal filtering and contamination checks in the appendix are valuable, but the first public recording date cannot guarantee that malware families, near-duplicate code, or historical variants were unseen by the models. Family-disjoint evaluation, similarity audits, and cross-validation across different sandbox configurations would strengthen the benchmark. (pp. 4–6, 14, 25–26)

  6. The failure taxonomy has diagnostic value, but the causal interpretation of model-attributable failures remains insufficiently established.

    Table 4 attributes three failures involving execution windows longer than 300 seconds to the model. However, the appendix notes that selecting a longer duration had a reasonable justification, the submission interface allowed up to 1,800 seconds, and the effective limit was not disclosed to the model. All 18 detonations exceeding 300 seconds lack valid telemetry, and in some cases the sample process remains alive at the end of execution.

    It is necessary to distinguish among a model violating an instruction, a model's choice triggering a problem, and the underlying cause of an environmental failure. These are not equivalent. The statement that 26 of 63 failures are attributable to the model is therefore better treated as a descriptive statistic under the current classification criteria.

    Intervention-based reruns after fixing or explicitly documenting environmental limits are needed to distinguish model causes, environmental causes, and interaction effects. Agreement among annotators of failure causes should also be reported. (pp. 8, 22)

Questions for authors

Please refer to the listed weaknesses in previous section

8 comments

We are grateful for the encouraging review and constructive feedback! We conducted additional experiments and analysis to address your concerns. Additional results and clarifications will be added to the manuscript accordingly. Below we provide the point-by-point responses. For your convenience, we provided TL;DR summaries before lengthy detailed explanations to help you quickly grasp our main points.

W1: The scoring rules were developed on the same evaluation results, without independent validation of their semantic validity.

TL;DR: We applied frozen v1.6 scoring to a new season and conducted rule-sensitivity and implementation audits. The original dropper exception accounts for 147 category passes in S1–S3, and modifying or removing it changes F1 scores and some rankings. The additional analyses establish scoring sensitivity and new-season applicability, but independent expert validation remains unavailable.

The authors explicitly state that the scorer was revised through multiple rounds of manual analysis of these three seasons, and acknowledge that the rules may not apply to other models or new samples. Code reproducibility, counterexample tests, and implementation reviews can establish that the rules are executed correctly, but do not sufficiently establish that they measure the intended analytical capabilities.

Response: The rules were refined through manual examination of S1–S3, as disclosed in §4.1 and Appendix C.3. Frozen v1.6 scoring has now been applied unchanged to Sep H1: six models, 39 samples, and three repetitions per model. The additional checks include rule-variant rescoring and an implementation audit. The audit and new-season results are reported below in the responses to the dropper condition and validation request. Independent expert annotation is limited in the current submission.

One concrete example is the rule that accepts a dropper classification whenever a dropped executable is observed. This choice raises category precision by an average of 0.30 and recall by 0.26; removing it also changes the ordering of leading models. This demonstrates that the semantics of the scoring rules can substantially affect the results.

Response: The original exception passes 147 reports, representing 46.5% of the 316 category passes. Its average category-precision and category-recall increments are 0.301 and 0.262. Of the 147 reports, 29 pass only on runtime by-products: 24 PyInstaller extraction cases and five PowerShell policy-test scripts. A written executable subsequently appears as a process in 30 cases. External metadata corroboration is present for 134 of the 147 passes. Task reports do not contain hashes of written files.

We evaluated V0 (original), V7 (excluding runtime by-products and same-name files, and requiring external corroboration), and V3 (exception removed). V7 passes 102 reports. The S1–S3 and pooled F1 results are:

SeasonModelV0V7V3
S1GLM-5.20.758 (1)0.694 (1)0.614 (1)
S1Opus 4.80.744 (2)0.674 (2)0.608 (2)
S1Sonnet 50.684 (3)0.589 (3)0.524 (3)
S1Sol0.566 (4)0.566 (4)0.441 (4)
S1Luna0.510 (5)0.460 (5)0.352 (5)
S2Sonnet 50.718 (1)0.628 (2)0.561 (1)
S2GLM-5.20.698 (2)0.641 (1)0.556 (2)
S2Opus 50.695 (3)0.609 (3)0.517 (4)
S2Sol0.625 (4)0.582 (4)0.520 (3)
S2Luna0.581 (5)0.512 (5)0.421 (5)
S3Opus 50.840 (1)0.797 (1)0.681 (2)
S3Grok 4.60.795 (2)0.744 (2)0.691 (1)
S3GLM-5.20.764 (3)0.722 (3)0.644 (3)
S3Sonnet 50.730 (4)0.682 (4)0.558 (5)
S3DeepSeek0.674 (5)0.628 (5)0.563 (4)
S3Sol0.603 (6)0.590 (6)0.434 (6)
S3Luna0.564 (7)0.564 (7)0.399 (7)
PooledOpus 50.771 (1)0.708 (1)0.604 (2)
PooledGLM-5.20.740 (2)0.687 (2)0.606 (1)
PooledSonnet 50.711 (3)0.634 (3)0.548 (3)
PooledSol0.599 (4)0.580 (4)0.465 (4)
PooledLuna0.555 (5)0.517 (5)0.394 (5)

The new Sep H1 results are:

ModelV0V7V3K
Grok 4.60.845 (1/1/1)0.845 (1/1/1)0.735 (1/1/1)0.849 (1/1/2)
Opus 50.832 (2/2/2)0.832 (2/2/2)0.716 (2/2/2)0.846 (2/2/1)
Sonnet 50.807 (3/3/3)0.807 (3/3/3)0.660 (3/3/3)0.804 (3/3/3)
DeepSeek0.729 (4/5/4)0.729 (4/5/4)0.623 (4/4/4)0.741 (4/5/4)
Sol0.676 (5/4/6)0.676 (5/4/6)0.544 (5/5/6)0.690 (5/4/5)
Luna0.654 (6/6/5)0.645 (6/6/5)0.493 (6/6/5)0.636 (6/6/6)

On Sep H1, 154 delivered reports reach the dropper path; V0 passes 135 and V7 passes 134. V7 changes Luna’s second-repetition F1 from 0.672 to 0.644. External corroboration uses MalwareBazaar vendor fields and Triage signatures. Similarly, independent expert consensus annotations is limited in our submission.

The rules should be frozen before evaluating a new season, and their accuracy and coverage should be assessed against consensus annotations from independent security experts. (pp. 6–7, 19–20, 24)

Response: The v1.6 rules were developed using S1–S3. They have now been applied unchanged to Sep H1, with six models, 39 samples, and three repetitions per model. The rule audit separately examined the dropper condition, DLL-root rundll32 detection, network noise, and IOCs without traces. Correcting the latter three issues together changes 25 S1–S3 reports and group F1 by at most 0.011; GLM-5.2 and Opus 5 exchange second and third place in S2.

For Sep H1, the rule-variant results are mean F1 across three repetitions, with repetition ranks in parentheses:

ModelV0V7V3K
Grok 4.60.845 (1/1/1)0.845 (1/1/1)0.735 (1/1/1)0.849 (1/1/2)
Opus 50.832 (2/2/2)0.832 (2/2/2)0.716 (2/2/2)0.846 (2/2/1)
Sonnet 50.807 (3/3/3)0.807 (3/3/3)0.660 (3/3/3)0.804 (3/3/3)
DeepSeek0.729 (4/5/4)0.729 (4/5/4)0.623 (4/4/4)0.741 (4/5/4)
Sol0.676 (5/4/6)0.676 (5/4/6)0.544 (5/5/6)0.690 (5/4/5)
Luna0.654 (6/6/5)0.645 (6/6/5)0.493 (6/6/5)0.636 (6/6/6)

V0 is frozen v1.6; V7 adds exclusions for runtime artifacts and same-name files plus external corroboration; V3 removes the dropper exception; K corrects the three implementation issues together. Independent security expert evaluation cannot be completed given the limited time frame.

W2: The main score more closely measures reporting faithfulness to available evidence than the completeness of end-to-end evidence collection.

TL;DR: The main F1 evaluates reports against observed evidence and does not directly measure behavioral discovery completeness. Among 612 delivered reports, 422 (69%) contain at least one recognized behavior class, while technique recall is unjudgeable for the remaining 190. We now report delivery, behavioral activation, E2, and report scores together, with explicit treatment of missing scoring dimensions.

The denominator of technique recall consists of behaviors observed in the particular run. When no relevant behavior is observed, the dimension may be omitted. Delivery also requires only that the sample starts and produces telemetry, rather than that sample-specific malicious behavior is actually activated.

Response: Delivery and technique recall have distinct criteria: delivery verifies completion of the workflow with sample execution and telemetry, while technique recall measures whether the report covers the behavior classes observed in that run. Delivery does not require an observed malicious-behavior class. All 612 delivered tasks contain the sample’s own process, including 50 with process-only telemetry; 422 (69%) contain at least one of the 12 behavior classes. For the remaining 190 reports, technique recall is unjudgeable and excluded from the recall average. When a behavior class is observed but the report lists no techniques, technique recall is scored as zero; this occurs in 32 reports. We will clarify this distinction and report behavioral activation alongside delivery and report scores.

Consequently, a run that observes less may have a smaller recall denominator, and a high-scoring report does not necessarily correspond to comprehensive behavioral discovery. Although the paper includes a behavioral-capture metric, E2, it is only diagnostic and does not enter the main F1 score.

Response: E2 is diagnostic and does not enter F1. Across delivered runs in S1–S3, model–season mean E2 ranges from 0.919 to 0.974, with an overall median of 0.998; 80.2% have E2 of at least 0.9. The recorded per-report Spearman correlation between E2 and recall is −0.187 (p = 4 × 10⁻⁶). The 422 reports with an observed behavior class have P/R/F1 of 0.828/0.600/0.696; the other 190 have 0.848/0.557/0.673.

SeasonModelDeliv.PRMain F195% CIStrict F1E2 (deliv.)Activation
S1GLM-5.236/390.8520.6820.758[0.663, 0.839]0.6960.94722/36
S1Opus 4.834/390.7540.7340.744[0.648, 0.830]0.6460.96324/34
S1Sonnet 536/390.7530.6260.684[0.583, 0.777]0.6280.94821/36
S1Sol38/390.8930.4140.566[0.435, 0.684]0.5510.95723/38
S1Luna37/390.8670.3610.510[0.369, 0.631]0.4820.97422/37
S2Sonnet 534/400.8010.6510.718[0.623, 0.802]0.6100.95127/34
S2GLM-5.239/400.8190.6080.698[0.600, 0.785]0.6800.92131/39
S2Opus 537/400.8290.5980.695[0.608, 0.773]0.6430.94530/37
S2Sol34/400.8840.4840.625[0.491, 0.741]0.5320.92526/34
S2Luna36/400.8180.4500.581[0.453, 0.695]0.5230.94229/36
S3Opus 537/400.8620.8200.840[0.757, 0.912]0.7770.95024/37
S3Grok 4.633/400.8700.7320.795[0.711, 0.871]0.6560.93123/33
S3GLM-5.239/400.8110.7220.764[0.676, 0.846]0.7450.91926/39
S3Sonnet 535/400.7930.6760.730[0.630, 0.823]0.6390.93524/35
S3DeepSeek30/400.8640.5530.674[0.544, 0.791]0.5060.93220/30
S3Sol38/400.8740.4610.603[0.477, 0.716]0.5730.93325/38
S3Luna39/400.8280.4270.564[0.420, 0.688]0.5500.93325/39

Activation here denotes an observed class under the 12-class detector. E2 normalizes seven capture fields against maxima across all scored tasks on each sample, including invalid tasks and undelivered runs; a run uses its best task.

Furthermore, technique scoring covers only 23 predicates and 12 behavior classes. Among the 612 delivered reports, technique precision cannot be evaluated for 224 reports, and technique recall for 190. Omitting these dimensions can also give different models different effective scoring weights. Behavioral activation, evidence coverage, and report quality should be reported jointly, with an explicit policy for handling missing dimensions. (pp. 6, 14, 17–21)

Response: Per-report precision is the mean of judgeable category, technique, and IOC precision dimensions; per-report recall is the mean of judgeable category and technique recall dimensions. Reports with no judgeable dimension on a side are excluded from that side’s mean. Model–season F1 is the harmonic mean of the mean precision and mean recall, rather than the mean of per-report F1.

DimensionJudgeable reports
Category precision (category committed and a reference class exists)483 (79%)
Category recall (a reference class exists)554 (91%)
Technique precision (N6)388 (63%)
Technique recall (R2)422 (69%)
IOC precision (N1)611

Judgeable precision dimensions per report number 0/1/2/3 in 1/59/233/319 reports. Judgeable recall dimensions number 0/1/2 in 16/216/380 reports. Technique precision is judgeable for Sol 52/110 (47%), Luna 56/112 (50%), Opus 5 70%, GLM-5.2 73%, and Sonnet 5 75%. With missing dimensions scored as zero, the two GPT-5.6 tiers remain last in S1–S3 and each season’s top-three membership is unchanged. Restricting scoring to the 287 reports judgeable on all five dimensions leaves 8–25 reports per model–season cell; for example, Luna moves from seventh to third in S3.

Of 2,062 technique claims, 1,239 (60.1%) have no predicate; the share ranges from 40.3% to 71.6% across model–season cells. Delivery, activation, E2, and report scores are tabulated in the preceding response. The explicit missing-dimension policy appears in §4.1, Appendix C.1, and Appendix E.

Reviewer_geBtOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

SANDBOXLIVE evaluates LLM agents on end-to-end malware analysis: inspecting, detonating, and reporting on live samples. It uses four agent roles, a safety controller, and deterministic evidence-based scoring. Eight models are tested over 119 post-release Windows/Linux samples and 675 runs, with validation through rule audits, sensitivity analyses, rescoring, and trace-level failure studies.

Critical strengths
  • Important and underexplored evaluation setting using real Windows and Linux malware.
  • Substantial experimental effort: 119 samples, eight models, and 675 runs.
Critical weaknesses
  • The headline metric does not validly measure end-to-end malware-analysis quality. The main F1 is computed only over the 612 successfully delivered reports, excluding 63 of 675 runs, and evaluates each report against evidence collected by that same run. Behavior capture (E2) is only diagnostic and is not included. This choice materially affects conclusions: when undelivered runs count as zero, Sonnet 5 falls from first to third in S2 (0.718 to 0.610), while DeepSeek falls from fifth to seventh in S3 (0.674 to 0.506). Thus, a model can score well by faithfully summarizing sparse evidence, and the reported F1 does not establish end-to-end analysis quality.
  • Model comparisons are confounded by the tool interface and insufficient replication. Under native tool calling, Kimi’s delivery improves from 15/29 to 24/29 on the same samples; a separate native-interface S2 run delivers 39/40 reports with F1 0.737, exceeding the reported S2 leader at 0.718. Moreover, each model–sample pair is evaluated only once without confidence intervals. The only reruns changed several framework components and produced same-sample mean absolute F1 differences of 0.113–0.137, far exceeding the 0.023 gap among the top three S2 models. Consequently, the rankings cannot be confidently attributed to malware-analysis capability rather than interface, framework, or run variability.
Questions for authors
  1. How would model rankings change if delivery failures and behavior capture (E2) were included in the primary metric?
  2. How many dimensions and ATT&CK claims are unjudged per model, and how are empty or abstaining reports scored?
  3. Were the scoring rules validated against blinded expert judgments or an untouched season? If so, what is the agreement?
  4. Did you perform multiple independent runs for each model–sample pair under an unchanged harness? If so, how many repetitions were used, and how were confidence intervals computed?
6 comments

Rebuttal: We are grateful for the encouraging review and constructive feedback! With our best efforts, we conducted additional experiments and analysis to address your concerns. Additional results and clarifications will be added to the manuscript accordingly. We will carefully proofread and revise unclear parts in the updated version as well. Below we provide the point-by-point responses. For your convenience, we provided TL;DR summaries before lengthy detailed explanations to help you quickly grasp our main points.

W1: Design of the main evaluation (F1)

W1: The headline metric does not validly measure end-to-end malware-analysis quality. The main F1 is computed only over the 612 successfully delivered reports, excluding 63 of 675 runs, and evaluates each report against evidence collected by that same run. Behavior capture (E2) is only diagnostic and is not included. This choice materially affects conclusions: when undelivered runs count as zero, Sonnet 5 falls from first to third in S2 (0.718 to 0.610), while DeepSeek falls from fifth to seventh in S3 (0.674 to 0.506). Thus, a model can score well by faithfully summarizing sparse evidence, and the reported F1 does not establish end-to-end analysis quality.

Thank you for raising concerns about our design of F1.

TL;DR: The main F1 is designed based on realistic considerations (delivered reports, evaluation references and why E2 is not included) and it measures end-to-end analysis quality by evaluating the report quality. Faithful summarizing is not enough to get a high rank, while we also emphasize the benchmarking findings beyond F1. A new table shows more detailed comparison and supports our clarifications, and will be added to the revised version.

The detail of response is split into 6 points:

  1. Delivered reports. A failed run leaves no report that can be checked: of the 63 failed runs, only one has a report written by Analyst, and it never completed submission. We therefore report delivery separately (Sec. 4.3) and give strict F1 as a control (Appendix E), which counts undelivered runs as zero. These undelivered runs reflect delivery failures (e.g., Sonnet 5’s content-filter refusals and DeepSeek’s unstable tool calling) rather than analysis quality.
  2. Evaluation references. The category is checked against MalwareBazaar labels (Section 4.1, line 298). Techniques and IOCs have no configuration-independent ground truth, since a sample behaves differently under different sandbox configurations (line 311–312). And they mainly depend on the samples rather than the agent's choice given our harness. We therefore check them against the evidence of the run, which is “the most reasonable reference we can obtain for every run” (Appendix C.3, line 1031-1032).
  3. F1 does not include E2 for two reasons. First, E2 measures how many dynamic behaviours of the sample are captured, which almost depends on the sample and the environment. So it shows little about agent capabilities and it is almost unrelated to report quality. Second, since agent's choices about sandbox environment only have limited influence to behaviours due to the controller in safety harness, the E2 scores for different agents are roughly the same. So adding it into the main F1 does not show the difference between agent capabilities.
  4. Main F1 reflects the end-to-end analysis of a delivered run to a large extent. A delivered report is the product of the whole pipeline, i.e., Scout’s static triage, the detonation that Architect and Executor plan and run, and Analyst’s interpretation of the static and dynamic evidence into a category, techniques, and IOCs. F1 mainly evaluates the report quality (Sec. 4.2), which directly shows how reliable the agents analyze the malware end-to-end, i.e., interpreting the evidences beyond simply summarization.
  5. Simply restating the evidence does not score well. A report that only restates the evidence, without committing to a category or mapping the observed behaviors to techniques based on effective reasoning, loses recall: an abstention is skipped in precision but counts as zero in recall when a reference category or an observed behavior exists. For example, the two GPT-5.6 models commit to the fewest claims, thus Sol has the highest precision (0.874–0.893) in S1–S3, yet the two models have the lowest recall and F1.
  6. Ranking is only part of our benchmarking. Dynamic malware analysis is an important yet safety-critical task for agent, so delivering an informative and faithful report is not the end in our benchmarking. A lower F1 in our runs does not mean that a model is a weaker analyst in general (Appendix C.3, line 1060-1061). The analysis of failures and case studies (e.g., Sec. 4.3) are also very valuable, which further meet our goal, i.e., to expose where agents are unreliable (line 82-83) and better evaluate the end-to-end analysis quality beyond F1.

To further address your concern, we will add a joint presentation of delivery, main F1, strict F1 (undelivered runs assigned zero precision and recall), E2, and observed behavior-class technique in the revised version based on Table 10 in Appendix E:

SeasonModelDeliv.Main F1Strict F1E2 (deliv.)≥1\geq 1 technique
S1GLM-5.236/390.7580.6960.94722/36
S1Opus 4.834/390.7440.6460.96324/34
S1Sonnet 536/390.6840.6280.94821/36
S1Sol38/390.5660.5510.95723/38
S1Luna37/390.5100.4820.97422/37
S2Sonnet 534/400.7180.6100.95127/34
S2GLM-5.239/400.6980.6800.92131/39
S2Opus 537/400.6950.6430.94530/37
S2Sol34/400.6250.5320.92526/34
S2Luna36/400.5810.5230.94229/36
S3Opus 537/400.8400.7770.95024/37
S3Grok 4.633/400.7950.6560.93123/33
S3GLM-5.239/400.7640.7450.91926/39
S3Sonnet 535/400.7300.6390.93524/35
S3DeepSeek30/400.6740.5060.93220/30
S3Sol38/400.6030.5730.93325/38
S3Luna39/400.5640.5500.93325/39

Here, “≥1 technique” counts runs in which at least one of the 12 behavior classes is observed in the verified final task.

Several takeaways from the table to our response:

  1. Strict F1 changes the ranking mainly for models with lower delivery rates (Sonnet 5 in S2: 34/40; DeepSeek in S3: 30/40), which supports the first point: these changes come from delivery rather than report quality.
  2. E2 of different agents are roughly the same, as a support to the third point;
  3. The rates of observed technique mainly depend on the samples (S2 is generally higher than S3 and S1), as a support to the second point.

To sum up, we design the main F1 based on realistic considerations and it reflects end-to-end analysis quality while we also emphasize evaluations beyond F1. Additional details and clarifications will be added in the revised version.

W2: Tool interface and reruns

W2: Model comparisons are confounded by the tool interface and insufficient replication. Under native tool calling, Kimi’s delivery improves from 15/29 to 24/29 on the same samples; a separate native-interface S2 run delivers 39/40 reports with F1 0.737, exceeding the reported S2 leader at 0.718. Moreover, each model–sample pair is evaluated only once without confidence intervals. The only reruns changed several framework components and produced same-sample mean absolute F1 differences of 0.113–0.137, far exceeding the 0.023 gap among the top three S2 models. Consequently, the rankings cannot be confidently attributed to malware-analysis capability rather than interface, framework, or run variability.

Thank you for raising concerns about tool interface and reruns!

TL;DR: We already acknowledge that our results are based on our harness (e.g., interface, framework) in limitations. Additional details with further clarifications about the mentioned experiments will be added in the revised version. We also add a new season to check run variability as you concern.

Response:

  1. Models in main comparison use the same text-based interface. To clarify, we apply text-based protocol as default to all models in our harness (line 287-288), which is common practice for model-agnostic benchmarking of coding and cybersecurity agents (e.g., TerminalBench 2.0 [1], BountyBench [2]). So the ranking in main results is using the same standardized interface for fair comparison since different tool interfaces often lead to different performance. Moreover, We already acknowledge that the malware-analysis capability is evaluated under our harness and framework, which means that the ranking could change if they vary (Limitations, line 732).
  2. Kimi experiments do not hurt the ranking in main results. We report Kimi experiments in the appendix rather than the main table. First, We included Kimi K3 because it was one of the strongest open-weight coding models. However, its API was heavily loaded at the time, and calls frequently failed with null response. We also met exhausted API quota and wrong tool callings. So we turned to calling via coding plan endpoint, which was more stable, and we found that native interface mitigates the tool calling errors. In the end we report this finding in appendix, which does not hurt the fair comparison of main ranking.
  3. Two runs in Appendix F. For the S2 experiments, we found several design bugs in our harness in the first run. For example, a too strict and inflexible harness may lead to setup failures for some specific samples while agents have no rights to correct the wrong setting due to the controller. On the other hand, bugs in harness could mislead the agents. After the update, the harness was fixed for all other experiments in main results. To show that the experiments could be sensitive to harness changes, we present these experiments in appendix. We will add this clarification in the revised version.

Here is the details of Kimi comparisons:

RunEndpointModel IDDeliveredP / R / F1
Text protocol, S2api.moonshot.ai/v1kimi-k319/40 (15/29 on the first 29 samples)0.760 / 0.789 / 0.775
Native tool calling (fixed adapter), first 29 samples of S2api.moonshot.ai/v1kimi-k324/290.850 / 0.769 / 0.807
Native tool calling, coding plan, S2api.kimi.com/coding/v1k339/400.806 / 0.678 / 0.737

Regarding your concern about run variability, we add an additional experiment on a new season (September 1 as S4). Six models each completed three repetitions on 39 samples using the same framework, harness and evaluation to previous experiments. Related results are shown in our response to your Question 4.

References:

[1] M. A. Merrill et al., "Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces," ICLR, 2026.

[2] A. K. Zhang et al., "BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems," NeurIPS Datasets and Benchmarks Track, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.