acceptodds
Under review as a conference paper at ICLR 2027

MASTRIA: Automated Threat Modeling and Risk Analysis for LLM Multi-Agent Systems

Abstract

Assessing the security of LLM-driven multi-agent systems (MASs) is challenging because execution paths, task-delegation relationships, and attack surfaces can evolve dynamically through model reasoning and inter-agent interactions, limiting the effectiveness of conventional threat-modeling approaches. Existing threat modeling frameworks for agentic systems address some of these challenges but still rely on manually authored architecture descriptions, diagrams, or contextual inputs, making their effectiveness sensitive to the completeness and accuracy of the provided system representation. To address this gap, we present MASTRIA, a three-phase automated threat modeling and analysis approach for MASs. Given the source code and an inventory of protected assets, MASTRIA first constructs a system model by identifying entry and exit points, generating an enriched data-flow diagram of agents, tools, data stores, and inter-agent communication, and deriving trust boundaries and segmentation. Next, it applies three complementary threat-identification modules: a deterministic rule-based OWASP Agentic Security Initiative (ASI) module, an agentic OWASP ASI module, and an agentic MAS taxonomy module. The identified threats are linked to relevant assets, attack paths, and adversarial techniques, then semantically de-duplicated and validated using both deterministic rules and LLM-based modules. Finally, MASTRIA assigns likelihood- and impact-based risk scores and generates mitigation recommendations. The effectiveness of MASTRIA was evaluated on six open-source multi-agent applications, recovering human-identified threats with 97.5% recall and 94.8% precision. The consistency of the results across repeated runs further demonstrates the feasibility of reproducible, implementation-grounded threat modeling for MASs.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM, Multi-agent systems, AI Agents, Threat Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NZnoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MASTRIA, an automated framework for threat modeling and risk analysis within multi-agent systems powered by large language models. The framework constructs a system model by analyzing source code, encompassing agents, tools, data flows, and trust boundaries. It then employs a combination of rule-based analysis and LLM-driven methods to identify and verify potential security threats. Finally, it assesses the likelihood and impact of these threats and provides corresponding mitigation recommendations. The objective is to automate threat modeling while ensuring that the analysis results remain traceable to the system's actual implementation.

Critical strengths

none

Critical weaknesses
  1. The paper only compares the identified threats with human annotated ground truth, but does not test them with real attacks. It is unclear whether these threats can actually happen in real-world systems.

  2. The paper lacks comparison with existing threat modeling methods or a simple LLM baseline. It is hard to know how much improvement comes from the proposed framework.

  3. The paper includes risk assessment and mitigation recommendations, but the experiments mainly focus on threat identification. The accuracy of risk scores and the effectiveness of suggested mitigations are not evaluated.

Questions for authors
  1. How do the authors ensure the generated DFDs are correct and complete? Since the following threat analysis are based on these diagrams, any missing or incorrect connections may affect the final results. Have the authors considered evaluating the accuracy of generated DFDs against the actual system architecture?

  2. The performance drops a lot when changing the backbone model from GPT-5.5 to Kimi-2.7. Can the authors explain why there is such a large difference? Does this mean the framework is highly depend on the LLM being used?

  3. The evaluated applications have relatively small number of agents. How well can MASTRIA work on larger and more complex multi-agent systems? Especially when agents and their interactions can change during runtime, will the framework still be able to identify the threats correctly?

5 comments
AuthorsOpenReview ↗

Thank you for the comments.

W1: Test Real attacks:

To address this concern, we empirically validated a subset of the threats identified by MASTRIA by implementing them as concrete attacks. Specifically, for four of the evaluated applications (Trip Planner, Instagram, Email Assistant, and HalluCheck), we selected twelve high-risk threat records emitted by MASTRIA and built one attack per threat record. Each attack places an attacker payload at the entry point named in the threat record: a scraped web page, a search result, an email, a dataset row, or one of the user’s own answers, depending on the threat. All attacks rely on prompt injection and LLM manipulation techniques. Each attack was executed three times without modification to account for stochastic behavior, for a total of 36 executions. Of these, 31 succeeded, yielding an attack success rate (ASR) of 86.1%. Seven of the twelve attacks succeeded in all three executions, and the remaining five succeeded in two of three.

We note, however, that actual exploitability (i.e., successful execution of a threat) is not the evaluation target of threat modeling, whether traditional or automated as in MASTRIA. We follow NIST SP 800-30 (similar to STRIDE, PASTA,OWASP, ASTRIDE, STRIDE-AI), under which a threat is a potential adverse event against an asset, whereas exploitability is a property of a specific vulnerability in a specific deployment and is typically assessed in a subsequent red-teaming phase.

W2: Comparison with existing threat modeling methods or a simple LLM baseline:

In response to this comment, we added three baselines (updated results are presented Section 4, Table 2 of the revised paper). All baselines use GPT-5.6-sol (high reasoning):

  1. Naive LLM: A single prompt that generates all threat records in one response.
  2. Iterative LLM: Repeated LLM calls (20 iterations), each receiving the previously identified threats and requesting additional threats.
  3. LLM Harness: A general-purpose coding agent (Codex) that navigates and analyzes the application's code iteratively.

All baselines receive the same inputs as MASTRIA (source code and asset inventory), along with the ASI taxonomy and threat-record schema. Their outputs were labeled by the same human reviewers, following the same guidelines and against the same ground truth, which was fixed before running the baselines. The baselines use a newer model (GPT-5.6-sol) than MASTRIA's backbone (GPT-5.5). The average results across seven applications, including the newly added LangGraph application, are:

MethodAvg. RecallAvg. PrecisionTotal Cost ($)
Naive LLM0.2310.8642.42
Iterative LLM0.4150.81255.47
LLM Harness0.2100.9104.50
MASTRIA0.9690.95252.81

The results demonstrate that MASTRIA's primary advantage is threat coverage (recall). Although the baselines achieve relatively high precision (0.812--0.910), they identify only 9--33 threats per application, compared with 36--85 for MASTRIA. The strongest baseline, Iterative LLM, achieves a recall of 0.415, compared with MASTRIA's 0.969, at a comparable total cost ($55.47 vs. $52.81). These findings indicate that neither repeated prompting nor iterative code navigation is sufficient to achieve MASTRIA's threat coverage.

W3: Evaluating risk assessment and mitigation recommendations:

In the revised paper, we provide the evaluation of the risk scoring. Three experts independently rated all 373 correctly identified threats. MASTRIA’s ratings exactly matched the reconciled expert rating for 329 of them (88.2%). Agreement between MASTRIA and the experts (κ = 0.76) is comparable to agreement among the experts themselves (κ = 0.75), and disagreements are mostly over-ratings (7.8%) rather than under-ratings (4.0%) (see the new results in the Result Section 4, Table 3). For mitigations, MASTRIA follows standard threat-modeling practice: each identified threat is linked to established countermeasures according to the threat properties; in MASTRIA we use the OWASP Agentic AI Threats and Mitigations guidance based on the Agentic Security Initiative (ASI) threat categorization. We chose community-vetted guidance over LLM-generated defenses deliberately, so that every recommendation is traceable and auditable. The correctness of this linkage depends on the ASI mapping, which we evaluate: 315 of 373 correctly identified threats (84.5%) received a complete and correct mapping, and the remaining 58 a partial one (see Table 2). To further demonstrate how mitigation can affect MASTRIA results, we hardened two applications with the attached mitigation recommendation MASTRIA attached to a same of threats, and measured the effect by running MASTRIA on the new variants (further details are in Appendix F, Applied mitigations).

AuthorsOpenReview ↗

Thank you for the comments.

Q1: Ensure the generated DFDs are correct and complete:

The revised manuscript evaluates this directly against the source code of all seven applications (previous six applications plus the new fully evaluated LanGraph application) (presented in the Results Section 4, Table 2). MASTRIA recovered every one of the 139 components with its trust level, at perfect precision. It also recovered every one of the 355 edges with its boundary-crossing annotation, at 0.986 precision. Entry points reached 0.977 precision and 0.920 recall, and exit points 0.892 precision and 0.961 recall(presented in the Results Section 4, Table 1). These results confirm that the code-derived model is a reliable foundation for the threat analysis built on it. Correctness is also enforced by design (i) the Scanner extracts components and connections deterministically from the source code’s ASTs, (ii) elements it cannot resolve are raised as explicit handoff items rather than guessed, (iii) agentic updates are accepted only if the code and model elements they cite actually exist, and (iv) downstream, the Threat validator checks every attack-path step against a real DFD edge, so modeling errors cannot silently produce unsupported threats. The high threat recall (0.969) further suggests that the few remaining entry and exit-point omissions had limited effect on the final results.

Q2: Performance drops when changing from GPT-5.5 to Kimi-2.7:

The gap reflects the capability of the underlying LLM in the agentic stages, not a dependence on a specific model vendor. Kimi-2.7 focuses on agentic coding, whereas GPT-5.5 has strong cybersecurity capabilities. In addition, Kimi-2.7 has a limited context window (256K tokens), which constrains its reasoning over large inputs; as noted in the paper, this limitation prevented us from evaluating the RealEstate application. As a case in point, GLM-5.3, which has strong cybersecurity reasoning and a larger context window, performs similarly to GPT-5.5 on the common subset: 0.916 precision and 0.949 recall, compared with 0.944 and 0.978.

AuthorsOpenReview ↗

Thank you for the comments.

Q3: larger multi-agent systems:

Our initial selection of six applications was intended to cover diverse MAS domains, topologies (hierarchical, sequential, and LLM-routed loop), capabilities (local and remote MCP servers, RAG, and code execution), and effects ranging from advisory outputs to irreversible external actions. The derived system models of these applications range from 12 to 33 DFD nodes. We also believe that our evaluation which includes applications with 6-7 agents, consistent with representative multi-agent configurations. Pareek et al. (CHI 2026) identify 2-6 agents as typical ensemble sizes, while TAMAS (ACL 2026) demonstrates multi-agent security risks using only four agents. Yang et al. (2026) emphasize that multi-agent security risks arise primarily from interactions and trust boundaries rather than agent count alone. Kim et al. (Nature Machine Intelligence, 2026) show that increasing the number of agents does not necessarily improve performance and may introduce coordination overhead and error amplification. Thus, our evaluation captures representative and security-relevant multi-agent configurations, while spanning diverse architectures, interactions, and threat scenarios. Nevertheless, we acknowledge that scalability to substantially larger systems warrants further investigation. To partially address the reviewer’s comment, we extended the evaluation with 14 additional applications. During the rebuttal period, we fully annotated one of these applications, GitHub MCP application, implemented in LangGraph and computed both its precision and recall; the updated results in the paper include this application. For the remaining 13 new applications (see Table below) given the time constraints of the rebuttal period, we report only precision (94.30%). Measuring their recall requires independent manual threat modeling by the human annotators to construct the ground truth, which we have started and will be completed and included in the final version of the paper.

App nameFrameworkSourceTopology# AgentsToolsPrecision
Personal Assistant (router)CrewAIRIFT-BenchRouter + final-checker loop (10 cycles)42 local MCP servers (11 tools), local writer tool97.1%
Personal Assistant (orchestrator)CrewAIRIFT-BenchOrchestrator loop (10 cycles)42 local MCP servers (11 tools), local writer tool91.4%
Finance (orchestrator)CrewAIRIFT-BenchOrchestrator loop (8 cycles)51 remote MCP server (market data), 3 local MCP servers (7 tools)97.3%
Travel (router)CrewAIRIFT-BenchRouter + final loop (10 cycles)64 local MCP servers (16 tools), web search, local writer tool81.3%
Code-Exec AgentCrewAIRIFT-BenchSingle agent, uncapped reasoning loop1Code interpreter (in-process sandbox), remote GitHub MCP (read-only), web search (local MCP)95.8%
Surprise TripCrewAIGitHub (crewAI-examples)Sequential3Web search, web scraping100.0%
Marketing StrategyCrewAIGitHub (crewAI-examples)Sequential3Web search, web scraping100.0%
RecruitmentCrewAIGitHub (crewAI-examples)Sequential4Web search, web scraping, LinkedIn Selenium tool100.0%
Job PostingCrewAIGitHub (crewAI-examples)Sequential (5 tasks)3Web search, RAG website search, file read97.0%
News AgentCrewAIGitHub (abdulwasea89/News-Agent)Sequential4Web search, web scraping, file write100.0%
Lead GeneratorCrewAIGitHub (Folken2/lead-generator-ai)Sequential4Web search, web scraping100.0%
Mutual Funds AnalyserCrewAIGitHub (Sudhir-Dharan/Mutual-Funds-Analyser)Sequential4Web search, web scraping100.0%
Travel (single agent)CrewAIRIFT-BenchSingle agent14 local MCP servers, web search, plan-document writer86.7%
13 applications94.3%

Regarding runtime dynamics, MASTRIA models interactions that the code permits rather than a single observed execution. A delegation that an LLM chooses only at runtime is therefore still represented as an edge, given the code allows it. LLM selected handoffs are modeled as inter-agent edges annotated as LLM-driven. Tools that a remote MCP server exposes only at runtime are modeled through an MCP client node and a tool-poisoning boundary crossing. The Finance application demonstrates both cases. Where the structure cannot be determined statically, as with dynamic dispatch or agents constructed in loops, the Scanner explicitly marks the region as incomplete. The Agentic analyzer then recovers the missing elements from the code (Appendix B.2). Components with no evidence in the code at all, such as agents registered purely from external configuration, remain a limitation that we already stated in Section 5.

Reviewer_oLsfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an automated framework for security threat modeling of Multi-Agent Systems. The central idea is to perform threat modeling from the application's source code and a collection of protected assets. A scanner first builds a framework-independent model of agents, tools, data flows, entry and exit points, and trust boundaries. It then traces possible attack paths and combines OWASP-based rules with two LLM analyses to identify threats. Duplicate and unsupported findings are removed. Finally, it scores risks by likelihood and impact and then recommends mitigations.

To evaluate the framework, three security experts independently examined six open-source MAS applications using OWASP threat-modeling guidance and after resolving 28 disagreements, they established a reference set of 316 threats. Using GPT-5.5, MASTRIA then analyzed the same applications and generated 325 threat records. It identified 308 of the 316 expert-established threats and missed eight, achieving 97.5% recall. Human reviewers judged 266 records fully correct and 42 correct but incompletely labeled; 17 were false positives. This resulted in 94.8% precision.

Critical strengths
  1. Thoughtful combination of deterministic and LLM analysis: The paper does not rely entirely on LLM judgements. The inclusion of AST-based extraction, intermediate-representation construction, rule-based identification is sensible. In particular, tracking threats through assets, entry points, attack paths, and exit points is a useful engineering contribution.

  2. Evaluation with human expertise: The three security reviewers independently constructed threat inventories, with a reported mean pairwise F1 score of 0.911. This is significantly stronger than evaluating generated threats solely with another LLM. The constructed ground truth itself is a valuable and novel contribution.

  3. Useful ablation: Removing the MAS-taxonomy component reduced recall from 0.975 to 0.478. Additionally, authors show that disabling deduplication substantially harms reported precision. Ablations provide significant insight into how important the framework’s components are.

Overall, the paper has a solid engineering foundation, and the detailed threat-model representation and evaluation set are important contributions.

Critical weaknesses
  1. The novelty contribution needs more emphasis: The paper repeatedly emphasizes deriving threat models from source code rather than manual system descriptions or architecture diagrams. However published ATAG work (Gandhi et al., 2026) contains an Agent Modeler that derives application topology from source code. It extracts agents, tasks, tools, and interconnections from CrewAI implementations. Authors already include ATAG in MASTRIA’s references as a source of an evaluated application, but methodological relationship to MASTRIA is not properly addressed. I do not believe that makes MASTRIA not-novel. The intermediate representation, trust boundaries, human-created evidence-validation process are meaningful extensions.

  2. The ground truth establishes agreement with expert threat modeling, not threat validity: The ground truth was constructed through manual threat modeling using the OWASP threat-modeling process (Conklin, n.d.) and agentic-threat guidance (OWASP GenAI Security Project, 2025a). MASTRIA also itself uses OWASP taxonomies and questions (OWASP GenAI Security Project, 2025a, 2025b). Consequently, there is substantial overlap between what the framework was meant to enumerate and what the evaluation considers a threat. This is not inherently invalid. Threat taxonomies are useful and the three human-constructed threat sets are a strength. However, agreement with expert-generated hypothetical threats is not the same as establishing the real-world validity of those threats. A concrete concern appears in Appendix C.1, Table 20. Some deterministic rules produce confirmed threats on broad structural conditions. For example, a tainted LLM-driven agent with a write-capable tool can produce a tool-misuse finding. Other rules concern absent identity controls even when those controls are marked as unobservable. Likewise, the example in Appendix B.6, Table 19 describes a user request steering an agent's web-scraping tool as a tool-misuse and goal-manipulation threat. It is not immediately clear that the described behavior constitutes unauthorized misuse rather than ordinary use of an intentionally exposed capability. Threat modeling does not require reproducing every attack, and I would not reject this paper simply because it lacks a red-team benchmark. However, independent checks on a representative set of findings, including secure negative examples, would substantially improve confidence.

  3. Lack of evaluation of system modeling accuracy: One of the main contributions of MASTRIA is automatically constructing system models from source code. However, Section 4 only evaluates the final threat identification results and does not measure how accurately these system components and their connections are identified. Although Section 3.3 describes validation checks, these mainly verify whether the generated information follows the required structure and references valid evidence, rather than whether the system model is complete and correct. Since errors in system modeling could lead to missed or incorrect threats, comparing the generated models against manually verified architectures would strengthen this contribution.

  4. Risk assessment and mitigation recommendations are not evaluated: Although risk assessment and recommendations are important parts of MASTRIA (Section 3.5), the experiments mainly focus on threat identification performance. The authors do not evaluate whether the assigned risk levels agree with human expert judgements, leaving a significant part of the proposed framework without validation. Too many references to the appendix. The paper frequently refers readers to tables and sections in the appendix to explain and clarify important parts of the methodology. This makes the paper difficult to follow as I had to constantly switch between the main text and the appendix. While detailed explanations can be placed in the appendix, the main paper should provide enough information to understand the key ideas of the paper.

  5. Lack of code and dataset: The authors claim to release the MASTRIA implementation and the manually curated threat dataset, but neither the code nor the datasets appears to be provided with the submission. This makes it difficult to verify the implementation and reproduce the reported experimental results despite reproducibility being one of the paper’s stated contributions.

References

Conklin, L. (n.d.). Threat modeling process (historical). OWASP Foundation. https://community.owasp.org/Threat_Modeling_Process

Gandhi, P. A., Tayouri, D., Shukla, A., Ifland, B., Elovici, Y., Puzis, R., & Shabtai, A. (2026). ATAG: AI-agent application threat assessment with attack graphs. In Proceedings of the ACM Asia Conference on Computer and Communications Security (pp. 805–819). https://doi.org/10.1145/3779208.3785380

OWASP GenAI Security Project. (2025a, February 17). Agentic AI – threats and mitigations. https://genai.owasp.org/resource/agentic-ai-threats-and-mitigations/

OWASP GenAI Security Project. (2025b, April 23). Multi-agentic system threat modeling guide (Version 1.0). https://genai.owasp.org/resource/multi-agentic-system-threat-modeling-guide-v1-0/

Questions for authors

Q1. How does MASTRIA’s framework differ from automatic topology extraction in ATAG? Can the authors provide a component-level comparison?

Q2. Can the authors report quantitative accuracy of the automatically reconstructed DFDs, entry/exit points, and trust boundaries?

Q3. How closely do the predicted risk ratings agree with human experts? What is the run-to-run variance in the final threat sets and risk rankings with the same model and input?

7 comments
AuthorsOpenReview ↗

Thank you for the comments.

W1/Q1: The novelty contribution needs more emphasis

We thank the reviewer for highlighting ATAG (Gandhi et al., 2026) and recognizing MASTRIA's methodological extensions. We agree that source-code-based topology extraction is not novel by itself and clarified the differences from ATAG in the Related Work section. Although both frameworks extract application topology from source code, their components serve different purposes, and therefore, making a direct one-to-one comparison is difficult. At the system-modeling level, ATAG's Agent Modeler extracts agents, tools, and interconnections into MulVAL facts. MASTRIA’s Scanner and Model Builder instead construct a framework-agnostic intermediate representation (IR) and derive a security-aware DFD incorporating entry and exit points, trust boundaries, action authority, and protected assets. The Agentic Analyzer and IR Validator further address implementation ambiguities through evidence-grounded enrichment. Thus, MASTRIA’s contribution at this stage is the transformation of extracted topology into a security-aware, evidence-grounded system model for subsequent threat analysis and modeling. We position this as an extension beyond topology extraction, while acknowledging the extraction capability already demonstrated by ATAG. The following component-level comparison focuses on system modeling, excluding subsequent threat enumeration.

ComponentATAG: Agent ModelerMASTRIA: System-Modeling
InputsApplication source code (mainly CrewAI)Application source code & asset inventory
Topology extractionExtracts agents, tasks, tools, and interactions using a deterministic AST scanner
RepresentationEncodes topology as MulVAL factsConstructs an explicit, framework-neutral intermediate representation (IR)
Model structureAgent graph, with tools, LLMs, and memory represented as attributesDFD with agents, tools, and data stores as nodes, as well as trust levels and flow types
Entry and exit pointsIdentifies input/output agents and external interactions with these agentsBuilds separate catalogs recording entry mechanisms (agent and tool), reachability to LLMs (complete path), and exit-action authority
Trust boundariesFlags external interactions. Explicit trust segmentation is not providedFlags external interactions. In addition, assigns trust levels and zones and categorizes boundary crossings
Protected assetsAsset categorization is not providedAssociates inventory assets with model elements and entry/exit points
Unresolved implementation factsNo evidence-validation mechanismRecords handoff questions; an Agentic Analyzer proposes answers, and an IR Validator checks supporting code evidence
AuthorsOpenReview ↗

Thank you for the comments.

W2: The ground truth establishes agreement with expert threat modeling, not threat validity.

We agree that expert agreement alone does not establish practical threat validity. MASTRIA performs pre-deployment threat modeling to identify plausible security threats, consistent with NIST SP 800-30’s definition of threats in terms of their potential to cause adverse impact. We therefore complement expert evaluation with attack demonstrations and negative examples. Although shared OWASP guidance aligns the scope and vocabulary of MASTRIA and the experts, three experts constructed application-specific ground-truth threats without access to MASTRIA’s outputs. Moreover, given the same taxonomy, the newly added Naive LLM, Iterative LLM, and LLM Harness baselines recover 21-41% of expert threats, compared with MASTRIA’s 373/385 (96.9%; updated Table 2). This suggests that shared taxonomy alone is insufficient to explain MASTRIA’s coverage. To assess feasibility, we implemented twelve attacks corresponding to selected high-risk findings across Trip Planner, Instagram, Email Assistant, and HalluCheck. Each attack used prompt injection or LLM manipulation at the specified entry point and was executed three times without modification. Overall, 31/36 executions succeeded (86.1% aggregate ASR), with every attack succeeding at least twice. These results support feasibility for the selected findings under the evaluated conditions (Appendix F, Table 32). For secure negative examples, we reran MASTRIA on variants hardened with OWASP-recommended mitigations. In Email Assistant, adding either validation and filtering step or a human-in-the-loop gate before draft creation led the ASI identifier to reject the relevant cascading-hallucination and autonomy threat candidates emitted by R11 and R13. In Trip Planner, replacing the eval calculator tool that carries a known code-execution vulnerability with an arithmetic-only evaluator resulted in no code-execution threats at all (further details are in Appendix F, Applied mitigations). Regarding Table 20, R5 requires both attacker-controlled input reaching an LLM-driven agent and a tool capable of write or code execution owned by that agent. Tool availability alone does not trigger the rule, without a path from untrusted input, R5 emits no finding. Therefore, the breadth of the structural conditions for R5 is no broader than the breadth of OWASP T2 "Tool Misuse". For the example in Table 19, we agree that fetching a user-provided product URL can constitute ordinary use in the application. The security concern is that an adversary can steer the agent to abuse the tool, for example a prompt injection that manipulates the agent to use the tool to scrape an attacker control website, operating with the user's credentials, hijacking the legitimate task and usage of the tool to the attacker's needs. Our adversary model includes attackers as a legitimate user of the application (Section 3.1). This threat has been demonstrated as part of the attack exploitation in the attack evaluation: an internal address supplied through the product-details input was fetched (Table 32, ID 016).

W3/Q2: Lack of evaluation of system modeling accuracy:

The revised manuscript evaluates the system modeling accuracy against the source code of all seven applications (previous six applications plus the new fully evaluated LanGraph application); presented in the Results Section 4, Table 1. MASTRIA recovered every one of the 139 components with its trust level, at perfect precision. It also recovered every one of the 355 edges with its boundary-crossing annotation, at 0.986 precision. Entry points reached 0.977 precision and 0.920 recall, and exit points 0.892 precision and 0.961 recall. These results confirm that the code-derived model is a reliable foundation for the threat analysis built on it.

AuthorsOpenReview ↗

Thank you for the comments.

W4/Q3: Risk assessment and mitigation recommendations are not evaluated:

In the revised paper, we provide the evaluation of the risk scoring. Three experts independently rated all 373 correctly identified threats. MASTRIA’s ratings exactly matched the reconciled expert rating for 329 of them (88.2%). Agreement between MASTRIA and the experts (κ = 0.76) is comparable to agreement among the experts themselves (κ = 0.75), and disagreements are mostly over-ratings (7.8%) rather than under-ratings (4.0%) (see the new results in the Result Section 4, Table 3). For mitigations, MASTRIA follows standard threat-modeling practice: each identified threat is linked to established countermeasures according to the threat properties; in MASTRIA we use the OWASP Agentic AI Threats and Mitigations guidance based on the Agentic Security Initiative (ASI) threat categorization. We chose community-vetted guidance over LLM-generated defenses deliberately, so that every recommendation is traceable and auditable. The correctness of this linkage depends on the ASI mapping, which we evaluate: 315 of 373 correctly identified threats (84.5%) received a complete and correct mapping, and the remaining 58 a partial one (see Table 2). To further demonstrate how mitigation can affect MASTRIA results, we hardened two applications with the attached mitigation recommendation MASTRIA attached to a same of threats, and measured the effect by running MASTRIA on the new variants (further details are in Appendix F, Applied mitigations).

W5: Lack of code and dataset:

We acknowledge that the MASTRIA implementation and the manually curated threat dataset were not included in the submission. Their public release is currently subject to institutional procedures. We commit to publicly releasing both the complete MASTRIA implementation and the curated threat dataset, along with the necessary documentation and instructions for reproducing our experimental results, upon acceptance of the paper. We explicitly state this commitment in the revised manuscript.

Reviewer_2uRROpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents MASTRIA, an automated threat modeling and risk analysis framework for LLM-based multi-agent systems. Given source code and an inventory of protected assets, MASTRIA constructs a system model, identifies and validates potential threats using deterministic and LLM-based analysis, and assigns risk scores with mitigation recommendations. The framework is evaluated on six open-source multi-agent applications, reporting 97.5% recall and 94.8% precision against human-identified threats.

Critical strengths
  1. Code-driven threat modeling. MASTRIA automatically constructs system representations from source code, reducing reliance on manually authored architecture descriptions or diagrams. This makes the approach potentially more practical for analyzing multi-agent systems.

  2. Structured validation of LLM outputs. The IR validator constrains agentic outputs to a predefined set of operations, helping improve the consistency of the generated threat models.

Critical weaknesses
  1. Lack of comparative baselines. The paper does not provide comparisons with existing threat modeling approaches, such as ASTRIDE and STRIDE-AI, or simpler alternatives, such as general-purpose coding agents. Without these baselines, it is difficult to assess the benefits of MASTRIA over existing methods.

  2. Insufficient evaluation of reproducibility. The paper does not clearly demonstrate whether the complete MASTRIA pipeline was executed multiple times. Repeated end-to-end evaluations would provide stronger evidence for the claimed consistency and reproducibility of the results.

  3. Limited evaluation scale. The evaluation includes only six open-source applications, which may not adequately represent the diversity and complexity of real-world multi-agent systems. This limits the generalizability of the reported results.

  4. Lack of empirical validation of identified threats. The reported precision and recall are based on human-identified threats rather than threats confirmed through actual attack executions. Therefore, the results primarily measure agreement with the human-identified threat inventory and do not necessarily demonstrate that the identified threats are practically exploitable.

Questions for authors

Was the ground truth finalized before running MASTRIA, or was it revised after inspecting MASTRIA's outputs? Please clarify how the ground truth was constructed and whether it was kept independent of MASTRIA's predictions.

3 comments
AuthorsOpenReview ↗

Thank you for the comments.

W1: Lack of Comparative Baselines

In response to this comment, we added three baselines (updated results are presented Section 4, Table 2 of the revised paper). All baselines use GPT-5.6-sol (high reasoning):

  1. Naive LLM: A single prompt that generates all threat records in one response.
  2. Iterative LLM: Repeated LLM calls (20 iterations), each receiving the previously identified threats and requesting additional threats.
  3. LLM Harness: A general-purpose coding agent (Codex) that navigates and analyzes the application's code iteratively.

All baselines receive the same inputs as MASTRIA (source code and asset inventory), along with the ASI taxonomy and threat-record schema. Their outputs were labeled by the same human reviewers, following the same guidelines and against the same ground truth, which was fixed before running the baselines. The baselines use a newer model (GPT-5.6-sol) than MASTRIA's backbone (GPT-5.5). The average results across seven applications, including the newly added LangGraph application, are:

MethodAvg. RecallAvg. PrecisionTotal Cost ($)
Naive LLM0.2310.8642.42
Iterative LLM0.4150.81255.47
LLM Harness0.2100.9104.50
MASTRIA0.9690.95252.81

The results demonstrate that MASTRIA's primary advantage is threat coverage (recall). Although the baselines achieve relatively high precision (0.812-0.910), they identify only 9-33 threats per application, compared with 36-85 for MASTRIA. The strongest baseline, Iterative LLM, achieves a recall of 0.415, compared with MASTRIA's 0.969, at a comparable total cost ($55.47 vs. $52.81). These findings indicate that neither repeated prompting nor iterative code navigation is sufficient to achieve MASTRIA's threat coverage.

Regarding ASTRIDE and STRIDE-AI, we do not report direct numerical comparisons because these approaches differ from MASTRIA in scope, input requirements, and implementation (code) availability. ASTRIDE has no public implementation. Moreover, ASTRIDE relies heavily on three specially fine-tuned vision-language models. Without access to their weights or implementation, reproducing the approach would require an unvalidated approximation, making a numerical comparison potentially misleading. STRIDE-AI extends STRIDE to AI-specific threats, incorporating OWASP LLM Top 10 and MITRE ATLAS within an auditor-guided assessment workflow. Unlike MASTRIA, its emphasis is on LLM security assessment and it does not explicitly address agentic behaviors such as tool use, orchestration, and inter-agent interactions, nor does it specify an automated procedure for deriving threat models directly from source code. We therefore discuss ASTRIDE and STRIDE-AI as related work rather than evaluating them as automated baselines.

W2: Insufficient evaluation of reproducibility.

We have extended the evaluation to assess the reproducibility of MASTRIA with six independent end-to-end executions for each of the seven applications (presented in Section 4; full results in Table 30 in the Appendix). All runs used identical source code, asset inventories, prompts, model configurations, ground truth, and evaluation procedures, and all generated threat records were evaluated by the same human annotators. Across the seven applications, mean precision ranges from 93.0% to 100.0% (standard deviation 0.0-4.9 pp), and mean recall ranges from 90.1% to 96.1% (standard deviation 2.8-6.5 pp). The standard deviation in the number of generated threat records ranges from 1.0 to 2.9 per application. The largest variation occurs in the classification of valid threats as Correct or Correct (Incomplete), with standard deviations reaching 9.1 and 7.1 pp, respectively. The scanner and system-model builder are deterministic, so repeated executions on identical inputs produce identical intermediate representations and base system models. The observed variability therefore arises from the LLM-assisted stages: system-model enrichment, agentic threat identification, and validation. In the revised version, we clarify that Table 2 reports the results of an earlier single execution, which is not one of the six new runs summarized in Table 30.

ApplicationRecall (%)Precision (%)
Email Assistant92.7 ± 4.993.0 ± 2.8
HalluCheck96.1 ± 2.8100.0 ± 0.0
Trip Planner94.1 ± 6.594.8 ± 4.0
RealEstate91.4 ± 5.296.9 ± 4.9
Finance95.2 ± 4.098.3 ± 2.8
Instagram92.1 ± 5.195.0 ± 4.6
GitHub MCP90.1 ± 4.094.9 ± 2.8
AuthorsOpenReview ↗

Thank you for the comments.

W3: Limited evaluation scale.

Our initial selection of six applications was intended to cover diverse MAS domains, topologies (hierarchical, sequential, and LLM-routed loop), capabilities (local and remote MCP servers, RAG, and code execution), and effects ranging from advisory outputs to irreversible external actions. The derived system models of these applications range from 12 to 33 DFD nodes.

Nevertheless, to address this comment, we extended the evaluation with 14 additional applications. During the rebuttal period, we fully annotated one of these applications, GitHub MCP application, implemented in LangGraph and computed both its precision and recall; the updated results in the paper include this application. It also extends the evaluation beyond the framework used by the original applications, which supports the framework-agnostic design of MASTRIA’s intermediate representation. For the remaining 13 new applications (see Table below; RIFT-Bench == https://arxiv.org/abs/2606.23927 ; crewAI-examples == https://github.com/crewAIInc/crewAI-examples/tree/main/crews/) given the time constraints of the rebuttal period, we report only precision (94.30%). Measuring their recall requires independent manual threat modeling by the human annotators to construct the ground truth, which we have started and will be completed and included in the final version of the paper.

App nameFrameworkSourceTopology# AgentsToolsPrecision
Personal Assistant (router)CrewAIRIFT-BenchRouter + final-checker loop (10 cycles)42 local MCP servers (11 tools), local writer tool97.1%
Personal Assistant (orchestrator)CrewAIRIFT-BenchOrchestrator loop (10 cycles)42 local MCP servers (11 tools), local writer tool91.4%
Finance (orchestrator)CrewAIRIFT-BenchOrchestrator loop (8 cycles)51 remote MCP server (market data), 3 local MCP servers (7 tools)97.3%
Travel (router)CrewAIRIFT-BenchRouter + final loop (10 cycles)64 local MCP servers (16 tools), web search, local writer tool81.3%
Code-Exec AgentCrewAIRIFT-BenchSingle agent, uncapped reasoning loop1Code interpreter (in-process sandbox), remote GitHub MCP (read-only), web search (local MCP)95.8%
Surprise TripCrewAIGitHub (crewAI-examples)Sequential3Web search, web scraping100.0%
Marketing StrategyCrewAIGitHub (crewAI-examples)Sequential3Web search, web scraping100.0%
RecruitmentCrewAIGitHub (crewAI-examples)Sequential4Web search, web scraping, LinkedIn Selenium tool100.0%
Job PostingCrewAIGitHub (crewAI-examples)Sequential (5 tasks)3Web search, RAG website search, file read97.0%
News AgentCrewAIGitHub (abdulwasea89/News-Agent)Sequential4Web search, web scraping, file write100.0%
Lead GeneratorCrewAIGitHub (Folken2/lead-generator-ai)Sequential4Web search, web scraping100.0%
Mutual Funds AnalyserCrewAIGitHub (Sudhir-Dharan/Mutual-Funds-Analyser)Sequential4Web search, web scraping100.0%
Travel (single agent)CrewAIRIFT-BenchSingle agent14 local MCP servers, web search, plan-document writer86.7%
13 applications94.3%

W4: Lack of empirical validation of identified threats.

To address this concern, we empirically validated a subset of the threats identified by MASTRIA by implementing them as concrete attacks. Specifically, for four of the evaluated applications (Trip Planner, Instagram, Email Assistant, and HalluCheck), we selected twelve high-risk threat records emitted by MASTRIA and built one attack per threat record. Each attack places an attacker payload at the entry point named in the threat record: a scraped web page, a search result, an email, a dataset row, or one of the user’s own answers, depending on the threat. All attacks rely on prompt injection and LLM manipulation techniques. Each attack was executed three times without modification to account for stochastic behavior, for a total of 36 executions. Of these, 31 succeeded, yielding an attack success rate (ASR) of 86.1%. Seven of the twelve attacks succeeded in all three executions, and the remaining five succeeded in two of three.

We note, however, that actual exploitability (i.e., successful execution of a threat) is not the evaluation target of threat modeling, whether traditional or automated as in MASTRIA. We follow NIST SP 800-30 (similar to STRIDE, PASTA,OWASP, ASTRIDE, STRIDE-AI), under which a threat is a potential adverse event against an asset, whereas exploitability is a property of a specific vulnerability in a specific deployment and is typically assessed in a subsequent red-teaming phase.

AuthorsOpenReview ↗

Thank you for the comments.

Q1: clarify how the ground truth was constructed:

The ground truth was constructed before MASTRIA was run and was kept independent of its outputs. Three experts independently threat-modeled each application, following the OWASP Threat Modeling Process and the OWASP ASI guide, without access to any MASTRIA or baseline output. The 31 disagreements among them were reconciled before the first pipeline run, and the reconciled set was not modified afterwards. MASTRIA’s records were then labeled in a separate pass. We will state this protocol explicitly in Section 4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.