acceptodds
Under review as a conference paper at ICLR 2027

EviTrace-DE: Evidence-Gated Generation of Traceable and Executable Detection Rules from CTI with LLM Agents

Abstract

Detection engineering converts attack behaviors described in cyber threat intelli- gence (CTI) into executable detection rules over security telemetry, but it remains expert-intensive and manual. Tool-use LLM agents can retrieve reports, inspect schemas, and query telemetry, but retrieval alone does not prevent unsupported conditions or event relations from entering the final rule, producing false or unexe- cutable rules. We study this failure mode as the evidence-commit gap and present EVITRACE-DE, an evidence-gated framework that separates open-ended LLM reasoning from rule construction through explicit evidence verification, enabling accurate and traceable rule generation. An LLM agent proposes candidate predi- cates and event relations, and programmatic validators verify candidate evidence and close the corresponding behavioral, schema, telemetry, or relational precondi- tions. Only candidates that pass all required verifications are admitted to P-DIR, an intermediate representation backed by evidence that binds each admitted con- dition to its supporting evidence. Deterministic compilers then generate KQL and Sigma rules using only the predicates and event relations admitted into P-DIR. The evidence-verification state also provides each LLM call with only the context needed for the current verification step, avoiding repeated full-history prompts. In CTI-REALM-50, EVITRACE-DE improves the normalized score from 0.537 to 0.752 and KQL F1 from 0.496 to 0.804, achieves a valid-submission rate of 100% and reduces the mean token usage per task by 98.7% compared to ReAct. In an independent AWS CTI-to-Sigma task, adapting verification to threat reports and API field definitions improves API-call F1 by 35.9% over a ReAct-style agent using the same model.

Primary area:
foundation or frontier models, including LLMs
Keywords:
detection engineering, cyber threat intelligence, large language model agents, evidence admission, rule-writing control

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7cRGOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EVITRACE-DE, which constrains LLM-based detection-rule generation through evidence admission. Programmatic validators verify candidate predicates and event relations, admitted content enters P-DIR, and deterministic compilers generate KQL and Sigma rules. The framework incorporates execution feedback and local context to support rule revision and reduce token consumption. The authors report improvements in rule-generation quality, traceability, and efficiency on CTI-REALM-50 and AWS CTI-to-Sigma tasks.

Critical strengths
  1. The framework directly links evidence verification to rule-writing eligibility and retains condition-level provenance through P-DIR, providing a useful design for limiting unsupported conditions and supporting analyst audits.

  2. Targeted ablations of evidence admission and deterministic compilation support their contributions to rule-generation quality, providing empirical evidence for the core design.

Critical weaknesses
  1. The paper presents programmatic validators as a key distinction from LLM-based self-correction. However, behavioral verification is described only as using normalized expressions and platform-semantic mappings, without explaining how the program determines whether CTI supports a candidate condition, how mapping resources are constructed, or how qualifiers, instance-specific constants, and conflicting evidence are handled. Required precondition generation, failure localization, and retries are also largely represented by unspecified functions such as instantiate and diagnose. Although the overall control flow is understandable, the rules that determine whether a candidate’s preconditions become Closed remain unclear.

  2. The paper does not sufficiently explain how QueryPlan is represented, constructed, or how its composition logic is validated. The predicate/relation subset constraints in Equation 5 can restrict newly introduced atomic conditions, but do not by themselves guarantee preservation of Boolean composition or the complete detection logic. For example, A AND B and A OR B contain the same predicates but have different semantics. The compiler may also omit content unsupported by the target language, yet the projection policy and its effects on detection scope are unspecified. The renderer ablation supports the value of deterministic compilation, but does not resolve these semantic-boundary concerns; the Direct Generation comparison also does not isolate the contribution of P-DIR itself.

  3. The experiments disable thinking for DeepSeek and Qwen and configure GPT-5 with low reasoning effort, without explaining these choices or testing whether the advantages persist under stronger reasoning configurations. The exact version or snapshot of DeepSeek-V4-Flash is also unspecified. Moreover, the Qwen and GPT experiments compare only against ReAct, while stronger workflows such as CRITIC and RulePilot, along with the key ablations, are evaluated primarily using DeepSeek. Consequently, the results do not establish whether the method retains a stable advantage over stronger baselines under more substantial reasoning budgets.

Questions for authors
  1. How does behavioral verification programmatically determine whether CTI supports a candidate condition? Please explain how normalized expressions, platform-semantic mappings, and required preconditions are constructed, and provide a complete verification example.

  2. How is the Boolean composition of QueryPlan constructed and validated? Does Equation 5 guarantee only constraints on the provenance of conditions, or full semantic preservation? When the compiler omits unsupported content, how are the effects of projection on detection logic checked and documented?

  3. Please specify the exact version of DeepSeek-V4-Flash and explain why thinking was disabled and GPT-5(low) was used. Are there results under higher reasoning budgets, including comparisons against stronger baselines such as RulePilot using GPT or Qwen, showing that the advantages persist?

Reviewer_ZdK5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The research work uses LLM agents to convert Cyber Threat Intelligence (CTI) into traceable detection rules. It attempts to reduce the risk of unsupported conditions entering rules despite access to relevant evidence. Programmatic validators detect proposed predicates and event relations before admitting them into a provenance-linked representation. Their evaluation result on benchmark datasets reflects an improvement of F1 score to .804.

Critical strengths
  1. Evaluation result on CTI-REALM-50 shows that the method improves KQL F1 from 0.496 to 0.804 over ReAct and achieves a 100% valid-submission rate.

  2. Condition level evidence path helps the analysis to trace through the compilation of its admitted object. This path supports the inspection of CTI, Schema, telemetry.

  3. Research work is dealing with an interesting direction, specially I like how they wanted to bridge the evidence between investigation and executable rule generation. This is helpful and timely.

Critical weaknesses
  1. Baseline compraison is missing from the manuscript. I know the research work has mentioned about some of this work in the manuscript (such as, [D]) but evaluation result would be helpful. Also it would be very helpful if they can report their tool's performance on the reported paper's dataset (such as. [C])

A. https://www.usenix.org/system/files/usenixsecurity26-cai.pdf B. https://dl.acm.org/doi/epdf/10.1145/3696410.3714798 C. https://arxiv.org/pdf/2511.12224 D. https://www.ndss-symposium.org/wp-content/uploads/2026-s199-paper.pdf

  1. What is the robustness of the proposed approach? The manuscript currently has not investigated that.

  2. Some of the analysis is very hard to interpret and not comprehensive. In Table 1, they haven't tested with other LLMs for CRITIC and RulePilot. Having the complete result will tell us the performance of those baseline as well. For example, LLMCloudHunter used OpenAI’s GPT-4o for their evaluation. It might be the case that the research work would see the improvement of result if they apply the same LLM used in those paper while comparing the baseline.

Also Table 1 has many columns but they didn't discuss how good they are across those modalities.

Minor:

  1. Figure 2 is hard to read. Text are very small.
Questions for authors
  1. Can the manuscript discuss more on how to extend it to a different platform. I read a small discussion on this in Section 6 but it is not clear. It would be helpful if you can provide more details on this.

  2. How does evidence references help supported conditions to identify the uncovered stages?

  3. Extend the analysis with other LLMs.

  4. Have you investigated the robustness of the proposed approach?

Reviewer_iJ46OpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EviTrace-DE, an evidence-gated framework for generating executable detection rules from CTI using LLM agents. The framework verifies candidate conditions before admitting them into an intermediate representation (P-DIR), from which KQL and Sigma rules are deterministically compiled.

Critical strengths

The paper addresses an important challenge in ensuring the correctness and traceability of LLM-generated detection rules.

The proposed method substantially improves detection performance and token efficiency over existing agent-based baselines.

The paper investigates the contributions of evidence admission, deterministic compilation, and context management.

Critical weaknesses

To me, I'm not sure about the novelty part. The novelty seems limited, as the method mainly integrates LLM prompting with existing rule-based verification and compilation techniques, making it more of an engineering effort than a novel research contribution.

Questions for authors

It would be much better if the authors could further explain the novelty part.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.