- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes EVITRACE-DE, which constrains LLM-based detection-rule generation through evidence admission. Programmatic validators verify candidate predicates and event relations, admitted content enters P-DIR, and deterministic compilers generate KQL and Sigma rules. The framework incorporates execution feedback and local context to support rule revision and reduce token consumption. The authors report improvements in rule-generation quality, traceability, and efficiency on CTI-REALM-50 and AWS CTI-to-Sigma tasks.
Critical strengths
-
The framework directly links evidence verification to rule-writing eligibility and retains condition-level provenance through P-DIR, providing a useful design for limiting unsupported conditions and supporting analyst audits.
-
Targeted ablations of evidence admission and deterministic compilation support their contributions to rule-generation quality, providing empirical evidence for the core design.
Critical weaknesses
-
The paper presents programmatic validators as a key distinction from LLM-based self-correction. However, behavioral verification is described only as using normalized expressions and platform-semantic mappings, without explaining how the program determines whether CTI supports a candidate condition, how mapping resources are constructed, or how qualifiers, instance-specific constants, and conflicting evidence are handled. Required precondition generation, failure localization, and retries are also largely represented by unspecified functions such as instantiate and diagnose. Although the overall control flow is understandable, the rules that determine whether a candidate’s preconditions become Closed remain unclear.
-
The paper does not sufficiently explain how QueryPlan is represented, constructed, or how its composition logic is validated. The predicate/relation subset constraints in Equation 5 can restrict newly introduced atomic conditions, but do not by themselves guarantee preservation of Boolean composition or the complete detection logic. For example, A AND B and A OR B contain the same predicates but have different semantics. The compiler may also omit content unsupported by the target language, yet the projection policy and its effects on detection scope are unspecified. The renderer ablation supports the value of deterministic compilation, but does not resolve these semantic-boundary concerns; the Direct Generation comparison also does not isolate the contribution of P-DIR itself.
-
The experiments disable thinking for DeepSeek and Qwen and configure GPT-5 with low reasoning effort, without explaining these choices or testing whether the advantages persist under stronger reasoning configurations. The exact version or snapshot of DeepSeek-V4-Flash is also unspecified. Moreover, the Qwen and GPT experiments compare only against ReAct, while stronger workflows such as CRITIC and RulePilot, along with the key ablations, are evaluated primarily using DeepSeek. Consequently, the results do not establish whether the method retains a stable advantage over stronger baselines under more substantial reasoning budgets.
Questions for authors
-
How does behavioral verification programmatically determine whether CTI supports a candidate condition? Please explain how normalized expressions, platform-semantic mappings, and required preconditions are constructed, and provide a complete verification example.
-
How is the Boolean composition of QueryPlan constructed and validated? Does Equation 5 guarantee only constraints on the provenance of conditions, or full semantic preservation? When the compiler omits unsupported content, how are the effects of projection on detection logic checked and documented?
-
Please specify the exact version of DeepSeek-V4-Flash and explain why thinking was disabled and GPT-5(low) was used. Are there results under higher reasoning budgets, including comparisons against stronger baselines such as RulePilot using GPT or Qwen, showing that the advantages persist?