acceptodds
Withdrawn from ICLR 2027

VeriIR-Agent: Risk-Aware and Auditable Planning for LLM-Based Incident Response

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models can generate remediation recommendations for cybersecurity incident response, but natural-language outputs are difficult to verify field by field and to audit trade-offs among recovery benefits, execution risks, and business impacts. This paper proposes VERIIR-AGENT, which decomposes response planning into three separately evaluable layers: generation, selection, and verification quality. For generation, typed action records replace free-form responses, followed by deterministic model-independent checks of evidence, commands, policies, completeness, and attack paths. For selection, a multi-objective ranking method considers recovery benefits, evidence support, execution risks, business impacts, and rollback capability over a fixed candidate pool. For verification, a conditional framework decomposes safety guarantees into “coverage gaps” and “false passes on covered properties,” corresponding to checker scope and precision. Experiments use 2,219 incident-response cases while holding the model, test cases, retrieval context, and standard decoding parameters constant, comparing free-form generation, prompt-based structuring, and schema-constrained decoding. Schema constraints increase schema validity from 46.2% to 98.0% and completeness from 61.8% to 90.8%, while action-type matching increases only from 72.7% to 75.1%. On 4,438 candidate actions, risk-aware ranking reduces unsafe selection from 17.6% to 10.9% and high-impact selection from 43.8% to 35.7%, while increasing rollback coverage from 21.3% to 31.8%; recovery relevance decreases from 83.1% to 80.2% as the associated cost. These results show that structural compliance, selection preferences, and verification guarantees should be evaluated separately rather than collapsed into a single “reliability” metric, and none should be equated with successful real-world execution or production-grade safety.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Cybersecurity Incident Response, Large Language Models, Structured Generation, Deterministic Verification, Risk-Aware Decision Making, Multi-Objective Ranking

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_6uUYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies LLM-based incident-response recommendations. It introduces VeriIR-Agent, which combines graph-conditioned typed action generation, deterministic field-level checks, and risk-aware ranking. It also provides a conditional analysis of verification coverage and false passes. The experiments compare output control settings on 2,219 cases and ranking objectives over 4,438 fixed candidate actions.

Critical strengths
  • it is interesting to distinguish structural compliance, selection quality, and verification guarantees. The evaluation includes some reasonable controls: raw outputs are assessed before repair, and both ranking objectives use the same candidates, This helps separate formatting improvements from changes in selection.
Critical weaknesses
  • the main experimental details are not clear enough to assess the results. Eq2 introduces five scoring terms without adequately explaining their computation or providing the actual weights. It is also unclear whether the unsafe and high-impact labels are established independently of the scores used for ranking. I cannot tell whether the gains reflect better recommendations or mainly better performance on the same proxies being optimized.
  • The data sources, candidate construction, and controlled-run configuration also need clearer documentation. Section 5.2 explicitly acknowledges that the historical configuration does not fully specify the newer experiments.
  • I am not convinced that the demonstrated contribution goes much over enforcing output structure and changing ranking preferences. Schema constraints mainly improve validity and completeness, while risk-aware ranking reduces evaluator-defined risk at some cost to recovery relevance.
  • The theoretical analysis also provides limited additional support. Prop1 assumes reliable mandatory checks, and Eq3 follows from a union bound. Neither coverage gaps nor check false-pass probabilities are measured.
Questions for authors

Q1: How are the five ranking scores computed, and what weights are used? Are the unsafe and high-impact labels established independently of those scores?

Q2: Where do the cases and references labels come from, and how are the two candidates per case constructed? Please provide the actual controlled-run configuration.

Q3: How does the method compare with selecting the highest-recovery candidate subject to the fixed risk and rollback requirements with the same candidate pool and independently assessed outcomes?

Reviewer_rS5NOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper aims to make AI-generated security incident responses easier to check and less risky to use; it organizes model's suggestions into a consistent format, checks for potential problems, and chooses actions based on their expected benefits and risks. The evaluation focuses on offline recommendations rather than actual execution. Results on 2k+ cases show that the outputs become much easier to process and that the selected actions are less often labeled unsafe, with a small drop in recovery relevance.

Critical strengths

[S1] The paper does a good job separating well-formatted outputs from genuinely better actions. The controlled comparison shows large improvements in output validity and completeness, but only small gains in action-type matching and command validity. This distinction is useful, as better formatting alone should not be taken as evidence of better decisions.

Critical weaknesses

[W1] The improvements in risk-aware selection are not fully convincing without a clearer explanation of the evaluation labels. The ranking method directly considers execution risk, business impact, and rollback, which closely overlap with the reported evaluation metrics. It is unclear how these labels and scores are obtained or whether they are assessed independently. This makes it hard to tell how much the improvements reflect truly better decisions.

[W2] The evaluation is still quite limited for assessing the practical value of the proposed system. The ranking experiment considers only two candidates per case, and there is no actual execution or rollback testing. The checker is also not evaluated against independently verified unsafe actions, and over 90% of admitted actions in the historical action-generation subset still carry warnings. The safety bound is a straightforward conditional argument, with neither its coverage gaps nor false-pass probabilities measured. These limitations leave the actual safety improvement uncertain.

Questions for authors
  1. How are the unsafe and high-impact labels assigned, and are they independent of the scores used by the ranking method?
  2. What are the exact ranking weights, and how are the individual scoring components calculated?
  3. Is there any independent evaluation showing that the newly selected actions in the 16.3% of changed cases are actually safer or more effective?
  4. How does the system handle cases where all candidate actions are blocked, and how are these cases accounted for in the ranking evaluation?
Reviewer_ZkW6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VeriIR-Agent separates LLM-based incident response into structured generation, risk-aware selection, and verification quality. Typed action records support deterministic checks, while a multi-objective score ranks a fixed candidate pool. A conditional analysis separates checker coverage gaps from false passes. Experiments on 2,219 cases compare output-control mechanisms and ranking objectives, while explicitly limiting claims about real-world execution safety.

Critical strengths
  1. Separating machine-checkable output, action prioritization, and verification scope addresses a meaningful reliability problem in LLM-based incident response.

  2. Fixed-case, shared-candidate comparisons help isolate selection effects. The manuscript appropriately distinguishes structural validity from execution success and acknowledges the limits of deterministic checks.

Critical weaknesses
  1. Contribution scope. Integrating typed outputs, checks, and ranking is useful, but the methodological advance is modest. The standard union-bound analysis characterizes assumptions more directly than it establishes a new verification mechanism.

  2. Selection evidence. Two fixed candidates per case and first-action evaluation provide a controlled selection test, but limited evidence about multi-step response quality. Conclusions should track this scope.

  3. Independent safety assessment. Ranking includes risk and impact terms closely related to evaluation criteria. Their relationship to evaluation labels needs clarification before improvements can be interpreted as independent safety gains.

  4. Matched alternatives and verification. A simpler risk-aware ranking on identical candidates would clarify the formulation's added value. Empirical evidence on checker false passes and coverage gaps would also strengthen the proposed verification-quality account.

Questions for authors
  1. What is the principal insight gained from integrating the established mechanisms?
  2. Which conclusions extend beyond the fixed-candidate first-action test?
  3. How are candidate scores constructed relative to evaluation labels?
  4. What do simpler risk-aware baselines and checker audits establish?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.