- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies LLM-based incident-response recommendations. It introduces VeriIR-Agent, which combines graph-conditioned typed action generation, deterministic field-level checks, and risk-aware ranking. It also provides a conditional analysis of verification coverage and false passes. The experiments compare output control settings on 2,219 cases and ranking objectives over 4,438 fixed candidate actions.
Critical strengths
- it is interesting to distinguish structural compliance, selection quality, and verification guarantees. The evaluation includes some reasonable controls: raw outputs are assessed before repair, and both ranking objectives use the same candidates, This helps separate formatting improvements from changes in selection.
Critical weaknesses
- the main experimental details are not clear enough to assess the results. Eq2 introduces five scoring terms without adequately explaining their computation or providing the actual weights. It is also unclear whether the unsafe and high-impact labels are established independently of the scores used for ranking. I cannot tell whether the gains reflect better recommendations or mainly better performance on the same proxies being optimized.
- The data sources, candidate construction, and controlled-run configuration also need clearer documentation. Section 5.2 explicitly acknowledges that the historical configuration does not fully specify the newer experiments.
- I am not convinced that the demonstrated contribution goes much over enforcing output structure and changing ranking preferences. Schema constraints mainly improve validity and completeness, while risk-aware ranking reduces evaluator-defined risk at some cost to recovery relevance.
- The theoretical analysis also provides limited additional support. Prop1 assumes reliable mandatory checks, and Eq3 follows from a union bound. Neither coverage gaps nor check false-pass probabilities are measured.
Questions for authors
Q1: How are the five ranking scores computed, and what weights are used? Are the unsafe and high-impact labels established independently of those scores?
Q2: Where do the cases and references labels come from, and how are the two candidates per case constructed? Please provide the actual controlled-run configuration.
Q3: How does the method compare with selecting the highest-recovery candidate subject to the fixed risk and rollback requirements with the same candidate pool and independently assessed outcomes?