- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
GRID constructs CTI article-graph training pairs through traceable graph extraction and graph-conditioned article rewriting, then post-trains an LLM with an offline task bank of multi-select and regex-based rewards. It evaluates the resulting extractor on 249 CTI articles from five sources against CTI graph-construction systems and post-training variants, including rewriting, curriculum ordering, open-weight data generation, and cross-backbone task-bank reuse.
Critical strengths
-
The paper turns a difficult open-ended extraction reward into auditable local tasks without reducing the final objective to trivial multiple-choice training.
-
The evaluation is broader and more transparent than is typical for this application.
-
The task-bank formulation converts open-ended KG extraction into efficiently verifiable tasks, substantially reducing reliance on expensive online LLM-as-judge rewards while enabling supervision reuse.
Critical weaknesses
-
The Task-bank and End2End Reward methods differ substantially in training data, batch sizes, and optimization steps, with End2End receiving only 13 RL steps. Consequently, the results demonstrate cost-constrained effectiveness rather than the inherent superiority of the proposed reward formulation. A controlled comparison under matched training budgets is needed.
-
Section 3.4 repeatedly evaluates models on the same 50-article subset and reports the best-performing checkpoints. This introduces potential selection bias, particularly since the improvement from article rewriting is only 0.005 F1. Checkpoints should be selected on a separate validation set.
-
The proposed rewriting strategy removes security facts that are not captured by the automatically extracted KG, potentially discarding valid information due to extraction errors. This may introduce incomplete supervision and reinforce the initial extractor's omissions. The authors should evaluate how much factual information is lost during rewriting and its impact on downstream KG completeness.
Questions for authors
-
For the RQ3 results in Table 3 and Figure 6, was the fixed 50-article evaluation subset used to choose the reported best checkpoint, or was checkpoint 100 chosen independently before inspecting this subset? If the latter, please state the pre-specified selection rule; if not, a held-out validation protocol would materially increase my confidence in the rewriting and curriculum conclusions.
-
Were the primary Task-bank Reward checkpoint at 280 steps and the Table 1 comparison selected without repeatedly inspecting the 249-article benchmark?