- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper evaluates the evidence cited by three prompted LLMs when predicting the original security classification of declassified documents. It compares model quotations with recovered institutional portion markings and measures prediction changes after deleting selected text. Model quotations are behaviorally influential but overlap recovered institutional portions less often than randomly positioned spans of equal length.
Critical strengths
- The paper is clearly written, with detailed descriptions of data processing and evaluation.
- Historical portion markings provide an interesting external reference for evaluating model evidence.
- The deletion experiments include useful length and position controls, alongside checks across prompts and models.
Critical weaknesses
- Limited incremental insight. The main contribution applies existing rationale-evaluation procedures to a new domain. The institutional reference is potentially useful, but the current results do not convincingly establish a substantial broader insight beyond the limitations of the evaluated classifiers.
- Classification performance complicates interpretation. Four-class accuracy is approximately 42–46%, below the 51% majority baseline, although macro-F1 is higher. It remains unclear whether evidence misalignment persists when the models make successful decisions. Please report alignment and deletion effects separately for correctly classified documents.
- The institutional reference needs stronger validation. Portions are recovered in only 2,299 of 37,862 documents and cover a median of 8.9% of their text. Coordinate consistency does not establish correct semantic boundaries or representative recovery. A small manual audit and an analysis by recovery coverage would strengthen the central alignment result.
- The diagnosis offers limited practical guidance. The lexical and positional analyses identify selection patterns, but do not isolate their causal role or test a remedy. A focused cue intervention or comparison using relevant classification guidance would help explain the failure and establish what practitioners can learn from it.
Questions for authors
- Please see above.