acceptodds
Withdrawn from ICLR 2027

Influential but Misaligned: LLM Evidence Drives Predictions Yet Misses What Institutions Classified

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

High-reliability security document classification requires knowing not only how accurately language models predict but also what evidence supports their decisions. Prior work has evaluated the predictive influence of model rationales and their agreement with human annotations, but research annotations alone cannot establish alignment with records of actual classification practice. We use portion markings in declassified documents released through the CIA Electronic Reading Room as a comparison reference. Portion markings record classification levels for document spans; where the highest-level rule applies, the highest-classified portions are directly linked to the document label. Using texts with classification markings removed, we evaluate the predictive influence and selection consistency of self-rationales from three open-weight language models and compare them with institutional markings in the portion-recovered subset. In a stratified sample, deleting the top self-rationale changed predictions 10.5–12.4 percentage points more often than length-matched random deletion. Rationale selection was stable under identical inputs but varied with prompt wording and model choice. In the portion-recovered subset, model rationales overlapped institutional portions less than a within-document random-position baseline (lift 0.65–0.75), and we found no evidence that deleting classified portions changed predictions more often than amount-matched random deletion. This divergence also left a positional and lexical signature. Llama rationales appeared near the beginning of documents more often than institutional portions and contained classification and secrecy vocabulary 26.6 percentage points more often in a sentence comparison matched for position and length. This pattern is consistent with a preference for sentences that overtly signal classification. The predictive influence of model rationales and their alignment with institutional span-level classification records therefore require separate evaluation.

Primary area:
interpretability and explainable AI
Keywords:
large language model rationales, security document classification, interpretability, institutional alignment, portion markings

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_n5RSOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper evaluates the evidence cited by three prompted LLMs when predicting the original security classification of declassified documents. It compares model quotations with recovered institutional portion markings and measures prediction changes after deleting selected text. Model quotations are behaviorally influential but overlap recovered institutional portions less often than randomly positioned spans of equal length.

Critical strengths
  • The paper is clearly written, with detailed descriptions of data processing and evaluation.
  • Historical portion markings provide an interesting external reference for evaluating model evidence.
  • The deletion experiments include useful length and position controls, alongside checks across prompts and models.
Critical weaknesses
  • Limited incremental insight. The main contribution applies existing rationale-evaluation procedures to a new domain. The institutional reference is potentially useful, but the current results do not convincingly establish a substantial broader insight beyond the limitations of the evaluated classifiers.
  • Classification performance complicates interpretation. Four-class accuracy is approximately 42–46%, below the 51% majority baseline, although macro-F1 is higher. It remains unclear whether evidence misalignment persists when the models make successful decisions. Please report alignment and deletion effects separately for correctly classified documents.
  • The institutional reference needs stronger validation. Portions are recovered in only 2,299 of 37,862 documents and cover a median of 8.9% of their text. Coordinate consistency does not establish correct semantic boundaries or representative recovery. A small manual audit and an analysis by recovery coverage would strengthen the central alignment result.
  • The diagnosis offers limited practical guidance. The lexical and positional analyses identify selection patterns, but do not isolate their causal role or test a remedy. A focused cue intervention or comparison using relevant classification guidance would help explain the failure and establish what practitioners can learn from it.
Questions for authors
  • Please see above.
Reviewer_YDdpOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The submission addresses the automatic classification of sensitive documents. The main contribution is a dataset of documents curated from the CIA Electronic Reading Room collection. The document annotations include their overall sensitivity level and localized information about document paragraphs. A subset also includes recovered institutional portions. The evaluation compares three ~70B models for global classification and provides more detailed results for Llama. The results indicate that the global task is challenging and that the model rationales align well with the human ones.

Critical strengths

The proposed dataset focuses on a challenging task with real-world applications. The proposed curation process makes sense, as does the evaluation setup. I appreciate the multifaceted evaluation using multiple metrics and different levels of analysis. However, the submission has multiple flaws that clearly outweigh its merits.

Critical weaknesses

The main weakness is the subpar quality of the implemented evaluation. Several aspects need substantial improvement for the paper to be accepted at a reputed venue such as ICLR. First, the authors cite several related works that tackle the same problem. Importantly, while the topic is quite niche, there are other works that use large AI models in closely related settings. For instance, https://www.mdpi.com/2076-3417/16/11/5661 and https://ieeexplore.ieee.org/document/10990101 . They should have been cited. Also, at least a subset of these works should have been used as baselines. Second, the evaluation uses only three similarly sized models, while a well-designed benchmark should include more models to assess the variability with model families and sizes. Also, the three tested models are outdated, leaving the question of how newer models would behave unanswered. Third, results show that even global classification is a hard problem. The experiments use a rather straightforward prompt, which could have been refined if an error analysis was carried. Beyond prompt optimization, the tested models could have been fine-tuned to adapt them to the task specificities, which do not seem well-handled by the pretrained models. Fourth, the comparison of the AI models and human results should have been deeper. For instance, it would be interesting to know if AI models are more or less conservative than humans, and how much the human assessments vary if different analysts read the documents.

A second important weakness is the lack of clarity in some of the submission’s aspects and main text self-containment. For instance, domain-specific terms, such as “institutional portion” and abbreviations should be defined the first time they are used. Second, the document selection criteria should be summarized and more clearly motivated. For instance, why were short and long documents removed from the collection, and how were the length bounds selected? Third, some of the metrics used in the experiments (random deletion change rate and excess change rate) are not that common and should have been formally defined.

Questions for authors

-Q1: How would baselines from the literature compare with the proposed approach?

-Q2: What is the error distribution across classes, and how would you use it to improve results?

-Q3: What would be the effect of prompt optimization, for instance, including definitions of the sensitivity levels?

-Q4: How would the tested models behave if fine-tuned on a subset of the proposed dataset?

-Q5: How would newer versions of the tested models and models from other families behave?

Reviewer_LisROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether the rationales generated by LLMs when classifying declassified CIA documents match actual institutional classification records. The authors find that while the rationales selected by the models do affect their final predictions, they rarely align with the specific text segments originally redacted by the CIA. The setup is clever and backed by thorough experiments. Overall, the paper offers a solid empirical look at how LLM explanations compare against real-world, high-stakes human decisions.

Critical strengths
  1. Evaluating LLM explanations against real-world institutional records, rather than standard crowd-sourced annotations, is a clever and grounded approach.
  2. The experiments are thorough. The authors test their claims across a large dataset using deletion tests, varied prompts, and multiple models, which makes the empirical results quite solid.
Critical weaknesses
  1. Model Performance: All three tested models underperform a trivial "always-Secret" baseline. Because the models are essentially failing at the primary task, it is hard to know how much weight to give the misalignment results. I wonder if these findings would still hold for a model that actually performs the task well, or if the analysis was restricted only to the subset of correctly classified documents.
  2. Limited Novelty: The main findings substantially overlap with prior work on the faithfulness and human alignment of LLM rationales [1,2]. While the use of institutional records is interesting, I am not convinced that the paper offers sufficiently new insights beyond applying existing approaches to a different setting.
  3. Real-world classification decisions rely heavily on external context, historical timing, and internal agency policies that aren't explicitly written in the text. The current evaluation doesn't seem to account for this external knowledge gap, which makes it hard to tell if the models are failing due to poor reasoning or simply because they lack the necessary off-page context.

[1] Evaluating Human Alignment and Model Faithfulness of LLM Rationale (Fayyaz et al., arXiv 2024)

[2] A Systematic Comparison between Extractive Self-Explanations and Human Rationales in Text Classification (Brandl and Eberle, TrustNLP 2026)

Questions for authors

Please see the weakness above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.