- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether evidential support for generated tokens can be decoded from LLM internal representations. The authors construct controlled two-part questions from financial filings, selectively removing supporting evidence to create labels without human annotation. A linear probe trained on residual-stream activations achieves 0.862 within-answer AUC across 493 evaluable answers, outperforming next-token entropy and the Lookback Lens baseline. Additional experiments examine token-level localization, distinguish evidential support from factual correctness, and investigate effects of context length and network depth.
Critical strengths
-
The paper investigates an important research question: whether information about evidence availability is encoded in LLMs' internal representations. This is an interesting and important direction that could motivate further research into understanding how LLMs use contextual information. However I do have some issues with the analysis, see W1.
-
The paper is well written, clearly organized, and easy to follow. The authors explain their methodology and experimental findings clearly, making the paper accessible and easy to understand.
Critical weaknesses
-
To my understanding, the probe learns to assign a low support score to text for which evidence was not available in the context. Nowadays, LLMs generate much of their text from parametric knowledge, which means the probe could flag a large number of tokens. However, this is not particularly useful, since the real challenge then becomes distinguishing between factually incorrect tokens and correct tokens generated from parametric knowledge, which may still be valuable to the user. Maybe I am missing something, but I find the practical value of the proposed method unclear. If the authors could also provide a way to make this distinction (on top of flagging ungrounded tokens), their approach would be considerably more valuable.
-
Even if I am missing something regarding the practical utility of the method, the empirical scope of the paper remains quite narrow, relying on a highly artificial experimental setup. The experiments are conducted on a single dataset and only two LLMs, with most evaluations performed on one model. This is considerably less comprehensive than what I would expect from an empirical paper at ICLR. I suggest extending the evaluation to additional datasets and domains, potentially using the same data construction methodology, as well as a broader range of LLMs to assess the generalizability of the findings.
-
To be more convinced that the probe actually learns to distinguish grounded from ungrounded tokens, I would like to see some evidence of generalization beyond the specific two-part question construction. For example, testing the same probe on standard, single-question answers from financial reports (with proper annotations for grounded and ungrounded tokens), without retraining, would strengthen the authors' claim about really learning to distinguish grounded from ungrounded tokens
Questions for authors
-
Connecting to W1: Given that the probe detects evidence availability rather than factual correctness, what practical applications do the authors envision, particularly when LLMs generate correct information from parametric knowledge?
-
Connecting to W3: Would the probe generalize to a different question format, such as single-question answers from the same financial reports, without retraining? This would help establish whether it learns evidential support rather than patterns specific to the two-part construction.