- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
Paper studies whether LLMs distinguish between a record that supports neither of two claims and a record that supports both. The authors build synthetic records where swapping entity roles changes the evidence state while keeping the same words. They show that the native yes/no answers distinguish absent from conflicting evidence poorly, while linear probes on hidden states do it almost perfectly. They then test steering along the probe directions and transfer to a harder task with source and time scope.
Critical strengths
-
The question of whether models separate missing from conflicting evidence is relevant for RAG and evidence reporting.
-
The stimulus construction is clean, and controlling for the exact word set rules out simple lexical shortcuts.
-
The authors include reasonable controls, such as random labels, text baselines and matched scalar decoders.
Critical weaknesses
The main finding is expected.
That linear probes recover more than the model answer is already known from prior work on latent knowledge and truth directions. The paper applies this to a new label set, but I do not see a new insight about how models process evidence.
The setup is very limited.
All experiments use short synthetic records with invented entities, a single yes/no token, and three 7B models. The test sets are small and probes are near ceiling, so it is hard to judge differences. I am not sure the results tell us anything about real documents, longer contexts or free generation.
The baselines are weak.
The scalar comparator is a classifier on a single output margin, so it is expected to lose to a classifier on the full hidden state. Simple prompting changes, few-shot examples, or asking the model to state the evidence state directly are not tested, although they may close the gap at the output level.
The causal results are mixed.
Steering works within one question for Mistral but is weak for Qwen. Directions do not transfer across questions, and for Qwen a plain answer direction works better than the state direction. Transfer to the source-scoped task works only for Mistral. So the causal claims are narrow, and the paper does not explain why the models differ.
The writing is hard to follow.
The text introduces many custom terms and reports results mostly as raw counts. A clearer presentation of the main result would help.