- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether LLMs confuse repeated records with independent evidential support even when provenance is explicitly supplied. It introduces the Explicit Evidence Groups task, in which each source denotes one independent observation, multiple records from the same source are declared redundant, and the correct answer is determined by the number of independent sources. The main experiment fixes a one versus two source minority and majority relation while increasing the minority source's record count from to .
Across seven selected open weight models, the paper reports a 17.2 to 40.6 percentage point loss in decision accuracy in the label free setting. It also evaluates repeated answer labels, neutral context extension, balanced repetition, and a genuine increase in independent sources. The paper then separates information availability from information use through linear probes of source and occurrence counts, explicit numerical reports, activation times gradient head selection, and final query head ablations. Finally, it tests a count first prompt that requires the model to report source counts before deciding.
The behavioral result is clear within the controlled task: repeated records can move choices toward a source minority, while all seven models respond to a genuine increase in independent support. Source and repetition information is often decodable or reportable, but this does not guarantee a correct direct decision. Count first prompting improves all models at low repetition, but becomes mixed at high repetition as count reporting deteriorates.
Critical strengths
-
The primary behavioral diagnostic is controlled and interpretable. The source majority and correct answer remain fixed while repetition dose changes, both label assignments are evaluated, and the label free condition removes repeated answer letters from added records. The genuine source increase control is particularly important because it shows that the evaluated models are not simply insensitive to source structure. Five of seven models also show positive paired intervals against both neutral extension and balanced repetition controls.
-
The paper provides a substantial and transparent decomposition of the failure. The probe experiment uses disjoint source identifier pools across family level train, validation, and test splits, with length only and shuffled label controls. The report experiments reveal a meaningful difference between exact count accuracy and preservation of source ordering. The head analysis uses frozen discovery selections, held out surfaces, matched random groups, stratified bootstrap intervals, and multiple comparison correction. The count first experiment is also appropriately reported as dose dependent rather than as a universal solution.
Critical weaknesses
-
The novelty relative to existing work is limited. The related work section does cite the closest precedents, but the distinctions it draws from them are narrow. Schuster et al. (2026), "Whose Facts Win? LLM Source Preferences under Knowledge Conflicts" (ACL 2026, arXiv:2601.03746), already show that repeating information from a less credible source reverses the source preferences of 13 open weight LLMs, and the paper's own related work attributes to it both the same source repetition versus additional source attribution comparison and a separate validation of source recognition and table reading; the present paper's stated difference is only that its equal weight rule fixes the correct choice. GroupQA (Naphade, 2026, Findings of ACL 2026) already reports that paraphrasing an argument is more persuasive than providing distinct independent support, and, according to the paper's own related work, also checks conflict detection and document level stance attribution. CAMA (Lin et al., 2026, arXiv:2608.19701) already formulates correlated memories as a false majority and estimates the effective number of independent evidence sources; the present paper's stated difference is that the correct groups are supplied rather than inferred. The general distinction between a correct intermediate computation and the computation that controls the output is likewise attributed in the paper's own related work to Zhou et al. (2026) on negation processing (arXiv:2605.03052). The present paper's strongest remaining novelty is therefore the particular combination of an explicit source count task, count and occurrence probes, and cross task head ablations, rather than the discovery of the behavioral failure or the general availability versus use distinction.
-
The evidence does not yet establish the proposed mechanistic explanation or broad generality. Count and Direct use different prompts and different output spaces, numerical tokens versus evidence group letters, while ablation is applied only at the final query position. The resulting interactions establish different relative final query contributions, but not a distinct counting circuit or failure to recruit that circuit during Direct. The paper also does not test whether the selected heads mediate count first improvements, which it correctly acknowledges. Separately, model inclusion depended on observed baseline accuracy, repetition sensitivity, and reporting behavior; the reported 64 surface estimates include 16 discovery surfaces; four of seven models are from the Qwen family; and the surfaces vary identifiers, order, and wording within shared templates rather than constituting independent tasks. Neutral and balanced controls also do not exactly match token length or insertion position. These choices support a controlled within model phenomenon, but not a prevalence claim or strong generalization to open ended evidence synthesis.
Questions for authors
-
Beyond supplying the source groups explicitly, how do the authors distinguish the central contribution from Schuster et al. (ACL 2026, arXiv:2601.03746) and CAMA (arXiv:2608.19701)? Schuster et al. already show that repetition reverses source preferences even for less credible sources, and the paper's own related work credits them with validating source recognition separately from the decision; CAMA already estimates the effective number of independent evidence sources. Please state precisely which empirical or mechanistic conclusion remains unique after accounting for these works, and in particular what the equal weight source majority rule establishes that explicit source attribution under a credibility rule does not.
-
Can the authors report the primary dose effects in Table 3 using only the 48 surfaces withheld during model selection, excluding the 16 discovery surfaces? Please also state whether the 32 surfaces used for the context controls in Table 4 overlap with the discovery surfaces, and whether the 104 family count grid and the 192 family count first grid played any role in model selection. Please also provide the complete candidate model inclusion and exclusion process. Because repetition sensitivity was one selection criterion, a fully held out estimate is important for judging the robustness and magnitude of the effect.
-
Is there direct evidence that count selected heads are recruited by the count first prompt or causally mediate its gains? A matched output formulation, intervention transfer test, or mediation analysis would help distinguish source counting from differences caused by producing numerical rather than letter outputs. Without such evidence, should the claim be narrowed from a mechanism explaining the behavioral gap to task selective final query contributions?