- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
-studies epistemic transparency under knowledge conflict: when contextual evidence contradicts knowledge that a model appears to already have. -proposes a taxonomy of behaviors beyond answer accuracy -builds a conflict-generation pipeline from advanced-domain questions, varies evidence format and task stakes, and evaluates five frontier models in both multiple-choice and more realistic task settings.
Critical strengths
moving from “which source did the model follow?” to “did the model make the conflict visible to the user?” is a key shift, especially for real deployments. I consider it a key strength.
experimental setup is broader than most prior work in this area. The authors go beyond MCQs, vary stakes and document format, and evaluate several models with repeated samples, which gives the paper real empirical substance.
Critical weaknesses
the evaluation stack largely depends on LLM judges, including conflict scoring and plausibility scoring, and some plausibility analyses are based on the same family of models being studied. This is not fatal, but it limits how hard I can lean on the fine-grained empirical conclusions.
dataset is still fairly small and heavily synthetic. The final MMLU-based set appears to come from 86 retained questions, and the paper does not provide a strong human-validation study showing that the generated conflicts and the taxonomy labels match expert judgment at a meaningful scale.
Questions for authors
Can you add a human evaluation on a nontrivial subset of items for the main taxonomy labels, especially conflict detection and attribution explicitness?
How sensitive are the main trends to judge choice?
2 comments
I appreciate the response. I will maintain my positive review.
We appreciate the reviewer’s recognition of the shift our work represents, and their feedback has led to a revision that we believe meaningfully strengthens the paper’s empirical foundations.
We ran a small human audit of the LLM judge (9 auditors scoring 5-6 model responses each, see full results in Appendix B.5). On conflict detection and compliance, the two central dimensions for our main results, annotators agree strongly with each other (Krippendorff's α = 1.00 and 0.81) and the judge closely matches the human majority (Cohen's κ = 0.72 and 0.77); the same holds for correction (α = 0.82, κ = 0.88). Agreement is weaker on attribution explicitness (α = 0.03) and reader agency preservation (α = 0.17), and we have too few refusals to evaluate propose-alternative robustly. Our interpretation is that the weaker agreement reflects these behaviors being more subjective and harder to define. The goal of this work is to show that epistemic transparency can be measured under realistic conflicts, rather than to rank models precisely on every dimension. Defining attribution and reader agency more precisely, ideally with subject-matter experts, is left for future work.
We re-scored our 23 human-validated responses with a second judge from a different model family (GPT-5.4) using the identical prompt, taxonomy and central-claim hint. The two judges agree closely on conflict detection, compliance and correction (Cohen's κ between 0.83 and 0.91). Agreement is lower on attribution explicitness and reader agency preservation (κ of about 0.5 to 0.6). These are the same two dimensions where human annotators also agreed least, and we already flag results on them as more subjective. The sample is small, so we see this as a consistency check rather than a precise measure of reliability. We have added the full comparison to Appendix B.6.