- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper assesses the impact of style vs. rhetoric on the ability of LLMs to assess factuality. It aims to assess different capabilities in the "belief function" (discriminating truth from falsehood) and "comparator function" (identifying factual equivalence) via a factorial design. The conditions are spelled out in Table 1, with conditions indicating neutral (N) vs. rhetorical (R) and truth (T) vs. falsehood (F). The use WiCE to create the dataset and main results are that there is wide variation when rhetoric is manipulated (top part of Table 2), but T and F are held constant.
Critical strengths
Understanding the impact of style with LLMs is important and the paper does a reasonable job setting up a simple experiment to probe the impact of stylistic variations.
Critical weaknesses
While the paper describes "belief functions" and "comparator functions" these are not actual mechanisms in the model.
It is really unclear to me whether the paper is isolating rhetorical effect. The example on Figure 2 illustrates what is confusing. The RT condition introduces a seeming redundancy, while the RF condition introduces an internal inconsistency. But one could argue that living with a family is different than a state of residency. Moreover, the neutral conditions appear to make clearer what evidence the model can condition on, that is less clear in the R conditions. In measurement science, we try to be clear about the construct, and "rhetoric" feels quite underspecified.
The neutral paraphrase in Table 7 itself suggests that there is a lot of variability in this condition alone, so the marginal effect of rhetoric is unclear.
Human validation on a sample of 200 has a 9% error rate, and the authors should report an error analysis.
Questions for authors
Can you more precisely define and isolate rhetoric?
Construct Isolation, 4-Axis Figure 2 Replacement, Marginal Effect Over Neutral Paraphrase, and n=200 Human Audit Error Taxonomy
We thank you for the rigorous measurement-science critique. Your feedback help us to sharpen our construct definitions, replace Figure 2, isolate the marginal directional effect of each rhetorical axis over baseline paraphrase noise, and report the complete human-audit error taxonomy (all highlighted in blue in the revised manuscript).
1. Behavioral vs. Mechanistic Framing
We fully agree that "belief function" and "comparator function" describe functional input-output behaviors rather than internal neural mechanisms. We have removed all mechanistic terminology throughout the manuscript and explicitly define them as a behavioral, decision-theoretic decomposition (Abstract, Sec. 1, Fig. 1, Sec. 8).
2. Defining and Isolating the Construct of "Rhetoric" & Replacing Figure 2
You rightly noted that the original Figure 2 showed only the
distractor_truthaxis which created apparent redundancy in RT and internal tension in RF.evidence_attributionandpseudo_reasoning)—from Content Packaging (ΔI⊂E) (distractor_truth), which prepends a shared, verifiably true background fact from E to test whether judges conflate background context coverage with primary-claim veracity.3. Isolating the Marginal Effect of Rhetoric Over
neutral_paraphraseYou pointed out that
neutral_paraphrasein Table 7 already exhibits non-trivial total InvFail (7.9%–91.1%). In Section 6 and Appendix D, Table 8, we decompose Truth Parity (Exp 1: NT vs. RT) by axis into bidirectional-consistent preferences (RT%/NT%) vs. position inconsistency (Inc%):This decomposition clarifies two key measurement facts:
neutral_paraphrasefailure is position inconsistency (Inc), not directional preference: Bidirectional-consistent preference (RT+NT) onneutral_paraphraseis only 2.0%–8.1% across calibrated models.neutral_paraphrase,evidence_attribution(ΔI=∅) systematically elevates consistent RT preference (up to25.2%;73%–99%Style Dividend), whereaspseudo_reasoning(ΔI=∅) induces a strong anti-style shift toward plain NT (73%–95%of decisive verdicts select NT, penalizing ungrounded premise clauses), anddistractor_truth(ΔI⊂E) drives consistent RT preference to18.2%–60.2%(and up to96.1%–100.0%Style Dividend on 7 of 8 models in our new n=100 2-sentencecompound_synthesistest, Appendix K, Table 18). Pooled cluster-robust regression (Table 3, Model M2) confirms that both Pure Framing (OR=1.87,p<10−3) and Content Packaging (OR=4.50,p<10−9) significantly increase failure overneutral_paraphrase.4. Error Analysis of the 9% (18/200) Human Validation Sample
In our blind expert audit of n=200 quadruples (stratified 50/axis), 182 (
91.0%) passed strictly, 11 (5.5%) were rated ambiguous (unclear), and 7 (3.5%) failed (fail). We added the full taxonomy to Section 3.3 and **Appendix H, Table 12 **:Per-Axis Pass Rates (n=50 each):
neutral_paraphrase: 96% (48 pass, 0 unclear, 2 fail);evidence_attribution: 90% (45 pass, 2 unclear, 3 fail);pseudo_reasoning: 90% (45 pass, 5 unclear, 0 fail);distractor_truth: 88% (44 pass, 4 unclear, 2 fail).Negligible Impact on Headline Metrics (Appendix H, Table 13): Excluding all 18 human-flagged items (plus 45 heuristic-flagged records; n=63 total, n=2,383 clean) shifts global InvFail by ≤0.20pp and AttrRF by ≤0.03pp across all 8 models.