acceptodds
Under review as a conference paper at ICLR 2027

RhetoriFact: Decomposing Belief Robustness and Comparator Fragility in Grounded LLM Judges

Abstract

Recently, Large Language Models (LLMs) have demonstrated remarkable capabilities as automatic evaluators for fact-checking. However, existing LLM judges remain susceptible to stylistic biases, often favoring authoritative rhetoric over factual substance. This raises a fundamental question: whether these biases are intrinsic to LLM evaluators or artifacts of ungrounded evaluation designs. To systematically investigate this vulnerability, we formalize factuality judgment as a decomposition of a belief function (discriminating truth) and a comparator function (recognizing factual equivalence), and introduce RhetoriFact. Our benchmark comprises 2,445 validated claim quadruples systematically permuted along a 2×2 factorial of factuality and rhetorical presentation. Extensive evaluations across eight state-of-the-art LLMs reveal the dual nature of rhetorical contamination: while the belief function is robust—models overwhelmingly resist direct falsehoods when anchored with oracle evidence—the comparator function fails pervasively, frequently flipping verdicts on factually identical pairs due to pure rhetorical variation. Logit-level analysis and scoped-prompt ablations further confirm that this instability stems from model probability distributions rather than sampling noise. Finally, we demonstrate that incorporating Chain-of-Thought (CoT) reasoning can partially mitigate these failures, though no output-surface predictor fully secures the rankings. Our findings conclude that while evidence grounding secures individual factuality verdicts, it does not eliminate the stylistic biases that skew automated rankings.

Primary area:
datasets and benchmarks
Keywords:
LLM evaluator, judge bias, style bias

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DWArOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper assesses the impact of style vs. rhetoric on the ability of LLMs to assess factuality. It aims to assess different capabilities in the "belief function" (discriminating truth from falsehood) and "comparator function" (identifying factual equivalence) via a factorial design. The conditions are spelled out in Table 1, with conditions indicating neutral (N) vs. rhetorical (R) and truth (T) vs. falsehood (F). The use WiCE to create the dataset and main results are that there is wide variation when rhetoric is manipulated (top part of Table 2), but T and F are held constant.

Critical strengths

Understanding the impact of style with LLMs is important and the paper does a reasonable job setting up a simple experiment to probe the impact of stylistic variations.

Critical weaknesses

While the paper describes "belief functions" and "comparator functions" these are not actual mechanisms in the model.

It is really unclear to me whether the paper is isolating rhetorical effect. The example on Figure 2 illustrates what is confusing. The RT condition introduces a seeming redundancy, while the RF condition introduces an internal inconsistency. But one could argue that living with a family is different than a state of residency. Moreover, the neutral conditions appear to make clearer what evidence the model can condition on, that is less clear in the R conditions. In measurement science, we try to be clear about the construct, and "rhetoric" feels quite underspecified.

The neutral paraphrase in Table 7 itself suggests that there is a lot of variability in this condition alone, so the marginal effect of rhetoric is unclear.

Human validation on a sample of 200 has a 9% error rate, and the authors should report an error analysis.

Questions for authors

Can you more precisely define and isolate rhetoric?

1 comment
AuthorsOpenReview ↗

Construct Isolation, 4-Axis Figure 2 Replacement, Marginal Effect Over Neutral Paraphrase, and n=200 Human Audit Error Taxonomy

We thank you for the rigorous measurement-science critique. Your feedback help us to sharpen our construct definitions, replace Figure 2, isolate the marginal directional effect of each rhetorical axis over baseline paraphrase noise, and report the complete human-audit error taxonomy (all highlighted in blue in the revised manuscript).

1. Behavioral vs. Mechanistic Framing

We fully agree that "belief function" and "comparator function" describe functional input-output behaviors rather than internal neural mechanisms. We have removed all mechanistic terminology throughout the manuscript and explicitly define them as a behavioral, decision-theoretic decomposition (Abstract, Sec. 1, Fig. 1, Sec. 8).

2. Defining and Isolating the Construct of "Rhetoric" & Replacing Figure 2

You rightly noted that the original Figure 2 showed only the distractor_truth axis which created apparent redundancy in RT and internal tension in RF.

  • Construct Formalization (Sec. 3.2): We formally separate Pure Rhetorical Framing (ΔI=∅\Delta\mathcal{I}=\emptyset)—which alters only the epistemic or discourse wrapper while adding zero propositions or redundancy (evidence_attribution and pseudo_reasoning)—from Content Packaging (ΔI⊂E\Delta\mathcal{I}\subset E) (distractor_truth), which prepends a shared, verifiably true background fact from EE to test whether judges conflate background context coverage with primary-claim veracity.
  • New 4-Axis Figure 2: We replaced Figure 2 with a complete 4-axis quadruple displaying all four benchmark axes side-by-side on the same base claim. As shown in prompt our Pure Framing axes contain neither redundancy in RT nor internal inconsistency in RF instruction, while preserving identical core claim structure.

3. Isolating the Marginal Effect of Rhetoric Over neutral_paraphrase

You pointed out that neutral_paraphrase in Table 7 already exhibits non-trivial total InvFail\mathrm{InvFail} (7.9%–91.1%7.9\%\text{--}91.1\%). In Section 6 and Appendix D, Table 8, we decompose Truth Parity (Exp 1: NT vs. RT) by axis into bidirectional-consistent preferences (RT%/NT%\mathrm{RT}\% / \mathrm{NT}\%) vs. position inconsistency (Inc%\mathrm{Inc}\%):

This decomposition clarifies two key measurement facts:

  1. roughly 60%–80% (62%–79% across the six calibrated models) of neutral_paraphrase failure is position inconsistency (Inc), not directional preference: Bidirectional-consistent preference (RT+NT\mathrm{RT}+\mathrm{NT}) on neutral_paraphrase is only 2.0%–8.1% across calibrated models.
  2. Construct-specific, opposing directional effects over control: Relative to neutral_paraphrase, evidence_attribution (ΔI=∅\Delta\mathcal{I}=\emptyset) systematically elevates consistent RT\mathrm{RT} preference (up to 25.2%; 73%–99% Style Dividend), whereas pseudo_reasoning (ΔI=∅\Delta\mathcal{I}=\emptyset) induces a strong anti-style shift toward plain NT\mathrm{NT} (73%–95% of decisive verdicts select NT\mathrm{NT}, penalizing ungrounded premise clauses), and distractor_truth (ΔI⊂E\Delta\mathcal{I}\subset E) drives consistent RT\mathrm{RT} preference to 18.2%–60.2% (and up to 96.1%–100.0% Style Dividend on 7 of 8 models in our new n=100n=100 2-sentence compound_synthesis test, Appendix K, Table 18). Pooled cluster-robust regression (Table 3, Model M2) confirms that both Pure Framing (OR=1.87,p<10−3\mathrm{OR}=1.87, p<10^{-3}) and Content Packaging (OR=4.50,p<10−9\mathrm{OR}=4.50, p<10^{-9}) significantly increase failure over neutral_paraphrase.

4. Error Analysis of the 9% (18/20018/200) Human Validation Sample

In our blind expert audit of n=200n=200 quadruples (stratified 50/axis), 182 (91.0%) passed strictly, 11 (5.5%) were rated ambiguous (unclear), and 7 (3.5%) failed (fail). We added the full taxonomy to Section 3.3 and **Appendix H, Table 12 **:

  • Per-Axis Pass Rates (n=50n=50 each): neutral_paraphrase: 96% (48 pass, 0 unclear, 2 fail); evidence_attribution: 90% (45 pass, 2 unclear, 3 fail); pseudo_reasoning: 90% (45 pass, 5 unclear, 0 fail); distractor_truth: 88% (44 pass, 4 unclear, 2 fail).

  • Negligible Impact on Headline Metrics (Appendix H, Table 13): Excluding all 18 human-flagged items (plus 45 heuristic-flagged records; n=63n=63 total, n=2,383n=2{,}383 clean) shifts global InvFail\mathrm{InvFail} by ≤0.20pp\le 0.20\text{pp} and AttrRF\mathrm{Attr}_{\mathrm{RF}} by ≤0.03pp\le 0.03\text{pp} across all 8 models.

Reviewer_1DE8OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses an important problem in trustworthy LLM evaluation: whether providing factual evidence eliminates the influence of rhetorical style on automated judgments. Its central finding is interesting: models can reliably distinguish true from false claims while remaining sensitive to rhetorical differences when comparing factually equivalent responses.

Critical strengths

The paper studies a relevant weakness in LLM-as-a-judge systems, particularly those used for preference evaluation and fact-checking.

Critical weaknesses

The paper's central contribution is the decomposition of LLM factuality judgment into a belief function and a comparator function. However, these functions are defined through different behavioral evaluation tasks rather than through independently identifiable model components.

Equation (4) defines invariance failure as the sum of style preference, anti-style preference, and inconsistency. But the inconsistency component can arise from position bias, response instability, or failure to follow the bidirectional evaluation protocol. It does not necessarily indicate rhetorical preference.

Questions for authors

How much of the reported invariance failure remains after excluding position-inconsistent comparisons? Does the ranking of models change when only stable rhetorical preferences are considered? Can the authors provide matched experiments without oracle evidence to demonstrate that grounding causally improves factual discrimination and reduces rhetorical falsehood attraction?

3 comments
AuthorsOpenReview ↗

Complete Inconsistency-Excluded Decomposition, Rank Correlation (rho=0.81), Full n=200 8-Model Ungrounded Experiments, and Behavioral Framing

We thank you for the clear, constructive review and for highlighting the practical value of our central finding. Below we address your conceptual point on behavioral framing and provide exact quantitative answers them.

1. Behavioral vs. Mechanistic Framing

We agree that our "belief function" and "comparator function" are defined via distinct behavioral tasks rather than isolated internal model circuits. In the revised manuscript (Abstract, Sec. 1, Fig. 1, Sec. 8), we explicitly define these terms as a decision-theoretic, behavioral decomposition of black-box judge outputsand have removed all claims of internal mechanistic localization.

2. How much invariance failure remains after excluding position-inconsistent comparisons? Does the ranking of models change?

In Table 2 and Section 5, we now explicitly decompose Equation (4) into bidirectional-consistent StylePref\mathrm{StylePref} (selecting the rhetorical option across both (A,B)(\mathrm{A},\mathrm{B}) and (B,A)(\mathrm{B},\mathrm{A}) orders), bidirectional-consistent AntiStylePref\mathrm{AntiStylePref} (selecting the neutral option across both orders), and Inconsistency\mathrm{Inconsistency}, alongside the Consistent Non-Tie Rate (CondNonTie=StylePref+AntiStylePrefTie+StylePref+AntiStylePref\mathrm{CondNonTie} = \frac{\mathrm{StylePref}+\mathrm{AntiStylePref}}{\mathrm{Tie}+\mathrm{StylePref}+\mathrm{AntiStylePref}}), which strictly excludes position-inconsistent comparisons from both numerator and denominator:

ModelPooled InvFail (#1+#2)Pooled CondNonTie (excl. Inc)Exp 1 InvFail (#1: NT vs. RT)Exp 1 CondNonTie (excl. Inc)Exp 1 StylePref\mathrm{StylePref} (RT %)Exp 1 AntiStyle\mathrm{AntiStyle} (NT %)Exp 1 Inconsist\mathrm{Inconsist} (%)
GPT-5.422.2% (Rank 1)10.3% (Rank 3)38.6%21.9%14.2%3.1%21.3%
Kimi-K325.5% (Rank 2)14.3% (Rank 4)41.7%28.5%18.7%4.5%18.5%
GLM-5.226.3% (Rank 3)8.5% (Rank 1)26.2%12.0%8.5%1.5%16.2%
Gemma 4 26B32.0% (Rank 4)8.7% (Rank 2)26.2%11.4%7.1%2.4%16.8%
Qwen 3.6 397B34.9% (Rank 5)20.8% (Rank 5)57.2%43.0%23.9%8.4%24.9%
Qwen 3.8 27B63.1% (Rank 6)37.7% (Rank 6)45.5%25.2%10.0%8.3%27.2%
Qwen 3.6 35B82.7% (Rank 7)49.9% (Rank 7)87.5%70.9%18.6%12.0%56.9%
DeepSeek-V494.7% (Rank 8)76.9% (Rank 8)97.8%77.8%5.0%2.9%89.9%
  1. Substantial invariance failure remains after excluding all position-inconsistent trials: Conditioning strictly on bidirectional-consistent trials, 11.4% to 77.8% of Exp 1 comparisons still fail to tie (e.g., 21.9% on GPT-5.4, 28.5% on Kimi-K3, 43.0% on Qwen-397B), and 8.5% to 76.9% fail to tie pooled across Exp 1 & 2. Moreover, among bidirectional-consistent decisive verdicts on Exp 1, every model favors RT\mathrm{RT} over NT\mathrm{NT} 54.6% to 85.3% of the time (p<10−4p < 10^{-4})—a metric mathematically free of position bias.
  2. Model rankings are strongly preserved (Spearman ρ=0.810,p=0.015\rho = 0.810, p = 0.015): Comparing full InvFail\mathrm{InvFail} against pooled inconsistency-excluded CondNonTie\mathrm{CondNonTie}, the four most vulnerable models (Qwen-397B, Qwen-27B, Qwen-35B, DeepSeek-V4) preserve their exact rank order (Ranks 5, 6, 7, 8), and the top four models (GPT-5.4, Kimi-K3, GLM-5.2, Gemma 4 26B) remain the top four (with GLM-5.2 and Gemma 4 moving from Ranks 3–4 to Ranks 1–2 once position flips are excluded).

3. Matched Experiments Without Oracle Evidence

We ran a matched Ungrounded Evaluation (−E-E vs. +E+E) across all 8 models on the exact same stratified n=200n=200 benchmark subset (50 per axis), as well as on n=100n=100 2-sentence persuasive synthesis quadruples (compound_synthesis; General Response Table; Sec. 5 Finding 1; Appendix K, Tables 16 & 18):

  • Causal improvement in factual discrimination: Removing oracle evidence causes Exp 5 truth accuracy to drop by 36.5 to 67.5 percentage points (falling to 20.0%–62.0%), and Exp 3 accuracy to fall to 23.5%–62.0%.
  • Causal suppression of Rhetorical Falsehood (RF\mathrm{RF}) attraction: Without oracle evidence, Exp 5 AttrRF\mathrm{Attr}_{\mathrm{RF}} rises to 5.6%–9.3% of all non-tie verdicts and 8.7%–28.6% of decisive verdicts across all 8 models (2.1×–5.3×2.1\times\text{--}5.3\times higher than unframed falsehood selection in Exp 3 on 6 of 8 models)—and reaches 11.6%–65.7% decisive under 2-sentence compound_synthesis across all 8 models (Table 18). Providing oracle evidence EE causally collapses AttrRF\mathrm{Attr}_{\mathrm{RF}} to 0.0%–1.0% on single-clause axes across all 8 models and to 0.0%–1.0% on 7 of 8 models under multi-sentence compound_synthesis (4.6×4.6\times reduction, 23.2%→5.0%23.2\% \to 5.0\%, on DeepSeek-V4-Flash), while increasing the Exp 1 Style Dividend on 7 of 8 models.

the belief function and comparator function are defined through different behavioral evaluation tasks. they are not independently identifiable model components. This is the major concern.

AuthorsOpenReview ↗

Clarifying Behavioral vs. Architectural Identifiability: Why Defining Capabilities via Factorial Probing Establishes Empirical Double Dissociation

Thank you for following up and clarifying your core concern. We want to address this directly, because we fully agree with you at the architectural level while clarifying why our behavioral claim does not assume separate internal model components:

  1. Our Paper Claims Behavioral-Level Capabilities, Not Architectural Model Components: We do not claim to identify two separate internal modules, sub-networks, or circuits inside the LLM. In the revised manuscript, we explicitly state this in the Abstract, Section 1, and Section 8 (Limitations):

"We use these terms strictly as a decision-theoretic, behavioral decomposition of black-box judge outputs—analogous to separating sensitivity from tie/decision criteria—rather than claiming separate internal neural circuits." (Sec. 1)

  1. Why Behavioral Capabilities Defined Across Matched Factorial Pairs Are Independently Identifiable: In psychometrics, Signal Detection Theory (SDT), and Bradley–Terry–Davidson tie models, a single black-box observer's discrimination sensitivity (d′, separating true from false when Δ_(fact) ≠ 0) and equivalence/tie criterion (θ, recognizing parity when Δ_(fact) = 0) are always defined and identified through complementary behavioral trial types over the same stimulus space. Crucially, \circled1–\circled2\circled{1}\text{--}\circled{2} (parity) and \circled3–\circled6\circled{3}\text{--}\circled{6} (discrimination/conflict) are not separate tasks with different prompts, instructions, or domains: they are all (4 choose 2) = 6 pairings of the exact same claim quadruples (c_(NT), c_(NF), c_(RT), c_(RF)) evaluated under the exact same evidence E and system prompt.
  2. Empirical Proof That These Two Behavioral Capabilities Vary Independently (Double Dissociation): If truth discrimination ("belief") and equivalence recognition ("comparator") were merely two labels for one single undifferentiated behavioral score, they would covary across interventions and across models. Instead, our experiments demonstrate a functional double dissociation:
  • Orthogonal Response to Interventions:
    1. Adding Oracle Evidence (− E → + E, Table 16): Selectively rescues truth discrimination (\circled5\circled{5} accuracy rises + 36.5–67.5,pp; Attr_(RF) drops from 8.7%–28.6%8.7\%\text{--}28.6\% to ≤1.0%\le 1.0\%), yet does not reduce parity bias—in fact, the \circled1\circled{1} Style Dividend increases on 7 of 8 models (e.g., 63.6%→83.9%63.6\% \to 83.9\% on GPT-5.4; 13.3%→51.6%13.3\% \to 51.6\% on Qwen-3.8-27B).
    2. Prompt Scoping & CoT Reasoning (Tables 5–6): Selectively shifts parity failure (InvFail drops by up to − 57.1,pp on Qwen-3.6-35B/Qwen-3.8-27B, or regresses + 4.0,pp under hardened prompting on Gemma-4), while conflict truth accuracy (Truth5) remains invariant at ceiling (e.g., 98.8%→98.6%–98.7%98.8\% \to 98.6\%\text{--}98.7\% on GPT-5.4; 98.9%→99.1%–99.2%98.9\% \to 99.1\%\text{--}99.2\% on Kimi-K3).
  • Cross-Model Statistical Independence (Spearman ρ = − 0.23, p = 0.59): Across the 8 models (Table 2), truth discrimination accuracy does not predict parity stability. For instance, Qwen 3.8 27B (99.0% \circled3\circled{3} accuracy) and Qwen 3.6 397B (99.1%) match GPT-5.4 (99.3%) and outperform Gemma 4 26B (94.7%) on truth discrimination, yet suffer 37.7% and 20.8% consistent non-tie failure (CondNonTie)—4.3 × and 2.4 × worse than Gemma 4 (8.7%).

In short: while a single neural network produces all verdicts, truth discrimination and equivalence recognition are empirically orthogonal, doubly dissociated behavioral dimensions—which is precisely why grounding a judge with evidence secures fact-checking (Δ_(fact) ≠ 0) without securing pairwise leaderboards or RLAIF rankings at parity (Δ_(fact) = 0). We hope this clarifies our exact behavioral claim and resolves your concern.

Reviewer_8Q3cOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces the RhetoriFact benchmark to test whether minimal pair changes to the truth can change the judge's assessment. They use plain/rhetorical × true/false pairings. They took the claims from WiCE and they find that rhetorical framing does not lead to models judging something incorrectly.

Critical strengths

The minimal-pair construction is a clean design choice.

Critical weaknesses

Poor presentation

I am familiar with this topic, but the writing was not clear, and I had a really hard time understanding the main text. Many parts of the manuscript were not understandable to me. The writing and presentation should be much more careful. I had to read this paper multiple times to get a grasp of what it says and had a very hard time.

I also do not think this paper has enough evidence for its claims. The framing changes are just one short sentence and are not really a rhetorical framing to make a difference? They say this is a floor, but that does not make it robust. I do not believe any of these weak perturbations can be used as evidence of their robustness, especially when we have major evidence that models can change their judgement based on their values, persuasion, and many other techniques. I also find their mechanistic claims to be overclaimed. This is just an output-level analysis, and the novelty can be seen as incremental.

Questions for authors

I strongly suggest that the authors rewrite the manuscript and make sure the text is clear and understandable and has a good flow.

3 comments
AuthorsOpenReview ↗

Full-Sample Multi-Sentence Persuasion & Ungrounded Manipulation Checks, Removal of Mechanistic Overclaims, and Full Clarity Rewrite

We thank you for recognizing our minimal-pair design and for the candid feedback on writing clarity, perturbation potency, and mechanistic framing. We have substantially rewritten the manuscript (changes in blue) and conducted three full-sample experiments (Sec. 5 Finding 1; Appendix K, Tables 16–18) to directly test your concerns.

1. Clarifying Our Core Finding & Proving Perturbation Potency (Single-Sentence vs. Multi-Sentence Rhetoric)

We respectfully clarify a key point regarding your summary that "they find that rhetorical framing does not lead to models judging something incorrectly" and your concern that "the framing changes are just one short sentence and are not really a rhetorical framing to make a difference."

Our central finding is a sharp functional asymmetry: while grounded judges resist rhetoric when one answer is true and one is false, single-clause wrappers, all models (Exp 5, AttrRF≤5%\mathrm{Attr}_{\mathrm{RF}} \leq 5\%%), those exact same single-sentence wrappers break equivalence judgments in 22.2%–94.7% of factually identical pairs (Exp 1 & 2)—and in 11.4%–77.8% of Truth Parity trials even after excluding all position-inconsistent comparisons, where the rhetorically framed truth wins 54.6%–85.3% of decisive verdicts (Table 2). For demonstrating parity fragility, a minimal 1-sentence wrapper is the strictest test: if a single clause flips rankings up to 85% of the time at parity, pairwise leaderboards are highly vulnerable.

Furthermore, to directly test whether our wrappers are "too weak" to sway Exp 5 or whether richer multi-sentence rhetoric overcomes oracle grounding, we ran three full-sample, manipulation-checked experiments:

  1. Same Single-Sentence Wrappers Without Oracle Evidence (n=200n=200 Across All 8 Models; Appendix K, Table 16): Evaluating the exact same single-sentence wrappers without oracle evidence (−E-E) across all 8 models drops Exp 5 truth accuracy by 36.5–67.5pp and causes decisive RF\mathrm{RF} attraction to surge to 8.7%–28.6% (5.6%–9.3%5.6\%\text{--}9.3\% of all non-tie verdicts; 2.1×–5.3×2.1\times\text{--}5.3\times higher than unframed falsehood selection in Exp 3 on 6 of 8 models, e.g., 4.2%→8.7%4.2\% \to 8.7\% decisive on GPT-5.4, 3.0%→10.8%3.0\% \to 10.8\% on Qwen-397B, 3.9%→10.8%3.9\% \to 10.8\% on Kimi-K3, and 5.3%→28.6%5.3\% \to 28.6\% on DeepSeek-V4). Restoring oracle evidence collapses AttrRF\mathrm{Attr}_{\mathrm{RF}} to 0.0%–1.0%, proving our single-sentence wrappers do persuade ungrounded judges and that oracle grounding is what neutralizes them.

  2. Multi-Sentence Editorialized Prose (verbose_polish, 2–3 Sentences, 2.6×2.6\times Length, n=100n=100 Across All 8 Models; Appendix K, Table 17): Expanding claims from 19.9 words to 51.7 words of confident editorial prose across all 8 models yields 0.0% RF attraction under oracle evidence (94.0%–100.0% truth accuracy), while Exp 1 parity failure reaches 18.0%–100.0% because judges penalize unverified editorializing in favor of plain NT.

  3. Multi-Sentence Persuasive Synthesis (compound_synthesis, 2 Sentences, 2.1×2.1\times Length, Full n=100n=100 Across All 8 Models; Appendix K, Table 18): To ensure our multi-sentence test exerts strong positive persuasive pull rather than triggering anti-verbosity penalties, we constructed n=100n=100 validated 2-sentence quadruples (mean 38.0 vs. 18.4 words) pairing one verified contextual sentence from EE with natural source-attributed claim synthesis, and evaluated all 8 models on the exact same n=100n=100 items under both −E-E and +E+E:

  • Both positive manipulation checks pass decisively: Without evidence (−E-E), compound_synthesis drives decisive RF selection to 11.6%–65.7% (8.0%–26.3%8.0\%\text{--}26.3\% of all non-tie verdicts), and with evidence at parity (Exp 1 +E), judges overwhelmingly prefer RT over NT (67.6%–100.0% Style Dividend, and 96.1%–100.0% on 7 of 8 models).
  • Belief remains robust across calibrated judges while exposing lightweight-judge limits: Even under a multi-sentence wrapper that wins 96%–100% of parity trials on 7 of 8 models, oracle grounding collapses Exp 5 +E RF attraction to 0.0% on 6 of 8 models and 1.0% (1/96) on Gemma-4, while slashing lightweight DeepSeek-V4-Flash's non-tie RF attraction by 4.6×4.6\times (23.2%→5.0%23.2\% \to 5.0\%; decisive 65.7%→7.9%65.7\% \to 7.9\%, lifting truth accuracy +46.0pp+46.0\text{pp} from 12.0%12.0\% to 58.0%58.0\%).

2. Removing All "Mechanistic" Overclaims

We agree that our study and logprobs analyses characterize black-box behavioral outputs and token probabilities rather than internal neural circuits. We have removed all "mechanistic" terminology throughout the paper and explicitly frame our Belief/Comparator split as a decision-theoretic behavioral decomposition.

3. Full Clarity Rewrite

We rewrote the Abstract, Introduction, Figure 1, Figure 2 (now showing all 4 axes side-by-side), and Sections 3–9 in clear, direct prose, eliminating dense jargon

Reviewer_8Q3cOpenReview ↗

Rebuttal Acknowledgement

I have read the authors' rebuttal, and I keep my score as my concerns stand. The attached revised paper has been heavily changed. I believe it needs another round of review to reread the paper, as most of the main paper has changed, and with the limited time, I can't make sure whether their revised version is sound or not. The presentation still has problems, with tables bleeding into the margins in the appendix and also Table 6 in the main paper. I thank the authors for removing the mechanistic part I suggested. I believe it helped the paper, but the paper has changed significantly, and I believe it needs a new round of review.

AuthorsOpenReview ↗

Table Margin Overflows Fixed & 3-Minute Map of Exact Substantive Additions

We thank you for reading our rebuttal, for noting that removing the mechanistic framing improved the paper, and for catching the table margin overflows.

1. Formatting Fixed (Table 6 & All Appendix Tables)

We have uploaded an updated PDF fixing the column spacing in Table 6 and the width formatting in Appendix Tables 12 and 17.

2. Why the Blue Highlighting Appeared Extensive—and a 3-Minute Soundness Map

We completely understand the limited time available during the discussion period. In our previous upload, the blue highlighting appeared extensive (~37% of main text) only because we wrapped entire paragraphs in blue where we made wording-only clarity edits and removed mechanistic terminology as requested—not because the underlying benchmark, methodology, or core experimental results changed. In the newly uploaded PDF, we have narrowed blue highlighting strictly to substantive additions (15% of main text) so every change can be verified at a glance.

For convenience, here is the complete map of what is 100% unchanged from the original submission vs. what was added in response to reviewers:

  • Core Benchmark, Methodology, & All Original 8-Model Runs — UNCHANGED: The N=2,445N=2{,}445 WiCE quadruples, the 2×22\times 2 design and 6 comparison tracks (\circled1–\circled6\circled{1}\text{--}\circled{6}), the bidirectional consensus operator (Eq. 3), the factorial logistic regressions (Table 3), the logprobs experiment (Table 4), the scoped/CoT/hardened experiments (Tables 5–6), and all underlying model outputs are identical to the original submission.
  • Prose & Terminology Edits (Abstract, Sec. 1–2, Sec. 5–9) — Wording Only: Replaced mechanistic phrasing with behavioral terminology and simplified dense phrasing, directly following your feedback and that of Reviewers 2 and 3.
  • Table 2 & Sec. 5 (Finding 2) — Re-tabulation of Existing Predictions Only: Added the explicit sub-components of Eq. (4) (StylePref\mathrm{StylePref}, AntiStylePref\mathrm{AntiStylePref}, CondNonTie\mathrm{CondNonTie}, and Spearman ρ=0.810\rho = 0.810) computed on the exact same original model predictions (requested by Reviewer 2).
  • Fig. 2, Sec. 3.3, & Sec. 6 (1st Paragraph) — Existing Benchmark Diagnostics Only: Expanded Fig. 2 to show all 4 existing axes instead of 1, added the 3-category breakdown of the existing 18/20018/200 human-audit items (Sec. 3.3 / Table 12), and reported the per-axis breakdown of existing Exp 1 results (Sec. 6 / Table 8, requested by Reviewer 3).
  • New Control Ablations — Self-Contained in Appendix K (Tables 16–18) + 1 Paragraph in Sec. 5 (Finding 1): All newly executed rebuttal controls (matched ungrounded −E-E evaluation and multi-sentence verbose_polish / compound_synthesis checks across all 8 models) are isolated in Appendix K and summarized in a single paragraph of Sec. 5 (Finding 1).

We hope this concise map and the cleaned PDF layout make it straightforward to verify the soundness of the revision within a few minutes.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Full-Sample 8-Model Ungrounded & Multi-Sentence Persuasion Experiments, Eq. (4) Decomposition, and Manuscript Revisions

We thank all three reviewers for their constructive feedback. We have uploaded a revised manuscript (all changes highlighted in blue), supported by three full-sample experiments and three diagnostic analyses that directly resolve every concern raised:

1. Behavioral, Decision-Theoretic Framing of "Belief vs. Comparator Functions"

We agree with all three reviewers that our decomposition into a belief function (discriminating truth from falsehood given evidence) and a comparator function (recognizing factual equivalence and declaring a tie at parity) is a functional, decision-theoretic decomposition of black-box judge outputs—analogous to separating sensitivity (d′d') from tie/decision criteria—not a claim about isolated internal neural circuits or weights.

  • Revision (Abstract, Sec. 1, Fig. 1, Sec. 5–9): We removed all mechanistic terminology ("mechanistically entangled", "mechanistic etiologies", "topographies") and rewrote the prose for clarity and directness.

2. New Experiment 1: Matched Ungrounded (−E-E) vs. Oracle-Grounded (+E+E) Evaluation Across All 8 Models (n=200n=200 Stratified)

To test whether oracle evidence causally improves factual discrimination and suppresses rhetorical falsehood attraction (AttrRF\mathrm{Attr}_{\mathrm{RF}}), we evaluated all 8 models on the exact same stratified n=200n=200 subset (50 per axis) without oracle evidence (−E-E) vs. with oracle evidence (+E+E) (Sec. 5 Finding 1; Appendix K, Table 16):

  • Takeaway: Removing oracle evidence drops Exp 5 truth accuracy by 36.5–67.5pp and causes decisive RF\mathrm{RF} share to reach 8.7%–28.6% (5.6%–9.3%5.6\%\text{--}9.3\% of all non-tie verdicts, exceeding unframed NF\mathrm{NF} selection in Exp 3 by 2.1×–5.3×2.1\times\text{--}5.3\times on 6 of 8 models). Restoring oracle evidence collapses AttrRF\mathrm{Attr}_{\mathrm{RF}} to 0.0%–1.0% across all 8 models while increasing the Exp 1 Style Dividend on 7 of 8 models.

3. New Experiments 2 & 3: Multi-Sentence Rhetoric—Editorialized Prose (2.6×2.6\times) and Persuasive Synthesis (2.1×2.1\times) (n=100n=100)

To test whether multi-sentence rhetoric overcomes oracle grounding (Appendix K, Tables 17 & 18):

  • Editorialized Multi-Sentence Prose (verbose_polish, 2.6×2.6\times length, n=100n=100 across all 8 models, Table 17): Expanding claims into 2–3 sentences of unverified authoritative commentary yields 0.0% RF attraction across all 8 models under +E+E (94.0%–100.0% truth accuracy), while triggering anti-verbosity preference for plain NT in Exp 1 (0%–4%0\%\text{--}4\% Style Dividend).
  • Persuasive Multi-Sentence Synthesis (compound_synthesis, 2.1×2.1\times length, n=100n=100 across all 8 models, Table 18): To ensure multi-sentence rhetoric exerts strong positive persuasive pull rather than anti-verbosity penalties, we constructed n=100n=100 2-sentence quadruples pairing one verified contextual sentence from EE with natural source-attributed synthesis. Across all 8 models on the exact same n=100n=100 records, compound_synthesis wins 67.6%–100.0% of decisive parity verdicts under +E+E (96.1%–100.0% on 7 of 8 models) and captures 11.6%–65.7% of decisive conflict verdicts when ungrounded (−E-E), yet oracle grounding collapses Exp 5 AttrRF\mathrm{Attr}_{\mathrm{RF}} to 0.0% on 6 of 8 models, 1.0% (1/96) on Gemma-4, and cuts lightweight DeepSeek-V4-Flash's non-tie RF attraction by 4.6×4.6\times (23.2%→5.0%23.2\% \to 5.0\%; decisive 65.7%→7.9%65.7\% \to 7.9\%).

4. Full Eq. (4) Decomposition, Construct Isolation, and Human Audit Error Taxonomy

  • Table 2 Updated: Explicitly decomposes Eq. (4) into StylePref\mathrm{StylePref}, AntiStylePref\mathrm{AntiStylePref}, Inconsistency\mathrm{Inconsistency}, the inconsistency-excluded Consistent Non-Tie Rate (CondNonTie\mathrm{CondNonTie}), and Style Dividend. Rankings between full InvFail\mathrm{InvFail} and pooled CondNonTie\mathrm{CondNonTie} are strongly preserved (Spearman ρ=0.810,p=0.015\rho = 0.810, p = 0.015).
  • Figure 2 Replaced & Marginal Axis Decomposition (Fig. 2; Appendix D, Table 8): Shows all 4 axes side-by-side and isolates each rhetorical axis's directional preference over neutral_paraphrase.
  • Human Audit Error Taxonomy (Sec. 3.3; Appendix H, Tables 12–13): Full root-cause breakdown of the 18/200 non-pass audit items (excluding them shifts headline metrics by ≤0.20pp\le 0.20\text{pp}).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.