- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces ReliLogic, a behavioral evaluation framework for propositional-logic entailment. It has two components. The Representation Set contains 4,200 solver-verified instances, each rendered in four forms intended to be logically equivalent (Natural, Stripped, Code, Cosmetic-Code). The Intervention Set contains 400 base instances in Stripped form with 1,200 logic-preserving and 765 logic-changing variants.
Beyond per-representation accuracy, the paper reports joint correctness across the four forms (ALL-4), representation disagreement (RepDis), conditional intervention sensitivity and specificity with a Youden-style combination , and item-level disagreement over three repeated runs. From an evaluation of 74 model configurations, the authors conclude that aggregate accuracy can hide inconsistency across equivalent forms, inappropriate responses to interventions, and instance-level instability. The appendix adds rank correlations under distribution shift and on external benchmarks, reasoning on/off comparisons, reasoning-budget scaling, and an answer-format ablation.
Critical strengths
-
Verified, paired construction. Labels are checked by both SymPy and truth-table enumeration, and every instance is paired across four renderings and across pre- and post-intervention versions, which enables instance-level analysis. The reported numbers are internally consistent: recomputing from Table 11, Overall equals the mean of the four per-representation accuracies and = Sens + Spec − 1 for all 74 rows, and the Section 4.2 statistics (49/74, 5.87, 33, 0.65) reproduce. This is why I trust the data even though I question its interpretation.
-
Breadth, plus two informative controlled comparisons. The study covers 74 configurations. The within-family reasoning on/off comparison (Table 12) and the answer-format ablation (Appendix B.5) are useful; for example, enabling reasoning moves Ministral-3-14B from 63.69 to 96.79 overall accuracy. These are the parts I learned from.
Critical weaknesses
1. The central claim is not supported by the paper's own numbers, and the new metrics lack baselines.
- Recomputing from Table 11, Spearman(Overall, ALL-4) = 0.992 and Spearman(Overall, ) = 0.904 (Pearson 0.979). Among the 44 configurations with Overall ≥ 95%, only one has ALL-4 below 90 (87.05), and 33 configurations have ≥ 0.99. The reported failures therefore appear only in models whose accuracy on at least one representation is already 50–80%, which per-representation accuracy already reveals. The headline example, Qwen3-30B-A3B Thinking, scores 80.05% on Natural.
- ALL-4 is close to what independent errors would produce. Its Pearson correlation with the product of the four marginal accuracies is 0.982; for example 68.69 vs 67.47 for Qwen3-30B-A3B Thinking and 14.21 vs 11.18 for Llama-3.1-8B.
- There is no noise baseline. Section 4.4 reports 15.03% item-level disagreement across three identical runs for Qwen3-30B-A3B Thinking, so much of its RepDis (31.31%) and of its roughly 13% specificity failures may be sampling noise rather than sensitivity to representation or intervention.
- The metrics reward degenerate predictors. Always answering "entailed" yields RepDis = 0, ALL-4 = 50% and Spec = 100%. Gemma-4-31B-IT is essentially this case (Code 50.26, Cosmetic-Code 50.19, ALL-4 49.88, and more than 90% "entailed" predictions by the authors' own account), yet Section 4.2 interprets it as showing higher representation invariance than Llama-3.1-8B.
2. The evaluation protocol is undocumented and appears to drive the headline gaps.
- The paper gives no prompts (in particular, how the task is stated for Code), temperature, seeds, maximum output tokens, per-model reasoning-effort settings, or unparsed-response rates, while unparsed outputs are counted as incorrect.
- About nine instruct configurations sit at roughly 50% on Code or Cosmetic-Code, and some fall below chance on a balanced binary task (Granite 3.3 8B at 41.12 on Cosmetic-Code, Devstral Small 2 at 43.93 on Code). This points to constant outputs or parse failures rather than weaker reasoning.
- In Appendix B.5, moving the answer from the first line to the final line raises Qwen3 8B (thinking off) from 80.29% to 98.60% on Stripped. If the main protocol is first-line and single-token, non-reasoning models are prevented from reasoning before answering, and a large share of the instruct-versus-reasoning gap and of the "unreliability" findings is a protocol effect. This belongs in the main text.
- gpt-5.4 (78.83) scores far below gpt-5-nano (98.41), which suggests differing default reasoning settings and makes cross-model comparison hard to interpret.
- The Figure 5 means differ from the main tables for the same configurations by several reported standard deviations (Qwen3 8B thinking off: 71.49 ± 0.084 vs 72.05; Qwen3-30B-A3B Thinking: 90.51 ± 0.200 vs 91.01), so it is unclear what data or settings the repeated runs used.
Questions for authors
1. The central claim is not supported by the paper's own numbers, and the new metrics lack baselines.
- Recomputing from Table 11, Spearman(Overall, ALL-4) = 0.992 and Spearman(Overall, ) = 0.904 (Pearson 0.979). Among the 44 configurations with Overall ≥ 95%, only one has ALL-4 below 90 (87.05), and 33 configurations have ≥ 0.99. The reported failures therefore appear only in models whose accuracy on at least one representation is already 50–80%, which per-representation accuracy already reveals. The headline example, Qwen3-30B-A3B Thinking, scores 80.05% on Natural.
- ALL-4 is close to what independent errors would produce. Its Pearson correlation with the product of the four marginal accuracies is 0.982; for example 68.69 vs 67.47 for Qwen3-30B-A3B Thinking and 14.21 vs 11.18 for Llama-3.1-8B.
- There is no noise baseline. Section 4.4 reports 15.03% item-level disagreement across three identical runs for Qwen3-30B-A3B Thinking, so much of its RepDis (31.31%) and of its roughly 13% specificity failures may be sampling noise rather than sensitivity to representation or intervention.
- The metrics reward degenerate predictors. Always answering "entailed" yields RepDis = 0, ALL-4 = 50% and Spec = 100%. Gemma-4-31B-IT is essentially this case (Code 50.26, Cosmetic-Code 50.19, ALL-4 49.88, and more than 90% "entailed" predictions by the authors' own account), yet Section 4.2 interprets it as showing higher representation invariance than Llama-3.1-8B.
2. The evaluation protocol is undocumented and appears to drive the headline gaps.
- The paper gives no prompts (in particular, how the task is stated for Code), temperature, seeds, maximum output tokens, per-model reasoning-effort settings, or unparsed-response rates, while unparsed outputs are counted as incorrect.
- About nine instruct configurations sit at roughly 50% on Code or Cosmetic-Code, and some fall below chance on a balanced binary task (Granite 3.3 8B at 41.12 on Cosmetic-Code, Devstral Small 2 at 43.93 on Code). This points to constant outputs or parse failures rather than weaker reasoning.
- In Appendix B.5, moving the answer from the first line to the final line raises Qwen3 8B (thinking off) from 80.29% to 98.60% on Stripped. If the main protocol is first-line and single-token, non-reasoning models are prevented from reasoning before answering, and a large share of the instruct-versus-reasoning gap and of the "unreliability" findings is a protocol effect. This belongs in the main text.
- gpt-5.4 (78.83) scores far below gpt-5-nano (98.41), which suggests differing default reasoning settings and makes cross-model comparison hard to interpret.
- The Figure 5 means differ from the main tables for the same configurations by several reported standard deviations (Qwen3 8B thinking off: 71.49 ± 0.084 vs 72.05; Qwen3-30B-A3B Thinking: 90.51 ± 0.200 vs 91.01), so it is unclear what data or settings the repeated runs used.
6 comments
Q1 (3): There is no noise baseline. Section 4.4 reports 15.03% item-level disagreement across three identical runs for Qwen3-30B-A3B Thinking, so much of its RepDis (31.31%) and of its roughly 13% specificity failures may be sampling noise rather than sensitivity to representation or intervention.
Let d denote the proportion of instances for which the answer changes when the same question is asked again. Under sampling noise alone, RepDis across the four representations is at most 2d. For all eight configurations evaluated over repeated runs, the observed disagreement across representations exceeds this baseline.
Sampling noise would affect the four representations equally and produce disagreements without a systematic directional bias. Our results, however, show strong biases in the errors associated with different representations. This pattern indicates that the inconsistency of Qwen3-30B-A3B Thinking arises from sensitivity to representation rather than sampling noise.
| Metric | Natural | Stripped | Code | Cosmetic | Overall |
|---|---|---|---|---|---|
| Accuracy | 80.05% | 85.98% | 98.19% | 99.83% | 91.01% |
Q1 (4): The metrics reward degenerate predictors. Always answering "entailed" yields RepDis = 0, ALL-4 = 50% and Spec = 100%. Gemma-4-31B-IT is essentially this case (Code 50.26, Cosmetic-Code 50.19, ALL-4 49.88, and more than 90% "entailed" predictions by the authors' own account), yet Section 4.2 interprets it as showing higher representation invariance than Llama-3.1-8B.
J_I was designed specifically to identify this behavior. A central argument of our work is that logical reliability must be assessed through multiple metrics jointly, and this example illustrates why that is necessary. A predictor that always answers entailed has J_I = 0. Gemma-4-31B obtains J_I = 0.06, the second-lowest score among the 74 configurations, indicating that it lacks reliable logical reasoning ability.
Q2 (1): The paper gives no prompts (in particular, how the task is stated for Code), temperature, seeds, maximum output tokens, per-model reasoning-effort settings, or unparsed-response rates, while unparsed outputs are counted as incorrect.
To support fair comparisons, we use a common set of instances, prompts, response formats, and parsing rules across the evaluated configurations. The only exceptions concern parameters that cannot be controlled externally for a small number of API models. The core findings of ReliLogic are derived from comparisons within each configuration and hold for every configuration evaluated.
All experiments reported in the main text follow the “first-line, single-token” protocol. For each instance, the prompt presents the premises and conclusion, provides the definition of entailment, and then specifies the required answer format. The definition reads: “A claim is 'entailed' only if it must be true in every situation where all the premises are true.” The decoding temperature is fixed at 0.6 except for API configurations that do not expose this parameter. The seeds used to generate the instances are 0, 1, and 2, and the maximum output length is set to 8,192 tokens for all models. We now report the reasoning settings and unparsed-response rates for all configurations in Table 14.
Prompt text:
[Problem]
A claim is "entailed" only if it must be true in every situation where all the premises are true.
On the first line of your reply, write exactly one of:
Answer: entailed
Answer: not-entailed
Manuscript revision: We have added the complete prompt in Figure 13 (p. 21) and an experimental-setup section describing the shared protocol, decoding temperature, and maximum output length (Appendix B.6, p. 25). Table 14 reports reasoning settings and UNPARSED rates for all 74 configurations (p. 22), and Tables 15–16 give their prediction distributions across the four representations (p. 23–24).
Q2 (2): About nine instruct configurations sit at roughly 50% on Code or Cosmetic-Code, and some fall below chance on a balanced binary task (Granite 3.3 8B at 41.12 on Cosmetic-Code, Devstral Small 2 at 43.93 on Code). This points to constant outputs or parse failures rather than weaker reasoning.
The near-50% results are primarily driven by a pronounced tendency to predict the same label across instances. We examined the two models you highlighted and confirmed that some responses could not be parsed. However, parsing failures are not the dominant explanation for their performance, and the weaknesses in their reasoning remain evident after restricting the evaluation to successfully parsed responses, as shown below. All models receive the same instructions and use the same response format, ensuring that the evaluation conditions remain consistent and fair.
| Config. / Rep. | Acc | Unparsed rate | Acc on successfully parsed responses | not-entailed |
|---|---|---|---|---|
| Granite-3.3-8B / Cosmetic | 41.12% | 5.27% | 43.41% | 89.86% |
| Devstral-Small-2-24B / Code | 43.93% | 5.80% | 46.63% | 71.45% |
Q2 (3): In Appendix B.5, moving the answer from the first line to the final line raises Qwen3 8B (thinking off) from 80.29% to 98.60% on Stripped. If the main protocol is first-line and single-token, non-reasoning models are prevented from reasoning before answering, and a large share of the instruct-versus-reasoning gap and of the "unreliability" findings is a protocol effect. This belongs in the main text.
Thank you for highlighting the influence of answer placement.
Our protocol was chosen to ensure a consistent evaluation across configurations. Every configuration is tested on the same instances with the same prompts, response format, and parsing rules. The main-text experiments require a “first-line, single-token” answer, and all conclusions in ReliLogic are based on comparisons within a configuration.
Regarding the validity of this evaluation, the distinction between the two protocols is the order of the answer and the written reasoning. With a first-line answer, the model answers before presenting its analysis. With a final-line answer, it presents its analysis before answering. Both approaches are commonly used and widely accepted in LLM evaluation.
The results in Appendix B.5 show that reasoning before answering performs better on propositional entailment tasks. Our interpretation is that these tasks require successive deductions along a proof chain. Allowing the model to reason first enables it to write out intermediate conclusions and use each conclusion as the basis for the next step. When the answer must come first, the entire derivation has to be completed before the first answer token is generated.
Manuscript revision: We have added Section 4.6 and placed a brief comparison. Due to the page limit of the main text, we also expanded the discussion in Appendix B.5.
We thank you for the detailed review and for examining our results so carefully. We address your concerns point by point below and hope that our responses accurately reflect your questions and resolve the issues raised. If we have misunderstood any aspect of your comments, we would welcome further clarification and discussion.
Q1 (1): Recomputing from Table 11, Spearman(Overall, ALL-4) = 0.992 and Spearman(Overall, ) = 0.904 (Pearson 0.979). Among the 44 configurations with Overall ≥ 95%, only one has ALL-4 below 90 (87.05), and 33 configurations have ≥ 0.99. The reported failures therefore appear only in models whose accuracy on at least one representation is already 50–80%, which per-representation accuracy already reveals. The headline example, Qwen3-30B-A3B Thinking, scores 80.05% on Natural.
Thank you for carefully recomputing the reported statistics. ReliLogic's additional metrics provide information beyond accuracy even among configurations with near-perfect accuracy. Of the 39 configurations with accuracy ≥ 98%, 22 make markedly more errors in a particular representation, with those errors consistently favoring the same label. Within this subset, the rank correlation between J_I and accuracy falls to 0.66, reflecting differences in how models respond to logical interventions.
Q1 (2): ALL-4 is close to what independent errors would produce. Its Pearson correlation with the product of the four marginal accuracies is 0.982. For example 68.69 vs 67.47 for Qwen3-30B-A3B Thinking and 14.21 vs 11.18 for Llama-3.1-8B.
The observation that ALL-4 is close to the product of the four marginal accuracies supports the paper's central finding for the following reasons.
(1) Judgments grounded in propositional logic should have consistent correctness across representations. If a model bases its judgments on propositional logic, it should succeed or fail on the same instances across all four representations. ALL-4 should therefore be close to the accuracy of each representation. When ALL-4 instead approaches the value predicted under independence across representations, this suggests that whether an instance is answered correctly depends primarily on how it is expressed. This lack of reliability across representations can only be measured by matching the four versions of each instance.
For Hermes-4-70B, for example, ALL-4 could reach 96.36% if correct and incorrect predictions were fully aligned across representations. The prediction under independence is 92.93%, compared with the observed value of 93.19%. The near-independence you identify is precisely the lack of representation invariance that ALL-4 reveals.
(2) Predictions that agree across representations are more likely to be correct, supporting invariance as a measure of reliability. On instances for which all four representations yield the same answer, accuracy is substantially higher than overall accuracy. Thus, higher ALL-4 scores indicate that a model's answers are more trustworthy. Among the 29 configurations with ALL-4 below 80%, 27 achieve an accuracy of 78.1–100% on these consistently answered instances. The two exceptions are Gemma-4-31B and ERNIE-4.5-VL, whose agreement across representations arises from a bias toward a single label. Their J_I scores of 0.06 and −0.05, respectively, identify this behavior.
(3) ALL-4 is a distinct empirical measurement, while the four marginal accuracies only constrain its possible range. Predicting ALL-4 from their product requires assuming independence across representations, which is itself a property that ALL-4 is intended to examine. For configurations with higher error rates, this assumption is contradicted by the observed results.