acceptodds
Under review as a conference paper at ICLR 2027

More Than Meets the Eye: Measuring Logical Reasoning Beyond a Single Accuracy

Abstract

An increasing number of large language models (LLMs) have achieved high accuracy on existing logical reasoning benchmarks, demonstrating impressive performance. However, producing the correct answer does not necessarily imply that a model can reliably identify and respond to the logical relations underlying a problem. Recent studies have highlighted the limitations of using overall accuracy as the primary measure of logical reasoning ability and have turned to examining models' reasoning behavior. In this work, we introduce ReliLogic, a behavioral reliability evaluation framework based on propositional-logic entailment questions. The framework consists of a Representation Set containing four logically equivalent forms and an Intervention Set constructed through logic-preserving and logic-altering interventions. Beyond final accuracy, ReliLogic evaluates cross-representation reliability, selective logical responsiveness, and instance-level prediction stability, providing a more comprehensive characterization of logical reasoning ability from the perspective of model behavior. We conduct a series of evaluations across 74 language-model configurations and uncover multiple failure modes that are not reflected by accuracy alone.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large language models, Logical reasoning, Model evaluation, Metamorphic testing, Behavioral evaluation

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_7XzVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces ReliLogic, a behavioral evaluation framework for propositional-logic entailment. It has two components. The Representation Set contains 4,200 solver-verified instances, each rendered in four forms intended to be logically equivalent (Natural, Stripped, Code, Cosmetic-Code). The Intervention Set contains 400 base instances in Stripped form with 1,200 logic-preserving and 765 logic-changing variants.

Beyond per-representation accuracy, the paper reports joint correctness across the four forms (ALL-4), representation disagreement (RepDis), conditional intervention sensitivity and specificity with a Youden-style combination JIJ_I, and item-level disagreement over three repeated runs. From an evaluation of 74 model configurations, the authors conclude that aggregate accuracy can hide inconsistency across equivalent forms, inappropriate responses to interventions, and instance-level instability. The appendix adds rank correlations under distribution shift and on external benchmarks, reasoning on/off comparisons, reasoning-budget scaling, and an answer-format ablation.

Critical strengths
  1. Verified, paired construction. Labels are checked by both SymPy and truth-table enumeration, and every instance is paired across four renderings and across pre- and post-intervention versions, which enables instance-level analysis. The reported numbers are internally consistent: recomputing from Table 11, Overall equals the mean of the four per-representation accuracies and JIJ_I = Sens + Spec − 1 for all 74 rows, and the Section 4.2 statistics (49/74, 5.87, 33, 0.65) reproduce. This is why I trust the data even though I question its interpretation.

  2. Breadth, plus two informative controlled comparisons. The study covers 74 configurations. The within-family reasoning on/off comparison (Table 12) and the answer-format ablation (Appendix B.5) are useful; for example, enabling reasoning moves Ministral-3-14B from 63.69 to 96.79 overall accuracy. These are the parts I learned from.

Critical weaknesses

1. The central claim is not supported by the paper's own numbers, and the new metrics lack baselines.

  • Recomputing from Table 11, Spearman(Overall, ALL-4) = 0.992 and Spearman(Overall, JIJ_I) = 0.904 (Pearson 0.979). Among the 44 configurations with Overall ≥ 95%, only one has ALL-4 below 90 (87.05), and 33 configurations have JIJ_I ≥ 0.99. The reported failures therefore appear only in models whose accuracy on at least one representation is already 50–80%, which per-representation accuracy already reveals. The headline example, Qwen3-30B-A3B Thinking, scores 80.05% on Natural.
  • ALL-4 is close to what independent errors would produce. Its Pearson correlation with the product of the four marginal accuracies is 0.982; for example 68.69 vs 67.47 for Qwen3-30B-A3B Thinking and 14.21 vs 11.18 for Llama-3.1-8B.
  • There is no noise baseline. Section 4.4 reports 15.03% item-level disagreement across three identical runs for Qwen3-30B-A3B Thinking, so much of its RepDis (31.31%) and of its roughly 13% specificity failures may be sampling noise rather than sensitivity to representation or intervention.
  • The metrics reward degenerate predictors. Always answering "entailed" yields RepDis = 0, ALL-4 = 50% and Spec = 100%. Gemma-4-31B-IT is essentially this case (Code 50.26, Cosmetic-Code 50.19, ALL-4 49.88, and more than 90% "entailed" predictions by the authors' own account), yet Section 4.2 interprets it as showing higher representation invariance than Llama-3.1-8B.

2. The evaluation protocol is undocumented and appears to drive the headline gaps.

  • The paper gives no prompts (in particular, how the task is stated for Code), temperature, seeds, maximum output tokens, per-model reasoning-effort settings, or unparsed-response rates, while unparsed outputs are counted as incorrect.
  • About nine instruct configurations sit at roughly 50% on Code or Cosmetic-Code, and some fall below chance on a balanced binary task (Granite 3.3 8B at 41.12 on Cosmetic-Code, Devstral Small 2 at 43.93 on Code). This points to constant outputs or parse failures rather than weaker reasoning.
  • In Appendix B.5, moving the answer from the first line to the final line raises Qwen3 8B (thinking off) from 80.29% to 98.60% on Stripped. If the main protocol is first-line and single-token, non-reasoning models are prevented from reasoning before answering, and a large share of the instruct-versus-reasoning gap and of the "unreliability" findings is a protocol effect. This belongs in the main text.
  • gpt-5.4 (78.83) scores far below gpt-5-nano (98.41), which suggests differing default reasoning settings and makes cross-model comparison hard to interpret.
  • The Figure 5 means differ from the main tables for the same configurations by several reported standard deviations (Qwen3 8B thinking off: 71.49 ± 0.084 vs 72.05; Qwen3-30B-A3B Thinking: 90.51 ± 0.200 vs 91.01), so it is unclear what data or settings the repeated runs used.
Questions for authors

1. The central claim is not supported by the paper's own numbers, and the new metrics lack baselines.

  • Recomputing from Table 11, Spearman(Overall, ALL-4) = 0.992 and Spearman(Overall, JIJ_I) = 0.904 (Pearson 0.979). Among the 44 configurations with Overall ≥ 95%, only one has ALL-4 below 90 (87.05), and 33 configurations have JIJ_I ≥ 0.99. The reported failures therefore appear only in models whose accuracy on at least one representation is already 50–80%, which per-representation accuracy already reveals. The headline example, Qwen3-30B-A3B Thinking, scores 80.05% on Natural.
  • ALL-4 is close to what independent errors would produce. Its Pearson correlation with the product of the four marginal accuracies is 0.982; for example 68.69 vs 67.47 for Qwen3-30B-A3B Thinking and 14.21 vs 11.18 for Llama-3.1-8B.
  • There is no noise baseline. Section 4.4 reports 15.03% item-level disagreement across three identical runs for Qwen3-30B-A3B Thinking, so much of its RepDis (31.31%) and of its roughly 13% specificity failures may be sampling noise rather than sensitivity to representation or intervention.
  • The metrics reward degenerate predictors. Always answering "entailed" yields RepDis = 0, ALL-4 = 50% and Spec = 100%. Gemma-4-31B-IT is essentially this case (Code 50.26, Cosmetic-Code 50.19, ALL-4 49.88, and more than 90% "entailed" predictions by the authors' own account), yet Section 4.2 interprets it as showing higher representation invariance than Llama-3.1-8B.

2. The evaluation protocol is undocumented and appears to drive the headline gaps.

  • The paper gives no prompts (in particular, how the task is stated for Code), temperature, seeds, maximum output tokens, per-model reasoning-effort settings, or unparsed-response rates, while unparsed outputs are counted as incorrect.
  • About nine instruct configurations sit at roughly 50% on Code or Cosmetic-Code, and some fall below chance on a balanced binary task (Granite 3.3 8B at 41.12 on Cosmetic-Code, Devstral Small 2 at 43.93 on Code). This points to constant outputs or parse failures rather than weaker reasoning.
  • In Appendix B.5, moving the answer from the first line to the final line raises Qwen3 8B (thinking off) from 80.29% to 98.60% on Stripped. If the main protocol is first-line and single-token, non-reasoning models are prevented from reasoning before answering, and a large share of the instruct-versus-reasoning gap and of the "unreliability" findings is a protocol effect. This belongs in the main text.
  • gpt-5.4 (78.83) scores far below gpt-5-nano (98.41), which suggests differing default reasoning settings and makes cross-model comparison hard to interpret.
  • The Figure 5 means differ from the main tables for the same configurations by several reported standard deviations (Qwen3 8B thinking off: 71.49 ± 0.084 vs 72.05; Qwen3-30B-A3B Thinking: 90.51 ± 0.200 vs 91.01), so it is unclear what data or settings the repeated runs used.
6 comments

We thank you for the detailed review and for examining our results so carefully. We address your concerns point by point below and hope that our responses accurately reflect your questions and resolve the issues raised. If we have misunderstood any aspect of your comments, we would welcome further clarification and discussion.


Q1 (1): Recomputing from Table 11, Spearman(Overall, ALL-4) = 0.992 and Spearman(Overall, ) = 0.904 (Pearson 0.979). Among the 44 configurations with Overall ≥ 95%, only one has ALL-4 below 90 (87.05), and 33 configurations have ≥ 0.99. The reported failures therefore appear only in models whose accuracy on at least one representation is already 50–80%, which per-representation accuracy already reveals. The headline example, Qwen3-30B-A3B Thinking, scores 80.05% on Natural.

Thank you for carefully recomputing the reported statistics. ReliLogic's additional metrics provide information beyond accuracy even among configurations with near-perfect accuracy. Of the 39 configurations with accuracy ≥ 98%, 22 make markedly more errors in a particular representation, with those errors consistently favoring the same label. Within this subset, the rank correlation between J_I and accuracy falls to 0.66, reflecting differences in how models respond to logical interventions.


Q1 (2): ALL-4 is close to what independent errors would produce. Its Pearson correlation with the product of the four marginal accuracies is 0.982. For example 68.69 vs 67.47 for Qwen3-30B-A3B Thinking and 14.21 vs 11.18 for Llama-3.1-8B.

The observation that ALL-4 is close to the product of the four marginal accuracies supports the paper's central finding for the following reasons.

(1) Judgments grounded in propositional logic should have consistent correctness across representations. If a model bases its judgments on propositional logic, it should succeed or fail on the same instances across all four representations. ALL-4 should therefore be close to the accuracy of each representation. When ALL-4 instead approaches the value predicted under independence across representations, this suggests that whether an instance is answered correctly depends primarily on how it is expressed. This lack of reliability across representations can only be measured by matching the four versions of each instance.

For Hermes-4-70B, for example, ALL-4 could reach 96.36% if correct and incorrect predictions were fully aligned across representations. The prediction under independence is 92.93%, compared with the observed value of 93.19%. The near-independence you identify is precisely the lack of representation invariance that ALL-4 reveals.

(2) Predictions that agree across representations are more likely to be correct, supporting invariance as a measure of reliability. On instances for which all four representations yield the same answer, accuracy is substantially higher than overall accuracy. Thus, higher ALL-4 scores indicate that a model's answers are more trustworthy. Among the 29 configurations with ALL-4 below 80%, 27 achieve an accuracy of 78.1–100% on these consistently answered instances. The two exceptions are Gemma-4-31B and ERNIE-4.5-VL, whose agreement across representations arises from a bias toward a single label. Their J_I scores of 0.06 and −0.05, respectively, identify this behavior.

(3) ALL-4 is a distinct empirical measurement, while the four marginal accuracies only constrain its possible range. Predicting ALL-4 from their product requires assuming independence across representations, which is itself a property that ALL-4 is intended to examine. For configurations with higher error rates, this assumption is contradicted by the observed results.

ConfigurationALL-4 range permitted by the four marginal accuraciesPrediction under independence (product)Observed ALL-4
Llama-3.1-8B-Instruct0–50.57%11.18%14.21%
Qwen2.5-14B-Instruct0–56.62%21.35%42.60%
Qwen3-30B-A3B Instruct0–63.40%30.62%46.88%

Q1 (3): There is no noise baseline. Section 4.4 reports 15.03% item-level disagreement across three identical runs for Qwen3-30B-A3B Thinking, so much of its RepDis (31.31%) and of its roughly 13% specificity failures may be sampling noise rather than sensitivity to representation or intervention.

Let d denote the proportion of instances for which the answer changes when the same question is asked again. Under sampling noise alone, RepDis across the four representations is at most 2d. For all eight configurations evaluated over repeated runs, the observed disagreement across representations exceeds this baseline.

Sampling noise would affect the four representations equally and produce disagreements without a systematic directional bias. Our results, however, show strong biases in the errors associated with different representations. This pattern indicates that the inconsistency of Qwen3-30B-A3B Thinking arises from sensitivity to representation rather than sampling noise.

MetricNaturalStrippedCodeCosmeticOverall
Accuracy80.05%85.98%98.19%99.83%91.01%

Q1 (4): The metrics reward degenerate predictors. Always answering "entailed" yields RepDis = 0, ALL-4 = 50% and Spec = 100%. Gemma-4-31B-IT is essentially this case (Code 50.26, Cosmetic-Code 50.19, ALL-4 49.88, and more than 90% "entailed" predictions by the authors' own account), yet Section 4.2 interprets it as showing higher representation invariance than Llama-3.1-8B.

J_I was designed specifically to identify this behavior. A central argument of our work is that logical reliability must be assessed through multiple metrics jointly, and this example illustrates why that is necessary. A predictor that always answers entailed has J_I = 0. Gemma-4-31B obtains J_I = 0.06, the second-lowest score among the 74 configurations, indicating that it lacks reliable logical reasoning ability.

Q2 (1): The paper gives no prompts (in particular, how the task is stated for Code), temperature, seeds, maximum output tokens, per-model reasoning-effort settings, or unparsed-response rates, while unparsed outputs are counted as incorrect.

To support fair comparisons, we use a common set of instances, prompts, response formats, and parsing rules across the evaluated configurations. The only exceptions concern parameters that cannot be controlled externally for a small number of API models. The core findings of ReliLogic are derived from comparisons within each configuration and hold for every configuration evaluated.

All experiments reported in the main text follow the “first-line, single-token” protocol. For each instance, the prompt presents the premises and conclusion, provides the definition of entailment, and then specifies the required answer format. The definition reads: “A claim is 'entailed' only if it must be true in every situation where all the premises are true.” The decoding temperature is fixed at 0.6 except for API configurations that do not expose this parameter. The seeds used to generate the instances are 0, 1, and 2, and the maximum output length is set to 8,192 tokens for all models. We now report the reasoning settings and unparsed-response rates for all configurations in Table 14.

Prompt text:

[Problem]

A claim is "entailed" only if it must be true in every situation where all the premises are true.

On the first line of your reply, write exactly one of:
Answer: entailed
Answer: not-entailed

Manuscript revision: We have added the complete prompt in Figure 13 (p. 21) and an experimental-setup section describing the shared protocol, decoding temperature, and maximum output length (Appendix B.6, p. 25). Table 14 reports reasoning settings and UNPARSED rates for all 74 configurations (p. 22), and Tables 15–16 give their prediction distributions across the four representations (p. 23–24).


Q2 (2): About nine instruct configurations sit at roughly 50% on Code or Cosmetic-Code, and some fall below chance on a balanced binary task (Granite 3.3 8B at 41.12 on Cosmetic-Code, Devstral Small 2 at 43.93 on Code). This points to constant outputs or parse failures rather than weaker reasoning.

The near-50% results are primarily driven by a pronounced tendency to predict the same label across instances. We examined the two models you highlighted and confirmed that some responses could not be parsed. However, parsing failures are not the dominant explanation for their performance, and the weaknesses in their reasoning remain evident after restricting the evaluation to successfully parsed responses, as shown below. All models receive the same instructions and use the same response format, ensuring that the evaluation conditions remain consistent and fair.

Config. / Rep.AccUnparsed rateAcc on successfully parsed responsesnot-entailed
Granite-3.3-8B / Cosmetic41.12%5.27%43.41%89.86%
Devstral-Small-2-24B / Code43.93%5.80%46.63%71.45%

Q2 (3): In Appendix B.5, moving the answer from the first line to the final line raises Qwen3 8B (thinking off) from 80.29% to 98.60% on Stripped. If the main protocol is first-line and single-token, non-reasoning models are prevented from reasoning before answering, and a large share of the instruct-versus-reasoning gap and of the "unreliability" findings is a protocol effect. This belongs in the main text.

Thank you for highlighting the influence of answer placement.

Our protocol was chosen to ensure a consistent evaluation across configurations. Every configuration is tested on the same instances with the same prompts, response format, and parsing rules. The main-text experiments require a “first-line, single-token” answer, and all conclusions in ReliLogic are based on comparisons within a configuration.

Regarding the validity of this evaluation, the distinction between the two protocols is the order of the answer and the written reasoning. With a first-line answer, the model answers before presenting its analysis. With a final-line answer, it presents its analysis before answering. Both approaches are commonly used and widely accepted in LLM evaluation.

The results in Appendix B.5 show that reasoning before answering performs better on propositional entailment tasks. Our interpretation is that these tasks require successive deductions along a proof chain. Allowing the model to reason first enables it to write out intermediate conclusions and use each conclusion as the basis for the next step. When the answer must come first, the entire derivation has to be completed before the first answer token is generated.

Manuscript revision: We have added Section 4.6 and placed a brief comparison. Due to the page limit of the main text, we also expanded the discussion in Appendix B.5.

Reviewer_oh5EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReliLogic, a behavioral evaluation framework for the logical reasoning capabilities of large language models. The authors argue that final-answer accuracy alone is insufficient to determine whether a model truly and reliably makes predictions based on underlying logical relations. ReliLogic builds on automatically verifiable propositional-logic entailment tasks and supplements accuracy evaluation along three dimensions: (1) cross-representation reliability—whether the model maintains correct and consistent predictions across four logically equivalent input forms (Natural, Stripped, Code, and Cosmetic-Code); (2) selective logical response—whether the model correctly updates predictions when logical relations change, while preserving predictions under operations that do not alter logic (e.g., adding irrelevant premises, reordering premises, or renaming symbols); and (3) instance-level stability—whether the same model yields consistent results across repeated runs on identical instances.

The authors define metrics including ALL-4, Representation Disagreement, Intervention Sensitivity, Intervention Specificity, Youden-style JIJ_I, and Item-level Disagreement, and evaluate 74 model configurations. Experiments show that models with similar overall accuracy can differ substantially in cross-representation consistency and intervention responsiveness; overall accuracy can remain stable across runs while masking high per-item prediction inconsistency. Supplementary experiments further examine proof depth, distracting premises, distribution shift, external logical benchmarks, reasoning configurations, inference budget, and model scale.

Critical strengths

A unified, verifiable behavioral evaluation design. ReliLogic jointly examines four equivalent representations, logic-preserving vs. logic-altering interventions, and stability across repeated runs on the same underlying logical instances, with labels verified using formal tools. This allows different failure modes to be separated and compared under relatively controlled conditions rather than only observing final accuracy.

Critical weaknesses
  1. The core contribution is incremental relative to existing “beyond accuracy” logical evaluations, and the paper has not sufficiently established how it differs. Prior work has already studied logical-equivalence rewrites, contrastive edits, premise order, irrelevant perturbations, proof verifiability, and sensitivity to causally relevant vs. irrelevant interventions. ReliLogic’s main contribution appears to be unifying these dimensions in one framework and adding code representations and repeat-run stability, rather than proposing new evaluation principles. The related-work section summarizes these studies only in prose, without systematic comparison with RobustLR, Stolfo et al., and other closest frameworks on tasks, interventions, metrics, and new findings. It is therefore difficult to judge which capabilities ReliLogic measures uniquely.

  2. Insufficient task difficulty and coverage lead to clear saturation on strong models and weaken the framework’s discriminative power as a future benchmark. The main task involves only synthetic propositional logic, a limited set of seven inference rules, single-literal conclusions, and relatively shallow proof chains. Irrelevant premises involve new atoms that do not share variables with the proof chain, rather than highly confusable distractors that share entities or variables. GPT-5, Qwen3.5-397B, and several smaller reasoning models score near ceiling on almost all metrics; when proof depth increases from 2 to 4, accuracy drops for high-performing models are all under 5 percentage points. This makes the paper look more like a way to separate weak models or configurations without reasoning enabled, and does not convincingly show that the framework can diagnose logical reliability problems that frontier models still fail to solve. Although the authors report external benchmarks and distribution-shift results, they mainly use rank correlation for validation and have not shown that the new metrics provide independent information beyond ordinary accuracy on more complex tasks.

Questions for authors
  1. What are the specific ranges and distributions of proof depth dd and number of distractors kk in the main Representation Set? Can the authors report ReliLogic metrics under deeper proof chains and harder distractors—for example, distractor premises that share variables with the proof chain and form approximate but invalid proof paths? This would materially affect my judgment of whether the benchmark has lasting discriminative power.

  2. Compared with RobustLR, Stolfo et al., and existing logical contrastive-edit/invariance tests, which interventions and metrics in ReliLogic are not covered by prior frameworks? Can the authors add an item-by-item comparison and show whether ReliLogic reveals model behaviors that existing evaluations or their metrics cannot predict?

  3. On FOLIO, PrOntoQA, and ProofWriter, do ALL-4, JIJ_I, and Item-level Disagreement still reveal differences that accuracy does not? Current rank correlations mainly indicate that overall model rankings transfer; they do not show that the behavioral metrics proposed here still provide independent value on more complex tasks.

  4. At the same parameter scale, reasoning configurations improve all metrics far more than simply scaling parameters. Have the authors analyzed whether this gain comes from longer explicit reasoning, specific training, or output format/decoding strategy? The paper provides behavioral correlations but does not yet explain the mechanisms behind these failures and improvements.

5 comments

Thank you for your careful assessment and constructive suggestions. We appreciate the opportunity to clarify the benchmark's design and the additional insights provided by its reliability metrics.


Q1 (Q1 & Weakness2): What are the specific ranges and distributions of proof depth (d) and number of distractors (k) in the main Representation Set? Can the authors report ReliLogic metrics under deeper proof chains and harder distractors—for example, distractor premises that share variables with the proof chain and form approximate but invalid proof paths? This would materially affect my judgment of whether the benchmark has lasting discriminative power.

Problem difficulty is governed by the generator parameters and can be increased further. The main set deliberately targets a regime in which accuracy is close to saturation but reliability still varies across models. This is precisely where accuracy offers limited discrimination and additional metrics are most informative. We hope the response below addresses your concern as we understand it. If we have overlooked or misunderstood any aspect of your comment, we would appreciate further clarification and welcome continued discussion.

(1) Distributions of d and k in the Representation Set. In the main Representation Set, proof depth d ranges from 1 to 5, and the number of distractor premises is k ∈ {0, 2, 4}. The dataset is uniformly distributed across all 5 × 3 = 15 combinations of d and k, with 280 base instances and 1,120 questions per combination.

(2) Reliability remains discriminative even at near-perfect accuracy. Consider the 39 configurations whose accuracies fall within a narrow range of 1.95 percentage points (98.02–99.98%). Their ALL-4 scores span 7.17 percentage points (92.74–99.90%); 22 have RepDis ≥ 2%, including seven with RepDis ≥ 5%. Were the errors in all four representations concentrated on the same instances, the ALL-4 error rate would be close to the error rate in any single representation. The larger observed gap indicates that most such failures arise when a model answers an instance correctly in three representations but incorrectly in the remaining one.

The intervention results provide a complementary distinction. As illustrated below, Hermes-4-70B and GPT-5 Nano have nearly identical accuracy and ALL-4 scores, yet differ in their responses to logic-changing interventions: Hermes correctly revises its judgment in 100% of cases, whereas GPT-5 Nano does so in 95.33%. ReliLogic is designed to capture precisely these differences in failure behavior. Characterizing reliability among high-scoring configurations therefore requires considering the metrics together. Across all 74 configurations, only eight satisfy both ALL-4 ≥ 99.5% and J_I ≥ 0.995.

ConfigurationAccALL-4SensSpecJ_I
Qwen3-8B (Thinking ON)99.2697.2499.8199.971.00
Hermes-4-70B99.1993.19100.0099.921.00
GPT-5 Nano98.3093.8195.3397.060.92

(3) Greater complexity makes reliability differences more pronounced. The generator allows us to increase proof depth and the number of distractor premises. Under these more demanding conditions, ALL-4 declines by 3.3–4.4 times as much as accuracy. The existing structural out-of-distribution (OOD) experiments examine depths of 6–10, 8 or 16 distractor premises, held-out rule sequences, and lexical transfer. The following results illustrate the changes for four configurations whose original accuracies exceed 98%.

ConfigurationAccALL-4RepDis ↓SensSpecJ_I
Gemini 3.1 Pro (4096)99.54→99.1298.57→96.881.43→3.1298.15→97.6898.91→98.350.97→0.96
Qwen3-8B Thinking ON99.26→98.6097.24→94.692.76→5.3199.81→99.1499.97→99.671.00→0.99
GPT-5 Nano98.30→97.5393.81→90.456.19→9.5595.33→92.6097.06→95.980.92→0.86
Hermes-4-70B99.19→97.0593.19→89.486.81→10.52100.00→98.5699.92→99.241.00→0.98

(4) More challenging distractor premises. We introduce distractors that share atoms with the proof chain and create plausible but invalid proof paths. For example, starting from P = {p, p → q, q → r} with conclusion c = r, we remove p → q and add one of the following distractor pairs: a reversed implication (p → s, r → s), a polarity mismatch (p → s, ¬s → r), or a missing link (p → s, t → r). The resulting metrics for four high-scoring configurations are reported below.

ConfigurationAccALL-4RepDis↓SensSpecJ_I
Gemini 3.1 Pro (4096)93.1986.1712.1491.3597.580.89
Qwen3-8B Thinking ON96.5791.527.4696.3298.630.95
GPT-5 Nano90.7481.2216.6986.9292.260.79
Hermes-4-70B94.8783.5715.1195.1698.080.93

Manuscript revision: Appendix A.2 now states the ranges and distribution of proof depth and distractor count in the Representation Set (p. 15).

Q2 (Q2 & Weakness1): Compared with RobustLR, Stolfo et al., and existing logical contrastive-edit/invariance tests, which interventions and metrics in ReliLogic are not covered by prior frameworks? Can the authors add an item-by-item comparison and show whether ReliLogic reveals model behaviors that existing evaluations or their metrics cannot predict?

Prior frameworks typically focus on individual properties, particularly behavior under logical interventions. ReliLogic brings together three complementary aspects of logical reliability: consistency across four representations, intervention metrics conditioned on correct predictions for the original instances, and item-level stability across repeated runs. These aspects are evaluated jointly on variants derived from the same base instances, exposing different failure modes from complementary perspectives. Our statistics and paired comparisons with the original instances also make the findings interpretable and allow failures to be traced to specific instances and evaluation conditions.

MethodNumber of representationsLogic-preserving interventionsLogic-changing interventionsRepeated runsScoring
RobustLR1Logically equivalent reformulationsChanges to logical connectives; introduction of negation—Accuracy on each perturbation set
Stolfo et al.1Changes to surface text that do not affect the answerChanges to numerical operands and arithmetic operators—Causal effects on the output distribution
Chen et al.; Jiang et al.1Premise shuffling; substitution of words irrelevant to the inference——Accuracy changes; hypothesis tests for bias
Tian et al.1Perturbations——Accuracy; robustness; traceability
ReliLogic4Addition of irrelevant premises; premise shuffling; renamingConclusion negation; removal of premises used in the proof; flips in literal polarity; interchange of ∧ and ∨Three runs with item-level comparisonsSens, Spec, and J_I computed only for correctly answered original instances; ALL-4 and RepDis

Behaviors revealed by combining the tests:

Evaluation in isolationWhat that evaluation would suggestWhat the combined evaluation reveals
Testing only irrelevant changes, as in Chen et al. and Jiang et al.After premise shuffling, Gemma-4-31B remains correct in 97.3% of cases, comparable to Gemini 3.5 Flash (97.9%).When the conclusion is negated, it changes its answer in only 1.0% of cases, with J_I = 0.06. Its apparent “robustness” reflects an almost invariant preference for entailed.
Testing interventions in a single representation, as in RobustLR and Stolfo et al.Thirty-one configurations achieve J_I ≥ 0.99.Eight of these configurations nevertheless change their judgments when the same problems are presented as code.
Evaluating only one supposedly “hardest” representationA single representation could serve as a proxy for the others.Among 22 high-scoring configurations with significant performance gaps across representations, the worst-performing representation is Code for eight, Natural for six, Stripped for five, and Cosmetic for three.
Comparing only aggregate scores across repeated runsThe standard deviation of accuracy is ≤ 0.34 percentage points.Item-level disagreement ranges from 0.93% to 22.89%.

Manuscript revision: We have added a comparison in Table 1 (p. 2). The related-work discussion now points to this table (Section 2, p. 4).

Q3: On FOLIO, PrOntoQA, and ProofWriter, do ALL-4, (J_I), and Item-level Disagreement still reveal differences that accuracy does not? Current rank correlations mainly indicate that overall model rankings transfer; they do not show that the behavioral metrics proposed here still provide independent value on more complex tasks.

We compare each configuration's accuracy on the three external benchmarks with its three reliability metrics on ReliLogic. These results show that configurations with comparable accuracy, including those ranked highly on the external benchmarks, can differ substantially in their reliability on ReliLogic.

ConfigurationFOLIOPrOntoQAProofWriterReliLogic ALL-4ReliLogic J_IReliLogic item-level disagreement
Gemini 3.1 Pro (4,096)89.00%100.00%99.25%98.29%0.9710.76%
Gemini 3.5 Flash (medium)89.75%100.00%98.50%92.33%0.9343.38%
GPT-5 Nano85.75%98.00%100.00%93.90%0.9242.74%
Hermes-4-70B87.00%98.25%80.50%93.19%0.9992.75%
Qwen3-8B ON86.75%97.75%97.25%97.05%0.9960.93%
Ministral-3-14B Reasoning85.50%89.00%92.00%90.19%0.9543.53%
Reviewer_y3CDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces ReliLogic, a framework for evaluating logical reasoning beyond aggregate accuracy. It combines four equivalent representations of propositional logic problems, interventions that preserve or change entailment labels, and repeated evaluations. Experiments across 74 model configurations examine consistency across representations, responsiveness to logical changes, and prediction stability.

Critical strengths
  • The paper addresses an important limitation of evaluating logical reasoning through aggregate accuracy alone. The results reveal differences in model behavior that similar accuracy scores can obscure.

  • The experiments cover a broad range of model configurations. The use of SymPy and exhaustive truth tables to verify labels provides a sound basis for the controlled evaluations.

Critical weaknesses
  • The synthetic problems are generated from seven predefined inference rules and exhibit substantial performance saturation. In Table 11, 39 of the 74 configurations achieve at least 98% overall accuracy, while some also approach perfect scores on the proposed reliability metrics. This limits the benchmark’s ability to distinguish stronger models and measure further progress in logical reasoning.

  • The natural language instances are generated using fixed templates, with entities and properties serving only as names for atomic propositions. This restricted language can, in principle, be parsed back into the underlying propositional formulas using the known templates. Consequently, the benchmark provides limited coverage of the semantic ambiguity and interpretation challenges encountered in less controlled natural language reasoning problems.

Questions for authors
  • Would increasing problem complexity reveal reliability differences among the models that currently approach perfect scores?

  • How would model accuracy and reliability change if the problems were expressed in more varied natural language rather than fixed templates?

4 comments

We sincerely thank you for your thoughtful comments and for raising these important concerns. We address them below.


Q1 (Q1 & Weakness1): Some approach perfect scores on the proposed reliability metrics. This limits the benchmark’s ability to distinguish stronger models and measure further progress in logical reasoning. Would increasing problem complexity reveal reliability differences among the models that currently approach perfect scores?

ReliLogic retains its ability to distinguish models even when accuracy approaches saturation. Moreover, problem complexity can be increased through the generator parameters, amplifying reliability differences among configurations with high accuracy.

(1) Reliability metrics continue to distinguish configurations in the accuracy saturation regime. For the 39 configurations whose accuracies span only 1.95 percentage points (98.02–99.98%), ALL-4 spans 7.17 percentage points (92.74–99.90%). Among these configurations, 22 have RepDis ≥ 2%, and seven have RepDis ≥ 5%. If errors across all representations occurred on the same instances, the ALL-4 error rate would be approximately equal to the error rate for an individual representation. The observed gap therefore indicates that most of these errors involve an instance being answered correctly in three representations but incorrectly in the fourth. The table below provides a further example: Hermes-4-70B and GPT-5 Nano have similar accuracy and ALL-4 scores, yet the intervention tests distinguish them. Under logic-changing interventions, Hermes correctly updates its judgment in 100% of cases, compared with 95.33% for GPT-5 Nano. Such failure modes are precisely what ReliLogic is designed to measure, demonstrating the need to consider multiple metrics jointly when characterizing reliability differences among configurations with high accuracy. Of the 74 configurations, only eight achieve both ALL-4 ≥ 99.5% and J_I ≥ 0.995.

ConfigurationAccALL-4SensSpecJ_I
Qwen3-8B (Thinking ON)99.2697.2499.8199.971.00
Hermes-4-70B99.1993.19100.0099.921.00
GPT-5 Nano98.3093.8195.3397.060.92

(2) Increasing problem complexity amplifies reliability differences among configurations with high accuracy. Difficulty is controlled by the generator parameters. When we increase reasoning depth and the number of distractor premises, the decline in ALL-4, for example, is 3.3–4.4 times the decline in accuracy. Our existing OOD experiments cover depths of 6–10, 8 or 16 distractor premises, held-out rule sequences, and lexical transfer. The table below shows the changes for four configurations with original accuracies above 98%.

ConfigurationAccALL-4RepDis ↓SensSpecJ_I
Gemini 3.1 Pro (4096)99.54 → 99.1298.57 → 96.881.43 → 3.1298.15 → 97.6898.91 → 98.350.97 → 0.96
Qwen3-8B Thinking ON99.26 → 98.6097.24 → 94.692.76 → 5.3199.81 → 99.1499.97 → 99.671.00 → 0.99
GPT-5 Nano98.30 → 97.5393.81 → 90.456.19 → 9.5595.33 → 92.6097.06 → 95.980.92 → 0.86
Hermes-4-70B99.19 → 97.0593.19 → 89.486.81 → 10.52100.00 → 98.5699.92 → 99.241.00 → 0.98

Q2 (Q2 & Weakness2): The benchmark provides limited coverage of the semantic ambiguity and interpretation challenges encountered in less controlled natural language reasoning problems. How would model accuracy and reliability change if the problems were expressed in more varied natural language rather than fixed templates?

The use of fixed templates to generate instances is a deliberate design choice. It ensures that the underlying logic remains consistent and that each problem has a unique correct answer. A change in a model's prediction when the same problem is expressed differently can therefore be attributed to model reliability rather than to a change in the problem itself. We observe model failures even under these controlled conditions; more complex natural-language reasoning would present further challenges.

(1) Lexical substitution. In the lexical transfer experiment in Figure 9, we replace all names and property terms with a new vocabulary not used in the main experiments, while preserving the underlying logic. Across all 10 configurations, accuracy remains essentially unchanged or improves slightly, with changes ranging from −0.13 to +2.51 percentage points.

(2) More complex natural-language paraphrases. We preserve the original logic of each problem while introducing more complex syntactic paraphrases, including active–passive alternations and clause reordering. Each paraphrase is checked programmatically for logical equivalence to the original instance, followed by manual spot checks. Results for four models are shown below; each cell reports original / paraphrased results.

ConfigurationOverall accuracyALL-4RepDis ↓SensSpecJ_I
Gemini 3.1 Pro (4096)99.54 / 98.2398.57 / 96.211.43 / 3.7998.15 / 95.4398.91 / 96.860.97 / 0.92
Hermes-4-70B99.19 / 97.4593.19 / 90.416.81 / 9.59100.00 / 98.7099.92 / 97.881.00 / 0.97
Qwen3-8B Thinking ON99.26 / 98.4997.24 / 95.482.76 / 4.5299.81 / 99.6899.97 / 99.481.00 / 0.99
Qwen3-8B Thinking OFF72.05 / 53.8139.40 / 36.6054.82 / 57.0249.09 / 24.8388.62 / 88.990.36 / 0.14

Thanks to the authors for their response and clarifications. I will keep my score.

Dear Reviewer y3CD,

Thank you for your thoughtful review, constructive feedback, and consideration of our responses. We sincerely appreciate your continued support.

Best regards,
The Authors

Reviewer_mR7xOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors propose ReliLogic, which is an evaluation framework for logical reasoning in LLMs. In the proposed evaluation framework, the authors argue that aggregate accuracy hides important failures. It uses verified propositional-logic entailment problems and has two components. The Representation Set renders 4,200 instances in four equivalent forms: natural language, abstract symbols, executable code, and code-like strings. The Intervention Set has 2,365 cases with logic-preserving edits (reorder, rename, neutral premise) and logic-changing edits (negate, delete, flip, swap). The introduced evaluation metrics measure joint correctness across forms (ALL-4) and prediction disagreement across forms (RepDis). They also measure responsiveness to interventions (Sens_I and Spec_I, combined as a Youden-style J_I) and item-level disagreement across repeated runs. Across 74 model configurations, high accuracy often coexists with these failures. Qwen3-30B-A3B Thinking scores 91.01% accuracy but only 68.69% ALL-4. Code is the weakest form for 33 configurations. DeepSeek-R1-0528 reaches only 53.49% Sens_I. Gemma-4-31B-IT reaches only 8.61% Sens_I. Eight configurations were each run three times. Their aggregate accuracy varies by at most 0.34% SD, yet 0.93–22.89% of instances receive differing predictions across runs. Errors are more frequent on missing-premise negatives and on deeper proofs. The appendix reports four further findings. Model rankings correlate with results on the datasets like ProofWriter, PrOntoQA and FOLIO (ρ=0.821, 10 configurations). Enabling reasoning modes improves accuracy, ALL-4 and J_I. Gains from a larger token budget saturate beyond 1,024 tokens. Parameter count correlates only weakly with accuracy.

Critical strengths

There are two strength points that I pinpoint in this paper:

First of all, this evaluation framework provides verified labels to make the failures attributable to the models. Every evaluation instance is checked by both SymPy and exhaustive truth tables. It is kept only if both confirm the intended label. Premise sets must be satisfiable. Intervened instances are re-verified, and those with mismatched labels are discarded. Sens_I and Spec_I are conditioned on a correct base answer. This matters because the paper’s core claims depend on exact ground truth. The claims are that models change answers across equivalent forms or ignore logic-changing edits. Double verification rules out label noise as an explanation.

Second, across 74 configurations, it is unveiled that different LLMs with near-identical accuracy diverge on the new metrics. Llama-3.1-8B and Gemma-4-31B-IT score 58.36% and 57.45% accuracy, but 14.21% and 49.88% ALL-4. Qwen2.5-Coder-14B and Gemma-4-26B-A4B-IT match on accuracy and ALL-4, but their J_I is 0.36 vs 0.60. Gemma-4-31B-IT has a 97.35% Spec_I but only an 8.61% Sens_I, which reveals an entailed bias that accuracy hides. Repeated runs keep aggregate SD at or below 0.34%, yet up to 22.89% of instances receive differing predictions.

Critical weaknesses

The novelty of the logic correctness evaluation needs to be elaborated. As cited by this paper, Sanyal et al. (2022) proposed a sensitivity and consistency test of logical edits under equivalent formulations. As well, Chen et al. (2024) investigated premise-order effects, which the Reorder intervention re-tests. The metrics are standard: ALL-4 is joint accuracy and RepDis and item-level disagreement are disagreement rates. Though the introduction claims that equivalent-representation reliability has received “comparatively limited attention, this paper never compares directly with these frameworks, and Appendix B.2 compares only accuracy rankings. The contribution is therefore a combination of known tests. The paper does not show that this combination reveals anything that existing frameworks would miss.

I have also read through the appendices. In Appendix B.5, it shows strong format sensitivity for Qwen3-8B with turning off the thinking mode. This check covers only three LLM choices, two of them strong and robust LLMs. It also uses only the Stripped form. It does not include the four weaker LLMs behind the main-text contrasts. Also, it does not include Code, where the largest drops occur. On Code and Cosmetic-Code, threre are four weaker models scoring close to 50%, which is close to a random chance of hit. In theory, random guessing, a constant answer, or frequent unparsed outputs counted as wrong may all produce such observations in LLMs' responses. However, the paper reports neither prediction distributions nor unparsed rates, so these potential causes cannot be told apart.

Questions for authors

1/ Could you elaborate on the difference between your work and previous works like Sanyal et al. and Chen et al.? Did you just present a combination of existing evaluation frameworks ? Can you elaborate on the novelty in your evaluation framework ?

2/ For each model and each form, what is the predicted label distribution and the UNPARSED rate? I am most interested in Llama-3.1-8B-Instruct, Gemma-4-31B-IT, Qwen2.5-Coder-14B and Gemma-4-26B-A4B-IT on Code and Cosmetic-Code.

3/ Is the B.5 “first-line, single-token” baseline the protocol for all main-text results? If so, does it require non-thinking models to answer before any written reasoning? In addition, can you provide the prompts and decoding temperatures used in your evaluation?

5 comments

We sincerely thank you for your time and valuable feedback. We address your remaining concerns below.


Q1 (Q1 & Weakness1): Could you elaborate on the difference between your work and previous works like Sanyal et al. and Chen et al.? Did you just present a combination of existing evaluation frameworks ? Can you elaborate on the novelty in your evaluation framework ?

ReliLogic advances prior work through four representations of the same logical instances, intervention metrics conditioned on correct predictions for the original instances, and instance-level stability across repeated runs. Together, these dimensions reveal distinct failure modes. The statistics we report, together with paired comparisons against the original instances, make the results readily interpretable and allow us to identify precisely which instances fail and under what conditions.

MethodNumber of representationsLogic-preserving interventionsLogic-changing interventionsRepeated runsScoring
Sanyal et al.1Logically equivalent reformulationsModifying logical connectives; introducing negation—Accuracy on each perturbation set
Chen et al.1Shuffling premises; substituting words irrelevant to the logical inference——Changes in accuracy; tests for bias
ReliLogic4Adding irrelevant premises; shuffling premises; renamingNegating the conclusion; removing premises used in the proof; flipping literal polarity; interchanging ∧ and ∨Three runs, compared at the instance levelSens, Spec, and J_I, conditioned on correct predictions for the original instances; ALL-4 and RepDis

Comparison of diagnostic findings:

Evaluation approachWhat the prior approach would revealWhat ReliLogic reveals
Testing only premise shuffling, as in Chen et al.Gemma-4-31B shows almost no change after premise shuffling (97.3%) and appears as “robust” as Gemini 3.5 Flash (97.9%).When the conclusion is negated, Gemma-4-31B updates its prediction in only 1.0% of cases, with J_I = 0.06. In the Code representation, it predicts entailed on 99.7% of instances. Its apparent “robustness” stems from predicting the same label almost invariably.
Testing interventions within a single representation, as in RobustLRThirty-one configurations achieve near-perfect accuracy on the instances after intervention, suggesting reliable logical reasoning.Eight of these configurations change their predictions when the same instances are expressed in the Code representation.

Manuscript revision: We have added Table 1 (p. 2), which compares ReliLogic with prior evaluations in terms of representations, logic-preserving and logic-changing interventions, conditioning on correctly answered base instances, repeated runs, and metrics. Section 2 now refers to this comparison (p. 4).

Q2 (Q2 & Weakness2): The paper reports neither prediction distributions nor unparsed rates, so these potential causes cannot be told apart. For each model and each form, what is the predicted label distribution and the UNPARSED rate? I am most interested in Llama-3.1-8B-Instruct, Gemma-4-31B-IT, Qwen2.5-Coder-14B and Gemma-4-26B-A4B-IT on Code and Cosmetic-Code.

Thank you for your careful reading and constructive comments. Your questions have helped us improve the paper.

We have added prediction distributions for all 74 configurations across the four representations in Tables 15–16, together with configuration-level UNPARSED rates and reasoning settings in Table 14. Accuracy near 50% is primarily attributable to models consistently predicting a single label. Our comparisons vary the representation while keeping the model, prompt, decoding settings, and response format fixed. The differences in accuracy across the four representations are therefore attributable to the representation itself. Moreover, even when considering only successfully parsed responses, 50 of the 74 configurations perform worse on Code than on Natural. The same model can distinguish entailed from not-entailed instances in natural-language and symbolic representations, yet predict almost exclusively one label when the instances are expressed as code. This is precisely the failure mode associated with changes in representation that ReliLogic is designed to reveal.

The prediction distributions for the four models of particular interest in the Code and Cosmetic representations are shown below. Each representation contains 4,200 instances, evenly divided between entailed and not-entailed.

ModelRepresentationAccuracyPredicted entailedPredicted not-entailedUNPARSED
Llama-3.1-8B-InstructCode50.86%2.67%97.33%0.00%
Cosmetic50.57%45.24%54.40%0.36%
Gemma-4-31B-ITCode50.26%99.74%0.26%0.00%
Cosmetic50.19%99.81%0.19%0.00%
Qwen2.5-Coder-14BCode54.74%8.50%91.48%0.02%
Cosmetic55.10%13.48%86.50%0.02%
Gemma-4-26B-A4B-ITCode50.10%99.90%0.10%0.00%
Cosmetic50.12%99.88%0.12%0.00%

For comparison, the same four models achieve accuracies of 62.76–80.36% on Natural / Stripped: 69.24% / 62.76% for Llama, 64.55% / 64.79% for Gemma-4-31B, 77.60% / 73.45% for Qwen2.5-Coder, and 80.17% / 80.36% for Gemma-4-26B. The highest parsing failure rate in these conditions is 0.14%.

Regarding the three possible explanations raised in your question:

  • Consistently predicting a single label (the primary explanation): In seven of the eight conditions, one label accounts for at least 86.50% of predictions.
  • Random guessing: Only Llama in the Cosmetic condition exhibits this pattern, with approximately balanced predictions (45.24% / 54.40%) and an accuracy of 50.57%.
  • Parsing failures: These are negligible, with a maximum rate of 0.36% across the 16 conditions.

Regarding Appendix B.5: The response-format control experiments for the four weaker models are reported below. All values are accuracies (%).

ModelProtocolNaturalStrippedCodeCosmetic
Llama-3.1-8BFirst-line69.2462.7650.8650.57
Final-line80.1674.8863.3862.76
JSON-only64.6957.4550.8750.25
Multi-token61.3053.2650.0850.19
Gemma-4-31BFirst-line64.5564.7950.2650.19
Final-line91.8892.2076.5277.76
JSON-only61.7962.1252.3350.07
Multi-token57.4159.1751.0850.13
Qwen2.5-Coder-14BFirst-line77.6073.4554.7455.10
Final-line89.7886.1172.5574.38
JSON-only74.2969.3253.6655.70
Multi-token72.5368.5652.3553.27
Gemma-4-26B-A4BFirst-line80.1780.3650.1050.12
Final-line93.3494.5578.4277.48
JSON-only77.8577.9151.7551.23
Multi-token74.0975.3550.1450.31

Manuscript revision: We have added prediction distributions for all 74 configurations across the four representations in Tables 15–16 (pp. 23–24). Table 14 reports the UNPARSED rate and reasoning setting for each configuration (p. 22).

Q3: Is the B.5 “first-line, single-token” baseline the protocol for all main-text results? If so, does it require non-thinking models to answer before any written reasoning? In addition, can you provide the prompts and decoding temperatures used in your evaluation?

All configurations are evaluated using the same instances, prompts, response format, and parsing rules to ensure fair comparisons as far as possible. The only exceptions concern parameters that cannot be externally controlled for a small number of API models. ReliLogic's core conclusions are based on comparisons within each configuration and hold for every configuration. All results reported in the main text use the “first-line, single-token” response protocol. The prompt first presents the problem, including its premises and conclusion, then defines entailment (“A claim is "entailed" only if it must be true in every situation where all the premises are true”), and finally provides the response instructions. We use a decoding temperature of 0.6 for all models except API configurations that do not allow this parameter to be set.

Prompt text:

[Problem]

A claim is "entailed" only if it must be true in every situation where all the premises are true.

On the first line of your reply, write exactly one of:
Answer: entailed
Answer: not-entailed

Manuscript revision: We have added the full prompt in Figure 13 (p. 21) and the evaluation protocol, decoding temperature, and maximum output length in Appendix B.6 (p. 25).

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

We sincerely thank all reviewers for the thoughtful feedback and constructive discussion, which helped us further clarify our results. As the discussion period closes, We briefly summarize the discussion below.

Strengths highlighted in the reviews

  • Verified, instance-level evaluation framework. ReliLogic evaluates logical reliability through four logically equivalent representations, logic-preserving and logic-changing interventions, and item-level comparisons across repeated runs. Labels are checked by both SymPy and exhaustive truth-table enumeration, help to distinguish different reasoning failures on individual instances.
  • Evidence beyond accuracy. Our evaluation of 74 configurations reveals reliability differences even among models under accuracy. The reviews also highlighted the controlled comparisons of reasoning modes and answer formats.

Discussion outcomes and revisions

Reviewer mR7x confirmed that our responses addressed their concerns and maintained their rating of 3. We clarified the distinctions from prior frameworks, added prediction distributions, and documented the prompt and evaluation settings.

Reviewer y3CD confirmed that our responses maintained their rating of 3. Our response demonstrated how reliability metrics distinguish high-scoring configurations and provided additional results with greater problem complexity and more varied natural-language formulations.

Reviewer 7XzV also confirmed that their concerns were resolved and raised their rating to 3. We clarified the complementary roles of the metrics, the relationship between ALL-4 and marginal accuracies, and the analysis of repeated-run variability. We also added the protocol analysis, documented model-specific reasoning settings, and explained the different instance sets underlying Fig.5 and the main tables.

Reviewer oh5E has not yet followed up. Our response addresses every weakness and question raised:

  • Difficulty and discriminative power (Q1 & W2). We specified the ranges and distributions of proof depth and distractor count. We also reported results for deeper proofs and more challenging distractors that share atoms with the proof chain and create plausible but invalid proof paths.
  • Comparison with prior frameworks (Q2 & W1). We provided a comparison of representations, intervention types, scoring conditions, and repeated-run evaluation, now included in Table 1. Concrete examples demonstrate the diagnostic value of combining these dimensions.
  • Reliability beyond accuracy (Q3). We compared accuracy on FOLIO, PrOntoQA, and ProofWriter with the corresponding configurations’ reliability metrics on ReliLogic. Configurations with similar external-benchmark accuracy exhibit substantial differences in representation consistency, intervention responses, and stability across runs.
  • Reasoning configurations and protocols (Q4). We provided comparisons examining thinking modes, answer placement, output formats, reasoning budgets, and reasoning effort. Section 4.6 now presents the answer-placement and tokenization analysis, with further discussion in Appendix B.5.

The manuscript revisions are highlighted in blue. Thank you for considering our responses and revisions.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.