acceptodds
Under review as a conference paper at ICLR 2027

Evidence Over Memory: Do Long-Context LLMs Reason from What They Read?

Abstract

Long-context models can read an entire scientific paper, but do their conclusions actually depend on what they read? We introduce MainLogic, a benchmark that tests selective responsiveness: a model should change its judgment when decisive evidence changes, withhold judgment when evidence becomes insufficient, and remain stable when irrelevant content changes. Starting from human-verified scientific citations, we construct matched families that systematically vary the claim and its evidential basis while preserving the underlying scientific problem. Across ten frontier models, performance on natural claims substantially overstates evidence-grounded reasoning: models reaching over 90% accuracy on the original claims correctly solve at most 32.4% of complete evidence-intervention families. Moreover, when supplied evidence conflicts with a model’s independently measured prior, accuracy drops by 6–51 points, and errors systematically drift toward that prior even when the evidence is insufficient. MainLogic reveals a gap between reading evidence and letting evidence determine the answer.

Primary area:
datasets and benchmarks
Keywords:
long-context, reasoning, benchmarking

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.60 over 5 reviews, above 70% of the others.
Reviewer_LyuKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MainLogic, a benchmark for evaluating whether long-context LLMs reason from provided evidence rather than relying on prior knowledge. It constructs matched scientific claims through evidence reversal and removal. Experiments across ten frontier models reveal substantial performance drops under evidence interventions and conflicts with measured priors.

Critical strengths
  1. The paper addresses an important problem of evidence-grounded reasoning in LLMs.
  2. The matched evidence-intervention design provides a more rigorous evaluation than conventional accuracy metrics.
  3. The experiments cover ten frontier models and provide useful analyses of prior–evidence conflicts and evidence accessibility.
Critical weaknesses
  1. The reliability of prior elicitation is unclear. Evidence-free judgments may be sensitive to prompt wording, answer-option ordering, and response format. Have the authors tested whether the measured priors remain stable across different prompt configurations?
  2. The quality of automatically constructed counterfactual evidence is concerning. The human audit found defects in 6 out of 40 sampled anchors. How can the authors ensure the validity of the remaining samples? An evaluation restricted to verified anchors would be helpful.
  3. The novelty over existing benchmarks is not entirely clear. Previous works have studied counterfactual evidence, contextual faithfulness, and evidence sufficiency. What additional evaluation capabilities does MainLogic provide beyond combining these existing approaches?
Questions for authors

See weaknesses.

7 comments

We sincerely thank the reviewer for their encouraging review and the time they gave to our work. Their concern about item quality led us to substantially expand our human audit, as detailed below.

W1: Reliability of prior elicitation

Evidence-free judgments may be sensitive to prompt wording, answer-option ordering, and response format. Have the authors tested whether the measured priors remain stable across different prompt configurations?

We thank the reviewer for this question. We re-ran the evidence-free probe with reworded instructions and with the answer options reversed, and recomputed RQ2 using only priors that agree across all three configurations (new Appendix L, Tables 22–23).

1. Stability across probe configurations. We re-probed all 216 claims (108 original claims and their contrary versions) on four models, with two additional configurations:

  • Reworded: "Is the following statement correct? Reply Supported if it is, Contradicted if it is not, or Uncertain if you cannot tell. Reply with one word."
  • Reversed: the original wording with the three answer options in reverse order.

The one-word response format and the decoding settings are unchanged, with one response per claim in each configuration. A prior is stable when all three probes give the same definite answer (Supported or Contradicted). 78–91% of definite priors are stable. Agreement with each alternative probe is 0.82–0.95 (κ 0.68–0.86).

Table 22 (Appendix L). Probe stability. Stable percentages use the definite priors as denominator. Flips are Supported↔Contradicted reversals among definite priors across the two alternative configurations.

SystemDefiniteStable across all threeReworded: agreement / κReversed: agreement / κFlips
GPT-6 Astra182147 (81%)0.86 / 0.760.82 / 0.723%
GPT-5.6 Luna201157 (78%)0.87 / 0.740.83 / 0.688%
GPT-5.6 Terra204165 (81%)0.84 / 0.700.84 / 0.7012%
GPT-4.1208189 (91%)0.88 / 0.680.95 / 0.865%

As a further check on response format, an earlier run of the same claims on seven models, with a different user-message format, gives κ 0.48–0.82 and points the same way on 85–99% of the claims where both runs give a definite answer.

2. RQ2 restricted to stable priors. We recomputed the three prior-conditioned measures using only stable priors. The alignment gap and ErrDir increase for all four models. DirComp stays high, at 85–92%: it increases for three models and moves from 93% to 92% for GPT-6 Astra.

Table 23 (Appendix L). Reported → stable-prior results. Brackets are anchor-bootstrap 95% intervals. Gap is the accuracy difference between prior-aligned and prior-conflicting evidence; ErrDir and DirComp are the share of prior-consistent answers among conflict errors and among determinate answers under insufficient evidence.

SystemGap: reported → stableErrDir: reported → stableDirComp: reported → stable
GPT-6 Astra0.06 [−0.01, 0.13] → 0.11 [0.03, 0.19]0.70 → 0.740.93 → 0.92
GPT-5.6 Luna0.30 [0.22, 0.38] → 0.33 [0.22, 0.43]0.70 → 0.720.79 → 0.85
GPT-5.6 Terra0.23 [0.15, 0.31] → 0.25 [0.16, 0.35]0.70 → 0.740.85 → 0.87
GPT-4.10.50 [0.42, 0.58] → 0.56 [0.48, 0.64]0.84 → 0.850.84 → 0.88

For GPT-6 Astra, the alignment gap rises from 6 to 11 points, and its interval moves from [−1, 13] to [3, 19]. All four stable-prior gaps have intervals that exclude zero.

Conclusion. The central RQ2 findings hold under a stricter definition of the prior. Models remain more accurate when the evidence agrees with their measured prior, and their errors and unsupported commitments still mostly follow that prior.

W2: Quality of the constructed evidence, and evaluation on verified examples

The human audit found defects in 6 out of 40 sampled anchors. How can the authors ensure the validity of the remaining samples? An evaluation restricted to verified anchors would be helpful.

We thank the reviewer. We ran a new per-row human review of 320 further rows (new Appendix R), and evaluated every system both on the 34 anchors that passed the original audit and on the 337 rows that pass review (Appendices Q and S).

The per-row review replaces the original protocol, which reviewed one sampled row and, if it passed, counted the whole six-cell anchor as verified. Reviewing every row on its own catches a poor edit, a weak negation or an incomplete evidence removal in the row where it occurs, independently of the other rows of the anchor.

1. What the original six failures were. The six anchors failed for different reasons (Appendix Q, Table 31). Three have correct labels but quality problems, two with a bad claim and one with a negation that is too easy. The other three affect labels. In two, the contrary claim is arguably not contrary, and in one, the evidence removal left behind a passage that still settles the claim. Together these affect at most 8 core cells (the six main cells of each anchor, excluding the deletion controls). Excluding all six anchors changes Acc6 by at most 1.1 points, and the ranking is unchanged (Appendix Q, Table 32).

2. How we ensure the validity of the remaining samples. The extended audit covers 320 rows, 40 from each of the eight cells (a cell is one combination of premise version and claim), spanning 105 anchors and 26 citing papers. Together, the two audits cover 368 rows from 106 of the 108 anchors. The auditor first judges the claim from the documents without seeing the reference label, then checks the claim, the contrary claim, and whether the edits or deletions are complete.

Extended-audit outcomeRowsShare
Passed29491.9%
Construction flawed, label still intended (residual evidence or ambiguous wording)237.2%
Label in doubt30.9%
Total320100%

All 26 failing rows are excluded from the validated subset. The auditor's blind judgment agrees with the reference label on 90.6% of all 320 rows and 96.6% of the 294 passing rows (Appendix R, Table 33). The main weakness is evidence removal, which leaves a passage that still settles the claim in 16 of the 80 P0P^0 rows. We now report this in the paper.

3. Evaluation restricted to verified examples. We report two restricted evaluations:

  • the 34 anchors that passed the original audit, as the reviewer asks;
  • the 337 individually validated rows from both audits, of which 250 are core rows. Acc6 is computed on these 250 and reweighted to match the full benchmark's mix of cells.

Values are percentages, from Appendix Q (Table 32) and Appendix S (Table 34).

SystemAll 108 anchors: Acc6 / GC634 audit-passing anchors: Acc6 / GC6Validated core rows: Acc6
Claude Fable 5.177.5 / 28.779.4 / 35.380.0
GPT-6 Astra75.8 / 32.478.4 / 35.377.2
Claude Opus 572.1 / 21.374.5 / 26.574.0
DeepSeek Flash70.8 / 19.471.6 / 20.673.7
Gemini 3.8 Flash69.4 / 20.472.5 / 26.569.7
Claude Sonnet 567.5 / 17.967.0 / 15.269.7
GPT-5.6 Terra65.6 / 13.063.2 / 8.867.1
Gemini 2.5 Flash63.7 / 12.062.3 / 8.862.9
GPT-5.6 Luna62.0 / 11.164.7 / 14.763.6
GPT-4.153.1 / 2.859.3 / 8.854.9
  • On individually validated rows, Acc6 is within −0.8 to +2.9 points of the headline value, and the ranking holds apart from two near-ties.
  • On the 34 audit-passing anchors, a third of the benchmark, scores move more (−2.4 to +6.2) and some mid-table systems swap; the top three and the last are unchanged.
  • GC6 needs all six cells of an anchor to be validated, which the row-level subset does not yet provide, so we report it only for the 34-anchor subset.

Conclusion. Reviewed row by row, 92% of rows pass and 0.9% have a label in doubt, and the main weakness, leftover passages after evidence removal, is now measured and reported. Restricted to validated rows, scores move by at most 3 points and the main findings are unchanged.

W3: Novelty over existing benchmarks

The novelty over existing benchmarks is not entirely clear. Previous works have studied counterfactual evidence, contextual faithfulness, and evidence sufficiency. What additional evaluation capabilities does MainLogic provide beyond combining these existing approaches?

We thank the reviewer. MainLogic adds three capabilities that separate counterfactual, faithfulness and sufficiency benchmarks cannot provide, because it tests all the required behaviours on the same scientific claim, using natural claims and the complete cited documents.

1. All required behaviours on the same problem. Each anchor crosses two opposing claims with three premise states (original, reversed and insufficient evidence). This tests whether judgments follow the evidence when the claim is fixed, and follow the claim when the evidence is fixed. A matched deletion control compares removing the decisive evidence with removing a similar amount of nonessential text. The model must answer Uncertain after the first and keep its answer after the second, which tests whether abstention responds to losing the evidence rather than to a shorter document. The control covers 82 anchors and is scored as a pair.

2. Joint correctness that per-condition scores cannot show. GC6 counts an anchor as correct only when all six core judgments are. Consider six anchors. Model A makes one error on every anchor, each time in a different condition. Model B is fully correct on five anchors and wrong on every condition of the sixth. Both score 83.3% overall and in every condition, but their GC6 is 0% and 83.3%. Only GC6 shows whether a model meets all the requirements together on the same problem; Appendix C formalises this.

3. Diagnosis against measured priors and evidence access. The same claims are probed without evidence and evaluated with full documents, highlighted documents and evidence passages only. This links three questions within one design:

  • Does accuracy change when the evidence agrees or conflicts with the model's measured prior?
  • Does the model fall back on that prior when the documents leave the claim open?
  • Do these patterns persist when the evidence is easier to find?

Comparison with representative prior work.

Related workWhat it establishesWhat MainLogic adds
VitaminC (Schuster et al., 2021); CrossAug (Lee et al., 2021)Contrastive claim–evidence pairs, and crossed original and modified claims and evidenceJoint correctness across two claim orientations and three evidence states for one natural scientific claim
FaithEval (Ming et al., 2025)Faithfulness to unanswerable, inconsistent and counterfactual contextsWhole-family evaluation, and comparisons conditioned on the prior, across fixed evidence states of the same claims
Sufficient Context (Joren et al., 2025); SURE-RAG (Qiu et al., 2026)Sufficiency analysis and selective answering; SURE-RAG adds grouped evidence conditions with missing informationPaired decisive-versus-redundant deletions that test whether abstention tracks the specific evidence the claim needs
CheckList (Ribeiro et al., 2020)Behavioural testing through directional changes and invarianceEdits to scientific premises with an explicit correct answer for each, six-cell joint scoring and matched deletion controls

Conclusion. What is new is the unit of evaluation and the comparisons it allows. MainLogic measures which evidence requirements a model meets together on the same scientific problem, and how its failures vary with prior alignment and evidence access. These results cannot be recovered by pooling scores from separate counterfactual, faithfulness and sufficiency benchmarks.

Reviewer_F56dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

MainLogic asks whether models base their judgments on the scientific papers they read or repeat familiar conclusions. The authors edit key evidence to test whether judgments change, remove evidence to test whether models withhold judgment, and edit irrelevant text to test stability. Across ten models, strong performance on the original claims does not translate into consistently following the evidence across these controlled variants, especially when the evidence conflicts with a model's prior judgment.

Critical strengths
  1. The paper asks an interesting question and tests more than ordinary accuracy. Whether a model's answer actually depends on the supplied evidence is an important question. Changing the evidence, removing it, and removing a similar amount of irrelevant text are useful complementary tests. Requiring correct behavior across the same claim family makes it harder for a model to succeed through a fixed answer preference or indiscriminate abstention.

  2. The paper includes meaningful diagnostic checks. The authors measure each model's evidence-free judgment instead of assuming that every counterfactual conflicts with its prior. The evidence-access comparisons, prompt ablations, and human audit also help assess the results. I appreciate the explicit acknowledgment that prior-consistent errors do not establish a causal mechanism of memory override.

Critical weaknesses
  1. A fundamental alternative explanation remains unresolved.

    I agree that the question is interesting and that the authors have run many experiments to make the evaluation rigorous. However, the central uncertainty remains: is the model ignoring the evidence, or rejecting material that conflicts with other knowledge and may itself contain defects? A prior-consistent answer is compatible with either explanation. The measured prior, minimal edits, and evidence-access controls do not settle this distinction.

    I understand that the task asks what follows from the supplied documents (§3). A response that rejects those documents can still be wrong under this instruction. However, that score does not establish whether the model failed to understand the evidence or declined to accept it. The authors acknowledge that their results do not identify a causal mechanism of memory override; I appreciate this qualification, but it leaves the central explanatory question open.

    Related work considers both accepting correct new information and resisting incorrect external material. ClashEval evaluates both directions and finds that models are less likely to adopt more unrealistic errors in the supplied content. Astute RAG iteratively consolidates internal and external knowledge with source awareness and bases its final answer on information reliability. These works motivate directly checking whether the reliability of the conflicting material affects the observed failures. MainLogic should discuss these studies and explain how its controls address this distinction.

  2. The additional experiments do not directly resolve this concern.

    The authors mark the relevant passages or provide only those passages to make the evidence easier to find. But even with the evidence in front of it, a model may answer incorrectly because it struggles with the reasoning or questions the material itself. I would like to see an experiment that directly distinguishes these possibilities. For example:

    • A low-inference control in which the source explicitly states the answer, after first establishing that the model has a stable conflicting prior. For example, use a documented correction to the reported Titanic death toll: establish the model's prior answer, provide a verified record explicitly correcting that figure, and ask how many people died according to the correction. The answer is directly stated and requires almost no inference. This would help separate basic information-reading failures from difficulty assessing scientific reasoning.
    • Unmodified, independently checked refutation papers, scientific corrections, or evidence-backed debunking articles that directly address the same claim. These would test responses to genuine corrections of earlier beliefs.
    • Papers presenting new mathematical solutions or proofs, independently checked and selected with evidence that the model has not previously encountered the result or proof. These would provide a real-material control for using new results; they would test prior revision only where a conflicting prior is measured.
    • A controlled multi-turn exchange that elicits a specific objection to the material, checks that objection, and addresses it with relevant evidence before retesting the judgment. The model's stated objection would need verification; its explanation alone would not establish the cause of its original answer.

    These are possible routes to addressing the same concern. A convincing test need not implement every option, but it should directly investigate the alternative explanation. Treating this only as a general limitation or future-work suggestion leaves the paper's central interpretation insufficiently supported.

Questions for authors
  1. Clarification: What evidence in the current experiments distinguishes ignoring the supplied evidence from rejecting a source because of conflicts with other knowledge or perceived defects? Please discuss this distinction in relation to ClashEval and Astute RAG.

  2. Additional evidence that would affect my assessment: Can the authors directly test this distinction through a low-inference control, genuine corrections or refutations, previously unseen mathematical solution papers, or a controlled multi-turn exchange that checks and addresses the model's objections? A convincing result along one of these routes would materially affect my assessment.

11 comments

We thank the reviewer for these suggestions. We implemented the low-inference control (W2 (a)), a test on a real, unedited refutation of a widely assumed hypothesis (W2 (b)) and the controlled multi-turn exchange (W2 (c)), covering all three routes the reviewer proposes. We first summarise what they show together (Q1 and W1), then give each in turn.

Q1&W1 (a): What distinguishes ignoring the evidence from rejecting the source?

What evidence in the current experiments distinguishes ignoring the supplied evidence from rejecting a source because of conflicts with other knowledge or perceived defects?

We thank the reviewer for this question. We probed 170 prior-consistent conflict errors across four models to test whether they come from rejecting a source the model has understood or from misreading what it establishes (Appendix J, O).

These are cases where the documents contradict the model's prior (its answer to the same claim with no documents) and the model answered with its prior. We ask each model for the passages that support and oppose its answer, its reasons for setting evidence aside, and separate judgments of what the documents establish and what it believes. Quotations are checked against the documents, and a separate model classifies the explanations and checks substantive objections.

  • The models find the evidence but misreport it. 84–100% of responses quote the located evidence (the evidence passages found by our pipeline), but only 0–6% correctly report what it establishes.
  • Explicit rejection is rare. Rejection because of outside knowledge or distrust of the source accounts for about 3% of the sampled errors. The most frequent explanation is a misreading of the evidence (41%), followed by weighing it against other passages (29%).
  • Showing the evidence again rarely helps. It corrects only 4–18% of errors, little more than a generic request to reconsider.

Three further controls rule out simpler explanations:

  • Directly stated conflicts are mostly handled. When the conflicting evidence is stated outright and shown as passages only, pooled accuracy is 84%, against 58% when it must be derived (Appendix M).
  • Real refutations are followed. All 15 model–claim pairs with a measured opposing prior switch to the document's judgment when given an unedited refutation (Appendix N).
  • Models can recognise insufficiency. They answer Uncertain on 98–100% of claims paired with unrelated papers (Appendix P).

The prior effects also persist when the prior is measured with alternative probes (Appendix L).

Conclusion. These results point neither to a general inability to use conflicting evidence nor to a general refusal to revise a belief. In the sampled errors, models rarely acknowledge what the source says and then reject it; they most often describe the source as saying what they already believe.

W1(b): Comparison with ClashEval and Astute RAG

Related work considers both accepting correct new information and resisting incorrect external material. […] MainLogic should discuss these studies and explain how its controls address this distinction.

We thank the reviewer for these references, and we have added both to Related Work.

ClashEval measures how often a model adopts information in its context over its own prior as the error in that context is made larger; Astute RAG combines internal and retrieved knowledge with source awareness to answer correctly when retrieval is imperfect. Both aim at the factually correct answer.

MainLogic asks something different. It asks whether the model's judgment follows what the supplied scientific premises establish, and this changes what counts as correct.

  • In factual question answering, corrupting the context does not change the right answer, and good prior knowledge can make up for poor retrieval.
  • In MainLogic, a validated edit to the premises changes the right answer, and removing an indispensable piece of evidence makes a claim undetermined even when the model knows the original result.

Our controls address the reviewer's distinction in three ways. First, the experiments above separate misreading from rejection (Appendices J and O). Second, the low-inference control and the real refutation test whether models accept explicit, reliable sources that oppose their prior (Appendices M, N). Third, the evaluation scores what the documents establish separately from whether the model endorses them, which is what checking a citation or a scientific argument requires. Varying the reliability of the source, as ClashEval does, is an extension that we now propose in the Conclusion.

Conclusion. ClashEval and Astute RAG ask whether a model reaches the factually correct answer when its context may be wrong. MainLogic asks whether its judgment follows what the supplied premises establish, and our new experiments show that, in the sampled errors, models misread those premises far more often than they reject them.

W2 (a) and Q2: A low-inference control with a stable conflicting prior

A low-inference control in which the source explicitly states the answer, after first establishing that the model has a stable conflicting prior. […] This would help separate basic information-reading failures from difficulty assessing scientific reasoning.

We thank the reviewer for this suggestion. We ran two analyses that meet the three conditions of the control (a conflicting measured prior, a source that states the answer outright, and verified material). One uses a real refutation (Appendix N); the other isolates the lowest-inference conflict cases within MainLogic (new Appendix M, Tables 24–25).

1. Real, verified material. The ETH set (Appendix N, detailed in W2 (b) below) uses as premise a recently released refutation of the Exponential Time Hypothesis.

  • The refutation contradicts 8 of the set's 11 claims with an explicit statement (for example, "both ETH and SETH are false").
  • Across seven models, the evidence-free probe found 15 model–claim pairs with a prior of Supported. All 15 switched to Contradicted given the refutation, and all 77 refutation-only rows were correct.

2. Within MainLogic (new Appendix M, Tables 24 and 25). We isolate the benchmark's lowest-inference conflict cases from runs already in the paper:

  • P−P^- rows, where the edited documents settle the claim against the original;
  • stated anchors: claim families (anchors) in which some edited passage states the counterfactual outright (48 of 108);
  • the passages-only rendering, where only the located passages are shown;
  • a conflicting prior, where the model's evidence-free judgment is definite and opposite to the reference label.

As in RQ2, each claim is compared with itself on the original papers, where the same prior agrees with the reference label. The difference is the cost of the conflict on identical claims.

Table 24 (Appendix M). Matched pairs pooled over the ten systems: accuracy on the original papers, where the prior agrees, and on the edited papers, where it conflicts, for the same claims.

AnchorsRenderingPairsOriginal, prior agreesEdited, prior conflictsCost
Statedfull documents67694%83%+12
Statedpassages only67694%84%+10
Derivedfull documents77884%54%+30
Derivedpassages only77980%58%+23

Table 25 (Appendix M). The low-inference control by system: stated anchors, passages only. Original is accuracy on the same claims in the original papers, where the prior agrees; Control is accuracy on the edited papers, where it conflicts; Cost is the difference. Brackets are anchor-bootstrap 95% intervals. The last column gives derived anchors under the same conditions.

SystemPairsOriginalControlCostDerived, control
Claude Fable 5.18499%95% [90, 99]+4 [−1, +9]76%
Claude Opus 58399%95% [90, 99]+4 [+0, +8]76%
GPT-6 Astra7699%95% [88, 100]+4 [+0, +9]72%
DeepSeek Flash5997%90% [82, 97]+7 [+2, +13]60%
Claude Sonnet 56198%87% [78, 95]+11 [+5, +20]51%
GPT-5.6 Terra8196%85% [78, 92]+11 [+4, +19]62%
Gemini 2.5 Flash4386%84% [72, 93]+2 [−13, +17]52%
GPT-5.6 Luna7189%77% [66, 87]+11 [+0, +24]46%
GPT-4.158100%67% [54, 80]+33 [+20, +46]23%
Gemini 3.8 Flash6072%57% [45, 68]+15 [+4, +27]36%
  1. For most models, basic reading holds even against a conflicting prior. With the answer stated and in view, the conflict costs 2–7 points for five models and 11 points for three. Pooled, it costs 10 points: 94% on the original papers against 84% on the edited ones.
  2. The large prior effects sit where the counterfactual must be derived. On derived anchors the conflict costs 23 points with the passages alone and 30 with the full documents.
  3. Access explains little of it. Showing only the passages lowers the cost by 2 points on stated anchors and 7 on derived anchors.
  4. One model fails the control. GPT-4.1 answers 100% of the original-paper rows correctly but only 67% of the conflicting ones, a 33-point cost even when the correction is stated and in view. For GPT-4.1, part of the failure is basic reading. Gemini 3.8 Flash is low under the passages-only rendering in general, including on the original papers (72%), as RQ3 notes.

Conclusion. Most models follow explicit conflicting evidence; their large difficulties arise when the consequence of the evidence must be derived. GPT-4.1 is the exception, failing even the explicit control, so the control separates models with basic reading failures from those whose failures lie in the reasoning.

W2 (b) and Q2: Unmodified scientific material and previously unseen results

Unmodified, independently checked refutation papers, scientific corrections, or evidence-backed debunking articles that directly address the same claim. […] Papers presenting new mathematical solutions or proofs, independently checked and selected with evidence that the model has not previously encountered the result or proof.

We thank the reviewer for these suggestions. We evaluated seven systems on a recently released, formally verified refutation of the Exponential Time Hypothesis, using unedited documents and measuring each model's prior on every claim (new Appendix N, Tables 26–27).

1. Material. The Exponential Time Hypothesis (ETH) states that 3-SAT on nn variables cannot be solved in time 2o(n)2^{o(n)}. It is unproven but widely assumed, and many published lower bounds depend on it. We use A 2o(n)2^{o(n)} Time Algorithm for 3-SAT (Claude, Anthropic, October 2026), a publicly released manuscript that gives a subexponential algorithm and so refutes ETH and related hypotheses. It meets the reviewer's criteria as follows:

  • Checked. Its proofs are formalised in Lean 4, and the release includes the procedure for checking them, so its correctness does not rest on trust in the author.
  • Previously unseen. It was released after all evaluated systems were trained. The manuscript is authored by Claude, and Claude models are among those evaluated; none of the evaluated models, these included, could have seen it in training.
  • Conflicts with a prior. The evidence-free probe measures each model's prior on each claim (below).

We selected 11 claims from seven earlier papers that the refutation contradicts. Eight are contradicted by an explicit statement and three need one inference. We excluded conditionals such as "assuming ETH, X", because refuting the assumption does not refute the conditional, and bounds that the result does not settle. Appendix N, Table 26 lists every claim, its source and how the refutation contradicts it. Each claim is evaluated under three conditions, with no edits to any document.

Supplied premisesWhat the documents establishReference label
Original source paperThe hypothesis is assumed; the unconditional claim is not establishedUncertain
Refutation manuscriptThe result contradicts the claimContradicted
Both, original source firstThe refutation contradicts the claim, although the earlier paper assumes itContradicted

2. Measured prior conflict. Across the 77 model–claim pairs, the evidence-free probe gives 15 Supported, 61 Uncertain and one Contradicted. The 15 Supported cases are the ones where the material conflicts with the model's prior.

Table 27 (Appendix N). Each system evaluates 11 claims. Initially supported: claims judged Supported with no documents. Prior-conflict cases answered C: of those, how many were judged Contradicted given the refutation. The last three columns are accuracy under each condition; U and C in the headers are the reference labels (Uncertain, Contradicted).

SystemInitially supported claimsPrior-conflict cases answered CSource only (U)Refutation (C)Both (C)
Claude Fable 5.10–100%100%100%
GPT-6 Astra0–91%100%100%
Claude Opus 511/191%100%100%
Claude Sonnet 577/755%100%100%
GPT-5.6 Terra11/136%100%100%
GPT-4.155/536%100%100%
GPT-5.6 Luna11/127%100%91%
All seven1515/1562%100%99%
  • Models follow unedited refuting material. All 77 refutation-only judgments are correct, as are 76 of 77 with both documents. All 15 cases with a conflicting prior are judged Contradicted once the refutation is supplied.
  • Models differ in telling assumptions from established results. With the original source alone, Claude Fable 5.1, GPT-6 Astra and Claude Opus 5 answer Uncertain on 10–11 of 11 claims. GPT-5.6 Terra, GPT-4.1 and GPT-5.6 Luna do so on only 3–4, often treating an assumed hypothesis as established.

Conclusion. Models can follow unmodified material that opposes a measured prior. The same set exposes a separate difficulty, telling what an original scientific source establishes from what it only assumes.

Reviewer_bQzUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MainLogic, a benchmark that tests a model's selective responsiveness: whether LLMs make judgments based on the evidence provided in the context rather than on prior knowledge. Specifically, it tests whether models flip their answer when decisive evidence flips, answer "undetermined" when the evidence is removed, and stay stable under irrelevant edits. Starting from 108 human-reviewed claims taken from citing papers, each paired with the full text of the cited work(s), the authors build six-cell families (the original claim and a contrary claim, crossed with original, counterfactual, and evidence-deleted premises), plus a length-matched deletion control for 82 anchors, giving 812 instances in total.

Ten frontier models are evaluated on three research questions:

  1. Does the model's judgment depend on the evidence in the context?
  2. When the evidence disagrees with the model's prior, which one does the model follow?
  3. Does the difficulty come from locating evidence in a long context or from using it?

Main findings: There is an evident gap between having evidence in context and actually using that evidence (or its absence) to make a judgment. Although 8 of 10 models exceed 82% accuracy on the original claims, the best model answers all six cells of a family correctly only 32.4% of the time. Accuracy is lower when the evidence conflicts with the model's prior than when it agrees. Models also tend to follow their prior when committing to an answer on undetermined instances. Easier evidence access helps only some models, mainly on the cell where the evidence contradicts the claim the model believes.

Critical strengths

The matched-family framework is very strong. It separates true evidence dependence from mere claim familiarity. Because premise-blind and always-undetermined policies cannot pass a full family, this design exposes behavioral flaws that pooled accuracy hides. For example, GPT-4.1 is within two points of the best model on the original claims but has the lowest GC6 score.

The paper also presents a strong prior analysis. By probing evidence-free priors per model and separating prior reliance from a generic label preference, the analysis yields a non-obvious dissociation: the two models with the smallest alignment gaps (Claude Fable 5.1 and GPT-6 Astra) still follow their prior in 97% and 93% of their committed answers on undetermined instances.

Critical weaknesses

The benchmark's label validity is under-verified, which directly limits the reliability of the headline metrics. Labels are assigned by construction rather than through independent annotation. The automatic check for residual evidence is "intentionally light" for counterfactual premises and absent for evidence-deleted premises, and the blinded human audit covers only 48 rows from 40 of the 108 anchors. This makes it difficult to interpret the drop from natural-claim accuracy to the strict GC6 metric. Without a human baseline or an estimate of the label error rate, one cannot tell whether the models are truly failing or the test items are flawed.

Key protocol details are also missing. The paper omits the decoding and reasoning settings used for evaluation, as well as the prompts, number of repetitions, and stability rule used to probe the evidence-free priors on which RQ2 depends. Apart from the redundancy-deletion control, the introduction/conclusion rewrites defined in Section 3 and the shuffled insufficiency pairs mentioned in Appendix B are not evaluated. Finally, the dataset is limited in scale (27 predominantly machine-learning papers), and the prior-conditioned metrics rely on small subsets.

Questions for authors

Does GPT-6 Astra's advantage persist if a different planner model produces the counterfactual edits, even on a subset of anchors?

Are the 6 failed audited anchors included in results in tables 2-5?

7 comments

We warmly thank the reviewer for their close engagement with our work and their recognition of the matched-family design. Their questions led to some of our most informative new experiments, presented below.

W1: Label validity and reliability of the reported results

The benchmark's label validity is under-verified […] the blinded human audit covers only 48 rows from 40 of the 108 anchors. This makes it difficult to interpret the drop from natural-claim accuracy to the strict GC6 metric. Without a human baseline or an estimate of the label error rate, one cannot tell whether the models are truly failing or the test items are flawed.

We thank the reviewer. We ran a new per-row human review of 320 further rows, which gives both a human baseline and a label error rate (new Appendix R), and evaluated every system on the rows that pass (new Appendix S).

The review covers 40 rows from each of the eight cells (a cell is one combination of premise version and claim), spanning 105 anchors and 26 citing papers; with the original audit, 368 rows from 106 of the 108 anchors are now reviewed. The original protocol reviewed one sampled row and, if it passed, counted the whole anchor as verified. Now every row is reviewed on its own, so a poor edit or an incomplete evidence removal is caught in the row where it occurs.

1. Human baseline and label error rate. The auditor first judged each claim from the documents, before seeing the cell, the reference label or the contrary claim. Only then did they check the claim, the contrast between the two claims, and whether the edited or reduced documents still settle the claim.

  • Human baseline. The blind judgment agrees with the reference label on 290 of 320 rows (90.6%, 95% CI [86.8, 94.2]), and on 284 of the 294 rows that pass the audit (96.6%) (Appendix R, Table 33).
  • Error rate. 26 of 320 rows fail (8.1%, 95% CI [4.9, 11.6]). In 23 of these the construction is flawed but the label is still the intended answer; only 3 (0.9%) put the label itself in doubt.
  • Where the failures are. Most are residual evidence, in 16 of the 80 evidence-deleted rows and 6 of the 80 counterfactual-premise rows. All 26 failing rows are excluded from the validated subset, including those whose label is still correct.
Audit outcomeRowsShare
Passed29491.9%
Construction flawed, label still intended (residual evidence or ambiguous wording)237.2%
Label in doubt30.9%
Total320100%

2. Results on validated rows and without the original failures. The two audits pass 337 rows, of which 250 are core rows (the six main cells of each anchor, excluding the redundancy-deletion controls). Appendix S recomputes each system's accuracy on the validated core rows, reweighted to match the full benchmark's mix of cells. Appendix Q separately recomputes the metrics without the six anchors that failed the original audit. Values are percentages.

SystemHeadline Acc6 / GC6Six failed anchors excluded: Acc6 / GC6Validated core rows: Acc6
Claude Fable 5.177.5 / 28.778.6 / 30.480.0
GPT-6 Astra75.8 / 32.476.5 / 33.377.2
Claude Opus 572.1 / 21.373.0 / 22.574.0
DeepSeek Flash70.8 / 19.471.2 / 20.673.7
Gemini 3.8 Flash69.4 / 20.469.9 / 21.669.7
Claude Sonnet 567.5 / 17.968.4 / 19.069.7
GPT-5.6 Terra65.6 / 13.065.8 / 13.767.1
Gemini 2.5 Flash63.7 / 12.063.9 / 12.762.9
GPT-5.6 Luna62.0 / 11.162.1 / 11.863.6
GPT-4.153.1 / 2.854.2 / 2.954.9
  • On validated rows, Acc6 changes by −0.8 to +2.9 points; without the six failed anchors, by at most 1.1 points, with the ranking unchanged.
  • The best GC6 without the failed anchors is 33.3%, against 32.4% in the headline results.

Conclusion. Item quality does not explain the drop from natural-claim accuracy to GC6. A human auditor, answering blind, agrees with 96.6% of the labels on rows that pass review; only 0.9% of reviewed rows have a label in doubt; and the drop persists both on validated rows and without the failed anchors. The models are failing, not the items.

W2 (a): Protocol details (decoding, prompts, repetitions and prior stability)

The paper omits the decoding and reasoning settings used for evaluation, as well as the prompts, number of repetitions, and stability rule used to probe the evidence-free priors on which RQ2 depends.

We thank the reviewer. We have added all of these details to Appendix H (Table 15), and ran a new check of how stable the evidence-free probe is (new Appendix L, Tables 22–23).

Evaluation (all tables).

  • Prompt. Every model and row gets the same system prompt, given verbatim in Appendix H. The user message lists the premise documents, then the conclusion, then "Based only on the premise documents, is the conclusion Supported, Contradicted, or Uncertain?".
  • Repetitions. One sample per row.
  • Decoding. Claude, Gemini and DeepSeek models run at temperature 0. The OpenAI API does not accept temperature 0 for the GPT-5.6 family, so we ran all four OpenAI models (GPT-6 Astra, GPT-5.6 Luna, GPT-5.6 Terra, GPT-4.1) at the API default, giving them one shared setting. No reasoning-effort or extended-thinking parameter is set, so each model uses its provider default. Output is capped at 4,000 tokens. Table 15 lists each model's id, provider and temperature.
  • Parsing. We take the first label word in the reply. A small synonym map covers words such as "undetermined", which counts as Uncertain.
  • Missing answers. A row with no parseable answer is excluded from that model's denominators. This happened for 2 of 8,120 model–row pairs, both from Claude Sonnet 5.
  • Instruction wording. Appendix H re-runs four systems under a stricter and a looser instruction. The differences are mostly within noise.

Table 15 (Appendix H). Models and decoding settings.

SystemModel idProviderTemperature
GPT-6 Astragpt-6-astraOpenAIAPI default
GPT-5.6 Lunagpt-5.6-lunaOpenAIAPI default
GPT-5.6 Terragpt-5.6-terraOpenAIAPI default
GPT-4.1gpt-4.1OpenAIAPI default
Claude Opus 5claude-opus-5Anthropic0
Claude Sonnet 5claude-sonnet-5Anthropic0
Claude Fable 5.1claude-fable-5-1Anthropic0
Gemini 2.5 Flashgemini-2.5-flashGoogle0
Gemini 3.8 Flashgemini-3.8-flashGoogle0
DeepSeek Flashdeepseek-flashDeepSeek0

Evidence-free probe (RQ2).

  • Input. The claim alone, with no premises and no section headers.
  • System prompt. "If the conclusion follows state Supported, if not state Contradicted, if you are uncertain answer Uncertain. Answer with only one word."
  • Orientations. Both claims are probed separately, 108 C+C^+ and 108 C−C^-, for 216 claims per model. Decoding and parsing are as above. All ten models answered all 216.
  • Prior assignment. One sample per claim. Supported gives b=1b=1, Contradicted gives b=0b=0, and Uncertain gives b=⊥b=\bot, which leaves the claim out of the prior-conditioned analyses.

The submitted text called this a "stable" judgment and said "unstable responses" were excluded. That overstated what the probe does, and we have corrected it.

Stability. We measured stability in two ways.

  1. New probe variants. We re-asked the 216 claims on four models with a reworded prompt and with the options in reverse order.
    • Agreement with the reported probe is 0.82–0.95 (Cohen's κ 0.68–0.86).
    • 78–91% of definite priors are identical across all three variants (Table 22).
    • Restricting RQ2 to these stable priors raises the alignment gap and ErrDir for all four models; DirComp stays at 0.85–0.92 (Table 23). GPT-6 Astra's gap rises from 0.06 to 0.11, and its interval, which included zero, now excludes it.
  2. An earlier run in a different format. On seven models, an earlier run used a different user-message format. It agrees with the reported probe at κ 0.48–0.82, and the two runs point the same way on 85–99% of the claims where both gave a definite answer.

Table 22 (Appendix L). Agreement of the reported probe with the two variants. Flips are Supported↔Contradicted reversals, as a share of definite priors over both variants.

SystemDefiniteStableAgree / κ (reworded)Agree / κ (reversed)Flips
GPT-6 Astra182147 (81%)0.86 / 0.760.82 / 0.723%
GPT-5.6 Luna201157 (78%)0.87 / 0.740.83 / 0.688%
GPT-5.6 Terra204165 (81%)0.84 / 0.700.84 / 0.7012%
GPT-4.1208189 (91%)0.88 / 0.680.95 / 0.865%

Table 23 (Appendix L). RQ2 measures with all definite priors (reported) → stable priors only. Brackets are anchor-bootstrap 95% intervals. Gap is the accuracy difference between prior-aligned and prior-conflicting evidence; ErrDir and DirComp are the share of prior-consistent answers among conflict errors and among determinate answers under insufficient evidence.

SystemGapErrDirDirComp
GPT-6 Astra0.06 [−0.01, 0.13] → 0.11 [0.03, 0.19]0.70 → 0.740.93 → 0.92
GPT-5.6 Luna0.30 [0.22, 0.38] → 0.33 [0.22, 0.43]0.70 → 0.720.79 → 0.85
GPT-5.6 Terra0.23 [0.15, 0.31] → 0.25 [0.16, 0.35]0.70 → 0.740.85 → 0.87
GPT-4.10.50 [0.42, 0.58] → 0.56 [0.48, 0.64]0.84 → 0.850.84 → 0.88

Conclusion. Unstable priors dilute the reported prior effects rather than create them. The full protocol and settings are now in Appendix H (Table 15), and the probe variants and stability results in new Appendix L (Tables 22–23).

W2 (b): Introduction/conclusion rewrites and shuffled insufficiency pairs

Apart from the redundancy-deletion control, the introduction/conclusion rewrites defined in Section 3 and the shuffled insufficiency pairs mentioned in Appendix B are not evaluated.

We thank the reviewer. We built and evaluated the shuffled insufficiency pairs (new Appendix P, Table 30), and removed the introduction/conclusion rewrites from the design.

For each of the 108 anchors, both claims (C+C^+ and C−C^-) are shown with the original, unedited papers of an unrelated anchor, the donor (216 rows, reference Uncertain). The donor comes from a different citing paper in a distant field; for example, ML claims receive physics or classical-robotics papers. Donors are length-matched to the anchor's own premises (median ratio 0.99, range 0.71–1.49). We searched every donor's papers for the claim's names, numbers and acronyms; the only matches are generic words such as "performance". Unlike P0P^0, which deletes evidence from on-topic papers, the label here is certain by construction. We scored six systems with the main evaluation's prompt and decoding.

Table 30 (Appendix P). Accuracy (answers Uncertain) with an unrelated anchor's papers (Shuffled) and with the evidence deleted from the on-topic papers (P0P^0), on the same 108 anchors. Pair: both claims correct. Diff.: shuffled minus P0P^0 accuracy in points, with an anchor-bootstrap 95% CI. Prior-dir.: share of determinate P0P^0 answers that match the model's evidence-free prior.

SystemShuffled C+ / C−P0 C+ / C−Shuffled, pairP0, pairDiff.Determinate, shuffled / P0Prior-dir.
Claude Fable 5.1100 / 10066 / 56100%49%+39 [31, 47]0 / 8496%
Claude Opus 5100 / 10048 / 45100%36%+53 [45, 62]0 / 11593%
GPT-6 Astra100 / 10059 / 61100%54%+40 [31, 49]0 / 8691%
GPT-5.6 Terra100 / 10042 / 51100%32%+54 [46, 62]0 / 11685%
GPT-5.6 Luna99 / 9934 / 5498%30%+55 [47, 62]2 / 12181%
GPT-4.1100 / 9828 / 3998%23%+66 [58, 74]2 / 14483%
  • Unrelated papers are recognised as insufficient. Every system answers Uncertain on 98–100% of shuffled rows. We checked the four determinate answers by hand. Their donors are unrelated (a CMB dark-matter paper, a galaxy survey, motion planning), and three of the four match the model's prior.
  • The same claims fail when the papers are on topic. With the evidence deleted from the cited papers, the same systems commit on 84–144 of 216 rows. Accuracy is 39–66 points lower, and pair accuracy 46–75 points lower.
  • Those commitments follow the prior. 81–96% of determinate P0P^0 answers match the evidence-free prior.

Introduction/conclusion rewrites. We have removed the rewrites from the design. They would test an edit that leaves the relation between premises and claim unchanged, which P=P^=, the redundancy-deletion control, already tests. Section 3 now states that P=P^= is the only relation-preserving edit in MainLogic, and Appendix B and RQ1 point to the shuffled pairs in Appendix P.

Conclusion. The P0P^0 failures are not a general reluctance to answer Uncertain. Models abstain on plainly unrelated papers, but fall back on their prior when the papers look relevant.

Reviewer_qqs3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MainLogic, a benchmark that evaluates whether long-context language models adjust their judgments appropriately when the evidence in scientific documents changes. Models classify a claim as supported, contradicted, or undetermined based on the supplied documents. Each original claim generates six related cases by crossing two opposing claims with three evidence conditions: original, reversed, and insufficient. A separate control removes a similar amount of irrelevant text while preserving the evidence. Together, these tests examine whether models update their answers, recognize insufficient evidence, and remain stable under irrelevant changes.

MainLogic contains 812 instances derived from 108 original claims across 27 papers, mostly in machine learning. The inputs include complete cited papers, with a median length of approximately 16,000 tokens. The original claims are human-reviewed, while altered documents are constructed with LLM assistance. A blinded audit covers 48 instances from 40 claim groups, with 34 groups passing the reported checks. The authors evaluate ten models, separately measure their judgments without evidence, and compare full documents with highlighted evidence and evidence-only inputs.

Several models exceed 90% accuracy on original supported claims, but accuracy across the six core conditions ranges from 53.1% to 77.5%. Under the stricter requirement of answering all six cases correctly, the best score is 32.4%. Accuracy is 6–51 percentage points lower when evidence conflicts with a model’s measured prior judgment than when it agrees. Incorrect definite answers under insufficient evidence also tend to follow those priors. Highlighting or extracting evidence helps some models but does not consistently resolve the errors. These findings demonstrate inconsistent responses to supplied evidence, although they do not directly identify the internal mechanism responsible.

Critical strengths
  1. Controlled tests of evidence dependence. The strongest aspect is the use of matched variants of the same scientific claim. Crossing two opposing claims with original, reversed, and insufficient evidence tests whether a model updates its judgment when required and withholds judgment when the documents no longer settle the question. The matched deletion control is particularly valuable: removing irrelevant text changes the document length without changing the correct answer. This helps distinguish responsiveness to evidence from sensitivity to deletion itself. Requiring all six cases to be correct also exposes inconsistencies that average accuracy can obscure. This design is consequential to my assessment because it provides a concrete framework for testing evidence dependence within complete scientific documents, with clear expectations for both appropriate changes and appropriate stability.

  2. Analysis that explains different failure patterns. The paper separately measures each model’s judgments without evidence, allowing it to identify conflicts with that model’s own prior rather than treating every counterfactual document as a conflict. It also distinguishes the frequency of errors from their direction, including whether unsupported definite answers follow those priors. These distinctions make the reported accuracy gaps of 6–51 percentage points more informative than a simple comparison of original and edited documents. Comparisons of full documents, highlighted passages, and evidence excerpts further show that making relevant information easier to access does not consistently eliminate errors. Although these comparisons do not identify an internal mechanism, they provide useful behavioral diagnostics. This supports acceptance because the analysis helps researchers understand which aspects of evidence use remain unreliable and suggests focused directions for subsequent evaluation and model improvement.

Critical weaknesses
  1. Incomplete validation of the constructed examples. The central results depend on the edited documents supporting the intended alternative claim and on evidence removal leaving both claims unresolved. However, the exhaustive human review covers the original claims, while the subsequent audit examines only 48 instances from 40 of the 108 claim groups; six audited groups contain defects (Section 4.1; Appendix E). Moreover, a group receives verified status when its sampled rows pass, which does not establish that every variant is valid. Remaining evidence elsewhere in a paper could make an intended “undetermined” label incorrect, while weak claim contrasts could introduce shortcuts. These issues can directly affect both accuracy and complete-family scores. I would need clearer handling of failed groups, a broader audit of complete families, and results on a fully validated subset before treating the reported performance gaps as reliable.

  2. Limited coverage of scientific reasoning. The 812 instances represent only 108 original claims drawn from 27 papers, mostly in machine learning. The multiple variants provide valuable comparisons within each claim, but they do not constitute 812 independent scientific problems. Although resampling at the anchor level appropriately accounts for dependence, the small and narrow source collection limits how broadly the results can be interpreted. In particular, the paper distinguishes claims that are explicitly stated from those requiring inference, but does not clearly report performance separately for these categories. Without that breakdown, it is difficult to assess how much the benchmark measures integration of scientific evidence versus locating and checking a stated fact. This matters to the claimed contribution on long-context reasoning. Broader coverage, or at least a clear analysis by reasoning requirement, would make the conclusions more convincing.

Questions for authors
  1. Clarification and validation: How do audit failures affect the reported results? Section 4.1 reports an audit of 48 instances from 40 anchor groups, with six groups failing, while the main evaluation covers 108 anchors. Could you clarify whether the failed groups or affected cells remain in the headline results, and how any retired cells are handled when computing six-case correctness? Please report the main metrics after excluding known failures and, if feasible, on the 34 groups marked verified. Because verification is based on sampled rows, please specify which variants were checked within that subset. These comparisons would help establish whether the reported weaknesses in evidence use remain substantial under the available quality checks. My assessment would improve if the main patterns persist after accounting for the identified construction defects.

  2. Clarification: What reasoning is required beyond checking explicitly stated facts? Appendices D.2 and F.1 distinguish claims explicitly stated in the documents from claims that require inference. Could you provide the number of anchors in each category and their separate per-case and six-case accuracy? For the inference category, a representative example showing the necessary passages and the dependency removed in the insufficient-evidence version would clarify the intended reasoning demand. This matters because the contribution is framed around scientific evidence reasoning, whereas a result dominated by checking directly stated facts would support a narrower interpretation. I would view the contribution more favorably if the main findings also hold for claims requiring inference. These are clarification and analysis requests concerning the existing dataset; extending the benchmark to additional disciplines would be future work.

6 comments

The author sincerely thank the reviewer for their exceptionally careful and detailed review. Their comments led to several new analyses, which we describe below.

Q1 and W1: How do audit failures affect the reported results?

Could you clarify whether the failed groups or affected cells remain in the headline results, and how any retired cells are handled when computing six-case correctness? Please report the main metrics after excluding known failures and, if feasible, on the 34 groups marked verified. Because verification is based on sampled rows, please specify which variants were checked within that subset.

We thank the reviewer for this question. We report the requested analyses for the original audit (Appendix Q), ran a new per-row human review of 320 further rows (new Appendix R), and evaluated every system on the 337 rows that pass review (new Appendix S).

The per-row review replaces the original protocol, which reviewed one sampled row and, if it passed, counted the whole six-cell anchor as verified. Reviewing every row on its own catches a weak negation or an incomplete evidence removal in the row where it occurs, independently of the other rows of the anchor.

1. Headline results and retired cells. The six failed anchors remain in all headline results (108 anchors, 812 rows), but they failed for different reasons (Appendix Q, Table 31). In three, the labels are still correct; two have a bad claim and one an easy negation. In two, the negation is arguably not contrary to the original claim, so the C−C^- labels are doubtful. In one, the evidence removal left behind a passage that still settles the claim, so the two P0P^0 labels are wrong. In total, at most 8 of the 648 core cells (the six main cells of each anchor, excluding the deletion controls) carry a doubtful label.

In a separate analysis we apply the audit's retirement rules. A bad claim retires all six core cells, a failed negation the three C−C^- cells, and residual evidence the two cells that share the premise. This retires 23 cells. Acc6 is computed over the remaining cells, and GC6 only over anchors that keep all six. Missing model answers are dropped, as in the paper.

Table 32 (Appendix Q). Acc6 / GC6 (%).

SystemAll 108 anchorsSix failed anchors excluded (102)Original audit-passing anchors (34)Retired cells removed
Claude Fable 5.177.5 / 28.778.6 / 30.479.4 / 35.378.1 / 30.4
GPT-6 Astra75.8 / 32.476.5 / 33.378.4 / 35.376.2 / 33.3
Claude Opus 572.1 / 21.373.0 / 22.574.5 / 26.572.6 / 22.5
DeepSeek Flash70.8 / 19.471.2 / 20.671.6 / 20.671.2 / 20.6
Gemini 3.8 Flash69.4 / 20.469.9 / 21.672.5 / 26.569.8 / 21.6
Claude Sonnet 567.5 / 17.968.4 / 19.067.0 / 15.268.1 / 19.0
GPT-5.6 Terra65.6 / 13.065.8 / 13.763.2 / 8.865.9 / 13.7
Gemini 2.5 Flash63.7 / 12.063.9 / 12.762.3 / 8.863.5 / 12.7
GPT-5.6 Luna62.0 / 11.162.1 / 11.864.7 / 14.762.1 / 11.8
GPT-4.153.1 / 2.854.2 / 2.959.3 / 8.853.9 / 2.9
  • Excluding the six anchors raises Acc6 by 0.1 to 1.1 points; removing only the retired cells changes it by −0.2 to +0.8. The ranking is unchanged in both.
  • On the 34 audit-passing anchors, a third of the benchmark, Acc6 moves by −2.4 to +6.2 points. The top three systems and the last are unchanged; some mid-table systems swap places on this smaller set.

2. Which variants were checked. In the original audit, 33 anchors had one row reviewed, six had two and one had three. No anchor had all its cells checked, which is the weakness the reviewer identifies.

Audit coverage by cell. Rows reviewed in each audit, and the validated rows used in Appendix S.

CellOriginal audit (40 anchors)Original audit, 34 passing anchorsExtended auditValidated rows, both audits
P+C+544044
P+C−754044
P−C+554043
P−C−744041
P0C+984045
P0C−654033
P=C+764045
P=C−224042
Total4839320337

The two audits together cover 368 rows from 106 of the 108 anchors.

3. Results of the per-row review (Appendix R, Table 33). The auditor first answered each row from the documents, without seeing the reference label, and then checked the claim, the contrast between the two claims, and whether the edited or reduced documents still contain evidence that settles the claim.

  • 26 of the 320 rows fail (8.1%, 95% CI [4.9, 11.6]), 22 for residual evidence, 2 for a bad claim and 2 for a broken negation.
  • In 23 of the 26, the construction is flawed but the label is still the intended answer. Only 3 rows (0.9%) put the label itself in doubt.
  • Most residual evidence comes from evidence removal. 16 of the 80 P0P^0 rows keep a passage that still settles the claim, against 6 of 80 P−P^- rows.
  • The auditor's blind answer matches the reference label on 290 of 320 rows (90.6%), and on 284 of the 294 rows that pass (96.6%).
  • Under the per-row criterion, an easy negation is recorded but does not fail the row.

4. Results on validated rows (Appendix S, Table 34). The two audits pass 337 rows from 105 anchors, 294 from the extended audit and 43 from the original audit; 250 of them are core rows. Because the review is now per row, no anchor yet has all six core cells validated, so GC6 cannot be computed on this subset. We report per-cell accuracy, and Acc6 over the 250 validated core rows with each cell weighted as in the full benchmark, so that it is comparable with the headline value. Inputs and reference labels are unchanged.

Table 34 (Appendix S). Accuracy (%). "Remaining" is the core rows outside the validated subset, reweighted in the same way.

SystemAcc6, allAcc6, validatedAcc6, remainingP+C+P+C−P−C+P−C−P0C+P0C−
Claude Fable 5.177.580.075.8898991886758
GPT-6 Astra75.877.274.9809186786267
Claude Opus 572.174.071.0919184854448
DeepSeek Flash70.873.769.4758479805173
Gemini 3.8 Flash69.469.769.5438484418779
Claude Sonnet 567.569.766.4828267715661
GPT-5.6 Terra65.667.164.5758467735152
Gemini 2.5 Flash63.762.964.3757572734042
GPT-5.6 Luna62.063.661.2867751783158
GPT-4.153.154.952.4895033832748
  • Acc6 on validated rows is within −0.8 to +2.9 points of the headline value. The ranking holds apart from two near-ties. Claude Sonnet 5 and Gemini 3.8 Flash are level at 69.7, and GPT-5.6 Luna and Gemini 2.5 Flash swap places, 0.7 points apart.
  • The hard cells stay hard on human-validated rows. Systems other than Gemini 3.8 Flash answer 27–67% of P0C+P^0C^+ rows correctly, and GPT-4.1 falls from 89% on P+C+P^+C^+ to 33% on P−C+P^-C^+.

Conclusion. The six failed anchors remain in the headline results, and three of them affect labels in at most 8 cells. Excluding them, keeping only the 34 audit-passing anchors, or restricting to the 337 individually validated rows leaves the main patterns in place, and the ranking changes only among close systems. The per-row review shows that 8.1% of rows fail and 0.9% have a label in doubt, mainly because evidence removal sometimes leaves a settling passage behind.

W2: Limited coverage of scientific reasoning

The 812 instances represent only 108 original claims drawn from 27 papers, mostly in machine learning. […] the small and narrow source collection limits how broadly the results can be interpreted. In particular, the paper distinguishes claims that are explicitly stated from those requiring inference, but does not clearly report performance separately for these categories. […] Broader coverage, or at least a clear analysis by reasoning requirement, would make the conclusions more convincing.

We thank the reviewer. We ran the requested analysis by reasoning requirement (new Appendix K, Table 21; see Q2), and the revised Limitations section now reports the field composition of the collection.

Analysis by reasoning requirement. This is answered in our reply to Q2 below (new Appendix K, Table 21). The gaps are largest where the counterfactual claim is not stated in any single sentence and must be derived from several passages, so the results are not driven by locating and checking stated facts.

Coverage. We agree that the collection is weighted towards machine learning, and that the 812 instances are 108 matched families rather than 812 independent problems; this is why all confidence intervals are computed by resampling anchors. Of the 108 anchors, 74 come from ML papers, but the collection is not limited to ML:

  • Other fields. The remaining anchors come from robotics (15), physics (14) and optimisation and optimal transport (5).
  • The pipeline is field-agnostic. It needs only a citing paper and the full text of the works it cites, so extending to other disciplines requires no change to the method.
  • A check outside the main field. We also built a small set in computational complexity, using a recently released refutation of the Exponential Time Hypothesis, a widely assumed conjecture, as the premise (Appendix N).

Conclusion. The revised Limitations section reports this composition. The collection is modest and weighted towards ML, but it spans four fields, and the pipeline extends to others without change.

Q2: What reasoning is required beyond checking explicitly stated facts?

Could you provide the number of anchors in each category and their separate per-case and six-case accuracy? For the inference category, a representative example showing the necessary passages and the dependency removed in the insufficient-evidence version would clarify the intended reasoning demand. […] I would view the contribution more favorably if the main findings also hold for claims requiring inference.

We thank the reviewer for this request. We ran a new analysis that splits every result by whether the counterfactual is stated outright in an edited passage or must be derived from several passages (new Appendix K, Table 21). Below we give the counts and accuracy per category (i), a derived example (ii), and whether the main findings hold on derived claims (iii).

(i) Categories, counts and accuracy.

  • Stated: some edited passage states the counterfactual claim C−C^- outright.
  • Derived: no single sentence does, so a reader must combine the edited passages to reach it.

During construction, an automatic check looks for any sentence that states either claim word for word. When it finds one, the model that writes the edits is asked once to make an edit from which the claim must be derived instead; if a literal statement remains, the anchor is recorded as stated. Of the 108 anchors, 48 are stated and 60 derived. Per-cell accuracy for every system and category, with Acc6 and GC6, is in new Appendix K (Table 21, below).

(ii) A representative derived example (also in Appendix K). The citing paper (vLLM) claims that for the 13B-parameter OPT model "the KV cache of a single token demands 800 KB of space, calculated as 2 (key and value vectors) × 5120 (hidden state size) × 40 (number of layers) × 2 (bytes per FP16)". The cited OPT paper never states 800 KB. The reader must combine three passages:

  1. Table 1's caption, which defines the columns: number of layers (#L) and embedding size (dmodeld_{model});
  2. the 13B row (#L = 40, dmodeld_{model} = 5120);
  3. the statement that the weights are kept in FP16.

In P−P^-, one table cell changes: #L goes from 40 to 41. No sentence states the new total, so the reader must recompute 2 × 5120 × 41 × 2 bytes = 820 KB; C−C^- is the same claim with 820 KB and × 41. P0P^0 deletes the table header, the 13B row, the caption and the FP16 sentence, about 350 of the paper's 70,000 characters. The paper still discusses OPT-13B, but the inputs to the calculation are gone, so neither claim is settled.

(iii) Do the main findings hold for claims requiring inference? Yes (Table 21; anchor bootstrap, 2,000 resamples).

  1. The largest difference is in the conflict cell. Accuracy on P−C+P^-C^+ (the original claim against reversed evidence) is 21–44 points lower on derived anchors for all ten systems, and every interval excludes zero. For example, Claude Sonnet 5 drops from 85% to 41%, and GPT-6 Astra from 90% to 68%.
  2. It is not only that derived claims are harder. The drop from P+C+P^+C^+ to P−C+P^-C^+ compares each claim with itself, so harder claims do not inflate it. This drop is 4–26 points larger on derived anchors for all ten systems, and four intervals exclude zero.
  3. The aggregate scores agree. Acc6 is lower on derived anchors for all ten systems, by 5–14 points (six intervals exclude zero). GC6 is lower for seven of the ten.
  4. One effect goes the other way. Both insufficient-evidence cells are answered better on derived anchors, for all ten systems. When no sentence states the claim, removing the premises it rests on is easier to notice.
Reviewer_5wiiOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper identifies the difficulty of distinguishing the problem of judging the correctness of a claim, from knowing whether it is supported by the purported evidence, especially in the setting of current AI systems ability to look for evidence across long contexts. The paper uses a corpus of arXiv papers and identifies claims within them. It also identifies the source of the premises supporting those claims within their context. The paper intervenes on the premises and on the claims, using AI tools and humans review, in order to generate conditions of contradiction and non-support.

These interventions constitute a small corpus, MainLogic, with measured quality.

The paper then reports on the ability of a fairly large number of both older and recent AI models to identify the truth of the natural claims and to identify its potentially counterfactual or unsupported truth under intervention. The idea is to distinguish reasoning from the provided evidence from reasoning from the model's background knowledge (styled as beliefs in the paper). All models suffer from a significant drop in accuracy with respect to the intended, supported, contradicted, or unsupported states under this intervention.

Substantial work is done to avoid confounds from the editing process, or from the abilty of the models to identify the relevant premise material in the long context.

The paper spends, perhaps, a little too much space and partial repetition on rigourous mathematical formalism for a quite simple labelling proceedure; this space might have been used better later in discussing the scope of the results.

Critical strengths

The paper develops a useful corpus, using a highly replicable method; this corpus and the work done with it in the paper will be broadly useful to research on understanding of evidence-based material by AI systems, and in particular to development of scientific paper writing and review quality support too. The paper identifies meaningful weaknesses in the current state of the art, and will drive research. The paper is careful to eliminate some potential confounds.

Critical weaknesses

None

Questions for authors

It is not clear whether and when AI systems should operate in as purely a counterfactual mode as that tested here; one might imagine that they could usefully both say that the evidence contradicts (or does not support) the claim, and that the evidence is probably wrong and the claim right, or vice versa (or absent, but known). How might the evaluation be extended to better characterise whether the model's prior knowledge relates to the broader epistemological process.

The corpus size is cited as a limitation; what isn't mentioned is that this limits MainLogic as a basis of training. How might a sizeable training corpus be built, would the quality limitations identified for the current corpus affect that, and would (re the previous question) training on such a corpus actually be likely to be beneficial.

5 comments

The authors sincerely thank the reviewer for their generous review and for recognising the value of the corpus to the wider community. Your questions helped us strengthen the paper, and we address each below.

Q1: Should models judge purely from the documents, and how can the role of the prior be characterised?

It is not clear whether and when AI systems should operate in as purely a counterfactual mode as that tested here; one might imagine that they could usefully both say that the evidence contradicts (or does not support) the claim, and that the evidence is probably wrong and the claim right, or vice versa (or absent, but known). How might the evaluation be extended to better characterise whether the model's prior knowledge relates to the broader epistemological process.

We thank the reviewer for this question. We ran two new experiments. In the first, models report what the documents establish and what they believe as separate judgments (new Appendix J, Table 20). In the second, they are given a real, unedited refutation of a widely held assumption (new Appendix N, Tables 26–27).

We agree that a useful system need not set aside what it knows. It can report that the documents contradict a claim and, separately, that it believes the claim. MainLogic scores only the first, because that is the judgment needed when checking a citation or reviewing an argument. In both experiments, a model's prior is the answer it gives when asked to judge the claim with no documents.

1. Asking for the document judgment and the belief separately (new Appendix J, Table 20). For four systems we took two kinds of error in which the answer followed the model's prior:

  • 170 conflict errors: the documents settle the claim against the prior, and the model answered with its prior;
  • 80 undetermined commitments: the documents leave the claim open, and the model committed to its prior anyway (20 per system).

We showed each model its recorded answer again and asked it to state separately (a) what the documents alone establish and (b) what it believes about the claim, regardless of the documents.

Table 20 (Appendix J). Share of the sampled errors in which the model's report of the documents matches the reference label, and in which its stated belief matches its prior. Number of rows in brackets.

SystemConflict errors: documents = referenceConflict errors: belief = priorUndetermined: documents = UncertainUndetermined: belief = prior
GPT-6 Astra0% (21)95%5% (20)95%
GPT-5.6 Luna6% (50)82%5% (20)95%
GPT-5.6 Terra0% (49)98%0% (20)100%
GPT-4.12% (50)80%0% (20)75%
  • Separating the two judgments does not recover the document judgment. The report of the documents is correct for only 0–6% of the conflict errors and 0–5% of the undetermined commitments.
  • The models attribute their belief to the documents. The stated belief matches the prior in 75–100% of cases, and the report of the documents usually says the same thing. The models do not say "the documents contradict this, but I believe it"; they say the documents support what they believe.
  • Asking for the deciding passage changes some answers. When asked to quote the passage that settles the claim, 15–45% of the undetermined commitments change to Uncertain.

2. A real, unedited refutation of a widely held assumption (new Appendix N, Tables 26–27). The Exponential Time Hypothesis (ETH) is a standard conjecture in computational complexity. It states that 3-SAT on nn variables cannot be solved in time 2o(n)2^{o(n)}. It has never been proven, but it is widely assumed, and many published lower bounds are stated conditionally on it. A paper released in October 2026, after the evaluated systems were trained, gives a 2o(n)2^{o(n)}-time algorithm for 3-SAT and so refutes ETH and several related hypotheses. Its proofs are formalised in Lean 4. This is the case the reviewer describes, a real document that we did not edit and that contradicts what models are likely to believe from their training data.

We took 11 claims from seven earlier papers that rely on ETH or a related hypothesis (for example, "3-SAT on nn variables cannot be solved in 2o(n)2^{o(n)} time"). We evaluated seven systems on each claim under three conditions, with no edits to any document:

  • Source only: the earlier paper, which assumes the hypothesis without proving it; reference Uncertain.
  • Refutation only: the 2026 paper; reference Contradicted.
  • Both: the earlier paper followed by the refutation; reference Contradicted.

We also asked each system for its judgment of each claim with no documents, to probe prior conflict.

Table 27 (Appendix N). Initially supported: claims the system judged Supported with no documents. Prior-conflict cases answered C: of those, how many it judged Contradicted once given the refutation. The last three columns are accuracy over the 11 claims under each condition; U and C in the headers are the reference labels (Uncertain, Contradicted).

SystemInitially supported claimsPrior-conflict cases answered CSource only (U)Refutation (C)Both (C)
Claude Fable 5.10–100%100%100%
GPT-6 Astra0–91%100%100%
Claude Opus 511/191%100%100%
Claude Sonnet 577/755%100%100%
GPT-5.6 Terra11/136%100%100%
GPT-4.155/536%100%100%
GPT-5.6 Luna11/127%100%91%
All seven1515/1562%100%99%
  • Given a real document that refutes their belief, models follow it. All 77 refutation-only judgments are Contradicted, including all 15 cases where the system had judged the claim Supported without documents. With both documents, 76 of 77 judgments are correct.
  • Systems differ on whether an assumption counts as established. Given only the earlier paper, Claude Fable 5.1, GPT-6 Astra and Claude Opus 5 answer Uncertain on 10–11 of the 11 claims. GPT-5.6 Terra, GPT-4.1 and GPT-5.6 Luna do so on only 3–4, treating the assumed hypothesis as proven.

How the evaluation can be extended. The revised Conclusion and Appendix J propose two extensions:

  1. Score the two judgments separately. Ask for the document judgment and the belief as separate labels. The first is scored against the reference; the second is compared with the prior measured without documents. This would give credit to a model that says "the documents contradict this, but I believe the claim", the behaviour the reviewer describes.
  2. Vary how reliable the source is. Systematically vary the accuracy and plausibility of the premises to measure when a model should accept a source and when it should reject it. This is close to the knowledge-conflict setting of ClashEval (Wu et al., 2024), which varies how large the errors in the supplied context are.

Conclusion. Models can follow a real document that refutes what they believe; all 15 such cases in the ETH set were judged correctly. In the MainLogic errors we sampled, however, models rarely acknowledge a disagreement between the documents and their belief. Asked for both separately, they report that the documents establish what they believe. These errors are therefore misreadings of the documents rather than reasoned disagreements with them. Scoring the two judgments separately and varying the reliability of the source are natural extensions of the matched design.

Q2: Could the pipeline build a training corpus, and would training on it help?

The corpus size is cited as a limitation; what isn't mentioned is that this limits MainLogic as a basis of training. How might a sizeable training corpus be built, would the quality limitations identified for the current corpus affect that, and would (re the previous question) training on such a corpus actually be likely to be beneficial.

We thank the reviewer for this question. We fine-tuned an open model on the current corpus and tested it on held-out papers and on the ETH set, a real, unedited refutation of a widely assumed complexity hypothesis (new Appendix T; the ETH set is described in Q1 and Appendix N). We take the three parts in order.

(i) How a sizeable corpus could be built. It would need many more independent anchors (an anchor is one original claim together with its matched variants), drawn from a broader set of papers and fields. The existing pipeline already produces the candidates, by extracting citation claims, retrieving the cited full texts, locating the evidence and building the matched premise–claim variants. None of these steps is specific to the current papers.

(ii) Would the quality limitations affect it? Yes, and they can now be measured. The extended human audit (Appendix R) finds that 8.1% of reviewed rows fail, and only 0.9% have a label in doubt. Most failures (22 of 26) are edits or deletions that leave a passage that still settles the claim, above all where the evidence was removed (16 of 80 P0P^0 rows). In training, such rows would teach a model to answer Uncertain when the documents in fact settle the claim. A training corpus should therefore use only rows that pass the per-row check (the 337 validated rows are a start, Appendix S), and strengthen the residual-evidence check on P0P^0, the weakest construction.

(iii) Would training be beneficial? We fine-tuned Qwen3.5-9B with LoRA (rank 32, 3 epochs, the paper's prompt). We used three-fold cross-fitting by citing paper, with about 540 training rows per fold, so every row is scored by a model that never saw its paper.

On MainLogic the gain is large.

  • Acc6 rises from 52.6 to 77.3 (+24.7 points, 95% CI [+20.7, +28.5]), level with Claude Fable 5.1 (77.5).
  • GC6 rises from 3.7 to 33.3, higher than any system in Table 2.
  • Most of the gain is in the two conditions with the evidence removed, where P0C+P^0C^+ rises from 42 to 89 and P0C−P^0C^- from 34 to 87.
  • The model is not just spotting deletions. On the pair that contrasts deleting the decisive evidence with deleting a similar amount of redundant text (P0C+P^0C^+ with P=C+P^=C^+), it is right on both members 71% of the time, against 33% before training and 61% for Claude Fable 5.1.

On the ETH set, with unedited sources and a real refutation, training hurts.

  • Accuracy falls from 69.7% to 39–58% across the three fold models.
  • Given the claim's source paper and the refutation together, the untrained model answers Contradicted on 9 of 11 claims; the trained models do so on 2–6 of 11.
  • The trained models learned that the claim's own source paper means Supported.

Conclusion. Training on the constructed rows improves performance on held-out citing papers, but the ETH results show that the gain does not carry over automatically to unedited scientific evidence. A training corpus would need natural sources that only assume a claim, leave it unresolved, or are challenged by later work, so that the presence of the original source stops being a reliable cue for Supported. It would also need per-row validation, given the residual-evidence failures above. The pilot does not yet show how performance scales with corpus size, so MainLogic remains primarily an evaluation benchmark, and we treat it as showing both that a separate training corpus could help and what it would need.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.