- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces MainLogic, a benchmark for evaluating whether long-context LLMs reason from provided evidence rather than relying on prior knowledge. It constructs matched scientific claims through evidence reversal and removal. Experiments across ten frontier models reveal substantial performance drops under evidence interventions and conflicts with measured priors.
Critical strengths
- The paper addresses an important problem of evidence-grounded reasoning in LLMs.
- The matched evidence-intervention design provides a more rigorous evaluation than conventional accuracy metrics.
- The experiments cover ten frontier models and provide useful analyses of prior–evidence conflicts and evidence accessibility.
Critical weaknesses
- The reliability of prior elicitation is unclear. Evidence-free judgments may be sensitive to prompt wording, answer-option ordering, and response format. Have the authors tested whether the measured priors remain stable across different prompt configurations?
- The quality of automatically constructed counterfactual evidence is concerning. The human audit found defects in 6 out of 40 sampled anchors. How can the authors ensure the validity of the remaining samples? An evaluation restricted to verified anchors would be helpful.
- The novelty over existing benchmarks is not entirely clear. Previous works have studied counterfactual evidence, contextual faithfulness, and evidence sufficiency. What additional evaluation capabilities does MainLogic provide beyond combining these existing approaches?
Questions for authors
See weaknesses.
7 comments
W2: Quality of the constructed evidence, and evaluation on verified examples
The human audit found defects in 6 out of 40 sampled anchors. How can the authors ensure the validity of the remaining samples? An evaluation restricted to verified anchors would be helpful.
We thank the reviewer. We ran a new per-row human review of 320 further rows (new Appendix R), and evaluated every system both on the 34 anchors that passed the original audit and on the 337 rows that pass review (Appendices Q and S).
The per-row review replaces the original protocol, which reviewed one sampled row and, if it passed, counted the whole six-cell anchor as verified. Reviewing every row on its own catches a poor edit, a weak negation or an incomplete evidence removal in the row where it occurs, independently of the other rows of the anchor.
1. What the original six failures were. The six anchors failed for different reasons (Appendix Q, Table 31). Three have correct labels but quality problems, two with a bad claim and one with a negation that is too easy. The other three affect labels. In two, the contrary claim is arguably not contrary, and in one, the evidence removal left behind a passage that still settles the claim. Together these affect at most 8 core cells (the six main cells of each anchor, excluding the deletion controls). Excluding all six anchors changes Acc6 by at most 1.1 points, and the ranking is unchanged (Appendix Q, Table 32).
2. How we ensure the validity of the remaining samples. The extended audit covers 320 rows, 40 from each of the eight cells (a cell is one combination of premise version and claim), spanning 105 anchors and 26 citing papers. Together, the two audits cover 368 rows from 106 of the 108 anchors. The auditor first judges the claim from the documents without seeing the reference label, then checks the claim, the contrary claim, and whether the edits or deletions are complete.
| Extended-audit outcome | Rows | Share |
|---|---|---|
| Passed | 294 | 91.9% |
| Construction flawed, label still intended (residual evidence or ambiguous wording) | 23 | 7.2% |
| Label in doubt | 3 | 0.9% |
| Total | 320 | 100% |
All 26 failing rows are excluded from the validated subset. The auditor's blind judgment agrees with the reference label on 90.6% of all 320 rows and 96.6% of the 294 passing rows (Appendix R, Table 33). The main weakness is evidence removal, which leaves a passage that still settles the claim in 16 of the 80 rows. We now report this in the paper.
3. Evaluation restricted to verified examples. We report two restricted evaluations:
- the 34 anchors that passed the original audit, as the reviewer asks;
- the 337 individually validated rows from both audits, of which 250 are core rows. Acc6 is computed on these 250 and reweighted to match the full benchmark's mix of cells.
Values are percentages, from Appendix Q (Table 32) and Appendix S (Table 34).
| System | All 108 anchors: Acc6 / GC6 | 34 audit-passing anchors: Acc6 / GC6 | Validated core rows: Acc6 |
|---|---|---|---|
| Claude Fable 5.1 | 77.5 / 28.7 | 79.4 / 35.3 | 80.0 |
| GPT-6 Astra | 75.8 / 32.4 | 78.4 / 35.3 | 77.2 |
| Claude Opus 5 | 72.1 / 21.3 | 74.5 / 26.5 | 74.0 |
| DeepSeek Flash | 70.8 / 19.4 | 71.6 / 20.6 | 73.7 |
| Gemini 3.8 Flash | 69.4 / 20.4 | 72.5 / 26.5 | 69.7 |
| Claude Sonnet 5 | 67.5 / 17.9 | 67.0 / 15.2 | 69.7 |
| GPT-5.6 Terra | 65.6 / 13.0 | 63.2 / 8.8 | 67.1 |
| Gemini 2.5 Flash | 63.7 / 12.0 | 62.3 / 8.8 | 62.9 |
| GPT-5.6 Luna | 62.0 / 11.1 | 64.7 / 14.7 | 63.6 |
| GPT-4.1 | 53.1 / 2.8 | 59.3 / 8.8 | 54.9 |
- On individually validated rows, Acc6 is within −0.8 to +2.9 points of the headline value, and the ranking holds apart from two near-ties.
- On the 34 audit-passing anchors, a third of the benchmark, scores move more (−2.4 to +6.2) and some mid-table systems swap; the top three and the last are unchanged.
- GC6 needs all six cells of an anchor to be validated, which the row-level subset does not yet provide, so we report it only for the 34-anchor subset.
Conclusion. Reviewed row by row, 92% of rows pass and 0.9% have a label in doubt, and the main weakness, leftover passages after evidence removal, is now measured and reported. Restricted to validated rows, scores move by at most 3 points and the main findings are unchanged.
W3: Novelty over existing benchmarks
The novelty over existing benchmarks is not entirely clear. Previous works have studied counterfactual evidence, contextual faithfulness, and evidence sufficiency. What additional evaluation capabilities does MainLogic provide beyond combining these existing approaches?
We thank the reviewer. MainLogic adds three capabilities that separate counterfactual, faithfulness and sufficiency benchmarks cannot provide, because it tests all the required behaviours on the same scientific claim, using natural claims and the complete cited documents.
1. All required behaviours on the same problem. Each anchor crosses two opposing claims with three premise states (original, reversed and insufficient evidence). This tests whether judgments follow the evidence when the claim is fixed, and follow the claim when the evidence is fixed. A matched deletion control compares removing the decisive evidence with removing a similar amount of nonessential text. The model must answer Uncertain after the first and keep its answer after the second, which tests whether abstention responds to losing the evidence rather than to a shorter document. The control covers 82 anchors and is scored as a pair.
2. Joint correctness that per-condition scores cannot show. GC6 counts an anchor as correct only when all six core judgments are. Consider six anchors. Model A makes one error on every anchor, each time in a different condition. Model B is fully correct on five anchors and wrong on every condition of the sixth. Both score 83.3% overall and in every condition, but their GC6 is 0% and 83.3%. Only GC6 shows whether a model meets all the requirements together on the same problem; Appendix C formalises this.
3. Diagnosis against measured priors and evidence access. The same claims are probed without evidence and evaluated with full documents, highlighted documents and evidence passages only. This links three questions within one design:
- Does accuracy change when the evidence agrees or conflicts with the model's measured prior?
- Does the model fall back on that prior when the documents leave the claim open?
- Do these patterns persist when the evidence is easier to find?
Comparison with representative prior work.
| Related work | What it establishes | What MainLogic adds |
|---|---|---|
| VitaminC (Schuster et al., 2021); CrossAug (Lee et al., 2021) | Contrastive claim–evidence pairs, and crossed original and modified claims and evidence | Joint correctness across two claim orientations and three evidence states for one natural scientific claim |
| FaithEval (Ming et al., 2025) | Faithfulness to unanswerable, inconsistent and counterfactual contexts | Whole-family evaluation, and comparisons conditioned on the prior, across fixed evidence states of the same claims |
| Sufficient Context (Joren et al., 2025); SURE-RAG (Qiu et al., 2026) | Sufficiency analysis and selective answering; SURE-RAG adds grouped evidence conditions with missing information | Paired decisive-versus-redundant deletions that test whether abstention tracks the specific evidence the claim needs |
| CheckList (Ribeiro et al., 2020) | Behavioural testing through directional changes and invariance | Edits to scientific premises with an explicit correct answer for each, six-cell joint scoring and matched deletion controls |
Conclusion. What is new is the unit of evaluation and the comparisons it allows. MainLogic measures which evidence requirements a model meets together on the same scientific problem, and how its failures vary with prior alignment and evidence access. These results cannot be recovered by pooling scores from separate counterfactual, faithfulness and sufficiency benchmarks.
We sincerely thank the reviewer for their encouraging review and the time they gave to our work. Their concern about item quality led us to substantially expand our human audit, as detailed below.
W1: Reliability of prior elicitation
We thank the reviewer for this question. We re-ran the evidence-free probe with reworded instructions and with the answer options reversed, and recomputed RQ2 using only priors that agree across all three configurations (new Appendix L, Tables 22–23).
1. Stability across probe configurations. We re-probed all 216 claims (108 original claims and their contrary versions) on four models, with two additional configurations:
The one-word response format and the decoding settings are unchanged, with one response per claim in each configuration. A prior is stable when all three probes give the same definite answer (Supported or Contradicted). 78–91% of definite priors are stable. Agreement with each alternative probe is 0.82–0.95 (κ 0.68–0.86).
Table 22 (Appendix L). Probe stability. Stable percentages use the definite priors as denominator. Flips are Supported↔Contradicted reversals among definite priors across the two alternative configurations.
As a further check on response format, an earlier run of the same claims on seven models, with a different user-message format, gives κ 0.48–0.82 and points the same way on 85–99% of the claims where both runs give a definite answer.
2. RQ2 restricted to stable priors. We recomputed the three prior-conditioned measures using only stable priors. The alignment gap and ErrDir increase for all four models. DirComp stays high, at 85–92%: it increases for three models and moves from 93% to 92% for GPT-6 Astra.
Table 23 (Appendix L). Reported → stable-prior results. Brackets are anchor-bootstrap 95% intervals. Gap is the accuracy difference between prior-aligned and prior-conflicting evidence; ErrDir and DirComp are the share of prior-consistent answers among conflict errors and among determinate answers under insufficient evidence.
For GPT-6 Astra, the alignment gap rises from 6 to 11 points, and its interval moves from [−1, 13] to [3, 19]. All four stable-prior gaps have intervals that exclude zero.
Conclusion. The central RQ2 findings hold under a stricter definition of the prior. Models remain more accurate when the evidence agrees with their measured prior, and their errors and unsupported commitments still mostly follow that prior.