- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks whether LLM judges on paper-derived scientific benchmarks reward answer quality or simply benefit from access to the source paper. It proposes ASD, which holds the question, candidate answer, and judge fixed while varying the judge’s evidence. At the paper level, ASD compares matched evidence sets that differ only in whether the source paper replaces a retrieved relevant paper (E1+ vs E1−, E3+ vs E3−), plus a no-evidence baseline. At the passage level, it edits single passages of the source paper (REMOVE, FILL, REWRITE) and aggregates verdict changes by LLM-assigned rhetorical role. The authors claim three contributions: the ASD audit framework, evidence that judges are source-dependent, and a passage-level account locating that dependence mainly in Methods and Results. Validation covers 356 tasks across BiomniBench, BixBench, and LABBench2, with three judges each grading evidence pools built by the other two models. The source paper raises verdict accuracy in 70 of 72 matched contrasts, adding non-source papers can lower accuracy, and Methods and Results passages account for over 60% of the absolute intervention effect.
Critical strengths
The paper proposes a clean, controlled design for a timely question. The matched-pair construction swaps exactly one paper in the same prompt position, and the cross-model arrangement keeps judges from grading evidence they retrieved themselves. Together these isolate the evidence variable well, and the question of how evidence shapes LLM-judge verdicts on paper-derived benchmarks matters as Agent-as-a-Judge setups spread.
Critical weaknesses
- It’s unclear the main finding is a problem at all. The paper shows that judges get more accurate when they can see the source paper. But the source paper is, by design, the document most relevant to the question, so you’d expect better evidence to produce better judgments. That seems like a good thing, not a flaw. The authors acknowledge this in their limitations but never explain when or why this dependence would actually be harmful. The size of the effect is also inflated by the setup. The judge is told to abstain when unsure, and without evidence it abstains almost all the time (82–99%), so much of the “gain” is just the judge finally committing to an answer once it has something to read.
- The passage-level results may be mostly noise. Each input is judged only once, and the paper’s own appendix shows that re-running the exact same prompt changes the verdict about 8–10% of the time. Many of the reported passage-level flip rates fall in that same range, so it’s hard to tell real effects from random variation. Tellingly, paraphrasing a passage (which should change nothing) flips verdicts almost as often as deleting it. The claim that Methods and Results passages drive most of the effect also has a simpler explanation: those sections make up about 65% of all paragraphs in the papers. When you look at how often an individual passage flips a verdict, every role behaves about the same, and Methods passages actually flip slightly less often than Background or Conclusions. Since this passage-level analysis is the paper’s main novel contribution, these problems weaken it substantially.
Questions for authors
- What is the threat model? Under what concrete scenario is higher judge accuracy with the source paper harmful rather than desirable? If the authors can describe a realistic case where source access leads judges to reward answers they shouldn’t, or to penalize correct answers that differ from the paper, the paper-level finding becomes much more meaningful. Without one, it reads as “better evidence helps.”
- How were topic blocks constructed? Section 3.1 describes contiguous TextTiling spans within a section, while Appendix B describes grouping all sentences with the same rhetorical role, mostly non-contiguous. If the second definition was used, the role-level analysis for topic blocks is partly circular, since blocks are defined by the very labels they are later grouped by.
- How do the passage-level flip rates compare to run-to-run variability? Appendix I reports that identical prompts re-run change the verdict 8–10% of the time. Can the authors show that the reported passage effects, especially the role differences and the REWRITE vs REMOVE gap, are distinguishable from this noise level?
- Does the Methods + Results finding hold per passage? The 60–65% share closely matches the fraction of paragraphs that are Methods or Results. Can the authors report per-passage effect rates by role, and explain how the conclusion survives Table 12, where per-passage flip rates are nearly flat across roles?
6 comments
So what are potential solutions for the future to mitigate source dependence of LLM judges? We think that there are three axes:
-
Identify problematic scientific domains and tasks through our source dependence framework, and do additional LLM mid-training e\.g\. through direct preference optimization and post-training data and tasks that essentially train an LLM to develop the abilities to critique open-ended science problems.
-
Improving the quality of relevant information access through knowledge databases.
-
Use agent-as-judge so that additional harnesses and checks can be utilized to assist with critiquing.
\.\.\.The size of the effect is also inflated by the setup.
much of the “gain” is just the judge finally committing to an answer once it has something to read.
We agree that abstentions explain a large part of the gain (Table 2). But the matched comparisons already give the judge three papers to read, and it still abstains until one of them is the source. The effect also remains without abstention. In the rubric experiment, the judges return a 0-100 score, they never abstain on inputs with evidence when they fill in the official rubric, and the source still adds 19.1 points .
Can the authors show that the reported passage effects, especially the role differences and the REWRITE vs REMOVE gap, are distinguishable from this noise level?
Yes, for the judge, Qwen3.5 27B evaluated every intact and edited input five times with the source present (Table 10):
| Input | Verdicts that change |
|---|---|
| Same input, called again | 0.3% |
| One passage paraphrased (REWRITE) | 0.5% |
| One passage replaced by neutral text (FILL) | 4.6% |
| One passage removed (REMOVE) | 5.0% |
Each operation is compared with repeated calls of its own intact inputs. Both the REMOVE and FILL interventions yield results that are higher than the baseline noise level with neither confidence interval including zero.
Does the Methods + Results finding hold per passage?
The finding does not hold for every decomposition, and we revised Sec. 5.3. For the GPT judge of the main text, a single paragraph changes 8.2-15.5% of the verdicts across the seven role labels, and Methods and Results paragraphs are not at the top of this range. So the 60-65% proportion reflects how common these paragraphs are. We compare the role labels across four decompositions under the open-weight judges, Results passages change verdicts more often for top-level sections and topic blocks Results passages vs\. the range over the other roles, Table24:
| Passages | REMOVE | FILL | REWRITE |
|---|---|---|---|
| Paragraphs | 4.0% vs. 3.0-3.6% | 3.9% vs. 2.6-3.7% | 2.2% vs. 1.7-2.4% |
| TextTiling blocks | 4.4% vs. 3.0-4.5% | 4.3% vs. 3.3-5.0% | 2.3% vs. 2.0-2.5% |
| Top-level sections | 12.5% vs. 1.8-11.2% | 13.5% vs. 1.8-10.8% | 3.3% vs. 1.8-3.6% |
| Topic blocks | 11.1% vs. 3.8-8.6% | 12.2% vs. 4.9-9.1% | 4.0% vs. 2.4-4.7% |
Under REMOVE and FILL, Results passages are less than 1 pp above the other roles for paragraphs and TextTiling blocks, and 5.0-5.8 pp above them for sections and topic blocks. Methods passages come second there . Under REWRITE there is no difference.
\.\.\.How were topic blocks constructed?
If the second definition was used, the role-level analysis for topic blocks is partly circular, since blocks are defined by the very labels they are later grouped by.
The second definition produced the reported block-level results, and Sec. 3.1 now gives both constructions and states which results use which. The reported block-level results use the construction of Appendix B: a topic block groups all sentences with the same rhetorical role, keeping them in the order in which they appear in the paper.
For the revision, we also ran TextTiling blocks and top-level sections, judged by open-weight judges App\. K, Table 26:
| Passages | REMOVE | FILL | REWRITE |
|---|---|---|---|
| Paragraphs | 3.48% | 3.44% | 2.16% |
| TextTiling blocks | 3.87% | 3.73% | 2.19% |
| Top-level sections | 8.53% | 8.88% | 3.20% |
| Topic blocks | 6.28% | 7.14% | 3.61% |
On LABBench2, a repeated call of intact input changes 0.62% of verdicts , as the mean over the same seven judges. Under all four block constructions, REMOVE and FILL are well above this noise floor, and REWRITE exceeds it by a smaller margin.
Regarding the issue of circular reasoning: The decomposition in Stage 1 Phase 4 is intended to provide intervention units at different levels for the subsequent judge intervention. The topic block is one form of decomposition, used to structurally divide the content of the paper. The rhetorical role is used only after the intervention is completed and for grouping subsequent results together, it does not participate in the interventions. We use it to semantically categorize the structural units that have been intervened upon, thereby analyzing the content or role corresponding to each intervention object.
The annotation targets of these two steps are also different. The decomposition in Stage 1 Phase 4 is annotated at the minimal unit, the sentence; the subsequent semantic analysis is annotated on the entire intervention object. Therefore, structural decomposition and semantic analysis differ in purpose, stage, and annotation granularity: the former determines the objects of intervention, while the latter explains what content the intervened objects are, and the two do not constitute circularity.
Thank you for the detailed and critical review. Table and appendix numbers refer to the revised PDF, and pp means percentage points.
We agree that the source paper is expected to contain better evidence, which helps an LLM judge on a benchmark task. However, there are at least two scenarios in which this dependence is a real problem:
1 Scientific Benchmarking at Scale: Current scientific benchmarks are carefully curated and manually labelled by domain experts, so they have few questions although they cover many subfields Mitcheneretal,˙2025;Laurentetal,˙2026;Quetal,˙2026. Coding benchmarks such as SWE-bench Jimenezetal,˙2024 can take many verifiers and unit tests directly from codebases. Scientific problems, in contrast, are often open-ended, cannot be verified automatically and require costly human labelling. If an LLM judge can only give an accurate verdict when it has the source, it does not judge independently: it needs the solution in front of it at all times, and solutions are scarce. This is a limit of current LLMs as judges of scientific problems. In an ideal world, we would have a scientific arena with a large number of questions and the corresponding apparatus or environments, where agents solve the tasks and judges, supported by a carefully designed knowledge base (with or without the solution), assess the agents accurately. For open-ended problems that lead to new scientific breakthroughs, no mark scheme or source paper exists, and so the reliability of judges is crucial for benchmarking real scientific discovery abilities of agents.
2 Self-Improving Agents and LLMs: Related to 1, the community is now developing agents and LLMs that improve themselves and aim at real scientific breakthroughs Novikovetal,˙2025;Luetal,˙2026;Zhangetal,˙2026. We argue that this is not yet possible, because the signal that these systems use to assess their own attempts depends on source-like information. Take On-Policy Self-Distillation OPSD;Zhaoetal,˙2026 as an example. One LLM is both the student and its own teacher theself−teacher, and the whole teaching signal comes from privileged information that only the teacher receives, such as the reference solution. The known problems of these algorithms come from this dependence: the student learns shortcuts that it cannot use at test time, and the method fails when the privileged information is specific to each task Zhuetal,˙2026;Robert&Qader,2026. Many later works try to strengthen the self-teacher, or the policy itself, with experience distilled into skills Wangetal,˙2026;OPID,Yangetal,˙2026, skill graphs Lietal,˙2026 and synthetic data Sicketal,˙2026. On open-ended scientific tasks the reference solution does not exist, so the self-teacher has only related literature, and our results show that the same models assess answers far less accurately with related literature than with the source. We therefore expect source dependence to be a bottleneck of these methods as well.
To further isolate the effects of source dependence on a judge, we added 1 an experiment where we replaced the candidate answer with the ground truth reference answers of LABBench2 and 2 an experiment on BiomniBench where the judges return a 0-100 score with and without the rubric in the prompt; in both, we compared the source-absent and source-present inputs. Without the source, correct answers are penalized, as the judge rejects most of the correct reference answers, and five relevant papers cannot make up for it accuracy stays 41\.1 pp lower, Fig\. 13. With the source, answers are rewarded that should not be. Only 2 of the 21 BiomniBench answers pass the official rubric in our independent grading, yet the mean judge score rises from 25.0 to 70.5 points once the source is given. Adding the rubric to the prompt lowers the source effect from 45.5 to 19.1 points, and a rubric that a model wrote from the question alone lowers it less to 25\.4 points, Fig\. 28. The judges mostly reward a match with the results reported in the source.