acceptodds
Under review as a conference paper at ICLR 2027

ASD: Auditing Source Dependence in LLM Judges For Scientific Benchmarking

Abstract

LLM judges are widely used to evaluate AI agents on open-ended scientific benchmarks, where expert grading is prohibitively costly. These benchmarks are often constructed from published papers, which raises the question of whether a verdict reflects the quality of the answer or access to the source paper, since judges can often consult the same literature. We propose Auditing Source Dependence (ASD), a two-stage pipeline that measures source dependence as the change in a judge's verdict when the source content in its evidence is varied. ASD first builds a catalog of relevant and irrelevant papers for each task and splits the source paper into passages (paragraphs and topic blocks) labeled by rhetorical role. It then judges the same answer while swapping the source paper in and out of the judge's evidence. To locate source dependence within the paper, it further applies counterfactual edits to single passages of the source paper. Across BiomniBench, BixBench, and LABBench2, replacing a relevant paper with the source paper raises judge accuracy, and editing a single passage can reverse the verdict on an unchanged answer. Methods and results passages account for over 60% of the most influential passages, and edits to different passages shift verdicts in opposite directions, so small average effects can mask large passage-level effects. These findings motivate reporting the source sensitivity of LLM judges alongside agent performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-as-a-Judge, AI for Science, LLM Reliability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JxqrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether LLM judges on paper-derived scientific benchmarks reward answer quality or simply benefit from access to the source paper. It proposes ASD, which holds the question, candidate answer, and judge fixed while varying the judge’s evidence. At the paper level, ASD compares matched evidence sets that differ only in whether the source paper replaces a retrieved relevant paper (E1+ vs E1−, E3+ vs E3−), plus a no-evidence baseline. At the passage level, it edits single passages of the source paper (REMOVE, FILL, REWRITE) and aggregates verdict changes by LLM-assigned rhetorical role. The authors claim three contributions: the ASD audit framework, evidence that judges are source-dependent, and a passage-level account locating that dependence mainly in Methods and Results. Validation covers 356 tasks across BiomniBench, BixBench, and LABBench2, with three judges each grading evidence pools built by the other two models. The source paper raises verdict accuracy in 70 of 72 matched contrasts, adding non-source papers can lower accuracy, and Methods and Results passages account for over 60% of the absolute intervention effect.

Critical strengths

The paper proposes a clean, controlled design for a timely question. The matched-pair construction swaps exactly one paper in the same prompt position, and the cross-model arrangement keeps judges from grading evidence they retrieved themselves. Together these isolate the evidence variable well, and the question of how evidence shapes LLM-judge verdicts on paper-derived benchmarks matters as Agent-as-a-Judge setups spread.

Critical weaknesses
  1. It’s unclear the main finding is a problem at all. The paper shows that judges get more accurate when they can see the source paper. But the source paper is, by design, the document most relevant to the question, so you’d expect better evidence to produce better judgments. That seems like a good thing, not a flaw. The authors acknowledge this in their limitations but never explain when or why this dependence would actually be harmful. The size of the effect is also inflated by the setup. The judge is told to abstain when unsure, and without evidence it abstains almost all the time (82–99%), so much of the “gain” is just the judge finally committing to an answer once it has something to read.
  2. The passage-level results may be mostly noise. Each input is judged only once, and the paper’s own appendix shows that re-running the exact same prompt changes the verdict about 8–10% of the time. Many of the reported passage-level flip rates fall in that same range, so it’s hard to tell real effects from random variation. Tellingly, paraphrasing a passage (which should change nothing) flips verdicts almost as often as deleting it. The claim that Methods and Results passages drive most of the effect also has a simpler explanation: those sections make up about 65% of all paragraphs in the papers. When you look at how often an individual passage flips a verdict, every role behaves about the same, and Methods passages actually flip slightly less often than Background or Conclusions. Since this passage-level analysis is the paper’s main novel contribution, these problems weaken it substantially.
Questions for authors
  1. What is the threat model? Under what concrete scenario is higher judge accuracy with the source paper harmful rather than desirable? If the authors can describe a realistic case where source access leads judges to reward answers they shouldn’t, or to penalize correct answers that differ from the paper, the paper-level finding becomes much more meaningful. Without one, it reads as “better evidence helps.”
  2. How were topic blocks constructed? Section 3.1 describes contiguous TextTiling spans within a section, while Appendix B describes grouping all sentences with the same rhetorical role, mostly non-contiguous. If the second definition was used, the role-level analysis for topic blocks is partly circular, since blocks are defined by the very labels they are later grouped by.
  3. How do the passage-level flip rates compare to run-to-run variability? Appendix I reports that identical prompts re-run change the verdict 8–10% of the time. Can the authors show that the reported passage effects, especially the role differences and the REWRITE vs REMOVE gap, are distinguishable from this noise level?
  4. Does the Methods + Results finding hold per passage? The 60–65% share closely matches the fraction of paragraphs that are Methods or Results. Can the authors report per-passage effect rates by role, and explain how the conclusion survives Table 12, where per-passage flip rates are nearly flat across roles?
6 comments

Thank you for the detailed and critical review. Table and appendix numbers refer to the revised PDF, and pp means percentage points.

What is the threat model? Under what concrete scenario is higher judge accuracy with the source paper harmful rather than desirable?

We agree that the source paper is expected to contain better evidence, which helps an LLM judge on a benchmark task. However, there are at least two scenarios in which this dependence is a real problem:

  • 11 Scientific Benchmarking at Scale: Current scientific benchmarks are carefully curated and manually labelled by domain experts, so they have few questions although they cover many subfields Mitcheneretal,˙2025;Laurentetal,˙2026;Quetal,˙2026Mitchener et al\., 2025; Laurent et al\., 2026; Qu et al\., 2026. Coding benchmarks such as SWE-bench Jimenezetal,˙2024Jimenez et al\., 2024 can take many verifiers and unit tests directly from codebases. Scientific problems, in contrast, are often open-ended, cannot be verified automatically and require costly human labelling. If an LLM judge can only give an accurate verdict when it has the source, it does not judge independently: it needs the solution in front of it at all times, and solutions are scarce. This is a limit of current LLMs as judges of scientific problems. In an ideal world, we would have a scientific arena with a large number of questions and the corresponding apparatus or environments, where agents solve the tasks and judges, supported by a carefully designed knowledge base (with or without the solution), assess the agents accurately. For open-ended problems that lead to new scientific breakthroughs, no mark scheme or source paper exists, and so the reliability of judges is crucial for benchmarking real scientific discovery abilities of agents.

  • 22 Self-Improving Agents and LLMs: Related to 11, the community is now developing agents and LLMs that improve themselves and aim at real scientific breakthroughs Novikovetal,˙2025;Luetal,˙2026;Zhangetal,˙2026Novikov et al\., 2025; Lu et al\., 2026; Zhang et al\., 2026. We argue that this is not yet possible, because the signal that these systems use to assess their own attempts depends on source-like information. Take On-Policy Self-Distillation OPSD;Zhaoetal,˙2026OPSD; Zhao et al\., 2026 as an example. One LLM is both the student and its own teacher theself−teacherthe self-teacher, and the whole teaching signal comes from privileged information that only the teacher receives, such as the reference solution. The known problems of these algorithms come from this dependence: the student learns shortcuts that it cannot use at test time, and the method fails when the privileged information is specific to each task Zhuetal,˙2026;Robert&Qader,2026Zhu et al\., 2026; Robert \& Qader, 2026. Many later works try to strengthen the self-teacher, or the policy itself, with experience distilled into skills Wangetal,˙2026;OPID,Yangetal,˙2026Wang et al\., 2026; OPID, Yang et al\., 2026, skill graphs Lietal,˙2026Li et al\., 2026 and synthetic data Sicketal,˙2026Sick et al\., 2026. On open-ended scientific tasks the reference solution does not exist, so the self-teacher has only related literature, and our results show that the same models assess answers far less accurately with related literature than with the source. We therefore expect source dependence to be a bottleneck of these methods as well.

To further isolate the effects of source dependence on a judge, we added 11 an experiment where we replaced the candidate answer with the ground truth reference answers of LABBench2 and 22 an experiment on BiomniBench where the judges return a 0-100 score with and without the rubric in the prompt; in both, we compared the source-absent and source-present inputs. Without the source, correct answers are penalized, as the judge rejects most of the correct reference answers, and five relevant papers cannot make up for it accuracy stays 41\.1 pp lower, Fig\. 13. With the source, answers are rewarded that should not be. Only 2 of the 21 BiomniBench answers pass the official rubric in our independent grading, yet the mean judge score rises from 25.0 to 70.5 points once the source is given. Adding the rubric to the prompt lowers the source effect from 45.5 to 19.1 points, and a rubric that a model wrote from the question alone lowers it less to 25\.4 points, Fig\. 28. The judges mostly reward a match with the results reported in the source.

Source absentSource present
Proportion of correct LABBench2 reference answers that pass4.9%52.5%
LABBench2 accuracy of the API judges (3 papers in the evidence)7-37%60-74%
Mean 0-100 judge score on BiomniBench without a rubric officialtotals:mean340˙official totals: mean 34\.025.070.5

So what are potential solutions for the future to mitigate source dependence of LLM judges? We think that there are three axes:

  • Identify problematic scientific domains and tasks through our source dependence framework, and do additional LLM mid-training e\.g\. through direct preference optimization and post-training data and tasks that essentially train an LLM to develop the abilities to critique open-ended science problems.

  • Improving the quality of relevant information access through knowledge databases.

  • Use agent-as-judge so that additional harnesses and checks can be utilized to assist with critiquing.

The size of the effect is also inflated by the setup.

\.\.\.

much of the “gain” is just the judge finally committing to an answer once it has something to read.

We agree that abstentions explain a large part of the gain (Table 2). But the matched comparisons already give the judge three papers to read, and it still abstains until one of them is the source. The effect also remains without abstention. In the rubric experiment, the judges return a 0-100 score, they never abstain on inputs with evidence when they fill in the official rubric, and the source still adds 19.1 points 455˙withouttherubric45\.5 without the rubric.

Can the authors show that the reported passage effects, especially the role differences and the REWRITE vs REMOVE gap, are distinguishable from this noise level?

Yes, for the judge, Qwen3.5 27B evaluated every intact and edited input five times with the source present (Table 10):

InputVerdicts that change 9595% CI
Same input, called again0.3% 00˙−09˙0\.0-0\.9
One passage paraphrased (REWRITE)0.5% 00˙−15˙0\.0-1\.5
One passage replaced by neutral text (FILL)4.6% 25˙−70˙2\.5-7\.0
One passage removed (REMOVE)5.0% 28˙−76˙2\.8-7\.6

Each operation is compared with repeated calls of its own intact inputs. Both the REMOVE and FILL interventions yield results that are higher than the baseline noise level with neither confidence interval including zero.

Does the Methods + Results finding hold per passage?

The finding does not hold for every decomposition, and we revised Sec. 5.3. For the GPT judge of the main text, a single paragraph changes 8.2-15.5% of the verdicts across the seven role labels, and Methods and Results paragraphs are not at the top of this range. So the 60-65% proportion reflects how common these paragraphs are. We compare the role labels across four decompositions under the open-weight judges, Results passages change verdicts more often for top-level sections and topic blocks Results passages vs\. the range over the other roles, Table24:

PassagesREMOVEFILLREWRITE
Paragraphs4.0% vs. 3.0-3.6%3.9% vs. 2.6-3.7%2.2% vs. 1.7-2.4%
TextTiling blocks4.4% vs. 3.0-4.5%4.3% vs. 3.3-5.0%2.3% vs. 2.0-2.5%
Top-level sections12.5% vs. 1.8-11.2%13.5% vs. 1.8-10.8%3.3% vs. 1.8-3.6%
Topic blocks11.1% vs. 3.8-8.6%12.2% vs. 4.9-9.1%4.0% vs. 2.4-4.7%

Under REMOVE and FILL, Results passages are less than 1 pp above the other roles for paragraphs and TextTiling blocks, and 5.0-5.8 pp above them for sections and topic blocks. Methods passages come second there 86˙−112˙8\.6-11\.2%, Table 24. Under REWRITE there is no difference.

How were topic blocks constructed?

\.\.\.

If the second definition was used, the role-level analysis for topic blocks is partly circular, since blocks are defined by the very labels they are later grouped by.

The second definition produced the reported block-level results, and Sec. 3.1 now gives both constructions and states which results use which. The reported block-level results use the construction of Appendix B: a topic block groups all sentences with the same rhetorical role, keeping them in the order in which they appear in the paper.

For the revision, we also ran TextTiling blocks and top-level sections, judged by open-weight judges App\. K, Table 26:

PassagesREMOVEFILLREWRITE
Paragraphs3.48% 29˙6−40˙32\.96-4\.033.44% 29˙2−39˙92\.92-3\.992.16% 17˙7−26˙11\.77-2\.61
TextTiling blocks3.87% 32˙9−45˙13\.29-4\.513.73% 31˙9−42˙83\.19-4\.282.19% 17˙8−26˙41\.78-2\.64
Top-level sections8.53% 75˙8−94˙67\.58-9\.468.88% 79˙9−97˙87\.99-9\.783.20% 26˙4−37˙52\.64-3\.75
Topic blocks6.28% 56˙3−69˙65\.63-6\.967.14% 64˙5−78˙56\.45-7\.853.61% 30˙7−41˙83\.07-4\.18

On LABBench2, a repeated call of intact input changes 0.62% of verdicts 03˙9−08˙50\.39-0\.85, as the mean over the same seven judges. Under all four block constructions, REMOVE and FILL are well above this noise floor, and REWRITE exceeds it by a smaller margin.

Regarding the issue of circular reasoning: The decomposition in Stage 1 Phase 4 is intended to provide intervention units at different levels for the subsequent judge intervention. The topic block is one form of decomposition, used to structurally divide the content of the paper. The rhetorical role is used only after the intervention is completed and for grouping subsequent results together, it does not participate in the interventions. We use it to semantically categorize the structural units that have been intervened upon, thereby analyzing the content or role corresponding to each intervention object.

The annotation targets of these two steps are also different. The decomposition in Stage 1 Phase 4 is annotated at the minimal unit, the sentence; the subsequent semantic analysis is annotated on the entire intervention object. Therefore, structural decomposition and semantic analysis differ in purpose, stage, and annotation granularity: the former determines the objects of intervention, while the latter explains what content the intervened objects are, and the two do not constitute circularity.

Reviewer_YgqJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Auditing Source Dependence (ASD), a framework for testing whether LLM judges on scientific benchmarks rely on the original source paper used to construct a task. ASD holds the question, candidate answer, and judge fixed while varying the evidence provided to the judge. It measures effects at both the paper level (source paper vs. relevant non-source papers) and passage level (REMOVE, FILL, REWRITE edits). Across BiomniBench, BixBench, and LABBench2, the authors find that source-containing evidence often improves judge accuracy and that editing individual passages can change verdicts.

Critical strengths
  • Important and timely problem: scientific benchmarks increasingly rely on LLM judges and retrieval-based evidence.
  • Clear intervention-based design: holding answer and judge fixed while changing evidence is stronger than correlational analysis.
  • Fine-grained analysis at paper, paragraph/topic-block, and rhetorical-role levels.
Critical weaknesses
  • The paper does not sufficiently distinguish source dependence from proper access to more relevant evidence. Since benchmark tasks are constructed from source papers, it is expected that the source paper is especially useful.
  • “Judge accuracy” depends on benchmark correctness labels, some of which may themselves come from automated evaluators. This may measure agreement with existing benchmark evaluation rather than true scientific correctness.
  • The appendix notes that some edits were mistakenly applied to relevant papers rather than source papers, yet these records remain in reported results. These should be removed or analyzed separately.
  • Other factors, such as document order, and source-paper position are not sufficiently controlled.
Questions for authors
  • How do you distinguish problematic source dependence from the expected benefit of giving a judge the most relevant paper?
  • How are correctness labels established for each benchmark, especially LABBench2? Are they independently human-validated?
  • Why are results based on one judge call per condition despite observed prompt-level variability?
  • Why were accidental relevant-paper edits retained in the main results, and do conclusions change after removing them?
5 comments

Thank you for the comments. Table and appendix numbers refer to the revised PDF, and pp means percentage points.

How do you distinguish problematic source dependence from the expected benefit of giving a judge the most relevant paper?

We agree that the source paper is expected to contain better evidence and this was also raised by Reviewer Jxqr. We see at least two settings where depending on the source is a real problem:

  • 11 Scientific Benchmarking at Scale: Today's scientific benchmarks are built and labelled by hand by domain experts, which keeps them small even though they span many subfields Mitcheneretal,˙2025;Laurentetal,˙2026;Quetal,˙2026Mitchener et al\., 2025; Laurent et al\., 2026; Qu et al\., 2026. Coding benchmarks such as SWE-bench Jimenezetal,˙2024Jimenez et al\., 2024 do not have this limit, because codebases already provide a large supply of unit tests and verifiers. Scientific questions are usually open-ended, have no automatic check, and need expensive expert labels. An LLM judge that is accurate only when it sees the source is not judging on its own, since it needs the solution every time and solutions are rare. We consider this a limitation of current LLMs as scientific judges. Ideally, the community would run a scientific arena with many questions and their apparatus or environments, in which agents attempt the tasks and judges evaluate them accurately with the help of a well-designed knowledge base, whether or not it contains the solution. Open-ended problems that could lead to new discoveries come with no source paper and no mark scheme, so reliable judges are essential for benchmarking the real scientific discovery ability of agents. This is the problem we want to point out, both for the benchmarks being built and for the LLM judges used on them.

  • 22 Self-Improving Agents and LLMs: In connection with 11, there is growing work on agents and LLMs that improve themselves with the goal of real scientific discovery Novikovetal,˙2025;Luetal,˙2026;Zhangetal,˙2026Novikov et al\., 2025; Lu et al\., 2026; Zhang et al\., 2026. We believe this cannot work yet, because these systems assess their own attempts with a signal that relies on source-like information. On-Policy Self-Distillation OPSD;Zhaoetal,˙2026OPSD; Zhao et al\., 2026 is one example. The same LLM acts as student and as teacher theself−teacherthe self-teacher, and all of the teaching signal comes from privileged information given only to the teacher, for instance the reference solution. The known weaknesses of these algorithms follow from this reliance, as the student picks up shortcuts that are unavailable at test time and the method breaks down when the privileged information differs from task to task Zhuetal,˙2026;Robert&Qader,2026Zhu et al\., 2026; Robert \& Qader, 2026. Later work tries to make the self-teacher or the policy stronger with experience distilled into skills Wangetal,˙2026;OPID,Yangetal,˙2026Wang et al\., 2026; OPID, Yang et al\., 2026, skill graphs Lietal,˙2026Li et al\., 2026 and synthetic data Sicketal,˙2026Sick et al\., 2026. For open-ended scientific tasks there is no reference solution, so the self-teacher is left with related literature, and our results show that the same models assess answers much less accurately with related literature than with the source. We therefore expect source dependence to limit these methods too.

To further isolate the effects of source dependence on a judge, we added 11 an experiment where we replaced the candidate answer with the ground truth reference answers of LABBench2 and 22 an experiment on BiomniBench where the judges return a 0-100 score with and without the rubric in the prompt; in both, we compared the source-absent and source-present inputs. Without the source, correct answers are penalized, as the judge rejects most of the correct reference answers, and five relevant papers cannot make up for it accuracy stays 41\.1 pp lower, Fig\. 13. With the source, answers are rewarded that should not be. Only 2 of the 21 BiomniBench answers pass the official rubric in our independent grading, yet the mean judge score rises from 25.0 to 70.5 points once the source is given. Adding the rubric to the prompt lowers the source effect from 45.5 to 19.1 points, and a rubric that a model wrote from the question alone lowers it less to 25\.4 points, Fig\. 28. The judges, therefore, mostly reward agreement with the results reported in the source.

Source absentSource present
Proportion of correct LABBench2 reference answers that pass4.9%52.5%
LABBench2 accuracy of the API judges (3 papers in the evidence)7-37%60-74%
Mean 0-100 judge score on BiomniBench without a rubric officialtotals:mean340˙official totals: mean 34\.025.070.5

So what are potential solutions for the future to mitigate source dependence of LLM judges? We think that there are three axes:

  • Identify problematic scientific domains and tasks through our source dependence framework, and do additional LLM mid-training e\.g\. through direct preference optimization and post-training data and tasks that essentially train an LLM to develop the abilities to critique open-ended science problems.

  • Improving the quality of relevant information access through knowledge databases.

  • Use agent-as-judge so that additional harnesses and checks can be utilized to assist with critiquing.

How are correctness labels established for each benchmark, especially LABBench2? Are they independently human-validated?

Reference answers and rubrics come from the benchmarks. In the submission, an LLM grader GPT−6AstrainCodex,withoutwebaccessGPT-6 Astra in Codex, without web access compared each candidate answer with the reference answer (BixBench, LABBench2) or scored it with the rubric (BiomniBench, pass at 60 points), and two co-authors with biological expertise reviewed the labels. For the revision, a second, independent grader gpt−rosalind−astragpt-rosalind-astra regraded all 356 answers under the official grading rule of each benchmark: the deterministic verifiers of the BixBench release (string match, numeric range, semantic comparison), the evaluator prompts of the LABBench2 release (exact or semantic match by question type) and the rubric of each BiomniBench task (pass at 60 points). Domain experts among the authors checked a sample of its labels by hand and agreed with it on every answer of the sample. The results are in App. E, Table 4:

BixBenchLABBench2BiomniBench
Answers where the label equals the grade published by the benchmarkn/a297 / 309n/a
Answers where the label equals the regraded label gpt−rosalind−astra,officialrulesgpt-rosalind-astra, official rules26 / 26299 / 30912 / 21
Matched comparisons that keep their sign under the regraded labels24 / 2424 / 2424 / 24

n/a: BixBench and BiomniBench publish no grades for the answers we use, because we generate these answers ourselves.

The source effect holds under both sets of labels: under the regraded labels all 72 matched comparisons keep their sign. On BixBench and LABBench2, accuracies change by at most 2.3 pp and matched comparisons by at most 1.6 pp. On BiomniBench the two gradings differ 2 vs\. 9 answers pass, so we report it under both label sets: all 24 comparisons stay positive, and their range is 4.8–42.9 pp instead of 4.8–52.4 pp.

Why were accidental relevant-paper edits retained in the main results, and do conclusions change after removing them?

We thank the reviewer for pointing this out. This was due to a task string matching issue which caused them to be included. We removed these edits. All LABBench2 passage-level results in the revision use only the edits of each question's own source paper (Table 20 and 21). With these edits removed, REWRITE still gives the smallest mean change in accuracy on LABBench2 at both resolutions: 8.18 pp against 8.49 and 8.71 pp for REMOVE and FILL for paragraphs, and 7.21 pp against 10.50 and 10.56 pp for topic blocks (Table 20).

Other factors, such as document order, and source-paper position are not sufficiently controlled.

Position sensitivity is a known property of LLMs: in multi-document question answering, accuracy depends on where the document with the answer is placed Liuetal,˙2024Liu et al\., 2024, and LLM judges are sensitive to the order of their inputs Shietal,˙2025Shi et al\., 2025. In our experiments, the papers of an input are ordered by retrieval rank, and every input follows this rule.

For the revision we added an experiment on the position of the source paper. In E3+ we kept the order of the relevant and the irrelevant paper and placed the source paper first, second or third App\. I, Fig\. 14. On LABBench2, the accuracy of Qwen3.8 27B over the three evidence pools and both passage resolutions is 71.0-73.4% with the source paper first, 73.2-77.7% with it second and 74.3–79.4% with it third, against 32.7-35.3% without the source paper E3−E3-. The position of the source paper changes accuracy by 2.7-5.4 pp. The source effect remains: on all three benchmarks, accuracy is above E3− in every position, by 36.0-40.1 pp on LABBench2.

Why are results based on one judge call per condition despite observed prompt-level variability?

The input of the judge is identical across calls and we use temperature 0, so variation only comes from decoding. We measured it with a fixed open-weight model and the source present: a repeated call changes 0.3% of verdicts 00˙−09˙0\.0-0\.9, and removing one passage changes 5.0% 28˙−76˙2\.8-7\.6 (Table 10).

Reference:

Reviewer_SJpkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ASD, a pipeline for measuring how scientific LLM judges depend on source papers. It holds questions and candidate answers fixed while replacing papers or removing, filling, and rewriting individual passages. Experiments cover three biological benchmarks and three judge models. Source access generally improves grading accuracy, while passage edits sometimes change verdicts. The authors recommend reporting judges’ source sensitivity alongside benchmark scores.

Critical strengths
  1. I found the focus on judges’ evidence useful. Holding the answer fixed while changing the evidence makes it possible to study a factor that benchmark scores can otherwise obscure. Comparing source papers against relevant alternatives also provides a more informative comparison than simply removing all evidence.

  2. The paper-level results are fairly consistent across the evaluated settings: source access improves recorded accuracy in 70 of 72 matched comparisons. I also appreciate that the authors distinguish source dependence from improper source use or invalid grading. This makes the recommendation to report judges’ evidence conditions more actionable.

Critical weaknesses
  1. I would be more cautious about attributing individual verdict flips to particular passages. Each input is judged only once, while Appendix I reports roughly 8–10% disagreement between repeated calls on identical source-free prompts. A corresponding source-present control would help establish how much passage-level sensitivity exceeds ordinary judge variability. This limits my confidence in the fine-grained attribution, although the broader paper-level finding remains useful.

  2. Also, I think the Methods/Results finding needs a clearer interpretation. These roles already account for roughly 64% of pooled paragraph edits in Table 12, so their reported 60–65% share of intervention effects may largely reflect their prevalence. Their per-paragraph flip rates are also similar to Background and Conclusions. The stronger Results effect for topic blocks is interesting, but differences in passage length and grouping make the role-level conclusion less clear.

Questions for authors
  1. On a representative subset, do passage effects exceed disagreement between repeated calls on identical source-present inputs? Do the same passages remain influential across repetitions?

  2. Does the Methods/Results conclusion hold after accounting for role prevalence and passage length?

  3. Section 3.1 describes contiguous topic blocks produced by TextTiling, whereas Appendix B describes grouping sentences by rhetorical role. Which representation produced the reported results? Please also reconcile the source-only passage comparison in Section 3.2 with the three-paper evidence used experimentally.

5 comments

Thank you for the suggestions. Table and appendix numbers refer to the revised PDF, and pp means percentage points.

On a representative subset, do passage effects exceed disagreement between repeated calls on identical source-present inputs? Do the same passages remain influential across repetitions?

Yes to both. The API providers have updated their models since the submission. For example, DeepSeek upgraded DeepSeek V4.1 Flash to DeepSeek-V4.1-Flash-1017 on 17 October 2026, and the model name deepseek-flash now serves the new version [DeepSeek API change log](https://api-docs.deepseek.com/updates/#date-2026-10-17). A rerun of the API judges would not use the models of the submission, so we used an open-weight judge, Qwen3.5 27B. It judged every intact and every edited input five times (Table 10):

InputVerdicts that change 9595% CI
Same input, called again0.3% 00˙−09˙0\.0-0\.9
REWRITE0.5% 00˙−15˙0\.0-1\.5
FILL4.6% 25˙−70˙2\.5-7\.0
REMOVE5.0% 28˙−76˙2\.8-7\.6

Each operation is compared with repeated calls of its own intact inputs. The difference is +5.0 pp for REMOVE 28˙to76˙2\.8 to 7\.6 and +4.6 pp for FILL 25˙to70˙2\.5 to 7\.0, and neither interval includes zero.

Passage influence remains consistent across five repetitions. Most verdict flipping edits under REMOVE 13/1813/18 and FILL 12/1612/16 succeed in all five repeats. This stability is supported by the 0.86 Jaccard index and a 0.95 rank correlation for pass probability changes (Table 11). No REWRITE edit flips a verdict across all five repetitions.

Does the Methods/Results conclusion hold after accounting for role prevalence and passage length?

The conclusion does not hold for every decomposition, and we revised Sec. 5.3 For the GPT judge of the main text, a single paragraph changes 8.2-15.5% of the verdicts across the seven role labels, and Methods and Results paragraphs are not at the top of this range. So the 60-65% proportion reflects how common these paragraphs are. We compare the role labels across four decompositions under the open-weight judges. Results passages change verdicts more often for top-level sections and topic blocks Results passages vs\. the range over the other roles, Table24:

BlocksREMOVEFILLREWRITE
Paragraphs4.0% vs. 3.0-3.6%3.9% vs. 2.6-3.7%2.2% vs. 1.7-2.4%
TextTiling blocks4.4% vs. 3.0-4.5%4.3% vs. 3.3-5.0%2.3% vs. 2.0-2.5%
Top-level sections12.5% vs. 1.8-11.2%13.5% vs. 1.8-10.8%3.3% vs. 1.8-3.6%
Topic blocks11.1% vs. 3.8-8.6%12.2% vs. 4.6-9.1%4.0% vs. 2.4-4.7%

Under REMOVE and FILL, Results passages are less than 1 pp above the other roles for paragraphs and TextTiling blocks, and 5.0-5.8 pp above them for sections and topic blocks. Methods passages come second there 86˙−112˙8\.6-11\.2%, Table 24. Under REWRITE there is no difference.

On role prevalence: per paragraph, the five roles of the paper body flip 8.9-10.7% of the verdicts of the three API judges (Table 22), so the share of a role in the flips follows its share of the paragraphs. On passage length: against blocks of other roles of similar length, Results blocks flip 4.0 pp more verdicts under REMOVE 25˙−55˙2\.5-5\.5 and 4.0 pp more under FILL 25˙−56˙2\.5-5\.6, and −0.1 pp under REWRITE −06˙to05˙−0\.6 to 0\.5 (Table 23).

Section 3.1 describes contiguous topic blocks produced by TextTiling, whereas Appendix B describes grouping sentences by rhetorical role. Which representation produced the reported results? Please also reconcile the source-only passage comparison in Section 3.2 with the three-paper evidence used experimentally.

For question 1:

Thank you for pointing this out. The reported block-level results use the role-based construction of Appendix B: a topic block groups all sentences with the same rhetorical role. In the original work, we considered three passage decompositions: paragraphs, topic blocks, and TextTiling blocks. TextTiling results were erroneously left out. We have now fixed this and apologize for the confusion this caused.

In the revision, we also added a fourth decomposition: top-level sections Sec31˙Sec 3\.1, which follow the authors' top-level section headings. Sec. 3.1 now describes all four and states which results use which. The main experiments use paragraphs and topic blocks, and the two further decompositions are reported with all seven open-weight judges App\. K, Table 26. Flip rates on LABBench2:

PassagesREMOVEFILLREWRITE
Paragraphs3.48% 29˙6−40˙32\.96-4\.033.44% 29˙2−39˙92\.92-3\.992.16% 17˙7−26˙11\.77-2\.61
TextTiling blocks3.87% 32˙9−45˙13\.29-4\.513.73% 31˙9−42˙83\.19-4\.282.19% 17˙8−26˙41\.78-2\.64
Top-level sections8.53% 75˙8−94˙67\.58-9\.468.88% 79˙9−97˙87\.99-9\.783.20% 26˙4−37˙52\.64-3\.75
Topic blocks6.28% 56˙3−69˙65\.63-6\.967.14% 64˙5−78˙56\.45-7\.853.61% 30˙7−41˙83\.07-4\.18

On LABBench2, a repeated call of intact input changes 0.62% of verdicts 03˙9−08˙50\.39-0\.85, as the mean over the same seven judges. Under all four block constructions, REMOVE and FILL are well above this noise floor, and REWRITE exceeds it by a smaller margin.

For question 2:

On the passage comparison of Sec. 3.2: all passage-level experiments use the three-paper input. The intact input of every edit is E3+ (the source paper, one relevant paper and one irrelevant paper), and the edited input is E3+ with one passage of the source paper edited; the two other papers are unchanged. Sec. 3.2 of the submission wrote the two inputs as the source paper alone, which was a notation error; Sec. 5.1 (Evidence conditions) already described the three-paper setup. We have corrected Sec. 3.2 the definition above Eq\. 7. The intact verdict of every edit is therefore the E3+ verdict of the paper-level experiment, and each passage-level effect is the effect of editing one passage while the rest of the source paper and the two other papers stay in the input.

AuthorsOpenReview ↗

Quick Follow Up

Dear Reviewer,

Thanks a lot for your reviews again. We were just wondering if you have had time to take a quick look at our responses and whether you have any follow up questions or concerns, as we approach the 2 days of the discussion period? We would be glad to follow up if anything is unclear, thanks!

Best, Authors

Comments on the paper

4 comments
AuthorsOpenReview ↗

Reply to All Readers and Reviewers

We thank all three reviewers for their valuable comments. For the convenience of the reviewers and the area chair, we give a unified reply here, and we answer the specific questions of each reviewer in the individual reply.

In short. A judge that can check answers against a paper may still struggle to judge new discoveries. It may also reward poor answers just because they agree with the paper. A brief summary of our contributions:

  • ASD, an interventional audit that estimates source effects from verdicts on evidence that differs in one paper or one edited passage. All three reviewers value this controlled design.

  • Evidence of source dependence. The source paper raises accuracy in 70 of 72 matched comparisons and we were not able to attain this level of accuracy with the addition of relevant papers.

  • How to pinpoint source-dependence within a source paper . Removing a passage changes verdicts far more often than a repeated call, a paraphrase does not, and Results blocks matter most.

All three reviewers agree on two strengths of the paper:

  • the controlled design (the answer and the judge are fixed, and only the evidence changes) for studying how evidence affects the verdicts of LLM judges.

  • the consistent paper-level result (70 of 72 matched comparisons).

The reviewers raised three main concerns. We answer them here, and we answer the specific questions of each reviewer in the individual comments.

11 Whether source dependence is a real problem for LLM judges, since the source paper is expected to contain the best evidence for judging.

We agree that the source paper is expected to contain the best evidence. The problem is trusting a judge that performs well when given the published answer to assess new research where no answer is known. Two scenarios make this concrete:

  • Scientific benchmarking at scale. In an ideal world, a scientific arena would hold a large number of open-ended questions, agents would solve them, and judges would assess them from a knowledge database, often without a solution or source paper. A judge that needs the source paper cannot support this, because new problems have no source paper or mark scheme.

  • Self-improving agents and LLMs. These systems improve by assessing their own attempts. In On-Policy Self-Distillation Zhaoetal,˙ICML2026Zhao et al\., ICML 2026, the teaching signal comes from privileged information such as the reference solution. For new problems it does not exist, so we expect source dependence to limit these methods as well.

Two new results Table 1, Fig\. 13 and Fig\. 28 show this:

- Correct answers do not pass without the source. Only 4.9% of the LABBench2 reference answers pass 525˙52\.5% with the source. Furthermore, Figure 13 shows that supplying five other relevant papers cannot substitute for the missing source paper because the accuracy on candidate answers still remains 41.1 pp below the E3+ baseline under those conditions Fig\. 13.

- Weak answers are rewarded with the source. Only 2 of the 21 BiomniBench answers pass the official rubric in an independent grading, but the mean score of the judges rises from 25.0 to 70.5 points when they see the source Fig\. 28.

So what are potential solutions for the future to mitigate source dependence of LLM judges? We think that there are three axes that future research can focus on:

  • Identify problematic scientific domains and tasks through our source dependence framework, and do additional LLM mid-training e\.g\. through direct preference optimization and post-training data and tasks that essentially train an LLM to develop the abilities to critique open-ended science problems.

  • Improving the quality of relevant information access through knowledge databases.

  • Use agent-as-judge so that additional harnesses and checks can be utilized to assist with critiquing.

The full argument and the references are in our replies to Reviewers YgqJ and Jxqr. Both settings are now in the paper Sec\. 1 and Sec\. 6.

22 Whether the passage effects exceed call-to-call noise.

They do for REMOVE and FILL. REWRITE stays at the level of noise, and this is the result we intended. REWRITE is a control that changes the wording of a passage and keeps the information in it, so it measures how much a verdict moves when only the text of the prompt changes. Because the verdicts do not move beyond the noise under REWRITE, the flips under REMOVE and FILL come from the information that was removed, and not from the change to the prompt.

The passage edits are applied to inputs that contain the source paper E3+E3+, so we measured the noise again on these inputs. The API providers have updated their models since the submission. For example, DeepSeek upgraded DeepSeek V4.1 Flash to DeepSeek-V4.1-Flash-1017 on 17 October 2026, and the model name deepseek-flash now serves the new version [DeepSeek API change log](https://api-docs.deepseek.com/updates/#date-2026-10-17). A rerun of the API judges would not use the models of the submission, so we used an open-weight judge, Qwen3.5 27B. It judged every intact and every edited input five times (Table 10):

InputVerdicts that change 9595% CI
Same input, called again0.3% 00˙−09˙0\.0-0\.9
One passage paraphrased (REWRITE)0.5% 00˙−15˙0\.0-1\.5
One passage replaced by neutral text (FILL)4.6% 25˙−70˙2\.5-7\.0
One passage removed (REMOVE)5.0% 28˙−76˙2\.8-7\.6

Each operation is compared with repeated calls of its own intact inputs. Both the REMOVE and FILL interventions yield results that are higher than the baseline noise level with neither confidence interval including zero.

33 How to read the Methods/Results finding.

At the paragraph level, the 60-65% share of Methods and Results partly reflects how common these paragraphs are, and we revised this claim Sec\. 5\.3. What holds is an effect on larger passages, where Results passages change verdicts most often and Methods passages come second.

In addition, the table shows this for four ways of splitting the source paper, with the open-weight judges on LABBench2. Each cell gives the flip rate of Results passages against the range over the other roles (Table 24):

PassagesREMOVEFILLREWRITE
Paragraphs4.0% vs. 3.0-3.6%3.9% vs. 2.6-3.7%2.2% vs. 1.7-2.4%
TextTiling blocks4.4% vs. 3.0-4.5%4.3% vs. 3.3-5.0%2.3% vs. 2.0-2.5%
Top-level sections12.5% vs. 1.8-11.2%13.5% vs. 1.8-10.8%3.3% vs. 1.8-3.6%
Topic blocks11.1% vs. 3.8-8.6%12.2% vs. 4.6-9.1%4.0% vs. 2.4-4.7%
  • Paragraphs and TextTiling blocks. Results passages are less than 1 pp above the other roles.

  • Top-level sections and topic blocks. Under REMOVE and FILL, Results passages change 11.1-13.5% of verdicts, and Methods passages come second with 8.6-11.2%. Top-level sections are built without role labels, so the effect does not depend on how topic blocks are defined.

  • At similar length. Results passages stay above the other roles under REMOVE, by 3.3 pp for top-level sections and 3.8 pp for topic blocks on LABBench2 (Table 23).

  • REWRITE. The roles do not differ.

Overall, we added the following new experiments below. Table, figure and appendix numbers refer to the uploaded revised PDF, and pp means percentage points.

New experimentResultWhere
Every input judged 5 times with the source present Qwen35˙27BQwen3\.5 27BRepeated call: 0.3% of verdicts change. REMOVE: 5.0%. FILL: 4.6%. REWRITE: 0.5%Table 10
Comparisons among Paragraphs, TextTiling blocks, top-level sections and topic blocksLABBench2: REMOVE flips 3.5-8.5% of verdicts and FILL 3.4-8.9%. A repeated call changes 0.6%Table 26
Results blocks vs. blocks of other roles of similar length+4.0 pp under REMOVE +4.0 pp under FILL, -0.1 pp under REWRITE.Table 23
Paper-level comparison E3+againstE3−E3+ against E3−+43.8 pp 370˙−503˙37\.0-50\.3 for Qwen 3.8 27BTable 16
Source paper is placed in the first, second or third position within E3+Accuracy stays above E3- in every position on all three benchmarksFig. 14
1 to 5 relevant papers, with and without the sourceAdding more relevant papers does not raise accuracy without the source paper, which stays far below accuracyFig. 13
Published reference answers as candidates4.9% pass without the source, 52.5% with it on LABBench2Table 1
0–100 scores with and without the BiomniBench rubricJudge scores rise sharply once the source paper is in evidence, and this gain shrinks most when the official rubric is in the prompt.Fig. 28
All 356 labels regraded by gpt-rosalind-astraAll 72 matched comparisons keep their signTable 4

In the revised manuscript, we have marked all revisions in red, in particular the additional results in the main and appendix sections, abstract, introduction, experiments and added a discussion of when source dependence is a problem Sec\. 1. We look forward to discussing the paper further with the reviewers. Please let us know if you have any further questions!

Reference

Zhao, S., Xie, Z., Liu, M., Huang, J., Pang, G., Chen, F., and Grover, A. Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. In Proceedings of the 43rd International Conference on Machine Learning (ICML), volume 306 of Proceedings of Machine Learning Research, pp. 162433–162448, 2026. URL https://proceedings.mlr.press/v306/zhao26be.html.

Area_Chair_yLM4OpenReview ↗

Please review the authors’ responses and join the discussion

Dear Reviewers,

The authors have responded to all reviews. Please read their responses and the other reviewers’ comments, then post a brief follow-up indicating which concerns, if any, remain and whether the responses have changed your assessment. If your concerns have been resolved, a short confirmation would also be helpful.

Thank you for your time and for contributing to the discussion.

Best regards,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.