- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates the reliability of evaluating scientific agents using reference-based LLM grading. The authors introduce NicheLedger, a spatial transcriptomics agent benchmark, and audit evaluation outcomes by independently replacing reference answers and graders while keeping model responses fixed. Their experiments show that both reference quality and grader choice can substantially affect evaluation scores and statistical conclusions. Similar audits on BixBench, however, reveal smaller effects and stable model rankings. The paper highlights important issues in benchmark reliability, but its primary contribution is empirical auditing rather than a novel evaluation methodology.
Critical strengths
-
By replacing reference answers and graders independently while keeping model responses fixed, the authors provide a straightforward way to isolate how these evaluation components affect benchmark outcomes. This controlled design makes the observed score changes interpretable.
-
The authors investigate both reference quality and grader variability in spatial transcriptomics and extend the analysis to BixBench. The results provide useful evidence that evaluation reliability should be examined beyond aggregate scores, even though the effects are not equally pronounced across benchmarks.
Critical weaknesses
-
The finding that automatically generated references can be inaccurate is relatively unsurprising, and the paper does not establish a substantially new method for addressing this problem. The more interesting contribution is the sensitivity of evaluation outcomes to grader changes, but the work primarily documents this issue rather than developing a principled approach to making evaluations more reliable. This limits the methodological contribution beyond an empirical audit of existing benchmarks.
-
The grader effects are substantial on NicheLedger but much smaller on BixBench, where model rankings remain stable. The paper does not sufficiently explain when grader variability meaningfully undermines model comparisons or establish a validated evaluation protocol that improves grading consistency and reliability. Consequently, the experiments reveal a potential problem but do not convincingly establish a generalizable solution or sufficiently broad methodological insight.
Questions for authors
-
The grader replacement experiments reveal substantial score variability on NicheLedger, but much smaller effects on BixBench. Can the authors explain what properties of the benchmark or grading setup account for this difference, and identify when grader variability is likely to affect model comparisons rather than only absolute scores?
-
Beyond documenting grader instability, what specific methodological contribution does this work offer for improving evaluation reliability? Can the authors demonstrate a grading protocol that is more consistent across grader versions, better aligned with expert judgments, and more robust in comparative evaluations?
-
Can the authors disentangle semantic disagreement from technical failures in the grading pipeline? The paper reports that one grader configuration frequently fails to return a parseable verdict, and such responses are counted as incorrect. Since only aggregate parsing statistics are retained, it is unclear how much of the observed score variation reflects differences in semantic judgment versus output-format failures. How would the conclusions change if these failures were handled separately?
-
How robust are the findings given the limited expert validation? The reference-replacement analysis relies on 40 niche-naming items and consensus from three clinicians. Since expert consensus is not equivalent to biological ground truth, can the authors establish whether the observed effects generalize across a broader set of tasks, annotators, and scientific domains?
-
How much of the reported comparative evidence is independent? Six of the seven evaluated systems derive from the same 9B model, and the external validation on BixBench shows much smaller score changes and stable model rankings. Can the authors demonstrate that their conclusions hold across more diverse agent architectures and benchmarks, rather than primarily reflecting issues specific to NicheLedger and its evaluation pipeline?
4 comments
Blinded Three-Clinician Readout Results for Reviewer n8j3
The independent three-system hand review is complete, and new Appendix R (Tables R.1 and R.2) reports physicians A and B under the prespecified both-yes primary rule alongside a third independent physician, C, as a supplementary individual rater. All three graded the stored answers of the frontier model (GPT-5.5) and the fine-tuned and base Qwen3.5-9B single-pass arms without thinking across the 36 consensus items/slides under both references, with the 216 conditions shown as 156 deduplicated blind rows per physician. None constructed either reference or saw other ratings, and all worked without discussion, AI tools or web search. C used B's row order and neither adjudicated nor replaced the primary ratings.
Under the prespecified A+B rule, the frontier's lead over the fine-tuned 9B arm is +8.3 points [0.0, 19.4] with the original reference (11/36 versus 8/36) and +44.4 [27.8, 61.1] with consensus (31/36 versus 15/36), a change of +36.1 [19.4, 52.8] matching the stored grader's +36.1 (+11.1 to +47.2). Under C, the frontier's lead is +38.9 [22.2, 55.6] under each reference (24/36 versus 10/36 and 31/36 versus 17/36), giving a change of 0.0 [−19.4, 19.4]. The fine-tuned minus base 9B gap changes by +2.8 points [0.0, 8.3] under A+B (7/36 and 13/36 for base 9B) and by 0.0 [−8.3, 8.3] under C (8/36 and 15/36 for base 9B).
A and B each gave 69 yes, 27 no and 60 unsure judgments, agreeing on all 156 displayed pairs, while C gave 85 yes, 34 no and 37 unsure judgments and agreed with each on 122/156 pairs. Unsure responses stay in the denominator of 36, and the earlier frontier-only hand study in Appendix E remains separately reported. The exploratory 95% intervals use 10,000 paired slide resamples conditional on fixed judgments rather than estimating variation across the physician population.
The prespecified A+B primary supports widening of the frontier's lead, whereas individual physician ratings show that widening is rater-dependent. The introduction, Section 4.4 and Section 6 now state this dependence directly. This study adds cross-system expert evidence on one 36-item naming task without establishing a generally reliable grading protocol, histological ground truth or cross-domain robustness, and six of the seven benchmarked systems still share the Qwen3.5-9B backbone.
Thank you for the detailed response and the additional experiments. I think some of my concerns have been addressed, especially regarding human evaluation.
However, my main concern is still about the novelty of this work. I agree that reference quality and grader instability are important problems, but I am not convinced that the proposed audit procedure provides enough new methodological insights. The results on BixBench also suggest that the findings may be quite benchmark-specific.
Overall, I appreciate the authors' efforts, but I don't think the rebuttal has fully addressed my concerns about the contribution of this paper. I will keep my original score.
Thank you for reviewing our three-clinician study and additional analyses, and for noting that the human-evaluation concerns were addressed. We appreciate your candor and briefly share how we view the core contribution and the BixBench contrast (in the 15 November revised PDF via the PDF icon beside the title, changes in blue).
1. Separating instrument sensitivity from agent capability. Rather than proposing a new automated judge, NicheLedger provides a lightweight, reusable diagnostic (D1–D4, Section 3 and Appendix A): holding agent outputs fixed while replacing the reference or the grader one at a time separates score-level shifts, ranking changes, and paired gap distortions that standard single-number leaderboards bundle into "agent reasoning." In that light, the contrast with BixBench is informative rather than a negative result. An audit should distinguish fragile setups from stable pooled leaderboards: on BixBench, our 24 jointly adopted reference replacements move overall accuracy by at most 1.4 percentage points at Table L.2's precision (all intervals including zero), while separate grader replacement spans 4.5 points on Claude yet preserves overall ordering (Table K.1, alongside near-complete 0/1 parsing versus lower flash parse completion in NicheLedger, Table G.1). Even though task openness, reference provenance, prompts, and parsing vary together across the two benchmarks, running the identical fixed-answer replacement check tells practitioners immediately whether a headline gap rests on a stable or fragile instrument.
2. Checking claims at the granularity where they are used. A post hoc capsule breakdown of those same 24 BixBench replacements illustrates why releasing fixed outputs and replacement checks still matters when a benchmark mean barely moves: beneath the <= 1.4-point overall shift, individual capsule scores swing from −38 to +15 percentage points (with one or two descriptive within-capsule lead flips per cell) that cancel in the average. Likewise, on the 36 NicheLedger slides (Appendix R), widening of the frontier-minus-fine-tuned gap is substantial under our prespecified two-physician rule (+36.1 points) yet absent under a third physician (0.0), making rater dependence directly inspectable. Releasing fixed agent outputs, raw judge responses, parse rates, and one-at-a-time replacement checks gives benchmark authors a practical safeguard before attributing score gaps to model capability, while broader task and backbone families remain open for future work. Thank you again for your constructive engagement throughout the review process.
Author Response to Reviewer n8j3
Thank you for the review. Our contribution is a controlled audit procedure, not a new grader or general remedy for instability. Replacing the reference or grader one at a time on fixed answers, alongside derangement and four checks (D1–D4, Section 3, Appendix A), separates score levels, rankings, test separations and direct paired gap changes while releasing fixed answers, verdicts, parse counts and served-model fields for re-testing (Appendices K, P, Q).
Reliable grading protocol. We cannot offer a protocol validated jointly for version stability, expert alignment and comparative robustness (Section 5.3, Appendix G). On the 45-pair conditional human-agreement sample, majority voting reaches κ=0.69–0.81 versus 0.951 for the best single grader. The open trio changes 16/400 verdicts across runs (Table G.4). Pinned 9B/14B/32B graders fail prespecified checks. The 14B/32B reruns change 13–35/400 temperature-zero verdicts, with hardware and serving version also changing for unquantized 32B (Table G.3). Forward grading reaches κ=0.951 against 0.687 for bidirectional AND, whose seven misses are hypernym pairs (Table G.5). Separately, bidirectional grading inflates the human-machine gap by about six points on 40 naming items (Table G.2). Neither result validates cross-version or cross-domain robustness.
Why NicheLedger and BixBench differ. Task openness, reference provenance, prompt and output resolution vary together and are not isolated factorially (Section 6). Niche naming grades synonyms and granularity shifts against uncurated clustering strings, whereas BixBench's
0/1parser reaches 98.8%–100.0% parse completion under shared graders (Table K.1), versus 76%–79% fordeepseek-flashin Table G.1 and 53% in pooled diagnostic calls (Section 5.1).A stricter grader can change score levels without changing rankings. In NicheLedger, D4's penalty varies from +7.5/+8.8 points to +3.8 (p=0.25) under flash, and strict domination (b=0) and exact-zero claims change outcome (Table 2). These are within-grader claim tests, not cross-grader interaction tests, and do not isolate parsing from semantics. In new Appendix Q.1, the flash-minus-chat change in BixBench's Claude-minus-GPT-4o gap is −0.65 [−2.11, +0.68] and −0.35 [−1.72, +0.99] points with images and −0.32 [−1.91, +1.06] and −0.42 [−1.60, +0.64] without images (10,000 paired capsule resamples of the 296-question/53-capsule intake, missing released answers counted wrong, distinct from the 278-question/50-capsule test in Appendix K). Every interval includes zero, neither resolving a gap shift nor proving grader equivalence.
Parsing and semantics. NicheLedger's 47.5% versus 20.0% result combines grading and parsing because only aggregate parse counts remain, and the pooled 53% diagnostic rate is not that comparison's rate (Section 5.1). Dropping unparsed calls there would condition coverage on
flashformatting without recovering a semantic-only difference. On BixBench (Appendix Q.1, Table Q.1), 63/82 GPT-4o and 78/106 Claude first-pass image disagreements have both replies parsed, and net chat-minus-flash differences split as 43+19=62 and 50+28=78 (missing released answers lie outside Table Q.1). Restricting to both-parsed rows leaves those 63 and 78 external disagreements intact without recovering NicheLedger's historical mechanism.Expert support and independent coverage. Completed three-system hand grading on 36 slides (Section 4.4, Appendix R and the accompanying results comment) gives a frontier minus fine-tuned 9B single-pass no-thinking gap of +8.3 points [0.0, 19.4] on original references and +44.4 [27.8, 61.1] on consensus (+36.1 change [19.4, 52.8]) under prespecified A+B both-yes, while supplementary physician C gives +38.9 [22.2, 55.6] on each and a 0.0 change [−19.4, 19.4] (10,000 paired slide resamples conditional on fixed ratings, unsure in denominator, A/B agreeing on 156/156 pairs and C with each on 122/156). Widening is rater-dependent, and hand grading tests answer-reference equivalence rather than biological ground truth. Separately, PanglaoDB favors consensus markers on all five mappable class-1a items (Appendix F). Post-hoc
gpt-6.1-solgrading (Appendix Q.2) has session-sensitive widening (2.8 and 16.7 points) and cannot replace physicians.Six of seven NicheLedger arms share Qwen3.5-9B, leaving GPT-5.5 versus Qwen3.5-9B and BixBench's Claude versus GPT-4o as the cross-family pairs (Section 6). On BixBench (165 questions/29 capsules, 159/28 with images, Appendix L, Table L.2), the 24 physician-adopted replacements from selected longer proposals move accuracy by at most 1.4 points and narrow Claude's lead by 1.0–1.3 points, all intervals including zero. The audit procedure transfers, while effect sizes and comparative shifts remain setting-specific.