acceptodds
Under review as a conference paper at ICLR 2027

NicheLedger: Replacing References and Graders on Fixed Answers of Spatial Transcriptomics Agents

Abstract

Some scientific-agent benchmarks grade open-ended answers with a language model against a fixed reference. Replacing either can move a score on unchanged answers. Using NicheLedger, a spatial-transcriptomics agent benchmark, we replace each on fixed answers and track levels, orderings and test outcomes. On 36 niche-naming items under one grader, a frontier model's answers score 27.8% against clustering-derived references and 77.8% against clinician-consensus references, two physicians' hand grading gives 30.6% and 86.1%, and item-derangement controls score at most 8.3%. Of 18 gained items, seven clustering-derived references name a cell type no clinician named, three name no niche, and eight are wording, granularity or unresolved cases. One of 21 system-pair orderings reverses. The same model's 40 niche-naming answers score 30.0%, 47.5% and 20.0% under three grader states, two reached by names reporting one model, counting unparsable replies incorrect. Three claims each change outcome under some state. On BixBench, the ordering survives three graders and model-proposed references that two physicians prefer, which move accuracy by up to 4.5 and 1.4 points respectively. References and graders thus move levels and test outcomes while most orderings hold, and clinician consensus is not biological ground truth.

Primary area:
datasets and benchmarks
Keywords:
benchmark evaluation, reference answers, LLM-as-a-judge, grader drift, measurement validity, scientific agents, spatial transcriptomics, human annotation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_n8j3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the reliability of evaluating scientific agents using reference-based LLM grading. The authors introduce NicheLedger, a spatial transcriptomics agent benchmark, and audit evaluation outcomes by independently replacing reference answers and graders while keeping model responses fixed. Their experiments show that both reference quality and grader choice can substantially affect evaluation scores and statistical conclusions. Similar audits on BixBench, however, reveal smaller effects and stable model rankings. The paper highlights important issues in benchmark reliability, but its primary contribution is empirical auditing rather than a novel evaluation methodology.

Critical strengths
  1. By replacing reference answers and graders independently while keeping model responses fixed, the authors provide a straightforward way to isolate how these evaluation components affect benchmark outcomes. This controlled design makes the observed score changes interpretable.

  2. The authors investigate both reference quality and grader variability in spatial transcriptomics and extend the analysis to BixBench. The results provide useful evidence that evaluation reliability should be examined beyond aggregate scores, even though the effects are not equally pronounced across benchmarks.

Critical weaknesses
  1. The finding that automatically generated references can be inaccurate is relatively unsurprising, and the paper does not establish a substantially new method for addressing this problem. The more interesting contribution is the sensitivity of evaluation outcomes to grader changes, but the work primarily documents this issue rather than developing a principled approach to making evaluations more reliable. This limits the methodological contribution beyond an empirical audit of existing benchmarks.

  2. The grader effects are substantial on NicheLedger but much smaller on BixBench, where model rankings remain stable. The paper does not sufficiently explain when grader variability meaningfully undermines model comparisons or establish a validated evaluation protocol that improves grading consistency and reliability. Consequently, the experiments reveal a potential problem but do not convincingly establish a generalizable solution or sufficiently broad methodological insight.

Questions for authors
  1. The grader replacement experiments reveal substantial score variability on NicheLedger, but much smaller effects on BixBench. Can the authors explain what properties of the benchmark or grading setup account for this difference, and identify when grader variability is likely to affect model comparisons rather than only absolute scores?

  2. Beyond documenting grader instability, what specific methodological contribution does this work offer for improving evaluation reliability? Can the authors demonstrate a grading protocol that is more consistent across grader versions, better aligned with expert judgments, and more robust in comparative evaluations?

  3. Can the authors disentangle semantic disagreement from technical failures in the grading pipeline? The paper reports that one grader configuration frequently fails to return a parseable verdict, and such responses are counted as incorrect. Since only aggregate parsing statistics are retained, it is unclear how much of the observed score variation reflects differences in semantic judgment versus output-format failures. How would the conclusions change if these failures were handled separately?

  4. How robust are the findings given the limited expert validation? The reference-replacement analysis relies on 40 niche-naming items and consensus from three clinicians. Since expert consensus is not equivalent to biological ground truth, can the authors establish whether the observed effects generalize across a broader set of tasks, annotators, and scientific domains?

  5. How much of the reported comparative evidence is independent? Six of the seven evaluated systems derive from the same 9B model, and the external validation on BixBench shows much smaller score changes and stable model rankings. Can the authors demonstrate that their conclusions hold across more diverse agent architectures and benchmarks, rather than primarily reflecting issues specific to NicheLedger and its evaluation pipeline?

4 comments
AuthorsOpenReview ↗

Author Response to Reviewer n8j3

Thank you for the review. Our contribution is a controlled audit procedure, not a new grader or general remedy for instability. Replacing the reference or grader one at a time on fixed answers, alongside derangement and four checks (D1–D4, Section 3, Appendix A), separates score levels, rankings, test separations and direct paired gap changes while releasing fixed answers, verdicts, parse counts and served-model fields for re-testing (Appendices K, P, Q).

Reliable grading protocol. We cannot offer a protocol validated jointly for version stability, expert alignment and comparative robustness (Section 5.3, Appendix G). On the 45-pair conditional human-agreement sample, majority voting reaches κ=0.69–0.81 versus 0.951 for the best single grader. The open trio changes 16/400 verdicts across runs (Table G.4). Pinned 9B/14B/32B graders fail prespecified checks. The 14B/32B reruns change 13–35/400 temperature-zero verdicts, with hardware and serving version also changing for unquantized 32B (Table G.3). Forward grading reaches κ=0.951 against 0.687 for bidirectional AND, whose seven misses are hypernym pairs (Table G.5). Separately, bidirectional grading inflates the human-machine gap by about six points on 40 naming items (Table G.2). Neither result validates cross-version or cross-domain robustness.

Why NicheLedger and BixBench differ. Task openness, reference provenance, prompt and output resolution vary together and are not isolated factorially (Section 6). Niche naming grades synonyms and granularity shifts against uncurated clustering strings, whereas BixBench's 0/1 parser reaches 98.8%–100.0% parse completion under shared graders (Table K.1), versus 76%–79% for deepseek-flash in Table G.1 and 53% in pooled diagnostic calls (Section 5.1).

A stricter grader can change score levels without changing rankings. In NicheLedger, D4's penalty varies from +7.5/+8.8 points to +3.8 (p=0.25) under flash, and strict domination (b=0) and exact-zero claims change outcome (Table 2). These are within-grader claim tests, not cross-grader interaction tests, and do not isolate parsing from semantics. In new Appendix Q.1, the flash-minus-chat change in BixBench's Claude-minus-GPT-4o gap is −0.65 [−2.11, +0.68] and −0.35 [−1.72, +0.99] points with images and −0.32 [−1.91, +1.06] and −0.42 [−1.60, +0.64] without images (10,000 paired capsule resamples of the 296-question/53-capsule intake, missing released answers counted wrong, distinct from the 278-question/50-capsule test in Appendix K). Every interval includes zero, neither resolving a gap shift nor proving grader equivalence.

Parsing and semantics. NicheLedger's 47.5% versus 20.0% result combines grading and parsing because only aggregate parse counts remain, and the pooled 53% diagnostic rate is not that comparison's rate (Section 5.1). Dropping unparsed calls there would condition coverage on flash formatting without recovering a semantic-only difference. On BixBench (Appendix Q.1, Table Q.1), 63/82 GPT-4o and 78/106 Claude first-pass image disagreements have both replies parsed, and net chat-minus-flash differences split as 43+19=62 and 50+28=78 (missing released answers lie outside Table Q.1). Restricting to both-parsed rows leaves those 63 and 78 external disagreements intact without recovering NicheLedger's historical mechanism.

Expert support and independent coverage. Completed three-system hand grading on 36 slides (Section 4.4, Appendix R and the accompanying results comment) gives a frontier minus fine-tuned 9B single-pass no-thinking gap of +8.3 points [0.0, 19.4] on original references and +44.4 [27.8, 61.1] on consensus (+36.1 change [19.4, 52.8]) under prespecified A+B both-yes, while supplementary physician C gives +38.9 [22.2, 55.6] on each and a 0.0 change [−19.4, 19.4] (10,000 paired slide resamples conditional on fixed ratings, unsure in denominator, A/B agreeing on 156/156 pairs and C with each on 122/156). Widening is rater-dependent, and hand grading tests answer-reference equivalence rather than biological ground truth. Separately, PanglaoDB favors consensus markers on all five mappable class-1a items (Appendix F). Post-hoc gpt-6.1-sol grading (Appendix Q.2) has session-sensitive widening (2.8 and 16.7 points) and cannot replace physicians.

Six of seven NicheLedger arms share Qwen3.5-9B, leaving GPT-5.5 versus Qwen3.5-9B and BixBench's Claude versus GPT-4o as the cross-family pairs (Section 6). On BixBench (165 questions/29 capsules, 159/28 with images, Appendix L, Table L.2), the 24 physician-adopted replacements from selected longer proposals move accuracy by at most 1.4 points and narrow Claude's lead by 1.0–1.3 points, all intervals including zero. The audit procedure transfers, while effect sizes and comparative shifts remain setting-specific.

AuthorsOpenReview ↗

Blinded Three-Clinician Readout Results for Reviewer n8j3

The independent three-system hand review is complete, and new Appendix R (Tables R.1 and R.2) reports physicians A and B under the prespecified both-yes primary rule alongside a third independent physician, C, as a supplementary individual rater. All three graded the stored answers of the frontier model (GPT-5.5) and the fine-tuned and base Qwen3.5-9B single-pass arms without thinking across the 36 consensus items/slides under both references, with the 216 conditions shown as 156 deduplicated blind rows per physician. None constructed either reference or saw other ratings, and all worked without discussion, AI tools or web search. C used B's row order and neither adjudicated nor replaced the primary ratings.

Under the prespecified A+B rule, the frontier's lead over the fine-tuned 9B arm is +8.3 points [0.0, 19.4] with the original reference (11/36 versus 8/36) and +44.4 [27.8, 61.1] with consensus (31/36 versus 15/36), a change of +36.1 [19.4, 52.8] matching the stored grader's +36.1 (+11.1 to +47.2). Under C, the frontier's lead is +38.9 [22.2, 55.6] under each reference (24/36 versus 10/36 and 31/36 versus 17/36), giving a change of 0.0 [−19.4, 19.4]. The fine-tuned minus base 9B gap changes by +2.8 points [0.0, 8.3] under A+B (7/36 and 13/36 for base 9B) and by 0.0 [−8.3, 8.3] under C (8/36 and 15/36 for base 9B).

A and B each gave 69 yes, 27 no and 60 unsure judgments, agreeing on all 156 displayed pairs, while C gave 85 yes, 34 no and 37 unsure judgments and agreed with each on 122/156 pairs. Unsure responses stay in the denominator of 36, and the earlier frontier-only hand study in Appendix E remains separately reported. The exploratory 95% intervals use 10,000 paired slide resamples conditional on fixed judgments rather than estimating variation across the physician population.

The prespecified A+B primary supports widening of the frontier's lead, whereas individual physician ratings show that widening is rater-dependent. The introduction, Section 4.4 and Section 6 now state this dependence directly. This study adds cross-system expert evidence on one 36-item naming task without establishing a generally reliable grading protocol, histological ground truth or cross-domain robustness, and six of the seven benchmarked systems still share the Qwen3.5-9B backbone.

Thank you for the detailed response and the additional experiments. I think some of my concerns have been addressed, especially regarding human evaluation.

However, my main concern is still about the novelty of this work. I agree that reference quality and grader instability are important problems, but I am not convinced that the proposed audit procedure provides enough new methodological insights. The results on BixBench also suggest that the findings may be quite benchmark-specific.

Overall, I appreciate the authors' efforts, but I don't think the rebuttal has fully addressed my concerns about the contribution of this paper. I will keep my original score.

Thank you for reviewing our three-clinician study and additional analyses, and for noting that the human-evaluation concerns were addressed. We appreciate your candor and briefly share how we view the core contribution and the BixBench contrast (in the 15 November revised PDF via the PDF icon beside the title, changes in blue).

1. Separating instrument sensitivity from agent capability. Rather than proposing a new automated judge, NicheLedger provides a lightweight, reusable diagnostic (D1–D4, Section 3 and Appendix A): holding agent outputs fixed while replacing the reference or the grader one at a time separates score-level shifts, ranking changes, and paired gap distortions that standard single-number leaderboards bundle into "agent reasoning." In that light, the contrast with BixBench is informative rather than a negative result. An audit should distinguish fragile setups from stable pooled leaderboards: on BixBench, our 24 jointly adopted reference replacements move overall accuracy by at most 1.4 percentage points at Table L.2's precision (all intervals including zero), while separate grader replacement spans 4.5 points on Claude yet preserves overall ordering (Table K.1, alongside near-complete 0/1 parsing versus lower flash parse completion in NicheLedger, Table G.1). Even though task openness, reference provenance, prompts, and parsing vary together across the two benchmarks, running the identical fixed-answer replacement check tells practitioners immediately whether a headline gap rests on a stable or fragile instrument.

2. Checking claims at the granularity where they are used. A post hoc capsule breakdown of those same 24 BixBench replacements illustrates why releasing fixed outputs and replacement checks still matters when a benchmark mean barely moves: beneath the <= 1.4-point overall shift, individual capsule scores swing from −38 to +15 percentage points (with one or two descriptive within-capsule lead flips per cell) that cancel in the average. Likewise, on the 36 NicheLedger slides (Appendix R), widening of the frontier-minus-fine-tuned gap is substantial under our prespecified two-physician rule (+36.1 points) yet absent under a third physician (0.0), making rater dependence directly inspectable. Releasing fixed agent outputs, raw judge responses, parse rates, and one-at-a-time replacement checks gives benchmark authors a practical safeguard before attributing score gaps to model capability, while broader task and backbone families remain open for future work. Thank you again for your constructive engagement throughout the review process.

Reviewer_zscpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper measures how much scientific-agent evaluation depends on references and graders while holding system answers fixed. Using NicheLedger, a spatial-transcriptomics agent benchmark, the authors replace clustering-derived references with blinded-clinician consensus references and re-grade fixed answers under several grader states. Reference replacement moves scores substantially for a frontier model on the covered naming items, with independent physician grading supporting the direction of the change and a derangement control indicating the effect is not mere wording familiarity. Most system orderings remain stable, though score gaps and some comparative test outcomes change. The audit procedure transfers to an external benchmark, where reference effects are smaller and ordering is preserved. The authors caution that clinician consensus is not biological ground truth and that effect sizes are benchmark-specific.

Critical strengths
  1. The fixed-answer replacement design cleanly isolates changes in the evaluation instrument from changes in the generated answers: only one input is replaced at a time while answers remain fixed, and the instrument itself is probed for potential influence on reported values. Independent physician grading corroborates the main reference-replacement effect in direction, strengthening the evidence beyond a single LLM grader.
  2. The paper carefully separates score levels, system orderings, score gaps, and statistical test outcomes, and reports negative results, external validation, and substantial limitations—including the distinction between clinician consensus and biological ground truth. Its reproducibility design makes the audit informative and its conclusions easy to assess.
Critical weaknesses
  1. The demonstrated contribution beyond a benchmark-specific audit remains limited. The largest effects arise on a newly constructed benchmark, on items where reference quality and coordinate–target correspondence are questionable, and most evaluated systems derive from the same backbone. Part of the repaired benchmark is only partially evaluated and lacks demonstrated spatial validity. On the external benchmark, physician-supported reference replacements move individual system scores only slightly without reversing the ordering. These findings are useful, but the current evidence is insufficient to establish broader significance beyond existing observations about reference and judge sensitivity.
  2. The central grading results are not exactly reproducible: the hosted grader is reached by name, and that name no longer returns the model that produced the published numbers, with no version-pinned identifier available.
  3. The main grader-sensitivity result does not separate semantic disagreement from output-parsing failure. Verdict-parsing rates differ markedly across aliases, and unparsable replies are counted as incorrect, but only coarse parsing records were kept, so the largest score difference cannot be decomposed. The authors acknowledge this, and the pipeline sensitivity is real; its mechanism and implications remain unresolved.
  4. The reported agreement with human adjudicators is conditional on a subset of pairs both adjudicators could decide and agreed on, and does not establish reliability across all evaluated answers or grader states.
Questions for authors
  1. On the fixed NicheLedger answers re-graded in the same period, when both grader aliases return resolvable verdicts, can you separately report resolution failures and inconsistent verdicts, and the remaining score difference after accounting for resolution behavior?
  2. Does the system score gap persist under independent hand grading on a stratified sample spanning multiple systems and both reference conditions? The current physician grading mainly corroborates the reference effect for the frontier model.
  3. Is the repaired release primarily intended as a cluster-label audit resource or as a spatial-capability benchmark? For its intended use, what verification would establish the correspondence between query coordinates, provided evidence, and reference targets?
4 comments
AuthorsOpenReview ↗

Author Response to Reviewer zscp

Thank you for these questions. Archived NicheLedger logs cannot split the 27.5-point grader shift, whereas retained BixBench flags separate both-parsed disagreements (Appendix Q.1). Completed three-system hand grading (Appendix R) gives a positive point estimate of the frontier's lead across references and raters, while gap widening depends on the physician. The repaired release targets cluster-label auditing, not spatial capability.

Semantic disagreement versus resolution failure. We cannot recover that split for NicheLedger's 27.5-point difference (47.5% versus 20.0% on 40 naming answers, Section 5.1). That run kept per-item verdicts and aggregate parse counts without per-call replies or parse flags, and the 53% rate is a pooled diagnostic figure rather than that comparison's rate. The 27.5-point shift therefore measures the combined grading-and-parsing pipeline, with no recoverable both-resolved subset.

Offline BixBench decomposition (Appendix Q.1, Table Q.1) separates rows with retained flags. In the first image pass, GPT-4o has 2,181 both-parsed and 20 chat-only parsed rows of 2,201, and Claude has 2,154 and 35 of 2,189, with no flash-only or neither-parsed rows. The 729/629 missing released answer rows lie outside Table Q.1 and differ from unparsed grader replies. Of 82 and 106 verdict disagreements, 63 and 78 occur with both replies parsed, and net chat-minus-flash correct-count differences split into both-parsed plus unparsed-involving rows as 43+19=62 and 50+28=78. Parsing thus does not explain all external disagreement, without identifying expert-confirmed semantic errors or recovering NicheLedger's historical split.

Independent hand grading of system gaps. The blind three-system review is complete (Section 4.4, Section 6, Appendix R and the accompanying results comment). Three independent physicians graded stored answers of GPT-5.5 and the fine-tuned and base Qwen3.5-9B single-pass no-thinking arms on all 36 consensus slides under both references (216 conditions as 156 deduplicated rows, unsure kept in the denominator of 36). Under prespecified A+B both-yes, the frontier minus fine-tuned 9B gap is +8.3 points [0.0, 19.4] on original references and +44.4 [27.8, 61.1] on consensus, a +36.1 change [19.4, 52.8] matching the stored grader's +36.1, whereas supplementary physician C gives +38.9 [22.2, 55.6] on each, a 0.0 change [−19.4, 19.4] (10,000 paired slide resamples conditional on fixed judgments). A and B agree on 156/156 pairs and C with each on 122/156, so the frontier's point-estimate lead stays positive while widening is rater-dependent. The original-reference A+B interval includes zero. In the separate 100-pair sample (Table G.5), both clinicians gave yes/no on 46 and agreed on 45, so κ=0.951 remains conditional.

In a post-hoc non-human check (Appendix Q.2, Table Q.2), cross-family gpt-6.1-sol judged the same two shuffled forms in separate sessions. Under both-yes, the frontier minus fine-tuned 9B gap is +33.3 points [19.4, 50.0] on original references and +50.0 [33.3, 66.7] on consensus, widening by +16.7 [2.8, 30.6] versus 36.1 for the stored grader. That widening varies by session (2.8 versus 16.7 points, intervals conditional on fixed judgments). It credits 23/36 original-reference frontier pairs against 11/36 for physicians A and B (C credits 24/36), and its sessions share one model and agree on 149/156 rows, so it cannot replace hand grading.

Intended use and spatial validity. The repaired release supports cluster-label auditing (Section 3, Appendix E). The example context in Appendix I tags the queried cluster and supplies its five markers. Clinicians assigned labels from those markers (Section 3), and derangement tests item-label correspondence (Table 1). Eleven of 40 query coordinates lie more than 1.5 spot spacings from a spot, and both references describe clusters rather than locations, so label replacement does not establish spatial validity. Stored predictions cover 15 of 22 filtered spatial items and separate none of 21 arm pairs (Table E.3). Spatial benchmarking would require unperformed item-level checks of spot registration, section membership, crop coverage and validated spatial targets.

Historical reproducibility. The hosted alias gives no version-pinned identifier, and a matching served-model field does not guarantee identical weights or routing (Section 5.1, Appendix P). Retained verdicts recompute published figures without regenerating historical calls. The 24 BixBench physician replacements move accuracy by at most 1.4 points with intervals including zero (Appendix L), so neither study claims broad cross-domain invariance.

AuthorsOpenReview ↗

Blinded Three-Clinician Readout Results for Reviewer zscp

Thank you for requesting independent hand grading across multiple systems and both reference conditions. That study is complete, and new Appendix R (Tables R.1 and R.2) reports all three physicians. Every physician's point estimate preserves the frontier model's lead under both references, while gap widening holds under the prespecified two-physician primary rule and is unresolved under the third physician. The introduction, Section 4.4 and Section 6 now state that gap widening depends on the rater.

Physicians A and B independently graded the stored answers of the frontier model (GPT-5.5), the fine-tuned Qwen3.5-9B arm in a single pass without thinking, and the base Qwen3.5-9B arm under the same configuration. The study covers all 36 consensus items, each from a distinct slide, under both the original and consensus references. Those 216 conditions appeared as 156 deduplicated rows per physician in separately shuffled forms, with system identity, reference source and stored verdicts withheld. Neither physician constructed either reference. Under the primary rule fixed before any form returned, an item is credited only when A and B both answer yes, and unsure responses remain in the denominator of 36.

A third independent physician, C, completed the same 156 rows in B's presentation order without seeing other ratings or constructing either reference. All three physicians worked without discussion, AI tools or web search. C is reported separately as a supplementary individual rater rather than an adjudicator, replacement rater or majority-vote tie-breaker.

Frontier minus fine-tuned 9B, percentage pointsOriginalConsensusChange
A+B, prespecified primary+8.3 [0.0, 19.4]+44.4 [27.8, 61.1]+36.1 [19.4, 52.8]
C, supplementary+38.9 [22.2, 55.6]+38.9 [22.2, 55.6]0.0 [−19.4, 19.4]
Stored grader+11.1+47.2+36.1

The exploratory 95% intervals use 10,000 paired resamples of the 36 slides conditional on fixed judgments, so they do not estimate variation across the physician population. For the fine-tuned minus base 9B contrast, the gap changes by +2.8 points [0.0, 8.3] under A+B and by 0.0 [−8.3, 8.3] under C (compared with +2.8 under the stored grader). No majority-vote rule replaces the prespecified primary.

A and B each recorded 69 yes, 27 no and 60 unsure judgments, agreeing on all 156 displayed pairs. C recorded 85 yes, 34 no and 37 unsure judgments, agreeing with A and B on 122/156 pairs each. Out of 36 items under original and consensus references, A+B credit 11 and 31 for the frontier, 8 and 15 for the fine-tuned 9B, and 7 and 13 for the base 9B. Physician C credits 24 and 31, 10 and 17, and 8 and 15, while the stored forward grader credits 10 and 28, 6 and 11, and 5 and 9. Appendix R reports each physician separately and the either-yes sensitivity, which coincides with A+B, while the earlier frontier-only hand study in Appendix E remains separately reported.

The frontier's point-estimate lead over the fine-tuned 9B arm remains positive under both references for all three physicians, whereas widening of that lead holds under the prespecified A+B rule (+36.1 points [19.4, 52.8]) and is not consistent across individual physicians. The study spans 36 items of one naming task, two of the three systems share the Qwen3.5-9B backbone, and clustering strings can reveal reference provenance. Because the physicians judged answer-reference equivalence rather than histological ground truth, these results neither recover the historical 27.5-point parsing-versus-semantics split nor establish general cross-domain reliability.

Thank you for your detailed responses. I have increased my scores for Soundness and Informativeness in light of your clarifications. Nevertheless, given the paper’s remaining substantial weaknesses, I will maintain my original overall rating.

Thank you for engaging so carefully with our responses and for raising your scores on Soundness and Informativeness. We appreciate your candor and respect your overall assessment.

For completeness on the record, we briefly note one follow-up finding we added in response to Reviewer yzAm after our earlier reply to you: breaking down the 24 jointly adopted BixBench reference replacements by capsule (Table L.2) shows that the small benchmark-wide shift (at most 1.4 percentage points) reflects cancellation between 17 to 21 jointly unchanged capsules and 8 to 11 changed capsules whose scores swing from −38.0 to +14.8 percentage points, with one or two descriptive within-capsule lead reversals per cell. While broader task types and model backbones remain outside our current study, this study-level heterogeneity illustrates why releasing fixed agent outputs and paired replacement checks adds diagnostic value even when pooled leaderboard rankings stay intact. Thank you again for your thoughtful review.

Reviewer_yzAmOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper audits how reference answers and LLM graders affect scientific-agent evaluation. Holding candidate answers fixed, it replaces either the references or the grader and examines changes in scores and comparative conclusions on NicheLedger and BixBench.

Critical strengths
  • Controlled audit design: Holding candidate answers fixed while changing one evaluation input at a time helps attribute score changes to the references or the grader.

  • Multiple checks on the reference effect: Clinician annotations, independent physician grading, mismatched-reference controls, and marker-registry comparisons provide complementary evidence supporting the observed reference effect.

Critical weaknesses
  • Limited external evidence for large effects: The external experiments support the audit procedure, but the most consequential reference effects remain concentrated in the authors’ own benchmark. This limits the breadth of the empirical contribution.

  • Narrow task and model coverage: The main reference audit covers 36 niche-naming items, while six of the seven evaluated systems derive from the same 9B backbone. The evidence therefore remains concentrated in a narrow task setting and a limited range of model families.

Questions for authors

Could the authors extend the reference audit to more test items and task types, and report how reference disagreements and the resulting score changes vary across tasks or BixBench capsules?

6 comments
AuthorsOpenReview ↗

Author Response to Reviewer yzAm

Thank you for the review. Our fixed-answer audit transfers to BixBench, but the reference effect differs sharply between benchmarks. It is large on NicheLedger's naming task and small for the physician-supported BixBench replacements, whose 95% intervals all include zero. Revised Appendix Q.1 adds a BixBench capsule grader breakdown, Appendix R adds three-system hand grading, and broader coverage remains open.

The NicheLedger reference effect. Starting from 40 blindly annotated T1 items, 36 reach agreement from at least two clinicians and enter replacement (Sections 3 and 4.2, Table 1). On those 36 fixed frontier answers, the LLM grader credits 10/36 (27.8%) against clustering references and 28/36 (77.8%) against consensus. Under the primary both-yes rule, which keeps unsure responses in the denominator without credit, two independent physicians who constructed neither reference credit 11/36 and 31/36 (Appendix E), corroborating the frontier gain without establishing biological ground truth or cross-system gaps. In the completed three-system hand review on the same 36 slides (Section 4.4, Appendix R), the frontier minus fine-tuned 9B single-pass no-thinking gap changes by +36.1 points [19.4, 52.8] under prespecified A+B and by 0.0 [−19.4, 19.4] under a third physician, so widening is rater-dependent. Elsewhere in NicheLedger, 25 of 65 held-out T1 items are unlabeled, T2 (52 items) is a retrieval control, and filtered T3 covers 15 scored items separating no arm pair (Tables E.1 and E.3).

The external reference audit. Appendix L covers 165 BixBench questions across 29 capsules with archives of at most 20 MB (159 in 28 with images, Table L.2). Two blinded physicians reviewed 39 source-validated substantive disagreements plus four controls and jointly adopted 24 proposals, which move agent accuracy by at most 1.4 percentage points and narrow Claude's lead by 1.0–1.3 points, every interval including zero. This transfers the audit procedure without reproducing NicheLedger's large effect.

A third physician adjudicated the 15 contested items in the submitted study. Replacing the 33 references adopted by both primary physicians or a three-physician majority moves accuracy by at most 2.1 points and narrows the lead by 1.0–1.5 points, every interval including zero (Appendix L). Because proposals were selected for disagreement from size-bounded archives and carry longer derivations, this sample cannot estimate task-wide error prevalence.

New capsule analysis and its boundary. Appendix Q.1 breaks down BixBench grader replacement across capsules rather than adding clinician reference labels. In the first image pass, the DeepSeek aliases disagree in 23/53 intake capsules for GPT-4o and 21/53 for Claude, with positive, negative and zero net chat-minus-flash capsule counts of 21/2/30 and 18/2/33. In the second pass, 20/53 capsules disagree for each system.

The direct flash-minus-chat change in the Claude-minus-GPT-4o gap is −0.65 [−2.11, +0.68] and −0.35 [−1.72, +0.99] points in the two image passes, and −0.32 [−1.91, +1.06] and −0.42 [−1.60, +0.64] without images. These exploratory 95% intervals use 10,000 paired capsule resamples of the 296-question/53-capsule intake, counting missing released answers incorrect and retaining replicas within capsules. Retained image rows cover 289 questions/52 capsules for GPT-4o and 285/51 for Claude, distinct from the 278-question/50-capsule primary test in Appendix K. No interval resolves a gap change.

What remains open. Within BixBench, the 39 source-supported alternatives span 21 capsules. On the released zero-shot answers, only six distinct questions change any grader verdict and the subset margin change is +2.4 points under each grader ([−7.1, 15.1] and [−7.9, 14.3], Appendix L). We have not computed the capsule distribution of the 24 adopted replacements, collected new BixBench clinician labels or tested reference heterogeneity on new task types, and Appendix Q.1 does not fill that gap. Six of seven NicheLedger arms also share Qwen3.5-9B, so broader task and model coverage remains missing.

Thank you for the new blinded physician ratings and the capsule-level BixBench analysis, which clarify how the gap between systems varies by physician and how graders disagree across capsules.

However, the new physician ratings still cover the same 36 niche-naming items. It also remains unclear how replacing reference answers affects scores across BixBench capsules. For now, I lean toward keeping my rating at 2.

Thank you for pointing to the exact remaining question on how the 24 jointly adopted BixBench reference replacements behave across capsules. Computing that post hoc breakdown directly from our saved verdicts (with no new model calls, recovering the rounded physician-arm estimates in Table L.2 of the 15 November revised PDF via the PDF icon beside the title, changes in blue) clarifies why a 1.4-point benchmark-wide shift masks substantial variation across individual studies.

1. Capsule coverage. The 24 jointly adopted questions appear in both the no-image set (165 questions across 29 capsules) and the paired image set (159 shared questions across 28 capsules), spanning 15 capsules in each arm: ten capsules have one replacement, three have two (bix-24, bix-32, bix-36), bix-53 has 3/7, and bix-49 has 5/6. As in Table L.2, we average replica verdicts within each question, questions within each capsule, and capsules equally, counting blank answers as wrong.

2. Capsule-level shifts across the four evaluated settings. In each cell below, 8 to 11 capsules change score for at least one agent (across 11 to 15 question means), whereas 17 to 21 capsules show zero net change in both agents (including occasional offsetting replica or question changes). Values below are in percentage points, with margin defined as Claude minus GPT-4o:

Answers / graderChanged capsulesClaude + / − / 0; range (pp)GPT-4o + / − / 0; range (pp)Margin change range (pp)Strict lead flips
No image / DeepSeek82 / 3 / 24; −38.00 to +14.813 / 3 / 23; −12.00 to +6.35−26.00 to +14.812
Image / DeepSeek92 / 3 / 23; −32.00 to +11.114 / 3 / 21; −22.50 to +6.25−14.29 to +11.112
No image / Qwen81 / 4 / 24; −30.00 to +14.814 / 2 / 23; −12.00 to +6.35−18.00 to +14.812
Image / Qwen113 / 5 / 20; −20.00 to +11.115 / 2 / 21; −12.50 to +6.25−14.29 to +11.111

The strict within-capsule lead flips (bix-24/bix-36, bix-43/bix-54, bix-36/bix-54, and bix-54, excluding ties) are descriptive point-estimate reversals rather than statistically resolved local ranking changes.

3. Why opposite-sign capsule shifts cancel in the headline mean. Individual capsules move substantially in opposite directions: on bix-36 (2/5 references replaced), Claude drops by 20 to 38 percentage points across cells and GPT-4o drops by 12 to 22.5 points, shifting the capsule margin by down to −26 points; conversely, on bix-49 (5/6 replaced), Claude rises by 11.11 points with images and 14.81 without images under both graders while GPT-4o is unchanged, and capsules such as bix-54 and bix-16 (with images) modestly raise GPT-4o. Because 17 to 21 capsules are jointly unchanged and positive and negative shifts offset one another, the equal-capsule benchmark average moves by at most 1.4 points at Table L.2's one-decimal precision (all pooled intervals including zero, with Claude ahead in all four cells; the 33-reference majority arm across 19 capsules shows the same −38.00 to +14.81 range and two strict flips per cell).

In short, even when BixBench's pooled leaderboard average is stable, per-capsule evaluations of biological research agents can shift substantially under physician-preferred references. While our three-clinician hand grading covers the same 36 NicheLedger slides and these BixBench items come from disagreement-screened <= 20 MB capsules with derivation-inclusive proposals (rather than estimating task-wide error prevalence or broader task/backbone coverage), we hope this capsule-level resolution directly answers your question and supports reconsidering your provisional rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Brief Post-Revision Discussion Summary for the Area Chair

Thank you for overseeing the review of our submission. We briefly summarize the key developments from the discussion following our 15 November revision:

  1. Human evaluation and soundness updates: Reviewer zscp increased both Soundness and Informativeness after reviewing the completed three-clinician hand grading (Appendix R) and BixBench parsing decomposition (Appendix Q.1), and Reviewer n8j3 noted that the human-evaluation concerns were addressed.

  2. New capsule-level BixBench reference analysis: In direct response to Reviewer yzAm's follow-up question on the 24 jointly adopted BixBench reference replacements (Table L.2), we computed a post hoc capsule breakdown from the saved verdicts (no new model calls). While the pooled equal-capsule benchmark shift is at most 1.4 percentage points (all intervals including zero), that small average reflects cancellation between 17 to 21 jointly unchanged capsules and 8 to 11 changed capsules whose scores swing from −38.0 to +14.8 percentage points, with one or two descriptive within-capsule lead reversals per cell (and the same range in the 33-reference majority arm). This reinforces why releasing fixed outputs and paired replacement checks adds diagnostic value below headline averages, while broader task types and model families remain clearly scoped as open work.

We remain available if any further clarification would be helpful.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.