acceptodds
Under review as a conference paper at ICLR 2027

What Can a Leaderboard Certify? Compositional Controllability for Fair Evaluation and Training of Biomedical Literature-Review Agents

Abstract

Leaderboards rank long-horizon agents by their final outputs. Yet a higher score alone does not establish whether two systems are comparable or which stage accounts for the difference. Unequal evidence, inputs, or budgets can affect scores, and statistical corrections do not remove this mismatch. We introduce compositional controllability to address these questions. A comparison window covers one stage, several stages, or the whole agent. Our central result bounds the gap between observed and controlled score differences using only nuisance outside the window. This yields an admissibility test applied before scores are inspected. Inadmissible comparisons are refused. For admissible pairs, an ordering is certified only when the score gap exceeds the combined sampling and nuisance radii; otherwise, it remains undecided. These decisions give each system a rank interval. We introduce BioLitBench, a benchmark of 2,042 biomedical articles represented as structured claim graphs. We use it to compare published pipelines, commercial agents, and our own agent under a shared reference list or a shared frozen corpus. Among seven published pipelines, a conventional statistical analysis declares a winner in 14 of 21 pairwise comparisons. Yet the top-ranked system alone received the target review's bibliography, giving it access to evidence the others were not supplied. To isolate pipeline performance, our test requires matched inputs and a fixed backbone model. It rejects 11 of the 21 comparisons, including every comparison involving the top-ranked system. Seven of the 14 conventional conclusions fall within these rejected pairs. The same comparison windows support stage-level training. We train SCRIBE on a 27B open model using rewards measured at each stage's exit. Under matched evidence, SCRIBE achieves a certified rank interval of [1,2], with certified advantages over all evaluated published pipelines and the evaluated Claude and OpenAI agents. The code is available at https: //anonymous.4open.science/r/LitReview-SCRIBE-8E4E

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, agent evaluation, benchmarking, compositional analysis, literature review writing, certified evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.80 over 5 reviews, above 27% of the others.
Reviewer_ruZhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to fairly compare multi-stage agents when they differ in inputs, resources, or intermediate components. The authors propose compositional controllability, which bounds the discrepancy between observed performance differences and controlled comparisons using nuisance factors outside a selected comparison window. This framework leads to admissibility tests and certified partial rankings. The paper also introduces BioLitBench, a benchmark for biomedical literature review agents, and SCRIBE, an agent trained using stage-level rewards. Experiments show that many statistically significant comparisons between existing pipelines are not admissible under the proposed framework, and that SCRIBE performs competitively against existing systems.

Critical strengths
  • The paper studies important evaluation problem. The paper highlights that statistically significant performance differences do not necessarily imply fair comparisons when systems receive different inputs or resources. I appreciate the distinction between comparing entire systems and attributing performance differences to individual components. The experiments on published pipelines provide a concrete demonstration of this issue.
  • The authors construct a large-scale benchmark and evaluate multiple published and commercial agents. The controlled comparisons and evidence-ablation experiments provide useful insights into the limitations of existing agent evaluation practices.
Critical weaknesses
  • In the main experiments, inputs are matched, so certification essentially reduces to conventional statistical testing. When inputs differ, the nuisance bounds are often loose, and the empirically estimated sensitivity constants do not necessarily provide rigorous upper bounds. It is unclear how much the proposed framework improves upon standard controlled evaluation practices.
  • Although stage-level training is presented as a major contribution, only the planning stage is trained, and its improvement over the untrained harness is not statistically certified. The strong leaderboard results therefore seem to demonstrate the effectiveness of the overall agent design rather than the proposed training method.
Questions for authors
  • Can the authors demonstrate settings where the proposed framework provides meaningful certification beyond matched-input evaluation and conventional statistical testing? How reliable are the empirical nuisance bounds?
  • Can the authors provide stronger evidence that stage-level training improves performance, independently of the overall SCRIBE architecture?
  • The evidence-ablation experiments suggest that some benchmark metrics improve even without supporting evidence. To what extent do these metrics capture the intended literature-review capabilities rather than similarity to reference reports?
2 comments

Thank you for separating two questions: comparing whole systems, and attributing a difference to one component. The revision is rewritten around exactly this distinction.

W1 / Q1. With matched inputs certification reduces to a standard test. Can the framework give meaningful certification beyond that, and how reliable are the empirical bounds?

On the overall ranking and composite score of the main leaderboard, the framework indeed adds nothing: with matched inputs, certification is the standard test, by design. What it adds does not come from a different test but from what is compared and whether it may be compared. First, the output of each stage is scored on its own against the expert review, so systems the composite cannot separate can be taken apart stage by stage. For example, untrained SCRIBE and Claude Code (Opus 5) differ by only 0.018 and are not separated, yet about 0.016 of that comes from Planning, while on Synthesis Claude Code is ahead (0.747 vs 0.683). Second, it decides before any test which comparisons may be made. When inputs differ, that step gives conclusions a standard test cannot.

Comparisons that look fair but should not be made. In the shared-corpus evaluation every system faces the same corpus, searches it itself, and writes to the same length, a common design. Take Claude Code (Sonnet 5) and Claude Science (Opus 5): on the five axes after retrieval, Claude Code is 0.032 higher, still significant after multiplicity correction, and the usual conclusion would be that Claude Code writes better. But these axes measure writing after the papers are in hand, and the two systems searched separately and entered writing with different papers, so the audit refuses that conclusion before any score is read. Our main leaderboard gives both systems the same reference list, and there the same five axes differ by only 0.005, so the 0.032 was not a difference in writing. The table lists this example and three more of the same kind (gaps are first minus second):

Two systemsAxis comparedGap when each searches for itselfGap when both get the same reference list
Claude Code (Sonnet 5) vs Claude Science (Opus 5)mean of Synthesis, Reasoning, Planning, Writing, Form+0.032, significant after correction (threshold 0.030)+0.005, not significant (p = 0.60)
AutoSurvey vs SurveyG (both gpt-5.6-luna)Form+0.082, significant after correction (threshold 0.065)+0.002, not significant (p = 0.89)
LLM×MapReduce vs OpenAI tool loop (both gpt-5.6-luna)Reasoning−0.099, significant after correction (threshold 0.085)−0.037, not significant (p = 0.11)
OpenAI tool loop (GPT-6.1 Sol) vs Claude Science (Opus 5)Writing+0.075, significant after correction (threshold 0.068)−0.023, reversed, not significant (p = 0.15)

Planted mismatches with a known answer (Figure 6). One system is compared with copies of itself that differ in one way each (double retrieval budget, a citation tool that can reach the expert's references, a doubled target length, 30% of input papers removed, 30 unrelated papers added, a cutoff one year later), so the two sides are equally capable. The audit refused all six before scoring and admitted two identical re-runs as controls. A conventional test was fooled on single readouts: with double budget it called six retrieval readouts "better", and with doubled length it found five readouts different at full length. It also missed the citation tool, although on 10 tasks that tool raised the share of the expert's references in the input from 23% to 72%.

Comparisons that should not be made at all. Among the 21 pairs of the seven published pipelines, a careful conventional analysis gives 14 orderings, and 7 fall in pairs the audit refuses (Table 2), e.g. LiRA > SurveyG, where only LiRA received the reference list.

Reliability of the empirical bounds. You are right that estimated constants are not rigorous upper bounds. The conclusions do not hinge on their exact value:

CheckResult
Sweep L_M from 0.4 to 1.2 (Figure 5)all five LiRA refusals hold for L_M ≥ 0.554, every estimate lies in 0.863–0.967, so the constant would have to be overestimated by more than 1.61× to change a conclusion
Two planted copies whose inputs differ only partly (70–75% of papers shared)radius 0.13–0.16, the realized gap is 9–15% of the radius, the bound holds on every readout, with no false certification
SCRIBE's eight real multi-stage chains, constants fitted on half the tasks and tested on the other half (Appendix F.4)the bound holds on 96–99% of tests, it is below 1 at the synthesis and planning exits, and loose after writing

W2 / Q2. Can you show that stage-level training helps independently of the SCRIBE architecture?

The training effect is already measured with the architecture fixed: both arms run the same SCRIBE pipeline, synthesis input and writer, and differ only in the planning (outline) module. To check that the trained stage still helps outside SCRIBE's own downstream, we also had the last step, turning the outline into a report, done by another model, gpt-5.6-luna (the model of six published pipelines). The two arms differ only in whether the outline came from the untrained or the trained planner, and the decision rule was fixed before generation. Each cell is the composite gain of the trained over the untrained planner, with 95% intervals in brackets:

Our 27B writes the reportgpt-5.6-luna writes the report
71 held-out tasks+0.021, +0.024, +0.018 (three seeds, first two certified)+0.028 [+0.009, +0.047], certified (three seeds pooled: +0.020 [+0.002, +0.038])
50 test tasks+0.012, not certified (planning +0.082, certified)+0.022 [+0.001, +0.043], certified (the paper row's planner: +0.016 [+0.003, +0.030])

Where the gain comes from when gpt-5.6-luna writes the report:

TasksOutline fromCompositePlanningBroken outlines
71 held-outuntrained planner0.5700.2529 / 71
71 held-outtrained planner (seed 17)0.5980.3422 / 71
50 testuntrained planner0.5710.2430 / 50
50 testtrained planner (seed 17)0.5930.3251 / 50

On the held-out tasks the gain comes mainly from repairing broken outlines (9 down to 2). On the test tasks no untrained outline was broken, and the gain comes from better outlines (planning 0.243 → 0.325). So with a different downstream writer, the trained stage still improves the final report.

The recipe does not depend on SCRIBE either: the teacher sees the expert's top-level titles while drafting, and the student sees no hint at run time, so it applies to any pipeline whose stage output can be scored on its own. You are right that the strong leaderboard result mainly reflects the overall design, and Section 6.1 now says so.

Q3. Do the metrics capture literature-review ability, or similarity to the reference report?

They measure a specific ability: given a set of papers, stating the findings those papers support that an expert judged important, and linking and organizing them as the expert did. The reference review fixes which findings are worth stating, not the wording:

  • Matching is by meaning, not shared words. Four of the six synthesis readouts are stratified by whether the cited abstracts support the claim, so the score comes from stating what the papers actually support.
  • The scale is not set by one review: each task has 4–9 other expert reviews on the same topic, and 0 is their 10th percentile. Style readouts such as length or section count give full marks anywhere within the other experts' range (Appendix I).
  • Beyond coverage, the readouts check whether findings the expert kept apart are lumped together, and whether each sentence cites a paper that supports it.

These readouts are of the same kind as common metrics in existing benchmarks. What differs is how they are scored:

What we measureSimilar metrics in existing benchmarksHow we score it
stating the findings an expert judged importantSurGE reference recall, ResearchQA coverageagainst the expert's claim graph for the same papers, with the scale set by several same-topic experts
section structureSurGE heading soft recallagreement with the expert's top-level sections
whether citations support sentencesNLI-based citation recall and precision in AutoSurvey and SurGEthe new citation-support check (Figure 4), plus whether a finding cites the paper the expert cited for it
links between findingsrarely measurednew: seven relation types between the expert's claims, e.g. mechanism and contrast

AutoSurvey, SurveyForge and others score with LLM judges, which measures an LLM's preferences. Our scoring is deterministic and uses no LLM judge. It measures the expert's choices: which findings are worth stating, how they connect, and how they are organized.

On the evidence ablation: a writer given no papers still scores well on some content readouts, because a strong model knows established findings. But it scores clearly lower on writing (−0.071), because writing checks whether each claim cites its own paper. The revision therefore states that content readouts measure agreement with the expert's content, not use of the supplied evidence. The latter is measured by the evidence-agreement readout and the new citation-support check. Swapping 60% of citations for other papers from the same list lowers citation support by about 32 points.

Reviewer_B6wuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the problem of fair evaluation and performance attribution for long-horizon AI agents. The authors propose compositional controllability, a framework that determines whether two agents can be fairly compared by bounding the effects of mismatched inputs, observation channels, and evaluation protocols. It introduces admissibility tests and certified rank intervals to avoid potentially misleading leaderboard rankings.

The authors also introduce BioLitBench, a benchmark containing 2,042 biomedical articles represented as structured claim graphs, and SCRIBE, a literature-review agent trained using stage-level rewards. Experiments show that 11 of 21 pairwise comparisons among published pipelines are inadmissible, including seven conventionally significant comparisons. SCRIBE achieves the highest composite score.

Critical strengths
  1. Principled theoretical framework. The compositional controllability framework provides a systematic approach to separating performance differences within a comparison window from nuisance factors outside it. The resulting admissibility tests and certified rank intervals are conceptually appealing.
  2. Meaningful empirical findings. The observation that seven statistically significant comparisons are inadmissible provides compelling evidence that conventional evaluation procedures can support unjustified conclusions.
  3. Connection between evaluation and training. The use of controlled stage-level rewards in SCRIBE is an interesting application of the proposed framework, demonstrating its potential beyond leaderboard analysis.
Critical weaknesses
  1. Practical limitations of the theoretical guarantees. The framework relies on estimating nuisance bounds and Lipschitz constants, which may be difficult for complex, black-box agents. Moreover, the main leaderboard experiments use matched conditions where the nuisance radius is zero, leaving the usefulness of nonzero bounds less thoroughly demonstrated.
  2. Limited validation of multi-stage compositionality. Although the framework supports arbitrary comparison windows, the multi-stage composition bound is tested primarily on synthetic chains rather than real agent pipelines. Some bounds also become loose or vacuous as the number of stages increases.
  3. Concerns about benchmark validity. BioLitBench relies heavily on automatically extracted claims and semantic similarity measures. The evidence-ablation results suggest that some metrics may reward content similarity without adequately measuring evidence grounding or factual correctness. More human-expert validation would strengthen the benchmark.
  4. Insufficient evidence for training effectiveness. SCRIBE achieves strong performance, but its improvement over the untrained harness is not statistically certified. Additionally, only the planning stage is trained, limiting the evidence for the broader stage-level training framework.
Questions for authors
  1. The proposed framework relies on estimating nuisance bounds and Lipschitz constants, which may be challenging for complex agents. How sensitive are the certification results to estimation errors in these quantities? Can the authors provide more evidence that the bounds remain informative when nuisance factors are nonzero?
  2. While the framework supports comparisons across multiple stages, the composition bounds are primarily evaluated on synthetic chains. Could the authors provide experiments on real multi-stage agent pipelines, such as SCRIBE, to demonstrate the practical effectiveness of these guarantees?
  3. SCRIBE achieves the highest composite score, but its improvement over the untrained baseline is not statistically certified. Can the authors provide additional ablations or comparisons against final-output reward training to demonstrate that stage-level rewards offer a meaningful advantage?
2 comments

Thank you for asking us to compare stage-level and final-output rewards (Q3). The comparison came out even, and that changed how the paper presents training.

W1 / Q1. Nuisance bounds and Lipschitz constants are hard to estimate. How sensitive is certification to estimation error, and are the bounds informative when nuisance is nonzero?

On the main leaderboard two systems receive byte-identical inputs, so the nuisance radius is zero and no estimated constant is used. Constants matter only when inputs differ. In the paper these are LiRA against the five self-retrieving pipelines: LiRA received the reference list, while the others searched the shared corpus. For each pair the radius is L_s × L_M × (the two sides' input mismatch), and a readout is refused when its radius exceeds the tolerance (the typical gap between two human reviews of the same topic):

PairInput mismatch ε_inRadius (median)Tolerance (median)Readouts refusedConventional test
AutoSurvey – LiRA0.9750.7250.20815 / 15not separated
SurveyForge – LiRA0.9680.7200.20815 / 15not separated
SurveyG – LiRA0.9720.7230.20815 / 15LiRA better (+0.059, threshold 0.057)
SurveyGen-I – LiRA0.9420.7000.20815 / 15not separated
LLM×MapReduce – LiRA0.9790.7280.20815 / 15LiRA better (+0.105, threshold 0.063)

A radius of about 0.72 is more than three times the gap between two human reviews: the input difference alone can move the score more than the gap between two experts. A conventional test declares LiRA better in two of these pairs, but the bound says that gap cannot be credited to the system.

Sensitivity to the constant (Figure 5): we swept L_M from 0.4 to 1.2 with everything else fixed. For every L_M ≥ 0.554, all five pairs are refused on all 15 readouts. Every estimate lies between 0.863 and 0.967 (selected value 0.890), so the constant would have to be overestimated by more than 1.61× before any refusal changed.

The bound does not only refuse. In our known-answer experiment, two copies of one system differ from it only partly (30% of input papers removed, or 30 unrelated papers added). The radius is then small, and the bound lets some readouts be compared with a widened interval:

Plant (reference list given)Papers shared by the two sidesRadius (median)Realized gap / radius (median)Readouts where the bound holdsReadouts compared after wideningFalse certifications
30% of input papers removed70%0.1610.145100%90
30 unrelated papers added75%0.1330.089100%110

The realized gap is only 10–15% of the radius, the bound holds on every readout, and no readout is falsely certified after widening.

W2 / Q2. The composition bound is tested mainly on synthetic chains. Can you test it on a real multi-stage pipeline such as SCRIBE?

Yes, on SCRIBE (Appendix F.4). Testing the composition bound requires re-entering each stage from a specified intermediate artifact. Published pipelines cannot do this, but SCRIBE's three stages (synthesis, planning, writing) each start from a sealed artifact. We ran each stage on either Qwen3.8-27B (Q) or gpt-5.6-luna (L), giving all eight chains. For example, QLQ runs synthesis on Qwen, planning on Luna and writing on Qwen. Scores on the 50 test tasks (reference list given, matched length). This composite excludes the planning axis, so it is not directly comparable with the leaderboard:

ChainSynthesisPlanningWritingComposite (four axes)
QQQQwenQwenQwen0.662
QQLQwenQwenLuna0.665
QLQQwenLunaQwen0.677
LQQLunaQwenQwen0.664
QLLQwenLunaLuna0.672
LQLLunaQwenLuna0.659
LLQLunaLunaQwen0.664
LLLLunaLunaLuna0.661

We then tested the bound on 13 pairs of chains: at each stage exit we measured the real distance between the two chains' outputs and compared it with the bound's prediction. The per-stage replacement errors ε_k and Lipschitz constants L_k were fitted on half of the tasks and tested on the other half, splitting once by task and once by subfield. The table gives the range over the two splits:

Stage exitReal distance between chains (mean)Bound (mean)Tests where the real distance stays within the boundTests where the bound is below 1 (informative)
synthesis0.0880.101–0.10796.8–97.8%100%
planning0.2560.562–0.56497.5–98.8%100%
writing0.3841.081–1.25396.0–99.2%30–58%

The bound holds on 96–99% of out-of-sample tests. How tight it is depends on the stage: at the synthesis and planning exits it is below 1 on every test (distances lie in [0, 1], so only a bound below 1 is informative). After writing, which amplifies earlier differences, it is below 1 on only 30–58% of tests. This matches your point that the bound loosens with depth, and the revision says so.

W3. Benchmark validity: automatically extracted claims, semantic similarity, the evidence ablation, and the need for human validation.

The claims are not extracted by a model. We parse the article XML with regular expressions: every sentence that carries a citation marker is a claim, kept verbatim with its citations and its section (Section 4). Each claim is therefore the expert's own sentence and cannot be rewritten or invented. Two bioinformatics PhD students also checked about a dozen claim graphs against the source articles, and we revised the construction accordingly. The only model used in scoring is a sentence encoder that decides which report sentence matches which expert claim. We checked it three ways.

First, re-scoring every stored report with another public encoder (e5-large-v2) keeps the ranking of the 16 systems (Kendall τ = 0.88) and all 9 certified wins of untrained SCRIBE.

Second, defect injection, with the expected effects written down before injecting (each defect applied to 60% of the content):

Injected defectChange in scores
60% of paragraphs deletedsynthesis −0.187, reasoning −0.128
60% of claims replaced by fillersynthesis −0.118
60% of citations swapped for other papers from the same listwriting −0.107, citation support −31.9 points
conclusions reversedbarely visible in the readouts
sentences moved to the wrong sectionbarely visible in the readouts

Third, the evidence ablation (Appendix D): a writer given the full evidence scores 0.071 higher on writing than one given none. Averaged over six pipelines, only the evidence-agreement readout separates full evidence from both no evidence (+0.145) and shuffled evidence (+0.078). The paper therefore now states that the content readouts measure agreement with the expert's content, and reports citation support separately (Figure 4: for most systems 81–85% of cited sentences are supported by a cited abstract, against 69% for the expert reviews themselves).

W4 / Q3. Training evidence is insufficient and only planning is trained. Does a stage-level reward beat a final-output reward?

We ran the comparison. We used the same training method and the same teacher drafts, and selected training examples either by a stage reward (each candidate outline scored at the planning exit) or by a final-output reward (each candidate outline handed to the writer, written into a full report, and the report scored on the composite). On the 71 held-out tasks:

Training examples selected byCompositevs untrained plannerPlanning
(untrained planner)0.561—0.224
stage reward (outline scored)0.591+0.0300.321
final-output reward (full report scored)0.578+0.0170.342

Both beat the untrained planner. The stage reward is slightly ahead, but the difference is not significant (−0.013, 95% interval [−0.029, +0.003]), so we do not claim it trains better. Its advantage is cost: it scores outlines on a CPU, while the final-output reward first has to write the candidate outlines into full reports, about a thousand for this cell.

We trained only planning because it is the axis on which all 16 systems are furthest from the expert (planning 0.16–0.32, every other axis above 0.44), and our open 27B breaks outlines on large inputs (25% of tasks in the largest third). Results are below. "Report written by" is the model that turns the outline into the report: our 27B (SCRIBE's default) or gpt-5.6-luna (the model of six published pipelines). The two arms differ only in whether the outline came from the untrained or the trained planner (seed 17):

Our 27B writes the reportgpt-5.6-luna writes the report
71 held-out tasks+0.021 (0.567 → 0.588), certified+0.028 (0.570 → 0.598), certified
50 test tasks+0.012 (0.566 → 0.578), not certified (planning +0.082, certified)+0.022 (0.571 → 0.593), certified

Training mostly repairs broken outlines: on the held-out tasks the broken rate falls from 12.5% to 3.0%. Of the outlines that crammed every claim into one or two sections, 96% are repaired, as are 92% of the cut-off outlines, and only 3% of sound outlines are newly broken. The revision presents training as the repair step of diagnose-then-repair, and states that its gain over the same pipeline untrained is not certified.

Reviewer_R1ZoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AI agent leaderboards can be misleading: when agents are given different inputs, models, or budgets, a “statistically significant” score gap may reflect unfairness rather than real architectural wellness. To fix this, the authors propose “compositional controllability,” a framework that first checks whether two agents can be fairly compared, and then declares a winner only when the score gap exceeds the combined error from randomness and unfairness. They test it on BioLitBench, a new benchmark built from 2,042 human-written biomedical reviews, where their test refuses 11 of 21 comparisons among published agents and invalidates half of the conclusions a standard statistical analysis would have drawn. They also build their own agent, SCRIBE, which ranks at the top of the leaderboard.

Critical strengths
  • The paper points out that "fair comparison" are often bypassed in leaderboards across agents. As agent benchmarks multiply, this observation is important for future benchmark design.
  • The paper contributes a large, structured benchmark resource (2,042 biomedical reviews converted into graphs of about 246,000 claims and 832,000 relations, with a separate training split), although its evaluation set of 50 tasks is comparatively small.
Critical weaknesses
  • The advantage of training is not fully established. In Appendix B on the same pipeline, training changes the score by only +0.011 and −0.010, while the main tables compare against an older pipeline. Training also used a single run with reward weights tuned on the evaluation tasks, so the paper’s central training claim needs multiple training seeds and a clean held-out setup before it can be supported.
  • The admissibility test’s refusals are shown on only 21 pairs among 7 pipelines in one domain, and they catch only obvious mismatches.
Questions for authors
  • Which parts of SCRIBE are responsible for its lead over the other agents? Could the authors provide a component analysis and qualitative examples of its reports compared with other agents’ outputs to show which design choices drive its performance?
  • Could the authors apply the framework to other agent leaderboards, including cases with subtler mismatches like differing budgets, tools or output lengths, to show that it makes correct and useful decisions more broadly?
3 comments

Thank you for checking Appendix B against Table 1. The two same-pipeline numbers you picked (+0.011 and −0.010) are exactly where the paper's training claim was weak. We agree, and we ran the clean setup you asked for.

W1. The training advantage is not established. It needs several seeds and a clean held-out setup.

Agreed that the original paper did not establish a training gain. First, a correction: the original Section 5 said the reward weights were "calibrated on leaderboard tasks". That was wrong. The planning reward uses fixed weights carried over from earlier planning runs, and they were never tuned on any evaluation task. The revised Section 5 says so.

The clean setup. Training tasks are disjoint from the 50 test tasks and their 399 same-topic reviews, and the training code refuses them. No test task is used for training, calibration or checkpoint choice. We also hold out 71 tasks of the training split that are never trained on and are used only to choose checkpoints and for intermediate checks. We train only planning (the stage that writes the outline): the same 27B drafts several candidate outlines per training task, the best by planning-exit score is kept, and the planning module is fine-tuned on these outlines. At run time it uses exactly the same prompt as the untrained model. We use three seeds (17, 29, 53), and the decision rule of every test run was written before generation.

Two ways of writing the report. A SCRIBE report is produced in three steps: synthesis (organize the claims from the papers) → planning (write the outline) → writing (turn the outline into the report). Since only planning is trained, we measure its effect on the final report with two writers for the last step: our own 27B (SCRIBE's default), and gpt-5.6-luna (the model of six published pipelines). In the second setting the two arms differ only in whether the outline came from the untrained or the trained planner, and the synthesis input and the writing model are identical. This tests whether the trained planning stage still improves the final report when a different model writes it.

Results are below. Each cell is the composite gain of the trained over the untrained planner, both arms from the same run:

Our 27B writes the reportgpt-5.6-luna writes the report
71 held-out tasks+0.021, +0.024, +0.018 (three seeds, first two certified)+0.028, certified
50 test tasks+0.012, not certified (planning +0.082, certified)+0.022, certified

You also pointed out that the main tables compare against an older pipeline. We ran a second test run inside the paper row's own pipeline, changing only the planner:

The paper row's own pipeline (50 test tasks)Compositevs untrained planner
untrained planner0.597—
the paper row's planner0.605+0.008, not certified
retrained planner (mean of three seeds)0.604+0.007, not certified

The retrained planner reproduces the paper row's score, but neither is certified above the same pipeline's untrained planner. We also placed the same-pipeline untrained row of Appendix B (0.595) on the board: the trained row is 0.011 above it against a threshold of about 0.037, not certified, and it keeps all 13 of its certified wins. So the leaderboard position comes mainly from the overall design, not from training, and Section 6.1 now says so.

What training changes. Mostly it repairs broken outlines. On large inputs the untrained 27B breaks the outline: either all claims are crammed into one or two sections, or the output runs too long and is cut off. On the held-out tasks the broken rate falls from 12.5% to 3.0%: 96% of crammed outlines and 92% of cut-off outlines are repaired, and only 3% of sound outlines are newly broken. One held-out example (a review of cathinone pharmacology): the untrained planner produced one section, leaving 362 of 471 claims unassigned, and a 2,896-word report. The trained planner produced 6 sections (the expert used 8) and left 87 claims unassigned. Its report had 9,329 words, and the composite rose from 0.531 to 0.681. Test tasks have smaller inputs, and in the first test run the untrained planner broke no outline, so there was little to repair. This is why the composite gain with our 27B writer is not certified, while it is certified with the gpt-5.6-luna writer.

W2 / Q2. Refusals are shown on only 21 pairs and catch only obvious mismatches. Can the framework handle subtler mismatches (budgets, tools, output length) and other leaderboards?

Agreed that the 11 refusals in the paper are obvious. For the budget, tool and length mismatches you named, we ran an experiment with a known answer (Figure 6). The system under test is SCRIBE-Luna (the SCRIBE pipeline with gpt-5.6-luna at every stage). We compare it with a copy of itself whose code, prompts and model are unchanged except for one planted mismatch, and, as a control, with an identical re-run. Both sides are the same system, so any verdict that one is more capable is wrong. Each setting has three plants and one control:

SettingThe copy differs only inAudit (before scores)Readouts a conventional test calls different (matched length / full length)
reference list given (38 readouts)30% of the input papers removedrefused: input differs from the reference list2 / 1
reference list given30 unrelated same-subfield papers addedrefused: same0 / 0
reference list giventarget length doubledrefused: task specification differs1 / 5 (four are coverage readouts)
reference list givencontrol: identical re-run, inputs byte-identical on 50/50 tasksadmitted0 / 0
shared corpus (44 readouts)retrieval budget doubledrefused: budget audit6 / 6 (all six retrieval readouts)
shared corpusa citation tool that can reach the expert review's references, opened on 10 tasksrefused: retrieval-route audit0 / 0
shared corpusliterature cutoff one year laterrefused: cutoff audit1 / 1
shared corpuscontrol: identical re-run, retrieved papers identical on 50/50 tasksadmitted0 / 0

The conventional test is the careful one: a per-readout paired test clustered by subfield, with Bonferroni correction. It never erred on the composite, but it was fooled on single readouts. With double budget it calls six retrieval readouts "better", although the copy only retrieved twice as much. With doubled length it finds five readouts different at full length, i.e. "writing longer finds more of the expert's findings". Hardest to see is the citation tool: on those 10 tasks it raised the share of the expert's references in the input from 23% to 72%, nearly handing over the answer, yet the conventional test found nothing, because the tool shows up only in the per-call logs. The audit refused all six plants before any score was read and admitted both identical re-runs.

The framework does not only refuse. For the two copies whose inputs differ only partly (30% removed or 30 added, so the two sides still share 70–75% of their papers), the bound gives a radius of only 0.13–0.16, so 9 and 11 readouts can still be compared with a widened interval, and none was falsely certified.

Real systems also produce comparisons that look fair but are correctly refused. In the shared-corpus evaluation every system faces the same corpus, searches it itself, and writes to the same length, a common design. Take Claude Code (Sonnet 5) and Claude Science (Opus 5): on the five axes after retrieval, Claude Code is 0.032 higher, still significant after multiplicity correction, and the usual conclusion would be that Claude Code writes better. But these axes measure writing after the papers are in hand, and the two systems searched separately and entered writing with different papers, so the audit refuses that conclusion before any score is read. Our main leaderboard gives both systems the same reference list, and there the same five axes differ by only 0.005, so the 0.032 was not a difference in writing. The table lists this example and three more of the same kind (gaps are first minus second):

Two systemsAxis comparedGap when each searches for itselfGap when both get the same reference list
Claude Code (Sonnet 5) vs Claude Science (Opus 5)mean of Synthesis, Reasoning, Planning, Writing, Form+0.032, significant after correction (threshold 0.030)+0.005, not significant (p = 0.60)
AutoSurvey vs SurveyG (both gpt-5.6-luna)Form+0.082, significant after correction (threshold 0.065)+0.002, not significant (p = 0.89)
LLM×MapReduce vs OpenAI tool loop (both gpt-5.6-luna)Reasoning−0.099, significant after correction (threshold 0.085)−0.037, not significant (p = 0.11)
OpenAI tool loop (GPT-6.1 Sol) vs Claude Science (Opus 5)Writing+0.075, significant after correction (threshold 0.068)−0.023, reversed, not significant (p = 0.15)

Given the same papers, all four gaps disappear, and the last one reverses.

As for other leaderboards: the audit reads only what each run records (input papers, task specification, budget, cutoff, retrieval route and length convention), so it applies to any leaderboard that records these fields, closed products included.

Q1. Which parts of SCRIBE drive its lead? Component analysis and qualitative examples.

Mainly the outline. Each row below changes one part of SCRIBE and keeps the rest fixed:

What we change (everything else fixed)TasksResult
Outline: the same, on the test tasks50 testplanning 0.270 → 0.688, composite 0.595 → 0.656 (+0.060)
Outline: train only the planning module (clean retraining)50 testplanning +0.082 (certified), composite +0.012 (our 27B writes), +0.022 (gpt-5.6-luna writes, certified)
Model: switch each of the three stages between our 27B and gpt-5.6-luna, all 8 combinations50 testthe four axes other than planning stay within 0.659–0.677
Model: run the whole pipeline on gpt-5.6-luna (SCRIBE-Luna) and compare with published pipelines on the same model50 testcertified above all five that run with the reference list (+0.046 to +0.092)

So a better outline raises the composite clearly. The choice of model barely moves the other axes. And with the model fixed, SCRIBE's pipeline still leads the published ones.

Against the strongest frontier agent the difference is also the outline (means over the 50 test tasks, reference list given):

Untrained SCRIBEClaude Code + Opus 5The expert review itself
composite0.5830.564—
synthesis / reasoning / planning0.683 / 0.600 / 0.2760.747 / 0.527 / 0.195—
central findings stated (of 27.2)21.721.727.2
mechanism links reproduced (of 26.1)16.214.426.1
unrelated findings kept in separate paragraphs80%94%100%
top-level sections7.811.66.1

Both state the same number of central findings. SCRIBE's sections are closer to the expert's and it reproduces more links, while Claude Code keeps unrelated findings apart more often. Of their 0.018 composite gap, about 0.016 comes from planning.

A qualitative example, chosen by a rule fixed in advance (the test task with the smallest composite gap between the two: a review of neutrophil chemoattractant receptors, SCRIBE 0.620, Claude Code + Opus 5 0.621). The expert wrote 6 top-level sections: "The neutrophil: an introduction" / "Regulation of neutrophil trafficking, priming and activation by chemoattractants" / "General aspects of chemoattractant-induced receptor signaling" / "Expression levels and functions of neutrophil chemoattractant receptors in health and disease" / "Final remarks" / "Supplementary information". SCRIBE wrote 7 in a similar order: introduction / receptor families and ligands / signaling pathways / regulation of receptor expression and function / infection and host defense / inflammatory and autoimmune disease / therapeutic implications. Claude Code split the same material into 14 sections (e.g. "The destructive edge I", "The destructive edge II", "Pathogen subversion of chemoattractant receptors", "Open questions and future directions"). On this task SCRIBE reproduces more of the expert's mechanism links (0.40 vs 0.20), while Claude Code's sentences more often cite the claim's own paper (0.54 vs 0.18). Appendix H compares the two reports sentence by sentence on another test task.

Reviewer_gSXJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that a higher leaderboard score alone does not show whether two agents were compared fairly, or which step caused the difference. Agents may get different evidence, inputs or budgets, and a better final report may come from better search rather than better writing.

In the authors' proposed method, before comparing two agents, they first estimate how much outside differences (in inputs, scoring, or setup) could move the score. If this amount is too large, the comparison is refused. If it is small, one agent is certified above the other only when the score gap is bigger than the noise plus this amount; otherwise, the pair is marked 'undecided'. Eventually, each agent gets a range of ranks instead of a single rank.

Seven pipelines, five commercial agents (four reported), and three agents built by the authors themselves are tested on one benchmark, BioLitBench.

Critical strengths

The idea is intuitive and interesting. The writing is generally easy to follow. The problem tackled is important.

Critical weaknesses
  1. Only one benchmark was tested, with only 50 tasks.
  2. Mathematical formulas dominate the paper. Although the writing is fluent and natural, the heavy formulas and the lack of figures make the paper more difficult to follow.
  3. The main demonstration is weak, which makes the main claim of the paper unconvincing. The paper builds a test that says when a comparison between two AI systems is unfair. The authors show it on 21 comparisons. The test calls 11 of them unfair. But those 11 are unfair for obvious reasons, so a careful researcher would not compare them anyway. The test agrees with common sense but does not show it catches hidden unfairness. For example, in DR-Tulu vs. SurveyG, DR-Tulu runs on its own AI model. SurveyG runs on GPT (gpt-5.6-luna). We can’t tell whether DR-Tulu’s design is worse or its model is just weaker. Anyone can see this unfairness without a new test.
  4. In the main leaderboard, the new method becomes an ordinary significance test. A is certified above B only if the score gap is bigger than random noise plus an unfairness allowance, which grows when the two systems get different inputs, scoring or setup. When all of these are the same, as in the main leaderboard, the allowance becomes 0, and the rule shrinks to: A beats B if the gap is bigger than the noise. That is a standard significance test. The core idea mirrors standard controlled comparison. The new part, the limit on unfairness, is very loose: in Appendix F.2, the typical limit is 0.955, while the real error it has to cover is typically 0.060. Of the 11 refusals, only the five LiRA ones use a computed limit; the six DR-Tulu refusals come from the model difference. Both were already obvious.
  5. SCRIBE’s certified advantage depends on the authors’ own choices. The main results cut each report to the task’s target length. Under fixed input, when full reports are scored, SCRIBE’s lead over the best reported system not built by the authors shrinks from 0.065 to 0.042, and its rank range widens from [1, 2] to [1, 4] (Tables 1 and 5).
Questions for authors
  1. With the same model as six of the seven published pipelines, is SCRIBE-Luna clearly better than each of them by your test? If not, how much of SCRIBE’s lead comes from its model?
  2. The main results cut every report to each task’s target length. When full reports are scored, SCRIBE’s lead over the best reported system not built by the authors shrinks. Why is the shortened version the main one, and did you choose it before seeing results?
  3. In the main leaderboard, all systems share the same entry condition, so the method gives the same result as a normal test. Which results in the paper could a normal test not have produced?
  4. Can you show a comparison that looks fair at first glance, but that your test correctly rejects?
3 comments

Thank you for Q4: asking for a comparison that looks fair but is correctly refused is exactly the right test of our method. We answer it first.

W3 / Q4. The refusals are obvious. Can you show a comparison that looks fair but is correctly refused?

Yes. Section 6.2 now gives such comparisons between real systems. We first walk through one example, then list the others that follow the same pattern.

Example: Claude Code (Sonnet 5) vs Claude Science (Opus 5).

  1. The setting looks fair. This is the shared-corpus evaluation: both systems face the same frozen corpus, search it themselves, and write reviews of the same length. Many evaluations are designed this way, and nothing in the configuration looks wrong.
  2. A conventional test gives a significant result. On the mean of the five axes after retrieval (Synthesis, Reasoning, Planning, Writing, Form), Claude Code scores 0.032 above Claude Science. With subfield clustering and Bonferroni correction over 105 pairs, the gap is still significant (threshold 0.030). The usual conclusion: Claude Code writes better reviews.
  3. What is wrong. These five axes measure writing after the papers are in hand, but the two systems searched separately, so they entered writing with different papers. In fact Claude Science retrieves better (0.119 higher on retrieval). The 0.032 could therefore come from writing skill or from the different papers, and a conventional test cannot tell which.
  4. What the audit does. Before any score is read, the audit checks one thing: whether the two sides enter the compared stage with the same inputs. Here they do not, so it refuses the conclusion that Claude Code writes better. The whole-system comparison, retrieval included, is unaffected.
  5. How we know the refusal is right. Our main leaderboard gives both systems the same reference list, so their inputs are identical. There, the same five-axis gap is only +0.005 (p = 0.60). The 0.032 was not a difference in writing.

Three more pairs show the same pattern. Gaps are first minus second. When each system searches for itself, a conventional test declares one side better on that axis. Given the same papers, every gap disappears, and the last one reverses:

Two systemsAxis comparedGap when each searches for itselfGap when both get the same reference list
Claude Code (Sonnet 5) vs Claude Science (Opus 5)mean of Synthesis, Reasoning, Planning, Writing, Form+0.032, significant after correction (threshold 0.030)+0.005, not significant (p = 0.60)
AutoSurvey vs SurveyG (both gpt-5.6-luna)Form+0.082, significant after correction (threshold 0.065)+0.002, not significant (p = 0.89)
LLM×MapReduce vs OpenAI tool loop (both gpt-5.6-luna)Reasoning−0.099, significant after correction (threshold 0.085)−0.037, not significant (p = 0.11)
OpenAI tool loop (GPT-6.1 Sol) vs Claude Science (Opus 5)Writing+0.075, significant after correction (threshold 0.068)−0.023, reversed, not significant (p = 0.15)

In the second and third rows both systems use the same model, so the usual conclusion would credit the scaffold, e.g. "AutoSurvey's scaffold writes more like an expert". The audit needs no hindsight. It does not know that these gaps will vanish, only that the inputs differ, and it sees this before scoring.

A test with a known answer. We also checked whether the audit itself errs: one system (SCRIBE-Luna) is compared with a copy of itself that has a single planted mismatch and is otherwise identical. Both sides are the same system, so any "better" verdict is wrong (Figure 6):

The copy differs only inAudit (before scores)Readouts a conventional test calls different (matched / full length)
reference list given: 30% of the input papers removedrefused: input differs from the reference list2 / 1
reference list given: 30 unrelated same-subfield papers addedrefused: same0 / 0
reference list given: target length doubledrefused: task specification differs1 / 5
shared corpus: retrieval budget doubledrefused: budget audit6 / 6 (all six retrieval readouts)
shared corpus: a citation tool that can reach the expert's references, opened on 10 tasksrefused: retrieval-route audit0 / 0
shared corpus: literature cutoff one year laterrefused: cutoff audit1 / 1
control: identical re-run (once in each setting)admitted0 / 0

The audit refused all six plants before scoring and admitted both controls. The conventional test never erred on the composite but was fooled on single readouts. Hardest to see is the citation tool: on those 10 tasks it raised the share of the expert's references in the input from 23% to 72%, nearly handing over the answer, yet the conventional test found nothing, because the tool shows up only in the per-call logs.

W4 / Q3. On the main leaderboard the method is an ordinary test. Which results could a normal test not produce?

You are right that on the overall ranking and the composite score of the main leaderboard, certification is the standard test. All systems receive the same inputs, and a fair comparison needs no extra correction. But an ordinary leaderboard compares only the total score of the final report, so at most it tells us whose total is higher. What we add does not come from a different test. It comes from what is compared, and whether it may be compared.

1. Which stage a gap comes from. The output of each stage is scored on its own against the expert review, so two systems that the composite cannot separate can be taken apart stage by stage. Untrained SCRIBE and Claude Code (Opus 5) differ by only 0.018 in composite (threshold 0.032) and are not separated, yet their strengths are opposite:

SCRIBE (untrained)Claude Code (Opus 5)
composite0.5830.564
Synthesis0.6830.747
Reasoning0.6000.527
Planning0.2760.195
Writing0.5700.561
Form0.7840.788

About 0.016 of the 0.018 gap comes from Planning. A stage can also be compared on its own: every SCRIBE stage starts from a sealed input, so we can put different models into the same planning slot with exactly the same input. On the 71 held-out tasks, the open 27B breaks outlines on large inputs (25% of tasks in the largest third), while frontier models in the same slot were never cut off in 852 replies. This is why we then trained only planning. None of this is visible from the total score or the rank.

2. Which comparisons should not be made. This step happens before any test, and a normal test never asks it:

  • Comparisons that should not be made. Among the 21 pairs of the seven published pipelines, a careful conventional analysis (subfield clustering, then Bonferroni) gives 14 orderings, and 7 of them fall in pairs the audit refuses (Table 2). Examples are LiRA > SurveyG and LiRA > LLM×MapReduce: only LiRA received the reference list, while the other two had to search for themselves.
  • Comparisons that look fair but are not. The four examples under W3/Q4 above: significant after multiplicity correction, gone once the inputs match.
  • Conditions that must be fixed in advance. AutoSurvey cannot take a length instruction and writes a median of about 250,000 words. Scored in full, it rises from 0.500 to 0.580 (Figure 3) and becomes the certified winner among the published pipelines (Table 9), by writing more, not better.

You are also right that the bound is loose. A loose bound only makes the method more conservative, with more refusals and more undecided pairs, never a false certification. And the five LiRA refusals it decides do not hinge on the constant: they hold for L_M as low as 0.554, while every estimate lies between 0.863 and 0.967 (Figure 5).

W5 / Q2. Why is truncation the primary view, and was it chosen before seeing results?

Honestly, the truncation rule was not pre-registered. We chose it because the length is set by the task, not tuned: each report is cut at the first section boundary past the length of the expert review it is compared with, and shorter reports stay whole. That length is within the normal range of human reviews (median 6,290 words over our 2,042 expert reviews, longest 54,585). Systems that take a length instruction write a median of 0.73–1.07× the target, while published pipelines that cannot take one write 2.7–47×. We report both views:

View (reference list given)Trained SCRIBEUntrained SCRIBE
matched length (primary)0.605, 1–3 of 16, 13 wins0.583, 1–7, 9 wins
full length0.608, 1–5, 11 wins0.585, 1–9, 7 wins

At full length, SCRIBE's lead over the best external system falls from 0.041 to 0.028 and its interval widens from [1, 3] to [1, 5], but it stays first by point estimate. The lead narrows not because truncation helped SCRIBE: SCRIBE itself barely moves (+0.003), since its reports are mostly shorter than the target and the cut rarely touches them. What changes is that long reports gain from their length (AutoSurvey +0.080). The known-answer control above supports the cut: when one system is asked to write twice as long, a conventional test finds 5 "different" readouts at full length (four of them coverage) and 1 after cutting.

Q1. With the same model, is SCRIBE-Luna clearly better than each published pipeline? How much of the lead comes from the model?

Yes. SCRIBE-Luna runs every SCRIBE stage on gpt-5.6-luna, the model of six of the seven published pipelines:

SCRIBE-Luna minusReference list givenShared corpus
LLM×MapReduce+0.092, certified+0.101, certified
LiRA+0.078, certified(LiRA runs only with the reference list)
SurveyGen-I+0.067, certified+0.021, not certified
AutoSurvey+0.062, certified+0.058, certified
SurveyG+0.046, certified+0.056, certified

Given the reference list, all five pipelines in the table are certified below SCRIBE-Luna, and SCRIBE-Luna is also certified above the OpenAI tool loop on the same model (+0.033). As for the model: the same harness on our 27B (0.583) and on gpt-5.6-luna (0.562) differs by 0.021, not certified. Most of the lead over published pipelines therefore holds with the model fixed.

W1. Only one benchmark, with 50 tasks.

Agreed that it is one domain, and the paper states this as a limit. The 50 test tasks span 23 subfields and certification clusters by subfield, so a different draw of tasks is accounted for. We released 1,842 training tasks so others can extend it.

W2. Formulas dominate and there are too few figures.

Agreed. Section 3 is about 30% shorter. We added an overview (Figure 1), redrew the leaderboard (Figure 2), and added four appendix figures: output length, citation support, sensitivity to the constant, and planted mismatches.

Reviewer_iC3vOpenReview ↗
Rating
1
Confidence
2
Informativeness
2(+1)
Presentation
1
Soundness
3(+2)
Read review
Paper summary

This paper introduces a method for better evaluation of literature review AI systems, demonstrated on biomedical literature. The method, compositional controllability, checks whether agents were evaluated under comparable experimental settings and certifies a ranking when the score difference exceeds statistical uncertainty. Along with that the authors introduce a benchmark dataset BioLitBench, and a literature review agent "SCRIBE". Across 50 topics, the authors report that SCRIBE scored 0.606 with supplied references and 0.487 when finding papers itself, based on how closely its output matched the claims, citations, and structure of human-written reviews.

The most novel and worthwhile contribution from the paper, in my opinion would be the BioLitBench dataset. Had the paper been structured around the dataset itself, in my opinion, that would've been a clearer contribution.

The authors have provided the dataset and the code, and in the limited time I have before the review period ends it seems to match what was state in the paper - dataset and implementation-wise. I couldn't find the hold out test set to evaluate the actual numbers on an isolated set of examples.

Critical strengths

As presented in the paper, in my opinion, the most meaningful contribution is BioLitBench. It represents biomedical articles as structured graphs containing claims, typed relations, and section hierarchies, enabling evaluation of AI generated summaries/reviews. I managed to reconstruct the claim graphs for all 50 evaluation reviews and the 399 same-topic reviews used for calibration. I didn't manage to go in further depth in the dataset since it was provided on the last day of the review period.

Critical weaknesses

I understand the motivation for tackling this issue and what the paper is about in general, but it does not clearly connect this motivation to the broader challenges of evaluating these systems in an easily comprehensible way. It introduces many ideas without a clear narrative explaining how they fit together. As a result, it reads more like a collection of technical details than a coherent argument, making the central contribution difficult to identify, except for the benchmark dataset which would be a nice contribution, had I been able to access/reconstruct it from the code.

Authors have updated the baselines to provide some additional frontier models, but I still find it somewhat lacking and non-comprehensive. I would've liked to see a wider scope of models and harnesses, although the update is a good first step. Had it not been updated in the review I would've test additional frontier models like (Astra) to see if it held up across a larger frontier.

It is still a bit unclear to me what the results mean for someone who wants to use a literature review agent. I can follow the individual components of the paper but I do not see a clear connection between the reported scores and the benefits a user should expect. What does a higher certified rank tell me about the usefulness or reliability of the resulting review? This framing around convincing me that this is important is really lackluster and hard to comprehend with how the paper presents it at the moment.

Better writing would push my overall rating to a weak rejection and I think there might be a paper somewhere in here that could clear the bar, but I think this paper leaves a lot to be desired in it's current form.

Questions for authors
  1. How do I access BioLitBench so I can validate the dataset with the code and with it the claims made in this paper?

  2. What criteria guided the selection of baseline systems and model backbones? Can you run it against some of the newer actual frontier models? If I'm not wrong GPT-5.6 Sol and Claude Opus 5 were available well before the submission deadline.

  3. Given that the certificates concern agreement with reference review, what does a higher rank tell a user about the usefulness or reliability of the resulting review?

2 comments

Thank you for saying plainly that you could not verify the data, and for singling out the claim graphs as the most valuable part. The data is now public. We also agree with your concern about SOTA hunting, and the revision changes the framing along the lines you suggested.

Q1. How do I access BioLitBench?

Data: https://huggingface.co/datasets/anonymousubmissionusr/biolitbench. Code: https://anonymous.4open.science/r/LitReview-SCRIBE-8E4E. To respect copyright, the dataset contains no article text. It contains the expert claim graphs, annotations, reference lists, and the location and checksum of every passage. A script fetches the open-access text from PMC and PubMed, rebuilds each task and checks every passage against its checksum.

W1. No clear narrative.

Agreed. The revision follows one thread: diagnose, then repair (new Figure 1). A single score mixes several stages and hides where an agent is weak. BioLitBench lets the output of each stage be scored on its own against the expert review. Before any score is read, we check that two systems received the same papers, budget and length instruction, and we certify only gaps larger than sampling noise. Our results show that planning (the outline) is the lowest of the five axes for all 16 systems, and that untrained SCRIBE (27B) breaks outlines on large inputs (25% of tasks in the largest third). So we trained only that stage, which raised planning on the test tasks (certified). The theory is also shortened.

W2 / Q2. Baselines, frontier models, and SOTA hunting.

The original published pipelines were review-writing systems whose code we could run ourselves on the same inputs, six of them on one model (gpt-5.6-luna) so that differences among them come from the scaffolds, with SCRIBE on the open Qwen3.8-27B because stage training needs open weights. In the revision we added the two models you named, GPT-5.6 Sol and Claude Opus 5, plus GPT-6.1 Sol and Claude Science + Opus 5 (Figure 2, Table 1). No frontier agent is certified above untrained SCRIBE, and Claude Code + Opus 5 is not separated from it:

SystemCompositeRank interval (of 16)Untrained SCRIBE minus this row (threshold)
SCRIBE, untrained (Qwen3.8-27B)0.5831–7—
Claude Code + Opus 50.5642–8+0.018 (0.032), not separated
Claude Science + Opus 50.5434–12+0.038 (0.032)
Tool loop + GPT-5.6 Sol0.5195–15+0.064 (0.042)
Tool loop + GPT-6.1 Sol0.5089–16+0.075 (0.040)

Bold = certified.

On SOTA hunting, you are right. The revision states that trained SCRIBE is not certified above its own untrained harness, so its rank comes mainly from the overall design, not from training.

Q3. What does a higher rank tell a user?

The new Appendix I ("What a rank tells a user") and the worked example in Appendix H answer this. Put simply, if A is certified above B, then with the same inputs A's reports contain more of what the expert wrote, including the findings, how they connect and how they are organized, and the gap is larger than a different set of tasks would produce. If the two are not separated, the gap in total score is within noise, though they may still be good at different things. If the comparison is refused, the two sides were given different inputs, so the gap cannot be credited to the step being compared.

Every readout converts back to counts. For example, untrained SCRIBE and Claude Code + Opus 5 both state about 22 of the expert's 27 central findings per task. SCRIBE reproduces more of the expert's mechanism links (16.2 vs 14.4 of 26), and its top-level sections are closer to the expert's (7.8 vs 11.6, against 6.1 for the expert). Claude Code more often keeps apart findings the expert treats as unrelated (94% vs 80%). With totals this close, a user can choose by the strengths they need, by cost, or by privacy.

Thanks for providing the code, unfortunate it is so late in the review period. As far as I can tell I can't seem to access the hold out set to verify the actual performance. The dataset matches what was described in the paper as far as I can tell and that's a big improvement to the initial submission. I would still liked to have more time to run some of the examples myself. I will update the final review according to the new information that you provided. I understand that you might've needed more time to prepare the revisions, but hopefully you can also understand that I don't have enough time to verify the actual numbers.

Thank you!

Comments on the paper

2 comments

We sincerely thank all reviewers. We have uploaded a revised PDF with changes. In brief:

  1. Four frontier agents added: Claude Code and Claude Science with Opus 5, and our tool loop with GPT-5.6 Sol and GPT-6.1 Sol, for 16 systems in total. The leaderboard tables were recomputed with them and are shown in blue.
  2. Two new kinds of refusal: comparisons between real systems that look fair and are correctly refused (Section 6.2), and planted mismatches with a known answer (Figure 6).
  3. A clean retraining of the planning stage, with three seeds (Appendix B).
  4. One thread through the paper added. Each capability is scored against the expert review on its own, which shows where a system is weak. For all 16 systems the outline (planning) is furthest from the expert, so we repair that stage alone and check whether the final report improves. A new overview figure (Figure 1) shows this thread. A worked example (Appendix H) and a plain-language table of the readouts (Appendix I) explain what a rank means for a user.

Thank you again for your time and effort in reviewing our work.

Best regards,

The Authors

Dear Area Chair and Reviewers,

Thank you for handling our submission. Leaderboards shape which literature-review agents scientists use, yet existing leaderboards only check whether a score gap beats noise, not whether two systems can be compared at all. Our theory bounds how much an input mismatch can move the score. If both systems enter the compared stage with the same inputs, the bound is zero and our test is the standard one. If not, the bound decides: refuse, or compare with a wider interval. In our planted tests this never certified a wrong winner. This changes conclusions. Half of the significant results among published pipelines are refused, and the winner was the only system given the expert’s bibliography. Scoring each stage on BioLitBench shows that the outline is the weakest stage for all 16 systems. Fixing it alone can improve the final report.

Best,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.