- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to fairly compare multi-stage agents when they differ in inputs, resources, or intermediate components. The authors propose compositional controllability, which bounds the discrepancy between observed performance differences and controlled comparisons using nuisance factors outside a selected comparison window. This framework leads to admissibility tests and certified partial rankings. The paper also introduces BioLitBench, a benchmark for biomedical literature review agents, and SCRIBE, an agent trained using stage-level rewards. Experiments show that many statistically significant comparisons between existing pipelines are not admissible under the proposed framework, and that SCRIBE performs competitively against existing systems.
Critical strengths
- The paper studies important evaluation problem. The paper highlights that statistically significant performance differences do not necessarily imply fair comparisons when systems receive different inputs or resources. I appreciate the distinction between comparing entire systems and attributing performance differences to individual components. The experiments on published pipelines provide a concrete demonstration of this issue.
- The authors construct a large-scale benchmark and evaluate multiple published and commercial agents. The controlled comparisons and evidence-ablation experiments provide useful insights into the limitations of existing agent evaluation practices.
Critical weaknesses
- In the main experiments, inputs are matched, so certification essentially reduces to conventional statistical testing. When inputs differ, the nuisance bounds are often loose, and the empirically estimated sensitivity constants do not necessarily provide rigorous upper bounds. It is unclear how much the proposed framework improves upon standard controlled evaluation practices.
- Although stage-level training is presented as a major contribution, only the planning stage is trained, and its improvement over the untrained harness is not statistically certified. The strong leaderboard results therefore seem to demonstrate the effectiveness of the overall agent design rather than the proposed training method.
Questions for authors
- Can the authors demonstrate settings where the proposed framework provides meaningful certification beyond matched-input evaluation and conventional statistical testing? How reliable are the empirical nuisance bounds?
- Can the authors provide stronger evidence that stage-level training improves performance, independently of the overall SCRIBE architecture?
- The evidence-ablation experiments suggest that some benchmark metrics improve even without supporting evidence. To what extent do these metrics capture the intended literature-review capabilities rather than similarity to reference reports?
2 comments
W2 / Q2. Can you show that stage-level training helps independently of the SCRIBE architecture?
The training effect is already measured with the architecture fixed: both arms run the same SCRIBE pipeline, synthesis input and writer, and differ only in the planning (outline) module. To check that the trained stage still helps outside SCRIBE's own downstream, we also had the last step, turning the outline into a report, done by another model, gpt-5.6-luna (the model of six published pipelines). The two arms differ only in whether the outline came from the untrained or the trained planner, and the decision rule was fixed before generation. Each cell is the composite gain of the trained over the untrained planner, with 95% intervals in brackets:
| Our 27B writes the report | gpt-5.6-luna writes the report | |
|---|---|---|
| 71 held-out tasks | +0.021, +0.024, +0.018 (three seeds, first two certified) | +0.028 [+0.009, +0.047], certified (three seeds pooled: +0.020 [+0.002, +0.038]) |
| 50 test tasks | +0.012, not certified (planning +0.082, certified) | +0.022 [+0.001, +0.043], certified (the paper row's planner: +0.016 [+0.003, +0.030]) |
Where the gain comes from when gpt-5.6-luna writes the report:
| Tasks | Outline from | Composite | Planning | Broken outlines |
|---|---|---|---|---|
| 71 held-out | untrained planner | 0.570 | 0.252 | 9 / 71 |
| 71 held-out | trained planner (seed 17) | 0.598 | 0.342 | 2 / 71 |
| 50 test | untrained planner | 0.571 | 0.243 | 0 / 50 |
| 50 test | trained planner (seed 17) | 0.593 | 0.325 | 1 / 50 |
On the held-out tasks the gain comes mainly from repairing broken outlines (9 down to 2). On the test tasks no untrained outline was broken, and the gain comes from better outlines (planning 0.243 → 0.325). So with a different downstream writer, the trained stage still improves the final report.
The recipe does not depend on SCRIBE either: the teacher sees the expert's top-level titles while drafting, and the student sees no hint at run time, so it applies to any pipeline whose stage output can be scored on its own. You are right that the strong leaderboard result mainly reflects the overall design, and Section 6.1 now says so.
Q3. Do the metrics capture literature-review ability, or similarity to the reference report?
They measure a specific ability: given a set of papers, stating the findings those papers support that an expert judged important, and linking and organizing them as the expert did. The reference review fixes which findings are worth stating, not the wording:
- Matching is by meaning, not shared words. Four of the six synthesis readouts are stratified by whether the cited abstracts support the claim, so the score comes from stating what the papers actually support.
- The scale is not set by one review: each task has 4–9 other expert reviews on the same topic, and 0 is their 10th percentile. Style readouts such as length or section count give full marks anywhere within the other experts' range (Appendix I).
- Beyond coverage, the readouts check whether findings the expert kept apart are lumped together, and whether each sentence cites a paper that supports it.
These readouts are of the same kind as common metrics in existing benchmarks. What differs is how they are scored:
| What we measure | Similar metrics in existing benchmarks | How we score it |
|---|---|---|
| stating the findings an expert judged important | SurGE reference recall, ResearchQA coverage | against the expert's claim graph for the same papers, with the scale set by several same-topic experts |
| section structure | SurGE heading soft recall | agreement with the expert's top-level sections |
| whether citations support sentences | NLI-based citation recall and precision in AutoSurvey and SurGE | the new citation-support check (Figure 4), plus whether a finding cites the paper the expert cited for it |
| links between findings | rarely measured | new: seven relation types between the expert's claims, e.g. mechanism and contrast |
AutoSurvey, SurveyForge and others score with LLM judges, which measures an LLM's preferences. Our scoring is deterministic and uses no LLM judge. It measures the expert's choices: which findings are worth stating, how they connect, and how they are organized.
On the evidence ablation: a writer given no papers still scores well on some content readouts, because a strong model knows established findings. But it scores clearly lower on writing (−0.071), because writing checks whether each claim cites its own paper. The revision therefore states that content readouts measure agreement with the expert's content, not use of the supplied evidence. The latter is measured by the evidence-agreement readout and the new citation-support check. Swapping 60% of citations for other papers from the same list lowers citation support by about 32 points.
Thank you for separating two questions: comparing whole systems, and attributing a difference to one component. The revision is rewritten around exactly this distinction.
W1 / Q1. With matched inputs certification reduces to a standard test. Can the framework give meaningful certification beyond that, and how reliable are the empirical bounds?
On the overall ranking and composite score of the main leaderboard, the framework indeed adds nothing: with matched inputs, certification is the standard test, by design. What it adds does not come from a different test but from what is compared and whether it may be compared. First, the output of each stage is scored on its own against the expert review, so systems the composite cannot separate can be taken apart stage by stage. For example, untrained SCRIBE and Claude Code (Opus 5) differ by only 0.018 and are not separated, yet about 0.016 of that comes from Planning, while on Synthesis Claude Code is ahead (0.747 vs 0.683). Second, it decides before any test which comparisons may be made. When inputs differ, that step gives conclusions a standard test cannot.
Comparisons that look fair but should not be made. In the shared-corpus evaluation every system faces the same corpus, searches it itself, and writes to the same length, a common design. Take Claude Code (Sonnet 5) and Claude Science (Opus 5): on the five axes after retrieval, Claude Code is 0.032 higher, still significant after multiplicity correction, and the usual conclusion would be that Claude Code writes better. But these axes measure writing after the papers are in hand, and the two systems searched separately and entered writing with different papers, so the audit refuses that conclusion before any score is read. Our main leaderboard gives both systems the same reference list, and there the same five axes differ by only 0.005, so the 0.032 was not a difference in writing. The table lists this example and three more of the same kind (gaps are first minus second):
Planted mismatches with a known answer (Figure 6). One system is compared with copies of itself that differ in one way each (double retrieval budget, a citation tool that can reach the expert's references, a doubled target length, 30% of input papers removed, 30 unrelated papers added, a cutoff one year later), so the two sides are equally capable. The audit refused all six before scoring and admitted two identical re-runs as controls. A conventional test was fooled on single readouts: with double budget it called six retrieval readouts "better", and with doubled length it found five readouts different at full length. It also missed the citation tool, although on 10 tasks that tool raised the share of the expert's references in the input from 23% to 72%.
Comparisons that should not be made at all. Among the 21 pairs of the seven published pipelines, a careful conventional analysis gives 14 orderings, and 7 fall in pairs the audit refuses (Table 2), e.g. LiRA > SurveyG, where only LiRA received the reference list.
Reliability of the empirical bounds. You are right that estimated constants are not rigorous upper bounds. The conclusions do not hinge on their exact value: