acceptodds
Under review as a conference paper at ICLR 2027

Positional Coverage Bias: LLM Coverage Judges Verify Where Content Sits, Not Whether It Is There

Abstract

LLM judges that verify whether required content is present in a long-form report do not track content — they track position. We name and measure this failure mode of coverage-style LLM-judge metrics, Positional Coverage Bias (PCB): permuting a report's paragraphs, changing not a single word, makes the judge rescind more "present" verdicts than the agent's real content edits do — 0.108 against 0.043 patching and 0.058 regeneration on the deployed metric's own judge, Holm-confirmed, and still that judge's repeat noise ([2.1, 7.7]) when the permutation is restricted to within each section, so it is not discourse scrambling. It replicates on a second agent, two datasets and three judges under the full permutation but not under the discourse-preserving one: the effect is local, and largest where reports are weakly sectioned — the regime the audited benchmark's own agent produces, and the scope we claim. Verdicts track position far more than wording, so phantom break rates of the size deployed benchmarks attribute to agents arise with no content removed, and in our own measurements the artifact dominates the signal: a generalizability decomposition leaves the revision method only 0.6% of break-rate variance across four near-equivalent edit strategies, and a blind human pass cannot separate content-preserving controls from real loss. It afflicts every judge that reads the report and checks each item — three such protocols — and is removable by construction: in a content-determined order the judge's input is byte-identical under permutation and the break falls . The removal is exact, and held to our own threshold our construction is not the metric to adopt: it is more wording-fragile than the judge it repairs, so it relocates the nuisance instead of lowering the threshold. An existing off-the-shelf order-agnostic metric has the property already, at a threshold our own interval cannot separate from the others (+0.022 [-0.038, +0.077]). We report the negative result rather than present a dominated design as a fix. What we hand the field is the threshold: an identifiability model makes it estimable from a model-free control battery before any agent is run, below which no coverage metric can certify that content was lost, and it is non-transportable, so each benchmark must measure its own. We release the control battery, the instruments, judges and ledger.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-as-a-judge, Positional coverage bias, Deep research agents, Benchmark validity, Detection thresholds

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9FWTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper audits the break rate that MR-DRE (Chen et al., 2026) uses for multi-turn revision of deep-research reports. The break rate is the fraction of checklist items judged covered before a revision and not covered after it. The setup uses DR-Tulu-8B turn-1 reports, a DeepSeek reviser in four edit modes, and four judges. The headline result is that shuffling paragraphs without changing a word gives a break rate of 0.107 under GPT-4.1-mini. That is higher than the real patch and regeneration arms in paired, Holm-corrected contrasts (Fig. 1). The authors call this Positional Coverage Bias. They propose a FLOOR over content-preserving controls as a detection threshold. The paper also includes a generalizability-theory analysis and a single-rater human pass. Finally, the authors' own order-invariant construction removes the position effect exactly but is more sensitive to rewording. An off-the-shelf RAGAS metric is order-invariant as well.

Critical strengths
  1. Checking whether a new benchmark metric measures what it claims to measure is useful work. The main observation is also hard to ignore. A reorder that keeps every word breaks more coverage than real edits, on the same judge behind the published numbers.

  2. The statistics are done carefully, with a paired question-level bootstrap, pre-declared Holm families, and a G-study. To their credit, the authors withdraw readings that don't hold up on larger samples. They also report their own construction as a negative result.

  3. The point in Sec. 2 is a good one. Break rate only counts covered-to-uncovered flips, so symmetric judge noise shows up as one-sided loss. The byte-verifiable control generator is also something others can reuse.

Critical weaknesses
  1. Could a simple instruction remove the effect? The paper says Positional Coverage Bias is "a property of the protocol, not of a prompt" (§4.6). The support is three read-and-check instruments that differ in format (Table 2). None of them is described as telling the judge to read the whole report carefully, or to check whether content is still present after it has been moved. If a simple instruction of this kind lets the judge recognize moved content, the failure mode would be much narrower than the paper claims.

  2. Generality to newer judges remains unclear. The judges are DeepSeek-chat, Qwen3-8B, Qwen3-32B, GPT-4.1-mini, and GPT-4o. I understand that GPT-4.1-mini is the judge MR-DRE actually deployed, so it is the right audit target. Still, the title and contribution 1 describe a property of LLM coverage judges in general. The reported model versions and configurations do not establish whether the effect persists with current frontier or reasoning judges. Judge models are improving quickly. If the effect is much weaker in current models, its lasting relevance would be limited.

  3. No comparative cost accounting. A finding like this is most useful when it comes with a practical fix. The paper discusses order-invariant scoring, through its own retrieve-then-judge construction and RAGAS. It also proposes a FLOOR, measured on the control battery, as a threshold for interpreting break rates. Even taken as options rather than recommendations, these add steps such as retrieval, claim decomposition, or extra scoring passes. Beyond the "one extra scoring pass" statement, the paper does not provide comparative accounting of actual judge calls or tokens for the alternatives and the full control battery. This makes it hard to judge whether they are worth the extra cost in practice.

Questions for authors
  1. (W1) Could you test a simple instruction-level baseline on the identity and reorder controls? For example, the judge could be told to read the whole report carefully and to check whether each item is covered anywhere in it, regardless of position.

  2. (W2) Could you run the identity and reorder controls with a few current frontier or reasoning judges? This would show whether the effect persists in current models.

  3. (W3) Could you report judge calls or tokens for the order-invariant alternatives relative to the standard protocol, next to their positional and wording sensitivity in Table 2? Separately, what is the overhead of the full control battery, and what exactly does "one extra scoring pass" cover?

Reviewer_nAHnOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper propose LLM-based coverage metrics for multi-turn deep-research report revision. It identifies Positional Coverage Bias: reordering paragraphs without removing any content can cause a coverage judge to mark previously covered checklist items as missing. Across several judges and settings, the reported break rate from paragraph reordering is often comparable to or larger than that from actual report revision. The paper proposes measuring a metric-specific “FLOOR” using content-preserving perturbations, arguing that revision break rates below this level should not be interpreted as verified content loss. It also evaluates order-invariant alternatives and finds that removing positional sensitivity may introduce sensitivity to wording instead.

Critical strengths
  • The experiment is simple, compelling, and highly relevant: word-identical paragraph reordering should be a standard robustness check for coverage-based evaluation.
  • The paper makes a useful practical contribution. Benchmark designers should measure judge robustness to content-preserving perturbations before interpreting revision break rates as content loss.
Critical weaknesses
  • The paper is very hard to read, and very likely to be an auto-research artifact rather than a human curated paper. It contains many intermediate caveats and withdrawn interpretations, such as 67×"rather than" and 5דwe do not claim” in the paper.
  • The paper treats paragraph order as purely irrelevant to coverage. This is not fully justified: reordering can reduce discourse coherence, local context, and retrievability even when every word remains present. The within-section experiments help, but the full reorder should not automatically be interpreted as pure judge noise.
  • Several key analyses are based on relatively small samples, especially the second-agent replication, human adjudication, and comparisons among alternative metrics.
Questions for authors
  • Why should coverage be strictly order-invariant?
  • Why should full paragraph reorder define the same FLOOR used for revision methods that do not reorder paragraphs? Would separate wording, within-section, and cross-section robustness floors be more appropriate?
  • Can you add positive controls with known, deliberate content deletion? This would show whether the metric remains sensitive and calibrated for genuine loss.
  • Can you expand the human evaluation beyond one rater and 76 flagged cases, with multiple blinded annotators and inter-rater agreement?
6 comments

We thank the reviewer for a blunt and specific critique. We have acted on the central one rather than argued with it, and on one question we think the reviewer is simply right, so we adopt their framing, including the one result that goes against us.

W1. Readability, and the impression of an uncurated artifact.

Since the submission PDF can no longer be revised, we give the audit here so that the commitment is checkable rather than asserted. Three examples of what is being removed, with their replacements:

  1. Figure caption: "What a coverage break rate is made of, and what we decline to claim from it." becomes "What a coverage break rate is made of." A scope limit belongs in the text, not in the title of a figure.
  2. "The two biases run in opposite directions and the second dominates, so we do not claim a net direction for BRK_true and report every floor-corrected residual as an interval." becomes "The two biases run in opposite directions and the second dominates, so every floor-corrected residual is reported as an interval."
  3. "the additional break of a full reorder, which at 25 questions we declined to claim, is measurable at n=65" becomes "the additional break of a full reorder, not separable from zero at 25 questions, is measurable at n=65".

The rule we are applying is that a scope limit is stated once, in the limitations, as a fact about what the data support, and does not reappear as an apology attached to each result. The register is out of the final version in full.

The reviewer scores Presentation 1, and on the evidence of this review that is the score we take most seriously: four of the points raised are addressed in the submission, and if they did not come across, the failure is ours in where we put them rather than a disagreement about substance. We give the locations first, then what is genuinely new, including three experiments run during the discussion period.

point raisedwhere it stands in the submission
separate wording / within-section / cross-section floorsTable 1 prints two floors as separate rows, "FLOOR, all controls" and "FLOOR, order-preserving only" (0.040, 0.051, 0.023, 0.048 across the four judges). Section 3 prescribes which arms are read against which. Appendix F works it through arm by arm, including the cell where it costs us.
why coverage should be order-invariantSection 3, "Why order-invariance is the right requirement", which states both of our own counter-arguments in the main text rather than in an appendix.
a positive control with deliberate deletionTable 3: deleting the most content-bearing paragraph breaks 0.099 against 0.028 repeat noise.
the scope of the human studyAppendix E states one rater, n=47 control and n=29 real, the power calculation, and names a second blind rater with an inter-rater kappa as the next step.

W2 / Q1. Why should coverage be strictly order-invariant? Reordering reduces coherence and retrievability.

We agree a reorder is not costless to a reader, which is why Section 3 states both pieces of our own evidence for it in the main text rather than in an appendix: net coverage does fall under the reorder (-0.042 and -0.025 under two judges), and a blind human rater declines "present" on 23.7% of provably word-preserving flips, which the paper subtracts as an error floor rather than ignoring.

The reviewer is raising a mechanism question, and it is testable. If the breaks were caused by lost coherence or lost retrievability, then how far an item's supporting material moved should predict whether that item breaks. It does not. Across the permuted corpus, neither absolute displacement (+0.135 [-1.02, +1.41], p=.816), nor post-permutation depth (+0.049 [-0.024, +0.120], p=.186), nor whether the evidence lands in the top of the report or the rest (-0.022 [-0.064, +0.031], p=.351) predicts breaking. A coherence account predicts all three. None holds.

The second test is already in the control set. The within-section permutation leaves every section boundary, every heading and every section's contents intact, and moves paragraphs only inside their own section. It still breaks at 2.6 times the judge's own repeat noise. Whatever a reader loses to a full reorder, that is not what this is measuring.

So the narrow claim stands. The benchmarks that deploy this metric read its output as content presence, and the checklist asks whether an item is addressed, not where. Order-invariance is the desideratum implied by the metric's own use. That a reorder also costs a reader something is a reason to report both numbers, which Section 3 does.

Reviewer_BTguOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper identifies Positional Coverage Bias (PCB): LLM judges of content coverage respond to where information sits in a document, not only whether it is present. The setting is multi-turn deep-research agents, where break rate is used to measure loss of previously covered content during report revision. Using content-preserving perturbations, the authors show that reordering paragraphs without changing their text yields a higher break rate (0.107) than real revisions: 0.047 for constrained patching and 0.070 for full regeneration. They propose an instability floor, the maximum break rate across content-preserving transformations. Observed losses below this floor cannot be separated from evaluator artifacts. Experiments span several judges, datasets, and protocols, with paired bootstrap CIs, Holm correction, and Generalizability Theory. An order-invariant retrieve-then-judge variant removes the positional effect by construction but becomes more sensitive to wording, leaving a floor comparable to the original. A RAGAS-based metric is also tested. The contribution is mainly diagnostic: reported coverage regressions may partly reflect the evaluation procedure rather than the agents.

Critical strengths
  1. The core finding is simple and consequential. Reordering paragraphs alone produces a higher break rate than actually revising the report. This directly questions previously reported regression rates for deep-research agents, and anyone using break rate as an evaluation metric needs to account for it.
  2. The statistical treatment is careful. Paired bootstrap CIs, Holm correction, and G-theory are used, and negative results are reported honestly. The retrieve-then-judge fix trades positional sensitivity for lexical sensitivity, and the authors say so rather than overselling it. The instability floor gives practitioners a concrete check, even if it needs further validation (see weaknesses).
Critical weaknesses
  1. The instability floor has not been validated as a detector. It is defined as the maximum break rate over a chosen set of perturbations. It therefore depends on which perturbations are included, carries its own sampling variability, and is dataset- and judge-specific by the authors’ own account. The paper never checks it against known ground truth, such as controlled deletion of specific content or human-annotated losses. Without that check, we don’t know whether a regression above the floor is real, or whether real losses can fall below it. 2. For a measurement paper, this is the central gap. The evidence base is narrow, and the cause of the effect is uncertain. The main revision experiments use a single reviser model and few research questions. More importantly, the discourse-preserving within-section permutation does not replicate consistently across agents. That leaves open whether the judges respond to position as such or to the coherence disruption that full paragraph reordering causes. If it is the latter, reordering is not entirely content-preserving from a reader’s standpoint, and part of the 0.107 may be a legitimate penalty.
Questions for authors

Could change my assessment:

  1. With synthetic deletions of known content at varying rates, does the break rate rise above the instability floor, and at what deletion rate does it become detectable?
  2. When the within-section permutation fails to replicate, what distinguishes those agents’ reports? Can you separate the effect of position from the effect of disrupted coherence?
  3. How sensitive is the floor to which perturbations are included? For example, what happens if you take the max over a subset, or use a high quantile instead of the max?
  4. How does PCB differ from known positional and order sensitivity in LLM judges (e.g., position bias in pairwise comparisons, lost-in-the-middle)? What is new beyond applying these to coverage judging?

Suggestion (not affecting score): a second reviser model and more research questions would make the results more general. An evaluation method robust to both positional and lexical variation would be a natural follow-up, but I don’t think this paper needs to provide one.

1 comment

Q4. How does positional coverage bias differ from known positional and order sensitivity?

The empirical separation is in the paper, but we accept that it is too deeply buried, sitting in the appendix rather than beside the claim it supports. Lost-in-the-middle predicts that whether an item breaks should depend on where that item ends up. It does not. Neither post-permutation depth (+0.049 [-0.024, +0.120], p=.186), nor absolute displacement (+0.135 [-1.02, +1.41], p=.816), nor top-versus-rest position (-0.022 [-0.064, +0.031], p=.351) predicts which items break.

The increment we claim is at the level of measurement rather than of phenomenon. That judge scores shift under surface and order changes is established, and we cite that literature. What is new is that in coverage verification the positional artifact out-magnitudes the content effect the metric exists to measure, since a word-identical permutation breaks more than the agent's real edits on the metric's own judge, and that this comes with an estimable detection threshold. In the final version this contrast moves into the main text beside the claim.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.