- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper audits the break rate that MR-DRE (Chen et al., 2026) uses for multi-turn revision of deep-research reports. The break rate is the fraction of checklist items judged covered before a revision and not covered after it. The setup uses DR-Tulu-8B turn-1 reports, a DeepSeek reviser in four edit modes, and four judges. The headline result is that shuffling paragraphs without changing a word gives a break rate of 0.107 under GPT-4.1-mini. That is higher than the real patch and regeneration arms in paired, Holm-corrected contrasts (Fig. 1). The authors call this Positional Coverage Bias. They propose a FLOOR over content-preserving controls as a detection threshold. The paper also includes a generalizability-theory analysis and a single-rater human pass. Finally, the authors' own order-invariant construction removes the position effect exactly but is more sensitive to rewording. An off-the-shelf RAGAS metric is order-invariant as well.
Critical strengths
-
Checking whether a new benchmark metric measures what it claims to measure is useful work. The main observation is also hard to ignore. A reorder that keeps every word breaks more coverage than real edits, on the same judge behind the published numbers.
-
The statistics are done carefully, with a paired question-level bootstrap, pre-declared Holm families, and a G-study. To their credit, the authors withdraw readings that don't hold up on larger samples. They also report their own construction as a negative result.
-
The point in Sec. 2 is a good one. Break rate only counts covered-to-uncovered flips, so symmetric judge noise shows up as one-sided loss. The byte-verifiable control generator is also something others can reuse.
Critical weaknesses
-
Could a simple instruction remove the effect? The paper says Positional Coverage Bias is "a property of the protocol, not of a prompt" (§4.6). The support is three read-and-check instruments that differ in format (Table 2). None of them is described as telling the judge to read the whole report carefully, or to check whether content is still present after it has been moved. If a simple instruction of this kind lets the judge recognize moved content, the failure mode would be much narrower than the paper claims.
-
Generality to newer judges remains unclear. The judges are DeepSeek-chat, Qwen3-8B, Qwen3-32B, GPT-4.1-mini, and GPT-4o. I understand that GPT-4.1-mini is the judge MR-DRE actually deployed, so it is the right audit target. Still, the title and contribution 1 describe a property of LLM coverage judges in general. The reported model versions and configurations do not establish whether the effect persists with current frontier or reasoning judges. Judge models are improving quickly. If the effect is much weaker in current models, its lasting relevance would be limited.
-
No comparative cost accounting. A finding like this is most useful when it comes with a practical fix. The paper discusses order-invariant scoring, through its own retrieve-then-judge construction and RAGAS. It also proposes a FLOOR, measured on the control battery, as a threshold for interpreting break rates. Even taken as options rather than recommendations, these add steps such as retrieval, claim decomposition, or extra scoring passes. Beyond the "one extra scoring pass" statement, the paper does not provide comparative accounting of actual judge calls or tokens for the alternatives and the full control battery. This makes it hard to judge whether they are worth the extra cost in practice.
Questions for authors
-
(W1) Could you test a simple instruction-level baseline on the identity and reorder controls? For example, the judge could be told to read the whole report carefully and to check whether each item is covered anywhere in it, regardless of position.
-
(W2) Could you run the identity and reorder controls with a few current frontier or reasoning judges? This would show whether the effect persists in current models.
-
(W3) Could you report judge calls or tokens for the order-invariant alternatives relative to the standard protocol, next to their positional and wording sensitivity in Table 2? Separately, what is the overhead of the full control battery, and what exactly does "one extra scoring pass" cover?
We thank the reviewer for a blunt and specific critique. We have acted on the central one rather than argued with it, and on one question we think the reviewer is simply right, so we adopt their framing, including the one result that goes against us.
W1. Readability, and the impression of an uncurated artifact.
Since the submission PDF can no longer be revised, we give the audit here so that the commitment is checkable rather than asserted. Three examples of what is being removed, with their replacements:
The rule we are applying is that a scope limit is stated once, in the limitations, as a fact about what the data support, and does not reappear as an apology attached to each result. The register is out of the final version in full.