- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper discusses whether scientific evaluation benchmarks can measure meaningful judgements when labels are predicted from a set candidate word distribution and rankings depend on the scoring procedure. The authors audit ResearchBench and finds that candidate text alone identifies the source hypothesis with 98% accuracy, while tested LLMs lose in pairwise source recovery. Further experiments establish context sensitivity in specific settings.
Critical strengths
- The paper convincingly shows that ResearchBench’s source labels can be recovered without scientific context: the candidate-only classifier reaches 98.40% accuracy, and simply choosing the shortest candidate achieves 79.51%.
- Its paired procedure comparisons are also carefully analyzed, separating changes in rankings from ties, parsing failures, and repeated observations.
Critical weaknesses
- The strongest context experiment leaves a plausible alternative unresolved. Notably, after presenting fifteen additional text blocks, the prompt asks the model to score “this candidate,” which could create uncertainty about the intended target.
- More broadly, the paper establishes that construction cues predict labels and that scoring procedures change rankings, but does not establish whether those cues explain the ranking failures.
Questions for authors
- Does the scalar context loss persist when the final instruction explicitly names a clearly delimited target, with the same items, donors, order, and output schema?
- Does the Isolated–Vector gap persist on length-matched source–alternative comparisons? Given the strong shortest-candidate baseline, this would help assess whether length differences account for much of the effect.
- For the high-performing contemporary judges, what happens when the background and research question are removed or shuffled? This would help distinguish successful source identification through candidate-only cues from effective use of the scientific context.
2 comments
Thank you to the authors for their detailed response. While I appreciate the follow-up note and your added analyses, they still leave my core concerns unresolved. For example, in the length-matched subset, below-chance vector recovery is no longer supported (49.66% spanning the 50% baseline) while strong text-only cues persist. The causal mechanism behind the Isolated-Vector gap also remains open. Moreover, the background/question ablation also tests only a single contemporary judge and largely reflects ties rising to 92% under shuffling - the study still doesn't validate whether higher source recovery translates to better scientific screening. Given these remaining limitations, I will be keeping my score.
Thank you for noting that the paired procedure comparisons are "carefully analyzed, separating changes in rankings from ties, parsing failures, and repeated observations." Shared points are in the General Response (S1–S4).
Explicit delimited target (Q1). Done; the context loss persists in the tested setting. We reran C and D for GPT-5.6 Sol on a hash-selected subset of 40 items (39 complete dependency groups; selection uses no model outputs). Explicitly delimiting and naming the target leaves a C−D gap of 6.42 pp (95% CI [3.42, 9.92]), compared with 7.00 pp under the original wording on the same subset; items, donors, order, and score schema are unchanged apart from target-reference wording (S2; App. I.1, Table 41, p. 54). The change between wordings, −0.58 [−2.60, 1.42] pp, is unresolved. For reference, the primary gap on all 237 items is 11.53 [9.36, 13.85] pp (§5.1, p. 7). The dual-focus study (S4) is a distinct control, not a repair of the original target wording: it identifies both focal hypotheses but differs in sample, templates, orders, focal organization, and output action (§5.2, pp. 8–9; App. F.6, pp. 37–39).
Length-matched gap (Q2). Done; the gap persists, but this subset does not support below-chance Vector recovery. Restricting source–alternative length differences to at most 10% leaves an Isolated–Vector gap of 19.32 [16.30, 22.38] pp. Vector recovery is 49.66 [47.77, 51.58]%, an interval that includes the 50% random reference (S1; App. H.1, Table 39, p. 53). This restricts saved-output comparisons: the original Vector inputs are unchanged, and other construction cues remain.
Background/question for contemporary judges (Q3). Done for one judge. For GPT-5.6 Sol, pairwise source recovery is 99.83% with intact context, 88.51% with the background and question removed, and 50.90% with them jointly shuffled; ties rise from 0.1% to 22.5% and 92.0% (S3; App. I.2, Table 42, p. 55). This supports sensitivity to the supplied context, while the shuffled result largely reflects ties. The ablation covers one contemporary judge, does not separate background from question effects, and does not establish scientific screening quality.
Remaining limitation. These analyses do not establish that construction cues cause the ranking failures. The classifier-margin strata (App. H.2, Table 40, p. 53) are observational, and the two upper strata contain only 8 and 4 relations; we report them descriptively, not as a causal explanation.
The analyses for all three of your questions are now in the revised paper (Q3 for one contemporary judge); if they resolve your concerns, we would be grateful if you would consider raising your score.