- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work turns PubPeer comments, in which researchers raise questions about papers, into a benchmark, PubPeerBench-1000, of 1000 comments with claims that are mapped to evidence within the paper. Concretely, the question is whether models agree with expert humans in assessing a comment's validity and impact, given a comment and the relevant evidence. In addition to the benchmark, a finetuned Qwen 3.6 27B is shown to beat out many SoTA models, including GPT-6 Astra, on PubPeerBench.
Critical strengths
This work deals with a problem that is of high interest to the community. The paper is comprehensive: the benchmark consists of 1000 questions, many SoTA language models are tested, and a finetuned 27B parameter model is able to beat out GPT-6 Astra on PubPeerBench. I found the paper to be easy to follow, the figures well-made, the ablations to be interesting.
Critical weaknesses
PubPeerBench tests exact agreement on evidence support (supported / partially supported / unsupported / insufficient evidence) and impact (none / minor / moderate / major / unknown). This raises some concern, since the distinction between categories such as supported / partially supported or minor / moderate / major can be subjective. This is especially relevant given the small annotator pool of two experts, with a third only adjudicating disagreements. Coupled with the relatively small differences between models, and the fact that the finetuned model was trained on labels from the same annotators and may therefore have learned their preferences, this makes it harder to assess how meaningful the reported improvements are.
Additionally, another concern is why the model is given a preselected evidence bundle rather than the full paper. The 65k context window seems large enough, with the largest current input being only 36k tokens. This could matter if, for example, a valid concern mistakenly refers to Fig. 4 instead of Fig. 5, causing the relevant evidence to be missed. It would be interesting to see how giving models the full paper would change the results.
Questions for authors
- How robust are the results to the subjectivity of the labels, i.e. how would the results change if instead of exact match, partially supported and fully supported were grouped into one category? Also, could the finetuned model partly be learning the annotators’ preferences?
- How would giving the entire paper as input change the results?
3 comments
Author Response (Continued, Weakness 2, Question 2)
[W2] Additionally, another concern is why the model is given a preselected evidence bundle rather than the full paper. The 65k context window seems large enough, with the largest current input being only 36k tokens. This could matter if, for example, a valid concern mistakenly refers to Fig. 4 instead of Fig. 5, causing the relevant evidence to be missed. It would be interesting to see how giving models the full paper would change the results.
[Q2] How would giving the entire paper as input change the results?
[A2] Thank you for this valuable suggestion. We use preselected evidence bundles for two reasons. First, the benchmark is designed to evaluate verification given matched evidence, separately from retrieval. Second, the annotators judge from the same evidence bundle, which keeps the expert labels and the model inputs aligned. The full texts (excluding supplementary material) of the papers for all 200 test cases fit within the 65,536-token context window.
We agree that the full-paper setting is important, and that mis-referenced figures are a realistic failure mode of location-first retrieval. For concerns with an explicit location, the evidence bundle contains the complete rendered page of the located figure or table, so a mis-referenced figure on the same page is still included; however, the bundle may miss evidence on other pages. We therefore added a full-text input condition on all 200 test cases (WorkBuddy-27B was not retrained for full-text input):
| Condition (n = 200) | Verdict exact agreement | Macro-F1 | EQ MAE | Impact exact agreement |
|---|---|---|---|---|
| Evidence bundle (WorkBuddy-27B) | 48.00% | 37.00% | 0.90 | 39.50% |
| Full text (WorkBuddy-27B) | 44.50% | 36.12% | 0.95 | 41.50% |
| Evidence bundle (GPT-6 Astra) | 46.50% | 34.38% | 0.98 | 33.50% |
| Full text (GPT-6 Astra) | 42.00% | 32.50% | 1.05 | 33.00% |
In the settings we tested, full-text input did not bring an overall improvement. For WorkBuddy-27B, verdict exact agreement drops from 48.00% to 44.50%, Macro-F1 from 37.00% to 36.12%, and EQ MAE rises from 0.90 to 0.95; only impact exact agreement rises, from 39.50% to 41.50%. A fairer reference is GPT-6 Astra, which was not trained on the task: it also drops on verdict and Macro-F1 (46.50% → 42.00%, 34.38% → 32.50%). These differences are small. For WorkBuddy-27B, verdict differs by 7 cases and could be significant only if no more than 9 cases are discordant between the two conditions; impact differs by 4 cases and is not significant under any overlap (p ≥ 0.125). For GPT-6 Astra, verdict differs by 9 cases. We therefore conclude that full-text input brings no visible improvement. One possible reason is that irrelevant pages in the full text dilute the evidence related to the concern, and locating the questioned figure or table in the full text is itself an additional retrieval step. We keep the evidence bundle as the main setting.
For future work, we plan to extend the system to take the full paper directly as input: the model would first automatically extract a concern-related evidence bundle from the full text (locating the cited figures and tables, adjacent pages, and supplementary material), and then perform verification. Current long-context multimodal models already have the basic capability for this step, but the results above also show that finding the right evidence in the full text is not trivial. We will therefore first evaluate the evidence-extraction step on its own, for example by reporting the recall of extracted pages against expert-selected pages, and then report end-to-end verification results.
We hope these new results answer your questions.
Thank you for your efforts, my concerns have been clarified. I think the new experiments should be included in the paper or the appendix, as they seem to strengthen the paper's contributions.
Author Response (Weakness 1, Question 1)
Dear Reviewer j2MD,
Many thanks for your positive assessment of the paper's clarity, figures, and ablation experiments, and for your thoughtful questions. We respond point by point below.
[A1] Thank you for these insightful comments and questions. We answer in two parts.
(i) Coarse-grained labels. Following your suggestion, we merged
supportedandpartially_supportedinto one class, and collapsed impact into three levels, {none/minor, moderate/major, unknown}:Merging can only turn confusions between
supportedandpartially_supportedinto correct predictions, so 3-class exact agreement equals 4-class exact agreement plus the share of such confusions among all 200 cases. WorkBuddy-27B has 24 such confusions (23.08% of its 104 verdict errors), so its 3-class exact agreement is48.00% + 24/200 = 60.00%; the base model and GPT-6 Astra have 16 and 21, respectively. After merging, the gain of WorkBuddy-27B over the base model is 13.50 percentage points for verdict (9.50 in the 4-class setting) and 16.50 points for impact (16.00 with the original five levels). The impact gain is significant under any overlap (exact McNemar p ≤ 0.05698 for verdict and p ≤ 0.00994 for impact). The model ranking matches the 4-class setting: WorkBuddy-27B is highest on all three metrics, followed by GPT-6 Astra and then the base model.(ii) Annotator pool and annotator preferences. Each case is independently annotated by two experts. Before adjudication, κ is 0.89 for verdict and 0.92 for impact, and the weighted κ for evidence quality, the most fine-grained field, is 0.59. We agree that high agreement does not rule out conventions shared by the two annotators; moreover, training and test labels follow the same protocol, so our results measure agreement with this protocol (Appendix B). However, three observations limit how much of the gain "annotator preferences" can explain. First, the gain depends on paper evidence: without evidence (claim_only), the fine-tuned model falls to the level of the majority-class baseline (exact agreement 38.50%, Macro-F1 13.90%); with evidence from another paper (mismatched), it falls to 25.50% (96 vs. 51 correct, p ≤ 0.00026 under any overlap). Second, GPT-6 Astra, which was never trained on our labels, answers 93/200 correctly against the same gold labels, close to WorkBuddy-27B's 96/200, which shows that most of the agreement is attainable without exposure to the annotators. Third, we evaluated WorkBuddy-27B against each annotator's independent labels instead of the adjudicated labels. Its verdict agreement with Annotators 1 and 2 is 45.50% and 47.00%, and its impact agreement is 40.50% and 38.50%, close to its results against the adjudicated labels (48.00% and 39.50%). If WorkBuddy-27B were mainly fitting one annotator's habits, its agreement with that annotator should be clearly higher than with the other. This is not what we observe: the differences between its agreement with the two annotators are only 1.5 and 2.0 percentage points (3 and 4 cases), which are not significant under any prediction overlap (paired exact McNemar test, p ≥ 0.25 and p ≥ 0.125), and they point in opposite directions: verdict leans slightly toward Annotator 2, and impact slightly toward Annotator 1.