acceptodds
Under review as a conference paper at ICLR 2027

Student Geng’s WorkBuddy: Evidence-Grounded Verification of Scientific Concerns

Abstract

Post publication peer review supports scientific self-correction, but systematic, reproducible verification of public scientific concerns still relies heavily on manual effort. PubPeer is an open platform where researchers question published papers, share supporting material, and discuss the reliability of the work. It has consequently become a large collection of research concerns. Recently, Hongwei Geng, also known as Student Geng, used public PubPeer comments and related evidence to examine data anomalies in papers from leading journals, such as *Nature*. The resulting cases led to investigations and sanctions by organizations such as the National Natural Science Foundation of China. Such cases show that PubPeer comments provide scientifically and practically valuable leads for research scrutiny, but remain unverified and require substantial manual validation. To address this problem, we propose Student Geng's WorkBuddy-27B, an evidence verification assistant for PubPeer comments. For each comment and corresponding paper, WorkBuddy first reformulates the scientific concern as a neutral statement, removing subjective or emotional language while preserving verifiable factual claims. It then maps the concern to traceable evidence in paper text, figures, methods, supplementary materials, and source data. WorkBuddy assesses how strongly this evidence supports the concern, together with evidence quality, potential impact, and evidence location. For training and evaluation, we construct PubPeerBench-1000 with 1,000 cases annotated by experts. We train Student Geng’s WorkBuddy through supervised fine-tuning and compare it on an independent test set with leading large language models, including GPT-6 Astra. Against untuned Qwen3.6-27B, our model's verdict exact agreement rises from 38.50% to 48.00%, Macro-F1 from 28.13% to 37.00%, EQ MAE falls from 1.50 to 0.89, and impact exact agreement rises from 23.50% to 39.50%, showing the effectiveness of task specific fine tuning. *PubPeerBench-1000 and code will be publicly released.*

Primary area:
datasets and benchmarks
Keywords:
Scientific Concern Verification, Evidence-Grounded Verification, Post Publication Peer Review, WorkBuddy, PubPeerBench-1000

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_j2MDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work turns PubPeer comments, in which researchers raise questions about papers, into a benchmark, PubPeerBench-1000, of 1000 comments with claims that are mapped to evidence within the paper. Concretely, the question is whether models agree with expert humans in assessing a comment's validity and impact, given a comment and the relevant evidence. In addition to the benchmark, a finetuned Qwen 3.6 27B is shown to beat out many SoTA models, including GPT-6 Astra, on PubPeerBench.

Critical strengths

This work deals with a problem that is of high interest to the community. The paper is comprehensive: the benchmark consists of 1000 questions, many SoTA language models are tested, and a finetuned 27B parameter model is able to beat out GPT-6 Astra on PubPeerBench. I found the paper to be easy to follow, the figures well-made, the ablations to be interesting.

Critical weaknesses

PubPeerBench tests exact agreement on evidence support (supported / partially supported / unsupported / insufficient evidence) and impact (none / minor / moderate / major / unknown). This raises some concern, since the distinction between categories such as supported / partially supported or minor / moderate / major can be subjective. This is especially relevant given the small annotator pool of two experts, with a third only adjudicating disagreements. Coupled with the relatively small differences between models, and the fact that the finetuned model was trained on labels from the same annotators and may therefore have learned their preferences, this makes it harder to assess how meaningful the reported improvements are.

Additionally, another concern is why the model is given a preselected evidence bundle rather than the full paper. The 65k context window seems large enough, with the largest current input being only 36k tokens. This could matter if, for example, a valid concern mistakenly refers to Fig. 4 instead of Fig. 5, causing the relevant evidence to be missed. It would be interesting to see how giving models the full paper would change the results.

Questions for authors
  1. How robust are the results to the subjectivity of the labels, i.e. how would the results change if instead of exact match, partially supported and fully supported were grouped into one category? Also, could the finetuned model partly be learning the annotators’ preferences?
  2. How would giving the entire paper as input change the results?
3 comments
AuthorsOpenReview ↗

Author Response (Weakness 1, Question 1)

Dear Reviewer j2MD,

Many thanks for your positive assessment of the paper's clarity, figures, and ablation experiments, and for your thoughtful questions. We respond point by point below.


[W1] PubPeerBench tests exact agreement on evidence support (supported / partially supported / unsupported / insufficient evidence) and impact (none / minor / moderate / major / unknown). This raises some concern, since the distinction between categories such as supported / partially supported or minor / moderate / major can be subjective. This is especially relevant given the small annotator pool of two experts, with a third only adjudicating disagreements. Coupled with the relatively small differences between models, and the fact that the finetuned model was trained on labels from the same annotators and may therefore have learned their preferences, this makes it harder to assess how meaningful the reported improvements are.

[Q1] How robust are the results to the subjectivity of the labels, i.e. how would the results change if instead of exact match, partially supported and fully supported were grouped into one category? Also, could the finetuned model partly be learning the annotators’ preferences?

[A1] Thank you for these insightful comments and questions. We answer in two parts.

(i) Coarse-grained labels. Following your suggestion, we merged supported and partially_supported into one class, and collapsed impact into three levels, {none/minor, moderate/major, unknown}:

ModelVerdict exact agreement (3-class)Macro-F1 (3-class)Impact exact agreement (3-level)
GPT-6 Astra57.0042.3145.50
Qwen3.6-27B (base)46.5041.0030.50
WorkBuddy-27B60.0044.9347.00

Merging can only turn confusions between supported and partially_supported into correct predictions, so 3-class exact agreement equals 4-class exact agreement plus the share of such confusions among all 200 cases. WorkBuddy-27B has 24 such confusions (23.08% of its 104 verdict errors), so its 3-class exact agreement is 48.00% + 24/200 = 60.00%; the base model and GPT-6 Astra have 16 and 21, respectively. After merging, the gain of WorkBuddy-27B over the base model is 13.50 percentage points for verdict (9.50 in the 4-class setting) and 16.50 points for impact (16.00 with the original five levels). The impact gain is significant under any overlap (exact McNemar p ≤ 0.05698 for verdict and p ≤ 0.00994 for impact). The model ranking matches the 4-class setting: WorkBuddy-27B is highest on all three metrics, followed by GPT-6 Astra and then the base model.

(ii) Annotator pool and annotator preferences. Each case is independently annotated by two experts. Before adjudication, κ is 0.89 for verdict and 0.92 for impact, and the weighted κ for evidence quality, the most fine-grained field, is 0.59. We agree that high agreement does not rule out conventions shared by the two annotators; moreover, training and test labels follow the same protocol, so our results measure agreement with this protocol (Appendix B). However, three observations limit how much of the gain "annotator preferences" can explain. First, the gain depends on paper evidence: without evidence (claim_only), the fine-tuned model falls to the level of the majority-class baseline (exact agreement 38.50%, Macro-F1 13.90%); with evidence from another paper (mismatched), it falls to 25.50% (96 vs. 51 correct, p ≤ 0.00026 under any overlap). Second, GPT-6 Astra, which was never trained on our labels, answers 93/200 correctly against the same gold labels, close to WorkBuddy-27B's 96/200, which shows that most of the agreement is attainable without exposure to the annotators. Third, we evaluated WorkBuddy-27B against each annotator's independent labels instead of the adjudicated labels. Its verdict agreement with Annotators 1 and 2 is 45.50% and 47.00%, and its impact agreement is 40.50% and 38.50%, close to its results against the adjudicated labels (48.00% and 39.50%). If WorkBuddy-27B were mainly fitting one annotator's habits, its agreement with that annotator should be clearly higher than with the other. This is not what we observe: the differences between its agreement with the two annotators are only 1.5 and 2.0 percentage points (3 and 4 cases), which are not significant under any prediction overlap (paired exact McNemar test, p ≥ 0.25 and p ≥ 0.125), and they point in opposite directions: verdict leans slightly toward Annotator 2, and impact slightly toward Annotator 1.


AuthorsOpenReview ↗

Author Response (Continued, Weakness 2, Question 2)

[W2] Additionally, another concern is why the model is given a preselected evidence bundle rather than the full paper. The 65k context window seems large enough, with the largest current input being only 36k tokens. This could matter if, for example, a valid concern mistakenly refers to Fig. 4 instead of Fig. 5, causing the relevant evidence to be missed. It would be interesting to see how giving models the full paper would change the results.

[Q2] How would giving the entire paper as input change the results?

[A2] Thank you for this valuable suggestion. We use preselected evidence bundles for two reasons. First, the benchmark is designed to evaluate verification given matched evidence, separately from retrieval. Second, the annotators judge from the same evidence bundle, which keeps the expert labels and the model inputs aligned. The full texts (excluding supplementary material) of the papers for all 200 test cases fit within the 65,536-token context window.

We agree that the full-paper setting is important, and that mis-referenced figures are a realistic failure mode of location-first retrieval. For concerns with an explicit location, the evidence bundle contains the complete rendered page of the located figure or table, so a mis-referenced figure on the same page is still included; however, the bundle may miss evidence on other pages. We therefore added a full-text input condition on all 200 test cases (WorkBuddy-27B was not retrained for full-text input):

Condition (n = 200)Verdict exact agreementMacro-F1EQ MAEImpact exact agreement
Evidence bundle (WorkBuddy-27B)48.00%37.00%0.9039.50%
Full text (WorkBuddy-27B)44.50%36.12%0.9541.50%
Evidence bundle (GPT-6 Astra)46.50%34.38%0.9833.50%
Full text (GPT-6 Astra)42.00%32.50%1.0533.00%

In the settings we tested, full-text input did not bring an overall improvement. For WorkBuddy-27B, verdict exact agreement drops from 48.00% to 44.50%, Macro-F1 from 37.00% to 36.12%, and EQ MAE rises from 0.90 to 0.95; only impact exact agreement rises, from 39.50% to 41.50%. A fairer reference is GPT-6 Astra, which was not trained on the task: it also drops on verdict and Macro-F1 (46.50% → 42.00%, 34.38% → 32.50%). These differences are small. For WorkBuddy-27B, verdict differs by 7 cases and could be significant only if no more than 9 cases are discordant between the two conditions; impact differs by 4 cases and is not significant under any overlap (p ≥ 0.125). For GPT-6 Astra, verdict differs by 9 cases. We therefore conclude that full-text input brings no visible improvement. One possible reason is that irrelevant pages in the full text dilute the evidence related to the concern, and locating the questioned figure or table in the full text is itself an additional retrieval step. We keep the evidence bundle as the main setting.

For future work, we plan to extend the system to take the full paper directly as input: the model would first automatically extract a concern-related evidence bundle from the full text (locating the cited figures and tables, adjacent pages, and supplementary material), and then perform verification. Current long-context multimodal models already have the basic capability for this step, but the results above also show that finding the right evidence in the full text is not trivial. We will therefore first evaluate the evidence-extraction step on its own, for example by reporting the recall of extracted pages against expert-selected pages, and then report end-to-end verification results.

We hope these new results answer your questions.


Thank you for your efforts, my concerns have been clarified. I think the new experiments should be included in the paper or the appendix, as they seem to strengthen the paper's contributions.

Reviewer_Ns9pOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper publishes PubPeerBench-1000, a benchmark of 1,000 comments on 357 papers drawn from PubPeer. Each comment is rewritten as a neutralized concern, stripped of evaluative language, and paired with an automatically retrieved evidence bundle of relevant paper text, figures, and page images. Two independent experts annotate each case with a verdict (supported, partially supported, unsupported, or insufficient evidence), an evidence-quality score, an impact level, and a short rationale, with a third expert resolving disagreements. The authors then fine-tune Qwen3.6-27B with LoRA on 700 training cases, split by paper to avoid leakage, producing Student Geng's WorkBuddy-27B. On the 200-case test set, the fine-tuned model reaches 48% verdict exact agreement, outperforming GPT-6 Astra (46.5%) and other general-purpose models.

Critical strengths

The main strength of this paper is its problem setup. Post-publication peer review has become a central mechanism of scientific self-correction, and PubPeer in particular has accumulated a large body of concerns that, as the Geng case shows, can lead to real investigations. The paper identifies this gap and scopes the task well. The model is asked whether the paper’s text, figures, and data support a specific concern, not whether the paper is fraudulent or what the authors intended, and the label design reflects that restraint by keeping insufficient evidence separate from unsupported so that a missing figure is not treated as a refutation.

Critical weaknesses
  1. The submission’s primary area is “datasets and benchmarks,” yet the benchmark itself is described almost entirely through label histograms. The paper provides no worked example of a case, no information on the scientific domains, journals, or publication years of the 357 papers. The qualification of the human experts, their training, or how three people cover fields ranging from chemistry to statistics to image forensics is not discussed. The paper also lacks discussions about the license of the 357 papers they deal with.

  2. It is also unclear which version of each paper was collected. Papers are fetched by DOI at collection time with no check for corrections, errata, or retractions issued after the comment, so a concern that was valid when raised may be labeled against a version in which the error has already been fixed.

  3. “PubPeerBench-1000” suggests a 1,000-case evaluation benchmark, but 700 of those cases are consumed by training and 100 by validation, leaving a 200-case test set. 700 examples is a thin basis for fine-tuning a 27B model on a task with four verdict classes, five impact levels, and a five-point quality scale, and 200 test cases cannot distinguish models whose scores differ by a few points. The rarest verdict class (unsupported, 4% overall) is represented by roughly eight test cases.

  4. The paper lacks qualitative analysis on where the models under perform.

  5. Missing citation > https://arxiv.org/abs/2505.11855. The paper has an identical approach of using pubpeer to build a error detection benchmark.

Questions for authors

see weakness.

3 comments
AuthorsOpenReview ↗

Author Response (Weakness 1, 2)

Dear Reviewer Ns9p,

Many thanks for your valuable comments, and for recognizing our problem setup and the careful separation of insufficient_evidence from unsupported. We respond point by point below.


[W1] The submission’s primary area is “datasets and benchmarks,” yet the benchmark itself is described almost entirely through label histograms. The paper provides no worked example of a case, no information on the scientific domains, journals, or publication years of the 357 papers. The qualification of the human experts, their training, or how three people cover fields ranging from chemistry to statistics to image forensics is not discussed. The paper also lacks discussions about the license of the 357 papers they deal with.

[A1] Thank you for this important comment. Since our primary area is datasets and benchmarks, we agree that the dataset documentation in the original submission was insufficient, and we have added the following.

  • Worked example (new Appendix H). We provide a complete case showing the original comment, the neutralized concern, the evidence bundle with page and figure locations, both independent annotations, the adjudicated label, and the predictions of WorkBuddy-27B and GPT-6 Astra.
  • Coverage statistics (revised figure and appendix). We report the distribution of the 357 papers by scientific field and publication year, as well as the distribution of issue types within each field. Due to space limits, journals and publishers are listed in Appendix I.
  • Annotators. The annotation was performed by three experts. All three hold PhDs in biology, are also well versed in chemistry and statistics, and have more than ten years of research experience each. All three have cross-disciplinary research backgrounds and are familiar with the concern types common on PubPeer. Regarding how three people cover different fields, we clarify three aspects. (i) Task requirements. The annotators judge whether the materials already in the paper support a specific concern, for example, whether two image panels are identical, whether reported numbers are consistent with the statistical test, or whether the described methods can support the conclusion (Sections 3.1 and 3.5). They do not need to redo the original domain research. This work relies on careful evidence checking and general research training rather than specialized expertise in every subfield. (ii) Composition of concern types. The most common types in the benchmark are methodological validity (69.20%) and reporting issues or overclaiming (26.20%). Types requiring more specialized knowledge account for smaller shares, e.g., chemical or state assignment (12.10%) and image duplication (6.80%) (Figure 3a). A case can have multiple issue types. (iii) Procedural safeguards. Each case is independently annotated by two experts following the same written annotation guidelines (Appendix B), and disagreements are adjudicated by the third expert; no label is decided by any single person.
  • Licensing. All 357 papers were obtained by DOI from open-access sources (see our response to W2). The dataset now records, for each paper, the license type, license URL, version type, and retrieval date. Of these papers, 275 are under CC BY or CC0, and 82 are under CC licenses with restrictions (e.g., CC BY-NC, CC BY-NC-ND). We release the data in tiers accordingly: for CC BY/CC0 papers, we provide the page images and text in the evidence bundles; for papers under restricted CC licenses, these materials are provided for non-commercial research use only, in accordance with the license terms.

[W2] It is also unclear which version of each paper was collected. Papers are fetched by DOI at collection time with no check for corrections, errata, or retractions issued after the comment, so a concern that was valid when raised may be labeled against a version in which the error has already been fixed.

[A2] Thank you for raising this point, which affects label validity. Comments were collected on 2026-06-09, and the corresponding papers were obtained by DOI from open-access sources between 2026-06-09 and 2026-06-12. At retrieval, we checked each paper for correction, erratum, or retraction records; none of the 357 papers had such a record. Of these papers, 350 are publishers' versions of record and 7 are preprints (including reviewed preprints). Therefore, no paper in the benchmark had a formal correction issued after the corresponding PubPeer comment and before our retrieval, and the situation you describe, an original concern labeled against a corrected version, does not affect the labels of this benchmark. The dataset records the version type and retrieval date of each paper, and the ethics statement now states that labels apply only to the evaluated paper version and do not reflect correction notices or new author-provided data released after retrieval.


AuthorsOpenReview ↗

Author Response (Continued, Weakness 3)

[W3] “PubPeerBench-1000” suggests a 1,000-case evaluation benchmark, but 700 of those cases are consumed by training and 100 by validation, leaving a 200-case test set. 700 examples is a thin basis for fine-tuning a 27B model on a task with four verdict classes, five impact levels, and a five-point quality scale, and 200 test cases cannot distinguish models whose scores differ by a few points. The rarest verdict class (unsupported, 4% overall) is represented by roughly eight test cases.

[A3] We agree with this description and thank you for this comment; it led us to revise our main claim. Below, we quantify what the 200-case test set can and cannot support.

Name. PubPeerBench-1000 refers to the 1,000 annotated cases; all evaluations use the fixed 200-case test set. We now explicitly describe it as a training-and-evaluation resource with a 700/100/200 split.

What 200 cases can distinguish. Comparing models on the same cases is a paired comparison, whose precision depends on the number of cases on which the two models disagree (b + c), not only on the total sample size. Because b cannot exceed the number of cases model A answers correctly, and c cannot exceed the number model B answers correctly, the correct counts reported in the paper give an upper bound on the exact McNemar p-value over all possible overlaps. For impact, WorkBuddy-27B vs. the base model (79 vs. 47 correct) has an upper bound of p = 0.0055; for verdict, WorkBuddy-27B vs. the mismatched condition (96 vs. 51) has an upper bound of p = 0.00026. Both differences are significant however the correctly answered cases overlap. For verdict, the difference between WorkBuddy-27B and the base model (96 vs. 77) is not significant under every possible overlap: it is significant at the 0.05 level as long as there are no more than 83 discordant cases. The actual number of discordant cases is 39, giving p = 0.0034.

What 200 cases cannot distinguish. WorkBuddy-27B and GPT-6 Astra differ by only 3 cases (96 vs. 93). Even if all 3 of these cases favored WorkBuddy-27B, p would only be 0.25. In a paired design with a 10%–20% discordance rate, detecting a 1.5-point difference with 80% power requires roughly 3,500 to 7,000 test cases. We therefore agree that this test set cannot distinguish models that differ by a few points. We have removed the claim that WorkBuddy-27B outperforms all general-purpose models, and now state that it performs comparably to the strongest general-purpose model we evaluated.

The unsupported class. The test set contains 8 unsupported cases. Its low frequency partly follows from our label definitions: a lack of supporting evidence is not automatically labeled unsupported (Section 3.5, Appendix B). Because the test set contains only 8 unsupported cases, each additional correct identification changes the recall of this class by 12.5 percentage points.

Training-set size. We agree that 700 cases is a small number, and we do not claim that it suffices to train an optimal model. However, three facts indicate that it suffices to learn effects that transfer. (i) The data are split by paper association groups, and no group crosses subsets (Appendix C), so the significant gains on the test set come from papers never seen in training, not from memorization. (ii) Only a rank-16 LoRA adapter is trained, with the base model, vision encoder, and multimodal projection frozen, for a total of 132 optimizer updates (Appendix E). This adjusts judgment behavior rather than teaching new knowledge; prior work has adapted large models with about 1,000 examples (LIMA; Zhou et al., 2023). (iii) The checkpoint is selected only on the 100-case validation set, using the mean negative log-likelihood (NLL) of the answer tokens computed at the end of each epoch. For the reported model (seed 42), the lowest validation NLL occurs at the last checkpoint, i.e., epoch 3 (optimizer step 132); validation NLL was still decreasing at the end of training, with no sign of overfitting.


AuthorsOpenReview ↗

Author Response (Continued, Weakness 4, 5)

[W4] The paper lacks qualitative analysis on where the models under perform.

[A4] Thank you for this suggestion. We analyzed the 104 verdict errors of WorkBuddy-27B on the test set. The most frequent single confusion direction is gold partially_supported predicted as insufficient_evidence, with 22 cases, or 21.15% of all verdict errors. These two labels are easily confused because both correspond to incomplete evidence; they differ only in whether the evidence bundle can substantiate part of the concern. On such cases, the model tends to withhold judgment rather than give a partially supported verdict; insufficient_evidence is the largest class in the data (38.20% of the full set), which may reinforce this tendency. This type of error does not judge an unsubstantiated concern to be valid, but it underestimates concerns that are indeed partially valid. We list class-balanced training as a direction for future improvement.


[W5] Missing citation > https://arxiv.org/abs/2505.11855. The paper has an identical approach of using pubpeer to build a error detection benchmark.

[A5] Thank you for pointing out SPOT (Son et al., 2025). At submission time, we were not aware of this work, which is available only as an arXiv preprint; we now discuss it in the related work. The two works are related but address different tasks. SPOT asks a model to discover errors in a full paper without being given a concern, and its labels are 91 errors in 83 papers confirmed by errata or retractions. PubPeerBench-1000 instead asks, given a specific public concern, whether the paper's own evidence supports it. Our data comprise 1,000 comment-level cases from 357 papers, most of which have no author response or resolution; we therefore include unconfirmed concerns and distinguish insufficient_evidence from unsupported. We thus view the two as complementary: SPOT evaluates the discovery of confirmed errors, whereas our work evaluates evidence-grounded verification of unresolved concerns. We hope these additions address your concerns.


Reviewer_woz8OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces PubPeerBench-1000, a set of 1,000 PubPeer comments across 357 DOIs. Each comment is paired with: 1) a “neutralized” restatement of the concern, 2) a pre-retrieved evidence bundle of 0–11 rendered pages from the target paper, 3) expert labels: a 4-class verdict, evidence quality on a 0–4 scale, a 5-class impact rating, and a rationale.

Two experts annotate each case and a third adjudicates disagreements. The authors LoRA-finetune Qwen3.6-27B on 700 training cases to get WorkBuddy-27B, and evaluate on a 200-case test split grouped by paper association.

Critical strengths
  1. The task asks whether the paper’s evidence supports a specific concern. It explicitly does not ask the model to judge misconduct, and this framing is careful.

  2. The identity chain, versioned JSON, content hashes, and paper-grouped splits show good data hygiene.

Critical weaknesses
  1. The paper shows a lack of care in preparation that is difficult to overlook. Several figures contain misaligned boxes and arrows as well as typos (e.g., “No unfied benchmark” in Figure 1), and the reported numbers are internally inconsistent: the per-group accuracies in Table 2 do not reproduce the overall accuracies for Kimi K3 and DeepSeek-V4.1-Flash in Table 1, and the agreement statistics in Appendix F conflict with those in Section 3.5. This is especially concerning given the paper’s subject. A work proposing a tool to verify the integrity of published research should itself meet a high standard of verifiability and internal consistency, and the current manuscript does not.

  2. Several numerical results inconsistencies: 1) Table 2 does not reproduce Table 1. Multiplying the per-group accuracies by the group sizes (23/88/41/48): Kimi K3: 6+38+21+20 = 85/200 = 42.5%, but Table 1 reports 41.0%. DeepSeek-V4.1-Flash: 8+41+22+22 = 93/200 = 46.5%, but Table 1 reports 44.0%. The same calculation does check out for WorkBuddy (96/200) and the base model (77/200). The recomputed values coincide with the Qwen3.8-Max and GPT-6 Astra rows, which suggests rows were mislabeled or swapped. 2) The adjudication statistics conflict. The two independent annotators reach a weighted κ of 0.81 on evidence quality. Yet the final labels agree with the first annotator at only κ = 0.59. If adjudication only resolves disagreements, final-versus-first agreement should be at least the inter-annotator level. This suggests the adjudicator overrode cases where both annotators agreed, or a number is wrong. 3) The same condition has an evidence-quality MAE of 0.90 in Table 3 but 0.89 in Tables 1 and 4. 4) Percentages are truncated rather than rounded (26.08, 21.73, 40.66). 5) Some values coincide across tables: base-model verdict = claim-only verdict = 38.50, and base-model impact = text-only impact = 23.50.

  3. The claimed system is not what is evaluated. The abstract says WorkBuddy reformulates the concern, maps it to evidence, and reports evidence location. However, in the experiments: 1) the neutralized claim and the evidence bundle are pre-computed inputs. 2) The bundle excludes supplementary material and attachments (App. E). 3) Location and rationale outputs are never evaluated, and location is not even among the four output fields in §3.1. So what is evaluated is a classifier given oracle retrieval, not an end-to-end verification assistant. Key procedures are left unspecified: who or what performs neutralization, retrieval, and issue decomposition, how good they are, and whether they could leak label information.

Questions for authors

See in "Critical Weaknesses"

5 comments
AuthorsOpenReview ↗

Response to Reviewer woz8 (1/4): Weakness 1 and 2.1

Dear Reviewer woz8,

Thank you for your detailed and precise review, and especially for recomputing our tables yourself, which helped us find and correct several errors. We respond point by point below.

[W1] "The paper shows a lack of care in preparation that is difficult to overlook. Several figures contain misaligned boxes and arrows as well as typos (e.g., “No unfied benchmark” in Figure 1), and the reported numbers are internally inconsistent: the per-group accuracies in Table 2 do not reproduce the overall accuracies for Kimi K3 and DeepSeek-V4.1-Flash in Table 1, and the agreement statistics in Appendix F conflict with those in Section 3.5. This is especially concerning given the paper’s subject. A work proposing a tool to verify the integrity of published research should itself meet a high standard of verifiability and internal consistency, and the current manuscript does not."

[A1] We sincerely apologize for these errors. We fully agree that a work proposing a tool to verify published research must itself meet a high standard of verifiability and internal consistency.

  • Figures. In the revised PDF, we corrected "No unfied benchmark" to "No unified benchmark" in Figure 1, fixed another label in the same figure ("Unified" under the question-mark icon should be "Unverified"), and realigned the boxes and arrows in Figures 1 and 2.
  • Table 2 vs. Table 1. Two row labels in Table 2 were wrong because we transcribed numbers by hand (details in [A2]).
  • Appendix F vs. Section 3.5. Two evidence-quality agreement values were swapped; the verdict and impact statistics (κ = 0.89 and 0.92) are correct (details in [A3]).
  • Verifiability. We no longer transcribe numbers by hand. All tables are now regenerated by a single script directly from the per-case prediction files.

[W2.1] "Several numerical results inconsistencies: 1) Table 2 does not reproduce Table 1. Multiplying the per-group accuracies by the group sizes (23/88/41/48): Kimi K3: 6+38+21+20 = 85/200 = 42.5%, but Table 1 reports 41.0%. DeepSeek-V4.1-Flash: 8+41+22+22 = 93/200 = 46.5%, but Table 1 reports 44.0%. The same calculation does check out for WorkBuddy (96/200) and the base model (77/200). The recomputed values coincide with the Qwen3.8-Max and GPT-6 Astra rows, which suggests rows were mislabeled or swapped."

[A2] Your diagnosis is correct: this is a labeling error in Table 2, not an error in the results. The rows labeled "Kimi K3" and "DeepSeek-V4.1-Flash" are in fact the subgroup results of Qwen3.8-Max and GPT-6 Astra. Their row names were mislabeled during manual transcription, not swapped. Only the two row names were wrong, and Section 4.3 copied them.

The correspondence is exact. With group sizes 23/88/41/48, each percentage maps to a unique integer count. The two rows sum to 85/200 and 93/200, exactly Qwen3.8-Max (42.50%) and GPT-6 Astra (46.50%) in Table 1; they cannot produce Kimi K3 (82/200) or DeepSeek-V4.1-Flash (88/200). Recomputing these rows from the Qwen3.8-Max and GPT-6 Astra per-case prediction files gives the same counts. After relabeling and consistent rounding (see [A5]), every row of Table 2 reproduces Table 1:

Model0 issues (n=23)1 issue (n=88)2 issues (n=41)≥3 issues (n=48)Overall (= Table 1)
Qwen3.8-Max26.09% (6/23)43.18% (38/88)51.22% (21/41)41.67% (20/48)42.50% (85/200)
GPT-6 Astra34.78% (8/23)46.59% (41/88)53.66% (22/41)45.83% (22/48)46.50% (93/200)
Qwen3.6-27B21.74% (5/23)40.91% (36/88)43.90% (18/41)37.50% (18/48)38.50% (77/200)
WorkBuddy-27B30.43% (7/23)47.73% (42/88)58.54% (24/41)47.92% (23/48)48.00% (96/200)

Because the underlying counts are unchanged, the conclusion of Section 4.3 is unaffected: all four models reach their highest verdict exact agreement in the 2-issue group and decline in the ≥3-issue group.


AuthorsOpenReview ↗

Response to Reviewer woz8 (2/4): Weakness 2.2–2.5

[W2.2] "2) The adjudication statistics conflict. The two independent annotators reach a weighted κ of 0.81 on evidence quality. Yet the final labels agree with the first annotator at only κ = 0.59. If adjudication only resolves disagreements, final-versus-first agreement should be at least the inter-annotator level. This suggests the adjudicator overrode cases where both annotators agreed, or a number is wrong."

[A3] Thank you for this sharp observation. We agree that if adjudication only picks one of the two annotators' scores when they disagree, final-vs-first agreement should not fall clearly below inter-annotator agreement. Our verdict and impact statistics meet this expectation: given the label marginals, κ = 0.89 and 0.92 correspond to about 7.5% and 5.9% inter-annotator disagreement, while the final label differs from the first annotator in 5.2% and 3.5% of cases. The same holds for evidence quality once its values are corrected.

We rechecked the raw annotation records. The cause is your second possibility, a wrong number: the two evidence-quality values were swapped.

Evidence quality (0–4), linearly weighted κReportedCorrect
Annotator 1 vs. Annotator 2 (before adjudication)0.810.59
Final label vs. Annotator 10.590.81
After the correction, final-vs-first agreement exceeds inter-annotator agreement, as expected. To be clear about what this implies: inter-annotator agreement on evidence quality (0.59) is moderate, clearly below verdict (0.89) and impact (0.92). Evidence quality is the finest-grained field (a five-level ordinal scale), and 59.76% of inter-annotator differences are only 1 point. We now state this in Section 3.5. The error affects no label, model result, or conclusion.

[W2.3] "3) The same condition has an evidence-quality MAE of 0.90 in Table 3 but 0.89 in Tables 1 and 4."

[A4] The underlying value is the same; only the rounding differs. The unrounded EQ MAE is 0.895, i.e., a total absolute error of 179 over the 200 test cases. Tables 1 and 4 truncated it to 0.89; Table 3 rounded it to 0.90. We now round everywhere, so all three tables and the abstract, introduction, and Sections 3.7, 4.2, and 4.5 report 0.90. The Table 3 comparison (0.86 vs. 0.90) is unchanged.

[W2.4] "4) Percentages are truncated rather than rounded (26.08, 21.73, 40.66)."

[A5] You are right, and the rounding was also inconsistent within the paper: 26.08 (6/23 = 26.087), 21.73 (5/23 = 21.739), and 51.21 (21/41 = 51.220) were truncated, whereas 47.73 (42/88 = 47.727) and 58.54 (24/41 = 58.537) were rounded. All values are now generated by script from unrounded values and rounded to two decimals. The affected values are:

LocationReportedCorrected
Table 2, Qwen3.8-Max (originally labeled "Kimi K3"), 0 issues26.08%26.09%
Table 2, Qwen3.6-27B, 0 issues21.73%21.74%
Table 2, Qwen3.8-Max, 2 issues; Section 4.351.21%51.22%
WorkBuddy EQ MAE (see [A4])0.890.90
Table 4, text, EQ MAE1.231.24
Table 4, claim_only, EQ MAE; Section 4.51.691.70
Section 4.2, relative EQ MAE reduction40.66%40.33%

40.66% also had a derivation error, and we apologize. It was computed from the truncated 0.89, (1.50 − 0.89)/1.50 = 40.667%, and then truncated again. From the unrounded value, it is (1.50 − 0.895)/1.50 = 40.33%. None of these changes affects any conclusion, but they remind us that all derived values must be computed from unrounded values.

[W2.5] "5) Some values coincide across tables: base-model verdict = claim-only verdict = 38.50, and base-model impact = text-only impact = 23.50."

[A6] The first coincidence has a clear explanation. 77 of the 200 test cases are insufficient_evidence, so a majority-class baseline that always predicts it scores exactly 38.50% exact agreement and 13.90% Macro-F1, matching claim_only in Table 4. The per-case outputs confirm that under claim_only, WorkBuddy predicts insufficient_evidence for all 200 cases, the expected and arguably ideal behavior without paper evidence; the paper now states this.

The untuned Qwen3.6-27B reaches the same exact agreement (38.50%, 77/200) with a different prediction distribution: Macro-F1 28.13% (claim_only: 13.90%), EQ MAE 1.50 and impact 23.50% (claim_only: 1.70 and 11.00%). The base/text overlap at 23.50% (47/200) involves only one metric; the two runs differ on all others: verdict 38.50% vs. 42.50%, Macro-F1 28.13% vs. 29.10%, EQ MAE 1.50 vs. 1.24. With 200 cases, every accuracy is a multiple of 0.5 points, so a coincidence on one metric is not unusual. Comparing the two per-case prediction files directly, their verdict labels differ on 79 of the 200 cases. The two rows therefore come from two different runs, not from a duplicated entry.


AuthorsOpenReview ↗

Response to Reviewer woz8 (3/4): Weakness 3

[W3] "The claimed system is not what is evaluated. The abstract says WorkBuddy reformulates the concern, maps it to evidence, and reports evidence location. However, in the experiments: 1) the neutralized claim and the evidence bundle are pre-computed inputs. 2) The bundle excludes supplementary material and attachments (App. E). 3) Location and rationale outputs are never evaluated, and location is not even among the four output fields in §3.1. So what is evaluated is a classifier given oracle retrieval, not an end-to-end verification assistant. Key procedures are left unspecified: who or what performs neutralization, retrieval, and issue decomposition, how good they are, and whether they could leak label information."

[A7] The evaluated component is a verification model operating on a given neutralized concern and claim-related evidence bundle. Neutralization and retrieval belong to benchmark construction, not to the evaluated model. We accept your characterization: this is an idealized-retrieval setting, and end-to-end verification including retrieval remains future work; our new full-text input condition (see our reply to j2MD) is a first step. We have revised the abstract, introduction, and Section 3.1 accordingly.

Evidence location. In our setting, the evidence location is fixed before the model runs, not predicted by it. PubPeer comments usually name the questioned location (figure, panel, table, or page), and the neutralized concern keeps these identifiers (Section 3.3). For comments without one, retrieval determines it when the bundle is built (Section 3.4). Location is thus part of the input; evaluating it as an output would mainly measure whether the model can repeat a given location, not its ability to locate evidence. Hence it is not among the four output fields in Section 3.1 and is not evaluated. We have removed "evidence location" from the model-output description and explicitly assigned "mapping the concern to evidence" to benchmark construction.

Rationale. We treat the rationale as a record of each judgment's basis for human checking, not as an evaluated prediction target (Section 4.1). It is kept in the structured output as part of the SFT target (Section 3.7) but is not scored, for two reasons. First, its role differs: verdict, evidence quality, and impact are the judgments users act on, whereas the rationale lets users go back to the bundle to check them; it is not a label that must "agree" with experts. Second, there is no reliable automatic metric: the same judgment can be correctly phrased in many ways, and expert rationales are adjudication and quality-control records (Appendix B) of varying length and style, unsuitable as references; overlap metrics such as BLEU and ROUGE measure wording similarity, not reasoning correctness, which is a separate research question.

Key procedures (detailed in the new Appendix G).

  • Neutralization. Neutralized concerns are generated by GPT-5.6 Sol with a fixed prompt from the full comment text only. The neutralizer sees neither paper evidence nor any label, so the neutralized text carries no information from either. In effect, with paper evidence fixed (Table 3), the neutralized concern gives higher verdict exact agreement (48.00% vs. 45.50%) and Macro-F1 (37.00% vs. 29.20%) than the original comment, which is better on EQ MAE (0.86 vs. 0.90) and impact (42.50% vs. 39.50%). This shows that neutralization preserves the information needed for the verdict. To systematically assess whether the core scientific concern is preserved, we compared each original PubPeer comment with its neutralized concern on the 200 test cases. An automatic evaluator judged all 200 rewrites to preserve the core concern (200/200). We then sampled 80 and manually reviewed them against the original comments and found no omitted or altered core concern (80/80).
  • Retrieval. Retrieval is rule-based: explicit page, figure, and table identifiers first, then text anchors (Section 3.4). Annotators and models judge from the same bundle, so expert labels and model inputs match by construction. When key evidence is missing, the case is labeled insufficient_evidence, so a retrieval miss appears as insufficient evidence, not as a wrong conclusion label. Retrieval runs before annotation and has no access to labels.
  • Issue decomposition. GPT-6 Astra counts the independent substantive issues from the neutralized concern with a fixed prompt. The count is used only for subgroup analysis, never as model input. The count received no human validation, so Appendix G now states that the groups in Section 4.3 are based on automated counts.
  • Supplementary material. We apologize for the inaccurate wording in Appendix E. The bundle is not limited to the main text: it contains the claim-related pages selected by retrieval, which can include supplementary pages when the concern involves them.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Area Chair and Reviewers

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair and all reviewers (Reviewers woz8, Ns9p, j2MD) for their careful reading and constructive feedback. We are encouraged that the reviewers found the task framing careful and appropriately restrained: it asks only whether paper evidence supports a specific concern, rather than judging misconduct or author intent (Reviewers woz8, Ns9p). We are also glad that the reviewers recognized the data hygiene of the identity chain, versioned JSON, content hashes, and paper-grouped splits (Reviewer woz8), the importance of the problem for post-publication scientific self-correction (Reviewers Ns9p, j2MD), and found the ablations interesting (Reviewer j2MD).

We take the reviewers' concerns very seriously, especially the numerical inconsistencies identified by Reviewer woz8. We fully agree that a paper proposing a tool for verifying published research must itself meet a high standard of verifiability and internal consistency. To this end, we regenerated every table in the paper directly from the per-case prediction files using a single script, which we will release together with the per-case predictions. The main revisions are summarized below; details are given in the individual responses.

  1. Numerical corrections. We corrected the model labels of two rows in Table 2: the rows labeled "Kimi K3" and "DeepSeek-V4.1-Flash" report Qwen3.8-Max and GPT-6 Astra, respectively. The numbers in Tables 1 and 2 were correct; only these two row labels (and the model names copied from them into the text of Section 4.3) were wrong. We also corrected two swapped evidence-quality agreement values in Section 3.5 and Appendix F, and the inconsistent truncation and rounding of percentages. All percentages now use standard rounding to two decimals.
  2. Scope clarification. We revised the abstract and Section 3 to state explicitly that the evaluated component is the verification model operating on prepared inputs (neutralized concern + claim-related evidence bundle). We also specify who or what performs neutralization, retrieval, and issue decomposition.
  3. Dataset documentation (primary area: datasets and benchmarks). We added a worked example of a real case, the distribution of scientific fields, journals, and publication years, annotator background, licensing information, and the version type and post-comment correction/retraction status of each paper.
  4. New analyses (see our response to Reviewer j2MD). In our responses, we provide a robustness analysis under coarse-grained labels, agreement with each individual annotator, and results under a full-text input condition.
  5. Ethics. We clarified the source and scope of the permission, the naming and portrait issue, and the treatment of author replies.
  6. Related work. We added and discussed SPOT (Son et al., 2025).

We hope these revisions demonstrate our commitment to rigor. Thank you again for your feedback. We would be glad to continue the discussion if any further questions arise.

Best regards, Authors


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.