- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper introduces a new benchmark, MUSE, for paper recommendation. It argues that existing benchmarks or citation-related tasks recommend papers after projects have started or after the paper has been written. Thus, it positions itself before projects start. MUSE has a target cutoff date to block future leakage and provides an author-disjoint train/val/test split. It also proposes three levels of prior-citation exclusion (all future citations, new citations by the author, new citations by the author and co-author) and three levels of evidence sources (bibliography, citation context, and author labels). These combine to nine evaluation settings. The paper analyzes the benchmark with different retrievers and finds that publication history is a strong signal but cannot capture most new citations. And, that project title (future data) can significantly improve the performance. This shows substantial room for improvement.
Critical strengths
Overall, I found the work meaningful, and the analyses are comprehensive.
- The paper presents a clear and meaningful position on the benchmark. The benchmark is motivated by a question of whether an author's historical publications can predict future citations, including influential ones. As the analyses show, this is a challenging task, yet it offers a real-world application for all researchers.
- The benchmark itself is substantial. MUSE contains 2.3 million papers and 1 million cases. These also include 580 author-label cases. The paper clearly motivates and documents the train/val/test splits, evaluation settings, and metrics.
- The paper presents well-designed diagnostic experiments. The results reveal useful findings beyond just which retriever does best.
Critical weaknesses
Here are my comments to improve the paper.
- Section 4.2 discusses recommendations that are new to the author but not to the field, and they are more difficult to retrieve. This conclusion sounds stronger than the supporting evidence suggests. The reason might simply be that fewer targets are available to retrieve. Although Appendices B.2 and H provide a count-matched comparison, they do not directly control for target across the Next, New, and Isolated exclusion levels.
- The paper acknowledges that the precision of the citation context classifier is 0.6345 (Appendix B.2). Although we cannot extrapolate to the 6,707 generative use cases, it raises concerns about the reliability of the Context-Root-* targets. In addition, it raises concerns about the 46% missed conclusion (Figure 3A) and the project title conclusion (Figure 3C).
Questions for authors
- I understand that Context-Root-* offers a scalable proxy for analyses (Sections 4.2 and 4.3). However, the classifier's precision is questionable. Could you repeat these diagnostics on the other levels of evidence?
4 comments
W2 and Q1. Repeating the diagnostics beyond Context-Root
Yes. We repeated both diagnostics using bibliography targets and works that authors named. Figure 3A and 3C (p. 9) show that pooled rankings still miss targets in many cases and that adding the next paper’s title and abstract improves retrieval in hindsight for both additional sources.
The reported precision of 0.6345 comes from the development set used for model selection (Appendix B.3, p. 18). In a separate human check, annotators confirmed explicit generative use in 17 of 27 decidable passages labelled positive (Appendix D.3, p. 24, lines 1291–1295). The Bibliography and Author-Root analyses below use targets defined independently of the classifier.
(a) §4.2 and Figure 3A—Incomplete recovery across evidence sources
-
Expanded analysis and figure. Figure 3A now compares the best individual ranking with pooled recovery across 24 configurations, separately for the New tier of each evidence source. This extends the original figure beyond Context-Root and incorporates the additional retrieval baselines.
-
Finding and main-text update. Even after pooling, 36.47% of CiteNew cases and 40.60% of Author-Root-New cases have no target in any ranking’s top 100. Substantial gaps therefore also appear with bibliography and directly author-attested targets. Revised §4.2, lines 418–422, reports this broader finding.
-
Supporting appendix results. Appendix G.1, Table 24, reports all tiers at depths 100 and 1,000, extending beyond the New-tier, depth-100 comparison displayed in Figure 3A. The original 46% unreached figure concerned seven rankings on Context-Root-New at depth 1,000. With the updated cohort and 24 rankings, the unreached share at that depth is 31.12% (G.1, lines 2309–2313).
(b) §4.3 and Figure 3C—Hindsight gains across evidence sources
-
Expanded analysis and figure. We expanded the hindsight evaluation to three representative lexical and dense retrievers—BM25, SPECTER2-Proximity and GritLM—across all three evidence sources. Figure 3C now illustrates the cross-source comparison using SPECTER2-Proximity K16 on the New tiers, at depth 100.
-
Finding and main-text update. Adding the next paper’s title improves retrieval, and its abstract adds further gains across all three sources. Revised §4.3, lines 426–431 and 456–457, reports the cross-source result and identifies the broader supporting evidence.
-
Supporting appendix results. Appendix F.2, Table 23, provides Hit results across all three retrievers and every reported tier, at depths 100 and 1,000. The accompanying analysis also reports that both additions improve nDCG@1000; all paired 95% confidence intervals for these gains exclude zero (F.2, lines 2186–2189).
These additions show that both incomplete recovery and hindsight gains extend to bibliography and author-attested evidence, with the main figures illustrating the cross-source findings and the appendix documenting their broader scope.
The expanded pooled analysis also incorporates the added citation-graph and supervised baselines. Our “Shared revision summary” and “Presentation changes” comments describe the broader evaluation and provide a guide to the revised paper.
We would welcome your assessment of whether these revisions address your concerns and would be happy to clarify any remaining questions.
Thank you for the detailed response and additional analyses.
The new comparisons directly address my concern about how to interpret differences across exclusion levels. I also appreciate the revised wording on the prior citation history, which aligns with the level of empirical support.
The new results on the diagnostics for the other levels of evidence also address my main concern about the poor classifier precision. However, the classifier’s precision remains a limitation for Context-Root targets, as the human check does not directly improve the precision.
I think the paper is in good standing. I will raise my score.
Thank you for the thoughtful follow-up and for raising your score. To address the remaining limitation on classifier precision, we constructed higher-precision Context-Root labels and repeated every Context-Root result in the paper with them. All findings remain consistent with the original results, indicating that our conclusions are robust to the classifier's lower precision. We keep the original labels in the main text, so the reported results remain comparable with the reviewed version and reproducible without API access, and we report the new analysis in a new appendix.
Follow-up to W2 and Q1. Context-Root label precision
-
What we added. New Appendix H.4 (pp. 47–48, lines 2521–2578). The classifier's teacher, GPT-5.4-mini, reassesses every citation context with a classifier score above 0.2, using the same prompt (Appendix B.3); this includes borderline contexts below the classifier's 0.5 decision threshold. A cited work becomes a target when the teacher labels at least one of its contexts as generative. On the same human-coded contexts used in Appendix B.3, precision rises from 0.63 to 0.83 (recall 0.93 → 0.88).
-
What we learned. Every Context-Root result holds under the higher-precision labels (Table 29, p. 48). Each row gives the test used in the paper, the values for the reference retriever (SPECTER2-Proximity K16), and where the finding holds under the higher-precision labels. Values are percentages; differences and gains are in percentage points, with 95% CIs in brackets.
| Finding (section) | Test | Classifier labels | Higher-precision labels | Holds |
|---|---|---|---|---|
| Label quality (human-coded contexts) | precision / recall | 0.63 / 0.93 | 0.83 / 0.88 | — |
| Reach falls beyond prior citations (§4.2) | Next > New > Isolated, Hit@100 | 22.82 > 15.26 > 14.08 | 21.59 > 14.48 > 13.62 | ✓ 23/23 configurations |
| The Next-to-New drop persists after count matching (§4.2) | observed − expected < 0, New Hit@100 | −0.59 [−0.83, −0.36] | −0.36 [−0.59, −0.12] | ✓ 4/4 retrievers |
| History beats popularity (§4.1) | history > popularity, New Hit@100 | 15.26 > 2.30 | 14.48 > 2.26 | ✓ 23/23 configurations, all metrics and depths |
| The signal is researcher-specific (§4.1) | own ÷ matched histories > 1, New nDCG@1000 | 8.4× | 8.6× | ✓ 3/3 retrievers |
| Pooling leaves gaps (§4.2) | > 50% of New cases unreached by all 24 rankings, depth 100 | 58.07 | 59.16 | ✓ |
| Project information helps (§4.3) | gain over history > 0 with CI excluding 0, New Hit@100: + title / + title and abstract | +9.72 / +20.24 | +9.65 / +19.52 | ✓ 3/3 retrievers, also when excluding papers that name a target |
| Labels cover few author endorsements (D.3) | share of endorsements with a generative-use label | 7.6 | 6.0 | ✓ |
-
The appendix analyses based on Context-Root are also unchanged (placebo titles, pooling in every tier, publication-history size, prior-citation visibility and uncapped BM25 queries).
-
What changes. The only noticeable difference is in absolute levels: on the same cases, Hit@100 for the reference retriever is 0.7–2.1 percentage points lower with the higher-precision labels. Every comparison and conclusion stays the same.
-
What changed in the paper.
- New Appendix H.4 and Table 29 (pp. 47–48) describe the higher-precision labels and report the full comparison.
- Revised §5 (p. 9, lines 475–477) now reads:
"Is a classifier label trustworthy? Its precision is 0.63 on the contexts used to select it; a sensitivity analysis with higher-precision labels (0.83) leaves the conclusions unchanged (Appendix H.4); held-out data would better test generalization."
Thank you again for your careful engagement throughout this discussion. Your comments led directly to the target-count control across the Next, New and Isolated levels and now to this sensitivity analysis, which shows that our Context-Root conclusions do not depend on the classifier's precision. Both have made the paper considerably stronger, and we are grateful for your help in improving it. We would be happy to clarify any remaining questions.
Thank you for these suggestions. We added a direct target-count control and expanded the pooled-recovery and hindsight analyses across evidence sources. These changes strengthen the evaluation and make the scope of the findings more precise.
W1. Interpretation of tier differences and the target-count explanation
Fewer targets and changes in which cases remain eligible explain part of the decline. When we compare the same cases and match target counts, works the researcher had not previously cited are still harder to retrieve across all four tested retrievers and all three evidence sources. Table 26 (p. 45) gives the direct comparison. Revised §4.2 (p. 8, lines 409–410) states the qualified finding.
(a) Revised conclusion in §4.2
We agree that the original heading, “Hard means new to the researcher, not new to the field” (original §4.2, line 411), was stronger than the comparison established. We replaced it with “Retrieval beyond prior citations” (revised §4.2, line 403). The opening finding now states that “reach falls for works the researcher had not cited before” (revised line 405), describing the measured prior-citation distinction without attributing retrieval difficulty to unfamiliarity. The original heading also claimed that works new to the field are not harder to retrieve. We no longer make this claim in the main text; the supporting comparison now appears in Appendix G.2 (lines 2333–2338) as a descriptive result on Context-Root-New, which does not separate citation visibility from target age.
(b) Direct comparison after matching target counts
What we added. The earlier count-matched analyses compared bibliography and citation-context targets. We added Appendix G.3 and Table 26 to directly compare New versus Next and Isolated versus New, using the same cases and accounting for differences in target count. The analysis covers four representative lexical, dense and graph retrievers—BM25, SPECTER2-Proximity, GritLM and Personalized PageRank—across all three evidence sources (revised G.3, lines 2345–2356).
What we learned. Fewer targets alone do not explain the lower New-tier scores. Across all four retrievers and three sources, both Hit@100 and macro Recall@100 remain below the target-count-matched expectation, with 95% confidence intervals excluding zero. For example, SPECTER2-Proximity K16 would be expected to find an Author-Root-New target in 34.84% of cases after matching target counts, compared with 29.87% observed (Hit@100). Retrieval is 4.98 percentage points below expectation (95% CI for observed minus expected [−6.39, −3.60]; Table 26).
How we qualified the finding. Target count and changes in which cases remain eligible explain part of the aggregate decline. For example, restricting Context-Root-Next to the cases retained in Context-Root-New accounts for approximately 71% of the aggregate Hit@100 decline for SPECTER2-Proximity K16. After fixing the cases and matching target counts, the New-tier residual remains negative across all four tested retrievers and three evidence sources. The additional coauthor exclusion has smaller or less consistent effects after target-count adjustment (revised G.3, lines 2356–2367; Table 26).
What we added to the main paper. Revised §4.2, lines 409–410, explicitly connects the finding to this control: