acceptodds
Under review as a conference paper at ICLR 2027

MUSES: Retrieving the Literature That Shapes a Researcher's Next Paper

Abstract

Much of the work that shapes a researcher's next paper is not something they went looking for: it often arrives before the project has a question anyone could search for. How much of it can be anticipated from a researcher's past publications alone? We introduce MUSES, a benchmark of 1.04 million cases built from the Semantic Scholar Open Research Corpus (S2ORC), of which 140,278 form the held-out test split. Given a researcher's papers before a cutoff, a system ranks 2.33 million earlier candidates for their next paper and is scored primarily on works the researcher had not cited before. Because a citation is weak evidence of influence, we also asked authors directly which works shaped their papers, yielding 580 author-collected evaluable cases, a distinct construct from the citation-based targets. A classifier grounded in human coding adds citation contexts where the next paper builds on a work. Across retrievers from popularity to dense encoders and a supervised model, we find that a researcher's past reveals where their next paper will look, but least well what will be new to them. History-based retrievers place a newly cited work in their top 100 for 29–34% of papers, against 5% for popularity; among author-collected cases, one retriever finds a work the authors themselves named in 56% of cases (36% when that work was new to the researcher). Yet the signal fades for works new to the researcher, most of all for those that shaped the paper: for 46% of the 6,707 test papers with such labels, no retriever we evaluate finds one in its top 1,000. Much of the remaining gap is the project itself: adding only the next paper's title raises one retriever's reach on these papers from 29% to 48%, more than switching retrievers does. MUSES shows where a researcher's record runs out and the project itself must take over.

Primary area:
datasets and benchmarks
Keywords:
information retrieval, scientific literature discovery, researcher-conditioned retrieval, personalized retrieval, citation recommendation, prospective evaluation, AI for science, retrieval benchmarks

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_X4CjOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces a new benchmark, MUSE, for paper recommendation. It argues that existing benchmarks or citation-related tasks recommend papers after projects have started or after the paper has been written. Thus, it positions itself before projects start. MUSE has a target cutoff date to block future leakage and provides an author-disjoint train/val/test split. It also proposes three levels of prior-citation exclusion (all future citations, new citations by the author, new citations by the author and co-author) and three levels of evidence sources (bibliography, citation context, and author labels). These combine to nine evaluation settings. The paper analyzes the benchmark with different retrievers and finds that publication history is a strong signal but cannot capture most new citations. And, that project title (future data) can significantly improve the performance. This shows substantial room for improvement.

Critical strengths

Overall, I found the work meaningful, and the analyses are comprehensive.

  1. The paper presents a clear and meaningful position on the benchmark. The benchmark is motivated by a question of whether an author's historical publications can predict future citations, including influential ones. As the analyses show, this is a challenging task, yet it offers a real-world application for all researchers.
  2. The benchmark itself is substantial. MUSE contains 2.3 million papers and 1 million cases. These also include 580 author-label cases. The paper clearly motivates and documents the train/val/test splits, evaluation settings, and metrics.
  3. The paper presents well-designed diagnostic experiments. The results reveal useful findings beyond just which retriever does best.
Critical weaknesses

Here are my comments to improve the paper.

  1. Section 4.2 discusses recommendations that are new to the author but not to the field, and they are more difficult to retrieve. This conclusion sounds stronger than the supporting evidence suggests. The reason might simply be that fewer targets are available to retrieve. Although Appendices B.2 and H provide a count-matched comparison, they do not directly control for target across the Next, New, and Isolated exclusion levels.
  2. The paper acknowledges that the precision of the citation context classifier is 0.6345 (Appendix B.2). Although we cannot extrapolate to the 6,707 generative use cases, it raises concerns about the reliability of the Context-Root-* targets. In addition, it raises concerns about the 46% missed conclusion (Figure 3A) and the project title conclusion (Figure 3C).
Questions for authors
  1. I understand that Context-Root-* offers a scalable proxy for analyses (Sections 4.2 and 4.3). However, the classifier's precision is questionable. Could you repeat these diagnostics on the other levels of evidence?
4 comments

Thank you for these suggestions. We added a direct target-count control and expanded the pooled-recovery and hindsight analyses across evidence sources. These changes strengthen the evaluation and make the scope of the findings more precise.

W1. Interpretation of tier differences and the target-count explanation

Fewer targets and changes in which cases remain eligible explain part of the decline. When we compare the same cases and match target counts, works the researcher had not previously cited are still harder to retrieve across all four tested retrievers and all three evidence sources. Table 26 (p. 45) gives the direct comparison. Revised §4.2 (p. 8, lines 409–410) states the qualified finding.

(a) Revised conclusion in §4.2

We agree that the original heading, “Hard means new to the researcher, not new to the field” (original §4.2, line 411), was stronger than the comparison established. We replaced it with “Retrieval beyond prior citations” (revised §4.2, line 403). The opening finding now states that “reach falls for works the researcher had not cited before” (revised line 405), describing the measured prior-citation distinction without attributing retrieval difficulty to unfamiliarity. The original heading also claimed that works new to the field are not harder to retrieve. We no longer make this claim in the main text; the supporting comparison now appears in Appendix G.2 (lines 2333–2338) as a descriptive result on Context-Root-New, which does not separate citation visibility from target age.

(b) Direct comparison after matching target counts

  • What we added. The earlier count-matched analyses compared bibliography and citation-context targets. We added Appendix G.3 and Table 26 to directly compare New versus Next and Isolated versus New, using the same cases and accounting for differences in target count. The analysis covers four representative lexical, dense and graph retrievers—BM25, SPECTER2-Proximity, GritLM and Personalized PageRank—across all three evidence sources (revised G.3, lines 2345–2356).

  • What we learned. Fewer targets alone do not explain the lower New-tier scores. Across all four retrievers and three sources, both Hit@100 and macro Recall@100 remain below the target-count-matched expectation, with 95% confidence intervals excluding zero. For example, SPECTER2-Proximity K16 would be expected to find an Author-Root-New target in 34.84% of cases after matching target counts, compared with 29.87% observed (Hit@100). Retrieval is 4.98 percentage points below expectation (95% CI for observed minus expected [−6.39, −3.60]; Table 26).

  • How we qualified the finding. Target count and changes in which cases remain eligible explain part of the aggregate decline. For example, restricting Context-Root-Next to the cases retained in Context-Root-New accounts for approximately 71% of the aggregate Hit@100 decline for SPECTER2-Proximity K16. After fixing the cases and matching target counts, the New-tier residual remains negative across all four tested retrievers and three evidence sources. The additional coauthor exclusion has smaller or less consistent effects after target-count adjustment (revised G.3, lines 2356–2367; Table 26).

  • What we added to the main paper. Revised §4.2, lines 409–410, explicitly connects the finding to this control:

    “The Next-to-New decline also persists when comparing the same cases and accounting for fewer targets, in both Hit@100 and macro Recall@100.”

W2 and Q1. Repeating the diagnostics beyond Context-Root

Yes. We repeated both diagnostics using bibliography targets and works that authors named. Figure 3A and 3C (p. 9) show that pooled rankings still miss targets in many cases and that adding the next paper’s title and abstract improves retrieval in hindsight for both additional sources.

The reported precision of 0.6345 comes from the development set used for model selection (Appendix B.3, p. 18). In a separate human check, annotators confirmed explicit generative use in 17 of 27 decidable passages labelled positive (Appendix D.3, p. 24, lines 1291–1295). The Bibliography and Author-Root analyses below use targets defined independently of the classifier.

(a) §4.2 and Figure 3A—Incomplete recovery across evidence sources

  • Expanded analysis and figure. Figure 3A now compares the best individual ranking with pooled recovery across 24 configurations, separately for the New tier of each evidence source. This extends the original figure beyond Context-Root and incorporates the additional retrieval baselines.

  • Finding and main-text update. Even after pooling, 36.47% of CiteNew cases and 40.60% of Author-Root-New cases have no target in any ranking’s top 100. Substantial gaps therefore also appear with bibliography and directly author-attested targets. Revised §4.2, lines 418–422, reports this broader finding.

  • Supporting appendix results. Appendix G.1, Table 24, reports all tiers at depths 100 and 1,000, extending beyond the New-tier, depth-100 comparison displayed in Figure 3A. The original 46% unreached figure concerned seven rankings on Context-Root-New at depth 1,000. With the updated cohort and 24 rankings, the unreached share at that depth is 31.12% (G.1, lines 2309–2313).

(b) §4.3 and Figure 3C—Hindsight gains across evidence sources

  • Expanded analysis and figure. We expanded the hindsight evaluation to three representative lexical and dense retrievers—BM25, SPECTER2-Proximity and GritLM—across all three evidence sources. Figure 3C now illustrates the cross-source comparison using SPECTER2-Proximity K16 on the New tiers, at depth 100.

  • Finding and main-text update. Adding the next paper’s title improves retrieval, and its abstract adds further gains across all three sources. Revised §4.3, lines 426–431 and 456–457, reports the cross-source result and identifies the broader supporting evidence.

  • Supporting appendix results. Appendix F.2, Table 23, provides Hit results across all three retrievers and every reported tier, at depths 100 and 1,000. The accompanying analysis also reports that both additions improve nDCG@1000; all paired 95% confidence intervals for these gains exclude zero (F.2, lines 2186–2189).

These additions show that both incomplete recovery and hindsight gains extend to bibliography and author-attested evidence, with the main figures illustrating the cross-source findings and the appendix documenting their broader scope.

The expanded pooled analysis also incorporates the added citation-graph and supervised baselines. Our “Shared revision summary” and “Presentation changes” comments describe the broader evaluation and provide a guide to the revised paper.

We would welcome your assessment of whether these revisions address your concerns and would be happy to clarify any remaining questions.

Thank you for the detailed response and additional analyses.

The new comparisons directly address my concern about how to interpret differences across exclusion levels. I also appreciate the revised wording on the prior citation history, which aligns with the level of empirical support.

The new results on the diagnostics for the other levels of evidence also address my main concern about the poor classifier precision. However, the classifier’s precision remains a limitation for Context-Root targets, as the human check does not directly improve the precision.

I think the paper is in good standing. I will raise my score.

Thank you for the thoughtful follow-up and for raising your score. To address the remaining limitation on classifier precision, we constructed higher-precision Context-Root labels and repeated every Context-Root result in the paper with them. All findings remain consistent with the original results, indicating that our conclusions are robust to the classifier's lower precision. We keep the original labels in the main text, so the reported results remain comparable with the reviewed version and reproducible without API access, and we report the new analysis in a new appendix.

Follow-up to W2 and Q1. Context-Root label precision

  • What we added. New Appendix H.4 (pp. 47–48, lines 2521–2578). The classifier's teacher, GPT-5.4-mini, reassesses every citation context with a classifier score above 0.2, using the same prompt (Appendix B.3); this includes borderline contexts below the classifier's 0.5 decision threshold. A cited work becomes a target when the teacher labels at least one of its contexts as generative. On the same human-coded contexts used in Appendix B.3, precision rises from 0.63 to 0.83 (recall 0.93 → 0.88).

  • What we learned. Every Context-Root result holds under the higher-precision labels (Table 29, p. 48). Each row gives the test used in the paper, the values for the reference retriever (SPECTER2-Proximity K16), and where the finding holds under the higher-precision labels. Values are percentages; differences and gains are in percentage points, with 95% CIs in brackets.

Finding (section)TestClassifier labelsHigher-precision labelsHolds
Label quality (human-coded contexts)precision / recall0.63 / 0.930.83 / 0.88—
Reach falls beyond prior citations (§4.2)Next > New > Isolated, Hit@10022.82 > 15.26 > 14.0821.59 > 14.48 > 13.62✓ 23/23 configurations
The Next-to-New drop persists after count matching (§4.2)observed − expected < 0, New Hit@100−0.59 [−0.83, −0.36]−0.36 [−0.59, −0.12]✓ 4/4 retrievers
History beats popularity (§4.1)history > popularity, New Hit@10015.26 > 2.3014.48 > 2.26✓ 23/23 configurations, all metrics and depths
The signal is researcher-specific (§4.1)own ÷ matched histories > 1, New nDCG@10008.4×8.6×✓ 3/3 retrievers
Pooling leaves gaps (§4.2)> 50% of New cases unreached by all 24 rankings, depth 10058.0759.16✓
Project information helps (§4.3)gain over history > 0 with CI excluding 0, New Hit@100: + title / + title and abstract+9.72 / +20.24+9.65 / +19.52✓ 3/3 retrievers, also when excluding papers that name a target
Labels cover few author endorsements (D.3)share of endorsements with a generative-use label7.66.0✓
  • The appendix analyses based on Context-Root are also unchanged (placebo titles, pooling in every tier, publication-history size, prior-citation visibility and uncapped BM25 queries).

  • What changes. The only noticeable difference is in absolute levels: on the same cases, Hit@100 for the reference retriever is 0.7–2.1 percentage points lower with the higher-precision labels. Every comparison and conclusion stays the same.

  • What changed in the paper.

    • New Appendix H.4 and Table 29 (pp. 47–48) describe the higher-precision labels and report the full comparison.
    • Revised §5 (p. 9, lines 475–477) now reads:

      "Is a classifier label trustworthy? Its precision is 0.63 on the contexts used to select it; a sensitivity analysis with higher-precision labels (0.83) leaves the conclusions unchanged (Appendix H.4); held-out data would better test generalization."

Thank you again for your careful engagement throughout this discussion. Your comments led directly to the target-count control across the Next, New and Isolated levels and now to this sensitivity analysis, which shows that our Context-Root conclusions do not depend on the classifier's precision. Both have made the paper considerably stronger, and we are grateful for your help in improving it. We would be happy to clarify any remaining questions.

Reviewer_tYbhOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2(+1)
Presentation
1
Soundness
2(+1)
Read review
Paper summary

This paper introduces MUSES, a large-scale benchmark for predicting which scientific papers will influence a researcher's next publication based solely on their historical publication record. Beyond citation overlap, the authors construct two additional sources of evidence for research influence: author-reported influential works and citation-context. It states that a researcher’s past reveals where their next paper will look, and much of the rest depends on the project itself.

Critical strengths
  1. Additional human-grounded evidence of research influence. The author-attestation study is valuable; the authors directly contact researchers to identify literature that influenced their published work. The resulting 580 evaluable cases provide an additional perspective on scientific influence.
  2. The paper introduces a large-scale retrieval benchmark with approximately 1.04 million cases and a candidate pool exceeding two million scientific papers. It provides a useful evaluation setting.
Critical weaknesses
  1. The definition of novel research influence. The paper uses previously uncited literature as a proxy for works that are new to a researcher. Although the authors acknowledge these limitations, the proposed evaluation still captures changes only in citation behavior.
  2. Writing. The paper is hard to read, likely mostly generated by AI.
  3. Limited evidence of practical value for AI-assisted scientific research. The paper states that this work should improve scientific discovery, but the experiments establish predictive retrieval performance rather than improvements in real research workflows.
  4. Representativeness of author-attested influence. The author-attestation study contains 580 evaluable cases, mostly from life science fields. How representative are these cases of the broader scientific literature covered by MUSES?
Questions for authors

See weakness

5 comments

Thank you for recognizing the value of the author-attestation study and the large-scale evaluation setting. We address the four concerns below, identifying the relevant passages in the original submission and the changes made in the revision.

W1. Previously uncited works and the scope of the evaluation

“New” means the researcher had not previously cited the work. Their first exposure is unknown. Revised §3.2 (pp. 4–5) defines that citation filter. Section 3.3 (p. 5) separately explains how we identify targets from bibliographies, citation passages and authors’ direct reports of what shaped their papers.

(a) Previously uncited does not establish unfamiliarity

New is one of the three citation tiers defined in §3.2. The original Appendix B.1, lines 799–800, explicitly stated:

“These are recorded author/coauthor citation-history exclusions; they do not establish first exposure or author intent.”

We agree that shorthand such as “already knew” was broader than this definition and revised it:

  • Table 1 replaces “excludes works the researcher already knew” with “prior-citation exclusions.”

  • §3.2, lines 218–220, replaces the claim about separating anticipation from familiarity with “Together, these tiers evaluate how well retrieval extends beyond the researcher’s and recent coauthors’ prior citation histories.”

(b) The evaluation includes author testimony alongside citation evidence

Regarding “only citation behavior,” MUSES evaluates three evidence sources: Bibliography, identifying works the next paper cites; Context-Root, identifying cited works whose passages describe building on their ideas, framing or methods; and Author-Root, recording works authors themselves report as shaping or inspiring their paper (original §3.3). These sources provide complementary evidence connecting prior works to the next paper.

The large-scale benchmark is defined by citations; the author-collected set is our check against that construct, and the main findings hold on it: reach declines when prior citations are excluded, pooled rankings leave 40.6% of Author-Root-New cases unreached, and project text improves retrieval in hindsight (§§4.2–4.3; Appendices E.4, F.2 and G.1).

We retain the acknowledgment in the original §5 that citations and author reports observe only part of influence. To make the scope clearer, Table 1 replaces “direct evidence of influence” with “evidence beyond bibliography membership.” Revised §3.3, lines 249–252, also explains:

“The label describes how the paper presents its use of a work, which may differ from the influence reported by its authors.”

W2. Writing and presentation

Thank you for the feedback on readability. We revised the main text and appendix to make the task, comparisons and findings easier to follow:

  • Sections 4.2–4.3 (pp. 8–9) now use headings tied to the measured findings: “Retrieval beyond prior citations” and “Project information substantially improves retrieval.”

  • Section 3.7 (pp. 6–7) explicitly identifies the retrievers, evidence sources and comparisons used in each diagnostic.

  • Table 1 (p. 3) and §3.2 (pp. 4–5) now describe the tiers as prior-citation exclusions, keeping evidence sources and tiers distinct.

  • The appendix is reorganized by topic (Appendices C–H) and opens with a contents list on p. 14.

Our separate comment, “Presentation changes,” provides before-and-after wording and locations. The AI-use statement (p. 10, lines 515–517) discloses assistance with editing, code and figures and affirms that we reviewed the AI-assisted work and take responsibility for the final content.

W3. Practical value and evaluation in research workflows

MUSES measures whether a retriever can recover literature connected to a researcher’s subsequent paper. Evaluating whether recommendations help researchers in practice requires a user study. Revised §5 (p. 9, lines 469–473) now proposes testing this with working titles and early project descriptions.

The original submission explicitly acknowledged the further evaluation needed to establish workflow benefits. Original §5, lines 479–481, stated:

“Do entry points help researchers? Appendix D.4 shows examples, but only a user study can measure the benefit.”

We retain this limitation in revised §5, lines 489–490, with the examples now in Appendix D.2, Table 9.

We also narrowed the framing of practical value. The original introduction (lines 151–153) stated:

“Tools that ask researchers for a short description of their next project, such as a working title, could therefore recover a substantial share of what history alone cannot.”

The revised introduction (lines 151–153) now reads:

“These results motivate testing whether a short description of an emerging project, such as a working title, can provide similar gains beyond history alone.”

The revised abstract (lines 033–034) likewise positions MUSES as “a benchmark for developing systems that help researchers discover literature relevant to their next project.”

The revised discussion describes the next step in these terms:

“MUSES provides a common evaluation setting for developing stronger history-based retrievers; prospective user studies can then test whether these systems help researchers find useful literature for emerging projects.”

W4. Representativeness of the author-attested evaluation

The expanded panel spans all five field groups, with life sciences accounting for 55.3% of classified cases versus 19.0% in the MUSES reference sample. Table 7 (p. 23) gives the composition comparison. Voluntary participation limits how far the results generalize within and across fields.

(a) Expanded collection and field analyses

We increased the author-attested evaluation from 580 to 1,633 cases across 57 journals. The collection spans five field groups: life sciences (901 cases, 55.3%), physical sciences (337, 20.7%), health and medicine (172, 10.6%), social sciences and humanities (155, 9.5%), and computer science and mathematics (65, 4.0%). Percentages are among the 1,630 classified cases; three cases are unclassified (Appendix D.1, Table 7).

We added two analyses:

  • A composition comparison with MUSES. Table 7 compares the collection with a uniform sample of 20,000 MUSES papers, showing both field coverage and remaining differences in composition (Appendix D.1, lines 1173–1178).

  • Retrieval results by field. Table 21 reports SPECTER2-Proximity K16 Hit@100 with confidence intervals, making performance variation visible alongside the composition comparison (Appendix E.4). Retrieval is lower on Author-Root-New than on Author-Root-All in every field group (Table 21, p. 39), so this pattern does not depend on the life-sciences majority.

The main paper now reports the expanded collection and directs readers to both analyses (§3.3, lines 231–241).

(b) Retained limitations and clearer scope

The original submission acknowledged the selection limitations:

“They come from authors who chose to respond, and the bibliography-derived shortlist may anchor their answers.”

—Original §5, lines 465–466; retained in revised §5, lines 477–478

We further state the scope of the new composition analysis in Appendix D.1, lines 1177–1178:

“This comparison describes field coverage; voluntary participation and nonresponse limit generalization to the broader MUSES population.”

These additions strengthen the collection’s coverage and allow readers to assess how its composition and retrieval results relate to the wider benchmark.

The revision also expands the retrieval baselines and tests the main diagnostics across all three evidence sources, providing broader support for the empirical findings (§3.5; Figures 3A and 3C). Our “Shared revision summary” and “Presentation changes” comments provide an overview of these additions and a guide to the revised paper.

We would welcome your assessment of whether these revisions address your concerns, including specific examples of any passages that remain unclear.

Thank you for your detailed response. I believe the additional evidence you provided sufficiently addresses my concerns, and I would like to raise my score accordingly.

Reviewer_LfjJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks and seeks an answer to the question: From a researcher’s publication history alone, can a system anticipate the literature their next paper will build on?

The authors of this paper found that history works well for where the researcher will look but poorly for what will be new to them.

Critical strengths
  1. The task is well formulated. The paper uses a cut-off time and excludes previously cited work.
  2. This paper includes an evaluation done by the original authors of the papers under study, which adds value.
Critical weaknesses
  1. The authors use baselines that are too narrow and do not support L440 "the limit is information more than representation".
  2. The author-attested set is selective and reproduces the citation construct (which was supposed to be complemented).
Questions for authors
  1. Have you implemented any graph-based baselines? More specifically, have you compared against citation-graph-based retrieval baselines?
4 comments

Thank you for recognizing the value of the task formulation and author-attested evaluation. Your suggestions helped us broaden the retrieval comparison and clarify both the conclusions and the role of author evidence.

W1 and Q1. Baseline breadth, citation-graph retrieval and the claim at line 440

Yes. We added four citation-graph baselines and expanded the evaluation to 24 configurations. Table 2 (p. 7) includes Personalized PageRank in the main comparison. We withdrew the relative-importance claim at original line 440. Revised §4.3 (pp. 8–9) shows what adding project information contributes within each tested retriever.

(a) Expanded baselines and updated results

  • Citation-graph retrieval. We added Personalized PageRank, co-citation, bibliographic coupling and Adamic–Adar, conditioned on the researcher’s publication history and using citation links available before the cutoff. Appendix C.1 (p. 19) describes all four. Tables 11, 14 and 17 (pp. 27, 31 and 35) report their reach on Bibliography, Context-Root and Author-Root, respectively.

  • Additional retrieval and supervised baselines. We added uncapped BM25, the late-interaction retriever LateOn, Linear-Proximity with citation-neighbour features, fine-tuned SPECTER2-Proximity and Qwen3-Embedding-8B with LoRA, together with matched controls. The revised evaluation covers 24 configurations, described in §3.5 and Appendix C.

  • Effect on the findings. Personalized PageRank is competitive: it has the highest reach on CiteNext and on Context-Root-Next, and on Context-Root-New and -Isolated at depths 100 and 1,000, while dense and adapted retrievers lead elsewhere (Appendix E.5, p. 39). Two patterns persist in §§4.1–4.2 (pp. 7–8): all 23 history-based configurations outperform popularity, and their Hit@100 declines when the researcher’s prior citations are excluded, across all three evidence sources. Even when pooling all 24 configurations, 36–58% of cases across the three New tiers have no target in any ranking’s top 100 (Figure 3A, p. 9).

(b) Revised interpretation and broader hindsight evidence

We removed the claim at line 440 of the original submission, “the limit is information more than representation,” since the original comparison did not establish that relative importance. Revised §4.3 examines how adding project information changes retrieval within each method. We expanded this analysis to three representative lexical and dense retrievers (BM25, SPECTER2-Proximity and GritLM) across all three evidence sources, evaluating the inputs on the same cases within each comparison. Appendix F.2 reports the complete results, and updated Figure 3C illustrates the comparison across sources. Adding the next paper’s title, and then its abstract, improves retrieval throughout these comparisons, providing broader support for the benefit of project information in hindsight.

W2. Coverage of the author-attested set and its relationship to citation evidence

The expanded collection covers 1,633 cases across 57 journals. Table 7 (p. 23) compares its field composition with MUSES. Author reports also identify works absent from the bibliography and connections that the available citation passages do not explicitly describe (Appendix D.3, pp. 24–25).

(a) Broader coverage of the author-attested set

We expanded the evaluation from 580 to 1,633 cases across 57 journals. The collection spans five field groups: life sciences (901 cases, 55.3%), physical sciences (337, 20.7%), health and medicine (172, 10.6%), social sciences and humanities (155, 9.5%), and computer science and mathematics (65, 4.0%). Percentages are among the 1,630 classified cases; three cases are unclassified (Appendix D.1, Table 7). This substantially broadens the empirical basis of the author-attested results.

We added a comparison of the panel’s field composition with MUSES (Appendix D.1, lines 1173–1178; Table 7) and retrieval performance by field (Appendix E.4, Table 21). These analyses make both coverage and variation in retrieval performance visible. The main paper now reports the expanded collection and directs readers to these analyses (§3.3, lines 231–241).

We retain the original submission’s acknowledgment that respondents self-select and that the bibliography-derived shortlist may anchor their answers (revised §5, lines 477–478). The expanded collection and field analyses give readers a clearer basis for assessing coverage and the scope of the findings, while Appendix D.1, lines 1166–1178, explicitly discusses the limits to population-level generalization.

(b) What author reports add beyond citations

Because the shortlist came from the bibliography, most endorsed works are cited. The endorsement adds which cited works the authors regard as shaping the paper, information bibliography membership does not provide. Among endorsements with usable citing text, only 7.6% carry a generative-use label, and where reference lists are complete, 10.6% of endorsed works fall outside the bibliography altogether.

  • Connections absent from the bibliography. Among 313 cases with complete reference lists, 89 of 839 author-endorsed works are certified uncited (10.6%, 95% CI [8.0, 13.4]). These are connections that an evaluation based only on references would miss. Appendix D.3, Table 10 and lines 1309–1312, reports the result and the checks establishing absence. We also added this finding to the main results (§4.1, lines 399–401).

  • Citation-context labels cover only a small share of author endorsements. Of the 4,181 endorsements, 1,640 have usable citing text; of these, 124 receive a generative-use label (7.6%, 95% CI [6.1, 9.1]). The classifier assigns this label to passages judged to describe building on the endorsed work’s ideas, framing or methods (Appendix D.3, lines 1278–1284).

To investigate whether this limited overlap reflected missed descriptions, two annotators reviewed the available passages for 40 sampled endorsements without the label. They found no explicit description of building on the endorsed work. In these audited examples, authors reported influence that the available citation passages did not explicitly describe (Appendix D.3, lines 1291–1296).

We clarified this interpretation in the main paper (§3.3, lines 249–252):

“The label describes how the paper presents its use of a work, which may differ from the influence reported by its authors.”

These additions substantiate the complementary contribution of author testimony: it captures connections that bibliography membership and explicit descriptions of use only partly cover.

We also added a direct target-count control in response to another review. It tests how much of the prior-citation decline is explained by fewer targets and changes in eligible cases (§4.2; Appendix G.3). Our “Shared revision summary” and “Presentation changes” comments describe this and the other updates across the reviews.

We would welcome your assessment of whether these revisions address your concerns and would be happy to clarify any remaining questions.

AuthorsOpenReview ↗

Thank you for your review: a brief summary of our revisions

Thank you again for your thoughtful review. Your suggestions led us to add citation-graph retrievers, broadening the comparison to 24 configurations, and to examine more closely what author reports add beyond citations. Both made the evidence and conclusions of the paper considerably stronger, and we are grateful for them. Before the author–reviewer discussion closes, we wanted to check whether our response addresses your concerns.

W1 and Q1. "The authors use baselines that are too narrow and do not support L440 'the limit is information more than representation'"; "Have you compared against citation-graph-based retrieval baselines?"

  • Yes. We added four citation-graph retrievers (Personalized PageRank, co-citation, bibliographic coupling and Adamic–Adar) and expanded the evaluation to 24 configurations. Personalized PageRank proved competitive, with the highest reach on CiteNext and Context-Root-Next, and now appears in Table 2 (p. 7).
  • We withdrew the claim at original line 440; revised §4.3 instead reports what project information adds within each retriever.

W2. "The author-attested set is selective and reproduces the citation construct (which was supposed to be complemented)."

  • Selective: we expanded the set from 580 to 1,633 cases across 57 journals and five field groups, and compare its field composition with MUSES while stating the limits of voluntary participation (Appendix D.1, Table 7).
  • Reproduces the citation construct: author reports add information beyond citations. Where reference lists are complete, 10.6% of endorsed works are absent from the bibliography, and only 7.6% of endorsements with usable citing text carry a generative-use label (Appendix D.3).

New since our response: higher-precision Context-Root labels. Following another reviewer's comment, new Appendix H.4 (pp. 47–48) repeats every Context-Root result with labels of higher precision (0.63 → 0.83 on human-coded contexts). All findings remain consistent with the original results (Table 29, p. 48), indicating that our conclusions are robust to the classifier's lower precision. This includes the comparison we added in response to your W2: the generative-use share becomes 6.0%, so author reports remain largely complementary to citation evidence.

If you feel these revisions address your concerns, we would be grateful if you would consider reflecting them in your assessment. If any concern remains, we would very much appreciate hearing it and will do our best to address it before the discussion closes.

AuthorsOpenReview ↗

Thank you, and a gentle final reminder

Thank you again for your time and for your review. We understand how busy this period is, so we simply wanted to send a gentle final reminder before the discussion closes. Our summary above addresses each of your points (W1, Q1 and W2) directly, and we would be very grateful to know whether our revisions have addressed your concerns.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Shared revision summary

Thank you to all reviewers. Your questions led us to broaden the evaluation, test alternative explanations and make the conclusions more precise. Below we summarize the principal changes and identify the concerns they address. Each reviewer reply provides detailed results and manuscript locations.

Dark-grey margin bars identify selected substantive additions and revised interpretations discussed in our responses. The full submission diff shows the remaining edits.

  1. Broader retrieval baselines (LfjJ, W1/Q1). We expanded the evaluation to 24 configurations, adding four citation-graph retrievers, LateOn, uncapped BM25 and additional supervised models. Matched frozen controls for the adapted encoders help distinguish the effect of training from differences in scoring. Personalized PageRank now appears in Table 2. The full comparison shows that the leading method depends on the evidence source, tier, metric and depth, and that adaptation does not improve retrieval uniformly (§3.5; Appendices C and E).

  2. Expanded author-attested evidence (LfjJ, W2; tYbh, W4). We increased the collection from 580 to 1,633 evaluable cases across 57 journals and five field groups. New composition and retrieval-by-field tables help readers assess coverage and performance variation. We extended the comparison of author reports with bibliography membership and citation-context labels. Appendix D.1 documents how author responses become eligible targets; Appendix E.4 explains how record coverage affects graph and popularity results: eligible targets these methods cannot rank remain in the evaluation and count as misses (§3.3; Appendices D.1, D.3 and E.4).

  3. Broader evidence for the researcher-specific signal. The matched-history analysis now evaluates three representative lexical and dense retrievers across all three evidence sources and every reported tier. The researcher's own history outperforms the matched histories for both nDCG@1000 and Hit@100, with all reported confidence intervals excluding zero. Appendix F.1 specifies the matching rules for each evidence source and gives the paired comparisons (§4.1; Table 22).

  4. Pooled recovery across evidence sources (X4Cj, W2/Q1). We expanded pooled recovery from Context-Root-New to all three evidence sources and incorporated all 24 configurations. Figure 3A shows the best individual and pooled Hit@100 on the New tiers; Table 24 reports every tier at depths 100 and 1,000. Even this larger union leaves 36–58% of New-tier cases without a target in any ranking's top 100 (§4.2; Appendix G.1).

  5. Hindsight findings across sources and methods (X4Cj, W2/Q1). We expanded the hindsight analysis to BM25, SPECTER2-Proximity and GritLM across Bibliography, Context-Root and Author-Root. Adding the next paper's title, and then its abstract, improves Hit@100, Hit@1000 and nDCG@1000 across every reported tier, with paired confidence intervals excluding zero. Figure 3C illustrates the cross-source comparison; Appendix F.2 gives the broader results and sensitivity checks (§4.3; Table 23).

  6. A direct test of the target-count explanation (X4Cj, W1). We added comparisons across citation-exclusion tiers that hold cases fixed and account for target count. Target count and changes in eligible cases explain part of the aggregate decline. After fixing the cases and matching target counts, New-tier scores remain below expectation across the four tested retrievers and all three sources. The additional coauthor exclusion has smaller or less consistent effects. We incorporated this qualified finding into §4.2 (Appendix G.3, Table 26).

  7. More precise conclusions (LfjJ, W1; tYbh, W1; X4Cj, W1). We removed “the limit is information more than representation.” Revised §4.3 reports the gains from project text in hindsight across the tested methods and sources. We replaced familiarity-based shorthand with descriptions of recorded prior citations and renamed §4.2 “Retrieval beyond prior citations,” preserving the distinction between citation history and first awareness (Table 1; §§3.2 and 4.2–4.3).

  8. Higher-precision Context-Root labels (X4Cj, W2/Q1). New Appendix H.4 repeats every Context-Root result with labels of higher precision (0.63 → 0.83 on human-coded contexts). All findings remain consistent with the original results (Table 29, p. 48), indicating that our conclusions are robust to the classifier's lower precision.

We also expanded or updated the supporting analyses:

  • Sensitivity to history and query construction. Appendices H.1–H.2 examine centroid count and BM25 query construction, helping readers assess dependence on these design choices.
  • Hindsight sensitivity checks. Appendix F.2 extends the detected-target-mention check to the expanded Author-Root collection. Both Hit gains persist; incomplete metadata remains an explicit detection limit. It also reports the placebo-title comparison for the two dense retrievers on Bibliography and Context-Root.
  • Encoder exposure and adaptation overlap. Appendix H.3 reports a training-free BM25 control, a SciRepEval training-edge overlap audit for SPECTER2, and an encoder trained before the test period (original SPECTER), which shows the same decline across tiers. It also shows that the directions and confidence-interval conclusions of the adapted-versus-frozen Author-Root Hit@100 comparisons persist after excluding identified training- and validation-researcher overlap.
  • Evaluation populations and metrics. The revised results use a common eligible cohort within each evidence source and tier. Section 3.6 retains concise metric explanations, while Appendix E.1 consolidates formal definitions and evaluation rules. Appendix E reports reach, target coverage and ranking quality separately, showing where they lead to different method comparisons.

The separate “Presentation changes” comment explains how the expanded appendix, revised figures and table organization help readers interpret and navigate this evidence.

AuthorsOpenReview ↗

Presentation changes

We revised the main text, figures and appendix to help readers understand the task, follow the numerical comparisons and locate the supporting evidence. The changes below describe how the presentation has changed; the separate “Shared revision summary” covers the added experiments and revised conclusions. Page numbers below refer to the revised PDF.

Reading guide. Figure 1 and §§3.1–3.3 introduce the task and evidence sources. Table 2 and Figures 2–3 present the main comparisons and findings. Appendix E gives complete retrieval results; Appendices F–H give the diagnostics, controls and robustness checks.

Main-paper explanations and figures

We made the distinction between evidence sources and prior-citation tiers more explicit, replaced wording that implied knowledge or first exposure, and specified the scope of the reported results. Revised captions identify the methods and evaluation cohorts used in each panel.

AreaOriginal submissionRevised submission
§4.2 heading (p. 8)“Hard means new to the researcher, not new to the field”“Retrieval beyond prior citations”
§4.3 heading (p. 8)“Much of what is missing is the project itself”“Project information substantially improves retrieval”
§3.7 diagnostic scope (pp. 6–7)Matched-history and hindsight diagnostics centered on single-retriever analyses; pooling used seven rankings on Context-Root-NewEach diagnostic states its retrievers, evidence sources and comparison. Matched histories and hindsight cover three retrievers and all three sources; pooling covers 24 configurations across sources
Results explanations (§§4.1–4.3, pp. 7–9)Numerical examples and broader findings were not always distinguished explicitlyRevised numerical statements identify the relevant method, evidence source, tier, metric and depth; the text distinguishes figure examples from the broader findings supported by the appendix
Table 1 and §3.2 (pp. 3–5)“excludes works the researcher already knew”; “separates genuine anticipation from familiarity”“prior-citation exclusions”; “evaluate how well retrieval extends beyond the researcher’s and recent coauthors’ prior citation histories”
Metrics (§3.6, p. 6; E.1, p. 25)§3.6 included Hit and macro Recall equations alongside interpretation and evaluation-population details§3.6 gives concise metric explanations; E.1 consolidates formal definitions of Hit, macro Recall, nDCG and MRR and states the evaluation rules
Figure 1 (p. 2)Panel B “three novelty levels”; Author attestation 580 cases; Context-Root 6,707 casesPanel B “three prior-citation tiers”; Author attestation 1,633 cases; Context-Root 6,659 cases; panel C titles and values updated (“Reach beyond prior citations,” “Project information helps”)
Figure 2 (p. 8)“Newer works are harder”; matched histories scored with base SPECTER2“Reach beyond prior citations”; both panels use SPECTER2-Proximity with 16 centroids; sample sizes stated in the caption
Figure 3 (p. 9)A: seven rankings on Context-Root-New. B: cited versus uncited targets. C: hindsight on Context-Root-NewA: best individual versus pooled Hit@100 on each source’s New tier. B: CiteNew reach versus target coverage at three depths. C: hindsight Hit@100 across all three sources’ New tiers. The caption identifies the methods and cohort sizes

Appendix organization and tables

We reorganized the appendix to support verification as well as navigation. Parallel results sections explain what each metric shows, while the diagnostic tables place the relevant comparisons, case counts and uncertainty together. Accompanying explanations identify each analysis's question, scope and main finding.

AreaOriginal submissionRevised submission
Appendix organizationSections by topic area (e.g. “Matched histories and retrieval depth”)Appendices C–H: methods, E.1–E.5 complete results, F diagnostics, G complementarity, H robustness
Appendix navigationNo contents listAppendix contents list with page numbers
Complete results, E.2–E.4, Tables 11–21 (pp. 26–39)Tables and supporting discussion organized differently across evidence sourcesParallel source sections separate Hit, macro Recall and ranking quality; accompanying prose explains method, tier and depth patterns, helping readers assess where conclusions depend on the metric
Matched histories, Table 22 (p. 40)Base-SPECTER2 bibliography comparison of own and matched historiesThree source panels report paired own-history advantages and intervals for three retrievers, two metrics and the reported tiers, making the expanded scope and uncertainty visible together
Hindsight, Table 23 (p. 42)Context-Root hindsight levels for one retriever, with shared cases across inputsSource panels group H, H+T and H+T+A within each of three retrievers and state tier-specific case counts; this makes the same-case input comparisons visible across sources
Pooled recovery, Table 24 (p. 43)Seven-ranking Context-Root-New table listed individual methods and their unionBest individual and pooled reach appear side by side for all three sources and tiers at depths 100 and 1,000, showing complementarity and remaining gaps
Target-count control, Table 26 (p. 45)No direct table comparing successive citation-exclusion tiers after target-count adjustmentThree source panels align New versus Next and Isolated versus New for both metrics, with confidence intervals below estimates; the caption explains the meaning of negative differences

We also standardized terminology and corrected local wording, typography and cross-references throughout the manuscript. Dark-grey margin bars identify selected substantive revisions discussed in the replies; the full submission diff shows the remaining edits.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.