acceptodds
Under review as a conference paper at ICLR 2027

Topic Is Not Agenda: What Text Embeddings Miss in Scientific Retrieval and What Citation Structure Adds

Abstract

Research agents retrieve candidates by text similarity and read only the top of the list, so what text similarity misses, they never see. In this paper we measure how far text similarity is from finding papers that share a study's problem and approach, using the documents that authors cite as gold, and we propose a method that uses the citations of nearby papers to close part of this gap. We test on a public must-cite benchmark (MasterSet), a 4M-paper corpus in eight scientific domains, and US patents. The top 100 by text similarity miss 61% of the most essential papers an author cites, the baselines, benchmarks and methods its experiments build on. In the 4M-paper corpus, 30.8% of the cited papers that it holds lie beyond cosine rank 1,000, where no reranker of the text candidates can reach them, and updating the query vector does not help on average. Text embeddings find the topic of a paper, not its specific problem and approach. The method, neighbourhood citation retrieval, asks the query's nearest text neighbours what they cite, adds those documents as candidates, and ranks them by the neighbours' citations. Its training-free form, the neighbour vote, cuts these misses to 43%, and a learned form cuts them to 35%. On the same MasterSet candidates the vote beats citation popularity and a trained Citeomatic-style reranker, and adding the vote and related neighbourhood signals to that reranker raises R@100 from .556 to .660. Fine-tuning the encoder on citations does not replace the vote, which still adds to every fine-tuned encoder we tested. We also analyse where the gain comes from with several controls, including the removal of the query's own organisation (its co-authors or its assignee) from its neighbours. Because it needs no training and uses new citations as soon as they appear, the approach can serve as a retrieval interface for research agents in collections where cosine similarity is saturated, such as scientific papers and patents, where many documents share a topic and text alone cannot tell which of them share a study's problem and approach.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
scientific literature search, dense retrieval, citation recommendation, citation graph, research agents, retrieval evaluation, patent retrieval

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_63eQOpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies citation-retrieval failures of text embeddings and proposes neighbourhood citation retrieval, which adds and ranks papers cited by a query’s nearest textual neighbours. Across scientific papers and patents, it measures retrieval gains and their sources, raising MasterSet R@100 from 0.392 to 0.569 without training while testing popularity and organizational effects.

Critical strengths
  1. The training-free neighbour vote raises MasterSet R@100 from 0.392 to 0.569 with a simple, interpretable mechanism.
  2. Shared-candidate comparisons and degree-preserving rewiring distinguish useful citation structure from popularity and candidate-pool expansion.
  3. Temporal, co-author, assignee, and matched-removal controls reveal meaningful differences between scientific retrieval and patent citation recovery.
Critical weaknesses
  1. Citations and citation communities proxy research agenda, but no independent labels validate shared problems or approaches.
  2. The broad text-saturation claim exceeds the tested encoders; oracle fitted vectors do not establish limits of trained late-interaction retrieval.
  3. The 4M-paper corpus contains only 11% of query references on average, so its recall results concern a selective bibliography subset.
Questions for authors
  1. Can experts label shared problems and approaches for cited and uncited candidates, then evaluate whether voting improves agenda relevance?
  2. Can you test a trained late-interaction retriever under the same temporal and retrieval-budget constraints, and measure voting’s additional gain?
  3. Can you expand a query subset to near-complete reference coverage and report full-bibliography recall alongside in-corpus recall?
1 comment

Follow-up for Reviewer 63eQ: near-complete reference coverage (Q3)

We ran the test you asked for. With near-complete coverage the gain holds, with the same encoder and the same queries. A corpus of about 170M papers, built from OpenAlex, Crossref and Semantic Scholar records, holds 91% of each query's references on average (median 94%), against 11% in the paper's 4M corpus. Over each query's OpenAlex reference list, the vote adds +9.5 R@100 points [+9.0, +10.0] to cosine, and over the references in the corpus +10.5, against +8.0 in the 4M corpus (table). On the 532 queries with at least 95% of their references in the corpus it adds +9.6. So the low coverage of the 4M corpus did not inflate the gain.

Setup. The queries, rules and time constraint are those of the paper corpus (§3, App. B), and the follow-up to Reviewer 18DR gives the scale results of the same run. The text model is the public Qwen3-Embedding-0.6B with exact cosine search, and other versions of a query (the same title, or cosine at least .95) are removed like the query itself. The table uses the 1,168 of the paper's 1,171 queries that both corpora can score. Coverage is the number of a query's references in this corpus's citation graph divided by the length of its OpenAlex reference list. Full-list recall divides by all of the query's references in OpenAlex (or by the in-corpus count when larger, 12 queries), so references outside the corpus count as misses.

CorpusCoverageIn-corpus: cosine / voteGainFull OpenAlex list: cosine / voteGain
Paper's 4M corpus11%.470 / .550+8.0 [+6.8, +9.2]––
About 170M papers91%.138 / .243+10.5 [+9.9, +11.0].126 / .221+9.5 [+9.0, +10.0]

At 170M papers, the 1,025 of these queries with at least 80% of their references in the corpus gain +9.6 [+9.1, +10.2] over the full list, and the 532 with at least 95% gain +9.6 [+8.8, +10.4]. We will add these results to App. L.

Reviewer_Chv6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines the limitations of text similarity-based paper retrieval in distinguishing between literature that shares similar research topics but differs in specific problems or approaches. To address this limitation, the paper adds the references cited by papers similar to the query to the search candidates and combines a voting ranking based on the number of neighboring papers citing each document with the cosine similarity ranking. While search performance improved in scientific literature without additional training, its effectiveness on patent data varied depending on whether neighbors with the same patent holder as the correct citation were included.

Critical strengths

S1. Performance improves without additional training. Since MasterSet R@100 improves solely through neighbor voting, it has practical applications as it can be added to existing search systems without complex training.

S2. This paper verified whether the performance improvement was simply due to adding more candidates. Even when compared to popularity metrics for the same candidates, neighbor voting yielded better results, and, with feature elimination experiments included, the results provide evidence that neighbor citation information is helpful.

S3. It analyzed whether the improved performance was due to citations from the same laboratory or company. An example of this is comparing removing co-authors and patent holders’ neighbors with random removal.

Critical weaknesses

W1. Limited Novelty Compared to Prior Work. The paper makes only limited new contributions compared to previous research. The method of expanding the candidate set by referencing similar papers has been used before, and co-citations from the same candidate show performance similar to that of neighbor voting.

W2. Insufficient Analysis of Parameter Choices. The rationale for using 100 neighboring papers and 1,000 candidate papers is not sufficiently explained. Since there is no performance comparison when these numbers are changed, it is difficult to determine whether the effectiveness is limited to the current settings. Furthermore, since each neighbor is assigned one vote regardless of similarity to the query, it would be helpful to verify whether this choice is appropriate by comparing it to similarity-weighted voting.

W3. Limited Analysis of Failure Cases. The explanation regarding conditions under which performance declines is insufficient. In citations of semiconductor patent applicants, popularity outperforms neighbor voting, and removing neighbors with the same patent holder causes voting performance to drop to 0.174. Although the influence of organizations was analyzed, providing examples of which specific documents are incorrectly recommended would make it easier to understand the method’s limitations.

W4. Limited Benefits of Combining Multiple Signals. Combining multiple signals does not always lead to improved performance. Although the Shapley contribution was presented, it is unclear which documents are being promoted at the expense of correct answers, so additional analysis of cases where performance declines is needed.

W5. Insufficient Evidence for the Limitations of Text-Based Retrieval. It is difficult to conclude solely based on the experimental results that the text fails to identify the specific problems and approaches addressed in the papers. It is necessary to verify whether the same discrepancy occurs when using the full text or applying actual multi-vector search.

Questions for authors

Q1. What criteria were used to select the 100 neighboring examples and 1,000 candidate examples? Were there any results obtained by changing these numbers?

Q2. Did you compare this approach to a method that weights votes based on similarity to the query?

Q3. I reviewed the comparison of k values in the appendix, but why was k=60 chosen for the main experiment?

Q4. Were the Gemini embeddings generated only once? If regenerated, would the retrieval results remain consistent?

Q5. What is the main difference between the proposed method and existing citation-based retrieval methods?

Reviewer_18DROpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the limitations of dense text embeddings in scientific retrieval and citation recommendation, arguing that text similarity captures overall "topic" rather than a paper's specific "problem and approach" (its agenda). Using a 4M-paper corpus across 8 scientific domains, the MasterSet must-cite benchmark, and US patent datasets, the authors show that text similarity (cosine rank) misses a substantial fraction of essential citations—30.8% of cited papers in the 4M corpus lie beyond cosine rank 1,000. Updating query vectors or fine-tuning encoders on citations fails to close this gap.

To address this, the authors propose neighbourhood citation retrieval. The method identifies a query paper's top 100 nearest text neighbours, extracts the documents cited by those neighbours, adds them to the candidate set, and re-ranks candidates using a training-free local citation count termed the neighbour vote (fused via Reciprocal Rank Fusion) or a learned classifier. The neighbour vote increases MasterSet R@100 from 0.392 to 0.569 (and to 0.650 for the learned variant), outperforming corpus-wide citation popularity and a Citeomatic-style reranker. Finally, using organisation controls (removing co-authors or same-assignee patents), the paper demonstrates that in science, performance gains originate primarily from the broader research community rather than self-citation/co-author networks.

Critical strengths

Clear Empirical Demonstration of the "Topic vs. Agenda" Limitation: The paper provides a thorough empirical characterization showing where dense text embeddings saturate. Quantifying that 30.8% of gold references sit beyond cosine rank 1,000, where downstream rerankers cannot access them, is an interesting finding for researchers building retrieval-augmented LLM research agents.

Critical weaknesses

Missing Scalability and Deployment Cost Analysis: The paper explicitly motivates the method as a retrieval interface for AI research agents operating over large, constantly growing literature collections. However, empirical analysis of retrieval time, memory footprint, graph index overhead, or update costs is not provided in the paper. Furthermore, while real-world scientific search engines and agentic systems operate over tens of millions of documents, the evaluations are limited to a 4M-paper subset.

Questions for authors

Sensitivity to Neighborhood Size (kk): How sensitive is the neighbour vote to the choice of the number of nearest neighbours (and how is the $ k=100$ chosen)?

Cold-Start Performance: How does the method perform when query papers sit in sparse text embedding regions where the 100 nearest neighbours themselves have low cosine similarity to the query or very few references?

1 comment

Follow-up for Reviewer 18DR: the vote at about 170M papers

As promised, we ran the vote on a corpus of about 170M papers built from OpenAlex, Crossref and Semantic Scholar records. The gain holds at this scale. The vote adds +10.9 R@100 points [+10.4, +11.4] to cosine on the full corpus, more than on a random 4M subset (+6.6), and at every size it closes about the same share of what cosine misses, (vote − cosine)/(1 − cosine), as in the paper (13–16% here, against 14.0–14.8% at 1M and 4M in App. M).

What cosine misses grows with the corpus, and the vote keeps closing part of it. When the gold documents are fixed and only more papers are added, cosine recall falls by two thirds from 4M to 170M papers, because many more papers compete for the top 100. The vote keeps adding about 7 to 13 points at every size (Table B).

Cost. The citation graph of about 2.3B citations over about 220M nodes takes 11.1 GB, as estimated in our answer (4 bytes per citation and 8 per document). The vote reads only the 100 neighbours' reference lists, so its work is bounded by the length of those lists, not by the number of papers searched. The cost that grows is the first stage: exact search over all 170M papers took about 11 minutes for all 1,419 mapped queries, in one pass over the corpus on one CPU machine. A deployment would use an approximate index, with which the vote keeps its gain at 4M (App. M).

Setup. The queries, rules and time constraint are those of the paper corpus (§3, App. B), and the gold is each query's references in this corpus's citation graph. Other versions of a query (the same title, or cosine at least .95) are removed like the query itself. The text model is the public Qwen3-Embedding-0.6B with exact cosine search. On the paper's own 4M corpus (with this corpus's citation graph) this encoder gains +8.1 [+6.9, +9.4] from the vote, against +6.4 with Gemini, so the gain holds with both encoders. The 4M, 16M and 64M sets are nested random subsets. A query is scored on a set when at least one of its references is in it, so the number of queries differs by set.

Table A. Corpus size (R@100; gold = the query's references in each set).

CorpusQueriesCosineVoteVote − cosineShare of misses closed
4M (random)813.474.541+6.6 [+5.1, +8.3]13%
16M (random)1,326.343.449+10.6 [+9.3, +11.9]16%
64M (random)1,408.219.346+12.7 [+12.0, +13.4]16%
About 170M (all)1,411.148.257+10.9 [+10.4, +11.4]13%

Table B. Same gold, more papers (the 813 queries' references inside the random 4M subset, scored in each larger set).

CorpusCosineVoteVote − cosine
4M.474.541+6.6 [+5.1, +8.3]
16M.340.434+9.4 [+7.3, +11.5]
64M.215.343+12.9 [+10.7, +15.0]
About 170M.156.248+9.2 [+7.2, +11.3]

Paper's 4M corpus with the same encoder: cosine .469, vote .550; with Gemini .558 and .623. With HNSW at 4M the vote gains +6.2 against +6.4 with exact search (App. M). We will add these results to App. M.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.