- Rating
- 3(+1)
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper builds a small reading list covering distinct contribution-level threads likely to recur later. It extracts claims from titles and abstracts, builds threads via semantic similarity and an anchor gate, and weights them by early uptake or a cross-fitted predictor. A greedy coverage objective avoids double-counting, with an optional LLM prior trading coverage for landmark recall. On pre-2021 GNN and federated learning papers, with 2021–2022 supervision and 2023–2024 evaluation, the predicted variant beats external baselines on late-uptake coverage; observed-count weighting gives the highest coverage, LLM scoring the highest landmark recall.
Critical strengths
-
The paper treats a reading list as a whole, rather than simply picking the highest-scoring papers. Grouping shared contributions into knowledge threads gives it a clear way to avoid redundant content. It also usefully distinguishes recovering established classics from covering ideas that recur in later research.
-
The uptake-weighted selections cover more later semantic-continuation events than the external baselines in both fields. The authors check alternative text representations, matching thresholds, and stricter data splits, and openly report where the simpler observed-count variant performs better than the learned predictor.
Critical weaknesses
-
The method relies on DeepSeek-V4-Flash to extract contributions from titles and abstracts truncated to 2,000 characters. It is unclear whether this captures enough technical detail to identify meaningful knowledge continuation, rather than just similar descriptions. A small human audit of the extracted claims and matched contributions would help establish whether this representation is reliable enough for selecting papers worth reading.
-
The added value of the learned predictor is unclear. Simply weighting threads by their observed early uptake gives better overall coverage in both fields. The predictor could still help identify ideas before they gain attention, but the cold-start evidence is limited and does not yet make a strong case for when prediction is preferable to the simpler approach.
Questions for authors
-
Could you provide a small human audit of the extracted claims and continuation matches, using the source papers as a reference? This would help assess whether DeepSeek-V4-Flash with the 2,000-character abstract limit captures the key contributions, and whether the matches reflect meaningful technical continuation rather than just topical similarity.
-
The evaluation covers only two fields, graph neural networks and federated learning, both within machine learning. It remains unclear whether the approach works as well in fields with different writing conventions and patterns of knowledge development.
6 comments
Response to questions (part 2)
Q2: Only two fields, both in machine learning
We agree testing on another field is useful, and we have added lattice quantum chromodynamics (lattice QCD), the numerical study of the strong interaction in theoretical physics. It differs from machine learning both in writing conventions and in how knowledge develops: 1) Abstracts report numerical determinations with uncertainties rather than benchmark wins. Only 0.5% of extracted claims are classified as performance claims. 2) Progress accumulates through long-running collaborations that build on shared gauge-field ensembles and refine methods and precision over many years. Every setup is identical to the main paper except the corpus (240 in the ≤2020 history pool, 200 in 2021–22, and 120 in 2023–24).
| LUC (%) | K = 20 | K = 50 |
|---|---|---|
| WCov(pred) | 51.3 | 74.1 |
| WCov(obs) | 59.8 | 79.3 |
| citations | 27.8 | 54.8 |
| PageRank | 27.8 | 53.0 |
| LLM per-paper score | 23.3 | 47.4 |
| LLM set selection | — | 44.8 |
| GNN-CF | 15.2 | 42.8 |
| Random (20 seeds) | 24.3 | 48.4 |
WCov(pred) leads every external baseline by 19.3–36.1 points. With the independent lexical matcher it still leads by 13.7–27.2 points. We also perform a sanity check. The three threads with the most late continuations correspond to central lines of the field over this period:
- lattice computation of parton distributions via quasi- and pseudo-distributions (9 history papers; continued by 27 early and 16 late papers);
- reuse of the MILC highly improved staggered-quark ensembles (6 history papers; 26 early, 15 late);
- lattice calculation of the hadronic contributions to the muon g−2 (10 history papers; 13 early, 13 late).
The revision will add lattice QCD to the main results, and the appendix will document the corpus and the run.
Thank you for the detailed response. The truncation statistics make the 2,000-character limit less of a concern for me, and the lattice QCD results go a long way toward addressing my concern about whether the method works outside machine learning. The expert audit was also helpful, especially the distinction between a contribution appearing again and being substantively used or extended. The cold-start results make it clearer where the learned predictor might be useful.
My remaining reservation is about how broadly to interpret the results. The evidence is stronger for selecting papers whose ideas recur in later work than for identifying papers that later work substantively builds on. I’m also still unsure how well the method would hold up over longer time periods. Given my limited familiarity with this area, I’m less confident about how much weight to give these reservations. I appreciate the care that went into the response, and I don’t have any further requests for experiments in this round.
Response to questions (part 1)
We thank the reviewer for the clear summary. We also appreciate the recognition of the set-level formulation, of the distinction between recovering established classics and covering ideas that recur in later work, and of our transparent reporting. We hope to address your concerns.
Q1 & W1. Is extraction from titles and abstracts reliable, and do matches reflect technical continuation?
Titles and abstracts are where authors state their contributions, and every paper in a field has them. This lets the method scale to whole fields. The 2,000-character limit rarely binds. It affects 27 of 478 GNN abstracts (5.6%), 45 of 564 FL abstracts (8.0%), and 34 of 560 lattice-QCD (another field) abstracts (6.1%). And truncated abstracts lose a median of 237, 128, and 319 characters, i.e., the last one or two sentences. In GNN and FL these sentences mostly report results or code availability rather than contributions, and performance claims are excluded from threads in any case. In the revision we will report the results without truncation.
We conducted a blinded expert audit. To concentrate expert effort on difficult cases, we first screened a stratified sample of 971 items (400 extracted claims, 200 shared threads, and 371 thread-to-late-paper events) with two independent LLM passes. We sent every item on which the two passes disagreed or failed to a domain expert, subsampling within sampled merges and uptake events. For each item, the expert saw the original titles and abstracts, but not the method identity, predicted weights, similarity scores, or LLM verdicts.
For claim fidelity, of the 400 sampled claims, 387 were judged faithful by both LLM passes. Of the 13 remaining claims reviewed by the expert, 10 were judged faithful and none unfaithful. The other 3 could not be assessed because the source record contains no abstract. For uptake matches, the expert reviewed 132 distinct thread-to-late-paper events, covering 200 matched claim pairs. Of these events, 130 (98.5%) continue the content of the historical contribution, consistent with our definition of uptake as later claims that restate, use, or extend a thread. Among them, 22 (16.7% of all events) are substantive use or extension, while 108 continue the same contribution. Only 2 events (1.5%) were unrelated or mismatched. We will release the annotation guideline together with the labels after the discussion period.
Performance against baselines. Among audited events covered by WCov(pred) at K=50, 78 of 79 are continuations, 9 of them substantive. Among those covered by the strongest external baseline, 53 of 54 are continuations, 8 of them substantive. The single mismatched event is covered by both. We therefore find no evidence that the events credited to our selection are less reliable than those credited to the baseline.
W2: Added value of the learned predictor
Observed counts and the predictor are two instantiations of the same uptake weight, and the paper reports both. We agree that observed early uptake gives higher overall coverage when a dense early window is available, and the revision will recommend it as the default in that setting. The predictor serves a different case: threads on which early counts carry no information.
As for cold start evidence, we took the 10 cold threads ranked highest by the predictor's frozen cross-fitted scores, and counted how many were continued in the late window.