acceptodds
Under review as a conference paper at ICLR 2027

NovCompass: Scientific Novelty Analysis through Traceable Paper Comparison

Abstract

Automated novelty assessment is increasingly explored to support peer review. Existing systems typically retrieve related work and use language models to compare it with a submission. However, identifying relevant overlaps, distinguishing the aspects they concern, and supporting each judgment with evidence from both papers remain difficult. We present NovCompass, a system that analyzes novelty along three dimensions: task, problem and method. For each dimension, NovCompass retrieves candidate prior work and compares every candidate with the submission. A comparison is retained only if the passages it quotes can be found in both papers, and each overlap claim in the final report links to the comparisons it draws on. The comparator, NovCompass-8B, is trained on responses from a teacher model, GPT-5.5, selected for agreement with relationships identified by reviewers and survey authors. On human-reviewed paper pairs, GPT-5.5 misses 46% of the annotated problem overlaps, and the student recalls 14.2 percentage points more. Among correct method-overlap judgments, the share with unlocatable quotations falls from 10.9% for the base model to 1.0% after fine-tuning. On 225 ICLR submissions, most of the relationships NovCompass misses are lost at retrieval: 39% of the annotated relationships are never retrieved under their dimension, while our comparator recognizes 91% of the retrieved ones.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
scientific novelty assessment, scientific document understanding, evidence-grounded reasoning, scholarly information retrieval, large language models, AI-assisted peer review

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Rfe8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

NovCompass retrieves prior work per dimension (task/problem/method) and compares candidates with a fine-tuned 8B comparator (Qwen3-8B) trained on GPT-5.5 outputs filtered by reviewer/survey overlap relations; comparisons are kept only when both quotes are programmatically located. On NovGauge it recalls 14.2 points more annotated problem overlaps than its teacher (p = 5e-6) at 99.4% observed precision (32 problem negatives); end-to-end on 225 ICLR submissions it recovers 54.3% of reviewer-identified relations (ceiling 61.3%).

Critical strengths
  1. Principled supervision. Reviewer/survey relations filter teacher answers rather than serve as ground truth, addressing teacher conservatism (45.7% of positive problem candidates rejected).
  2. Judge-free evidence. The headline comparison uses paired bootstrap CIs and McNemar tests, avoiding the unstable LLM-judged Verified F1 (four Gemini judges accept only 31.3–50.3% of the same rationales).
  3. Strong evaluation practice. Unseen-paper split (+18.2), blind audit (κ = 0.82), wrong-paper control (3%), and evidence that retrieval dominates end-to-end loss.
Critical weaknesses
  1. Causal claim unidentified; shared provenance. No unfiltered-student control is reported, so the +14.2 gain is tied to filtering only correlationally. Training relations and NovGauge labels come from the same extraction pipeline (14.5% of benchmark papers appear in training), so annotation conventions may be learned instead of novelty.
  2. Thin, partly unfavorable precision. With 32 problem negatives, 99.4% vs. 98.5% is roughly one false positive; on method the student's FPR is higher (25.5% vs. 20.3%); audit confirmation is 66.4% vs. 82.6% with overlapping CIs, and the net utility of "more total, less often right" is not quantified.
  3. "Traceable" does not mean supported; the framing overreaches. Quote location checks string presence, not entailment; Mis is 27.8–45.9% (GPT-5.2) / 33.5–63.1% (Gemini) and judge sensitivity is high (31.3–50.3%; GPT-5.5 method VF1 falls from 63.7 to 43.1). The system detects three-dimension overlap, not novelty in the broader sense.
  4. Attribution and ceiling gaps. The +7.3 rewrite gain is confounded with route count and budget (61.3% vs. 54.0%); regeneration's net accuracy effect is unreported (9,697 regenerations; 362 vs. 355 flips); truncation (16k tokens/paper; 93,491 comparisons) is unanalyzed; and the 61.3% ceiling rests on incomplete annotations, with index coverage of the 33/71 missed AC-named works unknown.
  5. External validity and positioning. ICLR/ML only (225 submissions; 4 task instances); index, weights, and scripts are unreleased, so the dominant failure mode cannot be audited. Appendix I also shows GPT-5.5 without web captures 78% of AC positioning judgments vs. 40–49% for the full report.
Questions for authors
  1. Can you train the unfiltered-student control (same recipe and compute)?
  2. What is the net utility of the recall/precision trade-off, and is the lower confirmation rate concentrated in method?
  3. Of the 33/71 AC-named works missed by retrieval, how many are outside the index? What is the truncation impact, and what is regeneration's net effect on decision accuracy?
5 comments
AuthorsOpenReview ↗

Response to Reviewer Rfe8 (1/4): the unfiltered control

Thank you for a careful and detailed review. Two of your points will make the paper's claims more precise: most of the rewrite gain came from the larger budget, and the unfiltered control shows where the filter helps and where it does not. For this response we trained the unfiltered student with two seeds, had two annotators check training labels and report statements against the source texts, recomputed the rewrite gain at a matched budget, and analyzed regeneration, truncation and index coverage. The response is in four comments.

Q1 / W1. Unfiltered-student control

We trained a student on the unfiltered teacher outputs with the same recipe as NovCompass-8B: the same GPT-5.5 inference pools, 13,264 training rows matched to the filtered set on source, dimension and candidate label, the same NovGauge pair exclusion and validation set, and the same training settings (Qwen3-8B, full fine-tuning, one epoch, global batch 64, 208 steps). Its training set keeps the teacher's answer even when it disagrees with the relation label. We trained it with two seeds.

Recall on NovGauge (decision only, no LLM judge, mean of 3 samples):

TaskProblemMethod
Teacher (GPT-5.5)75.453.670.5
Unfiltered student, seed 169.549.252.5
Unfiltered student, seed 263.544.048.4
Filtered student (NovCompass-8B)74.267.573.3

The teacher row follows the paper (Table 5). Differences below are paired, on the instances both models answered. McNemar counts use majority votes, so they do not map exactly onto the mean recall.

  • Filtered minus unfiltered on problem: +18.3 (95% CI 14.0 to 22.8, McNemar 30 against 3) for seed 1 and +23.5 (18.9 to 28.4, 47 against 2) for seed 2. On method: +20.9 (16.9 to 24.8) and +24.9 (20.8 to 29.1).
  • Both unfiltered students stay below the teacher on every dimension. Fine-tuning on the teacher's own answers does not get past the teacher.

Two caveats apply to this comparison.

  • Operating point and ranking. The filtered student also asserts more overlaps on method negatives (false-positive rate 25.5%, against 15.5% and 12.9% for the two unfiltered seeds). To separate operating point from ranking, we read each student's probability of "true" at the verdict token and compared ROC curves. On method, the filtered student ranks better with both seeds (AUC 0.854 against 0.838 and 0.838, difference 0.016 with a 95% CI of 0.003 to 0.030 for each seed). Thresholding these probabilities instead of sampling answers, it recalls 80.0% at the teacher's false-positive rate, against 70.4% and 71.2%. On problem, the unfiltered students rank almost as well (AUC 0.953 against 0.939 and 0.946, differences not significant) but assert far fewer overlaps. So on problem, filtering mainly removes the conservatism the student would otherwise learn from the teacher, and on method it also improves ranking.
  • Target format. The raw teacher outputs usually omit a rationale when the teacher answers "different", so those targets carry an empty rationale in the unfiltered set. The control therefore tests the training policy as a whole, label filtering together with complete-response selection.

The filtered student is a single training run. The two unfiltered seeds differ by about 5 points on problem recall, and the gain from filtering (18.3 and 23.5 points) is several times larger. We will replace the limitation "the effect of filtering alone is not isolated" with these results and caveats.

AuthorsOpenReview ↗

Response to Reviewer Rfe8 (2/4): shared provenance, precision and net utility

W1. Shared provenance

We agree this concern is hard to rule out completely, and we offer three pieces of evidence.

  1. Unseen papers. On instances where neither paper appears in training or validation, which removes the overlap you point to, the problem gain over the teacher is +18.2 (95% CI 11.3 to 25.3), larger than on the full benchmark.
  2. Limits of the control. Both students see the same paper pairs from the same pipelines, and only the targets differ, so exposure to the pipeline alone does not produce the gain. The control cannot rule out that the filtered student learns labelling conventions that NovGauge shares. The evidence against this is the blind audit, which does not use these labels: about 60% of the assertions that only the student makes are confirmed by humans (Q2).
  3. Label quality. Two annotators, blind to label, teacher answer and source, labelled 120 pairs. They agreed on 30 of the 36 pairs both labelled (κ = 0.66). Most of these were contested pairs, where agreement was lower (18 of 23) than on training targets (12 of 13). Of the sampled pairs used as training targets, where label and teacher agree, humans confirmed 39 of 40 (97.5%). On the 80 label-positive pairs the teacher rejected, humans judged 40% to overlap (47.5% on problem). The teacher's rejections on training candidates therefore mix conservatism with label noise, and we will describe the 45.7% rejection rate that way. The evidence for conservatism is on NovGauge, whose positives were reviewed by humans and where GPT-5.5 misses 46% of the problem overlaps. The filter discards the contested pairs, which keeps the teacher's "different" answers on them out of training.

Q2 / W2. Precision and net utility

We agree with each point. With 32 problem negatives, 99.4% against 98.5% is about one false positive, so we will state that no loss of precision is observed and that 32 negatives are too few to compare. On method the student's false-positive rate is higher (25.5% against 20.3%, 9 against 4 discordant pairs, p = 0.27). The audit's confidence intervals overlap.

Is the lower confirmation rate concentrated in method? Yes. Breaking down the blind audit (Appendix H) by dimension, with the same stratum weighting:

Confirmation rate of NovCompass-8B's assertionsEstimate (95% CI)Audited cases
Task95.0 (83.4 to 100)15
Problem64.6 (47.0 to 80.9)33
Method49.3 (30.5 to 67.4)28
Assertions GPT-5.5 also makes77.5 (62.1 to 91.7)24
Assertions only NovCompass-8B makes59.5 (45.9 to 71.8)52

Where both comparators assert an overlap, the student's confirmation rate (77.5%) is close to the teacher's overall rate (82.6%) and to its rate on those same assertions (84.6%, 95% CI 70.2 to 96.5). The gap comes from the assertions that only the student makes, of which about 60% are still confirmed.

Net utility. Among the 1,315 unannotated comparisons in the audit pool, NovCompass-8B yields an estimated 59 more confirmed overlaps than GPT-5.5 (161 against 102), at the cost of 60 more unconfirmed assertions (81 against 21). The extra assertions are therefore about half confirmed, lower than the 60% above, because this count also subtracts the assertions that only GPT-5.5 makes. If a reader assigns a cost λ to checking an unconfirmed assertion, relative to the value of one confirmed overlap, the student is preferable whenever λ < 0.98 overall (0.86 on problem, 0.78 on method). Every assertion arrives with its two quotations, so checking one means reading two passages. Whether that is cheaper than missing a relevant prior work is a judgment the reader can make with these numbers. The per-dimension cells are small, so we report these figures as descriptive.

AuthorsOpenReview ↗

Response to Reviewer Rfe8 (3/4): "traceable", the rewrite gain, and regeneration

W3. Are the reports supported, not only traceable?

Quote location checks that both spans exist, not that they support the rationale, and the paper defines the term this way (Sections 3.1 and 3.5). Whether the reports are supported is an empirical question, and we have now measured it directly. The revised introduction will also state the scope precisely: the system analyzes each contribution against the earlier work it finds, showing with quoted evidence where they overlap, so that the reader can judge what is new.

To measure semantic support directly, the two PhD students from the original audit, working independently, checked the 77 comparative statements in 10 randomly drawn released reports (20 of them by both) against the full texts of both papers, not only against the quoted spans. The stated relation holds for 94.8% of the statements (95% CI 90.4 to 98.7, resampling reports) and holds partly for the rest. None was judged wrong. The facts stated about the two papers are correct for 96.1% and partly correct for the rest. The most common issue was an overstated overlap, in about four statements. On the 20 statements both labelled, they agreed on every relation judgment and on 19 of 20 factual judgments. This measures the final reports, which use only retained comparisons, and the report writer is instructed to make a narrower claim or omit a comparison when the quotations do not support it. The audit in Q2 measures the comparator's own assertions.

Our central comparison continues to rest on judge-free decision statistics, as you note.


Q3 / W4. Attribution and ceiling

Rewrite gain (+7.3). You are right that this is confounded with budget. We recomputed it from the saved top-250 rankings of every route, with the production admission rules. This reproduces the paper's settings within two instances: 60.6% for five routes and 53.3% for three. We then gave the three description routes the same candidate budget as the five routes (376 against 372 unique candidates per target):

RoutesCandidates per targetAll (302)Problem (101)Method (197)
5 (with rewrites), paper budget37260.664.458.9
3 (descriptions), paper budget21853.360.449.7
3 (descriptions), matched budget37658.666.354.8

At a matched budget, the rewrite routes add 2.0 points overall, not 7.3. Most of the original gain came from the larger budget. The remaining gain is on method (+4.1 points, 8 relations), consistent with their purpose of restating a method without the authors' coined names. On problem, spending the budget on the description route works as well or better. We will correct the attribution in the paper and report it by dimension.

Regeneration. Of the 9,697 regenerations, 91.1% of the comparable ones kept their judgment, and the flips were nearly balanced (362 from negative to positive, 355 from positive to negative). On the reviewer-annotated instances, 26 comparisons were regenerated: 18 recovered located quotations and none changed from positive to negative. On the annotated relations, regeneration therefore does not reduce decision accuracy, and it raises verified recall from 76.8% to 88.6%. For the unannotated flips, we sampled 100 in each direction and asked a judge from another model family (Gemini 3.1 Pro), blind to which answer came first, whether the papers overlap. It answered "no overlap" for 93 of the 100 negative-to-positive flips and 94 of the 100 positive-to-negative flips. It treats nearly all flipped pairs as non-overlapping, so this check alone says little about which answer is right. We therefore checked 20 of the flips by hand, 10 in each direction. We judged all 20 pairs non-overlapping, agreeing with the judge on 19. So the negative-to-positive flips were wrong and the positive-to-negative flips were right in equal numbers, and with balanced flips (362 against 355) regeneration has no net effect on decisions. The main effect of regeneration is to recover located quotations.

Reviewer_mgYPOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents NovCompass, a system for identifying task-, problem-, and method-level overlaps between scientific papers through structured extraction, embedding-based retrieval, and LLM-based comparison. Experiments on 225 ICLR submissions evaluate its retrieval and comparison components, showing improved overlap recall but persistent limitations in retrieval coverage and the reliability of overlap judgments.

Critical strengths

The paper addresses a practical problem in scientific literature analysis and provides a structured alternative to document-level similarity or unconstrained LLM judgments. Its evidence-linked reports and evaluation on real ICLR submissions offer practical value.

Critical weaknesses
  • Limited novelty and unclear long-term value. The engineering is solid, and I appreciate that the authors avoid reducing novelty to a single score, instead providing dimension-specific overlap judgments and supporting evidence. However, JEV-like approaches and related early work in paper evaluation (https://openreview.net/forum?id=rNl8XiSHiJ) raise questions about the long-term value and distinctiveness of this approach.

  • Insufficient methodological insight. The paper presents a feasible engineering pipeline but offers limited theoretical discussion of why this particular decomposition and comparison strategy is appropriate. The contribution feels more like an integration of existing techniques than a sufficiently distinctive methodological advance for ICLR.

  • Insufficient ablation studies and implementation analysis. The authors do not compare Qwen3-Embedding-0.6B, 4B, and 8B, or systematically evaluate alternative embedding models and reranking strategies. The PDF-to-Markdown conversion process is also underspecified. For arXiv papers, using their HTML versions through ar5iv could avoid some errors introduced by PDF parsing and layout analysis.

  • The necessity of fine-tuning is unclear. Why fine-tune Qwen3-8B for structured extraction when general-purpose model APIs could potentially perform the same task? The paper lacks comparisons with cost-effective API baselines, such as GPT-4.1 mini, GPT-5.6 Terra, and DeepSeek Flash, in terms of extraction quality, downstream retrieval performance, cost, and latency. The benefit of maintaining separate fine-tuned models is therefore not sufficiently established.

  • The comparator evaluation does not establish overall superiority. Table 3 shows higher recall for the fine-tuned model than GPT-5.5 on reviewer-annotated overlaps, but reviewer annotations are incomplete, and overlap does not necessarily imply a lack of novelty. In the human audit, only 66.4% of the fine-tuned model's sampled overlap assertions were confirmed, compared with 82.6% for GPT-5.5. Higher recall alone is therefore insufficient to establish better judgment quality or practical value.

Questions for authors
  • I would encourage the authors not to overemphasize fine-tuning simply to make the work look more like a conventional ML paper. The community welcomes meaningful contributions through problem formulation, rigorous evaluation, and valuable insights, even without fine-tuning or introducing new chain-of-thought methods. Could future work focus more on identifying the fundamental insights behind the task and establishing when specialized models are genuinely necessary?

  • Have the authors explored a simpler embedding-only baseline for novelty assessment? For example, could the similarity scores of the top-50 retrieved papers be aggregated to estimate novelty, with lower average similarity suggesting greater novelty? Such an experiment would help clarify whether the proposed multi-stage pipeline provides substantial benefits beyond embedding-based retrieval alone.

4 comments
AuthorsOpenReview ↗

Response to Reviewer mgYP (1/3): JEV, decomposition, ablations

Thank you for the careful review and the concrete suggestions. The API comparison you asked for made us state more precisely why the extractor runs locally. Your question on when specialized models are needed gave us a clearer way to state what the training adds. For this response we ran five new experiments: JEV on NovGauge, three embedding sizes and two rerankers, three API extractors (one of them also with downstream retrieval), a student trained without filtering, and the embedding-only baseline you suggested. The response is in three comments.


W1. JEV, NAIPv2 and long-term value

NAIPv2 gives one quality score to a whole paper and is trained on review scores. It does not say which earlier paper overlaps, or in what way. We will add it to the related work.

JEV gives a decision only. We ran JEV (typesafe/jev-1.13) on all 875 NovGauge instances, with the same prompt and both full texts as our comparator. It cost about $0.001 per pair. AUCs below are on the 866 instances that every method in this response scored. We also read our comparator as a decision model. It takes one forward pass on the same prompt, and we use its probability of "true" at the verdict token.

AUCTaskProblemMethod
JEV0.9090.9590.814
NovCompass-8B, one forward pass0.9170.9530.854

The two are tied on task and problem. On method, the hardest dimension, our comparator is better (difference 0.041, 95% CI 0.014 to 0.068). So cheap decisions are possible, and our model gives them too. A decision alone gives no evidence. A reader who sees "overlap: yes" still has to find where the overlap is. Our model gives the decision and, when it is positive, the two quotes and the reason. The two uses also combine well. On NovGauge, skipping the full output when the probability is below 0.01 skips 45.5% of the negative pairs and loses 3.6% of the positives.


W2. Why this decomposition

Much new research recombines known parts (see Q1), so whether a paper is novel depends on which part is new. We therefore describe overlap separately for the task, the problem and the method, and leave the judgment of novelty to the reader. The labels are consistent with this. Of the 463 positive pairs in NovGauge, 296 (64%) are labelled as overlapping on one dimension only, and 19 on all three. The other dimensions of these pairs are unlabelled rather than labelled as different. A single similarity score would mix these cases. Comparing pairs is what lets the system say which earlier paper overlaps and in which part, which you noted as a strength.


W3. Ablations and parsing

Embedding size and rerankers. Re-encoding the 2.2M-paper index with other models is not possible within the rebuttal period. On the same NovGauge pairs as in W1, we scored each pair from titles and abstracts, by embedding cosine similarity and by a reranker's relevance score:

AUCTaskProblemMethod
Qwen3-Embedding-0.6B0.8780.9700.724
Qwen3-Embedding-4B0.8720.9600.704
Qwen3-Embedding-8B0.8810.9680.651
Qwen3-Reranker-0.6B0.9010.9600.666
Qwen3-Reranker-4B0.9030.9430.698
NovCompass-8B comparator (reads both full texts)0.9170.9530.854

Larger embedding models and rerankers do not help here, and on method all of them stay well below the comparator. Embeddings are good at finding related papers, which is how we use them. They are not enough to decide whether two papers share a method. On problem, some of them score slightly higher than the comparator. That column rests on only 32 negatives, and these negatives are easier to separate by similarity than real retrieved candidates. Inside the pipeline, similarity still ranks reviewer-named works high, which is why we use it for retrieval. What the comparator adds there is shown in Q2.

Parsing. In our experiments, PDFs were converted to Markdown by a document-parsing service. The released code accepts Markdown directly, can parse PDFs locally with pymupdf, and takes any other parser through one interface. Before quotes are located, both texts are normalized for common PDF artifacts such as ligatures, hyphens at line breaks and spacing. We will describe these rules in the appendix. ar5iv HTML is a good suggestion for arXiv papers. It would mostly help on the side of earlier papers, since submissions under review are usually not on arXiv.

AuthorsOpenReview ↗

Response to Reviewer mgYP (2/3): fine-tuning and the audit

W4. Is fine-tuning needed?

Extractor. API models extract about as well as our extractor, and we keep a local one for cost and for consistency over the index. We ran GPT-4.1 mini, GPT-5.6 Terra and DeepSeek-V4-Flash on the extractor's 137 held-out validation papers with the same prompt. A GPT-5.1 judge, asked whether the five fields are well-formed, specific and useful for retrieval, gives all four extractors between 4.6 and 5.0 out of 5 (ours 4.64). What matters downstream is retrieval. In live retrieval for 75 submissions, our fields placed 59 of the 102 reviewer-named works in the candidate pool, and GPT-4.1 mini fields placed 55.

Every one of the 2.2 million papers in the index must go through the same extractor, since targets and index are matched field by field. At the token counts we measured, that would cost about USD 22,000 with GPT-4.1 mini and USD 59,000 with GPT-5.6 Terra, and again for every re-extraction. On latency, GPT-4.1 mini and GPT-5.6 Terra took 6.5 and 8.8 seconds per paper. The cheapest option, DeepSeek-V4-Flash, took 28.7 seconds and gave 12 invalid outputs out of 137. A local model, whose weights we will release, avoids this cost and does not change over time.

Pairwise comparator. Cost matters most here, since one submission needs about 416 pairwise comparisons. With GPT-5.5 at list price and without prompt caching, this costs about $60 per submission. Cheaper APIs lose recall: GPT-5.4 mini finds 39.7% of the problem overlaps and DeepSeek-V4-Flash 43.0%, against 67.5% for our comparator. Gemini 3.1 Flash-Lite has high recall, but its false-positive rate on method is 33.6%, and 72.8% of its correct problem rationales are not supported by its quotes, against 37.1% for ours under the same judge. A GPT-5.5 call takes 15 seconds and calls can run in parallel. One RTX 3090 runs about 320 of our comparisons per hour, about 1.3 hours per submission. So the cheaper APIs we tested either miss many problem overlaps or give many more rationales that their quotes do not support, which is why we train a local comparator.


W5. Higher recall is not better judgment

We agree, and we do not claim overall superiority. By dimension, the audit confirms 95.0% of the student's task assertions, 64.6% of problem and 49.3% of method. Where GPT-5.5 makes the same assertion, 77.5% are confirmed, close to its own 82.6%. The gap comes from the assertions that only the student makes, and about 60% of those are still confirmed. In counts, among the 1,315 unannotated comparisons in the audit pool, we estimate 161 confirmed overlaps against 102 for GPT-5.5, with 81 unconfirmed against 21. We also agree that overlap does not mean a lack of novelty. The system reports where a paper meets earlier work and does not decide whether it is novel (Appendix J).

In short, the trained comparator matters for method decisions, for the evidence it attaches, and for which works reach the report, at a small fraction of the cost of a frontier API.

AuthorsOpenReview ↗

Response to Reviewer mgYP (3/3): what lies behind the task, and the embedding baseline

Q1. Insights behind the task, and when specialized models are needed

Thank you sincerely for this comment in particular. It looks past the details to what the work is for. Here we answer what lies behind the task.

How we see novelty. Few papers are new in every part. Most research recombines known parts (Weitzman 1998, Uzzi et al. 2013). A paper may apply a known method to a new problem, or bring a new method to a known problem. So "is this paper novel" cannot be answered with one score. The real question is which part is known, which part is new, and whether the new thing is the combination itself. A paper can share its problem with one earlier work and its method with another, while no earlier work has both.

What we ask instead. We do not judge novelty. We do the step before it. For the task, the problem and the method, we find which earlier work overlaps, and where both papers say so. What remains, and whether the combination counts, is for the reader to judge. As our report rules state, separate precedents for separate parts do not show that one earlier paper did the combination (Appendix K).

Where the task is hard. The task has three steps.

  1. Finding the earlier work. 39% of the overlaps are never retrieved under their dimension, while 91% of the retrieved ones are recognized.
  2. Seeing that two things are the same when the words differ. Authors give new names to their methods, so the same mechanism appears under different names. At the same retrieval budget, queries that drop these names recover 2.0 more points of the overlaps, and 4.1 on method. Models have the same difficulty. 15 of the 16 models published with NovGauge find at most 64% of the problem overlaps, with high precision. On NovGauge pairs, a yes or no decision is the easier part, and JEV matches our comparator on task and problem, and plain embeddings match it on problem. Judging a shared method and showing where the overlap lies are harder.
  3. Deciding what the remaining difference means. This is a value judgment and should stay with people.

When a specialized model is genuinely needed. It is needed for the part of step 2 that general models do poorly. Our comparator is trained only on teacher outputs that agree with overlaps named by reviewers and survey authors. For this response, we trained a student with the same recipe and data size on the teacher outputs without this filter. It finds 49.2% of the problem overlaps (44.0% with a second training seed), below the teacher's 53.6%. The filtered student finds 67.5% (difference 18.3 points, 95% CI 14.0 to 22.8). So fine-tuning alone does not get past the teacher. What helps is the filtering signal, a record of expert judgment on which works are really the same thing. A specialized model is also needed when scale rules out an API, since each paper is compared with hundreds of earlier works. It is not needed for extraction quality, or for yes or no decisions alone, where a decision model such as JEV does about as well on task and problem.

Where this could lead. Knowing which earlier work each part of a paper overlaps with would let recombination be studied from the content of papers, not only from citations: which problems and methods have already been put together, and which combinations are new. This matters more as generated research ideas multiply. Retrieval for methods is still our weakest step, and overlap is not the same as a lack of novelty.

The revised introduction will state this view, with the trained comparator as one way to act on it.


Q2. Embedding-only baseline

For the 194 submissions with a usable meta-review, we averaged the similarity of the top 50 retrieved papers. This score hardly predicts whether the AC raised an overlap with earlier work (AUC 0.59) or whether the paper was rejected (AUC 0.51). A score from the number of overlaps in our reports does no better (0.55 and 0.50). So a single score does not work for either system.

The useful output is the list of specific works. We gave the embedding ranking the same number of works as each NovCompass report (65 on average). Counting a work as found if it appears anywhere in the report, under any dimension, NovCompass reports include 68.2% of the 236 reviewer-named works and the embedding ranking 50.0%, a difference of 43 works (95% CI 31 to 56). The paper's 58.1% also requires the reviewer's dimension and located quotes. For the 69 distinct AC-named works that resolve to a paper (Appendix I's 71 counts repeated mentions), it is 29 against 22 (difference 7, CI 1 to 14).

Thank you again for these suggestions. They made the claims of the paper more precise.

References

Weitzman, M. L. (1998). Recombinant growth. The Quarterly Journal of Economics, 113(2), 331-360.

Uzzi, B., Mukherjee, S., Stringer, M., and Jones, B. (2013). Atypical combinations and scientific impact. Science, 342(6157), 468-472.

Reviewer_mgYPOpenReview ↗

Comments

Thanks for the additional experiments. They address many of my concerns, but the reliability of method-overlap judgments remains questionable. I'll keep my original score of 2.

Reviewer_sBJ5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper presents NovCompass, a system that helps researchers find out how a new scientific paper is similar to earlier work. It looks for three kinds of aspect-specific similarities - whether the papers try to achieve the same goal, address the same problem, or use similar methods. The system uses several LLMs. Some identify similarities between papers mentioned in published reviews and surveys. These similarities are used to prepare training examples for a smaller LLM. When a new paper is submitted, another LLM describes its main task, problem, and method. The system then searches about 2.2 million scientific papers, compares the relevant ones, and prepares a report. The report includes quotations from both papers so that readers can examine the evidence. The authors test the trained model on 619 previously reviewed paper pairs.

Critical strengths
  1. A really interesting result is that a smaller trained model outperforms its much stronger teacher on a difficult scientific comparison task.

  2. The paper identifies where a complete scientific comparison system actually fails -- i.e., the retrieval is the main bottleneck.

Critical weaknesses
  1. It is unclear what new scientific knowledge or practical value this paper adds. The paper combines several existing ideas - using LLMs to find related papers, comparing papers on different aspects, checking quotations, and generating reports. The authors train an 8B model that improves problem-overlap recall over its teacher. This is a useful result, but it is unclear what we learn about scientific novelty assessment besides the performance of this particular system. For example, suppose Alice proposes a new method to make neural networks faster. Bob's earlier paper addresses the same problem using a different method. Charlie's earlier paper already uses a method very similar to Alice's. A system might correctly identify that Alice and Bob address the same problem. But that tells us little about whether Alice's proposed method is original. Finding Charlie's paper and understanding the similarity would be much more important.

    The practical results also raise questions. NovCompass captures only 49% of area-chair judgments about how a submission relates to earlier work, compared with 75% for the GPT-5.5 reference. Since the comparison uses different resources, they are not comparable. The paper demonstrates an improvement on an overlap-detection benchmark, but does not convincingly establish a new scientific insight or a better capability for assessing scientific contributions.

  2. Several important stages depend on LLM-generated outputs whose correctness has not been sufficiently checked. The proposed system uses LLMs at several stages:

  • LLMs extract relationships between papers from reviews and surveys.
  • Another agent checks these relationships and assigns task, problem, or method labels.
  • These labels determine which GPT-5.5 responses are kept for training.
  • A separately trained LLM extracts the task, problem, and method from a new paper.
  • The trained comparator judges overlaps, while another LLM writes the final report.

The human judgments are not sufficiently large and also not a representative set. As an example, the system can reject a correct teacher judgment because an earlier automated stage made a mistake. Also, agreement between two LLM agents does not establish that their scientific judgment is correct. I do not see that the reliability of the entire training-label collection is established. Similar problems can occur later as well. If the paper extractor misses an important contribution, retrieval may never search for its earlier precedents. At the same time, if the comparator makes an incorrect judgment, the report writer may carry that error into the final report. Also, finding real quotations does not mean that the quotations support the conclusion. The paper's own human audit reveals this problem. Only 66.4% of NovCompass's asserted overlaps were confirmed from their supplied evidence, compared with 82.6% for GPT-5.5. --- i.e., the presented evidence was insufficient to confirm many of the assertions.

Questions for authors

Questions

  1. What new scientific understanding or practically useful capability does the paper establish beyond combining existing techniques?
  2. How reliable are the scientific judgments produced by the several automated LLM stages, particularly the labels used for training?
  3. Do the final evidence-linked reports correctly explain the relationships between scientific contributions?

Suggestions

  1. A controlled comparison showing that NovCompass helps identify important prior work and correctly explain what is genuinely new in a submission, beyond what existing systems can do.
  2. Independently check the generated labels against human judgments, particularly cases where the teacher disagrees with the label. Also evaluate the paper extractor and measure how errors at each stage affect the final report. I suggest that human evaluation of the scientific correctness of the final report's claims, using the relevant source-paper context shoud be done.
7 comments
AuthorsOpenReview ↗

Response to Reviewer sBJ5 (1/3): what the paper establishes, and Alice, Bob and Charlie

Thank you for the careful review, and for the Alice, Bob and Charlie example, which explains what is at stake better than our paper did. You asked for human checks of the training labels and of the final reports, and we ran both for this response. Two annotators labeled 120 training pairs, 80 of them cases where the teacher disagrees with the label, and checked 77 report statements against the full texts of both papers. We also trained the unfiltered student with two seeds, traced where missed relations are lost, and compared our reports with an embedding ranking at equal length. The response is in three comments.


Q1 / W1. New understanding beyond combining techniques

We see three findings from our evaluation.

  1. Frontier models systematically miss shared problems. Recomputed from the released predictions of NovGauge, 15 of its 16 published models find at most 64% of the problem overlaps that reviewers and survey authors identified, at high precision. GPT-5.5 misses 46%.
  2. Relation-filtered training reduces this blind spot. For this response, we trained a student with the same recipe and data size on the teacher outputs without the filter. It finds 49.2% of the problem overlaps (44.0% with a second seed), below the teacher's 53.6%. The filtered student finds 67.5%. So fine-tuning on the teacher's answers alone does not get past the teacher, and the gain comes with the filtering policy (details and caveats are in our reply to Reviewer Rfe8).
  3. The remaining bottleneck is discovery. End to end, 39% of the annotated relations are never retrieved under their dimension, while the comparator recognizes 91% of those that are retrieved.

On practical value, the reports find works that an embedding ranking misses. Given the same number of works per submission, NovCompass reports include 68.2% of the 236 reviewer-named works, and the embedding ranking 50.0% (difference 43 works, 95% CI 31 to 56). It also runs locally: one submission needs about 416 comparisons, which cost about USD 60 with GPT-5.5 and take about 1.3 hours on one RTX 3090 with our comparator.


The Alice, Bob and Charlie example

We agree that finding Charlie matters more for judging whether Alice's method is original. Your example makes an important distinction: better recall of shared problems does not, by itself, establish better assessment of a claimed methodological contribution. The system keeps the two apart by comparing task, problem and method separately. Its report would say that Bob shares Alice's problem and that Charlie's method is close to Alice's. A single similarity score would merge the two. A paper can share its problem with one earlier work and its method with another, so the report states which part overlaps with which work.

Charlie is also where the system is weakest, and we will state this clearly in the revision. Retrieval misses 80 of the 197 method relations that reviewers named, against 35 of the 101 problem relations. In the blind audit, 49.3% of the student's method assertions are confirmed, against 64.6% for problem.

Once Charlie is retrieved, the comparator recognizes the overlap better than the alternatives we tested. On method, its decision AUC is 0.854, against 0.814 for the decision model JEV and at most 0.724 for embedding similarity and rerankers. It also ranks method overlaps better than the unfiltered students. Most of the loss is in finding Charlie, and, as you anticipated, it starts at extraction. Judging method overlap also remains imperfect, as the 49.3% above shows. Of the 117 reviewer-named relations that were not retrieved under their dimension, 56 were retrieved under another dimension, which points to a failure in routing the earlier paper to the relevant comparison rather than in finding it at all. We checked 20 of the 117 by hand, stratified by a first-pass LLM attribution. The extracted fields fully described the shared aspect in 2, described it only partly in 11, and did not describe it in 7. The typical gap is a specific element, such as the earlier model being reused, which the fields describe only in general terms, so retrieval has little to match. This points to a concrete improvement: keeping in the fields the specific earlier methods a paper builds on.

AuthorsOpenReview ↗

Response to Reviewer sBJ5 (2/3): AC judgments, and the reliability of each LLM stage

W2. Area-chair judgments (49% against 75%)

We agree the gap is real, and where it occurs is informative. NovCompass's full report and GPT-5.5 have similar capture rates on contribution summaries (90% against 89%) and on judgments that an aspect is new (58% against 56%), where GPT-5.5 more often contradicts the AC (25% of these judgments, against 11% for NovCompass). The gap lies in judgments linking an aspect to named prior work: NovCompass captures 40%, against 78% for GPT-5.5. Web search alone does not explain it, since GPT-5.5 without search still captures 78% of positioning judgments. Its capture is not higher on older submissions (69.0% on ICLR 2023 to 2025, 79.5% on ICLR 2026), which does not suggest memorization, though we cannot exclude it.

We read the gap as discovery again: 33 of the 71 resolved AC-named works (69 distinct) are absent from our reports, mostly because retrieval missed them, and all 23 of these we could check are in our index. Given the same list length, our reports still include more AC-named works than the embedding ranking (29 against 22, difference 7, CI 1 to 14). GPT-5.5's statements are not linked to located quotations, and letting a model propose candidates from its own knowledge while the comparator verifies them is the natural next step.


Q2 / W3. Reliability of the LLM stages

We agree that agreement between two LLM agents does not establish correctness, so this response adds human checks at each stage.

  • Training labels. Two annotators, blind to label, teacher answer and source, labeled 120 pairs (κ = 0.66 on the 36 both labeled). Of the sampled pairs used as training targets, humans confirmed 39 of 40 (97.5%). On the 80 cases you asked about, where the teacher disagrees with a positive label, humans judged 40% to overlap. This changes our interpretation of teacher disagreement: it reflects both missed overlaps and errors in the labels, not teacher conservatism alone. Neither source is reliable on these contested pairs, and the filter keeps only pairs where label and teacher agree. So an error in an earlier stage can remove a correct teacher answer, as you say, but it then removes the pair from training rather than adding a wrong target. An incorrect retained overlap label requires both the label and the teacher to be wrong.
  • Extractor. Our check in 1/3 shows that the extracted fields often describe the shared aspect only in general terms. Replacing our extractor with an API model did not improve retrieval in our test: for 75 submissions, GPT-4.1 mini fields placed 55 of 102 reviewer-named works in the candidate pool, and ours 59. What the fields keep is therefore at least part of the problem.
  • Comparator. Beyond NovGauge, whose positives were reviewed by humans, the blind audit confirms 95.0% of the student's task assertions, 64.6% of problem and 49.3% of method. Where GPT-5.5 makes the same assertion, 77.5% of the student's assertions are confirmed, close to GPT-5.5's overall rate (82.6%) and to its rate on those same assertions (84.6%, 95% CI 70.2 to 96.5). The gap to 66.4% comes from the assertions only the student makes, of which about 60% are confirmed. Of the 29 audited student assertions that were not confirmed, the annotators judged 27 as no overlap and 2 as cannot tell, so these are mostly wrong assertions rather than cases where the evidence was missing.
  • Report writer. See Q3 below. The report uses only retained comparisons, so a comparator error can reach it. The human check measures how often the stated relations still hold.

The samples were drawn at random within fixed strata before any annotation was read. They are modest in size, so we give the counts behind each number.

AuthorsOpenReview ↗

Response to Reviewer sBJ5 (3/3): report correctness and comparisons with existing systems

Q3 / S3. Do the reports explain the relationships correctly?

Two annotators, working independently, checked the 77 comparative statements in 10 randomly drawn released reports (20 of them by both) against the full texts of both papers, not only against the quoted spans. They judged whether the facts stated about each paper are correct and whether the stated relation holds. The relation holds for 94.8% of the statements (95% CI 90.4 to 98.7, resampling reports) and holds partly for the remaining 5.2%, with none judged wrong. The facts are correct for 96.1% and partly correct for the rest. The most common issue was an overstated overlap, in about four statements. On the 20 statements both labeled, they agreed on every relation judgment and on 19 of 20 factual judgments. The report writer is instructed to make a narrower claim or omit a comparison when the quotations do not support it (Appendix K). This check covers the final reports against the full texts, while the audit in 2/3 covers the comparator's assertions against its own quotations, so the two rates are not directly comparable.


S1. Comparisons with existing systems, and the value of additional prior work

The paper compares discovery with existing systems on 100 targets (Appendix F). Under the dimension the reviewer named, NovCompass finds 58.3% of the 132 relations, against 45.5% for GPT-5.5 with web search (McNemar p = 0.03) and 37.9% for Semantic Scholar search. GPT-5.5 finds slightly more of the works when the dimension is ignored (79.6% against 72.8%, not significant). The systems search different corpora and return very different numbers of candidates (about 374 candidates per target for NovCompass, compared with 22 for GPT-5.5), so this is a reference point rather than a controlled baseline. For agreement with AC judgments about which aspects are new, the reports and GPT-5.5 with search have similar capture rates, and GPT-5.5 more often contradicts these judgments by attributing the aspect to earlier work (2/3).

Beyond the equal-length comparison with an embedding ranking in 1/3, the reports go past what reviewers named. A report contains on average 64 earlier works that the reviewers did not name, and 92% of these are not cited by the submission. Eleven of them were later named by the AC in the meta-review. We took 30 of these works, chosen among those that overlap with the submission on at least two dimensions and are not cited by it, so they are the strongest candidates rather than a random sample. We mixed them with 10 reviewer-named works and judged them without knowing which were which. All 30 overlap with the submission at least partly (4 fully, 26 partly). We judged that the submission should have discussed 25 of them (83%, 95% CI 66 to 93) and maybe the other 5, and none was irrelevant. All 10 reviewer-named works were judged as ones to discuss. These are our own judgments, so we offer them as indicative. We agree that a fully controlled comparison against other systems would be the stronger test, and we will note it as a limitation.


S2. Labels against human judgment, and error propagation

The training-label check targets the teacher-disagreement cases you named (2/3), and the extractor and stage attributions are in 1/3 and 2/3.

Summary of revisions. (1) State the three findings above in the introduction, together with the scope: the system analyzes each contribution against the earlier work it finds, showing with quoted evidence where they overlap, and the judgment of novelty stays with the reader. (2) Report the unfiltered control and the human checks of training labels and report statements, and revise our reading of teacher disagreement accordingly. (3) Report the audit by dimension, with method as the weakest. (4) Add the stage-wise attribution of missed relations.

Thank you again. If you agree, we would like to use your Alice, Bob and Charlie example in the revision, credited to an anonymous reviewer, to explain why we keep the dimensions separate.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.