NovCompass: Scientific Novelty Analysis through Traceable Paper Comparison

Automated novelty assessment is increasingly explored to support peer review. Existing systems typically retrieve related work and use language models to compare it with a submission. However, identifying relevant overlaps, distinguishing the aspects they concern, and supporting each judgment with evidence from both papers remain difficult. We present NovCompass, a system that analyzes novelty along three dimensions: task, problem and method. For each dimension, NovCompass retrieves candidate prior work and compares every candidate with the submission. A comparison is retained only if the passages it quotes can be found in both papers, and each overlap claim in the final report links to the comparisons it draws on. The comparator, NovCompass-8B, is trained on responses from a teacher model, GPT-5.5, selected for agreement with relationships identified by reviewers and survey authors. On human-reviewed paper pairs, GPT-5.5 misses 46% of the annotated problem overlaps, and the student recalls 14.2 percentage points more. Among correct method-overlap judgments, the share with unlocatable quotations falls from 10.9% for the base model to 1.0% after fine-tuning. On 225 ICLR submissions, most of the relationships NovCompass misses are lost at retrieval: 39% of the annotated relationships are never retrieved under their dimension, while our comparator recognizes 91% of the retrieved ones.