- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
NovCompass retrieves prior work per dimension (task/problem/method) and compares candidates with a fine-tuned 8B comparator (Qwen3-8B) trained on GPT-5.5 outputs filtered by reviewer/survey overlap relations; comparisons are kept only when both quotes are programmatically located. On NovGauge it recalls 14.2 points more annotated problem overlaps than its teacher (p = 5e-6) at 99.4% observed precision (32 problem negatives); end-to-end on 225 ICLR submissions it recovers 54.3% of reviewer-identified relations (ceiling 61.3%).
Critical strengths
- Principled supervision. Reviewer/survey relations filter teacher answers rather than serve as ground truth, addressing teacher conservatism (45.7% of positive problem candidates rejected).
- Judge-free evidence. The headline comparison uses paired bootstrap CIs and McNemar tests, avoiding the unstable LLM-judged Verified F1 (four Gemini judges accept only 31.3–50.3% of the same rationales).
- Strong evaluation practice. Unseen-paper split (+18.2), blind audit (κ = 0.82), wrong-paper control (3%), and evidence that retrieval dominates end-to-end loss.
Critical weaknesses
- Causal claim unidentified; shared provenance. No unfiltered-student control is reported, so the +14.2 gain is tied to filtering only correlationally. Training relations and NovGauge labels come from the same extraction pipeline (14.5% of benchmark papers appear in training), so annotation conventions may be learned instead of novelty.
- Thin, partly unfavorable precision. With 32 problem negatives, 99.4% vs. 98.5% is roughly one false positive; on method the student's FPR is higher (25.5% vs. 20.3%); audit confirmation is 66.4% vs. 82.6% with overlapping CIs, and the net utility of "more total, less often right" is not quantified.
- "Traceable" does not mean supported; the framing overreaches. Quote location checks string presence, not entailment; Mis is 27.8–45.9% (GPT-5.2) / 33.5–63.1% (Gemini) and judge sensitivity is high (31.3–50.3%; GPT-5.5 method VF1 falls from 63.7 to 43.1). The system detects three-dimension overlap, not novelty in the broader sense.
- Attribution and ceiling gaps. The +7.3 rewrite gain is confounded with route count and budget (61.3% vs. 54.0%); regeneration's net accuracy effect is unreported (9,697 regenerations; 362 vs. 355 flips); truncation (16k tokens/paper; 93,491 comparisons) is unanalyzed; and the 61.3% ceiling rests on incomplete annotations, with index coverage of the 33/71 missed AC-named works unknown.
- External validity and positioning. ICLR/ML only (225 submissions; 4 task instances); index, weights, and scripts are unreleased, so the dominant failure mode cannot be audited. Appendix I also shows GPT-5.5 without web captures 78% of AC positioning judgments vs. 40–49% for the full report.
Questions for authors
- Can you train the unfiltered-student control (same recipe and compute)?
- What is the net utility of the recall/precision trade-off, and is the lower confirmation rate concentrated in method?
- Of the 33/71 AC-named works missed by retrieval, how many are outside the index? What is the truncation impact, and what is regeneration's net effect on decision accuracy?
5 comments
Response to Reviewer Rfe8 (2/4): shared provenance, precision and net utility
W1. Shared provenance
We agree this concern is hard to rule out completely, and we offer three pieces of evidence.
- Unseen papers. On instances where neither paper appears in training or validation, which removes the overlap you point to, the problem gain over the teacher is +18.2 (95% CI 11.3 to 25.3), larger than on the full benchmark.
- Limits of the control. Both students see the same paper pairs from the same pipelines, and only the targets differ, so exposure to the pipeline alone does not produce the gain. The control cannot rule out that the filtered student learns labelling conventions that NovGauge shares. The evidence against this is the blind audit, which does not use these labels: about 60% of the assertions that only the student makes are confirmed by humans (Q2).
- Label quality. Two annotators, blind to label, teacher answer and source, labelled 120 pairs. They agreed on 30 of the 36 pairs both labelled (κ = 0.66). Most of these were contested pairs, where agreement was lower (18 of 23) than on training targets (12 of 13). Of the sampled pairs used as training targets, where label and teacher agree, humans confirmed 39 of 40 (97.5%). On the 80 label-positive pairs the teacher rejected, humans judged 40% to overlap (47.5% on problem). The teacher's rejections on training candidates therefore mix conservatism with label noise, and we will describe the 45.7% rejection rate that way. The evidence for conservatism is on NovGauge, whose positives were reviewed by humans and where GPT-5.5 misses 46% of the problem overlaps. The filter discards the contested pairs, which keeps the teacher's "different" answers on them out of training.
Q2 / W2. Precision and net utility
We agree with each point. With 32 problem negatives, 99.4% against 98.5% is about one false positive, so we will state that no loss of precision is observed and that 32 negatives are too few to compare. On method the student's false-positive rate is higher (25.5% against 20.3%, 9 against 4 discordant pairs, p = 0.27). The audit's confidence intervals overlap.
Is the lower confirmation rate concentrated in method? Yes. Breaking down the blind audit (Appendix H) by dimension, with the same stratum weighting:
| Confirmation rate of NovCompass-8B's assertions | Estimate (95% CI) | Audited cases |
|---|---|---|
| Task | 95.0 (83.4 to 100) | 15 |
| Problem | 64.6 (47.0 to 80.9) | 33 |
| Method | 49.3 (30.5 to 67.4) | 28 |
| Assertions GPT-5.5 also makes | 77.5 (62.1 to 91.7) | 24 |
| Assertions only NovCompass-8B makes | 59.5 (45.9 to 71.8) | 52 |
Where both comparators assert an overlap, the student's confirmation rate (77.5%) is close to the teacher's overall rate (82.6%) and to its rate on those same assertions (84.6%, 95% CI 70.2 to 96.5). The gap comes from the assertions that only the student makes, of which about 60% are still confirmed.
Net utility. Among the 1,315 unannotated comparisons in the audit pool, NovCompass-8B yields an estimated 59 more confirmed overlaps than GPT-5.5 (161 against 102), at the cost of 60 more unconfirmed assertions (81 against 21). The extra assertions are therefore about half confirmed, lower than the 60% above, because this count also subtracts the assertions that only GPT-5.5 makes. If a reader assigns a cost λ to checking an unconfirmed assertion, relative to the value of one confirmed overlap, the student is preferable whenever λ < 0.98 overall (0.86 on problem, 0.78 on method). Every assertion arrives with its two quotations, so checking one means reading two passages. Whether that is cheaper than missing a relevant prior work is a judgment the reader can make with these numbers. The per-dimension cells are small, so we report these figures as descriptive.
Response to Reviewer Rfe8 (3/4): "traceable", the rewrite gain, and regeneration
W3. Are the reports supported, not only traceable?
Quote location checks that both spans exist, not that they support the rationale, and the paper defines the term this way (Sections 3.1 and 3.5). Whether the reports are supported is an empirical question, and we have now measured it directly. The revised introduction will also state the scope precisely: the system analyzes each contribution against the earlier work it finds, showing with quoted evidence where they overlap, so that the reader can judge what is new.
To measure semantic support directly, the two PhD students from the original audit, working independently, checked the 77 comparative statements in 10 randomly drawn released reports (20 of them by both) against the full texts of both papers, not only against the quoted spans. The stated relation holds for 94.8% of the statements (95% CI 90.4 to 98.7, resampling reports) and holds partly for the rest. None was judged wrong. The facts stated about the two papers are correct for 96.1% and partly correct for the rest. The most common issue was an overstated overlap, in about four statements. On the 20 statements both labelled, they agreed on every relation judgment and on 19 of 20 factual judgments. This measures the final reports, which use only retained comparisons, and the report writer is instructed to make a narrower claim or omit a comparison when the quotations do not support it. The audit in Q2 measures the comparator's own assertions.
Our central comparison continues to rest on judge-free decision statistics, as you note.
Q3 / W4. Attribution and ceiling
Rewrite gain (+7.3). You are right that this is confounded with budget. We recomputed it from the saved top-250 rankings of every route, with the production admission rules. This reproduces the paper's settings within two instances: 60.6% for five routes and 53.3% for three. We then gave the three description routes the same candidate budget as the five routes (376 against 372 unique candidates per target):
| Routes | Candidates per target | All (302) | Problem (101) | Method (197) |
|---|---|---|---|---|
| 5 (with rewrites), paper budget | 372 | 60.6 | 64.4 | 58.9 |
| 3 (descriptions), paper budget | 218 | 53.3 | 60.4 | 49.7 |
| 3 (descriptions), matched budget | 376 | 58.6 | 66.3 | 54.8 |
At a matched budget, the rewrite routes add 2.0 points overall, not 7.3. Most of the original gain came from the larger budget. The remaining gain is on method (+4.1 points, 8 relations), consistent with their purpose of restating a method without the authors' coined names. On problem, spending the budget on the description route works as well or better. We will correct the attribution in the paper and report it by dimension.
Regeneration. Of the 9,697 regenerations, 91.1% of the comparable ones kept their judgment, and the flips were nearly balanced (362 from negative to positive, 355 from positive to negative). On the reviewer-annotated instances, 26 comparisons were regenerated: 18 recovered located quotations and none changed from positive to negative. On the annotated relations, regeneration therefore does not reduce decision accuracy, and it raises verified recall from 76.8% to 88.6%. For the unannotated flips, we sampled 100 in each direction and asked a judge from another model family (Gemini 3.1 Pro), blind to which answer came first, whether the papers overlap. It answered "no overlap" for 93 of the 100 negative-to-positive flips and 94 of the 100 positive-to-negative flips. It treats nearly all flipped pairs as non-overlapping, so this check alone says little about which answer is right. We therefore checked 20 of the flips by hand, 10 in each direction. We judged all 20 pairs non-overlapping, agreeing with the judge on 19. So the negative-to-positive flips were wrong and the positive-to-negative flips were right in equal numbers, and with balanced flips (362 against 355) regeneration has no net effect on decisions. The main effect of regeneration is to recover located quotations.
Response to Reviewer Rfe8 (1/4): the unfiltered control
Thank you for a careful and detailed review. Two of your points will make the paper's claims more precise: most of the rewrite gain came from the larger budget, and the unfiltered control shows where the filter helps and where it does not. For this response we trained the unfiltered student with two seeds, had two annotators check training labels and report statements against the source texts, recomputed the rewrite gain at a matched budget, and analyzed regeneration, truncation and index coverage. The response is in four comments.
Q1 / W1. Unfiltered-student control
We trained a student on the unfiltered teacher outputs with the same recipe as NovCompass-8B: the same GPT-5.5 inference pools, 13,264 training rows matched to the filtered set on source, dimension and candidate label, the same NovGauge pair exclusion and validation set, and the same training settings (Qwen3-8B, full fine-tuning, one epoch, global batch 64, 208 steps). Its training set keeps the teacher's answer even when it disagrees with the relation label. We trained it with two seeds.
Recall on NovGauge (decision only, no LLM judge, mean of 3 samples):
The teacher row follows the paper (Table 5). Differences below are paired, on the instances both models answered. McNemar counts use majority votes, so they do not map exactly onto the mean recall.
Two caveats apply to this comparison.
The filtered student is a single training run. The two unfiltered seeds differ by about 5 points on problem recall, and the gain from filtering (18.3 and 23.5 points) is several times larger. We will replace the limitation "the effect of filtering alone is not isolated" with these results and caveats.