- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ClaimCrawl uses claim-level examiner citations to adapt a dense patent retriever and a language-model reranker. It also constructs a frozen corpus of over seven million patents and evaluates corpus recovery, ranking within shared candidate pools, and shortlist usefulness.
Critical strengths
The application is practically relevant, and separating retrieval from fixed-pool ranking is a sound evaluation choice. The same-encoder comparison demonstrates substantial gains from task adaptation, while reranking improves Recall@20 by 14.40 percentage points over crawler order. The blinded human study provides useful preliminary evidence beyond citation recovery.
Critical weaknesses
The methodological novelty is limited because the system primarily combines established contrastive retrieval, hard-negative mining, LoRA adaptation, and pointwise ranking. The evaluation leaves important deployment questions unresolved. Candidate retrieval lacks per-query temporal filtering, and application-level splits do not rule out overlap between related patent families. These choices limit conclusions about temporally eligible prior-art search and generalization. The 50-case human study is helpful, but lacks sufficient scale and paired analysis to resolve these concerns. Practical efficiency also remains unclear because the system evaluates 1,000 candidates per query without measured end-to-end latency or cost.
Questions for authors
Do the improvements persist under query-specific date filtering and patent-family-disjoint splits? Can the authors report family-overlap statistics and paired outcomes for the human evaluation? What are the measured latency and GPU cost per query, and how do they compare with smaller reranking budgets?
5 comments
Response to Reviewer Rj6a [Part 2/4]
Q2 Overlap statistics and paired human-evaluation results. Of 2,896 test queries, 197 (6.8%) share a patent family with training. Across the 6,202 test query–target links, 23.4% point to patents previously seen as training positives (response Table 3, revised Section 5.3 and Appendix E.5, manuscript Table 24).
Revised Section 5.4 now cites CLEF-IP 2013 for scale. That study separately evaluated 50 English patent topics using search-report judgments (Piroi et al., 2013). We compared all methods on the same 50 later applications, with three reviewers judging pooled candidates with method and rank hidden. ClaimCrawl and forced-output GPT-5.5 averaged 6.8 and 5.1 unanimously accepted patents per top 20 (paired sign test, Holm-adjusted p = 0.005). At least five patents were accepted in 29/50 and 22/50 cases, respectively (McNemar test, adjusted p = 0.118). These cases support a gain in accepted patents per shortlist. The difference in five-patent success rates remains uncertain.
ClaimCrawl has the most successful cases at every threshold from three to seven. Against each of the four other baselines, paired tests favor ClaimCrawl for both success at five and accepted counts per case (response Table 2 and revised Appendix E.6, manuscript Table 25).
Table 2(a). Human-evaluation success counts on 50 cases at thresholds of 3–7 unanimously accepted patents among the top 20.
| Method | ≥3 | ≥4 | ≥5 | ≥6 | ≥7 |
|---|---|---|---|---|---|
| ClaimCrawl | 40 | 34 | 29 | 24 | 19 |
| GPT-5.5 forced | 33 | 27 | 22 | 17 | 13 |
| DeepSeek V4-Pro forced | 28 | 22 | 17 | 13 | 9 |
| Dense-only | 26 | 20 | 15 | 11 | 7 |
| BGE-M3 zero-shot | 24 | 18 | 13 | 9 | 6 |
| BM25 | 22 | 16 | 11 | 7 | 4 |
Table 2(b). Mean accepted patents per top-20 shortlist and Holm-adjusted paired tests against ClaimCrawl on the same 50 cases. McNemar tests assess success at ≥5. Sign tests assess per-case accepted counts.
| Method | Mean accepted ↑ | McNemar p Holm at ≥5 | Sign-test p Holm |
|---|---|---|---|
| ClaimCrawl | 6.8 | — | — |
| GPT-5.5 forced | 5.1 | 0.118 | 0.005 |
| DeepSeek V4-Pro forced | 4.2 | 0.008 | <0.001 |
| Dense-only | 3.8 | 0.004 | <0.001 |
| BGE-M3 zero-shot | 3.4 | 0.002 | <0.001 |
| BM25 | 3.0 | <0.001 | <0.001 |
Table 3. Separate analyses of family overlap and previously seen positive targets.
| Test subset | Queries | Crawler R@20 ↑ | Selector R@20 ↑ | ΔR@20 |
|---|---|---|---|---|
| Full test set | 2,896 | 0.3652 | 0.5098 | +0.1446 |
| Exclude queries sharing a training family | 2,699 | 0.3641 | 0.5085 | +0.1444 |
| All target positives unseen during training | 2,104 | 0.3528 | 0.4927 | +0.1399 |
Response to Reviewer Rj6a [Part 3/4]
Q3 Measured latency and GPU cost per query. On 50 randomly sampled test queries, full 9B decoding of 1,000 candidates on one RTX 5090 takes 332 seconds per query (about 0.092 GPU-hours). Stopping after the score takes 281 seconds (0.078 GPU-hours) for 1,000 candidates and 141 seconds (0.039 GPU-hours) for 500. The separately timed retrieval and full 9B reranking stages sum to about 332.5 seconds per query (0.45 + 332). This is a two-stage estimate (response Table 4, revised Section 5.3 and Appendix E.4, manuscript Table 21).
In the submission’s separate depth evaluation, 500 and 1,000 candidates both gave R@20 of 0.5127, with R@100 of 0.7054 and 0.7332, respectively (submission Table 10(a)). Halving the pool nearly halved reranking time on the timing sample, while the larger pool raised R@100 from 0.7054 to 0.7332 in the depth evaluation.
Table 4. Single-GPU latency per query on 50 randomly selected test queries. Language-model timings use an RTX 5090, 4-bit NF4, and batch size 16. Submission quality figures are historical references. Relative time uses the 1,000-candidate Qwen3-Reranker-8B LoRA run (one forward pass) as 1.00×. The ClaimCrawl 4B setting uses matched hyperparameters. Crawler time includes GPU query encoding and CPU flat-vector search over 7,039,592 patents.
| Configuration (candidates) | s/query ↓ | Relative time ↓ | R@20 ↑ reference |
|---|---|---|---|
| Qwen3-Reranker-8B LoRA (1,000) | 261 | 1.00× | 0.4271 (submission) |
| ClaimCrawl 9B, full (1,000) | 332 | 1.27× | 0.5127 (submission) |
| ClaimCrawl 9B, stop after score (1,000) | 281 | 1.08× | Unchanged in decoding comparison |
| ClaimCrawl 9B, stop after score (500) | 141 | 0.54× | 0.5127 (submission Table 10a) |
| ClaimCrawl 4B, stop after score (1,000) | 134 | 0.51× | 0.5072 (4B comparison) |
| Crawler | 0.45 | — | — |
Response to Reviewer Rj6a [Part 1/4]
Thank you for recognizing the value of evaluating corpus retrieval and fixed-pool reranking separately. You also ask whether the ranking gains persist when candidates are filtered by query date or when queries sharing a patent family with the training set are excluded.
W1 Limited methodological novelty. The query claim determines both which patents the retriever should recover and which passages the reranker should read. We train both stages separately with claim-level examiner citations. Within a fixed text budget, the reranker selects passages by their match to the claim and the patent field in which they occur. The revised introduction and Section 2, manuscript Table 1(b), contrast this design with training a patent encoder or selecting from supplied candidates (response Table 5). In submission Table 2(a), citation supervision raises R@5000 by 0.0871 over the same zero-shot 4B encoder. Holding the candidate pool, training conditions and input budget fixed, the excerpt improves R@20 over prefix truncation by 0.0167 (response Table 6; revised manuscript Table 17, Appendix E.3). On citations directly attached to the query claim, Gold R@20 improves by 0.0209 over the prefix, with a 95% confidence interval of [0.0108, 0.0310] (revised Section 5.3 and Appendix E.3, manuscript Table 18). The matched excerpt comparison isolates the benefit of selecting what to read for each claim. More of its directly cited patents reach the top 20.
W2 Absence of query-specific temporal filtering. The main evaluation uses a shared corpus cutoff, and Q1 reports the reranking gain when each query’s effective filing date limits candidate eligibility.
W3 Application-level splitting does not exclude family overlap. PANORAMA groups claims by application, so different applications from one family can span partitions. Q1 gives results after excluding overlapping families, and Q2 quantifies the overlap.
W4 Human-study size and paired analysis. Blinded judgments on the same applications allow a paired comparison across methods, and Q2 gives the sample-size context, threshold sensitivity and both paired tests.
W5 Latency with 1,000 candidates and smaller budgets. Halving the candidate pool nearly halves reranking time under the same stop-after-score rule, and Q3 reports per-query costs and recall at both depths.
Q1 Gains under date filtering and family disjointness. Yes, under each restriction evaluated separately. We retained candidates filed on or before the query’s effective filing date. This removed 19 recorded targets, so each comparison evaluates crawler order and reranking against the same eligible targets within that setting. Reranking raises R@20 from 0.3779 to 0.5236 under query-date filtering (+0.1457) and from 0.3652 to 0.5098 under the shared cutoff (+0.1446, response Table 1; revised manuscript Table 23, Appendix E.5).
With the original training split, R@20 rises from 0.3641 to 0.5085 (+0.1444) on the 2,699 test queries sharing no family with training. A separate comparison on 2,104 queries whose targets were all unseen as training positives gives a gain of 0.1399 (response Table 3; revised manuscript Table 24, Appendix E.5). The joint effect of date and family restrictions and the effect of retraining on a new family-level split remain open (revised Sections 3.2 and 5.3 and Appendix E.5, manuscript Tables 23–24).
Table 1. Query-date-filtered comparison.