acceptodds
Under review as a conference paper at ICLR 2027

ClaimCrawl: Citation-Supervised Retrieval and Reranking for Patent Prior-Art Search

Abstract

Patent prior-art search requires matching the technical content of a claim to earlier patents expressed in different terminology and lengthy descriptions. Beyond finding relevant documents, a useful search system must place them within the small set a searcher can inspect. Our analysis shows that high recall at deep retrieval cutoffs can coexist with limited target coverage among the earliest results. This gap motivates task adaptation at both retrieval and ranking stages, with separate evaluation of their contributions. We present ClaimCrawl, a framework that uses claim-level examiner citations to supervise a dense retriever and a pointwise language-model reranker. The retriever learns from cited positives and complementary lexical hard negatives and cross-device negatives. The reranker uses claim-conditioned excerpts to predict structured citation-alignment labels and scores, prioritizing targets within a fixed candidate pool. We also construct PAR4PC-Retrieve, linking PANORAMA queries and citation targets to a frozen corpus of over seven million U.S. patents. Supervised retrieval improves Recall@5000 by 8.71 percentage points over the same unadapted encoder. Within fixed candidate pools, reranking improves Recall@20 by 14.40 points over the retriever's original ordering and 8.56 points over task-adapted Qwen3-Reranker-8B. Model comparisons and ablations examine the contributions of task adaptation, and a blinded human study assesses the usefulness of the returned shortlists. Together, these results support citation-supervised retrieval and reranking as complementary components of patent prior-art search. Code: https://anonymous.4open.science/r/ClaimCrawl_code-BFBF/

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
patent prior-art search, information retrieval, dense retrieval, reranking, citation supervision

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_Rj6aOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ClaimCrawl uses claim-level examiner citations to adapt a dense patent retriever and a language-model reranker. It also constructs a frozen corpus of over seven million patents and evaluates corpus recovery, ranking within shared candidate pools, and shortlist usefulness.

Critical strengths

The application is practically relevant, and separating retrieval from fixed-pool ranking is a sound evaluation choice. The same-encoder comparison demonstrates substantial gains from task adaptation, while reranking improves Recall@20 by 14.40 percentage points over crawler order. The blinded human study provides useful preliminary evidence beyond citation recovery.

Critical weaknesses

The methodological novelty is limited because the system primarily combines established contrastive retrieval, hard-negative mining, LoRA adaptation, and pointwise ranking. The evaluation leaves important deployment questions unresolved. Candidate retrieval lacks per-query temporal filtering, and application-level splits do not rule out overlap between related patent families. These choices limit conclusions about temporally eligible prior-art search and generalization. The 50-case human study is helpful, but lacks sufficient scale and paired analysis to resolve these concerns. Practical efficiency also remains unclear because the system evaluates 1,000 candidates per query without measured end-to-end latency or cost.

Questions for authors

Do the improvements persist under query-specific date filtering and patent-family-disjoint splits? Can the authors report family-overlap statistics and paired outcomes for the human evaluation? What are the measured latency and GPU cost per query, and how do they compare with smaller reranking budgets?

5 comments
AuthorsOpenReview ↗

Response to Reviewer Rj6a [Part 1/4]

Thank you for recognizing the value of evaluating corpus retrieval and fixed-pool reranking separately. You also ask whether the ranking gains persist when candidates are filtered by query date or when queries sharing a patent family with the training set are excluded.

W1 Limited methodological novelty. The query claim determines both which patents the retriever should recover and which passages the reranker should read. We train both stages separately with claim-level examiner citations. Within a fixed text budget, the reranker selects passages by their match to the claim and the patent field in which they occur. The revised introduction and Section 2, manuscript Table 1(b), contrast this design with training a patent encoder or selecting from supplied candidates (response Table 5). In submission Table 2(a), citation supervision raises R@5000 by 0.0871 over the same zero-shot 4B encoder. Holding the candidate pool, training conditions and input budget fixed, the excerpt improves R@20 over prefix truncation by 0.0167 (response Table 6; revised manuscript Table 17, Appendix E.3). On citations directly attached to the query claim, Gold R@20 improves by 0.0209 over the prefix, with a 95% confidence interval of [0.0108, 0.0310] (revised Section 5.3 and Appendix E.3, manuscript Table 18). The matched excerpt comparison isolates the benefit of selecting what to read for each claim. More of its directly cited patents reach the top 20.

W2 Absence of query-specific temporal filtering. The main evaluation uses a shared corpus cutoff, and Q1 reports the reranking gain when each query’s effective filing date limits candidate eligibility.

W3 Application-level splitting does not exclude family overlap. PANORAMA groups claims by application, so different applications from one family can span partitions. Q1 gives results after excluding overlapping families, and Q2 quantifies the overlap.

W4 Human-study size and paired analysis. Blinded judgments on the same applications allow a paired comparison across methods, and Q2 gives the sample-size context, threshold sensitivity and both paired tests.

W5 Latency with 1,000 candidates and smaller budgets. Halving the candidate pool nearly halves reranking time under the same stop-after-score rule, and Q3 reports per-query costs and recall at both depths.

Q1 Gains under date filtering and family disjointness. Yes, under each restriction evaluated separately. We retained candidates filed on or before the query’s effective filing date. This removed 19 recorded targets, so each comparison evaluates crawler order and reranking against the same eligible targets within that setting. Reranking raises R@20 from 0.3779 to 0.5236 under query-date filtering (+0.1457) and from 0.3652 to 0.5098 under the shared cutoff (+0.1446, response Table 1; revised manuscript Table 23, Appendix E.5).

With the original training split, R@20 rises from 0.3641 to 0.5085 (+0.1444) on the 2,699 test queries sharing no family with training. A separate comparison on 2,104 queries whose targets were all unseen as training positives gives a gain of 0.1399 (response Table 3; revised manuscript Table 24, Appendix E.5). The joint effect of date and family restrictions and the effect of retraining on a new family-level split remain open (revised Sections 3.2 and 5.3 and Appendix E.5, manuscript Tables 23–24).

Table 1. Query-date-filtered comparison.

SettingCrawler R@20 ↑Crawler R@5000 ↑Selector R@20 ↑ΔR@20
Shared corpus cutoff0.36520.93410.5098+0.1446
Query-date filtering0.37790.94180.5236+0.1457
AuthorsOpenReview ↗

Response to Reviewer Rj6a [Part 2/4]

Q2 Overlap statistics and paired human-evaluation results. Of 2,896 test queries, 197 (6.8%) share a patent family with training. Across the 6,202 test query–target links, 23.4% point to patents previously seen as training positives (response Table 3, revised Section 5.3 and Appendix E.5, manuscript Table 24).

Revised Section 5.4 now cites CLEF-IP 2013 for scale. That study separately evaluated 50 English patent topics using search-report judgments (Piroi et al., 2013). We compared all methods on the same 50 later applications, with three reviewers judging pooled candidates with method and rank hidden. ClaimCrawl and forced-output GPT-5.5 averaged 6.8 and 5.1 unanimously accepted patents per top 20 (paired sign test, Holm-adjusted p = 0.005). At least five patents were accepted in 29/50 and 22/50 cases, respectively (McNemar test, adjusted p = 0.118). These cases support a gain in accepted patents per shortlist. The difference in five-patent success rates remains uncertain.

ClaimCrawl has the most successful cases at every threshold from three to seven. Against each of the four other baselines, paired tests favor ClaimCrawl for both success at five and accepted counts per case (response Table 2 and revised Appendix E.6, manuscript Table 25).

Table 2(a). Human-evaluation success counts on 50 cases at thresholds of 3–7 unanimously accepted patents among the top 20.

Method≥3≥4≥5≥6≥7
ClaimCrawl4034292419
GPT-5.5 forced3327221713
DeepSeek V4-Pro forced282217139
Dense-only262015117
BGE-M3 zero-shot24181396
BM2522161174

Table 2(b). Mean accepted patents per top-20 shortlist and Holm-adjusted paired tests against ClaimCrawl on the same 50 cases. McNemar tests assess success at ≥5. Sign tests assess per-case accepted counts.

MethodMean accepted ↑McNemar p Holm at ≥5Sign-test p Holm
ClaimCrawl6.8——
GPT-5.5 forced5.10.1180.005
DeepSeek V4-Pro forced4.20.008<0.001
Dense-only3.80.004<0.001
BGE-M3 zero-shot3.40.002<0.001
BM253.0<0.001<0.001

Table 3. Separate analyses of family overlap and previously seen positive targets.

Test subsetQueriesCrawler R@20 ↑Selector R@20 ↑ΔR@20
Full test set2,8960.36520.5098+0.1446
Exclude queries sharing a training family2,6990.36410.5085+0.1444
All target positives unseen during training2,1040.35280.4927+0.1399
AuthorsOpenReview ↗

Response to Reviewer Rj6a [Part 3/4]

Q3 Measured latency and GPU cost per query. On 50 randomly sampled test queries, full 9B decoding of 1,000 candidates on one RTX 5090 takes 332 seconds per query (about 0.092 GPU-hours). Stopping after the score takes 281 seconds (0.078 GPU-hours) for 1,000 candidates and 141 seconds (0.039 GPU-hours) for 500. The separately timed retrieval and full 9B reranking stages sum to about 332.5 seconds per query (0.45 + 332). This is a two-stage estimate (response Table 4, revised Section 5.3 and Appendix E.4, manuscript Table 21).

In the submission’s separate depth evaluation, 500 and 1,000 candidates both gave R@20 of 0.5127, with R@100 of 0.7054 and 0.7332, respectively (submission Table 10(a)). Halving the pool nearly halved reranking time on the timing sample, while the larger pool raised R@100 from 0.7054 to 0.7332 in the depth evaluation.

Table 4. Single-GPU latency per query on 50 randomly selected test queries. Language-model timings use an RTX 5090, 4-bit NF4, and batch size 16. Submission quality figures are historical references. Relative time uses the 1,000-candidate Qwen3-Reranker-8B LoRA run (one forward pass) as 1.00×. The ClaimCrawl 4B setting uses matched hyperparameters. Crawler time includes GPU query encoding and CPU flat-vector search over 7,039,592 patents.

Configuration (candidates)s/query ↓Relative time ↓R@20 ↑ reference
Qwen3-Reranker-8B LoRA (1,000)2611.00×0.4271 (submission)
ClaimCrawl 9B, full (1,000)3321.27×0.5127 (submission)
ClaimCrawl 9B, stop after score (1,000)2811.08×Unchanged in decoding comparison
ClaimCrawl 9B, stop after score (500)1410.54×0.5127 (submission Table 10a)
ClaimCrawl 4B, stop after score (1,000)1340.51×0.5072 (4B comparison)
Crawler0.45——
Reviewer_sSE5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies claim-level patent citation recovery using a two-stage retrieval and reranking framework. It introduces PAR4PC-Retrieve, which links existing PANORAMA citation targets to a corpus of over seven million U.S. patents. The framework adapts a Qwen3 embedding retriever using citation supervision and hard negatives, followed by a Qwen3.5 pointwise reranker that uses claim-conditioned excerpts to predict citation-alignment labels and scores. Experiments include same-encoder and fixed-pool comparisons, ablation studies, repeated training runs, and a small-scale human evaluation. The paper also examines claim-element coverage under different inspection budgets.

Critical strengths

(1) PAR4PC-Retrieve provides a useful extension of claim-level citation evaluation. By extending the original task from eight supplied candidates to a shared corpus of approximately seven million patent records, it enables the evaluation of first-stage retrieval and retrieval depth. This broadens the scope of the original task and constitutes a meaningful resource contribution. (2) The experimental results show substantial improvements in both retrieval and reranking. Adapting Qwen3-Embedding-4B improves Recall@5000 from 84.94% to 93.65%. With the same 1,000-candidate pool, the proposed selector achieves 51.27% Recall@20, compared with 36.87% for the original crawler ranking and 42.71% for the adapted Qwen3 reranker. These results provide evidence for the effectiveness of task adaptation and reranking. The ablation studies and repeated runs further strengthen the empirical evaluation, while the blinded human study provides additional evidence of shortlist usefulness.

Critical weaknesses

(1) The motivating phenomena illustrated in Figure 1 are not particularly novel, as they represent common challenges in retrieval systems. The paper needs to clarify what challenges are unique to patent retrieval and how the proposed methods address them. Moreover, the proposed supervised retrieval and reranking methods have limited technical novelty, as they largely follow existing approaches. (2) The paper presents claim-conditioned input construction as one of its contributions. However, the excerpt selection rule in Eq. (4) is not compared with a simpler input construction method under the same token budget. Although the overall framework achieves promising results, the contribution of the proposed input construction remains unclear. A controlled comparison with simpler alternatives would help establish its effectiveness and technical contribution. (3) The role of the generated score in reranking remains unclear. As acknowledged by the authors, the comparison between the structured selector and the yes/no model in Table 3(b) changes both the training targets and the ranking policy, making it difficult to isolate the contribution of score generation. Furthermore, the score targets are fixed at 0.95 and 0.05 according to the binary labels, without explicitly distinguishing relevance levels within each class. A controlled ablation is needed to determine whether the generated scores improve ranking beyond the predicted labels and original crawler order.

Questions for authors

(1) Could the authors compare the excerpt construction in Eq. (4) with simple prefix truncation under the same token budget? A controlled comparison using the same backbone, training data, and ranking rule, with consistent input construction during training and evaluation, would help clarify the contribution of the proposed method. (2) Could the authors compare label–score–crawler ordering with label–crawler ordering using the same structured-selector checkpoint? By keeping the candidate pool, predicted labels, parsing and fallback rules, and output length fixed, this comparison would help isolate the contribution of the generated scores. Reporting R@20, R@100, MRR, and within-label score distributions and tie rates would further clarify their effectiveness. (3) Could the authors clarify how All-Gold@100 of 0.5817 in Table 10 is computed? This value does not appear consistent with an integer success count over 2,896 test queries rounded to four decimal places. Please clarify the denominator and whether any additional averaging is involved.

5 comments
AuthorsOpenReview ↗

Response to Reviewer sSE5 [Part 1/4]

Thank you for suggesting two controls: comparing the excerpt with a prefix under the same budget, and holding the model outputs fixed to isolate the contribution of the generated score.

W1 Motivation and methodological novelty. We agree that missed candidates and low-ranked targets are familiar retrieval problems. Our contribution is to adapt corpus retrieval using examiner citations indexed to individual claims within an application, and to use the query claim to choose which passages the reranker reads from each long patent. This lets us search millions of patents for citation targets and compare retrieved candidates against the particular claim within a fixed input budget. Response Table 6 links each design choice to its control (revised Appendix E.1, manuscript Table 12).

(1) Distinction from the closest work. PANORAMA–PAR4PC selects from eight supplied candidates, and FiNE-Patents locates claim-feature evidence within a supplied patent. ClaimCrawl retrieves candidates from a shared patent corpus and evaluates reranking separately on identical pools. Response Table 1 and revised Section 2, manuscript Table 1(b), compare the supervision, trained stages and evaluation scope of these studies.

(2) A concrete design for long patents. A passage matching the query claim can occur late in a patent description and fall outside a fixed prefix. We use distinct claim-term matches and patent-field priority to decide which passages fit the input budget. Section 4.3 illustrates the rule with the access condition from Figure 1(a).

(3) What each design contributes. Citation supervision raises corpus R@5000 by 0.0871 over the same zero-shot 4B encoder (submission Table 2(a)). Once candidates are given, claim-conditioned passage selection brings more citation targets into the top 20; Q1 isolates that effect under a matched input budget.

Table 1. Comparison with related methods. P: patent domain; C: training or evaluation data derived from claim-level citation annotations; R: training a first-stage retrieval model (merely invoking an existing retriever does not count); K: training a candidate-assessment model to score, rank, or select supplied candidates; S: retrieval evaluation with multiple queries over a shared patent corpus. Entries follow the settings reported in each original paper, excluding subsequent third-party evaluations.

MethodPCRKS
PaECTER (original in-house evaluation)✓✗✓✗✗
patembed / PatenTEB✓✗✓✗✓
QaECTER / Sophia-Bench✓✗✓✗✓
Stamatis et al. (2024), patent reranking✓✗✗✓✓
FiNE-Patents✓✓✗✗✗
PANORAMA–PAR4PC (SFT selection model)✓✓✗✓✗
Qwen3 Embedding / Reranker (two models)✗✗✓✓✗
RankGPT (including permutation distillation) / FIRST / Rank1 / ReasonRank✗✗✗✓✗
PRP✗✗✗✗✗
ClaimCrawl✓✓✓✓✓

W2 Excerpts versus prefix truncation under the same budget. Selecting passages for the query claim recovered more cited patents in the top 20 than reading the prefix under the same budget, as shown by the three input constructions compared in Q1.

W3 Isolating the ranking contribution of generated scores. Reordering the same outputs from one checkpoint shows that the generated score helps distinguish candidates with the same predicted label, as quantified by the ranking comparison and score distributions in Q2.

AuthorsOpenReview ↗

Response to Reviewer sSE5 [Part 2/4]

Q1 Comparing Eq. 4 with prefix truncation under the same budget. The three variants share the same Qwen3.5-9B initialization, 120,000 training pairs, QLoRA hyperparameters, candidate pools and ranking rule. Each uses a 1,200-word preprocessing budget and a 3,072-token input limit, with its own text construction applied at both training and test time. The three input constructions are prefix truncation, selection by claim-term overlap alone and the complete Eq. (4) excerpt (response Table 2, revised Section 5.3 and Appendix E.3, manuscript Table 17).

Table 2(a). Candidate-text construction under the same input budget on 2,896 queries. Top-20 recall, MRR, prompt length and 95% difference intervals.

Candidate textR@20 ↑MRR ↑Mean prompt tokens
Crawler order reference0.36520.2089—
Prefix truncation0.49310.33021,909
Overlap only, without field weight0.50170.33641,915
ClaimCrawl excerpts0.50980.34311,918
Excerpt minus prefix+0.0167+0.0129—
95% difference CI[0.0079, 0.0255][0.0066, 0.0192]—

Table 2(b). The same candidate-text comparison at other recall cutoffs, with 95% difference intervals.

Candidate textR@10 ↑R@50 ↑R@100 ↑
Crawler order reference0.29870.50710.5689
Prefix truncation0.41220.61980.7251
Overlap only, without field weight0.42150.62620.7288
ClaimCrawl excerpts0.43090.63270.7322
Excerpt minus prefix+0.0187+0.0129+0.0071
95% difference CI[0.0091, 0.0283][0.0052, 0.0206][0.0013, 0.0129]

R@20 is 0.4931 for the prefix, 0.5017 for term overlap alone and 0.5098 for the complete excerpt. The excerpt and prefix have similar mean prompt lengths (1,918 and 1,909 tokens). The excerpt gains 0.0167 in R@20 (95% CI 0.0079 to 0.0255) and 0.0129 in MRR (95% CI 0.0066 to 0.0192). For citations directly tied to the query claim, its R@20 gain is 0.0209 (response Table 3 and Appendix E.3, manuscript Table 18). Selecting passages for the claim brings more cited patents into the top 20 within the same input budget.

Table 3(a). Gold-only evaluation of the same outputs on 2,896 queries.

MethodGold R@10 ↑Gold R@20 ↑Gold R@100 ↑Gold MRR ↑
Crawler order0.31220.38100.58720.1958
Prefix selector0.43350.51620.74740.3083
ClaimCrawl excerpt selector0.45520.53710.75510.3221
Excerpt minus prefix+0.0217+0.0209+0.0077+0.0138

Gold R@20 improves by 0.0209, with a 95% confidence interval of [0.0108, 0.0310].

Table 3(b). Silver R@20 on 1,319 queries with silver targets. Gold-before-silver is the fraction of pooled gold–silver pairs with gold ranked first, among pairs with both members in the top 100.

MethodSilver R@20 ↑Gold before silver
Crawler order0.32850.551
Prefix selector0.43950.616
ClaimCrawl excerpt selector0.44640.639
Excerpt minus prefix+0.0069+0.023

Silver R@20 gains 0.0069; the gold-before-silver fraction rises from 0.616 to 0.639.

AuthorsOpenReview ↗

Response to Reviewer sSE5 [Part 3/4]

Q2 Same-checkpoint ranking comparison and score statistics. Using the same checkpoint and decoded outputs, we held the candidate pool, predicted labels, parser, fallback rules and output length fixed and changed only the sorting keys. The 95% difference intervals for R@10 and MRR exclude zero. R@20 moves from 0.5051 to 0.5098 and R@100 from 0.7318 to 0.7322. The intervals for R@20, R@50 and R@100 include zero (response Table 4, revised Section 5.3 and Appendix E.3, manuscript Table 20).

Predicted positives take 11 score values, with 86.4% at 0.95. Predicted negatives take seven, with 97.8% at 0.05. We count a candidate as tied if another candidate in the same query and predicted label has the same score to six decimal places. Pooled across queries, the tie rates are 0.912 for positives and 0.996 for negatives (response Table 5, revised Section 4.4 and Appendix E.3, manuscript Table 19). Candidates tied on both label and score retain crawler order.

Table 4(a). Ranking-key ablation using the same checkpoint and decoded outputs. R@10, MRR and 95% difference intervals.

Ordering, same checkpoint and outputsR@10 ↑MRR ↑
Crawler order reference0.29870.2089
Score → crawler0.43010.3425
Label → crawler0.42180.3329
Label → score → crawler0.43090.3431
Complete minus no-score+0.0091+0.0102
95% difference CI[0.0037, 0.0146][0.0058, 0.0147]

Table 4(b). The same ranking-key ablation at deeper recall cutoffs, with 95% difference intervals.

Ordering, same checkpoint and outputsR@20 ↑R@50 ↑R@100 ↑
Crawler order reference0.36520.50710.5689
Score → crawler0.50940.63250.7322
Label → crawler0.50510.63090.7318
Label → score → crawler0.50980.63270.7322
Complete minus no-score+0.0047+0.0018+0.0004
95% difference CI[-0.0001, 0.0095][-0.0010, 0.0046][-0.0010, 0.0018]

Table 5. Test-time decoded-score statistics. Within-label tie rate is the fraction of candidates sharing a score with at least one other candidate in the same query and predicted-label group. Scores are rounded to six decimals for tie detection; candidate counts are pooled across queries.

StatisticPositive labelNegative label
Share exactly at training target 0.95 / 0.0586.4%97.8%
Number of distinct scores117
Reported distribution positions5th percentile 0.80; max 0.97min 0.02; max 0.30
Within-label candidate tie rate0.9120.996

On the same decoded outputs, using the score to order candidates within each predicted label raises R@10 by 0.0091 and MRR by 0.0102 over retaining crawler order. Its clearest benefit is to move cited patents nearer the top.

Q3 Calculation and correction of All-Gold@100. Thank you for catching this error. The denominator for All-Gold@100 is 2,896 test queries. Each query counts once: it succeeds if its top 100 contains every recorded target (gold and silver), with no further averaging. In our initial calculations for the draft, we recorded All-Gold results to three decimal places. When we standardized reporting to four decimal places, we used the wrong denominator when checking All-Gold@100 for the 9B selector. We apologize for this error. We have recalculated this value using the correct denominator; the corrected result is 0.5815.

Reviewer_p679OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a pipeline for the analysis of prior art in patents. The novel content resides in the combination of a dense retrieval with a point wise reranker. The main idea is to separate the collection from the full corpus from the reranker of common pool.

Critical strengths
  • the distinction between retrieval and reranking is methodologically sound and interesting
  • the paper is well detailed about corpus, ablation, configuration and reproducibility
  • the experimental results show relevant improvements w.r.t. a traditional supervised approach
Critical weaknesses
  • it is not fully clear to this reviewer the point about the temporal filter. It seems to me that the main core of the paper claims that the is not temporal filter while appendix seems to say something different
  • please clarify whether there is leakage between train and test since the split are separated by patent family
  • the algorithmic novel content is incremental. Please address this point
  • clarify the end-to-end latency/cost and compare with stronger baselines
Questions for authors

See previous box

4 comments
AuthorsOpenReview ↗

Response to Reviewer p679 [Part 1/3]

Thank you for pointing out the ambiguity between the date filters described in the main text and the appendix. The two descriptions refer to different evaluation steps.

W1 Temporal filtering across evaluation stages. The main retrieval and reranking evaluation uses a corpus frozen on December 31, 2024, with no per-query date filter. The inspection analysis applies an additional condition when crediting support for a claim element: the candidate patent date must be on or before the query application date (Appendix B.7). That condition applies after ranking.

Response Table 1 addresses the retrieval question directly. It restricts candidates to patents filed on or before the query’s effective filing date, then reranks each resulting pool (revised Sections 3.2 and 5.3 and Appendix E.5, manuscript Table 23).

Table 1. Query-date-filtered comparison.

SettingCrawler R@20 ↑Crawler R@5000 ↑Selector R@20 ↑ΔR@20
Shared corpus cutoff0.36520.93410.5098+0.1446
Query-date filtering0.37790.94180.5236+0.1457

Query-date filtering removes 19 recorded targets (about 0.3%) and changes the candidate pools, so we compare reranking with crawler order within each setting. With filtering, reranking raises R@20 from 0.3779 to 0.5236, a gain of 0.1457. The corresponding gain under the shared cutoff is 0.1446. The reranking advantage therefore persists when retrieval excludes patents filed after the query’s effective filing date.

W2 Train–test overlap and patent-family grouping. We use the PANORAMA–PAR4PC split released by Lim et al. (2025). All claims from one application stay together, but different applications in one family can span partitions. We select configurations on Validation-500, which is disjoint from test. Response Table 2 examines two kinds of overlap separately. One is a test query sharing a family with a training query. The other is a test citation target appearing as a training positive (revised Section 5.3 and Appendix E.5, manuscript Table 24).

Table 2. Separate analyses of family overlap and previously seen positive targets.

Test subsetQueriesCrawler R@20 ↑Selector R@20 ↑ΔR@20
Full test set2,8960.36520.5098+0.1446
Exclude queries sharing a training family2,6990.36410.5085+0.1444
All target positives unseen during training2,1040.35280.4927+0.1399

Of the 2,896 test queries, 197 (6.8%) share a family with a training query. Excluding them leaves 2,699 queries and a reranking gain of 0.1444 in R@20. Looking separately at citation targets, 23.4% of the 6,202 test query–target links point to patents used as training positives. Reranking gains 0.1399 on the 2,104 queries whose targets were all unseen as training positives, compared with 0.1446 on the full test set.

The gain persists in each separate analysis, extending both to queries from families absent among training queries and to queries whose citation targets were all unseen as training positives.

AuthorsOpenReview ↗

Response to Reviewer p679 [Part 2/3]

W3 Incremental algorithmic novelty. In ClaimCrawl the query claim guides both which patents the retriever learns to recover from the corpus using examiner citations, and which passages the reranker reads under a fixed budget using claim-term matches and patent-field priorities. PANORAMA–PAR4PC selects among eight supplied candidates, while FiNE-Patents locates passages in a supplied patent. Response Table 3 and revised Section 2, manuscript Table 1(b), compare the stages each study trains and evaluates. In submission Table 2(a), citation supervision raises R@5000 by 0.0871 for the same 4B encoder. With the pool and input budget fixed, claim-conditioned excerpts gain 0.0167 in R@20 over a prefix (95% CI 0.0079 to 0.0255, response Table 4; revised manuscript Table 17, Appendix E.3). On citations directly tied to the query claim, Gold R@20 improves by 0.0209 over the prefix, with a 95% confidence interval of [0.0108, 0.0310] (revised Section 5.3 and Appendix E.3, manuscript Table 18). The first comparison shows that adaptation brings more citation targets into the pool. The second shows that allocating the reading budget by claim brings more of the retrieved targets into the top 20.

Table 3. Comparison with related methods. P: patent domain; C: training or evaluation data derived from claim-level citation annotations; R: training a first-stage retrieval model (merely invoking an existing retriever does not count); K: training a candidate-assessment model to score, rank, or select supplied candidates; S: retrieval evaluation with multiple queries over a shared patent corpus. Entries follow the settings reported in each original paper, excluding subsequent third-party evaluations.

MethodPCRKS
PaECTER (original in-house evaluation)✓✗✓✗✗
patembed / PatenTEB✓✗✓✗✓
QaECTER / Sophia-Bench✓✗✓✗✓
Stamatis et al. (2024), patent reranking✓✗✗✓✓
FiNE-Patents✓✓✗✗✗
PANORAMA–PAR4PC (SFT selection model)✓✓✗✓✗
Qwen3 Embedding / Reranker (two models)✗✗✓✓✗
RankGPT (including permutation distillation) / FIRST / Rank1 / ReasonRank✗✗✗✓✗
PRP✗✗✗✗✗
ClaimCrawl✓✓✓✓✓

Table 4(a). Candidate-text construction under the same input budget on 2,896 queries. Top-20 recall, MRR, prompt length and 95% difference intervals.

Candidate textR@20 ↑MRR ↑Mean prompt tokens
Crawler order reference0.36520.2089—
Prefix truncation0.49310.33021,909
Overlap only, without field weight0.50170.33641,915
ClaimCrawl excerpts0.50980.34311,918
Excerpt minus prefix+0.0167+0.0129—
95% difference CI[0.0079, 0.0255][0.0066, 0.0192]—

Table 4(b). The same candidate-text comparison at other recall cutoffs, with 95% difference intervals.

Candidate textR@10 ↑R@50 ↑R@100 ↑
Crawler order reference0.29870.50710.5689
Prefix truncation0.41220.61980.7251
Overlap only, without field weight0.42150.62620.7288
ClaimCrawl excerpts0.43090.63270.7322
Excerpt minus prefix+0.0187+0.0129+0.0071
95% difference CI[0.0091, 0.0283][0.0052, 0.0206][0.0013, 0.0129]
AuthorsOpenReview ↗

Response to Reviewer p679 [Part 3/3]

W4 Computational cost and stronger baselines. Response Table 6 (revised manuscript Table 21, Appendix E.4) reports stage timings. Response Table 5 (revised manuscript Table 16, Appendix E.2) compares reasoning reranker Rank1-7B, task-adapted Qwen3-Reranker-8B, and ClaimCrawl on identical candidate pools.

Table 5(a). Fixed-pool reranking of the same 1,000 candidates into 100 outputs. R@20 and seconds per query on one RTX 5090.

MethodR@20 ↑Seconds/query ↓
Crawler order0.3652—
Rank1-7B reasoning reranker, zero-shot0.40691,021
Qwen3-Reranker-8B LoRA0.4236261
ClaimCrawl selector 9B0.5098332

Table 5(b). Other ranking metrics for the same fixed-pool comparison.

MethodR@10 ↑R@100 ↑MRR top 100 ↑
Crawler order0.29870.56890.2089
Rank1-7B reasoning reranker, zero-shot0.33810.59170.2783
Qwen3-Reranker-8B LoRA0.35940.61870.2961
ClaimCrawl selector 9B0.43090.73220.3431

On identical pools, R@20 is 0.5098 for ClaimCrawl, 0.4236 for task-adapted Qwen3-Reranker-8B and 0.4069 for zero-shot Rank1-7B. Reranking 1,000 candidates on one GPU takes 332, 261 and 1,021 seconds per query, respectively (revised Section 5.3 and Appendix E.2, manuscript Table 16). The separately timed retrieval and full 9B reranking stages sum to about 332.5 seconds per query (0.45 + 332). This is a two-stage estimate. Relative to adapted Qwen3-Reranker, ClaimCrawl gains 0.0862 in R@20 at an additional 71 seconds per query. Relative to Rank1, which generates a reasoning chain per candidate, it gains 0.1029 in about one third of the time.

Table 6. Single-GPU latency per query on 50 randomly selected test queries. Language-model timings use an RTX 5090, 4-bit NF4, and batch size 16. Submission quality figures are historical references. Relative time uses the 1,000-candidate Qwen3-Reranker-8B LoRA run (one forward pass) as 1.00×. The ClaimCrawl 4B setting uses matched hyperparameters. Crawler time includes GPU query encoding and CPU flat-vector search over 7,039,592 patents.

Configuration (candidates)s/query ↓Relative time ↓R@20 ↑ reference
Qwen3-Reranker-8B LoRA (1,000)2611.00×0.4271 (submission)
ClaimCrawl 9B, full (1,000)3321.27×0.5127 (submission)
ClaimCrawl 9B, stop after score (1,000)2811.08×Unchanged in decoding comparison
ClaimCrawl 9B, stop after score (500)1410.54×0.5127 (submission Table 10a)
ClaimCrawl 4B, stop after score (1,000)1340.51×0.5072 (4B comparison)
Crawler0.45——

Dear Reviewer p679,

Thank you for highlighting the issues of temporal filtering, data overlap, novelty, and computational cost. We have worked carefully to address them by clarifying the date conditions at different evaluation stages and providing separate analyses of query-date filtering and train–test overlap. We also report passage-selection controls to explain the methodological contribution, alongside stronger reranking baselines and timing measurements.

We hope these additions help resolve your concerns. We would appreciate your feedback on our responses and would be happy to answer any remaining questions or clarify any points that are still unclear.

Best regards,
Authors of Submission #55484

Reviewer_jN16OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors propose ClaimCrawl, a two-stage patent prior-art search framework that adapts both a dense retriever and a pointwise LLM reranker using claim-level examiner citations. The crawler is trained via contrastive learning with BM25 hard negatives, and the selector is fine-tuned with QLoRA on claim-conditioned excerpts to output structured alignment labels and scores for reranking. They evaluate their method by performing corpus-recovery, fixed-pool reranking comparisons, and a blinded human study.

Critical strengths
  1. The authors construct PAR4PC-Retrieve by linking PANORAMA claim-level citation targets to a 7M-patent USPTO corpus, elevating prior-art evaluation from tiny candidate pools to realistic corpus scale.
  2. Forcing the dense retriever to learn semantic representations by combining query-specific BM25 hard negatives with cross-device in-batch negatives is novel.
  3. The proposed pipeline consistently outperforms both general-purpose and patent-specific retrieval and reranking baselines across key recall and MRR metrics
Critical weaknesses
  1. The selector is trained with token-level cross-entropy on literal strings where positives always have a score of 0.95, and negatives have 0.05, making it unclear how it would behave on continuous scores.
  2. The human evaluation relies on an arbitrary binary threshold (>= 5 hits) without statistical significance testing.
  3. No wall-clock latency or throughput comparisons for running autoregressive generation with a 9B LLM over 1000 candidates per query.
  4. All the models used for the selector are 8B; it is not clear how the performance scales with the model size and how it depends on the crawler used.
  5. General-purpose encoders (BGE-M3) and patent-specific encoders (PaECTER, patembed-large) were evaluated strictly zero-shot without fine-tuning on PAR4PC-Retrieve
Questions for authors
  1. Since the selector is trained exclusively on scores 0.95 and 0.05, what is the distribution of decoded score values at test time?
  2. What is the wall-clock latency of running the 9B selector across 1000 candidates compared to Qwen3.5-4B and Qwen3-Reranker-8B?
  3. What are the wall-clock times required to train the 4B crawler and 9B selector? What total time is required to encode and index the patent corpus?
  4. What is the peak per-GPU VRAM consumption during crawler training, selector training, and inference?
7 comments
AuthorsOpenReview ↗

Response to Reviewer jN16 [Part 1/4]

Thank you for asking which scores the model actually emits at test time and how much they contribute to ranking.

W1 Fixed score targets and the role of generated scores. Training assigns scores of 0.95 and 0.05 to positive and negative labels. At test time, scores concentrate around these targets, while variation within a label improves R@10 from 0.4218 to 0.4309 on the same decoded outputs. Q1 gives the score distribution and the controlled comparison.

W2 Human-evaluation threshold and statistical tests. We defined success for this study as at least five accepted patents. At least a quarter of the top 20 must be unanimously judged relevant or technically helpful by all three reviewers. In revised Section 5.4, we added two analyses of the original 50 cases. We varied the threshold from three to seven and ran paired tests of success at five and of accepted counts per case. Response Table 1 below and manuscript Table 25 in Appendix E.6 give the full results.

Table 1(a). Human-evaluation success counts on 50 cases at thresholds of 3–7 unanimously accepted patents among the top 20.

Method≥3≥4≥5≥6≥7
ClaimCrawl4034292419
GPT-5.5 forced3327221713
DeepSeek V4-Pro forced282217139
Dense-only262015117
BGE-M3 zero-shot24181396
BM2522161174

Table 1(b). Mean accepted patents per top-20 shortlist and Holm-adjusted paired tests against ClaimCrawl on the same 50 cases. McNemar tests assess success at ≥5. Sign tests assess per-case accepted counts.

MethodMean accepted ↑McNemar p Holm at ≥5Sign-test p Holm
ClaimCrawl6.8——
GPT-5.5 forced5.10.1180.005
DeepSeek V4-Pro forced4.20.008<0.001
Dense-only3.80.004<0.001
BGE-M3 zero-shot3.40.002<0.001
BM253.0<0.001<0.001

ClaimCrawl has the most successful cases at each threshold. At five accepted patents, ClaimCrawl succeeds on 29/50 cases and GPT-5.5 on 22/50. The paired difference in binary success is not significant (Holm-adjusted McNemar p = 0.118). On those same cases, ClaimCrawl averages 6.8 accepted patents per shortlist versus 5.1 for GPT-5.5 (Holm-adjusted paired sign-test p = 0.005). Against DeepSeek V4-Pro, Dense-only, BGE-M3 and BM25, the Holm-adjusted p-values are at most 0.008 for binary success and below 0.001 for accepted counts. Before adjudication, the three reviewers agreed unanimously on 82.6% of items; mean pairwise Cohen’s κ was 0.71.

On these 50 cases, the advantage over forced-output GPT-5.5 is an average of 1.7 additional unanimously accepted patents in each top-20 shortlist.

W3 Latency and throughput. Full decoding with the 9B selector is the slowest of the three configurations at 1,000 candidates per query. Q2 gives the latency and throughput on the same hardware and the time saved by stopping after the score.

AuthorsOpenReview ↗

Response to Reviewer jN16 [Part 2/4]

W4 Model scale and dependence on the crawler. Manuscript Table 2(b) includes a 4B selector as well as the 8B and 9B models. We distinguish two comparisons. To examine model size, response Table 2 (revised manuscript Table 15, Appendix E.2) holds the candidate pool and training hyperparameters fixed across Qwen3.5-4B and 9B; the original 4B row in manuscript Table 2(b) uses its own training configuration. To examine dependence on the crawler, response Table 3 (revised manuscript Table 14, Appendix E.2) holds the 9B selector fixed and reranks each retriever’s own top 1,000 candidates (revised Sections 5.2–5.3).

Table 2. Scale comparison with matched hyperparameters using one candidate pool. Times are for full decoding.

MetricQwen3.5-4BClaimCrawl Qwen3.5-9B
R@10 ↑0.41960.4309
R@20 ↑0.50720.5098
R@100 ↑0.72890.7322
MRR ↑0.32610.3431
Seconds/query ↓158332

Table 3. One selector applied to different first-stage retrievers, each reranking its own top 1,000 candidates into 100 outputs. The final row is output R@100 divided by pool R@1000. Parentheses give output R@100. ΔR@20 is selector recall minus input-order recall.

MetricBM25Qwen3-Embedding-4B zero-shotClaimCrawl crawler
Pool R@1000 ↑0.62670.72750.8102
Original R@20 ↑0.24770.31150.3652
Selector R@20 ↑0.40120.46140.5098
ΔR@20+0.1535+0.1499+0.1446
Output R@100 / pool R@100089.9% (0.5631)90.3% (0.6571)90.4% (0.7322)

With matched settings, moving from 4B to 9B gains 0.0113 at R@10 and 0.0170 in MRR, but only 0.0026 at R@20. Full decoding with 4B takes 47.6% of the 9B time. The larger model helps most at the earliest ranks; if the top 20 are the focus, 4B achieves nearly the same recall in less than half the time in this comparison. Holding the 9B selector fixed, reranking improves R@20 by 0.1446–0.1535 across the BM25, zero-shot Qwen3 and ClaimCrawl pools. The ranking gain extends to all three pools, but the targets available to rerank still depend on retrieval: the ClaimCrawl pool has the highest R@1000 (0.8102) and the highest final R@20 (0.5098).

W5 Adapting other encoders with the same supervision. We adapted BGE-M3, PaECTER and patembed-large using the same training data, query-specific BM25 hard negatives and cross-device negatives as the ClaimCrawl crawler. Revised Section 5.2 reports the matched-supervision comparison. Response Table 4 and Appendix E.2, manuscript Table 13, give the full results.

Table 4. Corpus retrieval on 2,896 test queries, comparing unadapted and task-adapted encoders. MRR uses up to 5,000 retrieved results.

EncoderR@1000 ↑R@5000 ↑MRR top 5000 ↑
BGE-M3 zero-shot, submission0.58110.72870.1514
BGE-M3 matched-data adaptation0.70310.84560.1738
PaECTER zero-shot, submission0.55760.72900.1320
PaECTER matched-data adaptation0.70120.82130.1689
patembed-large zero-shot, submission0.64820.80940.1522
patembed-large matched-data adaptation0.73950.87020.1810
ClaimCrawl crawler0.81020.93410.2117

Relative to their own zero-shot results, adaptation raises R@5000 by 0.1169 for BGE-M3, 0.0923 for PaECTER and 0.0608 for patembed-large. Adapted patembed-large is the strongest of these three external encoders at 0.8702, versus 0.9341 for the ClaimCrawl crawler in the same comparison. All three encoders benefit from the supervision, and ClaimCrawl retains an R@5000 advantage of 0.0639 over the strongest of them after matched adaptation.

Presentation and clarification. Section 3.1 defines the recorded target set and evaluation cutoffs. The revised method follows the access condition in the claim from Figure 1(a) as an example (Section 4.3). We distinguish corpus retrieval, fixed-pool ranking and post-ranking inspection in Section 4.1.

AuthorsOpenReview ↗

Response to Reviewer jN16 [Part 3/4]

Q1 Test-time score distribution. Among valid outputs, candidates labeled positive receive 11 distinct scores. Of these candidates, 86.4% receive 0.95, the fifth percentile is 0.80, and the maximum is 0.97. Candidates labeled negative receive seven scores from 0.02 to 0.30, with 97.8% at 0.05 (response Table 5; revised manuscript Table 19, Appendix E.3). In the analysis added to the revision, we tested whether within-label score variation matters by holding the checkpoint, candidates and decoded outputs fixed. Within each predicted label we compared crawler order with score order, breaking score ties by crawler rank (response Table 6; revised manuscript Table 20, Appendix E.3). Using the score raises R@10 by 0.0091 and MRR by 0.0102. Both 95% intervals for the differences exclude zero, whereas those for R@20, R@50 and R@100 include zero. Under binary supervision, the model uses a small set of uncalibrated scores to refine the earliest ranks. Revised Sections 4.4 and 5.3 report these analyses. With the checkpoint and decoded outputs fixed, this comparison supports using generated scores to place cited patents nearer the top of the ranked list.

Table 5. Test-time decoded-score statistics. Within-label tie rate is the fraction of candidates sharing a score with at least one other candidate in the same query and predicted-label group. Scores are rounded to six decimals for tie detection; candidate counts are pooled across queries.

StatisticPositive labelNegative label
Share exactly at training target 0.95 / 0.0586.4%97.8%
Number of distinct scores117
Reported distribution positions5th percentile 0.80; max 0.97min 0.02; max 0.30
Within-label candidate tie rate0.9120.996

Table 6(a). Ranking-key ablation using the same checkpoint and decoded outputs. R@10, MRR and 95% difference intervals.

Ordering, same checkpoint and outputsR@10 ↑MRR ↑
Crawler order reference0.29870.2089
Score → crawler0.43010.3425
Label → crawler0.42180.3329
Label → score → crawler0.43090.3431
Complete minus no-score+0.0091+0.0102
95% difference CI[0.0037, 0.0146][0.0058, 0.0147]

Table 6(b). The same ranking-key ablation at deeper recall cutoffs, with 95% difference intervals.

Ordering, same checkpoint and outputsR@20 ↑R@50 ↑R@100 ↑
Crawler order reference0.36520.50710.5689
Score → crawler0.50940.63250.7322
Label → crawler0.50510.63090.7318
Label → score → crawler0.50980.63270.7322
Complete minus no-score+0.0047+0.0018+0.0004
95% difference CI[-0.0001, 0.0095][-0.0010, 0.0046][-0.0010, 0.0018]

Comments on the paper

1 comment
AuthorsOpenReview ↗

Follow-up on our responses and revised manuscript

Dear Reviewers,

Thank you again for your reviews and suggestions. We have made every effort to address your questions in our individual responses and revised manuscript.

We provide controlled comparisons of passage selection and generated scores, adapted-baseline comparisons, separate analyses of date filtering and training overlap, paired analyses of the original human-evaluation cases, and timing and memory measurements. These additions address the concerns about our method, evaluation, and computational cost. The corresponding manuscript locations are given in each response.

We would be grateful if you could revisit our responses and let us know whether any questions or concerns remain. Your feedback would help us understand which points still need clarification, and we will do our best to answer them.

Thank you for your time and consideration.

Best regards,
Authors of Submission #55484

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.