acceptodds
Under review as a conference paper at ICLR 2027

AUDITING TRAINING–EVALUATION OVERLAP, NEGATIVE SAMPLING, AND RERANKING IN MULTILINGUAL PATENT RETRIEVAL

Abstract

Cross-language retrieval systems are usually evaluated against a fixed benchmark, and the resulting numbers are read as evidence about representation quality. We show that in multilingual patent retrieval this reading is fragile: three ordinary design decisions—how negative examples are mined, which overlap definition an audit adopts, and whether a first-stage candidate space is validated—each change what a headline conclusion appears to be. We present CLARITY (Cross-Language Audited Retrieval with Integrity and Transparency), an audited data and evaluation workflow for multilingual patent counterpart retrieval spanning Chinese–English, Japanese–English, and Korean–English (2.692M bilingual records; 23,622 queries; 3,000–4,854 target-language documents per language pair). We contribute (i) a leakage audit that separates two definitions of training–benchmark exposure and shows they differ by 71–837× across language pairs: 70–76% of evaluation queries appear in the reranker training corpus by querytext match, but only 0.08–1.06% by exact query/positive pair; a control sample of non-benchmark titles matches at 79.7%, and the benchmark loose-exposure rates are 3.5–9.9 pp below this baseline, indicating that query-text overlap reflects corpus vocabulary intersection rather than answer memorization; (ii) a reranking evaluation corrected under the strict definition, where the reranker still improves NDCG@10 by +0.0249, +0.1446, and +0.1397 on the three pairs; and (iii) a repaired hard-negative pipeline: an earlier miner placed the query itself among the negatives in 2.51–9.14% of rows, and after excluding the query, same-component rows, and duplicates, hard negatives beat random negatives on all three language pairs. Baseline measurements for BM25, LaBSE, mE5, BGE-M3, and Qwen3-Embedding show that point-estimate rankings vary across language pairs, and direction asymmetry reaches 0.2619 NDCG@10 for mE5 on Korean–English.

Primary area:
datasets and benchmarks
Keywords:
Multilingual Patent Retrieval, Benchmark Evaluation, Training–Evaluation Overlap, Hard Negative Mining, Reranking

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_w4pyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds a cross-language patent retrieval benchmark. Queries are patent titles in Chinese, Japanese or Korean, or in English for the reverse direction, and each query has one counterpart document in the other language (23,622 queries in total). The paper audits three choices that change what the benchmark appears to show.

First, it measures overlap between the reranker's training corpus and the benchmark under two definitions. Under the loose definition, the query text appears anywhere in training (70% to 76% of queries). Under the strict definition, the exact query and positive pair appears (0.08% to 1.06%). Non-benchmark control titles match at 79.7% under the loose definition, and the authors conclude that loose overlap reflects shared vocabulary rather than memorized answers. Second, the Qwen3 reranker still improves nDCG@10 on the strict-clean subset for all three language pairs. Third, an earlier hard-negative miner placed the query itself among the negatives in 2.5% to 9.1% of rows. After repair and retraining with three seeds, hard negatives beat random negatives on all pairs by small margins. The paper also reports five pretrained baselines with paired bootstrap intervals and shows large differences between query directions.

Critical strengths

The statistical reporting is careful. Intervals come from paired bootstrap over publications with both query directions kept together, model comparisons use randomization tests with Holm correction, and adaptation results use three seeds. The smallest effect is extended to eight seeds.

The central point of the audit is useful beyond patents. A reported leakage rate can change by two orders of magnitude depending on the overlap definition, and benchmarks rarely say which definition they use. The paper also states the limits of its own audit plainly (Appendix B).

Critical weaknesses
  1. The strict definition may miss the leakage that matters, and the control does not settle the question.The training data contains six views per publication (Eq. 2), including source title to English abstract. The benchmark document is the English title plus abstract. A query whose publication appears in training with a different field pair is strict-clean, yet the model has seen that query mapped to most of the benchmark answer. Appendix B confirms that the audit does not detect "query–answer pairs that co-occur in the training corpus with different field combinations". Section 3.4 says the matched training positives "differ from the annotated benchmark counterpart", but not whether they come from the same publication.

The control does not resolve this. The control titles are drawn "from the archived corpora" (Appendix E), which appear to be the source of the reranker training corpus, so a high match rate is expected. Loose matching is also an exact match of the whole normalized title, which is not vocabulary overlap in the usual sense. A publication-level audit is needed before the reranking gain can be read as a held-out result.

  1. The hard-versus-random conclusion depends on one temperature, and the benchmark's scope is narrow. In Table 12, random negatives score higher than hard negatives on JP–EN and KR–EN at τ = 0.05 and τ = 0.07. The best random-negative KR–EN score (0.8511) is higher than every hard-negative KR–EN score in Tables 11 and 12, the highest of which is 0.8250. Section 4.6 acknowledges this, but the abstract and contribution 3 state without qualification that hard negatives beat random negatives on all three pairs. τ was also chosen on the same dev partition used for the comparison, and the margins at τ = 0.1 are small (+0.0018 to +0.0199).

The task is also close to saturation (Qwen3 reaches 0.968 on CN–EN) and is closer to translation retrieval than to prior-art search, as the paper notes. This limits what the benchmark tells us about multilingual representations. Much of the contribution corrects the authors' own earlier pipeline, and the relation to that earlier study is not explained.

Minor points (these do not drive my rating).

  • The source and license of the patent records are not stated, which matters for a benchmark intended for release.
  • Only 49% of Japanese records have English fields (Table 1). The paper does not discuss whether this biases the JP–EN sample.
  • The title and the system name (CLARITY) differ, which makes the paper's identity harder to track.
Questions for authors

Questions that could change my assessment:

  1. Among loose-exposed queries, what fraction have their own publication in the reranker training corpus under any field combination? What is the reranking gain after removing all such queries?
  2. Are the 2,000 control titles drawn from the same archive used to build the reranker training corpus?
  3. Does the hard-negative advantage hold if τ is tuned separately for each condition on a split that is not used for the comparison?
  4. What is the "earlier study" mentioned in Sections 1 and 4.2? Has it been published, and how does this submission differ from it?

Clarifications:

  1. What is the source and license of the patent records, and will the benchmark data be released?
  2. Why do only 49% of Japanese records have English fields?
4 comments

Thank you for specifying the checks that could change your assessment. We are addressing them directly, and the controlled supplementary training is underway.

Same-publication exposure and filtered gains. Another field pairing from the same publication can expose answer content, so exact-pair exclusion is insufficient for a source-independent interpretation. Our checkpoint-specific recheck changes the submission’s exposure counts and the interpretation of its filtered subsets; we will replace the affected audit tables and corresponding claims. Source-linked patent and family metadata are now available. We are verifying the same-publication fraction among text-exposed queries and reranking gains after source exclusion, with coverage and ambiguity reported. Additional source/family-aware holdouts have been constructed, and independently retrained rerankers will be evaluated on them. We will provide the verified exclusion results in the follow-up.

Control titles. They are archive-derived, so the control does not establish independence from the training source. We will remove that interpretation and describe whole-title matching as title-text overlap, rather than vocabulary overlap.

Temperature and repair. Random, uncorrected-hard, and repaired-hard conditions receive the same temperature grid, three paired seeds, and a tuning partition separate from the final comparison. We will report both common-temperature and separately tuned contrasts, with hard-negative claims matched to the results.

Earlier study and task scope. “Earlier study” was an unpublished internal pilot. We will remove the ambiguous wording and use the new controlled comparison as the primary repair evidence. The patent task is bilingual counterpart retrieval. External validation uses MIRACL-zh for monolingual retrieval, adds HC4-zh for English-to-Chinese news retrieval, and uses BEIR’s NFCorpus, SciFact, and FiQA for cross-domain/task transfer. Models will be selected only on patent tuning data. Full-corpus retrieval and shared-top100 reranking will be reported separately, including HC4 nDCG@10 and full-ranking MAP with per-topic results, paired confidence intervals, and tests.

Data provenance and selection. The records originate from public patent-office sources, including CNIPA, EPO, WIPO, JPO, MOIP, etc. The revision will document acquisition, English-field provenance, Japanese bilingual-field selection, and the release route under applicable terms. The 49% figure is a field-availability yield; its representativeness will be analysed rather than assumed.

We will follow up with completed results and a concise account of the revised PDF.

Thank you for the response and the revised PDF. My factual questions are answered, and I appreciate that the revision reports negative results plainly, including the 82–91% answer-abstract overlap in the audit corpus ( Table 17 ) and the adapted reranker scoring below the unadapted one (Tables 9–10 )

However my main concerns about the contribution remain :

  1. The clearest positive effect (H1 over H0 ) comes from removing the query itself (23–26% of views) and other fields of the positive publication (35–40%, Table 2) from the negatives. On MIRACL-ZH, where known positives are already protected, the added filter gives no gain (Table 19), while H1 scores below uniform sampling for BGE-M3 on JP–EN and KR–EN (Table 24) and on all four public benchmarks (Table 18) .
  2. The English fields are Google Translate outputs (Section 3.1) and title-only candidates match or beat title-plus-abstract documents on JP–EN and KR–EN ( Table 8 ) so the task is still very close to translation retrieval.
  3. The raw patent records and translations will not be released ( Appendix E) which limits the paper's value for the Datasets and Benchmarks track.

For these reasons, I am keeping my original score of 2.

1.Publication identity, field reuse and the control titles

Among query-role matches, answer abstracts occur in any role for 369/405 Chinese, 1478/1679 Japanese and 1065/1294 Korean queries, or 91.1%, 88.0% and 82.3%. These are field-overlap rates, and we no longer use exact-pair absence as evidence of source independence. The held-out performance claims use the record-based isolated allocations. The claims concern controlled adaptation and evaluation, without inferring a pretraining memorization effect from exact matching.

The 2,000 control titles came from the same archived source collection. We removed their use as evidence against answer exposure and replaced the vocabulary-overlap interpretation with normalized title/field reuse. We removed the causal attribution of larger exposed-query gains to training exposure. The reranking comparison now uses source-isolated records and identical Qwen3 R seed-42 top-100 shortlists for unadapted and three adapted Qwen3-Reranker-0.6B versions. On the observed-family/text 100k test, unadapted nDCG@10 is 0.9366/0.8022/0.7150, versus 0.9308/0.7741/0.6907 for adapted seed means. Adapted minus unadapted effects are -0.0059/-0.0280/-0.0243, with jointly adjusted p values 0.0020/0.0018/0.0018. Adapted scoring improves over the dense stage in Japanese and Korean while scoring below the matched unadapted reranker. This distinguishes the value of a reranking stage from the value of this adaptation recipe. Cluster intervals condition on the fitted models, and seed variation is reported separately.

Manuscript changes Sections 3.1–3.2 and 5.3; Tables 9–10 and 17; Appendices A and C.

  1. Independent tuning and backbone dependence

Each language and condition independently selects temperature from 0.05, 0.07 and 0.1 on a separate nominal-3k tuning allocation, using the same three seeds and training budgets, then freezes that choice before the 100k test. Tuned H1 minus R nDCG@10 is +0.0190/+0.0387/+0.0195 for Qwen3-Embedding-0.6B but +0.0052/-0.0091/-0.0116 for BGE-M3 in CN/JP/KR. Both complete tuning grids and fixed-temperature controls are reported. The Abstract, Section 5.1 and Conclusion explicitly describe the score-selection benefit as backbone dependent.

Manuscript changes Abstract; Sections 4 and 5.1; Tables 5 and 23–25; Conclusion 8.

  1. Task scope and the earlier pilot

The benchmark is explicitly bidirectional machine-translated counterpart retrieval, with one labeled publication per query. We added nested 3k/10k/100k candidate pools and matched title-only, abstract-only and title-plus-abstract diagnostics. Title-only recovery can match or exceed the full-document point estimate in Japanese and Korean, which we use to characterize the counterpart task and its field dependence. We do not interpret these scores as exhaustive prior-art recall. HC4 provides separately judged cross-language news retrieval, while MIRACL and BEIR evaluations retain their own relevance labels and full candidate corpora.

The earlier study referred to an unpublished internal pilot. We removed that ambiguous wording and use the newly fitted matched experiments as the primary evidence, with their complete recipes and results.

Manuscript changes Sections 1, 3.1 and 5.2–5.4; Tables 8 and 14; Appendix A.

  1. Data provenance and public release

The records originate from public patent-office resources, including CNIPA, USPTO, EPO, JPO and MOIP, and the English fields were generated with Google Translate. Public distribution of the original patents, translations, derived benchmark and results is authorized. We corrected Appendix E and the release documentation and prepared a companion archive containing the three original language snapshots and the exact frozen training, tuning, test and candidate allocations. Publication numbers and available family metadata are preserved. The supplement provides negative-bank indices, query metrics, fitting recipes and statistical replay code; the companion records reconstruct the exact bank messages.

Manuscript changes Section 3.1; Appendix E; supplement.

  1. Japanese field availability

The revised snapshot denominator is 248,276 complete four-field records out of 400,000 Japanese records, or 62.07%, with 151,724 incomplete records retained for the availability audit. The earlier 49% figure does not describe this evaluation population. We compare publication-year and IPC-section distributions; for example, 1990s records account for 35.31% of the complete group and 12.61% of the incomplete group. The analysis documents the field-complete selection rather than assigning an unsupported cause to missing English fields.

Manuscript changes Section 3.1; Table 22; Appendix E.

  1. Title and framework name

The title and framework now consistently use CLEAR. It expands to Cross-Language Evaluation and Auditing of Retrieval; under Multi-View Supervision specifies the setting. The acronym therefore covers the framework name, while the final title phrase states its scope.

We agree that the strongest positive result is the controlled removal of structurally ineligible negatives, rather than evidence that hard negatives are universally better. Table 2 reports the query-self and positive-publication fields retained by H0; the matched H1 minus H0 comparison measures the joint eligibility-and-uniqueness intervention. Where known positives are protected in MIRACL, the extra article proxy has no established full-corpus gain. BGE-M3 on Japanese-English and Korean-English and the four public-transfer point estimates likewise favor R over H1. We retain these results and restrict the contribution to a source-aware benchmark, explicit multi-view eligibility checks and matched controls that make those different outcomes reproducible. We do not attribute the gain to negative difficulty alone.

We also agree about the task. Section 3.1 now consistently defines bidirectional machine-translated counterpart recovery, and Table 8 documents its dependence on title versus abstract fields. The title-only results characterize that setting; they are not evidence of broad prior-art relevance retrieval. The independent HC4, MIRACL and BEIR evaluations are reported as different tasks with their original relevance judgments, and their outcomes do not support a blanket transfer claim.

The release concern has led to a concrete correction. Appendix E no longer excludes raw patent records or translations. The patent data totals 1.28G. Due to its large size, it has not been uploaded yet. CLEAR_supplement.zip, with negative indices, query-level results, configurations and replay code. Patent publication numbers are preserved, and the companion records match the hashes used for bank reconstruction. This makes the benchmark inputs and derived evidence available as public-distribution materials, in addition to the controlled comparison.

Manuscript changes Sections 3.1, 5.1–5.4; Tables 2, 8–10, 18–19 and 23–25; Appendix E; release materials.

Reviewer_Cyw4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents CLARITY, an auditing and evaluation workflow for multilingual patent retrieval. It examines training-evaluation overlap, negative sampling defects, and reranking reliability across Chinese-English, Japanese-English, and Korean-English retrieval. The experiments show substantial differences between loose and strict exposure definitions, improvements after correcting negative sampling, and consistent reranking gains on strictly filtered evaluation subsets.

Critical strengths

The paper highlights practical evaluation problems in multilingual retrieval. Distinguishing query-text overlap from exact query-positive overlap is useful.

The experiments include multiple language pairs, several embedding models, statistical testing, and detailed audits of negative sampling and reranking.

Critical weaknesses

The paper mainly applies established data auditing, negative sampling, and reranking procedures. The generalizable contribution beyond correcting the specific experimental pipeline is not sufficiently clear.

The strict exposure definition relies on exact query-positive matching. It cannot identify semantic duplicates, patent-family overlap, or potential exposure during model pretraining, limiting the completeness of the leakage analysis.

The benchmark uses relatively small candidate collections and only one relevant counterpart per query.

The improvements from corrected hard negatives are modest, particularly for Korean-English retrieval. Moreover, Appendix H shows that hard negatives do not consistently outperform random negatives under different temperature settings.

The interpretation of exposure effects on reranking performance is not fully supported. Exposed queries also have substantially lower initial retrieval scores, so the larger reranking gains may reflect query difficulty rather than training exposure alone.

Questions for authors

Clarify the generalizable contributions beyond the specific pipeline corrections.

Discuss the limitations of exact-match exposure auditing.

Explain the representativeness of the benchmark for real patent retrieval.

Clarify the sensitivity of hard-negative gains and reranking improvements.

4 comments

Thank you for identifying where the contribution, exposure analysis, and evaluation scope need stronger evidence. The controlled supplementary experiments are underway.

Contribution and audit scope. We will focus the contribution on an auditable connection between overlap definitions, negative-mining invariants, and controlled performance comparisons, with reusable checks and manifests. Exact-pair overlap is one measurement, not a certificate of independence. Our checkpoint-specific recheck changes the submission’s exposure counts and the interpretation of its filtered subsets; we will replace the affected tables and corresponding claims. Source-linked metadata and additional source/family-aware holdouts are now available. These checks address adaptation-data exposure; they do not establish absence of foundation-model pretraining exposure.

Fair tuning and sensitivity. Random, uncorrected-hard, and repaired-hard conditions use identical query–positive pairs and budgets, three paired seeds, and the same temperature grid. Selection uses a separate tuning partition, followed by held-out evaluation. We will distinguish the common-temperature repair contrast from the separately tuned pipeline contrast and qualify hard-negative claims according to the completed results. Paired query-level intervals describe uncertainty conditional on fitted checkpoints; seed variation will be reported separately.

Representativeness and independent validation. The patent task will be described consistently as bilingual counterpart retrieval with one labeled counterpart, with its implications for prior-art retrieval clearly limited. Supplementary evaluation includes larger patent candidate pools, duplicate sensitivity, MIRACL-zh for monolingual retrieval, HC4-zh for genuine English-to-Chinese news retrieval, and BEIR’s NFCorpus, SciFact, and FiQA for cross-domain/task transfer. Public test scores will not select models or temperatures. We will distinguish full-corpus retrieval from shared-top100 reranking, report HC4 nDCG@10 and full-ranking MAP with per-topic results, confidence intervals, and paired tests, and retain each dataset’s original relevance labels. Difficulty-stratified reranking gains will be interpreted as associations because initial retrieval performance can confound an exposure-based explanation.

We will provide completed results and identify the manuscript changes in a follow-up comment.

The larger candidate pools and matched tuning budgets make the comparisons more convincing. I’m still cautious about how broadly the benefit of hard negatives carries over, since uniform sampling worked better for BGE-M3 on both Japanese--English and Korean--English. I’m keeping my original score.

Following our first response, we now report the completed supplementary experiments, their conclusions and the corresponding manuscript changes.

  1. Contribution

CLEAR contributes an evaluation design for multi-view retrieval. It connects patent publication identity and complete-field links to source-isolated splits, six matched positive views, and alternative negative banks. R and H1 share eligibility rules, while H0 and H1 share the frozen ranked cache. This separates the fixed-temperature constraint comparison from score selection under equal tuning. We completed 162 patent encoder fits, 81 per backbone, and provide the split records, bank indices, query-level results and replay code. The contribution is this reproducible benchmark and controlled comparison, rather than a claim that one negative-sampling policy is best across tasks.

Manuscript changes Introduction; Figure 2; Section 4; Table 4; Discussion 6; Conclusion 8.

  1. Exposure and source isolation

We replaced pair-only filtering with disjoint publication/full-field and observed-family/text allocations for training, tuning and testing. A publication ID means the patent publication number. It is present as publication_number in the raw data and retained as publication_id in the experimental records; it does not describe whether the data are public. The separate text-pair audit measures normalized field reuse. Among query-role matches, answer abstracts occur in any role for 369/405 Chinese, 1478/1679 Japanese and 1065/1294 Korean queries, or 91.1%, 88.0% and 82.3%. These are field-overlap rates, and we no longer use exact-pair absence as evidence of source independence. The held-out performance claims use the record-based isolated allocations. The claims concern controlled adaptation and evaluation, without inferring a pretraining memorization effect from exact matching.

Manuscript changes Sections 3.1–3.2; Table 17; Appendices A and C.

  1. Benchmark scope

The benchmark is explicitly bidirectional machine-translated counterpart retrieval, with one labeled publication per query. We added nested 3k/10k/100k candidate pools and matched title-only, abstract-only and title-plus-abstract diagnostics. Title-only recovery can match or exceed the full-document point estimate in Japanese and Korean, which we use to characterize the counterpart task and its field dependence. We do not interpret these scores as exhaustive prior-art recall. HC4 provides separately judged cross-language news retrieval, while MIRACL and BEIR evaluations retain their own relevance labels and full candidate corpora.

Manuscript changes Section 3.1; Sections 5.2 and 5.4; Tables 8 and 14; Limitations 7.

  1. Selection sensitivity and reranking controls

Each language and condition independently selects temperature from 0.05, 0.07 and 0.1 on a separate nominal-3k tuning allocation, using the same three seeds and training budgets, then freezes that choice before the 100k test. Tuned H1 minus R nDCG@10 is +0.0190/+0.0387/+0.0195 for Qwen3-Embedding-0.6B but +0.0052/-0.0091/-0.0116 for BGE-M3 in CN/JP/KR. Both complete tuning grids and fixed-temperature controls are reported. The Abstract, Section 5.1 and Conclusion explicitly describe the score-selection benefit as backbone dependent.

We removed the causal attribution of larger exposed-query gains to training exposure. The reranking comparison now uses source-isolated records and identical Qwen3 R seed-42 top-100 shortlists for unadapted and three adapted Qwen3-Reranker-0.6B versions. On the observed-family/text 100k test, unadapted nDCG@10 is 0.9366/0.8022/0.7150, versus 0.9308/0.7741/0.6907 for adapted seed means. Adapted minus unadapted effects are -0.0059/-0.0280/-0.0243, with jointly adjusted p values 0.0020/0.0018/0.0018. Adapted scoring improves over the dense stage in Japanese and Korean while scoring below the matched unadapted reranker. This distinguishes the value of a reranking stage from the value of this adaptation recipe. Cluster intervals condition on the fitted models, and seed variation is reported separately.

Manuscript changes Sections 4, 5.1 and 5.3; Tables 5, 9–10 and 23–25; Appendix A.2.

We agree that the BGE-M3 results limit the breadth of a hard-negative benefit. Uniform eligible sampling performs better on Japanese-English and Korean-English under the same tuning opportunity, and those negative H1 minus R results remain in the main comparison. The revised claim distinguishes two effects. Fixed-temperature H1 minus H0 supports the joint eligibility-and-uniqueness intervention with both backbones; tuned H1 minus R evaluates score selection and varies by backbone and task. The public transfer and MIRACL results reinforce that distinction. We therefore present CLEAR as a reproducible evaluation protocol with calibrated empirical findings, rather than as a universally superior hard-negative method. The Abstract, Section 5.1, Discussion and Conclusion state this scope explicitly.

Manuscript changes Abstract; Section 5.1; Tables 23–25; Discussion 6; Conclusion 8.

Reviewer_Hm4VOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents CLARITY, a workflow for auditing multilingual patent retrieval across Chinese–English, Japanese–English, and Korean–English. It distinguishes query-text overlap from exact query–positive document pair overlap, showing substantially different exposure rates of 70–76% and 0.08–1.06%, respectively. The paper further shows that reranking gains persist after excluding evaluation queries with exact pair-level overlap. It also identifies and corrects errors in hard-negative mining, including query-as-negative and duplicate negatives, and shows that corrected hard negatives outperform random negatives under the main experimental setting. Finally, an evaluation of pretrained retrievers reveals performance differences across language pairs and retrieval directions.

Critical strengths
  1. The paper presents a leakage-aware evaluation framework for multilingual patent retrieval, which is interesting and provides a novel perspective on retrieval evaluation.
  2. Extensive experiments are conducted to validate leakage-corrected reranking and the repaired hard-negative mining pipeline. The experiments are carefully designed and provide valuable insights into multilingual retrieval evaluation.
Critical weaknesses
  1. Limitations of the strict leakage definition. Figure 2 shows that training–evaluation overlap is very low under the strict exposure definition. However, training information may still be exposed through indirect semantic or logical connections that are not captured by exact query–positive pair matching. The paper needs to further examine whether the strict definition adequately captures potential leakage.
  2. Insufficient analysis of leakage effects on reranking. Section 4.3 shows that queries with strict exposure exhibit larger reranking gains, suggesting that leakage may affect reranking performance. However, the evaluation considers only exact pair-level overlap. Other forms of potential leakage, such as semantic overlap and patent-family overlap, should also be examined to provide a more convincing assessment of the reranking gains.
  3. Limited validation of the audit findings. Although the paper suggests broader applicability, the experiments are limited to the authors' patent dataset and pipeline. Independent validation on other datasets would strengthen the generalizability of the findings. Furthermore, while Table 3 confirms the removal of the identified negative-mining defects, Table 7 compares corrected hard negatives with random negatives rather than directly evaluating the effect of the correction. A controlled comparison between the original and corrected mining procedures is needed to establish the contribution of the repair.
Questions for authors
  1. Beyond the discussion in Section 2, is there empirical evidence that similar audit findings hold in an independent setting? Evaluation on another benchmark, if available, would help establish the generalizability of the findings.
  2. Given the limitations of the archived checkpoints discussed in Section 4.2, can the authors further clarify the specific contribution of the hard-negative miner repair? A controlled comparison between the original and corrected miners on the same text-isolated split would help distinguish the effect of the repair from that of hard-negative sampling.
  3. In Section 4.3, the authors state that exposure inflates the reranking gains. However, the exposed queries also have substantially lower first-stage retrieval performance. Can the authors clarify whether the larger gains are attributable to exposure rather than differences in query difficulty?
2 comments

Thank you for the specific suggestions on exposure, repair attribution, and independent validation. The controlled supplementary experiments are underway.

Broader exposure audit. Exact query–positive matching measures one form of textual overlap; it does not establish source independence. Our checkpoint-specific recheck changes the submission’s exposure counts and the interpretation of its filtered subsets; we will replace the affected audit tables and corresponding claims. Source-linked patent and family metadata are now available, and additional source/family-aware holdouts have been constructed. The revision will distinguish exact-field, same-publication, and observed family overlap, with metadata coverage reported.

Contribution of the repair. Random negatives (R), a controlled reconstruction of the uncorrected miner (H0), and the repaired miner (H1) share query–positive pairs, mining representations, candidate rankings, training budgWe are retaining MIRACL-zhets, and three paired seeds. All conditions receive the same temperature grid and separate tuning/testing partitions. The common-temperature H1−H0 comparison addresses the filter rules; H1−R addresses hard-negative sampling. We will report these separately rather than infer repair benefits from hard-versus-random results alone.

Difficulty and independent validation. Larger gains on exposed queries may reflect lower first-stage performance; difficulty-stratified comparisons will therefore be described as associations. We are using MIRACL-zh for large-scale monolingual retrieval and controlled external auditing, adding HC4-zh as the principal non-patent English-to-Chinese retrieval test, and evaluating BEIR’s NFCorpus, SciFact, and FiQA for cross-domain/task transfer. NFCorpus also supports a natural negative-validity audit against its training qrels. Transfer models will be selected only on independent patent tuning data, with shared reranking candidates. We will distinguish full-corpus retrieval from top100 reranking and report HC4 nDCG@10 and full-ranking MAP with per-topic results and paired uncertainty estimates. Training-seed variation will be reported separately.

We will follow up with completed results and the corresponding changes in the revised PDF.

Following our first response, we now report the completed supplementary experiments, their conclusions and the corresponding manuscript changes.

1.Broader exposure and source isolation

We replaced pair-only filtering with disjoint publication/full-field and observed-family/text allocations for training, tuning and testing. A publication ID means the patent publication number. It is present as publication_number in the raw data and retained as publication_id in the experimental records; it does not describe whether the data are public. The separate text-pair audit measures normalized field reuse. Among query-role matches, answer abstracts occur in any role for 369/405 Chinese, 1478/1679 Japanese and 1065/1294 Korean queries, or 91.1%, 88.0% and 82.3%. These are field-overlap rates, and we no longer use exact-pair absence as evidence of source independence. The held-out performance claims use the record-based isolated allocations. The claims concern controlled adaptation and evaluation, without inferring a pretraining memorization effect from exact matching.

Manuscript changes Sections 3.1–3.2; Table 17; Appendices A and C.

  1. Reranking and query difficulty

We removed the causal attribution of larger exposed-query gains to training exposure. The reranking comparison now uses source-isolated records and identical Qwen3 R seed-42 top-100 shortlists for unadapted and three adapted Qwen3-Reranker-0.6B versions. On the observed-family/text 100k test, unadapted nDCG@10 is 0.9366/0.8022/0.7150, versus 0.9308/0.7741/0.6907 for adapted seed means. Adapted minus unadapted effects are -0.0059/-0.0280/-0.0243, with jointly adjusted p values 0.0020/0.0018/0.0018. Adapted scoring improves over the dense stage in Japanese and Korean while scoring below the matched unadapted reranker. This distinguishes the value of a reranking stage from the value of this adaptation recipe. Cluster intervals condition on the fitted models, and seed variation is reported separately.

Manuscript changes Sections 3.2, 4.2 and 5.3; Tables 9–10; Appendix A.2; Conclusion 8.

  1. Controlled repair and independent evaluation

The new comparison uses identical ordered positive rows, a shared frozen cache, paired seeds, 512 updates and effective batch 256. At fixed temperature 0.1, H1 minus H0 nDCG@10 is +0.0060/+0.0410/+0.0479 for Qwen3-Embedding-0.6B and +0.0126/+0.0340/+0.0304 for BGE-M3. This directly evaluates the joint eligibility-and-uniqueness intervention under matched conditions. H1 minus R answers the separate question of score selection among eligible candidates. The common Qwen3-mined banks are reused by both encoders, so the second backbone tests the same bank recipe with a different fitted encoder.

We completed full-corpus patent-adapter transfer on HC4, NFCorpus, SciFact and FiQA, and nine independently trained MIRACL Chinese fits. Patent-selected H1 minus R point estimates are nonpositive on all four transfer sets; HC4 and SciFact intervals include zero. MIRACL evaluates all 393 official development queries and 4,934,368 passages. Its full-corpus H1 minus H0 is -0.0003 with 95% interval [-0.0033, 0.0026], and H1 minus R is +0.0019 [-0.0055, 0.0095], both adjusted p=1.0. The extra same-article proxy changes 140/959 training rows without establishing an added full-corpus gain. These independent results are reported alongside the patent effects and delimit where the tested selection rules help.

Manuscript changes Sections 4, 5.1 and 5.4; Tables 13 and 18–20; Tables 23–25; Appendices A, D and G.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.