- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper builds a cross-language patent retrieval benchmark. Queries are patent titles in Chinese, Japanese or Korean, or in English for the reverse direction, and each query has one counterpart document in the other language (23,622 queries in total). The paper audits three choices that change what the benchmark appears to show.
First, it measures overlap between the reranker's training corpus and the benchmark under two definitions. Under the loose definition, the query text appears anywhere in training (70% to 76% of queries). Under the strict definition, the exact query and positive pair appears (0.08% to 1.06%). Non-benchmark control titles match at 79.7% under the loose definition, and the authors conclude that loose overlap reflects shared vocabulary rather than memorized answers. Second, the Qwen3 reranker still improves nDCG@10 on the strict-clean subset for all three language pairs. Third, an earlier hard-negative miner placed the query itself among the negatives in 2.5% to 9.1% of rows. After repair and retraining with three seeds, hard negatives beat random negatives on all pairs by small margins. The paper also reports five pretrained baselines with paired bootstrap intervals and shows large differences between query directions.
Critical strengths
The statistical reporting is careful. Intervals come from paired bootstrap over publications with both query directions kept together, model comparisons use randomization tests with Holm correction, and adaptation results use three seeds. The smallest effect is extended to eight seeds.
The central point of the audit is useful beyond patents. A reported leakage rate can change by two orders of magnitude depending on the overlap definition, and benchmarks rarely say which definition they use. The paper also states the limits of its own audit plainly (Appendix B).
Critical weaknesses
- The strict definition may miss the leakage that matters, and the control does not settle the question.The training data contains six views per publication (Eq. 2), including source title to English abstract. The benchmark document is the English title plus abstract. A query whose publication appears in training with a different field pair is strict-clean, yet the model has seen that query mapped to most of the benchmark answer. Appendix B confirms that the audit does not detect "query–answer pairs that co-occur in the training corpus with different field combinations". Section 3.4 says the matched training positives "differ from the annotated benchmark counterpart", but not whether they come from the same publication.
The control does not resolve this. The control titles are drawn "from the archived corpora" (Appendix E), which appear to be the source of the reranker training corpus, so a high match rate is expected. Loose matching is also an exact match of the whole normalized title, which is not vocabulary overlap in the usual sense. A publication-level audit is needed before the reranking gain can be read as a held-out result.
- The hard-versus-random conclusion depends on one temperature, and the benchmark's scope is narrow. In Table 12, random negatives score higher than hard negatives on JP–EN and KR–EN at τ = 0.05 and τ = 0.07. The best random-negative KR–EN score (0.8511) is higher than every hard-negative KR–EN score in Tables 11 and 12, the highest of which is 0.8250. Section 4.6 acknowledges this, but the abstract and contribution 3 state without qualification that hard negatives beat random negatives on all three pairs. τ was also chosen on the same dev partition used for the comparison, and the margins at τ = 0.1 are small (+0.0018 to +0.0199).
The task is also close to saturation (Qwen3 reaches 0.968 on CN–EN) and is closer to translation retrieval than to prior-art search, as the paper notes. This limits what the benchmark tells us about multilingual representations. Much of the contribution corrects the authors' own earlier pipeline, and the relation to that earlier study is not explained.
Minor points (these do not drive my rating).
- The source and license of the patent records are not stated, which matters for a benchmark intended for release.
- Only 49% of Japanese records have English fields (Table 1). The paper does not discuss whether this biases the JP–EN sample.
- The title and the system name (CLARITY) differ, which makes the paper's identity harder to track.
Questions for authors
Questions that could change my assessment:
- Among loose-exposed queries, what fraction have their own publication in the reranker training corpus under any field combination? What is the reranking gain after removing all such queries?
- Are the 2,000 control titles drawn from the same archive used to build the reranker training corpus?
- Does the hard-negative advantage hold if τ is tuned separately for each condition on a split that is not used for the comparison?
- What is the "earlier study" mentioned in Sections 1 and 4.2? Has it been published, and how does this submission differ from it?
Clarifications:
- What is the source and license of the patent records, and will the benchmark data be released?
- Why do only 49% of Japanese records have English fields?
4 comments
Thank you for the response and the revised PDF. My factual questions are answered, and I appreciate that the revision reports negative results plainly, including the 82–91% answer-abstract overlap in the audit corpus ( Table 17 ) and the adapted reranker scoring below the unadapted one (Tables 9–10 )
However my main concerns about the contribution remain :
- The clearest positive effect (H1 over H0 ) comes from removing the query itself (23–26% of views) and other fields of the positive publication (35–40%, Table 2) from the negatives. On MIRACL-ZH, where known positives are already protected, the added filter gives no gain (Table 19), while H1 scores below uniform sampling for BGE-M3 on JP–EN and KR–EN (Table 24) and on all four public benchmarks (Table 18) .
- The English fields are Google Translate outputs (Section 3.1) and title-only candidates match or beat title-plus-abstract documents on JP–EN and KR–EN ( Table 8 ) so the task is still very close to translation retrieval.
- The raw patent records and translations will not be released ( Appendix E) which limits the paper's value for the Datasets and Benchmarks track.
For these reasons, I am keeping my original score of 2.
1.Publication identity, field reuse and the control titles
Among query-role matches, answer abstracts occur in any role for 369/405 Chinese, 1478/1679 Japanese and 1065/1294 Korean queries, or 91.1%, 88.0% and 82.3%. These are field-overlap rates, and we no longer use exact-pair absence as evidence of source independence. The held-out performance claims use the record-based isolated allocations. The claims concern controlled adaptation and evaluation, without inferring a pretraining memorization effect from exact matching.
The 2,000 control titles came from the same archived source collection. We removed their use as evidence against answer exposure and replaced the vocabulary-overlap interpretation with normalized title/field reuse. We removed the causal attribution of larger exposed-query gains to training exposure. The reranking comparison now uses source-isolated records and identical Qwen3 R seed-42 top-100 shortlists for unadapted and three adapted Qwen3-Reranker-0.6B versions. On the observed-family/text 100k test, unadapted nDCG@10 is 0.9366/0.8022/0.7150, versus 0.9308/0.7741/0.6907 for adapted seed means. Adapted minus unadapted effects are -0.0059/-0.0280/-0.0243, with jointly adjusted p values 0.0020/0.0018/0.0018. Adapted scoring improves over the dense stage in Japanese and Korean while scoring below the matched unadapted reranker. This distinguishes the value of a reranking stage from the value of this adaptation recipe. Cluster intervals condition on the fitted models, and seed variation is reported separately.
Manuscript changes Sections 3.1–3.2 and 5.3; Tables 9–10 and 17; Appendices A and C.
- Independent tuning and backbone dependence
Each language and condition independently selects temperature from 0.05, 0.07 and 0.1 on a separate nominal-3k tuning allocation, using the same three seeds and training budgets, then freezes that choice before the 100k test. Tuned H1 minus R nDCG@10 is +0.0190/+0.0387/+0.0195 for Qwen3-Embedding-0.6B but +0.0052/-0.0091/-0.0116 for BGE-M3 in CN/JP/KR. Both complete tuning grids and fixed-temperature controls are reported. The Abstract, Section 5.1 and Conclusion explicitly describe the score-selection benefit as backbone dependent.
Manuscript changes Abstract; Sections 4 and 5.1; Tables 5 and 23–25; Conclusion 8.
- Task scope and the earlier pilot
The benchmark is explicitly bidirectional machine-translated counterpart retrieval, with one labeled publication per query. We added nested 3k/10k/100k candidate pools and matched title-only, abstract-only and title-plus-abstract diagnostics. Title-only recovery can match or exceed the full-document point estimate in Japanese and Korean, which we use to characterize the counterpart task and its field dependence. We do not interpret these scores as exhaustive prior-art recall. HC4 provides separately judged cross-language news retrieval, while MIRACL and BEIR evaluations retain their own relevance labels and full candidate corpora.
The earlier study referred to an unpublished internal pilot. We removed that ambiguous wording and use the newly fitted matched experiments as the primary evidence, with their complete recipes and results.
Manuscript changes Sections 1, 3.1 and 5.2–5.4; Tables 8 and 14; Appendix A.
- Data provenance and public release
The records originate from public patent-office resources, including CNIPA, USPTO, EPO, JPO and MOIP, and the English fields were generated with Google Translate. Public distribution of the original patents, translations, derived benchmark and results is authorized. We corrected Appendix E and the release documentation and prepared a companion archive containing the three original language snapshots and the exact frozen training, tuning, test and candidate allocations. Publication numbers and available family metadata are preserved. The supplement provides negative-bank indices, query metrics, fitting recipes and statistical replay code; the companion records reconstruct the exact bank messages.
Manuscript changes Section 3.1; Appendix E; supplement.
- Japanese field availability
The revised snapshot denominator is 248,276 complete four-field records out of 400,000 Japanese records, or 62.07%, with 151,724 incomplete records retained for the availability audit. The earlier 49% figure does not describe this evaluation population. We compare publication-year and IPC-section distributions; for example, 1990s records account for 35.31% of the complete group and 12.61% of the incomplete group. The analysis documents the field-complete selection rather than assigning an unsupported cause to missing English fields.
Manuscript changes Section 3.1; Table 22; Appendix E.
- Title and framework name
The title and framework now consistently use CLEAR. It expands to Cross-Language Evaluation and Auditing of Retrieval; under Multi-View Supervision specifies the setting. The acronym therefore covers the framework name, while the final title phrase states its scope.
We agree that the strongest positive result is the controlled removal of structurally ineligible negatives, rather than evidence that hard negatives are universally better. Table 2 reports the query-self and positive-publication fields retained by H0; the matched H1 minus H0 comparison measures the joint eligibility-and-uniqueness intervention. Where known positives are protected in MIRACL, the extra article proxy has no established full-corpus gain. BGE-M3 on Japanese-English and Korean-English and the four public-transfer point estimates likewise favor R over H1. We retain these results and restrict the contribution to a source-aware benchmark, explicit multi-view eligibility checks and matched controls that make those different outcomes reproducible. We do not attribute the gain to negative difficulty alone.
We also agree about the task. Section 3.1 now consistently defines bidirectional machine-translated counterpart recovery, and Table 8 documents its dependence on title versus abstract fields. The title-only results characterize that setting; they are not evidence of broad prior-art relevance retrieval. The independent HC4, MIRACL and BEIR evaluations are reported as different tasks with their original relevance judgments, and their outcomes do not support a blanket transfer claim.
The release concern has led to a concrete correction. Appendix E no longer excludes raw patent records or translations. The patent data totals 1.28G. Due to its large size, it has not been uploaded yet. CLEAR_supplement.zip, with negative indices, query-level results, configurations and replay code. Patent publication numbers are preserved, and the companion records match the hashes used for bank reconstruction. This makes the benchmark inputs and derived evidence available as public-distribution materials, in addition to the controlled comparison.
Manuscript changes Sections 3.1, 5.1–5.4; Tables 2, 8–10, 18–19 and 23–25; Appendix E; release materials.
Thank you for specifying the checks that could change your assessment. We are addressing them directly, and the controlled supplementary training is underway.
Same-publication exposure and filtered gains. Another field pairing from the same publication can expose answer content, so exact-pair exclusion is insufficient for a source-independent interpretation. Our checkpoint-specific recheck changes the submission’s exposure counts and the interpretation of its filtered subsets; we will replace the affected audit tables and corresponding claims. Source-linked patent and family metadata are now available. We are verifying the same-publication fraction among text-exposed queries and reranking gains after source exclusion, with coverage and ambiguity reported. Additional source/family-aware holdouts have been constructed, and independently retrained rerankers will be evaluated on them. We will provide the verified exclusion results in the follow-up.
Control titles. They are archive-derived, so the control does not establish independence from the training source. We will remove that interpretation and describe whole-title matching as title-text overlap, rather than vocabulary overlap.
Temperature and repair. Random, uncorrected-hard, and repaired-hard conditions receive the same temperature grid, three paired seeds, and a tuning partition separate from the final comparison. We will report both common-temperature and separately tuned contrasts, with hard-negative claims matched to the results.
Earlier study and task scope. “Earlier study” was an unpublished internal pilot. We will remove the ambiguous wording and use the new controlled comparison as the primary repair evidence. The patent task is bilingual counterpart retrieval. External validation uses MIRACL-zh for monolingual retrieval, adds HC4-zh for English-to-Chinese news retrieval, and uses BEIR’s NFCorpus, SciFact, and FiQA for cross-domain/task transfer. Models will be selected only on patent tuning data. Full-corpus retrieval and shared-top100 reranking will be reported separately, including HC4 nDCG@10 and full-ranking MAP with per-topic results, paired confidence intervals, and tests.
Data provenance and selection. The records originate from public patent-office sources, including CNIPA, EPO, WIPO, JPO, MOIP, etc. The revision will document acquisition, English-field provenance, Japanese bilingual-field selection, and the release route under applicable terms. The 49% figure is a field-availability yield; its representativeness will be analysed rather than assumed.
We will follow up with completed results and a concise account of the revised PDF.