- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces MarkPatentBench, a multimodal benchmark for patentability assessment in drug discovery. It combines patent claims, Markush structure images, and source-linked examination evidence, and evaluates models on five dimensions—novelty, inventive step, support, clarity, and eligibility—as well as historical grant-outcome prediction. The paper also proposes EGPA, which derives reusable assessment rules from training examples and examiner feedback to improve downstream assessment and prediction.
Critical strengths
- Novel and practically relevant benchmark setting. The benchmark goes beyond structure recognition by requiring joint reasoning over claims, Markush structures, and examination evidence. This is a meaningful and underexplored problem in scientific and legal reasoning.
- Strong emphasis on provenance and expert review. The dataset preserves source identities and examination records, and includes domain-expert quality review. This substantially improves traceability compared with many automatically constructed benchmarks.
Critical weaknesses
- The evaluation does not directly establish patentability correctness. Task 1 relies mainly on embedding similarity and ROUGE-L against examiner passages, which measure textual alignment rather than whether the generated assessment is legally or chemically correct. The paper itself notes that similarity does not verify correct assessment of the supplied claim version.
- Historical grant outcome is an imperfect proxy for patentability. A recorded grant or non-grant outcome can reflect procedural events, amendments, withdrawals, or claim-version changes rather than the substantive patentability of the supplied claim. This weakens the interpretation of Task 2 as “patentability prediction.”
Questions for authors
- How should Task 1 scores be interpreted given that textual similarity to examiner passages does not guarantee substantive correctness?
- Could the authors clarify how often the historical outcome corresponds directly to the patentability of the exact supplied claim version?
- Would expert evaluation of model-generated assessments change the ranking of the main methods?
Response to Reviewer ELBA
W1: Textual metrics and substantive correctness
We agree that textual similarity alone cannot establish substantive correctness. Section 4.1 defines Task 1 metrics as alignment with available examiner passages. In the revision, we have recruited human domain experts to rate model outputs separately (Figure 5(b); Appendix K.7) as a complementary evaluation. Please refer to the updated manuscript.
W2: Historical labels and claim-version correspondence
Section 3.2 identifies the claim version before extraction and requires a substantive objection tied to that version for EP negatives. This correspondence is separate from later application status, including non-final historical records. We have updated our manuscript in Appendix K.1 to explain the distinction.
Q1: Interpreting Task 1 scores
Task 1 measures reference alignment, not the percentage of legally correct assessments. Null controls show a modest case-specific signal above within-domain background similarity; expert ratings assess output quality separately. Task 2 uses classification metrics for recorded outcomes. We have updated the manuscript in Sections 3.1 and 4.1 accordingly.
Q2: Correspondence to the exact supplied claim version
Claim numbers, dates, and version references locate the corresponding claim document. The input audit verifies consistency with selected records, while EP negative screening checks for a version-linked objection. These checks serve different purposes; record consistency alone does not establish correct version selection (Section 3.2; Appendix K.1).
Q3: Whether expert ratings change the method ranking
Blinded expert ratings preserve the ordering EPA > Fixed RAG > Naive, with paired intervals supporting the differences on the reviewed subset. The positive pooled correlation is accompanied by a weaker method-centered correlation and variable within-method estimates. This supports method-level comparison on this subset, not a general proxy for case-level correctness or outcome prediction (Figure 5(b); Appendix K.7).
Two reviewers completed 30 cases (15 positive, 15 negative), yielding 60 case–reviewer forms and 900 dimension ratings on a 0–2 rubric. Intervals use 10,000 case-bootstrap samples, keeping methods paired. Only completed ratings are included.