acceptodds
Under review as a conference paper at ICLR 2027

Beyond Structure Recognition: Benchmarking Patentability Assessment in Drug Discovery

Abstract

Assessing patentability in drug discovery requires more than recognizing chemical structures: models must interpret the scope jointly defined by structural drawings and claim language, preserve its critical limitations, and assess it against relevant evidence. Markush claims make this task particularly challenging, as a shared scaffold, variable substituents, and textual restrictions jointly define a family of compounds. We introduce **MarkPatentBench**, a multimodal benchmark for assessing the patentability of small-molecule drug claims, curated from patent examination records of the European and U.S. patent office websites. The benchmark contains claim text and Markush chemical structure images with source-linked examination reference documents. It supports overall patentability prediction and assessment generation across five core dimensions: novelty, inventive step, support by the description, claim clarity, and patent-eligible subject matter. Human domain experts review the benchmark for data quality and annotation reliability. We evaluate existing large language and vision-language models under claim-only, evidence-augmented, and agentic configurations. Our analysis highlights potential reasoning failures, including overlooking claim limitations, inferring obviousness from scaffold familiarity, and treating missing information as evidence of a substantive defect. **MarkPatentBench** provides a testbed for investigating whether models can move beyond structure recognition to preserve the claimed chemical scope and support their patentability assessments with applicable evidence.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
patentability assessment, multimodal benchmark, Markush claims, drug discovery, vision-language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ELBAOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces MarkPatentBench, a multimodal benchmark for patentability assessment in drug discovery. It combines patent claims, Markush structure images, and source-linked examination evidence, and evaluates models on five dimensions—novelty, inventive step, support, clarity, and eligibility—as well as historical grant-outcome prediction. The paper also proposes EGPA, which derives reusable assessment rules from training examples and examiner feedback to improve downstream assessment and prediction.

Critical strengths
  1. Novel and practically relevant benchmark setting. The benchmark goes beyond structure recognition by requiring joint reasoning over claims, Markush structures, and examination evidence. This is a meaningful and underexplored problem in scientific and legal reasoning.
  2. Strong emphasis on provenance and expert review. The dataset preserves source identities and examination records, and includes domain-expert quality review. This substantially improves traceability compared with many automatically constructed benchmarks.
Critical weaknesses
  1. The evaluation does not directly establish patentability correctness. Task 1 relies mainly on embedding similarity and ROUGE-L against examiner passages, which measure textual alignment rather than whether the generated assessment is legally or chemically correct. The paper itself notes that similarity does not verify correct assessment of the supplied claim version.
  2. Historical grant outcome is an imperfect proxy for patentability. A recorded grant or non-grant outcome can reflect procedural events, amendments, withdrawals, or claim-version changes rather than the substantive patentability of the supplied claim. This weakens the interpretation of Task 2 as “patentability prediction.”
Questions for authors
  1. How should Task 1 scores be interpreted given that textual similarity to examiner passages does not guarantee substantive correctness?
  2. Could the authors clarify how often the historical outcome corresponds directly to the patentability of the exact supplied claim version?
  3. Would expert evaluation of model-generated assessments change the ranking of the main methods?
1 comment
AuthorsOpenReview ↗

Response to Reviewer ELBA

W1: Textual metrics and substantive correctness

We agree that textual similarity alone cannot establish substantive correctness. Section 4.1 defines Task 1 metrics as alignment with available examiner passages. In the revision, we have recruited human domain experts to rate model outputs separately (Figure 5(b); Appendix K.7) as a complementary evaluation. Please refer to the updated manuscript.

W2: Historical labels and claim-version correspondence

Section 3.2 identifies the claim version before extraction and requires a substantive objection tied to that version for EP negatives. This correspondence is separate from later application status, including non-final historical records. We have updated our manuscript in Appendix K.1 to explain the distinction.

Q1: Interpreting Task 1 scores

Task 1 measures reference alignment, not the percentage of legally correct assessments. Null controls show a modest case-specific signal above within-domain background similarity; expert ratings assess output quality separately. Task 2 uses classification metrics for recorded outcomes. We have updated the manuscript in Sections 3.1 and 4.1 accordingly.

Q2: Correspondence to the exact supplied claim version

Claim numbers, dates, and version references locate the corresponding claim document. The input audit verifies consistency with selected records, while EP negative screening checks for a version-linked objection. These checks serve different purposes; record consistency alone does not establish correct version selection (Section 3.2; Appendix K.1).

Q3: Whether expert ratings change the method ranking

Blinded expert ratings preserve the ordering EPA > Fixed RAG > Naive, with paired intervals supporting the differences on the reviewed subset. The positive pooled correlation is accompanied by a weaker method-centered correlation and variable within-method estimates. This supports method-level comparison on this subset, not a general proxy for case-level correctness or outcome prediction (Figure 5(b); Appendix K.7).

MethodMean rating95% intervalWithin-method Spearman95% interval
Naive0.40[0.30, 0.51]0.104[-0.269, 0.433]
Fixed RAG1.10[1.01, 1.18]-0.084[-0.447, 0.295]
EPA1.66[1.60, 1.72]0.442[0.097, 0.715]
Paired or pooled statisticEstimate95% interval
EPA − Naive rating1.267[1.130, 1.393]
EPA − Fixed RAG rating0.560[0.457, 0.663]
Pooled Spearman0.394[0.309, 0.506]
Method-centered Spearman0.137—
Quadratic-distance agreement α0.606—

Two reviewers completed 30 cases (15 positive, 15 negative), yielding 60 case–reviewer forms and 900 dimension ratings on a 0–2 rubric. Intervals use 10,000 case-bootstrap samples, keeping methods paired. Only completed ratings are included.

Reviewer_jsvtOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces MarkPatentBench, a multimodal benchmark for assessing the patentability of small-molecule drug claims using patent claim text and Markush chemical structure images. It focuses on five core dimensions: novelty, inventive step, support by the description, claim clarity, and patent-eligible subject matter. The benchmark evaluates models on two tasks: core-dimensional assessment generation and grant-outcome prediction. The authors also propose an experience-guided assessment method (EGPA) to improve model performance.

Critical strengths
  1. The proposed benchmark combines Markush images, patent claims, examination records, and grant outcomes from both EPO and USPTO. This could be a useful resource for future research.
  2. The inclusion of domain-expert review to check the quality of the collected data is a positive aspect of the benchmark construction process.
Critical weaknesses
  1. In Table 4, the last row (EGPA on US) reports a very high recall of 0.982 and F1 of 0.827, despite a BA of only 53.20% and MCC of 0.156. This suggests that the model predicts almost all applications as granted rather than effectively distinguishing between the two classes.

  2. The paper is poorly organized. Several sections, particularly Sections 3.1 and 3.2, read more like a collection of notes than a proper methodological explanation. Most of the important implementation details are pushed to the appendix, making the main paper difficult to understand on its own.

  3. The dataset collection process is not sufficiently explained. The authors do not clearly report the patent filing/publication year range or collection cutoff. The paper also lacks an IPC/CPC classification breakdown, making it difficult to understand the dataset's coverage and possible selection bias.

  4. The benchmark contains only 1,159 patent applications, with a smaller subset used for the main experiments. More importantly, the coverage across the five dimensions is highly uneven. For example, the US dataset contains only seven eligibility references and 70 inventive-step references. This raises concerns about the statistical reliability of the results.

  5. The proposed EGPA method mainly combines prompt refinement with a supervised predictor, offering limited methodological novelty. The authors also acknowledge that EGPA does not significantly outperform the Naive-review control using the same predictor. This makes the actual contribution of experience-guided refinement unclear.

  6. The paper uses historical grant outcomes as a proxy for patentability. However, applications may be withdrawn or abandoned for reasons unrelated to their patentability. Reducing this complex process to a binary granted/not-granted prediction may not reflect whether the claimed invention is actually patentable.

Questions for authors
  1. How do the authors match each patent claim with the correct Markush image, especially when a patent contains multiple chemical structures or amended claims? Is this matching performed automatically or manually verified?

  2. The benchmark uses five common patentability dimensions for both EPO and USPTO records. However, these jurisdictions have different legal standards, particularly for subject-matter eligibility. How are these differences handled when constructing the reference assessments and evaluating the models?

2 comments
AuthorsOpenReview ↗

Response to Reviewer jsvt (1/2)

W1: Predominantly granted predictions on US cases and method-row attribution

The historical US results identify a grant-biased Naive configuration, distinct from archived Full Ours. The table separates both from an always-granted baseline, showing why accuracy must be read with BA and class recalls. These rows are separate from the expanded current evaluation, where EPA improves BA over Naive DS-Flash despite lower granted recall and accuracy (Section 4.2).

Archived methodAccuracy (%)BA (%)Positive recall (%)Negative recall (%)TP/FN/FP/TN
Always granted69.9450.00100.000.00114/0/49/0
Archived Naive (CLAIM)71.1753.2098.258.16112/2/45/4
Archived Full Ours53.9957.2149.1265.3156/58/17/32

We have updated our manuscript accordingly.

W2: Organization and readability

Thank you for the valuable suggestion. Section 3 now separates task definitions, construction, statistics and expert validation, and our method. Section 4.1 specifies settings and metrics, Section 4.2 discusses the two tasks and expert ratings separately, and Section 4.3 examines prediction errors. We hope that this organization makes the benchmark contribution and evaluation targets clearer.

W3: Collection dates and IPC/CPC coverage

Official application-number queries recovered filing dates, publication dates, and IPC/CPC classifications for the EP collection (Section 3.3; Appendix K.2). Filing, publication, and query dates are distinguished. The query date below does not establish the original collection cutoff, which remains unverified.

FieldRecorded/totalCoverage (%)
Earliest recorded filing date851/851100.00
Earliest recorded publication date851/851100.00
IPC851/851100.00
CPC851/851100.00
SubclassIPC: cases (% of 851)CPC: cases (% of 851)
A61K763 (89.66%)628 (73.80%)
C07D684 (80.38%)701 (82.37%)
A61P672 (78.97%)777 (91.30%)
A01N51 (5.99%)39 (4.58%)
C07C46 (5.41%)64 (7.52%)
C07F42 (4.94%)76 (8.93%)
Date range / collection detailValue
Earliest recorded filing years2000–2024
Earliest recorded publication years2000–2026
Filing-year bands: 2000–2009 / 2010–2019 / 2020–2024143 / 556 / 152
EPO metadata query14 November 2026

W4: Dataset size and uneven reference coverage

Table 2 distinguishes collection and split sizes. Task 1 uses only available dimension-specific references; a missing reference is not a negative assessment. The EP denominators below make uneven coverage explicit, particularly for support, whose score applies to a much smaller subset (Appendices G and K.5).

DimensionCases with valid referencesShare of EP test set (%)
Novelty47995.61
Inventive step48897.41
Support10520.96
Clarity40079.84
Eligibility32464.67

Current collection and split sizes follow Table 2. In the common EP test set, 498 cases have at least one valid reference dimension. Dimension-specific subsets overlap, and scores are averaged equally across dimensions (Section 4.1).

W5: Methodological contribution

The main contribution is the benchmark linking graphical Markush claims and textual restrictions to examination references and recorded outcomes. EPA is a simple baseline, combining training-derived assessment instructions with TF-IDF logistic regression for outcome prediction (Section 3.4). We would like to highlight that this work focues on benchmark construction, rather than developing a novel method.

We have updated our manuscript accordingly.

W6: Interpreting historical outcome labels

Task 2 predicts recorded application status, which is distinct from a definitive patentability judgment on the supplied claim. EP's version-linked objection requirement provides examination evidence without equating procedural termination with substantive refusal (Section 3.2; Appendix K.1). Please refer to the updated manuscript.

AuthorsOpenReview ↗

Response to Reviewer jsvt (2/2)

Q1: Matching claims to Markush structures

Section 3.2 and Figure 2 explain version-first extraction: claim text and drawings come from the document identified in examination records. Formula identifiers, source pages, crop coordinates, cross-page definitions, and exclusions preserve correspondence. Model-based page comparison and construction-stage human feedback support corrections. The supplementary checks below verify file identity and selected-record consistency.

CheckPassed/totalVerified scope
Input-file hashes versus frozen manifest851/851File identity
MarkPatentBench-EP claims versus selected records501/501Input-to-selected-record consistency

Q2: Differences between EPO and USPTO standards

The dimensions provide a shared framework without equating EPO and USPTO standards. References come from the respective examination records, results are evaluated separately, and interpretation follows the applicable jurisdiction—for example, inventive step in EP and non-obviousness in the US (Sections 3.1–3.2).

Reviewer_5zVjOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces MarkPatentBench: 851 EP and 308 US small-molecule applications. Each record pairs claim text and Markush images with source-linked examination documents and examiner reference passages for five dimensions: novelty, inventive step, support, clarity, and patent-eligible subject matter. The authors also propose EGPA. It combines rules learned by reflection on 48 training cases with a TF-IDF logistic-regression predictor over the generated reviews.

Critical strengths
  1. Existing benchmarks do not cover joint reasoning over Markush drawings and claim text across five legal dimensions.

  2. Document and page identifiers, input hashes, and an explicit separation between inputs and references are all provided.

  3. Balanced accuracy is an appropriate choice under class imbalance, and confusion counts are reported.

Critical weaknesses
  1. EGPA is the only method with a classifier trained on 120 labeled applications and a threshold tuned for BA. Every other method is zero-shot. Table 9 shows the predictor, not the reflection, drives the gain: without it EGPA is worse than Naive, and its margin over “Naive + predictor” is not statistically significant.

  2. Every candidate underperforms Naive on the selection split, and every joint score is negative. The chosen A48 ties with others and wins only on tie-breaking. With 30 cases, one example moves BA by 3.3 points

  3. “Not granted” includes withdrawals, such as fee lapses; the flagship example in Figure 1 is one. Granted cases appear to use the narrowed, post-amendment claim, while not-granted cases use the original one, so the outcome may be guessable from claim form. All 56 cases with an unrecorded target carry the negative label.

  4. There is no image ablation, despite the “beyond structure recognition” framing. For Qwen3.8-Max it is unclear whether the multimodal API was used, since the open weights are text-only.

Questions for authors
  1. What fraction of granted cases use the granted or amended claim version, and of not-granted cases the published version?

  2. What are the text-only (no-image) results? For Qwen3.8-Max, was the multimodal API used rather than the text-only open weights?

  3. What are the application years? Please give results stratified by each model’s knowledge cutoff.

  4. What is the variance across repeated runs and across providers for the same backbone?

  5. What similarity do the null baselines achieve (generic template, mismatched reference)?

  6. Which exact model did deepseek-flash return for your calls?

  7. What share of “not granted” outcomes are withdrawals or fee lapses rather than substantive refusals?

  8. What are the results on substantive refusals only?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 5zVj (1/2)

W1: Fair supervision and the contribution of refinement

We agree that EPA's gains over direct inference combine assessment refinement and supervised prediction. Section 3.4 now separates these components. The historical EP controls below match the training split and predictor-learning procedure. EPA has a higher BA point estimate than Naive + predictor, but the paired interval includes zero. The detailed results are as follows:

MethodAccuracy (%)BA (%)
Naive69.8655.80
Naive + predictor67.6662.15
EPA without predictor69.0652.72
EPA with predictor69.2665.14

These are historical single-run controls, separate from the current repeated-run means. Each supervised configuration retains its own fitted predictor and threshold.

EPA − Naive + predictorΔBA (pp)Paired 95% intervalHolm p
Historical MarkPatentBench-EP3.00[−0.44, 6.54]0.092

We have updated our manuscript in Appendix K.3.

W2: Selection-set size and tie-breaking

We expanded the selection set retrospectively, keeping candidates, predictors, thresholds, and the joint criterion fixed. A48_r0 ranks first on the expanded set, while C48_r0 ranks first on the original subset in the same rerun, indicating selection-set sensitivity. Reported test results retain their original frozen selection.

MethodBA (%)ΔBA (pp)Joint rank
Naive50.00—Baseline
A48_r064.00+14.001

We have updated our manuscript in Appendix K.6.

W3: Outcome labels, claim versions, and the 56-case source group

Section 3.2 links claim text and structures to the version identified in examination records. EP negative inclusion additionally requires a substantive objection to that version, with its rationale and source evidence. The extra candidates came from official EPO queries and passed the same screening pipeline; missing metadata concern acquisition categories. Excluding affected test cases yields a similar BA difference, although both intervals include zero (Appendix K.1).

Groupn (positive/negative)Naive + predictor BA (%)EPA BA (%)ΔBA [95% interval] (pp)
Original common group501 (341/160)62.1565.14+3.00 [-0.44, 6.54]
Excluding the originally unrecorded acquisition-category group466 (341/125)59.9162.67+2.76 [-1.28, 6.84]
Acquisition-category auditCases
Originally unrecorded56
Recovered / still unknown8 / 48
Included in the common EP test set35

W4: Image ablation and the Qwen interface

We clarify that Qwen requests include image fields, and image-reading probes support visual-input use in the audited historical configuration. The separate image ablation holds cases, predictors, and thresholds fixed while removing images from both branches. Full inputs improve the point estimates, but the paired BA interval includes zero. The experimental results are shown below:

Input conditionAccuracy (%)BA (%)
Full text and images in both branches70.6666.17
Images removed from both branches68.6664.70
Image audit / effectResult
US Qwen requests with image fields308/308
Image-reading probes passed3/3
Full inputs − no images, ΔBA (pp)1.47 [−1.93, 5.01], paired 95% interval

We have updated our manuscript in Appendix K.3.

Q1: Claim stages in granted and not-granted cases

We identify claims through claim numbers, dates, and version references in examination records, rather than assigning stages from outcome labels. Both groups include amended claims. The incomplete stage categories below are distinct from the document-level version binding used for extraction (Section 3.2; Appendix K.1).

StagePositive, n=341Negative, n=160
Intended for grant87 (25.51%)0 (0.00%)
Amended/request version8 (2.35%)18 (11.25%)
Claim document only23 (6.74%)27 (16.88%)
Unknown stage223 (65.40%)115 (71.88%)

"Claim document only" and "unknown stage" are not interpreted as original-version claims. These metadata do not establish matched claim-stage distributions.

Q2: Text-only results and multimodal calls

The text-only ablation removes images from both branches while retaining cases, predictors, and thresholds (See Appendix K.3). The Qwen request audit covers historical US calls, not every call in the expanded current evaluation.

AuthorsOpenReview ↗

Response to Reviewer 5zVj (2/2)

Q3: Application years and model knowledge cutoffs

We compared recovered publication dates, including known earlier family publications, with the documented model cutoffs. Almost all cases precede these cutoffs, precluding an informative before/after comparison. Publication timing indicates possible availability, not training-set inclusion, and cannot rule out contamination (Appendix K.2).

ModelPublished cutoffBefore cutoff / date knownWithin cutoff month / after (cases)
GPT-5.6 Sol2026-02-16501/501 (100.00%)0 / 0
Gemini 3.1 Pro2025-01 (month precision)499/501 (99.60%)1 / 1

Cutoffs apply to the recorded configurations. Gemini has month-level precision; publications within that month are reported separately.

Q4: Repeated-run variability

Thank you for the constructive suggestions. Table 3 and Table 4 now report repeated-run results, with average results and standard deviations. The supplementary EP controls below also apply frozen predictors to independently generated assessments. EPA with a predictor has the highest mean BA, while the difference from EPA without a predictor underscores the contribution of supervised prediction. The detailed results are as follows:

MethodAccuracy (%)BA (%)
Naive68.46±0.20050.63±0.312
Naive + frozen predictor63.27±4.56561.85±4.015
Fixed RAG68.06±0.59950.50±0.823
Fixed RAG + frozen predictor69.86±1.39763.26±1.303
EPA without predictor67.40±1.55050.51±1.413
EPA with frozen predictor70.26±1.03765.77±1.595

Q5: Generic-template and mismatched-reference controls

Generic-template and mismatched-case controls use the same dimension-specific references and scoring procedure. True EPA pairs score higher, but the small increment over mismatched references indicates substantial within-domain background similarity. This supports a case-specific alignment signal, not a correctness percentage (Appendix K.5). Expert evaluation provides complementary evidence.

Pairing/inputCosine (five-dimension macro mean)ROUGE-L (five-dimension macro mean)
EPA true pairs0.624860.11959
Generic template0.532860.11772
References from other cases0.612100.11715
Generated assessments from other cases0.604550.11672
Paired analysisResult
Valid cases498
Mismatch permutations1,000
True − mismatched-reference cosine0.01276 [0.00789, 0.01774], paired 95% interval

References are fixed within each dimension. These supplementary scores retain their original scale and aggregation and are separate from the repeated-run averages in Table 3.

Q6: The DeepSeek model identifier

Section 4.1 identifies the backbone as DeepSeek V4.1 Flash, abbreviated DS-Flash. Appendix K.4 records the experimental configurations and run groups; the recorded API identifier is deepseek-v4.1-flash.

Q7: Withdrawals, unpaid fees, and substantive refusals

We agree that procedural termination differs from substantive refusal. EP negative inclusion requires an objection to the selected claim version; withdrawal or non-payment alone is insufficient. However, that objection need not cause termination, and some historical records are non-final. Task 2 thus predicts recorded application status, not a definitive legal judgment. Section 3.2 and Appendix K.1 distinguish these records; a fee-specific termination proportion remains unverified.

Historical recorded categoryCasesShare of negatives (%)
Refusal3320.63
Withdrawal10163.13
Pending1710.63
Unknown95.63

We have updated our manuscript accordingly.

Q8: Results on cases with refusal documents

We added document-defined sensitivity analyses retaining all positives and restricting negatives to applications with refusal documents. EPA has a higher BA point estimate than Naive + predictor on both subsets (Appendix K.1).

Negative-case subsetn (positive/negative)Naive + predictor BA (%)EPA BA (%)ΔBA (pp)
Refusal-decision PDF available368 (341/27)72.0475.31+3.26
Grounds-for-refusal document also available362 (341/21)69.6673.98+4.32

Both subsets retain all 341 positives. Membership reflects document availability, not independently verified correspondence between every refusal decision and the supplied claim version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.