acceptodds
Under review as a conference paper at ICLR 2027

PatRe: A Full-Stage Office Action and Rebuttal Generation Benchmark for Patent Examination

Abstract

Patent examination is a complex, multi-stage process requiring both technical expertise and legal reasoning, increasingly challenged by rising application volumes. Prior benchmarks predominantly view patent examination as discriminative classification or static extraction, failing to capture its inherently interactive and iterative nature, similar to the peer review and rebuttal process in academic publishing. In this paper, we introduce **PatRe**, the first benchmark that models the full patent examination lifecycle, including Office Action generation and applicant rebuttal. PatRe comprises 480 real-world cases and supports both oracle and retrieval-simulated evaluation settings. Our benchmark reframes patent examination as a dynamic, multi-turn process of justification and response. Extensive experiments across various LLMs reveal critical insights into model performance, including differences between proprietary and open-source models, as well as task asymmetries between examiner analysis and applicant-side rebuttal. These findings highlight both the potential and current limitations of LLMs in modeling complex, real-world legal reasoning and technical novelty judgment in patent examination.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, Patent Examination, Legal Reasoning, Novelty Assessment

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.80 over 5 reviews, above 85% of the others.
Reviewer_eASnOpenReview ↗
Rating
4(+1)
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PatRe is a benchmark for generating patent Office Actions and applicant rebuttals from longitudinal USPTO prosecution records. It contains 480 histories spanning eight IPC sections, with 1,075 Office Action and 596 rebuttal instances, linked to claims, amendments, and cited references. The Office Action task varies access to evidence through no-reference, oracle-reference, and oracle-plus-distractor settings. Evaluation combines decision/statute agreement, lexical overlap, and multidimensional LLM judgments, supplemented by human assessment and multiple judges. The experiments suggest that models are substantially more capable at responding to explicit objections than at discovering examination problems, while professional writing quality exceeds substantive legal and technical soundness.

Critical strengths
  1. A longitudinal benchmark linking examiner actions, applicant responses, claim versions and references, which is the main reason to accept if the data are released.
  2. Well-controlled experiments that separate fluent drafting from substantive reasoning, though the scope and calibration of the human and multi-judge evaluations need clarification.
Critical weaknesses
  1. Dataset availability is the main uncertainty. My rating is 3 (weak accept), conditional on a usable release. The evaluation code is public (https://github.com/AIforIP/PatRe), but I could not find the processed dataset or the reference pool: the data link (https://huggingface.co/IPIntelligence) lists no public datasets at review time. Please provide stable access, release terms, the version behind the reported results, and the preprocessing and scoring configuration.
  2. The evaluation needs qualification. The human study (100 OAs, 100 rebuttals, seven models, three IP-specialist PhD students) supports the task asymmetry, but its rebuttal scores are much lower than the primary judge’s: GPT-5-mini 6.85 versus 9.18, DeepSeek-V3.2 5.05 versus 8.37. The samples differ, so matched-sample calibration is needed. The OA judge treats the historical disposition, statutes and citations as the only correct answer, and the OA and rebuttal prompts use different scoring rules, so raw score gaps do not isolate model capability. Please clarify sampling: 479 allowance notices in 480 trajectories raises the question of whether rejected or abandoned applications are covered. Finally, generation from historical turns and oracle evidence is not closed-loop examination, and the claims should be bounded accordingly.
Questions for authors

Clarifications that could materially affect my assessment:

  1. Can you provide an accessible version of all 480 processed trajectories, the reference pool, document/claim alignments, and the evaluation configuration used for the tables? What are the release terms and version identifiers? If release is pending, what assets can be made available during review and what is the concrete release plan? Dataset availability is central to my weak-accept recommendation.

  2. How were the 480 trajectories sampled and filtered, and how many distinct applications ultimately received allowance, were abandoned, or ended unsuccessfully? Please explain the 479 allowance notices and distinguish intermediate final rejections from application-level outcomes. How representative is the roughly balanced IPC sample of the intended evaluation population?

  3. On the exact human-reviewed samples, how do judge scores and model rankings compare with human scores? Please clarify the unit used for the reported correlations, the handling of legally defensible alternatives to the historical response, and how the differing OA/rebuttal judge criteria affect interpretation of the task gap. Existing analyses or clarifications would be sufficient for rebuttal.

  4. Are the reported multi-turn results conditioned on historical claims, OAs, and rebuttals at each step, or do generated outputs and amendments feed subsequent steps? Please specify which parts of the process and outcome are evaluated, and clarify that the noisy-reference setting tests selection with oracle evidence present rather than end-to-end prior-art search.

Related-work suggestion, distinct from the above acceptance conditions:

Please broaden the discussion of visual patent evidence and future multimodal extensions, such as Intermediate Domain Alignment and Morphology Analogy for Patent-Product Image Retrieval, NeurIPS 2025.

4 comments
AuthorsOpenReview ↗

Response by Authors (Part 1)

Dear Reviewer eASn,

We sincerely appreciate your careful review and constructive feedback. We are grateful for your recognition of PatRe's longitudinal examination trajectories, controlled evaluation settings, and diagnostic findings. We would like to clarify the concerns regarding reproducibility, dataset representativeness, evaluation validity, and benchmark scope, and kindly request your thoughtful reconsideration of our manuscript.

Weakness 1 and Question 1: Dataset Availability and Reproducibility.

Answer: We thank the reviewer for emphasizing the importance of dataset accessibility and reproducibility. We would like to clarify that the link mentioned in the review was not provided by the authors in the submission materials or through the official anonymous review process. To facilitate anonymous access, we provide a dedicated anonymized repository for PatRe: https://anonymous.4open.science/r/PatRe-5313/. We are committed to publicly releasing the benchmark dataset and evaluation code. Additionally, the evaluation prompts and inference configurations are already provided in Appendix E and Section 4.1, respectively, to support reproducibility.

Weakness 2 and Question 2: Dataset Sampling and Representativeness.

Answer: We thank the reviewer for this valuable observation. We agree that the sampling criteria and application-level outcomes could be more clearly presented. PatRe prioritizes complete prosecution histories to capture multi-turn examination processes. The distribution of Notices of Allowance reflects the examination records collected in our dataset, which also contain substantial numbers of intermediate rejections, applicant rebuttals, and claim amendments. Notably, a Final Rejection does not necessarily indicate an ultimately unsuccessful application, as subsequent proceedings may still lead to allowance.

We acknowledge that the distribution of final application outcomes is important for understanding dataset representativeness. Additionally, the relatively balanced IPC coverage is intended to facilitate cross-domain evaluation rather than reflect the overall USPTO application distribution. We will further clarify the sampling criteria, application-level outcomes, and potential limitations in the revised manuscript.

Weakness 2 and Question 3: Evaluation Validity and Task Comparability.

Answer: We thank the reviewer for these thoughtful observations. First, PatRe uses historical USPTO examination records as references rather than assuming they represent the only legally defensible outcomes. This is also why we incorporate LLM-as-a-Judge alongside traditional reference-based metrics, allowing evaluation of the legal and technical soundness of generated outputs beyond exact agreement with historical decisions.

Also we appreciate the reviewer's observation regarding the differences between human and LLM evaluation scores. Our human evaluation involves 100 generated OAs and 100 rebuttals sampled across seven models and all IPC sections, with human and LLM judgments conducted on the same sampled outputs. We agree that scores from the full benchmark and the human-evaluated subset should not be directly compared. We will make this distinction clearer in the revised manuscript. Additionally, our three judge models demonstrate strong inter-judge agreement, with an average Kendall's τ of 0.881.

We acknowledge that OA and rebuttal generation involve different evaluation criteria, including the Wrong Disposition Cap for OA Soundness. We will clarify that cross-task score differences should be interpreted as diagnostic findings rather than direct measures of intrinsic model capability.

AuthorsOpenReview ↗

Response by Authors (Part 2)

Dear Reviewer eASn,

Weakness 2 and Question 4: Multi-turn Evaluation Protocol and Benchmark Scope.

Answer: We thank the reviewer for this valuable suggestion. PatRe uses real longitudinal USPTO prosecution histories to evaluate model performance across different examination stages. Each generation is conditioned on the corresponding historical context, and model-generated outputs are not fed into subsequent stages. The benchmark focuses on evaluating OA and rebuttal generation at individual stages rather than simulating a fully closed-loop examination process.

Additionally, OA-RS evaluates evidence selection and reasoning under noisy reference conditions, where oracle references are included alongside retrieved distractors, rather than end-to-end prior-art retrieval. We agree that these distinctions deserve clearer explanation and will clarify the evaluation scope and limitations in the revised manuscript. We also recognize closed-loop examination simulation and end-to-end retrieval as valuable directions for future work.

Related Work: Multimodal Patent Evidence.

Answer: We thank the reviewer for recommending Intermediate Domain Alignment and Morphology Analogy for Patent-Product Image Retrieval. We agree that visual evidence is important for patent analysis. As PatRe currently focuses on textual prosecution records, we will discuss the relevance of this work and clarify multimodal patent examination as a potential extension. We sincerely thank you again for your constructive suggestions. We believe these clarifications will strengthen the transparency, reproducibility, and appropriate interpretation of PatRe, and we kindly request your thoughtful reconsideration of our submission.


Sincerely thank you again for your detailed and insightful review. We respectfully look forward to your response.

Best Regards,

Authors of #19962

The authors has partly addressed my concerns, so I raise my score to 4

AuthorsOpenReview ↗

Sincerely Thanks for Your Reconsideration

Dear Reviewer eASn,

We sincerely thank you for your thoughtful feedback and for taking the time to carefully reconsider our responses and reassess our work.

We truly appreciate your recognition of our efforts and your willingness to raise the score.

Your constructive comments and valuable suggestions are greatly appreciated and will help us further improve the quality of our work.

Best Regards,

Authors of #19962

Reviewer_wCDgOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript introduces PatRe, a benchmark for generating patent Office Actions and applicant rebuttals from real USPTO prosecution histories. The dataset contains 480 trajectories across eight IPC sections, with 1,075 Office Actions and 596 rebuttals, together with claim revisions and cited references. Office Action generation is evaluated without references, with oracle references, and with oracle references mixed with retrieved distractors. Rebuttal generation uses the examiner's Office Action and associated evidence. The evaluation combines decision and statute alignment, reference citation precision, lexical overlap, LLM-as-a-judge scores, and a human study. Experiments across proprietary and open-source LLMs report substantially higher scores for rebuttal generation than Office Action generation, and reduced citation precision when distractor references are introduced.

Critical strengths
  1. The problem is well motivated. Collecting Office Actions, applicant rebuttals, and claim revisions together is useful for studying patent examination beyond one-shot decision prediction without any to and fro.
  2. The evaluation is broad - it includes different reference conditions, multiple LLM families, LLM-as-a-judge, and a human study.
Critical weaknesses
  1. The prompts include prosecution history, but it is unclear whether model-generated OAs, rebuttals, and amended claims are carried forward in an end-to-end rollout. Performance by turn depth is also not reported, making it difficult to assess how well models handle longer examination histories.
Questions for authors
  1. Are model-generated OAs, rebuttals, and amended claims passed into later rounds, and does each input include the full preceding history? Please report performance by turn depth to show how scores and consistency change over longer histories.
  2. Does the rebuttal judge receive the underlying claims, specification, and prior art? Figure 11 does not explicitly list these inputs, so it is unclear how factual grounding and amendment support are verified.
4 comments
AuthorsOpenReview ↗

Response by Authors

Dear Reviewer wCDg,

We sincerely thank you for your careful review and constructive feedback. We appreciate your recognition of PatRe's research motivation and broad evaluation framework. We would like to clarify the concerns regarding the multi-turn evaluation protocol and rebuttal evaluation.

Weakness 1 and Question 1: Multi-turn Evaluation Protocol and Turn-depth Analysis.

Answer: We thank the reviewer for highlighting the importance of evaluating models across multiple examination rounds.

As described in Sections 3.1 and 3.3 and illustrated in Figure 1, PatRe reconstructs real USPTO prosecution histories and formalizes OA and rebuttal generation at different examination stages. Each generation is conditioned on the corresponding historical context, while model-generated outputs are not carried forward into subsequent rounds. Our benchmark therefore focuses on evaluating generation at different stages of real prosecution histories rather than simulating a fully closed-loop examination process. The results are summarized below.

  • Rebuttal generation performance across examination rounds. | Round | Soundness | Clarity | Construct. | Complete. | Coverage (%) | | :--- | :--- | :--- | :--- | :--- | :--- | | R1 | 5.182 | 7.377 | 5.627 | 5.498 | 53.93 | | R2 | 5.800 | 7.748 | 6.197 | 6.027 | 58.89 | | R3 | 5.760 | 7.698 | 6.160 | 5.990 | 57.83 | | R4 | 5.707 | 7.604 | 6.100 | 5.840 | 56.44 |

  • Office Action generation performance across examination rounds. | Round | Soundness | Clarity | Construct. | Complete. | | :--- | :--- | :--- | :--- | :--- | | R1 | 1.926 | 6.602 | 1.403 | 1.779 | | R2 | 2.304 | 7.061 | 1.747 | 2.562 | | R3 | 2.415 | 7.062 | 1.876 | 2.622 | | R4 | 2.445 | 6.952 | 1.891 | 2.607 |

As shown in Tables above, rebuttal generation exhibits the most noticeable improvements from Round 1 to Round 2, followed by relatively stable performance with slight declines in later rounds. For OA generation, soundness, constructiveness, and completeness generally improve across the first four rounds.

These results provide further insights into model performance across different examination depths. Overall, both tasks show improvements in several evaluation dimensions, with rebuttal generation exhibiting the most noticeable gains in earlier rounds and relatively stable performance thereafter.

We will clarify the historical context provided at each stage and incorporate the turn-depth analysis into the revised manuscript, with additional results reported in the reversion.

Question 2: Rebuttal Judge Inputs and Factual Grounding.

Answer: We thank the reviewer for this valuable observation. As shown in Figure 11, the rebuttal judge receives the corresponding Office Action, ground-truth rebuttal, and generated rebuttal. While the underlying claims, patent specification, and prior-art documents are not provided as separate inputs, relevant technical details, claim amendments, and supporting arguments may be reflected in the examination records.

In particular, the Office Action provides the examiner's objections and supporting reasoning, while the ground-truth rebuttal reflects the applicant's responses and arguments. Together, these records provide meaningful context for assessing the consistency, legal and technical reasoning, and overall quality of generated rebuttals. However, such assessments are based on the information available in the examination records and do not constitute independent verification against the complete underlying documents.

We agree that the evaluation inputs and scope could be more clearly explained, and we will provide further clarification in the revised manuscript.


Sincerely thank you again for your detailed and insightful review. We respectfully look forward to your response.

Best Regards,

Authors of #19962

AuthorsOpenReview ↗

Sincerely Looking Forward to Your Response

Dear Reviewer wCDg,

We hope this message finds you well. We understand that you may have a busy schedule, so we would like to kindly remind you about our responses to your comments. We sincerely appreciate the time and effort you have taken to provide valuable feedback on our manuscript. We deeply value your feedback and believe that our responses address the concerns you raised.

As the discussion phase will close in two days, we would be very grateful to receive any further feedback from you before the deadline. If you find our clarifications satisfactory, we kindly request your thoughtful reconsideration of our submission, and we look forward to your response.

Best Regards,

Authors of #19962

AuthorsOpenReview ↗

Sincerely Looking Forward to Your Response (One Day Remaining and Revision Uploaded)

Dear Reviewer wCDg,

We hope this message finds you well. As the discussion phase will close in one day, we would be very grateful to receive any further feedback from you before the deadline. If you find our clarifications satisfactory, we kindly request your thoughtful reconsideration of our submission, and we look forward to your response.

We would also like to kindly remind you that we have uploaded a revised version of our manuscript to address your concerns.

Best Regards,

Authors of #19962

Thank you for the revised version of the manuscript. With added experiments and clarifications on the multi-turn history-forced evaluation setting and LLM-as-a-judge configurations, the ambiguities are resolved.

Reviewer_krhfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

As one who has studied and worked with patent offices, they face growing backlogs, and prior AI work treats examination as a yes/no prediction or as extracting a label from an existing office action. Real examination is a conversation: the examiner issues an office action citing prior art and statutes, the applicant argues back and amends claims, and the cycle repeats until allowance or final rejection. The authors want a benchmark that captures this whole exchange so that models can be tested as examiners who must find and justify problems, and as applicants who must answer them, under realistic conditions where the relevant prior art may be handed to the model, hidden, or buried among distractors. The author(s) assembled several hundred complete prosecution histories from the USPTO, with every office action, rebuttal, claim version, and cited reference in order, and defined generation tasks for both roles along with automatic and judge-based scoring. Running a range of commercial and open models, they report a consistent asymmetry: models write fluent, well-structured legal prose but reason poorly as examiners, tend to reject applications that should be allowed, misapply the harder statutes, and lose track of the relevant references when distractors are present, while they perform much better when defending a position against a stated objection. The dataset and the role asymmetry are the real contributions. The evaluation design still leaves open whether the reported accuracies beat trivial strategies.

Critical strengths

This paper presents real multi-turn prosecution data with full chronology, which is useful, and the DP/RO/RS decomposition is sensible.

Multi-judge and human agreement checks are present.

Critical weaknesses
  • There are no trivial baselines. Notice of Allowance is 44.6% of OAs; the best decision accuracy is 51–53%. Figure 5 shows GPT-5-mini predicting rejection on 83% of true allowances, so its accuracy comes from the rejection classes. “Proprietary models superior” needs majority-class and random rows.

  • Statute precision without recall rewards citing one statute. Figure 6 has the FN data; report F1!

  • The judge treats one ground-truth OA as “the absolute reference” and caps soundness at 3 on disposition mismatch. But examiners differ; alternative valid examinations are scored as errors. Human soundness scores for every model sit at 2–3/10, so the judge mostly orders models within a compressed band. The 0.619/0.786/0.845 ensemble-vs-human correlations appear to be over model-level means. State the n.

  • The RS pool is five oracle plus five BM25 documents, so distractor pressure is mild, and RCA is already 74% in the oracle-only condition, which suggests the “valid reference” definition (examiner-cited plus applicant-cited) needs scrutiny and more direct evaluation.

  • The dataset records final dispositions, but the rebuttal task never uses them. Whether a rebuttal’s arguments actually led to allowance is the outcome that matters, and it goes unmeasured. Claim amendment (C_t -> C_{t+1}) is formalized and prompted (“REFINED CLAIMS”) but not evaluated.

  • The OA-DP prompt instructs the model to “evaluate statutory rejections,” which plausibly produces the “hyper-critical bias” reported as a finding; Gemini, at 0.62 correct on allowances, suggests that the bias is not universal.

  • Contamination: all patents published after 2024, within the training windows of several evaluated models. I don't see any check on this (e.g., perplexity).

  • Table 8 reports a 200% discussed-to-cited ratio for section C, which signals a parsing problem. “First full-stage benchmark” seems to overclaim relative to PANORAMA’s decision trails.

Questions for authors
  • What are majority-class baselines?

-What is the Statute F1?

  • What is the outcome-based rebuttal metric using final dispositions?

  • Is there a contamination/leakage analysis?

  • What is “valid” in RCA?

4 comments
AuthorsOpenReview ↗

Response by Authors (Part 1)

Dear Reviewer krhf,

We sincerely thank you for your detailed review and feedback. We particularly value your firsthand experience with patent offices and share your concern about growing examination backlogs, which is a key motivation behind PatRe's goal of supporting AI-assisted patent examination. We appreciate your recognition of PatRe's contributions. We also acknowledge your important concerns regarding baseline comparisons, evaluation metrics, human–judge agreement, reference validity, and the scope and reliability of our benchmark. We would like to address these issues and clarify the corresponding evaluation settings and limitations.

Weakness 1 and Question 1: Baseline Comparisons.

Answer: We thank the reviewer for this valuable suggestion. We would like to clarify that PatRe primarily focuses on Office Action and rebuttal generation rather than binary decision classification. Decision Accuracy is one component of our evaluation framework, which also assesses the legal and technical reasoning, statutory correctness, and overall quality of generated examination documents.

Specifically, patent examination involves multiple disposition categories, and our Decision Accuracy is computed by matching the normalized disposition labels extracted from generated and ground-truth Office Actions, rather than reducing all decisions to a binary Allowance-versus-Rejection classification. Therefore, the proportion of Notices of Allowance alone does not directly determine the majority-class baseline in our evaluation setting.

Weakness 2 and Question 2: Statute Recall and F1.

Answer: We thank the reviewer for this helpful suggestion. We agree that Statute Precision alone does not fully capture the completeness of statutory identification, as a model may achieve relatively high precision while omitting relevant statutory grounds.

To provide a more comprehensive evaluation, we additionally calculated Statute Recall and F1 using the same generated Office Actions. The results across all ten evaluated models and three OA settings are presented below.

  • Statute Recall / F1 (%) across OA settings. | Model | OA-DP | OA-RO | OA-RS | | :--- | :--- | :--- | :--- | | GPT-5-mini | 74.12 / 53.58 | 86.09 / 61.37 | 88.32 / 56.36 | | Gemini-2.5-Flash | 58.70 / 52.79 | 57.80 / 60.07 | 62.45 / 57.53 | | DeepSeek-V3.2 | 66.93 / 55.69 | 67.26 / 60.01 | 69.02 / 55.04 | | GPT-4o-mini | 86.48 / 58.01 | 73.28 / 61.06 | 75.72 / 58.09 | | LLaMA3.1-8B-it | 71.05 / 51.15 | 96.36 / 62.16 | 96.62 / 56.78 | | LLaMA3.3-70B-it | 62.88 / 53.12 | 68.86 / 60.81 | 78.00 / 59.77 | | Qwen3.5-9B | 72.40 / 49.65 | 90.15 / 62.27 | 88.75 / 56.46 | | Qwen3.5-27B | 66.81 / 51.64 | 81.71 / 65.57 | 84.88 / 59.89 | | Gemma3-12B-it | 76.28 / 54.20 | 73.53 / 60.08 | 78.98 / 54.32 | | Gemma3-27B-it | 49.91 / 46.12 | 56.46 / 55.74 | 69.45 / 56.29 |

Under the oracle-summary setting, Qwen3.5-27B achieves the highest Statute F1 of 65.57%, with a Precision of 54.76% and a Recall of 81.71%. These results complement the existing Statute Precision evaluation by accounting for both correctly identified and omitted statutory grounds.

We will incorporate these additional metrics into the revised manuscript to provide a more balanced assessment of statutory identification performance.

Weakness 3: Evaluation Validity and Human–Judge Agreement.

Answer: We appreciate the reviewer's comments on evaluation validity and human–LLM agreement.

First, PatRe uses historical USPTO examination records as evaluation references rather than assuming that each recorded decision represents the only legally defensible outcome. As described in Section 3.2, we combine reference-based metrics with multidimensional LLM-as-a-Judge evaluation to assess reasoning quality beyond exact historical agreement.

Regarding the Wrong Disposition Cap, the rule ensures that Soundness accounts for both examination disposition and substantive legal reasoning. It applies only to Soundness when the generated disposition differs from the historical reference; other dimensions are evaluated independently.

Finally, the reported human–LLM correlations (0.619/0.786/0.845) are computed at the instance level using 100 human-evaluated Office Actions (n = 100), with human and LLM judge scores aligned for each instance. We will clarify this in the revised manuscript.

AuthorsOpenReview ↗

Response by Authors (Part 2)

Dear Reviewer krhf,

Weakness 4 and Question 5: Reference Validity and Distractor Settings.

Answer: We thank the reviewer for raising these important questions regarding reference validity and the difficulty of the noisy-reference setting.

First, we would like to clarify that valid references in RCA are defined based on examiner-cited references in the ground-truth Office Actions, rather than a combination of examiner-cited and applicant-cited references. RCA therefore measures the extent to which references cited in generated Office Actions align with those cited by the actual patent examiners. We will make this definition more explicit in the revised manuscript.

Regarding the OA-RS setting, it is designed to evaluate models' ability to identify and utilize relevant evidence when oracle references are mixed with BM25-retrieved distractors. We agree that the current configuration represents a controlled evidence-selection setting rather than a comprehensive evaluation of end-to-end prior-art retrieval.

We also acknowledge that varying the number and difficulty of distractors could provide additional insights into model robustness. We will clarify the scope of the current setting and discuss more challenging reference configurations as potential extensions.

Weakness 5 and Question 3: Rebuttal Outcomes and Claim Amendment Evaluation.

Answer: We appreciate the reviewer's suggestion to consider prosecution outcomes and claim amendments in rebuttal evaluation.

PatRe preserves real prosecution histories, including successive Office Actions, applicant rebuttals, and claim revisions. However, the current benchmark evaluates generated rebuttals at individual examination stages using historical context, rather than conducting a closed-loop examination process.

Consequently, the final disposition of a historical application cannot be directly attributed to a model-generated rebuttal. Our current evaluation focuses on the quality of generated responses, including their legal and technical reasoning, rather than their causal effect on eventual allowance.

We agree that incorporating prosecution outcomes and more explicit claim amendment evaluation would further enrich the benchmark. We will clarify this distinction and highlight these opportunities for future extensions.

Weakness 6: Prompting Bias and Model Behavior.

Answer: We agree that prompt formulation may influence models' examination decisions and should be considered when interpreting rejection tendencies.

The OA-DP task is intended to assess models' ability to identify and justify potential statutory issues without access to oracle references. However, we acknowledge that wording emphasizing statutory rejections may contribute to rejection-oriented responses.

We also agree that the observed behavior should not be generalized uniformly across models, particularly given the differences in allowance recognition highlighted by the reviewer. We will clarify this distinction and moderate the interpretation of the reported hyper-critical tendency.

Weakness 7 and Question 4: Data Contamination and Temporal Coverage.

Answer: We appreciate the reviewer's concern regarding potential data contamination. PatRe prioritizes patents published after 2024 to reduce potential overlap with LLM pretraining data, while also preserving complete prosecution histories. Restricting the dataset to more recent applications could result in incomplete examination trajectories, limiting multi-stage evaluation.

We acknowledge that training-data coverage varies across evaluated models. To provide further transparency, we will clarify the temporal coverage of PatRe in relation to publicly documented model training cutoffs in the revised manuscript.

Weakness 8: Data Quality and Positioning Relative to PANORAMA.

Answer: We thank the reviewer for carefully examining the dataset statistics and for highlighting the relevant work PANORAMA.

Regarding Table 8, the reported 200% ratio for IPC Sec. C resulted from an inadvertent reversal of the numerator and denominator in the calculation. We will carefully recheck the statistics and correct the values in the revised manuscript.

Regarding PANORAMA, we acknowledge its valuable contribution to patent examination research. Unlike PANORAMA, which decomposes patent examination into sequential tasks involving prior-art identification, evidence localization, and novelty/non-obviousness assessment, PatRe focuses on generating complete Office Actions and applicant rebuttals across different stages of real prosecution histories. The two benchmarks therefore address complementary aspects of patent examination, with PatRe emphasizing document-level generation and the examiner–applicant interaction.


Sincerely thank you again for your detailed and insightful review. We respectfully look forward to your response.

Best Regards,

Authors of #19962

AuthorsOpenReview ↗

Sincerely Looking Forward to Your Response

Dear Reviewer krhf,

We hope this message finds you well. We understand that you may have a busy schedule, so we would like to kindly remind you about our responses to your valuable comments.

First, we are truly honored to receive feedback from a reviewer with extensive experience in both patent examination and AI research. We sincerely appreciate your expertise and contributions to the patent examination community. In particular, we highly value your firsthand experience working with patent offices and share your concern about the growing examination backlogs, which is one of the key motivations behind PatRe. Our goal is precisely to explore how AI can better understand and support the complex, real-world process of patent examination.

We sincerely appreciate the time and effort you have devoted to reviewing our manuscript. Your valuable insights have helped us improve our work, and we hope that our detailed responses have satisfactorily addressed your concerns.

As the discussion phase will close in two days, we would be very grateful to receive any further feedback from you before the deadline. If you find our clarifications satisfactory, we would sincerely appreciate your thoughtful reconsideration of our submission.

Best regards,

Authors of #19962

AuthorsOpenReview ↗

Sincerely Looking Forward to Your Response (One Day Remaining and Revision Uploaded)

Dear Reviewer krhf,

We hope this message finds you well. As the discussion phase will close in one day, we would be very grateful to receive any further feedback from you before the deadline. If you find our clarifications satisfactory, we kindly request your thoughtful reconsideration of our submission, and we look forward to your response.

We would also like to kindly remind you that we have uploaded a revised version of our manuscript to address your concerns.

Best Regards,

Authors of #19962

Reviewer_zDQ7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce PatRe, a new benchmark that tackles the full patent examination cycle by simulating both examiner Office Actions and applicant rebuttals across multi-turn interactions. Unlike previous methods, which treat the patent decision process as a binary classification or a claim revision task, PatRe provides a full-stage benchmark for multi-turn generation of OAs and rebuttals. The authors test proprietary and open-source LLMs under direct prompting, oracle, and retrieval-simulated settings. Their experiments show that proprietary models perform better than open-source models on rebuttals. They also show the decoupling between surface language style and internal logic across models.

Critical strengths
  1. Modeling the entire interactive lifecycle of Office Actions and subsequent applicant rebuttals is a novel idea
  2. The paper is well written and is easy to understand.
  3. Their analysis reveals that proprietary models outperform open models on adversarial rebuttals and show clear divergences between surface-level legal style and internal statutory logic.
Critical weaknesses
  1. There is no comparison with the basline methods on any of the metrics to understand if the multi-turn interactions are helping the final decision and the rebuttal generation
  2. Metrics like Statute Precision and RCA are based on the model's predictions and do not give a clear idea of how they compare against the ground truth.
  3. Figure 5 shows that major models like GPT-5-mini, DeepSeek-v3.2, and Llama-3.3-70B-it reject almost everything, making the decisions by these models unreliable
Questions for authors
  1. Can you report RCA and statute accuracy by comparing the model's predictions with the ground truth and report those numbers?
  2. Given that some models reject almost everything, how can we improve these models?
  3. How does PatRe compare with the baseline methods in terms of final decision and rebuttal generation?
  4. Can you do an ablation study to see whether the number of rounds of rebuttal helps the model make better decisions?
5 comments
AuthorsOpenReview ↗

Response by Authors (Part 1)

Dear Reviewer zDQ7,

We sincerely thank you for your careful review and constructive suggestions. We appreciate your recognition of the presentation and writing of our paper, PatRe's novelty and contributions to modeling patent examination processes. We would like to clarify the benchmark's evaluation objectives and address your concerns regarding baseline comparisons, evaluation metrics, model behavior, and multi-turn interactions.

Weakness 1 and Questions 3–4: Generation-Oriented Benchmark, Baseline Comparisons, and Multi-turn Evaluation.

Answer: We thank the reviewer for these valuable suggestions. We would like to clarify that PatRe is a generation-oriented evaluation benchmark rather than a decision classification benchmark or a method designed to improve examination outcomes. Its primary objective is to systematically assess the capabilities and limitations of existing LLMs in generating complete, legally and technically grounded Office Actions and applicant rebuttals across different stages of patent prosecution. Decision Accuracy and statute-level metrics serve as complementary diagnostic measures rather than the primary objectives of the benchmark.

PatRe incorporates real longitudinal prosecution histories to evaluate generation at different stages, rather than introducing a multi-turn interaction method intended to improve final decisions. As described in Sections 3.1 and 3.3, each generation is conditioned on the corresponding historical examination context, and model-generated outputs are not fed into subsequent rounds. Therefore, the current evaluation does not directly measure whether additional rebuttal rounds improve subsequent decisions.

To further examine the effect of examination depth, we conduct a turn-depth analysis of OA and rebuttal generation performance across the first four rounds. The results are summarized below.

  • Rebuttal generation performance across examination rounds. | Round | Soundness | Clarity | Construct. | Complete. | Coverage (%) | | :-------: | :-----------: | :---------: | :------------: | :-----------: | :--------------: | | R1 | 5.182 | 7.377 | 5.627 | 5.498 | 53.93 | | R2 | 5.800 | 7.748 | 6.197 | 6.027 | 58.89 | | R3 | 5.760 | 7.698 | 6.160 | 5.990 | 57.83 | | R4 | 5.707 | 7.604 | 6.100 | 5.840 | 56.44 |

  • Office Action generation performance across examination rounds. | Round | Soundness | Clarity | Construct. | Complete. | | :-------: | :-----------: | :---------: | :------------: | :-----------: | | R1 | 1.926 | 6.602 | 1.403 | 1.779 | | R2 | 2.304 | 7.061 | 1.747 | 2.562 | | R3 | 2.415 | 7.062 | 1.876 | 2.622 | | R4 | 2.445 | 6.952 | 1.891 | 2.607 |

Rebuttal generation shows the most noticeable improvement from Round 1 to Round 2, followed by relatively stable performance in subsequent rounds. For Office Action generation, Accuracy, Constructiveness, and Completeness generally improve across the first four rounds. We will incorporate these turn-depth results into the revised manuscript.

AuthorsOpenReview ↗

Response by Authors (Part 2)

Dear Reviewer zDQ7,

Weakness 2 and Question 1: Ground-Truth Alignment of Evaluation Metrics.

Answer: We thank the reviewer for raising this concern. We would like to clarify that Statute Precision is already computed against the ground-truth Office Action, as defined in Section 3.2 (Eq. 1) of our paper. This metric therefore directly assesses agreement with ground-truth statutory labels, rather than relying solely on model predictions.

Similarly, RCA evaluates the references cited in the generated Office Action against the examiner-cited references extracted from the ground-truth USPTO examination records, rather than relying solely on the model's predictions. The existing RCA results are also reported in Table 11.

To provide a more comprehensive assessment, we additionally calculated Statute Recall/F1 and Citation Recall/F1 against the ground-truth records. The model-level results across the three OA settings are presented below. Since Statute Precision and RCA have already been reported in the paper, we focus here on the additional Recall and F1 metrics.

  • Statute Recall / F1 (%) across OA settings. | Model | OA-DP | OA-RO | OA-RS | | :--- | :--- | :--- | :--- | | GPT-5-mini | 74.12 / 53.58 | 86.09 / 61.37 | 88.32 / 56.36 | | Gemini-2.5-Flash | 58.70 / 52.79 | 57.80 / 60.07 | 62.45 / 57.53 | | DeepSeek-V3.2 | 66.93 / 55.69 | 67.26 / 60.01 | 69.02 / 55.04 | | GPT-4o-mini | 86.48 / 58.01 | 73.28 / 61.06 | 75.72 / 58.09 | | LLaMA3.1-8B-it | 71.05 / 51.15 | 96.36 / 62.16 | 96.62 / 56.78 | | LLaMA3.3-70B-it | 62.88 / 53.12 | 68.86 / 60.81 | 78.00 / 59.77 | | Qwen3.5-9B | 72.40 / 49.65 | 90.15 / 62.27 | 88.75 / 56.46 | | Qwen3.5-27B | 66.81 / 51.64 | 81.71 / 65.57 | 84.88 / 59.89 | | Gemma3-12B-it | 76.28 / 54.20 | 73.53 / 60.08 | 78.98 / 54.32 | | Gemma3-27B-it | 49.91 / 46.12 | 56.46 / 55.74 | 69.45 / 56.29 |

  • Citation Recall / F1 (%) across OA settings. | Model | OA-DP | OA-RO | OA-RS | | :--- | :--- | :--- | :--- | | GPT-5-mini | 7.82 / 13.59 | 57.17 / 59.09 | 52.49 / 47.34 | | Gemini-2.5-Flash | 10.64 / 17.71 | 47.51 / 54.38 | 49.77 / 47.72 | | DeepSeek-V3.2 | 13.75 / 21.84 | 54.04 / 57.51 | 57.42 / 40.34 | | GPT-4o-mini | 5.07 / 9.19 | 41.24 / 50.25 | 40.06 / 42.87 | | LLaMA3.1-8B-it | 18.85 / 27.59 | 43.11 / 49.73 | 44.73 / 44.07 | | LLaMA3.3-70B-it | 6.79 / 12.17 | 38.44 / 48.27 | 48.34 / 45.38 | | Qwen3.5-9B | 17.93 / 26.90 | 47.28 / 51.54 | 48.05 / 41.26 | | Qwen3.5-27B | 17.13 / 26.21 | 46.36 / 52.69 | 44.17 / 39.54 | | Gemma3-12B-it | 12.52 / 20.34 | 35.45 / 45.19 | 30.12 / 34.68 | | Gemma3-27B-it | 12.62 / 20.62 | 38.54 / 47.74 | 35.26 / 39.53 |

We will incorporate these additional results into the revised manuscript.

Weakness 3 and Question 2: Rejection Bias and Model Behavior.

Answer: We thank the reviewer for highlighting the rejection bias observed in several evaluated models. We would like to clarify that PatRe is designed as an evaluation benchmark rather than a method for improving patent examination decisions. Our objective is to systematically assess the capabilities and limitations of existing LLMs in patent examination tasks, rather than to propose or optimize a new examination approach.

In this context, the tendency of certain models to overpredict rejection decisions, as shown in Figure 5, represents an important finding of our evaluation. As explicitly discussed in Finding 2, this systematic rejection bias reflects a limitation in the decision-making capabilities of existing LLMs, rather than a limitation of the benchmark itself. These results also highlight the importance of evaluating model behavior beyond aggregate performance metrics.

We agree that mitigating such biases is an important direction for future research. Potential improvements include more balanced training data, decision calibration, and targeted post-training to improve the models' ability to recognize allowable claims. PatRe provides a standardized evaluation framework for identifying these limitations and assessing whether future approaches can address them. We will clarify this distinction and further discuss the implications of the observed rejection bias in the revised manuscript.


Sincerely thank you again for your detailed and insightful review. We respectfully look forward to your response.

Best Regards,

Authors of #19962

AuthorsOpenReview ↗

Sincerely Looking Forward to Your Response

Dear Reviewer zDQ7,

We hope this message finds you well. We understand that you may have a busy schedule, so we would like to kindly remind you about our responses to your comments. We sincerely appreciate the time and effort you have taken to provide valuable feedback on our manuscript. We deeply value your feedback and believe that our responses address the concerns you raised.

As the discussion phase will close in two days, we would be very grateful to receive any further feedback from you before the deadline. If you find our clarifications satisfactory, we kindly request your thoughtful reconsideration of our submission, and we look forward to your response.

Best Regards,

Authors of #19962

Reviewer_VohwOpenReview ↗
Rating
3
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PatRe, a benchmark for evaluating LLMs across the patent examination lifecycle, including Office Action generation and applicant rebuttal generation. The benchmark contains 480 real-world patent examination trajectories, covering 1,075 Office Actions and 596 rebuttals. For Office Action generation, the authors introduce three evaluation settings: Direct Prompting (OA-DP), Reference-Oracle (OA-RO), and Retrieval-Simulated (OA-RS), which test models under different levels of access to prior art. The paper evaluates several proprietary and open-source LLMs using deterministic metrics and LLM-as-a-Judge scores.

Critical strengths
  1. The paper introduces a benchmark that captures the broader patent examination lifecycle. This is a meaningful extension beyond existing benchmarks.

  2. The three Office Action generation settings (direct prompting, oracle references, and retrieval-simulated references) provide a well-structured way to evaluate how different levels of prior-art information affect model performance.

  3. The evaluation framework considers multiple aspects of generated documents, including decision accuracy, statutory correctness, and legal and technical quality, rather than relying solely on text similarity.

Critical weaknesses
  1. The dataset is relatively small, with only 480 patent examination trajectories. Given the diversity of patent domains and the complexity of patent examination, this may limit the benchmark's representativeness.

  2. The authors provide limited details on how the oracle prior-art references are collected and selected for each examination round. Since these references play an important role in the experimental setup, a clearer explanation of this process would be helpful.

  3. The results in Tables 2 and 3 are quite low, even when the models are provided with oracle references. The authors should discuss whether the low performance is mainly due to task difficulty or to the prompting strategy and experimental design.

  4. The evaluation depends heavily on LLM-as-a-Judge scores, especially for measuring legal and technical soundness. Although the authors include human evaluation and correlation analysis, they only evaluate a relatively small subset of the generated outputs. Given the particularly low soundness scores in Table 3, a more detailed comparison between human and LLM judgments across different tasks and evaluation dimensions would help establish the reliability of these scores.

Questions for authors
  1. In Section 3.3, the authors mention applying automated filtering to remove noisy or incomplete records. Could the authors provide more details about the filtering criteria and how they identified noisy records? It would also be helpful to know how many records were initially collected and how many were removed during filtering.

  2. Figure 2(a) shows a nearly uniform distribution across the eight IPC sections. Could the authors provide a more detailed breakdown at the IPC class or subclass level? Also, was this balanced distribution achieved through a specific sampling strategy?

  3. Appendix A mentions that the patents were published after 2024. What was the reason for selecting this particular time period? It would also be helpful to provide the exact years covered by the dataset, along with the distribution of filing and examination years, to better understand its temporal coverage and representativeness.

4 comments
AuthorsOpenReview ↗

Response by Authors (Part 1)

Dear Reviewer Vohw,

We sincerely appreciate the time and effort you have invested in reviewing our paper and providing constructive feedback. We are grateful for your positive recognition of the presentation of our paper, the benchmark’s coverage of the broader patent examination lifecycle, the well-structured Office Action evaluation settings, and the comprehensive evaluation framework encompassing multiple aspects of legal and technical quality.

We would like to provide further clarification on your concerns and questions, and kindly request your thoughtful reconsideration of our manuscript.

Weakness 1 and Questions 1–2: Dataset Construction, Scale, IPC Coverage, and Representativeness.

Answer: We thank you for raising these important concerns regarding dataset scale and technical representativeness. As described in Section 3.3, PatRe focuses on complete prosecution trajectories rather than isolated documents. Its 480 patent records correspond to 1671 examination-level instances, including 1075 Office Actions and 596 applicant rebuttals (also mentioned in your Paper Summary section).

Regarding data construction, our pipeline (Section 3.3) applies automated filtering to exclude incomplete prosecution histories, followed by manual verification by three IP-specialized PhD students to ensure document completeness, consistency, and chronological alignment. We will specify the filtering criteria and exclusion counts in the revised manuscript.

As reported in Section 3.4 and Appendix A, PatRe covers all eight IPC sections (A–H). The relatively balanced distribution across IPC sections is intentional, enabling comparisons across technical domains while reducing the dominance of overrepresented fields. The following table summarizes the technical coverage, spanning 84 classes and 250 subclasses across sections A–H. Since a patent may be assigned multiple IPC subclasses, the subclass counts are non-exclusive and may exceed the number of patents in each section.

IPC SectionClassesSubclassesTop 3 subclasses (count)
A624A61B (44), A61K (35), A61P (33)
B2046B64D (26), B64C (22), B60W (15)
C1020C07K (42), C12N (25), C12Q (24)
D924D06F (30), D10B (10), D03D (6)
E621E21B (21), E04B (13), E04H (6)
F1548F05D (14), F16H (11), F21V (10)
G1237G06F (76), G06V (22), G01N (20)
H630H04L (28), H04N (19), H10P (13)

We will clarify this sampling strategy and include this breakdown in the revised manuscript. We acknowledge that expanding the number of prosecution trajectories, particularly for underrepresented technical subclasses and rare rejection categories, is an important future direction.

Weakness 2: Construction and Selection of Oracle Prior-Art References.

Answer: The oracle prior-art references are obtained directly from the Examiner Cited documents provided by the USPTO in the original prosecution records, as mentioned in Section 3.3. We extract the patent publication numbers listed in these documents and retrieve the corresponding prior-art documents. The references are then associated with their respective Office Actions, ensuring that the oracle reference sets reflect the actual evidence cited during examination.

AuthorsOpenReview ↗

Response by Authors (Part 2)

Dear Reviewer Vohw,

Weakness 3: Interpretation of Low Performance Scores.

Answer: We thank the reviewer for this insightful observation. We agree that the relatively low scores, particularly under the oracle-reference setting, warrant further discussion.

First, the results across OA-DP, OA-RO, and OA-RS provide insight into the role of reference availability. Although access to references can improve certain evaluation dimensions, the improvements are not uniform across models or metrics. This suggests that providing relevant evidence alone does not consistently translate into stronger substantive examination performance.

Even in the oracle-reference setting, models must independently identify relevant claim limitations, establish their correspondence with prior-art disclosures, determine appropriate statutory grounds, and construct legally and technically sound arguments. Thus, OA-RO reduces uncertainty about reference availability but does not eliminate the inherent reasoning challenges of patent examination.

Second, the relatively higher scores in Clarity and Language Style compared with Soundness suggest that models are more capable of producing professionally written Office Actions than ensuring their substantive technical and legal correctness.

We will further clarify the interpretation of these results and discuss the challenges of evidence interpretation and statutory reasoning in the revised manuscript.

Weakness 4: Reliability of LLM-as-a-Judge Evaluation.

Answer: We thank the reviewer for highlighting the importance of validating LLM-as-a-Judge evaluations, particularly for legal and technical soundness. As described in our human evaluation protocol, PatRe includes expert assessments of generated Office Actions and applicant rebuttals, together with correlation analyses between human and automated evaluation scores.

To provide a more detailed analysis, we additionally report soundness correlations for each task. For Soundness, the Pearson correlations are 0.574 for Office Action generation and 0.470 for rebuttal generation, indicating moderate positive associations between human and LLM judgments. Other substantive dimensions also show positive correlations, including Constructiveness for Office Actions (r = 0.635) and Completeness for rebuttals (r = 0.508).

These task-specific results provide additional evidence of agreement between expert and automated assessments of Soundness. We will incorporate these soundness correlation results in the revised manuscript.

Question 3: Temporal Coverage and Selection Criteria.

Answer: We thank the reviewer for raising this question regarding temporal coverage. The selection of patents published after 2024 aims to reduce potential data contamination from LLM pretraining. Meanwhile, we consider the completeness of prosecution histories, as very recent applications may not have undergone sufficient examination rounds, potentially affecting the evaluation of multi-stage interactions.

We agree that the temporal distribution should be made more explicit and will provide the exact year ranges and distributions of filing, publication, and examination dates in the revised manuscript.


Sincerely thank you again for your detailed and insightful review. We respectfully look forward to your response.

Best Regards,

Authors of #19962

Reviewer_VohwOpenReview ↗

Acknowledgement of the rebuttal

Thank you to the authors for the detailed rebuttal and clarifications.

  1. The 1,671 examination-level instances provide a reasonable dataset size. However, the relatively limited representation at the IPC subclass level still raises concerns about generalizability across specialized technical domains.

  2. The number of records before and after noise filtering remains unclear. The exact distribution of filing and examination years would also help establish temporal representativeness.

I appreciate the authors' responses and will maintain my original score.

AuthorsOpenReview ↗

Sincerely Thanks for Your Timely Feedback

Dear Reviewer Vohw,

We sincerely thank you for your timely and valuable feedback.

  1. Regarding the technical domain coverage, our dataset spans all 8 IPC sections, 16 subsections, and 84 classes, providing broad representation across diverse technical domains. We acknowledge that more fine-grained coverage could further improve generalizability. We kindly emphasize that the current distribution provides a reasonable foundation for evaluating patent examination capabilities across different technical fields.

  2. Regarding the temporal distribution, 49.85% of the instances are from 2024, while the remaining 50.15% are from 2025 and 2026. This indicates that our dataset reflects recent patent examination practices.

We would like to provide further clarification on your concerns and questions, and kindly request your thoughtful reconsideration of our manuscript.

Thank you again for your reply!

Best Regards,

Authors of #19962

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revision Uploaded

We sincerely thank all reviewers for their constructive feedback. We have uploaded a revised manuscript with the following key updates:

  • Reviewer Vohw: We clarify the dataset filtering criteria and the resulting 480 complete trajectories (lines 221–225), provide a detailed IPC breakdown covering 84 classes and 250 subclasses (line 231), clarify how oracle references are retrieved from USPTO Examiner Cited documents and associated with their corresponding Office Actions (line 219), and specify the human evaluation protocol and instance-level human–LLM correlation analysis (n = 100) (lines 456–463).

  • Reviewer zDQ7: We clarify the history-conditioned multi-turn evaluation protocol (line 143), extend Statute and Citation metrics with Recall and F1 (lines 189 and 193; Appendix Tables 12–13, lines 736–741), and add performance analyses across four examination rounds for both OA and rebuttal generation (Appendix Tables 14–15, lines 743–751).

  • Reviewer krhf: We clarify PatRe's distinction from PANORAMA (line 99), explain the multi-category Decision Accuracy evaluation (line 185), supplement Statute Recall/F1 results (line 189; Appendix Table 12, lines 736–741), clarify examiner-cited references as the citation ground truth (Appendix Table 13, lines 736–741), and specify the instance-level human–LLM correlation analysis (n = 100) (lines 464–471).

  • Reviewer wCDg: We clarify the history-conditioned, non-closed-loop evaluation protocol (line 143), specify the task-specific LLM-as-a-Judge inputs (line 208), and add turn-depth analyses across four examination rounds (Appendix Tables 14–15, lines 743–751).

  • Reviewer eASn: We clarify the dataset filtering criteria (lines 221–225), explain the intentional IPC balancing for cross-domain evaluation (line 232), specify matched human–LLM evaluation samples and instance-level correlations (lines 456–463), clarify the non-closed-loop evaluation protocol (line 143), and incorporate the suggested work on multimodal patent retrieval (line 1135).


Sincerely thank all reviewers again for your detailed and insightful review. We respectfully look forward to your response.

Best Regards,

Authors of #19962

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.