- Rating
- 4(+1)
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
PatRe is a benchmark for generating patent Office Actions and applicant rebuttals from longitudinal USPTO prosecution records. It contains 480 histories spanning eight IPC sections, with 1,075 Office Action and 596 rebuttal instances, linked to claims, amendments, and cited references. The Office Action task varies access to evidence through no-reference, oracle-reference, and oracle-plus-distractor settings. Evaluation combines decision/statute agreement, lexical overlap, and multidimensional LLM judgments, supplemented by human assessment and multiple judges. The experiments suggest that models are substantially more capable at responding to explicit objections than at discovering examination problems, while professional writing quality exceeds substantive legal and technical soundness.
Critical strengths
- A longitudinal benchmark linking examiner actions, applicant responses, claim versions and references, which is the main reason to accept if the data are released.
- Well-controlled experiments that separate fluent drafting from substantive reasoning, though the scope and calibration of the human and multi-judge evaluations need clarification.
Critical weaknesses
- Dataset availability is the main uncertainty. My rating is 3 (weak accept), conditional on a usable release. The evaluation code is public (https://github.com/AIforIP/PatRe), but I could not find the processed dataset or the reference pool: the data link (https://huggingface.co/IPIntelligence) lists no public datasets at review time. Please provide stable access, release terms, the version behind the reported results, and the preprocessing and scoring configuration.
- The evaluation needs qualification. The human study (100 OAs, 100 rebuttals, seven models, three IP-specialist PhD students) supports the task asymmetry, but its rebuttal scores are much lower than the primary judge’s: GPT-5-mini 6.85 versus 9.18, DeepSeek-V3.2 5.05 versus 8.37. The samples differ, so matched-sample calibration is needed. The OA judge treats the historical disposition, statutes and citations as the only correct answer, and the OA and rebuttal prompts use different scoring rules, so raw score gaps do not isolate model capability. Please clarify sampling: 479 allowance notices in 480 trajectories raises the question of whether rejected or abandoned applications are covered. Finally, generation from historical turns and oracle evidence is not closed-loop examination, and the claims should be bounded accordingly.
Questions for authors
Clarifications that could materially affect my assessment:
-
Can you provide an accessible version of all 480 processed trajectories, the reference pool, document/claim alignments, and the evaluation configuration used for the tables? What are the release terms and version identifiers? If release is pending, what assets can be made available during review and what is the concrete release plan? Dataset availability is central to my weak-accept recommendation.
-
How were the 480 trajectories sampled and filtered, and how many distinct applications ultimately received allowance, were abandoned, or ended unsuccessfully? Please explain the 479 allowance notices and distinguish intermediate final rejections from application-level outcomes. How representative is the roughly balanced IPC sample of the intended evaluation population?
-
On the exact human-reviewed samples, how do judge scores and model rankings compare with human scores? Please clarify the unit used for the reported correlations, the handling of legally defensible alternatives to the historical response, and how the differing OA/rebuttal judge criteria affect interpretation of the task gap. Existing analyses or clarifications would be sufficient for rebuttal.
-
Are the reported multi-turn results conditioned on historical claims, OAs, and rebuttals at each step, or do generated outputs and amendments feed subsequent steps? Please specify which parts of the process and outcome are evaluated, and clarify that the noisy-reference setting tests selection with oracle evidence present rather than end-to-end prior-art search.
Related-work suggestion, distinct from the above acceptance conditions:
Please broaden the discussion of visual patent evidence and future multimodal extensions, such as Intermediate Domain Alignment and Morphology Analogy for Patent-Product Image Retrieval, NeurIPS 2025.
4 comments
Response by Authors (Part 2)
Dear Reviewer eASn,
Weakness 2 and Question 4: Multi-turn Evaluation Protocol and Benchmark Scope.
Answer: We thank the reviewer for this valuable suggestion. PatRe uses real longitudinal USPTO prosecution histories to evaluate model performance across different examination stages. Each generation is conditioned on the corresponding historical context, and model-generated outputs are not fed into subsequent stages. The benchmark focuses on evaluating OA and rebuttal generation at individual stages rather than simulating a fully closed-loop examination process.
Additionally, OA-RS evaluates evidence selection and reasoning under noisy reference conditions, where oracle references are included alongside retrieved distractors, rather than end-to-end prior-art retrieval. We agree that these distinctions deserve clearer explanation and will clarify the evaluation scope and limitations in the revised manuscript. We also recognize closed-loop examination simulation and end-to-end retrieval as valuable directions for future work.
Related Work: Multimodal Patent Evidence.
Answer: We thank the reviewer for recommending Intermediate Domain Alignment and Morphology Analogy for Patent-Product Image Retrieval. We agree that visual evidence is important for patent analysis. As PatRe currently focuses on textual prosecution records, we will discuss the relevance of this work and clarify multimodal patent examination as a potential extension. We sincerely thank you again for your constructive suggestions. We believe these clarifications will strengthen the transparency, reproducibility, and appropriate interpretation of PatRe, and we kindly request your thoughtful reconsideration of our submission.
Sincerely thank you again for your detailed and insightful review. We respectfully look forward to your response.
Best Regards,
Authors of #19962
The authors has partly addressed my concerns, so I raise my score to 4
Sincerely Thanks for Your Reconsideration
Dear Reviewer eASn,
We sincerely thank you for your thoughtful feedback and for taking the time to carefully reconsider our responses and reassess our work.
We truly appreciate your recognition of our efforts and your willingness to raise the score.
Your constructive comments and valuable suggestions are greatly appreciated and will help us further improve the quality of our work.
Best Regards,
Authors of #19962
Response by Authors (Part 1)
Dear Reviewer eASn,
We sincerely appreciate your careful review and constructive feedback. We are grateful for your recognition of PatRe's longitudinal examination trajectories, controlled evaluation settings, and diagnostic findings. We would like to clarify the concerns regarding reproducibility, dataset representativeness, evaluation validity, and benchmark scope, and kindly request your thoughtful reconsideration of our manuscript.
Answer: We thank the reviewer for emphasizing the importance of dataset accessibility and reproducibility. We would like to clarify that the link mentioned in the review was not provided by the authors in the submission materials or through the official anonymous review process. To facilitate anonymous access, we provide a dedicated anonymized repository for PatRe: https://anonymous.4open.science/r/PatRe-5313/. We are committed to publicly releasing the benchmark dataset and evaluation code. Additionally, the evaluation prompts and inference configurations are already provided in Appendix E and Section 4.1, respectively, to support reproducibility.
Answer: We thank the reviewer for this valuable observation. We agree that the sampling criteria and application-level outcomes could be more clearly presented. PatRe prioritizes complete prosecution histories to capture multi-turn examination processes. The distribution of Notices of Allowance reflects the examination records collected in our dataset, which also contain substantial numbers of intermediate rejections, applicant rebuttals, and claim amendments. Notably, a Final Rejection does not necessarily indicate an ultimately unsuccessful application, as subsequent proceedings may still lead to allowance.
We acknowledge that the distribution of final application outcomes is important for understanding dataset representativeness. Additionally, the relatively balanced IPC coverage is intended to facilitate cross-domain evaluation rather than reflect the overall USPTO application distribution. We will further clarify the sampling criteria, application-level outcomes, and potential limitations in the revised manuscript.
Answer: We thank the reviewer for these thoughtful observations. First, PatRe uses historical USPTO examination records as references rather than assuming they represent the only legally defensible outcomes. This is also why we incorporate LLM-as-a-Judge alongside traditional reference-based metrics, allowing evaluation of the legal and technical soundness of generated outputs beyond exact agreement with historical decisions.
Also we appreciate the reviewer's observation regarding the differences between human and LLM evaluation scores. Our human evaluation involves 100 generated OAs and 100 rebuttals sampled across seven models and all IPC sections, with human and LLM judgments conducted on the same sampled outputs. We agree that scores from the full benchmark and the human-evaluated subset should not be directly compared. We will make this distinction clearer in the revised manuscript. Additionally, our three judge models demonstrate strong inter-judge agreement, with an average Kendall's τ of 0.881.
We acknowledge that OA and rebuttal generation involve different evaluation criteria, including the Wrong Disposition Cap for OA Soundness. We will clarify that cross-task score differences should be interpreted as diagnostic findings rather than direct measures of intrinsic model capability.