- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors study patent-drafting agents that cover the complete generation process, from a technical report to claims, a specification, and figures. Their analysis, drafting, and consistency-checking modules share a representation of the invention's problem, insight, solution, and effect (PISE). The authors compare drafting configurations and test revision with and without feedback from QA1, the LLM evaluator that scores the drafts. A separately configured LLM evaluator, QA2, and a patent attorney also assess the Instant drafts before and after revision with QA1 feedback. Both give higher mean scores after revision. The correlations of QA1 and QA2 scores with the attorney's scores vary across drafting configurations.
Critical strengths
S1. External assessments reveal revision gains and weaknesses in the judges. Both QA2 and the attorney give Instant drafts higher mean scores after QA1-guided revision than before it, providing external evidence of improvement. Their scores are not reported for revision without QA1 feedback, so this evidence does not establish the added benefit of that feedback.
The authors examine the correlation of automatic and human evaluation separately for agent, pro, and instant drafts. Although this analysis does not establish that either LLM judge can replace professional assessment, it shows where the judges track the attorney's scores poorly, weaknesses that a correlation across all three configurations would hide.
Critical weaknesses
W1. The experiments leave the contributions of PISE, individual skills, and the evaluators unclear. AutoPatent already combines planning, specialized writing agents, and feedback from an examiner agent. In my reading, this empirical study of PISE and reusable patent skills can be illuminating even absent a new algorithm. The authors acknowledge, however, that they do not test PISE or individual skills separately, so their experiments cannot show which components are needed for the reported gains.
PatentScore evaluates several dimensions of patent-claim quality through LLM judgments and compares its scores with those of domain experts. The present paper discusses both PatentScore and PatClaimEval, but does not compare either with QA1/QA2 on the same drafts using shared criteria for claim quality. Do QA1/QA2 scores agree more closely with expert assessments of claim quality than these methods' scores do? Please clarify which comparisons are possible, given what each evaluator assesses and whether it requires reference claims.
W2. Professional validation relies on one attorney, and some evaluation materials remain unavailable. The authors acknowledge that the study is preliminary, but with only one attorney providing human scores, it cannot measure inter-annotator agreement among practitioners or establish whether other attorneys would give similar assessments.
Across the agent, pro, and instant drafts together, QA1's and QA2's overall scores correlate more strongly with each other than with the attorney's, whereas for agent drafts alone, both judges' scores have near-zero Pearson correlations with the attorney's (Figure 7). The pooled results therefore do not establish that either judge reliably ranks agent drafts in the same order as this attorney, and the sample of nine inventions provides too little evidence to extend that finding to other inventions. Agreement in score values is a separate question: Pearson correlation measures the strength of a relation between two variables, not the agreement between them. In Table 5, both judges give higher scores than the attorney on average, and QA2's scores are farther from the attorney's on average than QA1's.
The disclosed procedure and rubric explain what aspect of quality is being evaluated and how, but do not supply all the instructions needed to reproduce the LLM assessments: the authors state that "Proprietary system instructions and skill internals are not reproduced". It is also unclear whether QA2's instructions were fixed before the attorney's scores were examined. For revised Instant drafts, the reported attorney scores are limited to means, leaving readers unable to check how the improvement in those scores varies across inventions.
Questions for authors
- What materials and scores could the attorney see during evaluation (W2)? Please specify the order in which items were presented, the instructions, questions or prompts the attorney received, and whether the drafting configuration was concealed. Were QA2's instructions fixed before the attorney's scores were examined?
- Please share any existing results comparing how closely QA1/QA2, PatentScore, and PatClaimEval scores agree with expert judgments on the same drafts and overlapping criteria for claim quality (W1). If differences in required inputs or evaluation criteria prevent this comparison, please explain them. Correlations from different datasets would not establish which evaluator agrees more closely with experts.
- For QA1/QA2, please provide the full set of instructions and existing scores for individual drafts, so readers can assess agreement between each evaluator's scores and the attorney's within each drafting configuration (W2). If QA2 or the attorney scored drafts revised without QA1 feedback, please share those scores alongside the scores for drafts revised with QA1 feedback. This would show whether the advantage found by QA1 also appears in either external assessment. These requests are limited to existing materials and results. Evaluation by several attorneys would be valuable future work rather than a requirement for the rebuttal.
3 comments
Additional experiment: isolating the contribution of PISE
We appreciate the reviewer's suggestion to evaluate the individual contributions of PISE and specialized patent-drafting skills. We have completed a targeted ablation of PISE.
We compared our GPT-5.6 Medium agent with and without PISE. Both configurations preserve the professional skill-orchestration framework, including the Insight Builder, Claim-Tree Builder, Invention Knowledge Dossier, Patent Figure Generator, and other specialized drafting capabilities. Only the PISE-specific intermediate representation and workflow were removed.
This design allows us to investigate PISE's incremental contribution while retaining the agent's professional drafting capabilities.
Table R2. PISE ablation: quality improvement with PISE (1–10 scale).
| Quality dimension | QA1 gain | QA2 gain |
|---|---|---|
| Prosecution Resilience | -0.08 | +0.36 |
| Claim Strength | +0.09 | +0.31 |
| Disclosure Strength | -0.06 | +0.23 |
| Figure Quality | +0.76 | +0.99 |
| Filing Readiness | +0.32 | +1.06 |
| Overall | +0.21 | +0.59 |
The agent with PISE achieves a mean score of 6.13 versus 5.92 under QA1, and 6.60 versus 6.01 under independent QA2.
The QA2 results are particularly encouraging: PISE improves the mean score in all five dimensions, with the largest benefits in Filing Readiness (+1.06) and Figure Quality (+0.99). These dimensions reflect the coordination and consistency of patent-package components.
The nine-invention paired analysis yields p=0.207 under QA1 and p=0.059 under QA2. It provides an encouraging, consistent directional result across all five quality axes.
The performance of the no-PISE configuration also demonstrates the effectiveness of the underlying professional skill orchestration. PISE provides an additional structured invention representation that can improve coordination across these existing capabilities.
We will add this ablation, the per-invention scores, and the associated evaluation protocol to the revised manuscript. Together with the previously reported AutoPatent comparison, these results distinguish the role of PISE from the contribution of the broader professional agent architecture.
Independent evaluation, agreement with professional assessments, and reproducibility
W2, Question 1. Blinding, evaluation timing, and information exchange.
The patent attorney was blinded to the drafting configuration. QA1, QA2, and the attorney performed their assessments independently, without exchanging scores, rankings, or feedback.
QA2 was instructed using the same professional rubric as the attorney. The shared rubric establishes common assessment criteria while preserving independent scoring outcomes.
| Evaluation | Information-flow protocol |
|---|---|
| QA1 | Feedback partly used to improve drafts for the revision loop |
| QA2 | Independent assessment; no QA1 feedback or attorney scores supplied |
| Attorney | Blinded assessment; no QA1 or QA2 evaluation results supplied |
| Between evaluators | No exchange of scores or feedback |
W2, Questions 2–3. Agreement with attorney evaluation.
We analyzed 27 identical invention-configuration conditions scored by QA1, QA2, and the attorney.
Table R3. Agreement with blinded attorney evaluations.
| Metric | QA1 vs. attorney | QA2 vs. attorney |
|---|---|---|
| Pearson correlation | 0.598 | 0.603 |
| Spearman correlation | 0.520 | 0.558 |
| Mean absolute error | 1.79 | 2.19 |
| Mean score bias | +1.79 | +2.19 |
Both AI evaluators achieve similar pooled correlations with the attorney. QA1 has lower absolute error in this sample.
We also examined the correlations within each drafting configuration.
| Drafting configuration | QA1–attorney r | QA2–attorney r |
|---|---|---|
| Meta-agent | -0.092 | -0.133 |
| Higher-reasoning generalist | 0.796 | 0.703 |
| Instant generalist | 0.332 | 0.868 |
These results demonstrate substantial configuration-dependent variation. We will report both pooled statistics and individual configuration results, distinguishing correlation from numerical score calibration.
W1/W2. Independent validation of QA-guided revision.
The completed three-evaluator analysis shows that improvements from QA1-guided revision transfer to independent QA2 and attorney evaluations.
Table R4. Quality differences across three evaluators.
| Comparison | QA1 | QA2 | Attorney |
|---|---|---|---|
| QA1-guided revision minus initial draft | +1.28 | +0.58 | +1.58 |
| AgentEx minus QA1-guided revised draft | +0.06 | +0.65 | +0.68 |
The attorney values are aggregate differences. QA2 rates the QA1-revised draft above the initial draft for eight of nine inventions.
The second comparison reveals an evaluator-specific effect. QA1 finds the two selected drafting conditions nearly equivalent, while QA2 and the attorney retain larger preferences for AgentEx.
These observations provide evidence of transferable improvement together with evaluator-specific optimization.
Comparison with generic revision.
Our original controlled experiment compares generic revision and QA1-guided revision under QA1. At round four, QA-guided revision scores +0.38 higher for Instant and +0.45 higher for Extra-High reasoning.
The existing independent QA2 and attorney evaluations cover the initial and QA1-guided revised drafts, but not the corresponding generic-revision outputs.
W1. Comparison with PatentScore and PatClaimEval.
PatentScore and PatClaimEval assess claim-level properties using different inputs, evaluation criteria, and potentially reference claims. Our evaluation addresses complete patent packages, including prosecution resilience, disclosure, figure quality, and filing readiness.
We do not have matched professional evaluations applying all three methods to the same claims under common criteria. Consequently, the correlations reported in their respective studies cannot establish which evaluator agrees more closely with patent attorneys.
Evaluation transparency.
We will expand the experimental documentation with available individual AI scores, evaluator instructions, rubric mappings, and condition-level results. We will also identify the attorney comparisons available only as aggregate scores.
Comparison with AutoPatent and clarification of contributions
We appreciate the reviewer's identification of AutoPatent as closely related work. AutoPatent establishes important mechanisms for patent generation, including specialized writing agents, hierarchical Patent Writing Guideline Tree (PGTree) planning, and Reference-Review-Augmented Generation (RRAG) with examiner feedback. We also acknowledge its published component ablations.
To strengthen the empirical comparison, we implemented an AutoPatent-inspired PGTree/RRAG pipeline as an installable skill within our meta-agent framework. The official AutoPatent repository provides examples and demonstration materials, while its complete agent implementation and training data have not been publicly released. Our implementation reconstructs the published workflow using prompt-based agents.
Both this reconstructed baseline and Vibe Patenting AgentEx used GPT-5.6 with Extra-High reasoning. We evaluated paired nine corresponding inventions, with five independent scoring repetitions per invention and evaluator.
Table R1. Five-axis patent-quality comparison (1–10, higher is better).
Under the five-axis QA1 evaluations, Vibe Patenting achieves a 1.74-point higher mean, with higher scores across all nine inventions. The largest observed differences concern Figure Quality (+3.33) and Disclosure Strength (+1.78).
The two approaches address complementary aspects of patent drafting. AutoPatent focuses on hierarchical planning and iterative refinement of long patent descriptions. Vibe Patenting focuses on coordinating invention understanding and protection strategy across claims, embodiments, specifications, figures, and quality evaluation through a shared PISE representation.
We will expand Related Work to explain these complementary contributions and acknowledge AutoPatent's prior PGTree/RRAG results. We will also make the available evaluation records accessible.
We present this numerical comparison as exploratory system-level evidence.