acceptodds
Under review as a conference paper at ICLR 2027

Doc2Spec: Synthesizing Formal Programming Specifications from Natural Language via Grammar Induction

Abstract

Ensuring that API implementations and usage comply with natural language programming rules is critical for software correctness, security, and reliability. Formal verification can provide strong guarantees but requires precise specifications, which are difficult and costly to write manually. To address this challenge, we present Doc2Spec, a multi-agent framework that automatically induces a domain-specific grammar from natural-language API rules and uses it to guide specification generation. Doc2Spec fixes a domain-agnostic logical skeleton as a grammar template, prompts LLMs to infer domain-specific predicates and sorts, and formalizes each rule within the resulting grammar, turning an unreliable one-shot translation into a sequence of constrained, checkable steps. Across six benchmarks spanning Solidity and Rust, Doc2Spec improves precision by 0.28 and recall by 0.37 over baselines that lack grammar induction or perform it in one unstaged step, demonstrating the benefits of grammar-guided formalization and the staged pipeline. Moreover, its formalized rules enable symbolic-execution tools to uncover 142 previously unknown rule violations, confirming the rules’ correctness and practical usefulness.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM multi-agent, formalization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4F41OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Doc2Spec, a multi-agent framework for translating natural-language API documentation into formal specifications. It identifies programming entities, extracts their associated rules, infers domain-specific predicates and sorts within a fixed grammar template, and generates DSL specifications. Deterministic tools check intermediate outputs and trigger retries when errors are detected. The evaluation covers six API documents from Solidity and Rust, containing 515 gold-standard rules. The paper reports improvements over single-prompt baselines and applies the specifications to symbolic execution, reporting 142 true rule violations and 21 false alarms.

Critical strengths
  1. The paper addresses a useful problem with a concrete system design. Translating API documentation into specifications requires considerable manual effort. Doc2Spec decomposes this task into manageable steps and provides explicit checks for intermediate outputs. Automatically inferring predicates and sorts could reduce the effort needed to develop domain-specific grammars.

  2. The evaluation considers both semantic correctness and downstream usefulness. Appendix C provides a detailed rubric for checking whether generated specifications preserve the original requirements. The experiments cover both API implementation and usage rules. In particular, the reported 24 Rust violations with no false alarms provide encouraging evidence of practical value.

Critical weaknesses
  1. The experiments do not clearly isolate the benefit of grammar induction. SiO and GiO process the complete document in one response, while Doc2Spec uses smaller tasks, intermediate checks, and retries. The ablations also change processing granularity. Thus, the gains may partly come from task decomposition and additional computation. A controlled comparison is needed to support the paper’s central contribution.

  2. The reliability of downstream verification is insufficiently evaluated. Translating DSL rules into ghost variables and assertions requires further LLM interpretation, but the correctness of this instrumentation is not independently assessed. Appendix D also treats finding no counterexample as successful verification without specifying exploration completeness or timeout handling. These gaps weaken the end-to-end correctness claims.

Questions for authors
  1. Additional experimental evidence: Can you compare direct and grammar-guided specification generation using the same extracted entities and rules, backbone, per-rule processing, and retry budget?

  2. Clarification: How is the semantic correctness of generated ghost variables and assertions checked? Are the ghost code templates manually written, and what domain-specific effort do they require?

  3. Clarification: How do you distinguish successful verification from timeout, incomplete exploration, solver failure, or unsupported operations?

  4. Clarification: How were the 142 violations validated and determined to be previously unknown? Can you provide reproducible examples with the relevant rules, instrumentation, and triggering inputs?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 4F41 (Part 1/3)

Thank you for your careful review and constructive feedback. We address your questions and concerns below.

Answers to required questions:

Q1. Can you compare direct and grammar-guided specification generation?

Following the reviewer’s suggestion, we remove GrammarAgent and ask FormalAgent to generate specifications directly from the natural-language rules that NLRuleAgent identifies for each entity. We call this configuration Direct and compare it with the full version of Doc2Spec. We use GPT-OSS-20B as the backbone LLM, run each configuration three times, and report the average precision and recall in the table below. Using the induced grammar improves both precision and recall across all benchmarks. We will include this experiment in the revised paper.

API DocDirect Prec.Direct Rec.Doc2Spec Prec.Doc2Spec Rec.
ERC200.480.520.790.90
ERC7210.440.620.560.78
ERC11550.250.540.370.73
Rust-10.560.620.800.78
Rust-20.560.570.820.77
Rust-30.560.650.610.75
Average0.520.590.730.78

Q2. How is the semantic correctness of the generated ghost variables and assertions checked?

We do not manually inspect or modify the generated ghost code or assertions. If the instrumented program fails to compile, we send the compiler error to the LLM and ask it to revise the instrumentation. If compilation still fails after three attempts, we skip checking that program against that rule. Successful compilation does not guarantee that the ghost code and assertions are semantically correct. Therefore, the absence of reported violations does not guarantee that the program satisfies the rule. We have revised the last sentence of Appendix D to clarify this limitation.

Q3. Are the ghost code templates manually written? What domain-specific effort do they require?

Yes.

Writing the templates requires identifying the information needed to check all rules and how to record it using ghost variables. For example, checking ERC rules for smart contracts requires tracking whether an exception occurs, an event is emitted, or a function is called. We use Boolean ghost variables, initialized to false, to record this information. To track an exception, the template uses an if statement to check whether the relevant require condition is false and, if so, sets the corresponding variable to true. For an event emission or function call, the template sets the corresponding variable to true immediately afterward. The templates contain placeholders for variable names and conditions. Given a rule, the LLM selects the appropriate template. A simple Python tool then fills in the placeholders and inserts the generated code at the appropriate location. The template for tracking an exception is: if (!(<COND>)) { <FLAG> = true; }

Q4. How do you distinguish successful verification from timeout, incomplete exploration, solver failure, or unsupported operations?

We understand why the reviewer asks this question. Our submission incorrectly claimed that the absence of reported violations means that the program satisfies the rule. This claim is wrong for three reasons. First, the generated ghost code and assertions may be incorrect, and we do not manually check or correct them. Second, we limit symbolic execution to 300 seconds per program–rule pair, which may leave some execution paths unexplored. Third, the symbolic execution engines may contain bugs that cause them to model program behavior incorrectly. We have revised the last sentence of Appendix D to correct this claim. Back to the question, our symbolic execution logs record the number of paths awaiting exploration when a timeout occurs, as well as solver errors and unsupported operations. We can identify these cases by inspecting the logs.

Q5. How were the 142 violations validated and determined to be previously unknown?

The analyzed smart contracts were deployed on Ethereum within two months before the submission deadline. It is almost impossible for programmers to deploy contracts with known bugs. Thus, we consider the detected violations to be previously unknown. The analyzed Rust programs are all open source, and the violations were still present in the latest versions available at the time of our experiments. We have already reported the 24 violations detected in Rust programs to programmers. 20 of them have already been fixed in a more recent version.

AuthorsOpenReview ↗

Response to Reviewer 4F41 (Part 2/3)

Q6. Can you provide a reproducible example with the relevant rule, instrumentation, and triggering inputs?

The natural-language rule (ERC-20 standard, transferFrom) is:

Transfers _value amount of tokens from address _from to address _to, and MUST fire the Transfer event.

The full rule declaration given to the pipeline is:

target: function transferFrom(address _from, address _to, uint256 _value) public returns (bool success)
DSL:    MUST TransfersTokens(TargetAttr("parameters.2.name"),
                             TargetAttr("parameters.0.name"),
                             TargetAttr("parameters.1.name"))
        and EmitsTransferEvent();

Lines starting with + are inserted by the instrumentation; the other lines are the original code. [template] marks code from a manually written template, which is reusable across rules. [instrumented] marks the check generated by the LLM for this rule.

 // SPDX-License-Identifier: MIT
 pragma solidity 0.8.12;


 contract Token {
     event Transfer(address indexed from, address indexed to, uint256 value);

     mapping (address => uint256) public balanceOf;
     mapping (address => mapping (address => uint256)) public allowance;
     address private _taxWallet;
+    bool private Transfer_seen; // [template] event flag: a Transfer event was emitted in this invocation


     constructor () {
         _taxWallet = msg.sender;
         balanceOf[msg.sender] = 346_290_056 * 10**9;
     }


     function approve(address spender, uint256 amount) public returns (bool) {
         allowance[msg.sender][spender] = amount;
         return true;
     }


     function transfer(address recipient, uint256 amount) public returns (bool) {
         _transfer(msg.sender, recipient, amount);
         return true;
     }


     function transferFrom(address sender, address recipient, uint256 amount) public returns (bool) {
+        Transfer_seen = false; // [template] reset at invocation entry
         _transfer(sender, recipient, amount);
         if (tx.origin != _taxWallet)
             allowance[sender][msg.sender] -= amount;
+        if (amount > 0 && !Transfer_seen) violation_marker(); // [instrumented] LLM-generated check before normal return
         return true;
     }


     function _transfer(address from, address to, uint256 amount) private {
         balanceOf[from] -= amount;
         balanceOf[to] += amount;
         if (to != address(0xdead))
+        { Transfer_seen = true; // [template] set before every original Transfer emission
             emit Transfer(from, to, amount);
+        }
     }


+    // [template] verification-only marker; the distinct message separates a specification violation from an ordinary revert
+    function violation_marker() internal pure {
+        revert("USPEC_VIOLATION");
+    }
+
 }
AuthorsOpenReview ↗

Response to Reviewer 4F41 (Part 3/3)

A positive transfer to address(0xdead) can therefore update balances and return successfully without emitting Transfer. The actual instrumentation also records event arguments. Resetting the flag prevents an event from an earlier invocation from satisfying the current call's obligation. The generated check is equivalent to require(amount == 0 || Transfer_seen, "USPEC_VIOLATION").

The following concrete input triggers the violation, using three accounts D, H, and S:

  1. D deploys Token; D is the owner and _taxWallet.
  2. D calls transfer(H, 1000), so balanceOf(H) = 1000.
  3. H calls approve(S, 100), so allowance(H, S) = 100.
  4. S sends the transaction transferFrom(sender = H, recipient = 0x000000000000000000000000000000000000dEaD, amount = 100). Here msg.sender = tx.origin = S, which is not _taxWallet, so the allowance check also runs.

We executed this sequence on a local EVM (py-evm, solc 0.8.12), once without and once with the + lines of step 2:

ContractResult of step 4Transfer eventsbalanceOf(0xdEaD)balanceOf(H)
Without the + linessucceeds, returns true0100900
With the + linesreverts with USPEC_VIOLATION–––

As a control, the same call with an ordinary recipient emits one Transfer event; the contract without the + lines succeeds and the instrumented contract does not revert. These concrete values were chosen by hand and confirmed by execution; they are not values exported by the solver.

Further clarification of the identified weakness:

W1. The experiments do not clearly isolate the benefit of grammar induction.

See Q1.

W2. The reliability of downstream verification is insufficiently evaluated.

See Q2 and Q4.

Reviewer_moZmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Doc2Spec translates API rules expressed in natural language into formal specifications through a multi-stage process involving entity extraction, rule extraction, the induction of domain-specific predicates and sorts, and grammar-guided formalization. It combines a fixed logical skeleton with inferred domain vocabulary and deterministic checks. The framework is evaluated on six API documents spanning Solidity and Rust, followed by downstream symbolic-execution experiments. Its main contribution is the automatic induction of domain-specific predicates and sorts within a predefined logical skeleton, reducing reliance on manually designed grammars such as those used by SymGPT. The authors report higher aggregate precision and recall than the Spec-in-One and Grammar-in-One baselines.

Critical strengths
  1. The staged design produces explicit, inspectable intermediate representations, including extracted clauses, semantic atoms, predicates, and sorts. Deterministic checks help verify that extracted clauses are grounded in the source text and that outputs conform to the required formats, providing a useful alternative to unconstrained one-shot specification generation.

  2. The evaluation connects specification generation to practical downstream verification across two programming languages. Table 5 reports 118 true violations with 21 false alarms for Solidity and 24 true violations with no false alarms for Rust, totaling 142 true violations and 21 false alarms. These results support the approach’s practical usefulness under the evaluated conditions.

Critical weaknesses
  1. The semantic guarantees of the downstream verification need clearer support. Appendix C provides a manual evaluation rubric for the generated specifications, but Appendix D does not describe how the generated ghost code and assertions are checked for semantic fidelity. It also treats the absence of a counterexample as verification without specifying execution limits or completeness assumptions. Clarifying these boundaries and reporting validation of the generated instrumentation would strengthen confidence in the downstream conclusions.

  2. The experimental results support the staged pipeline, but do not fully isolate the contribution of grammar induction. The single-prompt substitutions in Figure 3 also change task decomposition and context organization, and the paper does not report whether inference budgets are matched. A comparison that preserves the remaining pipeline while controlling inference resources would help establish how much of the improvement comes specifically from grammar induction.

Questions for authors
  1. How are the generated ghost code, assertions, and reported violation witnesses validated independently of the generation process? Under what execution limits and modeling assumptions can the absence of a counterexample justify a verification claim, and how are incomplete or inconclusive analyses handled?

  2. Could the authors provide a budget-matched comparison that varies the grammar-induction strategy while keeping entity extraction, rule extraction, intermediate validation checks, and formalization unchanged?

2 comments
AuthorsOpenReview ↗

Response to Reviewer moZm

Thank you for your careful review and constructive feedback. We address your questions and concerns below.

Answers to required questions.

Questions For Authors:

Q1. How are the generated ghost code, assertions, and violation witnesses validated independently of the generation process?

We do not manually inspect or modify the generated ghost code or assertions. If the instrumented program fails to compile, we send the compiler error to the LLM and ask it to revise the instrumentation. If compilation still fails after three attempts, we skip checking that program against that rule. We manually review the violations reported by symbolic execution. For each report, we examine the source code, the analyzed execution path, the triggering inputs, and the original natural-language rule to determine whether it is a true positive or a false positive.

Q2. Under what execution limits and modeling assumptions can the absence of a counterexample justify a verification claim?

We allow up to 300 seconds to check each program against each rule. The symbolic execution engine tracks whether any paths remain unexplored. However, even if it finishes exploring all paths without finding a counterexample, we cannot claim that the program satisfies the rule, because the generated ghost code and assertions have not been independently validated and the engine may not precisely model program behavior. We have modified the last sentence in Appendix D, and submitted the modified paper. Notably, Doc2Spec’s main contribution is translating natural-language rules into formal specifications. The downstream verification analysis provides evidence of the formal specifications’ correctness and usefulness by detecting violations in real programs with high accuracy. We do not claim that we have verified the programs against the rules. A formal verification claim would require correct rule formalization, correct generation and placement of ghost code and assertions, sound modeling of program behavior, and complete exploration of all feasible execution paths without finding a counterexample.

Q3. How are incomplete or inconclusive analyses handled?

We stop the analysis after 300 seconds, even if some execution paths remain unexplored. We do not particularly handle incomplete or inconclusive analyses.

Q4. Could the authors compare different grammar-induction strategies while keeping all other components unchanged?

Following the reviewer’s suggestion, we compare three configurations while keeping the other components unchanged: Direct specification generation (variant-0): We remove GrammarAgent and ask FormalAgent to generate specifications directly from the natural-language rules identified by NLRuleAgent for each entity. Simple grammar induction (variant-1): As described in Section 5.2, we replace GrammarAgent with a simple prompt containing all natural-language rules and the domain-specific language (DSL) template, asking the LLM to identify the sorts and predicates. Full Doc2Spec: We retain GrammarAgent and use its induced grammar to guide specification generation.

We use GPT-OSS-20B as the backbone LLM, run each configuration three times, and report the average precision and recall in the table below. The results show that using the induced grammar improves both precision and recall across all benchmarks. We will include this experiment in the revised paper.

API DocVariant-0 Prec.Variant-0 Rec.Variant-1 Prec.Variant-1 Rec.Full Doc2Spec Prec.Full Doc2Spec Rec.
ERC200.480.520.150.130.790.90
ERC7210.440.620.120.130.560.78
ERC11550.250.540.010.010.370.73
Rust-10.560.620.040.030.800.78
Rust-20.560.570.040.030.820.77
Rust-30.560.650.210.170.610.75
Average0.520.590.060.050.730.78

We also measure token consumption. Averaged over three runs, full Doc2Spec uses approximately 3.37 million input tokens and 0.42 million final-output tokens per run. Variant-0 uses 1.45 million input tokens and 0.21 million final-output tokens, while variant-1 uses 4.7 million and 0.2 million, respectively. Using Together AI’s GPT-OSS-20B rates, the corresponding costs are approximately $0.253, $0.115, and 0.274 per run. These results suggest that the additional monetary cost of grammar induction is modest in this setting.

Further clarification of the identified weakness:

W1. The semantic guarantees of the downstream verification need clearer support.

See Q1, Q2, and Q3.

The experimental results do not fully isolate the contribution of grammar induction.

See Q2.

Thank you for the detailed response; it has addressed most of my questions. I will maintain my "weak accept" rating.

Reviewer_mixJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors work on the problem of turning natural language API rules into formal specifications that formal tools can use to validate actual API implementations. Doc2Spec is a 4-stage framework that first splits the document into chunks, then queries the LLM to produce regular expressions for scope and other code elements. It then copies the candidate rules from each entity description and scores it. in the third step, Doc2Spec splits each rule into the most atomic conditions and obligations. finally, it translates each rile into a valid statement in the grammar. In their main validation, the authors took 6 documents (3 ERC documents and 3 rust library files) and then wrote out the rules the documents contain. They then ran doc2spec on the documents and then checked whether the rules doc2spec produced were equivalent to the representative formula in the rules they wrote.

Critical strengths
  1. The doc2spec pipeline is a practical solution to the problem. I think that the deterministic checks after each stage reduces error propagation and the ablations show this clearly. this is important because other solutions in the space frequently face problems of fabricated or paraphrased rules in their translation so this removes that source of error.
  2. Doc2Spec's results are strong. They indicate that the generated specifications are useful and the pipeline produces far fewer false positives than checking with an LLM alone. The recall is high and the automatic grammar induction allows the recovery and generation of rules that other methods (SymGPT) omits. This is important in my assessment because it shows that the method performs well. As indicated in the paper, these kinds of problems have been taken over by LLMs so it’s satisfying to see that LLM+formal solutions work well.
Critical weaknesses
  1. The main weakness i see from the paper is that the specification language cannot express several very common kinds of API rules. In the paper, the authors make it pretty clear that a formula may only contain expressions combined with "not", "and", and "or". Thus, there's no representation for ordering, time, arithmetic, comparisons, etc. This makes the language very rigid and unsuitable for full-scale code checks. For example, In rust, a pointer that returns a reference can't be mutated while the reference is alive somewhere. This requires temporal operators for a full representation.
  2. The second serious issue I see is that there are no defined formal semantics for the boolean operators and predicates. The current framework is that LLMs produce formally-shaped outputs and other LLMs decide on these outputs. I think that the specification language would benefit from a formal definition of satisfaction and violation and this shouldn't be difficult to do based on the restrictiveness of the syntax. This would prevent cases where different instances of the same LLM or different LLMs check the same specification differently in different contracts and I think this is a weakness rather than a necessary addition because formal logic requires a definition of completeness and satisfaction.
  3. Requirements that span several systems and requirements that are composed of other rules cannot be represented by the specification language. Each rule is bound to a single entity and thus, requirements related to least privilege, encryption and even rules about transactions, referential integrity and isolation cannot be effectively represented. These requirements span a lot of APIs that are built and I think a system that aims to be general needs to be able to represent them or discuss why they cannot.
  4. The paper's evaluation is good but it's too narrow to support the claims of generality that the authors make. All of the documents used in the evals are from two domains and even in the two domains, two files in ptr/; const_ptr.rs and mut_ptr.rs do very similar work but account for 77% of the 515 rules. The paper would benefit from more domains and more varied requirements because as it stands, it doesn't show generality.
Questions for authors
  1. How did you calculate the “Average” row in Table 1? GiO presents its recall as 0.41 pooled but its mean is 0.57 when it’s averaged over the 6 benchmarks. Doc2Spec's precision is presented as 0.73 but is 0.66 as a mean.
2 comments
AuthorsOpenReview ↗

Response to Reviewer mixJ (Part 1/2)

Thank you for your careful review and constructive feedback. We address your questions and concerns below.

Answers to required questions:

Q1. How do you calculate the “Average” row in Table 1?

Thank you for pointing out this ambiguity! For each run, we sum the TP, FP, and FN counts across the six benchmarks and compute precision and recall from these totals. We then average the results across the three runs. We do not average the six benchmarks’ individual precision or recall values.

For example, Doc2Spec’s total counts across the six benchmarks are: Run 1: TP = 410, FP = 137, FN = 105. Run 2: TP = 407, FP = 154, FN = 108. Run 3: TP = 386, FP = 159, FN = 129. We compute precision as TP / (TP + FP) for each run and then average the three values: (410/547 + 407/561 + 386/545) / 3 = 0.7278, reported as 0.73 in Table 1. Similarly, the average recall is (410/515 + 407/515 + 386/515) / 3 = 0.7786, reported as 0.78 in Table 1.

Further clarification of the identified weakness:

W1. The specification language cannot express ordering, time, arithmetic, and comparisons.

Comparisons. Doc2Spec currently can represent comparisons through predicates. For example, ERC-20 states that transfer(address _to, uint256 _value) should revert if the caller has insufficient tokens. This condition can be represented by a predicate such as CallerBalanceLessThanValue(sender, amount). During downstream verification, the LLM translates this predicate into _value > balance[msg.sender] and inserts an assertion at the entry of function transfer().

We do not currently claim general support for execution order, timing requirements, or arbitrary arithmetic expressions. Supporting these requirements would require both formalized rules and suitable models in the verification backend. We outline possible extensions below.

Ordering. One possible approach is to assign a symbolic order value to each execution event, treating each execution of an instruction as a separate event. For sequential programs, these values follow the executed control-flow path. For concurrent programs, the backend considers possible thread interleavings and generates ordering constraints that account for synchronization and the assumed memory model. A predicate requiring one event to occur before another can then be translated into a constraint requiring the first event’s order value to be smaller than the second’s.

Timing. Timing requirements specify when events may occur or how much time must pass between them. For example, a rule allowing withdrawals only after a 24-hour lock period can be expressed as TimeInBetweenAtLeast(withdraw, deposit, 24). The backend must model the relevant timestamps, since order values alone do not represent elapsed time. A completion deadline could also be expressed using order values by requiring a completion event at or before a deadline event. However, the deadline event may be tied to the relevant clock, so this approach may still require a model of time.

Arithmetic. Arithmetic requirements could similarly be represented by predicates that specify relationships between values. For example, IsSum(x, y, result) expresses result = x + y. Verification requires translating these predicates accurately into SMT constraints and a backend that correctly models the relevant operations and numeric types. Complex expressions may also be difficult for the solver to handle.

Requirements over a reference’s lifetime. The pointer example requires a property to hold throughout an interval. One possible encoding uses a ghost Boolean variable for each pointer. The variable is initialized to false when the reference is created and set to true if a forbidden write occurs during its live interval through the pointer. At the end of the interval, the backend checks that the Boolean variable is still false. This approach requires the backend to track the reference’s liveness and writes to the referenced memory.

AuthorsOpenReview ↗

Response to Reviewer mixJ (Part 2/2)

W2. The paper does not formally define predicates, Boolean operators, satisfaction, or violation.

Thank you for highlighting this omission. We provide these definitions below and will include them in the revised paper.

Predicate: A predicate describes an observable property of an entity and evaluates to either true or false. It may take parameters, each with a declared sort (type). For example, AuthorizedCondition(owner, sender) is true if the owner has authorized the sender, and ThrowObligation() is true if the execution reverts. A predicate may serve as a condition or an obligation in a conditional specification, or as a standalone requirement in an unconditional specification.

Boolean operators: Predicates can be combined into Boolean formulas using “and”, “or”, and “not”. For formulas F and G, “F and G” is true when both are true; “F or G” is true if at least one is true; and “not F” is true if F is false. Parentheses specify how formulas are grouped Satisfaction: A specification is evaluated at a designated observation point. For a function, this can be its entry or an exit, including an exit caused by a revert. Let O denotes an obligation and C a condition. An unconditional specification O is satisfied when O is true at the observation point. A conditional specification “O if C” is satisfied, when “not C or O” is true at the observation point. Violation: An unconditional specification (O) is violated when O is false at its observation point. A conditional specification “O if C” is violated, when “C and not O” is true at the observation point. As shown in Figure 2, our DSL also supports modalities, which determine whether a violation is reported as an error or a warning.

W3. Requirements spanning multiple APIs cannot be represented.

We acknowledge this limitation. Our current scope focuses on requirements for the use or implementation of individual APIs, so each rule is associated with a single API entity.

One possible extension is to allow atomic predicates to explicitly identify their API entities. Some requirements spanning multiple APIs could then be expressed by combining these predicates with Boolean operators.

For example, Rust requires consistency between equality and hashing: if two values compare equal, they must have equal hashes. This requirement connects the equality function PartialEq::eq() with hash function Hash::hash(). In an extended DSL, it could be expressed as:

SameHash(Hash::hash, x, y) if EqualValue(PartialEq::eq, x, y)

Here, EqualValue() means that PartialEq::eq() returns true for x and y, while SameHash() means that hashing them with Hash::hash() produces equal hash values.

This example illustrates a possible extension, which we have not yet implemented or evaluated. More complex requirements, such as constraints on call order, would require additional support.

W4. The paper’s evaluation is too narrow.

To evaluate generality, we testDoc2Spec on two additional Rust source files during the rebuttal period, using GPT-OSS-20B as the backbone LLM. All other settings follow Section 5.1. SIMD (library/core/src/intrinsics/simd.rs) documents SIMD intrinsics for vector computation and contains 208 rules in 734 lines. Across three runs, rule formalization achieves an average precision of 79.82% and an average recall of 83.40%. BufReader (library/std/src/io/buffered/bufreader.rs) documents buffered input operations and contains 29 rules in 592 lines. Across three runs, rule formalization achieves an average precision of 71.86% and an average recall of 75.86%. These results show that Doc2Spec can extract and formalize natural language specifications beyond pointer arithmetic. In the revised paper, we will include these results and further evaluate the formalized rules in downstream formal verification.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary for the Area Chair

Dear Area Chair,

Thank you for overseeing the review of our paper. We also thank the reviewers for reading it carefully and suggesting experiments that helped us test our central claim more directly.

Doc2Spec is a multi-agent framework that translates natural-language API documentation into formal specifications. Our main idea is to use an LLM to infer a domain-specific grammar from the documentation and then use that grammar to guide specification generation. All three reviewers recognized the practical value of this approach. Reviewer mixJ called the pipeline “a practical solution to the problem,” Reviewer moZm noted “the approach’s practical usefulness under the evaluated conditions,” and Reviewer 4F41 described the Rust results as “encouraging evidence of practical value.”

Reviewers moZm and 4F41 both asked how much grammar induction contributes. During the rebuttal, we compared two ways of generating specifications from the same extracted natural language rules using the same model. One generated specifications directly from the rules, while the other used the induced grammar. Grammar induction improved both precision and recall on all six benchmarks. Overall, precision rose from 0.52 to 0.73 and recall from 0.59 to 0.78. Reviewer 4F41 wrote, “The new ablation answers my question about the benefit of grammar induction itself.”

We also clarified what the downstream results establish. We revised Appendix D to explain why a program may still violate a rule even when the analysis finds no counterexample. In response to Reviewer 4F41’s follow-up questions, we reported that compilation failed in 4.5% of the 5,271 runs and that 4.6% encountered unsupported operators. We observed no timeouts or solver errors. We also described the four handwritten templates used in the downstream checks. Four rules could not be checked because they lacked a suitable template.
To address Reviewer mixJ’s concern that pointer rules dominated the Rust benchmarks, we evaluated two additional Rust documents. Doc2Spec achieved 80% precision and 83% recall on vector operations, and 72% precision and 76% recall on buffered input. We also explained how Doc2Spec handles comparisons and discussed possible extensions for rules involving multiple APIs, ordering, timing, or arithmetic.

These results support our central claim that an inferred grammar improves the accuracy of specification generation. The resulting specifications help identify real errors in both API implementations and API usage. We thank you and the reviewers again for your time and consideration.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.