- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents Doc2Spec, a multi-agent framework for translating natural-language API documentation into formal specifications. It identifies programming entities, extracts their associated rules, infers domain-specific predicates and sorts within a fixed grammar template, and generates DSL specifications. Deterministic tools check intermediate outputs and trigger retries when errors are detected. The evaluation covers six API documents from Solidity and Rust, containing 515 gold-standard rules. The paper reports improvements over single-prompt baselines and applies the specifications to symbolic execution, reporting 142 true rule violations and 21 false alarms.
Critical strengths
-
The paper addresses a useful problem with a concrete system design. Translating API documentation into specifications requires considerable manual effort. Doc2Spec decomposes this task into manageable steps and provides explicit checks for intermediate outputs. Automatically inferring predicates and sorts could reduce the effort needed to develop domain-specific grammars.
-
The evaluation considers both semantic correctness and downstream usefulness. Appendix C provides a detailed rubric for checking whether generated specifications preserve the original requirements. The experiments cover both API implementation and usage rules. In particular, the reported 24 Rust violations with no false alarms provide encouraging evidence of practical value.
Critical weaknesses
-
The experiments do not clearly isolate the benefit of grammar induction. SiO and GiO process the complete document in one response, while Doc2Spec uses smaller tasks, intermediate checks, and retries. The ablations also change processing granularity. Thus, the gains may partly come from task decomposition and additional computation. A controlled comparison is needed to support the paper’s central contribution.
-
The reliability of downstream verification is insufficiently evaluated. Translating DSL rules into ghost variables and assertions requires further LLM interpretation, but the correctness of this instrumentation is not independently assessed. Appendix D also treats finding no counterexample as successful verification without specifying exploration completeness or timeout handling. These gaps weaken the end-to-end correctness claims.
Questions for authors
-
Additional experimental evidence: Can you compare direct and grammar-guided specification generation using the same extracted entities and rules, backbone, per-rule processing, and retry budget?
-
Clarification: How is the semantic correctness of generated ghost variables and assertions checked? Are the ghost code templates manually written, and what domain-specific effort do they require?
-
Clarification: How do you distinguish successful verification from timeout, incomplete exploration, solver failure, or unsupported operations?
-
Clarification: How were the 142 violations validated and determined to be previously unknown? Can you provide reproducible examples with the relevant rules, instrumentation, and triggering inputs?
5 comments
Response to Reviewer 4F41 (Part 2/3)
Q6. Can you provide a reproducible example with the relevant rule, instrumentation, and triggering inputs?
The natural-language rule (ERC-20 standard, transferFrom) is:
Transfers
_valueamount of tokens from address_fromto address_to, and MUST fire theTransferevent.
The full rule declaration given to the pipeline is:
target: function transferFrom(address _from, address _to, uint256 _value) public returns (bool success)
DSL: MUST TransfersTokens(TargetAttr("parameters.2.name"),
TargetAttr("parameters.0.name"),
TargetAttr("parameters.1.name"))
and EmitsTransferEvent();
Lines starting with + are inserted by the instrumentation; the other lines are the original code. [template] marks code from a manually written template, which is reusable across rules. [instrumented] marks the check generated by the LLM for this rule.
// SPDX-License-Identifier: MIT
pragma solidity 0.8.12;
contract Token {
event Transfer(address indexed from, address indexed to, uint256 value);
mapping (address => uint256) public balanceOf;
mapping (address => mapping (address => uint256)) public allowance;
address private _taxWallet;
+ bool private Transfer_seen; // [template] event flag: a Transfer event was emitted in this invocation
constructor () {
_taxWallet = msg.sender;
balanceOf[msg.sender] = 346_290_056 * 10**9;
}
function approve(address spender, uint256 amount) public returns (bool) {
allowance[msg.sender][spender] = amount;
return true;
}
function transfer(address recipient, uint256 amount) public returns (bool) {
_transfer(msg.sender, recipient, amount);
return true;
}
function transferFrom(address sender, address recipient, uint256 amount) public returns (bool) {
+ Transfer_seen = false; // [template] reset at invocation entry
_transfer(sender, recipient, amount);
if (tx.origin != _taxWallet)
allowance[sender][msg.sender] -= amount;
+ if (amount > 0 && !Transfer_seen) violation_marker(); // [instrumented] LLM-generated check before normal return
return true;
}
function _transfer(address from, address to, uint256 amount) private {
balanceOf[from] -= amount;
balanceOf[to] += amount;
if (to != address(0xdead))
+ { Transfer_seen = true; // [template] set before every original Transfer emission
emit Transfer(from, to, amount);
+ }
}
+ // [template] verification-only marker; the distinct message separates a specification violation from an ordinary revert
+ function violation_marker() internal pure {
+ revert("USPEC_VIOLATION");
+ }
+
}
Response to Reviewer 4F41 (Part 3/3)
A positive transfer to address(0xdead) can therefore update balances and return successfully without emitting Transfer. The actual instrumentation also records event arguments. Resetting the flag prevents an event from an earlier invocation from satisfying the current call's obligation. The generated check is equivalent to require(amount == 0 || Transfer_seen, "USPEC_VIOLATION").
The following concrete input triggers the violation, using three accounts D, H, and S:
DdeploysToken;Dis the owner and_taxWallet.Dcallstransfer(H, 1000), sobalanceOf(H) = 1000.Hcallsapprove(S, 100), soallowance(H, S) = 100.Ssends the transactiontransferFrom(sender = H, recipient = 0x000000000000000000000000000000000000dEaD, amount = 100). Heremsg.sender = tx.origin = S, which is not_taxWallet, so the allowance check also runs.
We executed this sequence on a local EVM (py-evm, solc 0.8.12), once without and once with the + lines of step 2:
| Contract | Result of step 4 | Transfer events | balanceOf(0xdEaD) | balanceOf(H) |
|---|---|---|---|---|
Without the + lines | succeeds, returns true | 0 | 100 | 900 |
With the + lines | reverts with USPEC_VIOLATION | – | – | – |
As a control, the same call with an ordinary recipient emits one Transfer event; the contract without the + lines succeeds and the instrumented contract does not revert. These concrete values were chosen by hand and confirmed by execution; they are not values exported by the solver.
Further clarification of the identified weakness:
W1. The experiments do not clearly isolate the benefit of grammar induction.
See Q1.
W2. The reliability of downstream verification is insufficiently evaluated.
See Q2 and Q4.
Response to Reviewer 4F41 (Part 1/3)
Thank you for your careful review and constructive feedback. We address your questions and concerns below.
Answers to required questions:
Q1. Can you compare direct and grammar-guided specification generation?
Following the reviewer’s suggestion, we remove GrammarAgent and ask FormalAgent to generate specifications directly from the natural-language rules that NLRuleAgent identifies for each entity. We call this configuration Direct and compare it with the full version of Doc2Spec. We use GPT-OSS-20B as the backbone LLM, run each configuration three times, and report the average precision and recall in the table below. Using the induced grammar improves both precision and recall across all benchmarks. We will include this experiment in the revised paper.
Q2. How is the semantic correctness of the generated ghost variables and assertions checked?
We do not manually inspect or modify the generated ghost code or assertions. If the instrumented program fails to compile, we send the compiler error to the LLM and ask it to revise the instrumentation. If compilation still fails after three attempts, we skip checking that program against that rule. Successful compilation does not guarantee that the ghost code and assertions are semantically correct. Therefore, the absence of reported violations does not guarantee that the program satisfies the rule. We have revised the last sentence of Appendix D to clarify this limitation.
Q3. Are the ghost code templates manually written? What domain-specific effort do they require?
Yes.
Writing the templates requires identifying the information needed to check all rules and how to record it using ghost variables. For example, checking ERC rules for smart contracts requires tracking whether an exception occurs, an event is emitted, or a function is called. We use Boolean ghost variables, initialized to false, to record this information. To track an exception, the template uses an if statement to check whether the relevant require condition is false and, if so, sets the corresponding variable to true. For an event emission or function call, the template sets the corresponding variable to true immediately afterward. The templates contain placeholders for variable names and conditions. Given a rule, the LLM selects the appropriate template. A simple Python tool then fills in the placeholders and inserts the generated code at the appropriate location. The template for tracking an exception is:
if (!(<COND>)) { <FLAG> = true; }Q4. How do you distinguish successful verification from timeout, incomplete exploration, solver failure, or unsupported operations?
We understand why the reviewer asks this question. Our submission incorrectly claimed that the absence of reported violations means that the program satisfies the rule. This claim is wrong for three reasons. First, the generated ghost code and assertions may be incorrect, and we do not manually check or correct them. Second, we limit symbolic execution to 300 seconds per program–rule pair, which may leave some execution paths unexplored. Third, the symbolic execution engines may contain bugs that cause them to model program behavior incorrectly. We have revised the last sentence of Appendix D to correct this claim. Back to the question, our symbolic execution logs record the number of paths awaiting exploration when a timeout occurs, as well as solver errors and unsupported operations. We can identify these cases by inspecting the logs.
Q5. How were the 142 violations validated and determined to be previously unknown?
The analyzed smart contracts were deployed on Ethereum within two months before the submission deadline. It is almost impossible for programmers to deploy contracts with known bugs. Thus, we consider the detected violations to be previously unknown. The analyzed Rust programs are all open source, and the violations were still present in the latest versions available at the time of our experiments. We have already reported the 24 violations detected in Rust programs to programmers. 20 of them have already been fixed in a more recent version.