- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper releases a benchmark pairing natural-language intents with structured Ethereum transactions (contract name and address, function, typed parameters, ETH value). It inclues 30,246 single-step instances decoded from 300 days of mainnet calls plus 325 DAO governance proposals, and 1,758 multi-step instances composed by grouping one address's successive transactions. Intents for the mainnet portion are reverse-generated by an LLM from the decoded transaction. An execution-aware evaluator re-encodes calldata, executes reference and prediction on a forked mainnet one block before the original transaction, and compares account-level state changes within 1%. Sixteen LLMs are evaluated.
Critical strengths
In conclusion, there are three strengths in my opinion:
- The resource is substantial and genuinely useful: 30k decoded, schema-typed transactions with verified ABIs, a two-tier taxonomy, and a released execution harness.
- Execution-based evaluation is the right direction for this domain, and the static-plus-execution design is sensible in principle.
- The study is broad — 16 models, direct and retrieval settings, a LoRA scaling sweep, a strict cross-category split — and the analysis of why fine-tuning hurts on long-bytes payloads (59.1% format failure on instances with a bytes parameter) is an informative failure-mode result.
Critical weaknesses
- The benchmark's central construct is compromised by how the intents were generated. Intents are produced by an LLM asked to reverse-engineer the transaction, and Appendix D reports that 74.0% of intents contain at least one raw parameter, 45.7% contain the contract name and 34.7% contain the exact function name (multi-step: 89.5%, 62.4%, 53.0%); the generation prompt explicitly asks for addresses and IDs on niche calls. The input therefore already contains most of the answer, and the task is closer to "reconstruct a schema from a detailed paraphrase" than to "translate a high-level goal into a call". The disclosure statistics are reported as a transparency measure, but they read as a counter-argument to the task definition.
- The two evaluation families tell opposite stories and are not reconciled. Static pass@1 is at most 0.06 (single-step, direct) and 0.11 (multi-step), while the same models reach executability 0.20–0.68 and state equivalence 0.51–0.62, and up to 0.744/0.706 with retrieval. If half of all outputs are state-equivalent to the reference, "no model is reliable" is not the right summary; if pass@1 ≈ 0.01 is right, state equivalence is too permissive. The abstract's claim that outputs "frequently fail to achieve intended state transitions" is also not what Table 2 shows: the Exec–StateEq gap is 0.03–0.06 in most rows.
- The execution metric is under-specified and may be degenerate. The number of simulated instances is never given, nor which accounts and slots are compared, nor how empty or partial state diffs are treated, nor how the 1% tolerance interacts with a swap routed through a different pool. StateEq ≈ 0.5 in the direct setting sits uneasily beside Slog ≈ 0.44, since state equivalence should be bounded above by routing correctness. The trivial-equivalence rate should be reported.
- The retrieval headline rests on an oracle. The main results section is titled "Retrieval augmentation significantly improves performance", but the index is keyed on the gold <contract, function> pair of the test instance. The authors correctly call this an upper bound, but an upper bound computed from the answer cannot support a claim about method performance; the deployable intent-only retrieval variant is missing.
- The prior benchmarks are not compared. The related work argues that IntentTx-18K and EVM-QuestBench are unsuitable, but neither is evaluated, re-implemented on overlapping data, or otherwise used to show that this benchmark is harder or measures something the others do not. In that situation "the first benchmark for intent-to-transaction translation" is both unverified and unlikely to be literally true.
- Construction choices bias the set toward simple, direct calls. Keeping only successful, top-level atomic calls removes internal calls, which is where router, multicall and aggregator composition lives — precisely the structural heterogeneity the introduction identifies as the main difficulty. Multi-step instances are built by grouping one address's transactions in a short window, which is a user session rather than necessarily one intent, and the joining text is synthetic.
- No human upper bound and no ambiguity analysis. Without an expert pass@1 on even 50 instances, a score of 0.06 cannot be interpreted as a model failure rather than an under-specified task. Minor: Table 3 in Appendix B is mislabelled — the "Audited" column reports the rejection counts (28, 27) while the text says 600 and 300 audits.
Questions for authors
- What is model performance on intents written before the transaction is seen (human-written goals, or generated without access to parameters)? That is the subset that tests intent understanding.
- How many instances were simulated for Table 2, exactly what is compared in the state diff, and what fraction of state-equivalence matches are trivial?
- What are pass@1 and state equivalence for an intent-only retrieval index, rather than the oracle-routing one?
- How do models perform on the overlapping subset of EVM-QuestBench or IntentTx-18K, and on a 50-instance expert baseline?
- What is the gas threshold, and how does including internal calls change the category distribution and difficulty?