Intent2Tx: Benchmarking LLMs for Translating Natural Language Intents into Ethereum Transactions

Large Language Models (LLMs) are poised to revolutionize Web3 interfaces, yet current benchmarks fail to bridge the gap between high-level user intents and functionally correct on-chain execution. We present Intent2Tx, the first benchmark for intent-to-transaction translation, comprising 30,246 single-step and 1,758 multi-step instances reconstructed from 300 days of real-world Ethereum mainnet activity. Each instance pairs a transaction deterministically decoded from a successful mainnet call with a human-written or LLM-generated natural language intent, grounding language in complex, state-dependent protocol interactions across 11 DeFi categories. To validate functional correctness, we propose an execution-aware evaluation framework that employs differential state analysis on forked mainnet environments, moving beyond brittle surface-level metrics. Our evaluation of 16 state-of-the-art LLMs reveals a ”reasoning-to-execution” gap: even syntactically perfect outputs frequently fail to achieve intended state transitions. Intent2Tx provides a framework for developing verifiable and autonomous agents in intent-centric Web3 ecosystems. Code and data: https://anonymous.4open.science/r/Intent2Tx_Bench-97FF.