acceptodds
Withdrawn from ICLR 2027

Intent2Tx: Benchmarking LLMs for Translating Natural Language Intents into Ethereum Transactions

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large Language Models (LLMs) are poised to revolutionize Web3 interfaces, yet current benchmarks fail to bridge the gap between high-level user intents and functionally correct on-chain execution. We present Intent2Tx, the first benchmark for intent-to-transaction translation, comprising 30,246 single-step and 1,758 multi-step instances reconstructed from 300 days of real-world Ethereum mainnet activity. Each instance pairs a transaction deterministically decoded from a successful mainnet call with a human-written or LLM-generated natural language intent, grounding language in complex, state-dependent protocol interactions across 11 DeFi categories. To validate functional correctness, we propose an execution-aware evaluation framework that employs differential state analysis on forked mainnet environments, moving beyond brittle surface-level metrics. Our evaluation of 16 state-of-the-art LLMs reveals a ”reasoning-to-execution” gap: even syntactically perfect outputs frequently fail to achieve intended state transitions. Intent2Tx provides a framework for developing verifiable and autonomous agents in intent-centric Web3 ecosystems. Code and data: https://anonymous.4open.science/r/Intent2Tx_Bench-97FF.

Primary area:
datasets and benchmarks
Keywords:
Web3, Decentralized Finance, Ethereum, Large Language Models, Benchmark, Structured Generation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_NjCCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper releases a benchmark pairing natural-language intents with structured Ethereum transactions (contract name and address, function, typed parameters, ETH value). It inclues 30,246 single-step instances decoded from 300 days of mainnet calls plus 325 DAO governance proposals, and 1,758 multi-step instances composed by grouping one address's successive transactions. Intents for the mainnet portion are reverse-generated by an LLM from the decoded transaction. An execution-aware evaluator re-encodes calldata, executes reference and prediction on a forked mainnet one block before the original transaction, and compares account-level state changes within 1%. Sixteen LLMs are evaluated.

Critical strengths

In conclusion, there are three strengths in my opinion:

  1. The resource is substantial and genuinely useful: 30k decoded, schema-typed transactions with verified ABIs, a two-tier taxonomy, and a released execution harness.
  2. Execution-based evaluation is the right direction for this domain, and the static-plus-execution design is sensible in principle.
  3. The study is broad — 16 models, direct and retrieval settings, a LoRA scaling sweep, a strict cross-category split — and the analysis of why fine-tuning hurts on long-bytes payloads (59.1% format failure on instances with a bytes parameter) is an informative failure-mode result.
Critical weaknesses
  1. The benchmark's central construct is compromised by how the intents were generated. Intents are produced by an LLM asked to reverse-engineer the transaction, and Appendix D reports that 74.0% of intents contain at least one raw parameter, 45.7% contain the contract name and 34.7% contain the exact function name (multi-step: 89.5%, 62.4%, 53.0%); the generation prompt explicitly asks for addresses and IDs on niche calls. The input therefore already contains most of the answer, and the task is closer to "reconstruct a schema from a detailed paraphrase" than to "translate a high-level goal into a call". The disclosure statistics are reported as a transparency measure, but they read as a counter-argument to the task definition.
  2. The two evaluation families tell opposite stories and are not reconciled. Static pass@1 is at most 0.06 (single-step, direct) and 0.11 (multi-step), while the same models reach executability 0.20–0.68 and state equivalence 0.51–0.62, and up to 0.744/0.706 with retrieval. If half of all outputs are state-equivalent to the reference, "no model is reliable" is not the right summary; if pass@1 ≈ 0.01 is right, state equivalence is too permissive. The abstract's claim that outputs "frequently fail to achieve intended state transitions" is also not what Table 2 shows: the Exec–StateEq gap is 0.03–0.06 in most rows.
  3. The execution metric is under-specified and may be degenerate. The number of simulated instances is never given, nor which accounts and slots are compared, nor how empty or partial state diffs are treated, nor how the 1% tolerance interacts with a swap routed through a different pool. StateEq ≈ 0.5 in the direct setting sits uneasily beside Slog ≈ 0.44, since state equivalence should be bounded above by routing correctness. The trivial-equivalence rate should be reported.
  4. The retrieval headline rests on an oracle. The main results section is titled "Retrieval augmentation significantly improves performance", but the index is keyed on the gold <contract, function> pair of the test instance. The authors correctly call this an upper bound, but an upper bound computed from the answer cannot support a claim about method performance; the deployable intent-only retrieval variant is missing.
  5. The prior benchmarks are not compared. The related work argues that IntentTx-18K and EVM-QuestBench are unsuitable, but neither is evaluated, re-implemented on overlapping data, or otherwise used to show that this benchmark is harder or measures something the others do not. In that situation "the first benchmark for intent-to-transaction translation" is both unverified and unlikely to be literally true.
  6. Construction choices bias the set toward simple, direct calls. Keeping only successful, top-level atomic calls removes internal calls, which is where router, multicall and aggregator composition lives — precisely the structural heterogeneity the introduction identifies as the main difficulty. Multi-step instances are built by grouping one address's transactions in a short window, which is a user session rather than necessarily one intent, and the joining text is synthetic.
  7. No human upper bound and no ambiguity analysis. Without an expert pass@1 on even 50 instances, a score of 0.06 cannot be interpreted as a model failure rather than an under-specified task. Minor: Table 3 in Appendix B is mislabelled — the "Audited" column reports the rejection counts (28, 27) while the text says 600 and 300 audits.
Questions for authors
  1. What is model performance on intents written before the transaction is seen (human-written goals, or generated without access to parameters)? That is the subset that tests intent understanding.
  2. How many instances were simulated for Table 2, exactly what is compared in the state diff, and what fraction of state-equivalence matches are trivial?
  3. What are pass@1 and state equivalence for an intent-only retrieval index, rather than the oracle-routing one?
  4. How do models perform on the overlapping subset of EVM-QuestBench or IntentTx-18K, and on a 50-instance expert baseline?
  5. What is the gas threshold, and how does including internal calls change the category distribution and difficulty?
Reviewer_n8K8OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a benchmark and studies the performance of some SOTA LLMs on this benchmark. The benchmark is called INTENT2TX that goes from natural language description of what look like tasks/queries to structured text that can be executed. The paper is badly written such that many things are unclear. From what I understood, the following is the summary. This authors took 300 days of Ethereum mainnet activity. The word "intent" is a misnomer because I do not think careful analysis of users' intent was done. Rather than intent, the input seems to be a natural language query/task description. About 30,000 single-step and 1750 multi-step tasks are in the benchmark. The benchmark has been generated using LLMs in the benchmark construction and no evaluation of the accuracy of the methods used are given. Hence, the correctness of the benchmark is unknown rendering the whole work on shaky foundation. They then use this LLM-generated (with some human audits and one small portion human-generated, whose process and accuracy are not largely reported) benchmark to evaluate other SOTA LLMs to see how much their outputs agree with this "benchmark" of unknown correctness.

Critical strengths

The paper has no strengths because the foundation itself is iffy. Anything built on something that may or may not be correct, itself may or may not be correct. It is futile to talk about strengths of such a construct.

Critical weaknesses
  1. We do not know if the benchmark itself is accurate and what its accuracy is.

  2. This is a benchmark that is perhaps interesting to the blockchain/Ethereum community. I did not find any NLP-related novelty. Certainly no novelty with respect to "learning representations", the central focus of this conference. Perhaps this paper, after it is made sound, should be submitted to an application domain.

  3. I do not know how important this task is and what really this is saving. It seems like a text to structure query type task, which if someone builds a good tool is solved. There is nothing fundamentally interesting or different about this problem and while the authors claim some minor points where it is different from existing work, and, this may well be the first work dealing with this problem, I am not sure this has a significant impact in the blockchain field and certainly its impact on the NLP community is non-existant to minimal.

  4. The paper is badly written. This may be the key flaw. I cannot understand ot reproduce the methods because what exactly was done was not provided in the 9 pages. I have not read the appendix - nor should I have to. As the instructions say, the 9 pages should be standalone. There is no running example to show what extraction was done. No prompt example really for us to understand what type of things were done with the LLM. No exact steps/processes of the benchmark construction except at a very high level.

  5. The experiments and the benchmark seems to be constructed from a small sample of data. I do not know how general this is for even the field and whether the observations are specific to some quirks of that platform and in those 300 days or whatever the sample of data that was taken. Why was there not a better sampling done across platforms and across a longer span of time? That would make the benchmark more useful/general.

  6. Some lower granularity points: "LLM applications where the objective is to abstract away, not dictate, the underlying technical steps" claim made without evidence or citation. No example. In some cases the objective may be to abstract and in some cases to find the exact technical steps. "dictate" to whom?

The next line claims there is a pressing need … again without any evidence or built up logic with a running example of what does not work, what needs to be done, etc.

abi, gas, etc.

In phase 3, only 900 samples were audited. We do not know if the other part is junk or not. It is LLM created. Using LLM created stuff to check for accuracy of LLMs is basically unsound for benchmark construction - it is the LLM's opinions, which can be ad hoc and inaccurate, that you will be checking against.

No accuracy numbers are provided for these audits to gain even a semblance of confidence.

Line 260 and nearby: It is unclear who generated the oracle. It seems it is machine generated and thus we do not know if it is accurate or not.

The scaling experiments were stopped at 28.8k even though the gains had not saturated and the curves were growing up in a straight line. Why? Why not go further?

Questions for authors

See the weaknesses and address them. Especially about how I know that the benchmark's entries are acutally correct given it is LLM-generated largely.

Reviewer_UAMuOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper construct a benchmarking dataset to assess the performances of LLMs for translation between natural-language user input into smart contracts. The dataset comes from two different sources: one smaller part comes from real-world scenarios, the other part is generated by an LLM pipeline based on real-world traces in Ethereum. As in EVMQuestBench, the authors split the dataset into two parts: single-step (only one transaction happens) and multi-step (multiple transactions happen) and test the relatively up-to-date models.

Critical strengths

As far as my knowledge goes, the paper proposes the biggest benchmarking dataset up to this day for translation between user intent and smart contract execution. Unlike the closest related works, the dataset is based on real traces over a long period of time. Finally, the paper builds a part of its dataset based on real-world user intents.

Critical weaknesses

I have two major concerns about the paper.

First, I do not see how the dataset created by the authors differs in nature from the closest related work, EVMQuestBench. As this difference is one of the main justification for the paper, this is an issue that the authors should clarify. In more details, the paper claims that EVMQuestBench "provides inputs that resemble explicit, stepwise execution traces rather than goal-oriented intents"; but, as is, it is not possible to know what is meant here. First because I had a look at EVMQuestBench's dataset, and their inputs look a priori human-like and goal-oriented. Second because the author's dataset is not directly available, only the creation pipeline, which makes the comparison hard/not reproducible --- this is also a problem on its own in my opinion, the authors should release the actual dataset you obtained your results on, or at least show some excerpts. The authors should focus more on the comparison with the state of the art, which is not good enough as of now.

Second, I find it hard to draw conclusions from the part of the dataset that is generated by an llm which is given actual executions as inputs. The purpose of the paper is to generate code based on intentions; I feel that doing the converse, i.e. generating intentions from code, defies the purpose since human-like intentions may contain uncertainties, contradictions, or goals expressed in a clumsy way, that seem lost with the way the authors create the inputs. More generally, creating llm input based on llm output looks suspicious to me and may have effects that are hard to correctly interpret. I would expect the authors to put this process into context and refer to other works that investigate such effects or that generate data in such a way; but I could not find such an excerpt.

All in all, I find that the way the dataset was created may be a source of biases and (potentially) lead to a misleading testbench; this would be acceptable if the authors justified properly why they had to do it this way (I acknowledge that obtaining good data may be hard here), but the paper contains way too little discussion about the data quality, alignement with real-world scenarios, and comparison with related works.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.