- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces ChainQABench, a benchmark of 160 expert-curated questions covering asset flows, exchange attribution, MEV-related interactions, and operational wash-trading patterns on Ethereum. It provides task-specific rulebooks, independently recomputed references, and an evidence hierarchy connecting raw records to decoded actions and analytical units. Eleven LLMs are evaluated under four evidence conditions, with SQL and calculation tools available through the evaluation harness. Decoded actions provide the most consistent improvements, while further grouping and retrieval have mixed effects.
Critical strengths
-
The benchmark addresses concrete analytical questions that require connecting blockchain records across transactions, events, and addresses. The four task families cover complementary aspects of on-chain analysis and provide a useful setting for studying evidence-grounded question answering over complex structured data.
-
The task-specific rulebooks carefully distinguish what the available evidence can establish. In particular, separating structural associations from ownership or intent, and distinguishing insufficient evidence from direct contradiction, makes the evaluation targets more precise and avoids treating every observed transaction pattern as a definitive real-world conclusion.
-
The reference construction includes a separately implemented verifier that recomputes the required fields from frozen sources. Together with provenance links between derived objects and underlying records, this supports checking numerical results and tracing disagreements to their source evidence or adjudication rules.
-
The evaluation covers eleven closed and open-weight models under the same questions and reference answers across four evidence conditions. Reporting both question-level and field-level accuracy, together with worked examples of successes and regressions, provides useful visibility into partial correctness and the different effects of evidence preparation.
Critical weaknesses
-
The higher-level evidence already performs part of the reasoning being evaluated, but this contribution is not systematically separated from the model's remaining work. In Table 6, the analytical unit explicitly supplies the relation between the two outer swaps and an intermediate swap. In Appendix L.3, it directly provides the nine-block span requested by the question. These are useful inputs for studying evidence preparation, but the evaluation should distinguish fields that can be copied from supplied evidence from those requiring calculation or further inference. Without this distinction, the capability measured under each evidence condition remains difficult to interpret.
-
The experiments lack independent programmatic solving baselines and comparisons between representative tool-use harnesses. Although the current harness provides SQL, exact arithmetic, and some archive-query access, the reported comparisons primarily vary the model and evidence condition. The reference-generation program operates on fixed item specifications and therefore does not establish how a practical system would perform from the natural-language question. A programmatic pipeline and a representative code-execution or planning agent, evaluated with comparable evidence and budgets, would help establish where the benchmark presents difficulties beyond decoding, aggregation, and rule execution.
-
The evidence for benchmark coverage and generalization is limited. All 160 questions come from one week of Ethereum activity, with 40 questions per task and only 12 and 8 questions in the arbitrage and anomaly-screening topics, respectively. The paper does not provide a systematic account of shared transactions, addresses, or question templates across items. This makes it difficult to assess how many distinct analytical situations are represented and whether the reported comparisons would remain stable on another time window.
-
The experimental breadth is greater than the depth of the resulting explanation. The paper establishes that decoding often helps and that additional grouping or retrieval has mixed effects, but does not sufficiently identify the causes of these differences. For example, the strongest model's question accuracy decreases from 53.1% under RAW to 51.2% under +UNITS, while Appendix K.6 reports 53 recoveries and 57 regressions from +ACTIONS to +UNITS across the in-context evaluations. These mixed outcomes are acknowledged, but a systematic breakdown of evidence-availability, calculation, rule-application, and output errors is missing. Single runs without uncertainty estimates also limit the interpretation of small differences.
Questions for authors
-
What proportion of required answer fields is explicitly supplied at each evidence layer, particularly L3? Can the results be separated into direct extraction, calculation, and adjudication fields to clarify what reasoning remains for the model?
-
How would a programmatic pipeline and a representative code-execution or planning agent perform when given the same natural-language questions, admissible evidence, and comparable execution budgets? Which difficulties remain after reliable decoding and computation are delegated to tools?
-
How many questions share underlying transactions, addresses, pools, or construction templates? Can the authors provide evidence that the main conclusions are stable under resampling or evaluation on an additional time window?
-
Can the recoveries and regressions across evidence conditions be categorized by their causes, including missing evidence, decoding, arithmetic, rule application, and output completeness? This would help explain why additional analytical structure sometimes reduces performance.
5 comments
Official Comment by Authors (2/3)
[W2, Q2] (continued)
II. Comparison with public agents
We have evaluated the code-execution and planning agents from the public agent framework smolagents [1] with the same questions, evidence, and execution budgets. Both agents use SQL and Python execution. The model is Gemini-3.8-Flash, the best-performing model in the main text, and the evidence condition is +Retrieval. We have also run two conditions with tools removed from our pipeline. Our pipeline provides SQL, an exact arithmetic calculator, archive queries, and 19 tools that aggregate the ledger without a row limit. Because the ledger is inaccessible without tools for the 50 SQL path items, we run the conditions with tools removed on the 110 in-context items.
Table W2-1. Field accuracy (%) and input tokens per item by configuration
| Configuration | All 160 items | 110 in-context items | Input tokens per item |
|---|---|---|---|
| Ours | 76.2 | 79.1 | 321K |
| Code-execution agent | 46.2 | 42.8 | 541K |
| Planning agent | 43.9 | 41.8 | 1,307K |
| Ours w/o aggregation tools and evidence SQL | — | 66.8 | 267K |
| Ours w/o all tools | — | 67.8 | 81K |
Input tokens per item are averaged over the in-context items.
- The public agents perform worse than our pipeline. Across all 160 items, field accuracy is 76.2% versus 46.2% and 43.9%, meaning that the public agents answer fewer than half of the fields correctly, while using 1.7 to 4.1 times as many input tokens per item as our pipeline.
- Our pipeline outperforms the public agents even with all tools removed. On the 110 in-context items, field accuracy is 67.8% versus 42.8% and 41.8%.
- Half of the items remain incorrect even with tools available. With all tools available, in-context field accuracy rises from 67.8% to 79.1%, but question accuracy across all 160 items is 50.0% (Appendix J.9). Because the tools return exact values and exhaustive aggregations, the remaining difficulties lie in determining what to compute and applying the decision criteria in the rulebooks.
We have added the comparison with public agents, the tool-removal conditions, and the full results to Appendix J.9.
[1] Roucher et al., smolagents: a smol library to build great agentic systems, 2025. https://github.com/huggingface/smolagents
[W3, Q3] Sharing across items and evaluation on another time window
We agree with your point that sharing across items and evaluation on another time window should be reported. As requested, we have quantified sharing across items and constructed and evaluated 160 new items from another week. The same construction procedure produces items for the four tasks in another week, and the comparisons between evidence conditions hold on the new 160 items.
I. Sharing across items and item counts by topic
Table W3-1. Number of items confirmed to share with other items in the same task
| Task | Transactions | Addresses | Pools | Templates |
|---|---|---|---|---|
| T1 | 13 | 30 | — | 12 |
| T2 | 14 | 16 | — | 0 |
| T3 | 12 | 18 | 7 | 2 |
| T4 | 12 | 14 | 13 | 2 |
Addresses include tokens and contracts. Only 16 of the 160 items use identical question templates with differences only in identifiers and numerical values. Items share records because the same records can require different analyses. For example, the same swap is used for price calculation, sandwich adjudication, and wash-trading checks.
The 12 arbitrage items and eight anomaly-screening items you mentioned are topics assigned to items, rather than tasks. The topics overlap, and Section 4.3 limits the topic-level results to "overlapping subsets rather than independent tasks". We draw conclusions at the task level, with 40 items per task.
II. Evaluation on another time window
Using the same construction procedure, we have created 160 new items from another week (blocks 25,620,418 to 25,670,641), with 40 items for each of the four tasks. We have evaluated Gemini-3.8-Flash, the best-performing model in the main text, under four conditions using our pipeline.
Table W3-2. Question accuracy / field accuracy (%) by time window
| Condition | Blocks 25,369,414–25,419,597 | Blocks 25,620,418–25,670,641 |
|---|---|---|
| Raw | 53.1 / 77.4 | 47.5 / 76.1 |
| +Actions | 53.1 / 79.3 | 48.8 / 77.5 |
| +Units | 51.2 / 79.4 | 45.0 / 74.4 |
| +Retrieval | 50.0 / 76.2 | 49.4 / 77.0 |
- The benchmark is not tied to a particular week. Applying the same construction procedure to on-chain records from another week produces new items for all four tasks: asset flows, exchange attribution, MEV-related interactions, and AMM wash-trading patterns.
- The reported comparisons hold in the additional time window. Field accuracy increases from Raw → +Actions in both time windows (77.4 → 79.3, 76.1 → 77.5).
We have added the sharing across items to Appendix F.5 and the evaluation on the additional time window to Appendix J.10.
(Continued in Official Comment by Authors (3/3).)
Official Comment by Authors (3/3)
[W4, Q4] Causes of recoveries and regressions across evidence conditions
We agree with your point that the causes of recoveries and regressions should be identified. Following your proposed cause categories, we have classified pairs whose correctness changed using the execution records. Additional analytical structure reduces performance at the rule-application stage, rather than because of missing evidence or arithmetic errors. At the stage of selecting values for calculation, adding analytical units instead yields more recoveries.
I. Classification of recoveries and regressions by cause
For the 110 items that receive evidence in context, we have classified pairs whose correctness changed at each of three transitions: Raw → +Actions, +Actions → +Units, and +Units → +Retrieval. There are 347 pairs in total across 11 models. We have paired the failed and successful runs for the same model and item and compared their answers, cited evidence, and calculation formulas.
Table W4-1. Number of recovery / regression pairs by cause (11 models, 110 in-context items)
| Cause | Raw → +Actions | +Actions → +Units | +Units → +Retrieval |
|---|---|---|---|
| Missing evidence | 0 / 0 | 3 / 0 | 0 / 3 |
| Decoding | 45 / 0 | — | — |
| Value selection | 0 / 21 | 34 / 20 | 25 / 20 |
| Arithmetic | 4 / 2 | 3 / 5 | 5 / 0 |
| Rule application | 13 / 10 | 6 / 20 | 16 / 14 |
| Output completeness | 26 / 6 | 6 / 7 | 7 / 6 |
| Composite-structure fields (undeterminable) | 1 / 3 | 1 / 5 | 9 / 1 |
| Total | 89 / 42 | 53 / 57 | 62 / 44 |
Value selection denotes cases where, under decoded evidence, the ledger values used in the calculation differ from those in the successful run. Rule application denotes cases where a condition specified by the rulebook or the final verdict is incorrect.
- Rule application is where additional analytical structure reduces performance. From +Actions → +Units, rule application accounts for six recoveries and 20 regressions. At the same step, value selection accounts for 34 recoveries and 20 regressions, indicating that analytical units help select the values to use in calculations.
- Missing evidence and arithmetic errors do not explain these regressions. Of the 57 regression pairs from +Actions → +Units, none involve missing evidence and five involve arithmetic.
- From Raw → +Actions, decoding yields more recoveries than regressions. Differences in the ledger values used account for 45 recoveries against 21 regressions.
II. Analysis of the two cases you raised
- 53 recoveries and 57 regressions. Value selection yields a net recovery of 14 pairs, while rule application yields a net regression of 14 pairs; these two effects offset each other, leaving an overall difference of only four pairs. Of the 20 regression pairs in rule application, eight come from two sandwich items. Adding analytical units introduces multiple pair units from the same pool into the evidence, and the model examines a neighboring pair rather than the pair specified by the question, getting the conditions and verdict wrong.
- The decrease from Raw → +Units for the strongest model. The same pattern appears for this model. Across the 110 in-context items, rule application accounts for four of the seven regression pairs from +Actions → +Units. On these 110 items, Raw and +Units both yield 61 correct responses; the decrease of three items you noted comes from the 50 SQL items that query tables.
We have added the cause classification, its rules, and the analysis by cause to Appendix J.5. We also agree with your point about single runs, so we have repeated the runs five times for each of four models under the +Retrieval condition and have added the results to Appendix J.11. Your feedback helped us show more clearly what each evidence condition measures. Thank you again.
Official Comment by Authors (1/3)
Thank you for your careful review and for your positive assessment of the paper’s approach and contributions. We respond in four comments, pairing the weaknesses and questions from [W1, Q1] to [W4, Q4]. The table, section, and appendix numbers below refer to the revised manuscript.
[W1, Q1] What the evidence layers provide and what reasoning remains for the model
As requested, we have counted the fields whose values are supplied verbatim in the evidence and separated the results into extraction, calculation, and adjudication fields. Evidence rarely explicitly provides field values, and the reasoning remaining for the model consists mostly of calculation and adjudication.
I. The role of analytical units
The two examples you mention differ in nature. As you note, the analytical unit in Appendix K.3 directly supplies the value of the requested span. By contrast, the relation provided by the analytical unit in Table 7 is material for adjudication, not the answer to a field, and the model must use that relation to derive the answer. Analytical units (L3) are an evidence layer constructed and proposed in this paper, and the purpose of the evaluation is to measure what changes when they are supplied as input. Because all analytical units are constructed in the same way within the scope defined by the question, without using the reference, finding the required units, calculating, and adjudicating remain the model’s work.
II. Fields whose values are explicitly supplied in the evidence
Table W1-1. Required fields whose reference values appear verbatim in the evidence alongside their corresponding names (out of 740 fields across 110 items in the in-context path)
The six in +Actions are the borrower and liquidator addresses in three liquidation items, and the one added in +Units is the span in the K.3 item. Information that serves as material for adjudication, such as the relation in Table 7, is not included in this count.
III. Results by extraction, calculation, and adjudication field
We have divided the 943 required fields into extraction (fields that require finding and writing down addresses or names), calculation (quantity fields), and adjudication (fields that require choosing a verdict, grade, whether conditions are met, or a category).
Table W1-2. Field accuracy by field type (average across 11 models, %)
We have added Table W1-1, Table W1-2, and their criteria to Appendix J.8.
[W2, Q2] Comparison with programmatic pipelines and code-execution and planning agents
We agree with your point that comparisons with other agents are needed. As requested, we have evaluated public code-execution and planning agents with the same questions, evidence, and execution budgets, and have also run conditions with tools removed from our pipeline. Our pipeline outperforms the public agents even with all tools removed, and half of the items remain incorrect after computation is delegated to tools. The remaining difficulties lie in determining what to compute and applying the decision criteria in the rulebooks.
I. Programmatic pipelines
The program that generates the reference is used only to produce correct answers, not as a solving baseline. Starting from natural-language questions requires an LLM to interpret the questions, and we have not found a public programmatic pipeline that solves the four tasks without an LLM. We have therefore compared against representative public agents that solve problems by executing code.
(Continued in Official Comment by Authors (2/3).)