acceptodds
Under review as a conference paper at ICLR 2027

ChainQABench: A Benchmark for Evidence-Grounded Question Answering over Blockchain Data

Abstract

Public blockchains make every transaction observable, yet even simple economic questions, such as how much an address gained, require combining many raw records. Doing so involves decoding the records into protocol actions, grouping them into analytical units, and judging what the result can establish. Large language models (LLMs) find these steps hard to perform directly on raw records, where values are hex-encoded and a single action spans several logs and traces. To address this, we introduce ChainQABench, to our knowledge the first evidence-grounded question-answering benchmark for blockchain data. Its 160 expert-curated questions cover asset flows, exchange attribution, value extraction through transaction ordering (MEV), and wash-trading patterns on decentralized exchanges. Five domain experts wrote rulebooks fixing admissible evidence and when a claim must remain indeterminate; an independent verifier reproduces every reference from its raw records. Over 16M transactions and 105M traces, we build an evidence hierarchy that organizes raw records into protocol actions, analytical units, and attribution context while preserving provenance links. We evaluate eleven LLMs as evidence moves from raw records to protocol actions, analytical units, and retrieved evidence. Even the strongest LLM, Gemini-3.8-Flash, achieves a maximum question accuracy of only 53.1% across these evidence conditions. Decoding raw records into protocol actions yields the most reliable gains, exposing typed quantities that LLMs otherwise struggle to reconstruct. We release the benchmark, evidence hierarchy, and independent verifier to support research on how evidence representations shape on-chain reasoning.

Primary area:
datasets and benchmarks
Keywords:
blockchain, benchmark, evidence-grounded question answering, large language models, on-chain analysis

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_GhtkOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ChainQABench, a benchmark of 160 expert-curated questions covering asset flows, exchange attribution, MEV-related interactions, and operational wash-trading patterns on Ethereum. It provides task-specific rulebooks, independently recomputed references, and an evidence hierarchy connecting raw records to decoded actions and analytical units. Eleven LLMs are evaluated under four evidence conditions, with SQL and calculation tools available through the evaluation harness. Decoded actions provide the most consistent improvements, while further grouping and retrieval have mixed effects.

Critical strengths
  • The benchmark addresses concrete analytical questions that require connecting blockchain records across transactions, events, and addresses. The four task families cover complementary aspects of on-chain analysis and provide a useful setting for studying evidence-grounded question answering over complex structured data.

  • The task-specific rulebooks carefully distinguish what the available evidence can establish. In particular, separating structural associations from ownership or intent, and distinguishing insufficient evidence from direct contradiction, makes the evaluation targets more precise and avoids treating every observed transaction pattern as a definitive real-world conclusion.

  • The reference construction includes a separately implemented verifier that recomputes the required fields from frozen sources. Together with provenance links between derived objects and underlying records, this supports checking numerical results and tracing disagreements to their source evidence or adjudication rules.

  • The evaluation covers eleven closed and open-weight models under the same questions and reference answers across four evidence conditions. Reporting both question-level and field-level accuracy, together with worked examples of successes and regressions, provides useful visibility into partial correctness and the different effects of evidence preparation.

Critical weaknesses
  • The higher-level evidence already performs part of the reasoning being evaluated, but this contribution is not systematically separated from the model's remaining work. In Table 6, the analytical unit explicitly supplies the relation between the two outer swaps and an intermediate swap. In Appendix L.3, it directly provides the nine-block span requested by the question. These are useful inputs for studying evidence preparation, but the evaluation should distinguish fields that can be copied from supplied evidence from those requiring calculation or further inference. Without this distinction, the capability measured under each evidence condition remains difficult to interpret.

  • The experiments lack independent programmatic solving baselines and comparisons between representative tool-use harnesses. Although the current harness provides SQL, exact arithmetic, and some archive-query access, the reported comparisons primarily vary the model and evidence condition. The reference-generation program operates on fixed item specifications and therefore does not establish how a practical system would perform from the natural-language question. A programmatic pipeline and a representative code-execution or planning agent, evaluated with comparable evidence and budgets, would help establish where the benchmark presents difficulties beyond decoding, aggregation, and rule execution.

  • The evidence for benchmark coverage and generalization is limited. All 160 questions come from one week of Ethereum activity, with 40 questions per task and only 12 and 8 questions in the arbitrage and anomaly-screening topics, respectively. The paper does not provide a systematic account of shared transactions, addresses, or question templates across items. This makes it difficult to assess how many distinct analytical situations are represented and whether the reported comparisons would remain stable on another time window.

  • The experimental breadth is greater than the depth of the resulting explanation. The paper establishes that decoding often helps and that additional grouping or retrieval has mixed effects, but does not sufficiently identify the causes of these differences. For example, the strongest model's question accuracy decreases from 53.1% under RAW to 51.2% under +UNITS, while Appendix K.6 reports 53 recoveries and 57 regressions from +ACTIONS to +UNITS across the in-context evaluations. These mixed outcomes are acknowledged, but a systematic breakdown of evidence-availability, calculation, rule-application, and output errors is missing. Single runs without uncertainty estimates also limit the interpretation of small differences.

Questions for authors
  1. What proportion of required answer fields is explicitly supplied at each evidence layer, particularly L3? Can the results be separated into direct extraction, calculation, and adjudication fields to clarify what reasoning remains for the model?

  2. How would a programmatic pipeline and a representative code-execution or planning agent perform when given the same natural-language questions, admissible evidence, and comparable execution budgets? Which difficulties remain after reliable decoding and computation are delegated to tools?

  3. How many questions share underlying transactions, addresses, pools, or construction templates? Can the authors provide evidence that the main conclusions are stable under resampling or evaluation on an additional time window?

  4. Can the recoveries and regressions across evidence conditions be categorized by their causes, including missing evidence, decoding, arithmetic, rule application, and output completeness? This would help explain why additional analytical structure sometimes reduces performance.

5 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/3)

Thank you for your careful review and for your positive assessment of the paper’s approach and contributions. We respond in four comments, pairing the weaknesses and questions from [W1, Q1] to [W4, Q4]. The table, section, and appendix numbers below refer to the revised manuscript.


[W1, Q1] What the evidence layers provide and what reasoning remains for the model

As requested, we have counted the fields whose values are supplied verbatim in the evidence and separated the results into extraction, calculation, and adjudication fields. Evidence rarely explicitly provides field values, and the reasoning remaining for the model consists mostly of calculation and adjudication.

I. The role of analytical units

The two examples you mention differ in nature. As you note, the analytical unit in Appendix K.3 directly supplies the value of the requested span. By contrast, the relation provided by the analytical unit in Table 7 is material for adjudication, not the answer to a field, and the model must use that relation to derive the answer. Analytical units (L3) are an evidence layer constructed and proposed in this paper, and the purpose of the evaluation is to measure what changes when they are supplied as input. Because all analytical units are constructed in the same way within the scope defined by the question, without using the reference, finding the required units, calculating, and adjudicating remain the model’s work.

II. Fields whose values are explicitly supplied in the evidence

Table W1-1. Required fields whose reference values appear verbatim in the evidence alongside their corresponding names (out of 740 fields across 110 items in the in-context path)

Raw+Actions+Units+Retrieval
Field count0677
Proportion0%0.8%0.9%0.9%

The six in +Actions are the borrower and liquidator addresses in three liquidation items, and the one added in +Units is the span in the K.3 item. Information that serves as material for adjudication, such as the relation in Table 7, is not included in this count.

III. Results by extraction, calculation, and adjudication field

We have divided the 943 required fields into extraction (fields that require finding and writing down addresses or names), calculation (quantity fields), and adjudication (fields that require choosing a verdict, grade, whether conditions are met, or a category).

Table W1-2. Field accuracy by field type (average across 11 models, %)

Field typeField countRaw+Actions+Units+Retrieval
Extraction2648.676.268.569.9
Calculation45742.846.947.849.6
Adjudication41760.864.163.765.4
Composite structures (lists, objects)4317.520.719.720.3
  • Extraction fields improve substantially when protocol actions are added (+27.6 pp). This is because addresses and names appear in the decoded actions. Most of the decline at +Units comes from one item whose evidence exceeds the budget and is therefore shortened before being supplied.
  • Calculation and adjudication fields improve only slightly even as evidence layers are added. This is because the evidence only decodes and groups records, leaving the model to calculate values and apply the decision criteria in the rulebooks. These two types account for 93% of required fields, and even under +Retrieval, only 49.6% and 65.4% are correct.

We have added Table W1-1, Table W1-2, and their criteria to Appendix J.8.


[W2, Q2] Comparison with programmatic pipelines and code-execution and planning agents

We agree with your point that comparisons with other agents are needed. As requested, we have evaluated public code-execution and planning agents with the same questions, evidence, and execution budgets, and have also run conditions with tools removed from our pipeline. Our pipeline outperforms the public agents even with all tools removed, and half of the items remain incorrect after computation is delegated to tools. The remaining difficulties lie in determining what to compute and applying the decision criteria in the rulebooks.

I. Programmatic pipelines

The program that generates the reference is used only to produce correct answers, not as a solving baseline. Starting from natural-language questions requires an LLM to interpret the questions, and we have not found a public programmatic pipeline that solves the four tasks without an LLM. We have therefore compared against representative public agents that solve problems by executing code.

(Continued in Official Comment by Authors (2/3).)

AuthorsOpenReview ↗

Official Comment by Authors (2/3)

[W2, Q2] (continued)

II. Comparison with public agents

We have evaluated the code-execution and planning agents from the public agent framework smolagents [1] with the same questions, evidence, and execution budgets. Both agents use SQL and Python execution. The model is Gemini-3.8-Flash, the best-performing model in the main text, and the evidence condition is +Retrieval. We have also run two conditions with tools removed from our pipeline. Our pipeline provides SQL, an exact arithmetic calculator, archive queries, and 19 tools that aggregate the ledger without a row limit. Because the ledger is inaccessible without tools for the 50 SQL path items, we run the conditions with tools removed on the 110 in-context items.

Table W2-1. Field accuracy (%) and input tokens per item by configuration

ConfigurationAll 160 items110 in-context itemsInput tokens per item
Ours76.279.1321K
Code-execution agent46.242.8541K
Planning agent43.941.81,307K
Ours w/o aggregation tools and evidence SQL—66.8267K
Ours w/o all tools—67.881K

Input tokens per item are averaged over the in-context items.

  • The public agents perform worse than our pipeline. Across all 160 items, field accuracy is 76.2% versus 46.2% and 43.9%, meaning that the public agents answer fewer than half of the fields correctly, while using 1.7 to 4.1 times as many input tokens per item as our pipeline.
  • Our pipeline outperforms the public agents even with all tools removed. On the 110 in-context items, field accuracy is 67.8% versus 42.8% and 41.8%.
  • Half of the items remain incorrect even with tools available. With all tools available, in-context field accuracy rises from 67.8% to 79.1%, but question accuracy across all 160 items is 50.0% (Appendix J.9). Because the tools return exact values and exhaustive aggregations, the remaining difficulties lie in determining what to compute and applying the decision criteria in the rulebooks.

We have added the comparison with public agents, the tool-removal conditions, and the full results to Appendix J.9.

[1] Roucher et al., smolagents: a smol library to build great agentic systems, 2025. https://github.com/huggingface/smolagents


[W3, Q3] Sharing across items and evaluation on another time window

We agree with your point that sharing across items and evaluation on another time window should be reported. As requested, we have quantified sharing across items and constructed and evaluated 160 new items from another week. The same construction procedure produces items for the four tasks in another week, and the comparisons between evidence conditions hold on the new 160 items.

I. Sharing across items and item counts by topic

Table W3-1. Number of items confirmed to share with other items in the same task

TaskTransactionsAddressesPoolsTemplates
T11330—12
T21416—0
T3121872
T41214132

Addresses include tokens and contracts. Only 16 of the 160 items use identical question templates with differences only in identifiers and numerical values. Items share records because the same records can require different analyses. For example, the same swap is used for price calculation, sandwich adjudication, and wash-trading checks.

The 12 arbitrage items and eight anomaly-screening items you mentioned are topics assigned to items, rather than tasks. The topics overlap, and Section 4.3 limits the topic-level results to "overlapping subsets rather than independent tasks". We draw conclusions at the task level, with 40 items per task.

II. Evaluation on another time window

Using the same construction procedure, we have created 160 new items from another week (blocks 25,620,418 to 25,670,641), with 40 items for each of the four tasks. We have evaluated Gemini-3.8-Flash, the best-performing model in the main text, under four conditions using our pipeline.

Table W3-2. Question accuracy / field accuracy (%) by time window

ConditionBlocks 25,369,414–25,419,597Blocks 25,620,418–25,670,641
Raw53.1 / 77.447.5 / 76.1
+Actions53.1 / 79.348.8 / 77.5
+Units51.2 / 79.445.0 / 74.4
+Retrieval50.0 / 76.249.4 / 77.0
  • The benchmark is not tied to a particular week. Applying the same construction procedure to on-chain records from another week produces new items for all four tasks: asset flows, exchange attribution, MEV-related interactions, and AMM wash-trading patterns.
  • The reported comparisons hold in the additional time window. Field accuracy increases from Raw → +Actions in both time windows (77.4 → 79.3, 76.1 → 77.5).

We have added the sharing across items to Appendix F.5 and the evaluation on the additional time window to Appendix J.10.

(Continued in Official Comment by Authors (3/3).)

AuthorsOpenReview ↗

Official Comment by Authors (3/3)

[W4, Q4] Causes of recoveries and regressions across evidence conditions

We agree with your point that the causes of recoveries and regressions should be identified. Following your proposed cause categories, we have classified pairs whose correctness changed using the execution records. Additional analytical structure reduces performance at the rule-application stage, rather than because of missing evidence or arithmetic errors. At the stage of selecting values for calculation, adding analytical units instead yields more recoveries.

I. Classification of recoveries and regressions by cause

For the 110 items that receive evidence in context, we have classified pairs whose correctness changed at each of three transitions: Raw → +Actions, +Actions → +Units, and +Units → +Retrieval. There are 347 pairs in total across 11 models. We have paired the failed and successful runs for the same model and item and compared their answers, cited evidence, and calculation formulas.

Table W4-1. Number of recovery / regression pairs by cause (11 models, 110 in-context items)

CauseRaw → +Actions+Actions → +Units+Units → +Retrieval
Missing evidence0 / 03 / 00 / 3
Decoding45 / 0——
Value selection0 / 2134 / 2025 / 20
Arithmetic4 / 23 / 55 / 0
Rule application13 / 106 / 2016 / 14
Output completeness26 / 66 / 77 / 6
Composite-structure fields (undeterminable)1 / 31 / 59 / 1
Total89 / 4253 / 5762 / 44

Value selection denotes cases where, under decoded evidence, the ledger values used in the calculation differ from those in the successful run. Rule application denotes cases where a condition specified by the rulebook or the final verdict is incorrect.

  • Rule application is where additional analytical structure reduces performance. From +Actions → +Units, rule application accounts for six recoveries and 20 regressions. At the same step, value selection accounts for 34 recoveries and 20 regressions, indicating that analytical units help select the values to use in calculations.
  • Missing evidence and arithmetic errors do not explain these regressions. Of the 57 regression pairs from +Actions → +Units, none involve missing evidence and five involve arithmetic.
  • From Raw → +Actions, decoding yields more recoveries than regressions. Differences in the ledger values used account for 45 recoveries against 21 regressions.

II. Analysis of the two cases you raised

  • 53 recoveries and 57 regressions. Value selection yields a net recovery of 14 pairs, while rule application yields a net regression of 14 pairs; these two effects offset each other, leaving an overall difference of only four pairs. Of the 20 regression pairs in rule application, eight come from two sandwich items. Adding analytical units introduces multiple pair units from the same pool into the evidence, and the model examines a neighboring pair rather than the pair specified by the question, getting the conditions and verdict wrong.
  • The decrease from Raw → +Units for the strongest model. The same pattern appears for this model. Across the 110 in-context items, rule application accounts for four of the seven regression pairs from +Actions → +Units. On these 110 items, Raw and +Units both yield 61 correct responses; the decrease of three items you noted comes from the 50 SQL items that query tables.

We have added the cause classification, its rules, and the analysis by cause to Appendix J.5. We also agree with your point about single runs, so we have repeated the runs five times for each of four models under the +Retrieval condition and have added the results to Appendix J.11. Your feedback helped us show more clearly what each evidence condition measures. Thank you again.

Reviewer_uYLVOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a new benchmark for question answering with supporting evidence in the blockchain domain. The questions for the dataset were collected and verified by experts, while the data itself was gathered automatically, then filtered and verified by experts. Experiments were conducted on this benchmark using several models.

Critical strengths

A new benchmark for a rather niche domain—manually curated and verified—includes multiple dimensions and types of annotations. The benchmark demonstrates significant potential for improving the performance of LLMs on tasks of this nature.

Critical weaknesses

Much important information is gathered in the appendices. Without reading them, the paper is clearly incomplete. The comparison with existing benchmarks in this domain is somewhat biased toward favoring the current work. It would be worthwhile to discuss the strengths and weaknesses of one or two of the most popular benchmarks in greater detail (beyond what is already presented in the table).

Questions for authors

What were the qualifications of the experts who compiled the dataset, and how much time, on average, was spent preparing a single question? How much time was spent preparing the set of rules?

3 comments

Thank you for your careful review and for your positive assessment of the paper’s approach and contributions. Below, we respond to [W1] and [Q1]. The table, section, and appendix numbers below refer to the revised manuscript.


[W1] Self-contained main text and comparison with existing benchmarks

We agree with your point that the benchmark should be understandable from the main text alone. We have moved content that defines the benchmark into the main text and added descriptions of two representative benchmarks. The main text now explains the meanings of the verdicts required by the items and the evaluation metrics on its own, and we explain what existing benchmarks and ChainQABench each evaluate beyond what is presented in the table.

I. Self-contained main text

We have placed two aspects of what the benchmark asks and how it scores responses in the main text.

  • Definitions of claim verdicts: Because 102 of the 160 items require a verdict, the meanings of the four verdicts (supported, contradicted, indeterminate, out_of_scope) are necessary to understand what the benchmark asks. We have also summarized the difficulty distribution and answer formats.
  • Evaluation metrics: The definitions and formulas for question accuracy and field accuracy.

II. Comparison with existing benchmarks

Table 1 is intended to show differences in evaluation targets, rather than a ranking.

  • CryptoBench: Evaluates retrieval and prediction using real-time cryptocurrency information sources such as on-chain intelligence platforms, market data providers, and DeFi dashboards. Template-based questions are updated monthly, and an LLM judge scores responses against expert-cross-validated references and rubrics.
  • EVM-QuestBench: Evaluates the ability to generate transaction scripts from natural-language instructions. Scripts are executed on a forked EVM chain, and a verifier checks receipts and the post-execution state. It consists of single-action tasks and multi-step workflow tasks.
  • ChainQABench: Evaluates the analysis of recorded on-chain activity in the form of claims about that activity. Expert-defined rulebooks fix the admissible evidence and analytical boundaries, references are derived from a frozen snapshot and reproduced by an independent verifier, and evidence is provided in multiple layers leading to raw records.

The first two benchmarks evaluate retrieval and prediction over real-time information and the generation of executable actions, while ChainQABench evaluates the analysis of recorded activity.

We have moved the verdict definitions to Section 3.4 and the evaluation metrics to Section 3.6, and added descriptions of the two benchmarks to Section 2.2.


[Q1] Expert qualifications and time spent on construction

The benchmark is constructed by five domain experts, and we estimate that item construction takes approximately 307 person-hours and rulebook preparation takes approximately 50 person-hours, totaling approximately 357 person-hours.

I. Expert qualifications

Table Q1-1. Experts’ degrees and experience (experience in years)

ExpertDegree, fieldOn-chain data analysisExchange and finance practiceRole
E1B.S., Physics and Mathematics47Item authoring
E2B.S., Computational Biology32Item authoring
E3B.S., Information Systems24Item authoring
E4M.S., Data Science12Independent verification
E5Ph.D., Industrial Engineering102Final decisions

All five experts have experience in on-chain data analysis and in exchange and finance practice. The item authors and the expert who performs independent verification work without seeing each other’s implementations.

II. Preparation time per item

Each item takes an average of approximately 1.9 person-hours (approximately 1.5 for T1, 1.8 for T2, 2.3 for T3, and 2.0 person-hours for T4). This is the sum of the time spent by the experts involved in each item and includes four stages: candidate selection, ledger evidence collection, reference calculation, and independent verification. It excludes the time spent implementing the independent verifier and building the data collection infrastructure. Across all 160 items, the total is approximately 307 person-hours.

III. Rulebook preparation time

The rulebooks for the four tasks take approximately 50 person-hours to prepare and undergo six revisions from the initial draft to the final version. An independent verifier separate from the construction code reproduces the references for all 160 items, and a separate expert reviews the results again.

We have added the expert qualifications and time spent on construction to Appendix E. Your feedback helped us communicate the benchmark’s composition and construction process more clearly. Thank you.

Thanks for you answer and clarifications.

Thank you very much for your response. We're glad that our clarifications were helpful. We sincerely appreciate your time and thoughtful feedback throughout the review process.

Reviewer_FzzDOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ChainQABench is 160 questions over one frozen week of Ethereum mainnet (blocks 25,369,414 to 25,419,597), 40 each on asset flows, exchange attribution, MEV interactions and AMM wash trading. Each task has a written rulebook (Appendix D) which fixes the admissible evidence, the thresholds, and when the answer has to stay indeterminate or out of scope. References are typed fields computed from the raw records under that rulebook, and a second implementation has to match them before release. The authors also build an evidence hierarchy, raw records, decoded protocol actions, analytical units, plus attribution labels, and run eleven LLMs under four conditions, Raw, +Actions, +Units and +Retrieval, where question accuracy needs every required field correct. The best model is Gemini-3.8-Flash at 53.1%. Decoding gives the most consistent gain, grouping and retrieval are mixed.

Critical strengths

The problem is real and the references are built the right way. There is no LLM judge and no rubric anywhere. Every reference is a set of typed fields computed under a written rule from a frozen snapshot, with explicit indeterminate and out of scope outcomes (Table 12), a verdict plus an evidence grade, and a second implementation which has to match before an item is released (Appendix E). I am not aware of another on-chain benchmark which does this, and the verdict and grade split is something other domain benchmarks could copy. So this part is good.

Critical weaknesses

The decoding claim in the abstract is thinner than it reads. Everything is a single run per LLM and condition (Appendix K) on 160 items with no interval anywhere, so one item is 0.6 pp on the full benchmark and 2.5 pp per task. The “seven improve, none degrade” sentence in Section 4.1 counts Qwen3-VL-8B going from 5.0 to 5.6, which is one item, and four of the eleven models are ties. The mean gain is about 2.5 pp, which is four items per model. Of the 43 net question-level gains from Raw to +Actions summed over the 11 LLMs (Table 5), 30 come from the 13 liquidation items (Table 18), and T3 is the only task which moves by more than one item (Figure 4). Under Raw a liquidation item basically asks the model to recognize one of the eight event signature hashes in Table T3-4, which is exactly the step decoding does for it. So the most reliable gain in the abstract is mostly decoding telling the model which log is a liquidation. Please add paired intervals over items and say in the abstract where the gain comes from.

Questions for authors

For each task and condition, how many tokens of evidence does an in-context item carry, what context window did each open-weight model run with, and was any input truncated? The closed models average 180K to 800K input tokens per item (Table 26). If a Raw input for a T2 or T4 item does not fit in Gemma4-12B or Qwen3-VL-8B, then part of the Raw to +Actions gain for the open-weight models is the input fitting, not the decoding.

1 comment

Thank you for your careful review and for your positive assessment of the paper’s approach and contributions. We respond to [W1] and [Q1] below. The table, section, and appendix numbers below refer to the revised manuscript.


[W1] Statistical support for decoding gains and their concentration in liquidation items

We agree with your point that statistical support is needed for the decoding gains. We have added the requested paired intervals over items and analyses by scope. The results show that the 95% intervals for the Raw → +Actions gains are above zero for both question accuracy and field accuracy. These gains are largest on liquidation items, but are also observed in field accuracy on the other items.

I. Paired intervals and field accuracy

Table W1-1. Raw → +Actions changes (averaged over 11 models, 160 items)

MetricChange (pp)95% interval
Question accuracy+2.44[+0.51, +4.49]
Field accuracy+4.36[+2.48, +6.34]

We have computed the 95% intervals with a paired bootstrap over items, stratified by task. The statement in Section 4.1, "Moving from Raw to +Actions improves question accuracy for seven LLMs and degrades none", refers to question accuracy, which counts an answer as correct only when all required fields are correct and therefore does not reveal partial improvements. In field accuracy (Table 6), 10 of the 11 models improve: Qwen3-VL-8B-Instruct, which you noted, improves by +6.0 pp, and the four models with tied question accuracy also improve by +1.4 to +12.1 pp.

II. Liquidation items and the other items

Your count is correct: 30 of the 43 net gains come from liquidation items. However, the gains are not limited to liquidation.

Table W1-2. Raw → +Actions changes in field accuracy by scope (averaged over 11 models)

ScopeChange (pp)95% interval
Liquidation+18.94[+11.23, +26.24]
All items excluding liquidation+3.15[+1.36, +5.04]

Even on the 147 items excluding liquidation, field accuracy increases by +3.15 pp (in 10 of the 11 models), and the 95% interval is above zero. Moreover, because T3 includes liquidation alongside sandwich and cyclic arbitrage (Table 2, footnote 3), your observation that "T3 is the only task which moves by more than one item (Figure 4)" reflects the gains on liquidation items within T3.

III. The role of decoding in liquidation

You are correct that what decoding does for liquidation overlaps with the step required by the item, and this is the intended role of the protocol action layer. As explained in Section 4.3, liquidation is the topic where this effect is most direct.

While liquidation benefits most from decoding, some tasks benefit from later evidence layers. For example, T4, whose judgments depend on ordered swaps, flow closure, and actor relations, rises from 25.2% to 30.5% across the four conditions (Figure 4). As stated in Section 4.2, "no single evidence representation consistently improves all tasks", and the contribution of ChainQABench is to bring together tasks requiring different forms of reasoning and enable comparisons of the same items across multiple evidence layers.

We have added the intervals and tables above to Appendix J.7. In the abstract, we have specified that decoding gains are largest for questions where the required values appear directly in decoded protocol events.


[Q1] Evidence token counts, context size, and input truncation

All seven open-weight models are run with a 204,800-token context, and no request exceeds the context window.

Table Q1-1. Median tokens in the first request (in-context items; full prompt)

TaskRaw+Actions+Units+Retrieval
T177K73K73K23K
T2145K135K135K127K
T343K45K83K29K
T473K80K123K49K

No single request exceeds 197,950 tokens. To leave room for responses and tool-call history, we set an evidence budget of approximately 170,000 characters (approximately 150,000 tokens), and items exceeding this budget are shortened using the same rule across all four conditions before being passed to the models. The 180K to 800K in Table 34 refers to "Mean API tokens per item, summed across all requests" (Appendix L), not the length of a single input.

All T2 and T4 items for Gemma4-12B and Qwen3-VL-8B-Instruct, which you identified, fit within the context window even under Raw (a maximum of 164,653 tokens per request). Across T2 and T4 for these two models, the only net gain from Raw → +Actions comes from two T4 items for Gemma4-12B, and the full evidence is supplied under both conditions for both items. The case you were concerned about therefore does not occur for these two models.

We have added these figures to Appendix L. Your feedback helped us present the evidence for the decoding gains more clearly through paired intervals and analyses by scope. Thank you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.