acceptodds
Under review as a conference paper at ICLR 2027

CAIA: A Benchmark for Tool-Grounded Language Agents in Real-life Adversarial Cryptocurrency Tasks

Abstract

An agent answering a cryptocurrency question must find the relevant historical records and compute the requested quantity. Counting transactions, for example, does not answer a question about distinct wallets. We introduce CAIA, 178 expert-curated tasks with reference answers and checked tool sequences for obtaining them. We compare 17 models through a shared 23-tool interface, allowing up to five tool-use rounds within each run. Tools raise GPT-5's single-attempt success from 28.1% to 70.2%. Repeating the task in five independent runs produces a correct GPT-5 answer on 77.0% of tasks, but majority voting selects a correct answer on only 67.4%. This exposes an answer-selection bottleneck: correct candidates are available but lost during aggregation. Across 23,463 calls, an individual attempt's call count also has near-zero association with correctness. The interaction records connect these outcomes to source choice, historical scope, and calculation. We release the questions, expert reference routes, and records to support evidence-based analytical agents.

Primary area:
datasets and benchmarks
Keywords:
language agents, tool-augmented LLMs, agent evaluation, evidence grounding, tool use, retrieval and reasoning, cryptocurrency, blockchain analytics, adversarial information environments

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fLvFOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

CAIA introduces a benchmark of 178 expert-curated cryptocurrency tasks for evaluating tool-grounded language agents. It provides validated answers, tool-use routes, and interaction traces to analyze evidence retrieval, temporal reasoning, and computation. Experiments on 17 models show that tool use greatly improves accuracy, while repeated attempts reveal limitations in answer selection and majority voting.

Critical strengths
  1. The authors construct an Agent Benchmark for cryptocurrency analysis.
  2. CAIA contains real-world questions, which are reviewed by experts and annotated with complete trajectory analysis.
  3. The authors also analyze the errors of agents on CAIA.
Critical weaknesses
  1. Why does the evaluation limit the model to “5 rounds of tool calls”? How was this limitation of 5 determined? Could this limitation be too restrictive and prevent models from obtaining correct answers?
  2. The authors do not analyze whether the designed 23 tool interfaces are comprehensive and appropriate.
  3. Why does the evaluation harness not adopt more mature frameworks such as Codex or Claude Code? I am curious whether improvements to the harness would further improve model performance on CAIA.
Questions for authors
  1. The model evaluation ultimately uses binary correctness. Why not consider using the Reference Route for rubric-based scoring?
  2. Figure 2 in the paper is empty and should be replaced with an actual figure.
  3. It is recommended to provide more annotation details, such as expert backgrounds, annotation time, and inter-rater agreement.
1 comment

Thank you for the review. Its three weaknesses ask about design choices that the submission states, and none identifies an error.

W1: Five rounds. Five is a fixed setting of the shared scaffold (Section 3), which we did not vary (Appendix E). The cap limits decision rounds, not calls: a round can issue several calls, and the cap resets for each of the five runs. Two results suggest the cap is not the main limit. An attempt's call count is almost unrelated to its correctness (r ≈ 0.03; Section 4.3), so attempts that make more calls are not more often right. And depending on the model, 9.6–40.4% of tasks are solved in some runs but not in most; for these tasks five rounds were enough to reach the correct answer, and the lost accuracy comes from inconsistency across runs.

W2: The toolkit. Every task comes with "checked tool sequences for obtaining" its answer (submission abstract): Section 2.2's reference-route validation kept only the tasks whose answers the tools could obtain (186 → 178). The toolkit is therefore sufficient for every task in the benchmark; we do not claim it covers cryptocurrency analysis in general. Appendix M lists all 23 tools. Table 6 suggests that agents under-use the tools rather than lack them: code execution accounts for 1.5% of 23,463 calls and web search for 49.6%, although many tasks require aggregation.

W3: Codex or Claude Code. A benchmark compares models under a fixed protocol. A vendor harness changes prompting, context management and built-in tools at once, and several of the 17 models have no such harness, so differences would no longer be attributable to the model. A stronger harness can be tested on the released tasks with the tools of Appendix M; that is future evaluation, not a flaw of the benchmark.

Q1: Route rubric. Routes are not unique. A route rubric would penalize valid alternative routes and give credit for progress toward a wrong number, so we score the final answer and use routes for diagnosis (Appendix N).

Q2: Figure 2. Thank you for flagging it. The figure used transparency masks that some PDF viewers render blank; it is now flattened, and two other figures with similar display problems were repaired.

Q3: Annotation. Section 2.2 and Appendix G report part of this. Ninety-two specialists each reviewed 50 candidates; each candidate received at least four ratings and was ranked by trimmed mean; the top 200 were deduplicated to 186 and validated down to 178. A ten-member expert team agreed with the evaluator on all 500 sampled decisions, and 16 participants completed a stratified 10% subset. Expert backgrounds, annotation time and inter-rater agreement are not reported; we will add the backgrounds and annotation time in the final version.

On the rating. The review rates soundness 2 ("minor or fixable"). Its weaknesses are design questions that the paper answers, and the one defect it identifies is a rendering problem, now fixed. We respectfully ask you to reconsider whether these support rejection.

Reviewer_d4r3OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents CAIA, a benchmark of 178 expert-curated cryptocurrency analysis tasks with reference answers and validated tool routes. It evaluates 17 language models under a direct-answer setting and a shared 23-tool setting, measuring single-attempt accuracy, five-attempt candidate coverage, majority-vote selection, and tool-use trajectories. Results show that tool access substantially improves accuracy, while repeated runs often produce a correct candidate that majority voting fails to select, pointing to an answer-selection bottleneck.

Critical strengths
  1. The task bank is reasonably comprehensive: 178 expert-curated tasks across six categories, ranging from direct retrieval to multi-source synthesis.
  2. The model evaluation is broad, covering 17 proprietary and open-weight models under both direct-answer and tool-enabled settings with a shared tool suite.
Critical weaknesses
  1. The evaluation lacks a fixed evidence environment. Results depend not only on the models but on live external tools and data sources that change over time. As a consequence, it is often unclear whether a failure stems from tool availability, retrieval, or model reasoning, and the reported numbers cannot be reliably reproduced or compared across time. This also raises data-contamination concerns for tasks whose answers may appear in web sources. For a benchmark paper, non-reproducible measurements undermine the primary contribution.
  2. The empirical analysis is largely descriptive and offers limited new insight. Most findings: stronger models answer more correctly, repeated sampling improves candidate coverage, tool access helps, are expected. The one interesting result, the gap between Pass@5 and majority-vote accuracy, is identified but not pursued. In general, the main finding does not translate into actionable understanding, which limits its value beyond the task collection itself.
Questions for authors
  1. How does CAIA relate to closely related benchmarks such as DMind (Huang et al., KDD 2026) and CryptoBench (Guo et al., 2025)? These are not sufficiently discussed or compared, which makes the specific contribution hard to assess.
  2. The paper claims a “real-life adversarial” setting and discusses stale, promotional, and conflicting information, but does not appear to construct or evaluate controlled adversarial cases. Is there any quantitative evidence for this claim?
  3. The human reference is based on 16 participants completing a stratified 10% subset, while models are evaluated on all 178 tasks.
  4. The current task taxonomy is mainly topic-based rather than capability-based. The six categories describe application areas but questions within the same category may require very different capabilities. It might be more insightful to have the classification based on direct retrieval, historical-state alignment, statistical calculation, multi-source synthesis, and subjective analysis.
2 comments

Thank you for your review. Shared evidence is in the general response posted above. Here we answer your specific points.

W1: "The evaluation lacks a fixed evidence environment." The targets are fixed: each time-sensitive task is anchored to a block height or timestamp (Section 2.3), so its reference answer does not change. The 178 tasks and their verified answers, which are the benchmark, are fixed and released. What changes is what live services return, and that holds for every benchmark that evaluates agents on the live web, including CryptoBench, which you cite as a comparison and which releases new questions every month. The revision states that scores are a dated snapshot and should be compared within one period (Appendices C and E).

You also write that "it is often unclear whether a failure stems from tool availability, retrieval, or model reasoning." The submission provides the means to tell these apart and says so. Section 3: "Reading them together distinguishes a retrieval error from an incorrect aggregation of relevant records." Section 5: retained responses let an evaluator "distinguish changes in reasoning from changes in what the tools returned." Section 4.3 applies this to representative failures, and Appendix N.2 classifies the failure types, describing them as "categories for reading a trajectory rather than frequency estimates for the whole benchmark."

Contamination. Without tools, Pass@1 is 12.9–28.1%, which limits how much memorized answers can explain. With tools, finding the authoritative public record is the task itself, not contamination.

W2: "Most findings ... are expected ... identified but not pursued." Your examples (stronger models score higher, sampling and tools help) are context, not the findings. The introduction states two findings: the selection bottleneck, and that an attempt's call count is nearly unrelated to its correctness (r ≈ 0.03) even though call volume tracks model strength (r = 0.82). The review does not discuss the second.

The gap was not left unpursued. The submission's Section 5 set out how to test selectors on it, and the revision quantifies it (Section 4.2, Table 5, Appendix B). With tools, the gap ranges from 9.6 to 40.4 points, and the lower end of its 95% interval is at least 6 points for every model. That is about three times the gap without tools (median 17.4 vs 6.2 points). Five GPT-OSS-120B runs contain a correct answer for 72.5% of tasks at 0.033 USD, against 70.2% for one GPT-5 run at 0.115 USD. This is the headroom an evidence-aware selector could recover, and measuring such headroom is what a benchmark contributes.

Q1: DMind and CryptoBench. Now compared (Section 6, Appendix D). DMind tests Web3 knowledge without tools; CryptoBench uses one browsing tool, a 0–3 LLM-judge rubric and new questions each month. Neither has fixed historical targets with verified answers and reference routes, and neither measures the gap between coverage and selection.

Q2: Evidence for "adversarial." No. The submission uses the word descriptively (Section 2.3), as CryptoBench does, and claims no measurement of manipulation. The revision states this scope, and we will drop the word from the title if the AC and PC find it necessary.

Q3: 10% human pilot. The submission disclosed this (Section 4.1 and Appendix G, B in the submission) and uses the pilot only as a reference point; no result depends on it.

Q4: Capability taxonomy. The submission already classifies failures by capability (Appendix N.2): wrong object or source (retrieval), wrong time (historical alignment), wrong quantity and aggregation error (calculation), and unsupported synthesis. The topical categories describe what the benchmark covers; Appendix E notes that per-task capability labels are not provided.

On the rating. The review rates soundness 2 ("minor or fixable concerns"), lists strengths, and is submitted with confidence 2, yet recommends clear rejection. Its main objection applies equally to CryptoBench, which the review itself proposes as a comparison. Each concern is answered above, mostly from the submission's own text. We respectfully ask you to reconcile the score with your own ratings.

I thank the authors for the detailed response and revision. Several of my concerns have been addressed through the added comparisons with related work and the presentation of the findings. Some concerns remain. For example, fixing the evaluation targets does not fully address cross-time reproducibility because live tool outputs may change. The purpose of the 10% human pilot also remains unclear: if no result depends on it, what is it intended to demonstrate, and why was only this subset evaluated? Regarding the capability taxonomy, this is only a minor suggestion: I meant a per-task taxonomy based on the required capabilities rather than failure categories. Considering these revisions and clarifications, I will raise my score from 1 to 2.

Reviewer_WGJWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CAIA contains 178 expert-curated cryptocurrency questions with reference answers and validated evidence routes. Seventeen models are evaluated with and without a shared 23-tool interface and five sampled attempts per task. Tools increase GPT-5’s reported single-attempt success from 28.1% to 70.2%; five-attempt candidate coverage reaches 77.0%, while majority voting achieves 67.4%. The paper also examines tool usage and qualitative failures involving entity resolution, historical scope, and calculation.

Critical strengths
  1. Historical anchors and expert-checked evidence routes make the tasks substantively useful. Distinguishing wallet counts from transaction counts, or matching dates and metric definitions across sources, tests analytical correctness beyond plausible financial prose.

  2. Separating candidate coverage from answer selection identifies an actionable failure: correct answers can be generated but discarded by aggregation. The model panel and retained trajectories provide a useful basis for investigating evidence-aware selection.

Critical weaknesses
  1. The central aggregation result needs a more executable specification. The manuscript does not sufficiently identify the judge configuration, semantic answer-grouping and tie rules, or the precise Pass@1 estimator. The 500-response human agreement audit supports final-answer scoring but does not resolve those aggregation details. The reproducibility statement also describes a two-tool demo, whereas the reported experiment uses 23 tools.

  2. The adversarial component is not separately validated. Section 2.3 describes a contested information environment, but no task-level annotation or controlled comparison establishes how often misleading sources actually drive errors. This limits the interpretation of the benchmark as measuring adversarial cryptocurrency analysis.

Questions for authors
  1. Clarification: How are five free-form answers grouped for voting, how are ties resolved, and is Pass@1 a first-attempt score or an average across attempts? Please identify the evaluator model/prompt and the frozen tool responses or complete interface needed to reconstruct the reported selection gap, with paired uncertainty across tasks.

  2. Construct validity and positioning: Yang et al., “CRAG — Comprehensive RAG Benchmark” (NeurIPS 2024), not cited in the manuscript, already evaluates domain-diverse, temporally varying factual questions through search interfaces. Which CAIA tasks specifically require resolving verified misleading or conflicting evidence, and what are the results on that subset? This would clarify the cryptocurrency-specific adversarial contribution beyond time-sensitive retrieval and arithmetic.

https://proceedings.neurips.cc/paper_files/paper/2024/hash/1435d2d0fca85a84d83ddcb754f58c29-Abstract-Datasets_and_Benchmarks_Track.html

1 comment

Thank you for a constructive review. The revision supplies the specification you asked for, and no reported number changes.

W1/Q1: Specification. Appendix A gives the full specification, and Section 3 states the rules in brief.

  • Evaluator. Three judges, o3-2025-04-16 (reasoning effort medium), gpt-4.1 (temperature 0.2) and deepseek-r1-250528 (default sampling), each score every answer three times against the reference. An answer is correct when at least 5 of the 9 judgments accept it. The criterion is the same with and without tools, and the judge prompt is in the released code.
  • Grouping and ties. Majority vote counts a task as correct when at least 3 of its 5 runs are judged correct. It counts correct labels rather than grouping free-form answers, and five binary labels cannot tie. When the correct runs give the same answer, a plurality vote over answers would do at least as well, so the reported gap is an upper bound on what plurality voting would lose.
  • Pass@1 scores one designated run per task; it is not an average over runs.
  • Paired uncertainty. Every majority-correct task is also Pass@5-correct, so the gap is a count of tasks with a paired Wilson interval. GPT-5 loses 17 of 178 tasks (9.6 points, 6.0–14.8), and with tools the lower end of the interval is at least 6 points for all 17 models (Table 5, now in Section 4.2).

Reproducibility. The submitted statement mentioned the two-tool demo, but it also pointed to "the evaluated 23-tool interface, with the toolkit detailed in Appendix H" (now M). It now lists the release explicitly: the questions and reference answers, the evaluation code with the judge prompts and configuration, a script that computes Pass@1, Pass@5 and majority vote with paired intervals, and the catalogue of all 23 tools. Frozen tool responses are not released, so reruns query live services. The intervals in Table 5 can be recomputed from the reported counts.

W2/Q2: Adversarial. The submission uses the term descriptively, "in the practical sense that available sources can serve competing incentives" (Section 2.3), and claims no measurement of manipulation. We did not annotate a subset of verified misleading evidence, so there is no subset result to report; Section 2.3 and Appendix E now state this scope. No reported result depends on the word, and we will drop it from the title in the revision if the AC and PC find it necessary.

CRAG. Thank you for the pointer; CRAG is now cited and compared (Section 6, Appendix D). CRAG retrieves from mock web pages and knowledge-graph APIs, whereas CAIA queries 23 live interfaces, including blockchain ledgers at a given block height. Many CAIA answers must be computed from historical records, such as the number of distinct wallets active in a quarter, rather than looked up. Every CAIA task also has an expert-validated reference route and is run five times, which is what exposes the gap between coverage and selection.

Your review rates soundness "minor or fixable" and finds that separating coverage from selection "identifies an actionable failure." The specification is now complete, and the scope of "adversarial" is stated explicitly. We respectfully ask for a score reconsideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the Reviews

We thank the reviewers. All three rate soundness 2 ("minor or fixable concerns"), and none of their concerns shows a reported number to be wrong. The reviews also credit the task bank, the 17-model evaluation, and the separation of candidate coverage from answer selection. No new experiments were run, and every new number is computed from results already in the paper.

What changed in the PDF

  • Table 5 (new, Section 4.2) reports for every model the selection gap, the share of tasks with a correct run but no correct majority, with a paired 95% interval. With tools, the gap ranges from 9.6 to 40.4 points, and the lower end of its interval is at least 6 points for every model; for GPT-5 it is 17 of 178 tasks (9.6 points, interval 6.0–14.8).
  • Section 3 now states the scoring rule and the exact majority-vote rule: a task counts when at least three of its five runs are judged correct. When the correct runs agree on an answer, a plurality vote would do at least as well, so the gap is an upper bound on what plurality voting would lose. Section 4.2 explains what the gap means and what better selection could recover, and Section 2.3 states in one place what "adversarial" means here.
  • Appendix A (new) gives the evaluator configuration (three judges, nine judgments per answer, correct when at least five accept), every estimator as a formula, and why the interval is paired.
  • Appendix B (new) shows that with tools about three times as many tasks fall in the gap (median 17.4 vs 6.2 points), and that five GPT-OSS-120B runs contain a correct answer for 72.5% of tasks at $0.033, more than one GPT-5 run achieves (70.2% at 0.115 USD). This is an oracle bound that future selectors can be measured against.
  • Appendices C–E (new) explain what live tools do and do not fix, compare CAIA with CRAG, DMind, CryptoBench and CryptoAnalystBench (also Section 6), and discuss limitations (also Section 5).
  • Corrections: Figure 2 is fixed, and the reproducibility statement now describes the 23-tool interface and lists exactly what is released.

Why the concerns do not support rejection Several are already answered in the submission. Section 3 states that scores and tool records together distinguish "a retrieval error from an incorrect aggregation of relevant records" (d4r3). Every retained task comes with "checked tool sequences for obtaining" its answer (abstract; 186 → 178 after validation), so the toolkit can answer every task (fLvF). Other concerns are about choices the submission stated openly: live tools with fixed historical targets, the 10% human pilot, the descriptive use of "adversarial," and the five-round budget. A limitation stated openly should not count against a paper; otherwise reviewing rewards papers that leave their limitations out. To our knowledge, CAIA is the only cryptocurrency agent benchmark with fixed historical targets, verified answers and expert reference routes, and it measures for every model how many tasks it solves in some runs but not in most. We ask the reviewers to reconsider their scores.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.