- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CAIA introduces a benchmark of 178 expert-curated cryptocurrency tasks for evaluating tool-grounded language agents. It provides validated answers, tool-use routes, and interaction traces to analyze evidence retrieval, temporal reasoning, and computation. Experiments on 17 models show that tool use greatly improves accuracy, while repeated attempts reveal limitations in answer selection and majority voting.
Critical strengths
- The authors construct an Agent Benchmark for cryptocurrency analysis.
- CAIA contains real-world questions, which are reviewed by experts and annotated with complete trajectory analysis.
- The authors also analyze the errors of agents on CAIA.
Critical weaknesses
- Why does the evaluation limit the model to “5 rounds of tool calls”? How was this limitation of 5 determined? Could this limitation be too restrictive and prevent models from obtaining correct answers?
- The authors do not analyze whether the designed 23 tool interfaces are comprehensive and appropriate.
- Why does the evaluation harness not adopt more mature frameworks such as Codex or Claude Code? I am curious whether improvements to the harness would further improve model performance on CAIA.
Questions for authors
- The model evaluation ultimately uses binary correctness. Why not consider using the Reference Route for rubric-based scoring?
- Figure 2 in the paper is empty and should be replaced with an actual figure.
- It is recommended to provide more annotation details, such as expert backgrounds, annotation time, and inter-rater agreement.
Thank you for the review. Its three weaknesses ask about design choices that the submission states, and none identifies an error.
W1: Five rounds. Five is a fixed setting of the shared scaffold (Section 3), which we did not vary (Appendix E). The cap limits decision rounds, not calls: a round can issue several calls, and the cap resets for each of the five runs. Two results suggest the cap is not the main limit. An attempt's call count is almost unrelated to its correctness (r ≈ 0.03; Section 4.3), so attempts that make more calls are not more often right. And depending on the model, 9.6–40.4% of tasks are solved in some runs but not in most; for these tasks five rounds were enough to reach the correct answer, and the lost accuracy comes from inconsistency across runs.
W2: The toolkit. Every task comes with "checked tool sequences for obtaining" its answer (submission abstract): Section 2.2's reference-route validation kept only the tasks whose answers the tools could obtain (186 → 178). The toolkit is therefore sufficient for every task in the benchmark; we do not claim it covers cryptocurrency analysis in general. Appendix M lists all 23 tools. Table 6 suggests that agents under-use the tools rather than lack them: code execution accounts for 1.5% of 23,463 calls and web search for 49.6%, although many tasks require aggregation.
W3: Codex or Claude Code. A benchmark compares models under a fixed protocol. A vendor harness changes prompting, context management and built-in tools at once, and several of the 17 models have no such harness, so differences would no longer be attributable to the model. A stronger harness can be tested on the released tasks with the tools of Appendix M; that is future evaluation, not a flaw of the benchmark.
Q1: Route rubric. Routes are not unique. A route rubric would penalize valid alternative routes and give credit for progress toward a wrong number, so we score the final answer and use routes for diagnosis (Appendix N).
Q2: Figure 2. Thank you for flagging it. The figure used transparency masks that some PDF viewers render blank; it is now flattened, and two other figures with similar display problems were repaired.
Q3: Annotation. Section 2.2 and Appendix G report part of this. Ninety-two specialists each reviewed 50 candidates; each candidate received at least four ratings and was ranked by trimmed mean; the top 200 were deduplicated to 186 and validated down to 178. A ten-member expert team agreed with the evaluator on all 500 sampled decisions, and 16 participants completed a stratified 10% subset. Expert backgrounds, annotation time and inter-rater agreement are not reported; we will add the backgrounds and annotation time in the final version.
On the rating. The review rates soundness 2 ("minor or fixable"). Its weaknesses are design questions that the paper answers, and the one defect it identifies is a rendering problem, now fixed. We respectfully ask you to reconsider whether these support rejection.