acceptodds
Under review as a conference paper at ICLR 2027

Investment Research Agent Benchmark: Grounding Agent Evaluation in Real-World Tasks and Live Information

Abstract

As AI agents enter investment research workflows, their evaluation must reflect diverse professional objectives and changing evidence. This poses three challenges: task grounding, scoring alignment, and reference maintenance. To address these challenges, we introduce the Investment Research Agent Benchmark (IRAB), comprising 101 tasks adapted from a corpus of 252,015 practitioner requests. Focused primarily on Chinese markets, these tasks span 12 research intents, ranging from industry-chain transmission analysis to stock and fund screening. For each task, IRAB pairs a task-specific rubric with dated reference evidence maintained under explicit refresh policies. Each rubric selects and weights applicable criteria from an inventory of 27 reusable, expert-derived criteria. A multiplicative scoring rule then combines data capability, answer quality, and reliability, so that detected factual failures reduce the overall score. Using IRAB, we evaluate 58 hosted configurations of 42 base models. Descriptive comparisons show higher data timeliness and source authority in newer release cohorts, while case analyses reveal persistent factual and verification failures. Repeated-run analysis further identifies variation in cited sources, underscoring the need for evidence preservation. This work thus contributes a practitioner-grounded benchmark, a reliability-aware evaluation framework, and an empirical characterization of the capabilities and limitations of investment research agents.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Finance, Investment Research

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cxHXOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper is missing the mandatory AI use statement section required by the ICLR 2027 author guidelines.

Critical strengths

This paper is missing the mandatory AI use statement section required by the ICLR 2027 author guidelines.

Critical weaknesses

This paper is missing the mandatory AI use statement section required by the ICLR 2027 author guidelines.

Questions for authors

This paper is missing the mandatory AI use statement section required by the ICLR 2027 author guidelines.

1 comment

Dear Reviewer cxHX,

Thank you for pointing this out. We did disclose our AI use in the OpenReview submission form, but inadvertently omitted the corresponding statement from the main text. We will add the AI use statement to the paper in the revised version.

We would also welcome any further specific comments on the paper’s content, and are glad to address them during the discussion.

Best regards, Authors

Reviewer_bpg9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper describes the Investment Research Agent Benchmark (IRAB), a benchmark for evaluating AI agents on real-world investment-research workflows. IRAB contains 101 tasks adapted from a corpus of 252,015 requests submitted by institutional practitioners, covering 12 intents including company and sector research, factual lookup, event/mechanism analysis, valuation, screening, backtesting, and visualization. The benchmark primarily targets Chinese markets, with additional global tasks. Each task is associated with a subset of 27 reusable expert-derived criteria and dated reference evidence governed by explicit refresh policies.

The paper proposes DQR scoring, where data capability, answer quality and reliability are multiplied at the task level. Reliability is reduced by task-specific factual-failure penalties; in the extreme case, firing the sole penalty criterion makes the task score zero. The primary evaluator is Opus 4.8. Twelve experts reviewed judge outputs for 30 tasks during rubric development, with 82% of feedback marking the assigned score as reasonable; the authors explicitly note that this is calibration rather than held-out evaluator validation.

The empirical study evaluates 58 hosted configurations from 42 base models, reporting DQR, cost, execution time, token use, and tool calls. The paper further studies release-date trends, source selection, failure cases, repeated judge scoring, and repeated agent runs. A particularly interesting result is that identical-reference reruns can retrieve substantially different evidence: for 14 Opus 4.8 tasks rerun 12 days apart with unchanged gold evidence, median cited-document Jaccard overlap is only 0.12.

Critical strengths
  • Separating assessment criteria from dated evidence accommodates changing financial information without discarding the context of earlier evaluations. Refresh policies and archived versions preserve the reference basis used for a particular run. Retaining the sources cited by each report is especially useful when later retrieval cannot reconstruct the same evidence.
  • The evaluation provides a diagnostic view of research quality while ensuring that consequential factual failures affect the overall assessment. The evaluation distinguishes data capability, answer quality, factual failures, and resource use. Reliability penalties make consequential factual errors affect the overall assessment instead of disappearing within an average of positive criteria. The accompanying failure analyses connect scores to concrete problems such as numerical distortions, internal contradictions, and insufficient verification of retrieved claims.
Critical weaknesses
  • Some grading requirements are not stated in the task description, making scores depend on unstated expert expectations. For instance, in the East Money historical analysis task, full completeness credit requires monthly data, although the request does not specify a reporting frequency. A response can therefore lose credit for choosing a reasonable level of detail that differs from the expert reference. The resulting score mixes fulfillment of the stated research request with the ability to anticipate additional requirements.
  • Reasoning-angle coverage is defined by the explanatory mechanisms in the expert reference, but the treatment of a well-supported alternative explanation is unclear. Non-consensus insights and independent synthesis appear as separate criteria applied to selected tasks. These definitions do not specify when a different explanation fulfills the task's reasoning requirements and when it counts as missing coverage. This limits the interpretability of scores for reports that offer a defensible analysis differing from the reference.
Questions for authors
  • How do you determine which additional expert-defined requirements are necessary for a satisfactory answer when they are not stated in the request? Should requirements that materially change the work, such as monthly data resolution, be stated explicitly in the task?
  • How does the rubric distinguish omitting a required explanatory mechanism from presenting a well-supported alternative explanation?
1 comment

Thank you for the careful and positive review. Below we explain each point you concerned about.

1. Regarding requirements not stated in the task (Weakness 1 and Question 1). Thanks for this question. We would like to clarify that our intention is to assess an agent’s professional judgment when research requests leave some analytical choices unspecified. Real-world requests are rarely fully specified, but professional users have implicit expectations, and meeting them is what separates strong agents from weak ones.

  • Example. The East Money task asks for a 2012–2020 retrospective of the share price, PE/PS and operating metrics. An annual series would reduce the 2015 bull market, when the stock surged and crashed within a single year, to one data point, and that episode is the key to understanding this stock. Any experienced analyst would therefore plot monthly data without being told.
  • What we want to measure is whether an agent has this professional judgment and skill, not only whether it can follow explicit instructions. If every such convention were written into the prompt, all agents would follow the same checklist and the benchmark would mostly test instruction following.
  • These requirements are reviewed by experts for each task (Section 2.2) and graded in tiers, not pass/fail: annual values alone still earn a lower tier, not zero (Table 2).

2. Regarding alternative explanations (Weakness 2 and Question 2). We would like to clarify that a well-supported alternative explanation does not receive a low score. The rubric only checks whether the load-bearing mechanisms are covered, by meaning rather than wording.

  • Example. Suppose a task asks why a carmaker's gross margin rose, and the reference lists economies of scale from higher deliveries as a required mechanism.
    • A report that says "fixed costs were spread over more cars" covers it and gets full credit, even though the wording differs.
    • A report that also argues, with evidence, that cheaper battery materials helped is not penalized for the extra explanation.
    • A report that gives only the materials explanation and never mentions scale is marked as missing that mechanism, so it drops one tier (e.g. from "complete" to "partial"), but is not zeroed.
  • Evaluating insight involves a trade-off: once a view is written into a reference answer, it has usually already become consensus. So for insight-type criteria we do not write down "the" insight. Instead, the reference records the consensus view as a floor, and the judge checks whether the report goes beyond it with a well-supported conclusion: non_consensus_view gives credit only to views that are off-consensus, specific, material and correct.
  • Reasoning angles work the same way: the reference lists the consensus mechanisms, and a well-supported alternative is not treated as wrong; on tasks that use non_consensus_view, it earns additional credit.
  • We acknowledge this as a limitation. How an LLM judge should evaluate genuinely insightful ideas is still an open question, so we use these criteria sparingly: reasoning_angle_coverage appears in 10 of the 101 tasks and non_consensus_view in only 2.

3. Regarding the ethics concern. We have added an Ethics statement (page 10).

  • Each request was strictly de-identified, then rewritten so that only the research subject, time scope, constraints and analytical objective remain (Section 2.1). No task can be traced to a person, an institution or an actual investment decision.
  • Because tasks are graded against live evidence, what an agent produces on the benchmark also differs from any answer to the original request.

In addition, during the discussion phase we added the following experiments and data (all marked in blue in the revised PDF):

  • Independent expert scoring (Appendix B.2). Five buy-side analysts independently scored 45 reports; the judge agrees with a single expert more closely than two experts agree with each other (α 0.77 vs. 0.62).
  • Sensitivity analysis (Appendix B.4). Under the measured grading error, the leaderboard order is largely preserved (τ = 0.88), and all release-date trends hold even with three times the error.
  • Open-source harness pilot (Appendix C.4). Ten configurations run in both harnesses at the same time are ranked very similarly (τ = 0.78, ρ = 0.90).
  • Outdated-evidence analysis (Appendix E.5). 11.9% of reports rely on outdated core data, and these reports score far lower (25.1 vs. 67.2).

Thank you again for the constructive review.

Reviewer_2J42OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The Investment Research Agent Benchmark has 101 tasks from real investment-practitioner research requests related mostly to Chinese financial markets, spanning 12 types of research intents. Grading is done via expert-authored rubrics and a LLM-judge, through a multiplicative combination of data-capability scores, answer-quality scores, and reliability scores. A key aspect of this benchmark is the role of time/dates, and the information in key references is periodically refreshed on a schedule (in 69 tasks) as are the corresponding rubric criteria. Evaluating many models in a proprietary agent-harness that can use web-search, the authors find recurring failure modes: order-of-magnitude errors, conflicts across a long deliverable, and over-reliance on ready-made analyses.

Critical strengths

The benchmark relies on actual queries to a production agent, sensibly anonymized, which makes it more representative of real investment research.

The benchmark introduces the concept of reference-evidence refresh which enables it to deal with the key aspect of investment research: relying on up-to-date information.

Good coverage of domains under-covered by existing finance benchmarks (non-US financial markets).

Comprehensive evaluation of many models.

Critical weaknesses

For validating the grader (Appendix B), the authors asked experts to assess the judge’s assigned scores. Why not instead have experts assign their own scores independently and then report the judge’s accuracy against them? That is a far more standard way to validate LLM-judges and more trustworthy.

Furthermore, the expert acceptance rate is fairly low at 82%. What implications does this have for the benchmark? I didn’t see a sensitivity analysis of how robust benchmark conclusions are given this level of grader-accuracy.

It would be good to see a robustness analysis against other ways to combine the D / Q / R subscores. There seems to be no validation that your way of aggregating the scores correlates well with expert judgements of agent outputs.

The distinction between answer-quality subscore and reliability subscore should be better explained. What does a researcher conclude if their agent scores high on one but not the other?

How was the quality of the rubrics and reference answers evaluated/ensured? This paper does not convincingly establish the quality of its ground truth.

How does your grader check for the presence of hallucinated information in agent outputs, when this information is in statements that are not directly assessed by any rubric criteria?

The paper results are not reproducible due to the use of a proprietary harness. Why not also report some results in a standard open harness for comparison?

The reference-evidence refresh is an interesting idea, not a weakness, I am just concerned how you will ensure daily refreshes happen correctly. Could you describe the process that will ensure the refreshes run as specified? Somebody running this benchmark could easily get invalid results if it happens to not be properly refreshed at that time.

Fig 3 is not very informative, because the model IDs are hard to read and missing from B & C. Would use more types of markers and a legend to indicate specific models/families.

Questions for authors

The authors enable web search for agents running IRAB, but this gives agents many ways to cheat & reward-hack once your benchmark is public on the internet. What mitigations will you take to prevent that?

How often do agents fail due to relying on out-of-date information?

How did you decide on 12 research intents, and why these 12?

2 comments

Thank you for the detailed review and valuable suggestions; most of our new analyses come from them. Details are in the revised PDF (marked in blue); here we summarize the key messages.

1. Regarding the validation of the grader. We agree that independent expert scoring is the standard way to validate an LLM judge, and we have run such a study (Appendix B.2). Five buy-side analysts, none of whom took part in designing the rubrics or references, independently scored 45 reports from 15 tasks, three analysts per report. They used the same rubric and reference as the judge, and model identities were hidden.

  • Detailed results are in Table 6. Overall, the judge agrees with an individual expert at α = 0.77 (Krippendorff's α over all criterion scores), well above the agreement between two experts (α = 0.62). This echoes MT-Bench (Zheng et al., 2023), where GPT-4 agrees with human raters more often than they agree with each other (85% vs. 81%), and FinResearchBench II (arXiv:2607.12252), which finds the same pattern in financial research (Cohen's κ 0.69 vs. 0.62).
  • If we look only at which of two reports for the same task is better, the judge agrees with a single expert on 84% of pairs (two experts: 69%), and with the majority of the three experts on 98%. We therefore believe the judge's scores are representative of expert judgement.
  • We also acknowledge why we did not run this study at first. The rubric was designed for an LLM judge, and its criteria are not simple yes/no checks: data-verification criteria require comparing every figure in a report against the reference, and a report can contain more than a hundred, while other criteria require judging the authority of each source. Expert review is also how FinAutoRubric (arXiv:2609.35744) validates its rubrics, with three in-house analysts comparing them in a blind review. In our study, experts could only score these criteria by impression: they agree on the direction (α = 0.68) but give exactly the same score only 31% of the time. The experts also come from different sectors, and how readily they notice a distorted figure depends on whether the task falls in their own field. This lowers agreement between experts on analytical judgement and penalty verdicts (α = 0.36). Even so, on every criterion family the judge agrees with an expert more closely than two experts agree with each other.

2. Regarding the 82% acceptance rate and sensitivity. The 82% is the share of expert feedback in the final calibration round (Appendix B.1) that found the judge's existing scores reasonable. This is already above the 80% commonly recommended as the minimum acceptable agreement between raters (McHugh, 2012). It is also not a direct accuracy measure; the study in point 1 is.

We have added a sensitivity analysis (Appendix B.4 and Table 8, page 19; summarized in Section 2.5 and Section 3.2). We perturbed every criterion score with the grading error we measured and recomputed the leaderboard 1,000 times. The ranking is largely preserved (Kendall's τ = 0.82–0.88), and all ten release-date trends in Section 3 still hold, even when the error is tripled to the level at which two experts disagree with each other. Only configurations within a few points of each other may swap, and these should be read as ties.

3. Regarding other ways to combine D, Q and R. Thank you for this suggestion; we have added the analysis (Appendix A.4 and Table 5, page 15). We use a product because the three factors are not substitutes: as experts stressed during rubric design, a polished analysis cannot make up for an error in an important figure, and multiplicative aggregation limits this kind of compensation (OECD, 2008). Recomputing the leaderboard of the 58 configurations under alternative rules (subtracting penalties within D, or taking the mean, the product of means or the minimum of D, Q and R), the ranking changes little (Kendall's τ = 0.84–0.93), the top five stay the same, and at most one of the 944 configuration pairs separated by at least 10 points changes order. Only dropping R changes the ranking materially (τ = 0.74).

To validate against experts, we used the study in point 1. Because the formula was fixed in the submitted paper, before the study was run, this is an out-of-sample check. D·Q·R correlates with the experts' overall 1–7 rating at Spearman's ρ = 0.73, against 0.51 without R, and when the two rules order a pair of reports from the same task differently, the experts side with D·Q·R in 5 of 6 cases. We therefore believe that the exact combination matters little, but keeping reliability as a separate factor does.

4. Regarding Q vs. R. This follows from point 3. Q measures whether the report answers the question well (coverage, depth and a clear conclusion); R measures whether it avoids severe factual distortions and internal contradictions. Because the two are kept separate, they point to different problems:

  • High Q, low R: the report reads well and reaches a clear conclusion, but the conclusion rests on a wrong number or contradicts itself, for example a valuation argument built on a misstated quarterly profit. This is the most dangerous case for a practitioner, because it invites action. The agent needs better verification (Section 3.3).
  • High R, low Q: the facts are safe, but the report is shallow or misses parts of the question. The agent needs better analysis and coverage.

Because IRAB reports D, Q and R separately (Section 3.1), a researcher can see which of the two their agent needs to improve.

5. Regarding the quality of rubrics and reference answers. We agree this was not clear, and have added the process to Section 2.3.

  • Rubrics were derived from interviews with investment practitioners (Appendix A.2). Experts reviewed each task's criteria and weights, and their feedback was used to calibrate the grading anchors (Appendix B.1).
  • Reference answers are collected by a retrieval agent and checked by a separate verification agent against three different sources. Humans approve the first version of every reference and spot-check later refreshes.
  • Validation: in the new study (point 1), analysts who did not build the rubrics or references scored reports with them, and agree with the judge at least as closely as with each other.

6. Regarding hallucinations outside the rubric. Three criteria cover different kinds of claims:

  • Figures with a counterpart in the reference are checked by data_quality.
  • Important claims without a counterpart are covered by data_distortion, whose released definition explicitly includes hallucinated numbers, entities or dates stated as fact (example in Table 15, page 27).
  • Contradictions anywhere in the report are caught by data_inconsistency.

A boundary does remain: a fabricated detail that is plausible, peripheral and consistent with the rest of the report may go undetected. This is hard to avoid without verifying every sentence. We grade strictly by the rubric, and detection beyond the reference depends partly on the judge model's own knowledge. Every run keeps its cited sources for later audit.

7. Regarding reproducibility. The reference harness is open source. A pilot run before the leaderboard shows that it ranks configurations closely to the production harness (τ = 0.78, ρ = 0.90; Appendix C.4).

8. Regarding refreshes. Each evaluation uses the reference version valid on its date, all versions are archived, and a failed refresh is never used for scoring.

9. Regarding Figure 3. Revised, with model families marked across all panels.

Q1. Regarding web search and reward hacking. Web search is essential for investment research. Live references make gaming hard: answers go stale at each refresh, and figures are checked against the dated reference. We have not observed reward hacking.

Q2. Regarding outdated information. 11.9% of reports rely on outdated core data, falling from 18.8% for the oldest models to 6.8% for the newest, and these reports score far lower (Appendix E.5).

Q3. Regarding the 12 research intents. The 12 categories were not fixed in advance; they emerged from the practitioner requests. We clustered the requests with LLM assistance, and domain experts merged related clusters and named the results. This follows inductive content analysis, in which categories are derived from the data rather than from a preset taxonomy (Hsieh & Shannon, 2005). The number 12 is the outcome of this consolidation, not a target. The resulting categories, such as single-company analysis, sector research, event attribution and strategy backtesting, overlap with established active equity workflows (CFA Institute, 2026), while reflecting what practitioners actually ask a research agent to do. The task bank's intent mix tracks this real demand (r = 0.94, Figure 1; Section 2.1).

We hope this addresses your concerns and would be grateful if you could take these results into account.

Comments on the paper

1 comment

Thank you all for the reviews and the helpful suggestions. We have uploaded a revised PDF, with all new material marked in blue; we also fixed a number of minor logic and grammar issues throughout the paper. Here we summarize what we added during the discussion phase; details are in the paper and in our responses to each reviewer.

1. Independent expert scoring (Appendix B.2). Five buy-side analysts, none of whom took part in designing the rubrics or references, independently scored 45 reports with the same rubric and reference as the judge. The key messages are:

  • The judge agrees with an individual expert more closely than two experts agree with each other (α 0.77 vs. 0.62). This echoes MT-Bench (Zheng et al., 2023) and FinResearchBench II (arXiv:2607.12252).
  • Within a task, the judge agrees with the majority of three experts on which report is better in 98% of pairs.
  • The study is small because of the time limit. The rubrics were designed for an LLM judge, so experts could score the figure-checking criteria only by impression, which lowers agreement among them.

2. Sensitivity analysis (Appendix B.4). We perturbed every criterion score with the measured grading error and recomputed the leaderboard 1,000 times. The key messages are:

  • The leaderboard order is largely preserved (τ = 0.88), and configurations more than 10 points apart almost never swap.
  • All release-date trends in Section 3 still hold, even with three times the error.

3. Choice of aggregation rule (Appendix A.4). We explain why D, Q and R are multiplied and compare the product with alternative rules. The key messages are:

  • As long as reliability is kept, other rules (e.g., the mean or the minimum of D, Q and R) give almost the same leaderboard (τ = 0.84–0.93), with the same top five.
  • Dropping R changes the ranking materially and lowers agreement with the experts' overall ratings (Spearman's ρ 0.51 vs. 0.73).

4. Open-source harness pilot (Appendix C.4). Before building the leaderboard, we ran the same ten configurations in the open-source and production harnesses at the same time. The rankings agree closely (τ = 0.78, ρ = 0.90), which is why we ran the full leaderboard on the production harness.

5. Outdated-evidence analysis (Appendix E.5). 11.9% of reports rely on outdated core data, and these reports score far lower (25.1 vs. 67.2).

We also added Ethics, Reproducibility and AI use statements, and explained how the reference answers are built and verified (Section 2.3). We hope these additions address the main concerns, and thank you again for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.