acceptodds
Withdrawn from ICLR 2027

EPSBench: Benchmarking LLM Agents for Earnings Forecasting

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Quarterly earnings forecasting requires an agent to connect financial evidence to the income-statement line items that produce a company's reported earnings. We introduce EPSBench, a benchmark for studying this ability across 149 firm-quarters from 86 US companies. Agents read a fixed collection of filings, earnings-call transcripts, and news, then estimate company-specific income-statement drivers with cited evidence. A shared accounting calculator converts these estimates into GAAP diluted earnings per share (EPS), providing a common basis for comparing forecast assumptions. We evaluate six models under a common harness, assessing point forecast error relative to date-matched analyst consensus and checking whether their prediction intervals achieve the stated coverage. Three models show no detectable paired difference in point forecast error from consensus, and three have higher error. Across all six models, nominal 80% intervals contain reported EPS in only 37-70% of runs that return an interval, and most misses fall above the upper bound because the forecasts run low. The driver records trace most of the error gap between models to operating-income estimates.

Primary area:
datasets and benchmarks
Keywords:
large language model agents, financial forecasting, agent evaluation, forecasting benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_8Fh9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EPSBench, a benchmark for evaluating LLM agents on quarterly earnings forecasting. Agents estimate income-statement drivers using financial filings, earnings-call transcripts, and news, and a shared calculator converts these estimates into EPS forecasts. The authors evaluate six LLMs on 149 firm-quarters and compare their performance with analyst consensus and historical baselines. The results show that leading models perform similarly to analyst consensus, but all models have poorly calibrated prediction intervals.

Critical strengths
  1. The accounting-based framework is interesting. Breaking EPS predictions into individual financial drivers makes it easier to understand where forecasting errors come from.
  2. The paper provides useful experiments across six LLMs, including uncertainty calibration, information-source ablations, and harness comparisons.
Critical weaknesses
  1. The accounting-based framework is interesting. Breaking EPS predictions into individual financial drivers makes it easier to understand where forecasting errors come from.
  2. The paper provides useful experiments across six LLMs, including uncertainty calibration, information-source ablations, and harness comparisons.
Questions for authors
  1. Could you include stronger statistical or machine-learning forecasting baselines?
  2. How would the same LLMs perform if they directly predicted EPS without the accounting-based framework?
  3. How do you verify that the cited evidence actually supports the driver estimates?
Reviewer_arfbOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a language-model-based earnings prediction pipeline. The pipeline combines firm-specific income-statement drivers and uses earnings call transcripts and filings as input. The pipeline achieves an accuracy that is comparable to that of human analysts.

Critical strengths

[1] The authors' approach to constructing company-specific accounting graphs is clever and aligns well with practice.

[2] The paper is generally well written and easy to follow.

Critical weaknesses

[1] Given that more than 2,000 companies are traded in the US stock market, 86 companies are not sufficient to represent the universe. To establish generalisability, the authors need to expand the universe or at least confirm that their sample is representative of the entire universe (e.g., market cap comparison). This is concerning because the consensus PDE of 0.11 is low, compared to what one would get in practice. Also, the companies listed in Table 5 seem to be large-cap firms. If the sample turns out to be not representative of the universe, the authors should clearly disclose this as a limitation.

[2] In general, analysts have a broader source of information (e.g., private communications with managers, internal research). It is surprising that LLMs, with an apparent penalty in information sources, achieve a comparable accuracy. Therefore, it is important to understand when and why these models excel. To do this, the authors can consider cross-sectional or time-series splits based on economic variables (e.g., when economic uncertainty is high, do the analysts perform relatively better than models?).

[3] Financial news often contains analyst forecast revisions and consensus values. If the model is receiving these values directly, it may anchor to the specific numbers, which may explain why the model's PDE is similar to that of consensus. Please verify that the news articles or transcripts you use as model inputs do not include these numbers, and if so, the authors should redact them.

[4] While I like the authors' 'accounting graph' approach, the authors need to benchmark the model's performance against other methodologies. For instance, (i) the model can be asked directly to come up with a point estimate (without the graph) and the interval; (ii) the authors may train a tabular machine learning model based on firms' historical financial statements and use it to predict EPS; (iii) the model follows the same process but the authors restrict the use of the shared calculator.

[5] Relatedly, I am also confused by the paper's intended contribution. What do the authors wish to establish and emphasise? Is it the pipeline itself or the model's performance? Or, based on the authors' writing, is it 'a common evaluation protocol linking dated evidence and driver estimates to point and interval scores'?

A huge line of finance and accounting literature uses diluted EPS as a benchmark. Numerous papers compare their prediction methods with actual EPS. Analysts' forecast consensus is almost always benchmarked against realised EPS. That said, I cannot agree with the authors' stated contribution.

If the authors wish to emphasise the pipeline itself, they must show that it's the firm-specific pipeline that improves the forecast accuracy. In its current form, I can only see that recent commercial models do a reasonable job in earnings forecasting, which is what we already know.

Questions for authors

Please respond to the suggestions/comments in the Weakness section.

Reviewer_BExAOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces EPSBench, a benchmark for evaluating LLM agents on quarterly EPS forecasting using financial filings, earnings calls, and news. It evaluates six LLMs on 149 firm-quarters and finds that leading models achieve errors comparable to analyst consensus, while their uncertainty estimates remain poorly calibrated.

Critical strengths

The structured accounting framework makes model predictions interpretable and allows errors to be traced to individual financial drivers.

Critical weaknesses
  1. Missing essential baselines. EPS forecasting has been extensively studied using statistical methods and smaller machine learning models. The paper compares only LLMs, historical EPS, and analyst consensus, without evaluating established forecasting approaches. This makes it difficult to assess whether expensive LLM agents provide any meaningful improvement over simpler methods.

  2. Limited novelty. The main findings largely confirm that stronger LLMs perform better and struggle with uncertainty calibration. Without comparisons to existing forecasting methods, the contribution beyond benchmark construction is unclear.

Questions for authors
  1. Why are established statistical and machine learning EPS forecasting methods excluded from the evaluation?

  2. Can the authors demonstrate that LLM agents outperform simpler, substantially cheaper forecasting models under comparable information constraints?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.