acceptodds
Under review as a conference paper at ICLR 2027

FinRiskArena: Time Safe and Multilayer Evaluation of Financial LLM Decision Making

Abstract

Large language models (LLMs) are increasingly considered for high stakes financial decision support, yet conventional benchmark performance provides limited evidence of whether such systems are reliable. Financial risk evaluation must account for temporal validity, differences between imminent and latent risk, operational reliability, evidence grounded decisions, and the validity of the metrics used to assess these properties. We introduce **FinRiskArena**, a time safe and risk sensitive benchmark for multidimensional evaluation of financial LLM decision making. FinRiskArena contains 32,045 company–time episodes from Chinese A share firms constructed under strict point in time information constraints. It separates **imminent distress detection**, which retains legally available near term signals, from **latent distress forecasting**, which removes audit opinions and requires more than 90 days of prediction lead. The benchmark evaluates five complementary layers: **predictive discrimination and calibration, operational reliability, semantic evidence use, counterfactual behavior, and evaluator validity**. A strong structured baseline reaches an AUPRC of 0.439 in the imminent regime, falls to 0.162 after audit opinion information is removed, and reaches 0.047 in the latent regime, showing that superficially similar distress prediction tasks differ substantially in difficulty. LLMs retain meaningful risk ranking signal, but predictive performance does not consistently align with structured output reliability or semantic auditability. Counterfactual tests reveal heterogeneous behavior across perturbations, while a separately frozen blinded financial expert validation shows that selected automatic counterfactual scores diverge from expert semantic judgments. FinRiskArena therefore treats trustworthy financial LLM evaluation as a multilayer measurement problem in which predictive quality, operational compliance, semantic reliability, counterfactual behavior, and evaluator validity must be assessed separately.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, Financial Decision Making, Time Safe Evaluation, Reliability Auditing, Measurement Validity

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_atY4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces FinRiskArena, a benchmark of 32,045 company–year episodes from Chinese A-share firms. The task is to predict whether a firm will enter ST/*ST distress status. Inputs are restricted to information disclosed before an April 30 cutoff. The benchmark defines two tracks: an imminent track that keeps all information, and a latent track that removes audit opinions and requires a lead time of more than 90 days. It evaluates a logistic regression baseline and two LLMs along five layers: prediction, output validity, expert-rated semantic quality, counterfactual behavior, and validity of the automatic detectors. The authors conclude that these layers diverge and should be measured separately.

Critical strengths
  • The construction of the labels is careful: facts are aligned to their actual disclosure dates, and censored observations are handled explicitly instead of being labeled negative.

  • Invalid LLM outputs are kept in the end-to-end metrics rather than silently dropped.

Critical weaknesses
  • "Time safety" covers the inputs but not the model's memory. The test period (2023–2024, with outcomes extending into 2025) falls before the training cutoff of both evaluated LLMs. Removing firm names does not guarantee there's no re-identification, because exact financial figures of listed firms are widely available online. The model may therefore already know which firms were later designated ST, and the current design cannot separate genuine forecasting from recall. I suggest repeating the evaluation with models whose knowledge cutoff precedes the test period, or at least adding a re-identification or numeric-perturbation test.

  • Track I labels are largely rule-driven. The ST/*ST designation is mainly triggered by annual report indicators, such as negative net profit combined with revenue below a specified threshold, negative net assets, a disclaimer or adverse audit opinion, or failure to disclose the annual report on time. April 30 is the annual report disclosure deadline. The 108 positives removed from Track II suggest that roughly half of the 218 Track I positives are designated within 90 days of the cutoff. Therefore, Track I mostly measures whether a model can recognise that a rule has already been triggered, rather than if it can forecast distress. A baseline that directly applies the exchange listing rules is needed to show how much of the task involves genuine prediction.

  • The semantic audit is not quite transparent. The paper does not report how many experts took part, their qualifications, or inter-annotator agreement. Several dimensions have large shares of non-finalised labels (for example, only 129 of 222 for evidence support) without explanation.

Questions for authors

Please see the weakness section.

Reviewer_BcMqOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

FinRiskArena introduces 32,045 annual company-time episodes for Chinese A-share firms, using future ST/*ST entry as an institutional distress proxy. It defines an April 30 point-in-time cutoff and separates imminent detection, which retains legally available audit evidence, from a latent-risk track that removes audit opinions and excludes events within 90 days. The benchmark evaluates structured baselines and two frozen LLM deployments across predictive performance, output validity, expert-rated semantic support, six counterfactual perturbations, and validation of selected automatic evaluators. The results show large differences between the two risk regimes, between predictive and operational performance, and between automatic counterfactual flags and expert judgments.

Critical strengths
  1. Point-in-time task construction is treated as part of the benchmark definition. The paper distinguishes accounting periods from disclosure availability, excludes post-cutoff corrections and future target-revealing fields, retains censoring as unknown, and gives each episode traceable Fact IDs. The separate imminent and latent tracks also avoid treating near-event recognition and earlier warning as the same task. The authors explicitly state that Track II changes both the evidence set and event composition, so the performance gap is not presented as a causal estimate of removing audit opinions.

  2. The evaluation separates several failure modes and tests the evaluator itself. Predictive metrics are conditional on evaluable responses while JSON, schema, and action rates use all planned requests. The 222-response expert audit is labeled diagnostic rather than population-representative. For CF4 and CF6, the authors compare frozen detector outputs with a separate expert review, find substantial disagreement, and mark the detectors as not validated instead of retuning them on the validation cases. This is a useful and appropriately cautious benchmark contribution.

Critical weaknesses
  1. The central point-in-time guarantee is difficult to verify independently. Time-safe eligibility is a defining part of the benchmark, but the paper does not identify the underlying data sources and source vintages in enough detail to let readers check how disclosure dates and later corrections were handled. The reproducibility statement points to offline audit records and says experiment-ready materials will be released later. Until those materials are available, this leaves the benchmark's most important construction claim difficult to audit.

  2. The sampled LLM results depend on a selection and weighting procedure that is not clear enough to assess. Table 1 shows that all test positives are included, alongside 782 sampled negatives in each track, and the paper uses population-restoring weights. It does not explain how those negatives were selected or how the weights reflect their inclusion probabilities. Because valid-response counts also differ across models, it is difficult to tell how much of the reported comparison reflects model behavior versus sampling and response availability.

  3. The semantic audit is useful as a diagnostic, but its results have limited reach. The paper reports expert judgments on 222 responses and correctly says this is not a population-representative estimate. However, without a clear account of how those responses were selected and how consistently the experts applied the rubric, the reported semantic rates are hard to generalize even across the benchmark's evaluated outputs. This limits the strength of the paper's evidence about the prevalence of supported or directionally correct reasoning.

  4. The counterfactual results are not equally well validated across the six test families. The paper reports expert checks for CF4 and CF6 and finds that the corresponding automatic detectors do not align well enough with expert judgments to be considered validated. That is a valuable finding, but it also means the remaining detector-based results should be interpreted cautiously. As presented, the benchmark does not establish that its broader counterfactual scores reliably measure the intended behaviors across all six families.

Questions for authors
  1. Time-safe data construction: Which source datasets and source vintages underlie the statements, audit information, and ST/*ST transitions? Could you show one or two concrete episodes with the filing dates used to determine eligibility and explain how later corrections are kept out of earlier snapshots?

  2. LLM sample interpretation: How were the 782 negative episodes in each track selected, and how do their inclusion probabilities determine the population-restoring weights? Given the different valid-response counts across models, can you report uncertainty for the main comparisons on episodes evaluable for both models?

  3. Scope of the expert audit: How were the 222 responses selected, and what was the experts' agreement before adjudication? Could you clarify which conclusions the audit supports beyond this diagnostic sample, especially for the reported rates of evidence support and direction correctness?

  4. Meaning of counterfactual scores: Since expert validation found that the CF4 and CF6 detectors were not reliable enough to validate, how should readers interpret the results for the other counterfactual families? Which conclusions are supported by direct expert review, and which depend on automatic detector outputs?

Reviewer_3h13OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Using a large sample of Chinese firm data, the authors construct a benchmark for evaluating financial LLM decision-making. Instead of using only predictive accuracy, the benchmark additionally considers operational reliability, semantic evidence use, counterfactual behavior, and evaluator validity. A model with better predictive accuracy is not necessarily trustworthy.

Critical strengths

[1] I like the authors’ basic idea that financial decision-making should be evaluated not only on predictive accuracy but also on other factors such as semantic evidence and counterfactual behavior.

[2] The authors construct a sufficiently large sample to support their claims.

[3] I applaud the authors' efforts to collect a large sample of human answers.

Critical weaknesses

[1] I am not convinced by the paper’s incremental contribution. Constructing a point-in-time dataset, predicting financial distress, evaluating structured outputs, conducting counterfactual evaluations, and performing human expert validation are all important topics, but they have already been extensively studied. The contribution of this study is combining these components into a unified framework, which I believe does not clear the bar for ICLR. The fact that audit opinions carry substantial informational value is also not novel in the accounting literature (see Gutierrez et al., 2020, for example). The authors should clarify and sharpen their contribution.

[2] In CF6, human experts did not assign any FAIL cases. Therefore, one cannot assess the detector’s ability to identify real failures based on this set of ground-truth labels.

[3] Although this is a minor comment (which I generally do not make), it is difficult to generalize the authors’ claims and benchmark with only two models tested. At the very least, the authors could test several more frontier commercial models.

[4] Modern LLMs can enforce structured outputs in many other ways. One can use schema-based function calling and even retry after initial validation. After 2–3 retries, Qwen’s performance of 73.6% will likely improve substantially. Therefore, the authors should try other methods to enhance this step or otherwise establish why one-shot formatting is an economically meaningful measure.

[5] I am more interested in the failure cases. Why do human experts assess LLMs’ reasoning as mediocre? Can the authors identify some systematic characteristics of these answers? Are there any prompting techniques or deterministic transformations that can be applied to the answers to avoid these common mistakes?

Reference

Gutierrez, Elizabeth, et al. "Do going concern opinions provide incremental information to predict corporate defaults?" Review of Accounting Studies 25.4 (2020): 1344–1381.

Questions for authors

Please respond to my comments in the Weakness section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.