- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces FinRiskArena, a benchmark of 32,045 company–year episodes from Chinese A-share firms. The task is to predict whether a firm will enter ST/*ST distress status. Inputs are restricted to information disclosed before an April 30 cutoff. The benchmark defines two tracks: an imminent track that keeps all information, and a latent track that removes audit opinions and requires a lead time of more than 90 days. It evaluates a logistic regression baseline and two LLMs along five layers: prediction, output validity, expert-rated semantic quality, counterfactual behavior, and validity of the automatic detectors. The authors conclude that these layers diverge and should be measured separately.
Critical strengths
-
The construction of the labels is careful: facts are aligned to their actual disclosure dates, and censored observations are handled explicitly instead of being labeled negative.
-
Invalid LLM outputs are kept in the end-to-end metrics rather than silently dropped.
Critical weaknesses
-
"Time safety" covers the inputs but not the model's memory. The test period (2023–2024, with outcomes extending into 2025) falls before the training cutoff of both evaluated LLMs. Removing firm names does not guarantee there's no re-identification, because exact financial figures of listed firms are widely available online. The model may therefore already know which firms were later designated ST, and the current design cannot separate genuine forecasting from recall. I suggest repeating the evaluation with models whose knowledge cutoff precedes the test period, or at least adding a re-identification or numeric-perturbation test.
-
Track I labels are largely rule-driven. The ST/*ST designation is mainly triggered by annual report indicators, such as negative net profit combined with revenue below a specified threshold, negative net assets, a disclaimer or adverse audit opinion, or failure to disclose the annual report on time. April 30 is the annual report disclosure deadline. The 108 positives removed from Track II suggest that roughly half of the 218 Track I positives are designated within 90 days of the cutoff. Therefore, Track I mostly measures whether a model can recognise that a rule has already been triggered, rather than if it can forecast distress. A baseline that directly applies the exchange listing rules is needed to show how much of the task involves genuine prediction.
-
The semantic audit is not quite transparent. The paper does not report how many experts took part, their qualifications, or inter-annotator agreement. Several dimensions have large shares of non-finalised labels (for example, only 129 of 222 for evidence support) without explanation.
Questions for authors
Please see the weakness section.