acceptodds
Under review as a conference paper at ICLR 2027

FinRiskAtlas: A Decision-Aligned Benchmark for Financial Risk Review and Evidence Seeking

Abstract

Financial risk review requires more than knowledge of financial rules: a model must produce supported outputs from available evidence and decide when to seek additional information. We introduce FinRiskAtlas, an expert-reviewed Chinese-language benchmark for evaluating these complementary capabilities. Its static component contains 9,742 instances across 42 knowledge families and eleven review operations. Each operation is specified before instance construction by its visible evidence, decision object, required output, and scoring rule. FinRisk-Ask complements these tasks with offline evaluation of 680 pre-action states reconstructed from 104 professional review trajectories. Models receive only the record and history available before the reviewer acted, choose Ask or Proceed, and generate one focused evidence request when asking. Evaluation separates agreement with recorded reviewer actions from request alignment with expert-verified evidence needs. Later observations support target verification but remain hidden during inference. Across 33 model configurations, the knowledge leader attains the highest reported score on only one of the eleven review operations; selecting it for information extraction leaves an 18.01-point gap to the operation leader. On FinRisk-Ask, well-targeted requests can coexist with weak agreement on recorded decisions to proceed. FinRiskAtlas provides operation-specific profiles and evidence-seeking diagnostics for comparing models across review tasks.

Primary area:
datasets and benchmarks
Keywords:
large language models, financial risk review, decision-aligned evaluation, operation-centric evaluation, evidence acquisition

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_8ma3OpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces FinRiskAtlas, a Chinese benchmark for financial risk and compliance review. It combines a static suite of knowledge and review-operation tasks, each defined by an explicit operation contract, with FinRisk-Ask, which replays pre-action states from real enterprise review trajectories and asks models to decide whether to ask for more evidence or proceed. Evaluating many models zero-shot, the authors argue that aggregate knowledge scores do not identify the best model on each review operation, and that evidence-request quality and ask/proceed agreement are led by different models.

Critical strengths
  1. FinRisk-Ask is a well-constructed offline evaluation of evidence-seeking. It is built from real professional trajectories with later evidence withheld, has clean metric decomposition and an expert audit, and such data are scarce.
  2. The static benchmark is documented with unusual care, including operation contracts, native scorers and second-pass audits.
Critical weaknesses
  1. A single judge from one model family scores all generation and request-alignment results, and that family leads several tasks, with no test for family bias; this confounds the corresponding conclusions.
  2. Conclusions rest on point-estimate leader comparisons with small margins and unreported repetitions, so the "operation-specific selection" message is weakly supported. That no single model leads every operation is also expected for heterogeneous tasks, and the repetitive text leaves key facts hard to find.
Questions for authors
  1. Did you check the judge for self-preference, for example by re-scoring the leaders with a judge from a different family?
  2. How many repetitions were run per configuration, and how stable are the leaders across them?
  3. Can you rule out that evaluated models were trained on the internal collections used for the knowledge items, and what exactly will be released?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 8ma3 (Part 1/2)

We sincerely thank you for your careful review and for recognizing the value of the benchmark. Most additions are in the appendix, with brief updates to the main text. We address your questions on evaluator family bias below, followed by inference repetitions and data release in Part 2/2.

Q1. Evaluator family bias

To examine evaluator family bias, we added Qwen3.7-Max to compare model rankings and Claude-Opus-4.7 to rescore the two leading disputed-issue configurations. Rankings remain strongly associated across evaluators, and DeepSeek-V4-Pro also scores higher than Nemotron-3-Super under Claude.

Qwen3.7-Max and the original DeepSeek-V4-Flash evaluator scored identical responses from 26 configurations on 100 items per generation task and 120 recorded Ask states from 102 trajectories. Candidate identities were hidden, both evaluators' own configurations were excluded, and inputs, rubrics and aggregation were identical.

Table T7a. Cross-family ranking agreement on common evaluation samples.

OperationConfigurationsSamplesSpearman ρ
Legal-judgment generation261000.956
Decision-view generation261000.928
Disputed-issue generation261000.909
Request alignment (ERA)261200.988

In the paper's request-alignment results, the ERA leader is Ling-2.6-1T at 79.76 and the CRA leader is Qwen3.7-Max at 87.67; neither belongs to the DeepSeek family.

For disputed-issue generation, we rescored DeepSeek-V4-Pro and Nemotron-3-Super on all 500 items with both Claude-Opus-4.7 and DeepSeek-V4-Flash. Their paper scores are 77.12 and 76.67. Claude belongs to neither candidate's family. Both evaluators received the same responses, references and rubric and used the paper's aggregation.

Table T7b. Additional evaluator scores for the two leading disputed-issue configurations.

EvaluatorDeepSeek-V4-ProNemotron-3-SuperDifference95% paired CI
DeepSeek-V4-Flash76.4574.092.35[−1.11, 5.77]
Claude-Opus-4.770.3661.199.16[5.71, 12.47]

The 95% intervals use 10,000 paired item-bootstrap samples. DeepSeek-V4-Pro retains the higher score under both evaluators. The difference increases from 2.35 points under DeepSeek-V4-Flash to 9.16 under Claude, whose interval excludes zero.

The comparison supports the leading pair's ordering under an evaluator outside both candidates' families, while showing that the size of the difference depends on the evaluator. Appendix D.5 now presents both analyses. Appendix F.2.1 uses the same DeepSeek–Qwen scores to check Figure 5(c): the positive generation–request associations for legal judgment and decision view persist when different evaluators score the two outputs.

The structured-operation results provide additional evidence independent of semantic evaluation, including the repeated extraction comparison in Part 2/2.


To be continued in Part 2/2.

AuthorsOpenReview ↗

Response to Reviewer 8ma3 (Part 2/2)

Continued from Part 1/2.


Q2. Inference repetitions and leader stability

In the submitted results, each of the eight local vLLM configurations used five inference runs per task, with scores averaged across runs; each of the 25 API configurations used one run. Appendix B.2 now lists these counts for all 33 configurations.

We added three runs per configuration for the main extraction comparison and two close leader comparisons, covering all 249 extraction items, 500 person-matching items and 445 case-classification items. Within each task, configurations used the same inputs and scorer, with inference settings fixed across each configuration's three runs. Table T8 separates the paper scores from the additional runs and reports the mean and sample standard deviation of the three new scores.

Table T8. Paper scores and three additional inference runs.

OperationConfigurationPaper scoreRun 1Run 2Run 3Mean ± SD
Information extractionGPT-5.6-Luna85.9785.5784.6886.1785.47 ± 0.75
Information extractionGemini-3-Flash-preview67.9665.3766.3766.7766.17 ± 0.72
Person matchingKimi-K387.0086.4086.4086.0086.27 ± 0.23
Person matchingGPT-5.6-Sol86.0086.2085.8085.8085.93 ± 0.23
Case classificationKimi-K2.588.0988.0987.1985.8487.04 ± 1.13
Case classificationGPT-5.6-Sol87.4287.1987.6486.9787.27 ± 0.34
Case classificationKimi-K387.1987.8788.7686.5287.72 ± 1.13

The large extraction gap persists across the additional evaluations: all three Luna scores exceed all three Gemini scores. The new mean gap is 19.30 points, with a paired 95% interval of [16.26, 22.50].

Person matching shows a smaller separation. Kimi-K3 scores higher in each run, with a mean advantage of 0.33 points and an interval of [0.00, 0.73]. In case classification, the three means span 0.67 points, each configuration ranks first in one run, and all pairwise difference intervals include zero. We accordingly describe these configurations as a closely matched leading group.

The paired intervals use 10,000 shared item-bootstrap samples, retaining all three responses per sampled item and preserving field-weighted scoring for extraction. They measure item-sampling uncertainty; the reported standard deviations summarize variation across runs.

We agree that different tasks can favor different models. Our contribution is to quantify the performance loss from Knowledge-based selection and show that the main extraction gap persists across runs. The close comparisons identify where exact ordering is unstable. Section 5 and Appendix B.6.1 now report these distinctions and the additional results.

Q3. Training contamination and release scope

The Trade Document Collection and Risk Document Collection have not been publicly released or provided as training data to the teams responsible for the evaluated models, including teams within our institutions. We used these collections for benchmark construction and did not use either to train or fine-tune any evaluated configuration.

FinRisk-Ask is derived from non-public enterprise review trajectories. For items derived from public sources, including judicial records, overlap with undisclosed model pretraining data remains unmeasured, as stated in Appendix H.

We will release the complete evaluation-ready benchmark and evaluation code immediately upon acceptance. Table T9 distinguishes the materials provided with the submission from the planned release.

Table T9. Materials provided with the submission and planned release.

MaterialIncluded in submissionPlanned release
Static evaluation dataComplete inputs and references for 249 extraction and 58 quantitative-reasoning items; 292 illustrative examplesFull static benchmark inputs and references across the 9,742 items
FinRisk-Ask dataAll 680 recorded-action labels; 1,360 action/credit records for two configurations; six display cardsAll 680 de-identified pre-action contexts and recorded actions; evidence targets for the 583 recorded-Ask states
Evaluation frameworkReported score matrix, operation contracts, parsing/scoring utilities and analysis scriptsTask/evaluator prompts, parsers, scorers, rubrics and scripts for running the published protocol

Thank you again for your recognition and constructive feedback on our work.

Reviewer_ZVwPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces FinRiskAtlas, an expert-reviewed Chinese-language benchmark for financial risk and compliance review. It contains 9,742 static instances covering 42 domain knowledge families and 11 review operations, and each operation fixes the available evidence, the required output and the scoring rule. FinRisk-Ask adds 680 pre-action states drawn from 104 professional review trajectories to evaluate whether a model requests additional evidence, and whether its request targets the expert-verified evidence needs. The model sees only the pre-action record, chooses between Ask and Proceed, and writes one evidence request when it chooses Ask. Across 33 model configurations, the paper finds divergence between knowledge rankings and operation performance, and it separates agreement with the recorded reviewer action from the alignment of evidence requests.

Critical strengths

1.The benchmark is carefully constructed. Six domain experts defined the taxonomy, reviewed items, and adjudicated disagreements. Construction was checked by blinded second-pass audits, and these audits report high inter-annotator agreement. FinRisk-Ask is the most distinctive part of the paper. It reconstructs the information available before each recorded reviewer action, keeps later evidence hidden from the model, and scores the decision to ask separately from the content of the request. This separation is the right design, because an Ask decision can be correct while the request still misses the evidence that is actually needed.

2.The experiments and visualizations provide useful diagnostic guidance for operation-specific model selection in financial review workflows. The comparison of 33 configurations shows that a model's performance on domain knowledge and its performance on review operations can diverge. For example, two configurations with nearly identical knowledge scores (Gemini-3-Flash-preview at 80.69 and Kimi-K3 at 80.68) differ by 14.10 points on applicable-provision selection and by 8.47 points, in the opposite direction, on legal-outcome prediction. Reporting AskR, ProceedR, ERA and CRA separately also shows that models can write relevant requests yet often disagree with the recorded decisions to proceed. These results are informative for choosing models for specific review operations.

Critical weaknesses

1.The need for this benchmark, relative to existing Chinese financial and legal benchmarks, requires stronger justification. Methodologically, the operation contract formalizes a well-specified task definition (input, decision, output format and metric), and the evaluation relies on zero-shot prompting, deterministic task-specific parsers and a rubric-based LLM judge. The contribution therefore rests mainly on the data and on what the data reveal. However, the Applied Review layer consists largely of legal-dispute material (Table 5), and its task formats overlap with existing Chinese legal benchmarks such as LawBench (Fei et al., 2024), while risk classification accounts for only 366 of the 3,561 instances. The paper does not empirically compare model rankings on FinRiskAtlas with rankings on existing Chinese financial or legal benchmarks (e.g., CFinBench, FinEval, CNFinBench or LawBench) to show what it measures beyond them.

2.The practical significance of the FinRisk-Ask evaluation results remains uncertain. FinRisk-Ask uses single-step offline replay over an imbalanced set of 583 recorded Ask states and 97 Proceed states drawn from 104 trajectories. Although balanced accuracy (BAcc) accounts for class imbalance, it does not reflect the potentially different operational costs of unnecessary evidence requests and premature proceeding. Furthermore, Evidence-Request Alignment (ERA) evaluates request quality only on recorded Ask states and does not penalize unnecessary requests on Proceed states. Consequently, models can obtain high ERA scores by frequently requesting evidence without reliably identifying when sufficient information is already available. For example, Ling-2.6-1T achieves the highest ERA (79.76) but a BAcc of only 51.89, close to the constant-action baseline of 50. Although Appendix D.2 independently audits the acceptability of recorded actions, it does not provide a human performance baseline under the same evaluation protocol. Even the best model achieves a BAcc of only 59.46, making its practical performance difficult to interpret. Finally, evidence requests are not executed, so the benchmark cannot determine whether correctly targeted requests improve subsequent review decisions. These limitations do not invalidate FinRisk-Ask as an offline diagnostic benchmark, but they restrict its conclusions to agreement with recorded actions and evidence-request alignment rather than the effectiveness of evidence-seeking decisions in actual review workflows.

Questions for authors
  1. Could you report the correlation and the differences between model rankings on FinRiskAtlas and rankings on existing Chinese financial or legal benchmarks (e.g., CFinBench, FinEval, CNFinBench or LawBench), to clarify what FinRiskAtlas measures beyond them?

  2. Beyond the action-acceptability audit in Appendix D.2, could you report a human BAcc baseline in which experts independently choose Ask or Proceed from the same pre-action context under the model protocol?

  3. FinRisk-Ask uses recorded reviewer actions as ground truth, but a historical decision to Ask or Proceed does not necessarily represent the optimal action given the available evidence. Different reviewers may reasonably make different decisions, particularly when evidence acquisition involves costs or delays. Although Appendix D.2 audits action acceptability, how can the authors establish that agreement with historical actions is a meaningful measure of evidence-seeking quality rather than behavioral imitation? In particular, could they evaluate whether alternative model actions are independently judged acceptable or preferable, even when they disagree with the recorded action?

4 comments
AuthorsOpenReview ↗

Response to Reviewer ZVwP (Part 1/2)

Thank you for recognizing the benchmark's construction and diagnostic value. We added comparisons with existing benchmarks, a matched human baseline, and an assessment of acceptable alternative actions. We address the benchmark comparisons here and the human evaluation in Part 2/2.

Q1. Added value over Chinese financial and legal benchmarks

Across eight configurations, LawBench and FinEval rankings correlate strongly with FinRiskAtlas Knowledge and several review operations, but show weaker associations with person matching, risk classification and recorded-action BAcc.

We evaluated eight model versions from the paper using the external benchmarks' official zero-shot prompts and scoring methods. LawBench uses 100 fixed examples from each of 20 tasks, totaling 2,000 items, with equal task weights. FinEval Financial Academic Knowledge uses all 1,151 public validation items and pooled accuracy. Each external evaluation uses one inference pass per model; FinRiskAtlas scores are those reported in the paper.

Table T4a. Rank correlations between external benchmarks and FinRiskAtlas across eight configurations.

FinRiskAtlas metricLawBench ρFinEval Academic ρ
Knowledge0.8810.905
Case classification0.9290.905
Information extraction0.8570.833
Applicable-provision selection0.8330.810
Person matching0.1680.275
Risk classification0.3070.258
Recorded-action BAcc−0.0240.095

Comparisons between related LawBench tasks and FinRiskAtlas operations also show differences in model ordering.

Table T4b. Rank correlations between analogous LawBench tasks and FinRiskAtlas operations.

LawBench taskFinRiskAtlas operationSpearman ρ
3-1: Article predictionApplicable-provision selection0.690
2-6: Entity extractionInformation extraction0.143

These task-level comparisons use the same eight configurations and 100 LawBench items per task. Uniform normalization of entity-label formatting changes the extraction correlation from 0.143 to 0.214.

The evidence-seeking comparison gives a concrete example. Seed-2.0-Lite scores higher than Qwen3-32B on LawBench, 62.61 compared with 55.56, and FinEval, 92.44 compared with 84.80. Qwen3-32B instead scores higher on recorded-action BAcc, 58.19 compared with 44.36, a 13.83-point difference. The weak BAcc correlations persist when Qwen3-1.7B and Qwen3-8B are excluded and when FinEval uses subject-macro averaging.

Applied Review includes substantial legal analysis: 366 risk-classification items and 3,195 items for provision selection, outcome prediction and generated case analyses. These materials cover financial and commercial disputes, including lending, insurance, securities, payments and trade. Each operation is reported separately, with equal operation weights in the descriptive layer mean. FinRisk-Ask adds actions and evidence requests on 680 pre-action states from 104 professional trajectories.

The contribution lies in the reviewed data, operation-specific performance differences and trajectory-based evidence-seeking evaluation. Section 5.2 and Appendix F.5 report the external comparisons, full model scores, related-task comparisons and sensitivity analyses.


Continued in Part 2/2.

AuthorsOpenReview ↗

Response to Reviewer ZVwP (Part 2/2)

Continued from Part 1/2.


Q2. Expert BAcc under the matched evaluation protocol

Three financial risk-control experts from our team, each with at least three years of professional experience, independently chose Ask or Proceed under the model protocol. Their mean BAcc was 61.66, with a sample standard deviation of 0.29 across experts.

All experts evaluated 197 states from 103 trajectories: the last recorded Ask state from each of 100 selected trajectories and all 97 recorded Proceed states. They saw the original pre-action contexts, with recorded actions, model responses and later evidence withheld. Kimi-K3 and Qwen3-32B each completed one additional run on these states. Table T5 reports this comparison separately from the paper's results on the full evaluation set.

Table T5. Expert and model action agreement on the same 197 states.

Expert / modelAskRProceedRBAccBAcc 95% CI
Expert mean46.0077.3261.66[57.80, 65.55]
Expert 152.0071.1361.57[57.00, 66.21]
Expert 258.0065.9861.99[55.93, 68.08]
Expert 328.0094.8561.42[57.12, 65.89]
Kimi-K379.0024.7451.87[46.88, 56.95]
Qwen3-32B50.0057.7353.87[47.55, 60.23]

Mean expert BAcc exceeds Kimi-K3 by 9.79 points, with a paired 95% interval of [5.03, 14.40], and Qwen3-32B by 7.79 points, with an interval of [1.49, 14.08]. Experts have higher mean Proceed recall than both models. Despite similar BAcc scores, the experts differ in their action choices: pairwise agreement ranges from 69.54% to 77.16%, with Fleiss' κ = 0.401. All intervals use 10,000 shared bootstrap samples of the 103 trajectories. These scores provide a human reference for action agreement on the same states.

Request quality is assessed separately. On recorded Ask states, a valid, focused request earns full credit by directly addressing any one expert-verified need in the target set; historical wording is not required. Section 4.2 and Appendix C.4 define this rule. ERA averages request credit over recorded Ask states. Requests on recorded Proceed states reduce ProceedR and BAcc but do not enter ERA. Thus, Ling-2.6-1T's ERA of 79.76 is interpreted alongside its ProceedR of 7.22 and BAcc of 51.89.

Learn-to-Ask [1] studies questioning and stopping from offline expert logs; QuestBench [2] evaluates question selection for underspecified reasoning tasks. FinRisk-Ask separates action agreement and request targeting at common professional review states. Section 5.4 and Appendix D.4 report the new human comparison.

Q3. Recorded-action agreement and acceptable alternatives

We independently assessed whether Ask and Proceed were acceptable, including actions that differed from the recorded decision. Experts accepted 33 of Kimi-K3's 94 disagreements and 34 of Qwen3-32B's 91 disagreements.

After the action-choice experiment, the three experts separately assessed both actions on the same 197 states using only pre-action contexts. Recorded actions, model outputs and later evidence remained hidden. Both actions could be accepted; acceptance or rejection required at least two explicit judgments in agreement.

Table T6. Expert acceptance of model actions on 197 states.

ConfigurationAll statesRecorded AskRecorded ProceedDisagreement acceptance95% CI for disagreements
Kimi-K367.01%84.00%49.48%35.11%[25.74%, 45.16%]
Qwen3-32B65.99%70.00%61.86%37.36%[27.96%, 47.25%]

The overall and branch rates use 197, 100 and 97 states, respectively. Two unresolved Kimi judgments and one unresolved Qwen judgment remain in these denominators. Disagreement intervals use 10,000 shared trajectory-bootstrap samples over the 103 trajectories.

The three-expert majority assessment accepted 163 of 197 recorded actions and both actions on 20 states. The earlier two-expert assessment in Appendix D.2 accepted 184 recorded actions and both actions on 16 states after resolving disagreements. Its ranking-sensitivity analyses use those final judgments.

Under this independent assessment, both models' actions are accepted less often on recorded Proceed states than on recorded Ask states. BAcc reports agreement with recorded professional decisions; the expert assessment separately evaluates action acceptability. Section 5.4 and Appendix D.4 report these results. Evaluating evidence-acquisition costs and effects on subsequent decisions remains future work.

[1] Wei et al. Grounded in Reality. ICML 2026.

[2] Li et al. QuestBench. NeurIPS 2025.

Thank you for your substantial effort, especially the external benchmark comparisons, the human expert baseline (~61.7 BAcc), and the acceptable-alternatives analysis on FinRisk-Ask, which are very helpful for interpreting the results. The external comparisons suggest that tasks such as Person Matching and Risk Classification provide evaluation signals beyond existing benchmarks and are particularly relevant to financial risk review. However, my remaining concern is that these tasks account for a relatively small portion of the static benchmark (Table 5), and the paper provides limited detail on their source materials and construction process. It would be helpful if you could clarify at a conceptual level how these categories were sourced and constructed, and provide a concrete example to better illustrate their importance.

AuthorsOpenReview ↗

Response to Reviewer ZVwP

Thank you for your careful follow-up. We appreciate your comments on the external comparisons and expert assessments. Person Matching has 500 items and Risk Classification has 366. They account for 866 of the 9,742 static items, or 8.9%. Table 5 groups Person Matching with 250 Institution Matching items, which explains the 750 items in that row. Task sizes reflect the source material available and the items retained after review. We report each operation separately and give each operation equal weight in its layer average.

1. Person Matching

We collected name entries from public sources such as trade fair records and registration documents. Some records list two forms of the same person’s name, giving us direct evidence for a match. We used these entries to build questions that present two names and ask whether they refer to the same person. The pairs include Chinese and Latin-script forms, regional spellings, abbreviations, and differences in name order.

2. Risk Classification

We searched public case materials by risk topic, including fraud, sanctions, money laundering, and export control. The topic under which we collected each document supplied its initial category. We screened the materials for quality and used a language model to identify relevant passages and format the case text. Domain experts reviewed the retained cases and checked that the source material supported their reference categories. Each item asks the model to read the case description and select one risk category from the available choices.

The table gives three examples. The case descriptions are abridged, and the choices and reference answers are translated into English.

ExampleAnswer choicesReference answer
Do “CHAN, Wai Lik” and “陈伟力” refer to the same person?Yes or noYes (match)
A trading company and a freight forwarder fabricate bills of lading, packing lists, and invoices. The bank pays under a letter of credit, but the buyer discovers that no goods were shipped.Export control, fraud, money laundering, sanctions, prohibited or restricted sales, no riskFraud
A company declares waste-acid treatment equipment as wastewater treatment equipment for export.Same six choicesExport control

The name pair illustrates a common issue in cross-border settlement. A payment record may give a name in Latin script while a supporting document gives it in Chinese characters. Recognizing forms such as Chan for the surname 陈 helps the reviewer compare the full names across the two records.

The two risk cases illustrate different review decisions. In the letter-of-credit case, the bank paid against documents for a shipment that never took place. The trade finance risk lies in relying on those documents as evidence of shipment. In the equipment case, the issue is cross-border trade compliance. The discrepancy in the product description gives a reviewer reason to check whether the equipment requires an export license. The prosecutor who handled the equipment case published an account of it [1].

For comparison, our separate Case Classification task uses financial and commercial disputes selected from China Judgments Online [2]. We kept the type of dispute recorded in each judgment as the reference label. In one item, a policyholder claims a critical-illness benefit and the insurer disputes coverage. The reference answer is life-insurance contract dispute. Here the model identifies the legal claim. In the letter-of-credit example, it identifies the financial risk illustrated by the conduct.

Thank you again for your time and thoughtful review. We would be happy to answer any further questions about the data or how we constructed these tasks.

[1] Cao Beibei. Prosecutor’s account of the export-control case. Procuratorate Daily, 2017.

[2] Supreme People’s Court of China. China Judgments Online.

Reviewer_kFLaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FinRiskAtlas, a Chinese-language benchmark for financial risk and compliance review. It contains 9,742 static instances covering 42 knowledge families and 11 review operations. The authors also introduce FinRisk-Ask, which evaluates evidence-seeking decisions using 680 states from 104 professional review trajectories. Experiments across 33 model configurations show that strong financial knowledge does not necessarily translate into strong performance on specific review tasks, and that generating relevant evidence requests is different from deciding when to request information.

Critical strengths
  1. Practical task design. The benchmark goes beyond conventional financial QA by covering concrete review operations such as information extraction, provision selection, and review opinion generation. The operation contracts in Section 3 make the evaluation objectives reasonably clear.
  2. Interesting evidence-seeking evaluation. FinRisk-Ask is a useful addition. Separating the Ask/Proceed decision from the quality of the evidence request captures two different capabilities that are relevant to real financial review workflows.
  3. Broad experiments and careful validation. The authors evaluate 33 model configurations and provide expert annotation audits. The 18.01-point gap in information extraction (Section 5.3) is a convincing example of why knowledge-based model selection can be misleading.
Critical weaknesses
  1. Imbalanced evidence-seeking data. FinRisk-Ask contains 583 Ask states but only 97 Proceed states. Even the best model achieves a BAcc of just 59.46, compared with 50 for a constant-action baseline (Table 16). This makes it difficult to judge how reliably the benchmark distinguishes models that know when to proceed without requesting more evidence.
  2. Some scoring rules are too coarse. For applicable-provision selection, any nonempty proper subset of the correct answers receives 0.5 credit, while including an incorrect option results in zero (Equation 5). This does not distinguish between missing one correct provision and missing several, and may affect model rankings.
  3. Dependence on a single LLM evaluator. The three generation tasks and FinRisk-Ask request alignment all rely on DeepSeek-V4-Flash for semantic evaluation. Although the authors provide human validation (Table 12), it is unclear whether the reported cross-task correlations, particularly those in Figure 5(c), would remain similar with a different evaluator.
Questions for authors
  1. Could the authors report how stable the FinRisk-Ask BAcc rankings are under trajectory-level resampling, particularly given the limited number of Proceed states?
  2. How sensitive are the provision-selection rankings to alternative set-based metrics, such as F1?
  3. Have the authors considered using a second evaluator to check whether the correlations in Figure 5(c) remain consistent?
4 comments
AuthorsOpenReview ↗

Response to Reviewer kFLa (Part 1/2)

Thank you for your careful review and constructive feedback. Most additions are in the appendix, with brief updates to the main text. We address ranking stability and scoring sensitivity below, followed by evaluator sensitivity in Part 2/2.

Q1. Trajectory-bootstrap ranking stability

The results support Kimi-K3 and Qwen3-32B as a leading group, with different strengths in Ask and Proceed recall. Their relative BAcc ordering varies across resamples.

We used 10,000 shared bootstrap samples of the 104 trajectories to evaluate 24 configurations, including nine of the ten highest-BAcc configurations in Appendix E.4. Each sampled trajectory retained all its states and predictions. The 97 Proceed states belong to 97 distinct trajectories. Following the paper's aggregation, metrics were computed separately for each run and then averaged for each configuration, including five runs for Qwen3-32B and one for Kimi-K3. The intervals therefore measure trajectory-sampling uncertainty for the observed runs.

At least one of Kimi-K3 and Qwen3-32B occupies or shares first place in 84.71% of samples. Their top-three frequencies are 90.10% and 82.75%, respectively. Table T1a presents the nine leading configurations covered by this analysis; strictly-first frequencies exclude ties.

Table T1a. Trajectory-bootstrap rankings among 24 configurations.

ConfigurationMedian rank [IQR]Strictly first (%)Top three (%)
Kimi-K32 [1, 2]41.5690.10
Qwen3-32B2 [1, 3]43.0482.75
Qwen3-Next-80B9 [6, 12]0.597.87
GPT-5.6-Terra6 [4, 8]0.6315.74
Ling-2.6-Flash9 [5, 13]2.4613.29
Nemotron-3-Ultra7 [4, 9]0.8114.35
Kimi-K2.56.5 [4, 9]1.4616.35
GLM-58 [5, 11]1.1913.31
Qwen3-8B5 [3, 10]7.8533.39

The individual 95% BAcc intervals are [54.93, 64.31] for Kimi-K3 and [54.08, 64.82] for Qwen3-32B, both above the constant-action reference of 50. The paired comparisons distinguish their action profiles more clearly than their overall BAcc scores.

Table T1b. Paired differences and 95% trajectory-bootstrap intervals.

ComparisonDifference95% CI
BAcc: Kimi − Qwen1.27[−5.52, 5.49]
AskR: Kimi − Qwen23.15[18.52, 26.88]
ProceedR: Qwen − Kimi20.62[12.37, 32.99]

The paired intervals support higher Ask recall for Kimi-K3 and higher Proceed recall for Qwen3-32B; the BAcc-difference interval includes zero. Appendix F.2.2 reports the ranking distributions, paired comparisons and resampling procedure.

Q2. Provision-selection rankings under set-F1

We rescored the same predictions from 26 configurations on all 1,000 provision-selection items using mean per-item set-F1. Rankings remain strongly associated with the native metric, with Spearman ρ = 0.904 and Kendall τ-b = 0.742. GPT-5.6-Sol ranks first under both metrics. Table T2 shows selected configurations; Appendix F.1.1 reports all 26.

The metrics emphasize different errors. The native rule awards 1 for the exact provision set, 0.5 for a nonempty correct proper subset, and 0 otherwise. Set-F1 gives graded credit for overlap, distinguishing the extent of omissions and incorrect inclusions.

Table T2. Provision-selection scores and ranks among 26 configurations.

ConfigurationNative scoreSet-F1Rank: native → set-F1
GPT-5.6-Sol67.5591.881 → 1
Gemini-2.5-Flash66.2588.412 → 4
Gemini-3-Flash-preview65.7587.953 → 6
GPT-5.4-Mini55.3585.847 → 12
Kimi-K351.6588.0011 → 5
DeepSeek-V4-Pro41.4582.7317 → 16

For Gemini-3-Flash-preview and Kimi-K3, the native-score difference reflects different error profiles. Gemini includes an incorrect provision on 16.6% of items, compared with 40.7% for Kimi, but omits a reference provision more often, on 41.0% of items compared with 20.9%. Their set-F1 scores are consequently similar at 87.95 and 88.00. The revised discussion attributes the 14.10-point native-score gap to the penalty for incorrect inclusions.

GPT-5.6-Sol remains 3.93 points above the Knowledge-leading Gemini-3 configuration under set-F1. This supports the finding that Knowledge leadership does not ensure the highest provision-selection performance. Section 5.2 and Appendix F.1.1 now distinguish these two scoring perspectives.


To be continued in Part 2/2.

AuthorsOpenReview ↗

Response to Reviewer kFLa (Part 2/2)

Continued from Part 1/2.


Q3. Evaluator sensitivity of Figure 5(c)

The positive associations between generation quality and request alignment for legal judgment and decision view persist when different evaluators score the two outputs. We compared the original DeepSeek-V4-Flash evaluator, J1, with Qwen3.7-Max, J2, using identical responses from 26 configurations: 100 items per generation task and 120 recorded Ask states from 102 trajectories. Both evaluators' own configurations were excluded, and inputs, rubrics and aggregation were held constant.

Table T3. Evaluator sensitivity of the Figure 5(c) correlations.

OperationBAcc ρ: J1 → J2ERA ρ: J1/J1 → J2/J2CRA ρ: J1/J1 → J2/J2CRA ρ: J1/J2; J2/J1
Legal judgment0.223 → 0.2150.698 → 0.5810.716 → 0.5680.676; 0.617
Decision view0.301 → 0.3040.598 → 0.5790.705 → 0.5970.594; 0.700
Disputed issues0.128 → 0.161−0.017 → −0.1710.056 → −0.222−0.110; −0.033

Jg/Jr denotes the generation and request evaluators. BAcc scores are unchanged; their correlations use generation scores from J1 or J2. The J1 CRA correlations of 0.716 and 0.705 use this common subset; the corresponding 0.80 values in Figure 5(c) use the full 32-configuration results.

The 95% percentile intervals use 10,000 paired bootstrap samples of generation items and Ask trajectories, with configurations, responses and evaluator judgments held fixed.

All four cross-evaluator CRA correlations for legal judgment and decision view have 95% intervals above zero. For both operations, CRA correlations exceed the corresponding BAcc correlations in point estimates under all four evaluator pairings; under J2, ERA shows the same ordering relative to BAcc.

Under J2, the CRA–BAcc correlation difference is 0.353 [0.139, 0.531] for legal judgment and 0.293 [−0.054, 0.469] for decision view. The legal-judgment interval is above zero, while the decision-view interval includes zero. For disputed issues, the CRA intervals under both evaluators include zero.

These results support the generation–request associations for legal judgment and decision view. Appendix F.2.1 reports the analysis. Appendix D.5 reports model-ranking agreement using the same DeepSeek–Qwen scores.

Thank you again for your constructive feedback.

Reviewer_kFLaOpenReview ↗

Response to rebuttal

Thank you for your reply. As my score is already positive, I will maintain my score.

Thank you for maintaining your positive assessment of our work. We sincerely appreciate your time, careful review, and constructive suggestions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Additional Results

Dear Area Chair and Reviewers,

Thank you for your careful reviews and constructive feedback. We have provided point-by-point responses and added the following results to the revised manuscript:

  • Comparisons with existing benchmarks. Evaluations of eight configurations on LawBench and FinEval show strong associations with FinRiskAtlas knowledge scores and weak associations with Ask/Proceed agreement. These comparisons support evaluating evidence-seeking separately from financial and legal knowledge.
  • Human reference for FinRisk-Ask. Three experts independently chose Ask or Proceed on a matched sample of pre-action states used for additional model runs. A separate assessment evaluates the acceptability of both actions, including model choices that differ from recorded decisions.
  • Evaluator checks and repeated runs. Cross-family rescoring shows strong ranking agreement and supports the ordering of the leading disputed-issue pair. Three additional runs on selected configurations and operations preserve the large extraction gap, while person-matching and case-classification results identify closely matched models.
  • Ranking and scoring sensitivity. Trajectory resampling supports a leading group for FinRisk-Ask, with variation in its exact ordering. Provision-selection rankings remain strongly associated under set-F1, with the same configuration ranked first under both metrics.

FinRiskAtlas combines expert-reviewed financial tasks with offline replay of professional review trajectories. The additional results support operation-specific model selection and provide a clearer basis for interpreting action agreement separately from request quality.

We also clarified inference repetitions, data sources, and the materials currently provided and planned for release upon acceptance. Changes in the revised manuscript are marked in blue.

We remain available to clarify specific points in the responses. Thank you again for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.