- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces FinRiskAtlas, a Chinese benchmark for financial risk and compliance review. It combines a static suite of knowledge and review-operation tasks, each defined by an explicit operation contract, with FinRisk-Ask, which replays pre-action states from real enterprise review trajectories and asks models to decide whether to ask for more evidence or proceed. Evaluating many models zero-shot, the authors argue that aggregate knowledge scores do not identify the best model on each review operation, and that evidence-request quality and ask/proceed agreement are led by different models.
Critical strengths
- FinRisk-Ask is a well-constructed offline evaluation of evidence-seeking. It is built from real professional trajectories with later evidence withheld, has clean metric decomposition and an expert audit, and such data are scarce.
- The static benchmark is documented with unusual care, including operation contracts, native scorers and second-pass audits.
Critical weaknesses
- A single judge from one model family scores all generation and request-alignment results, and that family leads several tasks, with no test for family bias; this confounds the corresponding conclusions.
- Conclusions rest on point-estimate leader comparisons with small margins and unreported repetitions, so the "operation-specific selection" message is weakly supported. That no single model leads every operation is also expected for heterogeneous tasks, and the repetitive text leaves key facts hard to find.
Questions for authors
- Did you check the judge for self-preference, for example by re-scoring the leaders with a judge from a different family?
- How many repetitions were run per configuration, and how stable are the leaders across them?
- Can you rule out that evaluated models were trained on the internal collections used for the knowledge items, and what exactly will be released?
2 comments
Response to Reviewer 8ma3 (Part 2/2)
Continued from Part 1/2.
Q2. Inference repetitions and leader stability
In the submitted results, each of the eight local vLLM configurations used five inference runs per task, with scores averaged across runs; each of the 25 API configurations used one run. Appendix B.2 now lists these counts for all 33 configurations.
We added three runs per configuration for the main extraction comparison and two close leader comparisons, covering all 249 extraction items, 500 person-matching items and 445 case-classification items. Within each task, configurations used the same inputs and scorer, with inference settings fixed across each configuration's three runs. Table T8 separates the paper scores from the additional runs and reports the mean and sample standard deviation of the three new scores.
Table T8. Paper scores and three additional inference runs.
| Operation | Configuration | Paper score | Run 1 | Run 2 | Run 3 | Mean ± SD |
|---|---|---|---|---|---|---|
| Information extraction | GPT-5.6-Luna | 85.97 | 85.57 | 84.68 | 86.17 | 85.47 ± 0.75 |
| Information extraction | Gemini-3-Flash-preview | 67.96 | 65.37 | 66.37 | 66.77 | 66.17 ± 0.72 |
| Person matching | Kimi-K3 | 87.00 | 86.40 | 86.40 | 86.00 | 86.27 ± 0.23 |
| Person matching | GPT-5.6-Sol | 86.00 | 86.20 | 85.80 | 85.80 | 85.93 ± 0.23 |
| Case classification | Kimi-K2.5 | 88.09 | 88.09 | 87.19 | 85.84 | 87.04 ± 1.13 |
| Case classification | GPT-5.6-Sol | 87.42 | 87.19 | 87.64 | 86.97 | 87.27 ± 0.34 |
| Case classification | Kimi-K3 | 87.19 | 87.87 | 88.76 | 86.52 | 87.72 ± 1.13 |
The large extraction gap persists across the additional evaluations: all three Luna scores exceed all three Gemini scores. The new mean gap is 19.30 points, with a paired 95% interval of [16.26, 22.50].
Person matching shows a smaller separation. Kimi-K3 scores higher in each run, with a mean advantage of 0.33 points and an interval of [0.00, 0.73]. In case classification, the three means span 0.67 points, each configuration ranks first in one run, and all pairwise difference intervals include zero. We accordingly describe these configurations as a closely matched leading group.
The paired intervals use 10,000 shared item-bootstrap samples, retaining all three responses per sampled item and preserving field-weighted scoring for extraction. They measure item-sampling uncertainty; the reported standard deviations summarize variation across runs.
We agree that different tasks can favor different models. Our contribution is to quantify the performance loss from Knowledge-based selection and show that the main extraction gap persists across runs. The close comparisons identify where exact ordering is unstable. Section 5 and Appendix B.6.1 now report these distinctions and the additional results.
Q3. Training contamination and release scope
The Trade Document Collection and Risk Document Collection have not been publicly released or provided as training data to the teams responsible for the evaluated models, including teams within our institutions. We used these collections for benchmark construction and did not use either to train or fine-tune any evaluated configuration.
FinRisk-Ask is derived from non-public enterprise review trajectories. For items derived from public sources, including judicial records, overlap with undisclosed model pretraining data remains unmeasured, as stated in Appendix H.
We will release the complete evaluation-ready benchmark and evaluation code immediately upon acceptance. Table T9 distinguishes the materials provided with the submission from the planned release.
Table T9. Materials provided with the submission and planned release.
| Material | Included in submission | Planned release |
|---|---|---|
| Static evaluation data | Complete inputs and references for 249 extraction and 58 quantitative-reasoning items; 292 illustrative examples | Full static benchmark inputs and references across the 9,742 items |
| FinRisk-Ask data | All 680 recorded-action labels; 1,360 action/credit records for two configurations; six display cards | All 680 de-identified pre-action contexts and recorded actions; evidence targets for the 583 recorded-Ask states |
| Evaluation framework | Reported score matrix, operation contracts, parsing/scoring utilities and analysis scripts | Task/evaluator prompts, parsers, scorers, rubrics and scripts for running the published protocol |
Thank you again for your recognition and constructive feedback on our work.
Response to Reviewer 8ma3 (Part 1/2)
We sincerely thank you for your careful review and for recognizing the value of the benchmark. Most additions are in the appendix, with brief updates to the main text. We address your questions on evaluator family bias below, followed by inference repetitions and data release in Part 2/2.
Q1. Evaluator family bias
To examine evaluator family bias, we added Qwen3.7-Max to compare model rankings and Claude-Opus-4.7 to rescore the two leading disputed-issue configurations. Rankings remain strongly associated across evaluators, and DeepSeek-V4-Pro also scores higher than Nemotron-3-Super under Claude.
Qwen3.7-Max and the original DeepSeek-V4-Flash evaluator scored identical responses from 26 configurations on 100 items per generation task and 120 recorded Ask states from 102 trajectories. Candidate identities were hidden, both evaluators' own configurations were excluded, and inputs, rubrics and aggregation were identical.
Table T7a. Cross-family ranking agreement on common evaluation samples.
In the paper's request-alignment results, the ERA leader is Ling-2.6-1T at 79.76 and the CRA leader is Qwen3.7-Max at 87.67; neither belongs to the DeepSeek family.
For disputed-issue generation, we rescored DeepSeek-V4-Pro and Nemotron-3-Super on all 500 items with both Claude-Opus-4.7 and DeepSeek-V4-Flash. Their paper scores are 77.12 and 76.67. Claude belongs to neither candidate's family. Both evaluators received the same responses, references and rubric and used the paper's aggregation.
Table T7b. Additional evaluator scores for the two leading disputed-issue configurations.
The 95% intervals use 10,000 paired item-bootstrap samples. DeepSeek-V4-Pro retains the higher score under both evaluators. The difference increases from 2.35 points under DeepSeek-V4-Flash to 9.16 under Claude, whose interval excludes zero.
The comparison supports the leading pair's ordering under an evaluator outside both candidates' families, while showing that the size of the difference depends on the evaluator. Appendix D.5 now presents both analyses. Appendix F.2.1 uses the same DeepSeek–Qwen scores to check Figure 5(c): the positive generation–request associations for legal judgment and decision view persist when different evaluators score the two outputs.
The structured-operation results provide additional evidence independent of semantic evaluation, including the repeated extraction comparison in Part 2/2.
To be continued in Part 2/2.