- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
The paper introduces FinCDM, a framework for diagnosing financial LLM capabilities at the concept level using expert-verified question–concept mappings and non-negative matrix co-factorization. The authors construct CPA-KQA, containing 1,260 Chinese questions covering 70 concepts, and CFA-KQA, containing 738 English questions covering 41 concepts. They evaluate 20 LLMs and show that similar aggregate accuracy can coexist with different estimated concept profiles. They also propose SkillTune, which selects fine-tuning examples associated with diagnosed weak concepts. Validation includes held-out response prediction, an expert interpretation study, fine-tuning experiments on three models across both benchmarks, and a controlled data-selection comparison on Qwen 3 8B. CPA-trained models are additionally evaluated on the independent FinEval-KQA dataset.
Critical strengths
-
The expert-authored, concept-balanced datasets provide a useful resource for financial LLM evaluation. The explicit CPA/CFA taxonomies, independent review of questions, and reported annotation agreement support examining performance beyond aggregate accuracy. This resource contribution remains valuable independently of the particular capability estimator.
-
The paper connects diagnosis to an actionable training intervention and provides encouraging independent transfer results. SkillTune improves all six model–benchmark configurations, and the three CPA-trained models also improve on FinEval-KQA, which is excluded from concept selection, hyperparameter tuning, and checkpoint selection. The controlled study includes training-budget-matched random selection baselines. These experiments provide meaningful preliminary evidence for the practical value of targeted supervision.
Critical weaknesses
-
The interpretation and calibration of the diagnostic scores are insufficiently established. Equations (3)–(5) reconstruct X through EU and Q through EV, while defining concept scores as F = U^T V. For any positive scalar c, replacing E with cE, U with U/c, and V with V/c preserves both reconstructions and the objective but changes F to F/c². Thus, equivalent factorizations can produce different concept counts and training selections under the fixed threshold of 0.9. The specified initialization, seed, and iteration count define a reproducible operational score, and this ambiguity does not by itself invalidate the reported empirical improvements. However, they do not establish that the score measures concept-level competence or that its threshold reliably identifies weak concepts. Appendix E.2 examines nearby thresholds within a fitted representation, which does not resolve this measurement issue. Because the scores determine both the diagnostic conclusions and SkillTune’s selections, addressing this concern requires substantive validation beyond introducing a numerical normalization.
-
The validation does not sufficiently establish diagnostic reliability or its incremental benefit over simpler selection methods. Each concept is assessed using only three distinct questions. Held-out response prediction provides evidence of predictive fit, but does not directly validate the interpretation of concept scores; the expert study is limited to six responses already included in fitting. Selecting training concepts from an initial assessment is a reasonable application, but reevaluating on the same assessment cannot establish performance on an untouched test set. FinEval-KQA meaningfully addresses this concern for CPA, whereas comparable independent validation is absent for CFA. Furthermore, the controlled selection comparison uses one backbone and one training run for SkillTune and the strongest nonrandom baselines. The per-concept baseline applies two successive thresholds to empirical correctness, discarding information, and its selection budget is not explicitly matched to SkillTune. Consequently, the current evidence does not adequately isolate the contribution of latent diagnosis from ordinary weak-topic selection and training variability.
Questions for authors
-
Is there an unreported normalization or identification constraint that addresses the scaling ambiguity in F = U^T V? How was the threshold of 0.9 selected, and what evidence supports interpreting its decisions as identifying weak versus strong concepts?
-
What reference profiles are used to calculate cosine similarity, Pearson correlation, and RMSE in Appendix E.1? Do these comparisons involve independent question sets, and how stable are the diagnosed weak-concept sets across item samples and factorization initializations?
-
Evidence request relevant to the current claims: Can the authors compare SkillTune with selecting the same number of concepts by their mean empirical correctness, using matched training budgets and multiple fine-tuning seeds? Results on the independent FinEval-KQA set would materially clarify whether matrix co-factorization contributes beyond a simpler selection rule.
-
Appendix K specifies extracting the first match of (A|B|C|D). Is this the actual parser used? This rule would extract A from “Answer: C.” Please clarify whether such responses occur and whether an audit using a parser that recognizes standalone answer labels changes the reported results. The concern is a potential confound, rather than an established explanation of the gains, because letter-only fine-tuning could improve formatting compliance independently of financial knowledge.
5 comments
W2 and Q2: How reliable is a concept profile based on three questions per concept?
We thank the reviewer for recognizing the value of using an initial assessment to identify which concepts a model should practice. For this purpose, it matters how closely a profile based on three questions per concept agrees with one based on different questions about the same concepts.
We agree that Appendix E.1 should specify which questions formed the reference profile in Figure 7 and whether they also appeared in the profiles being compared. For each of the 70 CPA-KQA concepts, we fitted the reference profile using six questions: the three benchmark test questions and three additional questions on the same concept from the training pool. The profiles in Figure 7 used one to five of those six questions. In particular, the four- and five-question profiles added one and two training-pool questions, respectively, to the three test questions. Each comparison profile therefore shared its questions with the reference. This analysis was conducted separately from SkillTune.
We repeated the comparison using disjoint question sets from the same 420 questions. In each of 100 random splits, we reserved two questions per concept for the reference profile and estimated the comparison profile from one to four of the remaining questions. Both profiles used , the mean reconstructed correctness for each concept. The table reports their mean agreement across splits.
| Questions per concept in the comparison profile | Pearson correlation | Cosine similarity | RMSE |
|---|---|---|---|
| 1 | 0.405 | 0.876 | 0.380 |
| 2 | 0.494 | 0.911 | 0.314 |
| 3 | 0.542 | 0.925 | 0.288 |
| 4 | 0.570 | 0.931 | 0.274 |
With three questions per concept, the mean Pearson correlation with the disjoint reference was 0.542 (SD 0.030 across splits). The fourth question increased it by 0.028, compared with gains of 0.089 for the second question and 0.048 for the third. The fourth question also gave the smallest improvement in cosine similarity and RMSE. Thus, the diminishing gains reported in Figure 7 persist when the two profiles use different questions. The expert study in Appendix C serves a separate purpose: its six responses were included in fitting, and the experts’ judgments illustrate how the diagnosed weaknesses appear in the model’s reasoning.
To address this concern, we will revise Appendix E.1 and the Figure 7 caption to identify the reference questions in each analysis and report the disjoint-set results above. We will also describe the expert study as a qualitative illustration in the main text and Appendix C.
How do the selection rules compare in guiding fine-tuning?
We thank the reviewer for recognizing the practical motivation behind SkillTune: a concept-level assessment can identify which training examples a model needs. Table 3b compares fine-tuning on diagnosed weaknesses with fine-tuning on randomly selected material and on the full training set.
We agree that our paper did not explain why the two diagnostic methods selected different numbers of concepts. Each method selects all the concepts it identifies as weak for a given model. For Qwen 3 8B, SkillTune identified 64 weak concepts. The per-concept accuracy rule identified 57: on each of these, the model answered at least two of three assessment questions incorrectly. Requiring the empirical rule to select 64 would add seven concepts on which the model answered at least two questions correctly. Likewise, restricting SkillTune to 57 would omit seven concepts it diagnosed as weak. We retained each method’s full set so that its training material corresponds to the weaknesses it identifies for that model.
| Selection rule | Concepts covered | CPA-KQA training questions | Training runs | CPA-KQA accuracy | External fineval-KQA accuracy |
|---|---|---|---|---|---|
| SkillTune | 64 | 960 | 1 | 0.58 | 0.58 |
| Per-concept accuracy | 57 | 855 | 1 | 0.51 | 0.49 |
| Random questions | 70 | 960 | 5 | 0.51 | 0.50 |
| Random concepts | 64 | 960 | 5 | 0.52 | 0.51 |
| All concepts | 70 | 1,050 | 1 | 0.55 | 0.53 |
The 64 concepts identified by SkillTune provided 960 CPA-KQA training questions. Each random baseline used the same number of questions, with a fresh selection in each of five runs. On fineval-KQA, a separate external benchmark, SkillTune exceeded the five-run means by 8.3 percentage points for random-question selection and 6.9 points for random-concept selection. Both gains were statistically significant in a paired comparison across the 101 evaluation questions (95% bootstrap intervals: 4.5 to 12.1 and 3.4 to 10.4 points, respectively). SkillTune also exceeded full-data training by 5.5 points while using 90 fewer questions. These results support the value of using diagnosed weaknesses to guide fine-tuning: SkillTune outperformed the random baselines at the same budget and full-data training at a larger budget.
We also examined seed sensitivity at the diagnosis stage. Across 20 fits using , the standard deviation of the concept count per model was 1.1 concepts on CPA-KQA and 1.0 on CFA-KQA. Model rankings by concept count had Spearman correlations of at least 0.98 and 0.96, respectively, with the seed-42 rankings. For the fine-tuning experiments, we ran each random baseline five times, drawing new training material each time. Measuring SkillTune’s own training-seed variability would require repeating the full fine-tuning and evaluation procedure for each additional seed. The computational cost of those runs exceeded the resources available during the rebuttal period.
To address this concern, we will add the concept counts, training budgets, and run counts to Table 3b and explain the budget comparisons at lines 459 to 461. We agree that variation across fine-tuning seeds is a limitation of the current evaluation, and we will state in §7 that the reported SkillTune accuracy comes from a single fine-tuning run.
W1 and Q1: Interpretation of the 0.9 threshold and stability of the diagnostic findings
We thank the reviewer for recognizing the value of the expert-authored, concept-balanced benchmarks independently of the estimator used in FinCDM. They allow us to check a central finding of the paper directly against model responses: two models with the same overall accuracy can have different financial strengths.
We agree that the equations allow the factors to be rescaled while preserving the reconstructed responses. In our experiments, however, we fixed the initialization and fitting settings specified in Appendix F and used that same setup throughout. Table 2 therefore uses the scores on the scale produced by that fit; we applied no rescaling before counting mastered concepts. We chose τ=0.9 before fitting as a conservative cutoff on this scale. Table 5 shows that as τ varies from 0.85 to 0.95, the Spearman correlation with the original Table 2 ranking remains above 0.98 on CPA-KQA and 0.96 on CFA-KQA.
To check whether our findings depend on this choice of scale, we also computed S, the mean reconstructed correctness over the questions for each concept. Rescaling the factors leaves S unchanged. We ranked models by the number of concepts whose S reached the midpoint between chance and perfect accuracy, corresponding to 0.625 on CPA-KQA and two-thirds on CFA-KQA. The table compares the resulting rankings with Table 2 and with rankings based on observed per-concept accuracy.
The specific contrast in §5.1 also persists. Grok 3 and GPT-4 both score 0.62 overall on CPA-KQA. Under S, Grok 3 leads in Economic Law (0.85 vs. 0.68), while GPT-4 leads in Tax Law (0.80 vs. 0.58). Thus, both the broad ranking by concept coverage and the difference in strengths between these equal-scoring models remain visible under a scale-invariant score.
To address this concern, we will revise §4.2, lines 266 to 276, to explain the choice of τ=0.9 and the fitted scale on which it operates. We will report the scale-invariant comparison in Appendix E and add the reconstructed correctness values alongside the equal-score example in §5.1.