acceptodds
Under review as a conference paper at ICLR 2027

Know Thyself: Concept-Level Diagnosis for Financial Large Language Models

Abstract

Large language models (LLMs) are increasingly used in finance, but existing benchmarks provide limited evidence of what financial capabilities a model has actually mastered. As a result, models with similar benchmark scores may differ sharply in their underlying strengths and weaknesses. We introduce FinCDM, a framework for diagnosis at the concept level that combines the official CPA and CFA concept taxonomies, mappings between questions and concepts verified by experts, model response patterns, and a replaceable capability estimator.}We construct a bilingual benchmark suite derived from the CPA and CFA examinations, consisting of CPA-KQA (Chinese, 1,260 questions, 70 concepts) and CFA-KQA (English, 738 questions, 41 concepts). Across 20 representative LLMs, we find that score-equivalent models are often not capability-equivalent, and that benchmark imbalance and linguistic limitations can further distort apparent competence. We further propose SkillTune, a diagnosis-guided data selection strategy for fine-tuning that targets each model's weak concepts. SkillTune improves multiple LLMs and, in a controlled comparison, outperforms alternative data selection strategies. These results support financial LLM evaluation based on concept-level profiles and diagnosis-guided improvement. Datasets and scripts will be publicly available.

Primary area:
datasets and benchmarks
Keywords:
Cognitive diagnosis, Financial model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QPj6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper introduces FinCDM, a framework for diagnosing financial LLM capabilities at the concept level using expert-verified question–concept mappings and non-negative matrix co-factorization. The authors construct CPA-KQA, containing 1,260 Chinese questions covering 70 concepts, and CFA-KQA, containing 738 English questions covering 41 concepts. They evaluate 20 LLMs and show that similar aggregate accuracy can coexist with different estimated concept profiles. They also propose SkillTune, which selects fine-tuning examples associated with diagnosed weak concepts. Validation includes held-out response prediction, an expert interpretation study, fine-tuning experiments on three models across both benchmarks, and a controlled data-selection comparison on Qwen 3 8B. CPA-trained models are additionally evaluated on the independent FinEval-KQA dataset.

Critical strengths
  1. The expert-authored, concept-balanced datasets provide a useful resource for financial LLM evaluation. The explicit CPA/CFA taxonomies, independent review of questions, and reported annotation agreement support examining performance beyond aggregate accuracy. This resource contribution remains valuable independently of the particular capability estimator.

  2. The paper connects diagnosis to an actionable training intervention and provides encouraging independent transfer results. SkillTune improves all six model–benchmark configurations, and the three CPA-trained models also improve on FinEval-KQA, which is excluded from concept selection, hyperparameter tuning, and checkpoint selection. The controlled study includes training-budget-matched random selection baselines. These experiments provide meaningful preliminary evidence for the practical value of targeted supervision.

Critical weaknesses
  1. The interpretation and calibration of the diagnostic scores are insufficiently established. Equations (3)–(5) reconstruct X through EU and Q through EV, while defining concept scores as F = U^T V. For any positive scalar c, replacing E with cE, U with U/c, and V with V/c preserves both reconstructions and the objective but changes F to F/c². Thus, equivalent factorizations can produce different concept counts and training selections under the fixed threshold of 0.9. The specified initialization, seed, and iteration count define a reproducible operational score, and this ambiguity does not by itself invalidate the reported empirical improvements. However, they do not establish that the score measures concept-level competence or that its threshold reliably identifies weak concepts. Appendix E.2 examines nearby thresholds within a fitted representation, which does not resolve this measurement issue. Because the scores determine both the diagnostic conclusions and SkillTune’s selections, addressing this concern requires substantive validation beyond introducing a numerical normalization.

  2. The validation does not sufficiently establish diagnostic reliability or its incremental benefit over simpler selection methods. Each concept is assessed using only three distinct questions. Held-out response prediction provides evidence of predictive fit, but does not directly validate the interpretation of concept scores; the expert study is limited to six responses already included in fitting. Selecting training concepts from an initial assessment is a reasonable application, but reevaluating on the same assessment cannot establish performance on an untouched test set. FinEval-KQA meaningfully addresses this concern for CPA, whereas comparable independent validation is absent for CFA. Furthermore, the controlled selection comparison uses one backbone and one training run for SkillTune and the strongest nonrandom baselines. The per-concept baseline applies two successive thresholds to empirical correctness, discarding information, and its selection budget is not explicitly matched to SkillTune. Consequently, the current evidence does not adequately isolate the contribution of latent diagnosis from ordinary weak-topic selection and training variability.

Questions for authors
  1. Is there an unreported normalization or identification constraint that addresses the scaling ambiguity in F = U^T V? How was the threshold of 0.9 selected, and what evidence supports interpreting its decisions as identifying weak versus strong concepts?

  2. What reference profiles are used to calculate cosine similarity, Pearson correlation, and RMSE in Appendix E.1? Do these comparisons involve independent question sets, and how stable are the diagnosed weak-concept sets across item samples and factorization initializations?

  3. Evidence request relevant to the current claims: Can the authors compare SkillTune with selecting the same number of concepts by their mean empirical correctness, using matched training budgets and multiple fine-tuning seeds? Results on the independent FinEval-KQA set would materially clarify whether matrix co-factorization contributes beyond a simpler selection rule.

  4. Appendix K specifies extracting the first match of (A|B|C|D). Is this the actual parser used? This rule would extract A from “Answer: C.” Please clarify whether such responses occur and whether an audit using a parser that recognizes standalone answer labels changes the reported results. The concern is a potential confound, rather than an established explanation of the gains, because letter-only fine-tuning could improve formatting compliance independently of financial knowledge.

5 comments
AuthorsOpenReview ↗

W1 and Q1: Interpretation of the 0.9 threshold and stability of the diagnostic findings

We thank the reviewer for recognizing the value of the expert-authored, concept-balanced benchmarks independently of the estimator used in FinCDM. They allow us to check a central finding of the paper directly against model responses: two models with the same overall accuracy can have different financial strengths.

We agree that the equations allow the factors to be rescaled while preserving the reconstructed responses. In our experiments, however, we fixed the initialization and fitting settings specified in Appendix F and used that same setup throughout. Table 2 therefore uses the scores on the scale produced by that fit; we applied no rescaling before counting mastered concepts. We chose τ=0.9\tau=0.9 before fitting as a conservative cutoff on this scale. Table 5 shows that as τ\tau varies from 0.85 to 0.95, the Spearman correlation with the original Table 2 ranking remains above 0.98 on CPA-KQA and 0.96 on CFA-KQA.

To check whether our findings depend on this choice of scale, we also computed SS, the mean reconstructed correctness over the questions for each concept. Rescaling the factors leaves SS unchanged. We ranked models by the number of concepts whose SS reached the midpoint between chance and perfect accuracy, corresponding to 0.625 on CPA-KQA and two-thirds on CFA-KQA. The table compares the resulting rankings with Table 2 and with rankings based on observed per-concept accuracy.

Spearman correlation of model rankingsCPA-KQACFA-KQA
SS vs. original Table 2 ranking0.9670.854
SS vs. observed per-concept accuracy at the same cutoff0.9750.959

The specific contrast in §5.1 also persists. Grok 3 and GPT-4 both score 0.62 overall on CPA-KQA. Under SS, Grok 3 leads in Economic Law (0.85 vs. 0.68), while GPT-4 leads in Tax Law (0.80 vs. 0.58). Thus, both the broad ranking by concept coverage and the difference in strengths between these equal-scoring models remain visible under a scale-invariant score.

To address this concern, we will revise §4.2, lines 266 to 276, to explain the choice of τ=0.9\tau=0.9 and the fitted scale on which it operates. We will report the scale-invariant comparison in Appendix E and add the reconstructed correctness values alongside the equal-score example in §5.1.

AuthorsOpenReview ↗

W2 and Q2: How reliable is a concept profile based on three questions per concept?

We thank the reviewer for recognizing the value of using an initial assessment to identify which concepts a model should practice. For this purpose, it matters how closely a profile based on three questions per concept agrees with one based on different questions about the same concepts.

We agree that Appendix E.1 should specify which questions formed the reference profile in Figure 7 and whether they also appeared in the profiles being compared. For each of the 70 CPA-KQA concepts, we fitted the reference profile using six questions: the three benchmark test questions and three additional questions on the same concept from the training pool. The profiles in Figure 7 used one to five of those six questions. In particular, the four- and five-question profiles added one and two training-pool questions, respectively, to the three test questions. Each comparison profile therefore shared its questions with the reference. This analysis was conducted separately from SkillTune.

We repeated the comparison using disjoint question sets from the same 420 questions. In each of 100 random splits, we reserved two questions per concept for the reference profile and estimated the comparison profile from one to four of the remaining questions. Both profiles used SS, the mean reconstructed correctness for each concept. The table reports their mean agreement across splits.

Questions per concept in the comparison profilePearson correlationCosine similarityRMSE
10.4050.8760.380
20.4940.9110.314
30.5420.9250.288
40.5700.9310.274

With three questions per concept, the mean Pearson correlation with the disjoint reference was 0.542 (SD 0.030 across splits). The fourth question increased it by 0.028, compared with gains of 0.089 for the second question and 0.048 for the third. The fourth question also gave the smallest improvement in cosine similarity and RMSE. Thus, the diminishing gains reported in Figure 7 persist when the two profiles use different questions. The expert study in Appendix C serves a separate purpose: its six responses were included in fitting, and the experts’ judgments illustrate how the diagnosed weaknesses appear in the model’s reasoning.

To address this concern, we will revise Appendix E.1 and the Figure 7 caption to identify the reference questions in each analysis and report the disjoint-set results above. We will also describe the expert study as a qualitative illustration in the main text and Appendix C.

AuthorsOpenReview ↗

How do the selection rules compare in guiding fine-tuning?

We thank the reviewer for recognizing the practical motivation behind SkillTune: a concept-level assessment can identify which training examples a model needs. Table 3b compares fine-tuning on diagnosed weaknesses with fine-tuning on randomly selected material and on the full training set.

We agree that our paper did not explain why the two diagnostic methods selected different numbers of concepts. Each method selects all the concepts it identifies as weak for a given model. For Qwen 3 8B, SkillTune identified 64 weak concepts. The per-concept accuracy rule identified 57: on each of these, the model answered at least two of three assessment questions incorrectly. Requiring the empirical rule to select 64 would add seven concepts on which the model answered at least two questions correctly. Likewise, restricting SkillTune to 57 would omit seven concepts it diagnosed as weak. We retained each method’s full set so that its training material corresponds to the weaknesses it identifies for that model.

Selection ruleConcepts coveredCPA-KQA training questionsTraining runsCPA-KQA accuracyExternal fineval-KQA accuracy
SkillTune6496010.580.58
Per-concept accuracy5785510.510.49
Random questions7096050.510.50
Random concepts6496050.520.51
All concepts701,05010.550.53

The 64 concepts identified by SkillTune provided 960 CPA-KQA training questions. Each random baseline used the same number of questions, with a fresh selection in each of five runs. On fineval-KQA, a separate external benchmark, SkillTune exceeded the five-run means by 8.3 percentage points for random-question selection and 6.9 points for random-concept selection. Both gains were statistically significant in a paired comparison across the 101 evaluation questions (95% bootstrap intervals: 4.5 to 12.1 and 3.4 to 10.4 points, respectively). SkillTune also exceeded full-data training by 5.5 points while using 90 fewer questions. These results support the value of using diagnosed weaknesses to guide fine-tuning: SkillTune outperformed the random baselines at the same budget and full-data training at a larger budget.

We also examined seed sensitivity at the diagnosis stage. Across 20 fits using SS, the standard deviation of the concept count per model was 1.1 concepts on CPA-KQA and 1.0 on CFA-KQA. Model rankings by concept count had Spearman correlations of at least 0.98 and 0.96, respectively, with the seed-42 rankings. For the fine-tuning experiments, we ran each random baseline five times, drawing new training material each time. Measuring SkillTune’s own training-seed variability would require repeating the full fine-tuning and evaluation procedure for each additional seed. The computational cost of those runs exceeded the resources available during the rebuttal period.

To address this concern, we will add the concept counts, training budgets, and run counts to Table 3b and explain the budget comparisons at lines 459 to 461. We agree that variation across fine-tuning seeds is a limitation of the current evaluation, and we will state in §7 that the reported SkillTune accuracy comes from a single fine-tuning run.

Reviewer_pkLjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes FinCDM, which adapts cognitive diagnosis from educational assessment to evaluate financial LLMs at the concept level rather than by aggregate accuracy. The authors build expert-authored, concept-balanced item banks aligned with the CPA and CFA syllabi, and factorize the model-by-item response matrix jointly with the item–concept mapping to obtain per-concept scores. They claim that equal-accuracy models have different concept profiles, that cross-lingual reversals reflect language limitations, and that existing benchmarks cover concepts unevenly. They also propose SkillTune, which fine-tunes on training items from diagnosed weak concepts and reportedly beats several data-selection controls.

Critical strengths

A reusable, concept-balanced, expert-authored bilingual item bank aligned with official taxonomies and a documented annotation protocol, plus a concrete demonstration that an existing benchmark covers concepts unevenly.

Critical weaknesses
  1. The concept scores lack validity. The score is not fixed up to scale so the threshold is arbitrary, and concepts rest on very few items with no uncertainty or per-concept-accuracy baseline. The expected "equal score, different capability" finding is therefore not shown to exceed noise, and the central heatmaps are too unreadable to check it.
  2. SkillTune's selection advantage is unsupported. The selected weak concepts cover most of the pool, margins are small from single runs on small test sets, and the gains may come from any fine-tuning or answer-format learning.

minor

Figure 2 is really too unclear.

Questions for authors
  1. How is the scale of the concept score fixed so that the 0.9 threshold has meaning, and how much do the profiles vary across random seeds?
  2. How do the factorization profiles differ from raw per-concept accuracy and other trivial baselines?
  3. What are the invalid-response rates under the output cap and answer extraction, and how much does the SkillTune training set overlap with the all-concept set?
4 comments
AuthorsOpenReview ↗

W1: Reliability of concept profiles with three items per concept

We thank the reviewer for recognizing the importance of measuring a model’s strengths across individual concepts. A practical question for a concept-balanced benchmark is how much each additional item improves the profile when the evaluation must cover 70 concepts.

Appendix A gives the psychometric basis for our design. Each item measures one concept, and each concept has three items, so the rows of the Q-matrix can be arranged into three copies of the identity matrix. This satisfies the Q-matrix conditions for strict identifiability under DINA in Gu and Xu (2019, Theorem 1). Figure 7 examines the choice empirically for FinCDM by comparing profiles estimated from one to five items with a reference profile fitted to all six items per concept: three CPA-KQA test items and three additional items from the training pool. We also repeated the comparison using disjoint reference and assessment items. In each of 100 random splits of the same pool, two items per concept supplied the reference profile, and we estimated a second profile from one to four of the remaining items.

Items per concept used to estimate the profilePearson correlation with the referenceCosine similarityRMSE
10.4050.8760.380
20.4940.9110.314
30.5420.9250.288
40.5700.9310.274

At three items, mean Pearson correlation was 0.542, with an SD of 0.030 across splits. Adding the fourth item increased the correlation by 0.028, compared with gains of 0.089 from the second and 0.048 from the third. All three metrics show diminishing gains. Together, the psychometric design criterion and the observed agreement across distinct questions support three items as a practical choice for this benchmark.

To address this concern, we will add the disjoint-item comparison to Appendix E.1 and identify the reference items used in each analysis in the Figure 7 caption.

AuthorsOpenReview ↗

Q1: Interpretation of the 0.9 threshold and stability across random seeds

We thank the reviewer for recognizing that the number of mastered concepts is useful for comparing models when their relative rankings are stable. The paper examines this directly by varying the threshold used to count a concept as mastered.

Section 4.2 uses τ=0.9\tau=0.9 as a prespecified, conservative cutoff on the fitted concept score FF: raising τ\tau makes the mastery criterion stricter. Table 5 shows that, as τ\tau varies from 0.85 to 0.95, the Spearman correlation with the Table 2 ranking stays above 0.98 on CPA-KQA and 0.96 on CFA-KQA. The cutoff applies to the scale of the specified fit. To check whether the ranking also holds under factor rescaling, we computed SS, the mean reconstructed correctness across items for each concept. Diagonal rescaling leaves SS unchanged. Counting concepts whose SS reaches the midpoint between chance and perfect accuracy gives the following results:

Spearman correlation of model rankingsCPA-KQACFA-KQA
Ranking under SS vs. original Table 2 ranking0.9670.854
Lowest correlation across 20 fits with the seed-42 ranking, using FF0.790.72
Lowest correlation across 20 fits with the seed-42 ranking, using SS0.980.96

The scale-invariant score thus gives a ranking close to the published one on both benchmarks. The seed comparison also shows where the choice of score matters: rankings vary more across fits under FF and are more reproducible under SS.

To address this concern, we will revise §4.2, lines 266 to 276, to state the scale on which the cutoff operates and add the comparison across scoring rules and random seeds to Appendix E alongside the threshold sensitivity results.

AuthorsOpenReview ↗

Q2: Comparison with observed per-concept accuracy

We thank the reviewer for recognizing the value of checking our central finding directly against model responses: two financial LLMs can achieve the same overall score yet excel in different domains. The benchmark’s expert item labels make this comparison possible using observed accuracy alone.

Section 4.3, lines 308 to 311, defines the per-concept accuracy baseline, and Table 3b reports its use in guiding fine-tuning. We also checked the two equal-score examples discussed in §5.1 against observed accuracy. In both cases, the lead changes hands across domains:

Benchmark and model pair (A / B)Overall accuracy (A / B)Domain where A leads (A / B)Domain where B leads (A / B)
CPA-KQA: Grok 3 / GPT-40.62 / 0.62Economic Law: 0.91 / 0.67Tax Law: 0.53 / 0.79
CFA-KQA: GPT-4o / Llama 3 70B0.72 / 0.72Derivative Investments: 0.72 / 0.47Alternative Investments: 0.43 / 0.58

These differences are visible in the responses independently of the factorization and the mastery threshold. Table 3b shows that the downstream conclusion also holds: for Qwen 3 8B, selecting weak concepts by observed accuracy raises accuracy on the external fineval-KQA benchmark from 0.29 before fine-tuning to 0.49 afterward; selection by co-factorization raises it to 0.58. Both methods identify useful training targets, with the larger transfer gain coming from co-factorization in this comparison.

We agree that Figure 3 is too small for readers to inspect individual heatmap values at page size. To address this concern, we will report the observed accuracies alongside Figure 3 in §5.1, lines 397 to 401, and provide a full-size heatmap with labeled concept boundaries in the appendix.

AuthorsOpenReview ↗

W2 and Q3: Training-set overlap, selection effectiveness, and invalid responses

We thank the reviewer for recognizing the practical motivation behind SkillTune: an aggregate score gives little guidance on what to train next, while a concept-level diagnosis identifies weaknesses that can guide the choice of training examples.

SkillTune uses all training questions associated with the concepts diagnosed as weak for each model. Every selected question therefore also appears in the all-concept set. The percentages below show the share of the full training pool that SkillTune uses.

ModelCPA-KQA: conceptsCPA-KQA: questionsCFA-KQA: conceptsCFA-KQA: questions
Qwen 3 0.6B56/70840/1,050 (80%)24/41360/615 (59%)
Qwen 3 8B64/70960/1,050 (91%)26/41390/615 (63%)
Llama 3.1 8B65/70975/1,050 (93%)35/41525/615 (85%)

For Qwen 3 8B, SkillTune used 960 of the 1,050 CPA-KQA training questions (91%). All-concept training added the remaining 90 questions from six concepts diagnosed as mastered. Table 3b compares these two selections with two random baselines at SkillTune’s budget. We ran each random baseline five times, drawing a fresh set of questions or concepts each time, and evaluated all methods on CPA-KQA and on fineval-KQA, a separate external benchmark.

MethodTraining questionsRunsCPA-KQA accuracyfineval-KQA accuracySkillTune advantage on fineval-KQA, pp (95% CI)
SkillTune96010.580.58Reference
Random questions96050.510.50+8.3 [4.5, 12.1]
Random concepts96050.520.51+6.9 [3.4, 10.4]
All concepts1,05010.550.53+5.5 [1.7, 9.3]

Despite the 91% overlap, SkillTune achieved higher accuracy on both benchmarks than either five-run random baseline mean or full-data training. On fineval-KQA, all three advantages were statistically significant in paired comparisons across the 101 test questions (10,000 bootstrap resamples). These results support the value of targeting diagnosed weaknesses even when they cover most of the training pool.

We counted a response as invalid when the parser could not extract an option letter within the 64-token output limit. The mean invalid-response rate was 6.9% on CPA-KQA, 6.0% on CFA-KQA, and 5.7% on fineval-KQA.

To address these concerns, we will report training-set coverage in Table 3, add the run counts and paired confidence intervals to Table 3b, and explain the equal-budget and full-data comparisons at lines 459 to 461. We will report the dataset-level invalid-response rates alongside the output limit and extraction rule in Appendix K.

Reviewer_kpECOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FinCDM evaluates financial LLMs using CPA/CFA curriculum concepts. The authors construct CPA-KQA, with 1,260 Chinese questions covering 70 concepts, and CFA-KQA, with 738 English questions covering 41 concepts. For each concept, three questions are used for diagnosis and evaluation, while fifteen are available for training. A matrix co-factorization model combines the LLM responses with the question–concept annotations to produce concept scores. SkillTune then selects training examples from concepts whose scores fall below a fixed threshold. The experiments evaluate 20 LLMs and fine-tune three backbones. They also test whether CPA-KQA training transfers to the separate 101-question FinEval-KQA set. The paper argues that similar overall accuracy can hide different concept profiles, and that these profiles can help select useful fine-tuning data.

Critical strengths

The datasets would be useful even without the proposed estimator. The CPA/CFA taxonomies, equal coverage per concept, and documented expert authoring and review process make it possible to examine what a benchmark actually tests (Section 4.1 and Appendix D). This is a useful addition to reporting an overall financial QA score. The paper also goes beyond displaying capability heatmaps: SkillTune is tested on three backbones, and the CPA-trained models improve on FinEval-KQA. Since FinEval-KQA was excluded from concept selection and tuning, those results provide some independent evidence that the selected supervision is useful (Section 4.3, Table 3).

Critical weaknesses
  1. In Equations 3–4, the fitted quantities are X ≈ EU and Q ≈ EV, but the capability score in Equation 5 is F = U^T V. For any positive diagonal matrix D, setting E' = ED, U' = D^{-1}U, and V' = D^{-1}V leaves both reconstructions and the objective unchanged, while F becomes F' = U^T D^{-2}V. Even a common scaling changes which concepts cross the fixed threshold of 0.9, and different scalings across latent dimensions can change relative concept scores. I could not find a normalization or calibration that resolves this ambiguity. It matters because F determines the capability profiles, concept counts, and training selections. Fixing the seed in Appendix F makes the output reproducible, but does not establish what its scale means. The threshold analysis uses the same fitted scores throughout, and the six-response case study is explicitly in-sample. These checks do not establish that a low score identifies a concept the model will struggle with on new questions.

  2. The data-selection experiments leave some uncertainty as well. The same three labeled questions per concept guide the selection of training topics and are then used to report the CPA/CFA post-tuning results (Section 4.3). The training questions are disjoint, but the evaluation has still influenced the training procedure. The independent FinEval-KQA results help address this concern. However, the controlled comparison uses only Qwen 3 8B and 101 external questions, with one fine-tuning run for SkillTune and the strongest All-concept control. The external accuracies are 0.58 and 0.53, with no uncertainty estimates. The raw per-concept baseline also uses a different thresholding rule, and its training-example budget is not reported as matched, unlike the random controls. I therefore do not yet see enough evidence that co-factorization is a better guide than selecting the same amount of data directly from concept accuracy.

Questions for authors
  1. Is a normalization or calibration step missing from Equations 3–5 or Appendices F/K? Please explain how F = U^T V and the threshold of 0.9 should be interpreted given the scaling ambiguity above. Do the weak-concept assignments predict performance on new questions better than raw per-concept accuracy?

  2. For Figure 7, what are the reference quantities used to calculate cosine similarity, Pearson correlation, and RMSE? I could not tell how the question subsets were sampled or how much the results vary across subsets and factorization initializations. These details are needed to judge whether three questions support a reliable diagnosis.

  3. Please report how many concepts and training examples each controlled method selects. I would also like to see the comparison against a budget-matched rule using continuous mean concept accuracy. For the external FinEval result, can you provide matched fine-tuning seeds and paired uncertainty over questions, or results on another set withheld from diagnosis and selection?

3 comments
AuthorsOpenReview ↗

W1 and Q1: Scale of F = UᵀV and interpretation of τ = 0.9

We thank the reviewer for recognizing the value of concept-level evaluation: models with similar overall scores can have different financial strengths. The reviewer’s scaling argument gives us a direct way to check whether those differences persist when concept performance is measured on the response scale.

We agree that our paper did not explain how we chose τ=0.9\tau=0.9 or how to interpret it in Eq. (5). We set it before fitting as a conservative cutoff: under the initialization in Appendix F, a concept counts toward Con when its capped score exceeds 0.9 on the reported 0 to 1 scale. As the reviewer observes, diagonal rescaling of the factors preserves EUEU and EVEV while changing F=U⊤VF=U^\top V. The cutoff therefore applies to the scores produced by the specified fit.

To check the effect on Table 2, we used the same fitted factors to compute SS, the mean reconstructed correctness EUEU over the questions for each concept. This score is invariant to the rescaling. We counted concepts scoring at or above the midpoint between chance and perfect accuracy and compared the resulting model rankings:

Spearman correlation of model rankingsCPA-KQACFA-KQA
SS vs. original Table 2 Con0.9670.854
SS vs. observed per-concept accuracy at the same cutoff0.9750.959

The contrast in §5.1 also holds under SS: Grok 3 leads GPT-4 in Economic Law (0.85 vs. 0.68), while GPT-4 leads Grok 3 in Tax Law (0.80 vs. 0.58). Thus the difference in the two models’ strengths is visible under a score that is independent of factor scaling.

To address this concern, we will revise §4.2, lines 266 to 276, to explain the choice of τ\tau and the scale on which it operates. We will add the comparison with observed accuracy and the threshold and seed analyses to Appendix E, and refer to the scale-invariant concept comparison in §5.1.

AuthorsOpenReview ↗

W2 and Q3: Data selection, budgets, and uncertainty

We thank the reviewer for recognizing that concept-level diagnosis can guide the choice of training material and that fineval-KQA provides an independent test of this idea. The reviewer’s questions distinguish two claims: whether training on diagnosed weaknesses helps on new questions, and whether SkillTune selects better training material than a simpler rule at the same budget.

We agree that our paper did not clearly distinguish the roles of the assessment, training, and external evaluation questions (W2). On CPA-KQA and CFA-KQA, we use three questions per concept to identify weaknesses, train on separate questions covering those concepts, and evaluate again on the original assessment. All three models improved on both assessments. All three CPA-trained models also improved on fineval-KQA, a separate external benchmark unused in diagnosis, tuning, or checkpoint selection. The original assessments thus show that concepts identified during diagnosis can guide training on separate examples; fineval-KQA shows that the gains extend to unseen questions.

For the training budgets and uncertainty requested in Q3, the table gives raw accuracies for Qwen 3 8B fine-tuned on CPA-KQA. The final column gives SkillTune’s accuracy advantage in percentage points, with 95% paired bootstrap intervals over the 101 fineval-KQA questions (10,000 resamples).

Selection methodConcepts coveredTraining questionsCPA-KQA accuracyfineval-KQA accuracySkillTune advantage on fineval-KQA (pp)
SkillTune649600.580.58Reference
Per-concept accuracy (at least two errors in three questions)578550.510.49Not estimated
Random questions709600.510.50+8.3 [4.5, 12.1]
Random concepts649600.520.51+6.9 [3.4, 10.4]
All concepts701,0500.550.53+5.5 [1.7, 9.3]

At the same budget of 960 training questions, SkillTune exceeded both random controls on fineval-KQA. For the reviewer’s specific comparison of 0.58 versus 0.53, SkillTune exceeded full-data training by 5.5 percentage points, with a paired interval of 1.7 to 9.3 points across test questions.

For the continuous-mean rule requested in Q3, we ranked concepts by mean empirical correctness and selected the lowest 64, matching SkillTune’s budget of 960 training questions. The two rules selected 60 concepts in common and four distinct concepts each, so 60 of the 960 training questions in each selection differed.

To address this concern, we will revise §4.3, lines 293 to 296, to explain which questions guide selection and which provide independent evaluation (W2). We will add the per-method budgets and paired intervals to Table 3b and revise lines 459 to 461 to distinguish the measured fine-tuning outcomes from the matched-budget comparison of selected concepts (Q3).

AuthorsOpenReview ↗

Q2: Reference quantities in Figure 7

We thank the reviewer for recognizing the practical question behind Figure 7: how much does each additional assessment question contribute to a model’s concept profile? This matters when allocating a fixed question budget across 70 concepts.

We agree that our paper did not specify which questions formed the reference profile in Figure 7 or how they overlapped with the questions used for comparison. For each of the 70 CPA-KQA concepts, the six-question reference pool contained the three benchmark test questions and three additional questions on the same concept from the training pool. Figure 7 compares profiles based on one to five questions per concept with the reference profile fitted to all six. As described in Appendix E.1, the four- and five-question profiles supplement the three test questions with one and two training-pool questions, respectively. Each comparison profile therefore shares its questions with the reference. This analysis was run separately from SkillTune.

In the additional experiment, we used the same 420 questions (210 benchmark test questions and 210 supplementary questions from the training pool). Within each of 100 random splits, we reserved two questions per concept for the reference and used one to four of the remaining questions to estimate the comparison profile. We computed both profiles using the concept score SS, the mean reconstructed correctness for each concept. The table reports mean agreement across splits.

Questions per conceptCosine similarityPearson correlationRMSE
10.8760.4050.380
20.9110.4940.314
30.9250.5420.288
40.9310.5700.274

Adding the second, third, and fourth questions increased Pearson correlation by 0.089, 0.048, and 0.028, respectively. From three to four questions, cosine similarity increased by 0.006 and RMSE fell by 0.014. On all three metrics, the fourth question gave the smallest improvement, reproducing the diminishing gains in Figure 7 with disjoint assessment and reference items.

To address this concern, we will revise Appendix E.1 and the Figure 7 caption to distinguish the original analysis from the disjoint-reference check and report the sampling protocol and the four results above.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the Area Chair and Reviewers

Dear Area Chair and Reviewers,

We thank the Area Chair for overseeing the review process and the reviewers for their thoughtful feedback. Our work connects concept-balanced financial benchmarks, concept-level diagnosis, and targeted fine-tuning within FinCDM. Below, we summarize how our responses address the main categories of concerns.

We agree that our paper did not make several scoring conventions and evaluation protocols sufficiently explicit. Regarding score interpretation and diagnostic stability, we clarified the fixed fitting settings and the prespecified conservative threshold. We also checked the findings using a scale-invariant score and observed per-concept accuracy. Both preserve the contrasting strengths of equal-scoring models, supporting the paper’s central insight across different ways of measuring concept performance.

Regarding the number of questions per concept, we specified the reference profile and question sampling in the original analysis. We added a comparison using disjoint reference and assessment questions, which reproduced the diminishing gains beyond three questions.

Regarding SkillTune’s selection effectiveness, we reported the selected concepts, training examples, and repeated-run results. SkillTune outperformed the five-run means of both random baselines at the same training budget and exceeded full-data training on CPA-KQA and the separate external fineval-KQA benchmark. Paired bootstrap intervals supported all three external advantages. These results show that targeted selection remains useful even when the diagnosed weaknesses cover most of the training pool.

Regarding evaluation transparency, we audited answer extraction with a revised parser and reported the resulting accuracy changes and invalid-response rates. We also clarified the expert study’s inputs and qualitative role and prepared full-page heatmaps for inspecting the diagnostic profiles.

To address these concerns in the paper, we have prepared revisions to the scoring explanation, question-count analysis, training-budget tables, uncertainty reporting, and answer-extraction details. We also explicitly discuss the remaining limitation concerning fine-tuning seeds. The complete datasets and supporting selection and evaluation code will be released. We hope this summary helps the Area Chair assess the evidence provided during the discussion, and we remain available for further questions.

Sincerely, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.