acceptodds
Withdrawn from ICLR 2027

Learning to Fuse: Cost-Sensitive Credit Assessment via Hierarchical Multi-Agent Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Credit risk assessment requires both accurate prediction and structured decomposition of how heterogeneous evidence contributes to each decision. Monolithic Large Language Models can incorporate unstructured evidence and natural-language reasoning into such workflows, but in high-stakes underwriting they may be distracted by noisy inputs, miss rare but decisive risk cues, and offer limited control over policy-dependent decision thresholds. We present CreditAgent, a hierarchical credit review system with three stages: evidence filtering, specialist risk analysis by agents, and decision fusion. Our central contribution is holding the adapted backbone, specialist-agent outputs, hard-stop rules, and data split fixed, we vary only the final fusion strategy to isolate the effect of hierarchical fusion on underwriting quality. On a held-out set of 6,000 personal credit cases from Chinese financial institution, CreditAgent achieves 83.32% accuracy and a Business Efficiency Coefficient of 0.7647 outperform flagship model. We present these findings as an institution-specific case study while identifying which components (hierarchical fusion, GRPO training recipe) are mechanism-portable versus institution-specific (hard-stop rules, cost ratios). To ensure reproducibility, we make code and dataset publicly available at https://anonymous.4open.science/r/Credit_Agents-AC45

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LLM application, credit review

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_uebXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents CreditAgent, a hierarchical multi-agent system for credit review, decomposed into three layers: (1) evidence governance, where coarse-filter agents plus an XGBoost-guided feature-selection loop reduce 2,515 raw fields to 100 expert-validated fields; (2) specialist reasoning, where six QLoRA SFT+DPO fine-tuned agents (Seed-OSS-36B backbone) covering Macro/Basic/History/Solvency/Willingness/Fraud publish structured scores and evidence chains to a shared blackboard; (3) decision fusion, combining an attention-weighted risk score, a case-specific threshold learned by GRPO as a contextual bandit over a bespoke Business Efficiency Coefficient (BEC) reward, and deterministic hard-stop veto rules. On a temporally later, risk-enriched 6,000-case proprietary evaluation set (three groups of 2,000: approved-then-defaulted, non-defaulters in low-approval regions, rejected applicants whose default is inferred from external bureau records), the controlled comparison—holding backbone, specialist outputs, hard-stops, and split fixed while varying only fusion—reports 83.32% accuracy and BEC 0.7647 for GRPO fusion vs 78.40%/0.6862 for uniform-average fusion, and +3.15 points accuracy and +9.75 points on approval-boundary (G2) cases over threshold-tuned cost-sensitive XGBoost/CatBoost on a matched split.

Critical strengths
  1. A real institutional setting with an unusually thorough robustness battery around the controlled comparison. The fusion claim is isolated by holding the backbone, specialist outputs, hard-stops, and split fixed; it is then stress-tested against threshold-tuned static fusion (two tuning objectives), cost-sensitive XGBoost/CatBoost on a matched split with out-of-fold threshold selection, hard-stop removal, GRPO hyperparameter sensitivity, and temporal diagnostics across rate-regime shifts. The limitations section voluntarily discloses the metric inconsistencies, missing per-case diagnostics, and undisclosed match rate — a level of candor above the norm for this subfield.
  2. An auditable decomposition for high-stakes decision support, with code released. The governance→specialization→fusion pipeline yields a by-construction audit trail (field provenance, per-specialist evidence chains, veto provenance), and the anonymous code release with a worked example reproducing the reported numbers exactly (Appendix D.1) gives the claimed numbers a verifiability that most proprietary-data papers lack.
Critical weaknesses
  1. The headline metric is internally inconsistent and, as defined, degenerate. BEC appears under at least three incomparable accounting conventions across the paper (Table 3 vs Tables 10–11: 0.6862 vs 0.8055 for the same system and accuracy), with the discrepancy acknowledged but never explained. The consistency penalty is identical (0.970) across all fusion variants and thus uninformative in the core comparison. Most importantly, under the paper's own distribution (≈2/3 defaulters, CFN:CFP = 3:1), rejecting everyone achieves BECpre ≈ 0.889 > 0.7647, and BEC-thuned static fusion indeed reaches 0.900 by collapsing the approval rate to 4.43% — i.e., the metric does not measure the trade-off it is introduced to measure (approval yield enters only through CFP = 1). The paper's response is to pivot to accuracy and per-group accuracy, but BEC remains the abstract- and conclusion-level headline, and the Conclusion's "best BEC among the evaluated fusion variants, including threshold-tuned baselines" is contradicted by Table 10. A related unexplained observation: the learned GRPO policy approves 60–68% of cases even though near-universal rejection is the optimum of its own stated training reward — the source of this gap (capacity, sampling, or the unspecified stability regularizer) is never analyzed.
Questions for authors
  1. Please reconcile the BEC accounting: exactly why does the released evaluation code yield 0.8055 for average fusion where Table 3 reports 0.6862, with identical accuracy? Restate all results under one convention. Also, given that reject-all achieves BECpre ≈ 0.889 under the paper's distribution and BEC-tuned static fusion reaches 0.900 (Table 10), how do you defend the Conclusion's claim of "best BEC among the evaluated fusion variants, including threshold-tuned baselines"?
  2. The learned policy approves 60–68% of cases although near-universal rejection maximizes the stated training reward (BECpre under the 3:1 ratio and 2/3 default rate). What precisely is the stability regularizer, and what explains the gap between the learned policy and the optimum of its own objective?
    1. Can per-case AUC/ECE/confusion matrices for the GRPO system be regenerated from the trained policy, to enable ranking-quality comparison with the tabular baselines?
Reviewer_cUJaOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a three-stage hierarchical multi-agent system for credit assessment. In the first stage, an LLM-assisted, XGBoost-guided pipeline selects 100 features from financial records. In the second stage, six specialised agents, fine-tuned from Seed-OSS-36B using supervised fine-tuning (SFT) and preference optimisation, independently assess creditworthiness and provide their respective supporting explanations. Their outputs are stored in a shared memory, called a blackboard. In the final stage, an attention-based aggregation module combines the agents' scores, with its decision thresholds optimised using GRPO. The paper also introduces the Business Efficiency Coefficient (BEC), a metric that accounts for both asymmetric error costs and disagreement between agents. Experiments on 6K risk-enriched credit records from a single Chinese institution show that the proposed system achieves 83.32% accuracy and a BEC of 0.7647, compared with 78.40% accuracy and 0.6862 BEC for average fusion. The main finding is that learned score aggregation can improve credit assessment, particularly for borderline cases.

Critical strengths
  • Systematic and Modular design of the multi-agent pipeline
    Clearly separating evidence filtering, specialist assessment, and the final decision making gives the system clear components that can be inspected during error analysis and tuned independently. Structured specialist outputs and explicit veto rules would make the decision process easier to inspect.
Critical weaknesses

Inconsistencies in the manuscript

  • Utility definition and claims. Equations (6)-(7) define BEC_pre = 1 - (3*FN + FP)/(3*N). Table 11's full-set Avg counts (FN = 537, FP = 759, N = 6,000) give 0.8683, not 0.8307. Appendix D.2 acknowledges that different BEC scales are used. Further, the conclusion claims the best BEC even against threshold-tuned baselines, although Table 10 explicitly states that these values are not comparable. These inconsistencies make it difficult to reliably assess the claimed improvement in business utility.

  • Discrepancies in the paper and code. Equations (2)-(4) describe an MLP scorer and a learned threshold, while Appendix O requests a direct recommendation. The released scorer instead requests a 0-100 LLM score, with fixed-weight fallback. Its strategy code uses a configured threshold, permits "deep review", and has an explicit fraud check at <= 2, not the paper's <= 3. The feature selector also removes low feature-label correlations rather than the feature-feature redundancy described in Appendix C.1. The authors should clearly identify which implementation was used for the reported experiments and explain these differences.

  • Theoretical statements.
    Theorem B.9 claims that GRPO reduces disagreement between specialist agents, but Appendix D.3 states the specialist outputs are fixed. I cannot understand in what way the final aggregation policy can have any change on the disagreement that existed between those outputs, so I don't think the claimed theorem follows.

Missing baselines and incomplete controlled comparisons

  • The benefit of GRPO is not isolated.
    The paper compares the proposed method with Avg, DST, and tuned tabular baselines, but does not clearly report a supervised fusion classifier trained on the same frozen specialist scores and evidence embeddings. An MLP or XGBoost stacking baseline, using the same data and veto rules, would help establish whether GRPO is even necessary and whether it provides any benefit over standard supervised fusion. The hybrid experiment in Section 5.2 replaces the specialist agents with XGBoost scores while retaining Layer 3, so it does not answer this particular question. Further, evaluating the learned attention scorer with a globally tuned threshold on the validation set would help separate the benefit of learned score weighting from that of learned threshold selection.

  • The tuned baselines and GRPO should be tested on the same applicants.
    Table 10 evaluates the simpler methods on 4,104 applications after tuning their approval thresholds, whereas GRPO is evaluated on 6,000 applications. Although the authors describe this difference as representative, the reported performance gap could partly be due to the different test sets rather than the methods themselves. Evaluating both GRPO and the tuned baselines on the same held-out applicants would provide a fairer comparison and help establish whether GRPO retains its advantage over the tuned baselines.

  • Insufficient details
    The optimisation procedure is not described in sufficient detail. Section 4.1 introduces a trainable attention MLP, a threshold policy, and a stability regulariser, but does not clearly explain the MLP's training objective, how the threshold actions are sampled, or how the regulariser is defined. These details are important for understanding what GRPO actually optimises. Without a clear specification, it is difficult to reproduce the training procedure or determine which parts of the optimisation contribute to the reported improvements.

Overall comment

The contribution beyond prior work is not clearly established. Hierarchical financial agents and multi-stage credit assessment have already been explored in prior work, including FinCon and MASCA, while GRPO is an existing optimisation method. The potentially novel aspect of CreditAgent is its combination of evidence filtering with cost-aware learned fusion. While this combination could be useful, the paper does not isolate the end-to-end benefit of the filtering stage or compare GRPO against supervised fusion using the same specialist outputs. Along with the unresolved inconsistencies in the utility calculations, these gaps make it difficult to establish the methodological and empirical contribution of the paper beyond integrating existing techniques into a single system.

Questions for authors
  • Please give the exact per-case reward and advantage formulas. If rewards are standardized within each applicant’s group, how is the 3:1 false-negative/false-positive cost ratio preserved in the policy updates?
Reviewer_tLMHOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes CreditAgent, a three-layer system for personal credit approval. Layer 1 reduces 2,515 raw fields to 100 through LLM-based coarse filtering, iterative correlation and XGBoost-importance pruning, and expert review. Layer 2 consists of six specialist agents (Macro, Basic, History, Solvency, Willingness, Fraud), each a QLoRA-adapted Seed-OSS-36B model trained with SFT and DPO on a 40,000-case corpus. Each agent emits a 1–10 score and an evidence chain, and the agents are described as communicating through a shared blackboard. Layer 3 fuses the scores with an attention-weighted sigmoid, compares the result against a case-specific threshold from a GRPO-trained policy, and applies a fraud veto (rfraud≤3r_{\text{fraud}} \le 3). The evaluation set contains 6,000 cases from 2023 at a Chinese lender, in three equal groups: approved and later defaulted (G1), non-defaulters in a low-approval region (G2), and rejected applicants inferred to have defaulted (G3). The paper introduces a Business Efficiency Coefficient (BEC) with a 3:1 false-negative to false-positive cost ratio and a penalty on inter-agent disagreement. It reports 83.32% accuracy and BEC 0.7647 for the full system, against 78.40% / 0.6862 for uniform averaging of the specialist scores, and a 3.15-point accuracy advantage over threshold-tuned XGBoost and CatBoost.

Critical strengths
  1. The evaluation data include outcomes for rejected applicants and emphasize approval-boundary cases. These properties are absent from public credit benchmarks and relevant to deployment. The evaluation window is also temporally later than the SFT corpus.

  2. The appendix reports several analyses that work against the paper's own framing. For example, tuned tabular models obtain higher AUC and better calibration than the fused specialist score (Appendix D.4), and BEC computed by the released code does not match Table 3 (Appendix D.2). This candor allows the problems discussed below to be identified, but it does not resolve them.

Critical weaknesses
  1. The headline results are not obtained under a clean protocol and cannot be reproduced.

    • GRPO is trained on 1,200 of the 6,000 evaluation cases (Appendix D.4), yet the headline 83.32% is computed on all 6,000 (Table 3; Table 10 caption). This contradicts Section 5.1 ("the reported evaluation does not reuse adaptation data"). Table 15 likewise pairs a full-set GRPO row with tabular results on the 4,800 held-out cases, although Appendix D.5 shows that per-case GRPO decisions can be regenerated.
    • BEC does not follow from Eq. (6)–(7). The reported group accuracies imply a GRPO BEC of about 0.908 before the consistency penalty (about 0.881 after it), not 0.7647. Table 11's Avg confusion matrix implies about 0.868 before the penalty, against 0.8307 in that table and 0.6862 in Table 3. BEC is also reported for XGBoost and Random Forest, for which the agent-dispersion term σ_agents is undefined.
    • A reject-all baseline (66.7% accuracy; BEC of about 0.889 before the penalty) is missing; GPT-5.1 falls below it. The group accuracies imply an approval rate of about 27.5%, inconsistent with the 60.1–67.9% reported in Section 5.2. No variance estimates are given.
  2. Key components are not implemented as described, and the hierarchy shows no discriminative advantage.

    • The Layer 2 prompts (Appendix O) contain no upstream outputs, and the 247–573 prompt tokens per specialist (Table 22) are too few to include upstream evidence chains of 874–1,340 tokens each. The shared blackboard, and Theorem B.4 that depends on it, therefore appear not to describe the evaluated system.
    • The "Macro" agent assesses application reasonableness rather than macroeconomic data (Appendix O, Table 9). The Basic agent outputs 1 in all four case studies, contrary to its own rule of returning 5 when data are missing. Layer 3 outputs Approve/Reject directly rather than a threshold, and the GRPO reward in Figure 4 (5.044) exceeds the stated bound of the reward.
    • The fused specialist score has lower AUC (0.815) than XGBoost or CatBoost trained on only 1,200 cases (0.846–0.847). These tabular baselines receive less data than the specialists (1,200 vs. 40,000 cases) and are restricted to the 100 selected fields.
    • G3 consists of applicants the institution rejected, so G3 accuracy can be obtained by imitating the existing rejection policy, and only G1 versus G2 tests default prediction. G3 also omits rejected applicants who later repaid, and its labels come from field families that overlap with the inputs (Table 18).
Questions for authors
  1. Please report all Table 3 rows and the Table 15 comparison on the 4,800 held-out cases only, excluding the 1,200 GRPO training cases. Since Appendix D.5 regenerates per-case GRPO decisions, can you also provide the GRPO confusion matrix, AUC, and ECE on these cases?
  2. Please give the exact formula behind the Table 3 BEC values and reconcile it with Eq. (6)–(7) and Table 11. How is σagents\sigma_{\text{agents}} defined for XGBoost and Random Forest? What are the BEC and accuracy of a reject-all policy?
  3. Do Layer 2 agents actually receive upstream outputs? If so, where do they appear in the prompts of Appendix O, and how is this consistent with the 247–573 prompt tokens in Table 22?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.