- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents CreditAgent, a hierarchical multi-agent system for credit review, decomposed into three layers: (1) evidence governance, where coarse-filter agents plus an XGBoost-guided feature-selection loop reduce 2,515 raw fields to 100 expert-validated fields; (2) specialist reasoning, where six QLoRA SFT+DPO fine-tuned agents (Seed-OSS-36B backbone) covering Macro/Basic/History/Solvency/Willingness/Fraud publish structured scores and evidence chains to a shared blackboard; (3) decision fusion, combining an attention-weighted risk score, a case-specific threshold learned by GRPO as a contextual bandit over a bespoke Business Efficiency Coefficient (BEC) reward, and deterministic hard-stop veto rules. On a temporally later, risk-enriched 6,000-case proprietary evaluation set (three groups of 2,000: approved-then-defaulted, non-defaulters in low-approval regions, rejected applicants whose default is inferred from external bureau records), the controlled comparison—holding backbone, specialist outputs, hard-stops, and split fixed while varying only fusion—reports 83.32% accuracy and BEC 0.7647 for GRPO fusion vs 78.40%/0.6862 for uniform-average fusion, and +3.15 points accuracy and +9.75 points on approval-boundary (G2) cases over threshold-tuned cost-sensitive XGBoost/CatBoost on a matched split.
Critical strengths
- A real institutional setting with an unusually thorough robustness battery around the controlled comparison. The fusion claim is isolated by holding the backbone, specialist outputs, hard-stops, and split fixed; it is then stress-tested against threshold-tuned static fusion (two tuning objectives), cost-sensitive XGBoost/CatBoost on a matched split with out-of-fold threshold selection, hard-stop removal, GRPO hyperparameter sensitivity, and temporal diagnostics across rate-regime shifts. The limitations section voluntarily discloses the metric inconsistencies, missing per-case diagnostics, and undisclosed match rate — a level of candor above the norm for this subfield.
- An auditable decomposition for high-stakes decision support, with code released. The governance→specialization→fusion pipeline yields a by-construction audit trail (field provenance, per-specialist evidence chains, veto provenance), and the anonymous code release with a worked example reproducing the reported numbers exactly (Appendix D.1) gives the claimed numbers a verifiability that most proprietary-data papers lack.
Critical weaknesses
- The headline metric is internally inconsistent and, as defined, degenerate. BEC appears under at least three incomparable accounting conventions across the paper (Table 3 vs Tables 10–11: 0.6862 vs 0.8055 for the same system and accuracy), with the discrepancy acknowledged but never explained. The consistency penalty is identical (0.970) across all fusion variants and thus uninformative in the core comparison. Most importantly, under the paper's own distribution (≈2/3 defaulters, CFN:CFP = 3:1), rejecting everyone achieves BECpre ≈ 0.889 > 0.7647, and BEC-thuned static fusion indeed reaches 0.900 by collapsing the approval rate to 4.43% — i.e., the metric does not measure the trade-off it is introduced to measure (approval yield enters only through CFP = 1). The paper's response is to pivot to accuracy and per-group accuracy, but BEC remains the abstract- and conclusion-level headline, and the Conclusion's "best BEC among the evaluated fusion variants, including threshold-tuned baselines" is contradicted by Table 10. A related unexplained observation: the learned GRPO policy approves 60–68% of cases even though near-universal rejection is the optimum of its own stated training reward — the source of this gap (capacity, sampling, or the unspecified stability regularizer) is never analyzed.
Questions for authors
- Please reconcile the BEC accounting: exactly why does the released evaluation code yield 0.8055 for average fusion where Table 3 reports 0.6862, with identical accuracy? Restate all results under one convention. Also, given that reject-all achieves BECpre ≈ 0.889 under the paper's distribution and BEC-tuned static fusion reaches 0.900 (Table 10), how do you defend the Conclusion's claim of "best BEC among the evaluated fusion variants, including threshold-tuned baselines"?
- The learned policy approves 60–68% of cases although near-universal rejection maximizes the stated training reward (BECpre under the 3:1 ratio and 2/3 default rate). What precisely is the stability regularizer, and what explains the gap between the learned policy and the optimum of its own objective?
-
- Can per-case AUC/ECE/confusion matrices for the GRPO system be regenerated from the trained policy, to enable ranking-quality comparison with the tabular baselines?