- Rating
- 3
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces CreditAgentBench, which is a benchmark that can be executed and measures what is known as aspirational constrained model repair. In particular, the notion of repair in a credit model setting differs from the typical financial benchmark setup and this is what the authors purport to fill the gap for through their proposed benchmark.
The contributions of the paper include a repair protocol, a scoreability-first bank of task cells, and evidence from over one thousand reported assignments.
Critical strengths
- the comparison to other executable benchmarks is well presented and Table 1 visualizes the differences clearly
- the evaluation methodology is described plainly and is easy to follow.
- the resultant findings directly stem from the experimental results and provide ample evidence for subsequent claims that are made.
Critical weaknesses
- the evaluation setup and methodology are extremely nuanced and domain specific. Is this intentional or are there generalizations to what is being evaluated? If not, then is the scope and representative ability of the proposed benchmark intended to be quite narrow ?
Questions for authors
See above.
Response to Reviewer 1g8R
W1: Is the domain-specific setup intentional?
Thank you for the review. Yes. A credit model must rank borrowers, give calibrated probabilities, keep errors low and stay stable over time, and these requirements pull against each other (§4), so the side effects of a repair can be measured. The protocol itself is not credit-specific:
What carries over and what is specific to credit.
We reused this scoring unchanged for training-data actions, a planning agent, a planning router and a coding agent (results in our reply to A4jW). We claim representativeness within tabular credit and financial-risk modeling (six sources; static, retrospective temporal and rolling temporal regimes), not across machine learning, and will add a scope paragraph to §1 and §8.