acceptodds
Under review as a conference paper at ICLR 2027

CreditAgentBench: Measuring Agentic Credit-Model Repair under Production Constraints

Abstract

Repairing an existing credit model differs from developing a new one. A model may rank borrowers well but miscalibrate default probabilities, or improve on a development sample while weakening under temporal shift. A repair is useful only if the other operational requirements remain intact. Credit and financial-risk data make this tension concrete because ranking, calibrated probabilities, prediction error, and temporal stability matter together. We introduce CreditAgentBench, an executable benchmark of aspirational constrained model repair. Each task fixes an incumbent, an improvement goal, and protected boundaries that must each be met; a participant explores on visible data and commits one final model. Evaluation reports progress toward the goal and the margin of the least-satisfied constraint. The task bank has 57 Natural and Controlled cells from six sources. In the matched 27-task Controlled comparison within the 1,026-assignment reported visible-development set, HPO attains higher average margins, while the dynamic agent crosses both boundaries slightly more often. That joint-validity advantage does not persist when the committed models are replayed on sealed data. Five further model–provider backends change both the agent–HPO comparison and the fraction of runs that yield a scoreable model. These results show why repair progress, joint validity, and final-model availability must be reported separately.

Primary area:
datasets and benchmarks
Keywords:
benchmarking, AI agents, credit risk, constrained model repair, AutoML

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1g8ROpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces CreditAgentBench, which is a benchmark that can be executed and measures what is known as aspirational constrained model repair. In particular, the notion of repair in a credit model setting differs from the typical financial benchmark setup and this is what the authors purport to fill the gap for through their proposed benchmark.

The contributions of the paper include a repair protocol, a scoreability-first bank of task cells, and evidence from over one thousand reported assignments.

Critical strengths
  1. the comparison to other executable benchmarks is well presented and Table 1 visualizes the differences clearly
  2. the evaluation methodology is described plainly and is easy to follow.
  3. the resultant findings directly stem from the experimental results and provide ample evidence for subsequent claims that are made.
Critical weaknesses
  1. the evaluation setup and methodology are extremely nuanced and domain specific. Is this intentional or are there generalizations to what is being evaluated? If not, then is the scope and representative ability of the proposed benchmark intended to be quite narrow ?
Questions for authors

See above.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 1g8R

W1: Is the domain-specific setup intentional?

Thank you for the review. Yes. A credit model must rank borrowers, give calibrated probabilities, keep errors low and stay stable over time, and these requirements pull against each other (§4), so the side effects of a repair can be measured. The protocol itself is not credit-specific:

What carries over and what is specific to credit.

Reused unchanged (§3)Specific to credit
Data roles, incumbent, goal, non-compensatory constraints, repair grammar, feedback, budget; one committed artifact; target and worst-constraint margins; sealed replayDatasets, protected metrics, MCID scales, production audits

We reused this scoring unchanged for training-data actions, a planning agent, a planning router and a coding agent (results in our reply to A4jW). We claim representativeness within tabular credit and financial-risk modeling (six sources; static, retrospective temporal and rolling temporal regimes), not across machine learning, and will add a scope paragraph to §1 and §8.

Reviewer_A4jWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the issue that existing machine learning agent benchmarks generally focus on development from scratch, ignoring the actual operation and maintenance of industrial existing models. The authors propose the CREDITAGENTBENCH benchmark, which formalises the problem of repairing existing models under production constraints. They establish a multi-scenario task set using a non-compensatory two-dimensional evaluation and a final submission-based assessment paradigm.

Critical strengths
  1. Extract the common industrial scenario of 'targeted repair of legacy models without affecting production metrics' into a standard academic problem.
  2. The proposed non-compensatory two-dimensional assessment, final submission evaluation, three-level data isolation mechanism, and acceptance criteria are quite close to real production.
  3. The paper is well written and the visualisation is appropriate.
Critical weaknesses
  1. The core contribution focuses on the benchmark framework and experimental observations, without proposing new agent architectures or optimisation algorithms; agents can only search within pre-set hyperparameter spaces, limiting decision space and showing weak technical innovation.
  2. The study only revolves around credit risk control scenarios and traditional tabular models, not covering deeper repair scenarios such as deep learning models, code-level repair, or feature engineering, limiting the generality and applicability of findings.
  3. Key findings are mostly confirmatory, lacking in-depth analysis at the principle level or groundbreaking theoretical insights, with academic contributions leaning more towards engineering practice than theoretical breakthroughs.
Questions for authors
  1. The agents in the paper only adopt a simple three-vehicle routing strategy. Would introducing stronger agent architectures with long-term planning or code generation capabilities significantly improve repair performance? Why was a more complex agent paradigm not chosen for comparison?
  2. The benchmark currently supports model repair only at the hyperparameter level. Are there plans to extend it to more sophisticated repairs such as feature engineering, model structure modification, or code-level repair? What core challenges would such an expansion pose to the existing task definition and evaluation framework?
  3. Experiments show that different model backends greatly affect agent performance. Were the calling configurations for each backend (prompt templates, inference parameters, token limits, etc.) kept consistent? Could differences at the calling level interfere with comparisons of backend capabilities?
Reviewer_cNZqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CreditAgentBench, an executable benchmark for repairing an existing credit-risk model while satisfying protected metric constraints. Participants explore candidates using development feedback, commit one final model, and have that model evaluated on sealed data. The benchmark separates target improvement, constraint satisfaction, and the availability of a scoreable submission. It contains 57 task cells across six data sources, covering natural and controlled defects. Experiments compare LLM-based controllers, HPO, rule-based control, and native modeling systems. HPO achieves better average margins in the matched controlled setting, while the agent’s small development-set advantage in joint validity does not persist on sealed evaluation.

Critical strengths

I think the paper addresses an important practical problem. When improving an existing model, a gain on one metric is not enough if it comes at the expense of other requirements. The benchmark makes this trade-off explicit by separating target progress from constraint satisfaction. This lets us see both how much a method improves the intended weakness and whether the resulting model remains within the specified boundaries. I particularly like that good performance on one metric cannot compensate for a violation elsewhere. That makes the evaluation useful for studying model maintenance, where several requirements need to hold at the same time.

The controlled comparison is another strength. Giving the search methods the same candidate space, tools, and candidate budget makes it easier to assess what the controller contributes. The decision to evaluate the actual committed model is also well motivated: finding a good candidate during search and choosing the right model to submit are different abilities. Scoring only the submitted artifact on sealed data keeps that distinction visible. Reporting score coverage alongside predictive outcomes further helps readers distinguish a method that produces reliable submissions from one that performs well only when it manages to return a scoreable result. Together, these choices give a fuller picture of the methods’ behavior than reporting their best search result alone.

Critical weaknesses

I am still unsure how to interpret the low success rates. None of the Natural-track methods achieves joint validity in the original campaign, and both the agent and HPO fail on all static Controlled tasks. I understand that the benchmark deliberately does not guarantee a feasible solution. Still, it is hard to tell whether these results reflect poor search, a restrictive model space, or goals that cannot be met together with the constraints. The continuous margins are useful for showing progress, but they do not tell us which explanation applies. A larger-budget search on a few representative tasks would help, as would checking whether the conclusions change with less demanding repair targets.

The paper emphasizes that finding a good candidate and choosing the right one to submit are different problems, but the experiments leave their individual effects unclear. The archive-to-final counts use different evaluation stages, as the authors acknowledge, so the gap does not establish a selection failure. The precommitted baseline also changes two things at once: it follows a fixed search schedule and submits the last candidate. Its worse performance could come from either choice. I would find this analysis more convincing if different selectors were compared on the same candidate archive, using the same visible feedback.

Questions for authors

Same as the weakness section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.