- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies realized algorithmic recourse when applicants may implement advice imperfectly, return at action-dependent times, or never return, while the classifier may change. It formulates a cell-level recourse contract bounding terminal failure, characterizes the risk interval compatible with unpaired behavioral and model-path marginals, and develops finite-sample certificates combining behavioral tests with path-wise model tolerance sets. It also derives a certification floor and a recommendation-leverage decomposition. Synthetic experiments probe dependence, timing, and data limitations; a LendingClub study evaluates five generators using real covariates and refitted classifiers with simulated responses and return times. The reported bootstrap-based certificates concern estimation uncertainty around known model updates; a historical backtest yields abstention in every cell, limiting evidence for certification under genuine future drift.
Critical strengths
- The setup is interesting and seems comprehensive, accounting for realized responses, non-return, and the model in force when an applicant returns (Lines 79–89). This broad formulation is an appealing aspect of the work, although I wonder whether all of its machinery is necessary.
- The distinction between finite-sample uncertainty and uncertainty from an unidentified coupling is conceptually valuable. The sharp finite-support characterization identifies what additional marginal data alone cannot resolve (Lines 108–133).
- The seemingly comprehensive literature review also helps situate the work (Lines 323–354).
Critical weaknesses
- Sections 3–5 are very dense and difficult to follow (Lines 108–318). Running examples across different marginal settings, together with a brief explanation of what to do with each proposition, would help convey why the results matter. The contribution summary (Lines 54–67) did not help me appreciate the work sufficiently. Strassen’s condition is also invoked before it is explained (Lines 20–22, 56–58, 123–126), adding to the difficulty.
- The experiments provide limited evidence of practically useful certification under genuine model drift. On LendingClub, only two of ten cells are certified, with failure bounds of 0.77 and 0.98; the historical backtest yields abstention in every cell (Lines 444–462). Moreover, the guarantee assumes historical increment paths are i.i.d. draws from the deployment increment law (Lines 193–195), whereas the authors acknowledge that the historical paths overlap and are not i.i.d. (Lines 1824–1827). These limitations leave the practical scope of the guarantees unclear, even if the certificates are valid under the stated assumptions.
Questions for authors
-
Clarification: Is the explicit response kernel needed in the problem formulation, and could you explain its role relative to the response-record distribution? The setup seems comprehensive, but I wonder whether it is more elaborate than necessary (Lines 79–89).
-
Clarification: What exactly is meant by the recommendation “logs” and the “available logs”? A concrete example of the records in the two unpaired sources would help me understand the setup (Lines 46–50, 96–100, 105–107).
-
Applicability: Under what realistic institutional data-collection process would the i.i.d. model-increment assumption hold, and how much independent model history would be available (Lines 193–195, 1824–1827)? Given the high certified failure bounds and abstention in the historical backtest, in what deployment settings would the resulting contracts be useful (Lines 444–462)?
4 comments
Response to Reviewer 6yvL (Part 2/4)
W2 · Evidence under genuine drift, and the i.i.d. assumption. We agree with the reviewer's reading.
The bootstrap study. The bootstrap paths measure estimation noise around known updates, so the two certified cells (0.77 and 0.98) test the pipeline under a correctly specified drift law, which Section 7 and Appendix D now call the oracle drift law; they are not evidence for certification at issuance. Under that law the eight uncertified cells have , so no dependence-robust contract below that level exists there at any sample size.
Genuine drift (new). We refit the classifier monthly on the previous 12 months of loans (37 models) and, at 17 issue dates in 2013–2015, calibrate RAC only on updates that had already happened, using non-overlapping windows of two monthly updates (consecutive windows share one endpoint model), under a protocol frozen before the run. Every contract RAC issued held: ROAR and Wachter in were certified at 0.90–0.97, against realized failure of 0.35–0.40. The levels are high because 17 windows allow only model budgets (Proposition 4.7); Q3 gives the settings in which levels are useful.
The i.i.d. assumption. We agree that the 2008–2009 paths of the submitted backtest are not i.i.d., and our theory does not cover dependent histories. Their dependence does not explain the abstention, however. RAC abstained because those paths drift far more than the bootstrap paths around the 2014–2015 refits (median score 33.9 against 2.5) and are nearly copies of one another (correlation 0.97 between neighbors, 0.83 over all pairs): their two-year update windows all overlap, so a backtest restricted to non-overlapping paths has . With one path, RAC abstains by construction, and by Proposition 4.7(b) no distribution-free procedure that certifies with probability can certify below (about 0.94 at and ). Overlap could only have made certificates overconfident, and none was issued.
In the new backtest the windows do not overlap. Their scores, which is all the guarantee uses, show no trend (Spearman , ) and no serial correlation, and no realized path is unusual for its history (conformal ), although the updates themselves are mildly correlated between neighboring windows (0.27, against 0.08 over all pairs; a check added after the run). The data are therefore consistent with the assumption in the form the guarantee uses, although 17 windows cannot confirm it.
Response to Reviewer 6yvL (Part 3/4)
Q1 · Is the response kernel needed? No, and the revision removes it from the formulation. Sections 2–7 and Algorithm 1 use only , the law of the response record in a cell. Its role: is the joint law of (from context ), (from ) and , so is the part of that the recommendation acts on. Certification needs only ; is needed only to compare two recommendations (leverage), so it is defined in Appendix A.3 with those results.
The rest of the setup is there because each part changes the answer:
- realized responses: certificates that ignore them are violated in at least 96% of synthetic repetitions and understate realized risk in 7 of 10 LendingClub cells;
- return times: per-update model sets, which ignore when applicants return, miss up to 0.157 against a target of 0.10 (Section 7, Q3);
- non-return counted as failure: keeps the loss computable from records.
Q2 · What are the logs? Two separate tables, and no row of one refers to a row of the other; the new Table 1 shows one record from each.
- Response log (the “available logs”): one row per rejected applicant who received randomized advice, with context , recommendation , features , the realized action and the return time (or no return within the horizon). For example, a near-miss applicant advised to cut the debt-to-income ratio from 31% to 26% cuts it to 28% and returns after one update. Randomization within each context makes the log identify (Assumption 4.2).
- Model history: one row per past update path, the coefficient changes of the refits one and two updates after a past date. On LendingClub it holds either 200 bootstrap paths (the oracle study) or 19 paths from monthly origins in 2008–2009 (the backtest); the new backtest uses the windows of the 2013–2015 monthly refits.
The two sources are unpaired: each logged cohort meets only one realized model path, and the deployment cohort will meet a future path that neither table contains. On LendingClub the covariates and model updates are real; the responses and return times are simulated. In the revision: Assumptions 4.2 and 4.3 and Algorithm 1 use the names of Table 1.
Response to Reviewer 6yvL (Part 4/4)
Q3 · When is the i.i.d. assumption realistic, how much history is available, and where are the contracts useful?
When it is realistic. Updates can be close to i.i.d. when they are routine refits of a stable pipeline, driven by independent shocks, and the update windows of different paths do not overlap; histories can also be pooled across portfolios not driven by the same shock. What the guarantee needs is weaker: it uses past paths only through their scores, so it is the past scores and the deployment score that must be i.i.d. The assumption fails across regime changes: on LendingClub the 2016 update lies above the entire bootstrap distribution (Appendix D.10). Remaining mismatch is what the model-law budget prices. In the revision: Section 4 states this scope under Assumption 4.3.
How much history. Required: a model budget needs about independent update paths when it is fixed in advance (Proposition 4.7), i.e. about 30 for : five years of monthly refits of one portfolio, which add six non-overlapping two-update paths a year, and less when the histories of independent portfolios are pooled. Our experiments choose the budget from a grid of ten at a stricter error level, which needs about twice as many paths. Available: our 2013–2015 monthly refits gave at most 17 windows, enough only for on that grid. Model history, not behavioral data, is therefore the bottleneck.
Where the contracts are useful. Contracts are useful in three settings:
- The institution commits on the model channel. Judging returning applicants, for the contract horizon, by the model that issued the advice needs no model history: on LendingClub RAC then certifies all ten cells (0.37–0.69; eight if applicants exactly on the old model's boundary count as failures), with ROAR and Wachter in at 0.42 and 0.37 instead of 0.77 and 0.98 under the oracle law. Its cost, under our yearly refits, is approving 9–30% of a cell whom the new model rejects, typically within 1–3 points of its predicted-repayment threshold; approving whenever either model accepts adds no rejection. A lighter commitment, capping how far updates may lower advised applicants' scores, certifies ROAR and Wachter in at 0.51 and 0.54, with no override under the 2013–2015 monthly refits (a check added after the run) and overrides for 5–18% of these applicants under yearly updates.
- Enough history exists, for example pooled across portfolios: under the oracle law, ROAR in is certified in every subsample from , and its level falls steadily as history grows, from 0.90 to 0.77 at (Table 20).
- A shared environment is recorded in both sources (Proposition 3.4); in our synthetic experiments this was the tightest certificate valid at every shock intensity.
Outside these settings, only advice that leaves a margin above the decision boundary can be certified (here ROAR and Wachter), and only at high levels; RAC then serves as an audit, showing when the data support no guarantee.
We hope the revision, in particular the clearer Sections 3–5 and the new evidence under genuine drift, addresses the reviewer's concerns, and we would be grateful if the reviewer would consider raising the score.
Response to Reviewer 6yvL (Part 1/4)
We thank the reviewer for a careful reading and an accurate summary, and for finding the setup comprehensive, the separation between finite-sample and identification uncertainty conceptually valuable, and the related work useful. The summary states the key limitation of our experiments precisely; the revision now states it up front and adds the evidence it calls for (W2, Q3). Line numbers refer to the submitted version; section, appendix, table and theorem numbers refer to the revision.
W1 · Density of Sections 3–5. We agree. In the revision, five changes:
(i) A running example now accompanies Sections 3 and 4, across the data settings the reviewer mentions. Applicants advised to lower utilization either comply or substitute by raising income, and the next model rewards either utilization or income, so each model rejects exactly one of the two responses.
(ii) Each main result is now followed by one sentence on what an institution can do with it; for example, after Theorem 4.5: calibrate on whole update paths, not single updates; after Proposition 4.7: a model budget γ needs about 3/γ independent update paths.
(iii) The introduction now opens with a concrete case (an applicant advised to cut the debt-to-income ratio by five points cuts it by three, while the lender retrains its model in a downturn), and the contribution summary is rewritten in plain words, each contribution stating what it means.
(iv) Strassen's condition is no longer used in the abstract or the introduction; it first appears right after Corollary 3.2, with its plain meaning: responses can be paired with future models so that every response meets a model that rejects it.
(v) Section 5 is reduced to its consequence for deployment, retitled “Which logs are needed”, and side results move to the appendix.