- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper formalizes sequential credit lending as a performative MDP in which the deployed policy shapes both wealth dynamics and future applications. It builds Eutopia, a simulator that joins an RL lender, realized repayments, and social dynamics. Experiments compare policy gradient against its performative counterpart (PePG) across two fairness perspectives, outcome parity versus true-positive-rate parity, and three constraint forms. The paper reports that performative learning performs best, and that opportunity-based utilities reproduce the delayed impact of fairness.
Critical strengths
-
The testbed is a significant contribution. Existing lending simulators move credit scores in response to decisions but sample arrivals exogenously; here the applicant pool itself reacts to approval history.
-
The micro-analysis in Section 4.3 and Appendix F.2.4 is the strongest part of the paper. Lorenz curves, reach rates, and never-served shares show that similar macro metrics hide opposite credit allocations, and reproducing the delayed impact of fairness inside a performative environment connects the paper concretely to Liu et al. (2018) and D'Amour et al. (2020).
-
Reproducibility is strong. Rule-based policies validate the simulator before any learning claim, and code, seeds, and hyperparameters are complete.
Critical weaknesses
-
Table 2 mixes two training protocols. Five cells learn the multiplier by dual ascent; the cell for FL under Equality of Outcome reports the fixed-multiplier policy of Appendix F.3. Under dual ascent, that cell earns a fifth of the profit with a wider gap (Table 7). The conclusion that FL under Outcome closes the wealth gap while keeping two thirds of the maximal profit (Sections 1 and 4.2) therefore describes the fixed-multiplier policy, while Figure 3 and the surrounding discussion describe the dual-ascent run. The authors ought to report the dual-ascent numbers in Table 2, present the fixed-multiplier policy as a separate result, and restate the conclusion. The fixed-weight result is useful on its own and deserves a named place in the main text.
-
SAC weakens the claim that the performative agent beats standard RL on all metrics in every scenario (Sections 1, 4.1, and 5). Under FL with Equality of Outcome, SAC earns more profit with a smaller wealth gap and an inequality ratio closer to one than either performative variant (Table 15). Appendix F.4 argues SAC is a fixed cutoff that concentrates loans within each group, but the fixed-multiplier policy shows similar concentration, and SAC's never-served share is not reported. The authors ought to scope the claim to policy-gradient learners and report SAC's micro metrics for this cell.
-
The comparisons in Table 2 report no significance tests, and several bolded wins are smaller than their standard deviations (Table 7). The wealth gap itself is computed on seed-averaged trajectories, so gaps of opposite sign cancel across seeds, and Appendix F.5 notes the per-seed mean gap is more than three times the value in Table 7. The authors ought to report the per-seed absolute gap and paired per-seed comparisons for the Section 4.1 claims.
Questions for authors
-
Is the FL-under-Outcome cell in Table 2 the fixed-multiplier run of Appendix F.3 while the other cells use dual ascent, and what are the dual-ascent numbers for that cell?
-
For SAC under FL with Equality of Outcome, what are the never-served share, reach rate, and per-group growth rates?
-
Do the Section 4.1 results hold under a paired per-seed comparison on the per-seed absolute gap?
5 comments
Follow-up on our response
Dear Reviewer 5qX8,
Thank you again for your constructive feedback on our work. We submitted our responses a few days ago but have not yet heard back from you. As the discussion period is coming to an end, we would appreciate it if you could revisit your score if our responses have addressed your concerns.
If you have any further questions, please feel free to let us know. We look forward to hearing your thoughts.
Kind regards, Authors
Thank you for your response and the new appendix experiments. My Q2 and Q3 are sufficiently answered. One question remains on Q1:
How was γ* = 1.87 selected? Appendix F.3 says it is where the Outcome and Opportunity multipliers meet during pre-training; the response describes it as the strength that maximizes utility at the dual-ascent wealth gap, which would use deployment outcomes. Also related, Table 2 and contribution (c) still report PeRL-γ* as the FL/Outcome result.
Official Rebuttal – All concerns and questions are addressed in the revised manuscript
We thank the reviewer for their detailed review and appreciating the nuances offered by our proposed simulator. We respond to the concerns raised below.
W1 & Q1. Yes, the FL+Outcome entry of Table 2 corresponds to PeRL-γ⋆ (Appendix F.3), while the other entries use the end-values of learned multiplier. With the end-value of the learned multiplier, this cell yields a profit of 1.304±0.974 (×106), a wealth gap of 71.9±298.0, and ρ=1.059±0.094 (Table 7).
We highlight that the two policies are two points on the same fairness-utility trade-off of FL+Outcome. Under Outcome, the Matthew effect keeps pushing the wealth gap open, so the constraint keeps binding and dual ascent drives the multiplier towards its fairness-holding extreme: the gap closes and loans are spread evenly, at the cost of profit. Under Equality of Opportunity, the constraint is met early and the multiplier decays towards the profit-earning extreme (Figure 3). γ⋆=1.87 is the strength at which we achieve maximal utility with wealth gap as same as with end-value of the Lagrangian multiplier. PeRL-γ⋆ is, thus, the balanced point learned by the same dual-ascent used for other objectives.
With the end-value of the learned multiplier throughout, FL+Outcome remains the best utility for every learner under relaxed wealth dynamics and on two additional datasets (Appendix G.3 and G.4 in the revised manuscript).
W2 & Q2. We agree with the reviewer on the scope i.e., the claim in Sections 1, 4.1 and 5 concerns PeRL and its passive counterpart, i.e., policy gradient, which is the comparison that isolates the effect of performativity. We have added a detailed comparison with SAC in Appendix G.2 in the revised manuscript, and summarise the observations below.
Thus, SAC reaches parity only under FL+Outcome, there it does not improve on PeRL-γ⋆, and only PeRL offers the evenly-served end of the trade-off. For the detailed results, we refer to Appendix G.2.
W3 & Q3: Statistical significance. Yes, the results of Section 4.1 hold under paired per-seed comparisons. The standard deviations of Table 7 measure the variation across seeds, whereas the claim of Section 4.1 is about the difference between the two agents on the same seed, which is what a paired test isolates. We ran exact two-sided paired Wilcoxon signed-rank tests over the 10 seeds on the per-seed profit, the per-seed absolute wealth gap, and the per-seed distance ∣ρ−1∣, with Holm correction over the 21 comparisons. The table below reports, for PeRL against RL as in Section 4.1, the number of seeds (out of 10) in which PeRL is better, along with the per-seed absolute wealth gap of both the agents. Here, ∗ denotes a significant difference.
We observe that PeRL is better than RL in 9 or 10 of the 10 seeds on profit and on ρ for nearly every utility, and on the per-seed absolute gap in every utility. These conclusions remain unchanged when RL is additionally given PeRL's pre-training (Appendix G.1 in the revised manuscript).
We hope we have satisfactorily addressed all the concerns. We are eager to discuss further if there are any further questions/concerns.