- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
When a model is used to make decisions, people may change their behavior in response to it. Some changes improve real outcomes, while others only change the inputs used by the model. The paper first shows that historical data alone cannot distinguish these effects: two predictors can agree and make optimal decisions on historical data, yet perform very differently once people respond. It then gives conditions under which these responses improve decisions, including a requirement that enough effort goes into beneficial actions. The authors also propose a test using separate data that approves deployment only when there is statistical evidence of improvement. They evaluate the approach through controlled simulations and credit-data experiments in which actions, responses, and causal effects are simulated. The results support the approach in these settings, although the guarantees depend on assumptions about people’s behavior and the effects of their actions.
Critical strengths
The paper connects strategic behavior, causal effects, and decision quality. It clearly distinguishes encouraging useful effort from actually improving decisions. The paper handles smooth and hard decision rules separately and keeps model selection independent of final testing. The controlled experiments check the main formulas, and the authors clearly state the limits of their experiments.
Critical weaknesses
(i) The guarantees depend on assumptions about causal effects and human responses that may be difficult to check. The credit experiments use synthetic actions, responses, and costs, so they do not validate these assumptions for real borrowers.
(ii) The credit experiments mainly compare different selection rules over the same 84 candidates. They therefore provide limited evidence about the proposed training algorithm and its ability to handle interacting agents.
(iii) The gains over Causal SC-DFL are small. In Table 1, the worst-environment gain is 0.0880 versus 0.0875, and several additional results are nearly identical. Repeated data splits and a direct statistical comparison would help establish whether this difference is reliable.
Questions for authors
In Sections 4.2–4.3, can the effort certificate satisfy for the same predictor and response model? Please provide one fully specified example with a nonzero equilibrium response, and compute both thresholds from the agents' utility, causal effects, and downstream optimizer. In particular, verify that holds along the entire response path. This would show that the incentive and regret guarantees can jointly certify a strict improvement.
6 comments
Response to Critical Weaknesses: Assumption Validity, Training with Interacting Agents, and Statistical Comparisons(2/2)
Critical Weaknesses
W:(ii) The credit experiments mainly compare different selection rules over the same 84 candidates. They therefore provide limited evidence about the proposed training algorithm and its ability to handle interacting agents.
A: the proposed training algorithm and its ability to handle interacting agents. We added a UCI-based semi-synthetic experiment testing end-to-end policy training with interacting agents. We jointly optimize the covariate scoring parameters and three action coefficients over 160 gradient updates. Each action coefficient enters both the score and its corresponding posted incentive, with a fixed action-specific incentive multiplier.
In each cohort, 20 borrowers compete for six slots. Borrowers choose actions using their expected selection probabilities under independent logistic score noise. Complete solves the coupled response and differentiates through the equilibrium during training. Price-taking uses independent responses during training. Both methods share initialization, training batches, and calibration procedures. Both are evaluated under the same full coupled response model, with each realized allocation selecting exactly the six highest scores after adding noise.
We use 128 training cohorts, 128 calibration cohorts, and 256 held-out test cohorts from one UCI data split. Each reported worst-environment metric is the minimum of its four environment means. Gains are relative to the same status-quo policy.
| Worst-environment metric | Complete | Price-taking |
|---|---|---|
| Raw-cost gain ↑ | 0.057615 | 0.045664 |
| Regret gain ↑ | 0.011294 | 0.009011 |
| Alignment ↑ | 0.076206 | 0.057086 |
Complete achieves higher empirical values on all three metrics in this experiment. Both methods select calibration-feasible policies. These single-split empirical results demonstrate the feasibility of training through endogenous quota responses. Formal significance testing and a final deployment certificate remain to be established for this run.
W:(iii) The gains over Causal SC-DFL are small. In Table 1, the worst-environment gain is 0.0880 versus 0.0875, and several additional results are nearly identical. Repeated data splits and a direct statistical comparison would help establish whether this difference is reliable.
A: To assess the reliability of the gain difference reported in Table 1 (0.0880 vs. 0.0875) , we added an experiment using the repeated data splits and direct statistical comparison you suggested. Specifically, we used a shared grid of 84 candidates (12 total incentive budgets × 7 allocations between two beneficial actions), four environments and causal-effect drift in the non-nominal environments.
We repeated the experiment with 10 data-split seeds. For each split, we refit the risk model, selected each method’s policy on the calibration set, and directly compared the two policies on the same 5,000 held-out test cohorts using a paired bootstrap.
| Worst-environment raw-cost gain | Result across 10 splits |
|---|---|
| CS-DFL, mean | 0.06468 |
| Causal SC-DFL, mean | 0.02309 |
| Paired difference, mean | +0.04159 |
| Splits with a positive difference | 10/10 |
| Splits with a paired bootstrap 95% interval entirely above zero | 10/10 |
And the results are clear that CS-DFL achieved a higher worst-environment gain in every split, which means CS-DFL achieves better performance than Causal SC-DFL.
Response to Questions: Joint Effort and Regret Certification with a Nonzero Equilibrium
Questions For Authors:
A: Yes, they can. We provide a fully specified analytical example showing that the effort and regret certificates can hold simultaneously for the same predictor, response model, and downstream optimizer, with .
Consider one strategic agent and one passive competitor sharing one allocation unit, with feasible allocation set and quadratic regularizer . For nonnegative actions , we specify the true costs and predictor as
The downstream optimizer minimizes predicted allocation cost plus the quadratic regularizer, yielding
z_1(a)=\min\left\\{1,\frac12+\frac15a_D+\frac1{10}a_U\right\\}.
The agent maximizes , where . On the unsaturated allocation branch, the first-order conditions are
Solving these conditions yields the unique nonzero equilibrium response
This response lies in the unsaturated region for every and is globally optimal because the allocation function is concave in actions and the quadratic effort cost makes the utility strictly concave.
The corresponding effort returns are and . Their uniform bounds yield and , hence
Along every response path , , the cost gap lies in , and the true unregularized oracle is . Since , the beneficial-action derivative terms satisfy
The corresponding terms for the other action are and . Thus holds uniformly along the entire response path, with and . Regret is throughout these paths.
The uniform derivative bounds yield
Therefore, the regret-improvement threshold is
Combining this with the effort certificate derived above, both certificates hold simultaneously:
The joint certificate guarantees a strict regret improvement:
We further verify the construction symbolically and numerically, including 12 independent optimization checks. At , the equilibrium response is , and the exact regret decreases from to .
A fixed status-quo predictor induces zero effort and regret , so the calculation also establishes improvement over that baseline. Full derivations are in the appendix, “A Fully Specified Joint Incentive and Regret Certificate.”
We are eager to know whether we have addressed your questions, and we would be happy to discuss further at any time to assist with your re-evaluation.
Response to Critical Weaknesses: Assumption Validity, Training with Interacting Agents, and Statistical Comparisons(1/2)
Critical Weaknesses
A: The guarantees admit explicit corrections for bounded assumption violations. We assume valid causal action maps to evaluate post-response regret from changes in features and true costs. Karimi et al. (2021) use structural interventions for causal recourse. Intervention data covering deployment actions can test predicted effects while historical fit cannot establish off-support effects. Bounded mechanism errors enter our guarantees as loss penalties.
We model agents as maximizing decision benefits minus effort costs, following Hardt et al. (2016). Common separable strictly convex costs yield our explicit effort-share formula and Shavit et al. (2020) provide the quadratic special case. With identified payoffs, new incentive conditions can test response predictions and optimality gaps. Our sensitivity analysis permits heterogeneous, coupled costs.
Strong monotonicity of a continuous game operator on a nonempty compact convex action set gives a unique equilibrium, with perturbation stability governed by its monotonicity modulus. Jacobian bounds can check this property within a specified model. This follows the variational-inequality approach of Parise and Ozdaglar (2019).
For our extension, let the nominal operator have modulus μ>0, let its uniform distance from the true operator be at most δ, and let the actual response have true VI dual gap at most ε. Its distance from the nominal equilibrium is bounded by
∣a−a0∣≤2μδ+δ2+4με.
This displacement bound also applies to nonunique true responses satisfying the gap condition. Small μ weakens the bound, and hard decisions near ties require separate analysis.
In our single-agent experiment with exact responses and 500 paired parameter settings, we perturb the effort-cost Hessian to Q=I+ρE, where E=E⊤ and ∣E∣2=1. Counts require positive oracle-regret gain and nonnegative alignment at β=0.6:
The counts above evaluate deterministic perturbation bounds across simulated parameter settings. The pathwise regret envelope converts aligned effort into decision improvement. Shavit et al. (2020) distinguish outcome improvement from prediction accuracy; our envelope adds a decision-specific sufficient condition. Given an identified mechanism, pathwise derivative checks can falsify it. On absolutely continuous paths, integrated positive violation S subtracts S from the improvement bound. In the two-action example, halving the beneficial causal coefficient from 2 to 1 gave mean gain 0.07833 and mean corrected lower bound 0.04667 across 501 fixed grid points for v∈[1,1.5].
Mechanism coverage makes the certificate applicable to deployment, paralleling the valid model-error bounds in Petrik et al. (2016). Held-out interventions can expose misspecification, but nonrejection cannot establish coverage. With valid candidate and baseline loss-error bounds e,e0, the corrected guarantee is Gtrue≥LCBG−e−e0. These experiments assess conditional robustness; validity for real borrowers still requires intervention and response evidence from that population.