- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces PP-Gym, a framework for evaluating performative prediction algorithms under different response dynamics and feedback conditions. It provides 13 synthetic tasks and compares 26 methods and variants using a shared evaluation protocol. The authors also propose TR-CLS, which combines loss surrogate modeling, historical response estimation, and trust-region optimization. TR-CLS outperforms the selected baselines on nine tasks, while additional experiments examine how environmental changes and evaluation settings affect algorithm performance.
Critical strengths
- PP-Gym provides a useful common interface for comparing performative prediction methods. Its treatment of deployment budgets, feedback access, and population dynamics makes the comparisons more consistent.
- The experimental evaluation is fairly extensive. The authors compare multiple algorithms, report statistical tests, and examine environmental changes and learning costs. The distinction between online performance and final-rule performance is particularly useful.
Critical weaknesses
- The contribution of PP-Gym relative to existing frameworks, particularly performativeGYM, is not sufficiently demonstrated. The paper describes additional features, but does not provide a direct comparison showing their practical benefits in terms of extensibility, implementation effort, or experimental coverage.
- The evaluation is limited to low-dimensional synthetic environments. TR-CLS relies on relatively simple loss surrogates and response models, which appear well suited to these settings. Its performance on more complex or misspecified response models remains unclear. Moreover, the proposed algorithm combines several established techniques, and the experiments do not fully establish its advantages over other surrogate-based approaches.
Questions for authors
- What practical advantages does PP-Gym offer over performativeGYM? Could the authors provide a direct comparison using the same tasks or algorithms?
- How does TR-CLS perform in higher-dimensional settings or with more complex response functions that cannot be well approximated by low-order polynomials?
- Does TR-CLS retain its advantages when compared with other surrogate-based methods under similar computational budgets?
5 comments
2. How does TR-CLS perform in higher-dimensional settings or with more complex response functions that cannot be well approximated by low-order polynomials?
Thank you for suggesting higher-dimensional tasks and more complex responses. This helped us add experiments with real data and nonlinear responses to better assess the method's applicability.
Credit-11D uses borrowers' financial and credit features from GiveMeSomeCredit to predict serious delinquency. Its deployed classifier has 10 feature weights and an intercept, giving 11 parameters. Spambase-58D uses 57 email features from UCI Spambase to distinguish spam from legitimate email. It has 58 deployed parameters with the intercept. Both tasks use real data and simulate how features respond to the deployed classifier. Table R2 shows that TR-CLS achieves the lowest mean online loss on both tasks.
(For detailed descriptions of the two tasks, see Section 3.1 on p. 5 and Appendix A.10 on p. 18.)
Table R3 reports results for the 58-dimensional task with saturating responses. TR-CLS has the lowest mean excess online population risk at all six saturation levels. controls response saturation. At , the response is linear. Larger values lead to earlier saturation.
Table R2. Mean online loss on the real-data tasks.
| Method | Credit-11D ↓ | Spambase-58D ↓ |
|---|---|---|
| TR-CLS | .647 | .201 |
| Damped RRM | .650 | .218 |
| Anderson RRM | .650 | .212 |
| Pooled RRM | .647 | .202 |
| Momentum RGD | .658 | .204 |
| Two-stage plug-in [1] | .657 | .202 |
Table R3. Mean excess online population risk on the 58-dimensional task with saturating responses.
| Saturation parameter | TR-CLS ↓ | Pooled RRM ↓ | Two-stage plug-in [1] ↓ | Momentum RGD ↓ | RGD ↓ |
|---|---|---|---|---|---|
| 0 | .00721 | .0194 | .0220 | .0165 | .0157 |
| 0.25 | .00721 | .0190 | .0215 | .0163 | .0155 |
| 0.5 | .00721 | .0180 | .0202 | .0156 | .0149 |
| 1 | .00732 | .0148 | .0165 | .0136 | .0135 |
| 2 | .00822 | .0103 | .0111 | .0120 | .0129 |
| 4 | .00895 | .00928 | .00927 | .0138 | .0156 |
3. Does TR-CLS retain its advantages when compared with other surrogate-based methods under similar computational budgets?
Thank you for suggesting comparisons with other surrogate-based methods under similar computational budgets. This helped us account for computational cost and compare performance under the same resource limits.
We compare the two strongest surrogate baselines we evaluated. Two-stage plug-in [1] uses a response model, and Pooled RRM uses empirical risk from historical data. On Spambase-58D, TR-CLS has the lowest mean online population risk under both .3-second and 1-second learning budgets (Table R4). This holds for both CPU and wall-clock time.
Table R4. Mean online population risk under a shared learning-time budget on Spambase-58D.
| Timing method | Learning time budget (s) | TR-CLS ↓ | Pooled RRM ↓ | Two-stage plug-in [1] ↓ |
|---|---|---|---|---|
| CPU time | .3 | .201 | .202 | .202 |
| CPU time | 1.0 | .201 | .202 | .202 |
| Wall-clock time | .3 | .201 | .202 | .202 |
| Wall-clock time | 1.0 | .201 | .202 | .202 |
[1] John P. Miller, Juan C. Perdomo, Tijana Zrnic. Outside the Echo Chamber: Optimizing the Performative Risk. ICML, 2021.
Thank you for the rebuttal and clarifications. While some concerns have been addressed, my main concerns remain unresolved. Therefore, I maintain my original score.
Thank you for recognizing PP-Gym's unified interface, comparison protocol, and evaluation of both online performance and final rule quality.
1. What practical advantages does PP-Gym offer over performativeGYM? Could the authors provide a direct comparison using the same tasks or algorithms?
Thank you for suggesting a direct comparison with performativeGYM. This helped us add comparisons on the same tasks and clarify the practical differences between the frameworks.
Table R1 compares the frameworks on five tasks using RGD, RRM, and DFO. PP-Gym is faster in all 15 comparisons. Speedups range from 5.47× to 381×, with a median of 59.4×.
Table R1. Framework runtime and final loss. Speedup is the runtime of performativeGYM divided by that of PP-Gym.
PP-Gym can reuse the same SPSA implementation and interaction loop across all three settings:
The performativeGYM calls are: