acceptodds
Under review as a conference paper at ICLR 2027

PP-Gym: Unifying Environments, Learning, and Evaluation for Performative Prediction

Abstract

Performative prediction (PP) studies learning when deployed decisions reshape subsequent data, as prices change demand. Existing implementations often tie algorithms to particular response models and feedback assumptions, making reuse and comparison across settings depend on custom integration. We introduce PP-Gym, a unified framework that separates environments, observations, learning algorithms and evaluation through a common deployment–feedback interface. It supports reusable algorithm implementations and comparisons with controlled information access, population responses and deployment counts. Across 13 tasks spanning immediate and gradual responses and individual-record or aggregate feedback, we evaluate 26 methods and variants. Building on this shared interface, we introduce the Trust-Region Cross-Loss Surrogate (TR-CLS), a simple learner that models deployment-dependent loss and selects new decisions from individual records or aggregate feedback. Under the shared protocol, TR-CLS significantly lowers average loss during learning on nine tasks relative to baselines selected on development data, including a 14.3% increase in revenue in simulated ten-product pricing. These results demonstrate the value of studying shared learning strategies across PP settings; PP-Gym provides the experimental infrastructure to develop them and test where their gains persist.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
performative prediction, benchmarking, simulation environments, decision-dependent distribution shift, surrogate optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_162ZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PP-Gym, a framework for evaluating performative prediction algorithms under different response dynamics and feedback conditions. It provides 13 synthetic tasks and compares 26 methods and variants using a shared evaluation protocol. The authors also propose TR-CLS, which combines loss surrogate modeling, historical response estimation, and trust-region optimization. TR-CLS outperforms the selected baselines on nine tasks, while additional experiments examine how environmental changes and evaluation settings affect algorithm performance.

Critical strengths
  1. PP-Gym provides a useful common interface for comparing performative prediction methods. Its treatment of deployment budgets, feedback access, and population dynamics makes the comparisons more consistent.
  2. The experimental evaluation is fairly extensive. The authors compare multiple algorithms, report statistical tests, and examine environmental changes and learning costs. The distinction between online performance and final-rule performance is particularly useful.
Critical weaknesses
  1. The contribution of PP-Gym relative to existing frameworks, particularly performativeGYM, is not sufficiently demonstrated. The paper describes additional features, but does not provide a direct comparison showing their practical benefits in terms of extensibility, implementation effort, or experimental coverage.
  2. The evaluation is limited to low-dimensional synthetic environments. TR-CLS relies on relatively simple loss surrogates and response models, which appear well suited to these settings. Its performance on more complex or misspecified response models remains unclear. Moreover, the proposed algorithm combines several established techniques, and the experiments do not fully establish its advantages over other surrogate-based approaches.
Questions for authors
  1. What practical advantages does PP-Gym offer over performativeGYM? Could the authors provide a direct comparison using the same tasks or algorithms?
  2. How does TR-CLS perform in higher-dimensional settings or with more complex response functions that cannot be well approximated by low-order polynomials?
  3. Does TR-CLS retain its advantages when compared with other surrogate-based methods under similar computational budgets?
5 comments

Thank you for recognizing PP-Gym's unified interface, comparison protocol, and evaluation of both online performance and final rule quality.


1. What practical advantages does PP-Gym offer over performativeGYM? Could the authors provide a direct comparison using the same tasks or algorithms?

Thank you for suggesting a direct comparison with performativeGYM. This helped us add comparisons on the same tasks and clarify the practical differences between the frameworks.

Table R1 compares the frameworks on five tasks using RGD, RRM, and DFO. PP-Gym is faster in all 15 comparisons. Speedups range from 5.47× to 381×, with a median of 59.4×.

Table R1. Framework runtime and final loss. Speedup is the runtime of performativeGYM divided by that of PP-Gym.

TaskMethodperformativeGYM runtime (s) ↓PP-Gym runtime (s) ↓Speedup ↑performativeGYM final loss ↓PP-Gym final loss ↓
LinearRGD1.35.00354381×-3.99-3.99
LinearRRM.592.00268221×-3.99-3.99
LinearDFO1.38.023259.4×-2.01-2.01
NonlinearRGD.254.0033675.6×-0.0232-0.0232
NonlinearRRM.315.0039779.4×-0.0232-0.0232
NonlinearDFO.395.032312.2×-0.241-0.241
MixtureRGD.663.018735.4×-0.00379-0.00379
MixtureRRM25.1.221114×0.6040.604
MixtureDFO1.86.15312.1×-0.0407-0.0407
PricingRGD.988.00884112×-0.00000355-0.00000355
PricingRRM13.6.124110×00
PricingDFO1.30.034737.4×-1.92-1.92
CreditRGD3.14.057354.7×0.8440.844
CreditRRM29.2.84234.6×0.6230.623
CreditDFO2.29.4195.47×1.471.47

PP-Gym can reuse the same SPSA implementation and interaction loop across all three settings:

from ppbench.agents import SPSA
from ppbench.env_real import by_name_any, make_base_env

for name in ("Credit-11D", "Stateful-credit-11D", "Bandit-credit-11D"):
    task = by_name_any(name)
    env = make_base_env(task)
    agent = SPSA(task.public(), seed=0)
    obs, _ = env.reset(seed=0)
    for _ in range(env.horizon):
        action = agent.act(obs)
        obs, reward, *_ = env.step(action)
        agent.observe(action, obs, reward)

The performativeGYM calls are:

# Stateful response: custom distribution map
dmap = StatefulLinear(
    exp.n, exp.seed, exp.A0, exp.A1, exp.STD, rate=.2
)
params = exp.init_model()
opt = RGD(params, lr=exp.lr, loss_fn=exp.loss_fn,
          proj_fn=exp.proj_fn)
for _ in range(T):
    x, y = dmap.sample(params)
    params = opt.step(params, x=x, y=y)

# Scalar feedback: custom optimizer adapter
opt = ScalarDFO(exp.init_model(), exp.lr, exp.proj_fn, exp.seed)
for _ in range(T):
    query = opt.ask()
    x, y = exp.distribution_map.sample(query)
    opt.tell(float(jnp.mean(exp.loss_fn(query, x=x, y=y))))

2. How does TR-CLS perform in higher-dimensional settings or with more complex response functions that cannot be well approximated by low-order polynomials?

Thank you for suggesting higher-dimensional tasks and more complex responses. This helped us add experiments with real data and nonlinear responses to better assess the method's applicability.

Credit-11D uses borrowers' financial and credit features from GiveMeSomeCredit to predict serious delinquency. Its deployed classifier has 10 feature weights and an intercept, giving 11 parameters. Spambase-58D uses 57 email features from UCI Spambase to distinguish spam from legitimate email. It has 58 deployed parameters with the intercept. Both tasks use real data and simulate how features respond to the deployed classifier. Table R2 shows that TR-CLS achieves the lowest mean online loss on both tasks.

(For detailed descriptions of the two tasks, see Section 3.1 on p. 5 and Appendix A.10 on p. 18.)

Table R3 reports results for the 58-dimensional task with saturating responses. TR-CLS has the lowest mean excess online population risk at all six saturation levels. κ\kappa controls response saturation. At κ=0\kappa=0, the response is linear. Larger κ\kappa values lead to earlier saturation.

Table R2. Mean online loss on the real-data tasks.

MethodCredit-11D ↓Spambase-58D ↓
TR-CLS.647.201
Damped RRM.650.218
Anderson RRM.650.212
Pooled RRM.647.202
Momentum RGD.658.204
Two-stage plug-in [1].657.202

Table R3. Mean excess online population risk on the 58-dimensional task with saturating responses.

Saturation parameter κ\kappaTR-CLS ↓Pooled RRM ↓Two-stage plug-in [1] ↓Momentum RGD ↓RGD ↓
0.00721.0194.0220.0165.0157
0.25.00721.0190.0215.0163.0155
0.5.00721.0180.0202.0156.0149
1.00732.0148.0165.0136.0135
2.00822.0103.0111.0120.0129
4.00895.00928.00927.0138.0156

3. Does TR-CLS retain its advantages when compared with other surrogate-based methods under similar computational budgets?

Thank you for suggesting comparisons with other surrogate-based methods under similar computational budgets. This helped us account for computational cost and compare performance under the same resource limits.

We compare the two strongest surrogate baselines we evaluated. Two-stage plug-in [1] uses a response model, and Pooled RRM uses empirical risk from historical data. On Spambase-58D, TR-CLS has the lowest mean online population risk under both .3-second and 1-second learning budgets (Table R4). This holds for both CPU and wall-clock time.

Table R4. Mean online population risk under a shared learning-time budget on Spambase-58D.

Timing methodLearning time budget (s)TR-CLS ↓Pooled RRM ↓Two-stage plug-in [1] ↓
CPU time.3.201.202.202
CPU time1.0.201.202.202
Wall-clock time.3.201.202.202
Wall-clock time1.0.201.202.202

[1] John P. Miller, Juan C. Perdomo, Tijana Zrnic. Outside the Echo Chamber: Optimizing the Performative Risk. ICML, 2021.

Thank you for the rebuttal and clarifications. While some concerns have been addressed, my main concerns remain unresolved. Therefore, I maintain my original score.

Reviewer_FiDvOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PP-Gym, a unified benchmark for performative prediction that separates learning algorithms, observations, environment dynamics, and evaluation. It includes 13 synthetic tasks covering different response and feedback settings and evaluates 26 methods and variants. The authors also propose TR-CLS, a surrogate-based method that supports both sample-level and aggregate feedback and achieves promising empirical results. The paper further investigates how environmental changes and evaluation stages affect algorithm performance.

Critical strengths
  1. The paper provides a unified framework that separates learning algorithms, observations, environment dynamics, and evaluation. This facilitates comparisons across methods under consistent settings and enables the study of individual components.
  2. The benchmark covers three important settings: stateless environments with sample feedback, stateful environments with sample feedback, and aggregate-only feedback. Integrating these settings into a single framework is a useful contribution.
  3. The authors also introduce TR-CLS, with distinct approaches for sample-level and aggregate feedback. The empirical results are encouraging, showing significant improvements over the selected baselines on 9 of the 13 tasks.
  4. The distinction between online and frozen evaluation is particularly interesting. The results show that algorithm rankings can vary depending on the evaluation stage, highlighting an important consideration in performative prediction.
Critical weaknesses
  1. All 13 tasks are synthetic and relatively low-dimensional (with at most 10 parameters), and several have analytically tractable objectives. This raises concerns about generalization to more complex settings. As discussed in [1], performative effects can behave differently in high-dimensional and overparameterized regimes. Including such tasks could make the benchmark more comprehensive and reveal whether the observed advantages persist in these settings.
  2. Frozen evaluation is limited to 40 deployments. In stateful environments, this may be insufficient for the population to fully adapt, as also suggested by Proposition A.2. Reporting results over different evaluation horizons (e.g., 40, 120, and 500 deployments) would help assess the stability of the reported rankings.
  3. Although deployment and sample budgets are carefully controlled, computational costs receive limited attention. Comparing wall-clock runtimes and examining how surrogate-fitting costs scale with dimensionality would make the evaluation more complete.
  4. As a minor point, I could not find a clear link to the complete implementation in the manuscript. Since this is primarily a benchmarking paper, releasing the code and evaluation scripts would be valuable for reproducibility and future research.

[1] E. Cyffers, A. Mirrokni, and M. Mondelli. "Optimal Regularization for Performative Learning." International Conference on Machine Learning (ICML), 2026.

Questions for authors
  1. Have the authors considered including higher-dimensional or overparameterized tasks to evaluate how well TR-CLS and the baselines generalize beyond the current settings?
  2. How sensitive are the frozen evaluation results and algorithm rankings to the evaluation horizon? Could the authors report results for longer horizons?
  3. Could the authors provide wall-clock runtime comparisons and discuss how the computational cost of TR-CLS scales with dimensionality?
  4. Is the complete benchmark implementation, including baselines and evaluation scripts, available or planned for release?
3 comments

Thank you for recognizing the framework's unified support for the three settings and the value of online and frozen evaluation.


1. Have the authors considered including higher-dimensional or overparameterized tasks to evaluate how well TR-CLS and the baselines generalize beyond the current settings?

Thank you for suggesting higher-dimensional tasks. This helped us add experiments with real data and examine how the methods perform as dimensionality increases.

Credit-11D uses borrowers' financial and credit features from GiveMeSomeCredit to predict serious delinquency. Its deployed classifier has 10 feature weights and an intercept, giving 11 parameters. Spambase-58D uses 57 email features from UCI Spambase to distinguish spam from legitimate email. It has 58 deployed parameters with the intercept. Both tasks use real data and simulate how features respond to the deployed classifier. Table R1 shows that TR-CLS achieves the lowest mean online loss on both tasks.

(For detailed descriptions of the two tasks, see Section 3.1 on p. 5 and Appendix A.10 on p. 18.)

Table R1. Mean online loss on the real-data tasks.

MethodCredit-11D ↓Spambase-58D ↓
TR-CLS.647.201
Damped RRM.650.218
Anderson RRM.650.212
Pooled RRM.647.202
Momentum RGD.658.204
Two-stage plug-in [1].657.202

2. How sensitive are the frozen evaluation results and algorithm rankings to the evaluation horizon? Could the authors report results for longer horizons?

Thank you for suggesting a longer frozen evaluation horizon. This helped us check whether the comparison results change with evaluation length.

We extended frozen evaluation from 40 to 120 and 500 deployments. The risk ordering between TR-CLS and the best baseline remains unchanged on all 13 tasks (Table R2).

Table R2. Mean frozen population risk across evaluation horizons.

TaskMethod40 frozen deployments ↓120 frozen deployments ↓500 frozen deployments ↓
Stateful-10DTR-CLS-69.5-69.4-69.4
Stateful-10DTwo-point ZO-64.7-64.6-64.6
Stateful-fast-3DTR-CLS-20.9-20.9-20.9
Stateful-fast-3DTwo-point ZO-19.2-19.2-19.2
Stateful-slow-3DTR-CLS-19.6-19.8-19.9
Stateful-slow-3DHistory PerfGD-17.5-17.4-17.3

3. Could the authors provide wall-clock runtime comparisons and discuss how the computational cost of TR-CLS scales with dimensionality?

Thank you for suggesting runtime measurements across dimensions. This helped us quantify the costs of learning and response fitting and report them more fully.

From 8 to 58 dimensions, TR-CLS uses .65–.75 seconds of wall-clock learning time for 120 deployments. Response fitting accounts for .03–.07 seconds of this total. TR-CLS has the lowest mean online population risk at all four dimensions (Table R3).

Table R3. Mean online population risk and wall-clock learning time on Spambase feature subsets.

Deployed parameter dimension dTR-CLS population risk ↓Pooled RRM population risk ↓Two-stage plug-in [1] population risk ↓TR-CLS wall-clock learning time (s) ↓
8.494.494.494.650
16.445.445.446.664
32.284.285.286.699
58.201.202.202.748

Dimensionality increases by a factor of 7.25, while learning time increases by only about 15%. This indicates modest growth in computational cost with dimensionality.


4. Is the complete benchmark implementation, including baselines and evaluation scripts, available or planned for release?

Thank you for asking about code availability. Your question helped us clarify the contents of the submitted material.

The supplementary material open-pp-gym contains PP-Gym's core algorithms and implementation code.

[1] John P. Miller, Juan C. Perdomo, Tijana Zrnic. Outside the Echo Chamber: Optimizing the Performative Risk. ICML, 2021.

Thank you for the detailed rebuttal and the additional experiments. I appreciate the authors' efforts to address my concerns. I believe these additions have substantially strengthened the paper and addressed my main concerns to a reasonable extent. I will therefore maintain my original score, which remains supportive of acceptance.

Thank you for your continued support and for recognizing how the additional experiments strengthened the paper.

Reviewer_yaJWOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PP-Gym, a framework for performative prediction (PP), where deployed decisions change subsequent data distributions. Following Gymnasium style, It separates environments, observations, learners, and evaluation. It also proposes TR-CLS, which fits deployment-dependent loss and temporal dependence from samples or aggregate feedback, then searches an equilibrium surrogate within a trust region. On synthetic tasks and across several baseline methods, TR-CLS shows improved performance.

Critical strengths
  1. The unified interface, through Gymnasium, is useful for the community, though I'm not sure whether it has been widely adopted in this research area.
  2. The proposed TR-CLS method shows improved performance across multiple synthetic tasks and baseline methods.
Critical weaknesses
  1. The used benchmark is limited to synthetic tasks. It will be better to include more complex and realistic tasks to demonstrate the generality of the proposed method.
  2. It looks like the proposed method improves in most settings, but drops for 2 tasks, could the authors provide more analysis on this?

Since I'm not an expert in this area, I cannot comment more useful suggestions. Please correct me if I have any misunderstanding.

Questions for authors

Please see the weaknesses.

3 comments

Thank you for recognizing the value of the unified interface to the community and the performance of TR-CLS across multiple tasks.


1. The used benchmark is limited to synthetic tasks. It will be better to include more complex and realistic tasks to demonstrate the generality of the proposed method.

Thank you for suggesting more complex and realistic tasks. This helped us add experiments with credit and email data to further assess the method's performance on real data.

Credit-11D uses borrowers' financial and credit features from GiveMeSomeCredit to predict serious delinquency. Its deployed classifier has 10 feature weights and an intercept, giving 11 parameters. Spambase-58D uses 57 email features from UCI Spambase to distinguish spam from legitimate email. It has 58 deployed parameters with the intercept. Both tasks use real data and simulate how features respond to the deployed classifier. Table R1 shows that TR-CLS achieves the lowest mean online loss on both tasks.

(For detailed descriptions of the two tasks, see Section 3.1 on p. 5 and Appendix A.10 on p. 18.)

Table R1. Mean online loss on the real-data tasks.

MethodCredit-11D ↓Spambase-58D ↓
TR-CLS.647.201
Damped RRM.650.218
Anderson RRM.650.212
Pooled RRM.647.202
Momentum RGD.658.204
Two-stage plug-in [1].657.202

2. It looks like the proposed method improves in most settings, but drops for 2 tasks, could the authors provide more analysis on this?

Thank you for looking closely at the task results and pointing out these two differences. Your question helps us explain the task mechanisms and the learning process behind these losses. It also helps clarify why the benefit of TR-CLS varies across tasks.

Classification-2D learns an intercept and a slope. The feature distribution of one class stays fixed. The mean of the other class moves with the deployed slope. The learner must separate the classes while adapting to the distribution its own classifier creates.

Regression-1D learns a single coefficient, which also changes the target. For an input of 1, consider raising the coefficient from 0 to 0.4. The prediction rises from 0 to 0.4, while the mean target falls from 1 to 0.4. The prediction then matches the mean target.

These tasks have only 1–2 parameters and simple responses. Updates based directly on the current batch are already effective here. This leaves less room for response modeling to improve performance. TR-CLS learns the response from past deployments and then searches for the next rule. Several methods reach similar losses in these simple settings. Table R2 shows that multiple methods are within 5% of the lowest mean online loss on each task (bold).

Online loss also includes the cost of early learning. In classification, the mean online loss gap between TR-CLS and Anderson RRM is about .00164. About 84% of this gap comes from the first 20 deployments. In regression, the first 20 deployments also account for about 53% of the online loss gap between TR-CLS and RGD. A substantial part of the gap therefore arises during early learning.

For intuition, consider a spam filter with a 50% decision threshold. Spam probabilities of 88.0% and 88.1% both lead to a spam label. Probabilities of 12.0% and 11.9% both lead to a legitimate label. The confidence changes, while the predicted label can stay the same.

Table R2. Mean online loss on the two low-dimensional tasks.

MethodClassification-2D ↓Regression-1D ↓
TR-CLS.127.0534
Anderson RRM.126.0493
CMA-ES.224.192
Cross-loss.231.754
Cross-loss (diagonal).251.754
Cross-loss (no memory).227.736
Damped RRM.126.0495
Delayed RRM.13220.7
EXP3.8088.54
GP-LCB.191.422
Guided ZO.148.162
Hold.693.547
Lazy deploy.163.0591
Momentum RGD.154.0527
One-point DFO.177.341
RGD.157.0496
Random search.3931.41
SPSA.190.206
Scalar surrogate.265.763
Two-point ZO.154.163
UCB.5345.19
Parametric PerfGD.211.0523

[1] John P. Miller, Juan C. Perdomo, Tijana Zrnic. Outside the Echo Chamber: Optimizing the Performative Risk. ICML, 2021.

I appreciate the authors' response. Since I'm not an expert in this domain, I would like to maintain current assessment. Please also lower the weight of my assessment. Thank you.

Thank you for your kind words about our response and for the time you devoted to reviewing our paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Area Chair,

Thank you for the time and care you have devoted to overseeing our submission. For your convenience, we provide below a concise summary of each reviewer’s post-rebuttal stance and the main points discussed.

Reviewer Stance (Post-Rebuttal)

  1. Reviewer FiDv (initial score 3 → latest recorded score 3) supported the paper and noted in their follow-up that the additional experiments strengthened it. In the initial review, FiDv wrote, “The distinction between online and frozen evaluation is particularly interesting.” Their main questions were whether 40 deployments allowed enough time to evaluate a final rule and whether the results carried over to higher dimensions at a reasonable computational cost. Extending evaluation to 120 and 500 deployments preserved the ordering of TR-CLS and the best baseline on all 13 original tasks. On real email data, increasing the classifier from 8 to 58 parameters raised learning time from 0.65 to 0.75 seconds. On pricing and regression tasks with 8,192-dimensional responses, TR-CLS achieved lower mean online risk than all six baselines on each task.

  2. Reviewer yaJW (initial score 3 → latest recorded score 3) gave a positive initial recommendation and responded positively to our rebuttal. They described the unified interface as “useful for the community” and asked how the method performs on more realistic data and why it trails simpler methods on two original tasks. The two original tasks have simple one- or two-parameter responses that current-batch updates can track well. On the classification task, TR-CLS achieved a mean online loss of 0.127, close to the best reported value of 0.126. On real borrower and email records with simulated responses to deployment, TR-CLS matched the lowest reported mean online loss on Credit to three decimal places and achieved the lowest on Spambase.

  3. Reviewer 162Z (initial score 2 → latest recorded score 2) recognized the value of a common evaluation protocol and followed up on two main concerns: what PP-Gym offers over performativeGYM in practice, and whether TR-CLS remains effective when responses are harder to model. We implemented the same pricing, classification, and regression tasks in both frameworks and showed which deployment, feedback, and evaluation code PP-Gym lets researchers reuse. For a periodic response that the fitted model approximated poorly, adding the trust region reduced mean online risk by 3.97% with the response model and deployment history held fixed. On two California Housing tasks, TR-CLS achieved 8.04% and 6.97% lower mean online risk than a neural surrogate under the same computational budget. They have not yet replied to our follow-up.

Across five tasks and three learning algorithms, PP-Gym ran faster than performativeGYM in all 15 matched comparisons, with a mean speedup of 89.6× and a median of 59.4×. The two implementations reached the same final loss to the reported precision in every comparison. TR-CLS achieved the lowest mean online risk among the methods tested on both 8,192-dimensional tasks and on two tasks using California Housing data, one with nonlinear shifts and one with a dead-zone response. On MNIST digit classification, its mean online risk was 9.41%–21.91% lower than those of regularized pooled retraining, a kernel surrogate, and TuRBO-1.

Thank you again for your time and for guiding our submission through this review cycle. We hope the above summary helps streamline your decision process.

Warm regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.