acceptodds
Under review as a conference paper at ICLR 2027

Practical Performative Policy Learning with Strategic Agents

Abstract

This paper studies performative policy learning, where agents adjust their features in response to a deployed policy to improve their potential outcomes, thereby inducing endogenous distribution shifts. While there is growing interest in strategic environments—including strategic classification hardt2016strategic and performative prediction perdomo2020performative—existing approaches often rely on restrictive parametric assumptions, such as micro-level utility models in strategic classification or macro-level distribution maps in performative prediction, which limit scalability. In this paper, we relax parametric assumptions on both micro-level agent behavior and macro-level data distribution. We identify a practical information structure in performative distribution shifts and introduce the evaluation vector—defined as the policy values at all feasible points a given agent can reach—as an effective mediator on the causal path from the deployed model to the shifted data. We propose a gradient-based policy optimization algorithm that uses a differentiable classifier as a surrogate for the high-dimensional distribution map. Our methodology exploits batch feedback, achieving superior sample efficiency over bandit feedback that summarizes responses into scalars. We provide theoretical convergence guarantees and demonstrate the method's efficacy in challenging high-dimensional settings.

Primary area:
optimization
Keywords:
Performative prediction, strategic classification, causal inference, policy learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_AYwKOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies performative policy learning when agents strategically modify their features in response to a deployed policy. Its central idea is to represent the policy’s influence through an evaluation vector containing treatment probabilities at feasible feature choices. This enables learning a differentiable behavior model and estimating performative policy gradients, connecting causal policy learning with optimization under endogenous distribution shifts.

Critical strengths
  • Clear motivation and formulation. The paper motivates the problem well and presents an accessible, clearly posed setting.
  • A valuable conceptual connection. The evaluation-vector representation provides a compelling bridge between causal inference, strategic behavior, and performative prediction, without requiring a prespecified parametric utility model.
  • Simple and intuitive gradient decomposition. Equation (2) is a powerful tool that clearly separates the direct policy effect �0��0� from the environment shift effect �1��1�.
  • Convincing experiments. The synthetic and semi-synthetic results show substantial improvements over the baselines. I appreciate the noisy-response, behavior-model, and discretization experiments, alongside diagnostics of the performative correction.
Critical weaknesses
  • The theoretical guarantee could be stronger. Theorem 4.4 controls min⁡t≤T∥∇θV(πθt)∥2\min_{t\le T}\|\nabla_\theta V(\pi_{\theta_t})\|^2, with residual error (GE+Bn)2(G_E+B_n)^2 from estimation bias and batch noise. This establishes approximate stationarity of some iterate, rather than a vanishing optimality gap V⋆−V(πθT)V^\star-V(\pi_{\theta_T}), where V⋆V^\star is the best achievable performative value. For nonconcave neural policies, stationary points can be suboptimal. The authors acknowledge this distinction; an explicit value guarantee under additional structure would strengthen the analysis.
  • The theorem assumes uniformly accurate behavior estimation in an RKHS norm that controls derivatives. It remains unclear when the training procedure and warm-up ensure this condition. Deterministic best responses can also violate the smoothness and positivity assumptions, leaving a gap between the guarantee and the main experiments.
Questions for authors
  1. In the smooth multinomial-logit experiment, can you compare learned and oracle performative gradients, including directional agreement?
  2. At a fixed interaction budget, would deliberately perturbing warm-up policies improve response-gradient estimation?
  3. How robust are the gains to heterogeneous manipulation costs and weaker relationships between manipulable features and outcomes, particularly in the lending experiment?
Reviewer_tiXROpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper studies performative policy learning with binary treatment, where agents adjust their manipulable features in response to a deployed policy. The features are divided into a finite manipulable component and a fixed component. The authors assume that agents respond to the policy through its evaluation vector: the treatment probabilities at the alternatives available to them.

The method learns a conditional response model using this evaluation vector rather than the policy parameters. It combines the model’s derivatives with an estimated CATE to construct a strategic policy gradient. Synthetic and semi-synthetic experiments show improvements over some simple baselines.

Critical strengths

S1. Practical modeling choice.

Conditioning the response model on policy evaluations rather than policy parameters is a useful modeling choice. It reduces the dimension of the policy input when the set of manipulable alternatives is small and allows the same approach to be used across different policy architectures.

S2. Empirical improvements.

The improvements over the included baselines are substantial (although stronger baselines can be used).

Critical weaknesses

W1. Gap between the theory and the main experiments.

The theory studies smooth, positive response probabilities, whereas the main experiments generate deterministic switches between alternatives, which violate the smoothness assumption. The theoretical guarantee therefore does not cover these experiments.

W2. Missing baselines.

The experiments show gains over the included baselines, but do not directly compare against zeroth-order policy optimization. For example, a comparison with the DFO baseline evaluated by Miller et al. (2021), using the same CATE estimator and matched deployment and observation budgets, would help assess the claimed sample-efficiency advantage.

Questions for authors

Q1. How do the authors connect the smooth, positive response probabilities assumed by Theorem 4.4 with the deterministic best responses used in the main experiments?

Q2. How does SPG compare with the DFO baseline evaluated by Miller et al. (2021), using the same CATE estimator and matched deployment and observation budgets?

Reviewer_pnwTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies performative policy learning where a principal deploys a binary treatment policy, and agents adjust a manipulable feature in response. The key idea is to model the response distribution as a function of the evaluation vector i.e. the policy's values at all feasible manipulable states for an agent with fixed features vv. Two assumptions support this: the manipulable feature is discrete and finite (motivated by bounded rationality), and the policy affects responses only through the evaluation vector.

A differentiable behavior model (an MLP or GP classifier) estimates p(u∣v,ζ)p(u | v, \zeta) and is combined with a meta-learner CATE estimate in a REINFORCE-style gradient (Strategic Policy Gradient, SPG). Theorem 4.4 gives convergence to a neighborhood of a stationary point, with a radius governed by behavior-model, CATE, and sampling errors. In synthetic and semi-synthetic (loan default) experiments, SPG substantially outperforms the CATE cutoff rule, a vanilla policy gradient that ignores the shift, and an end2end baseline that conditions on θ\theta directly.

Critical strengths
  • The central idea is intuitive. Treating the deployed function, summarized by its evaluations at reachable states, as the intervention instead of its parameterization makes performative learning tractable for neural policies.

  • The proposed method turns each batch of agent responses into supervised training data for the behavior model, which is a meaningful sample-efficiency advantage over bandit-feedback approaches.

Critical weaknesses
  • There seems to be a mismatch between the algorithm and the theorem for the behavior model. In Algorithm 1, the behavior model is trained once on warm-up data (line 11) and not updated afterward. The warm-up data come from about 30 vanilla-gradient iterates near the cutoff policy, while SPG then moves the policy far from that region. Theorem 4.4, in contrast, assumes a time-indexed estimator qtq_t with a uniform RKHS-norm error bound ϵH\epsilon_H over all epochs, all vv, and the full domain Ω=[0,1]∥U∥\Omega = [0, 1]^{\|U\|}, including evaluation vectors never observed during training.

  • The experiments only cover beneficial manipulation. In both settings, raising the manipulable feature (linearly increasing weights on uu in the synthetic data, and g(u,v)g(u, v) increasing in uu in the semi-synthetic data) causally improves outcomes. The optimal policy therefore rewards higher uu for everyone, and Figs. 2b and 5b show SPG moving nearly all agents to the top level. There is no setting with non-causal features, where manipulation games the policy without improving outcomes, which is the central concern of the strategic classification literature the paper positions itself against.

  • The end2end baseline feeds a 25,741-dimensional θ\theta, concatenated with vv, into an MLP trained on about 100 distinct θ\theta values. Its failure confirms that raw parameters are a poor input. A stronger comparison could use a low-dimensional functional embedding of the policy, such as predictions at random probe points. Zeroth-order methods are excluded by argument rather than evaluated.

  • Because the response mechanism is known in simulation, the true performative optimum could be computed as an oracle upper bound. Without it, it is unclear how close SPG comes to optimal.

Questions for authors
  1. Under the deterministic best responses used in the main experiments, p(u∣v,ζ)p(u | v, \zeta) appears to be piecewise constant in ζ\zeta with zero-probability classes. How do the authors reconcile this with Assumptions 4.2 and 4.3.2? Is the exact score-function density term zero almost everywhere in this case, and if so, does the method's success rely on the smoothing introduced by the learned behavior model?

  2. Is the behavior model updated after warm-up? If not, how is the uniform error bound justified when the policy moves far from the warm-up region? How does performance change if the behavior model is retrained online?

  3. Can the authors evaluate a setting with non-causal (gaming) features, where some manipulation raises uu without improving outcomes?

  4. How does the method scale when the manipulable feature uu is multi-dimensional?

1 comment

Dear Reviewer,

Thank you for your time and for the comments provided in your initial review.

We have carefully addressed your concerns through detailed clarifications, additional experiments, and revisions to the manuscript. As we have not yet had the opportunity to hear your thoughts on these updates, we would sincerely appreciate it if you could revisit our responses and reconsider your assessment in light of the new evidence.

If any concerns remain unresolved, we would be more than happy to provide further clarification.

Thank you again for your time and consideration!

Comments on the paper

1 comment

Follow-up on Author Responses

We sincerely thank the reviewers and AC for their time and consideration.

As the discussion period is approaching its end, we would greatly appreciate any further feedback from the reviewers, particularly on whether our responses, additional experiments, and manuscript revisions have adequately addressed their main concerns.

We would also be grateful if the AC could help facilitate any remaining discussion. We are happy to provide further clarifications before the deadline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.