- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This submission studies settings where forecasts influence the outcomes they predict, along the lines of performative prediction. In Section 2, the authors introduce a repeated interaction where a "learner" publishes a forecast, and the "world" best-responds by picking a "response intensity" according to a quadratic cost function that results in a response intensity that is monotone in the learner's forecast. In Section 3, the authors show that, under sufficient regularity conditions, the learner can learn a calibrated forecast with logarithmic regret by using a simple update rule. In Section 4, they instead consider a setting where responses jump when a forecast crosses a participation threshold. They show that in this setting, unlike the previous section, achieving zero bias requires sacrificing performance accuracy. In Section 5, the authors study the setting where the learner cares about both prediction accuracy and the outcome itself, and characterize the "strategic forecast" which minimizes the forecaster's combined loss function over these two objectives. In Section 6, the authors show that learning the strategic forecast requires \Theta(T^{1/2}) regret, compared to the logarithmic regret, compared to the O(log T) regret which is required for learning an "honest" forecast which just tries to predict the expected outcome after strategization. Finally in Section 7, the authors ask whether it is possible to discourage manipulation when some parts of the model are unknown. They show that a binning-based approach results in O(T^{2/3}) regret in this setting.
Critical strengths
I appreciated that the authors took the time to list several open directions in an appendix.
Critical weaknesses
I had an extremely difficult time reading this submission and parsing its results. The paper is packed full of technical results and jargon, with very little room left for exposition. It is hard to evaluate the quality of the contributions as a result. Some examples of confusing or unnatural writing from just the first few sections are included in the questions section. There is also no justification for the “quadratic world” assumption, and key details such as model timing are often relegated to an appendix.
Writing issues aside, another weakness of this submission is that its conceptual novelty is incremental compared to the vast literature on performative prediction. For example, the original Performative Prediction paper already gives an example of a setting where discontinuous feedback leads to no stable forecast, and the setting of Section 5 where the learner wants to incentivize good outcomes is well-studied. The paper would greatly benefit from an expanded related work section which better delineates this paper's conceptual and technical contributions from the existing literature.
Questions for authors
None. Examples of confusing writing include:
L46: “Under this condition the response is monotone”, but no condition is mentioned
L46: “The outcome then reveals the residual of the current forecast, and monotonicity makes that residual point to a unique honest forecast, which can be learned at logarithmic regret at every level of responsiveness”
L48: “If the response jumps, as it does when agents of discrete types switch on, no forecast is calibrated”
L52: “A stake in the outcome does, and at the honest forecast its marginal incentive splits into conflict of interest and manipulation in the ratio 1 : ε, where ε is the elasticity of the world’s intensity” -> It's unclear what “elasticity of the world’s intensity” means, as it was not introduced before this point
L69: “A strategic learner is therefore miscalibrated, and a binned audit caps its gain at O(T 2/3 ); in the gap, though, the audit sees a departure from calibration but not the objective behind it” -> the “binned audit” terminology was never introduced
L98: “The timing is Stackelberg, with the learner leading”
L127: “If the world is a population of agents who pay a fixed cost to participate, type k switches on only when the forecast exceeds a threshold”