- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies providers competing for agentic-search exposure by rewriting website profiles. WildRegistry replays a real request stream, while Delivery-Evidence Credit uses past delivery to reward or withhold future opportunities. The mechanism improves website selection and retains most adaptive providers’ exposure gains in the tested simulations.
Critical strengths
- The benchmark captures a useful feedback loop: selection determines the evidence available for later provider adaptation. Credit and Withhold have clear operational roles, supported by component ablations and profile-replacement tests.
- Evaluation extends beyond headline scores. The appendix includes human relevance checks, session-level uncertainty estimates, alternative backbones and provider groups, and sensitivity to welfare weights. Offline relevance labels are excluded from the agents’ decisions.
Critical weaknesses
- The reported rewriting equilibrium establishes stability of one prescribed updater. The alternative provider strategies are mostly submitted once, whereas the main updater learns continually. These comparisons do not establish resistance to unilateral adaptive deviations. The appendix scopes this limitation; the main text should consistently use the narrower stability claim.
- Reported confidence intervals condition on realized trajectories. Backbone and provider-group checks are helpful, but independent repetitions of the full coupled adaptation process are missing. It remains unclear how reliably the long-run gains recur when early selections and provider updates change.
Questions for authors
- Can a provider profitably deviate from the settled profiles using a different continual updater while competitors remain fixed? Otherwise, could the paper consistently describe its endpoint as updater stability?
- Can the authors repeat the complete adaptation trajectory and report variation in selection quality and provider exposure, rather than only uncertainty over requests within a run?
4 comments
W2 & Q2. Two independent full replays reproduce DEC's gains.
Prior work shows that hosted LLM services may change over time [4] and that LLM-agent systems can vary across runs [5]. We therefore ran paired PA and DEC experiments in two different periods, keeping the request stream and all other settings fixed, to test the results under different runtime conditions. The two replays produce different early histories: in the first ten active days, 36.6% of requests select different adaptive websites.
| Replay | System | W-nDCG@5 | D-nDCG@5 | Provider GEO |
|---|---|---|---|---|
| 1 | PA | 51.51 | 40.75 | 7.39 |
| DEC | 52.89 | 42.06 | 7.01 | |
| 2 | PA | 50.76 | 40.56 | 7.89 |
| DEC | 52.76 | 41.82 | 6.87 |
- User-side result: In the two replays, DEC improves W-nDCG@5 over PA by 1.38 and 2.00 points, and D-nDCG@5 by 1.31 and 1.26 points.
- Provider-side result: DEC's Provider GEO ranges from 6.87 to 7.01, remaining above the corresponding Original and retaining 63.3%–83.3% of PA's exposure gain.
The experiments show that the variation across periods does not change the main result. The two full runs produce different histories and absolute results, but both reproduce DEC's user-side quality gains and preservation of most provider-exposure gains.
Added in the revised PDF: §5.4 and Table 4 summarize both complete trajectories (pp. 8–9); Appendix F.1 reports their cross-run allocation variation and full outcomes in Tables 14–15 (pp. 30–31).
We hope the clarified equilibrium definition, second-updater experiment, and independent full replays address your questions about long-horizon stability and reproducibility. If any specific concern remains, we would be glad to address it during the discussion.
References
[1] J. Perdomo, T. Zrnic, C. Mendler-Dünner, and M. Hardt. "Performative Prediction." ICML 2020.
[2] A. Narang, E. Faulkner, D. Drusvyatskiy, M. Fazel, and J. L. Ratliff. "Multiplayer Performative Prediction: Learning in Decision-Dependent Games." JMLR 2023.
[3] X. Wang, C.-Y. Yau, and H. T. Wai. "Network Effects in Performative Prediction Games." ICML 2023.
[4] L. Chen, M. Zaharia, and J. Zou. "How Is ChatGPT's Behavior Changing over Time?"
[5] H. Zhu et al. "OAgents: An Empirical Study of Building Effective Agents." Findings of EMNLP 2025.
Thanks for the clarification and additional experiments. The distinction between profile stability and Nash stability addresses my concerns, and the 60-day replays in Appendix F.1 confirm that the quality gains hold across different histories. My concerns are resolved, and I’m sticking with my weak-acceptance recommendation.
Thank you for your thoughtful feedback and encouraging assessment. We are glad that the clarified equilibrium definition and additional full-trajectory experiments have addressed your concerns. We appreciate your recognition of these revisions and your continued support!
We thank the reviewer for drawing a useful distinction between updater-induced stability and a website-level adaptive deviation, and for examining the reproducibility of the long-horizon result.
In short. The second continual updater reproduces the stable endpoint and DEC's gains under a different update rule, while two full-trajectory replays reproduce DEC's gains under different realized histories. We clarify the definition and answer each empirical question below, with the corresponding additions to the revised PDF.
W1 & Q1: We agree that we should describe more precisely what "equilibrium" means in our empirical setting.
Section 3.3 (pp. 3–4) defines our game with two players: the user agent and the adaptive content agents together forming the provider-side player, whose objective is their summed exposure. The provider-side strategy is the shared updater FP, applied separately to each website's provider-local history. The same section defines the empirical approximate rewriting-equilibrium criterion by stability of deployed profile terms under repeated FP updates, and Appendix D.2 (pp. 25–26) gives the tail-window measurements. Under this model, our evidence identifies an empirical approximation to a performative fixed point of the provider-side dynamics. This follows the fixed-point notion of [1]; multiplayer performative-learning work distinguishes this solution concept from Nash equilibrium [2,3].
The deviation in the Q1 instead changes one website's updater while keeping its competitors fixed, which is a website-level Nash test. Our empirical claim concerns provider-side fixed-point stability. We have therefore revise the endpoint in terms of empirical profile stability under the deployed updater (Abstract, §3.3, §5.5, and Appendix D.2). We call this fixed-point criterion an approximate rewriting equilibrium, and §5.4 now limits the best-response comparison to the evaluated strategies.
Robustness across continual updaters. We agree that robustness beyond the original FP is best tested with another continual updater. We therefore reran the complete 60-day PA and DEC trajectories after replacing the shared FP for the full adaptive provider side with a less restricted continual updater. This experiment tests whether DEC's empirical benefit depends on the particular continual updater used by the modeled provider side.
This updater removes the fixed eight slots and lets each content agent freely use a 75-word profile budget. Under PA and DEC, 18.1% and 4.7% of profiles contain more than eight terms, so the added rewriting space changes realized profiles. Over the final 20 active days, 91.5% and 91.0% of profiles keep the same term set, reproducing the long-horizon stability in the original paper.
Added in the revised PDF: The Abstract (p. 1), §3.3 (pp. 3–4), §5.5 (p. 9), and Appendix D.2 (pp. 25–26) clarify the endpoint terminology; §5.4 limits the best-response comparison to the evaluated strategies and Table 4 summarizes the updater-robustness result (pp. 8–9), with the full experiment in Appendix F.2 (p. 31).