• The paper does not validate nor justify the design choices of the user simulation model that is complexe and hand designed.
• The link between the measured proxies and compulsive use is missing. The claim would have been stronger if the proxies are shown to be tracking the user's latent measures that simulate compulsion (e.g. OverrideRate).
Most conclusions of the paper are conditional on the hand-designed user simulator, which is not justified. The paper calls the mathematical forms of this model "experimental assumptions," but it offers no theoretical analysis of the model, no validation of its dynamics against user-level data, and its calibration is not reported. The component ablations in Section 5.4 only show which parts of this particular model drive over-cap exposure. The hand designed model is complex and is used for a leaderboard benchmark, a purpose that requires realism, a validation is therefore warranted. As a result, there is no basis for judging whether the benchmerking capabilities of COMPULSIONBENCH and the reported scorecard (rankings and mitigation effects) reflect anything beyond modeling choices of the authors.
The calibration that is supposed to ground the simulator is not reported. Section 4.1 states that "Appendix C reports the full audit." (Line 286) and Appendix C promises "pre- and post-calibration fits for each target statistic, along with sensitivity to the public-log sessionization threshold Δsess and to key simulator parameters" (Line 1503). Appendix C contains only a qualitative summary (Table 10). This calibration is based on criteria that are mentionned in Section 4.1 ("The released operational objective is lexicographic rather than a flat weighted sum: it prioritizes session-length and session-item-count fit before gaps, normalized hazards, diurnal patterns, cluster watch means, and conditional-return summaries." Line 284) but no value of these criteria or of any fit statistic appears in the paper, and the "operating envelope of the benchmark" (Line 290) that calibration is said to document is never stated. The paper reports that the user simulator is too light-tailed (Line 323), with the longest session reaching 73 steps before stoping versus 9394 in the data (Appendix C: Table 10), and sessions start time more uniform in the time of the day compared to the data (Appendix C: Table 10). These are exactly the properties a benchmark of tail risk and night use depends on. This results in the simulation never reaching the 120 minute threshold by any policy (Section 5.5). The paper uses a 15 minutes threshold in its evaluation results' section, which according to the paper itself is "not a clinical boundary or public-log quantile" (Line 297).
Several claims in the abstract are direct consequences of the design choices rather than informative findings:
• "Over-cap incidence is essentially unchanged." (Abstract: Line 036) The paper sets T_cap = T_ref, then by design the cap does not intervene untill the session length exceeds the threshold that is used to measure the over-cap incidence.
• "Without a short-gap re-entry spike." (Abstract: Line 038) The gaps between sessions are sampled from a logNormal distribution that have fixed paramaeters and that is independent of the state of the simulated user and the recommendation policy. No policy or mitigation strategy can change the length of that gap, so a re-entry spike cannot occur in this simulator.
Among evaluated policies, some optimized ones (such as PPO) don't outperform trivial ones (RoundRobinPolicy for deterministic scorecard Table 1 and Random for stochastic scorecard Table 11) in terms of cumulative watch time. In the deterministic case, round robin even outperforms PPO by apprx 14 min in the cumulative watch time metric. This bring the validity of the experimental results into question.