acceptodds
Under review as a conference paper at ICLR 2027

CompulsionBench: A Benchmark for Auditing Compulsion-Risk Proxies in Sequential Recommendation

Abstract

We introduce CompulsionBench, a reproducible benchmark for auditing and mitigating compulsion-risk proxies in sequential recommendation under explicit, non-clinical assumptions. CompulsionBench asks when engagement-oriented policies interact with long-horizon user dynamics to increase observable behavioral proxies, including prolonged sessions, late-night session starts, rapid-return diagnostics, and watch time accumulated beyond a fixed session threshold. It also asks which mitigation strategies reduce those proxies under matched evaluation. CompulsionBench combines a partially observed user simulator, tunable platform controls for personalization and reward variability, mitigation interventions including session caps and observable-risk constrained optimization, a public-log calibration and feasibility-audit protocol, and a standardized scorecard that separates observable compulsion-risk proxies from latent mechanism diagnostics. The released benchmark provides two plug-compatible user-response backends: a transparent parametric simulator used as the reference environment, and an LLM-augmented semantic robustness backend that preserves the same observation space, action space, interventions, calibration targets, and scorecard while perturbing the item-response kernel under the same benchmark interface. CompulsionBench defines two benchmark tasks, engagement maximization and observable-risk constrained optimization, and reports tail-risk and mitigation metrics for every policy. In the released active-threshold profile, high-watch deterministic heuristics produce higher over-cap exposure than PPO, while a simple session cap reduces PPO over-cap exposure from to minutes at a watch-time cost. Episode-level diagnostics further show that the cap is a tail-severity intervention: over-cap incidence is essentially unchanged, but deterministic p95 over-cap exposure falls from to minutes without a short-gap re-entry spike. A reference 120-minute profile shows that the same over-cap channel can be inactive, highlighting the need to report threshold provenance and proxy activation. CompulsionBench makes audits of compulsion-risk proxies and mitigation strategies reproducible while remaining explicit about construct-validity limits.

Primary area:
datasets and benchmarks
Keywords:
Benchmarking, Evaluation, Sequential Recommendation, Reinforcement Learning, User Simulation, Simulation Environment, Policy Evaluation, Long-Horizon Optimization, Constrained Reinforcement Learning, Tail Risk (CVaR), Engagement Optimization, User Behavior Modeling, AI Safety, Responsible AI

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_djRSOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents a reproducible benchmark COMPULSIONBENCH for auditing and mitigating compulsion-risk proxies in sequential recommendation under explicit, non-clinical assumptions. COMPULSIONBENCH asks when engagement-oriented policies interact with long-horizon user dynamics to increase observable behav-ioral proxies. This benchmark also evaluates which mitigation strategies reduce those proxies under matched evalua-tion. Experiments compare simple heuristics, PPO, Lagrangian PPO, and a session cap. The main finding is that the cap substantially reduces over-threshold watch time at about a 4% cost in total watch time.

Critical strengths
  1. The released benchmark provides two plug-compatible user-response backends. I think it will be user-friendly and contribute to the community.

  2. I also like the separation between observable behavior and hidden psychological variables. I think the author uses the same over-cap measure for training and evaluation will be very helpful and interesting.

Critical weaknesses
  1. The abstract is so long and it is so difficult to understand. The paper content is also difficult to follow.

2.D.4 reports 5,000 evaluation episodes per seed, while D.6 reports 500 test episodes. So the paper is very unrigorous and rough.

Questions for authors
  1. Which episode counts and training seeds produced Tables 1 and 11?

  2. The text contains numerous inconsistencies and is highly redundant and difficult to follow. Please simplify and revise this paper carefully.

Reviewer_17XGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces COMPULSIONBENCH, a benchmark to audit “compulsion-risk proxies” in recommendation policies and to compare mitigation strategies. The benchmark consists of a hand-designed, partially observable user simulator calibrated to aggregate statistics from KuaiRand, two benchmark tasks (engagement maximization and optimization constrained by time over a fixed session threshold), an LLM-augmented version of that simulator, and a scorecard. The measured proxies are: the cumulative watch time, the expected length of tail sessions, the time over a fixed session threshold, the late-night cumulative time, the fraction of late-night cumulative time over the total cumulative time, the late-night session start rate, and the short re-entry gap. Experiments compare multiple recommendation policies and conclude that a cap on session length successfully reduces the over-cap time at a small watch-time cost.

Critical strengths

•⁠ ⁠The paper treats a relevant well-defined problem. It asks whether engagement optimization in recommendation policies increases observable compulsion-risk proxies and how much can mitigation strategies reduce them. This problem is tied clearly to literature on engagement vs. reflective value. •⁠ ⁠Reproducibility effort: Default parameters (Table 4), seeds, pseudocode, compute accounting, linked code, etc..

Critical weaknesses

•⁠ ⁠The paper does not validate nor justify the design choices of the user simulation model that is complexe and hand designed. •⁠ ⁠The link between the measured proxies and compulsive use is missing. The claim would have been stronger if the proxies are shown to be tracking the user's latent measures that simulate compulsion (e.g. OverrideRate).

Most conclusions of the paper are conditional on the hand-designed user simulator, which is not justified. The paper calls the mathematical forms of this model "experimental assumptions," but it offers no theoretical analysis of the model, no validation of its dynamics against user-level data, and its calibration is not reported. The component ablations in Section 5.4 only show which parts of this particular model drive over-cap exposure. The hand designed model is complex and is used for a leaderboard benchmark, a purpose that requires realism, a validation is therefore warranted. As a result, there is no basis for judging whether the benchmerking capabilities of COMPULSIONBENCH and the reported scorecard (rankings and mitigation effects) reflect anything beyond modeling choices of the authors.

The calibration that is supposed to ground the simulator is not reported. Section 4.1 states that "Appendix C reports the full audit." (Line 286) and Appendix C promises "pre- and post-calibration fits for each target statistic, along with sensitivity to the public-log sessionization threshold Δsess\Delta_{sess} and to key simulator parameters" (Line 1503). Appendix C contains only a qualitative summary (Table 10). This calibration is based on criteria that are mentionned in Section 4.1 ("The released operational objective is lexicographic rather than a flat weighted sum: it prioritizes session-length and session-item-count fit before gaps, normalized hazards, diurnal patterns, cluster watch means, and conditional-return summaries." Line 284) but no value of these criteria or of any fit statistic appears in the paper, and the "operating envelope of the benchmark" (Line 290) that calibration is said to document is never stated. The paper reports that the user simulator is too light-tailed (Line 323), with the longest session reaching 73 steps before stoping versus 9394 in the data (Appendix C: Table 10), and sessions start time more uniform in the time of the day compared to the data (Appendix C: Table 10). These are exactly the properties a benchmark of tail risk and night use depends on. This results in the simulation never reaching the 120 minute threshold by any policy (Section 5.5). The paper uses a 15 minutes threshold in its evaluation results' section, which according to the paper itself is "not a clinical boundary or public-log quantile" (Line 297).

Several claims in the abstract are direct consequences of the design choices rather than informative findings: •⁠ ⁠"Over-cap incidence is essentially unchanged." (Abstract: Line 036) The paper sets T_cap = T_ref, then by design the cap does not intervene untill the session length exceeds the threshold that is used to measure the over-cap incidence. •⁠ ⁠"Without a short-gap re-entry spike." (Abstract: Line 038) The gaps between sessions are sampled from a logNormal distribution that have fixed paramaeters and that is independent of the state of the simulated user and the recommendation policy. No policy or mitigation strategy can change the length of that gap, so a re-entry spike cannot occur in this simulator.

Among evaluated policies, some optimized ones (such as PPO) don't outperform trivial ones (RoundRobinPolicy for deterministic scorecard Table 1 and Random for stochastic scorecard Table 11) in terms of cumulative watch time. In the deterministic case, round robin even outperforms PPO by apprx 14 min in the cumulative watch time metric. This bring the validity of the experimental results into question.

Questions for authors

⁠ •⁠ ⁠What are the values of the calibration criteria (The lexicographic objective in Section 4.1) after calibration? Could you show the simulation versus the data distribution of the target statistics? •⁠ ⁠What justifies the validity of the functional forms of the backend model? •⁠ ⁠How was the 15 minutes threshold (Tcap=TrefT_{cap} = T_{ref}) chosen? Was it fixed before the experiments were conducted? •⁠ ⁠Are the compulsion-risk proxies associated with OverrideRate (defined in Appendix E.3)? Can this association be shown theoretically (or empirically)? •⁠ ⁠the between sessions gaps are modeled by a lognormal and are independent from other states of the system. What justifies these assumptions? how well do they fit the data? •⁠ ⁠How was the PPO tuned? what justifies its underperformance (similar performance in the case of the stochastic scorecard) compared to the trivial policies (round robin and random)?

Additional feedback (not considered in the scoring)

•⁠ ⁠Section 3 would be easier to follow if it briefly defined the symbols in the main equations and if T_ref was defined before its first use in Eq 8. •⁠ ⁠The LLM backend is mentionned as a contribution but its results are not reported (just placeholders in Table 8). reporting these results or reframing the LLM version as an optional extension rather than a contribution would align the claims to the content. •⁠ ⁠Presenting a diagram that illustrate the different components of the system and their interaction would add clarity to the paper.

Reviewer_uu6COpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CompulsionBench, a simulation-based benchmark for auditing observable compulsion-risk proxies in sequential recommendation systems. The benchmark evaluates how engagement-oriented recommendation policies affect long-horizon behavioral outcomes, including prolonged sessions, over-cap watch-time exposure, late-night session starts, and rapid-return diagnostics. CompulsionBench provides an observable user simulator with two compatible response backends: one is a transparent parametric simulator, and another is an optional LLM-augmented semantic backend. It defines two evaluation tasks, engagement maximization and observable-risk-constrained optimization, and uses public short-video interaction logs to calibrate aggregate behavioral statistics. The authors evaluate heuristic recommendation policies, PPO, Lagrangian PPO, and session-cap interventions using a standardized reward-risk scorecard. The results show that higher-watch-time policies can exhibit greater over-cap exposure, while a session cap substantially reduces tail exposure at a modest watch-time cost.

Critical strengths

1. Clear benchmark formulation and transparent evaluation assumptions. The paper presents a well-structured benchmark that clearly defines the simulation environment, evaluation tasks, and metrics. Furthermore, it confines its conclusions to the scope of the conditional evaluation and discloses limitations regarding calibration, avoiding any exaggeration of the real-world significance of its synthetic data results. This transparency not only enhances the interpretability of the benchmark but also lays a valuable foundation for reproducible research.

Critical weaknesses

1. Insufficient evidence regarding the validity and robustness of the benchmark evaluation. Although the authors candidly acknowledge the limitations of their synthetic environment, the calibration results reveal significant discrepancies between simulated and real-world behavioral patterns. Despite the benchmark being explicitly designed for conditional evaluation, the paper fails to adequately demonstrate the robustness of its key conclusions—regarding strategy comparisons and risk mitigation—across a range of other plausible user dynamics and threshold configurations. Consequently, the generalizability and utility of the benchmark within its intended synthetic evaluation scenarios remain questionable.

2. Insufficient empirical validation of the LLM-augmented backend. The paper presents the LLM-augmented user-response backend as one of its primary contributions. However, the experimental evaluation focuses predominantly on the parametric simulator, and quantitative evidence supporting the added value of the LLM extension is very limited. The paper lacks a quantitative comparative analysis to demonstrate whether the LLM backend—as opposed to the parametric one—substantially alters policy rankings, risk metrics, or conclusions regarding risk mitigation. Consequently, it remains unclear whether the LLM extension provides a fundamentally different evaluation environment or merely introduces additional noise without delivering proven benefits.

3. Lack of a clear architectural overview of the benchmark. Although the paper provides detailed mathematical formulations and experimental results, it lacks an overview figure illustrating the overall architecture and construction pipeline of CompulsionBench. Relying primarily on textual descriptions makes the framework difficult to understand intuitively.

Questions for authors

Q1. Robustness of the Benchmark under Alternative User Dynamics Calibration results reveal significant discrepancies between the synthetic simulator and real interaction logs. How do the authors justify that the policy rankings and conclusions regarding risk mitigation derived from this benchmark remain robust despite these discrepancies? Have the authors attempted to use user response dynamics or calibration settings that better reflect real-world conditions, and do the main research conclusions remain consistent in such cases?

Q2. Sensitivity of the Risk Thresholds Although Appendix F.4 describes sensitivity analyses with different session-cap thresholds, the corresponding quantitative comparisons are not clearly presented. Could the authors provide additional results across intermediate evaluation thresholds to demonstrate whether policy rankings and mitigation trade-offs remain stable?

Q3. Empirical Contribution of the LLM-Augmented Backend The LLM-augmented backend is presented as a major contribution, but the paper provides limited quantitative evidence of its added value. Could the authors provide the actual calibrated fusion weights, the results of the proposed validity tests, and direct comparisons between the PARAM and LLM backends? In particular, does the LLM backend produce meaningful changes in policy rankings, risk measurements, or mitigation conclusions?

Q4. Clarification of the Evaluation Protocol Appendix D.4 states that trained policies are evaluated over 5,000 episodes per seed, while Appendix D.6 reports 500 test episodes for the main scorecard tables. Could the authors clarify which evaluation configuration was used to generate the main reported results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.