acceptodds
Under review as a conference paper at ICLR 2027

From Global Budgets to Per-User LTV Health: Offline Reinforcement Learning for Personalized Coupon Allocation under Extreme Population Skew

Abstract

Coupon incentives are widely used to promote purchases and sustain user engagement. However, existing systems typically apply uniform rules within user tiers or optimize aggregate returns under a global budget, overlooking differences in users' lifetime value (LTV). This limitation is particularly severe in e-commerce referral platforms dominated by low-activity users, where sparse histories and highly skewed populations further complicate LTV estimation and offline policy learning. To address these challenges, we formulate daily joint coupon allocation as an offline reinforcement learning problem for personalized fee-rate control with user-specific, LTV-aware penalties. We propose a group-relative offline RL algorithm built on implicit Q-learning, termed LIGER, that represents evolving LTV health through time-varying state features and a structured reward balancing immediate incentive efficiency with long-term value changes. To mitigate majority-group dominance, LIGER combines group-balanced sampling with within-group reward and advantage normalization. It further replaces a separately fitted state-value network with a temperature-controlled softmax Bellman target derived from twin Q-critics, avoiding an additional value-function fit on imbalanced data. Offline evaluations and online A/B tests in a real-world e-commerce setting demonstrate the effectiveness of LIGER.

Primary area:
reinforcement learning
Keywords:
application of e-commerce market, personalized coupon allocation, offline reinforcement learning, customer Lifetime value, user heterogeneity

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_NgxwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper formulates personalized coupon allocation as an offline RL problem with per-user, dynamically estimated LTV safety constraints and proposes LIGER, an implicit Q-learning variant tailored to extreme population skew. LIGER introduces a group-relative learning scheme (group-balanced sampling, reward normalization, and advantage standardization), a softmax-Q Bellman target that removes the separate value network, and a KAPPA-FLOOR mechanism to calibrate penalties arising from underestimated, sparsely estimated LTV caps. Offline FQE analysis and a month-long online A/B test on a large e-commerce platform show improvements in GMV, conversion, and cost control relative to baselines and the incumbent rule.

Critical strengths

The clearest strength of this work is its practically meaningful reformulation of coupon allocation from aggregate budget optimization to per-user LTV-aware decision-making. This directly addresses an important limitation of global-budget approaches, where overspending on persistently unprofitable users can be masked by underspending elsewhere. The formulation is well aligned with the production setting and is supported by encouraging online A/B results, showing higher GMV and conversion with essentially unchanged coupon expenditure.

Critical weaknesses

Please refers to the questions.

Questions for authors
  1. Can you quantify the truncation bias experiment motivating KAPPA-FLOOR (bias vs. history length, per-tier error bars) and show how κfloor was chosen and validated? How sensitive are results to this choice?
  2. Since you balance sampling across groups in critic training, how do you address the mismatch between this training distribution and the deployment distribution in value estimation? Did you consider importance corrections or alternative reweighting?

The reviewer is willing to increase the rating if all questions are well addressed.

Reviewer_FTHvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper formulates daily coupon face-value allocation as offline RL with per-user lifetime-value (LTV) targets and a platform fee-rate ceiling. LIGER uses group-calibrated reward and advantage normalization, balanced sampling, a softmax Bellman target over twin Q critics instead of a separately fitted value network, and KAPPA-FLOOR to prevent sparse-history safety estimates from dominating the reward. It reports model-based FQE on a production stream and a month-long online comparison with the incumbent tier-based rule.

Critical strengths
  1. The problem formulation reflects a real operational limitation: global subsidy budgets can hide persistently unprofitable users. Separating an estimated user-level LTV reference from a platform ceiling, while keeping the policy action space compatible with the existing coupon system, is practically motivated.

  2. The reported deployment result is encouraging. Against the incumbent rule, the paper reports +3.90% GMV and +1.40% visit-to-purchase rate with −0.12% coupon expenditure over approximately 300,000 users per arm. The state-conditioned action analysis and group-relative ablations also make the intended behavioral effect concrete.

Critical weaknesses
  1. The offline-RL evaluation remains vulnerable to severe out-of-distribution action error.

Equation (5) computes the critic target using a softmax over all 121 actions, even though the policy is learned from logged actions only. There is no behavior-support constraint, action masking, conservative penalty, or reported per-action coverage of the training data. The learned policy selects 100–109 distinct actions per stratum, which the paper interprets as desirable diversity, but in offline RL this can instead indicate extrapolation into poorly supported actions.

Twin critics do not resolve this issue. The FQE evaluator is also model-based and its accuracy, action-conditional error, and support coverage are not reported. The bootstrap procedure only quantifies variation over test initial states; it does not quantify FQE model bias. Moreover, the comparison is limited to BC and standard IQL, without CQL, BCQ, TD3+BC, CPQ, COptiDICE, or BCORLE-style baselines. Therefore, the reported FQE gains cannot yet establish that LIGER improves offline decision quality rather than exploiting critic or evaluator extrapolation.

  1. The claimed per-user LTV safety constraint is a heuristic reward penalty without direct validation.

The cap (κuest\kappa_u^{est}) is formed from the maximum of several historical LTV/GMV ratios, but no uncertainty estimate, calibration analysis, or reliable-history versus sparse-history validation is provided. KAPPA-FLOOR replaces small estimated caps with a larger floor in the training penalty. This may preserve the platform-wide ceiling, but it explicitly relaxes the claimed per-user constraint, and the paper reports no per-user violation rate, cap calibration error, or sensitivity to the floor value.

The reward is also heavily hand-designed: the supplement lists ten components but gives explicit equations for only the investment-return and health-transition terms. The remaining terms and their interactions are not sufficiently specified to assess whether the learned policy is optimizing LTV or simply responding to manually tuned proxies. The online A/B test reports GMV, conversion, and expenditure, but does not report realized per-user LTV, safety-cap violations, or long-term profitability. Since the data stream is only 27 days while the method uses 30- and 90-day LTV features and reports 30-day outcomes, the paper should also establish that all such features are strictly backward-looking and free of temporal leakage.

Questions for authors

N/A

Reviewer_5Sm7OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper formulates daily joint coupon allocation as an offline RL problem and proposes a new offline RL algorithm, LIGER. LIGER is built on IQL with LTV-aware constraints, group-balanced sampling and normalization, and a softmax-based Q-value target. Offline evaluations and real-world online A/B tests demonstrate the effectiveness of LIGER over several baselines.

Critical strengths
  • This paper studies a meaningful and practically relevant problem of personalized coupon allocation under user heterogeneity and per-user LTV constraints.
  • It clearly identifies the limitations of existing global-budget formulations and key challenges in standard offline RL for this scenario.
Critical weaknesses
  • Empirical validation is insufficient. LIGER is only compared with BC and standard IQL, without evaluation against SOTA methods for coupon allocation. Moreover, the lack of controlled experiments with varying population skew makes it difficult to assess the impact of population imbalance and the robustness of LIGER across different user distributions.

  • The design choices of LIGER lack sufficient justification. Despite introducing multiple components, the paper lacks adequate ablations to demonstrate their individual contributions to policy performance. Furthermore, the claimed connection to GRPO appears overstated, as the method primarily performs within-group normalization based on user categories rather than adopting GRPO's core optimization mechanism.

These issues undermine the claims regarding the effectiveness and novelty of LIGER.

Questions for authors
  • How does LIGER compare with existing SOTA methods for coupon allocation?
  • How robust is LIGER under different levels of population skew, and how does skew affect decision-making performance?
  • Can the authors provide ablation studies to directly demonstrate the individual contributions of each component to policy performance?
  • What is the substantive connection between LIGER and GRPO beyond the use of normalization?
Reviewer_jAAiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates personalized coupon allocation under extreme population imbalance. Existing approaches typically optimize aggregate returns under global budget constraints, without adequately accounting for differences in users' lifetime value (LTV). Moreover, the dominance of low-activity users in the population can bias offline reinforcement learning toward the majority group. To address these challenges, the authors propose LIGER, an offline reinforcement learning framework incorporating per-user LTV constraints.

Critical strengths
  1. This paper addresses a practically relevant and well-motivated problem.

  2. The proposed method is specifically designed to address extreme imbalance in user populations.

  3. Proposed framework integrates long-term value modeling with a safety-threshold calibration mechanism.

  4. Experiments are conducted in a real-world industrial setting and include large-scale online A/B testing.

Critical weaknesses
  1. The proposed per-user LTV safety constraints lack strict feasibility guarantees. One of the paper's central contributions is replacing traditional global budget constraints with per-user LTV constraints. However, the proposed algorithm primarily handles the estimated per-user LTV limits through reward penalties, without providing a strict guarantee of constraint satisfaction. In particular, the KAPPA-FLOOR mechanism raises the reference threshold used in the safety penalty to mitigate underestimation caused by sparse user histories. While this design may improve policy learning, it may also relax constraints for genuinely low-value users. If a low LTV limit reflects a user's actual long-term profitability risk rather than estimation error, such calibration could lead to excessive subsidies.

  2. The Softmax-Q Bellman target may introduce out-of-distribution (OOD) action estimation errors, potentially conflicting with the core motivation of offline RL. The proposed algorithm builds upon Implicit Q-Learning (IQL), which avoids evaluating out-of-distribution actions by relying on actions observed in the offline dataset. However, when computing the Softmax-Q Bellman target, LIGER evaluates all actions in the discrete action space and assigns weights according to their estimated Q-values. For state-action pairs that are rarely observed or insufficiently covered by the offline dataset, Q-network predictions may be unreliable. Since the softmax weights depend on these Q-values, overestimated actions may receive disproportionately large weights, potentially biasing the Bellman targets. Although taking the minimum of twin Q-critics can mitigate overestimation to some extent, it does not fundamentally guarantee that the evaluated actions lie within the support of the offline data.

  3. Group-wise normalization and balanced sampling alter the original optimization objective, but their effects on per-user and aggregate economic returns are insufficiently analyzed.

  4. The experimental baselines are insufficient to isolate the individual contributions of the proposed components, and the online evaluation does not adequately validate the core long-term LTV objective.

Questions for authors

Please refer to Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.