- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes an approach for incorporating temporal gaps (how to capture a users changing interest over time - TIS-MDP) in recommendation systems. If a user takes an action A and then comes back after a time, how much credit should flow back or be attributed to action A - this is the core contribution. While for a single user the interest decay is exponential, for a population it follows power law. By using mutual information to measure the correlation between historic item categories, and future clicks (CTTC), the authors propose a survival function based on the powerlaw to incorporate the temporal gap into Bellman. They prove the contraction properties and measure the effectiveness of TIS-MDP.
Critical strengths
The temporal gaps in user interactions is a real and under studied problem. How does this affect the effectiveness of of RL? Where does credit assignment fail? The paper motivates this, proposes a rigorous approach for addressing this. The paper grounds the contributions in established approaches (semi-MDP and Bellman, Law of large numbers). That makes it readable and relatable to researches. While the gamma distributed heterogeneity has some limitation in my POV, the observation of law of exponents for individual decay vs power law for population decay is interesting.
Critical weaknesses
One weakness of this paper are the assumptions made in the soundness of the approach, which are even partly addressed in the appendix (e.g. the population-level approximation). It would be even more valuable to perform ablation tests here. Also, it would be very good to show some tradeoffs when using the TIS-MDP in the experiments section, for example, why the improvements are different for the RL backbones (BCQ improves the length, but not the reward; CQL improves the reward, but not the length). Do they even need more data to estimate s(Δ) accurately? Simply mentioning this would already be sufficient.
Questions for authors
Ablation studies on interest continuation using only temporal gap for ablation. How sensitive is CTTC to biases outlined earlier? Since most of section 5 is standard RL material, more experimental work on these gaps would be very interesting to read about.
2 comments
Official Comment by Authors (2/2)
More experimental analysis. We agree that Section 5.1 is largely standard theory, and it serves only to confirm that the backup is well-posed. Therefore, we add a standard MDP baseline with the gap as input, and a per-backbone comparison with Semi-MDP discounting. These results are summarized in Tables I–III below.
Table I. Synthetic environment: cumulative reward (mean ± std over 5 seeds)
| Backbone | Standard MDP | Standard MDP + gap input | TIS-MDP (exponential) | TIS-MDP (power-law) |
|---|---|---|---|---|
| DQN | 47.29±0.23 | 49.68±0.23 | 52.67±1.99 | 53.27±2.55 |
| A2C | 52.81±0.14 | 53.99±3.38 | 56.07±1.00 | 55.47±0.53 |
| PPO | 51.86±1.52 | 53.12±1.72 | 55.15±0.59 | 55.91±0.67 |
Table II. KuaiRand (EasyRL4Rec): mean ± std over 5 seeds
| Backbone | Method | Len | ||
|---|---|---|---|---|
| BCQ | Standard MDP | 1.930±0.10 | 4.800±0.25 | 0.400±0.001 |
| Standard MDP + gap input | 1.990±0.06 | 4.936±0.19 | 0.404±0.005 | |
| TIS-MDP (exponential) | 2.065±0.02 | 5.106±0.04 | 0.409±0.002 | |
| TIS-MDP (power-law) | 2.080±0.06 | 5.140±0.27 | 0.410±0.009 | |
| CQL | Standard MDP | 0.850±0.01 | 3.130±0.01 | 0.270±0.004 |
| Standard MDP + gap input | 0.902±0.01 | 3.114±0.01 | 0.290±0.005 | |
| TIS-MDP (exponential) | 0.993±0.04 | 3.086±0.02 | 0.325±0.012 | |
| TIS-MDP (power-law) | 0.980±0.05 | 3.090±0.00 | 0.320±0.016 | |
| CRR | Standard MDP | 1.260±0.01 | 5.830±0.03 | 0.220±0.001 |
| Standard MDP + gap input | 1.264±0.01 | 5.858±0.02 | 0.220±0.000 | |
| TIS-MDP (exponential) | 1.269±0.01 | 5.893±0.03 | 0.220±0.001 | |
| TIS-MDP (power-law) | 1.270±0.02 | 5.900±0.06 | 0.220±0.000 |
Table III. KuaiSim cross-session retention: mean ± std over 5 seeds
| Method | Return day ↓ | Retention ↑ |
|---|---|---|
| TD3 | 2.2842±0.1366 | 0.8264±0.0212 |
| RLUR | 2.2415±0.5217 | 0.8032±0.0781 |
| TD3 + gap input | 2.2287±0.0419 | 0.8467±0.0067 |
| TD3 (TIS) | 2.1454±0.1053 | 0.8772±0.0156 |
We thank reviewer for the time to review our paper and hope that our responses address the reviewer's concerns so reviewer can re-evaluate our work. Please let us know if you have any further questions, and we will do our best to address them.
Official Comment by Authors (1/2)
We thank the reviewer for the constructive comments and for recognizing the motivation of our work. We respond to your questions as follows.
Assumptions. The individual exponential decay and the population power law are not assumed a priori. They form a conjecture drawn from the data, which Appendix B.3 checks in three steps. The exponential fits better for 41 of the top-50 users, the fitted rates are compatible with a Gamma distribution under a K–S test, and the implied Lomax curve matches a direct power-law fit (R2=0.700 vs. 0.703). We use the population-level approximation because per-user curves are sparse and noisy (Table 4). Theorems 5.3 and C.1 quantify the resulting aggregation and misspecification errors, and the gap-only continuation model is further tested by ablations on the survival form (Table II) and on the decay parameters α and τ (Tables 9 and 10).
Varying improvements across backbones, and trade-offs. RL optimizes the cumulative reward, and TIS-MDP improves it for all three backbones on KuaiRand (+7.8%, +15.3% and +0.8% over 5 seeds, Table II). Length and per-step reward are the two factors of this quantity (Rcumu≈Len×Ravg) and are not optimized separately. How the gain splits between them depends on how each backbone constrains its policy (Section 6.2), and this split is stable across seeds. BCQ (TIS) gains mainly in length (5.140 vs. 4.800) and CQL (TIS) mainly in per-step reward (0.320 vs. 0.270), and the exponential form shows the same pattern. If one component matters more for an application, it can be targeted through the reward definition.
The main trade-off of TIS-MDP lies in the strength of the decay. An overly aggressive decay also removes genuine long-term credit, and there is no gain at α=5 (Table 9). In addition, the method brings little benefit when interactions are dense and the gaps carry little temporal information, as on MovieLens (Table 6).
Does TIS-MDP require more data to estimate s(Δ)? The answer is No. Its two parameters are not estimated from the RL training data. The exponent α=0.185 is obtained once on Amazon and kept fixed, and τ is selected from a small set. The value fitted on KuaiRand itself (about 0.25) is not the best-performing one, and the gains hold for all α∈[0.1,1.0] (Table 9). What matters is therefore the decay shape rather than a precisely calibrated rate.
Sensitivity of CTTC to data volume and outlying points. The conclusion does not rest on the sparse and noisier long-gap points. When the fit is restricted to the first five days, where observations are densest, the power law is still closer to linear on the log-log scale than the exponential on the semi-log scale (R2 of 0.973 vs. 0.863, Appendix B.2). The same holds over the full 30 days (0.936 vs. 0.725, Fig. 2), on KuaiRand-1K (0.977 vs. 0.881, Fig. 9) and on Foursquare (0.854 vs. 0.806, Fig. 10).
Sensitivity of CTTC to the biases in Appendix F. The three datasets are affected by different biases. Amazon uses sampled negatives, KuaiRand items are surfaced by a feed recommender, and Foursquare consists of user-initiated check-ins, yet all three show the same decay shape. Moreover, only this shape is used downstream, and the RL results are robust to its exact calibration (Table 9).