- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the issue that in long-term recommendation reinforcement learning, the shared value function TD updates are mutually interfered with and error propagation occurs due to the heterogeneity of different users' interactions, and proposes the HAVL method. This method first performs heterogeneity-aware experience grouping based on expected rewards, continuous interaction probabilities and successor state features, then conducts transfer-constrained value learning, and makes minimal modification corrections to the original TD updates under the constraint of group reference TD loss. The paper provides relevant theorem derivations, conducts experiments on two KuaiRand datasets based on four reinforcement learning backbones including DQN, A2C, DDPG and TD3, sets up multiple groups of comparison and ablation experiments, also performs cross-user update interference diagnosis, and is supplemented with online A/B tests to verify business benefits.
Critical strengths
-
Instead of grouping by value or user activity which are widely adopted in prior work, this paper groups transitions based on reward, continuation probability and successor‑state features that directly characterize Bellman‑backup compatibility. Theoretical theorem further establishes the connection between the grouping metric and backup discrepancy. This provides a principled way to identify compatible experiences for shared value‑function learning.
-
The method is tested on four distinct RL backbones to demonstrate general applicability. Carefully designed ablation and diagnostic experiments verify the function of each component and confirm that the constrained update effectively mitigates cross‑user TD‑update harm. Beyond offline simulation results, real‑world online A/B test provides evidence of practical business improvement.
Critical weaknesses
-
The HAVL framework introduces many new hyperparameters including group number K, distance scaling coefficients, group radius calibration rule, loss tolerance , trust‑region , and minimum effective sample‑size threshold for activating constraints. The paper only reports results under one set of calibrated configurations. There is little evidence showing how performance degrades when these hyperparameters deviate from the selected values. Since the whole constraint mechanism can become ineffective or overly restrictive with poor hyperparameter choices, this creates uncertainty about whether the observed gains can transfer to new datasets and settings.
-
Theorem 1 only characterizes backup discrepancy under a predefined bounded continuation‑value class, and Theorem 2 gives local bounds on reference‑group TD‑loss increase instead of guarantees on policy cumulative reward. Theoretical analysis cannot rule out cases where constrained TD‑updates still produce worse final policy performance. While offline diagnostics show reduced TD‑loss harm on held‑out audit contexts, there lacks theoretical linkage connecting these per‑update improvements to the ultimate long‑term recommendation objective. As a result, theory serves only as partial motivation rather than rigorous justification for practical performance gains.
Questions for authors
-
Many critical hyper‑parameters are calibrated using reference‑set statistics in the current implementation. Could you provide sensitivity results for key hyper‑parameters such as group number K, minimum effective sample size threshold, and loss tolerance ? Specifically, how large is the performance drop when these hyperparameters deviate from the adopted configuration?
-
The theoretical analysis only bounds local reference TD‑loss growth rather than policy cumulative reward. In your experiments, is there observable correlation between the measured reduction of cross‑user TD‑update harm and the final policy cumulative‑reward improvement? Could you supply quantitative evidence to verify that mitigating TD‑update interference indeed drives the performance gains of HAVL?
-
The auxiliary predictor is essential for computing the grouping descriptor. What level of prediction error of the auxiliary predictor will substantially degrade the overall performance of HAVL? Have you conducted controlled experiments to simulate predictor noise to test the robustness of the grouping procedure?
-
Under some conditions the update will be entirely skipped by the back‑tracking acceptance check. What is the empirical frequency of skipped updates in your main experiments? Can frequent update‑skipping stall critic learning and limit final performance?
We respectfully thank you for your careful review and for recognizing the interaction-structure grouping and empirical evaluation. We understand your main concerns to be dependence on precise tuning and whether local TD-loss control benefits policy learning. We address these alongside predictor errors and skipped updates below. We have also revised the manuscript; please refer to the updated PDF for the changes.
W1 / Q1. Hyperparameter sensitivity and transfer
Your concern is whether modest configuration changes could make grouping or constraints ineffective. We therefore varied K∈{4,8,16}, minimum effective sample size ESS∈{4,8,16}, and the loss-budget coefficient η∈{0.005,0.01,0.02}, one at a time. Relative reward changes range from -0.33% to +0.73% against the default, indicating that performance did not depend on one precise setting in this sweep.
We also examined controller behavior. Reducing η from 0.02 to 0.005 increases the correction rate from 19.12% to 66.64%, as expected from tighter budgets, without an appreciable reward decrease in either configuration.
Regarding new datasets, our design uses data calibration to reduce manual specification: distance scales and grouping radii are determined from reference data, while K, ESS, and the budget remain the main adjustable parameters. We will distinguish these settings and include the sensitivity results in the revision.
W2 / Q2. Connecting local control to policy performance
You raise a key question: how does controlling reference TD loss relate to final policy return? We address this through the role of the theory and additional observations.
The analysis first characterizes experiences with similar expected Bellman backups, then establishes conditions for shared updates to satisfy group-loss budgets. The controller also minimizes deviation from the original update to retain its learning signal. It therefore balances learning progress against adverse update effects, rather than minimizing harm alone: a zero update eliminates update-induced harm but cannot learn.
We added paired evaluations at 12 training snapshots, measuring the effects of the same candidate update on independent receiver-user contexts. HAVL reduces mean positive TD-loss change by 21.7% relative to the ordinary update. Separately, the 10,000-update comparison yields 1.50% higher final reward than Base. These provide evidence for the local control effect and the full method's policy benefit, respectively.
We also compared a controller that only reduces step size without changing direction. It reduces audit harm by 29.4%, yet achieves lower fixed-update-budget reward than HAVL. This suggests that lower harm need not imply higher return; preserving useful learning signals also matters. We will clarify that the theory guarantees local loss control and the experiments assess final return, rather than attributing all performance gains to harm reduction.
Q3. Predictor-noise robustness
As also discussed in our response to Reviewer 7Kun, we added dimension-wise zero-mean Gaussian noise to the predictor's existing outputs. Noise standard deviations were 0.1, 0.3, and 1.0 times the corresponding reference-descriptor standard deviations. Perturbations affect grouping, while rewards and TD-target definitions remain unchanged. Relative cumulative-reward changes were -0.38%, -0.82%, and -2.07%, respectively.
Regarding the error level at which degradation becomes noticeable, changes remain below 1% at the two smaller scales, whereas noise at 1.0 times the descriptor standard deviation produces a larger 2.07% decrease. This provides a concrete observation under the Gaussian-perturbation setting: lighter tested perturbations have limited effects, while noise comparable to the descriptors' own variation produces more pronounced degradation.
Q4. Skipped updates and stalled learning
Your concern is reasonable: frequent rejection could impede critic learning. We therefore recorded the full supplementary run. Across 10,000 updates, no updates were skipped; 43.79% of proposals were corrected, and the mean accepted update norm was 91.64% of the original norm. Accepted reference losses remained within their budgets. In 0.72% of steps, no groups met the effective-sample-size requirement; these steps used ordinary updates rather than being skipped. Control in this run therefore primarily adjusted updates rather than stopping learning.