acceptodds
Under review as a conference paper at ICLR 2027

COMPASS: Completion-based Process Advantage Supervision for Sequential Recommendation

Abstract

Delayed rewards make it difficult to determine which actions contributed to an outcome. Completing alternative actions from the same interaction prefix can supply useful comparisons, but the resulting returns reflect the policy used for completion. They need not remain suitable value targets as the learning agent improves. We propose COMPASS, which fits offline completion advantages and uses their centered predictions to supervise action-value differences during reinforcement learning. The agent continues to learn from environmental rewards through its TD objective, and the fitted model is reused across training runs. We analyze the resulting value updates and characterize how changes in continuation policy affect the validity of an action comparison. On sequential recommendation tasks, COMPASS improves DQN's mean held-out return by 20.7% and 43.1% on RL4RS-A and RL4RS-B. The same model improves IQN across eight seeds, and a slate-preference extension improves DDPG by 12.5% on KuaiSim. Experiments examine the choices involved in using this evidence, including pretraining, centering, and the duration of supervision. Deployment uses the learned policy alone.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, recommender systems, credit assignment, counterfactual learning, process supervision

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jMUzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

COMPASS collects alternative-action completions under a fixed policy, fits a reusable action-advantage predictor, centers its scores over legal actions, and uses them as an auxiliary loss alongside ordinary TD learning. Centering preserves action gaps while discarding stale return levels. The method is evaluated with DQN and IQN on RL4RS and via slate preferences with DDPG on KuaiSim, accompanied by theoretical analyses of centered updates, policy-shifted comparisons, and decision loss.

Critical strengths
  1. The paper develops a clear, useful idea: old completions may retain reliable relative action information after absolute returns become stale. The centered interface is simple and cleanly separated from environmental reward and TD targets.
  2. The evidence is unusually diagnostic. Reward injection, pretraining, no-centering, permutation, reversed signs, guidance duration, and completion statistics are controlled; the same frozen model improves two value learners on every paired seed and transfers to a slate setting. Theory exposes the acceleration–bias trade-off.
Critical weaknesses
  1. Validation is confined to learned recommendation simulators. Since completion collection also relies on branching within such environments, robustness to simulator misspecification and transfer to logged or real systems remain unclear.
  2. Offline preparation is substantial:135,000 prefixes and fitted return distributions, and is not included in the common online interaction budget. Baseline comparisons therefore do not capture total data and compute cost, and schedule/K choices differ materially across environments.
Questions for authors
  1. How does COMPASS compare at matched total environment interactions, including completion collection?
  2. What are completion collection/training costs and the break-even point when reusing the frozen model?
  3. How sensitive are gains to simulator error or to completions generated from logged/world-model approximations?
3 comments
AuthorsOpenReview ↗

Response to Reviewer jMUz

We thank the reviewer for the positive assessment and the precise questions.

W2 / Q1. "Offline preparation is substantial […] and is not included in the common online interaction budget." / "How does COMPASS compare at matched total environment interactions, including completion collection?"

We now report the completion interactions next to the online budget, and the gap remains under the two step-matched comparisons we can make. On RL4RS-A the offline stage completes 270,000 intervention/reference branches over 135,000 prefixes with 418,496 rollouts (1–4 adaptive repeats), at most 2.2M simulator steps, about four online runs' worth (504k steps each). We list the two separately rather than summing them: the completions are taken by the fixed behavior-cloned policy μ, which never learns or acts online, and they are collected once and shared by every run. Step-matched comparisons:

RL4RS-A, DQNHeld-out return
DQN continued for another 500 epochs, same configuration (seed 0)125.4 → 129.6 (Appendix E.5)
DQN given the same completions as data: reward injection / MC-return pretraining / comparison pretraining (seeds 0–2)126.6 / 135.9 / 132.3 (Figures 3–4)
DQN / COMPASS, 500 epochs (8 seeds)128.2 / 150.3

Q2. "What are completion collection/training costs and the break-even point when reusing the frozen model?"

The offline stage costs about 1.2 online runs and is paid once. Same RTX 4090, each run alone (Appendix E.5, Table 25):

RL4RS-AGPU-minutes
Offline completion + fitting (once)80 (2 GPUs × 40 min)
One online run, DQN (500 epochs, 504k steps)66
One online run, COMPASS76 (+10 for teacher queries)

The frozen teacher is shared by every guided run on RL4RS-A (364 in our logs: all seeds, all K, all ablations and sweeps). Best-of-K is computed analytically from one fitted return distribution, so all K come from the same collection.

W1 / Q3. "Validation is confined to learned recommendation simulators. […] How sensitive are gains to simulator error or to completions generated from logged/world-model approximations?"

Logged approximations recover part of the gain but not the part that comes from comparing alternative actions. A teacher fitted on logged returns alone, with no branching in the simulator, reaches 139.5 vs. 153.5 for the completion-based teacher on the same seeds (Appendix E.6). On simulator error: completions and training use the same simulator, as do all baselines, so we use no information source the baselines lack. Both simulators are trained on real interaction logs and the completion prefixes are logged sessions, and on KuaiSim the gain holds on 1,958 users never used for completion (9.85 vs. 8.95–9.44 for the four baselines; Appendix E.2). We agree that validation on a live system is beyond this work and have expanded the limitations accordingly.

W2 (second part). "[…] and schedule/K choices differ materially across environments."

The schedule follows where the guidance enters. On a bootstrapped value (DQN, IQN) persistent guidance biases the targets, so β must decay (constant β: 119.1 vs. 150.3); on a policy (the KuaiSim actor) it does not enter bootstrapping and can stay on. K sets how optimistic the teacher is; a value learner uses it only early and a policy learner throughout, so the preferred K differs. We report the sweep for each environment (Appendix B.7, Table 8; Appendix C.5, Table 18) and state the rule in Section 6.4.

We hope the interaction counts, the cost table and the held-out results answer these questions, and would be grateful if the reviewer took them into account in the final assessment.

Thank you for the cost analysis, which resolves Q2. However, simulator sensitivity (Q3) remains untested, and the logged-return teacher performing like a zero target shows the gain depends on simulator branching, so I keep my score at 2.

AuthorsOpenReview ↗

Follow-up: scope, and a second simulator

We thank the reviewer for the follow-up and are glad the cost analysis resolved Q2.

Scope. We agree: the gain depends on branching. COMPASS compares alternative actions at the same state, which logs alone cannot provide, and the logged-return teacher (139.5, close to the zero target at 140.7) shows exactly this. This is the setting of the paper and of all baselines we compare against, which also learn by interacting with the same simulator; COMPASS additionally queries it offline with a fixed behavior-cloned policy μ. Learning purely from logs is outside our scope; our limitations already state that COMPASS "assumes access to alternative continuations from a shared prefix, limiting use where branching is unavailable or prohibitively expensive."

Simulator sensitivity. We trained a second RL4RS-A simulator from scratch (same DIEN architecture, a quarter of the logged sessions; validation AUC 0.91) and repeated the whole pipeline in it with all hyperparameters unchanged. The two simulators differ substantially: on the same 135,000 prefixes, completion returns correlate 0.82 and the sign of the completion advantage agrees in 69% of states.

RL4RS-A, DQN learner (seeds 0 / 1 / 2)Original simulatorNew simulator
DQN125.4 / 135.8 / 124.6 (mean 128.6)171.6 / 179.0 / 185.4 (mean 178.7)
COMPASS-DQN157.3 / 154.6 / 148.6 (mean 153.5)268.6 / 280.5 / 274.5 (mean 274.5)

The gain holds without retuning (+95.8 vs. +24.9), so it is not tied to one particular simulator.

Reviewer_j4F2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on RL-based sequential recommendation methods for delayed-reward modeling. The key argument is that the fixed completion policy μ\mu may become stale as the online learner improves, and the authors propose to mitigate the problem with action-comparison approaches. The proposed method is validated on two simulator environments.

Critical strengths
  • Clear motivation and reasonable solution.
  • Theory 1 provides some useful insights on separating the acceleration effect from biased guidance.
Critical weaknesses
  • The motivation on fixed action-completion may not hold in online settings, especially for dynamic environments of recommender systems.
  • Insufficient description of the background and related work. Should at least include a section on RL-based recommendation solutions.
  • Confusing method designs: Eq.(1) applies Cx(a)Cx(a) for x=Tkx=T_k, which calculates an “advantage of advantage”: TkT_k calculates the advantage, and CxC_x also take the format of the advantage function. This would make CTk≈TkCT_k \approx T_k, and it doesn’t seem necessary.
  • Needs more careful experimental support for the method’s generalization claims: weak statistical significance, and the design is strongly task-dependent.

Minor issues:

  • Should make the symbols consistent: CC vs. CsC_s.
Questions for authors

See "weaknesses"

2 comments
AuthorsOpenReview ↗

Response to Reviewer j4F2

We thank the reviewer for the careful reading and the concrete points on robustness, significance and notation.

W1. "The motivation on fixed action-completion may not hold in online settings, especially for dynamic environments of recommender systems."

μ does not need to keep up with the learner; it only needs to order actions better than chance, and the measured orderings stay valid as the learner changes. μ never acts online and never constrains the learner's updates directly: it enters only through centered comparisons, which the TD or policy loss can override. What is transferred is which action is better, not μ's return. Two new measurements:

  • On RL4RS-A, the agreement between an independently trained learner's ordering and the frozen teacher's rises from chance to 0.64 by epoch 100 and stays there through epoch 500 (eight IQN seeds; Appendix E.8, Figure 19).
  • On KuaiSim, the comparisons remain valid on 1,958 users μ never completed for: COMPASS 9.85 vs. 8.95–9.44 for DDPG, TD3, A2C and HAC (Appendix E.2).

Refreshing μ is also possible when it matters: on RL4RS-A, continuing a trained DQN with completions from its own earlier checkpoint reaches 146.2 vs. 129.6 for plain continued training (single seed; not in the paper).

W2. "Insufficient description of the background and related work. Should at least include a section on RL-based recommendation solutions."

Added. A paragraph on RL-based recommendation in Section 2, with the full discussion in Appendix E.1 (SlateQ, HAC, RL4RS, KuaiSim, and the delayed-reward baselines RRD and ARES already compared in Section 6.2 and Appendix B.5).

W3 + minor. "Eq.(1) applies C x(a) for x = T_k […] This would make C T_k ≈ T_k, and it doesn't seem necessary." / "Should make the symbols consistent: C vs. C_s."

T_K is not already centered, so the centering is needed. C_s is the legal-action centering operator, C_s x(a) = x(a) − mean_{a'∈L(s)} x(a'); it is a projection, so C_s C_s = C_s. T_K's reference is V_K(s), not the legal-action mean, and the κσ shrinkage is applied before centering, so C_s T_K ≠ T_K, and centering is what makes it comparable with C_s Q. We unified the notation to C_s throughout.

W4. "Needs more careful experimental support for the method's generalization claims: weak statistical significance, and the design is strongly task-dependent."

Significance. Paired over the eight seeds of Appendix B.8 (Table 9), RL4RS-A (Appendix E.4):

COMPASS − baselineMean diff.95% CIPaired t-test pWilcoxon pSeeds improved
over DQN+22.2[13.7, 30.6]4×10⁻⁴0.0088/8
over IQN+19.4[12.2, 26.6]4×10⁻⁴0.0088/8

For generalization, the KuaiSim held-out result above is on users disjoint from training. On RL4RS-B, as stated in Section 6.2, COMPASS improves DQN by 43% and comes close to, but does not exceed, the strongest policy-gradient baseline (378.8 vs. 388.0 for PPO).

Task dependence. The schedule and K follow one rule rather than per-task tuning: guidance on a bootstrapped value must be transient (constant β on DQN: 119.1 vs. 150.3), guidance on a policy may persist. We state this rule explicitly in Section 6.4.

If the new measurements and the revisions resolve the concerns about robustness and significance, we would appreciate a reassessment of the score.

AuthorsOpenReview ↗

Gentle reminder

Dear Reviewer j4F2, thank you again for your review. In our response above and in the revised PDF we addressed your points: whether a fixed μ remains useful as the learner changes (new ordering-consistency measurements), the added related-work section on RL-based recommendation, the centering in Eq. (1) with unified notation, and paired significance over eight seeds. We would greatly appreciate hearing whether these resolve your concerns, whether you have any further suggestions, and, if so, whether you would consider revisiting your rating.

Reviewer_fzDVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors presents a completion‑based auxiliary supervision framework COMPASS, to mitigate the credit‑assignment difficulty caused by delayed rewards in sequential‑decision / sequential‑recommendation settings. From offline intervention‑reference completion rollouts, they train a frozen completion‑advantage model that outputs relative action comparisons. During online RL, COMPASS applies state‑wise centering to both the completion‑derived action scores and the learner’s Q‑values, then optimizes an auxiliary Huber loss alongside the original TD objective, without modifying TD targets. The authors further extend COMPASS to a slate‑recommendation preference interface. They also provide theoretical analysis on convergence and transfer conditions for completion‑policy comparisons, and conduct extensive ablations on centering, action correspondence, Best‑of‑K statistics, and guidance decay schedules. Empirical evaluation is performed on two simulation environments: RL4RS and KuaiSim, showing consistent gains over vanilla DQN / IQN / DDPG across multiple random seeds.

Critical strengths
  1. Absolute returns from a fixed offline completion policy can become stale as the learner policy improves, but relative pairwise action advantages / ordering may remain valid. State‑wise centering removes shared state‑level offsets and offers a clean interface to inject this comparative supervision into standard value‑based RL without altering TD targets.
  2. This work includes formal analysis of value‑update dynamics, conditions for comparison transfer, plus comprehensive ablation suites. Improvements are observed across multiple backbones and two distinct recommendation paradigms, with multi‑seed results supporting the empirical benefits of the pipeline.
  3. Offline completion model is fitted once and frozen, reused for diverse downstream RL learners.
Critical weaknesses
  1. The completion model is trained on sparse offline intervention samples but queried for all legal actions at inference. The paper lacks diagnostics separating performance for state‑action pairs with direct completion evidence versus purely extrapolated pairs. It is unclear how much of the observed gain stems from valid completion comparisons versus uncontrolled model extrapolation artefacts.
  2. Ablation study shows that permuting action‑score correspondence still yields non‑trivial improvement over baseline. It raises the hypothesis that part of the benefit may come from generic regularization effects rather than meaningful completion‑derived action ordering. The paper has not isolated this confounder via random‑target or generic‑regularization baselines.
  3. COMPASS consumes substantial offline completion trajectories and compute overhead, but comparisons do not account for this extra resource budget.
Questions for authors
  1. Report the fraction of online‑queried state‑action pairs that have direct offline intervention completion evidence. Provide advantage‑prediction error and action‑ordering accuracy split for pairs with direct evidence vs purely extrapolated pairs, to characterize extrapolation reliability of the frozen completion model.
  2. Add comparison against simpler baseline: directly centered raw action‑value predictions without intervention‑reference completion rollouts. This helps validate whether the intervention‑reference completion formulation itself is necessary.
  3. Address the permuted‑correspondence ablation result: include random‑target / generic regularization baselines, to disentangle how much improvement originates from meaningful completion‑derived action ordering versus generic auxiliary‑loss regularization.
  4. For KuaiSim: run DDPG and COMPASS on the reserved disjoint held‑out user set with identical seeds and evaluation protocol, and state whether core performance conclusions hold on true held‑out data.
  5. Provide resource‑matched comparisons accounting for COMPASS’s offline completion data and computational overhead. Additionally, measure action‑ranking consistency across the course of online training, to empirically test the central assumption: whether completion‑derived action rankings remain valid as the learner policy changes.
6 comments
AuthorsOpenReview ↗

Response to Reviewer fzDV (1/3)

We thank the reviewer for the detailed and concrete questions. We ran every experiment requested; the results are below and in Appendix E of the revised PDF.

W1 / Q1. "The completion model is trained on sparse offline intervention samples but queried for all legal actions at inference. […] Report the fraction of online-queried state-action pairs that have direct offline intervention completion evidence. Provide advantage-prediction error and action-ordering accuracy split for pairs with direct evidence vs purely extrapolated pairs."

The fraction is zero by construction, and extrapolated pairs are somewhat less accurate than direct ones but of the same order. Completion prefixes come from the 50,000 construction sessions, and online training uses the disjoint training sessions (strict split, Appendix B.2), so every online query extrapolates the regressor to a new state. The direct/extrapolated split can therefore only be measured offline, on covered states, where the cached intervention action has direct evidence and the other legal actions do not. We compared the teacher with fresh μ completions on 1,024 covered prefixes (5 actions each, cached action always included, 16 completions per action; Appendix E.7, Table 26):

PairsOrdering accuracy (all / true gap ≥ 5)Advantage prediction (corr. / MAE)Label ceiling, split-half (all / gap ≥ 5)
Cached vs. extrapolated action0.64 / 0.740.39 / 7.20.76 / 0.93
Extrapolated vs. extrapolated0.60 / 0.680.26 / 7.60.77 / 0.94

In decision terms, choosing the teacher's preferred action among the five loses 3.7 return against the best of the five, versus 7.0 for a random choice: the teacher recovers about half of the available gap. Per pair the signal is weak (within-prefix gaps are 5.5 on a return scale of 84, and the completion labels themselves agree only 0.76 across halves); what the learner receives is this signal applied densely, on all legal actions of 135,000 states for 250 epochs. The controls in our reply (2/3) show that the aggregated signal is used and that its sign matters, and the state-independent control (136.9, the level of an uninformative target) shows that what is used is the state-specific part of the extrapolation, not a global action prior.

Q2. "Add comparison against simpler baseline: directly centered raw action-value predictions without intervention-reference completion rollouts."

Added; it reaches the level of an uninformative target, so the completions are needed. We fitted the same regressor on the logged next action and its logged return-to-go at the same 135,000 prefixes, centered over legal actions and used through the same loss (κ = 0, since it has no uncertainty estimate). Same seeds (Appendix E.6):

TeacherHeld-out return (3 seeds)
Logged return-to-go, no completions139.5
Uninformative (zero) target140.7
Completion-based teacher (COMPASS)153.5

The logged action and its return steer early collection, but the comparisons that lift DQN further require completing alternative actions. For reference, offline RL on the same logs without completions reaches 98.3 (BCQ) and 102.3 (CQL). On the reference branch: with κ = 0 centering cancels it exactly; with κ > 0 it decides which actions differ reliably from μ's own continuation before centering.

AuthorsOpenReview ↗

Response to Reviewer fzDV (2/3)

W2 / Q3. "Ablation study shows that permuting action-score correspondence still yields non-trivial improvement over baseline. […] include random-target / generic regularization baselines, to disentangle how much improvement originates from meaningful completion-derived action ordering versus generic auxiliary-loss regularization."

With the regularization held fixed, the ordering decides the outcome. On DQN the loss also gives a content-free gain (about +14), which appears only after guidance ends; on IQN it does not, and only the correct ordering helps. We replaced the teacher by targets that keep the interface (centering, β schedule, legal-action masking) but remove or invert the information (Section 6.5, Table 2; Appendix E.3, Table 24):

Target (seeds 0–2)DQNIQN
No auxiliary loss127.4130.2
Zero: all legal actions equal140.7136.3
Random ordering per state: permuted / Gaussian141.5 / 136.8132.8 / 142.6
State-independent (per-action mean of T_K)136.9133.0
Reversed ordering113.0135.2
COMPASS ordering153.5151.9

Regularization vs. ordering. Whatever the target, the loss pulls the learner's action gaps toward the target values; only the correct target also points them the right way. The reversed target separates the two: it uses the same numbers, schedule and loss as COMPASS, assigned to the opposite actions, so as a regularizer it does exactly what COMPASS does. On DQN it gives 113.0 against 153.5, with the random orderings in between. With the regularization fixed, the ordering alone moves the result by 40 points, and the wrong ordering falls below plain DQN.

The content-free part on DQN. The zero target adds about +14 over plain DQN, and the training curves (Appendix E.3, Figure 18) show that this gain appears only after β reaches zero. During guidance these targets flatten the action gaps, so collection is close to uniform over the legal actions (training return 60–77, against 92–117 without guidance); with the correct ordering, collection is also broad but tilted toward better actions (101–112). The correct ordering adds a further +8 on top.

IQN. Here the content-free part is essentially absent: all five content-free targets land at the level of IQN without guidance (136.0 over the 15 runs), while COMPASS reaches 151.9. Over eight paired seeds:

Paired difference, 8 seeds (95% CI, seeds improved)DQNIQN
Zero target − no auxiliary loss+14.2 ([+7.5, +20.8], 8/8)−4.5 ([−15.4, +6.3])
COMPASS − zero target+8.0 ([+0.3, +15.7], 7/8)+23.9 ([+13.9, +33.9], 7/8)

(8-seed means: DQN 128.2 / 142.3 / 150.3 and IQN 135.8 / 131.3 / 155.2 for no loss / zero / COMPASS.) So IQN is where the concern about generic regularization has the least room, and it is also where COMPASS gains the most. Finally, the ordering that helps is state-specific: a single action prior shared by all states (the per-action mean of T_K, which carries 2.4% of its variance) performs like an uninformative target on both learners.

AuthorsOpenReview ↗

Response to Reviewer fzDV (3/3)

Q4. "For KuaiSim: run DDPG and COMPASS on the reserved disjoint held-out user set with identical seeds and evaluation protocol, and state whether core performance conclusions hold on true held-out data."

The conclusions hold: on the 1,958 disjoint users, COMPASS is above all four baselines. We evaluated the same checkpoints, with the same seeds and per-batch response seeds, on users disjoint from training (133,967 logged prefixes, greedy slates, no exploration noise). A run's score is the mean over checkpoints 1,600/1,800/2,000, matching the tail average of the official KuaiSim protocol; this metric was fixed before COMPASS was evaluated (Appendix E.2, Table 23):

Methods11s13s17s19s23mean ± sd
DDPG8.818.929.228.898.898.95 ± 0.16
TD39.608.929.009.219.539.25 ± 0.30
A2C9.989.149.248.789.359.30 ± 0.44
HAC9.319.649.059.799.419.44 ± 0.29
COMPASS (K=1)9.749.909.4110.0610.119.85 ± 0.28
COMPASS (K=64)9.969.8310.019.749.849.88 ± 0.11

COMPASS (K=1) is above DDPG (+0.90, paired p = 0.008), HAC (+0.40, p = 0.007) and TD3 (+0.59, p = 0.017) on 5/5 seeds, and above A2C (+0.55) on 4/5; K=64 gives +0.93 (5/5), +0.43 (4/5), +0.62 (5/5) and +0.58 (4/5). The per-checkpoint curves (Figure 17) are also markedly more stable than DDPG's.

W3 / Q5. "Provide resource-matched comparisons accounting for COMPASS's offline completion data and computational overhead. Additionally, measure action-ranking consistency across the course of online training […]"

Cost. The offline stage costs 80 GPU-minutes once (270,000 branches, 418,496 rollouts, at most 2.2M simulator steps) and is shared by every guided run; one online run costs 66 GPU-minutes for DQN and 76 for COMPASS (Appendix E.5, Table 25). Two step-matched comparisons: another 500 online epochs under the same configuration barely move DQN (129.6, seed 0), and giving DQN the same completions as data does not close the gap (reward injection 126.6, MC-return pretraining 135.9, comparison pretraining 132.3; Figures 3–4). Details are in our reply to Reviewer jMUz (Q1–Q2).

Ranking consistency: the completion-derived ranking does not go stale. On 1,018 fixed held-out states we measured the pairwise agreement between the ordering of the learner's Q over legal actions and the frozen teacher's ordering, at all 21 saved checkpoints of the eight IQN runs (Appendix E.8, Figure 19). IQN trained without guidance rises from chance (0.505 at epoch 0) to 0.64 by epoch 100 and stays at 0.64–0.65 through epoch 500 (SD across seeds 0.011); the one DQN run with saved checkpoints shows the same curve (0.52 → 0.62). An independently trained learner converges toward the teacher's ranking and does not drift away. Guided IQN sits at 0.73 while β > 0 and relaxes to 0.62 after β reaches zero, so the learner is not locked to the teacher's ordering, and its gain persists after the relaxation.

If these results address the concerns about generalization and about the source of the gain, we would appreciate the reviewer's reassessment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: summary of new experiments and revisions

We thank the reviewers for their careful reading and constructive questions. Below is a short summary; each result is given in full in the reply to the reviewer who asked for it. All changes in the revised PDF are marked in blue.

New experiments (paper protocols and seeds throughout)

  1. KuaiSim on the 1,958 disjoint held-out users. COMPASS is above DDPG, TD3, A2C and HAC (9.85 vs. 8.95–9.44; Appendix E.2). (fzDV Q4)
  2. Content-free targets on DQN and IQN (zero, permuted, random, state-independent, reversed). They keep the loss, schedule and regularization of COMPASS but remove or invert the ordering. The ordering decides the outcome: on DQN 113.0 with the reversed ordering vs. 153.5 with the correct one; on IQN only the correct ordering helps (Section 6.5, Appendix E.3). (fzDV W2/Q3)
  3. A teacher fitted on logged returns, without completion rollouts: 139.5 vs. 153.5 for the completion-based teacher (Appendix E.6). (fzDV Q2, jMUz Q3)
  4. Teacher accuracy on direct vs. extrapolated action pairs (Appendix E.7). (fzDV W1/Q1)
  5. Ranking consistency across training, eight IQN seeds: the learner's own ordering converges toward the frozen teacher's and stays there (Appendix E.8). (fzDV Q5, j4F2 W1)
  6. Paired significance over eight seeds: +22.2 on DQN and +19.4 on IQN, 8/8 seeds each, p = 4×10⁻⁴ (Appendix E.4). (j4F2 W4)
  7. Cost of the offline stage next to one online run (Appendix E.5, Table 25). (fzDV W3/Q5, jMUz Q1–Q2)

Revisions to the paper

  • New Section 6.5 (held-out users and content-free targets) and Appendix E.1–E.8 with the results above.
  • Related work on RL-based recommendation (Section 2; Appendix E.1). (j4F2 W2)
  • Centering notation unified to C_s. (j4F2 W3)
  • The rule behind the schedule choice stated explicitly in Section 6.4. (jMUz W2, j4F2 W4)
  • Limitations on simulator-based evaluation expanded. (jMUz W1)
  • Offline completion counts in Appendix D.1 stated in one convention (270,000 branches, 418,496 rollouts).

We hope the new controls make the division of labor explicit: the completion policy supplies comparisons; the learner decides what to collect and what to keep.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.