- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
COMPASS collects alternative-action completions under a fixed policy, fits a reusable action-advantage predictor, centers its scores over legal actions, and uses them as an auxiliary loss alongside ordinary TD learning. Centering preserves action gaps while discarding stale return levels. The method is evaluated with DQN and IQN on RL4RS and via slate preferences with DDPG on KuaiSim, accompanied by theoretical analyses of centered updates, policy-shifted comparisons, and decision loss.
Critical strengths
- The paper develops a clear, useful idea: old completions may retain reliable relative action information after absolute returns become stale. The centered interface is simple and cleanly separated from environmental reward and TD targets.
- The evidence is unusually diagnostic. Reward injection, pretraining, no-centering, permutation, reversed signs, guidance duration, and completion statistics are controlled; the same frozen model improves two value learners on every paired seed and transfers to a slate setting. Theory exposes the acceleration–bias trade-off.
Critical weaknesses
- Validation is confined to learned recommendation simulators. Since completion collection also relies on branching within such environments, robustness to simulator misspecification and transfer to logged or real systems remain unclear.
- Offline preparation is substantial:135,000 prefixes and fitted return distributions, and is not included in the common online interaction budget. Baseline comparisons therefore do not capture total data and compute cost, and schedule/K choices differ materially across environments.
Questions for authors
- How does COMPASS compare at matched total environment interactions, including completion collection?
- What are completion collection/training costs and the break-even point when reusing the frozen model?
- How sensitive are gains to simulator error or to completions generated from logged/world-model approximations?
3 comments
Thank you for the cost analysis, which resolves Q2. However, simulator sensitivity (Q3) remains untested, and the logged-return teacher performing like a zero target shows the gain depends on simulator branching, so I keep my score at 2.
Follow-up: scope, and a second simulator
We thank the reviewer for the follow-up and are glad the cost analysis resolved Q2.
Scope. We agree: the gain depends on branching. COMPASS compares alternative actions at the same state, which logs alone cannot provide, and the logged-return teacher (139.5, close to the zero target at 140.7) shows exactly this. This is the setting of the paper and of all baselines we compare against, which also learn by interacting with the same simulator; COMPASS additionally queries it offline with a fixed behavior-cloned policy μ. Learning purely from logs is outside our scope; our limitations already state that COMPASS "assumes access to alternative continuations from a shared prefix, limiting use where branching is unavailable or prohibitively expensive."
Simulator sensitivity. We trained a second RL4RS-A simulator from scratch (same DIEN architecture, a quarter of the logged sessions; validation AUC 0.91) and repeated the whole pipeline in it with all hyperparameters unchanged. The two simulators differ substantially: on the same 135,000 prefixes, completion returns correlate 0.82 and the sign of the completion advantage agrees in 69% of states.
| RL4RS-A, DQN learner (seeds 0 / 1 / 2) | Original simulator | New simulator |
|---|---|---|
| DQN | 125.4 / 135.8 / 124.6 (mean 128.6) | 171.6 / 179.0 / 185.4 (mean 178.7) |
| COMPASS-DQN | 157.3 / 154.6 / 148.6 (mean 153.5) | 268.6 / 280.5 / 274.5 (mean 274.5) |
The gain holds without retuning (+95.8 vs. +24.9), so it is not tied to one particular simulator.
Response to Reviewer jMUz
We thank the reviewer for the positive assessment and the precise questions.
We now report the completion interactions next to the online budget, and the gap remains under the two step-matched comparisons we can make. On RL4RS-A the offline stage completes 270,000 intervention/reference branches over 135,000 prefixes with 418,496 rollouts (1–4 adaptive repeats), at most 2.2M simulator steps, about four online runs' worth (504k steps each). We list the two separately rather than summing them: the completions are taken by the fixed behavior-cloned policy μ, which never learns or acts online, and they are collected once and shared by every run. Step-matched comparisons:
The offline stage costs about 1.2 online runs and is paid once. Same RTX 4090, each run alone (Appendix E.5, Table 25):
The frozen teacher is shared by every guided run on RL4RS-A (364 in our logs: all seeds, all K, all ablations and sweeps). Best-of-K is computed analytically from one fitted return distribution, so all K come from the same collection.
Logged approximations recover part of the gain but not the part that comes from comparing alternative actions. A teacher fitted on logged returns alone, with no branching in the simulator, reaches 139.5 vs. 153.5 for the completion-based teacher on the same seeds (Appendix E.6). On simulator error: completions and training use the same simulator, as do all baselines, so we use no information source the baselines lack. Both simulators are trained on real interaction logs and the completion prefixes are logged sessions, and on KuaiSim the gain holds on 1,958 users never used for completion (9.85 vs. 8.95–9.44 for the four baselines; Appendix E.2). We agree that validation on a live system is beyond this work and have expanded the limitations accordingly.
The schedule follows where the guidance enters. On a bootstrapped value (DQN, IQN) persistent guidance biases the targets, so β must decay (constant β: 119.1 vs. 150.3); on a policy (the KuaiSim actor) it does not enter bootstrapping and can stay on. K sets how optimistic the teacher is; a value learner uses it only early and a policy learner throughout, so the preferred K differs. We report the sweep for each environment (Appendix B.7, Table 8; Appendix C.5, Table 18) and state the rule in Section 6.4.
We hope the interaction counts, the cost table and the held-out results answer these questions, and would be grateful if the reviewer took them into account in the final assessment.