acceptodds
Under review as a conference paper at ICLR 2027

MERA: Matrix-Entropy Reranking Actor-Critic for Sequential Recommendation

12 views

Abstract

Sequential reranking is a critical final stage in recommender systems. Given candidates and display positions, it makes sequential decisions over discrete ordered slates. Existing methods often optimize proxy scores, fusion weights, or latent controls before decoding them into slates, creating an action-interface mismatch. We propose Matrix-Entropy Reranking Actor-Critic (MERA), which models reranking as a Markov decision process with display-or-discard assignments as actions. Using the item–slot assignment probabilities of ranking policies, we define matrix-entropy-regularized Bellman operators and establish their contraction properties. We characterize the marginal actor problem's first-order stationarity through a KKT decomposition into an inner entropic transport problem and an outer projected score-alignment condition. MERA parameterizes feasible assignment plans using scores and solves the inner problem with Sinkhorn scaling. Critic action gradients at hard assignments sampled by Gumbel perturbation and Hungarian rounding guide projected score alignment. On two datasets with three simulators, MERA improves mean total session reward by up to 14.43% over the strongest of seven baselines per setting, with competitive session-level exposure diversity. MERA maintains low inference latency as candidate sets grow, supporting deployment in industrial reranking systems. Our code is available at https://anonymous.4open.science/r/MERA_code-CD4D/.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Recommendation System, Sequential Reranking, Differentiable Ranking

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_TM78OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MERA formulates sequential reranking as an item–slot assignment problem. The actor outputs a soft assignment via Sinkhorn scaling; at execution time, Gumbel perturbation plus Hungarian rounding produces a valid hard slate with no repeated items. An ordered-slate critic scores the actually executed action, and matrix entropy regularizes the soft plan. The setup is tested on KuaiRand Pure and MovieLens-1M through KuaiSim, RecSim, and RecoGym adaptations.

Critical strengths
  1. The assignment formulation neatly handles the "which item goes where" problem: hard rounding guarantees a valid, non-repeating slate, and the ordered-slate critic evaluates what was actually shown to the user. The match between action representation and evaluation is clean.
  2. The component study is well designed — it examines how stochastic execution, Ordered-Q, and the actor update interact, not just individual ablations. Testing across two datasets and three simulator variants gives useful evidence. The reported inference latency under 8 ms also suggests the approach is deployable.
Critical weaknesses
  1. There is a gap between what the theory analyzes and what the implementation does. The paper acknowledges that the clean (un-perturbed) plan doesn't necessarily match the hard sampler's marginals, and that matrix entropy isn't the entropy of the executed policy. The ablations show the update is useful, but they don't tell us whether the plan–sampler mismatch actually matters. Without measuring this discrepancy, it's hard to connect the theoretical motivation to the trained policy's behavior.
  2. The baselines and settings leave notable gaps. SPG and ListCVAE are included, but GFN4Rec is a closely related method and its absence makes it harder to pin down the assignment formulation's advantage. All results come from simulators, and the RecSim/RecoGym adaptations share representations from the KuaiSim pipeline. The α study and latency sweep are helpful but don't cover different shortlist/slate sizes or independent Gumbel scales, leaving the method's sensitivity to these practical choices largely untested.
Questions for authors
  1. Empirically, how large is the gap between the clean assignment plan and the hard sampler's marginals in the reported settings?
  2. Is a comparison with GFN4Rec feasible under matched conditions (same state info, candidates, simulator dynamics, training budget)? If not, what's the main obstacle?
  3. Beyond the α analysis, how does recommendation quality change with shortlist size, slate size, or Gumbel perturbation strength?
  4. Was Gumbel perturbation on during the main evaluation? Please report the eval_gumbel setting and any overrides used.
  5. Which representations and user-response components are shared across the three simulator adaptations, and which differ?
5 comments
AuthorsOpenReview ↗

Response to Reviewer TM78 (1/3)

We thank the reviewer for the careful review and for recognizing the match between the action representation and the executed-slate critic, as well as the design of the component study. We answer the five questions in turn, with new measurements on trained MERA policies.

Throughout, ZZ denotes the item–slot scores, P=Sinkhorn⁡(Z/α)P=\operatorname{Sinkhorn}(Z/\alpha) the clean assignment plan, and XX the executed hard slate.

W1/Q1. How large is the gap between the clean plan and the sampler's marginals?

The clean plan and sampled display marginals show high overlap on both datasets: 98.44% on KuaiRand Pure and 93.54% on ML1M. With M^\widehat M the empirical marginals of repeated executions, we report

\left|P_{ij}-\widehat M_{ij}\right|, \qquad \widehat M=\frac{1}{L}\sum_{\ell=1}^{L}X^{(\ell)}, \qquad \text{overlap}=1-\frac{d}{2}.$$ We use five checkpoints and four held-out requests per dataset, with $L=65{,}536$ executions per request at the training perturbation scale. | Dataset | Mean distance $d$ | Display-marginal overlap | | --- | ---: | ---: | | KuaiRand Pure | 0.0313 | 98.44% | | ML1M | 0.1292 | 93.54% | The measurements use the training entropy coefficients ($\alpha=0.2$ on ML1M, $0.7$ on KuaiRand Pure). **Optimization evidence complements the high marginal overlap.** Appendix C.3 preserves the Bellman contraction property with either the true-marginal or clean-plan entropy bonus. **Direct evaluation of the implemented update.** Additional RecSim evaluations summarize objective changes at evaluated checkpoints across training. We report $EQ=\mathbb E_X Q_\theta(O,X)$ and $F=EQ+\alpha H_{\rm mat}(P_{\rm clean})$, separating changes in expected hard-slate critic value from the clean-plan entropy contribution. The table gives mean changes and 95% confidence intervals, in units of $10^{-5}$. | Simulator | Dataset | $\Delta F$ [95% CI] | $\Delta EQ$ [95% CI] | | --- | --- | ---: | ---: | | RecSim | KuaiRand Pure | +7.876 [5.747, 10.004] | +6.231 [4.349, 8.112] | | RecSim | ML1M | +11.703 [9.659, 13.747] | +9.845 [8.584, 11.106] | **Both $\Delta F$ and $\Delta EQ$ have positive means and confidence intervals entirely above zero on both datasets.** These results support a significant improvement in the regularized objective and in expected hard-slate critic value over the evaluated training checkpoints. ## W2/Q2. Comparison with GFN4Rec under matched conditions **We compare with GFN4Rec [1] under the shared session benchmark, extending the original SPG and ListCVAE comparisons to an autoregressive generator–evaluator framework.** The comparison controls the four requested conditions: - **State information.** Both use the same state-encoder architecture and observation/history inputs. - **Candidates.** Both methods draw from the same full item pool. GFN4Rec generates slates autoregressively from it, and MERA learns its per-request shortlist from the same pool. - **Simulator dynamics.** Both use the same three simulators and two datasets and are evaluated by the same session-return metric. - **Training budget.** GFN4Rec uses the same number of training updates and the same checkpoint-saving rule as the corresponding MERA run in each setting. For the session-return task, we pair GFN4Rec's trajectory-balance generator with an evaluator using the same Ordered-Q architecture and set its trajectory reward to $\max(Q_\theta,0)+0.1$. This is a session-objective adaptation of the original immediate list-reward formulation. Both methods output one executed slate per request, without test-time beam search or candidate-slate reranking. The new five-run GFN4Rec means and MERA's reported Total rewards are: | Simulator | Dataset | GFN4Rec | MERA | | --- | --- | ---: | ---: | | KuaiSim | KuaiRand Pure | 10.0365 | 10.0576 | | KuaiSim | ML1M | 8.4121 | 9.1577 | | RecSim | KuaiRand Pure | 6.8873 | 7.3934 | | RecSim | ML1M | 6.7490 | 7.2336 | | RecoGym | KuaiRand Pure | 10.4264 | 13.9175 | | RecoGym | ML1M | 10.6415 | 13.1689 | **MERA matches the adapted GFN4Rec closely in mean Total reward on KuaiSim–KuaiRand Pure and achieves higher means in the other five settings.**
AuthorsOpenReview ↗

Response to Reviewer TM78 (2/3)

W2/Q3. Shortlist size, slate size, and Gumbel perturbation strength

Our paper studies the entropy coefficient α\alpha (Appendix B.3) and reports MERA's inference latency for K=100,200,400K=100,200,400 (7.782, 6.887, and 7.203 ms). We now add the effect of these sizes on recommendation quality, and of the perturbation strength.

Stable returns across shortlist and display sizes. We train each of four (K,N)(K,N) configurations with the original benchmark's training budget and report mean ± SD in KuaiSim on both datasets.

Shortlist size KKDisplay length NNKuaiRand Pure: Total rewardML1M: Total reward
1002010.0576 ± 0.10189.1577 ± 0.3718
2002010.0119 ± 0.16729.0628 ± 0.2773
4002010.0206 ± 0.09239.0150 ± 0.3102
100109.9621 ± 0.17629.1168 ± 0.2740

Every configuration retains at least 98.4% of the default (100,20)(100,20) mean return under the same training budget. Halving the display length to N=10N=10 retains 99.1% and 99.6% on KuaiRand Pure and ML1M, respectively, while halving item exposures per user–slate interaction. All four configurations also exceed the original N=20N=20 main-benchmark reference means: DDPG on KuaiRand Pure (9.7408 ± 0.1695) and HAC on ML1M (8.7534 ± 0.4922).

Stability across Gumbel scales. We freeze each checkpoint and its α\alpha, vary only the evaluation scale β\beta, and evaluate on 512 fixed initial-user contexts per checkpoint. Because the protocol differs from Table 1, we report variation across β\beta over five checkpoints per dataset:

DatasetGumbel scale β\beta → mean session return
KuaiRand Pure0 → 9.6207; 0.5 → 9.6104; 1 → 9.6091; 2 → 9.6007
ML1M0 → 7.8912; 0.25 → 7.7947; 0.5 → 7.8084; 1 → 7.8118

Across the tested scales, the mean-return range is only 0.21% on KuaiRand Pure and 1.22% on ML1M relative to each dataset's highest sweep mean, indicating limited sensitivity to evaluation perturbation. In training, the factorial study shows that stochastic execution lowers return with MLP-Q and DPG (T1 to T5) but raises it by 0.2153 and 0.6153 with Ordered-Q and score alignment (T3 to MERA).

Q4. Evaluation Gumbel setting

Yes. Gumbel perturbation was enabled during evaluation in all six main settings (three simulators × two datasets), with the same noise distribution and scale as in training. As specified in Section 4.4, independent noise εij∼Gumbel(0,1)\varepsilon_{ij}\sim\mathrm{Gumbel}(0,1) is added directly to the item–slot scores, so the evaluation noise scale is 1; the Sinkhorn temperature α\alpha then controls the transport map applied to the perturbed scores.

W2/Q5. Shared and simulator-specific components

The three simulators share the input representations, state interface, and session handling, and differ in how user responses are generated. Appendices A.1–A.2 describe this design:

ComponentCommon setupSimulator-specific part
User/item inputsDataset profiles, histories, candidate features, and pretrained KuaiSim-derived item encodingsRecSim and RecoGym construct their latent item factors from these encodings
State and candidate interfaceCommon observation/history handling and full candidate pool—
Response generationCommon action and feedback interfaceKuaiSim: learned multi-response probabilities and intra-slate similarity adjustment; RecSim: affinity, evolving satisfaction, and exposure memory; RecoGym: latent interests, item bias, and stochastic drift
Session handlingWhole-session history updates and the base temper-based termination mechanismThe response generated by each core drives the session updates

Holding the representation and session handling fixed isolates the effect of the response mechanism. We agree that, for the same reason, the three simulators should not be read as independent validations of real user dynamics. To complement this comparison, we validated the response model on held-out feedback (response-link AUC 0.6790 on KuaiRand Pure and 0.8321 on ML1M, with calibration results) and showed that MERA's advantage over HAC persists under perturbed response temperature, position decay, and termination hazard. Details are given in our response to Reviewer pSP3 (W1/Q1).

AuthorsOpenReview ↗

Response to Reviewer TM78 (3/3)

Summary

  • Q1. The clean plan and the executed display marginals overlap by 98.44% on KuaiRand Pure and 93.54% on ML1M. The reported RecSim evaluations across training checkpoints show positive average ΔF\Delta F and ΔEQ\Delta EQ on both datasets, with 95% intervals above zero.
  • Q2. Under matched state information, candidates, simulator dynamics, and training budget, MERA is comparable to the adapted GFN4Rec on KuaiSim–KuaiRand Pure and higher in the other five settings.
  • Q3. Every (K,N)(K,N) configuration retains at least 98.4% of the default return under the same training budget, and evaluation returns vary by at most 1.22% across Gumbel scales.
  • Q4. Evaluation used Gumbel(0,1) perturbation, identical to training, in all six main settings.
  • Q5. The simulators share representations, the state interface, and session handling, and differ in response generation; held-out response validation and mechanism perturbations complement this comparison.

We hope these answers address the reviewer's questions.

References

[1] Shuchang Liu et al. Generative Flow Network for Listwise Recommendation KDD, 2023.

Reviewer_pSP3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MERA, an actor–critic framework for sequential recommendation reranking. It represents each action as an item–position assignment with an additional discard column. The actor produces a feasible continuous assignment plan through Sinkhorn scaling, and Gumbel perturbation followed by Hungarian rounding generates the executed slate. An ordered-slate critic estimates long-term value, while the actor uses a projected score-alignment update motivated by a KKT characterization. Matrix entropy regularizes the assignment plan and enters the value-learning target. Experiments on two datasets under three simulator configurations report improvements in session reward, competitive exposure diversity, and millisecond-scale inference latency.

Critical strengths

The framework coherently connects structured slate generation with value learning on executed actions. The display-or-discard representation jointly expresses item selection and position assignment, while the critic evaluates the concrete slate shown to the environment. Although the underlying assignment formulation is classical, its integration with the proposed optimization procedure is technically coherent.

The internal empirical analysis is reasonably thorough. The paper reports five-seed results across six simulator–dataset settings and includes a complete three-factor ablation of stochastic execution, critic architecture, and actor update. The finding that stochastic exploration helps only in combination with suitable value learning and optimization is informative. The latency measurements also provide evidence of practical inference feasibility within the tested configurations.

Critical weaknesses

1、The experimental conclusions rely heavily on simulator fidelity, but this critical component is not adequately validated against real behavior. KuaiSim combines a learned immediate-response model with an explicit intra-slate similarity penalty and a temper-based termination rule. The RecSim and RecoGym adaptations change the response dynamics, but derive item factors from pretrained KuaiSim encodings and retain a shared session interface and termination mechanism. Consequently, the three environments provide useful variation, but should not be interpreted as independent validations of realistic user dynamics. The paper describes response-model training and data splits, but does not report sufficient quantitative validation of held-out response prediction, probability calibration, list-context effects, or session termination against real observations. Simulator rollouts evaluating the learned policy do not establish the simulator’s fidelity. The sensitivity analysis over the similarity-penalty coefficient is useful, but does not address the broader reliability of counterfactual slate responses. This issue is particularly consequential because only the executed slate receives observed feedback. The simulator must extrapolate to alternative slates, and the critic must subsequently generalize their long-term values. MERA additionally uses a local critic action gradient to guide optimization. The Bellman contraction and KKT results do not resolve these identification and approximation issues. I do not require an online experiment, but more direct offline validation is needed to connect the simulated improvements to credible behavioral value.

Questions for authors

1、How are the simulators validated as evaluators of alternative recommendation slates? Please provide independent evidence of response-prediction accuracy and calibration, session/termination fidelity, or policy-ordering consistency against real observations. How robust is MERA’s advantage to misspecification of response, position, and termination dynamics beyond the tested similarity-penalty variation?

2、How reliable is the practical critic-gradient approximation, and what does matrix entropy independently contribute? In a tractable small-action setting, can the proposed local gradient be compared with the actual sampler-induced expected-value gradient? Please also clarify whether the entropy bonus remains beneficial when transport temperature and perturbation settings are controlled separately.

3、How does MERA compare with strong alternative reranking frameworks under matched conditions? A representative generator–evaluator comparison should control the candidate pool, feedback budget, session objective, and inference/search budget. Please also clarify whether the reward gains and learning efficiency persist across different combinations of candidate size and display length.

4 comments
AuthorsOpenReview ↗

Response to Reviewer pSP3 (1/3)

We thank the reviewer for the detailed and expert assessment, and for recognizing the coherence of the assignment-based actor–critic and the thoroughness of the internal analysis. We address the three concerns in turn: (i) simulator fidelity, (ii) the practical update and the role of matrix entropy, and (iii) matched comparison and behavior across candidate and display sizes.

Throughout, ZZ denotes the item–slot scores, P=Sinkhorn⁡(Z/α)P=\operatorname{Sinkhorn}(Z/\alpha) the clean assignment plan, and XX the executed hard slate.

W1/Q1. Held-out feedback validation and robustness to misspecification

Response prediction and calibration. We evaluate the frozen response model, before the list-similarity adjustment, on 8,192 held-out test records per dataset, and fit a logistic calibration link on 8,192 disjoint validation records (training and splits in Appendices A.1–A.2).

DatasetResponse-link AUCBrier: original / calibratedECE: original / calibrated
KuaiRand Pure0.67900.2431 / 0.22560.1145 / 0.0159
ML1M0.83210.2420 / 0.16200.2608 / 0.1026

Validation-fitted calibration improves both Brier score and ECE on both datasets. KuaiRand Pure uses logged behavioral feedback; ML1M uses the paper's implicit-feedback construction (rated items as positives, sampled unrated items as negatives).

Termination, list-context effects, and policy ordering. Validating these against real observations requires logged sessions with natural exits and logs of alternative slates or policies shown to comparable users. The public KuaiRand Pure and ML1M data contain no such records, so these components cannot be validated directly. This limitation is shared by the simulators we build on and by prior work evaluated in them: KuaiSim [1] fits its response model to logged feedback but specifies session exit through a configurable leave mechanism; RecSim [4] and RecoGym [5] specify user dynamics rather than fit them to logged sessions; and HAC [6] and GFN4Rec [3] evaluate policies in such simulators. Following this common practice, we validate the response model on held-out feedback and assess the remaining components through robustness to misspecification.

Advantage under mechanism perturbations. Beyond the original similarity-coefficient study (Appendix B.3), we perturb response temperature, position decay, and termination hazard, and substitute the validation-calibrated response link. Policies are frozen and evaluated on fixed initial-user contexts; because this protocol differs from Table 1, we report the mean MERA − HAC [6] difference over five runs on KuaiRand Pure:

MechanismTested settingMean MERA − HAC
Reference conditionOriginal link+0.2206
Response temperature0.8 / 1.2+0.2825 / +0.1855
Position decay0.02 / 0.06+0.1436 / +0.0268
Termination hazard0.067 / 0.1+0.1300 / +0.1062
Response calibrationValidation-fitted link+0.1593

MERA's mean advantage over HAC remains positive in every tested condition, including the calibrated response link. These checks complement the simulator evaluation without treating it as direct validation of natural exits or counterfactual slate outcomes.

AuthorsOpenReview ↗

Response to Reviewer pSP3 (2/3)

Q2. Reliability of the practical update and the role of matrix entropy

Comparison with the sampler-induced gradient in a tractable small-action setting

The theoretical quantity and the practical target (Sec. 4.4, Appendix C.8) are

\qquad \widehat G_\varepsilon=\nabla_X Q_\theta(O,X_\varepsilon).$$ **We carried out the comparison the reviewer suggests on the trained model.** We fix 18 of the 20 positions and let four real shortlist items compete for the remaining two, so all 12 feasible slates can be enumerated and scored by the unchanged Ordered-Q. Four request contexts and two position pairs give eight action domains per dataset. The reference is the central-difference gradient of the sampler-averaged value, mapped to plan coordinates; reference, target, and evaluation use independent batches of 65,536 draws. | Dataset | Mean cosine: critic target vs. reference | Mean cosine: update vs. regularized gradient | Positive regularized directional estimates | | --- | ---: | ---: | ---: | | KuaiRand Pure | 0.997 | 0.989 | 8/8 | | ML1M | 0.980 | 0.984 | 8/8 | **The local critic gradient closely matches the sampler-induced gradient, and all 16 score directions increase the regularized objective** (stable across three finite-difference steps; minimum cosine above 0.998). **Direct evaluation of the implemented update.** Beyond this local comparison, additional RecSim evaluations summarize objective changes at evaluated checkpoints across training. We report $EQ=\mathbb E_X Q_\theta(O,X)$ and $F=EQ+\alpha H_{\rm mat}(P_{\rm clean})$, separating changes in expected hard-slate critic value from the clean-plan entropy contribution. The table gives mean changes and 95% confidence intervals, in units of $10^{-5}$. | Simulator | Dataset | $\Delta F$ [95% CI] | $\Delta EQ$ [95% CI] | | --- | --- | ---: | ---: | | RecSim | KuaiRand Pure | +7.876 [5.747, 10.004] | +6.231 [4.349, 8.112] | | RecSim | ML1M | +11.703 [9.659, 13.747] | +9.845 [8.584, 11.106] | **Both $\Delta F$ and $\Delta EQ$ have positive means and confidence intervals entirely above zero on both datasets.** These results support a significant improvement in the regularized objective and in expected hard-slate critic value over the evaluated training checkpoints. **Matched end-to-end evidence.** Table 1's T6 differs from MERA only in using a structured DPG update, and score alignment raises Total reward from 9.9555 to 10.0576 (KuaiRand Pure) and from 8.7784 to 9.1577 (ML1M). ### Separating transport temperature, perturbation, and entropy **Entropy and transport temperature are a single control in MERA, so the separable controls are entropy and perturbation. Varying each with the other fixed leaves performance stable, and the full formulation outperforms SPG at the same temperature.** MERA's inner problem is $$P_\alpha(Z)=\arg\max_{P\in\mathcal T_{K,N}} \{\langle Z,P\rangle+\alpha H_{\rm mat}(P)\} =\operatorname{Sinkhorn}(Z/\alpha),$$ so the entropy coefficient is the transport temperature ($\tau=\alpha$), while the Gumbel perturbation $\varepsilon_{ij}\sim\mathrm{Gumbel}(0,\beta)$ is set independently. The same conditions give $\Pi Z=\alpha\Pi\log P$, so entropy shapes both which policy is optimized and how its scores are updated. - **Entropy at fixed perturbation.** Appendix B.3 varies $\alpha$ with $\beta$ fixed; performance changes smoothly around the default $\alpha=0.7$. - **Perturbation at fixed entropy.** Holding each checkpoint and its $\alpha$ fixed, varying $\beta$ changes mean return by at most 0.21% (KuaiRand Pure) and 1.22% (ML1M) (details in our response to Reviewer TM78, Q3). - **Full formulation versus SPG [2].** Our SPG baseline shares MERA's partial-assignment shortlist ($K=100$), 20 Sinkhorn iterations, and transport temperatures (0.7 / 0.2): | Dataset | SPG | MERA | Relative gain | | --- | ---: | ---: | ---: | | KuaiRand Pure | 9.5754 | 10.0576 | +5.04% | | ML1M | 8.0930 | 9.1577 | +13.16% | Because both share the transport map at the same temperature, these gains reflect MERA's entropy-regularized stochastic formulation and its optimization rather than the Sinkhorn layer itself.
AuthorsOpenReview ↗

Response to Reviewer pSP3 (3/3)

Q3. Matched generator–evaluator comparison and fixed-budget scale robustness

GFN4Rec under matched conditions

We compare with GFN4Rec [3] as a representative generator–evaluator framework, pairing its trajectory-balance generator with an evaluator of the same Ordered-Q architecture. The requested conditions are matched as follows:

  • Candidate pool. Both draw from the same full item pool; MERA learns its per-request shortlist from it.
  • Feedback budget. Same number of training updates and checkpoint-saving rule as the corresponding MERA run.
  • Session objective. Same session reward and simulators; GFN4Rec uses max⁡(Qθ,0)+0.1\max(Q_\theta,0)+0.1 as its trajectory reward, a session-objective adaptation of the original formulation.
  • Inference budget. One executed slate per request, without beam search or candidate reranking.
SimulatorDatasetGFN4RecMERA
KuaiSimKuaiRand Pure10.036510.0576
KuaiSimML1M8.41219.1577
RecSimKuaiRand Pure6.88737.3934
RecSimML1M6.74907.2336
RecoGymKuaiRand Pure10.426413.9175
RecoGymML1M10.641513.1689

MERA is comparable to GFN4Rec in KuaiSim on KuaiRand Pure and achieves a higher mean Total reward in the other five settings.

Stable returns across candidate sizes and display lengths

We train each (K,N)(K,N) configuration with the original training budget and report mean ± SD in KuaiSim.

Shortlist size KKDisplay length NNKuaiRand Pure: Total rewardML1M: Total reward
1002010.0576 ± 0.10189.1577 ± 0.3718
2002010.0119 ± 0.16729.0628 ± 0.2773
4002010.0206 ± 0.09239.0150 ± 0.3102
100109.9621 ± 0.17629.1168 ± 0.2740

Every configuration retains at least 98.4% of the default (100,20)(100,20) return under the same training budget. Halving the display length retains 99.1% and 99.6% while halving item exposures per interaction. All four configurations also exceed the strongest N=20N=20 baselines in the original benchmark: DDPG on KuaiRand Pure (9.7408 ± 0.1695) and HAC on ML1M (8.7534 ± 0.4922).

Summary

  • (i) Q1. The response model is validated on held-out feedback, and MERA's advantage over HAC stays positive under every tested response, position, and termination perturbation; components that public data cannot validate are handled following common simulator practice.
  • (ii) Q2. In the tractable small-action setting, the local critic gradient matches the sampler-induced gradient (mean cosine 0.997 and 0.980), and RecSim evaluations across training checkpoints show positive average ΔF\Delta F and ΔEQ\Delta EQ on both datasets, with 95% intervals above zero. Entropy and temperature form a single control; performance is stable when entropy and perturbation are varied separately, and the full formulation outperforms SPG.
  • (iii) Q3. Under matched conditions, MERA is comparable to GFN4Rec on KuaiSim–KuaiRand Pure and higher elsewhere, and retains at least 98.4% of its return across (K,N)(K,N) within the same budget.

We hope these results address the reviewer's concerns.

References

[1] Kesen Zhao et al. KuaiSim: A Comprehensive Simulator for Recommender Systems, NeurIPS, 2023.

[2] Patrick Emami and Sanjay Ranka. Learning Permutations with Sinkhorn Policy Gradient. arXiv:1805.07010, 2018.

[3] Shuchang Liu et al. Generative Flow Network for Listwise Recommendation.,KDD, 2023.

[4] Eugene Ie et al. RecSim: A Configurable Simulation Platform for Recommender Systems, 2019.

[5] David Rohde et al. RecoGym: A Reinforcement Learning Environment for the Problem of Product Recommendation in Online Advertising. arXiv:1808.00720, 2018.

[6] Shuchang Liu et al. Exploration and Regularization of the Latent Action Space in Recommendation. WWW, 2023.

AuthorsOpenReview ↗

Follow-up on Our Responses and Additional Experiments

Dear Reviewer pSP3,

This is a gentle reminder that the author discussion period ends in two days. We would greatly appreciate your feedback on whether our responses and additional experiments address your concerns. If they do, please consider updating your assessment accordingly. If any points remain unresolved, please let us know, and we would be happy to clarify or discuss them further. Thank you again for taking the time to review our submission and for your comments, which have helped us improve the paper.

Reviewer_ieK5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MERA is an actor–critic method for sequential recommendation that optimizes the actual ordered slate shown to users, rather than proxy scores or latent controls, to maximize cumulative session reward.

Main contributions

  • Assignment-based actions: A (K \times (N+1)) matrix jointly represents item selection, display position, and discard decisions. Sinkhorn scaling produces a feasible soft assignment; Gumbel perturbations enable exploration, and Hungarian rounding produces the executed slate.
  • Matrix-entropy formulation: Entrywise assignment entropy provides a tractable regularizer for the combinatorial action space. The paper proves contraction and unique fixed points for the corresponding policy-evaluation and optimality Bellman operators.
  • Theory-guided actor update: A KKT characterization separates feasible entropic transport from projected score alignment. In practice, the actor fits its projected scores to detached action-gradient targets from a position-aware, attention-based critic trained on executed slates.

Validation

Across two datasets × three simulator settings, using seven baselines and five seeds, MERA achieves the highest mean session reward, per-request reward, and session depth. Total reward improves 3.25%–14.43% over the strongest baseline in each setting, with competitive session-level exposure diversity. Ablations indicate that stochastic exploration works best together with the ordered-slate critic and projected update; perturbation tests show generally more gradual ranking changes and smaller reward losses. Reported median inference latency is below 8 ms, including a 100–400-candidate shortlist sweep.

Critical strengths
  • A coherent, tractable formulation of slate-level sequential control. MERA jointly represents item selection, position, and omission through display-or-discard assignments, while its critic evaluates the slate actually executed. The KKT characterization connects feasible entropic transport with projected score alignment, providing a concrete optimization mechanism rather than just a new action encoding. This is consequential because it addresses the paper’s central action-interface mismatch while keeping the combinatorial decision problem computationally manageable.
  • Consistent empirical gains with evidence that the components work together. MERA achieves the highest mean total session reward across all six simulator–dataset settings, improving by 3.25%–14.43% over the strongest baseline in each setting while retaining competitive exposure diversity. The factorial ablations further show that exploration is most beneficial when paired with the ordered-slate critic and projected update. This strengthens the assessment because the gains are not confined to one setting, and the ablations substantiate the value of the integrated design rather than merely reporting an aggregate improvement.
Critical weaknesses
  • The connection between the theory and the implemented actor update remains insufficiently validated. The KKT result concerns the exact gradient of expected hard-action value, but training substitutes a detached local critic gradient at a sampled assignment. Moreover, the soft plan used for entropy regularization generally differs from the executed policy’s true marginals. The paper acknowledges both approximations but does not quantify their effects or validate the surrogate’s alignment with actual policy improvement. This is consequential because the theory-guided update is a central contribution, yet the guarantees do not establish that the implemented update optimizes the stated objective.
  • The evaluation provides less independent evidence of generalization than the six settings suggest. All results are simulator-based, and the RecSim and RecoGym adaptations reuse KuaiSim-derived item encodings and a common termination mechanism. Consequently, consistent gains may partly reflect shared modeling assumptions rather than robustness across substantially different recommendation environments. Five-seed means and standard deviations, without significance tests or uncertainty estimates for the improvements, also leave some comparisons unresolved. This limits confidence that the reported advantages will persist beyond the tested simulator family, particularly given the paper’s industrial applicability claims.
Questions for authors
  • How consequential are the practical approximations? What evidence establishes that the projected critic-gradient update improves expected executed-slate value, despite its bias? Have you measured the discrepancy between the clean Sinkhorn plan and the sampler’s actual assignment marginals? This would help determine whether the theoretical formulation meaningfully explains the implemented algorithm’s behavior.
  • How statistically reliable are the reported improvements? Were methods evaluated using matched users and simulator randomness, and are paired confidence intervals or significance tests available for total-reward differences—particularly in RecoGym, where baseline variability is substantial? This would distinguish consistent improvements from differences that remain uncertain with five seeds.
3 comments
AuthorsOpenReview ↗

Response to Reviewer ieK5 (1/2)

We thank the reviewer for the constructive review and for recognizing both the optimization mechanism and the factorial component study. We address the two concerns in turn: (i) whether the practical approximations are consistent with the theory, and (ii) the statistical reliability and generality of the improvements, with new diagnostics on trained models, held-out response validation, mechanism-sensitivity checks, and paired significance tests.

Throughout, ZZ denotes the item–slot scores, P=Sinkhorn⁡(Z/α)P=\operatorname{Sinkhorn}(Z/\alpha) the clean assignment plan, and XX the executed hard slate.

W1/Q1. How consequential are the practical approximations?

The actor uses a detached critic gradient at a sampled slate in place of the exact gradient of expected hard-action value, and the clean plan PP generally differs from the sampler's true marginals. We answer the three sub-questions in order.

(a) How do we assess the practical surrogate?

Theorem 1 requires G(P)=∇PEX∼S(P)Qθ(O,X)G(P)=\nabla_P\mathbb E_{X\sim\mathsf S(P)}Q_\theta(O,X), whereas MERA uses G^ε=∇XQθ(O,Xε)\widehat G_\varepsilon=\nabla_X Q_\theta(O,X_\varepsilon). We assess the practical consequence of this approximation through changes in the stated objective and expected hard-slate critic value after the implemented update. This directly tests the update's effect, without equating the surrogate with the exact gradient.

(b) Does the implemented update improve the objective?

Direct evaluation of the implemented update. Additional RecSim evaluations summarize objective changes at evaluated checkpoints across training. We report EQ=EXQθ(O,X)EQ=\mathbb E_X Q_\theta(O,X) and F=EQ+αHmat(Pclean)F=EQ+\alpha H_{\rm mat}(P_{\rm clean}), separating changes in expected hard-slate critic value from the clean-plan entropy contribution. The table gives mean changes and 95% confidence intervals, in units of 10−510^{-5}.

SimulatorDatasetΔF\Delta F [95% CI]ΔEQ\Delta EQ [95% CI]
RecSimKuaiRand Pure+7.876 [5.747, 10.004]+6.231 [4.349, 8.112]
RecSimML1M+11.703 [9.659, 13.747]+9.845 [8.584, 11.106]

Both ΔF\Delta F and ΔEQ\Delta EQ have positive means and confidence intervals entirely above zero on both datasets. These results support a significant improvement in the regularized objective and in expected hard-slate critic value over the evaluated training checkpoints.

End-to-end effect. In Table 1, T6 differs from MERA only in using a structured DPG update; score alignment raises Total reward from 9.9555 to 10.0576 on KuaiRand Pure and from 8.7784 to 9.1577 on ML1M.

The RecSim update diagnostic and the matched T6/MERA comparison provide complementary evidence: positive average objective and critic-value changes across evaluated training checkpoints, and higher executed-slate session reward under matched end-to-end conditions.

(c) How far are the sampler's marginals from the clean plan?

We measure the distance between the clean plan PP and the empirical marginals M^\widehat M of executed slates,

\left|P_{ij}-\widehat M_{ij}\right|, \qquad \widehat M=\frac{1}{L}\sum_{\ell=1}^{L}X^{(\ell)}, \qquad \text{overlap}=1-\frac{d}{2}.$$ We use five checkpoints and four held-out requests per dataset, with $L=65{,}536$ executions per request at the training perturbation scale. | Dataset | Mean distance $d$ | Display-marginal overlap | | --- | ---: | ---: | | KuaiRand Pure | 0.0313 | 98.44% | | ML1M | 0.1292 | 93.54% | **This gap affects neither the theoretical guarantees nor the diagnostics above.** Appendix C.3 shows that the contraction results hold whether the entropy bonus is computed from the true policy marginals or from the clean plan, and the update evaluation above concerns the actual stochastic sampler. The overlap is lower on ML1M because it uses a smaller entropy coefficient ($\alpha=0.2$ versus $0.7$): the plan is sharper, so Hungarian rounding of the perturbed plan departs from it more often.
AuthorsOpenReview ↗

Response to Reviewer ieK5 (2/2)

W2. Independence of the simulator evidence

We agree that the three simulators should not be read as independent validations of real user dynamics. They share profiles, item representations, and session handling, and vary the response mechanism: KuaiSim uses learned immediate responses with an intra-slate similarity adjustment, the RecSim adaptation adds satisfaction and exposure memory, and the RecoGym adaptation uses evolving latent interests with stochastic drift (Appendices A.1–A.2). Their role is a controlled comparison across response dynamics. We add two forms of evidence.

Held-out response validation. We evaluate the response probabilities, before the list-similarity adjustment, on 8,192 held-out test records per dataset, and fit a calibration link on 8,192 disjoint validation records.

DatasetResponse-link AUCBrier: original / calibratedECE: original / calibrated
KuaiRand Pure0.67900.2431 / 0.22560.1145 / 0.0159
ML1M0.83210.2420 / 0.16200.2608 / 0.1026

KuaiRand uses logged behavioral feedback; ML1M uses the paper's implicit-feedback construction, with rated items as positives and sampled unrated items as negatives.

Sensitivity to the simulator's mechanisms. Extending the original similarity-coefficient study, we perturb response temperature, position decay, and termination hazard, and also substitute the validation-calibrated response link. All policies are frozen and evaluated on a fixed set of initial user contexts. Absolute returns therefore differ from Table 1, and the quantity of interest is the sign of the MERA − HAC difference. On KuaiRand Pure:

MechanismSettingMean MERA − HAC
Reference response linkOriginal+0.2206
Response temperature0.8 / 1.2+0.2825 / +0.1855
Position decay0.02 / 0.06+0.1436 / +0.0268
Termination hazard0.067 / 0.1+0.1300 / +0.1062
Response calibrationValidation-fitted link+0.1593

MERA retains a positive mean advantage under every tested change to the response, position, and termination assumptions.

Scope of the industrial claim. Our deployment-related claim concerns inference cost: MERA's median latency stays below 8 ms as the shortlist grows to 400 candidates. We do not present the simulator results as evidence of online effectiveness, and we will state this distinction explicitly in the revision.

W2/Q2. Statistical reliability of the reported improvements

MERA achieves higher mean Total reward in all six settings, nominally significant gains in five, and lower run-to-run SD than the strongest original baseline in every setting. Methods share the candidate pool, simulator configuration, reward objective, and evaluation budget. Matched runs use the same users and simulator randomness and are paired by evaluation seed. We apply one-sided paired t-tests to the five run-level differences, testing whether the mean MERA-minus-baseline return is positive.

SimulatorDatasetStrongest original baselineMean MERA − baselineRelative gainOne-sided p-value
KuaiSimKuaiRand PureDDPG+0.3168+3.25%0.0026
KuaiSimML1MHAC+0.4043+4.62%0.0127
RecSimKuaiRand PureSPG+0.2577+3.61%0.0164
RecSimML1MSPG+0.2691+3.86%0.0085
RecoGymKuaiRand PureTD3+1.2446+9.82%0.0825
RecoGymML1MSPG+1.6607+14.43%0.0027

P-values below 0.05 are shown in bold. In RecoGym–ML1M, the +14.43% mean gain is supported by p≈0.0027. On RecoGym–KuaiRand, the +9.82% mean gain is accompanied by substantially lower variability: MERA's SD is 0.42 versus TD3's 2.06. Together, the consistent mean gains, five nominally significant comparisons, and lower variability strengthen the statistical evidence for MERA.

Summary

In summary: (i) the reported RecSim evaluations across training checkpoints show positive average objective and expected critic-value changes on both datasets, with 95% intervals above zero, and score alignment outperforms matched DPG end to end; (ii) the clean plan and executed marginals overlap by 98.44% / 93.54%, leaving the guarantees unchanged; (iii) the response model is validated on held-out feedback, and MERA's advantage persists under mechanism perturbations; and (iv) matched evaluation shows positive gains in all six settings, nominal significance in five, and lower SD throughout. We hope these results address the reviewer's concerns.

AuthorsOpenReview ↗

Follow-up on Our Responses and Additional Experiments

Dear Reviewer ieK5,

This is a gentle reminder that the author discussion period ends in two days. We would greatly appreciate your feedback on whether our responses and additional experiments address your concerns. If they do, please consider updating your assessment accordingly. If any points remain unresolved, please let us know, and we would be happy to clarify or discuss them further. Thank you again for taking the time to review our submission and for your comments, which have helped us improve the paper.

Reviewer_iZWROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MERA, a RL framework for sequential reranking. It identifies an action-interface mismatch in existing RL-based reranking methods, where policies often optimize intermediate scores rather than the actual ordered slates presented to users. To address this issue, MERA represents reranking actions as constrained item–position assignments with an additional discard slot. The framework combines matrix-entropy regularization, Sinkhorn-based assignment planning, Gumbel perturbation, Hungarian rounding, and an ordered-slate critic. This paper further derive a KKT-based decomposition of the marginal actor optimization problem and propose a projected score-alignment update guided by critic action gradients. Experiments across two datasets and three simulators demonstrate consistent improvements in cumulative session reward over several baselines, along with competitive exposure diversity and inference efficiency.

Critical strengths

The paper addresses the important problem of session-wise reranking and presents a coherent actor–critic framework that integrates structured assignment modeling with long-term value optimization. The method demonstrates consistent improvements in session rewards across multiple simulator–dataset settings, suggesting its practical potential for sequential recommendation.

Critical weaknesses
  1. While MERA presents a coherent framework for session-wise reranking, the paper does not sufficiently distinguish its application-level contribution from existing assignment-based ranking approaches, such as SPG. On the methodological side, although the KKT-based decomposition provides a structured interpretation of actor optimization, its implications for a genuinely improved optimization strategy remain unclear. Consequently, the paper does not convincingly establish what new capabilities or advantages MERA introduces beyond existing approaches.

  2. Theorem 1 characterizes stationarity using the exact marginal-policy gradient, whereas the implemented actor update relies on a biased critic action-gradient surrogate. The paper neither characterizes the approximation quality nor provides sufficiently controlled comparisons against alternative projected policy-gradient methods under matched architectures and sampling mechanisms. This makes it difficult to determine whether the proposed KKT-guided update offers a meaningful optimization advantage, weakening the empirical support for the paper's central methodological contribution.

Questions for authors
  1. What are the key advantages of the proposed item–slot–discard formulation over existing assignment-based ranking methods, particularly SPG? Beyond the unified representation, what additional capabilities does MERA introduce for session-wise reranking?

  2. What is the distinct methodological contribution of the KKT-guided score-alignment update compared with existing constrained or projected policy-gradient methods? In particular, what optimization advantages follow from the proposed decomposition beyond providing an alternative formulation of the first-order optimality conditions?

  3. Theorem 1 relies on the exact marginal-policy gradient, whereas the practical update uses a biased critic action-gradient surrogate (Section 4.4 and Appendix C.8). Can authors provide theoretical justification or empirical diagnostics demonstrating how well this surrogate approximates the intended policy-improvement direction?

  4. Table 1 does not fully establish its advantage over other constrained optimization strategies. Can authors clarify whether the improvements persist against projected DPG or alternative structured policy-gradient updates under matched actor–critic architectures, sampling mechanisms, and entropy settings?

5 comments
AuthorsOpenReview ↗

Response to Reviewer iZWR (1/3)

We thank the reviewer for the careful assessment and for recognizing the coherence of the actor–critic framework and the consistency of the session-reward gains. We first present matched end-to-end evidence for the actor update, then explain what MERA adds beyond SPG and what the KKT decomposition enables in practice, and finally report diagnostics on trained models.

Throughout, ZZ denotes the item–slot scores, P=Sinkhorn⁡(Z/α)P=\operatorname{Sinkhorn}(Z/\alpha) the clean assignment plan, and XX the executed hard slate.

W2/Q4. Matched end-to-end evidence for the actor update

Under matched architectures, stochastic execution, and entropy settings, score alignment improves Total reward on both datasets. In Table 1, T6 uses an entropy-regularized straight-through DPG update that propagates the critic gradient through the soft Sinkhorn path, whereas MERA fits its scores to detached critic-gradient targets at sampled hard slates. Both share Gumbel execution, Ordered-Q, α\alpha, Sinkhorn–Hungarian decoding, and the networks; the actor update is the only difference:

DatasetT6 (structured DPG)MERA (score alignment)Relative gain
ML1M8.77849.1577+4.32%
KuaiRand Pure9.955510.0576+1.03%

Across the four pairs in Table 1 that differ only in the actor update (T1/T4, T2/T3, T5/T7, T6/MERA), score alignment raises Total reward in seven of eight dataset comparisons, including both matched stochastic pairs.

W1/Q1. Beyond the representation: MERA optimizes the executed slate

We agree that the assignment representation is shared with SPG [1]; MERA's contribution lies in what is optimized on it. SPG uses the doubly stochastic matrix as a relaxation of the action. MERA makes the executed hard slate the object that is sampled, evaluated, regularized, and used as the actor's learning target:

SPGMERA
Executed actionHungarian rounding of the Sinkhorn planGumbel-perturbed plan rounded by Hungarian: a stochastic policy over complete slates
Critic evaluatesHard permutationHard slate, with position-aware Ordered-Q for long-term session value
Actor gradient taken atSoft Sinkhorn relaxationSampled hard slate (detached critic-gradient target)
Role of entropySinkhorn-temperature smoothing [2]; auxiliary entropy coefficient 0.01 in our baseline (Appendix A.4)Matrix entropy in the return and Bellman target, with coefficient equal to the transport temperature (Sec. 4.3)
Decision scopePermutations of a fixed item set (original SPG)Joint selection, placement, and omission via the discard column

In MERA, the actor is trained on the same object that the user sees and the critic evaluates.

The gains do not come from the representation. Our SPG baseline already uses a partial-assignment actor with a discard column and shares MERA's shortlist (K=100K=100), 20 Sinkhorn iterations, transport temperature (0.7 / 0.2), learning rates, and actor network (Appendix A.4). MERA still improves KuaiSim Total reward substantially:

DatasetSPGMERARelative gain
KuaiRand Pure9.575410.0576+5.04%
ML1M8.09309.1577+13.16%
AuthorsOpenReview ↗

Response to Reviewer iZWR (2/3)

W1/Q2. What the KKT decomposition enables in practice

Beyond restating the first-order conditions, the decomposition enables three practical properties: (i) the actor's target is taken at the executed hard slate; (ii) the entropy term is absorbed by the transport map, so the scores regress directly toward the critic gradient without a transport Jacobian or differentiation through Hungarian rounding; and (iii) as a result, stochastic exploration over complete slates becomes usable training signal.

In MERA, the objective depends on the scores through

\xrightarrow{\text{Sinkhorn}} \underset{\text{assignment plan}}{P} \xrightarrow{X\sim\mathsf S(P)} \underset{\text{hard slate}}{X} \xrightarrow{Q_\theta,\ H_{\rm mat}(P)} \underset{\text{objective}}{F(P)},$$ where $F(P)=V(P)+\alpha H_{\rm mat}(P)$ and $V(P)=\mathbb E_{X\sim\mathsf S(P)}Q_\theta(O,X)$. Direct chain-rule ascent faces two obstacles: Hungarian rounding has zero derivative almost everywhere, and backpropagating through Sinkhorn iterations requires the full normalization history. SPG's actor instead evaluates $Q_\theta(O,P)$ at the soft action and propagates its gradient through Sinkhorn [1]; SOFT top-$k$ [3] and implicit Sinkhorn differentiation [4] obtain transport derivatives without unrolling. MERA's score-residual update requires no transport Jacobian. Let $G=\nabla_PV$, and let $\Pi$ remove row- and column-wise offsets. Combining the outer actor condition with the inner transport condition gives $$\Pi(G-\alpha\log P)=0, \qquad \Pi Z=\alpha\Pi\log P \quad\Longrightarrow\quad D:=\Pi(G-Z)=0.$$ The inner relation absorbs the entropy term, so $G$ becomes a direct regression target for the scores. With exact $G$ and converged transport, chain-rule ascent follows $AD$ with transport Jacobian $A=\partial T_\alpha/\partial Z$, whereas free-score alignment follows $D$ itself. **This gives a different preconditioning of the ideal stationarity residual.** In practice, $G$ is replaced by the critic's action derivative $\widehat G_\varepsilon$ at a sampled hard slate (W2/Q3), held fixed by stop-gradient, and the actor minimizes $\tfrac12\|\Pi(Z_\phi-\operatorname{sg}[\widehat G_\varepsilon])\|_F^2$, giving $$\Delta Z\propto\Pi\!\left(\operatorname{sg}[\widehat G_\varepsilon]-Z\right).$$ The actor-loss gradient flows only through the score network (Appendix C.8). **The factorial study shows how the update interacts with stochastic execution and the critic.** Table 1: | Change in Total reward | KuaiRand Pure | ML1M | | --- | ---: | ---: | | Adding stochastic execution under DPG and MLP-Q (T1 → T5) | −1.40% | −2.95% | | Adding stochastic execution under DPG and Ordered-Q (T2 → T6) | +1.94% | +6.04% | | Adding stochastic execution under score alignment and Ordered-Q (T3 → MERA) | +2.19% | +7.20% | | Replacing DPG with score alignment under stochastic execution and MLP-Q (T5 → T7) | +2.11% | +12.70% | With MLP-Q and DPG, stochastic execution reduces return; with Ordered-Q, it benefits both updates. **Under matched stochastic execution, score alignment outperforms DPG with both critics (T5 → T7 and T6 → MERA): regressing toward targets at the explored slates turns exploration into a larger gain than differentiating through the relaxation.** ## W2/Q3. What is the practical effect of the implemented update? Theorem 1 requires $G(P)=\nabla_P\mathbb E_{X\sim\mathsf S(P)}Q_\theta(O,X)$, whereas MERA uses $\widehat G_\varepsilon=\nabla_X Q_\theta(O,X_\varepsilon)$. The approximation lies in using the critic derivative at a sampled slate as the target for $G(P)$; we assess its practical consequences through direct update evaluations. **Direct evaluation of the implemented update.** Additional RecSim evaluations summarize objective changes at evaluated checkpoints across training. We report $EQ=\mathbb E_X Q_\theta(O,X)$ and $F=EQ+\alpha H_{\rm mat}(P_{\rm clean})$, separating changes in expected hard-slate critic value from the clean-plan entropy contribution. The table gives mean changes and 95% confidence intervals, in units of $10^{-5}$. | Simulator | Dataset | $\Delta F$ [95% CI] | $\Delta EQ$ [95% CI] | | --- | --- | ---: | ---: | | RecSim | KuaiRand Pure | +7.876 [5.747, 10.004] | +6.231 [4.349, 8.112] | | RecSim | ML1M | +11.703 [9.659, 13.747] | +9.845 [8.584, 11.106] | **Both $\Delta F$ and $\Delta EQ$ have positive means and confidence intervals entirely above zero on both datasets.** These results support a significant improvement in the regularized objective and in expected hard-slate critic value over the evaluated training checkpoints.
AuthorsOpenReview ↗

Response to Reviewer iZWR (3/3)

Summary

  • (i) W1/Q1 — beyond SPG. With the assignment representation shared, MERA makes the executed hard slate the object that is sampled, evaluated, regularized, and used as the actor's target. It outperforms an SPG baseline that already uses a discard column and shares MERA's shortlist and transport map by 5.04% and 13.16%.
  • (ii) W1/Q2 and W2/Q4 — optimization advantage. The KKT decomposition yields a score-learning rule that requires neither sampler gradients nor a transport Jacobian, and it outperforms structured DPG under matched conditions on both datasets and in seven of eight factorial comparisons.
  • (iii) W2/Q3 — practical update effect. The reported RecSim evaluations across training checkpoints show positive average ΔF\Delta F and ΔEQ\Delta EQ on both datasets, with 95% intervals above zero, supporting improvement in the regularized objective and expected hard-slate critic value.

In short, MERA's novelty lies not in the assignment encoding but in making the executed slate the unit of exploration, evaluation, and policy improvement. We hope these clarifications and results address the reviewer's concerns.

References

[1] Patrick Emami and Sanjay Ranka. Learning Permutations with Sinkhorn Policy Gradient. arXiv:1805.07010, 2018.

[2] Gonzalo Mena et al. Learning Latent Permutations with Gumbel-Sinkhorn Networks, ICLR, 2018.

[3] Yujia Xie et al. Differentiable Top-k with Optimal Transport NeurIPS, 2020.

[4] Marvin Eisenberger et al. A Unified Framework for Implicit Sinkhorn Differentiation CVPR, 2022.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their constructive feedback. Below we summarize the main new evidence and the revisions; details are in the individual responses.

Novelty. The assignment representation is shared with SPG; MERA's contribution is to make the executed hard slate the object that is sampled, evaluated, entropy-regularized, and used as the actor's target, with the KKT decomposition turning this into a Jacobian-free score update. Against an SPG baseline that shares the shortlist, discard column, and transport map, MERA improves Total reward by 5.04% and 13.16% in KuaiSim.

New evidence

  • Practical approximation (all reviewers). The critic-gradient target closely matches the numerical sampler-averaged gradient (mean cosine 0.997 / 0.980), the implemented update improves both the regularized objective and expected critic value, and the clean plan overlaps with the executed marginals by 98.44% / 93.54%.
  • Simulators and significance (ieK5, pSP3, TM78). The response model is validated on held-out feedback, MERA's advantage over HAC persists under every tested mechanism perturbation, and the gains are positive in all six settings and nominally significant in five under paired tests.
  • Scale and perturbation (pSP3, TM78). Every tested (K,N)(K,N) configuration retains at least 98.4% of the default return, and evaluation returns vary by at most 1.22% across Gumbel scales.

Revisions (marked in red)

The theorems, main results, and conclusions are unchanged. The main text adds short clarifications in Sections 2, 4.4, and 5.1–5.3, and the abstract scopes the industrial claim to inference latency. New results appear in Appendix B.4 (significance), B.5 (KK/NN), B.6 (simulator validation), B.7 (Gumbel scale), and C.9 (approximation quality).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.