- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper recasts dense retrieval for recommenders as geometry. With item embeddings on the unit sphere, cosine ranking is angular ranking, so a query under a score threshold returns exactly a spherical cap whose area is proportional to how many items are probed. Retrieval becomes a covering problem: cover the future target set with the least total cap area. Theorem 1 bounds an m-cap cover against a single cap by a ratio in which the radius enters to the power d−1 while m enters only linearly, so small radius gains buy large savings. Theorem 2 explains why the natural construction fails: history cluster centers carry sampling error of order θ/√n plus drift error from moving interests, and the cost inflation grows exponentially in dimension again. The method therefore learns the placement: a Transformer over the history emits m heads, each producing a normalized center and a softplus radius, trained with a hinge miss term under a smooth softmin assignment plus an area penalty. On MovieLens-1M/20M, KuaiRand-27k and an internal short-video set it reports the best HR@10 and NDCG@10 on all four, with a Scan metric showing the probed corpus fraction falling from 24.1% at one head to 9.76% at three.
Critical strengths
-
The framing is the contribution and it is correct. Cosine ANN retrieval equals spherical-cap membership, and retrieval cost equals cap area, so multi-query retrieval is a covering problem whose radius advantage is exponential in dimension minus one. I have not seen this written down for recommender retrieval before, and it explains in one line why multiple interest vectors pay off at d = 128 but are much less obviously worth it at d = 32. This is framing that will outlive the specific model, and the connection to min-sum-radii in Footnote 3 is stated honestly, including the NP-hardness that then justifies the surrogate objective.
-
Theorem 2 is used to make a falsifiable prediction and the prediction is tested. The natural instantiation of history clustering, UIH_k-means, is the worst method in Table 1 on MovieLens-1M by a wide margin — HR@10 of 2.91 against SASRec's 28.3 — and ComiRec's clustering-derived variants are also weak on the short-video set. That is exactly what the sampling-plus-drift bound predicts, and it is a far more satisfying use of a theorem than the usual properties paragraph. The local-density check in Section 4.2 is also the right way to defend the area-as-cost proxy rather than waving at the uniformity assumption.
-
Reporting the Scan metric alongside HR and NDCG is the correct measurement decision, and the numbers are informative rather than decorative. Ranking quality alone cannot distinguish a method that finds the target from a method that finds it while probing less of the corpus. Table 4 gives Scan of 24.1% at m = 1, 9.76% at m = 3 and 1.12% at m = 16 on MovieLens-20M, and Table 2 shows that dropping the area term raises Scan from 9.8% to 38.5% while barely changing ranking quality — which is exactly the trade-off Theorem 1 says to manage, measured. The case study in Figure 2 closes the loop at the user level, with cap centers 0.89 radians apart tracking group centers 0.94 radians apart for a two-interest history.
Critical weaknesses
-
The theory and the experiment do not describe the same target set, and the 100%-recall idealization is not innocuous even in the authors' own numbers. Theorems 1 and 2 are about covering a set T of future items at full recall, and Footnote 1 makes that idealization explicit. Section 4.1 then evaluates with full-corpus leave-one-out, stating "exactly one item is held out for each user". When |T| = 1 there is no covering problem at all: one cap always suffices, θ_m = θ_1, and Theorem 1's advantage is identically zero. So the empirical section is not testing the object the theory is about; HR@k on a single positive is a ranking metric, and the geometric cost model never gets exercised as a covering claim on real multi-positive targets. Moreover, Table 2 reports full-corpus HR@200 of 75.6 against cap-restricted HR@200 of 73.0 for the full objective, so the learned union of caps misses on the order of 2.6pp of the targets that the ranking stage would otherwise have found — the coverage assumption the theory rests on does not hold in the deployed model, and the paper does not connect the gap. Figure 2a is the same phenomenon made visible: a dispersed history produces two caps of radius 1.25 and 1.16 radians probing 29.3% of the pool, the target is covered, and it still ranks 437th. The internal short-video dataset construction already uses a future engagement window, so a multi-positive protocol is answerable with a protocol change rather than a new model.
-
There is no variance anywhere in the main results. Table 1 gives seven methods across four datasets as bare numbers, with no seeds, no standard deviations, and no statement of how many runs produced each cell. Large gaps are safe, but the paper's "consistently improves" claim does not hold uniformly as stated: on KuaiRand-27k the method is worse than both single-interest baselines on H@200 (10.4 versus SASRec's 10.9 and HSTU's 11.5) and again on N@200 (2.29 versus 2.35 and 2.33), even though it wins H@10 and N@10 there. A tail regression on the behavioral dataset, in the metric closest to the retrieval budget the paper motivates, needs either error bars or a sentence of discussion rather than silence.
-
The serving argument is the motivation but is not measured. The whole point of minimizing total cap area is that the retriever touches fewer items, yet in practice m caps are executed against an ANN index and the cost is index traversals and candidate generation under a per-request budget. The paper reports neither candidate counts at matched recall in a real index nor latency, and there is no comparison against a cheap alternative at equal budget — for instance, retrieving top-K around each of m cluster centers with K tuned to the same total candidates. Without this, area is a proxy that has not been connected to the quantity an industrial system actually pays for.
-
The radius is set by a regularizer, not by a target, and there is no sensitivity study. The two loss-balancing coefficients λ_area and λ_ntp are fixed at 0.1 and 1 in the implementation paragraph and used unchanged across all four datasets, together with a softmin temperature of 0.05, yet these are the knobs that determine the learned cap widths — the paper's central object. Two consequences are unaddressed. First, a practitioner cannot obtain a required recall level, because the objective trades coverage against area through a coefficient rather than through a stated recall constraint. Second, the same constants are reused across four corpora that differ by more than an order of magnitude in item count and that are trained under different negative-sampling schemes (full-corpus scoring on the 3,883-item MovieLens-1M versus 128 sampled negatives elsewhere), so the reported cap compactness may partly reflect a lucky constant rather than a property of the geometry. A sweep over λ_area showing recall against probed area, ideally with a version that enforces a budget constraint, would fix this cheaply. On head allocation, the paper does analyze the aggregate version: Section 4.4 and Appendix G.2 report that as m grows, supervision concentrates on a subset of caps, leaving others with few assignments and weak gradients, which explains the m = 16 quality drop. What is missing is the per-user version. m is a global constant, so a user whose history is concentrated still emits m caps, and Figure 2b shows precisely that — two centers 0.13 radians apart with a Jaccard overlap of 0.87, which the paper calls safe overlap. It would be useful to know how much of the reported Scan budget goes to redundant heads, since a per-user allocation of active heads (or a sparsity term over head utilization) would presumably buy back area for free.
Questions for authors
-
Can you evaluate on a multi-positive future window and report the metric the theory speaks about — P@R at fixed recall, or the realized total cap area needed to cover the target set — on MovieLens-20M and the internal dataset? This is my main concern and I would raise my soundness score if the covering advantage shows up empirically. (Clarification, rebuttal-feasible.)
-
Why does the method regress on KuaiRand-27k H@200 while improving on H@10? I suspect radius calibration for low-activity users, i.e. heads widening past what a short history supports, but the paper neither reports a per-activity breakdown nor speculates.
-
Table 4 gives Scan as a function of the global m, and Figure 2b shows two nearly identical caps (Jaccard 0.87) for a concentrated history. Can you report, per user, the effective number of distinct caps, the share of total cap area contributed by redundant heads, and the distribution of learned radii? If head utilization were allocated per user rather than fixed at m, the m = 3 versus m = 8 trade-off in Table 4 should improve, which would be a stronger efficiency result than the current constant-m configuration.
-
How much of the internal-dataset gain is attributable to the fact that its target window contains multiple positives, unlike the leave-one-out academic sets? The largest gains are reported on exactly that dataset, and I would like that confound separated.
-
Have you tried replacing the analytic area penalty with a hard budget — minimize miss subject to Σ_j A(θ̂_j) ≤ c, swept over c? That would give the recall-versus-budget curve an engineer needs and would make Theorem 1's prediction directly checkable.
3 comments
Thanks for the additional experiments. The multi-positive protocol with precision at fixed recall is exactly the evidence I asked for, the λ_area sweep plus the Lagrangian argument answers the sensitivity question, and the measured 1.2 to 2.5 percent redundancy rules out my guess that per-user head allocation would recover much area. You also corrected my mistaken assumption that the internal set used a future window, and I accept that. I am raising soundness to 3. Two things are still missing, so the rating stays at 3: no seeds or standard deviations appear anywhere, which leaves "consistently improves" unsupported on cells like KuaiRand H@200, and there is still no index-level measurement of candidates or latency at matched recall, so the retrieval-cost claim remains an area proxy. One further thing I would like to see stated in the paper: if hyperparameters were chosen on the internal dataset and reused on the public ones, then the internal numbers are the optimistic ones and the public ones are conservative, which is the opposite emphasis from the current presentation. Please also say which values were selected that way.
We thank the reviewer for the thoughtful response and for raising the soundness score. We believe that the remaining two “missing things” are already addressed in the current version, and we apologize for not stating them clearly enough. We would be grateful if you would consider raising the rating in light of the following clarifications.
1. Seeds and standard deviations. For the online experiment, Figure 3a reports 95% confidence intervals for every metric, and all eight changes are statistically significant. For the offline experiments, the results in the new Appendix G.1 were already obtained over three random seeds, with standard deviations computed accordingly. We omitted them only to keep the formatting consistent with the other offline tables. We have now added the number of seeds and standard deviations to Table 6 in the latest version. The improvements of our method are statistically significant there as well.
2. Index-level measurement of candidates and latency at matched recall. This is precisely what Figure 3b reports. The top panel shows the number of candidates required by each method at matched recall. At the two marked operating points, one head requires approximately 2.2× and more than 2.5× as many candidates as three heads to achieve the same recall. Reading the bottom panel at these same candidate budgets further shows that the single-head arm has higher ANN index latency. Thus, at matched recall, the single-interest method has higher index latency.
Regarding hyperparameter selection, the selected hyperparameters include the number of Transformer layers, hidden dimension, number of interest heads, , , and learning rate. We would also like to clarify why we do not believe this makes the internal results too optimistic and the public results too conservative. First, the baselines were tuned on the internal dataset using the same procedure, so all methods are compared under the same tuning protocol. Second, the ablations in Tables 3, 4, and 5 were conducted on MovieLens, and the configurations they favor agree with those selected on the internal dataset. This suggests that the chosen hyperparameters are not specific to a particular corpus.
Author response
We thank the reviewer for the valuable comments. In response, we have conducted an extensive set of additional experiments and uploaded a revised version of the paper, with all changes highlighted in blue. Our point-by-point responses are provided below.
1. Multi-positive evaluation with precision at a given recall
Following your advice, we have added a new evaluation protocol in which the last 20% of each evaluation user's interaction sequence is used as the target set. We report precision at 50% and 80% recall (Section 4.1-Evaluation metrics; Appendix G.1). Our method consistently outperforms all baselines under this protocol. This provides further evidence of the effectiveness of our covering formulation in the multi-positive setting.
2. Regression on KuaiRand-27k HR@200
We would like to make three points.
The performance gap is approximately one percentage point, and our method remains the best at the top of the ranking.
As discussed in Section 4.3, HSTU is a particularly strong baseline on this dataset, as it was specifically designed to handle long and non-stationary sequences.
We believe this result reflects a trade-off induced by the area regularization. As described in Section 3.3, items are ranked by their minimum slack across predicted caps, while the area term encourages the model to produce compact caps. Compact caps lead to more discriminative rankings for items inside or close to a cap, improving retrieval quality near the top of the ranking. In contrast, items far from all predicted caps tend to receive similarly large slack values, making their relative ordering less discriminative at deeper ranking positions. Thus, the lower HR@200 is a consequence of the same cap compactness that reduces retrieval cost.
3. Adaptive number of caps
We would like to clarify three aspects.
We have already identified an adaptive number of retrieval regions as future work in our limitations (Appendix H). Realizing this would require substantial modifications to the model architecture, together with dedicated experiments. We believe this is better addressed as a separate research direction rather than as an incremental extension of the current work, and it was beyond the scope of the rebuttal period.
Our objective jointly encourages target coverage and penalizes the total area of the predicted caps. Consequently, heavily overlapping caps are generally discouraged, since they incur redundant area without providing additional coverage. The heavily overlapping caps shown in the case study represent a deliberately selected edge case: a user with a short and highly concentrated interaction history. This example is not representative of the general behavior of our model.
Following your advice, we have quantified the redundant fraction of the probed area. Let Ci denote the set of items covered by cap i. We define the redundant fraction as 1−∣⋃iCi∣/∑i∣Ci∣. Across different numbers of caps, with m ranging from 2 to 16, this fraction remains between 1.2% and 2.5%. This indicates that the predicted caps overlap only minimally in practice, and that very little of the probed area is counted more than once.
4. The target window on the internal dataset
We believe there may be a misunderstanding here, and we apologize if our description was unclear. The evaluation protocol is identical across all datasets: we use leave-one-out evaluation with a single held-out target per user. The internal dataset does not use a multi-positive target window. A more likely explanation for the larger gains on the internal dataset is that our hyperparameters were selected primarily based on the internal dataset and then reused unchanged on the public datasets. Therefore, the results on public datasets may actually be conservative compared with what could be achieved through dataset-specific hyperparameter tuning.
5. Hard budget versus the area penalty
The hard-budget and penalty-based formulations express the same underlying trade-off between missed targets and retrieval cost. Specifically, the constrained optimization problem minLmiss subject to Larea≤c has the Lagrangian Lmiss+λ(Larea−c), which, up to the constant term −λc, gives the penalized objective Lmiss+λLarea. Thus, varying λ provides a practical way to trace the recall–budget trade-off associated with different choices of c. Following your advice, we have conducted this sweep on MovieLens-1M:
The results illustrate the expected trade-off: reducing the retrieval budget comes at the cost of lower recall.