- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors release a large-scale dataset built from real user search interactions on the Kuaishou platform, supporting the discovery of multiple tasks (i.e., recall, ranking, and relevance judgment) in the search of e-commerce domain. Comprehensive analysis of the dataset are also provided.
Critical strengths
- The scale and richness of dataset could enable novel discovery in the ecommerce search domain.
Critical weaknesses
-
The value proposition of this dataset is not clear. While the paper provides several analysis on the dataset's patterns, distribution, and benchmarking, the broader value of the dataset (esp on "scale") is unclear.
- Experiments are mostly run on a subset (i.e., KuaiSearch-Lite). There is no scaling experiment showcase the value of the large-scale dataset which is claimed as one major contribution.
- The claimed positioning/superiority of KuaiSearch to other datasets are not supported by experiments. Including more empirical comparison with other datasets, e.g., their impact to model performance and transferrability/generalization capability would be helpful.
- The claimed contribution on personalization is not backed by experiment.
-
(Minor) Dataset quality and soundness.
- Potential misclaims: JDsearch actually reports 9.4MM searches, which is greater than the released dataset. The number in Table 1 shows test set size which could be misleading.
- Potential bias: certain customer behavior data may depend on listing or production environment, and may be biased when generalizing to common sense personalized preference.
Questions for authors
- Could you report results on the full dataset and a training-size scaling curve?
- Could you benchmark personalized retrieval and cold-start users, and report variance across seeds?
- Could you compare the lexical-gap and ambiguity statistics against Amazon-C4 and JDsearch?
On ethics
- Is there a takedown or opt-out process?
- Were follower counts and GMV coarsened? What fraction of users can be uniquely identified from (gender, age group, province, follower bucket)?
- Are queries screened for personal information?
4 comments
R2. On personalized retrieval and training-unseen users
User history supplies interest context beyond the current query. We compare query-only DPR with history-augmented DPR on Full, holding the separate-encoder architecture, candidate pool, and training protocol fixed. Click and purchase Recall@10 improve by 23.72% and 37.13% relative to query-only DPR, respectively.
| Feedback | Model | Recall@10 | Recall@50 | MRR@10 | HR@10 |
|---|---|---|---|---|---|
| Click | DPR (separate encoders) | 0.0721 | 0.1765 | 0.0420 | 0.1046 |
| Click | DPR (separate encoders) + history | 0.0892 | 0.1952 | 0.0577 | 0.1284 |
| Purchase | DPR (separate encoders) | 0.0808 | 0.1832 | 0.0398 | 0.0820 |
| Purchase | DPR (separate encoders) + history | 0.1108 | 0.2220 | 0.0565 | 0.1125 |
User groups: AUC / click NDCG@5
Users are grouped by training-period request count; items are grouped by training exposures n and clicks c. The 0 group contains users with no training-period requests; their available pre-request histories are retained. MostPop scores by n. DNN, DIN, and WuKong outperform both simple baselines in every user group on AUC/NDCG@5 and every item group on AUC.
| Training requests | Users | Test requests | MostPop | Smoothed CTR | DNN | DIN | WuKong |
|---|---|---|---|---|---|---|---|
| 0 (training-unseen) | 1,934 | 4,742 | 0.5264 / 0.3453 | 0.5754 / 0.3980 | 0.6566 / 0.4088 | 0.6563 / 0.4111 | 0.6541 / 0.4132 |
| 1–5 | 3,649 | 7,845 | 0.5116 / 0.3290 | 0.5695 / 0.3767 | 0.6561 / 0.3883 | 0.6572 / 0.3855 | 0.6589 / 0.3902 |
| 6–20 | 4,487 | 10,458 | 0.5074 / 0.3257 | 0.5663 / 0.3700 | 0.6627 / 0.3755 | 0.6617 / 0.3745 | 0.6629 / 0.3736 |
| >20 | 3,556 | 11,625 | 0.5019 / 0.3341 | 0.5650 / 0.3774 | 0.6677 / 0.3868 | 0.6695 / 0.3861 | 0.6700 / 0.3886 |
User NDCG@5 ranks the full exposed candidate set and macro-averages over click-positive requests; top-five misses score zero.
Item groups: AUC
| Item group | Test items | MostPop | Smoothed CTR | DNN | DIN | WuKong |
|---|---|---|---|---|---|---|
| n = 0 | 154,551 | 0.5000 | 0.5000 | 0.6547 | 0.6568 | 0.6597 |
| n > 0, c = 0 | 255,250 | 0.4868 | 0.5132 | 0.6463 | 0.6449 | 0.6473 |
| 1 ≤ c ≤ 5 | 175,890 | 0.4320 | 0.5842 | 0.6561 | 0.6563 | 0.6566 |
| 6 ≤ c ≤ 20 | 37,098 | 0.4037 | 0.6160 | 0.6690 | 0.6693 | 0.6685 |
| c > 20 | 5,976 | 0.4060 | 0.6834 | 0.7231 | 0.7230 | 0.7233 |
The three-run statistics for the seven ranking models appear in R1.
R3. On lexical-gap and ambiguity comparisons
We add a cross-dataset comparison of query length and lexical mismatch. Compared with JDsearch, KuaiSearch has lower mean lexical coverage (0.561 vs. 0.723) and a higher zero-overlap rate (14.9% vs. 8.2%), illustrating lexical mismatch in short queries. Amazon-C4 has substantially longer queries, representing a different query setting.
| Dataset | Mean query length (tokens) | Mean lexical coverage | Zero-overlap rate |
|---|---|---|---|
| KuaiSearch | 3.31 | 0.561 | 14.9% |
| JDsearch | 2.76 | 0.723 | 8.2% |
| Amazon-C4 | 44.65 | 0.080 | 12.3% |
We further use clicked-item entropy to characterize behavioral dispersion for the same query. KuaiSearch has higher mean and median entropy (3.85 and 3.90) than JDsearch (3.28 and 3.45), corresponding to more dispersed clicked-item distributions. This statistic is not applicable to Amazon-C4's single-target setting.
| Dataset | Mean clicked-item entropy | Median clicked-item entropy |
|---|---|---|
| KuaiSearch | 3.85 | 3.90 |
| JDsearch | 3.28 | 3.45 |
| Amazon-C4 | N/A | N/A |
R4. On dataset value
KuaiSearch combines readable queries and product content with pre-request user histories, exposure sets, and observed click/purchase feedback. This combination supports studying semantic matching together with behavioral personalization, complemented by a separate human-annotated relevance component.
| Dataset | Existing characteristics | What KuaiSearch adds |
|---|---|---|
| Amazon (Electronics, 5-core) [1] | Category-derived queries; 5-core filtering | Real search requests; low-frequency users and items retained |
| Amazon-C4 [2] | Generated complex queries and readable product text | Real queries and user-linked behavioral context |
| ESCI [3] | Real queries, readable product text, and human relevance labels | User histories, request-level exposures, and click/purchase feedback |
| JDsearch [4] | Real requests and user behavior; anonymized text | Readable queries and product content for language-model-based modeling |
The history comparison in R2 demonstrates a concrete use of this context: personalized DPR improves retrieval for both clicks and purchases. The scaling experiment in R1 further shows the value of additional training data.
R5. On JDsearch request counts
Thank you for pointing out the counting error. JDsearch [4] reports approximately 9.4 million search requests, compared with 2,574,949 for KuaiSearch. The earlier JDsearch figure referred to its test subset; the overall comparison should use the full request count.
R6. On production-environment bias
These observational logs come from the production serving environment, where exposure and position affect click/purchase feedback. Our personalization benchmark evaluates behavior prediction in this logged setting. Request-level exposure sets and pre-request histories provide context for studying these effects.
R7. On public-release authorization and privacy protection
Thank you for the detailed questions on responsible data release. The platform has authorized data collection, processing, and public research release to external researchers, with internal legal and data-security review. This use is covered by the applicable platform privacy policy; users have been informed and have given consent. Records of minors have been excluded from the public-release data.
Privacy measures include remapped identifiers, age groups, province-level location, relative time indices, and screening and handling of personal information in queries. Follower counts, following counts, and GMV retain their exact values; seller names and product titles retain their original text. We reviewed the proposed release for linkage risks and did not identify direct links to real platform user accounts.
References
[1] Qingyao Ai et al. 2017. Learning a Hierarchical Embedding Model for Personalized Product Search. SIGIR 2017, pp. 645–654.
[2] Yupeng Hou et al. 2026. Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoders. ACL 2026 (Volume 1: Long Papers), pp. 3251–3265.
[3] Chandan K. Reddy et al. 2022. Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Search. arXiv:2206.06588.
[4] Jiongnan Liu et al. 2023. JDsearch: A Personalized Product Search Dataset with Real Queries and Full Interactions. SIGIR 2023, pp. 2945–2952.
Thank you for the additional results and clarifications. I will raise my score to 2. However, I will keep the Ethics Review flag on in case of any potential leakage, such as users putting their addresses, names, card numbers, etc, as search queries.
We thank the reviewer for the comments on scale, personalization, comparative evidence, and responsible data release. We address the empirical and release-related questions separately below.
R1. On Full benchmarks and training-size scaling
We add Full results for seven ranking models and six retrieval methods. Ranking results report AUC, LogLoss, and request-level click NDCG@5 and MRR@5 as mean ± SD over three training runs.
Full ranking
Full retrieval
Click: 17,358 eligible requests
Purchase: 2,623 eligible requests
WuKong training-size scaling
We separately vary the fraction of Full training requests while fixing the validation set, test set, and candidate pool.
Metrics are means over three training runs; M denotes million.
From 10% to 100%, test AUC increases from 0.6233 to 0.6634, while click and purchase NDCG@5 increase from 0.3587 to 0.3885 and from 0.3631 to 0.3735. These results demonstrate the benefit of additional training data for ranking.