acceptodds
Desk rejected at ICLR 2027

KuaiSearch: An E-Commerce Search Dataset with Authentic Queries and Product Texts for Recall, Ranking, and Relevance

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

E-commerce search connects user needs with massive product inventories, yet real-world systems face challenges from ambiguous queries, noisy product texts, and diverse user preferences. Recent advances in large language models (LLMs) offer new opportunities for semantic understanding and intent modeling, but existing e-commerce search datasets remain limited by heuristically constructed queries, popularity-based filtering, anonymized texts, and incomplete coverage of the search pipeline. These limitations hinder realistic evaluation of LLM-based e-commerce search. To address this gap, we introduce KuaiSearch, a large-scale dataset built from real user search interactions on the Kuaishou platform. KuaiSearch preserves authentic queries and natural-language product texts, retains cold-start users and long-tail products, and covers three key stages of the search pipeline: recall, ranking, and relevance judgment. We further provide comprehensive analyses of users, products, and queries, together with benchmark experiments on representative search tasks. Results demonstrate the value of KuaiSearch as a realistic benchmark for e-commerce search research. The code is available at https://anonymous.4open.science/r/KuaiSearch-7230/.

Primary area:
datasets and benchmarks
Keywords:
E-commerce Search, Dataset, Personalized

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_sENROpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors release a large-scale dataset built from real user search interactions on the Kuaishou platform, supporting the discovery of multiple tasks (i.e., recall, ranking, and relevance judgment) in the search of e-commerce domain. Comprehensive analysis of the dataset are also provided.

Critical strengths
  • The scale and richness of dataset could enable novel discovery in the ecommerce search domain.
Critical weaknesses
  • The value proposition of this dataset is not clear. While the paper provides several analysis on the dataset's patterns, distribution, and benchmarking, the broader value of the dataset (esp on "scale") is unclear.

    • Experiments are mostly run on a subset (i.e., KuaiSearch-Lite). There is no scaling experiment showcase the value of the large-scale dataset which is claimed as one major contribution.
    • The claimed positioning/superiority of KuaiSearch to other datasets are not supported by experiments. Including more empirical comparison with other datasets, e.g., their impact to model performance and transferrability/generalization capability would be helpful.
    • The claimed contribution on personalization is not backed by experiment.
  • (Minor) Dataset quality and soundness.

    • Potential misclaims: JDsearch actually reports 9.4MM searches, which is greater than the released dataset. The number in Table 1 shows test set size which could be misleading.
    • Potential bias: certain customer behavior data may depend on listing or production environment, and may be biased when generalizing to common sense personalized preference.
Questions for authors
  • Could you report results on the full dataset and a training-size scaling curve?
  • Could you benchmark personalized retrieval and cold-start users, and report variance across seeds?
  • Could you compare the lexical-gap and ambiguity statistics against Amazon-C4 and JDsearch?

On ethics

  • Is there a takedown or opt-out process?
  • Were follower counts and GMV coarsened? What fraction of users can be uniquely identified from (gender, age group, province, follower bucket)?
  • Are queries screened for personal information?
4 comments

We thank the reviewer for the comments on scale, personalization, comparative evidence, and responsible data release. We address the empirical and release-related questions separately below.

R1. On Full benchmarks and training-size scaling

We add Full results for seven ranking models and six retrieval methods. Ranking results report AUC, LogLoss, and request-level click NDCG@5 and MRR@5 as mean ± SD over three training runs.

Full ranking

ModelAUC ↑LogLoss ↓Click NDCG@5 ↑Click MRR@5 ↑
WuKong0.6630 ± 0.00040.1586 ± 0.00020.3876 ± 0.00090.3287 ± 0.0012
DIN0.6626 ± 0.00020.1585 ± 0.00020.3872 ± 0.00050.3279 ± 0.0006
DeepFM0.6625 ± 0.00020.1584 ± 0.00000.3871 ± 0.00010.3279 ± 0.0006
DNN0.6625 ± 0.00020.1590 ± 0.00000.3868 ± 0.00050.3277 ± 0.0007
DCN-v20.6620 ± 0.00050.1586 ± 0.00020.3870 ± 0.00050.3280 ± 0.0003
DCN0.6620 ± 0.00020.1584 ± 0.00010.3859 ± 0.00090.3269 ± 0.0012
Wide & Deep0.6613 ± 0.00010.1584 ± 0.00010.3861 ± 0.00040.3275 ± 0.0006

Full retrieval

Click: 17,358 eligible requests

MethodRecall@10Recall@50MRR@10MRR@50HR@10HR@50
BM250.04970.10900.03260.03610.06990.1476
DocT5Query0.05520.12550.03510.03940.07850.1724
DPR (separate encoders)0.07210.17650.04200.04810.10460.2423
DPR (shared encoder)0.07010.17180.03950.04560.09920.2330
DSI0.06370.12090.04360.04740.09520.1771
LTRGR0.08430.16100.05920.06410.12660.2328

Purchase: 2,623 eligible requests

MethodRecall@10Recall@50MRR@10MRR@50HR@10HR@50
BM250.06150.11340.03380.03620.06250.1151
DocT5Query0.06160.12910.03190.03500.06290.1311
DPR (separate encoders)0.08080.18320.03980.04420.08200.1876
DPR (shared encoder)0.08060.17520.03770.04180.08160.1784
DSI0.08940.18310.04150.04620.09190.1868
LTRGR0.12160.23360.05740.06270.12470.2383

WuKong training-size scaling

We separately vary the fraction of Full training requests while fixing the validation set, test set, and candidate pool.

Training-request fractionTraining requestsTraining impressions (approx.)Test AUCClick NDCG@5Purchase NDCG@5
10%250,1947.95M0.62330.35870.3631
25%625,51119.81M0.64670.37510.3716
50%1,251,03439.66M0.65740.38350.3717
100%2,502,04479.31M0.66340.38850.3735

Metrics are means over three training runs; M denotes million.

From 10% to 100%, test AUC increases from 0.6233 to 0.6634, while click and purchase NDCG@5 increase from 0.3587 to 0.3885 and from 0.3631 to 0.3735. These results demonstrate the benefit of additional training data for ranking.

R2. On personalized retrieval and training-unseen users

User history supplies interest context beyond the current query. We compare query-only DPR with history-augmented DPR on Full, holding the separate-encoder architecture, candidate pool, and training protocol fixed. Click and purchase Recall@10 improve by 23.72% and 37.13% relative to query-only DPR, respectively.

FeedbackModelRecall@10Recall@50MRR@10HR@10
ClickDPR (separate encoders)0.07210.17650.04200.1046
ClickDPR (separate encoders) + history0.08920.19520.05770.1284
PurchaseDPR (separate encoders)0.08080.18320.03980.0820
PurchaseDPR (separate encoders) + history0.11080.22200.05650.1125

User groups: AUC / click NDCG@5

Users are grouped by training-period request count; items are grouped by training exposures n and clicks c. The 0 group contains users with no training-period requests; their available pre-request histories are retained. MostPop scores by n. DNN, DIN, and WuKong outperform both simple baselines in every user group on AUC/NDCG@5 and every item group on AUC.

Training requestsUsersTest requestsMostPopSmoothed CTRDNNDINWuKong
0 (training-unseen)1,9344,7420.5264 / 0.34530.5754 / 0.39800.6566 / 0.40880.6563 / 0.41110.6541 / 0.4132
1–53,6497,8450.5116 / 0.32900.5695 / 0.37670.6561 / 0.38830.6572 / 0.38550.6589 / 0.3902
6–204,48710,4580.5074 / 0.32570.5663 / 0.37000.6627 / 0.37550.6617 / 0.37450.6629 / 0.3736
>203,55611,6250.5019 / 0.33410.5650 / 0.37740.6677 / 0.38680.6695 / 0.38610.6700 / 0.3886

User NDCG@5 ranks the full exposed candidate set and macro-averages over click-positive requests; top-five misses score zero.

Item groups: AUC

Item groupTest itemsMostPopSmoothed CTRDNNDINWuKong
n = 0154,5510.50000.50000.65470.65680.6597
n > 0, c = 0255,2500.48680.51320.64630.64490.6473
1 ≤ c ≤ 5175,8900.43200.58420.65610.65630.6566
6 ≤ c ≤ 2037,0980.40370.61600.66900.66930.6685
c > 205,9760.40600.68340.72310.72300.7233

The three-run statistics for the seven ranking models appear in R1.

R3. On lexical-gap and ambiguity comparisons

We add a cross-dataset comparison of query length and lexical mismatch. Compared with JDsearch, KuaiSearch has lower mean lexical coverage (0.561 vs. 0.723) and a higher zero-overlap rate (14.9% vs. 8.2%), illustrating lexical mismatch in short queries. Amazon-C4 has substantially longer queries, representing a different query setting.

DatasetMean query length (tokens)Mean lexical coverageZero-overlap rate
KuaiSearch3.310.56114.9%
JDsearch2.760.7238.2%
Amazon-C444.650.08012.3%

We further use clicked-item entropy to characterize behavioral dispersion for the same query. KuaiSearch has higher mean and median entropy (3.85 and 3.90) than JDsearch (3.28 and 3.45), corresponding to more dispersed clicked-item distributions. This statistic is not applicable to Amazon-C4's single-target setting.

DatasetMean clicked-item entropyMedian clicked-item entropy
KuaiSearch3.853.90
JDsearch3.283.45
Amazon-C4N/AN/A

R4. On dataset value

KuaiSearch combines readable queries and product content with pre-request user histories, exposure sets, and observed click/purchase feedback. This combination supports studying semantic matching together with behavioral personalization, complemented by a separate human-annotated relevance component.

DatasetExisting characteristicsWhat KuaiSearch adds
Amazon (Electronics, 5-core) [1]Category-derived queries; 5-core filteringReal search requests; low-frequency users and items retained
Amazon-C4 [2]Generated complex queries and readable product textReal queries and user-linked behavioral context
ESCI [3]Real queries, readable product text, and human relevance labelsUser histories, request-level exposures, and click/purchase feedback
JDsearch [4]Real requests and user behavior; anonymized textReadable queries and product content for language-model-based modeling

The history comparison in R2 demonstrates a concrete use of this context: personalized DPR improves retrieval for both clicks and purchases. The scaling experiment in R1 further shows the value of additional training data.

R5. On JDsearch request counts

Thank you for pointing out the counting error. JDsearch [4] reports approximately 9.4 million search requests, compared with 2,574,949 for KuaiSearch. The earlier JDsearch figure referred to its test subset; the overall comparison should use the full request count.

R6. On production-environment bias

These observational logs come from the production serving environment, where exposure and position affect click/purchase feedback. Our personalization benchmark evaluates behavior prediction in this logged setting. Request-level exposure sets and pre-request histories provide context for studying these effects.

R7. On public-release authorization and privacy protection

Thank you for the detailed questions on responsible data release. The platform has authorized data collection, processing, and public research release to external researchers, with internal legal and data-security review. This use is covered by the applicable platform privacy policy; users have been informed and have given consent. Records of minors have been excluded from the public-release data.

Privacy measures include remapped identifiers, age groups, province-level location, relative time indices, and screening and handling of personal information in queries. Follower counts, following counts, and GMV retain their exact values; seller names and product titles retain their original text. We reviewed the proposed release for linkage risks and did not identify direct links to real platform user accounts.

References

[1] Qingyao Ai et al. 2017. Learning a Hierarchical Embedding Model for Personalized Product Search. SIGIR 2017, pp. 645–654.

[2] Yupeng Hou et al. 2026. Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoders. ACL 2026 (Volume 1: Long Papers), pp. 3251–3265.

[3] Chandan K. Reddy et al. 2022. Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Search. arXiv:2206.06588.

[4] Jiongnan Liu et al. 2023. JDsearch: A Personalized Product Search Dataset with Real Queries and Full Interactions. SIGIR 2023, pp. 2945–2952.

Thank you for the additional results and clarifications. I will raise my score to 2. However, I will keep the Ethics Review flag on in case of any potential leakage, such as users putting their addresses, names, card numbers, etc, as search queries.

Reviewer_L3avOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces KuaiSearch, a large-scale e-commerce dataset built from real user search interactions without applying population filtering, unlike many prior studies. Without population filtering, user-item interactions show a clear long-tail distribution.

KuaiSearch consists of three tasks: (a) Recall: retrieve clicked and purchased items given a user query; (b) Ranking: predict user-item interactions given user engagement history; and (c) Relevance: predict the relevance of an item to a given query, where relevance is annotated by experts.

The authors further conduct baseline experiments on a filtered subset, KuaiSearch Lite.

Critical strengths
  1. Custom population filtering is indeed a long-standing issue, as different filtering strategies can favor different methods and hinder universal comparisons. Addressing this issue makes the scope of this paper significant.
Critical weaknesses
  1. As discussed in Appendix F, the long-tail population can substantially influence the overall metrics (e.g., 0.66 Head, 0.62 Tail, and 0.63 Overall for DIN). Given the limited information available for long-tail users/items, it is concerning that simple heuristic baselines, such as popularity-based or domain-specific methods, might outperform more sophisticated approaches.

  2. Computational cost and efficiency are largely undiscussed, despite being crucial when considering the long-tail population. This paper aims to retain the long-tail population to better align with real-world scenarios. However, sophisticated approaches may achieve only marginal performance improvements on long-tail users/items while requiring major computational costs, potentially limiting their practical value.

  3. The authors themselves use the Lite subset throughout the experiments. It could be more straightforward to present Lite version as the main benchmark. Otherwise, future studies may resort to custom subsets due to computation/efficiency constraints, potentially prevent the goal of universal benchmarking.

  4. Given the small performance differences between some methods (e.g., Table 5), I suggest reporting the mean and standard deviation across multiple runs.

Questions for authors
  1. Why does the Relevance task use a random split instead of a chronological split, as in the other tasks?
4 comments

We thank the reviewer for highlighting population filtering, the importance of simple baselines, computational cost, and reproducible benchmarking. We respond to each concern below.

R1. On simple baselines for low-activity users and long-tail items

We directly evaluate MostPop and smoothed CTR alongside DNN, DIN, and WuKong on Full. User groups use training-period request counts; item groups use training exposures n and clicks c. MostPop scores by n.

DNN, DIN, and WuKong outperform both simple baselines in every user group on AUC and click NDCG@5, and in every item group on AUC. For users with 1–5 training requests, WuKong achieves 0.6589 AUC versus 0.5695 for smoothed CTR. For items with 1–5 training clicks, the corresponding AUCs are 0.6566 and 0.5842. These results establish useful learned baselines for the retained low-activity and long-tail populations.

User-group sizes

Training requestsUsersTest requestsImpressionsClick positives
0 (training-unseen)1,9344,742144,1786,420
1–53,6497,845225,2668,559
6–204,48710,458307,01011,923
>203,55611,625330,01212,474
Total13,62634,6701,006,46639,376

User performance: AUC / click NDCG@5

Training requestsMostPopSmoothed CTRDNNDINWuKong
00.5264 / 0.34530.5754 / 0.39800.6566 / 0.40880.6563 / 0.41110.6541 / 0.4132
1–50.5116 / 0.32900.5695 / 0.37670.6561 / 0.38830.6572 / 0.38550.6589 / 0.3902
6–200.5074 / 0.32570.5663 / 0.37000.6627 / 0.37550.6617 / 0.37450.6629 / 0.3736
>200.5019 / 0.33410.5650 / 0.37740.6677 / 0.38680.6695 / 0.38610.6700 / 0.3886

Item groups: n = training exposures; c = training clicks

Item groupTest itemsImpressionsClick positives
n = 0154,551172,9366,003
n > 0, c = 0255,250330,43211,730
1 ≤ c ≤ 5175,890313,36712,866
6 ≤ c ≤ 2037,098133,8655,698
c > 205,97655,8663,079

AUC

Item groupMostPop (n)Smoothed CTRDNNDINWuKong
n = 00.50000.50000.65470.65680.6597
n > 0, c = 00.48680.51320.64630.64490.6473
1 ≤ c ≤ 50.43200.58420.65610.65630.6566
6 ≤ c ≤ 200.40370.61600.66900.66930.6685
c > 200.40600.68340.72310.72300.7233

User NDCG@5 is macro-averaged over click-positive requests using the full exposed candidate set; a top-five miss scores zero.

R2. On model size and the practical value of predictive gains

The seven CTR models compared here have similar parameter counts and use the same training set.

In large-scale production systems, even small improvements in CTR prediction can translate into substantial business value across massive traffic volumes. For example, Wide & Deep [1] reports an offline AUC increase from 0.726 to 0.728 alongside a 3.9% relative improvement in online app acquisition rate (Table 1). This illustrates the practical importance of evaluating small predictive gains in industrial ranking.

R3. On the Lite benchmark and Full retrieval and ranking evaluation

We view Lite as the fixed benchmark for routine comparisons, with Full supporting larger-scale evaluation. Lite uses a predefined user subset and retains its first-month records; Full covers the two-month collection window.

StatisticLiteFull
Users102,086331,930
Search requests555,5532,574,949
Ranking impressions17,800,90481,401,477

Both versions have defined train/validation/test protocols and share 46,422 relevance annotations. Full retrieval and ranking results are provided below.

Full retrieval: @10/@50

Click: 17,358 eligible requests

MethodRecall@10Recall@50MRR@10MRR@50HR@10HR@50
BM250.04970.10900.03260.03610.06990.1476
DocT5Query0.05520.12550.03510.03940.07850.1724
DPR (separate encoders)0.07210.17650.04200.04810.10460.2423
DPR (shared encoder)0.07010.17180.03950.04560.09920.2330
DPR (separate encoders) + history0.08920.19520.05770.06390.12840.2640
DSI0.06370.12090.04360.04740.09520.1771
LTRGR0.08430.16100.05920.06410.12660.2328

Purchase: 2,623 eligible requests

MethodRecall@10Recall@50MRR@10MRR@50HR@10HR@50
BM250.06150.11340.03380.03620.06250.1151
DocT5Query0.06160.12910.03190.03500.06290.1311
DPR (separate encoders)0.08080.18320.03980.04420.08200.1876
DPR (shared encoder)0.08060.17520.03770.04180.08160.1784
DPR (separate encoders) + history0.11080.22200.05650.06140.11250.2249
DSI0.08940.18310.04150.04620.09190.1868
LTRGR0.12160.23360.05740.06270.12470.2383

Full ranking

The seven Full ranking models are evaluated over three training runs, with mean ± SD reported below.

ModelAUC ↑LogLoss ↓Click NDCG@5 ↑Click MRR@5 ↑
WuKong0.6630 ± 0.00040.1586 ± 0.00020.3876 ± 0.00090.3287 ± 0.0012
DIN0.6626 ± 0.00020.1585 ± 0.00020.3872 ± 0.00050.3279 ± 0.0006
DeepFM0.6625 ± 0.00020.1584 ± 0.00000.3871 ± 0.00010.3279 ± 0.0006
DNN0.6625 ± 0.00020.1590 ± 0.00000.3868 ± 0.00050.3277 ± 0.0007
DCN-v20.6620 ± 0.00050.1586 ± 0.00020.3870 ± 0.00050.3280 ± 0.0003
DCN0.6620 ± 0.00020.1584 ± 0.00010.3859 ± 0.00090.3269 ± 0.0012
Wide & Deep0.6613 ± 0.00010.1584 ± 0.00010.3861 ± 0.00040.3275 ± 0.0006

On Full, DIN achieves slightly higher mean AUC, click NDCG@5, and click MRR@5, together with lower mean LogLoss, than DNN across three training runs. These results illustrate the value of the attention-based DIN model for click prediction and ranking in this setting. The table below reports model differences, 95% confidence intervals, and p-values.

ComparisonMetricDifference95% CIp
DIN − DNNAUC+0.0001[−0.0004, +0.0006]0.68
WuKong − DINClick NDCG@5+0.0004[−0.0014, +0.0021]0.58

R4. On relevance data collection and random splitting

Retrieval and ranking use chronological splits of behavioral logs. The relevance data were manually collected and annotated and contain no timestamps, so we use a random 80:10:10 split by query–item pair to evaluate semantic relevance on held-out pairs. Training and testing contain no duplicate query–item pairs, and checkpoints are selected by minimum validation loss.

References

[1] Heng-Tze Cheng et al. 2016. Wide & Deep Learning for Recommender Systems. DLRS 2016, pp. 7–10.

I appreciate the authors' response and additional experiments. I would like to clarify W1 and follow up on R1.

Many existing datasets filter out cold-start users and long-tail items because they provide limited interaction signals for behavior-based models.

This raises a question for KuaiSearch that not applying such filtering: Do improvements on KuaiSearch reflect better behavioral modeling, or simply better semantic relevance matching / popularity estimation?

I believe the user/item subgroup analysis in R1 should be central to the paper, as the current evaluation does not clearly distinguish these factors.

For example, WuKong outperforms DIN in AUC across all seen-user groups, but not on unseen cold-start users, while DIN achieves higher overall AUC in Table 5. This suggests that the long-tail population may dominate the metrics, and whether this is desirable depends on the intended research focus.

Overall, my concerns regarding W2 and W4 have been largely addressed. However, W1 and W3 remain major concerns, and I'm not convinced the paper is ready in its current form.

Thank you for clarifying the remaining concerns. We address the sources of predictive gains, population-specific evaluation, and the fixed benchmark below.

On text and history-sequence modeling

We agree that the contributions of different information sources should be examined explicitly. We therefore evaluate complete DIN alongside versions without history-sequence modeling or text features on Full. All AUC and LogLoss values below are means over three training runs.

ConfigurationAUC ↑LogLoss ↓
Complete DIN0.66260.1585
Without history-sequence modeling0.66090.1586
Without text features0.64650.1597

Removing text reduces AUC by 0.0161, while removing history-sequence modeling reduces it by 0.0017. Text features produce the larger observed difference, and sequence modeling provides additional predictive information within this DIN setting.

The existing popularity-baseline comparison provides complementary evidence: DIN exceeds MostPop and smoothed CTR in every reported user and item group. For example, on items with 1–5 training clicks, DIN achieves 0.6563 AUC versus 0.5842 for smoothed CTR. Together, these comparisons characterize the value of text and sequence modeling alongside the simple popularity baselines.

On subgroup evaluation and the intended research focus

The original Table 5 reports Lite results, whereas the additional subgroup analysis uses Full. We align the overall and subgroup ablations here on Full. The previously reported changes in model ordering across user groups reinforce the need for population-specific evaluation.

A training-unseen user has no search requests in the training split, but may still have pre-request interaction history. Each request provides up to 20 preceding clicked items and up to 20 preceding purchased items; these histories can predate the behavioral-log collection window.

User groups: AUC

User training requestsTest impressionsImpression shareComplete DINWithout sequence modelingWithout text
0 (training-unseen)144,17814.33%0.65630.65810.6432
1–5225,26622.38%0.65720.65300.6371
6–20307,01030.50%0.66170.66240.6473
>20330,01232.79%0.66950.66640.6515

Sequence modeling increases AUC by 0.0042 and 0.0031 in the 1–5 and >20 request groups, respectively, while the complete model has lower AUC than the version without sequence modeling in the training-unseen and 6–20 groups. Text features improve AUC in all four user groups. These results show that the value of each information source varies across populations.

Item groups: AUC

Here, n and c denote training-period item impressions and clicks, respectively.

Item groupTest impressionsImpression shareComplete DINWithout sequence modelingWithout text
n=0172,93617.18%0.65680.65360.6346
n>0, c=0330,43232.83%0.64490.64450.6323
1≤c≤5313,36731.14%0.65630.65490.6420
6≤c≤20133,86513.30%0.66930.66870.6527
c>2055,8665.55%0.72300.72260.6973

Sequence modeling improves AUC in all five item groups, with the largest increase, 0.0032, on training-unexposed items. The first three low-feedback groups account for 81.15% of test impressions, making their performance important to understanding this benchmark.

Our intended research focus is the joint use of content and behavioral context across different levels of interaction sparsity. Preserving these groups allows their respective benefits and limitations to be measured directly. We will make the user/item subgroup analyses a central part of the main evaluation, reporting performance on the original test distribution alongside subgroup results and population shares.

On Lite as the fixed default benchmark

We agree with the suggestion to make the predefined Lite version the default benchmark for routine comparisons, with Full serving as the larger-scale extension. Lite uses a fixed user subset and first-month records, without additional minimum activity or item-popularity thresholds.

Standard comparisons will use the fixed membership, time window, train/validation/test partitions, candidate pool, preprocessing, and evaluation protocol of the designated version. This gives future studies a common reference, while Full supports evaluation at larger scale and coverage.

Reviewer_pt7WOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces KuaiSearch, a large-scale e-commerce search dataset comprising approximately 330K users and 18.61M items, with real search queries, product raw text, impressions, clicks, and purchase records, enabling both semantic matching and personalization research. The dataset retains long-tail items and users with varying history lengths, provides a Lite version, and covers three tasks: retrieval, ranking, and relevance.

Critical strengths
  1. The data combination has practical research value. With approximately 330K users and 18.61M items, combined with real queries, product raw text, impressions, clicks, and purchase records, the dataset supports both semantic matching and personalization research. This multi-task data resource that combines behavioral sequences with natural-language product text is uncommon in the e-commerce search domain and has tangible reference and reuse value for future research.

  2. The construction effort and quality control are relatively comprehensive. The dataset retains long-tail items and users with varying history lengths, provides a Lite version, and covers three tasks: retrieval, ranking, and relevance. The appendix provides three independent annotators, a reported Fleiss' κ = 0.89 for annotation consistency, feature ablation, and long-tail analysis, indicating that the authors have done substantive work in annotation quality control and data validation.

Critical weaknesses
  1. The experiments do not sufficiently validate the claimed search ranking and fine-grained relevance capabilities. Section 5.2 limits the ranking evaluation to click prediction and reports only an overall ROC-AUC, which cannot directly demonstrate whether the model can rank more suitable items higher within the same search session; ranking metrics such as NDCG computed per query or session are missing. Section 5.3 compresses the four-level relevance labels into a binary "grade 3 vs. all other grades" classification, without testing the model's ability to distinguish among relevance levels 0/1/2/3. The current evaluation coverage is weaker than the paper's contribution claims, making it insufficient to support its positioning as a reliable search evaluation benchmark.

  2. The data splitting and model selection protocols are insufficiently described, undermining benchmark credibility. Retrieval and ranking use the last day for testing and the remaining data for training, but the validation set, hyperparameter tuning, and checkpoint selection rules are not specified; the relevance task only states that 10% of samples are held out for testing, without clarifying whether the split is grouped by query, how duplicate query–item pairs are handled, or whether there is train/test overlap. Moreover, the comparison with key existing datasets (e.g., Amazon Shopping Queries/ESCI) is inadequate—the latter already provides customer queries, product text, and human-annotated relevance labels, serving as an important reference for assessing KuaiSearch's incremental value, yet the paper does not provide a precise comparison.

Questions for authors
  1. For the ranking task, could you supplement per-query or per-session metrics such as NDCG@K to directly measure the model's ability to rank more suitable items higher within the same search?
  2. For the relevance task, could you report four-level (0/1/2/3) classification or multi-level ranking results, rather than only a compressed binary classification?
  3. Please clarify the validation set split, hyperparameter tuning, and checkpoint selection rules. Is the 10% test split for the relevance task grouped by query? Are there duplicate query–item pairs across the training and test sets?
  4. In Table 5, DIN (AUC 0.6262) and DNN (AUC 0.6258) differ by only 0.0004. Could you provide standard deviations or confidence intervals to support the conclusion that attention-based interest modeling is advantageous?
  5. Could you provide a detailed comparison with the Amazon Shopping Queries/ESCI dataset to more precisely demonstrate KuaiSearch's incremental value in the combination of "real queries + natural-language product content + behavioral history"?
2 comments

We thank the reviewer for the constructive comments on evaluation coverage, data splitting, statistical support, and the comparison with ESCI. Our responses are provided point by point.

R1. On request-level ranking metrics

We supplement the Full ranking benchmark with request-level click and purchase NDCG@5 and MRR@5. The table reports mean ± SD over three training runs. We rank each request's complete exposure set and macro-average over 17,358 click-positive or 2,623 purchase-positive requests; a top-five miss scores zero.

ModelAUC ↑LogLoss ↓Click NDCG@5 ↑Click MRR@5 ↑Purchase NDCG@5 ↑Purchase MRR@5 ↑
WuKong0.6630 ± 0.00040.1586 ± 0.00020.3876 ± 0.00090.3287 ± 0.00120.3746 ± 0.00170.3126 ± 0.0015
DIN0.6626 ± 0.00020.1585 ± 0.00020.3872 ± 0.00050.3279 ± 0.00060.3761 ± 0.00310.3140 ± 0.0030
DeepFM0.6625 ± 0.00020.1584 ± 0.00000.3871 ± 0.00010.3279 ± 0.00060.3757 ± 0.00280.3131 ± 0.0025
DNN0.6625 ± 0.00020.1590 ± 0.00000.3868 ± 0.00050.3277 ± 0.00070.3764 ± 0.00110.3146 ± 0.0008
DCN-v20.6620 ± 0.00050.1586 ± 0.00020.3870 ± 0.00050.3280 ± 0.00030.3786 ± 0.00040.3161 ± 0.0009
DCN0.6620 ± 0.00020.1584 ± 0.00010.3859 ± 0.00090.3269 ± 0.00120.3782 ± 0.00080.3156 ± 0.0012
Wide & Deep0.6613 ± 0.00010.1584 ± 0.00010.3861 ± 0.00040.3275 ± 0.00060.3757 ± 0.00120.3133 ± 0.0013

R2. On four-level relevance evaluation

The release retains all four relevance labels. The original binary task detects full relevance: grade 3 is positive and grades 0–2 are negative. We now add direct four-class evaluation using the same 80:10:10 pair-level split. XLM-R achieves the highest Macro-F1 (60.31%) and accuracy (62.72%) among the reported settings.

ModelFormulationF1 (0)F1 (1)F1 (2)F1 (3)Macro-F1Accuracy
Qwen3-0.6BGenerative20.2116.9527.6152.2129.2539.52
Qwen3-1.7BGenerative42.0849.6767.2360.2654.8158.63
Qwen3-0.6BDiscriminative46.3550.8768.9359.8756.5159.96
Qwen3-1.7BDiscriminative57.0743.4869.5060.5157.6460.80
XLM-RDiscriminative55.3052.7370.9662.2560.3162.72

All values are percentages; F1 (0)–F1 (3) correspond to the four original labels.

R3. On data splits and checkpoint selection

For both versions, retrieval and ranking use the final day for testing, the preceding day for validation, and earlier records for training. Histories and statistical features use only pre-request information.

Relevance uses an 80:10:10 split by query–item pair, rather than by query. Training and testing contain no duplicate query–item pairs. Checkpoints for all tasks are selected by minimum validation loss.

R4. On attention-based interest modeling and the DIN–DNN comparison

On Full, DIN achieves slightly higher mean AUC, click NDCG@5, and click MRR@5, together with lower mean LogLoss, than DNN across three training runs. These results illustrate the value of the attention-based DIN model for click prediction and ranking in this setting. The table reports mean ± SD, DIN–DNN differences, 95% confidence intervals, and p-values.

MetricDIN (mean ± SD)DNN (mean ± SD)Difference (DIN − DNN)95% CIp
AUC ↑0.6626 ± 0.00020.6625 ± 0.0002+0.0001[−0.0004, +0.0006]0.68
LogLoss ↓0.1585 ± 0.00020.1590 ± 0.0000−0.0005[−0.0009, −0.0001]0.039
Click NDCG@5 ↑0.3872 ± 0.00050.3868 ± 0.0005+0.0004[−0.0008, +0.0016]0.42
Click MRR@5 ↑0.3279 ± 0.00060.3277 ± 0.0007+0.0002[−0.0013, +0.0016]0.76

R5. On dataset comparisons and the value of personalization

Thank you for highlighting Amazon Shopping Queries / ESCI [1], which we will cite. ESCI provides real queries, readable product content, and human relevance labels for non-personalized search. The comparison is shown below.

Table 1. Comparison of dataset scale, user context, and task support.

DatasetUsersItems / candidate poolRequestsText and query sourcePopularity filteringPersonalization informationTasks
Amazon (Electronics, 5-core) [2]192,40363,0013,221Readable; category-derived queriesYes: 5-coreYesPersonalized product retrieval
Amazon-C4 [3]20,8381,058,41721,223Readable; ChatGPT-generated queriesPopularity cutoff not specifiedNoComplex-query product retrieval
ESCI [1]Not released1,367,72927,643Readable; real queriesPopularity cutoff not specifiedNoRanking; relevance classification; adapted retrieval
JDsearch [4]173,83112,872,636≈9,400,000Anonymized term IDs; real queriesNo minimum popularity/history-length cutoffYesPersonalized ranking of exposed candidates
KuaiSearch-Lite102,0866,634,118555,553Readable; real queriesNo minimum popularity/history-length cutoffYesRetrieval; ranking; relevance judgment
KuaiSearch331,93018,605,5822,574,949Readable; real queriesNo minimum popularity/history-length cutoffYesRetrieval; ranking; relevance judgment

External benchmark sizes follow the evaluation settings in JDsearch Table 3 [4] and BLaIR Table 3 [3]; Amazon-C4 users denote review authors.

KuaiSearch combines real queries, readable product content, user histories, and request-level click/purchase feedback. It adds real requests relative to Amazon/Amazon-C4, personalization information relative to ESCI, and readable text relative to JDsearch. Retaining sparse users and low-frequency items supports semantic matching and personalized retrieval and ranking.

We evaluate this behavioral context on Full by comparing query-only DPR with DPR augmented by pre-request user history, holding the separate-encoder architecture, candidate pool, and training protocol fixed. Click Recall@10 improves by 23.72%, and purchase Recall@10 by 37.13%.

FeedbackModelRecall@10Recall@50MRR@10HR@10
ClickDPR (separate encoders)0.07210.17650.04200.1046
ClickDPR (separate encoders) + history0.08920.19520.05770.1284
PurchaseDPR (separate encoders)0.08080.18320.03980.0820
PurchaseDPR (separate encoders) + history0.11080.22200.05650.1125

These improvements show that user history provides useful predictive information beyond the query text, demonstrating the practical value of behavioral context for personalized retrieval. The separate human-annotated relevance component also supports complementary research on semantic relevance and behavioral preferences.

References

[1] Chandan K. Reddy et al. 2022. Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Search. arXiv:2206.06588.

[2] Qingyao Ai et al. 2017. Learning a Hierarchical Embedding Model for Personalized Product Search. SIGIR 2017, pp. 645–654.

[3] Yupeng Hou et al. 2026. Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoders. ACL 2026 (Volume 1: Long Papers), pp. 3251–3265.

[4] Jiongnan Liu et al. 2023. JDsearch: A Personalized Product Search Dataset with Real Queries and Full Interactions. SIGIR 2023, pp. 2945–2952.

Reviewer_pewEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • This paper Introduces KuaiSearch, a large-scale e-commerce search dataset collected from real Kuaishou user interactions.
  • Supports three search tasks: retrieval, ranking, and relevance judgment, with authentic queries, product metadata, and user behavior histories.
  • KuaiSearch-Lite which is a smaller version is also provided for efficient experimentation, along with dataset analysis and benchmarks covering lexical, dense, generative, and LLM-based methods.
Critical strengths
  • This paper provides a large-scale, realistic dataset with authentic user queries, natural-language product texts, and exposure/click/purchase signals.
  • Dataset is based on real unaltered customer queries and interactions on a large corporation e-commerce search website from China.
  • Supports modeling multiple search tasks retrieval, such as ranking, and graded relevance judgment within a common commercial domain.
  • Preserves sparse-history users and long-tail products, unlike several existing benchmarks.
  • Provides a potentially valuable resource for personalized search and LLM-based e-commerce research.
  • Definitely would help benchmarking models and methods on this dataset especially for researchers and practitioners interested in building search systems in Chinese language.
Critical weaknesses

Insufficient dataset characterization and comparison

  • Important benchmarks such as Amazon ESCI seems to be missing from comparison and is not cited either.
  • Two-month collection window limits analysis of longer-term seasonality and preference evolution.
  • The claimed integration of retrieval, ranking, and relevance is insufficiently demonstrated, particularly the linkage of relevance annotations to other components.
  • Moreover, this dataset is mostly limited for building search systems in Chinese language.
  • Table 1 can also include cold-start coverage, long-tail preservation, user history, and task support should be compared explicitly.
  • Tables 1–2 and Appendix C lack detailed distributional comparisons between KuaiSearch-Lite and the full dataset.

Limited experimental validation and overgeneralized conclusions

  • All experimental results are based on "Lite" version of the dataset; results on the full dataset are missing.Moreover the Lite version is not even highlighted in the abstract or introduction.
  • Ranking evaluation relies exclusively on ROC-AUC, without query-level NDCG@K, MRR@K, or purchase-oriented metrics.
  • Small performance differences are used to draw conclusions about architectural superiority without variance or significance estimates.
  • Relevance evaluation collapses four graded labels into binary classification, discarding valuable supervision.
  • Qwen vs Llama comparison and conclusions do not adequately account for differences in Chinese-language capabilities and training distributions.
  • Cold-start, long-tail, and exposure-bias analyses remain limited despite being central to the paper's motivation.
Questions for authors
  1. Why is Amazon ESCI missing from Table 1? Explicitly compare cold-start coverage, long-tail preservation, user histories, and supported search tasks across datasets in the same table.
  2. How representative is KuaiSearch-Lite? Can the authors report component sizes, distributional differences, and performance comparisons for both versions?
  3. How do two months of interactions capture preference evolution? Are weekly patterns or seasonal effects analyzed?
  4. What are the recommended train/validation/test splits for both versions? How is temporal leakage prevented, and can relevance labels be linked to retrieval and ranking records?
  5. Why is ROC-AUC the only metric in Tables 5, 7, and 9? Please provide query-level NDCG@K, MRR@K, and confidence intervals?
  6. Model comparisons: Are differences between DIN and simpler architectures statistically significant? Could unequal tuning or language-specific model capabilities explain the Qwen-versus-Llama results? Would another Chinese-capable LLM baseline help?
  7. Why convert four-level relevance judgments into binary labels? Can ordinal classification or graded ranking metrics better demonstrate the dataset's utility?
  8. What explains the head–tail performance gaps in Table 9? Can results be broken down by user-history length and item-popularity buckets, with sample counts?
  9. What is the broader research value beyond Chinese e-commerce search? Why was multimodal product information, particularly images, not considered for inclusion?
3 comments

We thank the reviewer for recognizing the value of the resource and for the detailed comments on characterization, evaluation, and interpretation. We address the questions individually below.

R1. On ESCI citation and dataset comparisons

Thank you for highlighting Amazon Shopping Queries / ESCI [1], which we will cite. ESCI provides real queries, readable product content, and human relevance labels for non-personalized search. The comparison is shown below.

Table 1. Comparison of dataset scale, user context, and task support.

DatasetUsersItems / candidate poolRequestsText and query sourcePopularity filteringPersonalization informationTasks
Amazon (Electronics, 5-core) [2]192,40363,0013,221Readable; category-derived queriesYes: 5-coreYesPersonalized product retrieval
Amazon-C4 [3]20,8381,058,41721,223Readable; ChatGPT-generated queriesPopularity cutoff not specifiedNoComplex-query product retrieval
ESCI [1]Not released1,367,72927,643Readable; real queriesPopularity cutoff not specifiedNoRanking; relevance classification; adapted retrieval
JDsearch [4]173,83112,872,636≈9,400,000Anonymized term IDs; real queriesNo minimum popularity/history-length cutoffYesPersonalized ranking of exposed candidates
KuaiSearch-Lite102,0866,634,118555,553Readable; real queriesNo minimum popularity/history-length cutoffYesRetrieval; ranking; relevance judgment
KuaiSearch331,93018,605,5822,574,949Readable; real queriesNo minimum popularity/history-length cutoffYesRetrieval; ranking; relevance judgment

External benchmark sizes follow the evaluation settings in JDsearch Table 3 [4] and BLaIR Table 3 [3]; Amazon-C4 users denote review authors.

KuaiSearch combines real queries, readable product content, user histories, and request-level click/purchase feedback. It adds real requests relative to Amazon/Amazon-C4, personalization information relative to ESCI, and readable text relative to JDsearch. Retaining sparse users and low-frequency items supports semantic matching and personalized retrieval and ranking.

R2. On the Full/Lite comparison and Full evaluation

Lite randomly samples 102,086 Full users and retains their first-month records. Both versions share 46,422 relevance annotations. Their sizes and distributions are summarized below.

StatisticFullLite
Users331,930102,086
Catalog items18,605,5826,634,118
Search requests2,574,949555,553
Ranking impressions81,401,47717,800,904
CTR3.769%3.833%
CVR7.380%7.464%
Mean query length (characters)6.74696.7486
Mean requests per user7.765.44
Zero-click items / unique exposed items90.32%91.62%
One-click items / clicked items69.44%78.15%

The query-length means are close. Lite covers the first month, with fewer observed requests per user and a higher one-click item share.

We also report Full retrieval with personalized DPR. Adding pre-request history improves click Recall@10 by 23.72% and purchase Recall@10 by 37.13% relative to query-only DPR.

Full retrieval: @10/@50

Click: 17,358 eligible requests

MethodRecall@10Recall@50MRR@10MRR@50HR@10HR@50
BM250.04970.10900.03260.03610.06990.1476
DocT5Query0.05520.12550.03510.03940.07850.1724
DPR (separate encoders)0.07210.17650.04200.04810.10460.2423
DPR (shared encoder)0.07010.17180.03950.04560.09920.2330
DPR (separate encoders) + history0.08920.19520.05770.06390.12840.2640
DSI0.06370.12090.04360.04740.09520.1771
LTRGR0.08430.16100.05920.06410.12660.2328

Purchase: 2,623 eligible requests

MethodRecall@10Recall@50MRR@10MRR@50HR@10HR@50
BM250.06150.11340.03380.03620.06250.1151
DocT5Query0.06160.12910.03190.03500.06290.1311
DPR (separate encoders)0.08080.18320.03980.04420.08200.1876
DPR (shared encoder)0.08060.17520.03770.04180.08160.1784
DPR (separate encoders) + history0.11080.22200.05650.06140.11250.2249
DSI0.08940.18310.04150.04620.09190.1868
LTRGR0.12160.23360.05740.06270.12470.2383

R3. On interaction depth and preference evolution

Statistics over the complete Full dataset cover 331,930 users and show substantial groups with repeated searches and positive item interactions.

Count thresholdUsers meeting search-request thresholdShareUsers meeting positive-item-interaction thresholdShare
>1063,72519.2%73,90322.3%
≥1071,59121.6%81,51224.6%
>2025,5667.7%34,33410.3%
≥2027,5978.3%36,64211.0%

Each session additionally includes up to the 20 most recent clicked items and the 20 most recent purchased items. These histories can precede the two-month collection window. Together with repeated interactions and the existing recent-versus-earlier interest analysis, they support research on personalized preferences and sequential interest changes.

R4. On data splits, leakage prevention, and task linkage

Both versions use the final day for testing, the preceding day for validation, and earlier records for training; checkpoints minimize validation loss. Histories and statistical features use pre-request information. Relevance uses an 80:10:10 query–item-pair split with no duplicate training/test pairs.

Retrieval and ranking share session IDs. Relevance is a separate annotation component without stable item/session-ID mappings to the behavior logs.

R5. On ranking metrics and confidence intervals

The Full results below report mean ± SD over three runs. Request-level NDCG/MRR rank the full exposed candidate set; top-five misses score zero. Click and purchase metrics average over 17,358 and 2,623 positive-feedback requests, respectively.

ModelAUC ↑LogLoss ↓Click NDCG@5 ↑Click MRR@5 ↑Purchase NDCG@5 ↑Purchase MRR@5 ↑
WuKong0.6630 ± 0.00040.1586 ± 0.00020.3876 ± 0.00090.3287 ± 0.00120.3746 ± 0.00170.3126 ± 0.0015
DIN0.6626 ± 0.00020.1585 ± 0.00020.3872 ± 0.00050.3279 ± 0.00060.3761 ± 0.00310.3140 ± 0.0030
DeepFM0.6625 ± 0.00020.1584 ± 0.00000.3871 ± 0.00010.3279 ± 0.00060.3757 ± 0.00280.3131 ± 0.0025
DNN0.6625 ± 0.00020.1590 ± 0.00000.3868 ± 0.00050.3277 ± 0.00070.3764 ± 0.00110.3146 ± 0.0008
DCN-v20.6620 ± 0.00050.1586 ± 0.00020.3870 ± 0.00050.3280 ± 0.00030.3786 ± 0.00040.3161 ± 0.0009
DCN0.6620 ± 0.00020.1584 ± 0.00010.3859 ± 0.00090.3269 ± 0.00120.3782 ± 0.00080.3156 ± 0.0012
Wide & Deep0.6613 ± 0.00010.1584 ± 0.00010.3861 ± 0.00040.3275 ± 0.00060.3757 ± 0.00120.3133 ± 0.0013

Pairwise differences over three seeds

ComparisonMetricDifference95% CIp
DIN − DNNAUC+0.0001[−0.0004, +0.0006]0.68
WuKong − DINClick NDCG@5+0.0004[−0.0014, +0.0021]0.58

R6. On CTR model differences and the Qwen/Llama comparison

Small AUC gaps also appear in established CTR work: DCN-V2, Table 6 [5] reports 0.8098 for DNN and 0.8099 for DeepFM. In industrial recommendation, gains on the order of 0.001 can accompany substantial online benefits; Wide & Deep, Table 1 [6] reports a 0.002 AUC increase alongside a 3.9% relative gain in app acquisition rate.

On Full, the DIN–DNN AUC difference is 0.0001 (p = 0.68), indicating similar performance. Confidence intervals are provided in R5; additional Welch comparisons follow.

MetricBetter resultComparatorDirectionp
AUCWuKongDCN-v2Higher is better0.047
AUCWuKongDCNHigher is better0.033
AUCWuKongWide & DeepHigher is better0.015
LoglossDCNDNNLower is better0.009

We will narrow the Qwen/Llama comparison to the tested models and configurations, considering differences in Chinese capability and pretraining.

R7. On four-level relevance evaluation

The binary benchmark detects full relevance: grade 3 is positive and grades 0–2 are negative. We add four-class evaluation with the same 80:10:10 split. XLM-R achieves the highest Macro-F1 (60.31%) and accuracy (62.72%) among the reported settings.

ModelFormulationF1 (0)F1 (1)F1 (2)F1 (3)Macro-F1Accuracy
Qwen3-0.6BGenerative20.2116.9527.6152.2129.2539.52
Qwen3-1.7BGenerative42.0849.6767.2360.2654.8158.63
Qwen3-0.6BDiscriminative46.3550.8768.9359.8756.5159.96
Qwen3-1.7BDiscriminative57.0743.4869.5060.5157.6460.80
XLM-RDiscriminative55.3052.7370.9662.2560.3162.72

All values are percentages; F1 (0)–F1 (3) correspond to the four original labels.

R8. On user activity and item popularity groups

Users are grouped by training-period request count, and items by training exposures n and clicks c. MostPop scores by n; smoothed CTR is the second simple baseline. DNN, DIN, and WuKong outperform both baselines across all user groups on AUC/NDCG@5 and all item groups on AUC.

User-group sizes

Training requestsUsersTest requestsImpressionsClick positives
0 (training-unseen)1,9344,742144,1786,420
1–53,6497,845225,2668,559
6–204,48710,458307,01011,923
>203,55611,625330,01212,474
Total13,62634,6701,006,46639,376

User performance: AUC / click NDCG@5

Training requestsMostPopSmoothed CTRDNNDINWuKong
00.5264 / 0.34530.5754 / 0.39800.6566 / 0.40880.6563 / 0.41110.6541 / 0.4132
1–50.5116 / 0.32900.5695 / 0.37670.6561 / 0.38830.6572 / 0.38550.6589 / 0.3902
6–200.5074 / 0.32570.5663 / 0.37000.6627 / 0.37550.6617 / 0.37450.6629 / 0.3736
>200.5019 / 0.33410.5650 / 0.37740.6677 / 0.38680.6695 / 0.38610.6700 / 0.3886

Item groups: n = training exposures; c = training clicks

Item groupTest itemsImpressionsClick positives
n = 0154,551172,9366,003
n > 0, c = 0255,250330,43211,730
1 ≤ c ≤ 5175,890313,36712,866
6 ≤ c ≤ 2037,098133,8655,698
c > 205,97655,8663,079

AUC

Item groupMostPop (n)Smoothed CTRDNNDINWuKong
n = 00.50000.50000.65470.65680.6597
n > 0, c = 00.48680.51320.64630.64490.6473
1 ≤ c ≤ 50.43200.58420.65610.65630.6566
6 ≤ c ≤ 200.40370.61600.66900.66930.6685
c > 200.40600.68340.72310.72300.7233

R9. On research value and multimodal extensions

Real short queries, readable product text, and user histories provide a common setting for semantic matching, personalization, and learning from sparse feedback. The current release focuses on text and behavior; images and multimodal information are directions for extension.

References

[1] Chandan K. Reddy et al. 2022. Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Search. arXiv:2206.06588.

[2] Qingyao Ai et al. 2017. Learning a Hierarchical Embedding Model for Personalized Product Search. SIGIR 2017, pp. 645–654.

[3] Yupeng Hou et al. 2026. Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoders. ACL 2026 (Volume 1: Long Papers), pp. 3251–3265.

[4] Jiongnan Liu et al. 2023. JDsearch: A Personalized Product Search Dataset with Real Queries and Full Interactions. SIGIR 2023, pp. 2945–2952.

[5] Ruoxi Wang et al. 2021. DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems. The Web Conference 2021, pp. 1785–1797.

[6] Heng-Tze Cheng et al. 2016. Wide & Deep Learning for Recommender Systems. DLRS 2016, pp. 7–10.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.