- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces RecoAtlas, a benchmark and RL environment for LLM shopping agents. Each task gives the agent a shopping request and asks it to search a big catalog and return a set of K items, each with a short exaplaination. Ground-truth sets come from real user behavior: items a user interacted with positively within 7 days on Amazon Reviews or H&M, filtered by GPT-5.6 for coherence. GPT-5.6 generates the query from those items. The agent gets three recommendation tools (relevance search, complementarity, de-duplication), each in a “semantic” version and a “behavior-trained” version.
Critical strengths
- The pipeline is complete and reusable: data split, query generation, tools, evaluator and RL interface all fit together, and code and artifacts are released. Others could apply it to new datasets.
- The cross-domain experiment is the convincing evidence that skills learned in RecoAtlas transfer beyond the training domain.
Critical weaknesses
- For the cross-domain experiment, it rests on a single data point. The authors train Qwen3-4B with GRPO for 512 updates on Amazon Musical Instruments and evaluate zero-shot on 406 hidden H&M Clothing queries. SetHit@K rises from 0.82% (base model) to 1.43% with the direct SetHit@K reward and 1.21% with the learned set reward. This is one source–target pair, one run per condition, and no seeds or confidence intervals. At such low absolute values, a 0.4–0.6 pp gap is hard to distinguish from noise.
- RL is front and center in the title, abstract, and contributions, but the experiments only use Qwen3-4B, each setting is run once with no seeds, and the two datasets point in opposite directions. That’s not enough to back a main contribution.
- For a benchmark paper, the key questions are: are the ground-truth sets reliable, does the query leak the answer, and does the metric reflect real utility? The paper doesn’t spend enough space on these.
Questions for authors
See weakness
5 comments
Part II
W2. More RL evidence
We agree that single runs on one backbone could not support the original claim. We now report three seeds on two backbones, evaluated on the full held-out set, and we clarify what the RL contribution is.
Table R2. Held-out SetHit@K (%) after RL, mean ± std over three seeds. Base = untrained model, averaged over the two reward arms.
| Backbone | Domain | Base policy | Direct (SetHit@K) reward | Learned reward |
|---|---|---|---|---|
| Qwen3-4B | Musical Instruments | 6.20 | 10.77 ± 0.46 | 8.76 ± 0.38 |
| Qwen3-4B | H&M | 0.93 | 1.08 ± 0.17 | 1.87 ± 0.17 |
| Gemma-4-E4B-it | Musical Instruments | 1.21 | 8.92 ± 0.75 | 8.08 ± 0.39 |
| Gemma-4-E4B-it | H&M | 0.29 | 1.81 ± 0.11 | 2.27 ± 0.10 |
RL improves mean SetHit@K over the base policy in all 8 settings (the smallest gain, Qwen3-4B on H&M with the direct reward, is within seed variation), and the trained policies transfer across domains (W1). The gains come from better behavior at each stage of the task (all metrics are in the revised RL table, Section 4.5):
- Search. Discovery recall (the share of reference items surfaced by any tool call) rises in 7 of 8 settings, e.g., Qwen3-4B on Musical Instruments: 15.48% → 20.98%.
- Selection. The share of discovered reference items kept in the final report rises in all 8 settings, e.g., Qwen3-4B on Musical Instruments: 40.11% → 51.12%.
- Report completeness. Gemma-4-E4B-it initially produces valid reports only 17–19% of the time; RL raises this to 97–100% under both rewards (Qwen3-4B is already ≥99.88% valid).
This is the RL contribution we will emphasize: RecoAtlas turns logged behavior into an environment where training reliably improves how recommendation agents search, select, and report.
The reward comparison is domain-dependent. On H&M, every learned-reward run beats every direct-reward run on both backbones. On Musical Instruments, direct optimization is better for both backbones. This shows a consistent, domain-dependent pattern. Precisely, GRPO learns from reward differences within a rollout group. When every report in a group is valid but recovers no reference item, the direct reward is constant and gives no learning signal, whereas the dense learned reward can still rank the reports. This matters more when recovery is sparse:
- H&M is a sparser starting point. Base SetHit@K is 4–7× lower than on Musical Instruments.
- The learned reward improves discovery where the direct reward stalls. On H&M, it raises discovery recall on both backbones, while direct training barely moves it.
Revision. We revise the contributions and Sections 1, 4.5, and 5 to present the RL contribution as the environment and its consistent training gains, and to state the reward comparison as domain-dependent.
Part III (Final)
W3. Reference-set reliability, query leakage, and metric utility
We agree these are the key questions for a benchmark, and we answer each with new evidence that the revised paper moves into a dedicated validity section.
Are the reference (ground-truth) sets reliable? A mined 7-day set is kept only if at least 2 of 3 LLM coherence votes find one common goal, a useful role for every item, and no redundancy (Section 3.1). Human annotators confirm the result: on 50 random held-out H&M query–set pairs, a majority judged 49/50 sets to be coherent bundles and 48/50 to satisfy their request (Table R3). The filter is also not specific to GPT-5.6: rebuilding Musical Instruments with Gemini-Flash-3.8 in every construction role gives 77.9% agreement on keep/reject decisions over 3,189 mined sets, and the leaderboard ranking of the four open-weight agents is unchanged (Section 4.1, Appendix K).
Do queries leak the answer? Each query passes deterministic leakage checks and an LLM query–set consistency check. No query received a majority leakage flag (0/50; 3.7% of individual judgments).
Table R3. Human verification of queries and reference sets.
| Question | Yes | Partly | No | Queries with majority "Yes" |
|---|---|---|---|---|
| Does the set satisfy the request? | 86.2% | 13.0% | 0.8% | 48/50 |
| Do the products make sense together as one bundle? | 85.4% | 12.2% | 2.4% | 49/50 |
| Does the request read like a real shopper's request? | 70.7% | 24.8% | 4.5% | 46/50 |
| Does the request leak the exact items? (lower is better) | 3.7% | 0.0% | 96.3% | 0/50 |
Does SetHit@K reflect real utility? Reference sets come from logged positive interactions within a seven-day window and are kept only if their products serve distinct, useful roles toward one goal (Section 3.1). Because the report size is fixed at , recovering more reference items means covering more of these roles, and every redundant product occupies a slot that could have recovered a missing role. To test this with humans, annotators compared two equal-size bundles for each of 50 held-out H&M queries, one with SetHit@K > 0 and one with SetHit@K = 0. They were blind to model identity, SetHit@K, and the reference set, and presentation order was balanced. They chose the bundle that better fits the request, checking coverage of every requested piece, the stated colour, style, and occasion, whether the items work as one outfit, and redundancy. Annotators preferred the bundle with positive SetHit@K in 81.0% of judgments (84.3% for external annotators), and SetHit@K agrees with individual annotators (κ = 0.62) more than annotators agree with each other (mean pairwise κ = 0.50) (Table R4). SetHit@K gives no credit to acceptable alternatives outside the reference set, a limitation shared with behavior-based metrics such as purchase-based Recall@K, which we state explicitly.
Table R4. Human agreement with SetHit@K.
| Agreement with SetHit@K [95% CI] | Cohen's κ with SetHit@K | |
|---|---|---|
| All annotators | 81.0% [74.0, 87.6] | 0.62 |
| Co-authors | 78.1% [69.4, 86.2] | 0.56 |
| External annotators | 84.3% [77.1, 91.1] | 0.69 |
| Majority vote (ties excluded) | 88.6% | — |
| Human–human (mean pairwise κ) | — | 0.50 |
Revision. We add a dedicated benchmark-validity section to the main paper (Section 4.1) covering reference-set reliability, query leakage, and metric validity, with the full annotation protocols in a new appendix.
We are very grateful for the reviewer's time. We hope these additions address the reviewer's concerns, and we would be grateful if the reviewer would consider updating their score.
Part I
We thank the reviewer for the careful reading. We address each weakness with new evidence: (i) three-seed RL results on two backbones (Qwen3-4B and Gemma-4-E4B-it), including cross-domain transfer in both directions, and (ii) two human annotation studies on reference-set reliability, query leakage, and metric validity (seven annotators: four co-authors and three external annotators with no AI/RecSys background), and (iii) a benchmark rebuilt with a different LLM family (Gemini-Flash-3.8). Section and appendix numbers refer to the revised PDF.
W1. Cross-domain transfer rests on a single data point
We agree that one run per condition on 406 queries cannot separate small differences from noise. We added three-seed experiments in both directions, evaluated on all agent_hidden queries of the target domain. We ran this study on Gemma-4-E4B-it because all of its in-domain checkpoints were available; due to storage constraints, not all Qwen3-4B checkpoints were retained after their in-domain evaluation. The original Qwen3-4B run shows the same direction (0.82% → 1.21–1.43% on H&M).
Table R1. Cross-domain transfer: SetHit@K (%) on the target domain, base → transferred model, mean ± std over three seeds.
Revision. We replace the single-run transfer result with these three-seed results in Section 4.5 and add the full tables (validity, discovery, and retention before and after transfer) in Appendix L.