- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
FIRM builds a shared fine-grained item-relation memory for LLM-based reranking. An Explorer proposes substitute/complement relations for selected item pairs. A Manager merges and splits those proposals into a relation tree. A Verifier checks assignments and revises definitions. At inference, relations among history items are summarized into a short user memory, and relations between history items and candidates are retrieved for the LLM ranker. Candidates come from SASRec (20 items). If the ground-truth item is missing, it replaces the lowest-ranked candidate — this happens for 83.5–88.7% of test users. On Sports / Home / Grocery, FIRM improves NDCG@10 over the strongest baseline by up to 10.03% on Grocery. A memory built once with Qwen2.5-7B can be reused by other reranking LLMs.
Critical strengths
- The motivating example is good. Regular vs decaf beans and small vs bulk packs are both "substitutes," but the comparison criteria differ. Storing a shared typed relation with a definition is a more sensible object than regenerating substitute/complement lists per user.
- The experimental package is careful and easy to inspect. Prompts, thresholds, and baseline adaptations are documented. The main table uses paired user-level tests. Ablations remove Manager, Verifier, user memory, and fine-grained labels. Transfer across Qwen sizes and to Llama-3.1-8B / Mistral-7B is reported, including cells where MemRec or AgentCF wins. Offline construction is 3.4–6.5 hours (cheaper than AgentCF); online reranking uses two LLM calls.
- Table 10 (coverage split) is the most informative analysis. Gains concentrate on users for whom some relation is actually retrieved. Users with no relation line look close to Zero-shot, and are slightly worse on Home/Grocery. That is exactly the kind of check a memory paper should include.
Critical weaknesses
- The evaluation is conditional on the target already sitting in a 20-item list. The target is inserted for 88.6% / 88.7% / 83.5% of test users. Since all methods share candidates, the comparison is fair as a reranking study. It does not tell us whether FIRM helps the pipeline a practitioner would actually run, where most of these targets were never retrieved in the first place. The "up to 10.03%" abstract number is a gain on that inserted list.
- The hierarchical part — which is the novel piece — moves the metric only a little. Removing both Manager and Verifier drops NDCG@10 by 3.2%, 2.0%, and 3.8%. Removing Verifier alone changes Home NDCG@10 by about 0.03%. There is no flat-clustering / length-matched control that feeds the Explorer's raw strings to the reranker under the same token budget. Extra relation text in the prompt can help without the tree being necessary.
- Relation quality is only measured indirectly via NDCG. There is no human (or independent judge) sample of pair→type assignments, and no real subtree example in the main text. The Verifier also leaves known-bad assignments in place: 676 / 2,651 / 1,136 pairs were judged misfit and kept because no revised proposal was produced (~6–10% of assigned pairs). A fluent but partly wrong taxonomy can still change a prompt.
- The large subgroup gains are not cleanly causal. On Home, users with relations on both sides already have Zero-shot NDCG@10 = 0.3205 vs 0.2847 overall, and FIRM's gain on that cell is +0.2015. Those users are easier before any relation text is added. A shuffled-relation control on the same users is missing. The global "remove relations" ablation does not answer this.
- Coverage is limited, and the relation inventory is narrower than the introduction suggests. History–history relations exist for only 27.1% / 31.5% / 49.5% of test users. The Explorer only emits Sub / Com / None, and most assigned pairs are substitutes. The paper does not analyze what the discarded None pairs actually contain.
Questions for authors
- Evidence: what happens if the target is not inserted? Please split users whose target SASRec already retrieved from users who needed insertion. This is the result that would most change my score.
- Evidence: against a control that passes the Explorer's raw (category, name, sentence), truncated to the same token budget, with no Manager/Verifier, how much gain remains? The current ablation still changes what text the ranker sees.
- Clarification: on a sample of assigned pairs, what fraction would a second judge say match the stored definition? How should the kept misfit pairs be counted?
3 comments
Response to Reviewer 7Jj4 (Part 2/2)
This continues our response in Part 1.
W3 Q3 Relation quality and retained misfits
1 Independent evaluation of relation quality
We sampled 200 pairs per domain from all final assignments, including retained misfits. Two external LLMs (GPT-6 Astra and Claude Fable 5.2) judged whether each pair matched its stored relation definition, using three labels: match, mismatch, and insufficient information. Excluding insufficient-information judgments, definition fit rates were 75.6–82.2%, showing that most evaluated pairs matched their stored definitions. Cohen's κ across all 200 pairs and three labels was .832 / .698 / .733 on Sports / Home / Grocery.
2 Counting and handling unresolved misfits
Uncorrected misfits are counted as unresolved, separately from assignments that passed verification. They remain in the final memory and may be retrieved for recommendation, so they were included in the sampling population for the external evaluation above.
| Processing status | Sports | Home | Grocery |
|---|---|---|---|
| Unique pairs retained without a revision proposal | 676 | 2,651 | 1,136 |
| Of these, explicit misfits remaining unresolved | 633 | 2,381 | 997 |
Table 9 counts unique pairs across all rounds. The first row includes failed responses, indeterminate judgments, and cases later judged valid; we separately report explicit misfits that remained unresolved.
3 Examples from the constructed relation hierarchy
The table shows selected branches of the relation hierarchy constructed on Home. Pair counts include assignments to each type and its descendants.
| Depth | Relation type | Stored definition (verbatim) | Pairs |
|---|---|---|---|
| 1 | size_difference | Items differ in physical size, allowing for substitution based on specific requirements and space needs, relevant to various sub-types including bedding, storage, and display dimensions. | 1,437 |
| 2 | dimensions | Items differ in size, allowing for substitution based on spatial needs, covering sub-types like use_occasion, capacity, and form. | 45 |
| 3 | capacity | Items vary in capacity, such as 6-quart vs 4.5-quart. | 1 |
| 2 | room_dimensions | Items differ in size to fit specific room dimensions or traffic flow. | 11 |
| 1 | functionality | Items offer similar core functions but differ in specific features or design, suitable for the same tasks. | 68 |
| 2 | specific_cooking_functions | Items offer a range of cooking functions, allowing buyers to choose based on specific cooking needs. | 10 |
| 2 | food_preparation_tasks | Items are designed for specific food preparation tasks, allowing buyers to choose based on the specific task. | 11 |
The path size_difference → dimensions → capacity refines a shared size-based criterion into dimensions and capacity. The 6-foot and 5-foot folding table pair is assigned to dimensions, making dimensional differences available as a comparison criterion.
W4 Shuffle control on the same user subgroup
We applied the “Shuffle” control to the same user subgroup with retrieved history–history and history–candidate relations (Table 10). “Shuffle” preserves candidates and item-pair edges, jointly shuffles relation names and definitions, and regenerates user memories.
| Domain | Users | FIRM | Shuffle | Difference |
|---|---|---|---|---|
| Sports | 2,151 | .2895 | .2813 | +.0082 |
| Home | 2,087 | .5220 | .4911 | +.0309 |
| Grocery | 2,417 | .4759 | .4564 | +.0195 |
NDCG@10 gains over “Shuffle” were .0082–.0309. This demonstrates that preserving relation semantics enables FIRM to rerank more effectively even with the same users and edges.
W5 Relation coverage and the meaning of None
FIRM targets substitute and complement relations, and the current memory is dominated by substitutes. Our main experiments evaluate all test users, including those with no retrieved relations. We sampled 200 pairs per domain labeled None by the Explorer. The same two external judges classified each pair as no relation identified, outside Sub/Com, valid Sub/Com, or insufficient information. The table reports judgment percentages as Astra / Fable and Cohen's κ across these four categories.
| Judgment category or metric | Sports | Home | Grocery |
|---|---|---|---|
| No relation identified | 55.5 / 48.0 | 83.5 / 84.0 | 79.0 / 80.0 |
| Outside Sub/Com | 35.0 / 44.5 | 5.0 / 9.5 | 5.5 / 7.0 |
| Valid Sub/Com | 9.5 / 7.5 | 10.0 / 5.5 | 14.5 / 12.0 |
| Insufficient information | 0.0 / 0.0 | 1.5 / 1.0 | 1.0 / 1.0 |
| Cohen's κ (4 categories) | .807 | .757 | .726 |
The None samples included relations outside Sub/Com, while 5.5–14.5% were judged valid Sub/Com relations. Thus, None does not imply that a pair has no semantic relation, and some substitute or complement relations may have been missed.
Follow-up on Our Responses
Dear Reviewer 7Jj4,
Thank you again for your careful assessment of our evaluation protocol and relation quality. We would appreciate your thoughts on the additional experiments and analyses provided in our responses. Please let us know if any concerns remain or if further clarification would be helpful.
Kind regards,
The Authors
Response to Reviewer 7Jj4 (Part 1/2)
Thank you for carefully examining the evaluation protocol and relation quality, and for proposing specific control experiments.
We will incorporate the additional experiments and analyses into the revised manuscript.
W1 Q1 Reranking by target retrieval status
We compared NDCG@10 between users whose targets were retrieved by SASRec (without insertion) and users who required insertion (evaluated with insertion). The groups contain 1,241 / 1,418 / 1,211 and 9,632 / 11,139 / 6,128 users on Sports / Home / Grocery, respectively. Values for the insertion-required group were estimated from the overall results in Table 1, the scores for target-retrieved users, and the two group sizes.
For users whose targets were retrieved, FIRM achieved the highest NDCG@10 in all three domains, demonstrating superior reranking on retrieved candidates (up to 8.03% over AgentCF on Grocery). For users who required insertion, FIRM ranked highest on Home and Grocery and performed comparably to MemRec on Sports.
W2 Q2 Raw and Flat under a matched relation token cap
We compared FIRM with “Raw” and “Flat” under a matched relation token cap. “Raw” directly uses the Explorer's relation categories, names, and explanations. “Flat” merges similar relations within each category from the same Explorer output and applies the same Verifier as FIRM, without constructing a hierarchy. “Raw” and “Flat” relation blocks were capped at the token count of each user's FIRM relation block, and user memories were generated from each condition's relations.
FIRM outperformed “Raw” by .0070–.0130 and “Flat” by .0051–.0083 in NDCG@10. This demonstrates that FIRM's relational memory is more effective for reranking than “Raw” and “Flat” even under the same relation token cap.