- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MemVista, a framework for domain-adaptive user memory evolution in LLM recommendation agents. The method aims to address the problem that compact user memories may discard useful historical information, which cannot be recovered by downstream recommendation policies. MemVista learns domain-shared memory facets from recommendation errors to guide evidence extraction and profile synthesis, while keeping the recommendation prompt fixed. Experiments on twelve domains and six LLM backbones show that MemVista improves recommendation performance over existing methods, and the learned facets can be transferred to other memory constructors.
Critical strengths
- The paper proposes learning domain-shared memory facets from recommendation errors to guide which historical information should be preserved, rather than directly updating user memories.
- The paper evaluates MemVista on twelve domains and six LLM backbones, with controlled experiments on facet learning, domain dependence, and transfer to other memory constructors.
Critical weaknesses
- Questionable baseline performance and limited evaluation settings. The performance of traditional recommendation baselines is concerning. In Table 1, SASRec achieves an average accuracy of only 0.136, which is much lower than BPR (0.191) and LightGCN (0.272). More importantly, SASRec performs worse than random selection in five domains, including Automotive (0.040) and Musical Instruments (0.067), where random selection among ten candidates has an expected accuracy of 0.100. This is difficult to explain, since SASRec is trained on the large background interaction data rather than only the 300 selected training users. These results raise concerns about the implementation and hyperparameter tuning of traditional baselines. Furthermore, the paper does not provide a public code repository, making it difficult to reproduce and verify these unusual results. Moreover, the main experiments evaluate recommendation by selecting one positive item from ten candidates, with nine negative items uniformly sampled from the same domain. This setting may be too simple to reflect practical recommendation tasks. Although the paper includes a harder 50-candidate evaluation, it does not report the traditional recommendation baselines under this setting. Therefore, the experimental results do not provide sufficient evidence for the claimed advantages over existing recommendation methods.
- Insufficient validation of the learned memory facets. The main idea of learning memory facets from recommendation errors is interesting, but the learning process relies heavily on LLM-generated diagnoses and prompt-based revisions. The method assumes that recommendation errors can reveal missing historical information in user memories. However, an incorrect recommendation does not necessarily indicate missing information. It may also result from limitations of the recommendation model or uncertainty in user behavior. The paper evaluates proposed facet sets using validation accuracy, but this does not directly verify whether the generated facets capture the missing information responsible for the recommendation errors.
Questions for authors
- Why does SASRec perform worse than random selection in several domains, and can the authors provide training details and additional results to verify its implementation and hyperparameter tuning?
- How does MemVista perform under more realistic candidate selection settings, such as popularity-based negative sampling or candidates retrieved by traditional recommendation models?
- How can the authors verify that the generated facets capture meaningful historical information missing from recommendation errors, rather than only improving validation accuracy?
10 comments
Response to Reviewer f69w (2/3)
2. Candidate selection and popularity-weighted negative sampling
"Moreover, the main experiments evaluate recommendation by selecting one positive item from ten candidates, with nine negative items uniformly sampled from the same domain. This setting may be too simple to reflect practical recommendation tasks."[W1]
As discussed in our response to Reviewer eRqr, “Understanding: shared facets, individualized memories, and research scope (2/2),” the ten-candidate evaluation format is also used in the closely related studies listed in Table R1. MemVista follows this format to facilitate comparison and extends the evaluation from 2–4 domains to 12 domains. To further examine sensitivity to candidate selection, we additionally evaluate popularity-weighted negative sampling below.
Table R1. Domain coverage, data sources, and main candidate setting.
| Study | Number of domains | Domains and data sources | Main candidate setting |
|---|---|---|---|
| AgentCF (WWW 2024) | 2 | Amazon: CDs and Vinyl; Office Products | 10-candidate evaluation |
| iAgent / i²Agent (Findings of ACL 2025) | 4 | Amazon: Books; Movies and TV. Other sources: Goodreads; Yelp | 10-candidate evaluation |
| MemRec (ACL 2026) | 4 | Amazon: Books; Movies and TV. Other sources: Goodreads; Yelp | 10-candidate evaluation |
| MARS (arXiv, 2026) | 4 | Amazon: Books; Movies and TV. Other sources: Goodreads; Yelp | 10-candidate evaluation |
| MemVista (Ours) | 12 | 10 Amazon Reviews 2023 categories, plus Goodreads and Yelp | 10-candidate evaluation |
"How does MemVista perform under more realistic candidate selection settings, such as popularity-based negative sampling or candidates retrieved by traditional recommendation models?"[Q2]
To examine sensitivity to negative sampling, we replace uniform sampling with popularity-weighted sampling while retaining the true next item and nine negatives (ten candidates in total). Negatives are sampled without replacement within each domain, with weights proportional to (c_i + 1)^0.25, where c_i is the item's interaction count in the original training data.
This weighting increases the probability of sampling popular items. The results are shown below.
| Domain | Uniform sampling | Popularity-weighted sampling |
|---|---|---|
| Goodreads | 0.67 | 0.69 |
| Yelp | 0.53 | 0.57 |
| Movies & TV | 0.61 | 0.59 |
| Music | 0.64 | 0.69 |
| Kindle | 0.65 | 0.72 |
| Baby | 0.31 | 0.34 |
| Automotive | 0.57 | 0.59 |
| Books | 0.59 | 0.57 |
| Games | 0.49 | 0.49 |
| Grocery | 0.40 | 0.41 |
| Health | 0.28 | 0.32 |
| Sports | 0.38 | 0.45 |
| Macro-average | 0.51 | 0.54 |
MemVista's macro-average accuracy increases from 0.51 to 0.54 under popularity-weighted sampling. Accuracy improves in nine domains, remains unchanged in one, and decreases in two. These results support its robustness to this change in the negative-sampling distribution.
3. Dependence on LLM-generated diagnoses and prompt-based revisions
“relies heavily on LLM-generated diagnoses”[W2]
Thank you for raising this point. While LLM-based recommendation inherently relies on the capabilities of the underlying backbone, our framework is designed to be model-agnostic and does not depend on a specific proprietary model. To verify this robustness, we evaluated our method across six different open- and closed-source models in Table 1, demonstrating consistent performance gains across varying model capacities.
“and prompt-based revisions”[W2]
We agree with this observation. Our key extraction and synthesis operations are indeed guided by natural-language prompts, and the evolved domain-shared facets () are stored as textual content.
Rather than relying on brittle prompt engineering, however, this design directly drives our generalization. By keeping all model weights, the memory architecture, and the ranking prompt completely fixed (the ranker never sees any facet text), we optimize what historical distinctions to retain via weight-frozen, error-driven evolution in textual space. Three results confirm that the evolved facets capture generalizable domain knowledge rather than prompt-specific artifacts:
- Outperforming static priors (Table 2): +10.3 pp over hand-crafted Fixed domain-prior facets (0.510 vs. 0.407).
- Cross-constructor transferability (Table 3): Once frozen, the learned facets transfer directly to completely different memory architectures—such as single-call Direct history summary (+12.3 pp) and hierarchical MARS (+11.8 pp)—without any re-tuning or prompt adaptation.
- Cross-backbone consistency (Table 1): The exact same evolution procedure consistently yields the best macro accuracy across all six open- and closed-source backbones.
Response to Reviewer f69w (3/3)
4. Whether learned facets capture meaningful missing information
“However, an incorrect recommendation does not necessarily indicate missing information. It may also result from limitations of the recommendation model or uncertainty in user behavior...The paper evaluates proposed facet sets using validation accuracy, but this does not directly verify whether the generated facets capture the missing information responsible for the recommendation errors.” [W2]
We agree that errors have multiple causes and that validation accuracy alone cannot confirm that an error stems from missing information. MemVista uses errors as feedback for proposing and revising facets, with validation performance determining whether the proposed revisions are retained, rather than assuming that every error reflects missing information.
While inherent LLM limitations and behavioral uncertainties cannot be eliminated, MemVista explicitly accounts for them by requiring extracted evidence to preserve uncertainty (Section 3.2, Appendix C) and evaluating the effectiveness / stability of facet revisions across three random seeds and six backbones (Tables 1–2, Figure 3).
Importantly, our evidence goes beyond validation accuracy: we examine what information the facets preserve and directly test whether the additional facts improve recommendation.
What information is preserved? As discussed in our response to Reviewer eRqr, "Understanding about domain facets," Appendix G analyzes facet composition, evolution, and associated held-out gains across all twelve domains (Figures 5–8). Figure 9 provides conceptual examples illustrating the distinctions these facets help preserve, including routine context, changing needs, and series progress. Appendix I provides concrete facets that retain relationships beyond broad preferences—for example, component roles and purchase order within a vehicle repair project, or dependencies between gaming hardware and accessories.
We apologize for not making these analyses more prominent given the limited space. We placed the detailed facet analyses in the appendix and prioritized the main text for comparisons with existing methods, which we consider the clearest evidence of our approach’s effectiveness.
Does the retained information help? Section 4.6 and Appendix E directly test whether historical facts preserved by MemVista—but omitted from reference profiles—improve recommendation. On 600 test users, 50 per domain, we replace at most two facts in each eligible reference profile with history-supported facts retained only by MemVista. We then repeat recommendation with the same ranker and candidates.
To isolate the value of the information, we standardize the profiles’ wording and match fact count and token length. The exchanged facts come from older history, while recent context and decoding remain fixed. Users are selected independently of prediction outcomes, and annotators verify facts without seeing candidates, targets, predictions, or method identities.
Replacing reference facts with MemVista-retained facts improves all three reference methods (Table 4 in MemVista):
| Reference memory | Before exchange | After exchange | Gain (pp) | Paired 95% interval (pp) |
|---|---|---|---|---|
| Adaptive facets w/o correctness | 0.407 | 0.425 | +1.8 | [0.7, 3.2] |
| Adaptive facets w/o target | 0.417 | 0.433 | +1.7 | [0.5, 2.8] |
| Independent facet proposals | 0.493 | 0.508 | +1.5 | [0.3, 2.7] |
Both columns use the standardized, length-matched profiles. All 600 users remain in the evaluation, including those without an eligible exchange. All three gain intervals are above zero. These results provide direct evidence that learned facets preserve useful historical information: transferring that information improves recommendation without increasing profile length or changing the ranking model.
The reverse exchange supports the same conclusion (§4.6, p. 9, lines 434–437; Appendix E). Replacing these additional historical facts in MemVista profiles reduces recommendation accuracy more than replacing other facts of similar age and length. This further supports the usefulness of the information preserved by the learned facets.
Together, these results support learning useful facets from imperfect error feedback. Validation and independent testing assess whether the proposed revisions remain useful beyond the diagnosed cases. The content analyses and controlled exchanges further show that the resulting profiles retain additional, history-supported facts that contribute to recommendation performance.
We hope these explanations resolve your concern about learning meaningful facets from imperfect error feedback and strengthen your confidence in the effectiveness of our approach for LLM-based recommendation.
Response to Reviewer f69w (1/3)
Dear Reviewer f69w,
Thank you very much for your interest in our work exploring LLM memory in agentic recommendation, and for recognizing the novelty of learning memory facets from recommendation errors.
We also appreciate your detailed and constructive questions regarding baseline reliability and the validation of learned facets. We address these concerns below with additional experiments and clarifications.
1. SASRec performance and reproducibility
Context from prior benchmarks and task difficulty: Low scores for ID-based/traditional baselines on sparse LLM-based reranking benchmarks are also consistent with recent literature. In MARS (arXiv, 2026), SASRec also trails both BPR and LightGCN in three of four domains under 10-candidate evaluation (Table 2 in MARS). Crucially, MARS evaluates on the instruction-conditioned InstructRec benchmark, where each query includes an explicit user-intent request derived from the target item's review. In contrast, our benchmark removes all target-derived intent instructions and evaluates purely history-only next-interaction prediction, making the task substantially harder even on overlapping domains.
Here, we investigated SASRec's performance by separating evaluation cases according to whether the correct item appeared in the training interactions. An item is considered cold-start if it has no interaction records in the training data.
Item cold start is particularly challenging for SASRec, which primarily relies on item IDs and interaction histories. The model needs to rely on content signals such descriptions, categories, brands, and attributes to make judgments. This situation also arises in practice when newly listed items have no training interactions and item metadata can help rank them.
The table below reports the proportion of evaluation cases with a cold-start target item, overall SASRec accuracy, and accuracy on the cold-start and non-cold-start subsets. Domains are ordered by their cold-start proportions.
SASRec exceeds the 10% random-choice accuracy on non-cold-start cases in all twelve domains. In Automotive, 87.0% of evaluation cases have a cold-start target. SASRec achieves 0.6% accuracy on these cases and 26.5% when the target appears in training, yielding 4.0% overall accuracy. In Music, 58.3% of cases have a cold-start target, and accuracy is 1.5% on those cases versus 14.0% when the target appears in training, yielding 6.7% overall accuracy. These results indicate that item cold start contributes substantially to the low overall scores.
In fact, before submission, we had also noticed these low scores and reran SASRec separately to check the evaluation pipeline and verify that the results were reproducible. While we initially withheld the codebase due to standard confidentiality considerations during the review stage, our constructive exchange with the reviewer has encouraged us to share it early. We provide the SASRec reproduction code in the supplementary materials, and will release the full training and facet evolution code upon acceptance. We appreciate your request for more implementation detail, which prompted us to report this breakdown.