- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes an evidence-supervised iterative retrieval framework, ADAR, for identifying implicit user preferences from dialogue histories. The method uses a teacher model with access to annotated evidence and reference answers to generate search trajectories, which are filtered based on information leakage, answer correctness, and evidence presence before being used to fine-tune a smaller student model. It also introduces an Abductive Distance Score (ADS) combining lexical coverage and semantic competition to provide feedback during iterative retrieval. Experiments on selected subsets of benchmark datasets demonstrate improvements over several retrieval and memory-based baselines.
Critical strengths
-
The paper addresses an important and practical problem of retrieving implicit user preferences from long-term dialogue histories (when supporting evidence may not be directly related to the current query).
-
The proposed framework provides a reasonable approach to leveraging annotated evidence for learning iterative search behavior through teacher-generated trajectories and supervised fine-tuning of small models.
-
Experiments on three benchmarks demonstrate improvements over several retrieval- and memory-based baselines, with ablation studies supporting the effectiveness of trajectory supervision and online feedback.
Critical weaknesses
-
The evaluation of retrieval effectiveness has more room for improvement. Although the proposed framework primarily aims to improve the retrieval of implicit supporting evidence, the experimental evaluation mainly focuses on downstream answer accuracy. Direct comparisons of evidence retrieval performance would better support the effectiveness of the learned search policy.
-
The quality and effectiveness of teacher-generated search trajectories are not sufficiently validated. Although the authors report trajectory filtering statistics and demonstrate the benefits of student SFT, these results may not verify the contributions of individual filtering strategies and the quality of intermediate search behavior. Additional ablations comparing filtered and unfiltered trajectories, along with qualitative case studies illustrating successful and failed teacher/student search trajectories, would help justify the proposed trajectory supervision pipeline.
-
The comparison and discussion of preference-aware memory construction approaches needs to be enhanced more comprehensively. Although the paper includes several memory-system baselines, the area is evolving rapidly and more recent papers (e.g., Mem-PAL, PGMem) explicitly organize user preferences and supporting information through structured memory representations. Further comparisons or discussions would help clarify the advantages and trade-offs of learning an iterative search policy over constructing preference-aware memories from various perspectives (e.g., retrieval effectiveness, computational efficiency, and adaptation to evolving user preferences).
Questions for authors
Please see weaknesses.
2 comments
Thank you again for your review. We followed up on the points you raised with a direct evaluation of evidence retrieval, an analysis of trajectory filtering and search behavior, and comparisons with preference-aware memory methods.
We’d welcome your feedback on whether our response addresses your concerns.
Thank you for the constructive review. Accuracy and coverage values are percentages.
1. Direct evidence retrieval. Quantitative retrieval evaluation on the personalization benchmark PM shows that ADAR finds more complete support and reaches the first supporting source earlier: coverage improves by 5.02pp over PGMem and MRR@25 rises from 0.59 to 0.66 (Table 1). This directly supports improved evidence search beyond final-answer accuracy.
Methods share a 25-source and visible-token budget. Memory outputs are linked to original sources. MRR uses ordered, deduplicated sources, with misses scored zero. Direct retrieval embeds the original question and returns top25 without reformulation or iteration.
Table 1. PM retrieval.
Two cases show how reformulation bridges the query-to-evidence gap. In pm_test_945, support discusses viewing the same play from front-row versus upstairs seats. Direct retrieval ranks it 100th (cosine 0.1591). ADAR's second-round query “theater performances art experiences” finds row33. In pm_test_1403, support discusses the conductor's and concertmaster's roles. It ranks 103rd (cosine 0.0243). The query “classical music concerts orchestras” finds row17 in round two. Both connect concrete dialogue details to broader interests.
2. Filtering and intermediate behavior. Evidence filtering improves both trace grounding and student performance. Answer-only filtering retains 22.80% of traces without annotated support. Full reduces this to 0% and improves PM by 3.08pp. Removing answer/evidence filtering lowers PM by 5.95pp (Table 2). Thus, answer correctness alone is insufficient to select evidence-supported search demonstrations.
Controls share the 4B initialization, 4,960 traces and optimization budget. All retain anti-leakage/validity checks. The last removes only answer/evidence thresholds. Zero denotes passing the corresponding data check.
Table 2. Selected-trace quality and student accuracy.
Success/failure cases distinguish source localization from integration. For c19f7a0b (arrival time, reference 6:30 pm), the student covers the answer session in five rounds and answers correctly. The teacher fails to cover it fully in four rounds and abstains. For c6853660 (coffee limit: one to two cups), the student reaches the answer session in one round but says “decreased”. The teacher answers “increased to two cups” after three rounds. Search supervision addresses reaching useful history, while the latter case shows why answer correctness is also checked.
3. Preference-aware memory and cost. ADAR's higher online cost comes from iterative search: 4.18 calls per question, versus 1.00 for PGMem/Mem-PAL. This additional computation accompanies higher PM accuracy and support coverage. Memory methods organize preferences before answering, enabling fewer online calls. Construction and later updates incur additional costs outside the online QA figures.
Table 3. PM accuracy and online QA cost per question.
ADAR's online tokens are 4.12/6.69 times those of PGMem/Mem-PAL. The included Native RAG/IRCoT comparisons also show the accuracy–cost trade-off against search baselines. These figures exclude judging and offline memory construction. For changing preferences, memory systems update stored representations, while ADAR searches source history at answer time. We will clarify where these approaches perform adaptation and separate online answering cost from memory construction and maintenance.