- Rating
- 3
- Confidence
- 3
- Informativeness
- 3(+1)
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper proposes QARS, a two-stage post-retrieval context selector for long-term conversational memory. Stage 1 partitions historical LLM responses into positive, uncertain, and discard groups via an incremental semantic-contribution score; Stage 2 trains two binary discriminators (F_keep, F_add) conditioned on a behavior-based LLM fingerprint derived from a diagnostic calibration set. Experiments on ConvoMem, HaluMem, and LongMemEval-S report accuracy gains over RAG and compression baselines with substantially fewer input tokens.
Critical strengths
- The paper offers a clear and well-motivated reframing of conversational memory: user queries provide compact semantic anchors, whereas historical LLM responses are treated as conditionally useful evidence rather than uniformly preserved context.
- The evaluation spans three datasets, turn- and session-level retrieval, six training-visible and two unseen LLMs, cross-dataset transfer, and targeted ablations, providing considerably broader empirical coverage than a single-benchmark efficiency study.
Critical weaknesses
- Although Appendix E.4 varies the upstream retriever, the semantic filtering encoder before Eq. (2) remains fixed to BGE-M3. Experiments with multiple embedding encoders are needed to establish that QARS generalizes across encoders rather than relying on one relevance geometry.
- The paper does not report training or held-out accuracy for and , making label learnability and discriminator reliability difficult to assess. Results over multiple random seeds, with variance and validation metrics, are needed to rule out initialization and sampling effects.
- Table 1 mixes answer accuracy and token count, so it is difficult to determine whether QARS consistently dominates alternatives or merely selects a different operating point. A Pareto-front plot, ideally with tunable operating points and confidence intervals, would make the accuracy-efficiency comparison substantially clearer.
Questions for authors
See weakness
4 comments
Response to Reviewer X8dq - Part (2/2)
Q3: Accuracy-token pareto analysis.
We introduced adjustable thresholds for QARS's discriminator output probabilities. For the baseline methods, we varied the compression ratio of LLMLingua2, the number of extracted sentences in RECOMP-Extractive, and the similarity threshold of EXIT to obtain operating points at different token consumption levels. We plotted Accuracy-Token curves on ConvoMem and LongMemEval using Qwen3.5-9B and Qwen3.5-122B as downstream LLMs. The complete curves are shown in Figure 9 in the appendix of the revised PDF.
We constructed empirical Accuracy-Token frontiers from the evaluated operating points without interpolation. At each token level, the frontier represents the highest accuracy among evaluated points using no more than that number of input tokens; the best baseline frontier takes the highest accuracy across all baselines under the same constraint. Over the common token range, we computed five metrics: QARS AUC, Best Baseline AUC, Mean Frontier Gap, Minimum Frontier Gap, and Frontier Superiority Coverage. Normalized AUC measures overall accuracy along the frontier; Mean Gap and Minimum Gap measure QARS's average and minimum accuracy advantages over the best baseline frontier, respectively; and Coverage measures the proportion of the token range where QARS's frontier achieves higher accuracy.
| Dataset / LLM | QARS AUC | Best Baseline AUC | Mean Frontier Gap | Minimum Frontier Gap | Frontier Superiority Coverage |
|---|---|---|---|---|---|
| ConvoMem / Qwen3.5-9B | 68.47 | 63.95 | +4.52% | +2.7% | 100% |
| ConvoMem / Qwen3.5-122B | 70.28 | 67.19 | +3.09% | +1.0% | 100% |
| LongMemEval / Qwen3.5-9B | 82.78 | 78.20 | +4.58% | +4.4% | 100% |
| LongMemEval / Qwen3.5-122B | 81.46 | 78.99 | +2.47% | +1.2% | 100% |
Across all four dataset-LLM settings, QARS achieves higher normalized AUC than the best baseline frontier, with average accuracy gains of 2.47-4.58%. Moreover, the Minimum Frontier Gap remains positive in all settings, and Frontier Superiority Coverage reaches 100% in every setting. This indicates that at different token consumption levels, QARS's empirical frontier consistently lies above the best baseline frontier, demonstrating a sustained accuracy advantage.
Together with the complete curves in the appendix, these findings demonstrate that QARS's advantages are not merely due to selecting different operating points, but reflect a consistently stronger accuracy-efficiency trade-off across token consumption levels.
Thanks author response and revision. I whould like to maintain my overall postitive score, and improve my Informativeness score and Soundness to 3.
Thank You for Your Positive Feedback
Thank you for your kind feedback and for acknowledging our responses and revisions. We sincerely appreciate your continued positive assessment of our work and the improved Informativeness and Soundness scores.
Response to Reviewer X8dq - Part (1/2)
Thank you for your thoughtful feedback. We have addressed your concerns in detail below.
Following your suggestion, we conducted an ablation study on the Stage 1 semantic filtering encoder. Keeping all other QARS components unchanged, we replaced the original BGE-M3 encoder with MPNet and GTE, respectively. The table below reports results averaged over six downstream LLMs.
Across the three encoders, accuracy varies by at most 1.9% on ConvoMem and 1.0% on LongMemEval, while input token counts differ by at most 25 and 107 tokens, respectively. Although BGE-M3 achieves the highest accuracy, these relatively small variations indicate that QARS maintains stable accuracy and compression efficiency without relying on a specific semantic filtering encoder.
We constructed independent evaluation sets for Fkeep and Fadd using test data from ConvoMem, HaluMem, and LongMemEval. We repeated discriminator training with two additional random seeds, resulting in three seeds in total, and evaluated both discriminators on these held-out sets. The results are reported as mean ± standard deviation in the table below.
Both discriminators demonstrate predictive performance on held-out data, with standard deviations ranging from 0.49% to 1.79% across datasets and random seeds. Although accuracy varies across datasets, these results provide evidence of label learnability and relatively stable discriminator performance across random seeds.