- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
CORAL combines GLiNER2 anchor extraction, LLM-based compression, and retrieval-augmented QA for conversational memory. The reported results suggest that substantial compression can preserve QA performance on a small example. However, the evaluation does not establish the proposed mechanism’s benefit or support the broader empirical conclusions.
Critical strengths
-
The pipeline is simple, interpretable, and accompanied by code and detailed outputs.
-
Reporting actual compression fractions and both QA and lexical preservation metrics is useful.
Critical weaknesses
-
The evaluation is severely insufficient and reuses the tuning data. All experiments use only the first ten questions from one LoCoMo conversation. Compression fraction and compressor size are selected using these same questions, without held-out evaluation. The headline improvement from 50% to 60% represents one additional correct answer. This undermines the empirical foundation of the method.
-
The model-family comparison contains an implementation failure. The compression code disables thinking only for Qwen and applies the short memory budget directly to generation. In the released DeepSeek run log (
experiment3_results.txt), all 100 retrieved snippet occurrences begin with unfinished compression reasoning (“Okay, so I need…”). Consequently, its 0% accuracy measures a pipeline producing truncated reasoning rather than valid compressed memories. -
The central methodological claim is not realized by the design. CORAL lacks explicit anchor prioritization and preservation checks. Anchors only condition the prompt, leaving the compressor to decide what survives and undermining the claimed separation between preservation and compression.
Questions for authors
-
Do the reported gains persist on held-out conversations when compression fraction and model size are selected on a separate development set? Please provide broader evaluation with uncertainty estimates.
-
Can you rerun the DeepSeek comparison with sufficient reasoning capacity and a separate budget for the final compressed memory? How does correcting the truncated-reasoning outputs affect the model-family conclusions?
-
How does CORAL prioritize competing anchors under a tight budget and verify their preservation? What mechanism substantively separates preservation decisions from those already made by the compression LLM?