- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper tries to come up with a context management methodology for long conversations. The build a method that converts turns into statements and keep a ledger. They show some improvements over some baselines on LongMemEval-S and LoCoMo benchmarks using fairly older models.
Critical strengths
I think this is an important topic to study, and especially could have useful implications in the for personalization and continued conversations.
Critical weaknesses
This is a very poorly written draft which is very difficult to understand. I suspect most of it is actually written by claude code, some examples from just the intro include: L44: Memory construction and retrieval have both improved, but the step that assembles the context has not. L52: It concerns the decision rather than the annotation, which is why it survives every field the write half adds L55: Coverage does not explain these failures: in our own runs the questions the system fails on every sampling draw are no less covered than the ones it answers
Everything is not motivated, for example: why do you use three routes under nomination? A lof of things are unclear: In figure 1, how is attestation getting info that there are two magazines? Since you bulid a candidate pool if the conversation is long enough won't it run into the same context issue?
The paper attributes the gains to the attestation, but attestation is not novel enough in itself. Mem0, A-Mem and others already use it.
I looked at the prompt and its very specific to the task itself and is overfitting.
ITEMS IN FLIGHT | anything the user still has to pick up, collect, return,
drop off, exchange, repair, collect from a person, lend back, borrow, or is
waiting to receive. Write the ITEM and, when stated, the shop/person and the
date, e.g. "The user still needs to return the boots to Zara (bought 2023-02-05)"
or "The user lent her green sweater to her sister".
This kind exists because counting questions ask how many things are in flight
and the answer is lost when only "I own X" statements are extracted: measured,
22.4% of the gold items of counting questions were absent from the store.
Questions for authors
Please look at the weakness.