- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The starting point of this article is good, as it addresses a problem we genuinely encounter: a LLM-based assistant performs worse if the user has a long history. They propose a benchmark called chronodossier. They create 311 persona dossler, with identities and timelines. Then use seven models to write long doc for these settings. They evaluate in two ways, intrinsic and extrinsic. They have some interesting findings which could be useful for the domain: - full-text as input have better results. - the precision of retrieval is good, but may not help the answering.
Critical strengths
-
The paper has a good idea of treating a persona dossier as a memory representation . Because the ground-truth state is available, the benchmark can evaluate both how the record is written and how it is read, and separate errors made at construction time from errors made at reading time, which most existing memory benchmarks cannot do.
-
The paper has some interesting and useful findings that I mentioned in the summary: - full-text as input have better results. - the precision of retrieval is good, but may not help the answering.
-
The writing is good, the strucutre is good. Dislike AI-generted papers, it's easy for me to follow the idea.
Critical weaknesses
-
As a benchmark and evaluation paper, the authors should present the dataset scale and coverage. I think there are very few lines are talking about this. - In section 3.1, the authors say they have 11 dimensions (ages, region, etc.), but What are the possible values for each dimension, and how many such values are there? For example, how many types of occupations are there, and which countries or regions are covered under the "region" dimension? - The actual distribution of these 311 individuals across various dimensions. for example, age distribution, gender ratio, and occupational distribution. - It is unclear how the 32 replay personas were selected from the 94 GPT-5.1 personas, and whether they are representative of the full cohort.
-
I checked the code for the function of build_section_excerpt and I found the excerpt is a lexical-overlap window centered on the sentence that best matches the query, but this choice is central to interpreting the results and should be stated in the paper. This packaging part also affects the finding. The authors say reading fails even when retrieval succeeds, but I think this is because the limtations of the excerpt, the decrease of performance looks like a mechanical consequence.
-
The settings is not that clear or clean. As I found in the text, the GPT5.1 has many roles: the item writer, the extrinsic evaluator, and one of the evaluated readers. The authors should be careful with the self-preference. I also concerns about the settings of model: Frontier readers run with minimal reasoning (GPT-5.1) or T=0 without extended thinking (Sonnet), with a 120-token output cap. This cap could partly explain the reported losses on some question types. And the settings for open-source models are not in the same with frontier, so the gap bettween them cannot be attributed to reading ability alone.
some sugeestions:
-
I feel that the term "state" needs to be better explained, especially in the abstract and introduction. Because I kept wondering about its meaning while reading, and it wasn't until Section 3 that I finally understood it.
-
The paper uses several counts: 431 item definitions, 520 rated turns, 8,880 total turns, and 208 unique frontier items. The relationship between these units, and in particular where the 208 comes from, is not explained.
Questions for authors
Q1. What values can each of the 11 seed dimensions take, and how many values are there per dimension? For example, how many occupations are included, and which countries or regions does the “region” dimension cover? Could you report the realized distribution of the 311 personas across these dimensions, such as the age distribution, gender ratio, and occupational distribution?
Q2. Why did you choose a lexical-overlap window centered on the best-matching sentence (build_section_excerpt) for constructing excerpts? Any better ways? If sections are longer in longer dossiers, how much of the RETRIEVE decline with length remains once the share of each section covered by the 550-character excerpt is controlled for?
These are two main issues I care about.