acceptodds
Under review as a conference paper at ICLR 2027

CHRONODOSSIER: AUDITING CONSTRUCTION AND TEMPORAL RECOVERABILITY OF LONG-FORM PERSONA MEMORY

Abstract

Long-horizon assistants rely on stored records to remain factually and temporally consistent as circumstances change. Their reliability depends on preserving state when a record is written and recovering the relevant evidence when that record is later queried. Long-form memory benchmarks evaluate records after construction, leaving record-construction failures confounded with evidence-access failures. Persona dossiers provide a controlled setting for separating these stages because they combine persistent attributes with dated, changing states. CHRONODOSSIER links construction and use across 311 synthetic persona dossiers of 1k–10k words produced by seven generators. Each dossier is built from a dated timeline, and every passage remains linked to the life phase it describes. A frozen question set is then answered by five models using either the full dossier or retrieved passages. Restricting each passage writer to source fields assigned to its life phase eliminates all detected reuse of fields assigned to other phases while preserving measured coverage of the intended fields. Compared with the frontier pipelines, the open-source pipelines exhibit model-specific bottlenecks in semantic concentration, answer completion, inference, unsupported-request handling, and retrieval robustness. Downstream evaluation reveals a distinct limitation in evidence use. For two frontier answer models, retrieved-context accuracy falls by 26–34 percentage points from 1k to 10k words. Among explicitly supported questions, time-dependent items trail timeless items by 12 percentage points under retrieval, although the top three retrieved passages contain the correct life phase for every unique frontier item. CHRONODOSSIER evaluates record construction alongside downstream use and traces failures to construction, evidence access, or answer formation.

Primary area:
datasets and benchmarks
Keywords:
language model memory, long-context evaluation, retrieval-augmented generation, temporal reasoning, persona modeling, synthetic data, datasets and benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Ym1VOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The starting point of this article is good, as it addresses a problem we genuinely encounter: a LLM-based assistant performs worse if the user has a long history. They propose a benchmark called chronodossier. They create 311 persona dossler, with identities and timelines. Then use seven models to write long doc for these settings. They evaluate in two ways, intrinsic and extrinsic. They have some interesting findings which could be useful for the domain: - full-text as input have better results. - the precision of retrieval is good, but may not help the answering.

Critical strengths
  1. The paper has a good idea of treating a persona dossier as a memory representation . Because the ground-truth state is available, the benchmark can evaluate both how the record is written and how it is read, and separate errors made at construction time from errors made at reading time, which most existing memory benchmarks cannot do.

  2. The paper has some interesting and useful findings that I mentioned in the summary: - full-text as input have better results. - the precision of retrieval is good, but may not help the answering.

  3. The writing is good, the strucutre is good. Dislike AI-generted papers, it's easy for me to follow the idea.

Critical weaknesses
  1. As a benchmark and evaluation paper, the authors should present the dataset scale and coverage. I think there are very few lines are talking about this. - In section 3.1, the authors say they have 11 dimensions (ages, region, etc.), but What are the possible values ​​for each dimension, and how many such values ​​are there? For example, how many types of occupations are there, and which countries or regions are covered under the "region" dimension? - The actual distribution of these 311 individuals across various dimensions. for example, age distribution, gender ratio, and occupational distribution. - It is unclear how the 32 replay personas were selected from the 94 GPT-5.1 personas, and whether they are representative of the full cohort.

  2. I checked the code for the function of build_section_excerpt and I found the excerpt is a lexical-overlap window centered on the sentence that best matches the query, but this choice is central to interpreting the results and should be stated in the paper. This packaging part also affects the finding. The authors say reading fails even when retrieval succeeds, but I think this is because the limtations of the excerpt, the decrease of performance looks like a mechanical consequence.

  3. The settings is not that clear or clean. As I found in the text, the GPT5.1 has many roles: the item writer, the extrinsic evaluator, and one of the evaluated readers. The authors should be careful with the self-preference. I also concerns about the settings of model: Frontier readers run with minimal reasoning (GPT-5.1) or T=0 without extended thinking (Sonnet), with a 120-token output cap. This cap could partly explain the reported losses on some question types. And the settings for open-source models are not in the same with frontier, so the gap bettween them cannot be attributed to reading ability alone.

some sugeestions:

  1. I feel that the term "state" needs to be better explained, especially in the abstract and introduction. Because I kept wondering about its meaning while reading, and it wasn't until Section 3 that I finally understood it.

  2. The paper uses several counts: 431 item definitions, 520 rated turns, 8,880 total turns, and 208 unique frontier items. The relationship between these units, and in particular where the 208 comes from, is not explained.

Questions for authors

Q1. What values can each of the 11 seed dimensions take, and how many values are there per dimension? For example, how many occupations are included, and which countries or regions does the “region” dimension cover? Could you report the realized distribution of the 311 personas across these dimensions, such as the age distribution, gender ratio, and occupational distribution?

Q2. Why did you choose a lexical-overlap window centered on the best-matching sentence (build_section_excerpt) for constructing excerpts? Any better ways? If sections are longer in longer dossiers, how much of the RETRIEVE decline with length remains once the share of each section covered by the 550-character excerpt is controlled for?

These are two main issues I care about.

Reviewer_QiArOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces ChronoDossier, a benchmark for memory-using LLM assistants. The authors create fictional people with dated life phases and use LLMs to turn these structured profiles into biographies. They then evaluate five models in scripted conversations. The model receives either the full biography or three short passages selected by a retrieval system using the current question. Most questions ask for facts explicitly stated in the biography. Some questions revisit older facts after a long scripted history or ask about a person at a particular point in time. Models generally answer better when they receive the full biography than when they receive three retrieved excerpts. The gap is largest for the longest biographies.

Critical strengths

The topic is important and has clear practical value.

The experiments contain useful observations about retrieval and packaging.

Critical weaknesses

The paper is motivated by user memory, but the benchmark contains no user or user-inspired data. The personas, memories, questions, conversation histories, answers, and most evaluations are generated by models. It is unclear what part of real user behavior this benchmark captures.

User memory is created from interaction. Users mention facts across many messages, change their minds, leave details implicit, and provide information of varying importance. The paper does model nothing from this process.

The paper assumes that a biography is the right form of memory. Real user memories are usually partial facts extracted from specific interactions, with missing context, uncertainty, and corrections. A dossier in this paper is a coherent life narrative generated from complete information - i.e., a biography. The paper never justifies why performance on biographies should reflect or resemble performance on user memories. The generator receives a complete and organized description of a fictional person and turns it into a polished dossier. This removes the difficult and realistic part of the task, which is extracting and updating memories from actual conversations.

Human evaluation is too limited to address this issue. The small human evaluation is about judge agreement and temporal labels. They do not ask whether the personas, memories, questions, or expected answers reflect real users and their expectations.

The main result on dossier length is confounded because each length bucket includes different personas and questions. It cannot separate the effect of length from differences in persona or question difficulty.

Questions for authors

none

Reviewer_mbYmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper looks at long personal dossiers as memory representations of a known latent state. Seven models are used to generate synthetic dossiers of varying lengths from structured timeline seeds, each section being linked to a particular life stage. The evaluation looks at the dossiers themselves (in terms of coverage, cross-phase leakage and collapse) as well as five readers who respond either to the whole dossier or to the retrieved passages. The errors are then ascribed to retrieval, packaging or the formation of the answer. The key result is that accuracy falls when retrieval is used for the longer dossiers, even though retrieval has already found the correct phase.

Critical strengths

• The fact that the latent state is known and that every passage can be traced back to a phase and a seed means that the design enables the writing problems to be separated from the reading problems. I agree with that. Two observations of practical value: it is possible for retrieval to get to the correct phase even if the answer is still wrong and the way in which the excerpts are packaged makes a huge difference to the outcome. • The paper is very transparent with null and negative results, the deficiencies in the configuration all laid out. Tests were performed with both the judge and the planner being substituted.

Critical weaknesses

• GPT-5.1 is doing too many things at once (producing the replays, writing the items, judging the IF items, acting as the tested reader and providing the SFT targets). It's very difficult to escape self-preference and simply swapping the judge has an impact on both the decisions and the absolute scores. We therefore need at least one other model family to produce the dossiers and the items and to check that the conclusions are stable. • The evaluation scale doesn't quite match the abstract. The abstract talks about several hundred dossiers from seven generators, the extrinsic evaluation only makes use of a few dozen GPT-5.1 personas and the other generators only show up in the intrinsic evaluation. The interventions are all run on the same panel or on even smaller panels and on the main panel none of them result in any significant change to the overall accuracy. • The open source results are very much affected by problems with the pipeline. In Table 3 the retrieval was performed without the use of the embedder and in some cases there were fewer than 3 sections per turn. When the retrieval is rerun in the appendix the loss is considerably reduced for all 3 readers and every interval contains zero. Most of the Qwen failures are simply missing answers and the effect of Gemma's intervention is to alter the results when the text after its end marker is removed. A large proportion of what is being measured in these rows is interface behaviour. • Length and persona go hand in hand with eight different personas per length category. Under Full access the frontier reader accuracy drops off dramatically at medium length and then picks up again which is a phenomenon that can't be easily explained on the basis of length alone. The drop-off also varies considerably depending on how the material is packaged as with the longer excerpts it can fall to half or less.

Questions for authors
  1. Would the length and attribution conclusions still stand in the case of a non-GPT model generating the dossiers, the items and the judgments? 2. What personas or types of item are responsible for the dip and then the recovery when we're at the middle values of length? Can we actually use the same personas to compare across the different lengths? 3. Given that there's an open source rerun of the code which includes the embedder, why does Table 3 still have the version that doesn't? 4. In the attribution table a number of the passes have no packet support detected. That would suggest that the keys or the section labels are incomplete in those cases. To what extent would that impact the attribution results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.