acceptodds
Under review as a conference paper at ICLR 2027

Human History Helps-But for How Long? Tracking LLM Dialogue Continuation Across Turns

Abstract

Can human response history keep language-model continuations close to recorded human responses-and for how long after new human responses stop being supplied? We introduce a controlled dialogue-replay protocol that fixes one participant’s recorded utterances while a language model generates the target participant’s response at each eligible turn. Five conditions vary the target-speaker history available to the model: no recorded human responses, the first one, two, or five human responses followed by model-generated responses, or all previous human responses. Earlier human responses remain in context after each finite prefix ends, while the current human reference is never exposed. We evaluate five language models across informal conversation (MaiChat), emotional support (ESConv), and persuasion (PersuasionForGood), with over 630,000 model-generated continuations saved before matched-window filtering. MaiChat and ESConv use dialogue history alone, whereas PersuasionForGood additionally includes fixed task context without strategy labels. We measure turn-level semantic distance to recorded human continuations and estimate condition-dependent trajectories over matched dialogue windows using mixed-effects models. Across all 15 model-dataset analyses, full human history has a lower fitted window-averaged semantic distance than the condition without recorded human response history, although its fitted response-step trend remains positive. A five-response human prefix retains a detectable benefit over that condition after model-generated responses enter the history in ESConv and PersuasionForGood. Yet continued human responses provide an additional detectable benefit over that prefix at one or more post-prefix positions for all five PersuasionForGood models and four of five ESConv models. Effects in MaiChat are less precise and more model-dependent. These findings distinguish the carryover of early human responses from the benefit of continually supplying new ones. Our protocol measures fidelity to recorded continuations under fixed interlocutor utterances, not general response quality, task success, or fully interactive dialogue.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language models, multi-turn dialogue continuation, human response history, human-AI behavioral alignment, semantic divergence

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WKHJOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper asks the question whether including human history improves model answer. At certain turns, the model is presented either with the history made of all the human turns, or a mixture of human and machine generated turns (in the same conversation), and the semantic similarity is measured between the ground-truth answer and the newly generated answer. The paper shows that there is only partial

Critical strengths

an interesting problem setup

Critical weaknesses

underpowered metrics (all the results are reported using a metric previously shown to be less correlated with human judgements compared with some other metrics) lack of connection to the real use case (when can we insert ground truth utterances instead of human ones) older methods lack of important related work (the paper claims it is the first study of this type, while the problem has been clearly studied in the past) limited dataset used (the paper does not use the most often used multi-turn datasets) writing clarity

Questions for authors

Why was the semantic similarity metric the only one used? Was there any study into human alignment of that metric against other evaluation metrics?

Why were the experiments not conducted on the newer versions of the models?

What were the exact versions of the models used?

Why were the 3 dataset used, and some more often used multi-turn dataset omitted (MtBench, AlpacaChat, MultiWoz, PersonaChat, MuTual).

Was there any "solvability" study done? What are the guarantees the tasks presented could be solved by the models?

How are differences in the semantic space interpreted? Are they statistically significant, and does that map to any sort of human understanding (e..g is the score of 0.55 suggesting that the answer is not suitable at all, and the score of 0.8 that the answer is paraphrased but keeps the same semantics?)

Why were the later utterances curtailed, is it not more realistic setup to keep last two human utterances instead of the first two?

What is the use case findings of the experiments support? In what situations do we, at inference time, have a "choice" to include the human "ground-truth" turns?

Reviewer_fLLgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how human-written conversation history affects subsequent LLM responses. It introduces a dialogue-replay protocol comparing five conditions, ranging from entirely model-generated response history to full human history, with intermediate conditions providing 1, 2, or 5 initial human responses. Experiments span five LLMs and three human dialogue datasets, using semantic distance to recorded human responses as the evaluation metric. The authors find that longer human history generally improves reference similarity and that benefits can persist after the model begins generating its own history.

Critical strengths
  • The paper investigates an interesting question about how historical responses influence LLM behavior across multiple turns.

  • The experiments are extensive, covering five models and three datasets. The controlled comparisons and statistical analysis are reasonably thorough.

Critical weaknesses
  1. Limited novelty and insufficient engagement with related literature.

The distinction between human-written and model-generated conversational history has been explored in prior work. The main extension here is to examine several finite human-prefix lengths and their effects on subsequent responses. While this is a reasonable experimental question, it is unclear that it provides a substantial new conceptual insight.

The related work section is also relatively thin and does not adequately position the paper within the broader literature on conversation history, context management, and multi-turn degradation. For example, Huang et al. (Do LLMs Benefit From Their Own Words?, COLM 2026) directly investigate how retaining prior assistant responses affects subsequent response quality. A more thorough discussion of related findings and a clearer articulation of the paper's distinct contribution would strengthen the work.

  1. The evaluation design limits the interpretation and significance of the findings.

The primary evaluation measures semantic similarity to a single recorded human response while keeping the other participant's utterances fixed. However, these utterances were originally produced in response to the actual human conversation. Once model-generated responses replace human history, subsequent fixed utterances may no longer be consistent with the generated dialogue. This potentially favors the human-history conditions independently of any improvement in conversational quality.

Furthermore, semantic similarity to one human reference does not necessarily indicate better response quality, coherence, or task success. The paper provides no human evaluation or alternative metrics to establish the practical significance of the observed differences.

Finally, the analysis remains largely descriptive. It does not distinguish potential explanations such as recency, speaker-specific information, or contextual consistency. The primary post-prefix analysis also covers only three or four response positions after the five-response prefix, limiting conclusions about how long the effects persist.

Overall, despite considerable experimental effort, the findings do not provide sufficiently deep insights into why conversational history matters or how it should be managed.

Questions for authors
  • How can the authors distinguish genuine benefits of human history from the greater compatibility of human history with the fixed recorded interlocutor responses?

  • How do the findings relate to existing work on assistant-history retention and multi-turn context management? What fundamentally new insight does the finite-prefix analysis provide?

  • Are the results robust to alternative evaluation metrics or human judgments, and can the authors explain the mechanisms underlying the observed differences?

Reviewer_LiEzOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper explores to the extent that human dialogue history helps AI in dialogue generation tasks. Authors use several dialogue datasets including MaiChat, ES-conv, and PersuasionForGood to isolate the effect that dialogue history has on AI dialogue continuation. For every task and step, authors use a fixed position window of dialogue supplied by humans in the dataset and then ask model to predict the continuation and score the difference between human output and model prediction. They do so by using mixed-effects regression model to observe the trend in human-AI drift. AUthors find that the context helps initially but grows dimininishing returns with more depth.

Critical strengths
  1. Formulation for eq2 and eq3 is easy to follow, and modeling decisions (like why mixed-effects was chosen) is easy to understand. Additionally task formulation is easy to follow (especially thanking figure 1).
  2. Large evaluation scale with 3 different types of dialogue dataset. Authors also included several models in their evaluation suite.
Critical weaknesses
  1. One large concern is novelty. I have read papers in past that model this dialogue divergence [1] by comparing AI chat simulation to human original respondent. However, many of these papers have some prompt robustness strategy. How can authors be sure that their final prompts are optimal for task? Many of these papers also include several evaluation metrics. Authors may want to consider borrowing inspiration from these previous works.

  2. Another concern is the feasibility. As to my understanding, there is usage of semantic distance but it doesn't account for the trajectory of topic in mulyiturn datasets. For example, users can switch conversation topics in the middle of general chatbots. In such case, how is this handled? Are dataset samples manually inspected to remove that (specific to MaiChat).

[1] https://arxiv.org/abs/2409.08330

Questions for authors
  1. Going to Weakness 2, were such "topic switches" handled manually? What was the criteria for exclusion if at all excluded?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.