- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper develops a theoretical framework to explain the performance degradation observed when task information is distributed across multiple conversational turns. The authors formulate conversational interaction as closed-loop inference over a fixed latent user intent, where user messages provide independent evidence regarding the true intent and model-generated responses reinforce the model's current interpretation.
Critical strengths
-
The formalization of feedback in multi-turn interaction is clear. The paper separates user-provided evidence from model-generated history and makes explicit how a model’s current estimate can influence what it later treats as evidence. This is a useful abstraction for studying self-reinforcing conversational errors.
-
The recovery, lock-in, persistent degradation, and intervention analyses fit together into a clear theoretical picture. The paper also distinguishes finite-horizon guarantees from asymptotic claims, particularly in its discussion of snowballing.
Critical weaknesses
-
The real-LLM validation is limited relative to the scope of the theoretical explanation. The experiments use 54 binary-intent instances and two 7B models to measure one-step preference shifts under controlled intent instructions. They do not test whether early errors persist during actual multi-turn rollouts or whether pruning reduces the gap to a matched single-turn baseline. The authors acknowledge this scope in Appendix B, but the central trajectory-level predictions remain unverified on real LLMs (Sec. 7; Appx. H).
-
The mapping from the formal model to actual LLM behavior remains underdeveloped. The analysis assumes specified likelihood families and additive scores with a fixed model-evidence weight, whereas real LLMs' use of history can depend on the context. The empirical probe measures changes in the log-likelihood of candidate intent descriptions; these changes are not calibrated to the theoretical drift quantities. This limits the practical interpretation of the recovery and lock-in thresholds (Secs. 2, 4, and 7).
-
The connection to fixed-information sharding needs clarification. Assumption 2.2 requires uniformly informative user shards after the first turn, so increasing the horizon also supplies continuing evidence. This differs from holding a complete task specification fixed and dividing it across more turns. Appendix B discusses a cumulative-information extension, but the implications of the asymptotic results for this fixed-budget setting remain unclear (Sec. 2; Appx. B).
-
The interpretation of unreliability is narrower than repeated-run consistency on the same instance. Theorem 5.2 concerns the variance of a binary terminal-error indicator over randomness that includes user shards and model responses. It establishes coexistence of persistently correct and incorrect trajectories, but does not directly establish persistent variability across repeated model runs with an identical user-shard sequence. This distinction should be clearer (Secs. 2 and 5).
-
Positioning relative to prior self-delusion analyses is incomplete. Ortega et al. (2021) (https://arxiv.org/abs/2110.10819) already explain how treating a model's own actions as evidence can distort inference about a latent task. A direct comparison would help identify the additional contribution of the present recovery and persistence guarantees and mitigation analysis (Appx. A).
Questions for authors
- Which aspects of the formulation, recovery and lock-in guarantees, and mitigation analysis are new relative to Ortega et al.'s (2021) self-delusion account of treating model-generated actions as evidence?
- Could a small real-LLM multi-turn evaluation test whether early intent errors persist and whether pruning assistant history narrows the gap to a matched single-turn baseline?
- Can the measured user and model preference shifts be related to the drift quantities in Section 4 in a way that supports identifying a recovery or lock-in regime in an actual LLM?
- How should the degradation guarantees be interpreted when a fixed total amount of task information is split across different numbers of turns, given Assumption 2.2's requirement of uniformly informative new shards?
- What supports treating each repetition of a user shard as an equal additive score contribution in the recap/snowball model, and which conclusions would change if the effect of repetition saturates?
- How does Theorem 5.2's variability over the stochastic conversation process relate to variability across repeated model runs with exactly the same user-shard sequence?
4 comments
W2 / Q3: What the model captures and how probes relate to drift
The model captures feedback from the current intent estimate into later inference. User messages arise from the true intent; model responses arise from the current estimate. Reusing those responses lets the current estimate shape subsequent evidence. A fixed still permits different feedback contributions: varies with the response and candidate intents, and the current estimate changes which responses are generated. Appx. B discusses context-dependent weights and response families. The practical insight is to distinguish changes to history weight, feedback timing, and user-evidence accumulation. The intervention analysis and the pruning results above connect this abstraction to actual history handling. Text deletion remains a concrete context intervention, rather than a measurement of .
What the new probes and continuations show. On the original 54 tasks, we follow each of ten wrong-seeded responses with a Weak, Medium, or Strong clarification. Here is the ambiguous initial request, a stored response generated under the wrong intent, the clarification at strength , and the probe margin. The joint margin and residual are:
| Model | Clarification | Joint margin | Residual comparison |
|---|---|---|---|
| Qwen2.5-7B | Weak | 1.805 | 0.395 |
| Qwen2.5-7B | Medium | 2.039 | 0.349 |
| Qwen2.5-7B | Strong | 1.962 | −0.105 |
| Mistral-7B-v0.3 | Weak | 1.248 | −0.141 |
| Mistral-7B-v0.3 | Medium | 1.581 | −0.050 |
| Mistral-7B-v0.3 | Strong | 1.195 | −0.358 |
Wrong-seeded histories alone shift the margin toward the alternative intent (mean −1.154 / −0.929), yet after clarification every joint margin is positive and is small and not consistently signed. In generated continuations, keeping the wrong-seeded answer does not detectably change error after the same clarification (Appx. H.4; details in our response to Reviewer Aomi, W1).
The drift quantities are hard to measure in an LLM, but the theory locates the boundary and explains the observed regimes. Our probe reads out which intent the model prefers; the theorems depend instead on how strongly each message separates the intents ( for user messages, and for model responses), which would require estimating divergences between intent-conditioned message distributions. We therefore do not place a deployed LLM in a regime numerically. What the theory provides is the boundary itself: both regimes exist, they are separated by versus (exactly at in the binary case), and the outcome moves with the history weight and with how informative each user turn is. This explains the contrast in our new experiments: a single rule-stating clarification is highly informative and the model recovers, whereas sharded tasks deliver partial evidence per turn while responses accumulate, and errors persist (W1/Q2).
Where these quantities are known, simulations confirm the boundary. In the formal model, wrong states persist when and errors vanish when (Appx. G.2). Moving only the snowball cap across cuts wrong final intents from 1,833 to 8 of 5,000 (Appx. G.6).
W4 / Q6: Conversation-level variability versus fixed-script repetitions
Thm. 5.2 links the coexistence of persistently correct and persistently incorrect trajectories, each with positive probability, to non-vanishing terminal-error variance over the conversation process, which includes randomness in both user messages and model responses. For terminal error and the realized user script , . A total-variance bound therefore does not imply a conditional bound for every script. Thm. 5.2 concerns variability across conversations, and Sec. 7.2 and Appx. I.3 keep the two notions separate.
We measure fixed-script variability separately as the fraction of run pairs that disagree in correctness, using the five runs per problem:
| Model | CONCAT | Four-turn KEEP | Four-turn PRUNE |
|---|---|---|---|
| Qwen2.5-7B | 0.080 | 0.160 | 0.074 |
| Mistral-7B-v0.3 | 0.184 | 0.104 | 0.128 |
Qwen's disagreement is higher under four-turn KEEP (0.160) than under CONCAT (0.080) or PRUNE (0.074). Mistral's lower KEEP disagreement accompanies a 91.8% error rate: repeated runs can agree because they repeatedly fail.
W3 / Q4: Continuing information versus a fixed budget
Assumption 2.2 deliberately isolates feedback-induced failure when user information is sufficient for consistent inference. At each horizon, both protocols receive the same realized user messages, and the theory identifies when feedback can still sustain an error. Splitting a fixed amount of user information across more turns supplies no growing budget. In the additive model, regrouping also preserves the terminal user-only score, so Thm. 5.1 does not transfer directly.
Theoretical extension: the lock-in bound of Thm. 4.1 carries over to a fixed budget. Splitting conditionally i.i.d. evidence atoms into equal groups weakens each user turn to , while each retained response still reinforces a wrong estimate by at least . Because the -group conversation matches the first turns of an infinite-horizon one, Thm. 4.1's bound lower-bounds terminal error whenever , while user-only error does not depend on . This is a sufficient condition, not a monotonicity result (Cor. E.1, Appx. E.3).
Evidence. In the matched-task evaluation (W1/Q2), four-turn KEEP error exceeds two-turn KEEP by 20.2 pp for Qwen and 26.4 pp for Mistral; grouping changes disclosure timing and the number of intermediate answers together. In the formal model, we fix 32 realized evidence atoms in each of 5,000 trajectories and regroup them into informative turns ():
| Number of informative groups | 1 | 2 | 4 | 8 | 16 | 32 |
|---|---|---|---|---|---|---|
| User-only error | 0.199 | 0.199 | 0.199 | 0.199 | 0.199 | 0.199 |
| SHARDED error | 0.243 | 0.300 | 0.436 | 0.654 | 0.772 | 0.799 |
User-only error is constant across groupings, while closed-loop error rises (Appx. G.5).
Q5: Why count repetitions equally, and what happens if their effect weakens?
Equal additive contributions follow from how the score treats context, and the reported gains of these protocols support them. Let and let be the model-evidence term of response . Our score adds one log-likelihood term for every user message present in the context. Recap and snowball add no new facts; they put copies of earlier messages back into the context, and the same rule counts each copy as one more term. Under snowball, is present in every later turn, so the margin at turn is with ; a recap adds one copy of each earlier message at a single turn. Laban et al. (2026) report that both protocols reduce multi-turn degradation, which a model in which repetition carries no weight could not explain. Equal weights are the simplest such choice. The analysis depends only on how fast the total user weight grows, so unequal weights enter through in the same way.
Saturation can be modeled by a counting window (Appx. B). If each message counts at most times, the weight becomes . Only recap and snowball are affected; the original protocol, pruning, and deferral involve no repetition.
- Recap: it still adds one copy unless , in which case it has no effect.
- Snowball: the user part of the margin now grows linearly, by about per turn, rather than quadratically, so the every- guarantee becomes a threshold. The argument of Thm. F.2 (snowballing) extends directly: the user part is still a sum of independent terms, now with weights at most , and the model part is the same martingale. The same concentration steps then apply with replaced by , which points to recovery when . Unlimited repetition is the limit .
We simulated capped snowball in the formal model. The cap includes each message's first appearance, and results are reported after 1,000 turns:
| Protocol | Maximum times each user message counts | Wrong final intents |
|---|---|---|
| No repetition (baseline) | 1 | 1,833 / 5,000 |
| Capped snowball | 2 | 8 / 5,000 |
| Capped snowball | 4 | 0 / 5,000 |
| Original snowball | No limit | 0 / 5,000 |
Thank the author for the detailed response, they have resolved some of my concerns. I will raise my rating.
We thank the reviewer for the detailed assessment and for recognizing the explicit separation between user evidence and model-generated history and the connection among recovery, persistence, and intervention results. If further clarification on any point would help your evaluation, please let us know.
W5 / Q1: Relation to Ortega et al. (2021)
We thank the reviewer for this reference, which we now discuss in the related work (Appx. A). Ortega et al. provide a closely related causal account of self-generated evidence. They distinguish an expert's actions, which reveal information about a latent task, from an agent's own actions, which must be treated as interventions. Under this treatment, the action likelihood drops out of the belief update, so P(θ∣do(a))=P(θ). They discuss interactive language models explicitly and propose training with factual and counterfactual teaching signals.
Both works build on the Bayesian view of in-context learning, which explains the shared structure. Our model follows statistical accounts of in-context learning as inference of a latent task from context (Xie et al., 2022; Zhang et al., 2024), and Ortega et al. adopt the same view of language models. Within this common view, the two treatments of self-generated evidence that they contrast become the endpoints of our family:
Our analysis covers the whole range 0<λ≤1 between these endpoints.
What we add. Ortega et al. explain why conditioning on one's own actions is wrong and how to avoid it; their opening example is a dialogue in which user corrections fail to remove such an error. We ask what happens when a model nonetheless reuses its responses with λ>0 while informative user evidence keeps arriving. To our reading, they do not give conditions or rates for this case. Our contributions are:
W1 / Q2: Actual multi-turn evaluation and matched pruning baseline
The new task evaluation shows that clearing assistant history narrows the gap to a single-turn baseline with the same user information. On 50 Actions and 50 Math problems from Laban et al. (2026), with five runs per problem and model, CONCAT, two-turn KEEP, and four-turn KEEP use the same ordered user shards. Before the fourth answer, PRUNE shares KEEP's first three answers but clears their bodies, keeping user messages and role slots.
Pruning reduces four-turn error by 23.4 pp [15.8, 30.8] for Qwen and 15.6 pp [9.6, 22.0] for Mistral. The remaining PRUNE−CONCAT gap is 2.4 pp [−2.2, 7.4] for Qwen, which is not distinguishable from zero, and 10.8 pp [3.2, 18.4] for Mistral. Because clearing text also changes context length, this contrast does not measure λ itself.
Early errors persist when history is retained. On a predefined panel of 20 problems per task with three runs each, we fork every run after the fourth answer into two branches: one retains A1–A4, and the other clears their bodies. Both receive the same two neutral retry prompts, without new facts or error notification. Among runs whose fourth answer fails (38 for Qwen; 106 for Mistral):