acceptodds
Under review as a conference paper at ICLR 2027

When Users Change Their Minds? Measuring and Repairing Intent Drift in LLM Agents

Abstract

LLM agents often operate over multi-turn interactions in which user intent changes before execution. We study intent drift: the failure mode in which superseded parts of the user's intent continue to influence the final answer or tool action. We introduce INTENTFLUX, an executable benchmark that converts verifiable tasks into dialogues with controlled intent changes while preserving their original graders. In a 627-case calibration, mean task score falls from to as dialogues contain more superseded and withdrawn information. Across eight models, the rate of fully correct solutions is significantly lower when the same final task must be recovered from an evolving dialogue rather than given directly in a single turn. We further introduce STATEFORGE, which explicitly maintains the active requirements before generation. On GENERAL-TEST, it improves mean task score from to . Providing the ground-truth final state improves performance further but still does not recover single-turn performance, indicating that state-estimation errors explain only part of the gap. These results establish intent drift as a measurable multi-turn failure mode and explicit state maintenance as a partial mitigation.

Primary area:
datasets and benchmarks
Keywords:
Intent Drift, LLM Agents, Agent Harness, Multi-turn Interaction, Post-Training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7CrDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Paper Summary

The paper studies intent drift: superseded or withdrawn user intent continuing to influence an agent's final output. It introduces IntentFlux, which converts verifiable tasks into simulated multi-turn dialogues using ADD, DELETE, and REPLACE edits. Each dialogue includes superseded values (variants) and withdrawn constraints (decoys), and every task keeps its original grader.

Performance degrades as the amount of stale information grows, and all eight evaluated models show a positive clean–drift gap. The authors then propose StateForge, which maintains an explicit active-intent state before generation. It improves the mean score from 0.367 to 0.467, and an oracle state reaches 0.549. Finally, they explore distilling this tracking ability into a small tracker and into the agent itself.

Critical strengths

Critical Strengths

  • The benchmark design is clean. Original graders are preserved, the final intent is known, and decoys are kept only when obeying them changes the grader's verdict, so failures can be attributed to stale intent.
  • Evaluation is careful. It uses paired confidence intervals, per-family results, and a turn-matched control, and the authors are candid about what each experiment does and does not establish.
Critical weaknesses

Critical Weaknesses

  1. Intent drift is not isolated from incremental disclosure. The control condition states the full intent in turn 1, whereas drift dialogues reveal goals across many turns. The gap therefore mixes the effect of supersession with the known sharding effect (Laban et al., 2025). The easy stratum already shows a large gap (0.19–0.30), so much of the effect may come from sharding. An ADD-only control is needed to separate the two.
  2. The novelty over Tack et al. (2026) is limited. That work also converts verifiable tasks into revision-based interactions while preserving graders. The paper does not compare the two directly.
  3. The ablation does not support the method claim. Recap baselines matched in calls and tokens perform comparably to StateForge (Table 12). The gain may therefore come from extra computation rather than from the explicit state representation.
  4. Numbers are inconsistent across tables. For Qwen3.5-122B under medium drift, the bare score is 0.240 in Table 8, 0.367 in Table 2, and 0.469 in Table 10. StateForge scores 0.467 in Table 2 but 0.428 in Table 12.
  5. The paper's scope is spread thin. The interactive transfer, the on-policy distillation, and the thinking distillation each use a single model or protocol and in-distribution data. CAR is uninformative for ToTTo and SummHay, which never reach full credit.
Questions for authors

Questions For Authors

  1. What is the clean–drift gap under an ADD-only control without variants or decoys?
  2. How do IntentFlux results compare to Tack et al. (2026)?
  3. Please reconcile the inconsistent scores in Tables 2, 8, 10, and 12.
  4. What evidence shows that the explicit state representation, rather than extra computation, drives StateForge's gain?
9 comments

Thank you for your detailed comments on progressive disclosure, novelty, the ablation evidence, numerical consistency, and the scope of the training studies. In response, we added controls and repeated comparisons, reconciled the reported scores, and clarified which conclusions the evidence supports. We respond to each concern below.

Q1. Incremental disclosure and the need for an ADD-only control

Intent drift is not isolated from incremental disclosure. The control condition states the full intent in turn 1, whereas drift dialogues reveal goals across many turns. The gap therefore mixes the effect of supersession with the known sharding effect (Laban et al., 2025). The easy stratum already shows a large gap (0.19–0.30), so much of the effect may come from sharding. An ADD-only control is needed to separate the two.

What is the clean–drift gap under an ADD-only control without variants or decoys?

A1. Thank you for suggesting this control; we have added it. On the same 125 GENERAL-TEST cases with Qwen3.5-122B, turn-matched ADD-only falls 0.110 below clean in mean score (paired 95% CI [0.042, 0.176]) and 0.072 in CAR, and natural ADD-only falls 0.120 in CAR. Dialogues with revisions lose a further 0.192 in mean score ([0.100, 0.286]) and 0.184 in CAR ([0.088, 0.272]) relative to matched ADD-only. Incremental disclosure is thus part of the gap, as the reviewer suspected, but the smaller part.

Easy-stratum gap. Both ADD-only CAR losses (0.072 and 0.120) are well below Qwen3.5-122B's easy-condition IDG of 0.352 (clean CAR 0.640 versus easy-drift CAR 0.288), so sharding alone does not explain most of that gap. Because the control is matched to medium dialogues and comes from a separate replay, we treat this as context, not as a decomposition of the easy-condition IDG; the directly controlled number is the medium-revision loss above.

Control construction. Both ADD-only conditions disclose only the final valid requirements in their original order, without variants, decoys, replacements, or withdrawals. Natural ADD-only presents one requirement per user turn. Matched ADD-only uses the same number of user turns as the corresponding medium-drift dialogue: we distribute the requirements approximately evenly across those turns and fill unused turns with waiting messages that contain no task requirements. When there are more requirements than turns, consecutive requirements are grouped. The closing request is shared, and the last turn does not recap the complete final specification. Thus, matched ADD-only retains progressive disclosure while matching the user-turn count and closing wording.

Replay results. All four conditions share one runtime and grading configuration, with temperature 0 and no format instruction beyond the shared closing request. The revision row comes from this replay, not from the headline bare run in Table 2, and since the replay uses a different summary judge (see our response to Reviewer GTUK, Q2), we compare scores only within it:

ConditionMean scoreCAR
Clean single-turn0.7650.616
Natural ADD-only0.6140.496
Matched ADD-only0.6550.544
Medium revision (same replay protocol)0.4630.360
Clean −- matched ADD-only0.110 [0.042, 0.176]0.072 [0.008, 0.136]
Matched ADD-only −- revision0.192 [0.100, 0.286]0.184 [0.088, 0.272]

The construction and results are now in Section 4.1 and the appendix.

Q2. Novelty and comparison with Tack et al. (2026)

The novelty over Tack et al. (2026) is limited. That work also converts verifiable tasks into revision-based interactions while preserving graders. The paper does not compare the two directly.

How do IntentFlux results compare to Tack et al. (2026)?

A2 (Part 1). Tack et al. (2026) is concurrent work on evolving user intent. We acknowledge the overlap: both frameworks construct multi-turn conversations that converge to a known source task and preserve its original grader. The distinction is not grader preservation or the use of revision alone, but the explicit construction of withdrawn constraints whose erroneous retention affects task success. We have added the following comparison to clarify this relationship.

AspectTack et al. (2026)INTENTFLUX
Intent representationA function and its arguments, with a final source-task anchor.Atomic requirements, an edit sequence, and a known final active set.
Edit constructionArgument reveal, argument revision, and function switching through predecessor tasks.Requirement-level ADD, REPLACE, and DELETE, with same-slot variants and separately constructed withdrawn decoys.
Shared mechanismCounterfactual argument values model revision of an existing requirement.Same-slot variants likewise replace an old value with a new value.
Withdrawal caseFunction switching introduces context and arguments from a related predecessor task.An extra requirement is withdrawn within the same task, without assigning a replacement value or changing the task.
Validation targetPredecessor-only arguments are checked for preserving the final task's answer (Appendix D.3).A decoy is retained only when obeying it would change the source grader's verdict.

Example: replacement versus withdrawal. Consider a task whose final output should contain department descriptions only. The following are illustrative requirement edits, not an additional experiment:

  • Replacement: “Return department names” followed by “Return department descriptions instead.” The output-field requirement remains active, but its value changes from department_name to department_description. The state update removes the old goal and inserts its replacement.
  • Withdrawal: “Return department descriptions and also include department names” followed by “Drop the names column; keep the descriptions.” The description requirement remains unchanged, while the extra requirement to include department_name is deleted. No new requirement replaces it.

Replacement therefore requires selecting the current value of a still-active requirement; withdrawal requires recognizing that an earlier requirement has lost its binding force while preserving the other valid requirements. Updating values alone does not specify when an extra obligation must disappear.

What withdrawal cases measure. Our decoy construction targets this second capability: can an agent stop enforcing a withdrawn constraint without discarding still-valid information? With a grader expecting description-only output, a candidate that returns both names and descriptions is incorrect even if the descriptions themselves are correct. Thus, the withdrawn requirement has an observable task consequence, rather than being merely irrelevant background. INTENTFLUX creates controlled cases in which deleting an obligation, without a successor value or task switch, is necessary for success. This complements the counterfactual value-revision mechanism shared with Tack et al.

The validation distinction matters for this measurement. Tack et al.'s predecessor check keeps predecessor-only arguments answer-preserving for the final task. Our decoy criterion instead selects extra constraints whose retention would alter the final output or action and the grader's verdict. It links requirement invalidation to behavior, not just to a different operator name. The benchmark records which requirements were withdrawn and which remain active, enabling output-level inspection of these failures. Aggregate CAR and mean score still summarize task outcomes; the construction alone does not assign every failure to withdrawal.

Variant and decoy candidate budgets are separately parameterized, although the reported difficulty sweep increases them jointly. This permits constructing replacement and withdrawal scenarios within the same source-task setup, without interpreting the existing sweep as a separate causal estimate for each factor.

Q2. Novelty and comparison with Tack et al. (2026)

The novelty over Tack et al. (2026) is limited. That work also converts verifiable tasks into revision-based interactions while preserving graders. The paper does not compare the two directly.

How do IntentFlux results compare to Tack et al. (2026)?

A2 (Part 2) Reported results. Tack et al. report that GPT-5.5 drops from 99.0% to 80.5% on GSM8K under six composed intent transitions. INTENTFLUX likewise observes degradation under evolving requirements: on the GENERAL-TEST GSM8K cases, Qwen3.5-122B falls from 0.952 in the clean condition to 0.524 under medium drift (Table 11), and every one of the eight evaluated models shows a positive clean–drift gap at every difficulty level. We read the two sets of findings as complementary, not as a ranking of benchmark difficulty, because models, cases, edit schedules, and metric aggregation differ. The revised manuscript identifies Tack et al. as concurrent work and gives the protocol comparison and the replacement/withdrawal example in Appendix A.1, which the benchmark description references.

Reviewer_GTUKOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies intent drift, a multi-turn failure mode in which superseded or withdrawn user requirements continue to affect an LLM agent’s final output or action. It introduces INTENTFLUX, a benchmark that converts verifiable tasks into evolving dialogues through controlled additions, deletions, and replacements while retaining the original task graders. Experiments across eight models show a consistent performance gap between evolving-dialogue and clean single-turn conditions, and a turn-matched control suggests that dialogue length alone does not explain the degradation. The paper further proposes STATEFORGE, which explicitly tracks the currently active requirements before generation. STATEFORGE improves performance over bare multi-turn execution and evaluated context-management harnesses. The paper also explores distilling state tracking into a smaller tracker and internalizing the behavior into a standalone agent.

Critical strengths
  1. Important and practically relevant problem. Users frequently revise or withdraw requirements during interaction, making reliable intent updating essential for deployed agents.

  2. INTENTFLUX preserves task-native graders for code, mathematics, SQL, generation, and tool-use tasks, providing more objective evaluation than purely judge-based conversational benchmarks.

  3. The distinction between replaced variants and withdrawn decoys provides a useful framework for constructing evolving-intent interactions. Requiring decoys to affect the original grader makes them behaviorally consequential.

  4. The benchmark includes multiple task families and eight recent models, and the degradation is observed across models and difficulty levels rather than being limited to one system.

Critical weaknesses
  1. INTENTFLUX defines intent drift as obsolete requirements continuing to influence the agent, but it only measures the performance gap between recovering a task from an evolving dialogue and receiving the final task directly. A failure may instead result from missed updates, loss of still-valid information, reference-resolution errors, or general task-solving degradation. The benchmark therefore measures evolving-intent recovery more directly than stale-intent influence itself. More detailed analysis of the reason could help to clarify the point.

  2. The turn-matched control rules out dialogue length alone, but it does not match the semantic competition, ambiguity, and reconstruction burden introduced by revisions. Thus, the observed gap cannot be causally attributed to obsolete information rather than the broader difficulty of processing revisions.

  3. The difference between this work and related work such as InterruptionBench or EvolIF is still marginal. A clearer clarification of the novelty is needed to emphasize the contribution except for the difference in tasks from web navigation to other agentic tasks.

  4. The experiments do not adequately compare against strategies such as reconstructing and confirming the final specification before execution, or solving from a final recap after discarding the conflicting dialogue. The interactive results show that explicitly restating the final intent largely removes the measured gap, suggesting that simpler confirmation mechanisms may already address much of the problem.

  5. Ground-truth final-state injection achieves only 0.549, compared with 0.778 in the clean single-turn condition. This suggests that state-estimation errors explain only part of the degradation, while retained-history interference, prompt placement, or downstream reasoning failures may be equally important. These residual errors are not sufficiently analyzed, weakening the framing of intent drift primarily as a state-maintenance problem.

Questions for authors
  1. What are the reasons that the agents fail to complete the task?

Others see Weakness.

10 comments

Thank you for your review and for identifying the need to distinguish the measured completion gap from specific failure mechanisms. Your comments also prompted us to clarify the contribution relative to related benchmarks, evaluate simpler reconstruction alternatives, and analyze failures that remain with oracle state information. We address these points below.

Q1. What the benchmark measures and why agents fail

INTENTFLUX defines intent drift as obsolete requirements continuing to influence the agent, but it only measures the performance gap between recovering a task from an evolving dialogue and receiving the final task directly. A failure may instead result from missed updates, loss of still-valid information, reference-resolution errors, or general task-solving degradation. The benchmark therefore measures evolving-intent recovery more directly than stale-intent influence itself. More detailed analysis of the reasons could help clarify this point.

What are the reasons that the agents fail to complete the task?

A1. We agree with this characterization. The aggregate clean–drift gap measures the cost of recovering and executing the final task from an evolving dialogue, not the rate at which stale intent is used. The revision keeps the two apart: “intent drift” names the failure mode, while IDG is an outcome-level measure that also includes other multi-turn errors (Sections 3.1 and 3.4).

Why agents fail. We inspected 25 sampled outputs from bare Qwen3.5-122B on medium dialogues, comparing each output with the user updates and the final requirements and assigning one primary label per case. Eleven (44%) are task execution or output-specification errors, such as incorrect code logic or returning arithmetic instead of the required function calls. Eight (32%) lose still-valid requirements or task evidence, for example by omitting source-supported summary facts or dropping a still-required hotel query. Four (16%) follow a superseded instruction, such as a replaced section-heading instruction. The remaining two (8%) are reference-resolution or update-binding errors, such as a temperature update applied to the wrong call, or are unresolved. Stale-instruction use does occur, but execution errors and the loss of still-valid information are more common.

Role of the construction. The construction is still what makes the stale cases identifiable. Each retained decoy changes the grader's verdict if obeyed, and the benchmark records which requirements were replaced or withdrawn, so an output can be checked against them directly. This is how the four cases above were found; they are not inferred from the aggregate gap. We have added this analysis to Section 6.1 and Appendix B.

Q2. Limitations of the turn-matched control and causal attribution

The turn-matched control rules out dialogue length alone, but it does not match the semantic competition, ambiguity, and reconstruction burden introduced by revisions. Thus, the observed gap cannot be causally attributed to obsolete information rather than the broader difficulty of processing revisions.

A2. We agree that the gap cannot be attributed to obsolete information alone. Besides stale values, revision dialogues involve competition between old and new values, ambiguity, and the work of reconstructing the final specification, and these are part of what processing a revision means. In the revision we therefore call the loss that remains after controlling for length and progressive disclosure revision-associated, not a pure effect of stale information. The claim we keep is narrower: with turn count and disclosure controlled, dialogues in which requirements are replaced or withdrawn are still substantially harder than dialogues that reach the same final task without revisions.

ADD-only control. This claim rests on a new ADD-only control, which reveals the final valid requirements one at a time without replacements or withdrawals. The matched version uses the same number of user turns and the same closing request as the medium-drift dialogue, with task-free waiting messages in unused turns. All four conditions below were replayed with Qwen3.5-122B on the same 125 cases under one runtime and judge configuration:

ConditionMean task scoreCAR
Clean single-turn0.7650.616
Natural ADD-only0.6140.496
Matched ADD-only0.6550.544
Medium revision (same replay protocol)0.4630.360

Compared with clean, matched ADD-only loses 0.110 in mean score (paired 95% CI [0.042, 0.176]). Medium revision loses a further 0.192 in mean score ([0.100, 0.286]) and 0.184 in CAR ([0.088, 0.272]) relative to matched ADD-only. Progressive disclosure explains part of the loss, and revisions add a larger gap on top of it. A1 discusses the individual failure causes.

Replay protocol. The replay uses scripted or frozen user messages, temperature 0, no extra format instruction, and Qwen3.5-122B as the summary judge, whereas Table 2 uses DeepSeek-V4-Flash as the judge. Its 0.463 for medium revision is therefore not on the same scale as the headline 0.367, and we compare conditions only within the replay. The construction and results are now in Section 4.1 and the appendix.

Q3. Novelty relative to InterruptionBench and EvolIF

The difference between this work and related work such as InterruptionBench or EvolIF is still marginal. A clearer explanation of the novelty is needed to emphasize contributions beyond extending the tasks from web navigation to other agentic tasks.

A3. The shared focus on changing requirements is real: both InterruptBench and EvolIF already include updates and removals. Our distinction is the evaluation object and construction, not the operator names or task-domain expansion alone.

AspectInterruptBenchEvolIFINTENTFLUX
Research objectWeb agents adapting during execution, including persistent environment state.Instruction following across interleaved topics and evolving constraints.Recovery and execution of a known final task from requirement edits and obsolete information.
ConstructionRewrite a WebArena-Lite request with missing, incorrect, or extra information; scheduled updates restore the original intent.Track topics, instructions, and constraints; synthesize queries after topic transitions and constraint addition, removal, or modification.Decompose source tasks into requirements; construct same-slot variants, withdrawn decoy pools, and dependency-constrained edit sequences.
EvaluationFinal web-task success, post-update success curves, action and token cost.Constraint/instruction satisfaction, endurance, recovery, and stability; dialogues end when modeled patience is exhausted.Original task graders, mean score and CAR, paired clean–drift outcomes, and an ADD-only disclosure control.
Capability emphasizedRepair ongoing behavior and environment state after interruption.Maintain instruction adherence and recover from failures as conversations deepen.Retain valid requirements and invalidate obsolete ones while solving the same final task; inspect known edits against task outcomes.

The central construction difference is that INTENTFLUX separately represents same-slot replacement candidates and extra constraints that will be withdrawn. A retained decoy must be consequential for the source grader: obeying it changes the verdict. This makes failure to invalidate that constraint an inspectable task error, while preserving the source task for comparison. InterruptBench also requires its updates to matter for the final intent, but its updates restore an originally specified web task; it does not construct withdrawn constraints within the task or separate them from same-slot replacements.

INTENTFLUX combines explicit requirement states, candidate pools, dependency-aware edits, original task grading, and fixed final-task comparisons. This supports inspecting obsolete-requirement handling alongside progressive-disclosure controls. StateForge uses the same representation to maintain active requirements and invalidate dependent conclusions before generation.

We have sharpened this distinction locally in Related Work, while retaining the existing benchmark and evaluation descriptions. The comparison follows InterruptBench's construction and evaluation sections and EvolIF's Sections 3.2–3.4.

Reviewer_hyA4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a new problem: whether a past user request that has since been obsoleted would negatively impact performance of LLMs in long-context conversations. It constructs a new benchmark to evaluate this problem and shows that even frontier LLMs suffer from this problem. It proposes a framework to solve this problem.

Critical strengths

This paper studies an important and imo very interesting problem: as the context gets longer and longer in a conversation, would some prior information "poison" the model? It proposes a benchmark and evaluation to support this hypothesis and proposes an approach to mitigate it.

It demonstrates that on the new benchmark, this is indeed a problem to be solved.

Critical weaknesses

Part of the writing is clear, but part is also very unclear. I have detailed my concerns below in the questions for authors section. In particular, the approach section (Sec 7 is too vague and too short. There is basically no explanation on Figure 2), raising concerns about the novelty of the paper if this part is not clearly explained.

Experiments on the improvement is only done with one model. Applying the technique on more models would improve the empirical validity of the paper's argument.

Questions for authors

When reading the abstract, someone would wonder the connection between 0.384 to 0.367. Why was the improvement not based on 0.384? Or why didn't the 627-case calibration just done on General-Test? This is something to consider to make the abstract easier to follow.

Line 46-48. Show example here or point to Figure 1. This definition here only is hard to understand.

Line 49: What is a "calibration pool"? What is a "drift score"?

Figure 1 (3) Edut -> Edit

Section 3 becomes harder to read. Although I think I can understand what it is trying to do mainly thru Figure 1 with the example, the text becomes hard to follow. For instance, what is a shard exactly?

I cannot understand some key parts of the evaluation. Most importantly, how exactly is the expected value taken in Section 4? I would understand if this is just reporting task accuracy - what exactly does expected value mean here?

It seems there are too many sections in the paper in general. Consider consolidate sections and organize better.

Sec 7 is too vague and too short. There is basically no explanation on Figure 2.

I would consider raising score if authors can clarify these questions, and upload a new version of PDF that has better presentation/

6 comments

Thank you for your detailed review and suggestions on the presentation, evaluation, and explanation of StateForge. We have used these comments to clarify the method, distinguish the evaluation settings and metrics, improve the organization, and extend the harness evaluation to additional models. We respond to each point below.

Q1. Writing clarity, Section 7 (now Section 5), and Figure 2

Part of the writing is clear, but part is also very unclear. In particular, the approach section (Section 7) is too vague and too short, and there is basically no explanation of Figure 2. If this part is not clearly explained, it raises concerns about the novelty of the paper.

A1. Thank you for pointing this out. We have expanded the method section (Section 7 in the original submission, now Section 5 after the consolidation described in A9), together with the caption and discussion of Figure 2.

How StateForge works. The revised section explains that StateForge separates requirement maintenance from downstream task solving. After each user turn, a tracker updates the active requirements: it adds new requirements, replaces superseded values, and removes withdrawn items. Requirements unaffected by the edit remain active. The tracker also distinguishes directly stated requirements from derived conclusions, so changing a requirement invalidates conclusions that depend on it. Before generation, the updated state is inserted between the cached dialogue history and the current user input. A separate format tracker maintains output conventions, and a final gate checks the generated answer against those conventions and can trigger regeneration. These roles are now stated in the text instead of being left to the diagram.

Figure 2 and the two training paths. In the example, the user first requests a Python parser and unit tests, then changes the language to Java. The language requirement becomes Java, and a derived choice of pytest must be discarded, while the unit-test requirement remains active. The ACTIVE, REMOVED, and DISCARD DERIVED entries distinguish valid information from an obsolete requirement and its dependent conclusion. This illustrates why removing an old requirement alone may leave an outdated implementation choice unresolved.

The two training paths address different questions. Path A uses on-policy distillation (OPD) to train a small external tracker, aiming to reduce the overhead of using a large model to manage the harness after every turn. The tracker learns structured state updates from teacher supervision while the task-solving model remains unchanged. Path B uses thinking-distillation SFT to test whether the agent can directly learn and internalize the harness's state-maintenance capability. Teacher reasoning traces contain explicit state updates, and the trained agent is evaluated without an external harness. OPD therefore targets cheaper external maintenance, while SFT studies internal state maintenance.

Novelty. The operation that StateForge adds is state folding: it determines which requirements and dependent conclusions are still valid, instead of only shortening the dialogue history. Figure 2 links this operation to two ways of learning it, in a small external tracker or inside the agent. We see this validity-aware design, and the study of how to deploy and learn it, as the contribution; the distillation algorithms themselves follow existing work, and the revised text says so.

Q2. Improvement experiments on more models

Experiments on the improvement are only done with one model. Applying the technique to more models would improve the empirical validity of the paper's argument.

A2. We have evaluated StateForge as a training-free harness on Claude Opus 4.7 and GPT-5.4, without fine-tuning the models. The results are:

ModelSettingMean scoreCAR
Claude Opus 4.7Bare0.3260.240
Claude Opus 4.7StateForge0.4850.368
GPT-5.4Bare0.3380.264
GPT-5.4StateForge0.3870.304

Opus 4.7 gains 0.159 in mean score and 0.128 in CAR, while GPT-5.4 gains 0.049 in mean score and 0.040 in CAR. Each row is a single execution on medium-drift GENERAL-TEST, with each base model itself serving as the StateForge tracker. These results extend the inference-time harness evaluation beyond Qwen3.5-122B. We have incorporated these findings into the StateForge results discussion and placed the numerical table in the appendix.

Q3. Abstract: the relationship between 0.384, 0.367, and the 627-case calibration

When reading the abstract, readers may wonder about the connection between 0.384 and 0.367. Why was the improvement not based on 0.384? Or why was the 627-case calibration not done directly on General-Test? The abstract should make these relationships easier to follow.

A3. Thank you, this was indeed unclear in the abstract. The two numbers come from different experiments, with different tested models, evaluation sets, and difficulty conditions, so 0.384 is not the starting point of the StateForge comparison:

Result in abstractEvaluation setTested modelConditionPurpose
0.476 ~ 0.384627-case sweepGemini 3.5 Flasheasy ~ hardbenchmark difficulty response
0.367 ~ 0.467GENERAL-TEST (125)Qwen3.5-122Bmediumbare vs. StateForge

Relation between 0.384 and 0.367. 0.384 is Gemini 3.5 Flash's hard-condition score on the 627-case sweep, whereas 0.367 is Qwen3.5-122B's bare medium-condition score on GENERAL-TEST, so the two are not directly comparable. For reference, Qwen3.5-122B scores 0.404, 0.367, and 0.272 on GENERAL-TEST for easy, medium, and hard (Table 9 of the revised manuscript).

Why the baseline is 0.367. The improvement is measured from 0.367 because StateForge is compared with bare execution of the same model on the same dialogues. We use Qwen3.5-122B for the method experiments because it is served locally, which lets us run the tracker-scale comparison (2B to 122B trackers), the ablations, and the distillation studies within one model family; all of these share the same frozen medium-condition trajectories.

Difficulty sweep on GENERAL-TEST. The sweep was in fact also run on GENERAL-TEST, for all eight models (Tables 1 and 9). For Gemini 3.5 Flash, GENERAL-TEST gives 0.482, 0.416, and 0.360 for easy, medium, and hard, close to 0.476, 0.419, and 0.384 on the 627-case set. We report the larger set for the sweep because it gives a more stable estimate across task families, and use the fixed 125-case set for the many repeated model and harness comparisons.

We have revised the abstract to name the set, model, and condition for both results, and replaced “calibration” with “difficulty sweep” throughout.

Q4. Lines 46–48: an example or a reference to Figure 1

Please show an example here or point to Figure 1. The definition alone is hard to understand.

A4. We have added a concrete example immediately after the definition of intent drift and included an explicit reference to Figure 1. A user first requests a parser in Python and later changes the required language to Java. If the agent still produces Python code, its output follows the superseded language requirement rather than the current one. This connects the definition to an observable output error before introducing the benchmark notation. Figure 1 is then referenced for how requirement replacements and withdrawals are represented in INTENTFLUX.

Q5. Line 49: definitions of calibration pool and drift score

What is a "calibration pool"? What is a "drift score"?

A5. Thank you for highlighting these ambiguous terms. We have replaced “calibration pool” with “627-case difficulty-sweep set” and defined its purpose at first mention: it compares performance across easy, medium, and hard dialogues with increasing loads of superseded and withdrawn requirements. It is not used to fit a calibration model. As explained in A3, this study characterizes benchmark difficulty, while GENERAL-TEST provides the fixed evaluation tasks.

“Drift score” meant the mean task score under the evolving-dialogue condition, not the clean–drift gap. We now use “mean task score under the drift condition” and explain it as the average score assigned by the original task graders to final outputs or actions produced from evolving dialogues. It retains partial credit from continuous task graders.

The revised metric description distinguishes three quantities:

  • Mean task score under drift: performance in the evolving-dialogue condition, including partial credit.
  • CAR: the fraction of cases receiving full task-native credit under the current final intent.
  • IDG: clean CAR minus drift CAR on the same cases, measuring the difference in full-credit task completion between the two conditions.

Thus, a drift-condition score describes one condition, whereas IDG compares two conditions. The explicit formulas and denominator are given in A8 and in the revised Evaluation Protocol.

Q6. Figure 1(3): typo correction

"Edut" should be changed to "Edit".

A6. Thank you for catching this typo. We have corrected “Edut” to “Edit” in panel (3) of Figure 1 and updated the figure used in the manuscript.

Q7. Section 3: readability and the definition of shard

Section 3 becomes harder to read. Although Figure 1 and its example help explain the intended idea, the text is hard to follow. For instance, what exactly is a shard?

A7. We have defined “shard” in plain language before introducing the formal intent-state notation and added a parser example. The revised explanation begins:

A shard is a separately represented task requirement whose content affects the correct output or action. For example, a request to implement a parser in Python that reads CSV input and returns a list of records contains requirements for the implementation language, input format, and return type. Changing Python to Java replaces the language requirement while leaving the input and return-type requirements unchanged.

We also clarify the relationship between a source shard and an intent goal. Source shards are converted into target goals that specify the final intent; for most task families, the mapping is direct, while task-specific loaders may construct goals from other source fields. Goals are the units of ADD, DELETE, and REPLACE edits. Variants and decoys are additional goals introduced during editing and need not correspond to a source shard.

The construction is now explained in order, before the equations:

  1. Decompose the source task into atomic requirements and construct the target goals.
  2. Construct same-slot variants and withdrawn decoys, distinguishing replacement of a value from removal of an extra constraint.
  3. Sample requirement edits subject to semantic dependencies.
  4. Apply the edit sequence to determine the final active requirements, which define the task evaluated by the original grader.

This explanation refers directly to Figure 1 and removes the repeated shard definition from the later construction paragraph. The formal state-update rules follow the plain-language description, so readers can understand the objects before encountering the notation.

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

Thank you for the detailed reviews. We have revised the manuscript to clarify the contribution, strengthen the comparisons, and make the scope of the evidence explicit. The main changes are summarized below, with references to the reviewer concerns they address. Full results and protocols are provided in the reviewer-specific responses, and manuscript revisions are marked in blue.

Main revisions

1. Clearer method and evaluation presentation (hyA4 Q1, Q3–Q10). We expanded the StateForge explanation and Figure 2, added concrete examples and a plain-language shard definition, clarified the evaluation sets and metrics, and consolidated the short sections. The revised text explains how active requirements are maintained and dependent conclusions invalidated. It also distinguishes the difficulty sweep from the method comparison and explains the separate motivations for OPD and thinking-distillation SFT.

2. Controls for progressive disclosure (GTUK Q2; 7CrD Q1). We added natural and turn-matched ADD-only controls to distinguish gradual disclosure from dialogues containing replacements and withdrawals. These comparisons show additional revision-associated loss beyond progressive disclosure. We narrowed the interpretation accordingly: this is not a pure causal estimate of obsolete-information influence.

3. Simpler baselines and repeated comparisons (GTUK Q4; 7CrD Q3). We added discard-and-solve and repeated the computation-matched recap comparison. Discarding the dialogue improves binary-task completion but loses source material needed by data-to-text and summarization tasks. We also clarified that correct user restatement supplies final-intent information externally. StateForge has higher observed recap averages, but the paired intervals include zero, so we claim neither a statistically separated advantage nor equivalence.

4. Failure analysis and a narrower framing (GTUK Q1, Q5). We added output-level analyses for bare execution and expanded the oracle-output inspection. The revised text separates superseded-instruction use from valid-information loss, execution errors, and reference/grader issues. IDG measures the recovery-and-execution gap, not the frequency of stale-instruction use. Correct oracle state does not eliminate downstream failures; prompt placement remains untested.

5. Broader harness evaluation and clearer learning scope (hyA4 Q2; 7CrD Q5). We added training-free StateForge evaluations on Claude Opus 4.7 and GPT-5.4, with improvements on both models. We clarified how the existing OPD and SFT studies examine external tracking and internal state maintenance, respectively, while retaining their single-setting and generalization limits.

6. More precise novelty positioning (GTUK Q3; 7CrD Q2). In the responses we compare with InterruptBench and EvolIF; the manuscript adds a protocol comparison with Tack et al. (Appendix A.1) and sharpens Related Work. The clarified distinction is the construction of within-task withdrawn constraints whose erroneous retention changes the grader's verdict, separately from same-slot replacements. A concrete replacement-versus-withdrawal example explains this difference.

7. Consistent numerical reporting (7CrD Q4, Q6; hyA4 Q3, Q5, Q8). We corrected the affected scores and table references, reconciled the bare baseline, and distinguished headline executions from component runs and repeated averages. We also report continuous task scores for ToTTo and SummHay rather than interpreting their zero full-credit CAR values as an absence of performance differences.

We believe the revision strengthens the benchmark controls and baseline coverage while making the method, related-work distinctions, and reporting easier to follow. We thank the reviewers and Area Chair for helping us clarify both the contribution and the limits of the current evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.