- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Paper Summary
The paper studies intent drift: superseded or withdrawn user intent continuing to influence an agent's final output. It introduces IntentFlux, which converts verifiable tasks into simulated multi-turn dialogues using ADD, DELETE, and REPLACE edits. Each dialogue includes superseded values (variants) and withdrawn constraints (decoys), and every task keeps its original grader.
Performance degrades as the amount of stale information grows, and all eight evaluated models show a positive clean–drift gap. The authors then propose StateForge, which maintains an explicit active-intent state before generation. It improves the mean score from 0.367 to 0.467, and an oracle state reaches 0.549. Finally, they explore distilling this tracking ability into a small tracker and into the agent itself.
Critical strengths
Critical Strengths
- The benchmark design is clean. Original graders are preserved, the final intent is known, and decoys are kept only when obeying them changes the grader's verdict, so failures can be attributed to stale intent.
- Evaluation is careful. It uses paired confidence intervals, per-family results, and a turn-matched control, and the authors are candid about what each experiment does and does not establish.
Critical weaknesses
Critical Weaknesses
- Intent drift is not isolated from incremental disclosure. The control condition states the full intent in turn 1, whereas drift dialogues reveal goals across many turns. The gap therefore mixes the effect of supersession with the known sharding effect (Laban et al., 2025). The easy stratum already shows a large gap (0.19–0.30), so much of the effect may come from sharding. An ADD-only control is needed to separate the two.
- The novelty over Tack et al. (2026) is limited. That work also converts verifiable tasks into revision-based interactions while preserving graders. The paper does not compare the two directly.
- The ablation does not support the method claim. Recap baselines matched in calls and tokens perform comparably to StateForge (Table 12). The gain may therefore come from extra computation rather than from the explicit state representation.
- Numbers are inconsistent across tables. For Qwen3.5-122B under medium drift, the bare score is 0.240 in Table 8, 0.367 in Table 2, and 0.469 in Table 10. StateForge scores 0.467 in Table 2 but 0.428 in Table 12.
- The paper's scope is spread thin. The interactive transfer, the on-policy distillation, and the thinking distillation each use a single model or protocol and in-distribution data. CAR is uninformative for ToTTo and SummHay, which never reach full credit.
Questions for authors
Questions For Authors
- What is the clean–drift gap under an ADD-only control without variants or decoys?
- How do IntentFlux results compare to Tack et al. (2026)?
- Please reconcile the inconsistent scores in Tables 2, 8, 10, and 12.
- What evidence shows that the explicit state representation, rather than extra computation, drives StateForge's gain?
9 comments
Q2. Novelty and comparison with Tack et al. (2026)
The novelty over Tack et al. (2026) is limited. That work also converts verifiable tasks into revision-based interactions while preserving graders. The paper does not compare the two directly.
How do IntentFlux results compare to Tack et al. (2026)?
A2 (Part 1). Tack et al. (2026) is concurrent work on evolving user intent. We acknowledge the overlap: both frameworks construct multi-turn conversations that converge to a known source task and preserve its original grader. The distinction is not grader preservation or the use of revision alone, but the explicit construction of withdrawn constraints whose erroneous retention affects task success. We have added the following comparison to clarify this relationship.
| Aspect | Tack et al. (2026) | INTENTFLUX |
|---|---|---|
| Intent representation | A function and its arguments, with a final source-task anchor. | Atomic requirements, an edit sequence, and a known final active set. |
| Edit construction | Argument reveal, argument revision, and function switching through predecessor tasks. | Requirement-level ADD, REPLACE, and DELETE, with same-slot variants and separately constructed withdrawn decoys. |
| Shared mechanism | Counterfactual argument values model revision of an existing requirement. | Same-slot variants likewise replace an old value with a new value. |
| Withdrawal case | Function switching introduces context and arguments from a related predecessor task. | An extra requirement is withdrawn within the same task, without assigning a replacement value or changing the task. |
| Validation target | Predecessor-only arguments are checked for preserving the final task's answer (Appendix D.3). | A decoy is retained only when obeying it would change the source grader's verdict. |
Example: replacement versus withdrawal. Consider a task whose final output should contain department descriptions only. The following are illustrative requirement edits, not an additional experiment:
- Replacement: “Return department names” followed by “Return department descriptions instead.” The output-field requirement remains active, but its value changes from
department_nametodepartment_description. The state update removes the old goal and inserts its replacement. - Withdrawal: “Return department descriptions and also include department names” followed by “Drop the names column; keep the descriptions.” The description requirement remains unchanged, while the extra requirement to include
department_nameis deleted. No new requirement replaces it.
Replacement therefore requires selecting the current value of a still-active requirement; withdrawal requires recognizing that an earlier requirement has lost its binding force while preserving the other valid requirements. Updating values alone does not specify when an extra obligation must disappear.
What withdrawal cases measure. Our decoy construction targets this second capability: can an agent stop enforcing a withdrawn constraint without discarding still-valid information? With a grader expecting description-only output, a candidate that returns both names and descriptions is incorrect even if the descriptions themselves are correct. Thus, the withdrawn requirement has an observable task consequence, rather than being merely irrelevant background. INTENTFLUX creates controlled cases in which deleting an obligation, without a successor value or task switch, is necessary for success. This complements the counterfactual value-revision mechanism shared with Tack et al.
The validation distinction matters for this measurement. Tack et al.'s predecessor check keeps predecessor-only arguments answer-preserving for the final task. Our decoy criterion instead selects extra constraints whose retention would alter the final output or action and the grader's verdict. It links requirement invalidation to behavior, not just to a different operator name. The benchmark records which requirements were withdrawn and which remain active, enabling output-level inspection of these failures. Aggregate CAR and mean score still summarize task outcomes; the construction alone does not assign every failure to withdrawal.
Variant and decoy candidate budgets are separately parameterized, although the reported difficulty sweep increases them jointly. This permits constructing replacement and withdrawal scenarios within the same source-task setup, without interpreting the existing sweep as a separate causal estimate for each factor.
Q2. Novelty and comparison with Tack et al. (2026)
The novelty over Tack et al. (2026) is limited. That work also converts verifiable tasks into revision-based interactions while preserving graders. The paper does not compare the two directly.
How do IntentFlux results compare to Tack et al. (2026)?
A2 (Part 2) Reported results. Tack et al. report that GPT-5.5 drops from 99.0% to 80.5% on GSM8K under six composed intent transitions. INTENTFLUX likewise observes degradation under evolving requirements: on the GENERAL-TEST GSM8K cases, Qwen3.5-122B falls from 0.952 in the clean condition to 0.524 under medium drift (Table 11), and every one of the eight evaluated models shows a positive clean–drift gap at every difficulty level. We read the two sets of findings as complementary, not as a ranking of benchmark difficulty, because models, cases, edit schedules, and metric aggregation differ. The revised manuscript identifies Tack et al. as concurrent work and gives the protocol comparison and the replacement/withdrawal example in Appendix A.1, which the benchmark description references.
Thank you for your detailed comments on progressive disclosure, novelty, the ablation evidence, numerical consistency, and the scope of the training studies. In response, we added controls and repeated comparisons, reconciled the reported scores, and clarified which conclusions the evidence supports. We respond to each concern below.
Q1. Incremental disclosure and the need for an ADD-only control
A1. Thank you for suggesting this control; we have added it. On the same 125 GENERAL-TEST cases with Qwen3.5-122B, turn-matched ADD-only falls 0.110 below clean in mean score (paired 95% CI [0.042, 0.176]) and 0.072 in CAR, and natural ADD-only falls 0.120 in CAR. Dialogues with revisions lose a further 0.192 in mean score ([0.100, 0.286]) and 0.184 in CAR ([0.088, 0.272]) relative to matched ADD-only. Incremental disclosure is thus part of the gap, as the reviewer suspected, but the smaller part.
Easy-stratum gap. Both ADD-only CAR losses (0.072 and 0.120) are well below Qwen3.5-122B's easy-condition IDG of 0.352 (clean CAR 0.640 versus easy-drift CAR 0.288), so sharding alone does not explain most of that gap. Because the control is matched to medium dialogues and comes from a separate replay, we treat this as context, not as a decomposition of the easy-condition IDG; the directly controlled number is the medium-revision loss above.
Control construction. Both ADD-only conditions disclose only the final valid requirements in their original order, without variants, decoys, replacements, or withdrawals. Natural ADD-only presents one requirement per user turn. Matched ADD-only uses the same number of user turns as the corresponding medium-drift dialogue: we distribute the requirements approximately evenly across those turns and fill unused turns with waiting messages that contain no task requirements. When there are more requirements than turns, consecutive requirements are grouped. The closing request is shared, and the last turn does not recap the complete final specification. Thus, matched ADD-only retains progressive disclosure while matching the user-turn count and closing wording.
Replay results. All four conditions share one runtime and grading configuration, with temperature 0 and no format instruction beyond the shared closing request. The revision row comes from this replay, not from the headline bare run in Table 2, and since the replay uses a different summary judge (see our response to Reviewer GTUK, Q2), we compare scores only within it:
The construction and results are now in Section 4.1 and the appendix.