acceptodds
Under review as a conference paper at ICLR 2027

StateWise: Teaching Language Models to Track and Obey Evolving Constraints in Multi-Turn Dialogue

Abstract

Users of conversational assistants rarely state all requirements up front: they add constraints, amend them, withdraw them, digress, and return. Instruction-tuned large language models (LLMs) degrade in this regime in two ways—active constraints stop being honored as turns accumulate (persistence failure), and withdrawn constraints keep being enacted (relapse)—and the only existing repair we are aware of is an external ledger that re-injects a compiled specification at every turn. To tackle this, we propose StateWise, a self-improving approach that teaches a single LLM to maintain its own revisable constraint state over a typed state with tombstones for revoked constraints, so that persistence and relapse are measured by executable checkers and ground-truth states are compiled for free. StateWise first fine-tunes the LLM on two complementary skills, summarizing a dialogue into its constraint state and responding conditioned on that state; it then self-trains on trajectories the model itself generates, keeping only turns that pass an exact round-trip check against the compiled state. This recipe teaches 7–8B models to track the state almost perfectly, but, surprisingly, not to obey it: on Llama-3.1-8B every supervised variant roughly doubles relapse, and on Qwen2.5-7B none reduces it. We therefore add a preference stage on self-generated relapse contrasts, in which the rejected response enacts a revoked rule and the chosen response passes the round-trip check. Experiments on two backbones with held-out constraint families and unseen phrasings show that StateWise cuts relapse by 8–22 points relative to supervised self-training, lifts conversation success by 8–19 points over the backbone, and beats an oracle compiled-ledger baseline on relapse with no inference-time scaffolding. We also characterize the recipe’s cost on cumulative-constraint and single-turn benchmarks and study weight interpolation as a mitigation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
instruction following, multi-turn dialogue, self-improvement, preference optimization, synthetic data, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6X4sOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a novel self-training method for improving the ability of LLMs to follow instructions that adapt and change over the course of long conversations, using both rejection-sampled SFT with self-generated data (4.2) and DPO preference tuning which contrast well-formed answers with those which ‘relapse’ or incorrectly follow a revoked constraint (4.3).

As noted elsewhere: the font and format are wrong, but the article is well within the page limit and otherwise appropriate for the venue, so I completed a full review.

Critical strengths

This paper introduces an interesting approach to modeling and evaluating instruction following over long conversations, addressing a key problem in LLMs which tend to lose the ability to follow an active set of instructions in longer conversations.

Critical weaknesses
  1. From the problem definition, it is not clear why relapse is undesirable/poor instruction following except in cases where a revoked constraint contradicts an active one. For example, if I ask for a bulleted-list where all sentences start with the letter ‘A’, then revoke the bulleted-list constraint, I have not supplied a requested alternative format: it's reasonable for the next answer to still be a bulleted list even if I do not require one. To my understanding, the problem setup treats this as unsuccessful, scoring 1/1 on persistence and 1/1 on relapse. Does this explain why relapse scores are generally high for models without an oracle ledger or preference training (<~0.4)? My thought is that the improvement from preference training comes from penalizing the following of constraints in RtR_t when I think a more natural interpretation of instruction following would be indifferent to RtR_t.
  2. A significant part of the main paper (Table 3 + 6.3) discusses a replication of the main experiments with a second random seed. I think this should go into the appendix and the main paper should instead include more analysis, which seems relatively light currently. For example, what kind of constraints are followed well, exhibit the most relapse, etc? At what turn indices?
  3. The dataset is fairly small, but if I understand right, the main evaluation datasets are synthetic and new. Pre-registration addresses statistical significance to some degree but I think a simpler approach would be to generate more data for evaluation, or just show that the current results are statistically significant, rather than pre-registering. Overall, the evaluation data for Table 1, etc, should be explained more clearly since it is introduced in this paper.
  4. Only models up to 8B are evaluated. While I think this is reasonable for the training experiments, an evaluation of larger models for inference only should be included to see if the overall instruction-following problem is resolved at larger model sizes.
Questions for authors
  1. The paper mentions pre-registration of hypotheses and includes timestamps and hypotheses in the Appendix. Are these also pre-registered with some verifiable registry? I understand these might be hard to share anonymously, but think this would be important for actually making claims about pre-registration in a camera-ready version. Though in my opinion, a better approach would be to just generate more data to ensure the findings are statistically robust.
Reviewer_3mH5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies multi-turn dialogue where constraints are added, amended, and revoked, and identifies two failures: forgetting active constraints and re-enacting revoked ones (“relapse”). It formalizes a revisable constraint state with tombstones and proposes StateWise, a three-stage self-improvement recipe: supervised skill induction, round-trip-filtered self-training, and relapse-contrastive DPO. The main claims are that tracking a state and obeying it are separable skills, that filtered self-training teaches only tracking and can worsen relapse, and that a short DPO stage on self-labeled relapse contrasts reduces relapse below the backbone and below an oracle ledger baseline.

Critical strengths
  1. Showing that supervised training roughly doubles relapse, and that a round-trip filter teaches state tracking without teaching obedience, is the paper’s most consequential contribution; Table 4 cleanly isolates the relapse-specific contrast as the necessary ingredient.
  2. Gates and test seeds were fixed before runs, and failed gates are reported as failures, which materially raises trust in the numbers.
Critical weaknesses
  1. The model is 6.5 points below the backbone on IFEval (failing the preregistered non-regression gate) and worse on Multi-IF at every turn; the obvious fix—mixing general instruction data—was not run, and interpolation only removes the cost by giving back the relapse gain.
  2. It is a prompt-level specification prepend rather than an external-ledger implementation, A6b also receives ground-truth state, and the construction/token-matching details are too thin to rule out an unfair comparison.
Questions for authors
  1. In Stage II, why are there many active-violation and relapse rejections if the filter already requires exact state match? What fraction of Stage-II rejections are state-correct but response-violating, and why was this signal not used until Stage III? .
  2. Is the DPO frozen reference the backbone rather than the Stage-II model? Please confirm the exact reference configuration and justify the choice.
Reviewer_Xx9pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The work tries to improve state tracking for instruction following in conversations. they use existing training recipe and first train to perform track rules which doesn't always translate to following them. They show that the real struggle is with rules that should not allow. They use DPO to prefer responses that follows all active rules and avoids using cancelled rules. They evaluate on short synthetic conversations that have formatting constraints. Additionally it leads to lower performance on broader instruction following benchmarks.

Critical strengths

They take a new route of following instruction along with state tracking, which is a nice idea. They perform experiments that show that just the state tracking is not useful. The work shows the need for state tracking to be combined with instruction following and identifies that most models fail with cancelled rules.

Critical weaknesses

The conversations used are kind of narrow, and short. They are also synthetically generated. Can you try to use these on StarV2 benchmark, or examples given in genieworksheets? Maybe the model follows more rules but is it able to complete the task?

There needs to be analysis on what kinds of cancellation rules are there. Sometimes cancellation can be ambiguous.

There is missing experiments on what would happen if you directly do DPO preference training with out state tracking. What would the numbers look like.

Also the performance actually decreases when you add not following cancellation rules training.

There is overfitting. I feel like the model is learning to track states for a given task, ideally it should generalize across tasks if the model was learning something useful. Additionally performance is worse on general instruction following tasks afterwards.

The submissions paper format is wrong.

Questions for authors

Look at weaknesses.

Additionally, why don't you provide confidence intervals for everything in table 2?

Where is arm A8 in the experiments, is it supposed to mean anything?

What is a tombstone? The paper you refer to ReBIND looks like AI slop.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.