acceptodds
Under review as a conference paper at ICLR 2027

PREDICTABLE BUT UNSTABLE : PROBING CONSTRAINT VIOLATIONS IN MULTI-TURN DIALOGUE

Abstract

In single-turn settings, linear probes over attention-head activations can predict whether a language model will violate an explicit constraint before generation. We extend this framework to multi-turn dialogue and find that mean AP does not systematically decline, yet probes become notably less stable when rebuilt from different development partitions—a pattern we term predictable but unstable. We identify two structural factors that may contribute: the first-violation task induces a survival structure that progressively thins supervision at later turns, even from an initial pool of 2,400 dialogues per configuration, and the relative construction stability of fixed attention-head banks varies across turns, so that a bank relatively stable at one turn may be less stable at another. These factors make probe construction sensitive to the turn composition of a development partition. We propose Frequency-Consensus head selection, which modifies only the aggregation rule while leaving the rest of the probing pipeline unchanged. On three 7B models, Frequency-Consensus improves model-averaged AP by 1.9 points over retaining all candidate heads while reducing cross-run SD by 37.5% and range by 34.8%, with the pattern extending to the evaluated 13–14B models.

Primary area:
interpretability and explainable AI
Keywords:
Large Language Model, Instruction Following, Constraint violation prediction, Attention head selection

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8DsdOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors study pre-generation prediction of the first violation of an explicit instruction in a multiturn conversation, conditional on the previous reponses remaining compliant. The main question that the paper looks at is whether the predictive performance remains reliable when the probing pipeline is reconstructed using different developmental partitions. The task uses first-event labels where compliant turns contribute negatives and the first violation is a positive, and subsequent turns are omitted. The main method proposed in the paper is frequency-consensus which fits logistic probes to individual attention head outputs at the final context token. Experiments are on scripted english dialogues per model-constraint configuration with up to 14 turns covering seven deterministically verifiable constraing families. The paper reports increased aggregated construction variability in matched single-turn vs multi-turn comparisons and against all-heads a 1.9 pp inmprovement.

Critical strengths
  1. The authors provide a useful distinction between predictive accuract and construction reliability.
  2. The table 2 results are interesting, and the authors compare with broad baselines in Table 3 and show that their method improves over the baselines.
  3. The authors describe the representation extracting before any token of the predicted response is generated, and excludes test observations from standardization, head selection and hyperparam selection, so there is no scope for potential leakage
Critical weaknesses
  1. The main improvemenets that the authors present are All-heads, and from Table 2 relative to Mean-AP, frequency-consensus reduces the AP SD by only 0.06 points on Gemma, 0.17 on llama and 0.06 on qwen, these improvements are very modest.
  2. In section 4.2 the authors discuss contributors to instability but the experiments do not isolate these. Such as expanding the training window, changing sample counts, class weighting and temporal coverage. ALso the 20 constructions re-partition one fixed development pool whereas the dialogue scripts, generated response labels and test set remain fixed, so the stability claim is limited in scope.
  3. The main comparisons report model level averages without per-constraint results or turn wise results, so it is hard to determine whether frequency-consensus helps broadly, or benefits late or rare events or only particular tasks.
Questions for authors

See weaknesses section above

Reviewer_eM35OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether a language model’s internal activations can predict its first violation of an explicit constraint before generating a response in multi-turn dialogue. The main observation is that average predictive performance does not systematically deteriorate, while performance becomes more sensitive to the development partitions used to construct the predictor. The authors investigate uneven supervision and turn-dependent head-bank stability as possible explanations.

They propose Frequency-Consensus, which selects attention heads based on how frequently they appear among the top-performing heads across repeated partitions. Experiments cover seven constraint families and five models. Compared with retaining all heads, the method improves average precision and reduces construction variability, although its advantage over other aggregation rules is smaller.

Critical strengths

What I find most interesting is the observation that a predictor can remain useful on average while becoming harder to construct consistently. This seems worth understanding, especially when a probe is going to be built using one development dataset and then reused. The effort to match class counts also makes the comparison more informative than simply comparing the original single-turn and multi-turn datasets.

I also appreciate the comparison between aggregation rules using the same repeated partitions and head-evaluation records. This helps separate the choice of aggregation rule from the benefit of repeated evaluation. Frequency-Consensus preserves mean AP and achieves the lowest reported SD across the three 7B models, with the clearest evidence of improvement on Llama.

Critical weaknesses

My first concern is how much additional benefit comes from the frequency rule itself. As the authors acknowledge in Appendix A, the procedure closely follows stability selection, so I think its empirical advantage over the other aggregation rules matters quite a bit. In Table 2, the improvement over Mean-AP is encouraging on Llama, but smaller on Gemma and Qwen, where the confidence intervals include zero improvement. This makes me less sure how broadly the headline reduction against All-Heads carries over to equally resampled alternatives. Could the authors provide per-task paired comparisons against Mean-AP and Mean-Rank, using the existing runs, to help clarify whether the improvements are spread across tasks or concentrated in a few configurations?

My second concern is about the explanation for the increased multi-turn variability. My understanding is that matching positive and negative observation counts does not necessarily match the amount of independent information, since several multi-turn observations can come from the same conversation. Grouped partitioning addresses leakage, but could differences in the number of independent conversations still explain part of the result? Could the authors report the number of distinct dialogue or script groups after matching and clarify whether the available evidence accounts for this difference? Similarly, Appendix E.6 explains that expanding the training window adds negative observations without adding positives, which makes me unsure how much Figure 3(b) tells us specifically about broader turn coverage. I may of course be missing a relevant control or interpretation here, and I would be very happy if the authors could help me understand which conclusions these experiments support.

Questions for authors

Please respond to the questions and concerns I have raised in the main review above.

Reviewer_Z9tXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper performs an empirical study of probing frozen attention-heads of a LLM during multi-turn dialogue conversations. The probe aims to predict the violation probability of any of the constraints given in the conversation. The authors mention a 'predictable but unstable' phenomenon - where the predictive performance of the probes show variability when their training datasets are built from different random partitions of the same dataset. The authors attribute this variability to two factors - lower supervision at later dialogue turns and the relative stability of attention-heads across the turns. They propose a approach for selecting the best (top K) attention heads - Frequency-Consensus, where repeated data partitions are performed on the same dataset, and the best attention heads are chosen based on the number of times they appeared in the top K attention heads in each partition. The experiments covered seven constraint families, three 7B models and two larger models, The experiments tracked the Average Precision scores and the standard deviations with partitioning of the data.

Critical strengths

Simple and Low-latency technique: The paper proposes using binary classifiers on frozen attention heads weights, which can be a simple and effective way of detecting constraint violation. Constraint violation can be one of commonly encountered problems in long muli-turn dialogues.

Thorough Experimentation on unstability: The authors performs various partition experiments on the datasets, and calculate the mean AP(average precision) and standard deviations. This showed the sensitivity of the scores to the turns in the dataset and also the attention heads chosen.

Simple Solution: The paper proposes Frequency-Consensus , which is a lightweight solution using binary inclusion votes across the different partitions, similar to reciprocal rank fusion, making it highly practical. The experiments show some empirical evidence in variance reduction with this approach

Critical weaknesses

Outdated models used for experimentation: The experiments are conducted on Gemma 7B, Llama-2 (7B/13B), and Qwen1.5 (7B/14B), These models do not incorporate the training data advancements and architectural improvements, so it uncertain that the scores and variability would hold for state of the art models.

Deterministic Constraints only - The paper focused on constraints focused on quotation markings, keyword mentions, word limits, etc. There is no approaches mentioned for broader semantic constraints such tonality of dialogue and safety checks.

Very modest gains of using Frequency-Consensus - The mean AP score differs by less than 0.1 points, and there is some standard deviation improvement for the Llama model, but this pattern is not across all the models, so it is uncertain if this approach will scale to variety of model architectures.

AP scores are still quite low for all the target models: The mean AP is ~ 50% for Gemma model, ~77% for Llama model and 62% for Qwen model. These score are quite low and cannot be used reliably in any production level violation detection systems.

Questions for authors

During Frequency-Consensus method, was there an experimental approach for selecting K for a model? K = 6 was chosen for Gemma and K = 8 for Llama.

Gemma-7B and Llama-7B have similar parameter sizes, yet the mean AP score for Gemma is 50.23 and Llama is 77.88. What could be the possible reasons for this ?

Can this work be extended to support non-deterministic semantic focused constraints such as tonality and safety ?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.