Do LLMs Follow a Behavioral SCRIPT? Measuring Structural Choreography and Rigidity from Positions and Transitions
Abstract
Individual-response evaluations can rate an answer highly while failing to assess whether a model consistently organizes its behaviour in the same way across users. To measure this, we introduce **Script**, the **S**tructural **C**horeography and **R**igidity **I**ndex from **P**ositions and **T**ransitions, a reference-free measure of behavioural scriptedness from span annotations. Script separates positional choreography from sequential momentum and calibrates both against a within-response permutation baseline that preserves behaviour frequency while disrupting arrangement. We evaluate it on 4,095 responses to 819 counselling prompts from five frontier models with matched therapist responses, and on 22 controlled prompting and training configurations. All five models exhibit recurring structure above the permutation baseline (–), while the matched therapist shows lower positional choreography under the same annotation scheme. For the models, response position explains substantially more behavioural variation than coded user state or conversation stage, and models respond more strongly to explicit cues than to affective distress. At matched behaviour mix, existing quality evaluation shows no detectable sensitivity to arrangement, and reducing cross-turn tactic recurrence does not necessarily remove within-response choreography. These findings identify behavioural arrangement as a distinct evaluation dimension and provide a practical framework for measuring not only what models do, but where and in what sequence they repeatedly do it. We release all analyses and codes.
est. 32% chance this paper gets accepted at ICLR 2027.
What do you think this paper will get?
All positions stay anonymous.