- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a metric to quantify the recurrence of behavioral patterns in an LLM's response, which is an axis distinct from measuring other forms of the quality of the response. This metric disentangles the relative position of the behavior from the sequential associations between different behaviors. The methodology for computing this metric can also be applied to quantify the recurrence of behavioral patterns throughout a conversation rather than merely within individual responses. The authors also explain how this metric can be standardized for use across different behavioral schemes.
The paper shows that for individual responses, nearly all of the explored LLMs exhibit similar recurrence of behavioral patterns, which is higher than the recurrence for human responses. For the conversation-level analysis, results show that LLMs behave consistently with humans relative to the user's state, and differ slightly with regard to the conversation turn.
The calculation of these metrics requires profiling the LLM, pooling the relative positions of each behavior and the transition from one behavior to another. Once such profiles are created, the authors highlight how they can be used to detect the LLM that generated a given response.
Critical strengths
- I think the proposed metric itself is useful and might be applied in different use cases. Similar to what the authors showed, it might be used for LLM watermarking, and it can also be used as an automatic reward that's applied during model training to encourage generating more diverse patterns (or more recurrent patterns, depending on the application).
- The method is mathematically rigorous to a good extent. An example of this is the disentanglement of choreography from momentum while also conditioning the momentum on choreography (this is how the measure of momentum can be isolated).
Critical weaknesses
-
Most of the results focus on showcasing the use of the proposed metric rather than validating whether the metric is reliable or not. The authors included a paragraph on that in the main body, but deferred all validation experiments to the appendix. Since the main contribution of the paper is the introduction of this metric, I think it's important to focus more on rigorously testing the validity and limits of this metric before showcasing its use. The experiments in the main body are only interpretable under the assumption that the metric works as intended.
-
Most of the results shown in the main body rely on annotations from a judge LLM. That's generally fine when the judge is validated, but the appendix shows that the judge agrees with human annotators for only 24% of the labels (within the correctly highlighted characters). This is a relatively weak agreement, and it only involves the LLM responses; it doesn't include the human responses. This limits the reliability of the results and conclusions from the conversation-level analysis in table 3 (the within-response results in table 2 also involve the judge, but the authors mention that these are similar to other results that were human-annotated, so those might be fine)
-
Based on the results in table 2, almost all models (except for claude) exhibit similar choreography and momentum (no statistical significance). Figure 3a shows that, given a response and several LLM profiles, one can detect the LLM that generated the response. Connecting both of these facts, it might be that the SCRIPT metric itself is limited in its ability to measure behavioral scriptedness and that a more representative metric can be computed from these profiles (the position and transition tables). Please let me know if I'm missing something in this interpretation.
-
(less critical but worth noting): the paper was hard to follow. For example, the purpose of the abstract is to give a summary of the work, but this work used a lot of new terms in the abstract (choreography, momentum), confusing the reader. The authors also mentioned conversational choreography (with no introduction to it) in one paragraph, described it in a later paragraph, and didn't end up using it in the main body. These are just examples suggesting that the manuscript needs substantial revision for better clarity.
-
(more of a comment than a weakness): The authors mention that this metric is reference-free. I think this is not worth claiming because, in all cases, one has to compare one model's SCRIPT metric to another model/human's metric (the absolute number isn't intuitive or meaningful), so "reference-free" doesn't really make sense.
Questions for authors
The authors use characters, not LLM tokens or words, to obtain the start of a span and subsequently compute the relative start position. Why is that the case?
5 comments
W3.
SCRIPT is already computed from the position and transition profiles:
- SCRIPT score: How strongly does a model follow a recurring pattern, beyond chance?
- SCRIPT profile: What does that pattern look like, and how does it differ between models?
1. Why can profiles distinguish models with similar SCRIPT scores?
Because models differ not only in how consistently they arrange behaviours, but also in which behaviours they use. SCRIPT controls for behaviour frequency by design, while profile-based identification uses both frequency and arrangement. For example, on held-out data (20% chance accuracy):
| Information used | 1 reply | 5 replies | 20 replies |
|---|---|---|---|
| Full profile (position + transitions) | 38% | 57% | 74% |
| Behaviour frequency only | 35% | 48% | 60% |
| Full profile, shuffled order | 34% | 50% | 62% |
2. Similar scores do not mean identical scripts.
Similar scores reflect a genuine similarity, not a lack of resolution. First, SCRIPT separates systems whenever their structure differs. It separates Claude from each of the other four models by Δ = 0.027–0.033 (Appendix B.5.1, Table B.23). When only prompting or training changes on a fixed task, positional choreography spreads the 22 Testbed 2 systems from C = 0.050 to 0.160, against 0.014 for human supporters (Appendix B.3, Table B.15). Second, the tie among the other four is measured precisely. Each score has a 95% interval of ±0.010–0.013 (Table B.23), so these models are scripted to the same degree within that precision.
W4.
We agree and will revise the manuscript to improve readability by:
- Simplifying the abstract and reducing technical terminology.
- Adding an intuitive example (empathy → advice → question) before introducing the equations.
- Defining C and M clearly: where behaviours occur and what follows what.
- Restructuring the paper: Present the metric and its validation first, followed by applications, and introduce conversation-level measures before using them.
These changes will make SCRIPT's novelty and findings easier to understand without sacrificing technical rigor.
W5.
We clarify that “reference-free” means SCRIPT does not require a gold-standard or human-written response, rather than that comparisons are unnecessary for interpreting its magnitude. SCRIPT is computed directly from a model's responses by comparing their behavioural arrangement with shuffled versions of those responses, while preserving behaviour frequencies. The score is meaningful on its own:
- SCRIPT ≈ 0 indicates no detectable structure beyond chance.
- SCRIPT > 0 indicates recurring structure, with the z-score quantifying statistical evidence.
Its magnitude is also interpretable. SCRIPT measures the additional share of behavioural uncertainty explained by position and preceding behaviour beyond the shuffled baseline. For example, SCRIPT = 0.10 indicates approximately 10% additional explained uncertainty (Appendix B.1). To provide further context, we normalize SCRIPT by a matched ceiling constructed from each corpus's existing behaviours. The five models reach 19–25% of this achievable rigidity (Appendix B.5.1).
Thus, human comparisons help contextualize the magnitude but are not required to compute or interpret SCRIPT. To avoid ambiguity, we will replace “reference-free” with “requires no reference response” throughout the manuscript.
Q1.
We use characters because they provide a consistent, precise measure of position across all models and human responses, without requiring a tokenizer. We chose characters for two reasons:
- Consistent across models: Different LLMs use different tokenizers, and three models in Testbed 1 are closed-source. Using one tokenizer would introduce an arbitrary choice. Characters work identically for all models and human therapists.
- Preserves behavioural order: Behaviours do not always align with sentence boundaries. In Testbed 1, 15% of highlights begin within a sentence, and 22% share a sentence with another highlight. Sentence-level positions can lose their order, while character positions preserve it.
New robustness test: Does the choice of unit change SCRIPT?
| Model | Characters | Words | Words + punctuation | Sentences |
|---|---|---|---|---|
| Claude | 0.118 | 0.117 | 0.115 | 0.073 |
| GPT | 0.098 | 0.097 | 0.098 | 0.072 |
| Gemini | 0.092 | 0.093 | 0.091 | 0.078 |
| Qwen | 0.091 | 0.092 | 0.092 | 0.075 |
| Llama | 0.083 | 0.084 | 0.083 | 0.085 |
Dear Reviewer,
Thank you for your time and review of our paper. We have provided a detailed rebuttal addressing all of your questions, along with several new analyses. We would greatly appreciate any further feedback and would be pleased to clarify any remaining concerns.
Thank you
We thank the reviewer for recognizing SCRIPT's contributions. To improve the clarity and accessibility of the methodology, we have also updated our project website (scriptmetric.com) with an interactive, step-by-step example demonstrating how SCRIPT is computed from a real conversation.
W1.
1. Every SCRIPT score includes a comparison with chance.
We compare real responses with 200 shuffled versions of the same responses and this comparison establishes reliability and validity of SCRIPT, preserving exactly which behaviours occur and how often, while changing their arrangement.
Thus, every score measures structure beyond what behaviour frequency and random ordering would explain. Table 2 reports the corresponding z-scores (22–44), and its random-label control returns approximately zero (−0.004).
2. Two new analyses (Total of six validation tests)
W2.
We tested Table 3 using three alternative annotation approaches, and the main findings remain consistent.
1. What does the 24% agreement mean?
The 23.6% exact-code agreement combines two types of labels:
2. Robustness checks: Do the findings change with different annotators?
We repeated Table 3 using only the six better-agreed codes and a position-blind rule-based annotator that labels each sentence independently.
The same pattern appears across all three approaches: Therapist advice decreases substantially as emotional intensity rises, while models change much less. Response position also remains more strongly associated with model behaviour than the coded user state.
3. New check: Can we reproduce the findings using only clinician annotations?
Since clinicians annotated all model responses, we independently repeated the model analysis without using the LLM judge.
Neither annotation approach shows the large reduction in advice observed for the therapist (−16 percentage points). This supports the conclusion that models adjust their advice less in response to increased emotional intensity.