- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes an exact decomposition of a post-training behavioral change, measured on models’ own generated trajectories, into a change in reach—what the predecessor would already have done at the successor’s contexts—and a change in propensity—how the successor’s probability differs from the predecessor’s at matched contexts. Applied to a census of released instruction-tuning and reinforcement-learning transitions, excess-vocabulary lists, connective and ordinal classes, and response termination, the split shows that counted changes vary widely in composition, that reach often carries most or all of a rise, and that propensity can even move against the count; targeted local and context interventions corroborate these readings. Because the decomposition is an exact identity computable from teacher-forced passes over stored trajectories, the authors recommend reporting the change in propensity alongside every counted behavioral change, while noting that the result depends on the chosen event, trajectory reference, scoring window, generation cap, and rendering.
Critical strengths
The paper’s main strength is that it turns the fuzzy claim that post-training makes a model “behave differently” into a clean diagnostic: split a counted change into reach and propensity. It’s simple, exactly identified on stored trajectories, and cheap to compute, yet it sharply separates a genuine shift in inclination from simply reaching contexts where the behavior was already likely. The case studies make this concrete—most vividly Olmo-3, where connectives rise sharply while propensity moves the other way—showing the tool can yield genuinely new, often counterintuitive, insight.
Critical weaknesses
Here are the weaknesses I see. First, I still don’t quite see the practical payoff: even if I can tell whether a change is carried by reach or propensity, what would I actually do to fix a bad pattern? It tells me what the model is doing, but not really what to do about it. Second, it seems hard to extract a consistent conclusion—the three RL examples point in quite different directions. A good analytical tool should ideally surface a finding that was not visible before; SAE, for instance, revealed that MLP representations organize into various concepts. This tool doesn’t quite deliver that kind of payoff. Finally, if “First/Second” counts as reach, couldn’t “Let’s plan” be the propensity behind it? Then how should these feature tokens be chosen in the first place? It feels like a bit of a regress.
Questions for authors
NA
6 comments
Response to Reviewer Wutv (2/3)
Weakness 2: "Second, it seems hard to extract a consistent conclusion—the three RL examples point in quite different directions. A good analytical tool should ideally surface a finding that was not visible before; SAE, for instance, revealed that MLP representations organize into various concepts. This tool doesn’t quite deliver that kind of payoff."
The examples differ in which part carries the change, and that difference is the finding: a count cannot show it. What is consistent is how the split reads them. Four of the reinforcement-learning cases, each trained from a base checkpoint (p is the change in propensity over the counted change):
| Successor; event; prompts | Counted change per response | p | Carried by |
|---|---|---|---|
| Olmo-3-7B-RL-Zero-Code; connectives; general prompts | +0.20 | −0.78 | reach |
| Olmo-3-7B-RL-Zero-Code; connectives; coding problems | −0.501 | 0.96 | propensity |
| Llama-3.1-8B-SimpleRL-Zoo; connectives; general prompts | −0.368 | 0.567 | propensity |
| MiMo-7B-RL-Zero; "wait"; MATH-500 | +24.21 | 0.30 | reach |
The split reads these cases consistently:
- Where counts disagree, the split agrees. On the first two rows the count rises on one prompt set and falls on the other, and the change in propensity is negative on both: on the same text, the successor is less likely to open a line with a connective (Section 4.2).
- The reading is stable over training. At each of the 26 intermediate checkpoints of Olmo-3-7B-RL-Zero-Code, the count rises while the change in propensity is negative (Section 4.3).
- The reading predicts fixes. The sign of p gave the direction of the local intervention's direct part in all five cases (Section 5.3).
- On overused words, ρ gives one answer across the census. On all 38 census cases, the predecessor in the successor's contexts writes at least 0.64 of the successor's count (Section 4.1).
The split surfaces a structure that counts hide: a change in behavior is made of two different things. What a model does at a given state and which states it goes to are separate quantities, and post-training can move them in opposite directions. On the Olmo-3 connectives, the successor goes more often to states where a connective is likely, while at those states it is less likely to write one (Section 4.2). Over training the two drift further apart: from the first public checkpoint (step 400) to the released model, the fall in propensity grows 5.74 times while the count's rise stays where it was (0.196 and 0.202 per response; Section 4.3, Appendix G.2). The same reversal appears in word classes selected by a fixed rule on one half of the prompts and counted on the other, on the three SimpleRL-Zoo transitions (Appendix G.7), and on the stop event at the 2,048- and 8,192-token caps (Section 4.3). A count reads both as one change; they have to be thought of separately.
Separating them changes what published readings of counts say:
- Overused words. Juzek & Ward (2026) link the rise of overused words to learning from human feedback. On the closest released pair in our census, Llama-3.1-8B to Llama-3.1-8B-Instruct, the predecessor in the successor's contexts already writes 0.86–0.94 of the successor's count of the 75 of their words that we count: most of the rise is where the model goes, not what it does there (Section 4.1, Appendix F.2).
- A training method. Shao et al. read the rise in code reasoning under random rewards as training amplifying a behavior the model already had. The split agrees, and shows that training acted on the first response token, where 92.1% of random@150's responses open with "To" against 52.6% of the predecessor's, not on writing "python" at the same context (change in propensity −0.003, interval including zero; Section 6).
- Fixes. Under the NoWait ban on MiMo-7B-RL-Zero, the other reflection words become more likely at the same contexts (direct part +0.40 per response), yet fall overall, entirely through the contexts the ban changes (induced part −3.58; Section 6).
Response to Reviewer Wutv (3/3)
Weakness 3: "Finally, if “First/Second” counts as reach, couldn’t “Let’s plan” be the propensity behind it? Then how should these feature tokens be chosen in the first place? It feels like a bit of a regress."
"Let's plan" can be part of it, but only a part, and the split of "First/Second" does not depend on it. A change in reach is the effect of the successor's earlier choices, which decide the contexts it reaches (Section 1), and a context, the prompt and the response so far, is built from all of those choices. Equation (3) sums the changed choices at every earlier position and over every token, each weighted by how much it changes the later count. A more likely "Let's plan" is one term of that sum: Equation (16) splits each position's contribution into the change in propensity of an earlier event and two terms for which tokens fill that event and its complement (Appendix B.3). Section 3 states this directly: a change in propensity at an earlier event is part of the change in reach of a later one.
The two readings therefore answer two questions, and both can hold at once. "First/Second" rising through reach and "Let's plan" rising through propensity are splits of two different events, each exact (Equation (2)). Neither has to be settled before the other, so reading a split involves no regress: the change in reach of the counted event is measured directly, from the predecessor's counts on the two sets of responses, without naming any earlier token.
Following a change back is a separate step, for finding where to act, and it ends. It works by position, not by token: the predecessor continues the successor's responses from a chosen point, and its count measures how much of the change in reach the whole response before that point accounts for (Equation (17)). We then split the event at that point; if its change is carried by propensity we stop, and if it is carried by reach we go further back. The first response token depends only on the prompt, which both models share, so a change there is a change in propensity, and every trace ends there at the latest (Appendix B.3). On the Olmo-3 connectives, the first paragraph accounts for 0.80 [0.70, 0.89] of the change in reach, and the paragraph break that ends it is carried by propensity (p = 0.91), so the trace stops after one step (Section 4.2). Such a point shows where a change enters, not which token in the prefix causes it (Section 7), and acting there removed the whole rise on held-out prompts (Section 5.2).
Which tokens to count is set by the claim being tested; the split then answers that claim. Four of the paper's five kinds of behavior are ones that published studies count: a published list of overused words (Kobak et al., 2025), "wait", code reasoning measured by the fraction of responses containing "python" (Shao et al., 2026), and response length through the stop event. The fifth, the connectives, are the single-word members of a list of discourse connectives chosen by hand, and none was chosen by its change (Appendix E.3). To check that this choice does not make the result, on three SimpleRL-Zoo transitions we selected words by a fixed rule on one half of the prompts (their share of all tokens rose, while their changes in propensity at both references were at most zero) and counted them as one class on the other half. Every class shows the reversal at both references, and on 5 of the 6 it holds without the class's largest word (Section 4.2, Appendix G.7).
In summary, the split tells what to do about a bad pattern, which fix and where, and acting where it points reduced the gap on held-out prompts in every case by point estimate. It reads different transitions consistently, and it shows that what a model does and where it goes are separate and can move in opposite directions. An earlier event's change in propensity is one part of a later event's change in reach, so reading a split involves no regress; following a change back ends at the first response token at the latest; and the counted events come from the claims being tested.
The revision is substantially larger and stronger than the version you reviewed. It adds a new Section 6 that applies the split to a published training method and to published fixes, splits every public checkpoint of a training run, tests every located decision on held-out prompts, and rewrites Sections 3 and 5 around the results. Beyond the new empirical findings, the revision shows that the split is of practical use: it explains how published methods and fixes act, it tells which fix to apply and where, and the locations it points to were tested on held-out prompts, where acting there reduced the gap in every case by point estimate. Each concern in your review is addressed above and marked in blue in the paper. We hope the revision resolves your concerns, and we would be grateful if you would consider raising your score.
Response to Reviewer Wutv (1/3)
We thank the reviewer for the careful review. The General Response summarizes the changes common to all reviews. The revision changes and adds a great deal, and we would be very grateful if you would look at it; revised content is marked in blue in the paper.
What the model is doing decides what to do about it: the split diagnoses which fix to apply and where to apply it, and acting where it points reduced the gap on held-out prompts in every case by point estimate. A count shows only that a pattern changed. The split shows how, and that decides the fix.
The change in propensity tells whether a fix aimed at the behavior itself will help. Our local intervention, the obvious such fix, sets the successor's probability of the behavior to the predecessor's at every context and changes nothing else. Its direct part, its effect at the same contexts, narrows the gap where p > 0 and widens it where p < 0, unless the run reweights contexts by more than a margin computed from the successor's responses before the run. The margin was at least 2 in all five cases, and the direct part's share had the sign of p in every one (Section 5.3; Appendix J.2). The same fix therefore helps in one case and hurts in another:
The change in reach tells where to act. When reach carries the change, the split finds where it enters: the located decision, the earliest point at which handing the successor the predecessor's text removes at least half of the gap. We chose each located decision on one set of prompts and tested it on another:
After a fix is run, the split shows how it worked: whether it closed the gap at the same contexts or through the contexts it changed. Published fixes are evaluated by counts, length or accuracy, which report only the sum of the two; the split gives each (Section 6; an example under Weakness 2).