acceptodds
Under review as a conference paper at ICLR 2027

What a Model Does Versus Where It Goes: Decomposing Post-Training Behavioral Change

16 views

Abstract

Post-training is often said to change a behavior, such as how often a model writes "delve" or opens a paragraph with "First", on the strength of counts in each model's own trajectories. Such a count mixes two effects. The post-trained model, the successor, may be more likely than its predecessor to produce the behavior in the same context: a change in propensity. Or its own earlier tokens may lead it into contexts where the behavior is common: a change in reach. For a given event, teacher-forcing the predecessor on the successor's trajectories splits the counted change exactly into these two changes. Across released transitions, from instruction tuning to reinforcement learning from a base model, the split varies widely, and the two changes can have opposite signs: Olmo-3-7B-RL-Zero-Code writes paragraph-opening connectives 4.7 times as often as its predecessor, yet at the successor's own contexts the predecessor assigns them slightly higher probability. The split also accounts for what two interventions do. Restoring the predecessor's probability of the behavior at every position (a local intervention) moves the behavior back toward the predecessor's rate on the transition where propensity dominates; on the two where reach dominates, giving the successor the predecessor's first paragraph (a context intervention) moves it back, and the local intervention does not. For response termination, the local intervention changes which contexts the successor reaches, and this induced change in reach cancels the change in propensity. We recommend reporting the split beside raw counts; it needs only the two checkpoints.

Primary area:
interpretability and explainable AI
Keywords:
post-training, language model behavior, behavioral evaluation, teacher forcing, causal mediation, lexical overuse

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_WutvOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an exact decomposition of a post-training behavioral change, measured on models’ own generated trajectories, into a change in reach—what the predecessor would already have done at the successor’s contexts—and a change in propensity—how the successor’s probability differs from the predecessor’s at matched contexts. Applied to a census of released instruction-tuning and reinforcement-learning transitions, excess-vocabulary lists, connective and ordinal classes, and response termination, the split shows that counted changes vary widely in composition, that reach often carries most or all of a rise, and that propensity can even move against the count; targeted local and context interventions corroborate these readings. Because the decomposition is an exact identity computable from teacher-forced passes over stored trajectories, the authors recommend reporting the change in propensity alongside every counted behavioral change, while noting that the result depends on the chosen event, trajectory reference, scoring window, generation cap, and rendering.

Critical strengths

The paper’s main strength is that it turns the fuzzy claim that post-training makes a model “behave differently” into a clean diagnostic: split a counted change into reach and propensity. It’s simple, exactly identified on stored trajectories, and cheap to compute, yet it sharply separates a genuine shift in inclination from simply reaching contexts where the behavior was already likely. The case studies make this concrete—most vividly Olmo-3, where connectives rise sharply while propensity moves the other way—showing the tool can yield genuinely new, often counterintuitive, insight.

Critical weaknesses

Here are the weaknesses I see. First, I still don’t quite see the practical payoff: even if I can tell whether a change is carried by reach or propensity, what would I actually do to fix a bad pattern? It tells me what the model is doing, but not really what to do about it. Second, it seems hard to extract a consistent conclusion—the three RL examples point in quite different directions. A good analytical tool should ideally surface a finding that was not visible before; SAE, for instance, revealed that MLP representations organize into various concepts. This tool doesn’t quite deliver that kind of payoff. Finally, if “First/Second” counts as reach, couldn’t “Let’s plan” be the propensity behind it? Then how should these feature tokens be chosen in the first place? It feels like a bit of a regress.

Questions for authors

NA

6 comments
AuthorsOpenReview ↗

Response to Reviewer Wutv (1/3)

We thank the reviewer for the careful review. The General Response summarizes the changes common to all reviews. The revision changes and adds a great deal, and we would be very grateful if you would look at it; revised content is marked in blue in the paper.

Weakness 1: "First, I still don’t quite see the practical payoff: even if I can tell whether a change is carried by reach or propensity, what would I actually do to fix a bad pattern? It tells me what the model is doing, but not really what to do about it."

What the model is doing decides what to do about it: the split diagnoses which fix to apply and where to apply it, and acting where it points reduced the gap on held-out prompts in every case by point estimate. A count shows only that a pattern changed. The split shows how, and that decides the fix.

The change in propensity tells whether a fix aimed at the behavior itself will help. Our local intervention, the obvious such fix, sets the successor's probability of the behavior to the predecessor's at every context and changes nothing else. Its direct part, its effect at the same contexts, narrows the gap where p > 0 and widens it where p < 0, unless the run reweights contexts by more than a margin computed from the successor's responses before the run. The margin was at least 2 in all five cases, and the direct part's share had the sign of p in every one (Section 5.3; Appendix J.2). The same fix therefore helps in one case and hurts in another:

  • On the Llama connectives (Llama-3.1-8B to Llama-3.1-8B-SimpleRL-Zoo), p = 0.609 [0.492, 0.767] on the responses the fix was run on, and the fix removes the gap by point estimate (share removed 1.33; Section 5.2).
  • On the paragraph-initial connectives (Olmo-3-1025-7B to Olmo-3-7B-RL-Zero-Code, grafted template), p = −0.084 [−0.101, −0.068], and the fix widens the gap: its direct part by 0.11 per response, and its whole effect too (share removed −0.09; Section 6, Tables 1 and 41).

The change in reach tells where to act. When reach carries the change, the split finds where it enters: the located decision, the earliest point at which handing the successor the predecessor's text removes at least half of the gap. We chose each located decision on one set of prompts and tested it on another:

  • On the Olmo-3 connectives, the located decision is the first paragraph. On held-out prompts, handing over the predecessor's first paragraph removed the whole rise (share removed 1.13 [1.02, 1.29]), while the fix aimed at the connectives themselves widened it (−0.76 [−0.96, −0.62]; Section 5.2, Appendix M).
  • On MATH-500, we located each decision on 100 problems and tested it on the other 400. Acting at the located decision removed a positive share of the gap in all 5 cases by point estimate, and by its interval in the 3 at the first response token (Section 5.3, Table 1, Appendix L.1).
  • On the released checkpoints of a published training method with random rewards (Shao et al., 2026), the split finds that training raised the probability of the first token leading to code, not of writing "python" at the same context. Acting at that first token removes the gap (Section 6, Table 1).

After a fix is run, the split shows how it worked: whether it closed the gap at the same contexts or through the contexts it changed. Published fixes are evaluated by counts, length or accuracy, which report only the sum of the two; the split gives each (Section 6; an example under Weakness 2).

AuthorsOpenReview ↗

Response to Reviewer Wutv (2/3)

Weakness 2: "Second, it seems hard to extract a consistent conclusion—the three RL examples point in quite different directions. A good analytical tool should ideally surface a finding that was not visible before; SAE, for instance, revealed that MLP representations organize into various concepts. This tool doesn’t quite deliver that kind of payoff."

The examples differ in which part carries the change, and that difference is the finding: a count cannot show it. What is consistent is how the split reads them. Four of the reinforcement-learning cases, each trained from a base checkpoint (p is the change in propensity over the counted change):

Successor; event; promptsCounted change per responsepCarried by
Olmo-3-7B-RL-Zero-Code; connectives; general prompts+0.20−0.78reach
Olmo-3-7B-RL-Zero-Code; connectives; coding problems−0.5010.96propensity
Llama-3.1-8B-SimpleRL-Zoo; connectives; general prompts−0.3680.567propensity
MiMo-7B-RL-Zero; "wait"; MATH-500+24.210.30reach

The split reads these cases consistently:

  • Where counts disagree, the split agrees. On the first two rows the count rises on one prompt set and falls on the other, and the change in propensity is negative on both: on the same text, the successor is less likely to open a line with a connective (Section 4.2).
  • The reading is stable over training. At each of the 26 intermediate checkpoints of Olmo-3-7B-RL-Zero-Code, the count rises while the change in propensity is negative (Section 4.3).
  • The reading predicts fixes. The sign of p gave the direction of the local intervention's direct part in all five cases (Section 5.3).
  • On overused words, ρ gives one answer across the census. On all 38 census cases, the predecessor in the successor's contexts writes at least 0.64 of the successor's count (Section 4.1).

The split surfaces a structure that counts hide: a change in behavior is made of two different things. What a model does at a given state and which states it goes to are separate quantities, and post-training can move them in opposite directions. On the Olmo-3 connectives, the successor goes more often to states where a connective is likely, while at those states it is less likely to write one (Section 4.2). Over training the two drift further apart: from the first public checkpoint (step 400) to the released model, the fall in propensity grows 5.74 times while the count's rise stays where it was (0.196 and 0.202 per response; Section 4.3, Appendix G.2). The same reversal appears in word classes selected by a fixed rule on one half of the prompts and counted on the other, on the three SimpleRL-Zoo transitions (Appendix G.7), and on the stop event at the 2,048- and 8,192-token caps (Section 4.3). A count reads both as one change; they have to be thought of separately.

Separating them changes what published readings of counts say:

  • Overused words. Juzek & Ward (2026) link the rise of overused words to learning from human feedback. On the closest released pair in our census, Llama-3.1-8B to Llama-3.1-8B-Instruct, the predecessor in the successor's contexts already writes 0.86–0.94 of the successor's count of the 75 of their words that we count: most of the rise is where the model goes, not what it does there (Section 4.1, Appendix F.2).
  • A training method. Shao et al. read the rise in code reasoning under random rewards as training amplifying a behavior the model already had. The split agrees, and shows that training acted on the first response token, where 92.1% of random@150's responses open with "To" against 52.6% of the predecessor's, not on writing "python" at the same context (change in propensity −0.003, interval including zero; Section 6).
  • Fixes. Under the NoWait ban on MiMo-7B-RL-Zero, the other reflection words become more likely at the same contexts (direct part +0.40 per response), yet fall overall, entirely through the contexts the ban changes (induced part −3.58; Section 6).
AuthorsOpenReview ↗

Response to Reviewer Wutv (3/3)

Weakness 3: "Finally, if “First/Second” counts as reach, couldn’t “Let’s plan” be the propensity behind it? Then how should these feature tokens be chosen in the first place? It feels like a bit of a regress."

"Let's plan" can be part of it, but only a part, and the split of "First/Second" does not depend on it. A change in reach is the effect of the successor's earlier choices, which decide the contexts it reaches (Section 1), and a context, the prompt and the response so far, is built from all of those choices. Equation (3) sums the changed choices at every earlier position and over every token, each weighted by how much it changes the later count. A more likely "Let's plan" is one term of that sum: Equation (16) splits each position's contribution into the change in propensity of an earlier event and two terms for which tokens fill that event and its complement (Appendix B.3). Section 3 states this directly: a change in propensity at an earlier event is part of the change in reach of a later one.

The two readings therefore answer two questions, and both can hold at once. "First/Second" rising through reach and "Let's plan" rising through propensity are splits of two different events, each exact (Equation (2)). Neither has to be settled before the other, so reading a split involves no regress: the change in reach of the counted event is measured directly, from the predecessor's counts on the two sets of responses, without naming any earlier token.

Following a change back is a separate step, for finding where to act, and it ends. It works by position, not by token: the predecessor continues the successor's responses from a chosen point, and its count measures how much of the change in reach the whole response before that point accounts for (Equation (17)). We then split the event at that point; if its change is carried by propensity we stop, and if it is carried by reach we go further back. The first response token depends only on the prompt, which both models share, so a change there is a change in propensity, and every trace ends there at the latest (Appendix B.3). On the Olmo-3 connectives, the first paragraph accounts for 0.80 [0.70, 0.89] of the change in reach, and the paragraph break that ends it is carried by propensity (p = 0.91), so the trace stops after one step (Section 4.2). Such a point shows where a change enters, not which token in the prefix causes it (Section 7), and acting there removed the whole rise on held-out prompts (Section 5.2).

Which tokens to count is set by the claim being tested; the split then answers that claim. Four of the paper's five kinds of behavior are ones that published studies count: a published list of overused words (Kobak et al., 2025), "wait", code reasoning measured by the fraction of responses containing "python" (Shao et al., 2026), and response length through the stop event. The fifth, the connectives, are the single-word members of a list of discourse connectives chosen by hand, and none was chosen by its change (Appendix E.3). To check that this choice does not make the result, on three SimpleRL-Zoo transitions we selected words by a fixed rule on one half of the prompts (their share of all tokens rose, while their changes in propensity at both references were at most zero) and counted them as one class on the other half. Every class shows the reversal at both references, and on 5 of the 6 it holds without the class's largest word (Section 4.2, Appendix G.7).

In summary, the split tells what to do about a bad pattern, which fix and where, and acting where it points reduced the gap on held-out prompts in every case by point estimate. It reads different transitions consistently, and it shows that what a model does and where it goes are separate and can move in opposite directions. An earlier event's change in propensity is one part of a later event's change in reach, so reading a split involves no regress; following a change back ends at the first response token at the latest; and the counted events come from the claims being tested.

The revision is substantially larger and stronger than the version you reviewed. It adds a new Section 6 that applies the split to a published training method and to published fixes, splits every public checkpoint of a training run, tests every located decision on held-out prompts, and rewrites Sections 3 and 5 around the results. Beyond the new empirical findings, the revision shows that the split is of practical use: it explains how published methods and fixes act, it tells which fix to apply and where, and the locations it points to were tested on held-out prompts, where acting there reduced the gap in every case by point estimate. Each concern in your review is addressed above and marked in blue in the paper. We hope the revision resolves your concerns, and we would be grateful if you would consider raising your score.

Reviewer_MgCfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper argues that a behavior becoming more frequent after post-training does not necessarily mean the model is more likely to produce it at the same context. It may instead generate contexts that encourage that behavior. By teacher-forcing the predecessor on the successor’s trajectories, the authors exactly separate frequency changes into propensity and reach components. Analyses of released model transitions show that these components vary widely and can have opposite signs. Local probability and prefix interventions further show why restoring local probabilities may not restore overall behavior rates. The main contribution is a practical framework for interpreting post-training behavior beyond raw counts.

Critical strengths

The main strength is separating changes in what a model does at the same context from changes in the contexts it generates. This gives a clearer picture of what post-training changes than output frequencies alone. The cases where the two components have opposite signs are particularly useful, since they show that raw counts can genuinely mislead our interpretation. Even though the decomposition itself is simple, I see value in it as a practical tool for understanding behavioral changes after post-training.

Critical weaknesses

My main reservation is that the paper currently offers a finding about how to interpret behavioral changes, without sufficiently showing how that finding can help improve models. The intervention studies provide useful examples, but do not yet establish reusable guidance for choosing training or inference changes that reliably help. While I find the analytical perspective useful, I remain uncertain about its broader impact beyond explaining the observed behavior.

Questions for authors

How can this decomposition help us improve a model? For example, if increased connective use mainly comes from changes in earlier context, should we change the training data, adjust the prompt, or intervene on the opening paragraph?

4 comments
AuthorsOpenReview ↗

Response to Reviewer MgCf (1/2)

We thank the reviewer for the careful review. The General Response summarizes the changes common to all reviews. The revision changes and adds a great deal, and we would be very grateful if you would look at it; revised content is marked in blue in the paper.

Weakness: "… without sufficiently showing how that finding can help improve models. The intervention studies provide useful examples, but do not yet establish reusable guidance for choosing training or inference changes that reliably help. … I remain uncertain about its broader impact beyond explaining the observed behavior."

The split helps improve a model by diagnosing the change: it tells which fix to apply and where to apply it, and acting where it points reduced the gap on held-out prompts in every case by point estimate. A count shows only that a behavior changed. The split shows how, and that decides the fix.

The change in propensity tells whether a fix aimed at the behavior itself will help. Our local intervention, the obvious such fix, sets the successor's probability of the behavior to the predecessor's at every context and changes nothing else. Its direct part, its effect at the same contexts, narrows the gap where p > 0 and widens it where p < 0, unless the run reweights contexts by more than a margin computed from the successor's responses before the run. The margin was at least 2 in all five cases, and the direct part's share had the sign of p in every one (Section 5.3; Appendix J.2). The same fix therefore helps in one case and hurts in another:

  • On the Llama connectives (Llama-3.1-8B to Llama-3.1-8B-SimpleRL-Zoo), p = 0.609 [0.492, 0.767] on the responses the fix was run on, and the fix removes the gap by point estimate (share removed 1.33; Section 5.2).
  • On the paragraph-initial connectives (Olmo-3-1025-7B to Olmo-3-7B-RL-Zero-Code, grafted template), p = −0.084 [−0.101, −0.068], and the fix widens the gap: its direct part by 0.11 per response, and its whole effect too (share removed −0.09; Section 6, Tables 1 and 41).

The change in reach tells where to act. When reach carries the change, the split finds where it enters: the located decision, the earliest point at which handing the successor the predecessor's text removes at least half of the gap. We chose each located decision on one set of prompts and tested it on another:

  • On the Olmo-3 connectives, the located decision is the first paragraph, and acting there on held-out prompts removed the whole rise, while the local intervention widened it (see the answer to your question).
  • On MATH-500, we located each decision on 100 problems and tested it on the other 400. Acting at the located decision removed a positive share of the gap in all 5 cases by point estimate, and by its interval in the 3 at the first response token (Section 5.3, Table 1, Appendix L.1).
  • On the released checkpoints of a published training method with random rewards (Shao et al., 2026), the split finds that training raised the probability of the first token leading to code, not of writing "python" at the same context. Acting at that first token removes the gap (Section 6, Table 1).

After a fix is run, the split shows how it worked: whether it closed the gap at the same contexts or through the contexts it changed. Published fixes such as bans of reflection words (NoWait) are evaluated by counts, length or accuracy, which report only the sum of the two. On MiMo-7B-RL-Zero, the NoWait ban raises the other reflection words at the same contexts (direct part +0.40 per response), and the words' fall comes entirely through the contexts the ban changes (induced part −3.58; Section 6).

AuthorsOpenReview ↗

Response to Reviewer MgCf (2/2)

Question: "How can this decomposition help us improve a model? For example, if increased connective use mainly comes from changes in earlier context, should we change the training data, adjust the prompt, or intervene on the opening paragraph?"

In your example, the split's diagnosis is to act on the opening paragraph rather than on the connectives, and acting there removes the whole rise. Your example is the Olmo-3 connectives: their count rises through reach, while at the same contexts the successor is less likely to open a line with a connective.

The diagnosis. When the predecessor continues the successor's responses after their first paragraph, that paragraph accounts for 0.80 [0.70, 0.89] of the change in reach (Section 4.2). Handing the successor the predecessor's first token removes 0.258 [0.166, 0.351] of the gap, its first sentence 0.436 [0.331, 0.544] and its first paragraph 1.013 [0.914, 1.128], so the first paragraph is the located decision (Appendix M).

The test on held-out prompts. Setting the connectives' probability to the predecessor's widened the gap (share removed −0.76 [−0.96, −0.62]), while handing the successor the predecessor's first paragraph removed all of it (1.13 [1.02, 1.29]), at a cost of 3.2 [0.6, 5.8] points of HumanEval+ pass@1 (Section 5.2, Appendix M). A count, which shows only that connectives rose, points to the connectives themselves, and the fix aimed there widened the gap.

The same diagnosis applies to any behavior (Section 6). Split the change. Whatever carries it, the sign of p tells whether the local fix will narrow the gap at the same contexts. If reach carries it, the located decision tells where any fix, in training or at inference, should act. Running the fix then measures its whole effect and its cost.

In summary, the split improves a model by telling which fix to apply and where to apply it. Acting where it points reduced the gap on held-out prompts in every case by point estimate, including your connective example, and Section 6 applies it to a published training method and to published fixes.

The revision is substantially larger and stronger than the version you reviewed. It adds a new Section 6 that applies the split to a published training method and to published fixes, splits every public checkpoint of a training run, tests every located decision on held-out prompts, and rewrites Sections 3 and 5 around the results. Beyond the new empirical findings, the revision shows that the split is of practical use: it explains how published methods and fixes act, it tells which fix to apply and where, and the locations it points to were tested on held-out prompts, where acting there reduced the gap in every case by point estimate. Each concern in your review is addressed above and marked in blue in the paper. We hope the revision resolves your concerns, and we would be grateful if you would consider raising your score.

Thank you again for your review. As the discussion period ends on November 18 (AoE), we would be grateful to know whether our response and the revised paper address your concerns. We are happy to clarify anything before the deadline.

As the discussion period closes in less than 24 hours (November 18, AoE), we wanted to thank you again for your review and send a gentle reminder about our response. Please do not hesitate to let us know if you have any remaining questions or concerns.

Reviewer_mH31OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The main goal of this paper is to isolate two different metrics that contribute to the behavioral changes attributed to post training. The first is propensity, which measures how likely a model performs a behavior in the same context, and reach, which measures where the model goes, or the contexts the earlier tokens navigate to during text generation. To measure these metrics the authors use teacher forcing to evaluate a model's behavior on the trajectories generated by another model. Specifically, the authors feed the model a predetermined trajectories sampled from a different model and evaluate the current model's next-token probability at every scored position along that sequence. They analyze ten model transitions, and found that behavioral changes in a model are often driven by reach rather than propensity, sometimes even moving in opposite directions, as demonstrated by the Olmo-3 model's 4.7-fold increase in paragraph-opening connectives. To validate this split, the authors tested live interventions during generation. They proved that modifying local token probabilities only reverts behavior if propensity drives the change, whereas swapping upstream text is required when reach dominates. Their experiments confirm that these two mechanisms independently influence model behavior, while changes in one can be offset by changes in the other. The authors conclude that both propensity and reach should be measured, rather than relying solely on raw behavioral counts, to accurately understand the effects of post-training.

Critical strengths

One of the strengths of the paper is the breadth of the experimental setup. The use 10 transitions spanning across 8 major model families is a robust and having results that are consistent across the different configurations demonstrates that the gap between raw counts and actual propensity appears consistently across reinforcement learning. Furthermore, this breadth of evaluation strengthens the generalizability of the paper’s conclusions, showing that the findings are not specific to a particular model or experimental setup, but instead hold across a diverse range of models and configurations.

Another strength is how heavily the authors stress-test their methodology. They prove the reach vs. propensity split isn't just a quirk of the evaluation setup. For example, they test against different prompt distributions by swapping model-written prompts for real human-written ones. They also scale the generation caps from 384 tokens all the way up to 8,192 tokens. Even at that extreme length, where the raw count of response termination flips, the underlying change in propensity keeps its negative sign. This setup makes the core claims highly convincing, showing that the metric isolates a true behavioral shift rather than just an artifact of the testing parameters.

Overall, the robustness and breadth of the experimental setup are major strengths of the paper. By demonstrating consistent findings across diverse model families, configurations, etc. the authors provide strong support for the validity of their methodology and increase confidence in the conclusions drawn.

Critical weaknesses

One of the biggest weaknesses of the paper is the writing and structure. The paper feels somewhat fragmented. Specifically it is difficult to follow the main argument without constantly referring back to the supplementary materials. For example, the context needed to interpret the intervention results in Figure 3 is almost entirely discussed in table 47 of the appendix, with very little explanation in the main text. The current structure makes it harder for readers to understand and evaluate the findings and moving more of this information to the main text would make the paper easier to follow.

Another concern is how the authors evaluate older base models. For base-to-instruct transitions, the authors "teacher-force under the chat-template rendering, whose control tokens they may not have been trained on". While they acknowledge this limitation and suggest that the observed differences could reflect a "register difference" rather than a genuine behavioral change, it raises an important concern about the reliability of the results. If base models are evaluated using a template they were never trained on, the evaluation itself could introduce off distribution results that could effect the results. This makes it difficult to determine if the observed reach and propensity differences are due to actual changes in model behavior or simply differences in how the models respond to unfamiliar formatting.

Questions for authors

In Section 7, you mention several limitations, including the mathematical shortcomings of the propensity metric across different baselines and the potential differences when prompting base models. Why don't these limitations fundamentally undermine the reliability of the core reach vs. propensity findings?

Also you mention that the teacher forcing base models on chat templates introduces out of distribution artifacts, which may just reflect a "register difference". What specific evidence ensures that the reported splits measure genuine behavioral changes rather than simply capturing the base model's failure to handle unfamiliar formatting?

11 comments
AuthorsOpenReview ↗

Response to Reviewer mH31 (1/3)

We thank the reviewer for the careful review. The General Response summarizes the changes common to all reviews. The revision changes and adds a great deal, and we would be very grateful if you would look at it; revised content is marked in blue in the paper. The review's summary describes the original claim about the interventions; the revision states it in a corrected form (General Response, point 5).

Weakness 1: "… it is difficult to follow the main argument without constantly referring back to the supplementary materials. For example, the context needed to interpret the intervention results in Figure 3 is almost entirely discussed in table 47 of the appendix …"

We agree, and we rewrote the paper so that its main argument can be followed without the appendix.

  • Interventions. The original Figure 3 is replaced by Table 1 (p. 9), which gives each case in one row, and Sections 3 and 5.1 define every quantity the table uses. The results no longer need the original Table 47 to be read. (The Qwen2.5-7B local intervention in the original Figure 3 is withdrawn; General Response, point 4.)
  • Measurement. Section 3 defines the terms of the split and its three validity checks in one place, and Section 4 states the setting behind its numbers at the start.
  • Appendices. They now hold the per-case tables and derivations, and the main text points to them where they are needed.

Weakness 2: "… If base models are evaluated using a template they were never trained on, the evaluation itself could introduce off distribution results …"

The original submission ran this test in its appendix (Appendices F.1 and F.5, and Table 26 for the Olmo-3 connectives). The revision makes it validity check (iii) in Section 3, recomputed per response, and the sign of the change in propensity still holds. The original main text reported the census's change in propensity under the template and listed the concern among its limitations. The check scores the predecessor with the prompt text alone (the bare rendering), the format a base model is trained on.

The template affects the predecessor in two places: when it reads the successor's responses, and when it writes its own.

  • Reading. At the successor's reference, the change in propensity and ρ use the predecessor only to read the successor's responses, so the template enters them only as the prompt in front of those responses. Removing it barely changes the predecessor's count on them: by at most 0.028 [0.015, 0.041] listed words per response on model-written prompts and 0.022 [0.014, 0.031] on human-written prompts, and by at most 0.168 on the two largest Qwen2.5 models. The sign of the change in propensity is unchanged on 7 of 8 transitions with model-written prompts and 9 of 10 with human-written prompts. On the one exception, Qwen2.5-14B-Instruct, the change in propensity is a small share of the rise either way (p = 0.09 with the template, −0.12 bare, on model-written prompts). Its rise is carried by reach under both renderings, and ρ is 0.96 and 1.04, so the predecessor writes about as many listed words as the successor at the same contexts (Appendix D.4; Tables 8, 18 and 19).
  • Writing. The predecessor's own responses enter the counted change, the change in reach and the fold, and here the template does change what the predecessor writes. The paper therefore reports each of these with its rendering. With the predecessor writing bare as well, the change is still carried by reach on 7 of 8 and 8 of 10 transitions, and on all 38 census cases, both renderings included, the predecessor in the successor's contexts still writes at least 0.64 of the successor's count (Section 4.1, Table 20).
AuthorsOpenReview ↗

Response to Reviewer mH31 (2/3)

Question 1: "… the mathematical shortcomings of the propensity metric across different baselines and the potential differences when prompting base models. Why don't these limitations fundamentally undermine the reliability of the core reach vs. propensity findings?"

Neither limitation overturns the core findings. We read "baselines" as the split's two references, the sets of responses whose contexts are held fixed, and answer that part here. The concern about prompting base models is answered by the bare check under Weakness 2 and by the four checks under Question 2.

The two references measure the same change in two sets of contexts, and when they disagree, it is the change that differs, not the measurement that fails. The successor's reference asks whether the predecessor, placed in the successor's contexts, would act as the successor does; the predecessor's reference asks the same in the predecessor's contexts. The split is exact at either. Equation (4) gives the exact difference between the two changes in propensity: they can differ in sign only when the change itself has opposite signs at contexts the two models visit at different rates. Suppose post-training raises a behavior at some contexts and lowers it at others. Each reference averages the same change over its own contexts, so if the successor visits mostly the first kind and the predecessor mostly the second, the two signs differ. Both are correct, and together they show what a count cannot: that the change is not the same everywhere.

The core findings hold at both references:

  • On the Olmo-3 connectives, the count rises while the change in propensity is −0.16 [−0.17, −0.14] at the successor's reference and −0.07 [−0.08, −0.07] at the predecessor's, and both stay negative at each of the 26 intermediate checkpoints (Sections 4.2 and 4.3, Appendix G.1).
  • On coding problems, the change in propensity accounts for most of the connectives' fall at both references (p = 0.96 and 0.71; Section 4.2). On Llama-3.1-8B-SimpleRL-Zoo, the connectives' fall is carried by propensity at both references (p = 0.567 and 0.589; Section 4.3).
  • In all five local-intervention cases, the change in propensity keeps its sign at the predecessor's reference (Section 5.3).
  • On the census, the change is carried by reach on 15 of 18 model-written and 16 of 20 human-written cases at the successor's reference, and on 14 and 18 at the predecessor's; cases whose labels differ are printed with both (Appendix F.2).

The original submission listed the sign change among its limitations, stated the reason in one sentence and gave the identity only in an appendix, as a residual. The revision states the identity in Section 3 as Equation (4) (p. 4) and reports both references in the main results.

AuthorsOpenReview ↗

Response to Reviewer mH31 (3/3)

Question 2: "… What specific evidence ensures that the reported splits measure genuine behavioral changes rather than simply capturing the base model's failure to handle unfamiliar formatting?"

We show this with four checks. In each, the sign of the change in propensity holds, with the one exception of the first check noted below.

First, the bare check above. With the template removed, the sign holds on 7 of 8 and 9 of 10 census transitions, and on the one exception the change in propensity is a small share of the rise under both renderings.

Second, the reversal on the Olmo-3 connectives holds under three renderings (Section 4.2; Appendix G.1, Table 22). Connectives per response over the first 384 tokens, Olmo-3-1025-7B to Olmo-3-7B-RL-Zero-Code:

Rendering, promptsPredecessor's count on its own responsesCounted changeChange in propensity, successor's referenceChange in propensity, predecessor's reference
Released format, model-written1.020.20−0.16−0.07
Grafted template, model-written0.2141.066−0.092−0.008
Released format, human-written0.700.19−0.15−0.09
Grafted template, human-written0.200.69−0.08−0.01
Predecessor bare, human-written0.530.36−0.13−0.07

Every interval in the last three columns excludes zero. The rendering moves the predecessor's count on its own responses from 1.02 to 0.214, and the counted change moves with it, but the change in propensity is negative at both references in every row. What the change in propensity at the successor's reference uses, the predecessor's count on the successor's responses, hardly moves: removing the template takes it from 0.97 to 1.02, while the count on the predecessor's own responses goes from 0.20 to 0.53 (Table 22, rows 6 and 7). The format moves the count, not the sign of the change in propensity, and this is why the paper reads behavior through the latter.

Third, the same results appear with no chat template at all. Llama-3.1-8B, Mistral-7B-v0.1 and their SimpleRL-Zoo successors all read the same plain question-and-answer prompt (Appendix E.4). There, word classes chosen on separate prompts show the reversal at both references (Appendix G.7), and the fall of the Llama connectives is carried by propensity at both references (Section 4.3).

Fourth, the predecessor is not simply copying at contexts it would not reach itself. This addresses the off-distribution part of the concern. On the fifth of the successor's responses that the predecessor finds relatively most likely, where it has least need to copy, the change in propensity keeps its sign on the Olmo-3 connectives under both formats and on "wait" for MiMo-7B-RL-Zero and Olmo-3-7B-RL-Zero-Math (Section 3, check (ii); Appendix D.4).

The "register difference" in the original submission concerned the predecessor's own responses, which enter the counted change and the change in reach but not the change in propensity at the successor's reference. It sat next to the sentence on scoring base models under a template, which does concern the change in propensity and is what the bare check now tests. The template's effect on the predecessor's own count is also why the headline now uses the released format, where the reversal is stronger (General Response, point 1). The 4.7-fold rise in the review's summary was the original grafted-template fold; with the corrected list and unit it is 5.97 under the grafted template and 1.20 under the released format.

In summary, the intervention results can now be read from the main text, with every case in Table 1. The template concern is a main-text validity check, backed by the reversal under three renderings and by transitions with no chat template. The two references are explained by Equation (4) and reported side by side.

The revision is substantially larger and stronger than the version you reviewed. It adds a new Section 6 that applies the split to a published training method and to published fixes, splits every public checkpoint of a training run, tests every located decision on held-out prompts, and rewrites Sections 3 and 5 around the results. Beyond the new empirical findings, the revision shows that the split is of practical use: it explains how published methods and fixes act, it tells which fix to apply and where, and the locations it points to were tested on held-out prompts, where acting there reduced the gap in every case by point estimate. Each concern in your review is addressed above and marked in blue in the paper. We hope the revision resolves your concerns, and we would be grateful if you would consider raising your score.

Reviewer_r5PqOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that the standard way of measuring post training behavioral shifts, where the post trained model is directly compared to the base model, conflates two different effects. To decompose this, the authors define two values called "reach" and "propensity". Propensity is defined as how much more or less likely the successor (the post trained model) is than the predecessor (usually the base model) to produce the behavior in the same context, and is measured by taking the successor’s frequency on its own outputs and subtracting the predecessor’s frequency on the successor’s outputs. Reach is how the successor's own earlier tokens led it into contexts where the behavior is common, and is measured by taking the predecessor’s frequency on the successor’s outputs and subtracting the predecessor’s frequency on its own outputs. The authors test 10 different model pairs, and find that models are all over the spectrum, from mostly reach to mostly propensity, and in some cases propensity runs opposite to the count. To check whether the propensity/reach split can help predict what reverses the behavioral shift, the authors test two intervention techniques. The first intervention sets the successor’s probability of the behavior at every position to the predecessor’s, while the second intervention gives the successor the predecessor’s first paragraph as context. They find that when propensity dominates, the first intervention sometimes moves the behavior back. The second intervention is found to be more effective when reach dominates. The authors suggest reporting the change in propensity alongside other reported values, to provide better clarity of model behavioral shifts.

Critical strengths
  • The paper's core idea is clearly presented, and the motivation behind the idea is also presented well. The theory behind propensity and reach is well-grounded, as the decomposition requires no modeling assumptions and connects well to established methods (e.g. Oaxaca/Kitagawa decompositions), which makes the concept easy to understand. Calculating these values is also cheap in terms of compute, as it does not require any additional training -- it only requires the two checkpoints and teacher-forcing passes, making it easy to adopt.
  • The method is also properly demonstrated on the headlining Olmo3 model, where the authors found that RL post training resulted in a 4.7x rise in writing specific connectives, but resulted in a negative gain in propensity when compared to the predecessor on the successor's contexts. The raw number would not have highlighted this effect.
Critical weaknesses
  • The paper suggests that propensity be adopted as an additional metric alongside any counted change. However, the propensity values reported here are setup-specific, and a variation in setup parameters can change the sign of the resulting propensity values (e.g., Qwen2.5-7B: −0.20 under the successor reference vs. +0.30 under the predecessor reference). The paper defaults to the successor reference but gives limited guidance on how to interpret results when references disagree, and doesn't show that its default generation cap and rendering are generally appropriate. Also, there is limited evidence for generalization (however, this is acknowledged by the authors): each transition is a single released checkpoint, and the RL-from-base transitions all use GRPO-family algorithms (per the models’ technical reports), so robustness across seeds and RL algorithms is not tested. Additional experiments of this nature could make the findings more robust.
  • Regarding the intervention experiments, they do not seem to fully establish a causal mechanism as tested. As an example, replacing the first paragraph (the context intervention) may change much more than paragraph structure or connective frequency. Although a paragraph from a different problem retains most of the effect (0.93 of the own-problem rate), suggesting topic is not the main driver, it seems like it can still change things like the reasoning process, response organization, etc. This is also suggested by the authors’ controls: in several experiments, the control class moves alongside the counted class. This makes it hard to draw conclusive results from this type of intervention.
  • Reach is largely a reflection of the contexts the successor reaches as a result of its earlier behavior. As the paper notes, a propensity change at an earlier position enters the reach of an event counted later. As a result, the split depends on which event is counted, i.e. the same underlying change can be classified as propensity for one event and reach for another. It would be helpful to be able to more cleanly separate what a model does from where it goes.
Questions for authors
  • What exactly does it mean to say that post-training primarily changed propensity rather than reach, and vice versa? Section 3.1 briefly discusses how the split connects to performance difference lemma, but are you able to elaborate how exactly it should be interpreted from an RL perspective? Could it help us diagnose when RL post training is not doing what we want?
  • How does the propensity/reach split vary across checkpoints? For the headlining Olmo3 model, does the negative propensity appear early on or does it appear in the later stages?
  • Is there a right or wrong answer on whether to report the successor or predecessor reference? Appendix B1 reports both references when they disagree, but the main text specifies only the change in propensity and uses the successor reference. Should the recommendation include both references, and how should researchers interpret a sign disagreement such as in the case of Qwen2.5-7B?
  • As mentioned above, would these findings hold across different seeds and RL algorithms?
8 comments
AuthorsOpenReview ↗

Response to Reviewer r5Pq (1/4)

We thank the reviewer for the careful review. The General Response summarizes the changes common to all reviews. The revision changes and adds a great deal, and we would be very grateful if you would look at it; revised content is marked in blue in the paper.

Weakness 1: "…a variation in setup parameters can change the sign of the resulting propensity values (e.g., Qwen2.5-7B: −0.20 under the successor reference vs. +0.30 under the predecessor reference)… robustness across seeds and RL algorithms is not tested."

We believe the review reads the reference as a setup parameter; it is not one. The successor's reference and the predecessor's reference do not measure one quantity under two settings. They ask two different questions about the same pair of models: over the texts the successor writes, would the predecessor act as the successor does? And over the texts the predecessor writes? The two Qwen2.5-7B values come from the same runs, prompts, rendering and window, and answer these two questions (Section 3). Equation (4) shows exactly when the answers can differ in sign: only when the successor is more likely to produce the behavior in some contexts and less likely in others, and the two models visit those contexts at different rates (Appendix B.4). The misunderstanding probably arose because the original submission listed the reference in Section 4.4 among the measurement choices, next to the rendering and the cap.

Where the paper does vary a setting, the sign of the change in propensity holds, with one exception.

  • On the Olmo-3 connectives, it is negative at both references under every rendering, prompt set and window we read (Appendix G.1, Table 22).
  • On the stop event, it is negative at every cap, while the count changes sign (Section 4.3, Appendix H.4).
  • With the predecessor scored bare, it keeps its sign on 7 of 8 census transitions on model-written prompts and 9 of 10 on human-written prompts; on the one exception, Qwen2.5-14B-Instruct, it is a small share of a reach-carried rise under both renderings (p = 0.09 and −0.12 on model-written prompts; Section 3, Appendix D.4).
  • On the raw softmax, with no temperature or truncation, no census case reverses its sign; on three, its interval comes to include zero (Appendix C.4).

The sign of the change in propensity is therefore not fragile to the settings the paper varies.

What a disagreement between the references means, and what still holds. On Qwen2.5-7B, over the texts the Instruct model writes, it is slightly less likely than the base model to use the listed words; over the texts the base model writes, it is slightly more likely. So "post-training made the model more inclined to these words" is neither simply true nor false: it depends on which texts one asks about, and the paper reports both. What does not depend on the reference is the main reading: most of the rise is a change in reach. The count rises by 0.48 [0.40, 0.56] listed words per response, and the change in propensity is a small share of it at either reference (p = −0.15 and 0.20, both below 1/2). Most of the rise therefore comes from the Instruct model reaching contexts where both models use these words. The same holds on this transition's other three census cases (Appendix F.2, Table 19). Across the census, the two references give different carrier labels on 8 of 38 cases, and the paper prints both (Section 4.1). Per response, the two values the review cites are now −0.07 and 0.09 (General Response, point 3). We moved the identity into Section 3 as Equation (4), with the question each reference asks, and added to Appendix B.4 the condition under which the two agree in sign.

The default cap and rendering. There is no single correct cap or rendering, so the revision does not claim one. It fixes defaults for stated reasons, states the setting of every result, and checks the main results under the alternatives. The default rendering is now the released format, the format each successor is used in, which both models read (General Response, point 1). The reversal holds under the grafted template, on human-written prompts, at the predecessor's reference and over 2,048 tokens (Section 4.2, Appendix G.1). On MATH-500, "wait" and the stop event keep their carrier labels at the successor's reference over windows of 4,096 and 8,192 tokens (Appendix K.5). Where a setting does change the answer, the paper reports it: the stop event's count changes sign with the cap, while its change in propensity stays negative (Section 4.3, Appendix H.4). This is itself an instance of the paper's point, that a count can move one way while the model's behavior at the same contexts does not. Because the answer can depend on the setting, every split in the paper is reported with the setting it was measured in.

AuthorsOpenReview ↗

Response to Reviewer r5Pq (2/4)

Seeds. The public releases include a stronger test of the same concern. Starting from the same Tülu-3 checkpoint, a PPO run and a GRPO run, which differ in their algorithm and not only in their seed, give the same census reading (Appendix H.2). Within a single run, the connectives' reversal holds at all 26 intermediate checkpoints of Olmo-3-7B-RL-Zero-Code (Appendix G.2). If changing the algorithm leaves the census reading unchanged, and the reversal holds throughout training, a change of seed alone is unlikely to change either.

RL algorithms. The split is not tied to any training method, and we applied it across several.

  • The census spans PPO (OLMo-2), SFT-then-DPO (Llama-3.1) and GRPO (Qwen2.5, Olmo-3) pipelines, and gives the same reading on all of them (Section 4.1, Figure 2).
  • PPO and GRPO runs started from the same Tülu-3 checkpoint give the same census reading (Appendix H.2).
  • Within runs, the connectives' reversal holds at all 26 intermediate checkpoints of Olmo-3-7B-RL-Zero-Code (Appendix G.2), "wait" is reach-carried at the 8 checkpoints of Olmo-3-7B-RL-Zero-Math (Appendix H.1), and across four runs of Shao et al. (2026) from one base model with different rewards, the rise in code reasoning is a change in reach (Appendix I.1).

The readings are therefore not specific to one algorithm or to one point in training. Table 13 records each successor's post-training stages and, where its developers document it, the algorithm; the 11 reinforcement-learning runs from a base checkpoint are GRPO-family, as the review notes.

Weakness 2: "…replacing the first paragraph (the context intervention) may change much more than paragraph structure or connective frequency. Although a paragraph from a different problem retains most of the effect (0.93 of the own-problem rate)… in several experiments, the control class moves alongside the counted class."

You are right that the original submission read too much into this experiment, and the revision limits the claim to what the experiment establishes. Its Appendix I.1 read the 0.93 for a paragraph written for a different problem as showing that the paragraph supplies "discourse register … rather than topic". Handing over a paragraph changes many things at once, so that reading does not follow, and the revision removes it. The context intervention is now presented as a probe of where a change enters, and Section 7 states that its share shows where a change enters, not what in the prefix causes it.

The narrower claim is the one that matters for fixing a behavior, and it is the one we could test. To decide where to act, one needs to know where the change enters, not which feature of the text carries it. The location was confirmed on held-out prompts: the first paragraph, located on one prompt set, removed all of the gap in the Olmo-3 connectives when the predecessor's first paragraph was handed to the successor (1.13 [1.02, 1.29]) (Section 5.2, Appendix M).

We agree that the handed-over text changes more than the behavior, and the revision measures this instead of assuming it away. Context interventions now report their cost in accuracy beside their effect (Table 1); at the first paragraph on Olmo-3-7B-RL-Zero-Code, the cost is modest, 3.2 [0.6, 5.8] points of HumanEval+ pass@1 (Section 5.2). The movement of the control class, frequent paragraph-initial words that are not connectives, is reported as a property of the probe rather than as a failed control (Appendix L.7). These measurements do not weaken the location; they tell what acting there costs. Where one wants to change only the behavior itself, the local intervention does that: it changes only the event's probability at each context, and its effect is reported in two parts, a direct part at the same contexts and an induced part through the contexts it changes (Equation (7); Appendix K.2).

AuthorsOpenReview ↗

Response to Reviewer r5Pq (3/4)

Weakness 3: "…the same underlying change can be classified as propensity for one event and reach for another. It would be helpful to be able to more cleanly separate what a model does from where it goes."

That the split depends on the counted event is not a limitation; it is part of what is being measured. A count is a count of a particular event, so any decomposition of a count is a decomposition for that event. Asking how much of a change is propensity without naming the event has no answer. What the review asks for, a cleaner separation of what a model does from where it goes, is what Equation (3) provides: it writes where the model goes in terms of what it does at earlier positions. It writes the change in reach as the successor's changed choices at earlier positions, each weighted by how many of the counted events the predecessor would write after it. Both parts therefore come from one change in the next-token distribution, and a change in propensity at an earlier event is part of the change in reach of a later one. On the Olmo-3 connectives, the successor writes more paragraph breaks at the same contexts (what it does), so it reaches more paragraph starts (where it goes), where it is less likely to open a line with a connective (what it does). The paragraph-break event is propensity-carried (p = 0.91), the first paragraph accounts for 0.80 [0.70, 0.89] of the connectives' change in reach, and Equation (16) gives the exact term through which the one enters the other (Section 4.2, Appendix B.3). Equation (3) also matches measurement: on random@150, its first-token term is 0.207 [0.187, 0.228] on the 400 held-out problems, and the measured change when the predecessor continues random@150's first token is 0.205 [0.177, 0.233] on all 500 (Section 3, Appendix D.2). We added Equation (3) to Section 3, the tracing identity to Appendix B.3 and the measured trace of the connectives to Section 4.2.

Question 1: "What exactly does it mean to say that post-training primarily changed propensity rather than reach, and vice versa? … how exactly it should be interpreted from an RL perspective? Could it help us diagnose when RL post training is not doing what we want?"

Meaning. "Primarily propensity" means that most of the counted change is the successor acting differently from the predecessor at the same contexts. "Primarily reach" means that the predecessor, placed in the successor's contexts, would already produce most of it. By the carrier rule, a change is propensity-carried if p's interval lies above 1/2 and reach-carried if it lies below. The fall of the line-opening connectives on Llama-3.1-8B-SimpleRL-Zoo is propensity-carried (p = 0.567 [0.525, 0.613]); the rise of the Olmo-3 connectives is reach-carried, with a change in propensity of the opposite sign (p = −0.78). For fixing a behavior, the sign of p matters more than the label (General Response, point 5).

RL perspective. With the behavior's count as the return, the counted change is the predecessor's advantage averaged over the successor's contexts and choices (the performance difference lemma). The change in propensity is the immediate-reward part of that advantage, and the change in reach is the continuation-value part. A propensity-carried change therefore sits mostly at the decisions whose next token is the behavior; a reach-carried change sits mostly at earlier decisions, where the successor moved probability toward tokens after which the predecessor itself would produce the behavior more (or less) often. Equation (3) writes out the continuation-value part, and Appendix B.2 gives the derivation (Equation (15)).

Diagnosis. Yes, and this is the use the revision develops most. The split shows what a count hides about what RL training changed, and where it entered:

  • Over the checkpoints of one run, the count barely moves while the fall in propensity deepens 5.74-fold (Section 4.3).
  • On general prompts and on coding problems the counts move in opposite directions, while the change in propensity is negative on both (Section 4.2).
  • On the released checkpoints trained with random rewards, the rise in code reasoning is a change in reach entering at the first response token, and acting at that token removes the gap (Section 6).

In each case, the count alone would misreport what RL post-training did.

Comments on the paper

6 comments
AuthorsOpenReview ↗

General Response (1/4)

We thank the reviewers for their detailed reviews. In response, we re-checked the settings of the original submission, ran new experiments, and rewrote the paper to make it easier to read. Content changes are marked in blue.

Below, we first summarize the changes to settings and claims, and then the new experiments.

Changes to settings and claims

1. The headline result is now read in the format the model is released with.

The original submission read the Olmo-3 connectives under Olmo-3-7B-Instruct's chat template applied to both models (now called the grafted template). The revision uses the released format, the chat template that Olmo-3-7B-RL-Zero-Code is released with. This is the format in which the model is used, and both models read it. The grafted template lowers the predecessor's own count (0.214 connectives per response, against 1.02 under the released format), which inflates the fold.

  • The fold is therefore smaller: 1.20, against 5.97 under the grafted template (5.97 is the original submission's 4.7-fold rise, recomputed with the corrections in points 2 and 3).
  • The reversal is stronger. The paper's central finding is that a count can rise while the model, at the same contexts, becomes less likely to produce the behavior. We summarize this with the original submission's signed ratio Δprop/Δgen, the share of the count's change that is a change in propensity, which the paper now writes as p. Under the released format, p = −0.78 [−0.99, −0.63], against −0.09 under the grafted template: the fall in propensity is 0.78 times the rise in the count, and over 2,048 tokens it exceeds the rise. A smaller fold with a larger fall in propensity makes the finding clearer, not weaker.
  • The reversal holds under both formats and, as in the original submission, at both references; it now also holds at every public checkpoint of the training run (Sections 4.2 and 4.3). The grafted template stays in the paper, as a robustness check and as the setting of the cases read under it (Appendix G.1).

2. The connective list now contains only connectives.

The original submission counted the connectives on 46 token ids, 15 of which are word pieces rather than connectives, such as "f", the first token of "furthermore", and it checked the 31 complete words only as a robustness check (its Appendix E.3). Every count and split of the connectives outside Section 5's local interventions now uses the 31 complete words, and the reversal holds on them (Section 4, Appendix E.3). Section 5's local interventions act on all 46 ids, as run (Appendix E.3).

3. Counts are now per response, not per 1,000 tokens.

A rate per 1,000 tokens divides each model's count by its own response length. If the successor writes longer or shorter responses, its rate changes even where it behaves exactly as before at the same contexts, so a difference in length is read as a difference in behavior. A count per response has no such term, and its change in reach is exactly Equation (3), the successor's changed choices at earlier positions (Section 3, Appendix C.5). For the stop event, which happens at most once per response, the count per response is the probability that a response ends before the cap, whereas a rate per 1,000 tokens also falls whenever responses get longer.

Recomputed per response, the findings of the original submission hold:

  • The split is exact, and how a count divides varies widely across released transitions; the two parts can have opposite signs.
  • On the census, the predecessor in the successor's contexts already writes most of the successor's count, on every case under both renderings and on both prompt sets.
  • The fall of the Llama connectives is mostly a change in propensity.
  • The stop event's count changes sign with the cap, while its change in propensity stays negative.
AuthorsOpenReview ↗

General Response (2/4)

4. We found an error in one local intervention and withdrew that result.

In the original submission's local intervention on Qwen2.5-7B to Qwen2.5-7B-Instruct, the run changed the list's probability by 1.43 times the change its target sets, outside the 0.95–1.05 band that the other local interventions meet, so it tested a different intervention, and the revision no longer reads it (Appendix M). We apologize for this error. The other local interventions deliver their targets (Appendices J.1 and M).

5. When the local intervention moves a behavior back.

The original submission predicted the effect of a fix from the carrier label: the local intervention, which sets the successor's probability of the behavior to the predecessor's at every context, moves the behavior back when propensity carries the change, and the context intervention does so when reach carries it. The revision predicts the local intervention from the sign of p (the signed ratio Δprop/Δgen; point 1) instead. The reason is simple: the local intervention changes only the behavior's probability at each context, so its effect at the contexts the model visits (its direct part) narrows the gap between the two models when p > 0 and widens it when p < 0.

  • The original submission's case still holds. A propensity-carried change has p > 1/2, and there the local intervention moved the behavior back (the Llama connectives, propensity-carried with nine responses per prompt; Section 5.2).
  • The sign also covers cases the label did not. On "wait", whose rise reach carries, p > 0 and the local intervention still removed part of the gap (0.62 on MiMo-7B-RL-Zero, 0.41 on Olmo-3-7B-RL-Zero-Math). Where p < 0, its direct part widened the gap (Tables 1 and 41).
  • The rule held in every case we computed. The direct part's share had the sign of p in all five cases. The rule holds unless the run reweights contexts by more than a margin computed from the successor's responses before the run, and the margin was at least 2 in every case (Section 5.3, Appendix J.2).
  • The other half of the original claim is kept as the located decision: the earliest position at which handing the successor the predecessor's response up to that point removes at least half of the gap. On the Olmo-3 connectives, the first paragraph, located on one prompt set, was confirmed on held-out prompts: the context intervention there removed all of it (1.13 [1.02, 1.29]), while the local intervention widened it (Section 5.2, Appendix M).
  • The original submission's stop-event result also holds. There, the local intervention's effect through the contexts it changes (its induced part) offsets about four fifths of its direct part (Section 5.3, Appendix K.4).

Each change corrects a setting, none was chosen for its result, and the main claims of the original submission hold; point 5 corrects the one about the local intervention.

New experiments and analyses

  • Is the sign of the change in propensity an effect of the chat template? (Section 3, Appendix D.4) A base model is not post-trained on a chat template, so reading it under one could distort its probabilities. The original submission already scored the predecessor on the prompt text alone (bare), the form it was trained on, in its Appendix F. The revision moves this into the main text as a validity check and states the result there: the census's change in propensity keeps its sign on 7 of 8 transitions on model-written prompts and 9 of 10 on human-written prompts. The one transition where it changed sign, Qwen2.5-14B-Instruct, has a change in propensity that is a small share of its rise under both renderings (p = 0.09 with the template, −0.12 bare, on model-written prompts), and its rise is reach-carried either way. The template does not drive the sign.
  • Is the predecessor's count in the successor's contexts just copying? (Section 4.1, Appendix F.4) When the predecessor reads a successor response, it may give a listed word high probability only because the word already appeared earlier in that response. If so, the census's high ρ (the predecessor's count over the successor's on the successor's responses; the original submission's predecessor share) would reflect copying, not the predecessor's own behavior. We recomputed ρ only up to the first listed word in each response, where nothing can yet be copied, and compared it with a cut at a matched position. ρ changes by at most 0.075, and its lower bound stays above 1/2 on every census case. The census reading does not come from copying.
  • Human-written prompts (Appendices E.2, F.1 and M). The original submission already showed the reversal and the census reading on human-written prompts from Chatbot Arena. The revision adds the held-out first-paragraph test on them.
AuthorsOpenReview ↗

General Response (3/4)

  • A published reading of overused words, split into its parts (Section 4.1, Appendix F.2). Juzek & Ward (2026) compare the counts of overused words in Llama-3.2-3B and Llama-3.2-3B-Instruct and link the rise to learning from human feedback. A count cannot tell whether post-training made the model more inclined to these words at the same contexts, or whether the model now reaches contexts where the base model would also write them. On the closest released pair in our census, Llama-3.1-8B to Llama-3.1-8B-Instruct, ρ is 0.86–0.94 on the 75 words of our census list that are among their significant increases: in the successor's contexts, the predecessor already writes most of the successor's count, so most of the rise is a change in reach. Their top-ranked words are the exception: those are propensity-carried or mixed (four words, at most 2.9% of the count). One count thus hides words that rise for different reasons, which is why a count should be split before a cause is assigned to it.
  • Post-training stages and training algorithms (Appendices E.1 and H.2). Table 13 gives the post-training stages of the successor of each of the 20 transitions, as its developers report them. Starting from the same Tülu-3 checkpoint, a PPO run and a GRPO run give the same census reading, so the reading is not specific to one RL algorithm.
  • Counts disagree across domains; the change in the model does not (Section 4.2, Appendices G.3 and G.6). On general prompts, the connectives' count rises; on coding problems, the successor's training domain, it falls. Read as counts, the two domains say opposite things about what RL did to the connectives. The split gives one answer: in both, the change in propensity is negative, so on the same text the successor is less likely to open a line with a connective. Counted at any position, the same words have a positive change in propensity, so the fall is specific to opening a line.
  • The reversal also appears in words we did not choose (Section 4.2, Appendix G.7). The connectives were chosen by hand, so the reversal could be a property of that choice. The original submission tested this in its Appendix L.1, on three SimpleRL-Zoo transitions and six classes of words selected by a fixed rule, and all six showed the reversal. The revision moves this test into Section 4.2, recomputes it per response, and checks the reversal without its largest word on all six classes (the original reported this on four). The rule takes every word whose share of all tokens rose (interval above zero) while its change in propensity was at most zero at both references (by point estimate), measured on one half of the prompts. We then counted the selected words as one class on the other half, and repeated with the halves swapped. Every class shows the reversal at both references, and five of the six keep it without their largest word.
  • How the split evolves over training (Section 4.3, Appendices G.2 and H.1). We split all 26 intermediate checkpoints of Olmo-3-7B-RL-Zero-Code. The reversal is present from the first checkpoint; from there to the released model the count barely changes, while the fall in propensity grows 5.74-fold. A count at any single checkpoint would miss this. "wait" on Olmo-3-7B-RL-Zero-Math is reach-carried at each of the 8 checkpoints we split.
  • Held-out tests of located decisions (Section 5.3, Table 1, Appendix L.1). Besides the first-paragraph test (point 5), we located decisions on 100 MATH-500 problems and tested them on the other 400. In all 5 cases the share of the gap removed is positive by point estimate, and in the 3 at the first response token its interval lies above zero.
  • Analysis of a published training method (Section 6, Appendix I). Shao et al. (2026) report that RL with random rewards raises code reasoning in Qwen2.5-Math-7B, and read this as training amplifying a behavior the model already had. A count cannot say where training acted. On their released checkpoints, the split shows that the rise is a change in reach that enters at the first response token: training did not make the model more likely to write code ("python") at the same context, but changed how it starts its response (92.1% of responses open with "To", against 52.6% for the base model), after which the base model itself goes on to write code. Handing the successor the base model's first token removes the gap. The split thus finds where this training acted, and acting at that position controls the change.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.