acceptodds
Under review as a conference paper at ICLR 2027

Flat by Theorem: What Truncated Chain-of-Thought Accuracy Curves Measure

Abstract

A growing literature asks when a chain of thought (CoT) decides its answer: truncate it, read out an answer, plot accuracy against the truncation point, read the rise as the moment of commitment. The quantity it measures, the probability that a trajectory ends correct given its prefix, is a bounded martingale under exact continuation, so its expectation is constant in the index and equals full-CoT accuracy; the companion agreement curve is a submartingale that rises for every model. Any rise is therefore an artefact; we decompose that of a published-style pipeline, rebuilt on our own traces, into three errors: a readout that intervenes rather than conditions, survivorship, and an index that is not a stopping time. On 8 models and 4 datasets (1,237,682 generations) the exact-continuation curve rises on no arm, equivalent to no change within 0.03–0.05; the forced-readout curve on the same traces climbs by +0.03 to +0.64 — by an amount that tracks the ordinary chain-of-thought gain closely enough to predict it: a rule registered on 6 arms before a 14B model was run put its effect at +0.520 [+0.396, +0.643]; it measured +0.633. Such a curve measures what the chain is worth, not when the answer was fixed. The readout dominates: survivorship alone manufactures declines of up to -0.464, but paired within a trace it and index leakage are null. Of 18 chat endpoints audited over one access path, 12 silently ignore assistant prefill, 10 in every sample; and a continuation conditioned on a re-encoded prefix passes every check a single-arm study can run, failing only the one that needs two. What survives is the corrected agreement curve's shape, free at half depth between 0.15 and 0.50. Code and records accompany this submission.

Primary area:
interpretability and explainable AI
Keywords:
chain-of-thought, reasoning, truncation curves, martingale, forced readout, survivorship bias, evaluation methodology

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_nXfgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies what truncated chain-of-thought accuracy and agreement curves measure. Through a martingale analysis, it shows that average correctness under exact continuation is constant, while answer agreement is non-decreasing. It analyzes readout intervention, survivorship, and indexing effects, and proposes a corrected continuation protocol with unbiased estimators and diagnostic checks. Experiments across eight models and four datasets compare these protocols, quantify measurement effects, and examine whether chat interfaces actually continue the supplied prefixes.

Critical strengths

The paper presents an interesting measurement framework, supported by extensive experiments and careful attention to implementation.

Critical weaknesses
  1. Limited conceptual novelty. Given the chosen estimand, flatness follows directly from standard conditional-expectation properties. The main theoretical contribution is a useful but modest formalization.
  2. Limited additional empirical insight. Most experiments reinforce the same protocol distinction. Their breadth strengthens confidence while adding limited understanding of when or how answers become determined.
Questions for authors

How does your new evaluation method deepen the community’s understanding of model reasoning?

2 comments
AuthorsOpenReview ↗

Reply to Reviewer nXfg

Thank you. On novelty we agree with the premise and say so in the paper: the martingale is standard, and Proposition 1 credits Falck et al., Kim et al. and Nikulkov. The contribution is what follows from applying it to an instrument the field now also uses as a training objective (Sivakumaran et al.): which published estimands violate its hypotheses and by how much on the same traces, a protocol that measures the curve correctly with checks that run inside one arm, and the condition under which a measured process is the continuation law at all (Prop. 6).

What it adds to the understanding of reasoning. New §4.2 fixes where the reported answer is computed: two problems per prompt, only A's answer reported, A solved first or last, on four models from 7B to 70B. In both orders the forced curve's extent tracks the chain's worth (e.g. Qwen2.5-7B: +0.650 vs. gain +0.600 with A first, +0.855 vs. +0.822 with A last), so the extent says nothing about when. After A's result line, while the model works on B, exact continuation reproduces the trace's eventual answer in 75–100% of draws: the answer is already highly predictable. Forced readouts understate this, and how they fail differs by model (Table 26, same cells): Llama-3.3-70B returns no parseable answer in 38% of draws but agrees with its final answer in 99% of those it parses; Qwen2.5-7B returns another parseable answer in 48% (B's answer in 14%); Mistral-Small agrees in 91%. Two targeted readouts (Table 27) change this for some models and not others, and before A's result line agreement is at most 0.41 across the three tested readouts, against 0.59–0.97 under exact continuation. So the forced curve's lag reflects how each model responds to being interrupted and asked, itself a finding about these models, rather than a time of commitment. Two further results are about reasoning rather than measurement: on natural tasks the corrected agreement curve's shape differs across models and tasks (0.20–0.46 of the distance closed by half the median length, absolute index), so once the readout is removed, how early a trajectory's answer settles differs across models and tasks; and on tasks where the chain is worth nothing (CommonsenseQA, ARC) the forced curve rises only slightly (+0.039 [+0.013, +0.064] on CommonsenseQA), so the familiar rising curve is evidence about how much reasoning helps on a task.

Thanks for the additional experimental results and clarifications. Most of my concerns have been resolved, and I have raised my rating accordingly.

Reviewer_3sruOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

A martingale-based framework examines what accuracy and answer-agreement curves measure when a chain of thought is interrupted at different positions. It models the final-answer distribution conditional on a prefix as a martingale, yielding a constant mean correctness probability under exact continuation and appropriate indexing, together with nondecreasing expected agreement. It then separates changes caused by forced readout, survivor selection, and indexing by the completed trajectory, and proposes diagnostics for continuation protocols. The experiments show that readout choices can substantially alter the observed curves, but the broader conclusions about prior commitment measurements and the interpretation of the proposed replacement metrics need stronger justification.

Critical strengths
  1. The distinction between continuation and intervention is useful and precise. Defining the conditional final-answer distribution makes clear why continuing the original process and requiring an immediate answer are different measurements. The flat-correctness identity provides a concrete reference for protocol validation.
  2. The experimental design directly compares protocols on shared trajectories. The main comparisons use the same prefixes, and the full combination of protocol choices separates readout, survivor, and indexing effects. This is more informative about measurement differences than comparing unrelated runs or published aggregate curves.
  3. Several plausible implementation explanations are examined. The analysis considers parsing failures, a longer readout window, closing reasoning tags, greedy base trajectories, and re-encoded prefixes. The diagnostic examples make the practical distinction between apparently valid and token-exact continuation tangible.
  4. The limitations and negative findings are informative. The paper reports a scorer correction, an unexplained flat-null discrepancy, and the fact that a simple unfitted prediction outperforms the fitted rule on the held-out example. These disclosures help delimit what the empirical evidence supports.
Critical weaknesses

W1. The discussion of prior work could distinguish measurement objectives more clearly. The flatness result concerns mean correctness under original continuation, whereas forced-answer protocols can target interruption or early-exit performance. Please distinguish the target quantity and readout protocol of each prior study when discussing commitment claims.

W2. The recommended agreement statistic uses future-dependent indexing. The evaluation selects positions using each trajectory's final length, conflicting with the recommendation to use absolute positions or stopping times. Please define the resulting target quantity and justify the length-normalized identity, or recompute the summaries at absolute positions.

W3. The diagnostic completeness claim needs a terminal-boundary condition. For a true conditional distribution P_t, the alternative Q_t = αP_t + (1−α)P_0 with 0 < α < 1 can satisfy martingale orthogonality and show increasing terminal agreement while differing from P_t. The completeness claim needs the exact terminal boundary Q_M = δ_Y together with the full martingale property; the weaker operational checks should be presented as diagnostics.

W4. Two directional deductions need further qualification. Counting EOS in the trajectory length T, consider paths A–EOS, B–EOS, A–X–C–EOS, A–X–W–EOS, B–X–C–EOS, and B–X–W–EOS with probabilities 4/11, 2/11, 1/11, 1/11, 2/11, and 1/11. Taking A–EOS and C–EOS endings as correct yields length–correctness covariance −4/121 but correctness bias +2/165 at T/2, contradicting the claimed sign relationship. Similarly, three equal-sized groups finishing in sequence with accuracies 1, 0.2, and 0.5 yield survivor accuracy rising from 0.35 to 0.5 despite cumulative finisher accuracy staying above the overall mean. Please correct these directional claims or state and prove the additional assumptions they require.

W5. The last-switch interpretation appears to have its direction reversed. A smaller post-decision fraction 1−t*/T places the last switch later, whereas the paper describes it as earlier settling.

Questions for authors

I sincerely encourage a focused response to W1–W4; satisfactory revisions to the measurement interpretation and mathematical arguments will lead me to raise my rating.

  1. Is the direction of the last-switch interpretation an editorial error, and does correcting it change the comparison with the cited commitment studies?
  2. Which claims are intended to generalize to answer-logit or next-token-probability readouts, and what evidence supports that scope?
2 comments
AuthorsOpenReview ↗

Reply to Reviewer 3sru (1/2)

Thank you for a precise report. W1–W5 were correct, and the revision fixes each at the source.

W1 (target quantities). New Table 3 separates, for Lanham et al., Ballon et al., Datta et al., Scalena et al., Zhang et al., Sivakumaran et al. and Bigelow et al., the plotted quantity, the readout, the index, and what the curve estimates. Two targets are untouched by Corollary 1 and the table says so: the accuracy of an interrupted model (what early-exit comparisons such as Datta et al.'s accuracy drop and Scalena et al.'s exit experiment need), and any statistic of the forced readout taken as an intervention. Where a study aims at the agreement curve GG but computes it under a forced readout, the table says that Proposition 2 describes the target, not the forced curve. What Corollary 1 constrains is the inference to when the trajectory's own answer was fixed, the reading those papers also give (quoted in §1).

W2 (future-dependent indexing). Correct: App. O read each trace at half its own length. The target is now G(t)=E[Pt(Y)]G(t)=\mathbb{E}[P_t(Y)] at absolute indices with absorption, normalised as (G^(t)−G^(t1))/(1−G^(t1))(\hat G(t)-\hat G(t_1))/(1-\hat G(t_1)) and read at the grid point nearest half of each arm's median length, one constant per arm; App. O labels this point a data-adaptive descriptive summary, since the bootstrap holds the selected grid point fixed. Across the 11 sampled-base exact-continuation arms it is 0.20–0.46 (prefix-ignoring control 0.06), against 0.15–0.50 under the length-normalised index; only the absolute-index version is reported. App. U is recomputed the same way (Spearman between published and corrected instruments +0.48 [0.00, 0.79]; the two are not monotone transforms of each other). Terminal agreement reads depths off each trace's length, and App. K now says so: it is a length-referenced summary, and the test's size is unaffected because under a prefix-ignoring readout P~t=P0\tilde P_t=P_0 at every index, so the paired difference has mean zero for any choice of indices. A significantly positive rise rejects that null; a null rise is inconclusive, and the paper now says so.

W3 (completeness needs the terminal boundary). Agreed, and it is now a result. Proposition 6: if the pseudo-beliefs, completed by Qt=δYQ_t=\delta_Y for t≥Tt\ge T, form an (Ft)(\mathcal F_t)-martingale, then Qt=PtQ_t=P_t. Your mixture shows the boundary cannot be dropped: unclamped, it is a martingale, satisfies Eq. (4) and has a rising agreement curve, but violates the boundary; its absorbed extension satisfies the boundary but loses the martingale property at the absorbing step. "Together they are complete" is removed. Own-answer calibration is the member aimed at the boundary: for the unclamped mixture at a fixed index its expectation is α(1−α)(E∥Pt∥2−∥P0∥2)>0\alpha(1-\alpha)(\mathbb{E}\lVert P_t\rVert^2-\lVert P_0\rVert^2)>0. We realised the absorbed mixture on the Qwen2.5-3B cells (each draw is the real continuation with probability α\alpha, a fresh draw otherwise; Table 22):

α\alphaflat nulldrift (BH)terminal-agreement riseconservation gapown-answer DD
3/4passpass+0.219+0.114 [+0.066, +0.164]+0.0111 [+0.0033, +0.0191]
1/2passpass+0.144+0.259 [+0.206, +0.309]+0.0144 [+0.0060, +0.0230]
1/4passpass+0.081+0.375 [+0.319, +0.431]+0.0095 [−0.0000, +0.0191]
0 (ignored)passfail (1)*+0.004+0.457−0.0004

*The one interval that survives Benjamini–Hochberg is one in which absorbed cells enter, whose value is set by the absorption convention rather than by the readout (App. T).

The checks that look only at increments before absorption pass, as your argument predicts. Conservation, which sees the absorbing step, rejects every mixture; own-answer calibration rejects α=3/4\alpha=3/4 and 1/21/2, its interval at α=1/4\alpha=1/4 ending at zero. The closed form above does not apply to the absorbed implementation on the fixed grid, where absorption selects which cells enter, so App. T reports these values empirically.

AuthorsOpenReview ↗

Reply to Reviewer 3sru (2/2)

W4 (directional deductions). Both counterexamples are correct; we verified them exactly. (i) Prop. 3 now states the level and the monotonicity conditions separately: S(t)<AccS(t)<\mathrm{Acc} iff F(t)>AccF(t)>\mathrm{Acc}, and S(t+1)<S(t)S(t+1)<S(t) iff the traces finishing at t+1t+1 are more often correct than S(t)S(t); your three-group example is in App. A. (ii) The sentence on the sign of the fraction-index bias is removed. Corollary 2's covariance form is exact; its sign is not determined by Cov(T,1{Y=a⋆})\mathrm{Cov}(T,\mathbf{1}\lbrace Y=a^\star\rbrace), App. A gives your six-path example (−4/121-4/121 vs. +2/165+2/165), and the sign is measured (App. G), which is what the paper's numbers relied on. Corollary 2 also states when the fraction index looks ahead: {T>s/f}\lbrace T>s/f\rbrace is Fs\mathcal F_s-measurable only if the first ss tokens settle whether the trace runs past s/fs/f.

W5 / Q1 (last-switch direction). Yes, App. I had the direction reversed. A smaller post-decision fraction places the last switch later: our forced readouts' last switch comes later than the no-commitment null (0.11–0.37 vs. 0.54–0.56), consistent with a readout that keeps moving with the prefix. The comparison with the cited studies is unaffected: §2 uses only the size of the null (about half the trace after the last switch for a process with no early commitment) against published values of 35–70%.

Q2 (answer-logit / next-token-probability readouts). Scope, as stated in §2.2: (a) any readout that appends a suffix and reads the answer distribution, by sampling or by next-token probabilities, is an intervention P~t\tilde P_t; Corollary 1 does not constrain its level, and Eq. (4) tests whether it is a conditional expectation. (b) A probe that equals the conditional expectation of the trajectory's eventual answer given features of the prefix, E[1{Y=a⋆}∣Gt]\mathbb{E}[\mathbf{1}\lbrace Y=a^\star\rbrace\mid\mathcal{G}_t] with Gt⊆Ft\mathcal G_t\subseteq\mathcal F_t, has a flat correctness curve too; a probe trained on forced answers targets the forced curve. (c) Empirically (App. X): on three BIG-Bench Hard tasks with Qwen2.5-7B we read the prefix as Lanham et al. do, the next-token probabilities over the option letters after "Given the above reasoning, the answer is" (0.95 of the mass on average), renormalised. That curve rises from 0.461 to 0.568, 4 of 13 intervals fail the drift test after Benjamini–Hochberg, and its paired effect against exact continuation on the same traces is +0.119 [+0.078, +0.159], the size of the sampled forced readout (+0.125 [+0.082, +0.167]); the exact-continuation curve on those traces is flat (slope −0.0034 [−0.0091, +0.0024]). Reading probabilities rather than sampling does not remove the intervention: the suffix does.

Reviewer_mXsKOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper shows that the accuracy curve of a truncated chain of thought, when measured by exactly continuing the trajectory, is flat by theorem, because the probability of ending correct given the prefix is a martingale. The rising curves that are reported in literature come mainly from the forced readout, which rises by about what the CoT is worth, so they do not indicate when the model decided its answer. The paper also proposes a corrected protocol with internal checks and an audit of chat endpoints, most of which silently ignore assistant prefill.

Critical strengths
  1. The paper clarifies what truncation curves actually measure, an increasingly used tool that is also used as a training objective. The theoretical result is simple but has concrete consequences for the literature
  2. The experimental design is solid and clean
  3. The paper has practical value, as it proposes a corrected protocol, specification tests that run on a single arm (each validated against a positive control), and an audit of chat endpoints useful to anyone using closed APIs.
Critical weaknesses

Major

  1. The framing of prior work sounds too strong. K(t)K(t) flat holds for any process, so K(t)K(t) cannot measure commitment even in principle. The forced readout is a different and legitimate quantity, as the paper acknowledges: it asks what the model would answer if interrupted now. The authors show that the forced curve is not K(t)K(t), but not that it carries no information about commitment. “Any rise is therefore an artefact” therefore goes a bit beyond what is shown.
  2. The alternative interpretation is not demonstrated. The abstract states that the curve measures “what the chain is worth, not when the answer was fixed”. Yet, Sec. 6 admits that, on the tasks used, the worth of the chain and the time of commitment are confounded, so ∆readout ≈ Acc − Acc0% is consistent with both readings. Moreover, the relation is close to an identity by construction (the readout at the first index is close to the 0%-CoT point, the one at the last index close to the full CoT), so r = 0.98 and the registered prediction add little. This is confirmed by the simple identity predicting better than the fitted rule. A task where the two quantities come apart would help strengthening this point
  3. Limited experimental scope. The open models are almost all from the Qwen family (with a single Llama), and between 1.5B and 14B of size, and the tasks are mostly mathematical with numeric answers. Decoding at temperature 1 without nucleus differs sensibly from the typical setups of the criticized literature. For the reasoning models, samples per question are a few (R = 3, K = 3).

Minor

  1. Since the theoretical part is not new, as the authors acknowledge, the contribution is its application. Thus, The the authors could compact the theoretical section a bit more.
  2. The “three errors” framing seems to overstate two of them. Survivorship is negative (it masks part of the rise), index leakage is small (at most 2.3 points, no test survives Benjamini–Hochberg), and both are null when paired. My reading is that one single error dominates (i.e., the readout).
  3. The endpoint audit is tangential to the main contribution, as it goes through a single aggregator, uses a heuristic classification (lowercase first character), and 24 samples per endpoint. I think it could be moved to the appendix.
  4. The deviation of R1-Distill-1.5B from the flat null remains unexplained.
Questions for authors

See weaknesses

4 comments
AuthorsOpenReview ↗

Reply to Reviewer mXsK (1/2)

Thank you. We agree with the main thrust of M1 and M2, have changed the claims, and ran the experiment you asked for.

M1 (framing). Agreed. Corollary 1 says the correctness curve under exact continuation cannot rise; it does not say the forced curve carries no information about commitment. The abstract now reads: "A rise in a published curve therefore comes from the measurement, and reading it as the time at which the trajectory's answer was fixed needs more than the curve." Table 3 states which targets of forced readouts are legitimate (interruption and early-exit accuracy). Whether a forced curve's shape tracks commitment is an empirical question, answered below.

M2 (alternative interpretation; identity by construction). We agree that Δreadout≈Acc−Acc0-CoT\Delta_{\text{readout}}\approx\mathrm{Acc}-\mathrm{Acc}_{0\text{-CoT}} is close to an identity because the forced curve's endpoints sit near the 0%-CoT and full-CoT accuracies, and the registered prediction is no longer in the abstract. What the identity establishes is that the vertical extent is a property of the endpoints. To test the shape we fixed where the reported answer is computed while keeping the problems the same (§4.2, Tables 2, 25–27, App. Y). Each prompt carries two GSM8K problems, A and B; the model solves both, closes each with a "Result" line, and reports only A's answer. In order A first A's answer is written halfway and the rest is work on B; in order A last it is written at the end. Four models from 7B to 70B follow the instructed order in ≥98.8% of traces.

modelorderAccCoT gainΔreadout\Delta_{\text{readout}} [95% CI]slope of K^\hat K [95% CI]share of rise while A is solved: cont. / forced
Qwen2.5-7BA first0.792+0.600+0.650 [+0.609, +0.689]+0.0014 [−0.0047, +0.0074]0.50 / 0.43
Qwen2.5-7BA last0.862+0.822+0.855 [+0.824, +0.882]−0.0004 [−0.0049, +0.0039]0.75 / 0.84
Llama-3.1-8BA first0.625+0.625+0.617 [+0.582, +0.651]+0.0014 [−0.0066, +0.0094]0.56 / 0.19
Llama-3.1-8BA last0.757+0.757+0.765 [+0.733, +0.797]−0.0029 [−0.0096, +0.0040]0.83 / 0.97
Mistral-Small-24BA first0.936+0.878+0.763 [+0.730, +0.797]+0.0016 [−0.0009, +0.0040]0.96 / 0.92
Mistral-Small-24BA last0.922+0.807+0.923 [+0.897, +0.944]−0.0031 [−0.0070, +0.0007]0.77 / 0.88
Llama-3.3-70B (FP8)A first0.954+0.950+0.943 [+0.921, +0.962]−0.0008 [−0.0025, +0.0008]0.99 / 0.57
Llama-3.3-70B (FP8)A last0.948+0.946+0.948 [+0.923, +0.968]−0.0005 [−0.0034, +0.0021]0.85 / 0.88

(Shares are the share of the mean rise in agreement with the trace's own answer, both instruments on the same traces; intervals in Table 25.)

AuthorsOpenReview ↗

Reply to Reviewer mXsK (2/2)

M2, results. (i) The flat null holds on all eight arms, and the extent tracks the chain's worth in both orders (within 0.12): it does not distinguish an answer written halfway from one written at the end. (ii) The shape. With A first, after A's result line and before the final line, exact continuations reproduce the trace's eventual answer in 0.75–1.00 of draws. Table 26 shows, on the same cells, what the forced curve's lag is made of, and it differs by model: while the model works on B, Llama-3.3-70B returns no parseable answer in 38% of draws but agrees with its final answer in 99% of those it parses; Llama-3.1-8B returns no parseable answer in 75%; Qwen2.5-7B returns another parseable answer in 48% (B's answer in 14%); Mistral-Small agrees in 91%. We also re-read the same traces with two targeted 32-token readouts, one naming problem A and one in the prompt's own "Result A:" format (Table 27). The first raises Qwen's agreement to 0.87 and removes B's answer; the second raises Llama-3.1-8B's to 0.48 but lowers Mistral's to 0.83; Llama-3.3-70B still returns no parseable answer in 45–54% of draws while working on B. Before A's result line, agreement is at most 0.41 across the three tested readouts, against 0.59–0.97 under exact continuation. So a forced curve's shape depends on how it asks, and the tested readouts understate how predictable the eventual answer already is. This is the empirical content of M1: a forced curve is a valid measurement of an interrupted model; reading its shape as evidence of answer stabilisation requires validating the readout, and exact continuation provides that reference in our study.

M3 (scope). App. X / Table 24 adds eight arms, generated and analysed as the main arms: Qwen2.5-7B at the creators' recommended setting (0.7/0.8/20/1.05) and untruncated (T=1), Qwen2.5-7B on three BIG-Bench Hard tasks, Mistral-Small-24B, Llama-3.3-70B (FP8 weights), R1-Distill-7B at DeepSeek's recommended T=0.6, an R1-Distill-1.5B replicate, and Phi-4-mini-reasoning. On all eight the flat null holds (e.g. Llama-3.3-70B −0.0015 [−0.0071, +0.0040]; Phi −0.0053 [−0.0117, +0.0006]), every internal check passes, and the readout effect stays within 0.079 of the chain-of-thought gain. On the non-mathematical task the chain is worth +0.115 and the readout effect is +0.125; an answer-logit readout there behaves like the sampled one (+0.119; reply to 3sru, Q2). The theory requires only that base traces and continuations share one decoding law, which every arm does; App. B lists each arm's sampling parameters. On R=3, K=3: inference is over questions (158 per reasoning arm, cluster bootstrap), and the estimators using K are unbiased for any K≥2 (App. D), so a small K widens intervals rather than biasing them; the replicate, the recommended-setting arm and Phi-4-mini-reasoning more than double the reasoning-model data, with the same result.

m1 (compact theory). We kept §2's statements in the main text, since other reports asked for more there (Prop. 6, the restated Prop. 3 and Cor. 2); the space for §4.2 came mainly from condensing the endpoint audit.

m2 (three errors). Agreed: the abstract says the readout dominates and that survivorship and index leakage are null when paired within a trace. Survivorship stays because an unpaired pipeline that drops finished traces still produces declines of up to −0.464, and leakage because its bias formula tells a reader which indices are safe.

m3 (endpoint audit). Condensed to one paragraph (§4.4), details in App. Q. On the single access path: we re-sent the identical request through the vendor's own API for every OpenAI and DeepSeek endpoint still served (9 endpoints, 24 samples each). All 9 fall in the same class as before and all 9 ignore the prefill there too (gpt-4o, the one mixed endpoint, continues in 38% of first-party samples vs. 21% before); 3 further OpenAI endpoints have since been retired, and the other vendors' endpoints were not available to us first-party. DeepSeek also documents a separate prefix-completion route (beta API); through it both DeepSeek endpoints continue the prefix in every sample, but it still hands the server a string to re-tokenise, the case only conservation catches (App. T). All raw replies are released.

m4 (R1-Distill-1.5B). It is the only one of the 11 sampled-base continuation arms whose flat-null interval excludes zero, and it does not survive Benjamini–Hochberg across them (bootstrap p=0.013p=0.013 vs. a threshold of 0.0045). We reran it with fresh base traces (App. X): same model, questions, nominal sampling parameters, grid and cap, generated by the later implementation that conditions on emitted token ids. The replicate's slope is +0.0045 [−0.0035, +0.0119], it passes all five checks, and its readout effect (+0.613) tracks its CoT gain (+0.679). The deviation did not recur; we have not identified its cause.

Reviewer_mXsKOpenReview ↗

Response to Authors (1/2)

I really thank the authors for their effort, and I think that this new evidence fully addresses my points M1 and M2.

Reviewer_mXsKOpenReview ↗

Response to Authors (2/2)

I am also satisfied regarding M3 and the minor points. I agree with the way you dealt with m1, as other reviewers asked for it. I'm willing to increase my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response (to all reviewers)

We thank the reviewers for careful reports. The revised PDF acts on every concrete request, with every change since the submission in blue; the individual replies give details.

  1. Mathematics (3sru W3–W5). New Proposition 6: the martingale property identifies the continuation law only together with the terminal boundary QT=δYQ_T=\delta_Y. The checks are presented as diagnostics, and App. T realises the reviewer's mixture αPt+(1−α)P0\alpha P_t+(1-\alpha)P_0 on real cells. Prop. 3 and Cor. 2 are restated with the reviewer's counterexamples (App. A), Prop. 5 is proved by a finite argument, and the last-switch direction in App. I is corrected.
  2. Targets of prior work (3sru W1). Table 3: for each study, what is plotted, how the prefix is read, where it is truncated, and what that curve estimates.
  3. Absolute-index shape (3sru W2). The agreement-shape summary and the ranking comparison are recomputed at absolute indices (App. O, U).
  4. Where the answer is computed, fixed by construction (mXsK M2, nXfg). New §4.2, Tables 2, 25–27 and App. Y: two GSM8K problems per prompt, only A's answer reported, A solved first or last, on four models from 7B to 70B. The flat null holds on all eight arms and the forced curve's extent tracks the chain's worth in both orders. Its shape is not by itself a reading of when the answer was fixed: after A's result line, exact continuation reproduces the eventual answer in 75–100% of draws, while the forced readout's lag on the same cells is, depending on the model, missing answers (Llama), other numbers (Qwen), or almost nothing (Mistral). Two targeted 32-token readouts reduce the lag for some models only, and before A's result line agreement is at most 0.41 across the three tested readouts, against 0.59–0.97 under exact continuation.
  5. Scope (mXsK M3, 3sru Q2). App. X / Table 24: eight further arms (a non-mathematical task, an answer-logit readout, recommended and untruncated decoding, Mistral-Small-24B, Llama-3.3-70B, Phi-4-mini-reasoning, an R1-Distill replicate). The flat null and every internal check pass on all eight.
  6. Audit (mXsK m3). Replicated through OpenAI's and DeepSeek's own APIs (Table 18); DeepSeek's documented prefix-completion route is now reported too (App. Q).
  7. Framing (mXsK M1). The abstract and conclusion say that a rise comes from the measurement and that a curve's shape is not by itself a reading of when the answer was fixed.

Smaller points raised in the reviews are also addressed: Cor. 2's measurability condition, the P~t=P0\tilde P_t=P_0 notation, accents in Lévy/François, App. L's Ft1\mathcal F_{t_1}-measurable restriction (the effect stays at +0.26 to +0.63 on the main arms), completeness-guard counts, and Sivakumaran et al.'s curve type in Table 3. App. B now states how prefixes were tokenised: for the main arms, the first tt tokens of the re-encoded base text; for the added arms, the token ids the server emitted. The two coincide on 96.2–100% of traces wherever both are recorded.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.