- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies what truncated chain-of-thought accuracy and agreement curves measure. Through a martingale analysis, it shows that average correctness under exact continuation is constant, while answer agreement is non-decreasing. It analyzes readout intervention, survivorship, and indexing effects, and proposes a corrected continuation protocol with unbiased estimators and diagnostic checks. Experiments across eight models and four datasets compare these protocols, quantify measurement effects, and examine whether chat interfaces actually continue the supplied prefixes.
Critical strengths
The paper presents an interesting measurement framework, supported by extensive experiments and careful attention to implementation.
Critical weaknesses
- Limited conceptual novelty. Given the chosen estimand, flatness follows directly from standard conditional-expectation properties. The main theoretical contribution is a useful but modest formalization.
- Limited additional empirical insight. Most experiments reinforce the same protocol distinction. Their breadth strengthens confidence while adding limited understanding of when or how answers become determined.
Questions for authors
How does your new evaluation method deepen the community’s understanding of model reasoning?
2 comments
Thanks for the additional experimental results and clarifications. Most of my concerns have been resolved, and I have raised my rating accordingly.
Reply to Reviewer nXfg
Thank you. On novelty we agree with the premise and say so in the paper: the martingale is standard, and Proposition 1 credits Falck et al., Kim et al. and Nikulkov. The contribution is what follows from applying it to an instrument the field now also uses as a training objective (Sivakumaran et al.): which published estimands violate its hypotheses and by how much on the same traces, a protocol that measures the curve correctly with checks that run inside one arm, and the condition under which a measured process is the continuation law at all (Prop. 6).
What it adds to the understanding of reasoning. New §4.2 fixes where the reported answer is computed: two problems per prompt, only A's answer reported, A solved first or last, on four models from 7B to 70B. In both orders the forced curve's extent tracks the chain's worth (e.g. Qwen2.5-7B: +0.650 vs. gain +0.600 with A first, +0.855 vs. +0.822 with A last), so the extent says nothing about when. After A's result line, while the model works on B, exact continuation reproduces the trace's eventual answer in 75–100% of draws: the answer is already highly predictable. Forced readouts understate this, and how they fail differs by model (Table 26, same cells): Llama-3.3-70B returns no parseable answer in 38% of draws but agrees with its final answer in 99% of those it parses; Qwen2.5-7B returns another parseable answer in 48% (B's answer in 14%); Mistral-Small agrees in 91%. Two targeted readouts (Table 27) change this for some models and not others, and before A's result line agreement is at most 0.41 across the three tested readouts, against 0.59–0.97 under exact continuation. So the forced curve's lag reflects how each model responds to being interrupted and asked, itself a finding about these models, rather than a time of commitment. Two further results are about reasoning rather than measurement: on natural tasks the corrected agreement curve's shape differs across models and tasks (0.20–0.46 of the distance closed by half the median length, absolute index), so once the readout is removed, how early a trajectory's answer settles differs across models and tasks; and on tasks where the chain is worth nothing (CommonsenseQA, ARC) the forced curve rises only slightly (+0.039 [+0.013, +0.064] on CommonsenseQA), so the familiar rising curve is evidence about how much reasoning helps on a task.