acceptodds
Under review as a conference paper at ICLR 2027

Drop the Act: Probe-Filtered RL for Faithful Chain-of-Thought Reasoning

Abstract

Reasoning models can keep reasoning even when they can already give the correct answer. We call these additional, post-commitment steps reasoning theater. We introduce ProFIL (Probe-Filtered Reinforcement Learning), a drop-in extension to Group Relative Policy Optimization (GRPO). Prefix counterfactuals identify post-commitment steps; a lightweight probe is trained once on activations of a base model and then frozen. During RL, the current policy produces rollouts, while a separate frozen copy of the base model teacher-forces the same rollout text to supply the probe activations. High-theater rollouts receive zero reward and zero policy advantage. Across GSM8K, LiveCodeBench, ToolUse, and MMLU-Redux and two model architectures, ProFIL reduces post-commitment theater by 11-100%, raises faithful fraction (including +24 percentage points on LiveCodeBench under an independent GPT-4.1 judge), and shortens chains by 4-19% in three of four domains, while preserving or improving the reported task-accuracy metric. A matched length-penalty baseline worsens theater, isolating commitment detection from generic compression. Frozen-probe audits, an independent pairwise judge, and inference-time steering controls further support the result and address the RL-obfuscation concern. Probe weights, training configurations, evaluation code, and rollout caches are released for all four domains.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
chain-of-thought faithfulness, reinforcement learning, activation probes, interpretability, reasoning efficiency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yUuiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ProFIL, a method designed to reduce reasoning steps generated after the correct answer has already become recoverable from an earlier prefix. ProFIL trains a lightweight probe on frozen base-model activations using forced-answer counterfactuals, then uses the frozen probe to filter rollouts during GRPO training. Rollouts with high predicted post-commitment reasoning receive zero reward and zero policy advantage.

Critical strengths

1.This paper identifies an important limitation of reasoning models: generating more reasoning tokens does not necessarily imply more useful reasoning.

2.The proposed method integrates into GRPO by filtering rewards and advantages, without requiring changes to the underlying policy architecture or additional policy-side objectives.

Critical weaknesses

1.The answer recoverability does not establish that the preceding visible reasoning logically supports the answer or caused the model's internal computation. Thus, the paper's evidence supports reduced post-commitment reasoning more directly than improved reasoning faithfulness in the broader sense.

2.The filter assigns zero reward and advantage to an entire rollout based on its average probe score. This could discard useful verification, alternative derivations, or uncertainty management, even when the response contains some post-commitment material.

Questions for authors

1.Since forced-answer counterfactuals measure when the correct answer becomes extractable rather than whether the visible reasoning supports the answer, what evidence justifies interpreting the reduction in post-commitment steps as an improvement in reasoning faithfulness?

2.How sensitive are the results to the training threshold τ\tau? Can the threshold be selected reliably without extensive domain-specific tuning, especially in settings where the cost of suppressing useful reasoning is high?

1 comment
AuthorsOpenReview ↗

Response: temporal faithfulness and filtering

Thank you for a fair and constructive review, and for recognizing the importance of the problem and the simplicity of the GRPO integration.

W1 and Q1: Faithfulness

We agree. Our claim is about temporal faithfulness. A model often settles on its answer internally before it stops writing, and we train its visible reasoning to end at that point. §4.1 separates this from logical and causal faithfulness, and §6 lists the distinction as a limitation. Temporal faithfulness is still a meaningful part of faithfulness, because text written after the correct answer is available was not needed to produce it. A GPT-4.1 judge that never sees the probe agrees: on LCB, the fully faithful fraction rises 0.49→0.730.49 \rightarrow 0.73 (§5.4). In the final version, we will use "temporally faithful" throughout and revise the first sentence of App. I.

W2: Rollout-level filtering

If the filter removed useful reasoning, accuracy would drop. It does not. Accuracy rises on GSM8K (+5.2 pts) and ToolUse (+3 pts), stays the same on MMLU-Redux, and shows no significant change on LCB (p=0.54p = 0.54). Three properties explain this:

  • Commitment requires the correct answer: A step counts as post-commitment only after the correct answer is already available. Verification, alternative derivations, and self-correction before that point are never filtered (§4.1, App. G).
  • No length penalty: A model that needs more tokens can use them. On MMLU-Redux, chains get 5% longer at equal accuracy.
  • Removed text is redundant: It consists of restarted derivations (72% of high-theater baseline rollouts vs. 0% under ProFIL, Table 5) and re-checks of cases already solved (Fig. 4). Both occur mostly on problems the model gets right (§5.5).

The probe reads commitment from internal activations with held-out AUROC of 0.92 to 0.99 (§4.2), so the filter follows the model's internal state, not the length of its output. Filtered rollouts get zero advantage, not a penalty. Filtering individual steps is a natural refinement that we leave to future work.

Q2: Sensitivity to τ\tau

The same τ=0.5\tau = 0.5 works across domains and on both backbones (Table 2). LCB, which has sparse rewards, uses τ=0.2\tau = 0.2, and a schedule based on reward density recovers this value automatically (App. E). Training with that schedule is future work. Setting τ=1\tau = 1 recovers standard GRPO, so practitioners can start with a lenient threshold when suppressing reasoning is costly.

Overall, ProFIL produces chains up to 19% shorter (Table 1) that are easier to monitor, while maintaining accuracy. We hope this addresses your concerns.

Reviewer_NaftOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ProFIL aims to reduce reasoning that continues after a correct answer becomes recoverable. Offline forced-answer queries identify the first correct prefix and label later steps as post-commitment reasoning. An attention probe is trained on frozen-base activations, then held fixed. During GRPO, a frozen copy of the base model teacher-forces policy rollouts; the probe's scores gate both verifier rewards and policy advantages. Experiments on four domains report lower probe-positive rates, shorter responses in three domains, and independent-judge preferences for less padding. The paper also presents a length-penalty control and probe audits. Task-accuracy conclusions depend on the evaluation protocol, particularly for LiveCodeBench.

Critical strengths
  1. The method offers a concrete training intervention beyond generic length penalties (§4; Table 3). Using a frozen activation-producing model as well as a frozen probe avoids changing the scorer's representation during policy updates. The matched length-penalty experiment is valuable: shorter responses alone do not reproduce the reduction in the targeted behavior.

  2. Independent checks make the evidence stronger than optimization of a single learned score (§5.4). The probe-blind GPT-4.1 comparison on LiveCodeBench reports a strict faithful-fraction increase from 0.49 to 0.73, and the randomized pairwise judgments provide complementary evidence of reduced padding. The paper also explicitly distinguishes temporal, logical, and causal faithfulness and discloses single-run training variance limitations.

Critical weaknesses
  1. The main behavioral metric remains insufficiently connected to the stated post-commitment construct (§§4.1–4.2; §5.4; Appendix A.2). For three domains, the headline “performativity ratio” is a thresholded pooled chain score, not the fraction of steps after the first correct forced answer. Uncommitted chains receive zero target ratio, and a first correct forced answer need not remain correct at later prefixes. On LiveCodeBench, correctness for this labeling uses a test-pass rate above 0.5. Thus, suppressing the probe-positive rate can differ from removing unnecessary reasoning while retaining useful verification. The independent judges support reduced padding, but the AUROC audit establishes ranking discrimination rather than directly quantifying changes in suffix length or commitment stability. This is consequential because the paper frames the result as improved faithfulness, while the clearest evidence concerns a narrower writing behavior.

  2. The main accuracy-preservation presentation combines incompatible evaluation points (§5.2; Tables 1 and 3). Table 1 reports LiveCodeBench's best sampled validation accuracy during training, 0.30 versus 0.37, alongside final greedy behavior measurements. At the final greedy checkpoint on all 131 problems, accuracy is 0.214 for baseline and 0.183 for ProFIL. The paper acknowledges this distinction, but the sampled peak cannot establish that the policy exhibiting reduced theater preserves task performance under that same evaluation. The small test set does not prove a significant accuracy loss either. A consistent checkpoint and decoding protocol is needed to substantiate the claimed behavior–accuracy trade-off.

Questions for authors
  1. From the fresh forced-answer audit already conducted, what are the actual post-commitment suffix ratios, uncommitted-chain rates, and rates of later incorrect forced answers after the first correct prefix? Please distinguish these from probe AUROC.
  2. Were the 0.30 and 0.37 sampled peaks selected on a validation set disjoint from the 131 final test problems? Which exact checkpoints were used for the probe, judge, and accuracy results, and are matched-checkpoint sampled results already available?
  3. How does the LiveCodeBench forced-answer procedure produce executable code within the stated 64-token limit, and does the above-0.5 pass-rate criterion also define the reported task accuracy? Clarifying these details would materially affect the interpretation of commitment and correctness.
2 comments
AuthorsOpenReview ↗

Response: metric validity and accuracy protocol

Thank you for a careful review. Evidence already in the paper addresses both weaknesses.

W1: Metric vs. construct

Our hypothesis is simple. A model often settles on its answer internally before it stops writing, and its visible reasoning should end at that point. This timing is the construct (§4.1), so the "narrower writing behavior" the review points to is exactly what we target. A step is labeled post-commitment only after the correct answer is already available. The frozen probe detects this from internal activations with held-out AUROC of 0.92 to 0.99 (§4.2).

Independent measures, none of which see the probe, confirm that PR tracks this behavior:

  • GPT-4.1 judge: It flags steps written after the model has decided. Strict faithful fraction rises 0.49→0.730.49 \rightarrow 0.73, with non-overlapping CIs (§5.4).
  • Blinded pairwise judge: It rates the baseline as more padded in 57 to 72% of pairs. The length-penalty model gets no advantage (51%).
  • Length-penalty control: It shortens chains but raises PR (Table 3), so PR is not a proxy for length.
  • Re-derivation headings: They appear in 31/43 high-theater baseline rollouts and in 0/131 ProFIL rollouts (Table 5).
  • Uncommitted chains: Conditioning on correctness gives the same result (§5.5), so these chains do not drive it.

Answer flips: Text written after the answer is available is post-commitment by definition. If the answer later flips to a wrong one, that extra text did harm. Reporting forced-answer suffix ratios next to PR is a useful addition that we leave to future work.

W2: Accuracy protocols

We agree that Table 1 should not place the sampled LCB peak next to greedy behavior metrics. In the final version, we will report both protocols side by side and describe LCB accuracy as statistically indistinguishable. This still shows no trade-off. The greedy gap is 4 of 131 problems (p=0.54p = 0.54), and the length penalty scores higher (30/131) while increasing theater. As the review notes, the test set "does not prove a significant accuracy loss." In the other three domains, accuracy rises or stays the same (Table 1).

Questions

  • Probe: It is trained once on frozen-base rollouts before RL and never updated (§4.2).
  • Checkpoints: For each condition, all behavior, accuracy, and judge results come from the same evaluated checkpoint.
  • 64-token cap: It applies to the boxed-answer domains, not to LCB. We will rewrite §5.1 to state each domain's labeling procedure clearly.
  • LCB scoring: Reported accuracy uses the same execution scorer as the training reward (App. B), applied identically to every condition.
Reviewer_NaftOpenReview ↗

Concern remains

Thank you for the clarifications. The 64-token cap, LCB scoring, and matched-checkpoint statements resolve several procedural questions. The independent judges and length-control experiment also support the narrower conclusion that ProFIL reduces padding or post-decision writing.

However, my main construct-validity concern remains only partially addressed. High probe AUROC and correlations with independent judges do not make the headline PR equivalent to the actual fraction of reasoning after a stable commitment. Treating all text after the first correct forced answer as post-commitment assumes that this answer reflects a settled decision—particularly when later forced answers may flip. The requested suffix ratios, uncommitted-chain rates, and answer-flip rates would directly test this assumption; otherwise, the faithfulness claim should be narrowed to the demonstrated writing behavior.

For accuracy, reporting both protocols side by side is appropriate. However, p=0.54p=0.54 establishes only that no difference was detected on 131 problems, not that there is “no trade-off.” The paper should use the more precise “statistically indistinguishable” formulation and clarify whether peak checkpoint selection used a disjoint validation set.

Overall, the response improves the presentation but does not fully resolve the principal metric-validity concern.

Reviewer_eDwkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Reasoning models can have a tendency to continue their chain of thought even when they have enough to produce the correct answer. The paper calls these excess steps post-commitment theatre. The paper proposes a method (ProFIL) to fix this issue. ProFIL works as follows:

  • Take an initial model (before RL) and generate reasoning traces from it.
  • Cut off the reasoning prefixes at several points and force the model to output an answer. For each such prefix, record whether the model can produce an answer or not.
  • Use this data to train a small probe model, that takes model activations corresponding to a prefix and predicts whether the prefix is enough for the model to answer correctly.
  • Now, during RL with GRPO, use this probe to mark traces that produce excessively long responses beyond the point enough to produce the correct answer. These traces are given 0 reward, and after advantage computation, there advantage is set to 0 as well.

Across benchmarks including GSM8K and LiveCodeBench, ProFIL is shown to substantially reduce post-commitment theatre, while preserving or improving accuracy.

Critical strengths
  • The problem tackles a timely problem of reasoning models producing thinking tokens even when they can stop.

  • After the initial processing step, the method is easy to integrate into existing GRPO pipelines.

  • The paper shows that merely using a length penalty doesn't fix this issue, and one may need to specifically focus on these post-commitment tokens to reduce them. I found this comparison helpful.

Critical weaknesses

While the method proposed in the paper is fairly natural and easy to integrate, it's evaluation needs to be strengthened significantly before one can reliably trust its usefulness.

  1. Use of greedy decoding: The paper says that the accuracy evaluation of the trained model is greedy@1. If this means evaluation is done with greedy decoding (or 0 temperature), this can significantly alter the results. Reasoning models are known to produce unstable looping like behavior with greedy decoding [1] and hence are recommended to be used at higher temperatures. Therefore, all evaluations should be done at higher temperature (e.g. 0.6-0.8) to ensure the conclusions are not affected by an unreasonable temperature choice.

  2. Missing prior work comparison: The prior work S-GRPO [2] also trains models to stop reasoning earlier when possible, and doesn't seem to need any pre-processing step of training a probe. Although the paper cites this work, the proposed method should also be evaluated against this arguably simpler baseline.

  3. Limited coverage of benchmarks: A natural concern is that training the model to stop thinking earlier can hurt accuracy, especially on challenging problems where such additional thinking can help with useful verification. Evaluation on more challenging benchmarks such as AIME and MATH500 can reveal such issues. The paper currently misses these evaluations. In fact, the prior work on S-GRPO [2] seems to include these, so just mimicking their training and eval setup for these benchmarks can be a useful starting point. This will significantly strengthen the claims in the paper.

[1] Pipis, Charilaos, et al. "Wait, Wait, Wait... Why Do Reasoning Models Loop?." ICML 2026.

[2] Dai, Muzhi, Chenxu Yang, and Qingyi Si. "S-grpo: Early exit via reinforcement learning in reasoning models." NeurIPS 2025.

Questions for authors

Addressing the points raised in weaknesses section would be helpful.

Another question on evals: For live code bench, the highest accuracy across training is reported. Is a validation set used to find the checkpoint with highest accuracy, and then a different held-out test set used to report the accuracy? If the two sets are the same, there could be overfitting.

3 comments
AuthorsOpenReview ↗

Response: decoding, S-GRPO, and benchmark difficulty

Thank you for your careful review. Some premises in this review conflict with the paper's data. We address each below.

W1: Greedy decoding

The looping explanation predicts the opposite of what we observe.

  • Same decoding for both: Baseline and ProFIL use identical decoding (§5.1), so decoding alone cannot create the gap between them.
  • Theater is not looping: Loops make outputs longer, yet baseline theater is highest in the shortest 20% of responses. That is exactly where ProFIL reduces it most (0.59→0.090.59 \rightarrow 0.09, Fig. 5).
  • Training uses sampling: The filter scores sampled GRPO rollouts (§4.3), so ProFIL learns its behavior under sampling.

W2: S-GRPO

S-GRPO is not simpler than ProFIL.

  • Cost: S-GRPO generates and scores early-exit answers at several points in every rollout, at every training step. On LCB, each exit also requires running another program. ProFIL finds the commitment point once, offline, and adds only one forward pass per rollout during RL (319 ms vs. 533 ms for a policy update, App. H).
  • Objective: S-GRPO's reward decays with exit position, which acts as a brevity penalty. Table 3 shows that brevity pressure increases theater (0.267→0.3740.267 \rightarrow 0.374), while ProFIL reduces it (0.0760.076).

We agree that S-GRPO is the closest prior method and will add an explicit comparison to §2.

W3: Harder benchmarks

For our backbone, AIME and MATH-500 are easier than LCB. DeepSeek reports R1-Distill-Llama-8B at 50.4 on AIME 2024, 89.1 on MATH-500, and 39.6 on LCB (DeepSeek-AI, 2025).

Our goal is to remove reasoning after the answer is settled while keeping accuracy, not to force short chains. The paper directly tests whether useful verification is lost:

  • No length penalty: The model can use as many tokens as it needs. On MMLU-Redux, chains get 5% longer at identical accuracy (Table 1).
  • Commitment requires the correct answer: A step counts as post-commitment only after the correct answer is already available. Earlier reasoning, including verification that fixes a wrong answer, is never filtered (§4.1, App. G).
  • Theater appears on solved problems: High-theater rollouts are 33% accurate, vs. 9% for low-theater ones (§5.5).

Evaluating on AIME is a natural extension that we leave to future work.

Q: LCB accuracy

Our LCB claim is theater reduction, measured with greedy decoding at a fixed checkpoint and no tuning on the theater metric. §5.2 presents the sampled peak as training telemetry, not as a held-out result. Under greedy decoding, accuracy differs by 4 of 131 problems (p=0.54p = 0.54), which is no meaningful difference in either direction.

Thanks for the response. Unfortunately, it doesn't address my concerns, so I will stick to my score. My concerns were mainly about the evaluations, and the response mostly explains why each of them may not be a problem. While that may well be true, only a more thorough evaluation can actually address these concerns.

Regarding greedy decoding, it is totally possible that it is not influencing the results. But since it is a non-standard evaluation choice for reasoning models, the results would be much more trustworthy if they were done at higher temperatures (e.g., 0.6), where these models are typically used. Similarly, without a direct experimental comparison against close baselines like S-GRPO, it is hard to understand the true value of the method. Also, evaluating on a diverse set of benchmarks like MATH-500 and AIME is pretty standard for this kind of work (e.g., the closely related S-GRPO work evaluates on these), and I don't think it can be left for future work.

AuthorsOpenReview ↗

New results at T = 0.6 on GSM8K, MATH-500, and AIME

Thank you. We ran the requested evaluations: T=0.6T = 0.6, top-p 0.95, 8 samples per problem, paired bootstrap CIs.

Faithfulness holds under sampling (GSM8K)

Performative chains fall from 19.5% to 1.7% (a 92% reduction; 95% CI for the difference: −0.192-0.192 to −0.166-0.166). The drop is the same within correct chains (0.197→0.0160.197 \rightarrow 0.016) and incorrect chains (0.180→0.0220.180 \rightarrow 0.022). ProFIL also uses 59% fewer tokens (325 vs. 792), at an accuracy of 0.908 vs. 0.924.

Zero-shot transfer to MATH-500 and AIME

Both checkpoints were trained only on GSM8K and never saw these benchmarks. Without any tuning:

BenchmarkBaselineProFILTokens
MATH-5000.880.8625% fewer
AIME 20240.380.35 (n.s.)about equal
AIME 20250.260.22 (n.s.)about equal

Caveats

  • Accuracy: Under sampling, ProFIL is 1.6 points lower on GSM8K. We will report T=0.6T = 0.6 as another protocol
  • Faithfulness scope: It was measured on GSM8K only, where the probe was trained.
  • S-GRPO: A matched training run did not fit in the discussion period. S-GRPO rewards early exit, which is length pressure, and our length-penalty baseline (Table 3) shows that length pressure alone increases theater.

Breadth

ProFIL is evaluated on math, competitive programming (LiveCodeBench), agentic tool use, and knowledge QA, across two model families. S-GRPO evaluates math and science QA, and does not test code or tool use.

We hope these address your concerns!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.