- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ProFIL, a method designed to reduce reasoning steps generated after the correct answer has already become recoverable from an earlier prefix. ProFIL trains a lightweight probe on frozen base-model activations using forced-answer counterfactuals, then uses the frozen probe to filter rollouts during GRPO training. Rollouts with high predicted post-commitment reasoning receive zero reward and zero policy advantage.
Critical strengths
1.This paper identifies an important limitation of reasoning models: generating more reasoning tokens does not necessarily imply more useful reasoning.
2.The proposed method integrates into GRPO by filtering rewards and advantages, without requiring changes to the underlying policy architecture or additional policy-side objectives.
Critical weaknesses
1.The answer recoverability does not establish that the preceding visible reasoning logically supports the answer or caused the model's internal computation. Thus, the paper's evidence supports reduced post-commitment reasoning more directly than improved reasoning faithfulness in the broader sense.
2.The filter assigns zero reward and advantage to an entire rollout based on its average probe score. This could discard useful verification, alternative derivations, or uncertainty management, even when the response contains some post-commitment material.
Questions for authors
1.Since forced-answer counterfactuals measure when the correct answer becomes extractable rather than whether the visible reasoning supports the answer, what evidence justifies interpreting the reduction in post-commitment steps as an improvement in reasoning faithfulness?
2.How sensitive are the results to the training threshold ? Can the threshold be selected reliably without extensive domain-specific tuning, especially in settings where the cost of suppressing useful reasoning is high?
Response: temporal faithfulness and filtering
Thank you for a fair and constructive review, and for recognizing the importance of the problem and the simplicity of the GRPO integration.
W1 and Q1: Faithfulness
We agree. Our claim is about temporal faithfulness. A model often settles on its answer internally before it stops writing, and we train its visible reasoning to end at that point. §4.1 separates this from logical and causal faithfulness, and §6 lists the distinction as a limitation. Temporal faithfulness is still a meaningful part of faithfulness, because text written after the correct answer is available was not needed to produce it. A GPT-4.1 judge that never sees the probe agrees: on LCB, the fully faithful fraction rises 0.49→0.73 (§5.4). In the final version, we will use "temporally faithful" throughout and revise the first sentence of App. I.
W2: Rollout-level filtering
If the filter removed useful reasoning, accuracy would drop. It does not. Accuracy rises on GSM8K (+5.2 pts) and ToolUse (+3 pts), stays the same on MMLU-Redux, and shows no significant change on LCB (p=0.54). Three properties explain this:
The probe reads commitment from internal activations with held-out AUROC of 0.92 to 0.99 (§4.2), so the filter follows the model's internal state, not the length of its output. Filtered rollouts get zero advantage, not a penalty. Filtering individual steps is a natural refinement that we leave to future work.
Q2: Sensitivity to τ
The same τ=0.5 works across domains and on both backbones (Table 2). LCB, which has sparse rewards, uses τ=0.2, and a schedule based on reward density recovers this value automatically (App. E). Training with that schedule is future work. Setting τ=1 recovers standard GRPO, so practitioners can start with a lenient threshold when suppressing reasoning is costly.
Overall, ProFIL produces chains up to 19% shorter (Table 1) that are easier to monitor, while maintaining accuracy. We hope this addresses your concerns.