- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper is mainly concerned with the validity of faithfulness estimates that modify or intervene on models' reasoning traces. Specifically, the paper wants to mitigate the cases where "replay" is invalid. And by "replay" the authors mean cases in which the authors extract the reasoning, append a fixed "Final answer:" cue, and allow a new continuation of up to 1,024 tokens, but instead of producing the final answer, the model just keeps reasoning.
Critical strengths
None
Critical weaknesses
I might be misunderstanding the paper, but it seems the authors are trying to solve a problem that appears just because they did not use proper formatting or chat template? In Appendix D.1, they say:
We did not test whether another cue, such as a native chat template with thinking delimiters, restores consistency on those pairs.
So that means that the failure they observe in models, continuing the reasoning after "Final answer:" cue, is probably attributable to inference setup issues.
Besides that, not all papers in the (un)faithfulness literature use modified reasoning traces, making the usefulness of this approach limited in that regard.
Presentation is also quite bad, and there's a lot of AI-generated content. I struggled to understand what this paper was about. The following is in the first paragraph of intro, which is already citing an Appendix section. What 6 decodes is this referring to? So confusing.
When we perturb 1 clause so that a finding of no association becomes a finding of harm, the model changes its answer from no to yes on every decode. The same edit does not change the answer on any of the 6 decodes for a second chain from that model and benchmark (Appendix E.6).
Also, authors use the expression "reproduce the model" and "replay" all the time, without really explaining what they mean.
Questions for authors
- Why is "replay" just appending "Final answer:" to the extracted reasoning?
- Do the experiments use the proper role markers, thinking channel delimiters, etc?
- What problem is this paper really trying to address?
Final remarks
Dear Reviewers and AC,
Thank you for your time and consideration. If any concern remains open, we would be glad to answer it before the discussion closes on 18th of November (AoE). Tables R1 to R8 are new work for this response. The revised PDF contains every change named below.
Confirmed in the discussion. Reviewer shFk wrote: "The threshold sensitivity analysis, the replay re-run at a larger answer budget, the accuracy breakdown under each generator, and the removal experiment for mathematics have addressed my concerns on those points." Reviewer N1y7 wrote: "The clarification about templates resolves my concerns on the formatting."
1. Replay setup (all reviewers). The replay failures came from an answer budget of 128 tokens. Of the 18,682 failed greedy replays, 18,612 end at this budget. We re-ran 22 of the 30 replay-invalid pairs at 4,096 tokens. All 22 pass replay validity (answer consistency 0.878 to 1.000, Table R1, i.e., Table 1 of the revision). Under the 128-token budget, the truncation baseline is negative on 6 of the failing pairs. A study that took this baseline as faithfulness would have concluded that the models of these 6 pairs ignore their chains.
2. Answer budget and the 77 verified estimates (Reviewer shFk). Some decodes end at the budget before they state an answer. If we count them as correct, the clinical mean changes by 0.023 and the GPQA mean changes from 0.495 to 0.350 (Table R7). On 18 verified pairs decoded again at 4,096 tokens, the clinical mean on the same items differs from the submitted mean by -0.011 [-0.031, 0.008] (Table R8). In mathematics, the estimate from the decodes cut at their 128th token matches the submitted estimate (difference -0.012 [-0.029, 0.003]). Scored on the whole decode, the estimate is lower (difference -0.147 [-0.175, -0.117]), because the models solve the problem again after the answer on their first line. In Section 4 and the caption of Table 2, we now state that the science values depend on the answer budget (Appendix D.16).
3. Stated conclusions in clinical chains (Reviewer shFk). In a test on 16 pairs of 5 models, removing every answer statement from the chains reduced the clinical estimates by 45% and left the mathematics estimates almost unchanged (0.444 to 0.436, Table R5b). After the removal, (M2) passes on 4 of the 8 clinical pairs. In the abstract, the caption of Table 2, and a new paragraph of the Discussion, we now state that a clinical estimate includes the dependence on the stated conclusion. We treat this result as preliminary.
4. Generator dependence (Reviewers QSSJ and shFk). At a matched 128-token budget, the estimates of 6 pairs under the 2 generators differ by 0.299 on average (0.341 as submitted, Table R2). The 6 pairs have almost the same order under both generators (Spearman 0.943). Neither the evaluator's recovery drop nor the change of the stated answer accounts for the difference. In the abstract, we now state that estimates can be compared under one generator but do not give an absolute level of faithfulness. In the Discussion, we name calibration across generators an open problem. Reviewer shFk wrote: "I accept that this is acknowledged as an open problem and I do not expect it to be solved in this paper."
5. Length and fluency matching (Reviewer QSSJ). With matching enforced per item, the estimate of a pair changes by a median of 0.046 under the length criterion and 0.056 under the fluency criterion (Table R3, Appendix D.13). On matched items, the estimates are higher on average. So the residual mismatch does not inflate them on average. We replaced "by construction" with "approximately, through one generator and one instruction".
6. NSF-CoT and recent work (Reviewer QSSJ). On 90 PubMedQA items of 3 verified pairs, we scored the chains with the LLM judge of NSF-CoT. The product of its groundedness and utility scores correlates with the CRISP effect at Spearman 0.336 (p = 0.001, Table R4). The 2 measures agree only in part. We now discuss NSF-CoT in Related Work and in Appendix D.15.
7. Problem, scope, and presentation (Reviewer N1y7). In the first paragraph of the revised introduction, we state the problem and define replay.
Contribution. CRISP tests the 2 assumptions of an intervention-based faithfulness estimate. In this study, replay validity detected the harness failure on 30 pairs. Intervention validity identified 28 more pairs. On these pairs, the perturbation changed the answer recovered by the evaluator on 29% of the checked items with a recoverable answer. On the verified pairs, it changed this answer on 55% (means per pair). At the cost of one teacher-forced condition and 3 evaluator calls per checked item, a study can establish that its numbers are interpretable before using them.
We hope that these results are useful as you finalize your recommendations.
The Authors