- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes an inference-time intervention method called QFD. The main novelty lies in formulating Q-based token filtering as a hypothesis-testing problem, deriving success and preservation guarantees for the resulting decoding policy, and implementing it through a lightweight Q-estimator. The Q-estimator fits a transformation using the frozen LLM's hidden states and output head. The theory explains why filtering low-Q tokens is a reasonable strategy. In particular, given exact Q-values, the filtering rule optimizes a lower bound on the model's success rate under a constraint on mistakenly filtering successful tokens, while providing a guarantee on preserving previously successful generations.
Critical strengths
The intervention method brings meaningful empirical improvements with lightweight implementation. The performance in the reported settings is quite strong and also has evidence that this intervention materially changes the outcomes. The theory is clean written. It gives evidence for connecting local filtering decisions to full-generation success and provides a preservation bound that does not explicitly grow with trajectory length.
Critical weaknesses
My main concern about the paper is the gap between the theory and the proposed algorithm. The hypothesis-testing perspective seems somewhat disconnected from the actual design of Q-filtered decoding, as the filtering rule itself appears to follow naturally from the Q-value interpretation without requiring this framework. It is therefore unclear to me whether the paper's main contribution lies in the hypothesis-testing perspective or in the algorithmic improvements over existing value-filtered decoding methods.
Also, to my understanding, the drop rate measures how likely a successful trajectory is to contain a token removed by the filter. I don't see why a large drop rate necessarily indicates poor preservation, as suggested in Figure 1. For example, a reweighting policy that changes to has a drop rate of exactly 0.5 for any $ 0<\epsilon < 1/2 $, assuming both tokens lead to success, even though the two distributions become arbitrarily close as . This makes me confused on what the drop rate is intended to show for comparing QFD among reweighting policies.
My current rating reflects these concerns, but I am open to raising it if the authors can adequately address them during the rebuttal.
Questions for authors
Please see the weaknesses above. I would particularly appreciate clarification on what the paper is intended to present and the interpretation of drop rate when comparing different reweighting policies.
We thank all reviewers for their valuable feedback and constructive suggestions. We have uploaded a revised manuscript that incorporates the feedback received. The main changes are summarized below:
Coupled sampling and a full redraw. Every method now samples with the same random draws as the base model, so that each completion of a method is paired with the base model’s completion from the same draws. To this end, we regenerated all training, validation and test completions in one software environment, and retrained every estimator and baseline with the same protocol as before. Coupled sampling lets us measure exactly how often a method leaves a successful generation of the base model unchanged.
Additional similarity metrics: We introduced two new metrics to better quantify preservation of the base model's behavior: Trajectory Retention-Rate (TR) and the fraction of unchanged successful responses. We also report the cosine similarity and the BLEU score between each successful completion and the base model's successful completions on the same prompt, which also credit completions that change only slightly. We also report the Anytime Retention Rate (AR), defined as 1-DR, to consistently express all metrics in terms of similarity rather than dissimilarity.
Additional intervention cost metrics: We also report the cost of each setting in a property that the criterion does not measure: over-refusal of safe requests on the harmful-compliance tasks, over-abstention on answerable questions on FaithEval, and affirmative answers to yes/no problems whose answer is negative on DeepMath.
Refined theoretical analysis: We reframed our theoretical analysis around the retention rate of successful base-model outputs, establishing how this quantity is controlled by the intervention threshold.
Additional Model-Dataset pairs: We added Gemma x DeepMath, WildJailbreak to the experiments.
Qualitative examples: We added qualitative examples in the appendix to illustrate the behavior of our method and highlight its strengths and limitations.
Clearer contributions: We revised the Introduction to better emphasize our main contributions and distinguish our approach from existing methods.
We appreciate the reviewers' time and thoughtful feedback, which have helped us improve the manuscript.