acceptodds
Under review as a conference paper at ICLR 2027

Filtering Low-Q Tokens: Improving Success Rates while Preserving Base-Model Successes

Abstract

LLMs are highly capable, yet they may still be imperfect with respect to a variety of target criteria, such as safety, contextual faithfulness, or reasoning. A natural goal is therefore to take a capable, trained LLM and augment it to increase its rate of successful responses. This augmentation should preserve the diverse successful outputs the base model already produces. We derive such an augmentation by testing each candidate token against the hypothesis that the final output will be successful. We show that the success probability of the full generation is lower-bounded by a quantity increasing in the power of this test. Our method, *Q-filtered decoding* (QFD), therefore follows from the Neyman-Pearson lemma, which identifies the optimal test of this form at a given false-rejection rate: a threshold on a token's probability of leading to a successful final output. At each decoding step, a small learned probe estimates this probability for every candidate token; those falling below the threshold are removed, and the remaining distribution is renormalized. Raising the threshold makes the filter intervene more aggressively, at the cost of altering more of the base model's successful outputs. Across mathematical reasoning, contextual faithfulness and harmful compliance, in seven task-model pairs, QFD significantly improves the success rate on six. We measure preservation by the drop rate, the probability that an intervention rejects a successful generation of the base model, and by the cosine similarity of successful completions to those of the base model. No baseline, including LoRA fine-tuning and activation steering, dominates QFD: at 28 of its 35 settings, no baseline reaches as low a drop rate, and at the rest, those that do are less successful. On four of the six pairs, QFD's largest improvement in success rate is comparable to the best baseline's, yet it drops 54-87% of the base model's successful generations instead of 96-100%, and its completions are closer to the base model's in cosine similarity. QFD slows generation only by 1.2% on average.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
inference-time intervention, controlled decoding, large language models, Q-function, hypothesis testing, LLM safety, Neyman-Pearson lemma, test-time alignment, steering, distribution preservation, jailbreak robustness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ip69OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an inference-time intervention method called QFD. The main novelty lies in formulating Q-based token filtering as a hypothesis-testing problem, deriving success and preservation guarantees for the resulting decoding policy, and implementing it through a lightweight Q-estimator. The Q-estimator fits a d×dd \times d transformation using the frozen LLM's hidden states and output head. The theory explains why filtering low-Q tokens is a reasonable strategy. In particular, given exact Q-values, the filtering rule optimizes a lower bound on the model's success rate under a constraint on mistakenly filtering successful tokens, while providing a guarantee on preserving previously successful generations.

Critical strengths

The intervention method brings meaningful empirical improvements with lightweight implementation. The performance in the reported settings is quite strong and also has evidence that this intervention materially changes the outcomes. The theory is clean written. It gives evidence for connecting local filtering decisions to full-generation success and provides a preservation bound that does not explicitly grow with trajectory length.

Critical weaknesses

My main concern about the paper is the gap between the theory and the proposed algorithm. The hypothesis-testing perspective seems somewhat disconnected from the actual design of Q-filtered decoding, as the filtering rule itself appears to follow naturally from the Q-value interpretation without requiring this framework. It is therefore unclear to me whether the paper's main contribution lies in the hypothesis-testing perspective or in the algorithmic improvements over existing value-filtered decoding methods.

Also, to my understanding, the drop rate measures how likely a successful trajectory is to contain a token removed by the filter. I don't see why a large drop rate necessarily indicates poor preservation, as suggested in Figure 1. For example, a reweighting policy that changes [1−ϵ,ϵ][1 - \epsilon, \epsilon] to [1−2ϵ,2ϵ][1 - 2 \epsilon, 2\epsilon] has a drop rate of exactly 0.5 for any $ 0<\epsilon < 1/2 $, assuming both tokens lead to success, even though the two distributions become arbitrarily close as ϵ→0\epsilon \rightarrow 0. This makes me confused on what the drop rate is intended to show for comparing QFD among reweighting policies.

My current rating reflects these concerns, but I am open to raising it if the authors can adequately address them during the rebuttal.

Questions for authors

Please see the weaknesses above. I would particularly appreciate clarification on what the paper is intended to present and the interpretation of drop rate when comparing different reweighting policies.

Reviewer_e4T7OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors aim to preserve the base model's successful trajectories during finetuning i.e. ensure that after finetuning, the base model answers similarly on questions in got right. They propose a decoding algorithm to solve this problem. Experiments show their method is at the pareto frontier of performance vs "preservation of the base model's successful trajectories" on a bunch of models x tasks compared to a few baselines.

Critical strengths

The authors covered three model families and many tasks. This shows the generalization of their method and reduces the likelihood it was tuned specifically for one setup.

Critical weaknesses
  1. Motivation: Why is the problem important? Why would one want to preserve base model's answers on questions it got right as long as overall accuracy is good? Also I do not understand how this necessarily preserves the diversity? why would one want to particularly minimize the metric drop rate?

  2. Evaluation: There is no comparison with GRPO or many of the other RL based fine-tuning methods. Also, I am not sure if drop-rate quantifies the probability with which original trajectory will necessarily be preserved especially across different methods.

Questions for authors

See weakness.

Reviewer_tjNTOpenReview ↗
Rating
3
Confidence
2
Informativeness
3(+1)
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Q-filtered decoding (QFD), an inference-time intervention that raises an LLM's success rate on a binary criterion (math correctness, contextual faithfulness, or safety) while limiting how often it alters generations the base model would already have gotten right. At each decoding step, a single d×d matrix reusing the frozen LM head estimates, for every candidate token, the probability that selecting it leads to a successful completion (the Q-function); tokens below a threshold c are masked and the rest renormalized, with c set from a user-specified level α, the fraction of the base model's successful validation completions the filter may alter. Framing token removal as a statistical test of whether the generation will succeed, the authors use the Neyman–Pearson lemma to show that, for a fixed rate of wrongly removing tokens from successful generations, filtering by Q-value is the optimal rule, and introduce the drop rate — how often an intervention changes a generation the base model would have gotten right — as a preservation metric that applies uniformly to interventions of different families (decoding-time methods, LoRA fine-tuning, and activation steering), allowing them to be compared under the same notion of preservation. Across seven task–model pairs, QFD significantly improves the success rate on six, no baseline matches its success rate at an equal or lower drop rate, and the overhead is at most 0.25% parameters and about 1.2% decoding time.

Critical strengths
  • S1. The motivation, theoretical grounding, and experimental results of the proposed method are all consistent with one another. The proposed decoding method also has the practical advantage of being explicitly controllable by setting α. The sweep experiments show that the drop rate is closely reproduced by α (32 of 35 settings). Moreover, on six of the seven task–model pairs the experiments show that the method significantly improves the success rate while preserving the base model's correct generations to the extent specified by α. (In contrast, the baselines tend to alter the base model's generations excessively, and in some cases also lower accuracy.)
  • S2. The provided code supports reproducibility. The equations were somewhat hard to follow, but they can be cross-checked against the code.
  • S3. The additional cost is small. The estimator is a single d×d matrix that reuses the existing LM head, adds at most 0.25% to the active parameters, and produces Q-values for all candidate tokens in a single computation. The increase in decoding time is 1.2% on average over the 35 settings, and at most 3.2% on six of the seven task–model pairs. Given that the method adds an intervention on top of an already-trained model, I consider this level of overhead highly practical.
Critical weaknesses
  • W1. Most of the paper is devoted to establishing the theoretical foundation. It would be helpful to add more empirical results on the effects of applying the method, such as changes in the number of generated tokens or a section with qualitative analysis of samples.
  • W2. There is no sensitivity analysis for hyperparameters other than α. The relationship to the base model's own sampling parameters (e.g., top-p, top-k) is discussed theoretically, but not examined empirically; an ablation varying top-p would show how sensitive QFD's success rate and drop rate are to the size of the candidate set.
  • W3. On SALAD-Bench with Nemotron at α=0.85, 25 of the 627 QFD completions (about 4%) contain runs of <unk> tokens, but the paper only mentions that these were counted as failures, without analyzing the cause. Since this appears to be a case where estimator error surfaces in the most aggressive setting, a discussion of the cause (e.g., fallback behavior on a model without top-k truncation) and possible mitigations is needed.
Questions for authors

Please see weaknesses.

Comments on the paper

1 comment

We thank all reviewers for their valuable feedback and constructive suggestions. We have uploaded a revised manuscript that incorporates the feedback received. The main changes are summarized below:

  1. Coupled sampling and a full redraw. Every method now samples with the same random draws as the base model, so that each completion of a method is paired with the base model’s completion from the same draws. To this end, we regenerated all training, validation and test completions in one software environment, and retrained every estimator and baseline with the same protocol as before. Coupled sampling lets us measure exactly how often a method leaves a successful generation of the base model unchanged.

  2. Additional similarity metrics: We introduced two new metrics to better quantify preservation of the base model's behavior: Trajectory Retention-Rate (TR) and the fraction of unchanged successful responses. We also report the cosine similarity and the BLEU score between each successful completion and the base model's successful completions on the same prompt, which also credit completions that change only slightly. We also report the Anytime Retention Rate (AR), defined as 1-DR, to consistently express all metrics in terms of similarity rather than dissimilarity.

  3. Additional intervention cost metrics: We also report the cost of each setting in a property that the criterion does not measure: over-refusal of safe requests on the harmful-compliance tasks, over-abstention on answerable questions on FaithEval, and affirmative answers to yes/no problems whose answer is negative on DeepMath.

  4. Refined theoretical analysis: We reframed our theoretical analysis around the retention rate of successful base-model outputs, establishing how this quantity is controlled by the intervention threshold.

  5. Additional Model-Dataset pairs: We added Gemma x DeepMath, WildJailbreak to the experiments.

  6. Qualitative examples: We added qualitative examples in the appendix to illustrate the behavior of our method and highlight its strengths and limitations.

  7. Clearer contributions: We revised the Introduction to better emphasize our main contributions and distinguish our approach from existing methods.

We appreciate the reviewers' time and thoughtful feedback, which have helped us improve the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.