- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper introduces prefix filters: short Python predicates, specific to one model and one domain, that flag a partial LLM output as likely to fail a validity oracle. Experiments cover MLIR, SMILES molecules, PII-free dialogue summaries and TypeScript, with 7 open models and SQL in the appendix.
Critical strengths
The technique proposed (prefix filter) is easy to compute and interpretable. Additionally, the paper points out the interesting phenomenon of a clustered failure mode on lower-resource domains, which is important for further training and also interesting to know.
Critical weaknesses
The most natural baselines are missing. This includes compute-matched rejection sampling, prompting with learned error patterns, constrained decoding, etc. Additionally, in figure 4, it seems that LoRA is excelling on domains like typescript and summaries.
Additionally, there are several soundness concerns. Soundness is checked on a limited number of samples (e.g., only 3-17 valid samples for MLIR). There is also possible leakage concerns in the typescript validation. Furthermore, several numbers disagree across sections: Qwen-14B MLIR with grammar-constrained decoding (sec. 1 vs. table 2), token overhead (sec. 5.4 vs. appendix E), and share of errors from Qwen-1.5B's top MLIR filter (fig. 5).
Questions for authors
- How many filter triggers would actually have produced an invalid sample under the oracle?
- Does the external TypeScript validation set include reference solutions for the 50 test problems?
- How does the method actually compares with the baselines pointed above? Or, why were some of the baselines not applicable?
We thank the reviewers for their reviews. This document serves as a short response to all three reviewers.
We plan to conduct an extra experiments comparing against rejection sampling, and using filters in the prompt, as suggested.
On the comparison against LoRA DPO, one can use the results from round1 PALLA if one wishes to make a comparison of performance achieved using the same samples. We believe it is not a valid comparison to compare PALLA against LoRA DPO trained on examples from all three iterations, as LoRA DPO only would not have been able to produce the round 2 and 3 training samples (as illustrated by its high collapse). We will revise the explanation of LoRA collapse and include the round1 PALLA results directly in the table.
On soundness for filter learning: please refer to Appendix C. MLIR does not check merely against 3-17 valid samples; we also validate filters against a set of fixed golden solutions (line 740).
On the request for a GPT experiment: our paper is about the fact that one can use prefix filters to better understand the error patterns in LLMs, while our experiments illustrate that these filters can be used to enhance inference for models in which prefix filters are successfully learned. We do not claim that our approach is a universal approach that can increase the performance of any model and any domain. We do not believe that the fact we use GPT 5.4 / OSS-120B for the synthesizers requires us to run an experiment on cloud GPT / Claude-scale models, and argue that the experiments in the paper provide ample evidence that learned prefix filters are an effective tool for enhancing inference capability. moreover, Algorithm 3 requires the full oracle trie to be visible, which cloud-based models will note expose.
We thank the reviewers for pointing out the issue in Def. 3.1. and various typos.
We plan to withdraw the paper.