acceptodds
Under review as a conference paper at ICLR 2027

Learning Error Patterns of Language Models

Abstract

When generating outputs for domains with specific validity constraints (e.g., a program should compile), LLMs often fail in a small number of focused ways: for example, by using Python function names when generating TypeScript. We observe that these error patterns can be represented using a small number of constraints that can be learned in practice. We propose prefix filters, which are per-domain-and-LLM symbolic functions, as objects to capture the error patterns, Palla as an algorithm to learn prefix filters efficiently in practice, and implement Palla. Prefix filters learned by Palla i) help us quantitatively analyze the error patterns of LLMs, and ii) can be used to constrain the outputs of a model via constrained sampling algorithms. For example, Palla learns 132 filters for Qwen2.5-1.5B on TypeScript generation, boosting compile rates by over 60% and allowing Qwen2.5-1.5B to achieve similar performance to Llama3.1-8B unconstrained.

Primary area:
generative models
Keywords:
Large Language Models, Error Patterns, Symbolic Error Patterns

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_qpJUOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces prefix filters: short Python predicates, specific to one model and one domain, that flag a partial LLM output as likely to fail a validity oracle. Experiments cover MLIR, SMILES molecules, PII-free dialogue summaries and TypeScript, with 7 open models and SQL in the appendix.

Critical strengths

The technique proposed (prefix filter) is easy to compute and interpretable. Additionally, the paper points out the interesting phenomenon of a clustered failure mode on lower-resource domains, which is important for further training and also interesting to know.

Critical weaknesses

The most natural baselines are missing. This includes compute-matched rejection sampling, prompting with learned error patterns, constrained decoding, etc. Additionally, in figure 4, it seems that LoRA is excelling on domains like typescript and summaries.

Additionally, there are several soundness concerns. Soundness is checked on a limited number of samples (e.g., only 3-17 valid samples for MLIR). There is also possible leakage concerns in the typescript validation. Furthermore, several numbers disagree across sections: Qwen-14B MLIR with grammar-constrained decoding (sec. 1 vs. table 2), token overhead (sec. 5.4 vs. appendix E), and share of errors from Qwen-1.5B's top MLIR filter (fig. 5).

Questions for authors
  • How many filter triggers would actually have produced an invalid sample under the oracle?
  • Does the external TypeScript validation set include reference solutions for the 50 test problems?
  • How does the method actually compares with the baselines pointed above? Or, why were some of the baselines not applicable?
Reviewer_qRobOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work observes that given an llm and a specific task, the model might repeatedly make mistakes that can be collected and classified into some error patterns. The authors propose "prefix filters" to learn such patterns, and use a PALLA algorithm to reject the erroneous outputs before they are generated by applying the learned prefix filters. Experiments evaluated 7 small-size llms on 4 tasks, where error patterns are collected and learned and synthesized by gpt-5.4-mini into python programs (the filters), and the results show that applying PALLA significantly improves the results on those tasks, while maintaining the output diversity (comparing to fine-tuning) and using a reasonable budget.

Critical strengths

S1: results on the selected 7 models show a significant improvement with PALLA compared to the unconstrained version; detailed comparison between PALLA and fine-tuning which shows that PALLA maintains output diversity.

Critical weaknesses

W1: My major concern is that the authors did not compare the proposed method to a direct evaluation of competent models (such as GPT-5.4 or more recent ones, or Claude equivalent). This significantly limits the practical value of this method. All evaluations are done with smaller LLMs, where improvements are observed with low cost. The authors are already using GPT-5.4-mini as the synthesizer, so it will be a lot more clear if the the proposed method is compared to a direct evaluation of GPT-5.4-mini where performance and costs are reported. Without those comparisons (under matched budgets), it is unclear whether the proposed method provides meaningful practical advantages over simple alternatives such as larger models (the authors also acknowledged that PALLA is ineffective on already competent models).

This weakness is consequential to my recommendation since the paper's primary motivation is to provide a cheap yet effective approach. Missing match-budget comparisons makes it hard to evaluate this central claim.

W2: The paper is poorly written, containing broken sentences, grammar mistakes and inconsistencies. The inconsistency in the definition and examples of prefix filters (which is one of the central contributions) makes it hard to follow: definition 3.1 states that a prefix filter triggers when it returns False. However, the filter example in after Def. 3.2 returns True when an invalid pattern is detected. A similar issue happens in section 5.2 examples, where upon matching the pattern (i.e. the filter triggers) returns True.

The poor writing quality and inconsistent specification/definition of the central part of the contribution significantly affects the validity, reproducibility and faithfulness of the proposed method.

Questions for authors

Q1: have the authors considered evaluating a competent model directly without PALLAR on the 4 tasks, and reporting the cost/performance?

Q2: can the authors clarify the definition of prefix filters and how they are implemented?

Reviewer_PZ3KOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces PALLA, which learns Python prefix filters from oracle-labeled model outputs and applies them through constrained adaptive rejection sampling. The filters are learned over three rounds and serve both to characterize recurring errors and to improve generation validity.

Critical strengths
  • The method connects error diagnosis with an intervention during generation. Learned filters make recurring failures inspectable and can directly prevent their recurrence. For example, three filters catch 51.7% of Qwen2.5-7B’s unconstrained MLIR errors (Figure 2B), providing concrete evidence that some frequent failures can be captured by compact rules.
  • The gains are substantial in several difficult settings and sometimes extend beyond validity. Qwen2.5-14B’s MLIR compile rate increases from 27.5% to 71.0%. For TypeScript/Qwen2.5-1.5B, compilation improves from 16.0% to 70.0%, while pass@10 increases from 13/50 to 29/50 tasks (Appendix E.4). The latter is useful evidence that filtering can improve task success.
  • The evaluation helps establish both applicability and limitations. The main experiments cover four domains and seven models. The grammar-plus-PALLA experiment demonstrates compatibility with existing constraints (Appendix G), while the SQL results show that higher validity does not necessarily improve correctness (Appendix E.5).
Critical weaknesses
  • The reported filter reliability needs an independent evaluation. Section 5.1 reports approximately 5% aggregate false rejection, but does not clearly identify an evaluation set separate from the valid examples used to construct and refine filters. Appendix C.1 also uses 390 external TypeScript solutions from a 390-task benchmark without specifying whether these include the 50 test tasks. These omissions leave uncertainty about rejection of previously unseen valid outputs and the separation between filter development and testing.
  • The DPO comparison does not sufficiently isolate the diversity advantage. DPO uses round-0 preference pairs, whereas final PALLA results follow three rounds of sampling (Appendix C.2). The abstract’s same-sample comparison therefore needs qualification. Figure 4 also does not specify whether distinctness is computed within valid outputs. Given the different validity rates, the current results leave unclear how much of the diversity difference reflects broader coverage of valid outputs.
  • The token-efficiency trade-off needs a direct comparison. For TypeScript/Qwen2.5-1.5B, the reported token counts and compile rates imply approximately 288 generated tokens per compiling output without filters, versus 667 with PALLA (Appendix E.4). The paper acknowledges the overhead, but does not report a comparison with ordinary compiler-based rejection sampling under the same token budget. Such a comparison would clarify whether prefix filtering offers an efficiency advantage when the compiler is available during inference. This calculation covers generated tokens only and excludes verification time.
Questions for authors
  • Do the 390 external TypeScript reference solutions include any test tasks? How was the approximately 5% aggregate false-rejection rate measured, and does it hold on independent valid outputs when all actual token prefixes are checked?

  • Does the diversity advantage persist when DPO and round-1 PALLA use the shared initial generations and diversity is measured among valid outputs? For TypeScript, how does PALLA compare with ordinary compiler-based rejection sampling in the number of compiling outputs obtained under the same generation-token budget?

1 comment

We thank the reviewers for their reviews. This document serves as a short response to all three reviewers.

  • We plan to conduct an extra experiments comparing against rejection sampling, and using filters in the prompt, as suggested.

  • On the comparison against LoRA DPO, one can use the results from round1 PALLA if one wishes to make a comparison of performance achieved using the same samples. We believe it is not a valid comparison to compare PALLA against LoRA DPO trained on examples from all three iterations, as LoRA DPO only would not have been able to produce the round 2 and 3 training samples (as illustrated by its high collapse). We will revise the explanation of LoRA collapse and include the round1 PALLA results directly in the table.

  • On soundness for filter learning: please refer to Appendix C. MLIR does not check merely against 3-17 valid samples; we also validate filters against a set of fixed golden solutions (line 740).

  • On the request for a GPT experiment: our paper is about the fact that one can use prefix filters to better understand the error patterns in LLMs, while our experiments illustrate that these filters can be used to enhance inference for models in which prefix filters are successfully learned. We do not claim that our approach is a universal approach that can increase the performance of any model and any domain. We do not believe that the fact we use GPT 5.4 / OSS-120B for the synthesizers requires us to run an experiment on cloud GPT / Claude-scale models, and argue that the experiments in the paper provide ample evidence that learned prefix filters are an effective tool for enhancing inference capability. moreover, Algorithm 3 requires the full oracle trie to be visible, which cloud-based models will note expose.

  • We thank the reviewers for pointing out the issue in Def. 3.1. and various typos.

We plan to withdraw the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.