acceptodds
Under review as a conference paper at ICLR 2027

Statistical Limits of Learning Rules from Exceptions

Abstract

Learning systems that encounter repeated deviations from an established regularity must decide whether they reflect transient noise, stable but idiosyncratic exceptions, or a reusable secondary rule. We formalize this decision through grouped observations, where each type may be observed through multiple noisy tokens, and study how a fixed observation budget should be divided between repetition and type diversity. For a finite separated rule class, we prove minimax token complexity . A rate-optimal design heavily repeats one type to establish persistence and uses singleton observations for structural learning, while under balanced acquisition the information-optimal repetition count is . Controlled simulations support both allocation laws. On CLINC150, persistent unstructured residuals are recoverable on observed intents but do not transfer to unseen intents, whereas structured residuals transfer and benefit from broader supporting-type coverage in all ten domains. On Omniglot, global character count is a poor proxy for relevant diversity, as directly increasing target-alphabet support at a fixed positive-token budget raises structured novel-type AUC from to (paired gain , 90% CI ), while unstructured controls remain at chance.

Primary area:
learning theory
Keywords:
learning theory, sample complexity, rule learning, memorization, generalization, repeated measurements, hypothesis testing, experimental design, exceptions, model selection

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_1G2xOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper focuses on modern learning systems across noise and local persistence scenarios. When a learning system repeatedly observes samples that violate established patterns, how should one determine which specific type of situation these deviations represent? This work observed that the distinction among noise, local persistence, and reusable structure becomes sharper. To that end, the authors proposed a statistical model based on binary residuals and latent variables, constructed it with symmetric noise, and also observed the same distinction in three views (simulation, CLINC150, and Omniglot).

Critical strengths
  1. The focused question is valuable, and a clear statistical framework has been established.

  2. The two theories yield clear and interpretable conclusions, and the appendix provides a relatively complete mathematical proof.

  3. The experimental results are not only in one dataset, but separate two goals.

Critical weaknesses
  1. The theoretical novelty of the proposed allocation laws relative to classical hypothesis testing and optimal experimental design. For example, the sparse-replication strategy in Theorem 1 is insightful, but the authors do not sufficiently articulate the fundamental differences between this conclusion and existing theories regarding grouped observations, optimal experimental design, or sequential hypothesis testing. (Can the authors also discuss Theorem 2?)

  2. In my view, the main contribution of this paper is that repetition establishes persistence, while supporting-type diversity enables generalization. I think the novelty is not sufficient.

  3. The experiment should be more reliable. Residual labels are manually constructed; N, E, and R may be found in one scenario in the real world; the authors need more baselines to compare.

  4. Several equations are not numbered, which makes them difficult to reference. The authors are encouraged to number these equations for better readability.

Questions for authors

See the weaknesses.

Reviewer_De8POpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This is a statistics theory paper. Its motivation is to understand whether classifier errors come from one of these three categories: random deviations (which are unpredictable), persistent exceptions (which the classifier could have memorized) and deviations with an underlying rule (which the classifier could have learnt). Concretely, they propose statistical models for each category, and study conditions under which they are distinguishable. In this setting, the learner needs to distinguish between the three models (N, E or R).

Their theoretical results are

  • A bound on the number of observations required to bound the worst case error, achieved with a construction that first uses repeated examples to distinguish N from E and R, and then new inputs to distinguish E from R.
  • Deriving the optimal repetition count per input, when each input must appear the same number of times
  • a sequential likelihood ratio test for how to sequentially distinguish between R and E

Experimentally, they test their theoretical results on synthetic data generated from their three statistical models and observe scaling similar to what is predicted by the theorems. Experiments on CLINC150 and Omniglot test intuitions behind the theorems (such as more distinct inputs help learn a shared rule), but don't directly test their theory.

Critical strengths

The studied topic is interesting, and it its important to know whether a model's errors are due to random, unpredictable noise, or structural information missed.

Critical weaknesses

Clarity and presentation. I have strong concerns about this. The paper is extremely hard to follow. The notation is only minimally explained, making it hard to follow what formulas mean. For example, section 1 does not introduce most components of the equation in lines 56-57. The surrounding text also doesn't help much in interpreting the contributions, and how they fit together. For example, Theorems 1 and 2 have the same underlying intuition (use repetition to isolate the random noise model, and new, diverse inputs to distinguish between E and R). The authors could have motivated both theorems with these intuitions, but instead most of this is only in the appendix. I believe in its current presentation, the paper should not be accepted.

Significance. While understand that this a theoretical paper, the authors never discuss how their proposed strategy of distinguishing between these error types could be used in practice. In addition, most conclusions don't come as surprising: Theorem 1 for example implies that distinguishing between the three models is more difficult if the rules are more similar or the observations are more noisier. It's unclear how to derive practical insights from this theory.

Questions for authors
  • In what practical setup does it make sense to observe every input the same number of times?
  • Could you experimentally test the proposed sequential test for distinguishing between E and R?
Reviewer_AppbOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper introduces a statistical framework to study when it is possible to distinguish observations generated by a generalizable rule from those generated from random associations or pure noise. In their setup they consider a finite collection of inputs (types), each associated with a noisy binary label (token). Within this framework, the paper studies how limited token budget should be allocated between repeated observation of the same type or sampling from different types (increasing diversity), to identify the label regime. They derive sample-complexity bounds and characterize the optimal repetition scale. They support their results by controlled simulations with CLINC150 and Omniglot datasets to investigate the effect of repetition and type diversity on generalization.

Critical strengths

The paper touches on an interesting question: the tension between learning generalizable shared rules and memorizing exception rules. They propose a concrete statistical hypothesis-testing problem to distinguish these two setups from observations and provide a theoretical analysis of the required sample complexity and budget allocation between data points.

Critical weaknesses
  1. Presentation: The paper is very jargon-heavy and difficult to follow. The terminology is not clearly established throughout the paper, and I needed multiple passes through the sections to make sense of the terminology and notation. As a result, I find it difficult to judge the paper in its current form.

  2. Motivation: I find a disconnect between the motivation and the actual setup. The paper begins by discussing that modern systems are trained on observations that may "violate established patterns/regularities," and the need for models to determine whether these observations represent "exception rules" or "reusable secondary rules." This is followed by a broader motivation that modern systems need to simultaneously learn generalizable rules and non-generalizable facts or exceptions, which is indeed a very interesting problem to study.

    However, the paper then considers a much simpler problem: given some observations, the model must determine whether the binary labels follow an exception rule or a generalizable rule. This is where I see two main issues:

    • I do not see what corresponds to the "established pattern" or the "secondary rule" in the proposed setup. This also relates to part of my previous concern about the terminology not being clearly grounded in the actual problem formulation.

    • As a reader, I would expect either a stronger justification for studying this simplified problem (given the current paper motivation) and/or a roadmap explaining how future work could build on the proposed setup and results to address the original, broader question.

  3. Theoretical results: The theoretical results also do not seem to follow a coherent story. Section 4 considers a class of functions G\mathcal{G}, where the model must distinguish which function g∈Gg \in \mathcal{G}, if any, was used to generate the data. Section 5 simplifies the setup to a single function gg, while Section 6 returns to considering the full function class. I wonder what distinguishes the exception setting (E) from the rule setting (R) when gg is known, as in Section 5. In particular, couldn't one also define the mapping between XiX_i and ZiZ_i as a function gg in the exception setting?

  4. Empirical results: I find the empirical section weak, as it is unclear to me what new insights these experiments provide. If the labels are randomized, we naturally cannot expect accuracy above random guessing. Conversely, when the labels are not randomized, we would expect performance to improve as the model observes more distinct samples. The empirical observations therefore seem rather expected, and I am not sure what my takeaway should be from these experiments.

Questions for authors

(see waeknesses)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.