acceptodds
Under review as a conference paper at ICLR 2027

Generalized Example Difficulty Metrics and Their Applications for Tabular Classification

Abstract

Quantifying example difficulty in ML is crucial for addressing prevalent data quality issues. Our goal is to design the most effective example difficulty metric for each data-centric application. To this end, we investigate questions about the design space itself: Which design axes matter most? Which values are the "best" for each axis, given all else equal? Existing surveys, which are catalogs of published metrics, cannot answer these questions rigorously. Furthermore, existing research under-represents the pervasive domain of tabular data, and none rigorously contrasts the architecture families of deep learning and gradient-boosted decision trees (GBDTs). We remedy these limitations by defining a broad taxonomy that generalizes the design knobs of existing metrics, yielding 96 metrics in a five-dimensional design space, spanning DL and GBDTs. We then study their behavior across 33 tabular classification tasks and their performance across five data-centric applications. We show the following: 1) Most published metrics correlate together despite a vast design space. 2) Model probing method matters more than other axes, such as whether to compute influence or direct train- or test-time measurements. 3) Final-model and snapshot-mean measurements are redundant, and train-time and test-time measurements are nearly redundant. 4) Training and measuring GBDTs is more effective for application tasks on tabular data than the DL equivalent on average; loss remains strong for many tasks. We conclude by identifying the best metric, from our experiments, for five data-centric application tasks.

Primary area:
interpretability and explainable AI
Keywords:
data-centric AI, tabular learning, tabular classification, example difficulty

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PaLwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies so called example difficulty metrics, i.e. trying to estimate how difficult is an example for a model to be learned. The authors first propose a generalization of existing metrics and then later on present an experimental study that analyzes all metrics that can be derived from the generalization (not only the already published ones).

As mentioned by the authors in the introduction, difficulty metrics could be potentially used for outlier detection, data cleaning, active learning (related to curriculum learning as well), etc.

Critical strengths
  • I appreciate the generalization aspects of the paper, this is something that should be considered more in our field. Proper formalizations lead to a more systematic exploration of variants and to see the similarities and relations between approaches. This is exactly what is paper is aiming for.
  • Example difficulty metrics are indeed needed in several areas and a rigorous evaluation of them is appreciated. However, there is the question whether each task (outlier detection, data cleaning, curriculum learning, etc.) indeed requires a different metric. Figure 6 for example shows that there is no clear winner across tasks, but there are design choices which influence slightly generalize across a subset of the tasks.
Critical weaknesses
  1. One of the main outcomes of the paper is that the model probing method (i.e. the model itself) matters significantly. This is not surprising, since how difficult an example is to be learned by a model, obviously should depend on the model. One could argue that this dependency should be especially high when comparing rather simple methods (linear models) with more advanced ones. Some examples would be nearly impossible to learn for linear models for example (think of a nearly linearly separable problem with an outlier).
  2. The generalization given in equation (1) is indeed very general due to the fact that the authors tried to make it fit to a large set of previous methods. However, the generalization is not properly motivated and its reasonability for measuring example difficulty only becomes clear later when looking at the terms for specific methods. Table 1 is great, but can Eq. 1 be motivated from an example difficulty perspective at all? I guess so, but currently the only motivation is that it fits to previous approaches.
  3. One of the major weaknesses I see is that all experiments are based solely on synthetic corruptions of the datasets (L306) - aren't there real-world datasets with proper corruptions and annotations? Furthermore, details about the synthetic corruptions are missing (L334ff is not enough for reproducability).
  4. Is RQ2 (L409) really the right question? The answer to RQ4 is that "relative importance of the dataset and design axes varies significantly across applications" (L428). Doesn't this render RQ2 useless, since it simply depends on the task at hand?
Questions for authors

Minor comments/suggestions and questions:

  1. Minor: The abstract uses terminology that only later become clear from the definitions like model probing method, final-model and snapshot mean measurements, etc. Furthermore, there is no proper motivation in the abstract of why example difficulty metrics are useful. Just at least one sentence about it in the abstract would reasonable and describing the results of the paper in more broader terms in the abstract could lead to a broader readership.
  2. Which of the GBDT methods (Table 2) is really used? Did you test all of them but then grouped the results under the label GBDT? (same question holds for "deep learning")
  3. Minor: Cut-off sentence: "We RQ2." (L320), "we pairs of metrics" (L323)
  4. What is the value of kk used for the F1@k metric? L339 mentioned that k is the number of target points, but I do not get this statement.
  5. Minor: Types of figures and legends should be more consistent (compare Figure 4 with Figure 2 - different descriptions of the design axis and different presentation of the variance analysis)
  6. Minor: typos in L337 (maybe that is a good sign nowadays)
  7. Cut-off "RQ4" in L344 - looks like there was a plan to add a sentence about RQ4 here.
3 comments
AuthorsOpenReview ↗

Response to Reviewer PaLw (1/2)

Thank you for the detailed review and insightful questions. We’ve collected the requested data on linear models, and clarify points raised regarding Equation 1 and distinguishing RQ2 vs. RQ4.

The model probing method (i.e. the model itself) matters significantly. (W1)

To clarify, we define the model itself as the model architecture; the model probing method is a function that extracts a scalar from a model snapshot’s internals (logits, activations, gradients).

This dependency should be especially high when comparing rather simple methods (linear models). (W1)

As we understand, the general hypothesis presented here is that the relative importance of different design axes depends on the model architecture, especially model complexity. This is an insightful question, though our conclusion upon further analysis leans negative.

To test it, we trained a logistic regression model with gradient steps across the AMLB datasets. We then repeated RQ2 within each of the seven architectures (three GBDT, three DL, and logistic regression) without cross-architecture pooling.

In pooled data, we ordered the relative importance of each axis as follows: probe method > dataset aggregation > snapshot aggregation > model architecture > shared.

As you may see in Figure 12, Appendix A3.4, the same ordering, excluding architecture, holds across all seven architectures. Furthermore, we observe no significant difference in the values for logistic regression compared to others. Although different model architecture changes which data points are more or less difficult, RQ2’s conclusions still apply.

Can Eq. 1 be motivated from an example difficulty perspective at all? (W2)​

We want to reiterate that defining a taxonomy and equation that fits the existing metrics, and articulating this design space, is one of our main goals. In addition, we’ve added prose to clarify the motivation for the equation and design axes. Please kindly refer to our revised exposition in Section 3.

We now state how the expectation term and the aggregation terms arise naturally out of necessity. We now also clarify that the five design axes naturally correspond to several alternate perspectives on defining example difficulty.

  1. Example difficulty from the perspective of model inductive bias: model architecture axis (Kwok et al., 2024; Ethayarajh et al., 2022).
  2. Bias-variance decomposition: dataset aggregation axis (Zhu et al., 2024).
  3. Training dynamics: snapshot aggregation axis (Toneva et al., 2019; Baldock et al., 2021).
  4. Leave-out attribution & influence: probe set & dataset aggregation axis.

All experiments are based solely on synthetic corruptions. (W3)

Thank you for this suggestion. As outlined in the general comment, we’ve validated the noisy label detection tasks on realistic noisy label datasets. We conclude that metric rankings broadly recur. Total metric rankings broadly agree, with a moderate correlation (0.51), and the top-10 overlap is well above chance. Furthermore, five out of the top-10 metrics from the held-in data repeat in the top-10 in three out of four distinct validation cohorts, as shown in Table 5. Please kindly refer to Table 4 in Appendix A3.6 for the data.

Details about the synthetic corruption are missing. (W3)

We now provide additional details on the synthetic corruption in Appendix A.2 per the reviewer’s suggestion.

The answer to RQ4 is that "relative importance of the dataset and design axes varies significantly across applications" (L428). Doesn't this render RQ2 useless, since it simply depends on the task at hand? (W4)

RQ2 and RQ4 do not contradict. RQ2 measures how each design axis affects the ranking of data points, whereas RQ4 measures how each design axis or the dataset changes the metrics' data-quality application performance. If our goal is to design a metric that 1) broadly agrees with existing metrics, and simultaneously 2) performs well for a certain application, RQ2 informs the former and RQ4 informs the latter, and only one would be incomplete.

Furthermore, RQ4 shows the relative importance of different design axes for data quality application performance, after factoring out the dataset’s effect. We can interpret this from Figure 4 by ignoring the “dataset” bars and comparing the heights of the blue bars for the other design axes. This reading shows that the ordering of relative importance between design axes, as asserted in RQ2, is replicated nearly identically in RQ4. We now clarify this in prose.

AuthorsOpenReview ↗

Response to Reviewer PaLw (2/2)

The abstract lacks motivation and uses undefined terminology. (Q1)

We appreciate the feedback and have updated the abstract accordingly.

Which GBDT/DL methods did you use? (Q2)

We trained and evaluated all three GBDT and DL architectures and pooled the data for each architectural family. Architectures within each family are much more similar to one another, and we want conclusions that hold broadly for the popular architecture families. Please refer to Section 3.2 for this point.

Typos & minor errors. (Q3, 6, 7)

We’ve fixed the identified typos and minor errors. We’ve bolded “RQ4.” and “RQ5.”, etc., at L344 to highlight that they are signposts.

What k is used for F1@k? (Q4)

We used k = 35 out of 350 of the observed points by default, which is equal to the number of corrupted rows for each application’s test split. (150 points were used as validation to determine the sign/direction of the metric to use in the test split for certain nontrivial metrics. Val/test split was drawn 20 times randomly. This is now documented in updated Appendix A.2.)

Figures and legends should be more consistent. (Q5)

As requested, we’ve made the labels consistent. Please let us know whether we addressed the intended inconsistencies.

Reviewer_PaLwOpenReview ↗

Thanks for the rebuttal and the revision

Thanks for the detailed rebuttal and the paper revisions. Motivation of the mathematical framework indeed improved significantly (although in my opinion designing equation (1) step-by-step with each step adding a different aspect of example difficulty would be even more readable - but this might be just personal taste).

Fig. 12 is indeed insightful - thanks!

Reviewer_FaNaOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the design of example-difficulty metrics for tabular classification. It introduces a five-axis taxonomy covering model architecture, probe set, probing function, aggregation across training-set conditions, and aggregation across training snapshots. By exhaustively enumerating legal combinations within selected domains, the authors construct 96 metric configurations, enabling controlled comparisons of individual design choices rather than comparisons only among existing, fully specified metrics.

Critical strengths
  1. The taxonomy-first design enables informative, controlled comparisons.
  2. The application experiments yield useful, differentiated conclusions.
Critical weaknesses

W1. Validation on realistic data-quality problems remains limited. W2. Practical cost and the external validity of pruning remain unresolved.

Questions for authors

Although I have some knowledge in the fields related to TFM, I have limited understanding of the data quality. My initial score is based on the overall completeness and contribution of the entire article. I will revise the score later according to the suggestions from other reviewers and the author's response.

1 comment
AuthorsOpenReview ↗

Response to Reviewer FaNa

Thank you for your positive assessment of our paper. We’re confident that the reviewers' feedback has only strengthened our paper.

Validation on realistic data-quality problems remains limited. (W1)

As outlined in the general comment above, we have now extensively validated the metric rankings on data-quality applications using 12 new external datasets. Metric ranking results for noisy labels, noisy features, hard and easy pruning are consistent in external validation.

Practical cost and the external validity of pruning remain unresolved. (W2)

Please also refer to the general comment and our comment on Weakness 1, where we have addressed the external validity concern.

We’d like to clarify that we use the data-centric applications primarily to validate the construct of the taxonomy, rather than prescribing a single workflow. The taxonomy itself does not determine which of the 96 competing definitions of example difficulty are most or least aligned with the abstract notion of difficulty. The data quality applications provide an external yardstick. This is a standard approach used in Seedat et al. (2024) and George et al. (2024), as clarified in the text.

Whether example difficulty measurement is practically cost-effective is contextual on the dataset size and compute vs. accuracy loss trade-offs. Our data nonetheless provides useful insights for making such decisions. From a cost perspective, Figure 7 shows which metrics are most reliable and therefore require fewer samples to approximate their true difficulty values. From an accuracy perspective, Appendix A3.6 lists the top candidates. While we measure the difficulty metrics with hundreds of replicates to measure them precisely for scientific analyses, in practice train- or test-time metrics incur no more overhead than extracting the fc layer activations or gradients, which is standard in existing example difficulty metrics.

Reviewer_RjbvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper organizes example difficulty metrics into a framework covering model architecture, probing signal, probed data, and aggregation across training runs and snapshots. It instantiates 96 metrics using neural networks and gradient-boosted trees and evaluates them on 33 tabular classification tasks. The analyses study relationships between metrics, influential design choices, and redundancy. Five applications cover label noise, feature noise, conflicting duplicates, missingness, and data pruning. The main findings emphasize the probing signal and model family, identify several apparently redundant choices, and recommend particular metrics for different applications.

Critical strengths
  1. The framework turns a fragmented literature into a controlled design space. Varying individual axes while holding the others fixed is more informative than comparing a small collection of named metrics, and helps explain why apparently similar difficulty measures behave differently.
  2. The measurement effort is substantial. Six model architectures, 300 independent splits, multiple training snapshots, and five applications provide a broad empirical basis. Including boosted trees is particularly valuable for tabular data, where neural-network signals alone would give an incomplete picture.
Critical weaknesses
  1. The central redundancy analysis pools examples across datasets before computing weighted Spearman correlations (§4.1). Equal dataset weights and reliability correction do not remove between-dataset differences in metric level or scale. Without documented within-dataset ranking or per-dataset correlation results, a high pooled correlation does not establish that two metrics rank examples similarly within a task. This directly affects the recommendations to discard apparently redundant design choices.
  2. The application recommendations have limited external validation. Difficulty is measured on at most 500 class-balanced points per task, and pruning is evaluated on this small pool. Only three tasks are held out; conflicting duplicates and missingness have no held-out evaluation. These results are useful exploratory evidence, but provide a narrow basis for recommending metrics for full, imbalanced tabular datasets.
Questions for authors
  1. Were metric values converted to within-dataset ranks before pooling? Please provide per-dataset correlations, or their distribution, for the main redundancy claims, particularly the final-snapshot versus mean-snapshot comparison.
  2. Which three datasets were held out, and how consistent are the application-specific metric rankings across those tasks? Are any conclusions supported by pruning results beyond the 500-point class-balanced pool?
1 comment
AuthorsOpenReview ↗

Response to Reviewer Rjbv

Thank you for the considerate suggestions and questions. We’ve conducted additional experiments to validate our results, as outlined in the general comment.

Were metric values converted to within-dataset ranks before pooling? (Q1)

Yes.

Do the main redundancy claims repeat per-dataset before pooling? (W1, Q1)

Yes. As a baseline, pooled and non-pooled inter-metric correlations agree at Spearman correlation of 0.69; please refer to new Figure 10 in Appendix A3.3.

Furthermore, RQ1 and RQ3’s redundancy claims repeat without pooling, as we now show in Figure 9 in Appendix A3.2 and Figure 13 in Appendix A3.5. The former shows that 17 published example difficulty metrics are more redundant (lower effective dimensionality) than 17 randomly sampled example difficulty metrics in our taxonomy for every single task. The latter demonstrates that axis-value redundancy claims repeat: mean vs. final snapshot are redundant (≥ 0.9) and train vs. test are nearly redundant (≳ 0.9) for most tasks.

Which three datasets were held out? (Q2)

The three held-out datasets from AMLB are bank-marketing, MiniBooNE, and okcupid-stem, as we now clarify in Table 3.

How consistent are the application-specific metric rankings across held-out tasks? (W2, Q2)

We emphasize that the held-in cohort includes a substantial collection of tabular learning tasks in an effort to collect broad data while simultaneously agreeing that additional external validation is valuable. We thus report the consistency of metric rankings between the held-in cohort and four types of validation tasks: AMLB held-out (3), AMLB/external full datasets (4/4), and real errors (3 per task), which is shown in Table 4 in Appendix A3.6.

The total rankings and top-10 metrics are consistent across noisy labels, noisy features, and data pruning, with moderate to high rank correlation (0.51-0.90) and top metrics recurring in the validation datasets well above chance. We now make this clear in the interpretation of RQ6.

We observe that conflicting duplicates and missing values do not repeat--these tasks have the fewest datasets in the held-in cohort. We’ve softened our claims regarding these applications.

Are any conclusions supported by pruning results beyond the 500-point class-balanced pool? (Q2)

Yes. Metric rankings when pruning based on the full, imbalanced datasets are broadly consistent with the held-in rankings. The total rankings show moderate to high rank correlation (0.52-0.73), and top-10 overlap is far above chance.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank the reviewers for their review and constructive criticism. The reviewers recognized the paper’s intended focus on a general and formal taxonomy coupled with extensive evaluations, and provided thoughtful questions and suggestions for strengthening the experiments. We’ll summarize the major improvements here in the main comment.

On the writing aspect, the key improvements are: clarifying the motivation in the abstract, elaborating on the motivations for Equation 1 in Section 3, and clarifying the purpose of data-centric applications as construct validity. We’ve also made our interpretations of the results more precise with new data.

We’ve strengthened our experiments and analysis through additional figures and extensive validation on 19 datasets. These additional analyses largely validate our findings by repeating our RQ1 & 3 redundancy claims across tasks, RQ2’s axis importance analysis across architectures, and the metric rankings in RQ6 in various validation tasks for noisy labels, features, hard and easy pruning.

(RQ6) Additional validation of metric recommendations for applications. (Rjbv W1, Q2; FaNa W1, W2; PaLw W3) As requested by the reviewers, we’ve repeated the data-centric application experiments on full, external, and realistic dirty and imbalanced tables. These are summarized in Table 4 in Appendix A3.6. Our existing results replicate with strong correlation and well-above-chance top-10 overlap for noisy labels, noisy features, hard data pruning, and easy data pruning. Only conflicting duplicate and missing values, which has the fewest datasets in AMLB, were not consistent and required softening our claims.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.