- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies so called example difficulty metrics, i.e. trying to estimate how difficult is an example for a model to be learned. The authors first propose a generalization of existing metrics and then later on present an experimental study that analyzes all metrics that can be derived from the generalization (not only the already published ones).
As mentioned by the authors in the introduction, difficulty metrics could be potentially used for outlier detection, data cleaning, active learning (related to curriculum learning as well), etc.
Critical strengths
- I appreciate the generalization aspects of the paper, this is something that should be considered more in our field. Proper formalizations lead to a more systematic exploration of variants and to see the similarities and relations between approaches. This is exactly what is paper is aiming for.
- Example difficulty metrics are indeed needed in several areas and a rigorous evaluation of them is appreciated. However, there is the question whether each task (outlier detection, data cleaning, curriculum learning, etc.) indeed requires a different metric. Figure 6 for example shows that there is no clear winner across tasks, but there are design choices which influence slightly generalize across a subset of the tasks.
Critical weaknesses
- One of the main outcomes of the paper is that the model probing method (i.e. the model itself) matters significantly. This is not surprising, since how difficult an example is to be learned by a model, obviously should depend on the model. One could argue that this dependency should be especially high when comparing rather simple methods (linear models) with more advanced ones. Some examples would be nearly impossible to learn for linear models for example (think of a nearly linearly separable problem with an outlier).
- The generalization given in equation (1) is indeed very general due to the fact that the authors tried to make it fit to a large set of previous methods. However, the generalization is not properly motivated and its reasonability for measuring example difficulty only becomes clear later when looking at the terms for specific methods. Table 1 is great, but can Eq. 1 be motivated from an example difficulty perspective at all? I guess so, but currently the only motivation is that it fits to previous approaches.
- One of the major weaknesses I see is that all experiments are based solely on synthetic corruptions of the datasets (L306) - aren't there real-world datasets with proper corruptions and annotations? Furthermore, details about the synthetic corruptions are missing (L334ff is not enough for reproducability).
- Is RQ2 (L409) really the right question? The answer to RQ4 is that "relative importance of the dataset and design axes varies significantly across applications" (L428). Doesn't this render RQ2 useless, since it simply depends on the task at hand?
Questions for authors
Minor comments/suggestions and questions:
- Minor: The abstract uses terminology that only later become clear from the definitions like model probing method, final-model and snapshot mean measurements, etc. Furthermore, there is no proper motivation in the abstract of why example difficulty metrics are useful. Just at least one sentence about it in the abstract would reasonable and describing the results of the paper in more broader terms in the abstract could lead to a broader readership.
- Which of the GBDT methods (Table 2) is really used? Did you test all of them but then grouped the results under the label GBDT? (same question holds for "deep learning")
- Minor: Cut-off sentence: "We RQ2." (L320), "we pairs of metrics" (L323)
- What is the value of used for the F1@k metric? L339 mentioned that k is the number of target points, but I do not get this statement.
- Minor: Types of figures and legends should be more consistent (compare Figure 4 with Figure 2 - different descriptions of the design axis and different presentation of the variance analysis)
- Minor: typos in L337 (maybe that is a good sign nowadays)
- Cut-off "RQ4" in L344 - looks like there was a plan to add a sentence about RQ4 here.
3 comments
Response to Reviewer PaLw (2/2)
The abstract lacks motivation and uses undefined terminology. (Q1)
We appreciate the feedback and have updated the abstract accordingly.
Which GBDT/DL methods did you use? (Q2)
We trained and evaluated all three GBDT and DL architectures and pooled the data for each architectural family. Architectures within each family are much more similar to one another, and we want conclusions that hold broadly for the popular architecture families. Please refer to Section 3.2 for this point.
Typos & minor errors. (Q3, 6, 7)
We’ve fixed the identified typos and minor errors. We’ve bolded “RQ4.” and “RQ5.”, etc., at L344 to highlight that they are signposts.
What k is used for F1@k? (Q4)
We used k = 35 out of 350 of the observed points by default, which is equal to the number of corrupted rows for each application’s test split. (150 points were used as validation to determine the sign/direction of the metric to use in the test split for certain nontrivial metrics. Val/test split was drawn 20 times randomly. This is now documented in updated Appendix A.2.)
Figures and legends should be more consistent. (Q5)
As requested, we’ve made the labels consistent. Please let us know whether we addressed the intended inconsistencies.
Thanks for the rebuttal and the revision
Thanks for the detailed rebuttal and the paper revisions. Motivation of the mathematical framework indeed improved significantly (although in my opinion designing equation (1) step-by-step with each step adding a different aspect of example difficulty would be even more readable - but this might be just personal taste).
Fig. 12 is indeed insightful - thanks!
Response to Reviewer PaLw (1/2)
Thank you for the detailed review and insightful questions. We’ve collected the requested data on linear models, and clarify points raised regarding Equation 1 and distinguishing RQ2 vs. RQ4.
The model probing method (i.e. the model itself) matters significantly. (W1)
To clarify, we define the model itself as the model architecture; the model probing method is a function that extracts a scalar from a model snapshot’s internals (logits, activations, gradients).
This dependency should be especially high when comparing rather simple methods (linear models). (W1)
As we understand, the general hypothesis presented here is that the relative importance of different design axes depends on the model architecture, especially model complexity. This is an insightful question, though our conclusion upon further analysis leans negative.
To test it, we trained a logistic regression model with gradient steps across the AMLB datasets. We then repeated RQ2 within each of the seven architectures (three GBDT, three DL, and logistic regression) without cross-architecture pooling.
In pooled data, we ordered the relative importance of each axis as follows: probe method > dataset aggregation > snapshot aggregation > model architecture > shared.
As you may see in Figure 12, Appendix A3.4, the same ordering, excluding architecture, holds across all seven architectures. Furthermore, we observe no significant difference in the values for logistic regression compared to others. Although different model architecture changes which data points are more or less difficult, RQ2’s conclusions still apply.
Can Eq. 1 be motivated from an example difficulty perspective at all? (W2)
We want to reiterate that defining a taxonomy and equation that fits the existing metrics, and articulating this design space, is one of our main goals. In addition, we’ve added prose to clarify the motivation for the equation and design axes. Please kindly refer to our revised exposition in Section 3.
We now state how the expectation term and the aggregation terms arise naturally out of necessity. We now also clarify that the five design axes naturally correspond to several alternate perspectives on defining example difficulty.
All experiments are based solely on synthetic corruptions. (W3)
Thank you for this suggestion. As outlined in the general comment, we’ve validated the noisy label detection tasks on realistic noisy label datasets. We conclude that metric rankings broadly recur. Total metric rankings broadly agree, with a moderate correlation (0.51), and the top-10 overlap is well above chance. Furthermore, five out of the top-10 metrics from the held-in data repeat in the top-10 in three out of four distinct validation cohorts, as shown in Table 5. Please kindly refer to Table 4 in Appendix A3.6 for the data.
Details about the synthetic corruption are missing. (W3)
We now provide additional details on the synthetic corruption in Appendix A.2 per the reviewer’s suggestion.
The answer to RQ4 is that "relative importance of the dataset and design axes varies significantly across applications" (L428). Doesn't this render RQ2 useless, since it simply depends on the task at hand? (W4)
RQ2 and RQ4 do not contradict. RQ2 measures how each design axis affects the ranking of data points, whereas RQ4 measures how each design axis or the dataset changes the metrics' data-quality application performance. If our goal is to design a metric that 1) broadly agrees with existing metrics, and simultaneously 2) performs well for a certain application, RQ2 informs the former and RQ4 informs the latter, and only one would be incomplete.
Furthermore, RQ4 shows the relative importance of different design axes for data quality application performance, after factoring out the dataset’s effect. We can interpret this from Figure 4 by ignoring the “dataset” bars and comparing the heights of the blue bars for the other design axes. This reading shows that the ordering of relative importance between design axes, as asserted in RQ2, is replicated nearly identically in RQ4. We now clarify this in prose.