- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper focuses on how to interpret disagreement between annotators in ML datasets. The main point is that the same human labels can be evaluated vs. a single correct answer (so disagreement may be error) or vs a distribution of responses (so variation is part of the target itself). Authors propose to make explicit the evaluation target and the evidence about the data generation process, in order to decide which conclusions, model comparisons and label actions are really justified. Exps. on synthetic profiles and on some real datasets show that changing the target can change model rankings and claim support, and that majority vote training increase divergence from the response distribution.
Critical strengths
The problem is relevant and real, noisy-label learning and distributional learning make opposite assumptions on what disagreement means and often this is left implicit. The idea of a framework that force to declare the target and connect it with the available evidence is interesting and potentially useful.
Critical weaknesses
The main issue for me is the presentation: the paper is quite hard to follow. Many method-specific terms (measurement contract, the reliability graph, the compiler etc.) are introduced in abstract and intro with very short, and abstract definitions, and what they mean in practice, become clear only later in sec. 3. The intro is very dense, the 8/10 annotators example at the beginning helps but then some expressions (e.g. "controlled profiles, fixed predictions, and matched training") are used before being explained. Related work is too short, for who don't already know this literature it is difficult to understand what existing methods do and how the paper is positioned w.r.t. them. Also the main text rely too much on the appendix, it shows mostly rankings and counts while the detailed numbers behind them (per-system values, the 12 conclusion contrasts, the full claim requirements, per-seed results) are only in appendix. The experiments section is a bit fragmented, the overview in 4.1 is very short and most datasets (FERPlus, MELD, GoEmotions, OK-VQA etc.) are described in one or two sentences. Finally there is no dedicated limitations / future work section, some limits are mentioned in the discussion but the conclusion is very short.
On the method side, is not clear why exactly six templates and which principle is behind them (there is a sort of correspondence with the evidence dimensions of the graph but is not discussed, and it is not said if they cover all relevant cases). Moreover, on real data the records depend on the analyst, who must understand correctly the dataset documentation and encode it well. The "unknown" state helps when docs are incomplete, but if the analyst misinterpret something with confidence the compiler only checks that the record is consistent with the contract, not that it is correct. This is mentioned in one sentence in the discussion but I think this is a central point for the practical reliability of the framework and needs more discussion. I would also like a concrete real-data example in the main text showing how an analyst go from the documentation to the evidence states, e.g. what is concretely a "known distinguishing corruption channel" in a real dataset. The toy 80/20 example and the ChaosNLI record help but they are not sufficient. Also the thresholds of the full requirements and the training setup are only in appendix, at least a short summary in the main text would help.
Questions for authors
-
Why six templates? Is there a principle behind them and do you think they cover all relevant cases? Can you add a concrete real-data example in the main text showing how their conditions are verified?
-
What happens if the analyst misinterpret the documentation when compiling the records (not missing info, that goes to "unknown", but a wrong interpretation)? Is there a way to detect or limit this kind of errors, e.g. measuring inter-analyst agreement? Would you consider to add a dedicated limitations and future work section?
2 comments
Thank you for your answer. I will keep my score.
Thank you for the questions. Presentation changes and revised-paper locations are described in the separate Revision Summary.
1. Why Six Rules, And How Actual Records Satisfy Them
The six rules cover common evidence structures and state sufficient conditions for their specified conclusions. They are not exhaustive, and several can apply together. Our reply to jWu4, point 1, explains their justification.
ChaosNLI item
mnli-100637ehas an old contradiction label and 100 new responses: 59 entailment, 18 neutral and 23 contradiction. Predicting the most frequent response in this sample uses entailment as the target. Predicting response frequencies uses (0.59,0.18,0.23).The item identifier links the responses to the same input, allowing this empirical distribution to be constructed. SNLI/MNLI identifiers locate the source dataset, so they support reporting by dataset, not by individual annotator. The resulting decision is to preserve the response frequencies, score predictions by KL and report by source. Removing the source links leaves the pooled score available but removes source-conditioned reporting. The relations therefore determine which comparisons the researcher can make from the same responses.
This record supplies neither an independent correctness reference nor a known corruption channel, so the counts alone do not justify correcting minority responses. The known-channel example is constructed, rather than a channel verified in a real dataset. With a known flip probability of 0.1, a recorded positive rate of 0.4 implies a pre-corruption rate of (0.4−0.1)/0.8=0.375. The distribution is identified, while individual flips remain unresolved.
HANS provides a concrete reference example of a different kind. In item
ex0, reversing the subject and object between premise and hypothesis yields the non-entailment reference. Its construction template and subcase identify the controlled relation and diagnostic group. This supports checking predictions against that logical reference.2. What Happens When The Evidence Is Misinterpreted
We test confident misinterpretation by changing each of 21 encoded conditions while keeping the observations and independent reference answers fixed. For example, marking an unavailable reference as present can make a correction rule appear satisfied. Both implementations receive identical changes and are scored against the unchanged reference.
Among non-conflicting cases, 183 omissions produce no unsupported decisions but remove supported options. The 1,245 inputs with added assertions give:
Actions are counted once per family and input. Separate rule evaluation retains more supported options, but a false premise can also activate more unsupported rules. The matcher's abstention when several rules match explains both its lower error count here and the valid options it discards in the clean comparison.
Four targeted misreadings produce unsupported decisions in 10/12 inputs for both implementations. They treat a vote as an independent reference, a source name as complete item links, partial group links as complete, or an association as an intervention. In the vote example, correction becomes eligible because of the asserted reference. This identifies what to verify: whether the source establishes correctness, rather than merely recording agreement.
These results separate verification of the evidence from checks on its use. A false reference assertion calls for inspecting the source, while a conclusion attached to the wrong target is caught by a binding check. The compiler and checked checklist reject all 191 designed record violations (reply to jWu4, point 4). The framework thus makes the premise behind a decision inspectable and checks that the evaluation uses that decision consistently. Direct analyst-agreement measurement remains future work.