acceptodds
Under review as a conference paper at ICLR 2027

What Do Labels Let Us Conclude?

Abstract

The same human responses can be evaluated against a single answer or a response distribution. Choosing between these targets can change what counts as a good prediction, and making sense of label disagreement requires evidence about how the data was labelled. We propose a framework for making these evaluation judgements explicit. A measurement contract states the target, and a Mediated-Label Reliability Graph records the production evidence. Analysts use shared rules to determine the supported conclusions. A compiler then combines this record with the requested performance claim to construct an evaluation protocol. Controlled evidence profiles show how removing target or process information changes rule matches and eligible actions while responses stay fixed. In real-data comparisons, changing the target and score changes two model rankings. Applying the complete requirements of twelve performance claims withdraws support for six and narrows one. Matched image and language experiments show a further consequence of target choice: majority reduction increases divergence from the response-distribution target. Together, these studies show how explicit targets and evidence guide model selection, the interpretation of performance gains and label treatment.

Primary area:
datasets and benchmarks
Keywords:
supervised learning, evaluation methodology, label uncertainty, measurement, distributional targets

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tEcZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper focuses on how to interpret disagreement between annotators in ML datasets. The main point is that the same human labels can be evaluated vs. a single correct answer (so disagreement may be error) or vs a distribution of responses (so variation is part of the target itself). Authors propose to make explicit the evaluation target and the evidence about the data generation process, in order to decide which conclusions, model comparisons and label actions are really justified. Exps. on synthetic profiles and on some real datasets show that changing the target can change model rankings and claim support, and that majority vote training increase divergence from the response distribution.

Critical strengths

The problem is relevant and real, noisy-label learning and distributional learning make opposite assumptions on what disagreement means and often this is left implicit. The idea of a framework that force to declare the target and connect it with the available evidence is interesting and potentially useful.

Critical weaknesses

The main issue for me is the presentation: the paper is quite hard to follow. Many method-specific terms (measurement contract, the reliability graph, the compiler etc.) are introduced in abstract and intro with very short, and abstract definitions, and what they mean in practice, become clear only later in sec. 3. The intro is very dense, the 8/10 annotators example at the beginning helps but then some expressions (e.g. "controlled profiles, fixed predictions, and matched training") are used before being explained. Related work is too short, for who don't already know this literature it is difficult to understand what existing methods do and how the paper is positioned w.r.t. them. Also the main text rely too much on the appendix, it shows mostly rankings and counts while the detailed numbers behind them (per-system values, the 12 conclusion contrasts, the full claim requirements, per-seed results) are only in appendix. The experiments section is a bit fragmented, the overview in 4.1 is very short and most datasets (FERPlus, MELD, GoEmotions, OK-VQA etc.) are described in one or two sentences. Finally there is no dedicated limitations / future work section, some limits are mentioned in the discussion but the conclusion is very short.

On the method side, is not clear why exactly six templates and which principle is behind them (there is a sort of correspondence with the evidence dimensions of the graph but is not discussed, and it is not said if they cover all relevant cases). Moreover, on real data the records depend on the analyst, who must understand correctly the dataset documentation and encode it well. The "unknown" state helps when docs are incomplete, but if the analyst misinterpret something with confidence the compiler only checks that the record is consistent with the contract, not that it is correct. This is mentioned in one sentence in the discussion but I think this is a central point for the practical reliability of the framework and needs more discussion. I would also like a concrete real-data example in the main text showing how an analyst go from the documentation to the evidence states, e.g. what is concretely a "known distinguishing corruption channel" in a real dataset. The toy 80/20 example and the ChaosNLI record help but they are not sufficient. Also the thresholds of the full requirements and the training setup are only in appendix, at least a short summary in the main text would help.

Questions for authors
  1. Why six templates? Is there a principle behind them and do you think they cover all relevant cases? Can you add a concrete real-data example in the main text showing how their conditions are verified?

  2. What happens if the analyst misinterpret the documentation when compiling the records (not missing info, that goes to "unknown", but a wrong interpretation)? Is there a way to detect or limit this kind of errors, e.g. measuring inter-analyst agreement? Would you consider to add a dedicated limitations and future work section?

2 comments

Thank you for the questions. Presentation changes and revised-paper locations are described in the separate Revision Summary.

1. Why Six Rules, And How Actual Records Satisfy Them

The six rules cover common evidence structures and state sufficient conditions for their specified conclusions. They are not exhaustive, and several can apply together. Our reply to jWu4, point 1, explains their justification.

ChaosNLI item mnli-100637e has an old contradiction label and 100 new responses: 59 entailment, 18 neutral and 23 contradiction. Predicting the most frequent response in this sample uses entailment as the target. Predicting response frequencies uses (0.59,0.18,0.23)(0.59,0.18,0.23).

The item identifier links the responses to the same input, allowing this empirical distribution to be constructed. SNLI/MNLI identifiers locate the source dataset, so they support reporting by dataset, not by individual annotator. The resulting decision is to preserve the response frequencies, score predictions by KL and report by source. Removing the source links leaves the pooled score available but removes source-conditioned reporting. The relations therefore determine which comparisons the researcher can make from the same responses.

This record supplies neither an independent correctness reference nor a known corruption channel, so the counts alone do not justify correcting minority responses. The known-channel example is constructed, rather than a channel verified in a real dataset. With a known flip probability of 0.1, a recorded positive rate of 0.4 implies a pre-corruption rate of (0.4−0.1)/0.8=0.375(0.4-0.1)/0.8=0.375. The distribution is identified, while individual flips remain unresolved.

HANS provides a concrete reference example of a different kind. In item ex0, reversing the subject and object between premise and hypothesis yields the non-entailment reference. Its construction template and subcase identify the controlled relation and diagnostic group. This supports checking predictions against that logical reference.

2. What Happens When The Evidence Is Misinterpreted

We test confident misinterpretation by changing each of 21 encoded conditions while keeping the observations and independent reference answers fixed. For example, marking an unavailable reference as present can make a correction rule appear satisfied. Both implementations receive identical changes and are scored against the unchanged reference.

Among non-conflicting cases, 183 omissions produce no unsupported decisions but remove supported options. The 1,245 inputs with added assertions give:

ImplementationUnsupported DecisionsSupported Actions Retained
Original Matcher114/1,245373/1,085
Complete Checklist214/1,245957/1,085

Actions are counted once per family and input. Separate rule evaluation retains more supported options, but a false premise can also activate more unsupported rules. The matcher's abstention when several rules match explains both its lower error count here and the valid options it discards in the clean comparison.

Four targeted misreadings produce unsupported decisions in 10/12 inputs for both implementations. They treat a vote as an independent reference, a source name as complete item links, partial group links as complete, or an association as an intervention. In the vote example, correction becomes eligible because of the asserted reference. This identifies what to verify: whether the source establishes correctness, rather than merely recording agreement.

These results separate verification of the evidence from checks on its use. A false reference assertion calls for inspecting the source, while a conclusion attached to the wrong target is caught by a binding check. The compiler and checked checklist reject all 191 designed record violations (reply to jWu4, point 4). The framework thus makes the premise behind a decision inspectable and checks that the evaluation uses that decision consistently. Direct analyst-agreement measurement remains future work.

Thank you for your answer. I will keep my score.

Reviewer_YSGuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a framework linking prediction targets, evidence about label production, and permissible evaluation claims. Its measurement contract defines what to predict; its evidence graph and analyst judgments guide evaluation and label treatment. Experiments show changes in model rankings, narrower support for performance claims, and worse response distribution prediction after majority label training.

Critical strengths
  • The framework distinguishes the prediction target, explanations of disagreement, and eligible actions. This seems to be helpful in preventing unjustified correction of minority responses and clarifies why unresolved mechanisms can still permit meaningful evaluation.
  • The evidence that majority labels hurt distributional prediction convincing seems to be convincing. The comparisons use the same initialization, data order, and optimization settings, with separate data for checkpoint selection and final evaluation. Majority-label training increases KL div. by 0.254 on cifar10 and 0.095 on chaos nli, with confidence intervals above zero under both bootstrap analyses.
Critical weaknesses
  • The framework's independent validation seems insufficient. The controlled study checks rules against their own expected outputs, and changing response order across seeds does little to test those decisions.
    • Representing 28 contracts shows broad coverage, but does not establish accurate or useful judgments.
    • It would be also meaningful to include agreement between analysts and comparison with a simpler checklist.
  • I'm not fully convinced that the claim-withdrawal results demonstrate the framework's benefit. The paper lists numerical tolerances and joint requirements, but does not clarify when they were chosen or adequately justify their relevance to the intended applications. Adding requirements naturally makes claims harder to support. I think the 6 withdrawals would be more convincing with independent justification for these requirements or an analysis of how the results change under different thresholds.
Questions for authors
  • Were the 12 claims, joint requirements, and tolerances fixed before examining their results? How sensitive are the withdrawal counts to reasonable alternative thresholds?
  • What evidence shows that independent analysts reach consistent judgments, and that the graph and compiler improve decisions over a structured checklist?
  • Can you report paired uncertainty estimates for the FERPlus ranking differences, particularly the 0.001-nat gap in the three-system comparison?
1 comment

Thank you for the questions.

1. Requirement Sources, Timing And Sensitivity

The twelve-claim analysis is retrospective. Numerical allowances come from existing comparison records:

  • FERPlus's Calibration-Plus-Edge and Residual-Edge specifications require mean KL/ECE not to increase and accuracy to improve on average and at two or more seeds.
  • The Calibrated-Utility And Geometry specifications allow GoEmotions micro-F1 and macro average precision to fall by 0.010 and 0.005, and ECE to rise by 0.002. ChaosNLI permits accuracy loss and ECE increase of 0.010, and OK-VQA permits accuracy loss of 0.010.
  • COCO-Search18's method specification permits normalised JS to rise and attention similarity to fall by at most 0.005. Its repeated-run record applies these limits at each seed.
  • CASE's Global-Fusion Report caps subject-probe accuracy at chance + 0.010 for weights, and the equal-weight reference + 0.010 for predictions and joint probes. MELD applies mean/seed criteria to existing outputs.

Available commits place five-system FERPlus, GoEmotions and OK-VQA conditions before their result records. Three-system FERPlus conditions follow the five-system results but precede their own. COCO's repeated-run conditions follow one run and precede the multi-seed summary. ChaosNLI conditions and results share a commit. CASE has no established earlier threshold commit.

These allowances attach to properties in the claim. Distributional fit can be assessed by its score alone, whereas a claim that also retains accuracy must meet the accuracy limit. We test how sensitive the resulting decisions are to these allowances.

Across 414 original or alternative settings, scaling existing non-zero tolerances by 0, 0.5, 1 or 2 leaves all twelve decisions unchanged. Relaxing zero boundaries or seed consistency changes three of the six withdrawals:

  • ChaosNLI geometry passes with a 0.001 local-decrease allowance, against an observed change of -0.000720.
  • GoEmotions geometry passes at 0.005, against -0.002700.
  • COCO-Search18 utility passes at 0.001 or with two of three seeds, because one seed's high-entropy divergence improvement is -0.000255.

Both FERPlus utility claims and OK-VQA geometry remain unsupported across these alternatives. MELD supports an average-benefit claim, although individual modality effects change sign across seeds. The mean criterion concerns the observed average, not statistical significance.

For ChaosNLI, the global gain of 0.024 supports a global claim. A joint claim must state the permitted local decrease. CASE retains its full task-and-probe claim.

2. Independent Validation And The Checklist Comparison

To assess decisions independently of the rules, we derive reference answers from finite probability laws. Each of 24 cases has three evidence views. We find the processes consistent with each view and calculate which actions are valid under all of them. The comparison reuses the 12 development and 12 held-out cases, with identical evidence for both implementations.

The original matcher acts only when one rule matches completely. A complete checklist applies our six rules separately. Across 68 non-conflicting views, it retains 59/66 supported action-family instances, versus 25/66 for the matcher, with zero unsupported decisions for both. Unnecessary abstention falls from 20/45 to 5/45 views with supported actions. A single-rule checklist reproduces all 1,604 matcher outputs, isolating the selection policy.

In a mixed-mechanism case, a paired reference supports both repairing recording corruption and preserving valid response variation. The matcher abstains because two rules match. The criterion allows researchers to compare both treatments under one target, retaining the evidence required for each.

For additional mechanisms, including selective release, retention rises from 9/14 to 12/14. Four conflicting views have an empty compatible set, but the Boolean encoding omits the conflicting constraints and both implementations return ordinary outputs. Raw-evidence consistency therefore needs checking before rule execution.

The 28 contracts establish coverage. These tests assess decision correctness and retained options. Our reply to jWu4, points 3-4, covers analyst agreement and the compiler/checklist comparison.

3. Paired FERPlus Uncertainty

To assess model selection, we calculate all 13 paired comparisons over the same 3,508 items and three seeds, using 10,000 shared-item/seed resamples. Pointwise 95% intervals give:

  • Residual Edge minus Global Control: KL -0.000836 [-0.026831,0.030031] nat. With trained seeds fixed, the interval is [-0.009942,0.007905]. This pair remains unresolved.
  • Edge minus Within-Class Control: KL -0.0216 [-0.0363,-0.0073] nat, with accuracy difference -0.3231 [-1.9574,1.3683] percentage points. This supports choosing Edge for distributional prediction, while its accuracy difference remains unresolved.
Reviewer_jWu4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper looks at how the choice of evaluation target and the evidence available about annotation processes affect the conclusions we draw from model performance. The authors argue that label disagreement should not automatically be treated as noise, since the same responses may represent errors against a single target or genuine variation in human judgements. They formalise this through a measurement contract and a Mediated-Label Reliability Graph (MLRG), using compatible measurement processes to determine whether conclusions are identified, partially identified or unidentifiable. Six evidence rules are then used to connect the available information to supported comparisons and label-treatment decisions.

The experiments examine how these choices affect evaluation outcomes. Changing from majority-label accuracy to distributional KL alters model rankings on FERPlus, while applying joint performance requirements withdraws support for six of twelve claims. The matched experiments on CIFAR10H and ChaosNLI also show that majority reduction increases divergence from the original response distributions. Overall, the paper attempts to formalise when evaluation conclusions are justified by the available evidence, although the practical framework relies on predefined rules and analyst judgements to make these decisions.

Critical strengths

1. I think the identification-based formulation is one of the stronger aspects of the paper. Rather than assuming that disagreement indicates label noise, the authors consider what conclusions remain valid across measurement processes compatible with the observations. The distinction between identified, partially identified and unidentifiable conclusions is useful, particularly when annotation provenance or independent reference information is unavailable. I also appreciate that they separate whether an action is justified from whether that action would improve model performance.

2. The matched experiments on CIFAR10H and ChaosNLI are reasonably well designed. Comparing distribution preservation, majority reduction and response sampling helps separate the effect of changing the target from differences in optimisation. The increase in KL under majority reduction is convincing for the declared response-distribution target, and I appreciate the sensitivity analysis around the bootstrap procedure. The authors are also careful not to overstate the non-inferiority results when those change under shared-item resampling.

3. I find the analysis of performance claims useful. The authors show that improvements in individual metrics do not necessarily support broader claims involving calibration, representation geometry or consistent modality contributions. In particular, the twelve real-data comparisons demonstrate how requiring several properties to hold jointly can change the interpretation of reported improvements. This is a useful contribution to evaluation methodology, although its reliability depends on how those requirements are specified.

Critical weaknesses

1. My main concern is the gap between the identification theory and its implementation. Section 3.2 defines supported conclusions over the set of compatible measurement processes, but the actual framework uses six predefined evidence rules and analyst-provided judgements. It is unclear how the complete or partial matching of these rules corresponds to identification over the compatible-process set. The rules may provide sufficient evidence for certain actions, but the paper does not establish whether they cover all relevant explanations or whether satisfying them guarantees the claimed identification status.

2. I am not convinced that the controlled experiments provide much validation beyond checking that the rules behave as designed. The authors construct profiles by making particular evidence conditions visible or hidden, and the resulting actions follow directly from those conditions. The reported 36/36 agreement is therefore expected. I would have liked to see cases involving conflicting evidence, overlapping explanations or incomplete annotation records where the correct conclusion is not already determined by the experimental setup. That would better test whether the framework can distinguish supported conclusions under realistic ambiguity.

3. The framework relies heavily on analyst interpretation, but the reliability of those judgements is not evaluated. Analysts specify the measurement contract, interpret evidence availability and assign source and action statuses. The compiler checks consistency with these inputs, but it cannot establish whether the underlying assumptions are correct. The authors also acknowledge that inter-analyst agreement has not been studied. This seems important because the framework is intended to make evaluation conclusions more defensible, yet two analysts could potentially reach different conclusions from the same dataset.

4. I also question how much the real-data comparisons validate the proposed MLRG and compiler. Switching from majority accuracy to distributional KL naturally changes what is being measured, and the FERPlus ranking reversals illustrate this clearly. Similarly, withdrawing support for broader performance claims follows from applying additional requirements. These are meaningful observations, but they do not establish that the proposed representation is necessary to obtain them. A simpler target specification with explicit evaluation conditions might lead to the same conclusions. I would want an ablation or comparison that isolates what the graph and compiler contribute.

5. The claim-support analysis is sensitive to the requirements chosen by the evaluator. The twelve comparisons use different combinations of metrics, tolerances and consistency conditions, but it is not clear how much the conclusions would change under other reasonable specifications. For example, requiring positive improvement at every seed is considerably stricter than testing whether the average improvement is statistically distinguishable from zero. Without examining the sensitivity of the results to these choices, the finding that six claims lose support is difficult to interpret as evidence of more reliable evaluation.

6. Finally, the majority-reduction results, while convincing, are somewhat expected under the selected target. When the evaluation objective is KL divergence from the empirical response distribution, replacing that distribution with a majority label removes information that the model is subsequently evaluated on. The experiments quantify the effect, but I am not sure they demonstrate a particularly new methodological insight. It would be more informative to understand when majority reduction is harmful beyond distributional fit, or under what conditions preserving disagreement actually improves the intended downstream evaluation.

Questions for authors
  1. How do the six evidence rules in Appendix B.3 correspond to the identification criterion in Section 3.2? Does a complete rule match guarantee identification over compatible measurement processes, or is it only a sufficient condition?

  2. The 36/36 agreement in Section 4.2 seems expected given that the profiles are constructed around the rule conditions. How would the framework handle conflicting evidence, overlapping rules or mechanisms outside the six templates?

  3. Since the MLRG and I/P/U judgements depend on analyst interpretation, have the authors measured inter-analyst agreement? Would different analysts produce the same evaluation protocols and conclusions?

  4. What does the MLRG and compiler achieve that a structured evaluation specification with explicit targets, metadata and requirements would not? Is there an ablation demonstrating the contribution of the graph?

  5. How were the thresholds and joint requirements in Appendix E.2 chosen? Would the six withdrawn claims remain unsupported under other reasonable thresholds or seed-consistency criteria?

  6. Since majority reduction discards information from the distributional target used for KL evaluation, isn't the observed degradation partly expected? What do these experiments establish beyond quantifying that loss?

1 comment

Thank you for the questions.

1. Why Six Rules, And What A Complete Match Establishes

Fix the target, then ask which conclusions hold across all processes consistent with the observations and assumptions. Each rule gives a sufficient way to establish a particular answer. Two rules concern label treatment and four concern comparisons:

  • A valid item reference establishes whether a recorded label is wrong. A known channel with a unique inverse identifies the underlying distribution.
  • Under an empirical response target, linked valid responses define the frequencies to preserve and score.
  • Source identities and item links determine which responses can be compared across sources.
  • Complete group links define group comparisons and the groups to keep separate across data splits.
  • Valid paired or controlled measurements compare responses under complete and incomplete information.
  • Context links and valid controls define comparisons between contexts. Causal attribution requires excluding remaining confounding.

These six common evidence structures are not exhaustive. Several can apply together. Additional mechanisms use the same compatibility criterion.

For example, two equally weighted sources with response rates 0.8 and 0.2 have pooled rate 0.5. Removing source links permits swapping the source assignments without changing the pool. Keeping the links fixes the comparison while its cause remains unknown.

For a non-empty compatible set, I means one answer remains. P means the evidence narrows the answers but leaves more than one. U leaves all original alternatives unresolved.

A complete match checks that all recorded conditions of one rule are satisfied. With valid premises, this is sufficient for its specified conclusion. Identifying the whole production mechanism or all eligible actions is a separate question. A partial match alone does not establish P. For example, channel inversion identifies a distribution while individual errors can remain unresolved.

2. Validation Beyond Prescribed Outputs

We test against answers calculated independently from probability laws, including overlapping mechanisms, incomplete records and conflicts. Our reply to YSGu, point 2, gives the reference construction and matched checklist comparison.

3. Analyst Reliability

We test how interpretation errors change decisions while holding the observations fixed. This locates the evidence assertion responsible for an unsupported action (reply to tEcZ, point 2). Direct measurement of analyst agreement is future work.

4. What The Graph And Compiler Contribute

The graph records the relations a comparison needs, such as which responses concern the same item. The compiler checks that the resulting conclusion is used with the right target and consistent records. We isolate these checks by comparing it with independent checked and unchecked checklists using identical inputs and output schemas.

Across 286 records from 32 software contracts, the compiler and checked checklist accept all 32 valid records and reject all 191 designed violations. These include wrong dataset/target bindings, inconsistent statuses and flags declaring result-dependent selection. The unchecked checklist accepts all 191, including 140 with active actions. Both checkers accept 31 records with cleared protocol fields, so completeness needs a separate check.

The checked checklist reproduces every acceptance decision and accepted protocol. All 32 graphs convert losslessly to node/relation tables with metadata retained. Thus a structured checklist can implement the framework by retaining its evidence relations and checks. The checks reject mismatched targets and inconsistent statuses, while the relations make the evidence behind each comparison inspectable.

5. Requirements And Sensitivity

Our reply to YSGu, point 1, gives the tolerance sources, timing and results across 414 settings, distinguishing stable claims from those requiring a narrower scope or a different seed criterion.

6. What Preservation Achieves Beyond KL

To test consequences for point decisions, we assess accuracy and error among confident predictions on the existing matched runs: 747 CIFAR10H and 242 ChaosNLI items, three seeds and 10,000 paired seed/item resamples.

Preservation improves modal accuracy over majority reduction by 6.96 [3.88,10.31] and 6.06 [0.96,11.43] percentage points, respectively (pointwise 95% intervals). Accuracy against the stored CIFAR-10 class improves by 6.69 [3.70,9.91]. Selective error falls at 50%, 70% and 90% coverage on CIFAR10H, and at 90% on ChaosNLI. These gains show benefits for predicting the mode and choosing confident decisions, beyond distributional fit. Accuracy and selective-error differences from response sampling have intervals including zero. The largest exploratory ChaosNLI KL benefit occurs at low entropy, so the benefit is not confined to highly disputed items.

Comments on the paper

1 comment

1. Organisation Of The Responses

We thank all three reviewers for their feedback. We considered all weaknesses and questions together and distributed the answers across the three reviewer replies. Shared concerns receive one detailed answer, with cross-references in the other replies to avoid repetition.

2. Concerns And Added Studies

W and Q denote weaknesses and questions in review order.

  • A. Independent validation (jWu4 W2/Q2, YSGu W1-W2): finite-process references and a matched checklist comparison.
  • B. Incorrect evidence interpretation (jWu4 W3, tEcZ Q2): paired changes to evidence conditions and targeted documentation misreadings.
  • C. Graph/compiler contribution (jWu4 Q4, YSGu Q2): record-check comparisons and graph-to-table conversion.
  • D. Requirement choice and sensitivity (jWu4 Q5, YSGu Q1): source/timing audit and alternative tolerance and seed requirements.
  • E. FERPlus ranking uncertainty (YSGu Q3): paired seed/item resampling.
  • F. Benefits beyond KL (jWu4 Q6): accuracy, selective prediction and response-group analyses on the matched training runs.

3. Reading Order And Main-Text Explanations

To address tEcZ's presentation concerns, the Abstract states the procedure in plain terms, and Sections 1-2 start from the research decision. The 80/20 example separates choosing a target from deciding that a response is wrong. Related Work then explains what existing approaches assume and where the framework fits.

Section 3 fixes the target first, so the reader knows which question the evidence must answer. It then records the relations needed for the supported comparison and specifies its evaluation requirements. Table 1 pairs rule conditions with conclusions, and ChaosNLI connects actual fields to a research decision. Section 4 first tests decision validity, then assesses model claims with predictions fixed. Finally, matched training tests label-treatment consequences. Key dataset context, requirements and results are now in the main text. Section 5 discusses implications, limitations and future user evaluation.

4. Study Purposes, Findings And Locations

A. Independent Validation. Reference answers from finite probability laws test decisions beyond prescribed rule outputs. A complete checklist applying our rules separately retains 59/66 supported action instances, versus 25/66 for the original matcher, across 68 non-conflicting views. Both produce no unsupported decisions. The mixed-mechanism case shows the value of retaining repair and preservation as separately justified options. Section 4.2, Table 2, and Appendix D.2, Tables 17-18, give the design and results.

B. Interpretation Errors. Paired encoding changes, with observations fixed, test how misreading evidence changes a decision. Omissions can remove supported options, whereas false assertions can activate unsupported rules. The majority-vote case traces an unsupported correction to the assertion of an independent reference, showing which premise needs source verification. Section 4.2, Appendix D.2.4 and Table 19 report these tests.

C. Representation And Checks. Checked and unchecked checklists isolate which checks prevent errors in using recorded conclusions. The compiler and checked checklist agree on 286 records and reject 191 designed violations, including wrong target bindings. Lossless conversion of 32 graphs shows that tables retain the same evidence relations. Both checkers accept cleared protocol fields, so completeness needs an additional check. Section 4.2 and Appendix G, Table 40, report this comparison.

D. Requirement Sensitivity. To test dependence on evaluator choices, we audit requirement sources and timing and examine 414 settings. Scaling non-zero tolerances leaves all twelve decisions unchanged. Relaxing zero boundaries or seed consistency changes three of six withdrawals. CASE retains its full claim, while ChaosNLI supports a global claim or a joint claim with a stated local allowance. Sections 3.4 and 4.3, Table 4, and Appendix E.2-E.5, Tables 21-26, give the requirements and comparisons.

E. Ranking Uncertainty. Paired resampling tests which FERPlus rank differences justify model selection. The approximately 0.001-nat gap between Residual Edge and Global Control remains unresolved, whereas Edge has lower KL than Within-Class Control. Section 4.3, Table 3, and Appendix E.6-E.7, Tables 27-32, give system values and paired uncertainty.

F. Prediction Consequences. Accuracy and selective-error analyses test benefits beyond distributional fit. Preservation raises modal accuracy over majority reduction by 6.96 and 6.06 percentage points on CIFAR10H and ChaosNLI. Error among the most confident 90% of predictions also falls on both datasets. Response-group comparisons locate where KL gains arise. Section 4.4, Table 5, and Appendix F.6-F.7, Tables 36-39, give these results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.