- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper asks when uncertainty in a model's predicted concepts can change its final decision, and which concepts warrant human inspection. It uses disagreement among several concept predictors to define a region around their average prediction, and tests whether a linear classifier gives the same answer throughout that region. It also ranks concepts by combining their uncertainty with the classifier's sensitivity to them. In experiments, this stability score performs similarly to confidence when choosing which predictions to keep, but the tested single-concept correction rules reduce average accuracy on the CEBaB restaurant-review dataset.
Critical strengths
-
The paper tackles a useful problem: uncertainty in the predicted concepts does not necessarily matter for the final decision. It gives a simple way to check whether a linear classifier's answer stays the same over a chosen set of concept values.
-
The evaluation tests the approach across three datasets and compares it with simple uncertainty baselines. It achieves higher average accuracy among retained predictions than raw concept disagreement and performs competitively with confidence-based selection.
Critical weaknesses
-
The experiments do not isolate what the ensemble-based region adds. For selective prediction, the reported margin-based results match or trail confidence. The higher repair and damage rates among flagged examples also lack a comparison with confidence or the gap between the top two class scores at the same review budget. These controls would show whether the method adds information beyond proximity to a decision boundary. Most variance-selected corrections fall outside the region, limiting the certificate's relevance to them. Varying the region's size would show the trade-off between certifying predictions and covering actual corrections.
-
The novelty needs clearer positioning. Mukherjee et al. (ACL 2026) already propagate ensemble-derived concept intervals through a linear classifier. Here, the certificate compares pairs of classes over an ellipsoid. Comparing the assumptions and guarantees would clarify the contribution. The Shapley identity follows directly from the additive game; its value is interpretive.
-
The correction experiment uses only robust training (Mode C), leaving its generality unclear. Repeating it with ordinary task-and-concept training (Mode A) would test whether the harm is specific to that regime. With only four concepts, testing every eligible single correction would also show whether helpful corrections exist but the rankings miss them.
Questions for authors
- At matched review budgets, does the margin better identify examples where corrections repair or damage predictions than confidence or the top-two score gap? How does region size affect certification and coverage of corrections?
- Does harm persist without robust training? How often can a single correction help, and how often do the rankings find it?
- Please clarify novelty with respect to Mukherjee et al.
3 comments
Response to reviewer ZQqz
Any prediction-changing replacement on a certified example must leave the region; the converse does not hold. We checked that no certified, in-region selected replacement changes the prediction at any tested radius. We will report the per-seed counts and denominators alongside these summaries in the revision.
Ordinary training and exhaustive single corrections. We enumerate every eligible single-concept gold replacement. An initial error is repairable if at least one such replacement fixes it. The conditional hit rate asks whether the rule selects a fixing concept among those repairable errors. For uniform-random selection, this conditional probability is calculated exactly as the fraction of eligible concepts that fix each repairable example, then averaged over repairable examples.
| Run | Initial errors | Repairable errors (%) | Gradient fixes/breaks | Gradient conditional hit (%) | Uniform-random conditional hit (%) |
|---|---|---|---|---|---|
| C0 | 343 | 159 (46.4) | 120/264 | 75.5 | 61.0 |
| C1 | 339 | 155 (45.7) | 109/255 | 70.3 | 65.1 |
| C2 | 353 | 134 (38.0) | 98/191 | 73.1 | 56.2 |
| A0 | 345 | 145 (42.0) | 104/254 | 71.7 | 59.1 |
| A1 | 356 | 161 (45.2) | 111/248 | 68.9 | 64.8 |
| A2 | 369 | 140 (37.9) | 104/194 | 74.3 | 55.4 |
Mean repairability is 43.3% in Mode C and 41.7% in Mode A. The gradient rule's conditional hit rate is 73.0% versus 60.8% for random in Mode C, and 71.7% versus 59.8% in Mode A. These conditional results do not imply positive intervention utility: the gradient rule also breaks many initially correct predictions, producing mean accuracy changes of −7.56 and −7.44 points. The variance, PL, and weight-only rules also reduce accuracy in every Mode A seed. Thus harm persists without robust training, although these experiments do not establish its causal mechanism.
As a derived hindsight upper bound, an oracle that chooses a fixing concept whenever one exists and otherwise makes no correction would improve Mode C accuracy by 159/1689, 155/1689, and 134/1689: 9.41, 9.18, and 7.93 percentage points. It would break no correct predictions. This uses hindsight and a no-correction option; it is not an achieved performance result or a deployable policy.
Relation to Mukherjee et al. Mukherjee et al. (ACL 2026) construct ensemble min–max concept intervals, propagate them through a linear classifier to obtain exact per-class logit bounds, and study epistemic–aleatoric decomposition. Our focus is pairwise decision stability over a specified ellipsoid and attribution of a fixed local loss penalty, followed by evaluation of correction outcomes. We will acknowledge the standard support-function and additive-game arguments and position the work within concept-space uncertainty analysis.
For the same feasible region, directly minimizing a logit difference avoids combining separately attained class extrema. This observation also applies to boxes and does not establish an advantage of ellipsoids. A diagonal ellipsoid is contained in the box with the same center and coordinate half-widths, but this comparison does not automatically apply to ensemble min–max intervals. We will distinguish these geometries, use the probability-box constraint , and avoid claiming that changing geometry alone establishes novelty or improved routing.
We remain available to answer further questions or queries.
Dear Reviewer As the author-reviewer discussion period ends soon we kindly request that you read our response and confirm whether your concerns are addressed. We remain committed to answering any further questions you have.
Response to reviewer ZQqz
We thank the reviewer for his thoughtful response and question especially specifying controls required to assess the disagreement region. We have now completed matched-budget and exhaustive-correction analyses for both ordinary training (Mode A) and joint robust training (Mode C), and extended the radius analysis to measure actual replacement containment. All results below use CEBaB seeds 0–2 and the same test examples
Matched review budgets. We hold the checkpoint, eligibility mask, and predicted-label gradient correction rule fixed, and vary only the score used to select examples for review. The margin is minc=y^dc/vcTΣvc, where vc=Wy^−Wc and dc=vcTμ+by^−bc. The baselines are the top-two logit gap and maximum softmax confidence. Each score flags its 495 lowest-scoring examples, including any without eligible annotations; only eligible flagged examples receive a correction. Unflagged examples remain unchanged.
Each cell reports errors captured / repairable errors captured / fixes / breaks; overall accuracy change in percentage points. The change is (fixes−breaks)/1689, multiplied by 100.
Margin captures fewer repairable errors than gap or confidence in every run. Correcting the flagged set reduces accuracy for all three scores in all six runs, with changes from −0.77 to −3.20 points. There is no consistent advantage for margin in net correction outcome. This experiment evaluates one common budget, not a curve over intervention budgets.
For routing without correction, the three-seed mean MRA values for margin/gap/confidence are .8575/.8647/.8646 in Mode C and .8531/.8588/.8588 in Mode A. MRA averages retained accuracy over coverages 0.50,0.60,0.70,0.80,0.90,0.95,1.00, with floor counts and stable input-order ties. Euclidean, trace-isotropic, and interval controls give .8623/.8580/.8586 in Mode C and .8590/.8549/.8552 in Mode A. Their denominators are respectively ∥vc∥2, ∥vc∥2tr(Σ)/K, and ∑k∣vck∣σk; the last uses the unclipped box μ±σ. These results do not establish improved routing over gap or confidence.
Region size and actual correction containment. Scaling Σ to t2Σ gives mt=m/t. It preserves the ranking and MRA, but changes certification to m>t and the flagged set to m≤t. We have now measured two distinct containment quantities at each radius: whether the selected single-concept gold replacement lies inside the region, and whether the vector matching all eligible annotations, with unknown coordinates left at their means, lies inside it. For diagonal geometry, leaving unknown coordinates at their means minimizes their contribution to the Mahalanobis distance, so the latter checks existence of an in-region completion.
The following are means over seeds, with Mode C values followed by Mode A in parentheses. Certification uses all 1,689 examples; both containment columns use the 1,583 eligible examples. Selection uses the predicted-label gradient rule and is held fixed as the radius changes.
These containment measurements are separate from the outcome rates among flagged examples:
The rates above are means of per-seed rates, not pooled ratios. Increasing the radius includes more actual replacements but certifies fewer examples. At unit scale, most selected replacements lie outside the specified region, limiting what this certificate can say about their outcomes.