acceptodds
Under review as a conference paper at ICLR 2027

Label-Free Aggregation Thresholds for Set-Valued Self-Consistency

Abstract

Drawing several samples and combining them is standard test-time practice. When outputs are sets, combining means keeping the candidates that appear in more than a fraction of samples, and practice defaults to . We show that can be read off the unlabeled samples themselves. A predictor built from sampling-sharpness statistics recovers 68% of the oracle gain with no annotation, beats a grid search costing ten labelled sentences (+0.013 F1, bootstrap CI above zero), and ties one costing twenty-five; as a prior it never does worse, even refit on task families it never saw. We package it as a label-free triage rule. It works because a sample frequency is not a probability of correctness: every model family we measure is overconfident, with calibration gaps as large as -0.261 on multi-label, and the default is below optimal in 192 of 246 pairs across 5 kinds of set-valued output and 31 checkpoints. What governs the miscalibration is sampling sharpness—temperature and instruction tuning are two routes to it rather than two mechanisms, and matching on sharpness makes the apparent tuning gap vanish. One consequence inverts the usual advice: aggregation buys precision, not recall, and its gain is largest where extraction is failing, so a large gain is a cue to fix the pipeline. The threshold condition itself is classical, and a sample frequency is not the calibrated probability it assumes; we contribute the miscalibration measurement, the sharpness mechanism, and the label-free procedure. We release every sample, so all numbers recompute without a GPU.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
large language models, test-time sampling, self-consistency, set-valued prediction, structured prediction, calibration, threshold selection, uncertainty quantification

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_W7MDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on test-time aggregation for set-valued prediction tasks with large language models, including named entity recognition, relation extraction, multi-label classification, and keyphrase extraction. Specifically, it studies how to select aggregation thresholds for self-consistency without requiring labeled data from the target task. The paper connects this behavior to the miscalibration of sample frequencies and investigates how sampling sharpness changes with temperature and instruction tuning. It then proposes a linear predictor that estimates the aggregation threshold from three statistics computed on unlabeled samples, together with a disagreement-based heuristic for deciding when aggregation is useful. In experiments, the proposed predictor improves F1 over majority voting.

Critical strengths
  1. The paper addresses a practical question and the empirical evidence is convincing. The analysis spans multiple model families, scales, languages, and output types.
  2. The precision–recall analysis is informative. The finding that majority aggregation primarily improves precision, with relatively little benefit to recall, helps clarify when additional sampling is actually worthwhile.
Critical weaknesses
  1. The novelty is limited. The threshold predictor proposed by the author essentially utilizes three sampling statistical features for linear regression. Is the paper primarily aimed at validating an existing theory rather than proposing innovative new methods?
  2. The predictor trained by the author only improved by an average of 0.003 F1 compared to directly using threshold 0.25. Since a simple fixed threshold can achieve similar results, why is it necessary to train additional predictors?
Questions for authors

None

Reviewer_w3AwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies self-consistency when the model output is a set. The usual rule keeps a candidate if it occurs in more than half of the k samples (θ\theta = 1/2). The paper asks where θ\theta must be, why 1/2 is often incorrect, and how to set θ\theta. In experiments, the authors utilized 31 models across 7 families and different sizes and 9 datasets over five output types.

Critical strengths
  • The θ\theta = 1/2 rule is common in NER and extraction pipelines. The paper shows that this rule loses F1.
  • Many studies hypothesize that fine-tuning hurts calibration. This paper test this hypothesis. It matches tuned and base models and then compares them.
Critical weaknesses

I could not understand the following: Initially, section 4 says that 124 tuned cells have a matched base cell. But It later says that 114 tuned cells are sharper than their base cell. If hte 114 cells are part of the 124? If this is the case, then we only have 10 cells enter the comparison?

Questions for authors

See weakness

Reviewer_ruBdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper shows that sampling sharpness explains the miscalibration mentioned across every intervention they can make.

Critical strengths

The proofs seem to be solid.

Critical weaknesses

The experiment is not sufficient. More experiments, especially those from real applications, should be introduced.

Questions for authors

Could authors ad more experiments, especially those from real applications?

Comments on the paper

1 comment

Thank you for your rebuttal. I now understand this better. I will keep my rating for acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.