- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper focuses on test-time aggregation for set-valued prediction tasks with large language models, including named entity recognition, relation extraction, multi-label classification, and keyphrase extraction. Specifically, it studies how to select aggregation thresholds for self-consistency without requiring labeled data from the target task. The paper connects this behavior to the miscalibration of sample frequencies and investigates how sampling sharpness changes with temperature and instruction tuning. It then proposes a linear predictor that estimates the aggregation threshold from three statistics computed on unlabeled samples, together with a disagreement-based heuristic for deciding when aggregation is useful. In experiments, the proposed predictor improves F1 over majority voting.
Critical strengths
- The paper addresses a practical question and the empirical evidence is convincing. The analysis spans multiple model families, scales, languages, and output types.
- The precision–recall analysis is informative. The finding that majority aggregation primarily improves precision, with relatively little benefit to recall, helps clarify when additional sampling is actually worthwhile.
Critical weaknesses
- The novelty is limited. The threshold predictor proposed by the author essentially utilizes three sampling statistical features for linear regression. Is the paper primarily aimed at validating an existing theory rather than proposing innovative new methods?
- The predictor trained by the author only improved by an average of 0.003 F1 compared to directly using threshold 0.25. Since a simple fixed threshold can achieve similar results, why is it necessary to train additional predictors?
Questions for authors
None
Thank you for your rebuttal. I now understand this better. I will keep my rating for acceptance.