- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a framework to calibrate LLM judges with human labels. In particular, it cross-fits a lightweight model that uses the scores, embeddings, and uncertainty features from multiple LLM judges to predict the human scores. Fitting is done a small subset of the dataset, so human labels are kept minimal. Experiment results show that the trained models achieve high item-level correlation with human scores and accurate population-level summary metrics. In addition, embedding features alone can achieve well-calibrated item-level and population-level results.
Critical strengths
- The method is simple and practical. It does not involve training the LLM judges and only requires fitting lightweight models on a small dataset.
- Ablation studies are extensive, covering single/multi-judges, with/without embeddings, and various human label budgets.
Critical weaknesses
- The novelty is very limited. The proposed method simply combines various features to fit a lightweight neural network, and there is no methodological novelty or theoretical novelty. The choice of LLM scores, embedding features, and uncertainty features also seems arbitrary and lacks justification.
- Important experimental results are missing in the main paper, and many results added in the Appendix are negative. Section 3.2 reports the item-level results on the PERSUADE dataset, while Section 3.3 reports the population-level results on the MQM dataset. Since the paper identifies both the item-level prediction and summary metric prediction as equally important use cases, it should report both of them on all datasets, instead of reporting each of them on a separate dataset. Although these results are reported in Appendix A.1 and Appendix A.2, the item-level results on the MQM dataset are much worse than on the PERSUADE dataset, with and .
Questions for authors
- How does your method handle cases where disagreement among human raters is high? Current results seem to be based on a setting with a single human score per item.
- How does your method compare with a training-based method that fine-tunes LLM judges on human ratings?