- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes TRUST, a framework for improving the estimation of target LLM failure rates by transferring judge reliability information from auxiliary benchmarks to a target benchmark. The framework leverages semantic clustering, confidence stratification, and target-guided filtering to identify local regions where judge behavior can be reliably transferred. The authors also provide a theoretical analysis and validate the proposed method across multiple benchmarks.
Critical strengths
-
The paper addresses an important problem, as accurately estimating LLM error rates is challenging given the high cost of human annotation.
-
The idea of transferring TPR/FPR rather than directly transferring error rates is novel.
Critical weaknesses
-
The proposed method focuses on binary correct/incorrect labels. However, LLM evaluation can also involve multi-level scoring tasks. It is unclear how the proposed method and its corresponding theoretical analysis could be extended to these broader settings.
-
The proposed pipeline is relatively complex, yet some of its components may not be essential. For example, Figure 12 shows that the performance is relatively insensitive to the filtering threshold.
-
The paper analyzes the mean squared error (MSE), which can be decomposed into squared bias and variance. A separate analysis of these two components would provide additional insights into the proposed method.
Questions for authors
See Weaknesses above.
Please indicate whether your concern has been addressed, or whether there are remaining concerns that should be discussed.