- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes a new method to examine LLM judgement structures with taking into account their confidence scores. The method, called ConfStruct, normalize confidence scores using CDF kernel so that normalized scores are uniform. Thanks to that, ConfStruct then can compare judgements from different LLMs, aggregate them in coherent trajectories and single scores. The paper goes further with introducing complementary panel-level diagnostics---consensus, verdict diversity, polarization, and tie prevalence.
In experiments, the paper demonstrates how ConfStruct is able to track judgement structures of 18 LLMs. The paper also shows that the proposed aggregated score ranking highly correlated with ground-truth annotation on a dataset.
Critical strengths
This is an interesting paper tackling a challenge (but very useful if solved), that is how to aggregate agents opinions. The ideas are interesting, and to me, novel. The paper presents pretty strong math framework.
The paper also presents lots of information in the appendices.
The results shown are interesting, demonstrating how the work can be used helpfully in the community.
Critical weaknesses
One of the major problems of the paper is presentation.
-
First of all, the maths presentation should come with clear explanations. For instance, what is the intuition behind equation 4? What are domains/ranges of introduced functions/values? w_i(x, j) is not well-formed when x doesn't contain y_j. The claim that R (equation 11) is well defined isn't correct when all models agree with each other (in this case, spread(lambda, c) == 0 and thus g == 0 always).
-
Second, in the experiments, I believe that it's extremely crucial to demonstrate that the proposed ConfStruct is faithful. Therefore, the key experiment results should be Table 1 and section 4.4. Hence, my suggestion is to move the results in table 1 and section 4.4. to the beginning of section experiments. Other results should be considered as how ConfStruct is used.
Following the second point above, ConfStruct should have tested on more datasets/benchmarks with groudtruth (like table 1) to prove its faithfulness. Besides, although section 4.4 is important, it's too brief, especially, it's unclear why d_{H,M} vs d_{M,M} can tell anything meaningful about ConfStruct (w/o reading the corresponding appendix).
Some minor points really need to be clarified.
- Line 156, wikjl = 1 − wijkl. Is it true when using LLMs? Very often an LLM can give different output for different input orders e.g. (y_i, y_k) vs (y_k, y_i)
- Line 255, "The factor 4 normalizes the score to peak at 1...", why?
- Line 431, "GPQA Diamond (Rein et al., 2023), where a correct answer exists but is never revealed to the judges" --- but is it true that the used LLMs weren't trained on this dataset?
Questions for authors
Please see above.