- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper conducts an empirical study using EGG dataset to audit a deep learning network (Conv1d + a four-layer transformer). Their results reveal that 1) there is correlation between uncertainty score and model error, and 2) uncertainty score can predict the masked channel in EGG data. The authors also conduct additional checks showing that 1) the masked channel would make the prediction different and 2) the results can not be interpreted by conditional ambiguity (through a toy example with 8 different possible oracles).
Critical strengths
- The study is thorough. The authors provide evidence to exclude the confounders of their conclusions.
Critical weaknesses
- The empirical results are not new. The two main takeaways (in my opinion) have all been studied by prior works [1, 2, 3].
- The scope of the paper is somehow limited. The authors only study one model and one dataset. It's hard to think whether the results can be generalized to other fields (especially the misinformation is only modeled as masked channels in EGG type of data).
[1] Xiong, Miao, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. "Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms." In International Conference on Learning Representations, vol. 2024, pp. 23650-23678. 2024.
[2] Gadgil, Soham, Ian Covert, and Su-In Lee. "Estimating conditional mutual information for dynamic feature selection." In International Conference on Learning Representations, vol. 2024, pp. 34979-35010. 2024.
[3] Hou, Bairu, Yujian Liu, Kaizhi Qian, Jacob Andreas, Shiyu Chang, and Yang Zhang. "Decomposing uncertainty for large language models through input clarification ensembling." arXiv preprint arXiv:2311.08718 (2023).
Questions for authors
- What is the new contribution offered by this paper over the existing works listed above?
- Is there any unique takeaway from this analysis on EGG data? This might be related to my previous concern about the generalization and new contribution.