- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper focuses on the problem that LLMs answer ToM questions inconsistently, while Bayesian ToM methods require reliable probabilities. The authors identify that existing methods depend on unavailable token probabilities or unreliable verbalized numerical probabilities. To address this gap, the authors propose OBToM, which asks an LLM for ordinal likelihood comparisons and convert them into Bayesian-graph probability tables, and uses exact inference to select the answer. They found that this framework achieves better accuracy on multiple ToM benchmarks.
Critical strengths
The design of the framework is very coherent. It turns simple ordinal LLM judgements into structured Bayesian ToM inference without relying on the model's internal statistics. The experimental results are really promising.
Critical weaknesses
-
I generally appreciate the framework for its coherent combination of multiple components. The authors showed promising results and compared model requests. However, the authors did not conduct any ablation study for the components, nor did they conduct a probability calibration evaluation. Therefore, it is unclear whether the gains come from ordinal estimation, low-rank sharing, the hard-coded perception gate, or their combination. This is very important to confirm the validity of the desired framework.
-
The experiments, while their results seem promising, are conducted only on one LLM backbone. I would be more convinced if the framework demonstrates more general performance improvement on other LLM backbones. Additionally, I would like to see some other tests beyond answer accuracy. For example, among answers assigned probability p, what percentage of them are actually correct? For a ToM question with known Bayesian graph CPTs, how far could the framework recover them? A case study might also be helpful.
Questions for authors
see weaknesses