- Rating
- 1
- Confidence
- 1
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates causal effects of interventions in large language models in a theory-of-mind-style dataset. I could not discern from the paper how the actual study design, dataset, and conclusions look like.
Critical strengths
The poor writing quality makes it hard to me to have enough confidence to judge the merits of the paper.
Critical weaknesses
The writing quality is extremely poor. The abstract seems to be partially a description of the experimental methodology, which is an unconventional choice, which is not necessarily bad per se, although it misses the high-level point and the emphasis on the contribution. I read both abstract and conclusion multiple times, and could not figure out what kind of study the authors conducted and what the high-level conclusion is. The experimental design details, what the three different settings actually mean (what is the operation? what would be a story? what does no component exchange mean?), and details in the tables are insufficiently explained to be sufficiently judged.
Unfortunately, the paper lacks several critical aspects, which make it hard to assess the contribution. Namely, the description of the dataset is missing - there is a brief summary of some parameters in Sec. 4, but no examples, prompt, etc. They refer to theory of mind and exchanging locations, but what does that mean exactly? What type of dataset is it? Whether there are any examples, the dataset size, etc., are not described in the manuscript, despite the fact that the paper has space.
Questions for authors
N/A