- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies the discrepancy between an "AI-likeliness score" a language model may assign to its generation during generation and after generation (from a retrospective viewpoint). In particular, the authors tested 3 evaluation schema: ask the model to evaluate its generation after each sentence, ask it to evaluate a completed generation one sentence at a conversation round, and finally ask it to grade all sentences in an completed response.
While the problem appears interesting, I find the problem of interesting not being well-defined and the experiment setup highly restrictive to a particular setting, which could be a good course project instead of a top-venue conference paper. Thus I recommend for rejection.
Critical strengths
None
Critical weaknesses
The phenomenon is not defined or tested with any generality. The current study is highly tied to the specific AI-likeliness setting under social media posts and specific chat templates used for interaction. The heterogeneous results also does not present a conclusive finding. Therefore I believe this paper shall be treated as an interesting standalone phenomenon presentation instead of a scientific research.
Questions for authors
None. I believe the modification required for acceptance would be substantially changing the current content of the paper.