- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper asks the question whether including human history improves model answer. At certain turns, the model is presented either with the history made of all the human turns, or a mixture of human and machine generated turns (in the same conversation), and the semantic similarity is measured between the ground-truth answer and the newly generated answer. The paper shows that there is only partial
Critical strengths
an interesting problem setup
Critical weaknesses
underpowered metrics (all the results are reported using a metric previously shown to be less correlated with human judgements compared with some other metrics) lack of connection to the real use case (when can we insert ground truth utterances instead of human ones) older methods lack of important related work (the paper claims it is the first study of this type, while the problem has been clearly studied in the past) limited dataset used (the paper does not use the most often used multi-turn datasets) writing clarity
Questions for authors
Why was the semantic similarity metric the only one used? Was there any study into human alignment of that metric against other evaluation metrics?
Why were the experiments not conducted on the newer versions of the models?
What were the exact versions of the models used?
Why were the 3 dataset used, and some more often used multi-turn dataset omitted (MtBench, AlpacaChat, MultiWoz, PersonaChat, MuTual).
Was there any "solvability" study done? What are the guarantees the tasks presented could be solved by the models?
How are differences in the semantic space interpreted? Are they statistically significant, and does that map to any sort of human understanding (e..g is the score of 0.55 suggesting that the answer is not suitable at all, and the score of 0.8 that the answer is paraphrased but keeps the same semantics?)
Why were the later utterances curtailed, is it not more realistic setup to keep last two human utterances instead of the first two?
What is the use case findings of the experiments support? In what situations do we, at inference time, have a "choice" to include the human "ground-truth" turns?