- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This work proposes a self-learning framework for creating a generalized prompt for an LLM facilitator that engages in a multi-party setting. The approach starts by using expert agents with manually created prompts tailored for different multi-party contexts in a human study. The data from that study then feeds into the framework, which iteratively refines a generalized facilitation prompt by comparing its projected performance with that of the previous iteration's prompt. The learned prompt is compared with the expert prompts using the test split of that dataset, showing an improvement in performance. It is also used in another human study which, compared to the first study, shows an improvement in participant experience.
Critical strengths
-
The proposed framework is well-designed. It can be repurposed for different applications, given that a judge can be given a meaningful rubric to close the loop.
-
The use of human studies grounds the results and strengthens the claims compared to relying on simulated participants. Using the manually-crafted experts in the first study provides a dataset with a very good starting point for the system to learn from. These experts also act as a strong baseline for comparison. When I read the paper, it wasn't clear to me whether the authors plan to release the datasets. I recommend they consider doing so, as it will be a good addition to the research community.
-
Rigorous data analysis highlighting the different set of rules that led to a better participant experience. Such rules can be transferred to other applications, such as therapeutic chatbots, where a good client experience is crucial.
Critical weaknesses
2 minor points:
-
In the second study, it would've been a better control if the expert agents were used with the same participants or even with different participants. This would've controlled for the participants' expectations of the LLM facilitators (humans' expectations towards AI change over time). I understand that the authors accounted for this in their results, but I still recommend that the authors acknowledge this in the limitations section.
-
Table 2 was a bit harder to understand (second set of items, not first one). I recommend that the authors try to improve its presentation by expanding the caption (or adding a footnote) to explain the ratios. For example, the authors mention that the expert prompts include 91 instructions, while the table has 97; it's unclear whether this is a mistake or an adjustment that wasn't clarified. Also, the authors are encouraged to revise the numbers reported in the results (e.g., the text and table mention 47 rules common between the experts and the generalized prompt, while Figure 3(a) mentions 46).
Questions for authors
N/A
We thank the reviewer for this thoughtful suggestion and agree that a within-subjects design would strengthen the comparison. Unfortunately, it was not feasible in our setting because of task exposure: participants from Study 1 had already seen the discussion scenarios, so assigning them the same tasks in Study 2 would have introduced familiarity effects. We fully agree that a concurrent, randomized control group with new participants would be the ideal design; while this was beyond the scope of the present work, we see it as an important direction for future studies. Following the reviewer's comment, we will note in the limitations that the between-study comparison does not fully control for cohort differences in prior attitudes toward AI facilitation, and we will revise Table 2 accordingly.