acceptodds
Under review as a conference paper at ICLR 2027

Learning to GUIDE the Group: Self-Learning LLM Facilitation Across Multi-Party Tasks

Abstract

As large language models (LLMs) are increasingly adopted as multi-party communication facilitators, there is a growing demand for agents capable of balancing diverse group dynamics across contexts. However, existing LLM facilitators are often fine-tuned for task-specific objectives; generalized LLM-based facilitation methods that can adapt to different types of tasks and diverse group dynamics are underdeveloped. To address this gap, we present GUIDE (Generalized Unified Intervention for Dynamic Exchanges), a framework for developing generalized LLM-based facilitators across different multi-party contexts, such as open-ended discussion, negotiation, and consensus-building. GUIDE evaluates facilitation across both objective task outcomes and subjective metrics, including psychological safety and perceived friction. Within this framework, we study whether LLMs can self-learn generalized facilitation prompt updates offline to real-time interventions. We first collect a corpus of multi-party human interactions guided by task-specific expert LLM facilitators (N=432 conversations). Grounded in these dialogues, we then employ a self-learning method in which an LLM induces an actionable strategic playbook and iteratively refines it using feedback from an LLM judge aligned with human annotators (N=20), developing a single generalized facilitator that outperforms the three task-specific expert facilitators on held-out conversations (61.4% win rate). Finally, we evaluate these improvements empirically; in a real-time online empirical study (N=462 conversations), the self-learned generalized facilitator preserves facilitator performance and group productivity, improves speaking comfort and reduces interpersonal friction, while using 24.7% fewer turns and 29.0% fewer words than the expert prompts. These findings provide empirical evidence that LLMs can learn generalizable, task-agnostic social facilitation from task-specific human interactions, on-par with hand-crafted task-specific prompts across diverse group settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
self-learn, human-centered AI, LLM intervention

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Jw3KOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This work proposes a self-learning framework for creating a generalized prompt for an LLM facilitator that engages in a multi-party setting. The approach starts by using expert agents with manually created prompts tailored for different multi-party contexts in a human study. The data from that study then feeds into the framework, which iteratively refines a generalized facilitation prompt by comparing its projected performance with that of the previous iteration's prompt. The learned prompt is compared with the expert prompts using the test split of that dataset, showing an improvement in performance. It is also used in another human study which, compared to the first study, shows an improvement in participant experience.

Critical strengths
  1. The proposed framework is well-designed. It can be repurposed for different applications, given that a judge can be given a meaningful rubric to close the loop.

  2. The use of human studies grounds the results and strengthens the claims compared to relying on simulated participants. Using the manually-crafted experts in the first study provides a dataset with a very good starting point for the system to learn from. These experts also act as a strong baseline for comparison. When I read the paper, it wasn't clear to me whether the authors plan to release the datasets. I recommend they consider doing so, as it will be a good addition to the research community.

  3. Rigorous data analysis highlighting the different set of rules that led to a better participant experience. Such rules can be transferred to other applications, such as therapeutic chatbots, where a good client experience is crucial.

Critical weaknesses

2 minor points:

  1. In the second study, it would've been a better control if the expert agents were used with the same participants or even with different participants. This would've controlled for the participants' expectations of the LLM facilitators (humans' expectations towards AI change over time). I understand that the authors accounted for this in their results, but I still recommend that the authors acknowledge this in the limitations section.

  2. Table 2 was a bit harder to understand (second set of items, not first one). I recommend that the authors try to improve its presentation by expanding the caption (or adding a footnote) to explain the ratios. For example, the authors mention that the expert prompts include 91 instructions, while the table has 97; it's unclear whether this is a mistake or an adjustment that wasn't clarified. Also, the authors are encouraged to revise the numbers reported in the results (e.g., the text and table mention 47 rules common between the experts and the generalized prompt, while Figure 3(a) mentions 46).

Questions for authors

N/A

1 comment

We thank the reviewer for this thoughtful suggestion and agree that a within-subjects design would strengthen the comparison. Unfortunately, it was not feasible in our setting because of task exposure: participants from Study 1 had already seen the discussion scenarios, so assigning them the same tasks in Study 2 would have introduced familiarity effects. We fully agree that a concurrent, randomized control group with new participants would be the ideal design; while this was beyond the scope of the present work, we see it as an important direction for future studies. Following the reviewer's comment, we will note in the limitations that the between-study comparison does not fully control for cohort differences in prior attitudes toward AI facilitation, and we will revise Table 2 accordingly.

Reviewer_eLoWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether an automatically learned prompt can replace task-specific, hand-written prompts for an LLM facilitator in multi-party chats. Three participants and an AI facilitator complete an open-ended policy discussion, a three-party negotiation, and a charity consensus task. Study 1 (144 groups, 432 episodes) uses three expert prompts and yields a corpus of transcripts, surveys, and outcomes. Starting from an empty playbook, an ACE-style loop edits natural-language rules over 30 generations. The loop has a conversation reflector, a curator, and a validation reflector. An LLM judge calibrated to 20 annotators (κ = 0.70) compares each candidate with the previous playbook, and per-rule credit drives pruning. The final 53-rule playbook wins 61.4% of 736 held-out decisions against the expert prompts. In Study 2 (154 new groups, 462 episodes), the playbook lowers friction (1.60 → 1.42) and within-group dispersion, using 25% fewer turns and 29% fewer words. Task outcomes do not differ significantly, and raw facilitator ratings are lower.

Critical strengths

The paper reports 894 participants in 298 real-time three-person groups. The logs include turn-level facilitator decisions, surveys, and free-text rationales. Multi-party human data with an AI facilitator at this scale is rare. The protocol is carefully controlled: task order is counterbalanced, and the two studies' configurations are verified identical by diff. The paper releases the full prompts, the playbook, the judge rubric, and an episode replay. It also reports unfavorable results, such as lower raw ratings and more negotiation bookkeeping errors. Two observations are useful beyond facilitation. First, an uncalibrated judge (κ = 0.29) writes its own biases into the playbook as rules. Second, per-citation credit cuts false blame from 27 to 5 of 38 citations.

Critical weaknesses

The reflector, curator, and incremental-edit design comes from Agentic Context Engineering. The paper's distinguishing claim is grounding in real human interactions. Yet the only signal ablation, run with the unaligned judge, puts judge-only feedback within 0.6 points of both signals (p = 0.68). There is no comparison with other prompt optimizers. There is also no "intervene less" variant of the expert prompts, although restraint rules account for 134 of the 274 wins. The 61.4% win rate is scored by the same judge and rubric that provided the training signal. The balanced speech/silence sampling and the mutual-silence ties make abstention low-risk, which the authors acknowledge. No test-set evaluation uses a different judge model or human raters. Study 2 is a non-randomized between-study comparison, and its participants arrived with lower prior attitudes toward AI facilitation. The abstract, introduction, and conclusion outrun several results:

  • Psychological safety (p = .057) is described as improved.
  • All task outcomes are lower. For example, bookkeeping errors in negotiation rose from 2.3% to 8.7%.
  • The productivity deficit survives the prior-adjusted ANCOVA.
Questions for authors
  1. How do the expert prompts perform with one added instruction to intervene less often, matched to the learned playbook's turn rate?
  2. What is the held-out win rate when decisions are judged by a different model family or by human annotators?
  3. With the aligned judge, does adding the human-corpus signal improve over judge-only feedback?
  4. Does the playbook transfer to a task type not seen during learning?
3 comments

Response to Reviewer 2

We thank the reviewer for the thorough and constructive feedback. We address each point below.


Methodology: Comparison with Other Prompt Optimizers

Early in the method design, we tried several prompt optimization approaches, including GEPA [1]. Not every method fits our setting: our prompt is a skill-based playbook, so a playbook-based self-learning framework is the natural choice. Our design follows the spirit of ACE (Zhang et al., 2025) but does not adopt it directly. ACE targets tool-use and reasoning tasks, and its original formulation performed poorly here. We made substantial changes to make it work for facilitation; most importantly, we edit individual skills rather than rewriting the prompt as a whole.


On the LLM Judge

We agree with the reviewer's central concern. The same judge supplied the training signal and scored the 61.4% offline win rate, so the playbook may partly reflect that judge's preferences. To clarify each model's role:

  • gemini-3-flash runs the facilitator policy, the reflector, and the curator.
  • gemini-3.7-flash is the judge only, both during learning and in the offline evaluation.

On Claims in the Abstract, Introduction, and Conclusion

We will revise these claims to match the results. Psychological safety (4.67 vs. 4.62, p = .057) will be reported as directionally higher but not significant. Task outcomes, including negotiation bookkeeping errors, and the productivity difference after the ANCOVA will be reported as limitations. Study 2 will be described as a non-randomized between-study comparison.


Questions for Authors

Q1: How do the expert prompts perform with one added instruction to intervene less often, matched to the learned playbook's turn rate?

We ran this experiment. We added "In general, you should intervene less" to all three expert prompts and re-evaluated on the same test set (87 episodes, 736 decision points) with the same judge. The modified expert now speaks less often than our playbook (20.8% vs. 34.4% of decision points). Our playbook still wins 58.3% of comparisons [95% CI 56.2, 60.2]. Most of those wins come from moments where the playbook intervenes and the expert stays silent.

So the gain does not come from intervening less. It comes from learning when to intervene. This matches Study 1, where participants said the facilitator sometimes failed to step in at critical moments: restraint without judgment is a failure mode of its own. We will include the full expert prompts, which already contain targeted restraint instructions, in the appendix.

Q2: What is the held-out win rate when decisions are judged by a different model family or by human annotators?

Different model families. We re-judged the held-out test set (87 episodes, 736 decision points) with judges from three model families: Gemini, GPT, and Claude. Each got the same rubric and identical inputs. The baseline is the "intervene less" expert from Q1, so the two facilitators do not differ simply in how often they speak. We report the 329 decision points where at least one facilitator speaks, since a point where both stay silent is a trivial tie. (Counting those ties, the Gemini win rate over all 736 points is 58.3%, as in Q1.) Restricting to these points also means abstention cannot inflate the results through mutual-silence ties.

JudgePlaybook win rate [95% CI]
Gemini 3.7 Flash (training judge)68.5% [63.7, 73.4]
GPT-5.665.8% [60.6, 70.7]
Claude Opus 5.562.8% [57.5, 67.9]

The playbook wins under all three judges, and every confidence interval lies above parity. That includes GPT and Claude, which played no part in learning, so the improvement is not specific to the training judge.

Human annotators. Having humans annotate every decision point is too costly, so we validated the judge against humans first and then used it at scale (see "On the LLM Judge" above): κ = 0.70 against the human majority vote, and 66% agreement with an individual annotator vs. 47% between two annotators. Study 2 adds human-reported outcomes that involve no LLM judge.

Q3: With the aligned judge, does adding the human-corpus signal improve over judge-only feedback?

Even though the model is optimized with an LLM judge, the human corpus remains essential. The judge can only compare two candidate responses. The human corpus is where the model learns how real people behave and react in group discussions. In the final playbook, 74% of rules (39/53) originate from or are refined by the human corpus, e.g., matching the group's language, acknowledging humor and gratitude, and validating personal disclosures. These social behaviors are hard to discover from judge feedback alone.

Q4: Does the playbook transfer to a task type not seen during learning?

Yes. We ran a held-out-task experiment. We re-ran our self-learning procedure three times. Each time, one task is held out entirely, and the playbook is learned only on the other two. We then test it on the held-out task against that task's own expert prompt. All other settings are identical to the main experiment. The results show that self-learning can transfer to unseen task.

Unseen test taskLearned onEpisodesDecision pointsWins / Losses / TiesWin rate vs. expert [95% CI]
Consensus-buildingNegotiation + Open-ended1441,068374 / 164 / 53059.8% [57.5, 61.9]
NegotiationConsensus-building + Open-ended1441,364438 / 252 / 67456.8% [55.2, 58.6]
Open-ended discussionConsensus-building + Negotiation1441,298455 / 185 / 65860.4% [58.7, 62.1]

Each task's decision points are split evenly between moments where the expert spoke and moments where it stayed silent:

Unseen test taskExpert spoke: N, win rate [95% CI]Expert silent: N, win rate [95% CI]
Consensus-building534, 63.1% [58.8, 67.1]534, 56.6% [55.0, 58.3]
Negotiation682, 57.3% [54.1, 60.5]682, 56.3% [54.9, 57.6]
Open-ended discussion649, 68.9% [65.8, 72.0]649, 51.9% [50.5, 53.3]

Please let us know if you have any further questions. If our responses have addressed your concerns, we would sincerely appreciate your consideration in raising the score. Thank you!

We have updated the revised version. Please let us know if you have any other questions. If you are satisfied with the changes, we would sincerely appreciate your consideration in raising the score. Thank you!

Reviewer_JreCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the development of general facilitation strategies for multi-party communications. It develops an iterative refinement pipeline, which updates an actionable strategic playbook based on LLM judge feedback, and shows that the learned general strategy outperforms specialized facilitators. The authors conducted large-scale human studies to quantify the effect of their method and proposed extensive metrics capturing participant experience and ratings of the AI facilitator.

Critical strengths
  • The overall concept of generalized facilitation is useful in practice, given that it’s hard to collect data to develop specialized facilitation strategies.
  • The evaluation framework with the metrics of participant experience and human ratings of the facilitator is useful and will guide future work in this area.
Critical weaknesses
  • The generalization claim is not well supported or qualified. Their method learns on the three types of tasks to accumulate the “playbook” and eventually also evaluates on the same types of tasks. What is learned is a playbook that applies to these covered tasks but there is no study of whether the learned strategies are actually general and can benefit unseen tasks.
Questions for authors

Did you notice any cases where the learned generalized playbook still includes content that is only useful/applicable to a specific task type, and how often do these appear?

4 comments

We thank the reviewer for this important point. To test whether the learned strategies generalize to unseen tasks, we ran an additional task-level generalization experiment.

Learn on two tasks, test on an unseen task

We re-ran our self-learning procedure three times. Each time, one task is held out entirely, and the playbook is learned only on the other two. We then test it on the held-out task against that task's own expert prompt. All other settings are identical to the main experiment.

Unseen test taskLearned onEpisodesDecision pointsWins / Losses / TiesWin rate vs. expert [95% CI]
Consensus-buildingNegotiation + Open-ended1441,068374 / 164 / 53059.8% [57.5, 61.9]
NegotiationConsensus-building + Open-ended1441,364438 / 252 / 67456.8% [55.2, 58.6]
Open-ended discussionConsensus-building + Negotiation1441,298455 / 185 / 65860.4% [58.7, 62.1]

Each task's decision points are split evenly between moments where the expert spoke and moments where it stayed silent:

Unseen test taskExpert spoke: N, win rate [95% CI]Expert silent: N, win rate [95% CI]
Consensus-building534, 63.1% [58.8, 67.1]534, 56.6% [55.0, 58.3]
Negotiation682, 57.3% [54.1, 60.5]682, 56.3% [54.9, 57.6]
Open-ended discussion649, 68.9% [65.8, 72.0]649, 51.9% [50.5, 53.3]

Win rate counts ties as half a win (50% = parity). 95% CIs are from an episode-level bootstrap with 2,000 resamples.

Why does it transfer?

We see two main reasons:

  1. It learns to read social signals better. The expert prompts follow fixed scripts and often step in regardless of how the group is doing. The playbook learns to read the room: whether participants are actively building on each other's points, losing engagement, or settling on a shared view. It adapts its intervention to these signals. Such signals reflect the group's dynamics, not the task.
  2. It learns to handle group events that occur in every task. Examples are members leaving, a group reaching agreement, and a value conflict surfacing. Such events are not specific to any one task.

On task-specific content

Only 3 of 53 rules (6%) in the final playbook relate to a specific task (negotiation), and they account for only 4% of rule citations at test time. Even these rules stay at a high level, e.g., "provide a final checklist of agreed-upon details" or "do not provide numerical templates that anchor participants". None of them refers to the task's actual definition, such as the $10 pot, the payoff table or the integer constraint, and no rule in the playbook names a task. We will qualify the generalization claim in the paper accordingly.

We will include the expert prompt in the appendix and task-generalization results in the main text.

Please let us know if you have any further questions. If our responses have addressed your concerns, we would sincerely appreciate your consideration in raising the score. Thank you!

We have updated the revised version. Please let us know if you have any other questions. If you are satisfied with the changes, we would sincerely appreciate your consideration in raising the score. Thank you!

Thanks for your response. The leave one task out evaluation and stats about task-specific content is helpful, but does not fully resolve my concern, so I will maintain my score. The three tasks follow similar formats and settings and the new results are only reported under the training judge. Regarding the task-specific content, the model was prompted to not use task-specific wording in rule creation, so the fact that the rules do not mention task details does not mean they will transfer to other tasks. They may still be general-sounding rules but specialized for these tasks.

Dear Reviewers and Chairs,

As the discussion period concludes, here is a concise summary of the concerns raised by Reviewer JreC (R1) and Reviewer eLoW (R2), the new experiments we conducted to resolve them, and the current status (alongside Reviewer Jw3K (R3), who recommended Clear Acceptance).


Reviewer JreC (Reviewer 1)

  • Issue: Questioned whether the playbook generalizes to unseen tasks rather than overfitting to the three training tasks, and asked how often task-specific rules appear.
  • What We Tried & Demonstrated:
    • Leave-One-Task-Out Experiments: Re-ran the full self-learning pipeline three times, holding out one task completely during training and testing against that unseen task's dedicated expert prompt. The learned playbook consistently beat the specialized experts: 59.8% on unseen Consensus-building, 56.8% on unseen Negotiation, and 60.4% on unseen Open-ended discussion.
    • Playbook Audit & Revision: Audited all 53 rules and found only 3 rules (6%) related to a specific task (negotiation, 4% of test citations), with zero references to task-specific mechanics. Added these results to the main text and qualified our generalization claims.
  • Status: Reviewer JreC explicitly acknowledged in their follow-up that our leave-one-task-out evaluation and task-specificity stats were helpful, though they maintained their score. Notably, the three tasks span distinct interaction structures (unstructured debate, quantitative bargaining, value allocation), and our cross-family experiments for R2 confirm that gains hold across independent judges.

Reviewer eLoW (Reviewer 2)

  • Issues:
    1. Abstention Confound: Whether gains came solely from speaking less rather than learning when to intervene.
    2. Same-Judge Bias & Ties: Whether the 61.4% offline win rate was inflated by evaluating with the training judge or by mutual-silence ties.
    3. Value of Human Corpus: Whether human logs added signal beyond judge-only feedback.
    4. Claims & Study 2 Limitations: Overstated claims regarding psychological safety (p=.057p = .057), negotiation bookkeeping errors, ANCOVA productivity controls, and Study 2's non-randomized design.
  • What We Tried & Demonstrated:
    • Matched-Turn-Rate Baseline (Q1): Added "In general, you should intervene less" to all three expert prompts. Even when the modified expert intervened less often than our playbook (20.8% vs. 34.4%), our playbook still won 58.3% [56.2, 60.2]—proving the gain comes from when and how to intervene, not mere silence.
    • Cross-Model-Family Judge Evaluation (Q2): Re-evaluated all 329 active decision points (excluding mutual-silence ties) against the "intervene-less" expert across three distinct model families: 68.5% under Gemini 3.7 Flash (training judge), 65.8% under GPT-5.6 (unseen), and 62.8% under Claude Opus 5.5 (unseen).
    • Human-Corpus Attribution (Q3): Showed that 74% (39/53) of playbook rules originated from or were refined by real human conversation logs (e.g., matching group tone, validating disclosures, recognizing humor), behaviors pairwise judge comparisons alone cannot surface.
    • Manuscript Revisions: Reframed psychological safety as directionally higher (p=.057p = .057), explicitly noted Study 2’s non-randomized between-study design, and reported negotiation bookkeeping errors and ANCOVA productivity differences as limitations.

We have addressed every empirical and textual concern raised during review through new leave-one-task-out studies, restraint-matched baselines, cross-family judge evaluations (GPT-5.6 and Claude Opus 5.5), and thorough manuscript revisions. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.