acceptodds
Under review as a conference paper at ICLR 2027

Beyond Mode Collapse: Generating Diverse Synthetic Expert Conversations via Generative Flow Networks

Abstract

High quality synthetic data is central to post-training LLMs for adaptive-AI applications that represent the diverse expert strategies and decisions in conversations. Prompting LLMs directly or conditioning them on end-use scenarios yields low-diversity data that collapses onto dominant modes. We propose a method to generate diverse high-quality synthetic data using Generative Flow Networks (GFlowNets). We show that training GFlowNets to generate latent conversation structure using a Gaussian-mixture density over key interaction features (e.g., confusion-episode dynamics, scaffolding–directive balance) enables sampling expert strategies in proportion to their prevalence in the training data. Across two structurally distinct domains, tutoring and emotional-support dialogues, our GFlow-based synthetic data generation approach offers a better balance of fidelity, mode-coverage and authenticity than reinforcement-learning and end-to-end LLM baselines, without copying training data. Evaluated on three downstream outcome prediction tasks, classifiers trained on synthetic GFlowNet-generated conversations provide a stronger training signal than competitive synthesis baselines.

Primary area:
generative models
Keywords:
gflow networks, synthetic data quality, synthetic data generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_A37pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a GFlowNet-based method for generating diverse synthetic expert conversations. It first uses a GFlowNet to generate discourse-move sequences, then turn them into natural dialogues via LLMs. Experiments on math tutoring and emotional support show better structural fidelity and diversity than RL-based and LLM-only generation baselines.

Critical strengths
  • The paper addresses an important limitation of synthetic conversation generation: generated dialogues fail to preserve deeper structure underlying diverse, adaptive decisions of conversation.
  • Comprehensive baseline comparisons. The paper compares GFlowNets with an architecture- and reward-matched REINFORCE baseline, and several LLM-only prompting strategies. This helps distinguish the contributions of structural generation and the sampling objective.
Critical weaknesses
  • Limited evaluation of conversation quality. The reward and most evaluation metrics rely on the same hand-designed conversation features. Improvements on these metrics are therefore expected, but can't indicate that the generated conversations are more coherent, realistic, or effective. Adding independent semantic or human evaluations would make the results more convincing.
  • Insufficient details on LLM-only baseline. The paper describes three prompting strategies but does not provide the full prompts or generation settings. The differences between B2 and B3 are also not entirely clear.
  • Limited evidence of downstream utility. The downstream evaluation is restricted to three classification tasks rather than actual LLM post-training. The classifiers also use discourse-move labels as inputs, making it unclear the real effectiveness of the generated content. Add evaluations on downstream model training can better support the claimed benefits.
Questions for authors
  1. The paper demonstrates improvements in structural fidelity and coverage, but it remains unclear whether training LLMs on these synthetic conversations leads to better abilities. Could the authors evaluate the generated data in LLM fine-tuning tasks to demonstrate its practical value? I have some concerns about the quality of the generated conversations.
  2. A straightforward alternative is to provide an initial role and discourse move (e.g., "student" and "question"), then let an LLM generate the conversation turn by turn, conditioning each new response on the preceding dialogue. This would allow the conversation to develop based on its actual content rather than a predefined structural sequence. Could the authors compare against this approach in terms of conversation quality and diversity?
  3. The LLM-only baselines (B1–B3) use few-shot examples, scenario conditioning, or mode-proportional sampling, but do not include instructions targeting the structural features used in evaluation. In contrast, the proposed method directly optimizes rewards based on these features. Could the authors provide a stronger LLM-only baseline with comparable structural guidance to better isolate the benefits of GFlowNet?
3 comments
AuthorsOpenReview ↗

Clarifications about evaluations and baselines

Thank you for your feedback! We summarize the concerns and our responses below.

Reward and evaluation metrics share the same hand-designed feature space

Reply: Thank you for this insightful comment! Our downstream fine-tuning evaluations to predict conversation outcome from its text already provide a strong feature-space independent signal of benefits of our method for dataset diversity.

To further address this concern, we will run a blind LLM-as-a-judge pairwise evaluation across all conditions (LLM-only, RL+LLM, GFlow+LLM), focusing on the tutor/counselor's effectiveness and the conversation's coherence. For realism, we will run pairwise evaluations between each synthetic condition and the held-out real conversations. These evaluations will provide additional insights independent of hand-designed feature-space evaluations.

Prompting details and generation settings on the LLM-only baseline and clarify B2 vs B3 difference.

Reply: We will add the full prompt details in the appendix, as well as the following details about generation settings: Our LLM condition prompts state the task of generating a complete conversation, describe the allowed discourse moves per role, and require each turn to self-label with one of them, followed by formatting instructions and 3 few-shot examples sampled from the training data. B2 differs from B3 in how it chooses the few-shot demonstrations: B2 draws them uniformly at random from the training corpus; B3 draws them with probability proportional to the real GMM-mode mixture (the same mixture the gmm_density reward is fit on), so rarer modes still get a chance to appear as examples instead of being drowned out by the majority mode.

All LLM generation conditions use k=3k{=}3 few-shot examples, temperature 0.70.7, the same structured-output JSON schema as our realization pipeline, and up to 2 retries with JSON-escape repair on malformed output.

A baseline that provides an initial role and discourse move, then lets an LLM generate the conversation turn by turn.

Reply: Both are good suggestions that will help us isolate the effect of GFlow on generating diverse conversation structures, and we'll add them as additional baselines.

Downstream evaluation should use real LLM fine-tuning, not a classifier probe

Reply: Thank you for the suggestion! We actually ran a downstream LLM fine-tuning evaluation and have shared the results in the general comment, which we will add to the existing Table 4. In sum, GFlow+LLM shows strong improvements over existing baselines in 5 of 6 scenarios, confirming the robustness of our results.

The additional fine-tuning results are encouraging and resolved my most concerns. However, the key baseline comparisons and independent evaluation of conversation quality are still missing. I will reconsider my rating later.

We are adding preliminary results for the requested evaluations and baselines as promised in our original response. Given the cost and time constraints of the rebuttal period, we ran it on a tutoring subset of 200 conversations and compared it with an existing equivalent random sample wherever applicable. We will compute these results on the full set and report them in the camera-ready version.

Additional LLM baselines

We added the two requested LLM-only baselines: 1) B4 - turn-by-turn generation from a sampled opening move and conditioned on the prior context (move and conversation text), and 2) B5 - B2's scenario and instructions augmented with conversation-level targets for the structural features for better parity with the structural GFlow and RL conditions.

B5 was the best LLM baseline. On MathMentorDB (n=200, Gemini), B5 narrows the gap with GFlow on pMSE (83.4 for B5 vs. 80.5 for GFlow+LLM) but still trails GFlow+LLM on fidelity (IPα 0.90 for GFlow+LLM vs. 0.46 for B5) and coverage (IRβ 0.55 for GFlow+LLM vs. 0.23 for B5). B5's authenticity also falls below the real-data reference (0.55 vs. 0.70), whereas GFlow+LLM stays above real-data authenticity (0.86). The results suggest that adding move-level and structural guidance improves the LLM’s ability to generate higher-quality conversations in its dominant mode (better pMSE), but it still struggles to cover the diverse expert strategies as well as GFlow trained on expert conversations.

Independent feature-agnostic evaluation of converation quality

We ran LLM-as-judge evaluation for effectiveness, coherence, and realism on a tutoring subset: a pairwise judge on 200 conversations per condition (each pair judged in both orders). The judge sees only conversation text. Below are the preliminary results.

Effectiveness. In a pairwise comparison between the conditions on effectiveness of the tutor in resolving the issue, GFlow is judged most effective amongst the synthetic conditions, followed by LLM-only (B5) and RL. GFlow conversations are judged more effective in 60% of cases compared with LLM-only, but they lose to the real conversations in 55% of cases.

Coherence. LLM-only is judged most coherent, followed by GFlow then RL. LLM-only is preferred slightly over GFlow (in 56% of cases) and even over real conversations (in 60% of cases). A possible explanation is that real tutoring chats come from a live Discord help channel. They often refer to context the transcript lacks (images, earlier messages), contain terse fragments and interleaved off-topic chatter, include tutor mistakes, and end unresolved. An LLM writing a whole conversation in one pass produces a self-contained, polished exchange with none of these imperfections. So high coherence partly reflects distance from real data, and the planned conditions' lower coherence partly reflects their structural fidelity to it.

Realism. In a pairwise comparison of each synthetic condition against the real, for realism, the judge always preferred the real conversation, but was least confident between GFlow+LLM and real. This suggests that real conversations still have markers that are easily distinguishable from any synthetic condition.

We will report the full evaluations, baselines and a deeper discussion of the results across both domains in the camera-ready version.

Reviewer_nLoAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses mode collapse and insufficient coverage in synthetic dialogue with a two-stage pipeline: GFlowNet plans dialogue structure and an LLM realizes it into natural language, using reward-proportional sampling to cover diverse expert strategies. It contributes a structural planning method, shows proportional sampling avoids RL's mode collapse and LLM-only's low coverage, and demonstrates across two domains that GFlow+LLM better balances fidelity, coverage, and authenticity than baselines without copying training data, with downstream TSTR showing stronger training signals.

Critical strengths

1、The paper addresses mode collapse and insufficient coverage in synthetic dialogue, and GFlowNet's core property of "sampling proportional to reward" rather than "reward maximization" directly matches the need to "cover diverse expert strategies" at the mechanism level, offering a new path distinct from end-to-end LLM generation and reward-maximizing RL.

2、The paper evaluates from multiple angles—structural fidelity, pMSE, fidelity/coverage decomposition, authenticity diagnostics, mode coverage, and downstream TSTR—thereby comprehensively examining the method's performance in terms of fidelity, coverage, authenticity, and downstream utility.

Critical weaknesses

1、Key hyperparameters not reported: The values and tuning rationale for ϵ,α, and β in Eq. (3) are not given, directly affecting the reward shape and reproducibility.

2、Inconsistent sources of action labels: GFlow/RL use planner-committed skeleton labels, while LLM-only uses self-reported labels; the fairness of structural metric comparisons needs clarification.

3、Meaning of the structural upper bound needs explanation: The authors should explain how it is constructed and make explicit that "the gap between GFlow and the upper bound mainly stems from skeleton planning rather than LLM realization."

4、Insufficient downstream evaluation coverage: TSTR is conducted only in the tutoring domain on three binary tasks; ESConv has no downstream experiments.

5、ESConv results are not robust enough: Under GPT realization, GFlow's IP_α is lower than LLM-only and its outlier rate is relatively high, which tensions with the core claim and requires discussion of the reasons.

6、w(t) in Eq. (3) is undefined: Neither the main text nor the appendix explains its meaning or values; it needs clarification or removal.

Questions for authors

See the Weaknesses above.

1 comment
AuthorsOpenReview ↗

Clarification about experiments and measurements

Thank you for your feedback! We summarize the concerns and our responses below.

Clarification about Eq. 3 hyperparameters (ϵ\epsilon, α\alpha, β\beta) and wtw_t

Reply: Thank you for catching that. We use the same values for every headline run in both domains: β=1.0\beta{=}1.0, α=0.10\alpha{=}0.10, ε=0.01\varepsilon{=}0.01. We will add these values to the manuscript.

We will add the missing definition of wtw_t: its a multiplier that nudges the per-step reward based on conversation state (e.g., more scaffolding reward when the student is confused). We experimented but didn't find it useful for our final evals, so it defaults to 1.

We will also add the training hyperparameters (Adam, lr$=0.01$, weight decay 10−410^{-4}, batch size 32, grad-norm clip 1.01.0, early stopping patience 10, up to 50 epochs).

Structural metrics need to use consistent move labeling across LLM, RL and GFlow conditions

Reply: We appreciate the reviewer’s suggestion about making our metric evaluations fairer. We will re-annotate all conditions' realized text with one external LLM annotator, blind to condition, and recompute all move-derived metrics.

To test the expected changes to results, in a pilot experiment, we randomly sampled 25 tutoring conversations per condition, re-annotated every turn with Claude Opus 5 blind to the condition and existing labels. Recomputing move-mix, transition, and ϕ(x)\phi(x) distances to real data did not change the condition rankings and shifted GFlow's gap to each baseline by at most 0.02. This suggests that using a held-out annotator for move labels would not significantly shift our metric evaluations.

Clarification on construction and interpretation of Structural upper bound

Reply: We have clarified the construction of the structural upper bound in the general response.

Insufficient downstream evaluation coverage (ESConv missing from TSTR)

Reply: We have provided a clarification for not covering ESConv in downstream evaluations in the general response.

For ESConv, GFlow's IP_α is lower than LLM-only, and its outlier rate is relatively high and should be discussed.

Reply: We attribute GFlow's lower IPα\mathrm{IP}_\alpha and higher outlier rate on ESConv to timing. GFlow matches ESConv's strategy mix, which improves coverage (IRβ\mathrm{IR}_\beta), but its synthetic counselors sometimes advise before building rapport. Real supporters almost always build rapport first, so early advice stands out as atypical even when the overall mix of strategies is right. Tutoring has no comparably fixed order, so the same timing slip goes unnoticed there. We will add the implications of such behaviors for improving future work.

Reviewer_hbXkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper uses GFlowNet to generate latent conversation structure over key interaction features, enabling sampling expert strategies in proportion to their prevalence in the training data.

Critical strengths
  1. High-quality synthetic data is an important source for the post-training of LLM, this paper proposes an eligible framework that utilises a GFlow network to generate a bunch of synthetic data.

  2. The visualisation figures in Figure 2 look cool, directly reflecting the generated synthetic data.

Critical weaknesses
  1. Idea is too straight-forward and idea is lack of novelty. Generally, this paper is using GFlowNet to generating synthetic data, not showcasing any novel ideas or findings in this track.

  2. Lack of a motivation case to help reviewers to understand how this framework works to generate synthetic data, and why these data have high-quality over those baseline models? Why these can can help with the post-training of LLMs? Why previoius methods cannot help?

  3. Some details in the experiments are not clear enough. For example, in Table 2, the structural upper bound is not clearly illustrated, which confused reviewers to a large extend.

Questions for authors

See above.

3 comments
AuthorsOpenReview ↗

Clarifications about novelty and contributions

Thank you for your feedback! We summarize the concerns and our responses below.

Novelty

Reply: We have provided clarifications about the novelty of our work in the general response.

Insufficient motivation: choice of data, advantages over baselines, and limitations of prior methods

Reply: We study tutoring and counseling, where effective support depends on understanding the help-seeker's state and expert’s choice of the strategy. In real MathMentorDB sessions, students often stay confused for many turns, and tutors must balance direct answers against hints; a tutoring agent needs such examples in its training data. Prompting an LLM, even with scenario- or mode-proportional conditioning, instead yields smoothly scaffolded sessions that look plausible but fall outside the real distribution (IP 0.20; 79--88% of samples lie farther from any real conversation than held-out real data does). RL concentrates on the highest-scoring patterns and covers little of the real distribution (IR 0.15). We show that our distribution-proportional sampling instead improves downstream SFT.

Clarification on impact of mode-proportional sampling from GFlow on post-training

Reply: We refer to the general discussion point where our post-training experiment results show the benefits of diverse synthetic data generation using GFlowNets over baselines for downstream prediction tasks.

Clarification on construction and interpretation of Structural upper bound

Reply: We have clarified the construction of the structural upper bound in the general response.

Thanks for the authors' responses. I would encourage you to consider these suggestions and your responses to reshape your motivation and presentation.

Thank you for your reply! We will revise the motivation and presentation along the arguments we presented in the rebuttal above, update the post-training results to more strongly ground the utility of our approach and additionally reference the papers that we have discussed in the responses. If you have remaining soundness concerns, or specific prior work you see as overlapping with ours, we would appreciate knowing so we can address them in the manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response about contributions and downstream utility

General response

We thank the reviewers for their constructive comments, which have improved our work. We summarize the key contributions and common responses below, and address the remaining feedback in individual reviewer responses.

  1. Key contributions and novelty

Our work is the first to use the reward-proportional sampling capabilities of GFlowNets to successfully address the mode-collapse problem that occurs with LLM-generated synthetic dialogue datasets that attempt to capture a rich, realistic diversity of expert conversation strategies.

Prior GFlowNet methods for LLMs fine-tune the LLM itself in order to improve reasoning accuracy [1, 2, 3] or preference alignment [0], by sampling diverse specific responses to a single input prompt. In contrast, our objective focuses on diversity across higher-level structure and strategies for the entire conversation and dataset, not individual responses. For example, prior work used flows defined over tokens [0, 1, 3] or over model reasoning steps toward a verifiable goal [2]; but none targeted the distributional properties of the entire dataset.

In our work, we seek dataset diversity at the level of expert strategies by learning from a small dataset of expert behaviors for downstream LLM training, not diversifying responses in LLM training policy. No single strategy is correct, so instead of a verifiable objective reward, the reward design needs to represent diverse expert strategies in a conversation. We achieve dataset diversity by optimizing flows over a smaller factored pedagogical move space and using a mode-specific Gaussian reward over cheaply computable expert features ϕ(x)\phi(x) derived from the real expert conversations. Optimizing flows over a smaller skeleton space allows us to train roughly 1200 trainable parameters, five to eight orders of magnitude fewer than the LLM policies fine-tuned by prior GFlowNet methods [0, 1, 2, 3, 4], which train in a few minutes on a CPU versus the expensive GPU training required for prior approaches.

[0] - GDPO: Learning to Directly Align Language Models with Diversity Using GFlowNets (Kwon et al., EMNLP 2024)

[1] - Zhu, Xuekai, et al. "Flowrl: Matching reward distributions for llm reasoning." International Conference on Learning Representations. Vol. 2026. 2026.

[2] - Yu, Fangxu, et al. "Flow of reasoning: Training llms for divergent reasoning with minimal examples." arXiv preprint arXiv:2406.05673 (2024).

[3] - Liu, Xiaodong, et al. "GFlowRL: Scaling Distribution-Matching RL to Large Language Models." arXiv preprint arXiv:2607.13394 (2026).

[4] - Hu, Edward, et al. "Amortizing intractable inference in large language models." International Conference on Learning Representations. Vol. 2024. 2024.

  1. Utility of diverse data generation for post-training.

We added a downstream experiment, fine-tuning an LLM on our synthetic data, to evaluate whether the diversity of our data improves training. We fine-tuned Qwen2.5-0.5B, LoRA, and the MiniLM encoder (22M) (originally used in the paper), on each condition’s synthetic conversations (without any move labels) for tutoring.

GFlow+LLM trains the best synthetic-data model in 5 of 6 cases. The exception is resolved with Qwen, where LLM-only leads slightly (0.517 vs. 0.488). The table below summarizes the results. We will update the existing table 4 in the paper with these results.

Tutoring, macro-F1 on real held-outReal-trainedLLM-onlyRL+LLMGFlow+LLMMajority
Qwen2.5-0.5B (LoRA)
resolved0.8880.5170.2910.4880.371
has_breakthrough0.8260.4290.4130.6290.413
has_off_topic0.7570.4840.4840.5310.484
MiniLM (full fine-tune)
resolved0.7970.4920.2910.5940.371
has_breakthrough0.7330.5040.4130.6900.413
has_off_topic0.7460.4840.4840.5270.484
  1. Lack of ESConv in downstream evaluations

ESConv outcome labels come from post-session surveys, not transcripts, so even models trained on real ESConv conversations predict them near chance (AUROC 0.50–0.55, vs. 0.93–0.96 for tutoring), and only 260 of 1,040 conversations are labeled. Without informative ground-truth labels, a meaningful downstream test on ESConv was therefore not possible.

  1. Clarification on construction and interpretation of Structural upper bound We will clarify in the manuscript that the structural upper bound condition resamples real skeletons verbatim from the training data and realizes them with the LLM. Since its skeletons are real by construction, its gap to the other conditions isolates skeleton-planning quality from realization quality.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.