- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a GFlowNet-based method for generating diverse synthetic expert conversations. It first uses a GFlowNet to generate discourse-move sequences, then turn them into natural dialogues via LLMs. Experiments on math tutoring and emotional support show better structural fidelity and diversity than RL-based and LLM-only generation baselines.
Critical strengths
- The paper addresses an important limitation of synthetic conversation generation: generated dialogues fail to preserve deeper structure underlying diverse, adaptive decisions of conversation.
- Comprehensive baseline comparisons. The paper compares GFlowNets with an architecture- and reward-matched REINFORCE baseline, and several LLM-only prompting strategies. This helps distinguish the contributions of structural generation and the sampling objective.
Critical weaknesses
- Limited evaluation of conversation quality. The reward and most evaluation metrics rely on the same hand-designed conversation features. Improvements on these metrics are therefore expected, but can't indicate that the generated conversations are more coherent, realistic, or effective. Adding independent semantic or human evaluations would make the results more convincing.
- Insufficient details on LLM-only baseline. The paper describes three prompting strategies but does not provide the full prompts or generation settings. The differences between B2 and B3 are also not entirely clear.
- Limited evidence of downstream utility. The downstream evaluation is restricted to three classification tasks rather than actual LLM post-training. The classifiers also use discourse-move labels as inputs, making it unclear the real effectiveness of the generated content. Add evaluations on downstream model training can better support the claimed benefits.
Questions for authors
- The paper demonstrates improvements in structural fidelity and coverage, but it remains unclear whether training LLMs on these synthetic conversations leads to better abilities. Could the authors evaluate the generated data in LLM fine-tuning tasks to demonstrate its practical value? I have some concerns about the quality of the generated conversations.
- A straightforward alternative is to provide an initial role and discourse move (e.g., "student" and "question"), then let an LLM generate the conversation turn by turn, conditioning each new response on the preceding dialogue. This would allow the conversation to develop based on its actual content rather than a predefined structural sequence. Could the authors compare against this approach in terms of conversation quality and diversity?
- The LLM-only baselines (B1–B3) use few-shot examples, scenario conditioning, or mode-proportional sampling, but do not include instructions targeting the structural features used in evaluation. In contrast, the proposed method directly optimizes rewards based on these features. Could the authors provide a stronger LLM-only baseline with comparable structural guidance to better isolate the benefits of GFlowNet?
3 comments
The additional fine-tuning results are encouraging and resolved my most concerns. However, the key baseline comparisons and independent evaluation of conversation quality are still missing. I will reconsider my rating later.
We are adding preliminary results for the requested evaluations and baselines as promised in our original response. Given the cost and time constraints of the rebuttal period, we ran it on a tutoring subset of 200 conversations and compared it with an existing equivalent random sample wherever applicable. We will compute these results on the full set and report them in the camera-ready version.
Additional LLM baselines
We added the two requested LLM-only baselines: 1) B4 - turn-by-turn generation from a sampled opening move and conditioned on the prior context (move and conversation text), and 2) B5 - B2's scenario and instructions augmented with conversation-level targets for the structural features for better parity with the structural GFlow and RL conditions.
B5 was the best LLM baseline. On MathMentorDB (n=200, Gemini), B5 narrows the gap with GFlow on pMSE (83.4 for B5 vs. 80.5 for GFlow+LLM) but still trails GFlow+LLM on fidelity (IPα 0.90 for GFlow+LLM vs. 0.46 for B5) and coverage (IRβ 0.55 for GFlow+LLM vs. 0.23 for B5). B5's authenticity also falls below the real-data reference (0.55 vs. 0.70), whereas GFlow+LLM stays above real-data authenticity (0.86). The results suggest that adding move-level and structural guidance improves the LLM’s ability to generate higher-quality conversations in its dominant mode (better pMSE), but it still struggles to cover the diverse expert strategies as well as GFlow trained on expert conversations.
Independent feature-agnostic evaluation of converation quality
We ran LLM-as-judge evaluation for effectiveness, coherence, and realism on a tutoring subset: a pairwise judge on 200 conversations per condition (each pair judged in both orders). The judge sees only conversation text. Below are the preliminary results.
Effectiveness. In a pairwise comparison between the conditions on effectiveness of the tutor in resolving the issue, GFlow is judged most effective amongst the synthetic conditions, followed by LLM-only (B5) and RL. GFlow conversations are judged more effective in 60% of cases compared with LLM-only, but they lose to the real conversations in 55% of cases.
Coherence. LLM-only is judged most coherent, followed by GFlow then RL. LLM-only is preferred slightly over GFlow (in 56% of cases) and even over real conversations (in 60% of cases). A possible explanation is that real tutoring chats come from a live Discord help channel. They often refer to context the transcript lacks (images, earlier messages), contain terse fragments and interleaved off-topic chatter, include tutor mistakes, and end unresolved. An LLM writing a whole conversation in one pass produces a self-contained, polished exchange with none of these imperfections. So high coherence partly reflects distance from real data, and the planned conditions' lower coherence partly reflects their structural fidelity to it.
Realism. In a pairwise comparison of each synthetic condition against the real, for realism, the judge always preferred the real conversation, but was least confident between GFlow+LLM and real. This suggests that real conversations still have markers that are easily distinguishable from any synthetic condition.
We will report the full evaluations, baselines and a deeper discussion of the results across both domains in the camera-ready version.
Clarifications about evaluations and baselines
Thank you for your feedback! We summarize the concerns and our responses below.
Reward and evaluation metrics share the same hand-designed feature space
Reply: Thank you for this insightful comment! Our downstream fine-tuning evaluations to predict conversation outcome from its text already provide a strong feature-space independent signal of benefits of our method for dataset diversity.
To further address this concern, we will run a blind LLM-as-a-judge pairwise evaluation across all conditions (LLM-only, RL+LLM, GFlow+LLM), focusing on the tutor/counselor's effectiveness and the conversation's coherence. For realism, we will run pairwise evaluations between each synthetic condition and the held-out real conversations. These evaluations will provide additional insights independent of hand-designed feature-space evaluations.
Prompting details and generation settings on the LLM-only baseline and clarify B2 vs B3 difference.
Reply: We will add the full prompt details in the appendix, as well as the following details about generation settings: Our LLM condition prompts state the task of generating a complete conversation, describe the allowed discourse moves per role, and require each turn to self-label with one of them, followed by formatting instructions and 3 few-shot examples sampled from the training data. B2 differs from B3 in how it chooses the few-shot demonstrations: B2 draws them uniformly at random from the training corpus; B3 draws them with probability proportional to the real GMM-mode mixture (the same mixture the gmm_density reward is fit on), so rarer modes still get a chance to appear as examples instead of being drowned out by the majority mode.
All LLM generation conditions use k=3 few-shot examples, temperature 0.7, the same structured-output JSON schema as our realization pipeline, and up to 2 retries with JSON-escape repair on malformed output.
A baseline that provides an initial role and discourse move, then lets an LLM generate the conversation turn by turn.
Reply: Both are good suggestions that will help us isolate the effect of GFlow on generating diverse conversation structures, and we'll add them as additional baselines.
Downstream evaluation should use real LLM fine-tuning, not a classifier probe
Reply: Thank you for the suggestion! We actually ran a downstream LLM fine-tuning evaluation and have shared the results in the general comment, which we will add to the existing Table 4. In sum, GFlow+LLM shows strong improvements over existing baselines in 5 of 6 scenarios, confirming the robustness of our results.