acceptodds
Under review as a conference paper at ICLR 2027

ThreadBench: Benchmarking Realism in Agentic Simulation of Real-world Discussions

Abstract

LLM agents are increasingly used to simulate real-world interactions, but it remains unclear whether simulated behaviors preserve the content patterns and interaction dynamics of real human behaviors. Existing evaluations often focus on individual comments or task performance, rather than whether simulated behaviors match human behavior at thread level. In this paper, we use Reddit discussions as a concrete testbed for real-world social simulation. Reddit provides public, topic-grounded, multi-party interactions with diverse opinions, experiences, and interaction patterns, making it a useful setting for evaluating whether LLM agents reproduce the distributional and structural properties of real online discussions. We introduce ThreadBench, a benchmark for Reddit discussion simulation built from 9,817 real Reddit threads spanning 8 diverse domains and 116 subdomains. ThreadBench uses statistical analyses to compare generated and real discussions across 12 metrics grouped in 4 families: uniformity, expression, tone, and interaction. We further develop Thread-Prompt-based Discussion Generation (ThreadPDG), a Planner–Writer framework that learns how a real discussion grows. The Planner produces thread- and comment-level plans from a real seed discussion, and the Writer generates comments from these plans. For each seed, we generate multiple candidate threads and combine their best branches using thread-level statistics. Across all domains and multiple LLMs, ThreadPDG achieves ThreadBench pass rates of 58–100%, compared with 0–17% for the baselines. Human evaluation with 436 annotators further shows that ThreadPDG is not distinguished from real discussions in 78.7% of pairs, outperforming existing simulation methods.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Agentic Simulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fB7vOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ThreadBench, a benchmark for evaluating how realistically LLM-based agents can simulate real-world Reddit discussions at the thread level. Unlike prior work that focuses on single-comment quality, role-play, or task success, ThreadBench compares generated discussion threads against 9,817 real Reddit threads spanning 8 domains and 116 subdomains. It defines 12 thread-level metrics across four families: uniformity, expression, tone, and interaction, and uses nonparametric tests (Mann–Whitney U and Kolmogorov–Smirnov) to decide whether generated discussions are statistically indistinguishable from real ones.

The authors further propose ThreadPDG, a Planner–Writer discussion generation framework that learns how real discussions grow, generates multiple candidate threads, and composes the best branches using thread-level statistics. Experiments across GPT-5.4-mini, GPT-4o-mini, and DeepSeek-V4.1-Flash show that ThreadPDG achieves 58–100% metric pass rates, while baselines OASIS and SynthPAI remain at 0–17%. A human Turing-test study with 436 annotators finds ThreadPDG indistinguishable from real discussions in 78.7% of pairs.

Critical strengths
  1. Thread-level evaluation is the right unit. Most prior benchmarks stop at comment-level fidelity; this paper evaluates structure + semantics + tone jointly.
  2. Strong empirical contribution. The paper presents a large-scale benchmark comprising 9,817 threads across 116 subdomains, evaluated along 12 metrics. The scale and breadth of this resource represent a substantial contribution to the community.
  3. Statistical rigor above average. Use of MWU + KS, real-vs-real calibration, effect sizes, and a consistency theorem is commendable.
Critical weaknesses
  1. Presentation could be improved. The paper is rich in implementation and metric details, but the high-level design rationale receives comparatively less attention. A clearer narrative that connects the design choices to the underlying research questions would make the contributions easier to appreciate.
  2. The contribution of the branch selection mechanism is not fully isolated. Since the selection step explicitly optimizes toward the target distribution, it is unclear how much of the reported gain comes from the Planner/Writer genuinely learning discussion dynamics, versus from the selection step itself performing metric fitting. An ablation that replaces branch selection with random sampling would help clarify this.
Questions for authors
  1. How strictly are seed pool / reference pool / test pool disjoint? Are root posts ever shared across splits?
  2. If ThreadPDG’s branch selection is disabled and branches are sampled randomly, how much of the gain remains?
  3. Why does gpt-4o-mini benefit much less than GPT-5.4-mini / DeepSeek? Is this planner competence or writer capacity?
5 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We are glad that the reviewer considers thread-level evaluation the appropriate unit and recognizes the benchmark’s scale and statistical rigor, including the real-vs-real calibration, effect sizes, and consistency theorem. We appreciate the suggestions for making the design rationale clearer and separating the contributions of generation and selection. We respond to these points and questions individually below.

1. Research questions and design rationale

We appreciate the reviewer’s suggestion. To make the motivation and design of our experiments clearer, we organize the Experiments section around three research questions. We have added the following paragraph at the beginning of the Section 5 and throughout the paper.

RQ1. Evaluating the realism of generated discussions. We examine how closely existing simulators and ThreadPDG match real Reddit discussions across eight domains. We use ThreadBench to assess different aspects of discussion behavior and complement the statistical comparisons with a human Turing test (Section 5.2).

RQ2. Understanding how ThreadPDG improves discussion realism. We examine the contributions of draft budget and branch selection by varying the number of drafts and comparing metric-based selection with random selection at the same draft budget (Section 5.3).

RQ3. Understanding differences in discussion behavior across methods and LLMs. Beyond overall pass rates, we analyze semantic diversity, comment length, tone, and reply structure to identify which characteristics of real discussions different methods capture well and which remain challenging (Section 5.4).

2. Contribution of branch selection

We truly thank the reviewer for the suggestion. We conducted an additional experiment replacing metric-based branch selection with random selection while keeping all other setups the same. Each random selection is repeated 50 times, and the tables below report the average metric pass rates across 8 domains.

The results show that branch selection contributes substantially to the performance improvement. At K=3, DeepSeek achieves 81.2% with metric-based selection compared with 49.6% under random selection, while GPT-5.4-mini achieves 81.2% compared with 46.5%. Random selection remains close to K=1 for both models (47.9% and 45.8%), suggesting that increasing the draft budget alone provides limited improvement. For GPT-4o-mini, selection provides a smaller gain (28.1% vs. 21.4%) and does not improve performance in every domain. These results show that branch selection accounts for much of the additional improvement at larger draft budgets, while the K=1 results reflect the generation performance without selection.

DeepSeek-V4.1-Flash: average pass rate (%) across domains

DomainK=1K=2 randomK=2 selectedK=3 randomK=3 selected
Camera33.334.775.037.891.7
Celebrity66.774.091.772.0100.0
Cellphone50.052.275.051.383.3
Game58.356.583.365.583.3
Health33.335.758.335.858.3
Movies50.052.283.351.383.3
News25.025.050.024.558.3
TV Series66.759.775.058.791.7
Mean47.948.774.049.681.2

GPT-5.4-mini: average pass rate (%) across domains.

DomainK=1K=2 randomK=2 selectedK=3 randomK=3 selected
Camera41.745.875.048.283.3
Celebrity33.326.775.027.891.7
Cellphone50.049.866.747.883.3
Game33.339.066.740.566.7
Health50.048.866.754.366.7
Movies50.050.066.749.291.7
News58.351.866.753.375.0
TV Series50.050.875.050.791.7
Mean45.845.469.846.581.2

GPT-4o-mini: average pass rate (%) across domains.

DomainK=1K=2 randomK=2 selectedK=3 randomK=3 selected
Camera16.719.316.719.516.7
Celebrity25.024.725.024.225.0
Cellphone33.322.033.322.241.7
Game16.720.316.718.725.0
Health16.720.025.020.525.0
Movies16.711.016.715.025.0
News33.333.233.332.833.3
TV Series25.019.725.018.333.3
Mean22.921.324.021.428.1
AuthorsOpenReview ↗

Rebuttal by Authors

3. Separation of seed, reference, and test pools

We appreciate the reviewer’s question. As described in Appendix E.1, the seed, reference, and test pools are separated by source-thread ID within each domain. No source thread, including its root post and comments, is shared across the three pools. The seed pool is used for generation, the reference pool provides statistics for branch selection, and the test pool is used only for final evaluation.

4. Planner and Writer diversity across backbones

We thank the reviewer for this insightful question. We conduct an additional analysis of Planner and Writer outputs at K=1, before branch selection, to understand why GPT-4o-mini shows smaller improvements. The experiments below examine the diversity of Planner and Writer outputs at K=1, before branch selection.

Planner. We measure plan diversity using semantic similarity and four-gram reuse. For each planned comment, we combine its planned response, topic, details, and context into a single text and use all-mpnet-base-v2 to measure similarity between plans. Plans with cosine similarity ≥0.70 to another plan in the same thread are considered near-duplicates. We also measure how often plans within the same thread share four-word sequences.

The results show that GPT-4o-mini generates more repetitive plans, with a mean near-duplicate rate of 58%, compared with 15% for DeepSeek and 24% for GPT-5.4-mini. Its four-gram reuse rate is also higher (10.1% vs. 2.3-2.5%).

Planner repetition by domain (%).

DomainNear-dup. DSNear-dup. 5.4-miniNear-dup. 4o-mini4-gram reuse DS4-gram reuse 5.4-mini4-gram reuse 4o-mini
Camera9%20%54%2.2%2.4%10.0%
Celebrity25%27%59%0.9%3.3%9.6%
Cellphone15%24%51%2.4%2.4%10.5%
Game16%28%59%3.3%3.0%10.1%
Health9%19%57%1.3%1.0%7.0%
Movies18%25%64%2.8%2.5%11.5%
News16%27%66%2.7%2.5%11.8%
TV Series15%25%56%3.2%2.9%10.6%
Mean15%24%58%2.3%2.5%10.1%

Writer. We measure comment diversity using within-thread cosine similarity and recurring four-word phrases. A phrase is considered recurring if it appears in at least 10% of threads within a domain–model group.

GPT-4o-mini also produces more repetitive comments, with a mean cosine similarity of 0.366, compared with 0.223-0.232 for the other backbones and 0.241 for real discussions. Recurring phrases appear in 24.3% of its comments, compared with 3.2-8.4% for the other backbones and 1.5% for real discussions. These results suggest that GPT-4o-mini produces less diverse plans and comments, leaving fewer distinct candidates for branch selection.

Writer within-thread comment cosine similarity.

DomainDS5.4-mini4o-miniReal
Camera0.2510.2500.3730.275
Celebrity0.1840.1730.3110.180
Cellphone0.2660.2400.3820.289
Game0.2270.2190.3460.221
Health0.2600.2690.4240.311
Movies0.2320.2310.3850.239
News0.2260.2030.3660.204
TV Series0.2100.1990.3370.214
Mean0.2320.2230.3660.241

Writer comments containing a recurring phrase (%).

DomainDS5.4-mini4o-miniReal
Camera8.4%15.5%41.5%4.7%
Celebrity3.5%5.7%13.9%0.0%
Cellphone2.4%11.2%21.5%0.4%
Game3.2%9.2%28.4%2.6%
Health0.7%3.7%31.4%0.0%
Movies1.3%7.2%18.6%1.9%
News5.3%10.6%28.3%1.6%
TV Series1.0%4.2%10.4%0.6%
Mean3.2%8.4%24.3%1.5%
AuthorsOpenReview ↗

Kind Reminder and Summary of Our Rebuttal

Dear Reviewer fB7v,

Thank you again for taking the time to review our work. Since the discussion period is ending soon, we would like to briefly summarize the key points of our rebuttal:

  • Research questions and design rationale (W1): We have reorganized Section 5 around three research questions (RQ1–RQ3), covering discussion realism, the contributions of draft budget and branch selection, and differences in discussion behavior across methods and LLMs.

  • Contribution of branch selection (W2, Q2): We conducted an additional ablation comparing metric-based branch selection with random selection over 50 repeated runs. At K=3, DeepSeek achieves 81.2% with selection versus 49.6% with random selection, while GPT-5.4-mini achieves 81.2% versus 46.5%. These results show that branch selection accounts for much of the improvement at larger draft budgets.

  • Separation of data pools (Q1): We clarify that the seed, reference, and test pools are separated by source-thread ID. No source thread, including its root post and comments, is shared across the three pools (Appendix E.1).

  • Planner and Writer diversity across backbones (Q3): We conducted additional analyses to explain GPT-4o-mini's smaller improvements. Its Planner produces 58% near-duplicate plans, compared with 15% for DeepSeek and 24% for GPT-5.4-mini. Its Writer also produces more recurring phrases (24.3% versus 3.2–8.4% for the other backbones). These results suggest that lower output diversity limits the benefits of branch selection.

Full details are provided in our responses above. We have also revised the paper accordingly, with the changes highlighted in blue for easy reference. We hope these additional experiments and clarifications address your concerns. We would appreciate any further feedback before the discussion period closes.

Best regards,
The Authors

Reviewer_q7AAOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ThreadBench, a benchmark for evaluating how closely LLM generated discussions resemble real threads. It contains 9,817 threads across 8 domains and uses 12 metrics covering content diversity, expression, tone and interaction patterns. The authors also propose ThreadPDG, a planner-writer framework that generates discussions using real thread structures and combines branches from multiple drafts. Experiments show substantial improvements over OASIS and SynthPAI across several LLMs, with additional support from a human evaluation involving 436 annotators.

Critical strengths
  1. A useful benchmark for discussion realism: The paper addresses an important evaluation gap by considering entire discussions rather than individual comments. The combination of content-based and structural metrics makes the benchmark potentially useful for future research.

  2. Strong empirical improvements: ThreadPDG consistently outperforms the two baselines across most domains and models. The inclusion of human evaluation provides additional evidence beyond the proposed automatic metrics.

Critical weaknesses
  1. Limited evidence of independently generated discussion dynamics: ThreadPDG directly inherits the comment count, reply structure and approximate comment lengths from real seed threads. This makes the strong performance on structural metrics less surprising and raises questions about whether the method actually learns realistic interaction dynamics or reproduces existing ones.

  2. Evaluation metrics overlap with optimization objective: ThreadPDG selects branches using the same metrics employed by ThreadBench. Although the authors use separate reference and test pools, this still makes it difficult to determine how much of the improvement reflects more realistic discussions rather than optimizations toward the benchmark’s measurements.

Questions for authors
  • How would ThreadPDG perform if it had to generate its own reply structure instead of inheriting the structure of a real discussion?

  • How sensitive are the reported pass rates to the evaluation sample size? Since passing a statistical test does not establish distributional equivalence, it would be useful to understand whether the conclusions remain stable with larger samples.

5 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We appreciate the reviewer’s thoughtful assessment and the recognition of ThreadBench as a useful resource for evaluating entire discussions. We also value the reviewer’s acknowledgment of the empirical improvements and the supporting human evaluation. We have carefully considered your questions and address them individually below.

1. Reliance on inherited structure and performance with independently generated reply structure

We thank the reviewer for this insightful question. We agree that inheriting comment counts, reply structure, and approximate lengths contributes to the strong performance on structural metrics. We would like to clarify that realistic interaction also involves how participants respond to one another, including their stances, tone, and disagreements. ThreadPDG uses real discussion examples to guide the Planner and Writer in generating these interaction patterns. ThreadBench evaluates them through metrics such as disagreement and politeness, alongside other discussion characteristics including emotion, storytelling, and content diversity.

Beyond structural metrics, Table 3 shows ThreadPDG at K=3 passes 78.8% of the remaining metrics with both GPT-5.4-mini and DeepSeek, compared with 1.3–13.8% for OASIS and 2.5–8.8% for SynthPAI across backbones, after excluding average depth and structural virality. In the human Turing test, ThreadPDG discussions are not distinguished from real ones in 78.7% of evaluated pairs, compared with 61.1% for OASIS and 58.7% for SynthPAI.

To assess performance without inherited structure, we evaluate a variant in which the Planner independently chooses comment counts, reply targets, and target lengths from the seed post and comment text. This uses GPT-5.4-mini at K=1, with other settings unchanged. The table below compares Free(Planner independently plans the reply structure along with comment length and counts) with the original ThreadPDG at K=1 and K=3 on the seeds shared by all three configurations, using the same real reference as Table 3.

DomainFree K=1PDG K=1PDG K=3
TV Series25.0%50.0%75.0%
Camera25.0%41.7%83.3%
News0.0%58.3%75.0%
Game16.7%33.3%66.7%
Health25.0%50.0%58.3%
Movies25.0%50.0%83.3%
Celebrity8.3%25.0%100.0%
Cellphone33.3%50.0%83.3%
Mean19.8%44.8%78.1%

This experiment shows that Free K=1 achieves an average metric pass rate of 19.8%, compared with 44.8% for ThreadPDG K=1 and 78.1% for K=3. The results suggest that learning structural properties and comment-length variation without a structural template remains challenging. Nevertheless, Free performs better on disagreement and impoliteness in some domains. Detailed per-metric means, MWU/KS p-values, and pass/fail decisions are provided in Appendix L.5, Table 29.

2. Overlap between selection and benchmark metrics

We thank the reviewer for raising this concern. We agree that branch selection uses the same metrics as ThreadBench. To assess whether improvements in these metrics also make generated discussions more realistic, we included a separate human Turing test (Table 2).

Across 240 discussion pairs and 436 annotators, ThreadPDG discussions are not distinguished from real ones in 78.7% of evaluated pairs, compared with 61.1% for OASIS and 58.7% for SynthPAI. These results provide additional evidence that the generated discussions resemble real ones beyond what is measured by the benchmark metrics.

AuthorsOpenReview ↗

Rebuttal by Authors

3. Sensitivity to evaluation sample size

We thank the reviewer for this important question. Following the reviewer’s suggestion, we conduct additional experiments to examine how evaluation results change with sample size. At each sample size, we randomly sample nn threads per group. The table below reports the average pass rates for ThreadPDG at K=3K=3.

The table below reports pass rates averaged across domains. We observe that pass rates generally decrease as sample size increases, with the largest drops occurring at smaller sample sizes. The changes become smaller as the sample size approaches 150. At n=130, DeepSeek and GPT-5.4-mini achieve pass rates of 80.8% and 81.9%, close to their results of 81.7% and 81.2% at n=150. GPT-4o-mini also changes only slightly, from 26.0% to 28.1%, while real-vs-real pass rates remain around 94%.

SampleReal–realDeepSeek-V4.1-Flashgpt-5.4-minigpt-4o-mini
n=1095.694.394.580.1
n=3094.490.390.557.6
n=5093.987.587.646.1
n=7094.185.886.139.1
n=9094.184.385.132.7
n=11094.182.683.829.8
n=13093.380.881.926.0
n=15094.381.781.228.1

We further extend the target sample size from 150 to 170 for GPT-5.4-mini at K=3K=3. The average pass rate changes only slightly, from 81.2% to 79.2%. These results suggest that our findings remain consistent with a larger sample. Detailed per-metric results, including MWU/KS p-values and pass/fail decisions, are provided in Appendix L.5, Table 30.

DomainK=1 when n=150 → n=170K3 when n=150 → n=170
Camera41.7 → 41.783.3 → 75.0
Celebrity33.3 → 25.091.7 → 91.7
Cellphone50.0 → 41.783.3 → 83.3
Game33.3 → 33.366.7 → 66.7
Health50.0 → 50.066.7 → 75.0
Movies50.0 → 50.091.7 → 91.7
News58.3 → 50.075.0 → 66.7
TV Series50.0 → 50.091.7 → 83.3
Mean45.8 → 42.781.2 → 79.2
AuthorsOpenReview ↗

Kind Reminder and Summary of Our Rebuttal

Dear Reviewer q7AA,

Thank you again for taking the time to review our work. Since the discussion period is ending soon, we would like to briefly summarize the key points of our rebuttal:

  • Reliance on inherited structure (W1, Q1): We conducted an additional experiment in which the Planner independently determines comment counts, reply targets, and target lengths. The independently planned variant achieves a 19.8% average pass rate, compared with 44.8% for ThreadPDG at K=1 and 78.1% at K=3. Beyond structural metrics, ThreadPDG at K=3 also passes 78.8% of the remaining metrics, showing improvements beyond inherited structure (Table 3, Appendix L.4).

  • Overlap between selection and evaluation metrics (W2): We provide additional evidence through a separate human Turing test involving 436 annotators and 240 discussion pairs. ThreadPDG discussions are not distinguished from real ones in 78.7% of evaluated pairs, compared with 61.1% for OASIS and 58.7% for SynthPAI (Table 2). These results support discussion realism beyond the benchmark metrics.

  • Sensitivity to evaluation sample size (Q2): We conducted additional experiments across sample sizes from 10 to 150 and further extended the evaluation to 170. Pass rates become more stable as sample size increases. For GPT-5.4-mini at K=3, the average pass rate changes only slightly from 81.2% at n=150 to 79.2% at n=170, supporting the consistency of our findings (Appendix L.2, Appendix L.5, Table 30).

Full details are provided in our responses above. We have also revised the paper accordingly, with the changes highlighted in blue for easy reference. We hope these additional experiments and clarifications address your concerns. We would appreciate any further feedback before the discussion period closes.

Best regards,
The Authors

Reviewer_jJLMOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces a benchmark for real social media discussion simulation, covering 8 domains and 116 subdomains with 12 thread-level evaluation metrics. The authors propose ThreadPDG, a Prompt-based Planner–Writer Discussion Generation method, which achieves pass rates of 58–100% with multiple LLMs and substantially outperforms existing baselines.

Critical strengths
  • The large-scale dataset covers real Reddit discussions spanning 8 domains and 116 subdomains.
  • The proposed Planner–Writer Discussion Generation method shows significantly better performances than baselines.
Critical weaknesses
  • What are the implications of generating free-form discussions? Because the task objectives are unclear, the definitions of the metrics lack strong motivation, especially for story_probability and hard_disagree_rate. There is no clear reason why these metrics can evaluate the quality of discussion simulations.
  • Quality filtering removes incomplete or unusable discussions according to predefined content and structural requirements. How can we verify the quality of the remaining data?
  • The figures and tables are all very hard to read and understand.
  • The Planner–Writer is an application of an existing multi-agent framework. The authors also do not provide related work for comparison to demonstrate their contribution.
Questions for authors

It would be better to use larger font sizes for the tables and figures. Currently, Table 2 is overwhelming to read, but the information it provides is limited.

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for the thorough review and appreciate the recognition of ThreadBench's coverage and ThreadPDG's stronger performance over the baselines. We address the concerns about metric motivation, data quality, presentation, and our contribution below.

1. Motivation for free-form discussion simulation and the evaluation metrics

We appreciate the reviewer for raising this point. We agree that the motivation for free-form discussion simulation and the choice of evaluation metrics were not sufficiently explained. We have revised and expanded the Introduction and Section 3.2 in the paper.

Why discussion simulation matters. Online discussions provide insights into how people exchange arguments, share experiences, and respond to others, including in studies of persuasion [1,2]. LLM-based discussion simulations allow researchers to vary conditions and examine how conversations develop across simulations [4]. For these simulations to be useful in studying human interaction, they need to reflect the communication patterns observed in real discussions.

Why realism needs empirical validation. [3] identifies validation against real observations as a central challenge for generative social simulation. [4] evaluates whether readers can distinguish generated Reddit discussions from real ones. [5] further shows that turn-taking and conversational equality predict conflict outcomes in real conversations but not in simulated ones. These findings highlight the importance of evaluating realism beyond apparent plausibility. ThreadBench therefore compares generated and real Reddit threads across uniformity, expression, tone, and interaction, using statistical distributions to identify differences in discussion patterns.

Motivation for storytelling. Storytelling is common in Reddit discussions. [6] estimates that 52% of sampled posts and comments across 291 subreddits contain stories. StoryARG [7] provides annotated examples of narratives and personal experiences, while [8] identifies 47,153 narrative comments in a corpus of over one million comments. These findings motivate the use of story_probability to evaluate whether generated threads capture the distribution of storytelling observed in real discussions within each domain.

Motivation for disagreement. An analysis of over 9,000 Reddit threads [9] identifies disagreement chains as characteristic patterns of argumentative discussions. DEBAGREEMENT [10] reports disagreement in 33–42% of annotated comment–reply pairs in its forums, while [11] identifies disagreement through users' stances toward named entities. Building on these studies, we use hard_disagree_rate to measure explicit disagreement between comments and replies and compare its distribution between generated and real threads within each domain.

2. Quality filtering and validation of the retained data

We agree that filtering alone does not establish the quality of the retained data. To address this concern, two authors jointly reviewed 500 retained discussion threads and found all of them suitable for inclusion in ThreadBench. We also assess the retained data through statistical comparisons and simulation performance. The repeated real-vs-real comparisons yield a joint metric pass rate of 94.3%, showing consistency across samples of real discussions(Section 3.4). Using the retained data as a reference, ThreadPDG achieves metric pass rates of 67–92% with GPT-5.4-mini and 58–100% with DeepSeek across eight domains, compared with 0–17% for the baselines (Table 3). In the human study with 436 retained annotators, ThreadPDG discussions are not distinguished from real ones in 78.7% of evaluated pairs, compared with 61.1% for OASIS and 58.7% for SynthPAI (Table 2). These results provide further evidence that the retained data can serve as a valid reference for generating and evaluating realistic discussions.

AuthorsOpenReview ↗

Rebuttal by Authors

3. Readability of figures and tables

Thank you for pointing this out. We agree that the original tables were difficult to read. We have enlarged the font sizes and adjusted the layouts of Tables 1-3 to improve readability.

4. Planner–Writer framework and related work

We thank the reviewer for suggesting that we clarify the relationship between our approach and prior work. We have added a related work subsection on planning–writing frameworks in the paper.

Planning–Writing MAS. Separating planning from text generation is a well-established approach. Plan-and-Write [12] generates a storyline before writing the story, while Re3 [13] combines planning, writing, and revision using prompted LLMs. STORM [14] constructs outlines from collected information to generate Wikipedia-style articles, and AgentWrite in LongWriter [15] plans and generates long-form text section by section. These studies demonstrate the use of explicit planning to guide text generation. AutoGen [16] further provides a general framework for coordinating specialized LLM agents.

Our contribution lies in combining the established planning–writing approach with branch-level generation and selection to model realistic discussions. Specifically, the Planner determines how discussion branches develop and what each comment contributes, while the Writer generates comments accordingly. ThreadPDG then selects and combines candidate branches based on thread-level statistics to better match real discussion patterns, including variation in expression, tone, and interactions across replies.

AuthorsOpenReview ↗

Rebuttal by Authors

References

[1] Tan et al. (WWW 2016). Winning Arguments: Interaction Dynamics and Persuasion Strategies in Good-faith Online Discussions. https://arxiv.org/abs/1602.01103

[2] Mirzakhmedova et al. (Findings of EMNLP 2023). Unveiling the Power of Argument Arrangement in Online Persuasive Discussions. https://aclanthology.org/2023.findings-emnlp.1048/

[3] Larooij and Törnberg (Artificial Intelligence Review 2026, 59:15). Validation is the Central Challenge for Generative Social Simulation: A Critical Review of LLMs in Agent-Based Modeling. https://doi.org/10.1007/s10462-025-11412-6

[4] Bouleimen et al. (Scientific Reports 2026, 16:32760). The Collective Turing Test: Large Language Models Can Generate Realistic Multi-User Discussions. https://doi.org/10.1038/s41598-026-62286-9

[5] D’Costa et al. (2024). What You Say or How You Say It? Predicting Conflict Outcomes in Real and LLM-Generated Conversations. https://arxiv.org/abs/2409.09338

[6] Antoniak et al. (ACL 2024). Where Do People Tell Stories Online? Story Detection Across Online Communities. https://arxiv.org/abs/2311.09675

[7] Falk and Lapesa (ACL 2023). StoryARG: A Corpus of Narratives and Personal Experiences in Argumentative Texts. https://aclanthology.org/2023.acl-long.132/

[8] Nabhani et al. (ArgMining 2025). Storytelling in Argumentative Discussions: Exploring the Use of Narratives in ChangeMyView. https://aclanthology.org/2025.argmining-1.21/

[9] Zhang, Culbertson, and Paritosh (ICWSM 2017). Characterizing Online Discussion Using Coarse Discourse Sequences. https://doi.org/10.1609/icwsm.v11i1.14886

[10] Pougué-Biyong et al. (NeurIPS Datasets and Benchmarks 2021). DEBAGREEMENT: A Comment-Reply Dataset for (Dis)agreement Detection in Online Debates. https://datasets-benchmarks-proceedings.neurips.cc/paper_files/paper/2021/hash/6f3ef77ac0e3619e98159e9b6febf557-Abstract-round2.html

[11] Lorge et al. (LREC-COLING 2024). STEntConv: Predicting Disagreement between Reddit Users with Stance Detection and a Signed Graph Convolutional Network. https://aclanthology.org/2024.lrec-main.1327/

[12] Yao et al. (AAAI 2019). Plan-and-Write: Towards Better Automatic Storytelling. https://arxiv.org/abs/1811.05701

[13] Yang et al. (EMNLP 2022). Re3: Generating Longer Stories With Recursive Reprompting and Revision. https://aclanthology.org/2022.emnlp-main.296/

[14] Shao et al. (NAACL 2024). Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models. https://aclanthology.org/2024.naacl-long.347/

[15] Bai et al. (ICLR 2025). LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs. https://proceedings.iclr.cc/paper_files/paper/2025/file/59f278de1619bdb6b53fd04e8e0976e0-Paper-Conference.pdf

[16] Wu et al. (COLM 2024). AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation. https://arxiv.org/abs/2308.08155

AuthorsOpenReview ↗

Kind Reminder and Summary of Our Rebuttal

Dear Reviewer jJLM,

Thank you again for taking the time to review our work. Since the discussion period is ending soon, we would like to briefly summarize the key points of our rebuttal:

  • Motivation and evaluation metrics (W1): We have expanded the Introduction and Section 3.2 to clarify the motivation for free-form discussion simulation and the choice of evaluation metrics. We also provide evidence from prior studies to support the use of storytelling and disagreement as important characteristics of real discussions.

  • Data quality and validation (W2): Two authors jointly reviewed 500 retained threads and found all suitable for inclusion. Real-vs-real comparisons achieve a 94.3% joint pass rate. In our human evaluation with 436 annotators, ThreadPDG discussions are not distinguished from real ones in 78.7% of evaluated pairs, compared with 61.1% for OASIS and 58.7% for SynthPAI.

  • Readability of figures and tables (W3, Q1): We have enlarged the font sizes and adjusted the layouts of Tables 1–3, including Table 2, to improve readability.

  • Planner–Writer contribution and related work (W4): We have added a related work subsection covering existing planning–writing approaches. We clarify that our contribution lies in combining planning–writing with branch-level generation and selection to better match real discussion patterns.

Full details are provided in our responses above. We have also revised the paper accordingly, with the changes highlighted in blue for easy reference. We hope these revisions address your concerns. We would appreciate any further feedback before the discussion period closes.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.