- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces ThreadBench, a benchmark for evaluating how realistically LLM-based agents can simulate real-world Reddit discussions at the thread level. Unlike prior work that focuses on single-comment quality, role-play, or task success, ThreadBench compares generated discussion threads against 9,817 real Reddit threads spanning 8 domains and 116 subdomains. It defines 12 thread-level metrics across four families: uniformity, expression, tone, and interaction, and uses nonparametric tests (Mann–Whitney U and Kolmogorov–Smirnov) to decide whether generated discussions are statistically indistinguishable from real ones.
The authors further propose ThreadPDG, a Planner–Writer discussion generation framework that learns how real discussions grow, generates multiple candidate threads, and composes the best branches using thread-level statistics. Experiments across GPT-5.4-mini, GPT-4o-mini, and DeepSeek-V4.1-Flash show that ThreadPDG achieves 58–100% metric pass rates, while baselines OASIS and SynthPAI remain at 0–17%. A human Turing-test study with 436 annotators finds ThreadPDG indistinguishable from real discussions in 78.7% of pairs.
Critical strengths
- Thread-level evaluation is the right unit. Most prior benchmarks stop at comment-level fidelity; this paper evaluates structure + semantics + tone jointly.
- Strong empirical contribution. The paper presents a large-scale benchmark comprising 9,817 threads across 116 subdomains, evaluated along 12 metrics. The scale and breadth of this resource represent a substantial contribution to the community.
- Statistical rigor above average. Use of MWU + KS, real-vs-real calibration, effect sizes, and a consistency theorem is commendable.
Critical weaknesses
- Presentation could be improved. The paper is rich in implementation and metric details, but the high-level design rationale receives comparatively less attention. A clearer narrative that connects the design choices to the underlying research questions would make the contributions easier to appreciate.
- The contribution of the branch selection mechanism is not fully isolated. Since the selection step explicitly optimizes toward the target distribution, it is unclear how much of the reported gain comes from the Planner/Writer genuinely learning discussion dynamics, versus from the selection step itself performing metric fitting. An ablation that replaces branch selection with random sampling would help clarify this.
Questions for authors
- How strictly are seed pool / reference pool / test pool disjoint? Are root posts ever shared across splits?
- If ThreadPDG’s branch selection is disabled and branches are sampled randomly, how much of the gain remains?
- Why does gpt-4o-mini benefit much less than GPT-5.4-mini / DeepSeek? Is this planner competence or writer capacity?
5 comments
Rebuttal by Authors
3. Separation of seed, reference, and test pools
We appreciate the reviewer’s question. As described in Appendix E.1, the seed, reference, and test pools are separated by source-thread ID within each domain. No source thread, including its root post and comments, is shared across the three pools. The seed pool is used for generation, the reference pool provides statistics for branch selection, and the test pool is used only for final evaluation.
4. Planner and Writer diversity across backbones
We thank the reviewer for this insightful question. We conduct an additional analysis of Planner and Writer outputs at K=1, before branch selection, to understand why GPT-4o-mini shows smaller improvements. The experiments below examine the diversity of Planner and Writer outputs at K=1, before branch selection.
Planner. We measure plan diversity using semantic similarity and four-gram reuse. For each planned comment, we combine its planned response, topic, details, and context into a single text and use all-mpnet-base-v2 to measure similarity between plans. Plans with cosine similarity ≥0.70 to another plan in the same thread are considered near-duplicates. We also measure how often plans within the same thread share four-word sequences.
The results show that GPT-4o-mini generates more repetitive plans, with a mean near-duplicate rate of 58%, compared with 15% for DeepSeek and 24% for GPT-5.4-mini. Its four-gram reuse rate is also higher (10.1% vs. 2.3-2.5%).
Planner repetition by domain (%).
| Domain | Near-dup. DS | Near-dup. 5.4-mini | Near-dup. 4o-mini | 4-gram reuse DS | 4-gram reuse 5.4-mini | 4-gram reuse 4o-mini |
|---|---|---|---|---|---|---|
| Camera | 9% | 20% | 54% | 2.2% | 2.4% | 10.0% |
| Celebrity | 25% | 27% | 59% | 0.9% | 3.3% | 9.6% |
| Cellphone | 15% | 24% | 51% | 2.4% | 2.4% | 10.5% |
| Game | 16% | 28% | 59% | 3.3% | 3.0% | 10.1% |
| Health | 9% | 19% | 57% | 1.3% | 1.0% | 7.0% |
| Movies | 18% | 25% | 64% | 2.8% | 2.5% | 11.5% |
| News | 16% | 27% | 66% | 2.7% | 2.5% | 11.8% |
| TV Series | 15% | 25% | 56% | 3.2% | 2.9% | 10.6% |
| Mean | 15% | 24% | 58% | 2.3% | 2.5% | 10.1% |
Writer. We measure comment diversity using within-thread cosine similarity and recurring four-word phrases. A phrase is considered recurring if it appears in at least 10% of threads within a domain–model group.
GPT-4o-mini also produces more repetitive comments, with a mean cosine similarity of 0.366, compared with 0.223-0.232 for the other backbones and 0.241 for real discussions. Recurring phrases appear in 24.3% of its comments, compared with 3.2-8.4% for the other backbones and 1.5% for real discussions. These results suggest that GPT-4o-mini produces less diverse plans and comments, leaving fewer distinct candidates for branch selection.
Writer within-thread comment cosine similarity.
| Domain | DS | 5.4-mini | 4o-mini | Real |
|---|---|---|---|---|
| Camera | 0.251 | 0.250 | 0.373 | 0.275 |
| Celebrity | 0.184 | 0.173 | 0.311 | 0.180 |
| Cellphone | 0.266 | 0.240 | 0.382 | 0.289 |
| Game | 0.227 | 0.219 | 0.346 | 0.221 |
| Health | 0.260 | 0.269 | 0.424 | 0.311 |
| Movies | 0.232 | 0.231 | 0.385 | 0.239 |
| News | 0.226 | 0.203 | 0.366 | 0.204 |
| TV Series | 0.210 | 0.199 | 0.337 | 0.214 |
| Mean | 0.232 | 0.223 | 0.366 | 0.241 |
Writer comments containing a recurring phrase (%).
| Domain | DS | 5.4-mini | 4o-mini | Real |
|---|---|---|---|---|
| Camera | 8.4% | 15.5% | 41.5% | 4.7% |
| Celebrity | 3.5% | 5.7% | 13.9% | 0.0% |
| Cellphone | 2.4% | 11.2% | 21.5% | 0.4% |
| Game | 3.2% | 9.2% | 28.4% | 2.6% |
| Health | 0.7% | 3.7% | 31.4% | 0.0% |
| Movies | 1.3% | 7.2% | 18.6% | 1.9% |
| News | 5.3% | 10.6% | 28.3% | 1.6% |
| TV Series | 1.0% | 4.2% | 10.4% | 0.6% |
| Mean | 3.2% | 8.4% | 24.3% | 1.5% |
Kind Reminder and Summary of Our Rebuttal
Dear Reviewer fB7v,
Thank you again for taking the time to review our work. Since the discussion period is ending soon, we would like to briefly summarize the key points of our rebuttal:
-
Research questions and design rationale (W1): We have reorganized Section 5 around three research questions (RQ1–RQ3), covering discussion realism, the contributions of draft budget and branch selection, and differences in discussion behavior across methods and LLMs.
-
Contribution of branch selection (W2, Q2): We conducted an additional ablation comparing metric-based branch selection with random selection over 50 repeated runs. At K=3, DeepSeek achieves 81.2% with selection versus 49.6% with random selection, while GPT-5.4-mini achieves 81.2% versus 46.5%. These results show that branch selection accounts for much of the improvement at larger draft budgets.
-
Separation of data pools (Q1): We clarify that the seed, reference, and test pools are separated by source-thread ID. No source thread, including its root post and comments, is shared across the three pools (Appendix E.1).
-
Planner and Writer diversity across backbones (Q3): We conducted additional analyses to explain GPT-4o-mini's smaller improvements. Its Planner produces 58% near-duplicate plans, compared with 15% for DeepSeek and 24% for GPT-5.4-mini. Its Writer also produces more recurring phrases (24.3% versus 3.2–8.4% for the other backbones). These results suggest that lower output diversity limits the benefits of branch selection.
Full details are provided in our responses above. We have also revised the paper accordingly, with the changes highlighted in blue for easy reference. We hope these additional experiments and clarifications address your concerns. We would appreciate any further feedback before the discussion period closes.
Best regards,
The Authors
Rebuttal by Authors
We are glad that the reviewer considers thread-level evaluation the appropriate unit and recognizes the benchmark’s scale and statistical rigor, including the real-vs-real calibration, effect sizes, and consistency theorem. We appreciate the suggestions for making the design rationale clearer and separating the contributions of generation and selection. We respond to these points and questions individually below.
1. Research questions and design rationale
We appreciate the reviewer’s suggestion. To make the motivation and design of our experiments clearer, we organize the Experiments section around three research questions. We have added the following paragraph at the beginning of the Section 5 and throughout the paper.
RQ1. Evaluating the realism of generated discussions. We examine how closely existing simulators and ThreadPDG match real Reddit discussions across eight domains. We use ThreadBench to assess different aspects of discussion behavior and complement the statistical comparisons with a human Turing test (Section 5.2).
RQ2. Understanding how ThreadPDG improves discussion realism. We examine the contributions of draft budget and branch selection by varying the number of drafts and comparing metric-based selection with random selection at the same draft budget (Section 5.3).
RQ3. Understanding differences in discussion behavior across methods and LLMs. Beyond overall pass rates, we analyze semantic diversity, comment length, tone, and reply structure to identify which characteristics of real discussions different methods capture well and which remain challenging (Section 5.4).
2. Contribution of branch selection
We truly thank the reviewer for the suggestion. We conducted an additional experiment replacing metric-based branch selection with random selection while keeping all other setups the same. Each random selection is repeated 50 times, and the tables below report the average metric pass rates across 8 domains.
The results show that branch selection contributes substantially to the performance improvement. At K=3, DeepSeek achieves 81.2% with metric-based selection compared with 49.6% under random selection, while GPT-5.4-mini achieves 81.2% compared with 46.5%. Random selection remains close to K=1 for both models (47.9% and 45.8%), suggesting that increasing the draft budget alone provides limited improvement. For GPT-4o-mini, selection provides a smaller gain (28.1% vs. 21.4%) and does not improve performance in every domain. These results show that branch selection accounts for much of the additional improvement at larger draft budgets, while the K=1 results reflect the generation performance without selection.
DeepSeek-V4.1-Flash: average pass rate (%) across domains
GPT-5.4-mini: average pass rate (%) across domains.
GPT-4o-mini: average pass rate (%) across domains.