- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
AudCraft derives a human-content reference and audience taxonomy from Qilin, then compares LLMs with human posts on audience orientation and audience-conditioned expression. It distinguishes expression level, cross-audience variation, and alignment across 55 segments. Results show concentrated default generations, incomplete alignment with human expression, and a blinded preference advantage for human posts.
Critical strengths
-
Separating expression level, adaptation magnitude, and adaptation alignment shows why lexical richness or large stylistic changes need not reproduce human audience patterns. Analysis across 55 segments and four models makes this more informative than a single overall quality score.
-
The taxonomy uses real content and engagement queries, and the study adds human validation. The 275 blinded, randomized-order comparisons favor human posts in 58% of cases versus 35% for GPT-5.4-mini, while recognizing the model's contextual-usefulness advantage. This supports the descriptive findings without treating every difference as a deficiency.
Critical weaknesses
-
The human reference may be selected independently of the model characteristics being evaluated. It removes 256,730 posts (13.9% of otherwise eligible content) with Fast-DetectGPT, without Chinese-platform calibration or filtering sensitivity. Removing human posts that resemble model output could enlarge the stylistic and originality gaps. This directly affects the central comparison.
-
The audience-design comparison matches domain-intent labels but not concrete topic or genre. Content specificity and topic mix can therefore drive stylometric profiles and human-derived vocabulary scores. The human preference study is valuable but uses the same broad matching. The evidence supports differences under this generation setup more securely than differences specifically attributable to audience adaptation.
Questions for authors
Do the main profile and human-preference findings persist with an unfiltered or manually verified human subset? What detector threshold and Chinese-data calibration were used?
Can topic/genre-matched comparisons establish which profile differences remain, or can the audience-design interpretation be limited to the current broadly conditioned setup?
Response for Reviewer WUbZ
Thank you for acknowledging our work and your constructive comments. Please kindly find our point-by-point responses below for your review.
Thank you for this question. To assess the impact of AI-text filtering, we repeated our analyses using an unfiltered human reference and compared the results with those obtained after filtering.
(1) Stability of Audience Distributions. We compared the unfiltered and filtered human references, each containing 100 posts.
Although filtering slightly reduces human concentration, the changes are small (ΔHHI = −0.0111 and −0.0135). Importantly, the unfiltered reference still exhibits substantially lower concentration than the LLMs, indicating that our main findings are not driven by detector-based filtering.
(2) Stability of Audience Adaptation Results. We further compared three adaptation results across the same 55 audience groups, each containing 56 estimates (4 models × 14 metrics).
The estimates remain highly consistent before and after filtering, with correlations of 0.990–0.999 and mean absolute changes of 0.021–0.040. None of these differences alters our qualitative conclusions, confirming the robustness of our findings to AI-text filtering.
(3) Detector Threshold and Chinese-data Calibration. We calibrated Fast-DetectGPT on Chinese data using 7,000 AI-generated texts from seven LLMs and 1,000 human-written posts. The threshold of 1.514155 was set to the median detection score of the AI-generated texts. At this threshold, the AI detection rate for LLM-generated texts was 4.46 times that for human-written posts, demonstrating discriminative ability on Chinese data. This calibration provides an empirical basis for filtering while limiting the removal of human-written content.
Thank you for this suggestion. Our study focuses on audience-group-level comparisons, examining how LLM-generated content differs from human posts within the same Domain–Intent segment. Importantly, audience adaptation involves not only how content is expressed, but also what content is selected for a target audience. Therefore, our evaluation intentionally allows specific topics and genres to vary, capturing the overall content and expression choices under shared audience conditions.
We agree that our findings should not be interpreted as audience effects isolated from topical or genre variation. We will explicitly clarify this scope in the manuscript. Topic/genre-matched comparisons would offer a complementary analysis of which differences persist under finer-grained controls, which we leave for future work.