- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether the similarity between LLM-generated social networks and real human social networks comes from the LLM agents' decision-making policies, or from the simulator architecture and input representations. To address this question, the authors propose the Simulator Identifiability Audit, which uses architecture-matched Random Twins, Matched Edge Counts, and Label Permutation to evaluate generative social simulators in three stages: policy detectability, structural fidelity improvement, and attribution to individuals. The authors conduct experiments across six simulator architectures, real and synthetic social networks, and multiple LLMs. They also release the ClassNetBench benchmark.
Critical strengths
The research question is quite novel. Existing studies on LLM-based social simulation rarely distinguish the contributions of LLM policies, simulator architectures, and input representations. This paper clearly identifies this attribution problem and proposes an architecture-matched Random Twin and a three-stage audit framework to address it.
Critical weaknesses
-
The authors project generated networks to a common edge count () to reduce the influence of network size differences on structural metrics. This is reasonable when comparing connection patterns at a fixed edge count. However, the number of edges can also be an important outcome influenced by the LLM policy itself. The paper uses the minimum edge count across generated networks as , while keeping the observed network unchanged. When the LLM generates far fewer edges than the real network, this projection may limit the higher-order structures that can be observed. Therefore, the paper should more clearly distinguish between the overall effect of the policy on the generated network and its effect on network structure after controlling for the edge count.
-
The experiments mainly build on a shared framework of candidate selection and edge accumulation, with additional mechanisms such as information exposure, choice volume, memory, and network backbones. These tasks are essentially structured social partner selection, which is still quite different from free-form conversations, two-way interactions, dynamic relationship development, and long-term planning in many real-world generative social simulation systems.
Questions for authors
-
The paper mentions that each architecture is calibrated using its Random Twin, and that the Random Twin is calibrated to the same structural targets used in evaluation. Which network statistics are used during calibration? Do these statistics overlap with the six metrics used to evaluate structural improvements? Are the same real networks used for both calibration and final evaluation?
-
Does the disappearance of T3's community improvement really show that the improvement is only caused by differences in edge counts, or could the current matching method be controlling away part of the effect of memory? Across different datasets, what proportion of the observed network's edges does the common edge count () represent for T3? How sensitive are the results to the choice of ()?