- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper argues that LLM social simulators built on persistent agents are ill-posed for open online platforms, where the set of participants changes over time and individuals are observed only when they post. It reframes simulation as recursive prediction of the observable communication history and proposes OpenSim: a communication-volume forecaster with participant activation from a user pool, a mean-field state transition (following M2Sim), and an LLM communication policy trained with a future-trajectory critic. On one Chinese social-media corpus (59 event units, 7 for test) and five Qwen backbones, OpenSim reports lower KL, Wasserstein and DTW than Direct LLM, Social Retrieval and MF-LLM. Intervention, boundary and ablation experiments are also reported.
Critical strengths
- Treating participation as a prediction target, instead of supplying the realised participant sequence, exposes an information leak in current mean-field simulators. The boundary experiment (Table 3) measures how much that leak helps (about 35% lower KL). If this holds up, it changes how such simulators should be evaluated.
Critical weaknesses
-
The headline comparison does not support the claim of simulation without future participant information. The main OpenSim variant samples from an event-specific pool "constructed from the corresponding event's observed participants" (App. D.2), that is, from the people who actually took part, including in the test horizon. This leaks less than supplying the ordered sequence, but it is still future information. With the event-agnostic SocioVerse pool (OpenSim†), the gain over MF-LLM largely disappears, and on two of five backbones it reverses (KL 2.08 vs 1.25 on Qwen2-1.5B; 1.57 vs 1.47 on Qwen2.5-1.5B). The 71.2% and 35.0% improvements in the abstract are therefore measured on the leaky variant.
-
The participation component is the paper's main conceptual addition, yet it is neither specified nor evaluated. The volume forecaster is never described. Table 5 lists participant sampling as uniform, so the activation likelihood appears not to be learned, and App. D.2 states that stepwise communication counts are held fixed in the pool comparison. No experiment reports how accurately volume or participant identity is predicted. The evaluation is also narrow: one Chinese corpus with 7 test units (5 topics), Qwen models only, a single training seed and a single rollout for the main results, and no variance or confidence intervals. The paper does not say whether the baselines are fine-tuned on the same 44 training units as OpenSim. M2Sim, the closest prior method, appears only as a boundary setting in Table 3, not as a baseline in Table 1.
Questions for authors
- Please report Table 1 with the event-agnostic pool as the main setting, or explain why sampling from the event's own observed participants does not leak future information.
- How is implemented and trained? At test time, is used, or is it replaced by observed counts? How accurate is it? Is learned, or uniform as Table 5 suggests?
- Are Direct LLM, Social Retrieval and MF-LLM fine-tuned on the training units? Can you add M2Sim as a baseline in Table 1?
- Can you report variance across training seeds and rollouts, with confidence intervals, for Tables 1 to 3?
- (Suggestion) Results on a second dataset, or with a non-Qwen model family, would make the generality claim much more convincing.