- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes a replay protocol that freezes semantic opportunities before a simulated policy intervention. Policy branches share the frozen pool and keyed randomness, while activation, exposure, and reply targeting can change. The resulting fixed-semantics effect is distinguished from an adaptive effect that permits new state-dependent language generation. A factorial study finds that controversy ranking increases participation more under conflict-responsive behavior and can produce shallower reply trees. The paper studies channel deletions, alternative response forms, learned topology, semantic catalogs, and external platform data, with stronger external support for volume than depth.
Critical strengths
-
The protocol specifies what stays fixed across policy branches and what may change. The pre-treatment pool, separate consumption within each branch, keyed random streams and exhaustion rule make this distinction executable. Proposition 1 gives the corresponding unbiasedness result and states the covariance condition for a variance reduction. I find this useful for studies that want to separate structural policy response from newly generated semantic content.
-
The mechanism study uses a four-cell interaction rather than interpreting the joint behavior-plus-ranking contrast as an interaction. Channel deletions, volume-matched comparisons, and the learned-topology experiment provide several checks on the implemented explanation. Reporting depth reversals and the weak external depth evidence makes the scope of the result easier to assess.
Critical weaknesses
1. The motivation and advantages of fixed-semantics replay are not sufficiently established.
It is unclear why fixing semantic opportunities is particularly useful for studying LLM-agent societies, where semantic adaptation is an essential part of the interactions. While the proposed method provides more controlled experiments, it is not obvious that the resulting counterfactual effect is more meaningful than the effect measured through conventional adaptive simulation.
The paper does include evaluations of simulation fidelity, including distributional similarity, intervention prediction, and semantic compatibility. However, these results do not clearly establish in which aspects Rehydration is preferable to existing methods. In particular, it would be helpful to demonstrate a scientific insight that becomes possible specifically because of the proposed control.
2. The comparison with conventional methods and the interpretation of the empirical findings are not fully convincing.
The theoretical justification largely follows standard ideas from causal inference and paired simulation. In the synthetic experiment, the ground truth is defined as the fixed-semantics effect, so it is not surprising that Rehydration estimates this quantity more accurately than adaptive replay, which targets a different effect. The empirical comparison demonstrates reduced variance, but the practical implications of this improvement remain unclear.
Similarly, while the authors provide mechanism ablations and robustness tests for Shallow Swarm, it is difficult to determine how much of this phenomenon arises from the specific behavioral and ranking mechanisms implemented in the simulator. The external evidence supports increased participation more clearly than the reduction in discussion depth. I would like to better understand the generality and significance of this finding beyond the chosen simulation settings.
3. The presentation could be substantially improved.
Several important definitions and distinctions are difficult to follow. For example, the random variables and appear in Section 3.1 without a formal definition of their distributions or coupling across interventions. The presentation also moves between fixed-semantics and adaptive effects, additive and multiplicative interaction measures, and different simulation configurations without always making their relationships sufficiently explicit.
Questions for authors
See Weaknesses section
4 comments
Response to Reviewer KWBD: W2 (1) Theoretical Contribution, Synthetic Validation, and Practical Implications
Thank you for this important comment. We clarify the scope of our theoretical contribution, the role of the synthetic experiment, and the practical implications of our additional variance analyses.
1. Theoretical contribution.
We agree that the unbiasedness argument and common-random-number covariance identity follow established probability results. Our contribution is not a new pairing theorem, but an operational protocol for fixed-semantics counterfactual experiments in generative-agent simulations. Rehydration specifies how to generate and seal semantic proposals before intervention, share them across policy branches, maintain branch-local consumption states, couple structural randomness by operation, and prevent post-intervention semantic regeneration from changing the replay target.
2. Role of the synthetic experiment.
We agree that the synthetic experiment does not establish superiority over adaptive simulation. It verifies whether Rehydration recovers its declared fixed-semantics estimand under a known ground truth. Adaptive replay instead targets an effect that includes policy-induced semantic adaptation. Therefore, their difference reflects distinct estimands, rather than estimation error by adaptive replay. We use the synthetic experiment for estimator verification, not as a general ranking of the two workflows.
3. Practical implications of uncertainty and repeatability.
We conducted additional crossed-catalog, same-estimand pairing, and matched-budget comparisons to distinguish semantic-generation uncertainty from replay variability and workflow-level repeatability.
Table R3. Uncertainty decomposition and matched-budget repeatability.
| Comparison | Result | Interpretation |
|---|---|---|
| Between-catalog variance share | 34.9% [29.3%, 40.2%] | Catalog generation contributes additional uncertainty |
| Independent / paired variance ratio, same estimand | 1.015 [0.994, 1.038] | No clear precision gain from pairing |
| Original volume variance ratio | 4.89 | Original workflow comparison |
| Same threads and statistic, original comparison | 3.68 [3.25, 4.67] | Advantage persists after alignment |
| Matched-budget volume variance ratio | 3.07 [2.48, 3.36] | Lower volume-effect dispersion under freezing |
| Matched-budget depth variance ratio | 0.97 [0.84, 1.30] | No clear stability advantage |
For the crossed-catalog experiment, we used 25 threads, with 10 independently generated catalogs and 20 replay seeds per catalog. Catalog variation accounted for 34.9% of paired log-volume effect variance. Under the same fixed-semantics estimand and branch marginals, the independent-to-paired variance ratio was 1.015, indicating no clear precision gain from pairing itself.
For matched-budget repeatability, both workflows used 78 usable generation requests and 156 paired policy contrasts per workflow on the same 10 threads. We matched the model snapshot, temperature, and requested seeds, although actual token counts differed. The adaptive-to-frozen variance ratio was 3.07 [2.48, 3.36] for individual log-volume policy contrasts, with lower variance under freezing on all 10 threads. For leaf depth, the corresponding ratio was 0.97 [0.84, 1.30], showing no clear advantage.
The previously reported 9.61 [7.58, 12.37] measures the ratio of average per-thread variances of budget-averaged log-volume effects, a different statistic from the 3.07 comparison. It should not be interpreted as an increase from the original 4.89 ratio.
The practical implication is that researchers should allocate experimental budgets across independently generated semantic catalogs as well as replay seeds, rather than repeatedly sampling only one catalog. Rehydration also provides more repeatable volume-effect estimates under the matched workflow, although this advantage does not extend clearly to depth.
Overall, Rehydration contributes an operationally specified experimental control, verifies recovery of its declared estimand, and provides practical guidance for separating semantic-generation uncertainty from replay variability. It complements adaptive simulation by enabling a different, explicitly defined counterfactual analysis rather than replacing conventional adaptive effects.
Response to Reviewer KWBD: W (2) Shallow Swarm — Mechanism, Robustness, and Generality
Thank you for raising this concern. We first clarify what Shallow Swarm means in the original paper, and then report additional experiments examining its mechanisms and generality.
1. Original definition of Shallow Swarm.
In the original manuscript (Section 5.1, Figure 3), we define Shallow Swarm as a behavior–platform interaction: controversy-oriented ranking is nearly neutral for baseline populations but amplifies participation under conflict-responsive behavior, while the resulting reply structures become shallower. Thus, it is not simply an increase in comment volume, but a joint pattern involving participation and discussion structure. Section 5.2 further investigates the mechanisms behind this pattern.
2. Mechanism separation.
On 79 held-out Reddit threads, the original policy reproduced both comment-volume amplification (1.230 [1.206, 1.253]) and mean leaf-depth compression (−1.517 [−2.615, −0.366]). Matching simulated trajectories at a common comment-count prefix retained the negative depth interaction (−1.692 [−2.906, −0.409]), indicating that volume differences alone do not explain depth compression.
An eight-cell activation-by-fatigue-by-ranking ablation further distinguished the behavioral pathways.
Table R4. Mechanism ablation and depth-tail robustness.
| Condition | Volume interaction [95% CI] | Depth interaction [95% CI] |
|---|---|---|
| Activation varied; fatigue off | 1.234 [1.228, 1.240] | +0.013 [−0.010, +0.038] |
| Activation fixed on; fatigue varied | — | −0.327 [−0.353, −0.299] |
| Common depth cap, 97-thread split | 1.240 [1.236, 1.244] | −0.332 [−0.373, −0.291] |
These results suggest that conflict-responsive activation primarily contributes to volume amplification, whereas depth fatigue contributes to the negative depth interaction in the learned-topology implementation. The common depth cap reduced maximum-depth normalized Wasserstein distance from 3.263 to 1.170, while both interactions persisted.
3. Dependence on decision policies.
We replaced the original hand-designed decision rule with alternative selectors to examine whether the joint pattern was specific to that rule.
Table R5. Fixed-pool decision-policy comparisons.
| Decision policy | Threads | Volume interaction [95% CI] | Depth interaction [95% CI] |
|---|---|---|---|
| Original rule | 79 | 1.230 [1.206, 1.253] | −1.517 [−2.615, −0.366] |
| Earlier learned selector | 79 | 1.000 [0.987, 1.012] | −0.155 [−0.481, +0.150] |
| Qwen-calibrated bounded selector | 50 | 1.057 [1.015, 1.098] | −0.112 [−0.181, −0.049] |
| Live-Qwen pilot | 12 | 0.691 [0.600, 0.800] | −0.287 [−0.389, −0.183] |
The earlier learned selector reproduced neither interaction. The exploratory 50-thread selector reproduced both, whereas the 12-thread Live-Qwen pilot retained negative depth interaction but not positive volume amplification. Thus, the joint pattern is not exclusive to the original hand-designed rule, but is not guaranteed under autonomous LLM decisions.
4. External evidence and generality.
| Evidence | Finding | Interpretation |
|---|---|---|
| Reddit conflict–volume regression | 10.375 [3.815, 16.982] | Supports conflict–participation association |
| 1,415-thread lexical-proxy analysis | No established depth compression | Does not validate the depth pathway |
| HN/Lemmy observational samples | No established depth compression | Limited external support for depth effects |
| r/anime ranking audit | No Best-versus-Controversy comparison | Target ranking effect not identified |
The observational evidence supports an association between early conflict and subsequent participation, but does not directly establish ranking-conditioned depth compression.
Overall, we characterize Shallow Swarm as a conditional behavior–platform interaction, not a universal empirical law. Its volume amplification and depth compression arise through distinct mechanisms, depend on behavioral and decision policies, and have different levels of external support. Its significance is to reveal why participation and discussion structure must be evaluated jointly: increased activity does not necessarily imply deeper discussion.
Response to Reviewer KWBD W3 : Definitions — Randomness, Contrasts, and Effect Scales
Thank you for identifying these presentation issues. We agree that the mathematical definitions and relationships among experimental quantities should be explicit in the main text rather than left implicit in the implementation. We will revise Section 3.1 and the experimental sections accordingly.
(1) Structural randomness and coupling.
Conditional on the initial context and frozen semantic catalog, each base structural draw is modeled as an independent Uniform(0,1) variable across distinct operation keys. The symbols and denote the collections of these draws for the two policy arms, rather than single scalar errors.
Paired replay uses the same draw for each matching operation key in both arms, whereas independent replay uses separate draws with identical marginal distributions. In the 79-thread implementation, keys include the post-level seed, simulation step, agent, and operation, but exclude the decision method, behavioral condition, and ranking. Thus, an abstention in one branch does not shift subsequent matching draws. Categorical choices transform shared base draws using branch-specific probabilities, so shared randomness does not force identical actions.
(2) . Relationships among estimands, contrasts, and simulation configurations.
We will explicitly distinguish these quantities in the main text (Table R5).
Table R5. Distinctions needed to interpret the policy comparisons.
| Distinction | Meaning |
|---|---|
| Fixed-semantics vs. adaptive effect | Shared pregenerated semantic proposals vs. policy-dependent semantic generation |
| Joint contrast vs. interaction | Changing behavior and ranking together vs. measuring whether the ranking effect differs across behavioral conditions |
| Additive depth interaction | Difference between ranking-induced depth changes across behavioral conditions; measured in layers, with null value 0 |
| Multiplicative volume interaction | Ratio of ranking effects across behavioral conditions; with null value 1 |
| Simulation configuration | The decision, exposure, timing, and other rules defining each experimental comparison |
Let and denote the corresponding depth and volume outcomes, where indexes behavioral conditions and indexes ranking conditions.
The interactions are defined as follows:
Thus, the additive depth interaction has a null value of zero, whereas the multiplicative volume interaction has a null value of one. For volume, we calculate the interaction on the scale and exponentiate the resulting contrast, yielding a ratio-of-ratios rather than a ratio of aggregate arithmetic means.
We will also add a setup-and-contrast key alongside the main results, distinguishing local robustness tests, global phase-boundary analyses, and policy-replacement experiments. These configurations differ in their decision rules and execution settings and will not be presented as interchangeable replications.
These revisions will make the relationship between each research question, its estimand, effect scale, and simulation configuration explicit, allowing readers to interpret the mathematical formulation and empirical results consistently.
Response to Reviewer KWBD: W1
Thank you for asking what scientific insight the proposed control provides. We do not argue that fixed-semantics replay is universally more meaningful than adaptive simulation. Rather, it addresses a complementary question: how do policies change conversational outcomes when semantic opportunities are held constant?
1. Why fix semantic opportunities?
The key distinction is between a policy changing which replies are generated and changing which available replies are actually realized. Conventional adaptive simulation captures both effects, since different policy-induced states may lead to different generated content. Rehydration holds semantic opportunities constant across policy branches, allowing us to isolate policy-dependent changes in realized interactions under a shared semantic reference.
This distinction matters for moderation. Hiding a comment may prevent not only that comment from being seen but also subsequent replies to it. In adaptive simulation, policy branches may generate different semantic content, making it difficult to attribute differences in subsequent conversations to the moderation rule alone. Fixed-opportunity replay makes these downstream effects traceable independently of newly generated content. It complements rather than replaces adaptive simulation.
2. A scientific insight enabled by this control.
We conducted a five-thread multi-round moderation experiment to ask: Can two policies produce nearly identical activity levels but different conversations?
Each thread received two independently generated, parent-bound catalogs, each containing six first-layer opportunities and twelve possible continuations. All 180 reply texts and parent links were sealed before policy assignment.
Within each catalog and paired seed, both policy branches shared the initial state, frozen opportunities, participation draws, and moderation allowance. Only the moderation rule changed: coarse-label moderation used a shared hash to break ties among proposals with the same antagonism label, whereas semantic moderation prioritized replies classified as containing attacks. Both policies could hide up to two replies after the first six opportunities, with matched realized allowances. A reply could be published only if its fixed parent existed and remained visible. Phase II published the sealed text verbatim, with no new generation or rewriting.
Across five threads, two catalogs, two policies, and ten paired seeds, we obtained 200 trajectories (100 matched policy pairs). Both policies hid 200 reply instances and published nearly identical totals (954 vs. 952), yet 27/100 pairs produced different realized reply sets, across three threads. All 180 distinct replies were published at least once. Independent Qwen and DeepSeek evaluations yielded minimum acceptability rates of 94.44% and 98.33%, respectively.
This shows that similar activity levels do not imply conversational equivalence.
To explain why these differences arise, we held each realized action history fixed and changed only the visibility mask, separating immediate masking from subsequent interaction-path effects.
Table R1. Exposure contrasts and fixed-history decomposition.
Note. Contrasts are Semantic minus Coarse-label moderation. Exposure is the cumulative count of visible new replies over 18 scheduled opportunities, divided by 18, not a percentage. Acceptable rebuttals must satisfy the respective evaluator's minimum acceptability criterion. Intervals use paired bootstrap resampling of five threads and are descriptive; catalogs and seeds are repeated measurements. The decomposition is a simulator diagnostic, not a real-world mediation estimate.
The subsequent path component reduced acceptable-rebuttal exposure under both evaluators, suggesting that hiding a comment can also prevent substantive replies to it. Although total rebuttal-exposure effects differed across evaluators, this analysis reveals how similar activity totals can conceal different conversational continuations. Rehydration makes these policy-dependent effects traceable under shared semantic opportunities, providing a content-level mechanism analysis complementary to adaptive simulation.