- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Society of Programs (SOP), a hierarchical compilation framework for scaling LLM-based social simulations. The core idea is to compile repeated LLM decision-making into reusable computational structures through three stages. Experiments show that SOP can substantially reduce LLM calls and computational cost while maintaining similarity to Full–LLM trajectories. The paper presents an interesting perspective that large-scale LLM social simulation can be optimized by exploiting behavioral redundancy across agents and time.
Critical strengths
SOP introduces an interesting compilation perspective for scaling LLM-based social simulation. Instead of repeatedly invoking LLMs for individual agent decisions, it exploits redundancy across behavior, population, and temporal dimensions. Behavior compilation replaces LLM inference with reusable programs, region compilation replaces individual execution with shared population transition matrices, and event-driven propagation reduces redundant timestep-level computation. By connecting LLM behavior modeling with program synthesis, population-level aggregation, and temporal state propagation, SOP offers a promising direction for improving the scalability of long-horizon LLM-based social simulations.
Critical weaknesses
-
The methodology lacks sufficient details in the main paper, with many important components deferred to the appendix. The paper does not clearly explain key algorithmic choices, including prototype selection, region construction, and transition estimation. Moreover, it remains uncertain whether prototype discovery in the first-stage provides essential benefits or merely serves as an intermediate grouping step that could be replaced by directly learning programs and regions (Stage 2).
-
The static region assumption is insufficiently justified. In long-horizon social simulations, agent states and external contexts may evolve over time, potentially causing region drift and accumulated errors in the learned transition operators. Although this issue is briefly mentioned in the limitations, it is a fundamental assumption underlying the effectiveness of region compilation.
-
SOP effectively transforms LLM-based simulation into a region-level Markov process, but the approximation error introduced by this transformation is not sufficiently analyzed. Aggregating agents into regions may lose individual heterogeneity, historical dependency, and complex interaction effects. More analysis is needed to characterize when this approximation is valid and how errors accumulate over long simulation horizons.
-
The fidelity evaluation is limited. The experiments mainly rely on JSD and Exact Match against the Full–LLM trajectory, which may not fully capture whether important social dynamics are preserved. Additional evaluation metrics, more diverse datasets, and analyses of emergent behaviors or long-term dynamics would provide stronger evidence for the effectiveness of SOP.
-
The experimental comparison is incomplete. The evaluation lacks comparisons with recent scalable LLM-based social simulators, such as ScaleSim (ICML 2026) and GASim (ACL 2026). More comparisons are needed to clarify whether SOP provides advantages over or complements existing approaches.
Questions for authors
See Weaknesses
5 comments
Comment 2: The Static Region Assumption and Long-Term Evolution
Thank you for raising this important question about the validity of region compilation during long-term dynamic evolution. We agree that the conditions under which region compilation applies and its long-term behavior should be explained more clearly in the methodology and supported by direct experimental evidence.
First, “fixed regions” in SoP refer to a partition of the static attribute space. The distribution of dynamic states within each region continues to evolve throughout the simulation. When the external environment changes, the system also switches to the corresponding context-conditioned transition operator . Regions are constructed based on program responses across dynamic states and external contexts. The key condition for region compilation is therefore that members have identical or similar conditional transitions under the relevant states and contexts; it does not require individual states or the environment to remain unchanged.
For approximate acceleration methods based on surrogate models, representative individuals, or population aggregation, the propagation of local approximation errors during closed-loop evolution is a common issue that needs to be quantified and evaluated. SoP uses closed-loop behavioral fitting, response-aware region construction, and hierarchical error analysis to reduce and assess approximation errors under explicit applicability conditions and computational budgets, thereby establishing a verifiable fidelity–computation trade-off. In particular, under fixed operators and valid time blocks, temporal merging is numerically equivalent to step-by-step region propagation and introduces no additional temporal approximation error.
To further address this concern, we added a continuous long-horizon region stability experiment. We froze the existing behavioral programs and used propagation induced by each individual’s own attributes and program as the reference. Under the same initial conditions, activation semantics, and step-by-step output requirements, we compared the current static regions with two levels of nested region refinement to directly measure the additional error introduced by region aggregation. For WVS, we progressively extended the duration of all eight complete contexts to a total horizon of 512 steps. For Reddit, we evolved each of the 32 topics continuously for up to 2,048 steps, without resetting states within a topic. All configurations covered three independent compilation runs, and we report both mean and peak errors.
As shown in Table R2, the time-averaged TV for WVS at 512 steps is 0.00898, and the maximum per-step TV across the tested configurations is 0.036. For Reddit, all tested horizons remain numerically equivalent to individual-level expected propagation within a tolerance of . Further verification of each member’s operator confirms that the current Reddit regions satisfy the response-equivalence condition over the covered states and contexts, allowing this equivalence to hold as dynamic states continue to evolve. These results provide direct evidence of the behavior of fixed regions during continuous evolution and quantify the additional distributional deviation relative to individual-level program propagation.
Table R2: Continuous long-horizon distribution error of the current static regions relative to individual-level program propagation
| WVS continuous steps | Mean TV ↓ | Peak TV ↓ | Reddit continuous steps | Mean TV ↓ | Peak TV ↓ |
|---|---|---|---|---|---|
| 8 | 0.00354 | 0.02354 | 32 | ||
| 32 | 0.00586 | 0.03396 | 128 | ||
| 128 | 0.00750 | 0.03600 | 512 | ||
| 512 | 0.00898 | 0.03600 | 2,048 |
Note: TV directly compares the distribution produced by the current regions with that induced by each individual’s own program. Mean TV is first averaged over time and topics within each compilation and then averaged across the three compilations. Peak TV is the maximum across all compilations, time points, and topics. The 32 Reddit topics evolve continuously and independently, without state resets within a topic. All results concern region aggregation error under the complete set of covered contexts.
In the revised manuscript, we will clarify these applicability conditions in the main text and add Table R2 and continuous error curves to provide a clearer view of the long-term behavior of region compilation and the range of its errors. The complete experimental configurations, region-refinement comparisons, and results for each compilation will be included in the supplementary material for you and readers to examine.
Comment 3: Approximation Error and Validity Conditions of Region Aggregation
Thank you for pointing out that region aggregation may lose individual heterogeneity, historical dependence, and complex interactions. We agree that region propagation does not inherently preserve all individual dynamics. To clarify its validity conditions and quantify long-term error, we have completed a region-granularity sensitivity experiment and a continuous long-horizon stability experiment.
1. Validity conditions for region approximation
One sufficient condition is that the dynamic state adequately carries the information needed for decisions and that individuals within the same region have identical program-induced transitions over reachable states and covered contexts. When responses are heterogeneous, a fixed region operator may lose attribute–state associations that change with history. SoP reduces such differences through adaptive partitioning driven by program responses, rather than merging individuals solely on the basis of demographic similarity.
Both supplementary experiments fix the behavioral programs and use the same initial states, contexts, activation mechanisms, and step-by-step output requirements. They directly compare region propagation with propagation based on each individual’s own program using TV distance, thereby examining the distributional deviation introduced by aggregation.
2. Region-granularity experiment
On WVS, refining the partition from one region per prototype to the default partition reduces mean direct TV from 0.02063 to 0.00354, a reduction of approximately 82.9%. This difference includes the effects of region aggregation and differences between the calibration and deployment populations; an independent diagnostic that recomputes operators using deployment members also shows a consistent downward trend. In the main Reddit experiment, responses within the current regions are already equivalent, so further refinement increases computation without improving accuracy. This indicates that organizing computational units according to behavioral responses enables SoP to reduce unnecessary aggregation loss and avoid redundant refinement once responses are consistent.
3. Continuous long-horizon experiment
We extended the duration of the covered contexts to examine how aggregation error changes during continuous evolution. The table reports time-averaged direct TV aggregated over three compilations:
| Dataset | Continuous horizon | Mean direct TV |
|---|---|---|
| WVS | 8 steps | 0.00354 |
| WVS | 32 steps | 0.00586 |
| WVS | 128 steps | 0.00750 |
| WVS | 512 steps | 0.00898 |
| 2,048 steps within each topic |
After the continuous WVS horizon increases by a factor of 64, mean direct TV remains below 0.01, with a peak of 0.036 over the full horizon. Although aggregation error increases, its growth is much smaller than the proportional increase in the horizon, and region propagation continues to preserve the population distribution induced by individual programs reasonably well. The improvement from additional refinement is small, indicating that simply increasing the number of regions does not necessarily yield further fidelity gains.
Each Reddit topic is initialized separately, attitudes are not reset within a topic, and 21 joint attitude–evidence-memory states are retained. During continuous evolution of up to 2,048 steps, direct TV remains below , consistent with response equivalence within the current regions. This shows that such regions can maintain numerical consistency during long-term propagation.
Together, the two experiments show that SoP can keep distributional deviation low in the evaluated settings while reducing redundant propagation. This result directly supports the design of constructing regions based on program responses and demonstrates its practical advantage in the trade-off between fidelity and computational efficiency. In the revised manuscript, we will add these results and clarify that the reliability of region approximation depends on how dynamic states represent historical and interaction information, as well as on the consistency of conditional transitions within each region. Temporal merging preserves propagation equivalence for the same region model when the stability conditions are met, but it does not eliminate errors already introduced by earlier aggregation.
Thank you again for your constructive comments, which have helped us further improve our analysis of the validity of region approximation and our evaluation of long-term error.
Thank you for reading our work carefully and for your positive assessment of the research problem, method design, and experimental results. Below, we address each of your questions.
Comment 1: Methodological Details and the Role of Prototype Discovery
Thank you for your comments. We will move the descriptions of the core algorithms into the main text. Specifically, prototype selection uses controlled 2×2 pre-decision tests to rank attributes according to the weighted score 0.7CR+0.3MG, where CR is the category change rate and MG is the normalized magnitude of ordinal change. Attributes are selected according to the prototype budget and split at the median of the training population.
Regions are constructed after the behavioral programs have been frozen. For WVS, we greedily split regions according to the reduction in program-response impurity. For Reddit, we group individuals with identical response signatures across the covered states and contexts into equivalence classes. To estimate transitions, we execute the frozen programs and state-update rules locally and construct transitions from the frequencies of region members’ successor states. For response-equivalent regions, a representative member can be used for this calculation.
Prototypes organize behavioral learning, whereas regions compress the execution of frozen programs. However, their different roles alone do not establish that prototype discovery is indispensable. The existing A1 ablation retains group-based learning, so we added a Global + regions comparison: it skips prototype discovery, learns a global conditional program directly from pooled cases, and then constructs regions without prototype boundaries.
This comparison uses the same data splits and paired compilation seeds, with capacity and numerical evaluation limits set according to the aggregate budget of the full method. The table below reports program-based closed-loop replay results for three compilations, expressed as mean ± sample standard deviation. Exact Match measures agreement with the fixed LLM reference decisions.
Under the tested configurations, SoP achieves higher average individual-level agreement, while Global achieves lower average population-level JSD, indicating a fidelity trade-off between the two ways of organizing learning. We will position prototype discovery as a task-dependent decomposition of learning and an inductive bias under a limited budget, narrow our claims about its necessity, and discuss both its individual-level agreement and its trade-off in population distribution error in the main text.
Thank you for your constructive comments, which have helped us clarify the roles of the modules and more accurately characterize the empirical benefits and limits of prototype discovery.