acceptodds
Under review as a conference paper at ICLR 2027

Society of Programs: Compiling LLM Behavior into Scalable Social Dynamics

Abstract

Imagine a social simulation in which the same kind of decision is made thousands of times: by different agents, under recurring contexts, and across successive timesteps. Asking an LLM to recompute each decision independently makes population-scale simulation expensive even when much of the induced behavior is reusable. We present Society of Programs (SoP), a hierarchical compiler that turns reference-LLM behavior into reusable social dynamics. SoP first compiles task-relevant behavior into executable prototype programs, then groups agents by the responses of these frozen programs and propagates their states through region-level transition operators, and finally merges consecutive updates whenever the propagation conditions remain unchanged. This changes the computational unit from repeated agent-level inference to reusable computation over programs, populations, and temporal blocks. Across WVS and Reddit social simulations, SoP preserves high fidelity to the Full–LLM reference while reducing redundant inference and propagation. In a multi-round opinion simulation with agents, it reduces LLM calls by approximately and build-plus-deployment time by approximately relative to Full–LLM. The gains arise from behavioral regularities that can be compiled and reused; temporal compression is applied only across stable intervals and required output boundaries.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM-based social simulation, scalable simulation, computational efficiency, agent based system

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_QHHQOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Society of Programs (SOP), a hierarchical compilation framework for scaling LLM-based social simulations. The core idea is to compile repeated LLM decision-making into reusable computational structures through three stages. Experiments show that SOP can substantially reduce LLM calls and computational cost while maintaining similarity to Full–LLM trajectories. The paper presents an interesting perspective that large-scale LLM social simulation can be optimized by exploiting behavioral redundancy across agents and time.

Critical strengths

SOP introduces an interesting compilation perspective for scaling LLM-based social simulation. Instead of repeatedly invoking LLMs for individual agent decisions, it exploits redundancy across behavior, population, and temporal dimensions. Behavior compilation replaces LLM inference with reusable programs, region compilation replaces individual execution with shared population transition matrices, and event-driven propagation reduces redundant timestep-level computation. By connecting LLM behavior modeling with program synthesis, population-level aggregation, and temporal state propagation, SOP offers a promising direction for improving the scalability of long-horizon LLM-based social simulations.

Critical weaknesses
  1. The methodology lacks sufficient details in the main paper, with many important components deferred to the appendix. The paper does not clearly explain key algorithmic choices, including prototype selection, region construction, and transition estimation. Moreover, it remains uncertain whether prototype discovery in the first-stage provides essential benefits or merely serves as an intermediate grouping step that could be replaced by directly learning programs and regions (Stage 2).

  2. The static region assumption is insufficiently justified. In long-horizon social simulations, agent states and external contexts may evolve over time, potentially causing region drift and accumulated errors in the learned transition operators. Although this issue is briefly mentioned in the limitations, it is a fundamental assumption underlying the effectiveness of region compilation.

  3. SOP effectively transforms LLM-based simulation into a region-level Markov process, but the approximation error introduced by this transformation is not sufficiently analyzed. Aggregating agents into regions may lose individual heterogeneity, historical dependency, and complex interaction effects. More analysis is needed to characterize when this approximation is valid and how errors accumulate over long simulation horizons.

  4. The fidelity evaluation is limited. The experiments mainly rely on JSD and Exact Match against the Full–LLM trajectory, which may not fully capture whether important social dynamics are preserved. Additional evaluation metrics, more diverse datasets, and analyses of emergent behaviors or long-term dynamics would provide stronger evidence for the effectiveness of SOP.

  5. The experimental comparison is incomplete. The evaluation lacks comparisons with recent scalable LLM-based social simulators, such as ScaleSim (ICML 2026) and GASim (ACL 2026). More comparisons are needed to clarify whether SOP provides advantages over or complements existing approaches.

Questions for authors

See Weaknesses

5 comments

Thank you for reading our work carefully and for your positive assessment of the research problem, method design, and experimental results. Below, we address each of your questions.

Comment 1: Methodological Details and the Role of Prototype Discovery

Thank you for your comments. We will move the descriptions of the core algorithms into the main text. Specifically, prototype selection uses controlled 2×2 pre-decision tests to rank attributes according to the weighted score 0.7CR+0.3MG0.7\mathrm{CR} + 0.3\mathrm{MG}, where CR is the category change rate and MG is the normalized magnitude of ordinal change. Attributes are selected according to the prototype budget and split at the median of the training population.

Regions are constructed after the behavioral programs have been frozen. For WVS, we greedily split regions according to the reduction in program-response impurity. For Reddit, we group individuals with identical response signatures across the covered states and contexts into equivalence classes. To estimate transitions, we execute the frozen programs and state-update rules locally and construct transitions from the frequencies of region members’ successor states. For response-equivalent regions, a representative member can be used for this calculation.

Prototypes organize behavioral learning, whereas regions compress the execution of frozen programs. However, their different roles alone do not establish that prototype discovery is indispensable. The existing A1 ablation retains group-based learning, so we added a Global + regions comparison: it skips prototype discovery, learns a global conditional program directly from pooled cases, and then constructs regions without prototype boundaries.

This comparison uses the same data splits and paired compilation seeds, with capacity and numerical evaluation limits set according to the aggregate budget of the full method. The table below reports program-based closed-loop replay results for three compilations, expressed as mean ± sample standard deviation. Exact Match measures agreement with the fixed LLM reference decisions.

DatasetSoP JSD ↓Global JSD ↓SoP Exact Match (%) ↑Global Exact Match (%) ↑
WVS0.1289 ± 0.05840.1094 ± 0.064167.02 ± 2.9459.95 ± 7.91
Reddit0.2658 ± 0.01550.2517 ± 0.024448.08 ± 0.3645.25 ± 3.30

Under the tested configurations, SoP achieves higher average individual-level agreement, while Global achieves lower average population-level JSD, indicating a fidelity trade-off between the two ways of organizing learning. We will position prototype discovery as a task-dependent decomposition of learning and an inductive bias under a limited budget, narrow our claims about its necessity, and discuss both its individual-level agreement and its trade-off in population distribution error in the main text.

Thank you for your constructive comments, which have helped us clarify the roles of the modules and more accurately characterize the empirical benefits and limits of prototype discovery.

Comment 2: The Static Region Assumption and Long-Term Evolution

Thank you for raising this important question about the validity of region compilation during long-term dynamic evolution. We agree that the conditions under which region compilation applies and its long-term behavior should be explained more clearly in the methodology and supported by direct experimental evidence.

First, “fixed regions” in SoP refer to a partition of the static attribute space. The distribution of dynamic states within each region continues to evolve throughout the simulation. When the external environment changes, the system also switches to the corresponding context-conditioned transition operator PR(et)P_R^{(e_t)}. Regions are constructed based on program responses across dynamic states and external contexts. The key condition for region compilation is therefore that members have identical or similar conditional transitions under the relevant states and contexts; it does not require individual states or the environment to remain unchanged.

For approximate acceleration methods based on surrogate models, representative individuals, or population aggregation, the propagation of local approximation errors during closed-loop evolution is a common issue that needs to be quantified and evaluated. SoP uses closed-loop behavioral fitting, response-aware region construction, and hierarchical error analysis to reduce and assess approximation errors under explicit applicability conditions and computational budgets, thereby establishing a verifiable fidelity–computation trade-off. In particular, under fixed operators and valid time blocks, temporal merging is numerically equivalent to step-by-step region propagation and introduces no additional temporal approximation error.

To further address this concern, we added a continuous long-horizon region stability experiment. We froze the existing behavioral programs and used propagation induced by each individual’s own attributes and program as the reference. Under the same initial conditions, activation semantics, and step-by-step output requirements, we compared the current static regions with two levels of nested region refinement to directly measure the additional error introduced by region aggregation. For WVS, we progressively extended the duration of all eight complete contexts to a total horizon of 512 steps. For Reddit, we evolved each of the 32 topics continuously for up to 2,048 steps, without resetting states within a topic. All configurations covered three independent compilation runs, and we report both mean and peak errors.

As shown in Table R2, the time-averaged TV for WVS at 512 steps is 0.00898, and the maximum per-step TV across the tested configurations is 0.036. For Reddit, all tested horizons remain numerically equivalent to individual-level expected propagation within a tolerance of 10−1210^{-12}. Further verification of each member’s operator confirms that the current Reddit regions satisfy the response-equivalence condition over the covered states and contexts, allowing this equivalence to hold as dynamic states continue to evolve. These results provide direct evidence of the behavior of fixed regions during continuous evolution and quantify the additional distributional deviation relative to individual-level program propagation.

Table R2: Continuous long-horizon distribution error of the current static regions relative to individual-level program propagation

WVS continuous stepsMean TV ↓Peak TV ↓Reddit continuous stepsMean TV ↓Peak TV ↓
80.003540.0235432<10−12<10^{-12}<10−12<10^{-12}
320.005860.03396128<10−12<10^{-12}<10−12<10^{-12}
1280.007500.03600512<10−12<10^{-12}<10−12<10^{-12}
5120.008980.036002,048<10−12<10^{-12}<10−12<10^{-12}

Note: TV directly compares the distribution produced by the current regions with that induced by each individual’s own program. Mean TV is first averaged over time and topics within each compilation and then averaged across the three compilations. Peak TV is the maximum across all compilations, time points, and topics. The 32 Reddit topics evolve continuously and independently, without state resets within a topic. All results concern region aggregation error under the complete set of covered contexts.

In the revised manuscript, we will clarify these applicability conditions in the main text and add Table R2 and continuous error curves to provide a clearer view of the long-term behavior of region compilation and the range of its errors. The complete experimental configurations, region-refinement comparisons, and results for each compilation will be included in the supplementary material for you and readers to examine.

Comment 3: Approximation Error and Validity Conditions of Region Aggregation

Thank you for pointing out that region aggregation may lose individual heterogeneity, historical dependence, and complex interactions. We agree that region propagation does not inherently preserve all individual dynamics. To clarify its validity conditions and quantify long-term error, we have completed a region-granularity sensitivity experiment and a continuous long-horizon stability experiment.

1. Validity conditions for region approximation

One sufficient condition is that the dynamic state adequately carries the information needed for decisions and that individuals within the same region have identical program-induced transitions over reachable states and covered contexts. When responses are heterogeneous, a fixed region operator may lose attribute–state associations that change with history. SoP reduces such differences through adaptive partitioning driven by program responses, rather than merging individuals solely on the basis of demographic similarity.

Both supplementary experiments fix the behavioral programs and use the same initial states, contexts, activation mechanisms, and step-by-step output requirements. They directly compare region propagation with propagation based on each individual’s own program using TV distance, thereby examining the distributional deviation introduced by aggregation.

2. Region-granularity experiment

On WVS, refining the partition from one region per prototype to the default partition reduces mean direct TV from 0.02063 to 0.00354, a reduction of approximately 82.9%. This difference includes the effects of region aggregation and differences between the calibration and deployment populations; an independent diagnostic that recomputes operators using deployment members also shows a consistent downward trend. In the main Reddit experiment, responses within the current regions are already equivalent, so further refinement increases computation without improving accuracy. This indicates that organizing computational units according to behavioral responses enables SoP to reduce unnecessary aggregation loss and avoid redundant refinement once responses are consistent.

3. Continuous long-horizon experiment

We extended the duration of the covered contexts to examine how aggregation error changes during continuous evolution. The table reports time-averaged direct TV aggregated over three compilations:

DatasetContinuous horizonMean direct TV
WVS8 steps0.00354
WVS32 steps0.00586
WVS128 steps0.00750
WVS512 steps0.00898
Reddit2,048 steps within each topic<10−12< 10^{-12}

After the continuous WVS horizon increases by a factor of 64, mean direct TV remains below 0.01, with a peak of 0.036 over the full horizon. Although aggregation error increases, its growth is much smaller than the proportional increase in the horizon, and region propagation continues to preserve the population distribution induced by individual programs reasonably well. The improvement from additional refinement is small, indicating that simply increasing the number of regions does not necessarily yield further fidelity gains.

Each Reddit topic is initialized separately, attitudes are not reset within a topic, and 21 joint attitude–evidence-memory states are retained. During continuous evolution of up to 2,048 steps, direct TV remains below 10−1210^{-12}, consistent with response equivalence within the current regions. This shows that such regions can maintain numerical consistency during long-term propagation.

Together, the two experiments show that SoP can keep distributional deviation low in the evaluated settings while reducing redundant propagation. This result directly supports the design of constructing regions based on program responses and demonstrates its practical advantage in the trade-off between fidelity and computational efficiency. In the revised manuscript, we will add these results and clarify that the reliability of region approximation depends on how dynamic states represent historical and interaction information, as well as on the consistency of conditional transitions within each region. Temporal merging preserves propagation equivalence for the same region model when the stability conditions are met, but it does not eliminate errors already introduced by earlier aggregation.

Thank you again for your constructive comments, which have helped us further improve our analysis of the validity of region approximation and our evaluation of long-term error.

Reviewer_w3mkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Society of Programs (SoP), a hierarchical compilation framework for reducing redundant LLM inference in large-scale social simulations. SoP consists of three stages: compiling LLM behaviors into executable programs, grouping agents into regions and constructing region-level state-transition operators, and merging consecutive timesteps to further reduce state updates. Experiments on WVS and Reddit show that SoP substantially reduces computational costs while approximating Full-LLM simulation results. In experiments with 100,000 agents, SoP reduces the number of LLM calls by approximately 5,163×.

Critical strengths
  1. Practical problem. The paper addresses redundant inference in large-scale LLM social simulations and aims to improve efficiency at the computational pipeline level.
  2. Clear framework design. The method progressively reduces computation through behavior compilation, region-level state propagation, and temporal merging, with a clear objective for each stage.
  3. Significant efficiency gains. Experiments on WVS and Reddit demonstrate substantial efficiency improvements, while large-scale evaluations and ablation studies examine the benefits of the proposed method and its individual components.
Critical weaknesses
  1. Potential loss of LLM-specific capabilities and insufficient generalization evaluation. SoP compiles LLM decisions into fixed programs and state-transition matrices, potentially limiting its ability to understand novel events, capture complex individual behaviors, and model multi-agent interactions. Experiments on complex interactions and unseen events are needed to assess its generalization and whether it preserves the unique advantages of LLM-based simulation.
  2. Insufficient analysis of the trade-off between Region granularity and fidelity. Larger Regions reduce computation but may obscure individual behavioral differences, while smaller Regions increase computational costs. The paper lacks a systematic evaluation of how Region granularity affects simulation fidelity and efficiency.
  3. Lack of comparison with simpler population modeling methods. Social decisions may be dominated by a few key variables. Identifying these variables, grouping agents accordingly, and directly learning group-level transitions from Full-LLM trajectories may achieve similar results without SoP’s multi-stage compilation. A baseline using key-variable-based grouping and transition modeling would help establish whether the proposed framework justifies its additional complexity.
  4. Limited practical benefits of Temporal Merging under certain conditions. Temporal Merging requires stable transition conditions and infrequent intermediate outputs. Appendix D.3, Table 5, shows no reduction in updates when every Reddit timestep must be reported. Even with outputs every eight steps, an 87.5% reduction in updates yields only a 3.50× speedup, raising questions about its benefits in dynamic simulations requiring frequent observations.
Questions for authors

See weaknesses.

3 comments

Thank you for reading our work carefully and for your positive assessment of the research problem, method design, and experimental results. Below, we address each of your questions.

Comment 1: Potential Loss of LLM-Specific Capabilities and Insufficient Evaluation of Generalization

Thank you for raising this important issue. We understand your concern about SoP’s ability to respond to new events and interactions. We would like to clarify that SoP fixes the decision mapping, rather than the agents’ inputs, states, or decision outcomes. The compiled programs still read individual information, current events, and historical states. As these inputs change, the same program can make different decisions.

In scenarios involving interactions, interaction information can also be incorporated into behavior compilation. For example, the distribution of neighbors’ attitudes, received messages, the states of interaction partners, and past interactions can be included as program inputs according to the needs of the task. When other agents’ behavior changes, this information is updated and can subsequently influence the current agent’s decisions. Therefore, as long as decision-relevant information can be sufficiently represented through inputs and dynamic states, behavior compilation can use a unified form to handle scenarios both with and without interactions. SoP reuses the process of computing decisions from this information, rather than directly repeating a decision obtained previously.

The same applies to the transition matrix. It represents the transition relationship under a given context and state representation; it does not require the same matrix to be used throughout the entire simulation. For new contexts that the program can represent, the corresponding transition operator can be computed using the frozen program. When events or interaction feedback change the propagation conditions, the operator and temporal block boundaries also need to be updated accordingly. Temporal merging is performed only over intervals in which the conditions remain stable, thereby preserving the computations needed for event changes and feedback updates.

To further examine whether existing programs can be reused in new contexts, we conducted an additional small-scale experiment. The experiment reused existing Full-LLM reference trajectories and fixed three behavior compilation artifacts. We evaluated them on 32 Reddit test posts that did not overlap with the training posts, with 100 individuals and 1,024 timesteps. The reference decisions for the test posts were used only for final evaluation, and the transition operators for the new contexts were computed locally using the frozen programs.

Table R1: Program reuse on held-out Reddit posts

Execution methodJSD ↓Individual agreement (%) ↑
Individual closed-loop program replay0.2658 ± 0.015548.08 ± 0.36
Regional propagation: original setting0.2686 ± 0.0143—
Regional propagation: additional setting0.2686 ± 0.0143—

Note: Values are the mean ± sample standard deviation across three compilations. The original setting constructs regions using known deployment contexts; the additional setting fixes the regional partition before encountering the test contexts. JSD covers all timesteps. Individual agreement is reported only for program replay and is calculated over decisions that were actually activated. Program replay uses the recorded activation schedule, whereas regional propagation uses Bernoulli activation with q=0.125q=0.125. These are different execution paths, so their difference in JSD cannot be directly interpreted as regional approximation error. “—” means not applicable.

The additional setting achieves the same JSD as the original setting without any new LLM calls. This indicates that, under the current test setting, existing programs can continue to be used in new contexts while maintaining their original predictive performance.

Given the time available for this revision, we have not yet repeated a larger-scale full-LLM simulation or conducted a systematic comparison, so we began with the small-scale validation described above. These results provide preliminary evidence of reuse in new contexts within the tested task. Future work should expand the population and range of events and introduce richer endogenous interactions to systematically evaluate SoP’s behavioral fidelity, generalization, and computational benefits in these settings.

Comment 2: Insufficient Analysis of the Trade-off Between Regional Granularity and Fidelity

Thank you for pointing this out. The original manuscript did not sufficiently explain what level of regional granularity is appropriate. We therefore conducted an additional sensitivity experiment in which we kept the learned behavior programs and other simulation settings fixed, changed only the regional granularity, and required all configurations to output results at every timestep. The main results are as follows:

ExperimentRegional settingMean TV difference from individual program propagationPropagation updates
WVS main experimentOne region per prototype → original default partition0.02063 → 0.00354Approximately 37 → 147 on average
Reddit main experiment8 regions → redundant subdivision into 24–32 regionsBoth below 10−1210^{-12}8,192 → 24,576–32,768
Additional Reddit analysis with heterogeneous artifactsPer-prototype budget of 1 / 2 / 40.03602 / 0.01333 / close to zeroApproximately 476 / 640 / 740 on average

The mean JSD relative to Full-LLM on WVS also decreased from 0.1427 to 0.1295. Since the TV difference also reflects differences between the calibration and deployment population compositions, we conducted a recalibration diagnostic and likewise observed benefits from finer partitioning. The main Reddit experiment shows that, when responses within the original regions are already consistent, further subdivision only increases overhead. These results support choosing regional granularity according to differences in program responses: overly coarse partitions may mask behavioral differences, while the benefits of finer partitioning can reach a saturation point. We will add this analysis to the revised manuscript. Thank you for your constructive feedback. It will help us present the trade-off among regional granularity, simulation fidelity, and computational cost more clearly.

Comment 3: Lack of Comparison with Simpler Population Modeling Methods

Thank you for your suggestion. We added a group-transition baseline and further examined four lightweight statistical models that fit decision probabilities directly from existing Full-LLM trajectories. To examine whether these methods can preserve individual-level behavior, we used an autonomous closed-loop evaluation in which each method continues its rollout based on its own generated historical states. The table below reports individual decision agreement, measured by exact match (EM).

MethodWVS EM ↑Reddit EM ↑
SoP68.98%61.67%
Ordered Logistic61.68%35.93%
Ordered Probit62.05%35.05%
Multinomial Logistic63.95%36.01%
Quadratic additive ordinal model65.12%34.13%

Under the settings shown in the table, SoP achieves higher individual agreement on both datasets. The central value of multi-stage compilation is that it connects behavioral representation with scalable execution: prototype discovery and rule induction organize behavioral differences, program synthesis converts them into executable and reusable state-dependent decision logic, and regional aggregation and temporal merging further reduce repeated computation during deployment. This division of work allows what is learned about behavior to be reused throughout large-scale simulations. We therefore emphasize the joint design of individual behavior modeling and execution efficiency as our contribution. The individual agreement results above provide additional evidence, while the full group-transition comparison and the cost results in the paper together define its advantages and scope of applicability.

Comment 4: Limited Practical Value of Temporal Merging Under Certain Conditions

We agree that the benefits of temporal merging depend on the simulation scenario. We originally considered this approach because agents do not necessarily act at the same pace in different scenarios. Some simulations are organized into rounds, with every individual required to make a decision in each round. For example, the Full-LLM reference simulation used in APS updates all individuals round by round; in public goods games, there are also settings in which all participants decide how much to contribute in every round. APS https://arxiv.org/html/2605.27419v1 and Corrupted by Reasoning https://arxiv.org/html/2506.23276v2 both contain such examples.

The pace of social media scenarios is somewhat different. In reality, not everyone is online all the time, and when a piece of information appears, not everyone sees it at the same time. Some people see it early, some see it later, and others never encounter it. OASIS accounts for this difference by using users’ activity probabilities to determine when to activate agents, rather than activating everyone at every timestep. https://arxiv.org/html/2411.11581v4#S2.SS5

This difference led us to consider whether, in addition to reducing the cost of individual decisions, there might also be room to optimize repeated progression over time. Of course, having only some agents activated does not necessarily mean that timesteps can be merged. Merging also requires the propagation conditions and activation mechanism to remain stable during the interval, with no intermediate queries or interventions that must be preserved. When these conditions are met, we can directly compute the result of evolution over the interval. If every timestep must be observed, or if the environment changes continuously, the simulation must still proceed step by step. Therefore, the dense-observation scenario you identified is indeed one in which the benefits of this component are limited.

For us, temporal merging is a conditional additional optimization. Our primary motivation is to transform the decision process, which originally requires individual LLM calls, into program computations that can be executed repeatedly, and then use regional sharing to reduce repeated computation across individuals. These two benefits do not require a lower observation frequency. Building on them, temporal merging further exploits stable intervals that exist in some scenarios; how much additional benefit it provides depends on the specific workload. We will clarify this positioning in the paper.

We hope these clarifications and additional analyses address your concerns about generalization, regional granularity, comparisons with simpler population modeling methods, and the practical value of temporal merging. We will incorporate the corresponding explanations and results into the revised manuscript, together with a clearer discussion of the scope and limitations of our findings.

Thank you again for your careful reading and constructive feedback. Your comments have helped us identify where the presentation and evaluation can be strengthened, and we appreciate the opportunity to improve the paper.

Reviewer_htz9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SoP, which reduces computational redundancy of large-scale LLM social simulation. It transforms reference-LLM behavior into reusable social dynamic programs. Region compilation and event-driven propagation partition agents and merges consecutive timesteps, respectively, and further reduce LLM calls. Efficiency is evaluated using two case studies, and ablation study is conducted to evaluate the effect of the components.

Overall, the targeted problem is important and the proposed approach is interesting. Experimental results are also promising. The paper is also well-written and easy to read. On the downside, there are several issues on the design of the approach and the experiments, which are stated in the "Critical Weaknesses" section.

Critical strengths

S1. Large-scale LLM-based simulation is inefficient and there is plenty of room for optimization.

S2. Treating simulation cost as a compilation problem is clean and well-motivated.

S3. The proposed approach has two components, which partition agents and merges consecutive timesteps, respectively, to further improve efficiency.

S4. Experiments are extensive and report promising results, with ablation studies conducted and multiple compiler LLMs evaluated to demonstrate that the performance gain can be transferred to many LLMs.

Critical weaknesses

W1. Inter-agent interaction is not modeled in the compiler. So it is unclear why the social dynamics, rather than individual behavior, can be captured by the proposed approach. It is unknown whether the two evaluated scenarios, WVS and Reddit, are interaction-rich. If they are, this point needs further investigation, i.e., why modeling individual behavior through the proposed compilation approach also handles interaction patterns well.

W2. The scaling experiment reports only cost but not fidelity. The fidelity also matters because adding new agents (as well as new interactions) may affect it.

W3. While RQ4 in the experiments vary compiler LLMs, the paper does not report the the result (both fidelity and cost) when the reference LLM is changed, especially when we vary the model size for the reference LLM, e.g., the models in the 7B and 30B range, up to 763B (DeepSeek-4.1-Flash). Using smaller reference LLMs may reduce the overall speedup, if DeepSeek-4.1-Flash is still used as the compiler LLM.

Questions for authors

Q1. The scalability test on Reddit varies only horizon (T) and uses N = 100. What if we vary population (N) on Reddit?

3 comments

Thank you for reading our work carefully and for your positive assessment of the research problem, method design, and experimental results. Below, we address each of your questions.

W1. Interactions Between Agents

Thank you for raising this question. We would like to clarify that SoP compiles an agent’s conditional decision rules given its own attributes, dynamic state, and external information. In interactive scenarios, external information can include the behavior of other agents, neighbors’ opinions, received messages, or the state of the population. When such information affects the reference LLM’s decisions, the corresponding variables should also be included as inputs to the compiled program, allowing the compilation process to learn how agents respond to social information.

Therefore, compiling individual decision rules does not imply that agents are assumed to be independent. When applied to interactive scenarios, social information needs to be continuously updated as the simulation evolves. The compiled programs then produce new decisions based on this information, collectively driving the population dynamics. The key requirement is that the interaction-related information on which decisions depend can be adequately represented.

The same requirement applies to subsequent region propagation and temporal merging: the region representation needs to preserve interaction states that affect behavioral transitions, and changes in the interaction context or propagation conditions need to establish new computational boundaries.

W2. Fidelity in the Scaling Experiments

Thank you for the reminder. Section 4.3 of the original paper primarily presented costs, so our reporting was indeed incomplete. Following the experimental configurations in the paper, we additionally evaluated fidelity at different population sizes and simulation horizons.

For this analysis, we reused the programs obtained from the three independent compilations reported in the paper. For WVS, we fixed the simulation at 8 event rounds and expanded the population to 100, 1,000, 10,000, and 100,000. For Reddit, we fixed the population at 100 and evaluated the first 16, 64, 256, and 1,024 steps.

DatasetPopulation NHorizon TRegion JSD (mean ± SD) ↓95% Conditional CI
WVS10080.1270 ± 0.0443[0.0964, 0.1703]
WVS1,00080.1295 ± 0.0589[0.1183, 0.1424]
WVS10,00080.1059 ± 0.0527[0.1001, 0.1119]
WVS100,00080.0986 ± 0.0514[0.0939, 0.1034]
Reddit100160.3709 ± 0.0278[0.3161, 0.4347]
Reddit100640.3491 ± 0.0170[0.3137, 0.3936]
Reddit1002560.2563 ± 0.0251[0.2412, 0.2792]
Reddit1001,0240.2686 ± 0.0143[0.2634, 0.2800]

The SD reflects variation across the three compilations. The CI was obtained through 5,000 paired resamples with the programs, existing reference trajectories, and events or topics held fixed. It reflects statistical uncertainty due to changes in population composition. For WVS, replicas of the same source profile were resampled together; for Reddit, complete agent trajectories were used as the resampling unit, with predictions and references kept paired.

For WVS, the mean JSD at a population of 100,000 is 0.0986, which is no higher than the 0.1295 observed at a population of 1,000. At least under the current profile distribution and event settings, we do not observe error increasing continuously with population size. This provides direct evidence about fidelity as the population scales. The Reddit results need to be interpreted in light of changes in the topics covered. The 16-step horizon covers only the first half of the first topic, whereas the 64-, 256-, and 1,024-step horizons contain 2, 8, and 32 complete topics, respectively. The topic reset and evidence memory mechanisms from the paper are retained throughout. Therefore, as the horizon grows, the topics and stages included in the average also change.

Regarding your point that new agents and interactions may affect fidelity, we also need to clarify what these experiments can answer. The larger WVS populations are resampled from the same empirical profile pool, while the Reddit experiment extends the horizon for a fixed population; both retain the original interaction mechanisms. Thus, these results show fidelity under scaling in the current settings. We will add this per-scale results table to the revised paper and explain the settings and limitations alongside it.

W3. Fidelity and Cost When Changing the Reference LLM

Thank you for pointing out this distinction. You are right: in RQ4, we changed the compiler LLM while keeping the reference LLM fixed. Therefore, the results for these nine models do not directly answer what happens to fidelity and cost when the reference model is changed.

There is also a practical resource constraint. One important reason we chose DeepSeek Flash as the reference model is that it is relatively inexpensive. Even so, API calls for the entire project have already cost approximately 1,000 US dollars. Using all of these models as reference models would require us to regenerate complete trajectories, recompile, and rerun the experiments, which would be too expensive for us (T_T). We also have only two H100 GPUs locally, making it difficult to avoid the cost of running large models through local deployment. We are currently trying to conduct complete experiments using smaller models from the Qwen family as reference models, but, frankly, we are not yet sure whether we can obtain complete results before the rebuttal period ends.

Regarding your point that a smaller reference model may reduce the overall speedup, we agree. A faster reference model would reduce the runtime of the Full-LLM baseline, so our relative advantage could indeed become smaller, especially if the compiler continues to use the same model. That said, the main cost our method aims to reduce comes from repeatedly calling the LLM for individual agents during simulation. Once compilation is complete, these repeated inference calls can be replaced by program execution. We therefore believe the approach remains valuable even with a smaller reference model.

We also conducted a small experiment to first examine how much model response time can change when only the thinking setting is varied for the same simulation decision. We selected 16 real decision contexts from the two datasets and tested five models under two settings each, repeating each context three times per setting, for a total of 480 tests. The same API endpoint was used for both settings of each model. The results are as follows:

ModelThinking settingMedian time for a complete response (seconds)
DeepSeek V4.1 Flashnone → max2.23 → 9.73
GPT-6.1 Sollow → max5.35 → 37.74
Qwen3.5-9Boff → on2.09 → 49.55
Qwen3-32Boff → on3.44 → 11.31
Qwen3.5-397B-A17Boff → on3.79 → 71.32

As shown above, even for the same model, increasing or enabling thinking can multiply the time for a single response by several times to more than twenty times. Given the time cost, our original experiments did not use the highest thinking setting for the reference model either. This test at least gives us a more concrete view of how much the thinking setting itself can affect reference inference cost, in addition to model size.

We also looked at the difference between the 9B and 397B models from the same Qwen3.5 family in this test. With thinking enabled, their response times were 49.55 and 71.32 seconds, respectively—a difference of approximately 1.44×, which we can roughly estimate as 1.5×. If we use this as a hypothetical assumption and suppose that changing to a smaller reference model makes the overall Full-LLM baseline 1.5× faster, while our compilation and program execution times remain unchanged, then in the WVS experiment with 100,000 people, the original time speedup of approximately 17.6×, including construction and deployment costs, would still be approximately 11.8× after adjustment. This corresponds to a reduction of approximately 91.5% in total time. Thus, at least under this estimate, the time savings from our method remain substantial even if a smaller reference model reduces the overall speedup.

Q1. Scaling the Reddit Population

Thank you for the reminder. Our original Reddit scalability experiment did indeed mainly increase the observation horizon T, while keeping the population fixed at N=100, so it did not directly answer what happens as the population grows. To address this, we fixed T at 1,024 and added experiments with populations of 300, 500, and 1,000, comparing them with the original result for 100 people.

The test topics, evidence update procedure, and per-step activation probability follow the original settings. We directly reused the three compilation results from the paper, then rebuilt the population mapping and region weights for each population size. The expanded populations were sampled with replacement from the original 100 profiles. Each added agent has its own activation schedule and history, and its reference trajectory was generated through actual Full–LLM calls. This allows us to examine how well the already-compiled programs perform as the number of people increases under the same empirical profile distribution.

The fidelity results are as follows:

Population NProgram Replay JSD ↓Region Propagation JSD ↓Exact Match (%) ↑
1000.2658 ± 0.01550.2686 ± 0.014348.08 ± 0.36
3000.2791 ± 0.01630.2809 ± 0.016348.12 ± 0.27
5000.2826 ± 0.01490.2834 ± 0.014748.38 ± 0.28
1,0000.2838 ± 0.01490.2838 ± 0.015048.56 ± 0.23

Values are the mean ± sample standard deviation across three frozen compilations. Program Replay JSD follows the same evaluation procedure as Table 1 in the paper; Region Propagation JSD evaluates the overall distribution, and Exact Match is calculated only over individual decisions made when agents are activated.

As the results show, our method maintains relatively stable performance as the population grows from 100 to 1,000. Exact Match remains between 48.1% and 48.6%, with the result at 1,000 people slightly higher than at the original population of 100. Although JSD increases, the change is relatively small: with a tenfold increase in population, Program Replay JSD and Region Propagation JSD increase by approximately 0.0180 and 0.0152, respectively. In particular, when the population increases from 500 to 1,000, Region Propagation JSD changes only from 0.2834 to 0.2838, remaining almost level.

We believe these results directly address performance as the population increases: within the population range tested, SoP maintains fidelity close to that at the original scale, without a substantial decline as the population grows, and its performance becomes more stable at the larger scales.

We also recorded the computational costs as the population increased:

Population NValid Full–LLM decisionsPopulation mapping (ms)Operator construction (s)Hot propagation (ms)
10012,7341.8410.25643.353
30038,0963.7940.25623.359
50063,9185.8940.25763.418
1,000127,83910.9990.25553.365

All population sizes use 8 regions. Times are averaged over the measurements corresponding to the three compilations; for each compilation, hot propagation time is the median of 10 timing runs. Hot propagation covers the full 1,024 steps, but preparation, mapping, operator construction, and file I/O must be counted separately. The signature preparation shared across population sizes takes approximately 3.065 seconds on average. Valid decision counts exclude failed requests and retries.

After a tenfold increase in population, the number of valid Full–LLM decisions also increases by approximately tenfold. For SoP, reusing the existing compilation results requires no new LLM calls, and region propagation time remains at around 3.4 ms. This is because the number of regions remains fixed at 8 in these experiments, with the added population primarily changing the population weights of the regions. In other words, under these settings, the existing compiled programs can support more agents while maintaining similar performance and continuing to avoid the growing cost of calling an LLM for each individual at each time step.

Of course, population growth still introduces some processing overhead. For example, population mapping time increases from approximately 1.8 ms to 11.0 ms. If complete trajectories are required for every individual, the cost of individual replay will also grow with population size. Therefore, it is the cost of the region propagation stage that remains approximately constant here; population mapping and individual outputs still need to be included in the total cost. We will add these supplementary results and experimental settings to the revised paper so that the Reddit experiments cover scaling along both the observation horizon and population dimensions.

Thank you again for these helpful question!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.