acceptodds
Under review as a conference paper at ICLR 2027

Simulation-Augmented Generation via Reinforcement Learning

Abstract

Simulation-augmented generation by Milli et al. (2026) is a recent framework for answering normative questions by querying generative simulations of individuals in a prompt-dependent target population and synthesizing their responses. Kraiczy et al. (2026) formalize representative selection in this setting using social choice and, to avoid simulating every individual, use proportional clustering over predicted viewpoint embeddings to route each prompt to a small subset among simulations. We introduce reinforcement learning from social choice axioms (RL-SCA), a framework for directly training representative routers using continuous measures of social choice guarantees as reward signals. In our setting, we use the mEJR+ factor, a continuous measure of proportional representation, and show that it can be computed in time, making it practical as a reward for reinforcement learning. This allows us to train a lightweight router end-to-end to map each prompt directly to a representative subset of simulations, replacing the two-stage process of predicting population-wide embeddings and subsequently clustering them. The resulting router requires only a single forward pass at inference time, avoiding predicted embedding computations for the full population while achieving significantly improved proportional representation across three datasets spanning politics, personal advice and book reviews.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
pluralistic alignment, social choice theory, reinforcement learning, simulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gA33OpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper gives an algorithm to calculate the mEJR+ factor which is used as a reward signal to train a router using RL.

Critical strengths

The paper provides a solution to compute a mEJR+factor in a low complexity, which enables to train a router using RL with such reward signal.

Critical weaknesses
  1. The problem formulation and framework is built on top of previous works, SAGE, which exhibits a limited novelty.
  2. The baseline comparison is limited. A supervised learning baseline should be compared.
Questions for authors

The RL train cost is not disclosed, which should be considered for baseline comparison as well.

Reviewer_A85DOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper improves the routing mechanism for simulation-augmented generation (SAGE). Specifically, given a prompt and a fixed population of n LLM simulations of individuals, the router is expected to choose a small slate of k≪nk \ll n representative simulations. The prior method, Predictive Route–then–Simulate (PRS), implements the routing by predicting a response embedding for each simulation and running SEAR on the predicted embeddings. But PRS's performance on the evaluation metric, the proportionality axiom of metric proportional justified representation+ (mPJR+), degrades substantially as k increases, e.g., at k=10k=10.

The paper proposes RL-SCA, which trains the router end-to-end on a reward signal derived from a proportionality axiom. The continuous reward signal is the negative log of the mEJR+ factor, which measures how far a slate is from satisfying the metric EJR+ (mEJR+) axiom, and the paper shows that this factor can be efficiently computed in O(n2(log⁡n+k))\mathcal{O}(n^2 (\log n + k)) time.

Experiments on Remesh (political viewpoints, n = 310), Humanual Opinion (personal advice, n = 300) and Humanual Book (book reviews, n = 103) show that at k=10k=10, RL-SCA raises mPJR+ satisfaction over PRS from 43% to 61% on Remesh, from 40% to 67.5% on Humanual Opinion, and from 53.7% to 83% on Humanual Book, and raises mEJR+ satisfaction by 28.5–39.9 points.

Critical strengths

The idea is simple, novel, and well-motivated. Instead of predicting embeddings and then optimizing, end-to-end training based on an approximated downstream metric could significantly boost the performance. The use of the mEJR+ factor is also well-grounded, given the computational difficulty of mPJR+. Empirical experiments also justify the effectiveness of the paradigm. The limitation of the experimental design is also honestly included in the discussion, and all notation has been precisely defined.

Critical weaknesses

The clarity and quality of the paper can be further improved.

  1. Quality: The experiment only compares with the baselines of random and PRS, while in the related field of prompt selection, many other methods could be adapted, such as [1][2]. I would recommend at least comparing with two more simple baselines: one is a fixed slate of simulations shared across all prompts, and the other is language-model-based selection of simulations.

  2. Quality: Multi-trial experiments are needed. Training is done with a single seed, and only 95% binomial confidence intervals are reported.

  3. Clarity: This domain is interdisciplinary and includes special terminology from both machine learning and computational social choice. For readers not specialized in either domain, understanding the paper will be hard. It is highly recommended to explain the background in the introduction or preliminaries to improve readability.

  4. Clarity: Minor issues

  • line 054: use citep here.
  • line 122: use citet for nouns.

References

[1]: Do, Viet-Tung, et al. "Automatic prompt selection for large language models." arXiv preprint arXiv:2404.02717 (2024).

[2]: Zhang, Haoting, et al. "Language model prompt selection via simulation optimization." arXiv preprint arXiv:2404.08164 (2024).

Questions for authors

Are there empirical evaluations of the algorithm efficiency, like FLOPs, latency, wall block time, etc?

Reviewer_tG5oOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies representative routing for simulation-augmented generation: for each prompt, select k≪nk\ll n of nn LLM simulations of individuals so that their responses proportionally represent the population in a viewpoint-embedding space. The prior method of Kraiczy et al. (2026), Predictive Route–then–Simulate (PRS), predicts an embedding for every simulation and runs SEAR on the predictions.

The authors define the mEJR+ factor β⋆\beta^\star, the smallest multiplicative relaxation under which a slate satisfies metric EJR+, and compute it exactly in O(n2(log⁡n+k))O(n^2(\log n+k)) time (Theorem 1). They then train a two-hidden-layer MLP router with REINFORCE on the reward −log⁡β⋆-\log\beta^\star (RL-SCA); at inference it selects the top-kk simulations from the prompt alone.

On three fixed populations, RL-SCA raises mPJR+ satisfaction at k=10k=10 from 43%, 40%, and 53.7% under PRS to 61%, 67.5%, and 83%.

Critical strengths

The paper turns a binary proportionality axiom into a graded score that can be computed exactly and uses it as a verifiable reward, which addresses the sparse-reward problem of binary satisfaction that the paper argues concretely (Section 2). The algorithm for β⋆\beta^\star is simple and checkable: the correctness proof states both directions of the sweep argument and the pointer invariant explicitly, and I found no problems in it or in the triangle-inequality argument showing that every mPJR+ slate has β⋆≤3\beta^\star\le3, which links the new score to the metric used in prior work. The decision-focused-learning motivation for replacing predict-then-optimize with end-to-end training fits the setting and removes the per-individual prediction step.

At k=10k=10, RL-SCA improves over PRS on all three datasets, on both mPJR+ and mEJR+, and reduces the largest mEJR+ factor across test prompts (Appendix D.4). A router trained only at k=10k=10 also yields useful prefixes for smaller kk. The paper is candid about its scope: it calls the results a proof of concept, states that mEJR+ is not uniquely privileged, and notes that SEAR need not return mEJR+ slates.

Critical weaknesses

My main concern is that it is unclear how much the router uses the prompt. The router scores a fixed population, and viewpoint structure may be largely stable across prompts, for example if the same individuals tend to be outliers or centers. A single learned ranking that ignores the prompt, such as the same MLP with a constant input or a free logit vector trained with the same reward, could then capture much of the gain, but no such baseline is reported. The Remesh test prompts are also close to the training prompts, since the 1,000 generated questions cover ten topics and are split at random (Appendix D.1.1). Without a prompt-agnostic baseline or a prompt-sensitivity analysis, such as the overlap of top-kk sets across prompts or a topic-held-out split, one cannot tell whether RL-SCA learns prompt-dependent routing or a good population-level ordering.

The results also lack a reference point and an ablation of the reward. For mPJR+, SEAR applied to the generated embeddings satisfies the axiom by construction (Section 1), but for mEJR+ there is no reference such as a greedy or local-search minimizer of β⋆\beta^\star, so it is unclear how much headroom remains or how often mEJR+ is satisfiable at all on these instances. The central design choice is also not ablated: a binary mEJR+ or mPJR+ reward versus the graded −log⁡β⋆-\log\beta^\star, or other transforms of β⋆\beta^\star. The sparse-reward argument for a graded signal is plausible but not tested empirically.

The comparison with PRS partly favors RL-SCA by construction, and PRS details are missing. RL-SCA is trained on the mEJR+ factor computed in the same generated embedding geometry used for evaluation, and mEJR+ implies mPJR+, whereas PRS is trained to predict embeddings; the authors acknowledge this for the mEJR+ comparison (Appendix D.4). This alignment is partly the point of end-to-end training, but the paper does not describe how the PRS predictor was trained or tuned on each dataset. This matters most for Humanual Book, which uses a different simulator from Kraiczy et al. RL-SCA was tuned over 18 configurations on Remesh and 4 on Humanual Opinion (Appendix D.3), and it is not stated whether PRS received comparable tuning.

The efficiency claim is not measured. Single-forward-pass inference is a headline claim, yet the paper reports no wall-clock, FLOP, or latency comparison with PRS, and SAGE still requires kk LLM simulation calls, so the practical saving should be quantified. The offline cost of generating the full response grid for every training prompt (155,000 responses for the 500 Remesh training prompts) is also not compared with what PRS requires.

The router's scope is limited, and the evaluation measures only a proxy. With one output per simulation, the model is tied to a fixed population of at most 310 individuals; it cannot add new individuals or select from subpopulations, and it does not use the individual-level context sis_i that PRS uses. The authors list these as future work (Section 4), but they matter for SAGE, where the target population depends on the prompt. In addition, all results are measured in the preference-embedding geometry, and there is no test of whether better proportionality leads to better SAGE outputs, for example syntheses judged against the synthesis from the full population.

Minor comments: all reported hyperparameter sweeps use training seed 0 and no variance across seeds is reported, and with 80 test prompts on Humanual Opinion the binomial intervals in Figure 1 are about ±10 percentage points. The contributions mention “three baselines”, but only PRS and random selection are evaluated. The training configuration for Humanual Book is not reported (Table 3 and Appendix D.3 cover only Remesh and Humanual Opinion). The paper also does not state which mPJR+ verifier is used for evaluation, whereas Kraiczy et al. used the FPT verifier with subsampling.

Overall, the paper offers a clean and reusable idea, an exactly computable proportionality reward, with sizeable gains at k=10k=10. The evidence would be more convincing with a prompt-agnostic baseline, a reference point for mEJR+, details and comparable tuning for PRS, and measured efficiency. I currently lean toward rejection, but would raise my score if these additions, especially the prompt-agnostic baseline, show that the router learns prompt-dependent selection.

Questions for authors
  1. How does RL-SCA compare with a prompt-agnostic learned ranking (a constant input or a free logit vector trained with the same reward), and how much do the top-kk sets vary across test prompts or under a topic-held-out split of Remesh?

  2. How often is mEJR+ satisfiable at k=10k=10 on these datasets, and how close does a greedy or local-search minimizer of β⋆\beta^\star come?

  3. How does training behave with a binary reward or with other transforms of β⋆\beta^\star, and how are slates with β⋆=+∞\beta^\star=+\infty handled in the reward (Algorithm 1, line 23)?

  4. How was the PRS predictor trained and tuned on each dataset, especially Humanual Book, and which mPJR+ verifier is used for evaluation?

  5. What are the measured inference and offline generation costs of RL-SCA and PRS, relative to the cost of the kk simulation calls?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.