acceptodds
Under review as a conference paper at ICLR 2027

SIMULATING RESPONDENTS, NOT SINGLE QUESTIONS: COHERENT SURVEY GENERATION WITH LARGE LANGUAGE MODELS

Abstract

Large language models are increasingly used to simulate response distributions in social surveys. Prior work has achieved accurate population-level simulation for individual questions. Real-world questionnaires, however, typically require each respondent to answer a sequence of related questions. A simulated respondent should therefore exhibit coherent preferences across the entire questionnaire, rather than merely produce accurate distributions for isolated items. Existing single-item simulation methods can closely match item-level response distributions, but they do not accurately reproduce how the same respondent answers a complete survey. To address this limitation, we propose FullRespondent-LLM (FR-LLM), a framework for simulating complete virtual survey respondents. FR-LLM first fine-tunes two specialized LLMs: a marginal model that estimates the response distribution of each item and a respondent-level autoregressive model that captures dependencies among answers across the questionnaire. We combine these models through Marginal-Constrained Joint Projection (MCJP), which projects the autoregressive joint distribution onto the set of distributions satisfying the item-level marginals learned by the marginal model. By modeling item distributions and cross-item relationships separately, FR-LLM generates complete questionnaires that reproduce realistic cross-item relationships while retaining the item-level accuracy of strong single-item simulators. Experiments on two real-world social survey datasets show that FR-LLM more accurately reproduces response patterns across multiple questions, while maintaining competitive single-item accuracy and generalizing better to unseen respondent populations and survey questions. We also conduct a small commercial survey, use the responses generated by each method to make the same business decision, and compare the resulting profits. FR-LLM produces the highest profit, demonstrating its potential for practical survey-based decision making.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Survey simulation, Distribution simulation, Large language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DA1aOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This article provides a simulation framework enabling coherent preferences across the entire questionnaire, rather than merely producing accurate distributions for isolated items. Targeting the cell-level distribution, the FR-LLM framework utilizes both the separate item prediction and the LLM-based joint correlation prediction, with MCJP method combining their information, and the method has shown consistent improvement compared to baselines.

Critical strengths
  • This article addresses an interesting problem: previous work has not paid enough attention to balancing separate item prediction and joint-level prediction, and thus the simulation serves poorly as a complete questionnaire. This article formulates this problem clearly.

  • This article provides a complete and theoretically guided framework, showing significant improvement compared to previous simulation baselines.

Critical weaknesses
  • W1: This article seems have several writing problems that make the proposed framework hard to follow:
      1. There is notation overloading: the first subscripts in agja_{gj} and aija_{ij} denote the population cell and the individual index, respectively.
      1. YiY_i represents both respondent ii’s answer sheet and the answer to the ii-th question.
      1. The notation in Figure 3 is 1yj=k1_{y_j=k}, whereas in the main text, it appears to be ⊮(Yj=k)⊮ (Y_j=k).
      1. Appendix F.1 seems to refer to Table 9, but it currently appears as an empty subsection.

  • W2: MCJP performs calibration at the population-cell level, consistent with the simulation target TgT_g defined in Section 2. Although this approach shows improvement comparing to the baselines, it uses aggregated marginal targets (agj)j(a_{gj})_j and PgP_g. It is unclear whether the choice of cell construction method is essential to the framework.
Questions for authors

Please address the weaknesses:

  • Q1: Could the author clarify the concerns listed in W1?

  • Q2: For W2, I would expect the author to clarify the choice of cell construction method and conduct an ablation study on it. Specifically, could the authors provide a small scale validation experiments for MCJP with the following setup:
    • Using individual-level marginal distributions (aij)(a_{ij}) with jj ranging over index for items, and Pi(y):=Eπ∼ΠPθπ(y∣xi,q1:m)P_{i}(y):= \mathbb E_{\pi\sim\Pi}P_\theta^\pi(y|x_i,q_{1:m}) for each person ii?
5 comments
AuthorsOpenReview ↗

Response to Reviewer DA1a

Thank you for reading our work carefully and for your constructive feedback. We clarify why FR-LLM calibrates at the population-cell level, report a new analysis of within-cell variation, and explain how we will revise the notation.

1. Population cells reflect the level at which many surveys are used

Thank you for asking why we calibrate by population cell. Many surveys aim to characterize a target group, for example to understand a customer segment or describe how preferences vary across demographic groups. Even when generating one respondent at a time, we need observable attributes to specify whom the model should simulate. In ESS11, we use country, gender, and age group for this purpose. These variables describe the respondent’s background, not their identity. MCJP matches aggregate item marginals by reweighting complete candidate questionnaires; it does not assign everyone in the cell the same answers.

2. Cell-level calibration does not erase within-cell variation

Thank you for suggesting individual-level calibration. Our evaluation follows prior work in holding out questions, populations, or both. For unseen questions, we lack repeated answers from the same person that would let us estimate their personal response probabilities. For unseen populations, the test respondents are different people whose linked answer histories were not available during training. A held-out answer is one observation, not a ground-truth individual response distribution.

An ablation using model-predicted individual marginals is possible, but it would test agreement with those predictions and changes in population-level fit, not whether the model recovered each person’s true distribution. With a finite candidate pool per person, some response options may also be absent and cannot be recovered by reweighting. Our analysis below addresses the related question of whether cell-level calibration suppresses within-cell variation; it does not replace the proposed ablation or establish that our grouping is optimal.

Table 1. Within-cell standard deviation error on ESS11 with Qwen3-4B. Lower is better. We average each questionnaire’s normalized item scores, compute the standard deviation across respondents in each cell, and compare it with the human reference. Absolute errors are weighted by cell size over cells with at least five respondents. Within each mode, all methods use the same evaluation respondents and number of output rollouts.

ModeSingle FTSequential FTJoint Model (no MCJP)FR-LLM
M1: unseen constructs0.04620.03920.04090.0386
M2: unseen countries0.03890.04420.03260.0287
M3: unseen constructs and countries0.03400.03380.02690.0201

FR-LLM has the lowest error in all three modes in this exploratory analysis. In M3, for example, the average within-cell standard deviation is 0.1044 for human respondents, 0.1274 for the uncalibrated joint model, and 0.1180 for FR-LLM. This shows that cell-level calibration can bring the amount of variation within groups closer to the human reference. It does not establish that rare responses occur at the correct frequency or that a generated questionnaire matches any particular individual.

3. We will clarify the notation, Figure 3, and Appendix F.1

Thank you for identifying these inconsistencies. In the revision, we will use gg for a population cell, ii for a respondent, and jj for an item. We will distinguish the complete response vector yi\mathbf{y}_i from the individual answer yijy_{ij}, and use separate notation for person-level predictions and cell-level targets. We will also align Figure 3 with the main text and explicitly introduce and reference Table 9 in Appendix F.1.

FR-LLM remains competitive when we change how population cells are defined.

Thank you for asking whether the results depend on our choice of grouping. We reran our method on ESS11 with Qwen3-4B using country × gender cells instead of country × gender × age group. We removed age only from the calibration grouping, not from the model inputs. The candidate questionnaires, marginal predictions, evaluation respondents, and output budgets were unchanged.

ModeMethodConstruct JSD ↓Two-construct JSD ↓
M1Zero-shot0.17020.4504
M1Single FT0.09000.3274
M1Sequential FT0.12510.3526
M1FR-LLM (original cells)0.05250.2614
M1FR-LLM (without age in cells)0.06300.2821
M2Zero-shot0.21300.5286
M2Single FT0.06220.3108
M2Sequential FT0.11870.3742
M2FR-LLM (original cells)0.02750.2453
M2FR-LLM (without age in cells)0.02660.2385
M3Zero-shot0.18830.4624
M3Single FT0.10960.3459
M3Sequential FT0.13990.3759
M3FR-LLM (original cells)0.08580.3074
M3FR-LLM (without age in cells)0.09300.3221

With either grouping, FR-LLM outperforms all three main baselines on both metrics in all three modes. Removing age slightly improves joint fit in M2 and weakens it in M1 and M3. Thus, the grouping affects the size of the gains, but the gains over the main baselines do not depend on retaining age in the calibration cells.

Thank you again for your valuable feedback. We realize that our original paper may not have sufficiently explained why FR-LLM performs calibration at the population-cell level and how this choice affects individual-level variation.

In our rebuttal and follow-up, we clarified the motivation for cell-level calibration and added experiments on within-cell variation and alternative cell constructions. These results suggest that FR-LLM's improvements are not dependent on a particular cell definition. We have also addressed the notation and presentation issues you identified.

We hope these clarifications and additional experiments have addressed your concerns. If you have any remaining questions, we would be happy to provide further explanations. If your concerns have been sufficiently resolved, would you consider updating your rating?

Reviewer_JusGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FR-LLM, which generates coherent synthetic survey respondents by combining two fine-tuned language models: a marginal model that estimates item-level response distributions and a respondent model that generates complete questionnaires. Marginal-Constrained Joint Projection (MCJP) reweights the full response vectors to match the target marginals while preserving the joint structure. On ESS11 and TALIS 2018, FR-LLM improves joint-distribution accuracy and questionnaire-level reliability and validity, and its item-level accuracy stays competitive. The paper also includes low-data experiments and a small commercial simulation.

Critical strengths
  • Separating marginal estimation from joint response modeling is intuitive and technically reasonable. MCJP provides a principled way to combine the two models without independently modifying individual answers. The theoretical analysis also clarifies the relationship between marginal calibration errors and residual dependence errors under idealized assumptions.

  • The paper evaluates two datasets, two LLM backbones, three transfer settings, and a low-data scenario. FR-LLM consistently improves construct-level JSD and the errors of Cronbach's alpha and AVE across the main experimental settings. The additional ablations, human-resampling reference, and synthetic-to-real evaluation provide useful complementary evidence.

Critical weaknesses
  • The core projection operation is closely related to classical KL projection, exponential tilting, and iterative proportional fitting. The main contribution therefore appears to lie in combining separately trained LLMs rather than introducing a fundamentally new joint-distribution learning method.

  • Sequential FT differs from FR-LLM in its training procedure and question-order handling, in addition to the absence of marginal calibration. Consequently, the comparison does not cleanly establish how much improvement comes from the respondent model, randomized question ordering, or MCJP. Although the component ablation is useful, additional comparisons using the same joint proposal with alternative calibration strategies would provide stronger evidence for the specific contribution of the proposed method.

  • Construct-level JSD evaluates joint response patterns within predefined constructs, while Cronbach's alpha and AVE capture only selected psychometric properties. These metrics do not establish that the generated respondents reproduce dependencies across constructs or the full questionnaire, nor do they guarantee realistic individual-level preferences.

Questions for authors
  • Given the close relationship between MCJP and established KL projection and iterative proportional fitting methods, what is the specific technical contribution beyond applying these methods to the outputs of two separately fine-tuned LLMs? How much of the observed improvement comes from the respondent model itself, and how much comes from marginal calibration compared with simpler reweighting baselines?

  • Can the authors report results across multiple fine-tuning seeds, with uncertainty estimates or significance tests, to establish that the reported improvements are robust to training variability?

  • How well does FR-LLM reproduce dependencies across different constructs or across the full questionnaire? Could the authors provide evidence that its improvements extend beyond matching within-construct distributions and selected psychometric summaries?

4 comments
AuthorsOpenReview ↗

Response to Reviewer JusG #1

Thank you for reading our work carefully and for your constructive feedback. We appreciate the opportunity to clarify the contribution of FR-LLM, separate the effect of calibration from that of the respondent model, and explain how we evaluate dependencies across constructs.

1. The contribution of FR-LLM goes beyond the use of MCJP

Thank you for pointing out the connection to classical projection methods. We agree that MCJP uses an established KL projection solved by iterative proportional fitting (IPF). It is one part of our framework, and we do not claim the projection itself as a new algorithm.

The broader contribution is to combine two separately fine-tuned LLMs that model different aspects of survey responses: one estimates the marginal distribution of each item, while the other generates complete candidate questionnaires that capture dependencies among answers. MCJP brings these predictions together by reweighting whole questionnaires without editing individual answers. The ability to simulate respondents for unseen questions or populations comes from the LLMs’ ability to generalize, not from the projection itself. Classical projection alone requires both a candidate distribution and target marginals to be supplied; it does not predict either for an unseen setting.

2. The randomized joint model is a component of FR-LLM, and we now report it without MCJP

Thank you for pointing out that Sequential FT differs from FR-LLM in both training and calibration. We introduced randomized question ordering and simultaneous supervision of multiple answers as part of our respondent-model design. Because this joint model was a component of our proposed framework rather than an external baseline, we originally used fixed-order Sequential FT as the end-to-end comparator. We agree, however, that this comparison cannot tell us how much MCJP adds. We therefore report the same randomized, multi-answer-supervised joint model without MCJP as a separate ablation.

Table 1. ESS11 with Qwen3.5-9B. All methods use three output rollouts. M1 holds out constructs, M2 holds out countries, and M3 holds out both. Lower is better.

ModeMethodAlpha MAEAVE MAEItem JSDConstruct JSDTwo-construct JSD
M1Sequential FT (fixed order)0.0840.0750.0440.0470.110
Joint Model (random order, no MCJP)0.0760.0700.0270.0280.071
FR-LLM0.0780.0710.0240.0210.059
M2Sequential FT (fixed order)0.0890.0820.0450.0490.165
Joint Model (random order, no MCJP)0.0310.0340.0140.0100.084
FR-LLM0.0280.0300.0140.0090.081
M3Sequential FT (fixed order)0.0990.0840.0540.0750.184
Joint Model (random order, no MCJP)0.0680.0590.0430.0490.129
FR-LLM0.0780.0660.0380.0450.123

The joint model already improves both construct-level JSD measures over Sequential FT. Adding MCJP further improves both measures in all three modes in this setting, though Alpha and AVE MAE do not always improve. This ablation tests the effect of adding calibration; it does not separate the effects of question-order randomization and multi-answer supervision.

To verify calibration directly, we also ran a separate ESS11 M1 control with Qwen3-4B. Holding the candidate pool, marginal targets, and output budget fixed, the expected marginal residual was 0.0509 with MCJP, compared with 0.0588 without calibration and 0.0913 with one-pass reweighting. This confirms that MCJP brings the weighted candidate distribution closer to the specified marginals. The Item JSD in Table 1 uses the marginal model’s probabilities and is not itself a measure of this calibration effect.

AuthorsOpenReview ↗

Response to Reviewer JusG #2

3. We evaluate joint response patterns both within and across constructs

Thank you for asking whether the evaluation captures relationships across constructs. For a long questionnaire, the possible response patterns form the Cartesian product of all item options. This space grows rapidly with the number of items, so most complete patterns are unobserved in a finite survey sample. A direct comparison of full-questionnaire distributions is then difficult to interpret: different metrics can yield very different results depending on how they handle sparse counts.

We therefore evaluate joint patterns at two levels. Construct JSD compares the joint distribution of the items within each construct and averages across constructs. Two-construct JSD takes every pair of constructs, combines their items into one response vector per respondent, compares the joint distributions of those combined vectors, and averages across pairs. In other words, the two groups of items are scored together, not separately. We use these two measures as complementary checks of how well the model learns questionnaire structure. They provide evidence about within- and cross-construct dependencies without claiming to recover the full high-dimensional joint distribution.

Thank you again for your valuable feedback. We realize that our original paper may not have sufficiently explained why FR-LLM performs calibration at the population-cell level and how this choice affects individual-level variation.

In our rebuttal and follow-up, we clarified the motivation for cell-level calibration and added experiments on within-cell variation and alternative cell constructions. These results suggest that FR-LLM's improvements are not dependent on a particular cell definition. We have also addressed the notation and presentation issues you identified.

We hope these clarifications and additional experiments have addressed your concerns. If you have any remaining questions, we would be happy to provide further explanations. If your concerns have been sufficiently resolved, would you consider updating your rating?

Thank you very much for your thoughtful review and constructive feedback. In our rebuttal, we have clarified the distinction between our framework's contributions and classical statistical calibration methods, particularly why these methods alone cannot address generalization to unseen questions and populations without learned marginal and joint distributions. We have also provided additional ablation experiments to clarify the contributions of randomized question ordering and MCJP.

We believe some of our key contributions may not have been sufficiently clear in the original submission, and we sincerely hope you could revisit our detailed responses and additional results. We would be very grateful for any further feedback, and if our clarifications have addressed your concerns, we would greatly appreciate your reconsideration of the rating. Thank you again for your time and careful consideration.

Reviewer_nFeBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a methodology FR-LLM (FR for Full Respondent) for generating synthetic survey respondents with joint distribution matching + marginal distribution matching. It combines two models, a model trained on individual questions (for marginal distribution matching) with a model trained on linked questionnaires (for intra-respondent joint distribution matching), then reweights linked responses to match predicted item-level distributions. Experiments on ESS 11 and TALIS 2018 show improved cross-item response structure while maintaining marginal distribution accuracy.

Critical strengths

I like the idea of jointly thinking about the marginal and intra-respondent response distribution. This work points out a meaningful limitation in that sense, since accurate individual question-level marginal distributions do not guarantee intra respondent-level response patterns. Also, the way the method is executed using a statistical calibration method is good.

Critical weaknesses
  1. From my understanding of the paper, evidence is limited to closed-ended questionnaires, two LLM families (of the similar size, 9B and 8B), and one fine-tuning random seed. More experiment setup – in particular, one more dataset, and one more LLM family model with additional training seed would be great to establish the validity of the method.

  2. The practical motivation of the methodology needs stronger justification. Because FR-LLM requires individual-level survey response data for training, which is hard to obtain, I think the practical use case of FR-LLM needs to be discussed in detail. It remains unclear to me when generating synthetic respondents in the FR-LLM offers an advantage over using conventional prediction models (yes, conventional prediction models should also be a baseline) or just training LLMs on marginal response distributions.

Questions for authors

Please refer to the weakness section.

5 comments
AuthorsOpenReview ↗

Response to Reviewer nFeB #1

Thank you for reading our work carefully and for your constructive feedback. We ran additional experiments in response to your comments and hope the following clarifications and results address your concerns.

1. FR-LLM is intended to extend existing surveys to new populations and questions

Thank you for asking us to clarify the real-world use case. We may not have made clear that FR-LLM does not require collecting a new large survey for every target population or questionnaire. It can be trained on existing public surveys with respondent-level records or on a small survey that collects linked answers. It then uses the LLM’s ability to generalize to simulate complete questionnaires for new questions or harder-to-reach populations, such as older adults who do not use the internet or people with limited time to participate. These are illustrative use cases, not groups we directly evaluated. Our held-out experiments show that FR-LLM can reproduce useful cross-item structure on new questions and in new populations, although performance varies by setting.

By cross-item dependence, we mean how answers within the same target questionnaire relate to one another, not how new answers correlate with answers from a previous survey. A marginal model may suffice if only a single item’s response distribution is needed. FR-LLM is intended for tasks where the combination of answers matters.

2. A smaller model and a third dataset provide additional evidence

To address your concern about the scope of our evaluation, we added experiments with Qwen3-4B on ESS11, and an exploratory experiment on a third dataset, World Values Survey Wave 7 (WVS7). M1 holds out constructs, M2 holds out countries, and M3 holds out both. Lower is better for all metrics; bold marks the best value within each mode at the reported precision.

Table 1. ESS11 results with Qwen3-4B.

ModeMethodAlpha MAEAVE MAEItem JSDConstruct JSDTwo-construct JSD
M1Zero-shot0.4840.3290.0970.1700.450
M1Single FT0.4800.3250.0690.0900.327
M1Sequential FT0.0670.0610.0970.1250.353
M1FR-LLM0.0930.0820.0690.0530.261
M2Zero-shot0.6430.4240.1060.2130.529
M2Single FT0.6490.4290.0310.0620.311
M2Sequential FT0.0730.0650.0880.1190.374
M2FR-LLM0.0530.0490.0310.0280.245
M3Zero-shot0.4760.3210.1040.1880.462
M3Single FT0.4980.3300.0670.1100.346
M3Sequential FT0.0900.0780.0940.1400.376
M3FR-LLM0.1330.1170.0670.0860.307

Table 2. WVS7 results with Qwen3-4B. This exploratory pilot uses questionnaire batteries on institutional confidence and interpersonal trust.

ModeMethodAlpha MAEAVE MAEItem JSDConstruct JSDTwo-construct JSD
M1Zero-shot0.6300.2590.1770.3450.535
M1Single FT0.6860.2510.0880.4150.681
M1Sequential FT0.0720.0620.1980.3890.576
M1FR-LLM0.0570.0520.0880.2810.651
M2Zero-shot0.7900.3180.3290.5670.675
M2Single FT0.7970.3600.0660.2690.643
M2Sequential FT0.1380.1050.0990.1800.470
M2FR-LLM0.3900.2440.0660.2120.531
M3Zero-shot0.6980.2570.2180.4810.618
M3Single FT0.6020.2890.1030.3540.632
M3Sequential FT0.0310.0280.1820.3200.590
M3FR-LLM0.0700.0560.1030.2890.656

On ESS11, FR-LLM achieves the lowest Construct JSD and Two-construct JSD in all three modes. Sequential FT has lower AVE MAE in M1 and M3, but its response distributions are farther from the human distributions on both JSD measures. This shows why matching a psychometric summary is not the same as matching the joint response distribution. On TALIS, FR-LLM also achieves the lowest Construct JSD in M1 and M2 and the lowest Two-construct JSD in M2, while Sequential FT leads on Construct JSD in M3. Across all nine settings, FR-LLM improves Construct JSD over Single FT, though it does not outperform Sequential FT in every setting.

FR-LLM’s Item JSD uses the Single FT marginal probabilities, so ties do not imply exact marginal agreement in sampled questionnaires. These experiments add evidence from a smaller backbone and a third dataset, but do not test an independent model family or additional training seeds.

AuthorsOpenReview ↗

Response to Reviewer nFeB #2

3. FR-LLM improves joint accuracy over conventional baselines on ESS11 M2

Following your suggestion, we added item-wise multinomial logistic regression and two hot-deck baselines on the original ESS11 M2 split (28,268 training and 7,312 held-out-country respondents). Linked hot-deck samples a complete answer vector from one demographically matched donor; independent hot-deck uses the same donor marginals but samples answers separately.

Table 3. Conventional baselines on ESS11 M2.

MethodAlpha MAEAVE MAEItem JSDConstruct JSDTwo-construct JSD
Multinomial logistic regression0.57100.38320.01610.04510.1579
Independent hot-deck0.55880.37880.01640.04460.1566
Linked hot-deck0.03820.04060.01640.01030.0971
FR-LLM (Qwen3.5-9B)0.02750.02950.01400.00910.0813

Logistic regression is already competitive on Item JSD. Keeping donor answers linked substantially improves joint accuracy without changing the expected item marginals. FR-LLM further reduces Two-construct JSD from 0.0971 to 0.0813 in this setting. The comparison illustrates where FR-LLM adds value: when the task depends on which answers occur together, not just how often each answer occurs.

Thank you again for your constructive feedback. We realize that our original submission may not have sufficiently clarified the practical motivation of FR-LLM, particularly its ability to extend existing surveys to unseen questions and populations without collecting a new large-scale survey for each setting.

In our rebuttal, we have clarified this motivation and provided additional evidence from a third dataset (WVS7), a smaller backbone (Qwen3-4B), and conventional prediction baselines, including logistic regression and hot-deck sampling.

We would be grateful to know whether these clarifications and additional experiments have addressed your main concerns. If any questions remain, we would be happy to discuss them further. If your concerns have been sufficiently resolved, would you consider revisiting your rating?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.