- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This article provides a simulation framework enabling coherent preferences across the entire questionnaire, rather than merely producing accurate distributions for isolated items. Targeting the cell-level distribution, the FR-LLM framework utilizes both the separate item prediction and the LLM-based joint correlation prediction, with MCJP method combining their information, and the method has shown consistent improvement compared to baselines.
Critical strengths
-
This article addresses an interesting problem: previous work has not paid enough attention to balancing separate item prediction and joint-level prediction, and thus the simulation serves poorly as a complete questionnaire. This article formulates this problem clearly.
-
This article provides a complete and theoretically guided framework, showing significant improvement compared to previous simulation baselines.
Critical weaknesses
- W1: This article seems have several writing problems that make the proposed framework hard to follow:
-
- There is notation overloading: the first subscripts in and denote the population cell and the individual index, respectively.
-
- represents both respondent ’s answer sheet and the answer to the -th question.
-
- The notation in Figure 3 is , whereas in the main text, it appears to be .
-
- Appendix F.1 seems to refer to Table 9, but it currently appears as an empty subsection.
-
- W2: MCJP performs calibration at the population-cell level, consistent with the simulation target defined in Section 2. Although this approach shows improvement comparing to the baselines, it uses aggregated marginal targets and . It is unclear whether the choice of cell construction method is essential to the framework.
Questions for authors
Please address the weaknesses:
- Q1: Could the author clarify the concerns listed in W1?
- Q2: For W2, I would expect the author to clarify the choice of cell construction method and conduct an ablation study on it. Specifically, could the authors provide a small scale validation experiments for MCJP with the following setup:
- Using individual-level marginal distributions with ranging over index for items, and for each person ?
5 comments
FR-LLM remains competitive when we change how population cells are defined.
Thank you for asking whether the results depend on our choice of grouping. We reran our method on ESS11 with Qwen3-4B using country × gender cells instead of country × gender × age group. We removed age only from the calibration grouping, not from the model inputs. The candidate questionnaires, marginal predictions, evaluation respondents, and output budgets were unchanged.
| Mode | Method | Construct JSD ↓ | Two-construct JSD ↓ |
|---|---|---|---|
| M1 | Zero-shot | 0.1702 | 0.4504 |
| M1 | Single FT | 0.0900 | 0.3274 |
| M1 | Sequential FT | 0.1251 | 0.3526 |
| M1 | FR-LLM (original cells) | 0.0525 | 0.2614 |
| M1 | FR-LLM (without age in cells) | 0.0630 | 0.2821 |
| M2 | Zero-shot | 0.2130 | 0.5286 |
| M2 | Single FT | 0.0622 | 0.3108 |
| M2 | Sequential FT | 0.1187 | 0.3742 |
| M2 | FR-LLM (original cells) | 0.0275 | 0.2453 |
| M2 | FR-LLM (without age in cells) | 0.0266 | 0.2385 |
| M3 | Zero-shot | 0.1883 | 0.4624 |
| M3 | Single FT | 0.1096 | 0.3459 |
| M3 | Sequential FT | 0.1399 | 0.3759 |
| M3 | FR-LLM (original cells) | 0.0858 | 0.3074 |
| M3 | FR-LLM (without age in cells) | 0.0930 | 0.3221 |
With either grouping, FR-LLM outperforms all three main baselines on both metrics in all three modes. Removing age slightly improves joint fit in M2 and weakens it in M1 and M3. Thus, the grouping affects the size of the gains, but the gains over the main baselines do not depend on retaining age in the calibration cells.
Thank you again for your valuable feedback. We realize that our original paper may not have sufficiently explained why FR-LLM performs calibration at the population-cell level and how this choice affects individual-level variation.
In our rebuttal and follow-up, we clarified the motivation for cell-level calibration and added experiments on within-cell variation and alternative cell constructions. These results suggest that FR-LLM's improvements are not dependent on a particular cell definition. We have also addressed the notation and presentation issues you identified.
We hope these clarifications and additional experiments have addressed your concerns. If you have any remaining questions, we would be happy to provide further explanations. If your concerns have been sufficiently resolved, would you consider updating your rating?
Response to Reviewer DA1a
Thank you for reading our work carefully and for your constructive feedback. We clarify why FR-LLM calibrates at the population-cell level, report a new analysis of within-cell variation, and explain how we will revise the notation.
1. Population cells reflect the level at which many surveys are used
Thank you for asking why we calibrate by population cell. Many surveys aim to characterize a target group, for example to understand a customer segment or describe how preferences vary across demographic groups. Even when generating one respondent at a time, we need observable attributes to specify whom the model should simulate. In ESS11, we use country, gender, and age group for this purpose. These variables describe the respondent’s background, not their identity. MCJP matches aggregate item marginals by reweighting complete candidate questionnaires; it does not assign everyone in the cell the same answers.
2. Cell-level calibration does not erase within-cell variation
Thank you for suggesting individual-level calibration. Our evaluation follows prior work in holding out questions, populations, or both. For unseen questions, we lack repeated answers from the same person that would let us estimate their personal response probabilities. For unseen populations, the test respondents are different people whose linked answer histories were not available during training. A held-out answer is one observation, not a ground-truth individual response distribution.
An ablation using model-predicted individual marginals is possible, but it would test agreement with those predictions and changes in population-level fit, not whether the model recovered each person’s true distribution. With a finite candidate pool per person, some response options may also be absent and cannot be recovered by reweighting. Our analysis below addresses the related question of whether cell-level calibration suppresses within-cell variation; it does not replace the proposed ablation or establish that our grouping is optimal.
Table 1. Within-cell standard deviation error on ESS11 with Qwen3-4B. Lower is better. We average each questionnaire’s normalized item scores, compute the standard deviation across respondents in each cell, and compare it with the human reference. Absolute errors are weighted by cell size over cells with at least five respondents. Within each mode, all methods use the same evaluation respondents and number of output rollouts.
FR-LLM has the lowest error in all three modes in this exploratory analysis. In M3, for example, the average within-cell standard deviation is 0.1044 for human respondents, 0.1274 for the uncalibrated joint model, and 0.1180 for FR-LLM. This shows that cell-level calibration can bring the amount of variation within groups closer to the human reference. It does not establish that rare responses occur at the correct frequency or that a generated questionnaire matches any particular individual.
3. We will clarify the notation, Figure 3, and Appendix F.1
Thank you for identifying these inconsistencies. In the revision, we will use g for a population cell, i for a respondent, and j for an item. We will distinguish the complete response vector yi from the individual answer yij, and use separate notation for person-level predictions and cell-level targets. We will also align Figure 3 with the main text and explicitly introduce and reference Table 9 in Appendix F.1.