- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how generating several survey respondents in one completion affects synthetic data. Generating ten respondents together produces larger correlation errors than generating them separately, mostly by making the relationships stronger than those in human data. The paper also tests how earlier answers affect later responses. Hiding earlier answers or correcting the data can reduce correlation error, but can make marginal distributions or regression estimates worse.
Critical strengths
-
I like the clear experimental setting: a comparison of the same profiles under one-person and ten-person requests while keeping the examples, questions, and decoding settings fixed.
-
The experiment that changes the pairings of earlier answers is useful. The values stay the same, but the relationships among later generated responses change, showing that answer history matters.
-
I also like that the paper checks correlations, marginal distributions, and regression estimates together. Improving one metric does not necessarily make the synthetic data better for other analyses.
-
The authors run additional checks with different temperatures, prompt wording, and output constraints, and report where the results are less consistent.
Critical weaknesses
-
The writing is difficult to follow. Terms such as “relationship strength,” “pattern agreement,” and “dependence fidelity” sound similar, but their differences are not clearly explained. Sentences and captions also contain too many experimental details and qualifications. It is often hard to tell what each experiment is trying to show.
-
It is still unclear why earlier answers make the relationships stronger, rather than simply changing them. More explanation of what the model might be doing would help readers understand the finding.
-
The paper shows that one-person generation and corrections can improve correlation error while making other measures worse. But it is unclear what readers should do with this result. A concrete example of how batching changes an analysis would make its importance clearer.
-
The main comparison uses one or ten respondents per request. It does not show how the effect changes between these settings, or how many preceding respondents are needed for the distortion to appear.
Questions for authors
-
Does the order of respondents matter? What happens if similar demographic profiles are grouped together rather than mixed?
-
Why does batching mainly exaggerate the strength of relationships? Do you think the model follows patterns in earlier answers or generates less individual variation?
-
When hiding history lowers correlation error, how much comes from recovering the human pattern and how much comes from simply making correlations weaker?
-
Can you show an example where batching changes the conclusion of an analysis, in addition to its average error?
-
Given that one-person generation improves correlations but can worsen marginal distributions, when would you recommend generating respondents separately?
Reply to Reviewer HzZX
Thank you for your careful review and for questions that shaped the new experiments.
Terminology. Section 2 now defines relationship strength, pattern agreement and correlation error with equations, and the paper uses only these terms. Table 1 and Figure 1 show what each experiment changes and what it is meant to show. We have also cut extraneous detail and qualifications from sentences and captions.
Q2. Why relationships become stronger. Each new respondent moves towards the respondents before it, and in a batch this builds up, so respondents spread along one axis. When people differ mainly along one shared dimension, every pair of variables tied to it correlates more strongly. The model therefore follows the pattern in earlier answers, rather than producing less individual variation. Batching multiplies the variance along that axis by 1.8–4.0, while variance in other directions changes much less (0.96–1.22; Section 3.3, Appendix C).
How many preceding respondents are needed. One is enough. Two-person requests are already worse than one-person requests in every seed. Error rises by 0.023 for Qwen and 0.085 for Llama with two-person requests, and by 0.094 and 0.223 with ten-person requests. Within a request, error also rises with position (Section 3.1, Figure 3A).
Q1. Order and grouping. Reversing the order within requests changes error little. Grouping similar profiles lowers error for Qwen and raises it for Llama, in every seed (Section 3.3).
Q3. Recovery or weakening when hiding history. The gain comes mostly from weakening. For Qwen, masking brings correlation error back to about its one-person level (Table 8). There is some recovery: the error left after rescaling the correlations falls from 0.081 to 0.065, and the Pearson agreement rises from 0.848 to 0.894. The rank ordering of pairs, however, barely changes (ρ from 0.770 to 0.782). For Llama, the ordering gets worse (ρ from 0.798 to 0.616). These results are from NHANES (Section 3.3, Table 7).
Q4. An example. In an exploratory regression of trust in police on income, happiness, satisfaction with democracy and age, the coefficient on satisfaction with democracy is 0.36 in the human data. Batching inflates it in all three models, for example from 0.34–0.37 to 0.49–0.61 for Llama. Signs and the ordering of predictors mostly survive, so batching changes how strong a relationship looks rather than whether it exists (Section 3.5).
Q5. When to generate separately. We recommend generating respondents separately whenever an analysis uses relationships, such as correlations or regressions, because batching inflates them and an instruction does not prevent this. Batching gives better marginals in our data, but through the same process, so it is not better data in general (Section 5).
We would be glad to answer any further questions before the discussion closes on 18 November.