- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper tests whether simulated choices respond proportionally to two sizes of change in an attribute. Aggregate responses often look closer to proportional than individual responses. Probing and activation interventions then show that a model's response to a perturbation can be carried by a low-dimensional displacement and transferred between respondents.
Critical strengths
- Testing responses to changes is a useful direction for evaluating synthetic respondents. The probability-based readout and relabeling controls help make the test cleaner. I also appreciate the reference-validity checks and the authors' decision to exclude electricity from the corrected conclusions. The intervention work includes several useful controls.
Critical weaknesses
-
My main concern is the interpretation of “wrong per respondent”. The individual human counterfactuals are not observed; the proportionality target follows from a fitted choice model. The curvature checks help, but they do not exclude other forms of nonlinear human response. Matching a small-to-large ratio also does not mean that the aggregate response has the correct absolute magnitude. I would describe this as a test against an operational reference, rather than direct evidence that the simulated individuals are wrong and the average is right.
-
I also have some concerns on the “systematic share”. As defined in Appendix B, it is the absolute mean response divided by the mean absolute response. This measures cancellation or signed coherence, rather than the fraction of movement caused by change size. Perfectly proportional individual responses with opposing signs could have a low share. The separate two-size analysis is evidence of scaling inconsistency, but it does not justify saying that the remaining movement has nothing to do with size.
-
The held-out respondents are used to select the peak probe layer before interventions, so they are not fully held out from analysis selection. More importantly, the transplanted displacement contains both proportional and non-proportional effects. Showing that it transfers, or scales approximately linearly along the tested path, does not isolate the error component or rule out downstream contributions in general.
Questions for authors
-
Do the behavioral conclusions survive more flexible human references, and can proportionality be reported separately from absolute human accuracy?
-
Can layer selection use a separate validation split, and can an intervention target the non-proportional component while preserving the common response?