- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a multi-sample LLM debiasing framework. By combining the outputs of multiple LLMs with predefined features, it trains a lightweight calibration network to improve the model's prediction accuracy for real human responses. The problem addressed in this paper holds certain research significance; the method is simple and practical, and the experiments are somewhat convincing. However, the overall approach is quite similar to traditional supervised classification, regression, and ensemble learning, with relatively limited innovation. Furthermore, the existing experiments have not yet fully demonstrated the necessity of introducing LLM outputs, nor do they include an in-depth analysis of individual and group prediction changes during calibration. Therefore, I believe this paper still needs to further clarify its methodological contributions and verify its effectiveness and reliability through more comprehensive experiments.
Critical strengths
- The research idea of this paper is commendable, as it focuses on the bias between LLM-simulated human responses and real human behavior, which holds certain research significance and practical application value.
- The method proposed in this paper is simple and easy to implement. Without requiring additional fine-tuning of the LLM, it merely employs a lightweight calibration network to improve prediction results, demonstrating a degree of practicality.
- The experiments in this paper are fairly convincing. The authors conducted evaluations on multiple datasets and considered prediction tasks at both individual and group levels, while also providing ablation studies and downstream application analyses, which to some extent support the effectiveness of the method.
Critical weaknesses
-
First, the task in this paper is essentially still a classification or regression task, which predicts human responses by training a simple MLP that combines LLM outputs with predefined features. Although this method follows some basic paradigms of existing LLM calibration, I find its application scenario rather narrow and lacking in novel methodological design. Essentially, it merely applies existing supervised learning methods to a new domain, resulting in relatively limited overall innovation.
-
Additionally, I noticed that in some tasks, LLM outputs seem unnecessary, as good predictive performance can be achieved using only predefined features. For example, in Table 3, w/o LLM even outperforms the method incorporating LLM outputs on certain metrics (e.g., Acc). This suggests that the model might fit the ground-truth answers well using only existing features. Then, is this paper calibrating LLM prediction bias, or directly training a new prediction model? I believe the authors need to further explain the necessity of introducing LLMs and their practical advantages over direct supervised learning.
-
Finally, although the authors evaluated predictive performance at both the individual and group levels and conducted some analysis on individual ranking, I consider these experiments still insufficient. Improvements in group average predictions do not necessarily mean that individual predictions have been effectively calibrated. For instance, drastic changes in some individual predictions might improve the group mean while simultaneously harming the prediction accuracy for other individuals. Therefore, I suggest that the authors evaluate both individual and group performance on the same cohort of respondents and further analyze whether improvements in group predictions come at the cost of sacrificing individual accuracy or response diversity.
Questions for authors
No.
General Response to All Reviewers, the (S)ACs, and the PCs
We sincerely thank the three reviewers, the (S)ACs, and the PCs for their time and careful engagement. The reviews were constructive and have substantially strengthened the paper.
Strengths noted by the reviewers. The problem has research significance and practical value, and the method is simple and keeps the LLMs frozen (HBW5, R9H4). Preserving the distributional variation of synthetic responses is interesting (zsJE), and the controlled Mean vs. Vector comparison supports keeping the full response vector (R9H4).
Key new evidence.
Revisions. We will report the label-free EEDI results in Sec. 5.2, add guidance on choosing K in Sec. 3.3, and add the new comparisons to the appendix.
We thank the reviewers again for their thoughtful feedback, and the (S)ACs and PCs for overseeing a careful review process.
Best,
Authors