- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether LLMs can accurately reconstruct consumer reactions to public events, exploring a relatively underexplored research area. The proposed dataset is large-scale, covering 1,553 topics and 23,122 atomic evaluation criteria.
Critical strengths
- The task is well defined: generate open-ended comments and score coverage of concrete reaction points.
- The scale is large: 1,553 topics, 23,122 criteria, and a multi-model leaderboard with bootstrap intervals.
- Pointwise yes/no criteria are more auditable than a single holistic score.
- The main pattern is clear: emotion and praise are easier than flashpoints and criticism.
Critical weaknesses
- Criteria are drafted by an LLM from topic summaries. The paper does not validate them against raw comments, so 98.4% measures judge agreement, not label fidelity.
- No human forecaster is tested, so 42.5% has no ceiling. Many criteria are defined as not inferable from the event text.
- Topics are public trends from March–September 2025, with no contamination or retrieval check.
- The reliability audit covers only 10 topics and about 100 criteria.
- The low GPT and Claude scores may reflect Chinese register or refusals. The paper does not separate these from consumer intuition.
- The metric is unweighted recall, judged on the first 3,600 characters. Prompt and agent gains are only 0.6–4.7 points on 100 topics.
- Miss citation: ConsintBench: Evaluating Language Models on Real-World Consumer Intent Understanding. https://arxiv.org/abs/2510.13499
- [Minor]: The abstract says 13 models, while Table 4 lists 14, including an undeclared GPT-6-astra.
Questions for authors
See Weaknesses.
3 comments
Thank you for the detailed response and additional checks, which address several minor concerns. However, my main methodological concerns remain. First, label fidelity and judge reliability are still primarily assessed through LLM self-checks, without sufficient human validation. Second, the human benchmark is limited to a single expert and 20 topics, making the claimed human ceiling unreliable. Third, the unweighted recall metric overlooks criterion prevalence and hallucinated reactions, while the entity-masking experiment does not fully rule out data contamination. Therefore, I maintain my current score.
Thank you for clarifying the three remaining concerns. Here, we address each remaining concern below.
1. Human validation of label fidelity and judge reliability.
To directly assess label fidelity, we additionally conducted a manual source-verification audit of 118 criteria across 30 topics, covering all four reaction families. The sample was fixed before annotation. One human expert examined each criterion alongside 1-3 retrieved original consumer posts. All 118 criteria were rated as explicitly supported by the supplied source evidence. This provides direct human verification that the audited reaction points reflect actual consumer discourse.
For judge reliability, our existing audit contains yes/no decisions on 13 cases selected for disagreement among automatic judges. The human-majority decision agreed with the original three-judge majority on 11 of the 13 disputed cases (84.6%).
We also report an automated consistency check. As reported in “Are pointwise judges reliable?”, we evaluate both inter-judge consistency and intra-judge stability on a calibration set of 100+ criteria across 10 topics. With the full scoring criteria, GPT-5.2, Gemini-3-Pro, and Claude-4.5-Opus achieve 92.1% three-judge agreement, compared with 65.8% for holistic judging.
These assessments address whether the reference reactions have source support, and whether generated responses satisfy the scoring criteria.
2. The human study and the interpretation of a “human ceiling.”
We recruited a second expert to independently assess the same 100 disclosed criteria using the original assessment form, without access to the first annotator’s labels.
| Judgment | Original annotator | Second annotator |
|---|---|---|
| Predictable | 74% | 83% |
| Reasonable, but not predictable | 24% | 10% |
| Unreasonable after disclosure | 2% | 7% |
| Cannot judge | 0% | 0% |
The second expert rated a larger proportion of the criteria as predictable (83% versus 74%). Across the two experts, 93–98% of the criteria were rated as either predictable or reasonable after disclosure. These findings provide additional support for the reasonableness of the criteria and suggest that most are predictable by domain experts, while also revealing some variation in individual judgments.
We will add more experts to this study after the acceptance of the paper.
Also, we do not suggest there is a "ceiling" for our benchmark. By this experiment, we want to show human experts can perform far better than current models on ConsumerSimBench, and that there are much space for improvement, because the unpredicted points are also reasonable.
3. Unweighted recall
Regarding weighting, we directly tested whether our model comparisons depend on equal family weights. Across the full benchmark of 1,553 topics, 23,122 criteria, and 13 models, we assigned 40% weight to each family in turn and 20% to each remaining family. Rankings remained highly stable, with Spearman correlations of 0.984–0.995 against the original ranking. Gemini-3.1-Pro remained first under all four settings, and each setting retained three or four of the original top four models. These results demonstrate that our main model comparisons are robust to the tested changes in family weighting.
4. Hallucinated reactions
We want to clarify that we do not equate "not observed in the finite corpus" with "wrong." The collected reactions provide reliable positive targets, but they cannot enumerate every plausible response to an event. Penalizing every corpus-unmatched reaction would therefore misclassify reasonable long-tail reactions as false positives. For this reason, we retain authentic-reaction coverage as the official metric rather than introduce an incomplete negative-label penalty, and we prevent packing by length limit..
Also, in marketing, we believe it is better to anticipate a broader range of reactions than to miss potentially consequential ones, as marketers can prepare for additional scenarios but cannot afford a failed or unpredicted campaign.
5. Contamination
As stated in the appendix, we do not expect no data contamination. Our insight and experiments prove that gaining the historical data does not help improving the performance, because we are measuring LLM's ability of consumer reaction reconstruction, instead of news memorization.
Thank you again for the constructive feedback, which has helped us clarify and strengthen the paper.
Rebuttal by authors
Thank you for recognizing the task, scale, and auditable scoring. Here, we try to address the concerns as follows.
Thanks for pointing this out, and we now validate them against raw comments. We use raw comments as input and validate them across the full benchmark (without the input length limit). This yields a 99.17% score, showing that almost all our criteria are possible to gain points.
We want to clarify that we already have human studies to measure the correctness of criteria drafting in Appendix H. On a 20-topic subset, one marketing expert achieves a 74% score. After seeing the ground-truth criteria, he marked another 24% as reasonable once disclosed but difficult to anticipate. Only 2% criteria is marked as requiring event-external context. Topic-bootstrap 95% intervals are [64, 83]% for predictability and [95, 100]% for reasonableness. Therefore, most reactions are judged anticipatable, nearly all are judged legitimate, and critique remains the hardest family (58% predicted vs 88% for praise). This means it is possible to get a nearly full score on ConsumerSimBench. There are plenty of space for agents to reach and surpass the expert's level.
We also want to clarify that
Many criteria are defined as not inferable from the event textis not true. In this experiment, only 2% criteria is marked as requiring event-external context.We acknowledge that temporal contamination may exist and models can memorize these events. However, our bench measures the model's ability to reconstruct consumer reactions rather than memorizing social events. Simply repeating all of the comments will definitely exceed the word limit, so the model must have a knowhow of representative aspects, which is exactly what we want to measure. Knowing the event in advance does not provide additional advantage for our tasks because the criteria itself remain hidden.
We additionally conducted an ablation study where we mask all the entities on 100 topics and 1,454 criteria. Coverage changes from 35.82% to 35.91%, a difference of +0.09 percentage points, which means these entities in LLM's memory do not affect the result.
Thanks for pointing this out. We have expanded the scoring audit to 1,000 distinct criteria across 100 topics, covering outputs from Gemini-3.1-Pro, GPT-5.2, and Claude-Opus-4.6, and use GPT 6.1 sol as judge. Scoring criteria individually versus together within each topic produced identical decisions in 96.9% of cases, supporting the consistency of our batched evaluation.
In Appendix G, we source from Youtube and find a very similar phenomenon. Therefore, these models' performance to reconstruct consumers' reaction is universal and irrelevant with the language or Platform.
We also conducted an automatic check. Screening all 1,553 outputs per model for short Han content, English-dominant text, and refusal/assistant-identity phrases flags 0 GPT-5.2, 4 Opus-4.6, and 1 Sonnet-4.6 outputs. Even assigning perfect scores to every flagged output could increase Opus and Sonnet by at most 0.26 and 0.07 percentage point, so this is not the dominant reason for the low score.
The length limit is desired because we want to ensure fair comparison between models and discourage lengthy comments. In our prompt, we have requested each comment to be 20-400 words. Lengthy comment out of that range will hardly appear in social media (2.24%).
We want to clarify that this is desired behavior. Our main contribution is the benchmark and we want to show improving LLM's empathy may require true insight beyond prompt-engineering. This aligns with the users' feedback that Gemini "seems to have a better EQ" than GPT and Claude, regardless of the prompt. We can not let GPT behave like Gemini in all ways.
Thanks and we have added this work, which focuses on intent understanding. In contrast, ComsumerSimBench focuses on consumer reaction forecasting and reconstruction.
Thanks for pointing this out! This is a typo and we will change it from 13 to 14 in our final version.
We hope they can address all your concerns shown in "Critical Weaknesses". We would be grateful if you could reassess your evaluation given the above response, and we would be happy to respond should you have further questions or unresolved concerns. Thank you!