acceptodds
Under review as a conference paper at ICLR 2027

Can LLMs Think Like Consumers? Benchmarking Crowd-Level Reaction Reconstruction with ConsumerSimBench

Abstract

LLMs are increasingly used as “digital consumers” to simulate public opinion, pre-test marketing decisions, and anticipate audience response. However, existing evaluations rarely ask whether a model can reconstruct the concrete reaction patterns that real consumers surface in public discourse. We introduce CONSUMERSIMBENCH, a benchmark built from 1,553 real Chinese social-media topics and 23,122 atomic, rule-audited criteria spanning four reaction families. Rather than scoring open-ended generations with a holistic preference judge, CONSUMERSIMBENCH decomposes each task into auditable yes-no decisions over concrete reaction points, raising three-judge agreement from 65.8% to 92.1% with 98.4% agreement between pointwise judge decisions and human-majority labels. Across 13 frontier generators, the strongest model, Gemini-3.1-Pro, covers only 42.5% of real reaction criteria, while GPT-5.2 and Claude-4.6 trail far behind despite their strength on technical benchmarks. The failures reveal a sharp gap between technical-benchmark performance and socially grounded consumer intuition. Our structured prompt, CONSUMERSCF, yields 0.6–3.8 percentage points higher coverage, while generate-reflect multi-agent refinement improves MiMo-V2.5-Pro from 32.9% to 37.6%. CONSUMERSIMBENCH reframes consumer simulation as a forecasting problem over real public-discourse reactions, showing that frontier LLMs remain far from reliably predicting what consumers will actually care about in high-context Chinese consumer discourse.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, Consumer Simulation, Social Media Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_sLGcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether LLMs can accurately reconstruct consumer reactions to public events, exploring a relatively underexplored research area. The proposed dataset is large-scale, covering 1,553 topics and 23,122 atomic evaluation criteria.

Critical strengths
  1. The task is well defined: generate open-ended comments and score coverage of concrete reaction points.
  2. The scale is large: 1,553 topics, 23,122 criteria, and a multi-model leaderboard with bootstrap intervals.
  3. Pointwise yes/no criteria are more auditable than a single holistic score.
  4. The main pattern is clear: emotion and praise are easier than flashpoints and criticism.
Critical weaknesses
  1. Criteria are drafted by an LLM from topic summaries. The paper does not validate them against raw comments, so 98.4% measures judge agreement, not label fidelity.
  2. No human forecaster is tested, so 42.5% has no ceiling. Many criteria are defined as not inferable from the event text.
  3. Topics are public trends from March–September 2025, with no contamination or retrieval check.
  4. The reliability audit covers only 10 topics and about 100 criteria.
  5. The low GPT and Claude scores may reflect Chinese register or refusals. The paper does not separate these from consumer intuition.
  6. The metric is unweighted recall, judged on the first 3,600 characters. Prompt and agent gains are only 0.6–4.7 points on 100 topics.
  7. Miss citation: ConsintBench: Evaluating Language Models on Real-World Consumer Intent Understanding. https://arxiv.org/abs/2510.13499
  8. [Minor]: The abstract says 13 models, while Table 4 lists 14, including an undeclared GPT-6-astra.
Questions for authors

See Weaknesses.

3 comments
AuthorsOpenReview ↗

Rebuttal by authors

Thank you for recognizing the task, scale, and auditable scoring. Here, we try to address the concerns as follows.

The paper does not validate them against raw comments

Thanks for pointing this out, and we now validate them against raw comments. We use raw comments as input and validate them across the full benchmark (without the input length limit). This yields a 99.17% score, showing that almost all our criteria are possible to gain points.

No human forecaster is tested.

We want to clarify that we already have human studies to measure the correctness of criteria drafting in Appendix H. On a 20-topic subset, one marketing expert achieves a 74% score. After seeing the ground-truth criteria, he marked another 24% as reasonable once disclosed but difficult to anticipate. Only 2% criteria is marked as requiring event-external context. Topic-bootstrap 95% intervals are [64, 83]% for predictability and [95, 100]% for reasonableness. Therefore, most reactions are judged anticipatable, nearly all are judged legitimate, and critique remains the hardest family (58% predicted vs 88% for praise). This means it is possible to get a nearly full score on ConsumerSimBench. There are plenty of space for agents to reach and surpass the expert's level.

We also want to clarify that Many criteria are defined as not inferable from the event text is not true. In this experiment, only 2% criteria is marked as requiring event-external context.

Topics are public trends from March–September 2025, with no contamination or retrieval check.

We acknowledge that temporal contamination may exist and models can memorize these events. However, our bench measures the model's ability to reconstruct consumer reactions rather than memorizing social events. Simply repeating all of the comments will definitely exceed the word limit, so the model must have a knowhow of representative aspects, which is exactly what we want to measure. Knowing the event in advance does not provide additional advantage for our tasks because the criteria itself remain hidden.

We additionally conducted an ablation study where we mask all the entities on 100 topics and 1,454 criteria. Coverage changes from 35.82% to 35.91%, a difference of +0.09 percentage points, which means these entities in LLM's memory do not affect the result.

The reliability audit covers only 10 topics and about 100 criteria.

Thanks for pointing this out. We have expanded the scoring audit to 1,000 distinct criteria across 100 topics, covering outputs from Gemini-3.1-Pro, GPT-5.2, and Claude-Opus-4.6, and use GPT 6.1 sol as judge. Scoring criteria individually versus together within each topic produced identical decisions in 96.9% of cases, supporting the consistency of our batched evaluation.

The low GPT and Claude scores may reflect Chinese register or refusals. The paper does not separate these from consumer intuition

In Appendix G, we source from Youtube and find a very similar phenomenon. Therefore, these models' performance to reconstruct consumers' reaction is universal and irrelevant with the language or Platform.

We also conducted an automatic check. Screening all 1,553 outputs per model for short Han content, English-dominant text, and refusal/assistant-identity phrases flags 0 GPT-5.2, 4 Opus-4.6, and 1 Sonnet-4.6 outputs. Even assigning perfect scores to every flagged output could increase Opus and Sonnet by at most 0.26 and 0.07 percentage point, so this is not the dominant reason for the low score.

The metric is unweighted recall, judged on the first 3,600 characters.

The length limit is desired because we want to ensure fair comparison between models and discourage lengthy comments. In our prompt, we have requested each comment to be 20-400 words. Lengthy comment out of that range will hardly appear in social media (2.24%).

Prompt and agent gains are only 0.6–4.7 points on 100 topics.

We want to clarify that this is desired behavior. Our main contribution is the benchmark and we want to show improving LLM's empathy may require true insight beyond prompt-engineering. This aligns with the users' feedback that Gemini "seems to have a better EQ" than GPT and Claude, regardless of the prompt. We can not let GPT behave like Gemini in all ways.

Miss citation

Thanks and we have added this work, which focuses on intent understanding. In contrast, ComsumerSimBench focuses on consumer reaction forecasting and reconstruction.

[Minor]: The abstract says 13 models, while Table 4 lists 14

Thanks for pointing this out! This is a typo and we will change it from 13 to 14 in our final version.

We hope they can address all your concerns shown in "Critical Weaknesses". We would be grateful if you could reassess your evaluation given the above response, and we would be happy to respond should you have further questions or unresolved concerns. Thank you!

Thank you for the detailed response and additional checks, which address several minor concerns. However, my main methodological concerns remain. First, label fidelity and judge reliability are still primarily assessed through LLM self-checks, without sufficient human validation. Second, the human benchmark is limited to a single expert and 20 topics, making the claimed human ceiling unreliable. Third, the unweighted recall metric overlooks criterion prevalence and hallucinated reactions, while the entity-masking experiment does not fully rule out data contamination. Therefore, I maintain my current score.

Thank you for clarifying the three remaining concerns. Here, we address each remaining concern below.

1. Human validation of label fidelity and judge reliability.

To directly assess label fidelity, we additionally conducted a manual source-verification audit of 118 criteria across 30 topics, covering all four reaction families. The sample was fixed before annotation. One human expert examined each criterion alongside 1-3 retrieved original consumer posts. All 118 criteria were rated as explicitly supported by the supplied source evidence. This provides direct human verification that the audited reaction points reflect actual consumer discourse.

For judge reliability, our existing audit contains yes/no decisions on 13 cases selected for disagreement among automatic judges. The human-majority decision agreed with the original three-judge majority on 11 of the 13 disputed cases (84.6%).

We also report an automated consistency check. As reported in “Are pointwise judges reliable?”, we evaluate both inter-judge consistency and intra-judge stability on a calibration set of 100+ criteria across 10 topics. With the full scoring criteria, GPT-5.2, Gemini-3-Pro, and Claude-4.5-Opus achieve 92.1% three-judge agreement, compared with 65.8% for holistic judging.

These assessments address whether the reference reactions have source support, and whether generated responses satisfy the scoring criteria.

2. The human study and the interpretation of a “human ceiling.”

We recruited a second expert to independently assess the same 100 disclosed criteria using the original assessment form, without access to the first annotator’s labels.

JudgmentOriginal annotatorSecond annotator
Predictable74%83%
Reasonable, but not predictable24%10%
Unreasonable after disclosure2%7%
Cannot judge0%0%

The second expert rated a larger proportion of the criteria as predictable (83% versus 74%). Across the two experts, 93–98% of the criteria were rated as either predictable or reasonable after disclosure. These findings provide additional support for the reasonableness of the criteria and suggest that most are predictable by domain experts, while also revealing some variation in individual judgments.

We will add more experts to this study after the acceptance of the paper.

Also, we do not suggest there is a "ceiling" for our benchmark. By this experiment, we want to show human experts can perform far better than current models on ConsumerSimBench, and that there are much space for improvement, because the unpredicted points are also reasonable.

3. Unweighted recall

Regarding weighting, we directly tested whether our model comparisons depend on equal family weights. Across the full benchmark of 1,553 topics, 23,122 criteria, and 13 models, we assigned 40% weight to each family in turn and 20% to each remaining family. Rankings remained highly stable, with Spearman correlations of 0.984–0.995 against the original ranking. Gemini-3.1-Pro remained first under all four settings, and each setting retained three or four of the original top four models. These results demonstrate that our main model comparisons are robust to the tested changes in family weighting.

4. Hallucinated reactions

We want to clarify that we do not equate "not observed in the finite corpus" with "wrong." The collected reactions provide reliable positive targets, but they cannot enumerate every plausible response to an event. Penalizing every corpus-unmatched reaction would therefore misclassify reasonable long-tail reactions as false positives. For this reason, we retain authentic-reaction coverage as the official metric rather than introduce an incomplete negative-label penalty, and we prevent packing by length limit..

Also, in marketing, we believe it is better to anticipate a broader range of reactions than to miss potentially consequential ones, as marketers can prepare for additional scenarios but cannot afford a failed or unpredicted campaign.

5. Contamination

As stated in the appendix, we do not expect no data contamination. Our insight and experiments prove that gaining the historical data does not help improving the performance, because we are measuring LLM's ability of consumer reaction reconstruction, instead of news memorization.

Thank you again for the constructive feedback, which has helped us clarify and strengthen the paper.

Reviewer_sZmaOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
1
Soundness
2(+1)
Read review
Paper summary

ConsumerSimBench evaluates whether LLMs can reconstruct real consumer reactions to trending topics. It contains 1,553 Chinese social-media topics from RedNote and 23,122 atomic criteria across four dimensions: sentiment flashpoints, emotion keywords, positive aspects, and negative aspects. Instead of holistic LLM judging, it uses criterion-level binary judgments to measure reaction coverage. Across 13 models, Gemini-3.1-Pro achieves the highest reported coverage (42.5%). The authors also explore structured prompting and multi-agent refinement, finding modest improvements. The paper argues that current LLMs struggle to reproduce specific, context-dependent consumer reactions.

Critical strengths

The paper tries to understand whether a model can reconstruct the concrete reaction patterns of real consumers through data collected from social media website RedNote. This is a very relevant problem in marketing and sales. The collected dataset is substantial covering 1553 topics across diverse domains.

Critical weaknesses

Criterion coverage may not be able to model consumer behavior fully.

Ground Truth is generated via an LLM. The paper does not provide any human study to establish if the criteria proposed by the LLM are correct or not.

Prompting and multi-agent improvements are evaluated on only 100 topics; broader validation would strengthen the findings.

Appendix sections are mentioned in the paper but the appendix is not there.

No citations for the models that were evaluated on the curated benchmark.

Questions for authors

Given that frontier models are heavily trained on public web and social media data, how did the authors ensure that high scores (e.g., Gemini-3.1-Pro) reflect true "consumer intuition" rather than data memorization of specific historical social threads?

In line 269, 100 topics are sampled for manual inspection. What is the result of the manual inspection?

Which LLM was used for data annotation?

Did the authors conduct any human study to measure the correctness of criteria drafting? These details would be helpful to understand if LLM annotations are correct.

How was the completeness and correctness of the reference criteria validated beyond judge agreement?

Does the metric penalize unsupported reactions, and how do model rankings change when controlling for output length?

4 comments
AuthorsOpenReview ↗

Thanks for the review. We are concerned that several major criticisms appear to overlook information explicitly provided in our submission. [1/2]

Thank you for recognizing the problem's relevance and dataset scale. We are concerned that several major criticisms appear to overlook information explicitly provided in our submission. Specifically:

  1. We have our appendix in “Supplementary Material: pdf”, so the appendix is not there does not hold.
  2. We have human study to measure the correctness of criteria drafting in Appendix H. On a 20-topic subset, one marketing expert achieves a 74% score. After seeing the ground-truth criteria, he marked another 24% as reasonable once disclosed but difficult to anticipate. Only 2% criteria is marked as requiring event-external context. Topic-bootstrap 95% intervals are [64, 83]% for predictability and [95, 100]% for reasonableness. Therefore, most reactions are judged anticipatable, nearly all are judged legitimate, and critique remains the hardest family (58% predicted vs 88% for praise). This is the result for "In line 269, 100 topics are sampled for manual inspection.", and "The paper does not provide any human study" does not hold.
  3. Many of our evaluated models do not have a technical report, so we do not agree that "No citations for the models that were evaluated on the curated benchmark." is a critical weakness. Nevertheless, thanks for pointing out and we have added the citations for their model card.

We try to address the remaining concerns as follows:

Criterion coverage may not be able to model consumer behavior fully.

ConsumerSimBench focuses on anticipating crowd reactions for the marketing industry. In this scenario, the brand cares what the consumers will say about a post (consumer simulation → our evaluation targets), and why (free-form generation).

As noted in The Verification Horizon, "Every verifier we can build is only a proxy for human intent, never the intent itself" [1]. ConsumerSimBench uses consumer reaction reconstruction as a deliberately scoped proxy for consumer simulation, rather than claiming to measure every aspect of human likeness. Five findings support coverage as a valid measure:

  1. The criteria come from authentic consumer reactions and are hidden from generators;
  2. Consumer reactions are open-ended and cannot be exhaustively enumerated in advance;
  3. Response length budgets prevent unlimited enumeration;
  4. A marketing expert judged 98/100 sampled criteria reasonable.
  5. Our experiment shows that coverage cannot be inflated through packing, repetition reduction, or broad guessing.

Please also see our response to reviewer 9HeU regarding the design of four criteria, and how it can be used in practice.

[1] Qwen team. "The Verification Horizon: No Silver Bullet for Coding Agent Rewards." arXiv:2606.26300 (2026).

Prompting and multi-agent improvements are evaluated on only 100 topics; broader validation would strengthen the findings.

To strengthen the findings, we broaden the number of topics to 1000. The result generally align with the one on 100 topics.

ModelVanilla → SCFImprovement [95% CI]
Gemini 3.1 Pro49.54% → 52.57%+3.03 [2.08, 3.96]
Gemini 3 Flash50.32% → 52.02%+1.71 [0.83, 2.58]
GPT-4o36.95% → 38.16%+1.22 [0.36, 2.09]

Given that frontier models are heavily trained on public web and social media data, how did the authors ensure that high scores (e.g., Gemini-3.1-Pro) reflect true "consumer intuition" rather than data memorization of specific historical social threads?

Our bench measures the model's ability to reconstruct consumer reactions rather than memorizing social events. Simply repeating all of the comments will definitely exceed the word limit, so the model must have a knowhow of representative aspects, which is exactly what we want to measure. Knowing the event in advance does not provide additional advantage for our tasks because the criteria itself remain hidden.

We additionally conducted an ablation study where we mask all the entities on 100 topics and 1,454 criteria. Coverage changes from 35.82% to 35.91%, a difference of +0.09 percentage points, which means these entities in LLM's memory do not affect the result.

Which LLM was used for data annotation?

The retained initial-drafting configuration requests gemini-3-flash-preview. It receives the event brief together with filtered source-post text and is instructed to ground each criterion in those reactions. Separate hardening and pruning configurations default to GPT-5.2.

AuthorsOpenReview ↗

Thanks for the review. We are concerned that several major criticisms appear to overlook information explicitly provided in our submission. [2/2]

Did the authors conduct any human study to measure the correctness of criteria drafting?

Yes. In appendix H, we have ensured the criteria drafting and the LLM's judgement to be well aligned with human. We have also tested a human expert's performance on our benchmark.

Additionally, on the same 100-topic subset, we have performed a QC to ensure all the criteria can be matched by a real comment.

We also gathered all original comments and judged them as input (without length limit). This yields a 99.17% score, which shows that our criteria can almost all be satisfied.

How was the completeness and correctness of the reference criteria validated beyond judge agreement?

As stated above, a benchmark can never guarantee "completeness", instead we try to align our criteria with the marketers' real expectations and ensure the quality and correctness. We have human study to measure the correctness of criteria drafting in Appendix H.

Does the metric penalize unsupported reactions?

We do not equate "not observed in the finite corpus" with "wrong." The collected reactions provide reliable positive targets, but they cannot enumerate every plausible response to an event. Penalizing every corpus-unmatched reaction would therefore misclassify reasonable long-tail reactions as false positives. For this reason, we retain authentic-reaction coverage as the official metric rather than introduce an incomplete negative-label penalty.

How do model rankings change when controlling for output length?

Controlling for output length changes some model positions, while preserving the top-ranked model. We re-scored frozen outputs from all 13 generators on the same 100 topics under 600-, 1,200-, and 2,400-character prefix caps, plus an exactly matched per-topic budget defined as the shorter of 1,200 characters and the shortest response across the 13 models. All conditions used GPT-6.1-sol as judge.

Gemini-3.1-Pro ranks first in all four conditions. The 1,200- and 2,400-character rankings have Spearman ρ = 0.973, whereas the exactly matched budget versus the 1,200-character cap gives ρ = 0.709. Under exactly matched lengths, Gemini-3.1-Pro scores 40.60%, exceeding GPT-5.2 by 6.78 percentage points (95% paired CI [3.49, 10.18]), Claude-Opus-4.6 by 11.10 points [7.87, 14.51], and Claude-Sonnet-4.6 by 13.25 points [10.09, 16.53]. Thus, equalizing visible output length preserves these performance gaps, although individual leaderboard positions depend on the length budget.

We hope they can address all your concerns shown in "Critical Weaknesses". We would be grateful if you could reassess your evaluation given the above response, especially update the Critical Weakness and Rating to an accurate version that reflect the true status of our paper. We would be happy to respond should you have further questions or unresolved concerns. Thank you!

Thank you for your detailed response and rebuttal. I want to apologize and acknowledge that I missed the supplementary material during my initial review phase. Thank you for pointing that out. The supplementary material, the expanded 1,000-topic scaling, and the length-control ablations successfully clarify some of my initial concerns. However, I have three remaining concerns:

The vanilla baselines shift considerably between the main paper (e.g., GPT-4o at 22.1%) and the rebuttal table (GPT-4o at 36.95%). Similar score shifts are observed for the Gemini variants as well. Could you please explain this discrepancy? For a large-scale benchmark containing 23,122 criteria across 1,553 topics, relying on a single expert with no inter-annotator agreement metrics makes it difficult to validate the correctness and quality of the gold-standard criteria. As mentioned in the rebuttal, gemini-3-flash-preview was used to draft the initial criteria, and Gemini models occupy the top #1 and #2 spots in Table 4. This raises the concern of self-preferential or model-family bias, and the paper does not provide an ablation or control ruling out this drafting bias.

In light of the supplementary material and rebuttal introducing further ablations, I am increasing my Informativeness and Soundness scores to 2, and raising my overall rating to 2. Further clarification around the baseline score shifts, single-expert validation limits, and potential Gemini drafting bias will make the submission much more complete.

Thanks for the follow-up feedback! We address the remaining concerns below.

1. Baseline discrepancy.

Thank you for highlighting this discrepancy. We traced it to a change in the judge used in the supplementary evaluation pipeline. Because GPT-5.2 was no longer available, we initially used GPT-6.1-sol-xhigh for the rebuttal experiments. Upon closer inspection, however, we found that its judgments on the marketing tasks was very poor, showed limited agreement with our expert annotations (only 64.10%), and differed substantially from those of other model families.

To enable a consistent comparison, we have now used GPT-5.4 to re-score the exact same 1,000 Vanilla responses alongside the newly generated SCF responses:

GeneratorHistorical GPT-5.2-Judge: VanillaGPT-5.4-Judge: VanillaGPT-5.4-Judge: SCF
Gemini 3.1 Pro42.51%42.68%44.32%
Gemini 3 Flash42.00%40.63%39.9%
GPT-4o22.23%20.28%20.25%

Under the same judge, SCF improves the score of Gemini 3.1 Pro by 1.64 percentage points, while slightly reducing the scores of Gemini 3 Flash and GPT-4o. These results suggest that SCF provides limited and inconsistent gains on our benchmark for the models tested.

2. Independent human assessment and agreement.

We recruited a second expert to independently assess the same 100 disclosed criteria using the original assessment form, without access to the first annotator’s labels.

JudgmentOriginal annotatorSecond annotator
Predictable74%83%
Reasonable, but not predictable24%10%
Unreasonable after disclosure2%7%
Cannot judge0%0%

The second expert rated a larger proportion of the criteria as predictable (83% versus 74%). Across the two experts, 93–98% of the criteria were rated as either predictable or reasonable after disclosure. These findings provide additional support for the reasonableness of the criteria and suggest that most are predictable by domain experts, while also revealing some variation in individual judgments.

We will add more experts to this study after acceptance of the paper.

3. Criterion-constructor sensitivity.

We evaluated sensitivity to the criterion-construction model on the 54 topics whose criteria were revised during rule refinement, as these topics may be particularly sensitive to construction choices. Using the same inputs, we redrafted the criteria with GPT-5.6 and Claude-Sonnet-5, then evaluated identical outputs from 11 generators against each criterion set. The resulting sets contained 875 original criteria, 576 GPT-drafted criteria, and 704 Claude-drafted criteria.

The top-four set of generators remained unchanged under both evaluation metrics. The model rankings obtained with the GPT- and Claude-drafted criteria were strongly correlated with the original rankings, with Spearman’s ρ of 0.927 and 0.964, respectively. These results suggest that, on the topics examined, the comparative rankings are largely robust to the choice of a strong criterion-construction model.

Thank you again for the constructive feedback, which has helped us clarify and strengthen the paper.

Reviewer_9HeUOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors propose a consumer simulation benchmark based on real user data from Chinese social media for crowd-level reaction reconstruction. They also propose fine-grained atomic reaction criteria to detect four different reaction behaviors, which can be better evaluated using an LLM as the judge. On a 100-topic subset, our structured prompt, CONSUMERSCF, yields 0.6–3.8 percentage points higher coverage without changing the model ordering.

Critical strengths
  • The social simulation of real consumer reactions is an interesting topic to explore. As mentioned in the paper, real consumers’ behavior can be complex and implicit, including satire or boasting, which requires social reasoning from LLMs. Meanwhile, the dataset is large-scale, covering four super-categories and twenty consumer-facing subfields, with detailed data statistics disclosed.
  • The authors propose atomic reaction criteria, which make the evaluation of consumer reaction simulation more interpretable.
  • Evaluation of existing SOTA LLMs shows that the task is still challenging for current LLMs, which could be a potential field for better model development.
Critical weaknesses
  • The implications of this dataset are unclear. How can it be used in practice for user attitude surveys? How accurate can attitude predictions be, even if they cover all four criteria?
  • The motivation for selecting these four criteria is unclear. How well can they indicate the quality of user reaction simulation? How should we understand the scale of this score? What constitutes a significant improvement in this score? How sensitive is it? Would it align better with human evaluation than open-ended questions? I would like to raise my scores if this concern can be resolved.
  • The improvement from the prompting method CONSUMERSCF is limited.
Questions for authors
  • Will you make the dataset public?
  • How sensitive are the evaluation metrics?
2 comments
AuthorsOpenReview ↗

Rebuttal by authors [1/2]

Thank you for recognizing the task's importance, scale, and interpretability. Here, we try to address your concerns as follows.

How can it be used in practice for user attitude surveys?

ConsumerSimBench tests whether digital consumers can anticipate the specific issues people actually raise, which is a critical capability for marketing decision-making. Before launching a campaign, decision-makers need to anticipate whether it will generate enthusiasm or trigger a viral backlash, and understand what will drive that response.

The marketing industry today does this bad. For example, Arc’teryx’s mountain fireworks campaign attracted public criticism over its environmental impact. Their marketing team failed to anticipate such concrete environmental concern that could turn a promotional event into a reputational crisis, resulting in substantial financial and reputational losses.

In contrast, ConsumerSimBench directly evaluates this capability by testing whether LLMs recover the specific flashpoints, emotions, praise targets, and criticism targets observed in real consumer discourse. It can evaluate whether an LLM or agentic approach is capable of anticipating the mass reactions and assist marketing decisions.

How accurate can attitude predictions be, even if they cover all four criteria? How well can they indicate the quality of user reaction simulation? ... The motivation for selecting these four criteria ...

From a marketer’s perspective, what they expect is exactly captured by our four criteria (that is: What will consumers focus on? Will they like it? Which positive or negative aspects will they discuss in their posts?) Therefore, a model that accurately predicts these criteria should be able to anticipate the consumer reaction before it is ever posted, which would be very useful for marketers.

We show these criteria are not redundant. Within a single topic, pairwise correlations across criteria is only 0.123–0.302. A model that captures one topic's flashpoints is barely more likely to capture its criticism targets, so collapsing any one of the dimension would discard much information. However, across the 13 generators, each family alone reproduces the leaderboard, where the rank correlation between the splits is 0.874–0.978. So the four behave like four near-independent indicators of one latent factor.

To further show the accuracy, we provide a case where the model's response covers all four criteria:

On the topic "A Guide to Capturing the Midsummer-Night Blue Hour,” Gemini-3.1-Pro achieves a perfect score of 100% (6/6). Its generated reactions include “The blue-hour atmosphere is absolutely stunning—cool and sophisticated,” “Try using a small cool-colored flashlight to illuminate the face,” and “The beautiful light lasted less than ten minutes; by the time I finished posing, it was already completely dark.” These correspond to reference expressions from real users such as “Under the Klein-blue sky, the pale-pink and white flowers look like silent snow,” “The orange-yellow light from a flashlight creates a beautiful contrast with the blue sky,” and “It is a little disappointing that I missed the best shooting spot.” The model therefore captures not only generic positive sentiment, but also the specific reactions consumers express, such as aesthetic appreciation, practical advice, satisfaction with successful outcomes, and disappointment when the ideal moment is missed.

As noted in The Verification Horizon, "Every verifier we can build is only a proxy for human intent, never the intent itself" [1]. ConsumerSimBench uses consumer reaction reconstruction as a deliberately scoped proxy for consumer simulation, rather than claiming to measure every aspect of human likeness. Five findings support coverage as a valid measure:

  1. The criteria come from authentic consumer reactions and are hidden from generators;
  2. Consumer reactions are open-ended and cannot be exhaustively enumerated in advance;
  3. Response length budgets prevent unlimited enumeration;
  4. A marketing expert judged 98/100 sampled criteria reasonable;
  5. Our experiment shows that coverage cannot be inflated through packing, repetition reduction, or broad guessing.

[1] Qwen team. "The Verification Horizon: No Silver Bullet for Coding Agent Rewards." arXiv:2606.26300 (2026).

AuthorsOpenReview ↗

Rebuttal by authors [2/2]

How should we understand the scale of this score?

The score indicates the percent of correct reactions the model can predict, and a gain means more coverage of the reference inventory under the protocol. In real world marketing scenario, the marketer usually expect most of the possible reactions to be predicted before release.

In the expert review of 100 criteria over 20 topics, one marketing expert achieves a 74% score. After seeing the ground-truth criteria, he marked another 24% as reasonable once disclosed but difficult to anticipate. Only 2% criteria is marked as requiring event-external context. Topic-bootstrap 95% intervals are [64, 83]% for predictability and [95, 100]% for reasonableness. This means it is possible to get a nearly full score on ConsumerSimBench. There are plenty of space for agents to reach and surpass the expert's level.

What constitutes a significant improvement in this score?

We interpret score improvements through the additional consumer reactions they recover, rather than fluency alone. Across 1,553 topics, Gemini-3.1-Pro outperforms GPT-5.2 by 13.80 percentage points, corresponding to 3,718 net additional covered reference criteria. The largest gains are in emotions (+24.76 points) and positive aspects (+16.71), followed by flashpoints (+8.11) and negative aspects (+5.62).

For a concrete example, on the Asia Pet Fair topic, both models capture pets socializing and crowded conditions. Gemini additionally captures satisfaction with free samples and discounts, positive exhibitor-service experiences, physical exhaustion, and intention to attend next year. The collected posts explicitly praise successful giveaway hauls and booth-specific generosity, complain about the physical burden of carrying those hauls, and express plans to return. With the same 15-comment request and identical saved 1,200-character judging windows, coverage increases from 13.99% to 60.12%. The improvement therefore changes the reconstructed audience profile: it captures not only the social appeal of the event, but also the benefits, brand interactions, and participation costs that visitors actually discuss.

How sensitive are the evaluation metrics?

We conducted judge reliability test in 5.3 and found our yes-no criteria yield substantially higher agreement both with inter-judge and with human. We also report 95% CI for each model's score in table 4.

Here, we also test different combinations to calculate the total score:

Alternative aggregationSpearman correlationTop-four overlap
Equal-family pooled coverage1.0004/4
All-criterion micro coverage0.9884/4
Topic macro excluding empty families0.9954/4
40% weight to each family and 20% to each other family0.984–0.9953-4/4

The broad comparison is stable across all seven tested alternatives, showing our metrics are stable.

Would it align better with human evaluation than open-ended questions?

We reckon that human evaluation itself is open-ended. Without specific rubrics and "ground-truth", it is impossible to give such a score. On the contrary, providing rubrics and "ground-truth" will downgrade the problem settings to our current scenario, and we have shown that LLM's yes-no judge yields no significant difference with human.

Nevertheless, we provide a paired comparison between each model's response. For each pair, the annotator chooses which comment is better on his own preference, like the Chat Arena. After 2,600 votes, Gemini still ranks #1, with an ELO of 1375.9. This shows that our result actually has both a good interpretation and alignment.

The improvement from the prompting method CONSUMERSCF is limited.

Thanks for pointing out, and we want to clarify that this is expected. This shows that our benchmark is generally robust to prompt-engineering methods, and improving LLM's empathy may require true insight. This aligns with the users' feedback that Gemini "seems to have a better EQ" than GPT and Claude, regardless of the prompt. We can not let GPT behave like Gemini in all ways.

Will you make the dataset public?

Yes. An anonymous link is already available in "ETHICS STATEMENT", and we will release it after acceptance.

We hope they can address all your concerns shown in "Critical Weaknesses". We would be grateful if you could reassess your evaluation given the above response, and we would be happy to respond should you have further questions or unresolved concerns. Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.