acceptodds
Withdrawn from ICLR 2027

Understanding and Evaluating Cultural Norm Grounded Reasoning in LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Cultural intelligence in large language models (LLMs) requires not only knowing cultural norms but also reasoning with them under context. Yet existing benchmarks predominantly evaluate cultural knowledge recall, leaving open whether models can effectively invoke acquired norms in multi-constraint reasoning scenarios. We introduce CultureForest, a benchmark for Cultural Norm Grounded Reasoning that explicitly tests this ability. Each of its 5,378 examples is grounded in atomic cultural norms, enabling verifiable and attributable evaluation across 8 domains and 53 countries/regions. CultureForest supports progressive evaluation from multiple-choice (Easy) through binary judgment (Medium) to open-ended generation (Hard), accompanied by a lightweight verifier for scalable open-ended assessment. Extensive experiments reveal that even top-tier models degrade substantially in open-ended settings, with pronounced cross-region disparities. Through multi-faceted analysis, we uncover consistent patterns: (1) test-time reasoning yields limited gains and may exacerbate inequity; (2) models share convergent regional preference structures; (3) responses grow markedly conservative under stricter norms; and crucially, (4) by disentangling cultural knowledge acquisition from reasoning, we demonstrate that the primary bottleneck is not knowledge availability but its effective utilization during reasoning. These findings point to a necessary shift from knowledge-centric evaluation toward measuring knowledge-grounded reasoning in cultural scenarios.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Cultural Intelligence, Cultural Norms, Knowledge-Grounded Reasoning, Open-Ended Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_CU8vOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces CultureForest, a benchmark of 5,378 items covering 53 countries and 8 domains. GPT-4.1 generates each item from three norms extracted from Cultural Atlas and evaluates it in multiple-choice, binary judgment, and open-generation formats. The authors train a small verifier for the open setting and claim that the main bottleneck of LLMs is the use of cultural knowledge during reasoning, not its availability.

Critical strengths

The open-ended evaluation is supported by a carefully validated verifier. C-Verifier is tested under leave-one-model-out and leave-one-domain-out protocols. It outperforms GPT-5.2, Gemini-3.1-Pro, and Qwen3.5-Plus used as judges. The error analysis shows that commercial judges collapse to Neutral in a large share of their disagreements. This finding is useful beyond this benchmark. This is the paper's strongest contribution. However, it is an evaluation tool and, by itself, does not support the central claims.

Critical weaknesses
  1. Reasoning is never measured. All three settings score only the final output. Reasoning traces are not evaluated beyond three qualitative cases. In Section 4, reasoning is defined as accuracy conditional on a knowledge probe. Everything the probe does not explain is attributed to reasoning, including format effects and the distractors added by the refinement agent. The probe itself is weak. Its chance level is 25%. Qwen3.5-0.8B reaches only 43% coverage, yet the paper claims that small models possess the knowledge but cannot use it. The design allows a direct test, since each wrong option violates exactly one norm. Errors could be linked to knowledge of that specific norm. This analysis is not reported. Composition across norms is also never tested. This matters because the knowledge versus reasoning claim is the main contribution stated in the title and conclusion.

  2. The ground truth is Cultural Atlas, and the paper ignores its limitations. Cultural Atlas is written by an Australian institution for an Australian audience. Its profiles are built from interviews with migrant communities living in Australia, and its own documentation states that content may reflect an Australian context. No annotator in the pipeline belongs to the evaluated cultures. Human validation only checks textual consistency between generated options and the written norm. Several norms in the paper’s own examples are not behavioral norms at all. Some are descriptive generalizations about a group, some state the absence of a norm, and one is a legal requirement. Country coverage is also highly unbalanced, from over 300 items to around 50. The benchmark therefore measures agreement with Cultural Atlas, not cultural competence. The regional disparity findings cannot separate model weakness from source bias or sampling variance.

Questions for authors

Can you report the accuracy of each option and link each error to the specific norm the model's knowledge violates?

How were the 53 countries selected from the more than 70 profiles in Cultural Atlas, and do the disparity results hold when controlling for the number of items per country?

Reviewer_SYFTOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents CultureForest, a benchmark of 5,378 everyday situations from 53 countries. Each situation rests on three norms from the Cultural Atlas website, all from one country and one topic. A chain of GPT-4.1 agents writes the scenario, one option that respects all three norms, and three options that each break exactly one norm. Further agents check the options against the norms and rewrite them when needed. The same items are tested in three formats. In Easy, a model picks one of four options. In Medium, it judges each option on its own as appropriate or not, and it counts as correct only if all four judgements are right. In Hard, it answers the question in free text, and a small fine-tuned model called C-Verifier decides whether the answer satisfies each of the three hidden norms. The authors test 19 models and a size ladder of Qwen3.5 and report four findings. Reasoning helps a little, models agree on which countries are hard, answers become more conservative under strict norms, and the main bottleneck is using cultural knowledge and not having it. The data, scripts and verifier are released.

Overall assessment This paper asks a good question and releases enough data and code for readers to check its claims. I used that release, and in its current form I could not confirm the main findings. I recommend rejection. The main reason is that the Easy and Medium results cannot yet be read as a measure of cultural reasoning, and the described validation never asks a member of the cultures concerned to judge an item. I believe the work can become a solid paper, and I list concrete steps below. The strengths, failures and questions fields hold the details.

What works well Testing whether a model can apply several norms to a concrete situation, with and without the norms in the prompt, is the right direction. The agent pipeline with an option-by-option check and repair step is sensible. In my view the most valuable part of the paper is the recombination of Atlas norms into many synthetic situations with LLM agents. Once it is refined and cleaned of the artifact described below, this could be the main contribution, because it makes a large and varied pool of situations cheap to produce. The option matrix, where every option has a known label on every norm, is also useful as an additional measurement for per-norm diagnostics. I appreciate the printed prompts, the released items and the scoring scripts, that allowed me to properly evaluate the work done.

Main concerns Firstly, the Easy and Medium answers can be predicted from the writing style of the options. A plain bag-of-words classifier (my check on the released data) that sees only the four options picks the key in about 90 percent of Easy items, and on Medium it reaches about two thirds, which is higher than any model in Table 2. The signal source is likely the sentence form, but not the cultural norms. It comes from how the pipeline narrates wrong options and describes the right one, and from the option design, where the key is always the option closest to the others. The paper runs no such baseline, so the drop from Easy to Medium, the Medium scores and the analysis of knowledge versus use all have a simpler explanation that is not ruled out.

Secondly, the validation compares text with text. The annotators were told to judge only from the norm text and not from their own knowledge, and the paper relies on the Atlas being expert-verified for the cultural validity of every norm. There is no human baseline and no comparison with other benchmarks, so the scores have no human anchor.

At third, the closest prior work is not engaged. NormAd (Rao et al. 2025) uses the same source, runs the same experiment with and without the norm, has a human baseline, and already shows that models fall short even when the norm is given. The Medium format is the CulturalBench-Hard (Chiu et al. 2025) design. Please state what is new relative to both and compare directly.

At fourth, the verifier learns the conventions of the pipeline. "Violate" is defined by how the wrong options are written, and "Neutral" means off-topic. Its advantage over the LLM judges on the synthetic set mostly reflects this convention, and its margins on real responses are small and come without a significance test. The paper also reads "Neutral" as caution, which weakens the Hard results and the claim that answers turn conservative under strict norms. Together with the issues above, the claim that knowledge is not the bottleneck is not supported at present.

Minor Issues

  • The released scripts leave Qwen3.5 thinking at the template default, which the model cards give as off for the 0.8B and 2B models, so the thinking rows for these sizes are probably non-thinking runs and the size curve mixes two settings
  • More than a fifth of the items rest on holiday names and dates, which test calendar recall more than norm reasoning, and I would report them separately.

A path forward ICLR review form doesn't have a hard character constraint, so I am trying to be as constructive with the feedback as possible at the weakness section. Much of the necessary adjustments can be done in the rebuttal period, but very likely the paper needs a revision and a resubmission. First, report choices-only and single-option baselines next to chance and re-score all models on the items where such a baseline fails. Second, run a style swap on a few hundred items. Third, ask insiders from a few countries to answer a sample and report their agreement with the keys. Fourth, add NormAd and CulturalBench-Hard to the comparison and report how the rankings relate. For a revision, regenerate the options in one neutral template with adversarial filtering, consider a design in which each wrong option breaks two norms, and train and test the verifier on human-labelled real responses with whole items and countries held out. I would be glad to read this work again after that, I believe it has some decent potential.

Critical strengths
  1. The question is worth asking. The paper tests whether a model can apply several norms to a concrete situation, and whether it does better when the norms are placed in the prompt. This goes beyond recalling facts, and it fits where the field is heading. Running the same items in three formats is a clean way to see how the format changes what is measured.

  2. The generation pipeline is built with care. Separate agents write the scenario, the question and the options, and every option is checked against every norm and repaired instead of dropped. The authors report high agreement between the intended pattern and the labels from their human check, and they print all prompts.

  3. The evaluation is broad and consistent. The same metrics are applied to a wide set of recent model families and to a size ladder of one family. Each item is run three times, and results are shown per country. This gives a useful first picture of how current models behave, even though some conclusions need the care I describe below.

  4. The design is clearly explained and the release is generous. The paper explains each construction step, and the authors released the 5378 items, the verifier weights and the scoring scripts. This is what allowed me to test the data directly, and I am thankful for it.

Critical weaknesses

Each point below ends with a way to fix it. The first two are the most serious, and the others either build on them or can be fixed on their own.

1. The right answer in Easy and Medium can be guessed from the writing style of the options, and the paper does not test for this.

I trained a plain bag-of-words classifier on the released items. It sees only the four option texts, with no question, no norms and no country, and it is tested on countries it has not seen. It picks the keyed option in about 90 percent of Easy items, which is close to the strong models in Table 2. On Medium it reaches about two thirds, which is higher than every model in the table. This does not show that a simple classifier beats the LLMs, because it is trained on the labels and the LLMs are not. It shows that the labels can be recovered without any cultural knowledge.

I then ran two more tests to see where the signal comes from. The first keeps only function words and punctuation and still finds the key in about three quarters of Easy items. The second trains on etiquette, business, greeting and communication items and tests on holiday-date items, where it still finds the right option about 86 percent of the time. So what carries over is form and not culture. A look at the words shows how the pipeline writes. Wrong options are narrated as departures ("but", "without", "instead of") and usually come with a friendly excuse. The right option carries compliance commentary ("ensuring", "respectfully") and completeness words ("all", "each", "both"). The rewrite steps meant to neutralize the wording did not remove this, and the signal is spread so widely that deleting the most telling words does not fix it.

This matters because the drop from Easy to Medium, the Medium scores and the whole analysis of knowledge versus use now have a simpler explanation that the paper does not rule out. To fix it, please report choices-only and single-option baselines next to chance, and re-score all models on the items where such a baseline fails. Please also run a style swap on a few hundred items, where the right option gets the departure style and a wrong option gets the compliance style while the actions stay the same. For the benchmark itself, regenerate the options in one neutral template and filter them adversarially until text-only classifiers fall close to chance.

2. The option design itself leaves a structural hint in every item.

Each wrong option breaks exactly one of the three norms, so the right option is always the one closest to the other three. Simply picking the option that looks most like the rest finds the key in about half of the items, where chance is one in four. I also checked every way of adding up to three extra attributes to the options, which makes ~4.4k designs. In each of them the key is either the closest option or the odd one out, so the hint cannot be hidden while each wrong option breaks one norm. If each wrong option breaks two norms instead, all four options are equally far from each other and the hint disappears. The price is that two of the three norms are then enough to find the key. Mixing both designs across items would soften this. Using Medium as the main format would also avoid this particular hint, because options are judged one at a time, but it would not remove the style hint from point 1.

3. Nothing in the validation is anchored in the cultures the benchmark claims to cover.

The paper describes the Cultural Atlas as expert-verified and relies on this for the cultural validity of every norm. The human check was done by professional NLP annotators who were told to judge only from the norm text and not from their own knowledge, and who were not required to belong to any of the cultures. As described, nobody was asked to judge an item as a member of the culture. This is a text-against-text check. It shows that an option matches a sentence, but not that the sentence, the combination of three norms, or the key is right for people in that country. Calling this validation, and calling these annotators experts, goes further than the check shows. More than half of the wrong options break only a norm that the paper itself labels as sometimes acceptable to break, and they come with a friendly excuse, so some keys may be contestable. There is also no human baseline and no comparison with other benchmarks, so a Medium score of 50 percent cannot be read as good or bad. To fix this, please ask insiders from a handful of countries to answer a stratified sample of items, with and without the norms, and report their agreement with the keys and their reasons for disagreeing. Please also show how model rankings on CultureForest relate to rankings on NormAd and CulturalBench-Hard.

4. The work is not placed against its closest prior studies, NormAd and CulturalBench-Hard.

The core idea, take Cultural Atlas norms, generate situations, and test whether models apply the norm with and without it in the prompt, is the idea behind NormAd (NAACL 2025). NormAd uses the same source and 75 countries, has a human baseline, and already reports that models fall well short of humans even when the relevant norm is given. In the submission it is cited once, inside a list, and described as a knowledge benchmark. The name NormAd never appears in the text, and the work is missing from the comparison table and the related work. The Medium format, four separate true/false judgements where all four must be right, is the CulturalBench-Hard design, which also comes with a human baseline. This changes what is new here. To fix this, please say clearly what CultureForest adds, for example three norms per situation, three formats, per-norm attribution and a current model panel. Please put both papers in the comparison table and show at least one conclusion where the added pieces change the result.

5. The option matrix produces training data for the pipeline's way of writing a violation.

The synthetic training pairs for C-Verifier come from the option matrix. A pair is labelled "Violate" when the option was written to break that norm, "Satisfy" when it was written to respect it, and "Neutral" when the norm comes from another topic. The verifier therefore learns what a violation looks like when the pipeline narrates it, which is an explicit and excused departure on the norm's topic. Real model answers are unlikely to violate a norm in that form. The paper's own error analysis says the verifier misses structural violations and is fooled by polite wording, and it finds violations in under 1 percent of cases for almost every model. This is the same kind of problem as in point 1, because the pipeline's way of writing becomes the label. The matrix is useful because every option has a known label on every norm, which allows per-norm diagnostics. It should be treated as an additional measurement and not as the main supervision. To fix this, please train and test the verifier on human-labelled real responses, hold out whole items, norms and countries, say which unit the 9:1 split uses, and report a confusion matrix on real responses.

6. On the synthetic set the verifier's advantage over the LLM judges reflects a label convention, and on real responses its margins are small.

On the synthetic validation set, the biggest group of judge disagreements, about four in ten, is "Satisfy" labelled as "Neutral". The judges say there is no clear evidence for an option that only complies implicitly. The design label says "Satisfy", because the pipeline instructed the writer that each option must involve and satisfy every norm it does not break. The fine-tuned verifier learned this convention. The paper calls the judges' behavior a hedging bias that comes from safety training, and it calls its own result superior robustness. Training on a convention and then scoring against the same convention measures agreement with the convention. On real responses the wins are mostly by a few points on 900 pairs per held-out model, and the verifier loses on the holiday-date and business domains, which together hold about 40 percent of the items. To fix this, please have people from the cultures re-label a sample of the cases where the judges and the design disagree, and report who is right. Please also state the rule for implicit compliance, add a significance test for the claimed wins, and show results on held-out countries. It might reduce amount of cultures in your benchmark in the end significantly, yet the quality of the work would be verifiable and that is important.

7. The Neutral class means off-topic by construction, yet the paper reads it as caution.

In the synthetic data a "Neutral" pair is a norm taken from a different topic of the same country, and in the human labels "Neutral" means thematically unrelated. A response that does not mention a norm is therefore "Neutral", even when it is good. The paper reads a higher "Neutral" share under strict norms as answers becoming more conservative. A simpler reading is that strict norms are mostly taboos, and a good answer about how to greet someone does not need to list what not to do. Under this reading nothing about caution has changed. This affects the Hard results and the claim about conservative answers. To fix this, please split the norms into do and do-not statements, report "Satisfy", "Neutral" and "Violate" rates separately, and control for answer length.

8. The headline claim that knowledge is not the bottleneck is not supported at present.

The claim compares how often models recognize a norm sentence that is shown to them with their Medium score, and it reads the missing part as a failure to use knowledge. Medium is the format that can be solved from writing style (point 1), some keys are contestable (point 3), and there is no human ceiling. So the remaining gap has several explanations that the paper does not separate. The largest effect in the paper also points the other way. Putting the norms into the prompt lifts Medium by about 20 points, which suggests that the models did not have the norms available when they needed them. The probe itself shows the model the norm sentence and asks for true or false, so it tests recognition and not recall. Nunchi-Bench, which the paper cites, already reports that models recognize cultural facts but struggle to apply them. To fix this, please re-derive the claim on cleaned items with a human ceiling, report the probe with several paraphrases of the negation, and say how much of the gap remains after contestable keys are removed.

Questions for authors
  1. How well does a classifier, or an LLM, do on Easy and Medium when it sees only the four options, with no question, no norms and no country? Please report this next to chance in the main table, and also after the four options of each item are rewritten into one neutral template of the same length.

  2. If you rewrite a few hundred items so that the right option uses the departure style of the wrong options and one wrong option uses the compliance style of the right one, with the actions unchanged, do the models follow the style or the action?

  3. Have you considered an option design in which each wrong option breaks two norms, or a mix of designs across items? How does the choices-only baseline change, and how many of the three norms does a model need to see to find the key?

  4. Could people from a handful of the 53 countries answer a stratified sample of Easy and Medium items, with and without the norms? How often do they agree with the keys, and what do they say about wrong options that break only a norm labelled as sometimes acceptable to break?

  5. How do the model rankings on CultureForest compare with the rankings on NormAd and CulturalBench-Hard for the same models, and what does CultureForest show that these two do not?

  6. Which unit was used for the 9:1 split of the synthetic verifier data? How does C-Verifier do when whole items, norms or countries are held out, and when violations are not narrated as an excused departure? Could you share the confusion matrix on the real responses?

  7. Where the LLM judges say Neutral and the design says Satisfy, who is right when people from the culture look at a sample? What is the rule for implicit compliance?

  8. Does the higher Neutral share under strict norms remain when norms are split into do and do-not statements and answer length is controlled?

  9. The released scripts leave Qwen3.5 thinking at the template default, which is off for the 0.8B and 2B models, and they use a sampling temperature that differs from the one stated in the paper. Were the thinking rows for these sizes run with thinking on? Could you rerun the size series with thinking set explicitly for every size and the same temperature?

  10. The Hard score is divided by one global constant, so the highest score a country can reach differs by country, from about 95 to about 119 in the released items. Could you normalize per country and report the country gaps and the agreement between model rankings again?

  11. More than a fifth of the items are built on holiday names and dates. What does it mean to violate such a norm, and what do the results look like when these items are reported separately?

Reviewer_FwnTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CultureForest, a benchmark consisting of 5,378 items spanning 53 different countries/regions and 8 domains, constructed from cultural norms in the Cultural Atlas. Each item combines three atomic cultural norms with a situation and four candidate behaviors. Only one behavior satisfies all three norms while other two violate a different norm. Based on this structure, the authors design 3 evaluation levels: Easy (multiple-choice), Medium (evaluate each option, only counted correct if all are classified correctly), and Hard (open-ended generation). In terms of Hard, the answers are evaluated using C-Verifier, a classifier 0.8B finetuned and verified by leave-one-model-out and leave-one-domain-out, and at the same time, compared with zero-shot LLM judges. Experiments on 19 models show a great drop in performance from Easy to Medium and Hard, with considerable cross-region disparities. The authors also find that test-time reasoning only brings limited improvement, and models tend to have difficulties in similar regions. Their responses become more non-committal under more stricter cultural norms. To further investigate these trendings, the paper analyzes the accuracy at Medium based on whether model recognises at least 2 out of 3 norms through True/False probes. And the paper also examines the effect of explicitly including those norms in the prompt. Based on these analyses, the authors conclude that the primary bottleneck lies not in the lack of cultural knowledge but in the ability of applying those knowledge in reasoning effectively.

Critical strengths

This paper has critical strengths in its design for attributable evaluation. Each item links three atomic cultural norms to four candidate behaviors through a predefined satisfaction. To ensure the reliability, the authors conduct human verification for both answer labels and the strictness labels of norms. They also report a high inter-annotator agreement. Furthermore, evaluating the same content across three formats, in which Medium and Hard independent to the position of the correct option, allows errors to be analyzed by each norm, format, and region. This level of attribution is not commonly explored in existing cultural benchmarks, which adds to the paper’s value.

Another strength is that the open-ended evaluation is carefully designed. C-Verifier is validated on held-out models and domains, giving higher performance than zero-shot LLM judges on most of test set. In addition, the paper also analyzes the error patterns and show that the model rankings in Hard mode are stable across different scoring weights. By releasing the dataset, verifier weights, scoring scripts, the paper makes open-ended cultural evaluation more accessible for future research.

Critical weaknesses
  1. The evidence supporting the main conclusion about the bottleneck in cultural reasoning could be stronger. The paper concludes that the primary bottleneck lies not in the lack of cultural knowledge but in the ability of applying those knowledge in reasoning effectively. However, establishing which factor is more important would require comparing both on a common scale. In the Figure 5a, explicitly providing the three gold norms would improve Medium accuracy by around 17-23% for the top tier models., yet 24-39% of items remain incorrect. The authors assume this as evidence that models struggle to apply knowledge they already possess, supported by a knowledge probe reporting 80-92% coverage. While this explanation reasonable, but probe only need to recognise True/False with norm text provided, similar to the closed form evaluations that the paper itself notes may overestimate model capabilities. Furthermore, the with/without-knowledge comparison is observational, meaning that items whose norms are recognised might be easier. Since this is the main conclusion of the paper, an additional analysis which can directly compare two factors would strengthen the claim.

  2. The novelty and contribution could be better positioned to closely related prior work. The paper presents the gap between possession gand applying cultural knowledge as largely overlooked, although two peer-reviewed studies have investigated closely related questions using the same data source. NormAd (Rao et al., NAACL 2025) construct scenarios from Cultural Atlas, compares settings with and without relevant norms and finds that models remain below human performance even when norms are explicitly provided. On the other hand, SocialCC (ACL 2025) uses Cultural Atlas and identifies a gap between cultural knowledge and its application. These finding are consistent with CultureForest, so the concern is primarily about how the contribution is positioned rather than the validity of the results that paper provide. CultureForest still offers valuable additions but an explicit comparison with these studies, while emphasizing these specific contributions would help clarify paper’s novelty.

Questions for authors
  1. Could the authors clarify which comparison supports the conclusion that knowledge utilization is the primary bottleneck? For example, comparing the performance gain from explicitly providing norms with the remaining error under the same settings. And using human performance as a reference, this would help strengthen the claim. In the human review process, annotators almost perfectly identify all gold labels when they are given norms. Reporting this result as Medium mode item-level accuracy would provide a clearer reference for interesting the remaining gap between models and humans.
  2. Does the accuracy gap between the with-knowledge and without-knowledge groups remain after controlling item difficulty? Besides, would the knowledge coverage 80-92% be remained similar if the probe did not explicitly display the norm text? For example, authors can compare the same item across models or asking models to identify relevant norms from a scenario before seeing them. This would help to distinguish independent knowledge retrieval from recognition. If coverage decreases, the improvement from providing norms may show that models have missing knowledge or limited capability in applying it.
  3. In addition to combining three norms per item and evaluating matched content across formats, could the authors clarify how CultureForest differs from NormAd’s Rule-of-Thumb and Country settings, as well as from SocialCC? Including explanation and both studies in the comparison table and related work section would help readers benefit from understanding the benchmark’s contributions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.