acceptodds
Under review as a conference paper at ICLR 2027

CRISP: Cultural Reward Modeling for Implicit Situated Propriety

Abstract

As large language models (LLMs) are increasingly deployed across countries and regions, the ability to recognize and respond appropriately to diverse cultural contexts becomes increasingly important. However, existing research has largely focused on cultural knowledge or tasks with predefined response spaces, while open-ended culturally situated behavior remains comparatively underexplored. In this work, we introduce CRISP-RM, a culturally situated reward model that assigns rewards according to cultural appropriateness in open-ended social scenarios. During policy optimization, we further introduce Norm Grounding Supervision (NGS), providing guidance that enhances the policy's sensitivity to relevant cultural norms. To construct culturally situated data, we employ a collaborative multi-agent framework that instantiates implicit cultural norms into diverse social scenarios and further curate NormCompass as a dedicated testbed. We conduct comprehensive experiments to evaluate the effectiveness of CRISP-RM in both reward modeling and policy optimization. Best-of-\(N\) experiments show that CRISP-RM consistently outperforms strong general reward models. During GRPO policy optimization, CRISP-RM consistently improves culturally situated behavior, while incorporating NGS yields further gains. Further analyses demonstrate the advantages of CRISP-RM in distinguishing culturally appropriate behavior beyond superficial fluency and politeness, while NGS provides complementary gains during policy optimization by improving norm grounding.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Cultural Alignment, Reward Modeling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ozH7OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper collects cultural norms from CulturalBench, FORK, and NormAd, and uses a multi-agent process to instantiate them in social scenarios with implicit norms. The data are split by norm, with 3,521, 449, and 222 scenarios for training, validation, and testing, respectively; the NormCompass test set contains only norms from NormAd. Multiple models generate candidate answers; GPT-5.6 Sol assigns scores from 1 to 5, and untied within-scenario scores are converted into preference pairs. CRISP-RM trains a scalar reward head on Qwen3-0.6B or Qwen3-4B with the standard Bradley–Terry loss. During policy training, the 4B reward model reads the scenario and final answer only; its score is calibrated with validation-set statistics, mapped through a sigmoid, and used by GRPO. An optional 8B Norm Grounding Supervisor (NGS) additionally reads the gold norm and observable rationale and supplies a Grounded probability as an auxiliary reward. The paper evaluates the cultural reward with Best-of-N selection, GRPO for three policy models, CultureForest and CulShield, and controlled triplet analyses. These experiments support gains from culturally adapted reward data and provide evidence that NGS can improve observable norm grounding.

Critical strengths
  1. The data isolation and external evaluation are strong enough to make the empirical gains meaningful.
  2. The experiments are substantial and provide evidence for the effectiveness of each major design module.
Critical weaknesses
  1. The reward is validated against a supplied norm, but the independent evidence for “cultural appropriateness” remains limited. Preference annotation, the NormCompass final score, norm-matching diagnostics, and the construction and filtering of controlled triplets all rely on GPT-5.6 Sol. This is not test-label leakage, but it can make both training and analysis favor one evaluator's interpretation of the norm. The evaluation of 100 responses by two human annotators is useful support (human–GPT Krippendorff's α = 0.732), but the paper does not report the annotators' cultural experience, coverage across cultures, disputed cases, or whether optimized responses were included. Thus, the evidence mainly establishes agreement with scoring under the supplied norm; it does not establish that the norm should take priority in each scenario.
  2. The incremental value over culture-specific reward methods is not tested directly enough. CRISP uses a conventional scalar reward model, Bradley–Terry training, and GRPO; the main additions are contextual preference data and norm-grounding supervision. Therefore, outperforming general reward models that were not adapted to this domain establishes that specialized data are useful, but does not establish the relative advantage of the proposed design. The paper cites and discusses Think-as-Locals but does not provide an experimental comparison, even though that work also addresses cultural preferences and surface-cue interference. Although CuSiR is included, Appendix C.3 describes only a CuSiR-style three-dimensional reward and does not provide its time-weighting formula or enough adaptation detail for verification. CuSiR-style and Skywork degrade by roughly 19–34 points on CultureForest, and the degradation remains large on valid outputs. This important comparison needs a reproducible implementation and failure analysis rather than an emphasis only on parameter scale.
Questions for authors
  1. Could evaluators familiar with the relevant cultures blind-rate responses before and after optimization, while allowing them to mark the target norm as inapplicable, identify reasonable exceptions, or accept multiple actions? Can report the model sources, cultural distribution, and annotator expertise for the existing 100-response human evaluation, as well as human–model disagreements rather than only aggregate agreement?
  2. If a same-sized general reward model is adapted using the same preference data, how much of CRISP's advantage remains?
  3. NGS simultaneously adds the gold norm, an 8B model, and rationale supervision. What happens in controls that use only NGS, allow the answer reward to access the gold norm, or supervise only the norm rationale? After obtaining 88.31% macro-F1 on 117 test examples excluding other failure types, does NGS remain reliable for optimization-induced errors, generic platitudes, and responses whose rationale is correct but whose action is opposite to the norm?
  4. Figure 2 and Figure 6 use the test set to show training progress and λ sensitivity. Were the final checkpoint and λ selected entirely on an independent validation set? Could you specify the selection rule, repeated-training results, the actual default λ, and the additional training cost of NGS?
5 comments
AuthorsOpenReview ↗

Response to Weaknesses

We sincerely thank the reviewer for the thoughtful and constructive feedback. We appreciate the concerns regarding cultural appropriateness, the contributions of CRISP, and the reliability of our experimental evaluation. These comments have motivated us to conduct additional experiments and provide further clarification. We address each concern in detail below.

Concerns About the Cultural Appropriateness of the Data. We thank the reviewer for raising these concerns. We provide additional clarification from three perspectives. First, to reduce reliance on a single evaluator, we independently rescored 100 human annotated responses using Gemini 3.8 Flash and DeepSeek V4 Pro. Both evaluators show strong correlations with GPT-5.6 Sol (Spearman ρ = 0.684/0.690), indicating a degree of consistency across different models. Second, our original human evaluation primarily aims to examine whether model scores align with human judgments of whether a response adheres to a given norm, rather than directly validating the applicability of these norms in specific contexts. Third, regarding concerns about norm applicability, we acknowledge that the broad cultural coverage of our study makes it challenging to recruit annotators with relevant expertise across all represented cultures. To address this, we additionally employ two top tier models, Gemini-4-Argon and DeepSeek-V4-Pro, to filter our testbed based on the contextual applicability of the target norms. As shown in the table below, our filtered testbed contains 204 instances, and the experimental results remain comparable to those obtained on the original testbed. Furthermore, optimization using our constructed training data yields performance improvements on both our testbed and CultureForest, providing additional evidence supporting the quality of our training data.

Performance Comparison on the Filtered NormCompass Testbed

Policy ModelsNormCompass
Qwen3-4B3.33
+CuSiR3.11
+Skywork-Reward-V2-Qwen3-4B3.56
+CRISP3.75
+CRISP + NGS3.95
Qwen3-8B3.75
+CuSiR3.65
+Skywork-Reward-V2-Qwen3-4B3.62
+CRISP4.00
+CRISP + NGS4.06
DeepSeek-R1-Distill-Qwen-7B2.36
+CuSiR2.17
+Skywork-Reward-V2-Qwen3-4B2.80
+CRISP3.03
+CRISP + NGS3.13

Concerns About the CRISP. We thank the reviewer for this suggestion. We would like to clarify that situated cultural preference data and the resulting trained reward model constitute the core contributions of CRISP. Our goal is not to introduce a new reward model architecture or optimization algorithm, but to develop a specialized reward model capable of distinguishing culturally appropriate behaviors in situated contexts.

We additionally compare against Think-as-Locals using Qwen3-4B as the policy model. Think-as-Locals achieves scores of 3.58 on NormCompass and 57.52 on CultureForest, both lower than those achieved by CRISP+NGS.

Regarding CuSiR, we implemented the three dimensional reward and dynamic curriculum weighting consistent with the original paper and adapted them to single turn GRPO training. We will provide additional implementation details in Appendix C.3.

Regarding the performance degradation on CultureForest, we believe it stems from reward models assigning excessively high rewards to responses that appear superficially plausible but violate cultural norms. Figure 3 has already shown that baseline models are susceptible to such superficial plausibility. Further analysis of the training logs reveals two representative cases: (1) In a Ukrainian flower giving scenario, Skywork assigned a high reward of 0.9985 to an incorrect response that insisted on giving six roses, rationalizing the number six as a symbol of luck, harmony, and love. (2) In a Korean workplace drinking scenario, CuSiR assigned a cultural reward of 0.95 to an inappropriate recommendation to maintain eye contact while drinking, which the policy justified as balancing humility and active engagement. These cases suggest that reward biases in baseline models may be further amplified during policy optimization.

AuthorsOpenReview ↗

Response to Questions For Authors-1

Q1: We thank the reviewer for this valuable suggestion. Following the reviewer's recommendation, we conducted an additional human evaluation comparing model responses before and after policy optimization to further validate the effectiveness of CRISP+NGS. Specifically, two human annotators independently rated responses generated before and after optimization with CRISP+NGS. As shown in the table, both annotators observed substantial improvements, with the average score increasing from 3.57 to 4.19. These results further demonstrate that CRISP+NGS not only improves automatic evaluation scores but also enhances culturally appropriate behavior as judged by human annotators, providing additional evidence of our method's effectiveness independent of the automatic evaluator.

Human Evaluation Results Before and After Policy Optimization

Human AnnotatorBefore Optimization+CRISP+NGSImprovement
Annotator 13.624.28+0.66
Annotator 23.524.10+0.58
Average3.574.19+0.62

Q2: We thank the reviewer for this suggestion. The core contribution of CRISP lies in constructing situated cultural preference data and training a specialized cultural reward model using these data, rather than designing a new model architecture. To investigate whether our approach depends on a specific model initialization, we further use Skywork Reward V2 as the base model and train it on the same CRISP preference data. Experimental results show that models with different initializations achieve comparable performance at both the 0.6B and 4B scales. These findings suggest that our cultural reward modeling approach is not dependent on a particular model initialization and can be effectively applied to existing general purpose reward models.

Best-of-N Results with Different Reward Model Initializations

RMSizeNormCompass @8NormCompass @16CultureForest @8CultureForest @16
Random—3.60143.610456.619957.1828
CRISP (Original)0.6B3.78383.828858.866158.7889
CRISP (Skywork Initialization)0.6B3.78383.799558.715558.6499
CRISP (Original)4B3.88963.914459.638260.2969
CRISP (Skywork Initialization)4B3.90323.934759.805060.4017

Q3-1: To investigate the contributions of individual NGS components, we conducted additional ablation experiments using Qwen3-4B. The results show that, compared with the base Qwen3-4B model, using NGS alone improves performance on NormCompass (3.79) but leads to a noticeable decline on CultureForest (53.06). Adding only norm grounding rationale supervision (CRISP + rationale) improves performance on CultureForest (58.97), but underperforms CRISP alone on NormCompass. In contrast, the full CRISP + NGS achieves the best results on both benchmarks (3.86 / 60.28), suggesting that cultural reward signals and norm grounding supervision are complementary, and their combination leads to more comprehensive performance improvements.

SettingNormCompassCultureForest
Qwen3-4B3.2756.18
+CRISP3.6957.60
+CRISP + Rationale3.4558.97
+NGS Only3.7953.06
+CRISP + NGS3.8660.28
AuthorsOpenReview ↗

Response to Questions For Authors-2

Q3-2: We thank the reviewer for raising concerns about the reliability of NGS. To further evaluate its reliability in complex error scenarios, particularly its ability to detect norm recognition errors arising during policy optimization and distinguish different types of incorrect responses, we conducted three additional experiments to systematically assess the effectiveness and limitations of NGS.

(1) Norm Error Detection in Real Training Trajectories. To assess the reliability of NGS during policy optimization, we randomly sampled 160 responses from the reinforcement learning trajectories of Qwen3-4B. After GPT based verification, we identified 79 rollouts that failed to correctly recognize the target cultural norm. Among these, NGS assigned a p(pass) below 0.5 to 66 responses (83.54%), with an average p(pass) of 0.1873 across all 79 responses.

(2) Generic and Vague Responses. We constructed generic variants of 50 culturally appropriate responses that lacked specific guidance grounded in cultural norms. NGS assigned an average p(pass) of 0.8898 to the original responses, compared with only 0.0036 for the generic variants. Moreover, NGS preferred the original responses in all 50 paired comparisons. (Paired comparison refers to comparing the p(pass) values of the original and modified responses to the same question.)

(3) Inconsistency Between Norm Understanding and Action. Using the same 50 original responses, we further constructed variants that correctly identified the cultural norm but recommended actions that violated it. NGS assigned these variants an average p(pass) of 0.2566 and preferred the original responses in 94% (47/50) of paired comparisons.

These results demonstrate that NGS can detect most norm recognition errors arising during actual policy optimization and effectively distinguish generic responses from those providing specific norm grounded guidance. Even in the more challenging cases where the norm is correctly understood but the recommended action is inappropriate, NGS maintains strong relative discrimination ability.

Q4: We thank the reviewer for their attention to our experimental setup. We provide the following clarifications. Regarding checkpoint selection, all GRPO experiments use a fixed training budget of 880 steps and are evaluated using the final checkpoint, without selecting checkpoints based on validation or test set performance. Regarding the choice of λ, all main experiments use λ = 0.1. Figure 6 is intended solely to analyze the impact of different λ values on model performance, rather than to select the final hyperparameter. Regarding the computational overhead of NGS, NGS introduces an additional forward pass through the Qwen3-8B classifier at each GRPO training step.

Reviewer_edavOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a reward model and a synthetically constructed dataset to help evaluate responses to open-ended questions that require some sort of cultural understanding. They have some sort of multi-agent loop that basically draws from norms extracted from previously released datasets. Their multi-agent loop constructs scenarios and questions across 19 cultures, which they call Norm Compass. 10 LLMs answer each of these constructed questions. A GPT-5.6 judge scores them, and these are converted into preference pairs, using which they train CRISP-RM, a reward model. This reward model generalizes beyond their constructed dataset to two more datasets they use, called Culture Forest and CulShield. They further show that extra supervision, which exposes the model to the cultural norm it needs to use to answer the question, helps.

Critical strengths

I think the dataset and model contributions are the biggest strengths. The dataset construction, although to me it seems a little complex in design, would be useful for testing open-ended behavior (which requires cultural norm understanding or how to respond in certain cultural behavior scenarios). That is important, and the model could be potentially used to improve this behavior downstream for models, because today getting good ratings on these sorts of tasks is very hard. People usually have to rely on human ratings which often tend to have disagreements themselves.

Critical weaknesses
  1. Even though the dataset construction seems interesting, the entire section 3.2 has multiple modules. They have a situated reasoner, a response advisor, a revision planner, a question reviser, and so on. This dataset construction is complex to understand. In their Figure 1, the left side is easy to understand, but the right side is hard. It's also not grounded in any sort of prior literature as to why you'd need these multiple modules. This makes me trust the dataset overall a little less, but if the authors have any explanation as to how they designed this, that might be helpful.

  2. Poor writing: I think there are several parts where the authors could improve the writing:

a) In the introduction they mostly just say that we improve things and that it gives additional gains. This would be better with some more concrete numbers even in the abstract.

b) Line 216, they say things like, "we evaluate model performance with norms provided as reference." I would appreciate them specifying what the instruction is, what the system prompt is, and some sort of qualitative example of how this is evaluated.

c) In section 4 (Culture Forest) and in section 5 (Culture Forest, CulShield, Skywork, and CuSIR), all are introduced without any explanation or citation. I don't think these are commonly known benchmarks or models. I would appreciate a section titled "Baselines" and "Datasets" where you specify which datasets you're using with citations and you also give a one-line explanation for each of these frameworks and datasets.

d) I would also appreciate moving Figure 4 to the top somewhere because it took me a while to actually understand what you are trying to do when you say you are generating scenarios and questions in section 3.1. It was very hard for me to understand what these might look like.

  1. I believe they use GPT-5.6 Sol for everything, like for constructing the dataset, for evaluating the models, etc., which introduces some sort of circular dependency. It's hard to understand if the gains would be different if they used a different model as an evaluator. Generally, I'd recommend using separate models for constructing your dataset and for evaluation, especially if you're evaluating on the same dataset that you constructed.
Questions for authors

Q1) Why do some frameworks use Llama 3 in Table 4? Is it fair to compare these baselines with your model, given that the pre-trained base model to begin with is different?

Q2) Why do the baselines drop so much for Culture Forest in Figure 2? Are you sure your model is not doing some sort of reward hacking behavior? Could you provide some qualitative examples?

Q3) How do you think your base Qwen model would perform in the generative setting, best-of-N, without your training?

Q4) how many ratings did you collect from humans to calculate the agreement between annotators and GPT in Table 3?

4 comments
AuthorsOpenReview ↗

Response to Weaknesses

We sincerely thank the reviewer for the thoughtful and constructive feedback. We appreciate the concerns regarding our data construction pipeline, evaluation reliability, and clarity of presentation. These comments have helped us identify areas that require further clarification and additional analysis. We address each concern in detail below.

About Data Construction. Our multi-agent pipeline aims to construct high quality cultural scenarios to support the training and evaluation of culture aware models. We argue that a well designed cultural scenario should not explicitly reveal the target norm, while providing sufficient and unambiguous contextual clues that allow the norm to be reasonably inferred and culturally appropriate behavioral decisions to be made. We assess these conditions through the responses of frontier models. When the target norm is explicitly revealed, or when even state of the art models fail to infer the target norm, we consider that the scenario or question may contain design flaws and revise it accordingly. In contrast, when a model correctly identifies the target norm and provides appropriate behavioral advice but fails to adequately recognize relevant contextual evidence or provide sound reasoning, this may also indicate potential issues with the scenario, such as insufficient contextual clues or flawed design. However, since the model has already identified the target norm and made an appropriate behavioral decision, such failures may instead stem from deficiencies in response generation rather than the scenario itself. Therefore, we prioritize providing targeted feedback to guide answer regeneration, rather than immediately modifying the scenario. If repeated attempts remain unsuccessful, we reconsider the validity of the scenario. Two representative examples illustrate the effectiveness of our iterative framework. (1) Indian Head Nodding (Norm Leakage): The target norm is that vertical head nodding indicates agreement. The initial scenario described Riya's gesture as a "confirming nod", explicitly revealing its meaning. Our framework revised it to "moves her head up and down once or twice", requiring the model to infer whether the gesture indicates agreement. (2) Ukrainian Flower Giving (Missing Key Clues): The target norm is that bouquets should contain an odd number of flowers. The initial scenario described Olena considering removing a broken flower from a bouquet intended for her piano teacher, but omitted the original number of flowers, making it impossible to determine whether the action would violate the norm.

Circular Dependency. To address concerns about potential circular dependencies in evaluation, we independently rescored 100 human annotated responses using Gemini 3.8 Flash and DeepSeek V4 Pro. The scores from both evaluators show strong positive correlations with those from GPT-5.6 Sol (Spearman ρ = 0.684/0.690). Furthermore, CRISP also achieves performance improvements on CultureForest, an independent benchmark developed by other researchers, providing further evidence that our method is not effective only on our self constructed testbed.

Writing and Presentation Issues. We apologize for the lack of clarity in certain parts of the manuscript and will make the following improvements in the revised version.

(a) Abstract and Introduction. We apologize for the lack of quantitative results in the current abstract and introduction. In the revised version, we will include key numerical results to more clearly demonstrate the performance improvements achieved by our method.

(b) Evaluation Setup. We have actually provided the prompt used for automatic evaluation in Table 19: Prompt used for automatic evaluation of culturally situated responses in the appendix. However, we acknowledge that this was not clearly referenced in the main text. We will add an explicit reference to this table and provide concrete evaluation examples to help readers better understand the evaluation process.

(c) Baselines and Datasets. We thank the reviewer for this suggestion. We will add dedicated Datasets and Baselines subsections to the main text, including relevant citations and brief descriptions. Specifically, CultureForest primarily evaluates models' ability to perform situated reasoning based on cultural norms, while CulShield focuses on understanding and adhering to cultural taboos. Skywork is a general purpose reward model, whereas CuSiR improves situated cultural reasoning through imitation learning and reinforcement learning. We will clarify these datasets and methods in the revised manuscript to improve the readability of our experimental setup.

(d) Placement of Figure 4. We apologize that the current organization makes it difficult for readers to intuitively understand the scenario and question generation process. In the revised version, we will move Figure 4 to an earlier position.

AuthorsOpenReview ↗

Response to Questions For Authors

  • Q1: The Llama 3 series models in Table 4 are general purpose reward models specifically trained for reward modeling, included for comparison with representative existing methods. Furthermore, we include Skywork Reward V2 as a baseline, which shares the same Qwen3 architecture and parameter scale as CRISP-RM, to minimize the impact of architectural differences and ensure a fairer comparison.
  • Q2: We believe the substantial performance degradation of baseline models on CultureForest stems from reward models assigning excessively high rewards to responses that appear superficially plausible but violate cultural norms. The controlled triplet experiment in Figure 3 has already revealed this issue. Further analysis of the training logs identifies two representative cases: (1) In a Ukrainian flower giving scenario, Skywork assigned a high reward of 0.9985 to an incorrect response that insisted on giving six roses, rationalizing the number six as a symbol of luck, harmony, and love. (2) In a Korean workplace drinking scenario, CuSiR assigned a cultural reward of 0.95 to an inappropriate recommendation to maintain eye contact while drinking, which the policy justified as balancing humility and active engagement. These cases suggest that reward biases in baseline models may be further amplified during policy optimization. Regarding concerns about potential reward hacking in CRISP, Figure 2 already demonstrates consistent performance improvements on the independent CultureForest benchmark, while Figure 3 further validates CRISP-RM's ability to distinguish culturally appropriate behaviors. Together, these findings suggest that CRISP's improvements reflect genuine gains in culturally appropriate behavior rather than merely exploiting reward model biases.
  • Q3: To address this question, we extend our Best-of-N experiments by including the original Qwen3 models at three parameter scales (0.6B, 4B, and 8B) without CRISP training. The results, using Qwen3-8B and Llama3.1-8B as policy models, are presented in the table below.
ModelsNormCompass @8NormCompass @16CultureForest @8CultureForest @16
Random3.6033.61256.62757.219
Qwen3-0.6B3.6053.60557.43157.431
Qwen3-4B3.6553.65257.96358.038
Qwen3-8B3.6823.67058.38358.523
CRISP-0.6B3.7863.82958.87958.787
CRISP-4B3.8923.91459.66960.320
  • Q4: We collected human ratings for 100 responses to assess the agreement between human annotators and GPT.

Thank you for your valuable feedback on our paper. We have carefully addressed your concerns in our rebuttal and provided additional experiments and analyses. We hope these responses adequately address your questions. As the discussion period is limited, we would greatly appreciate hearing your thoughts on our responses. If you have any remaining concerns or questions that require further clarification, we would be happy to continue the discussion and provide additional information. Thank you again for your time and effort. We look forward to your feedback.

As the peer review discussion period is approaching its end, we would like to kindly follow up once again on our rebuttal and additional experiments. We have carefully considered your questions and suggestions and made substantial efforts to address your concerns through additional experiments and further clarifications. We would greatly appreciate it if you could review our responses before the discussion period concludes and let us know whether they have adequately addressed your concerns. If any issues remain unresolved, we would be more than happy to provide further clarification and engage in additional discussion. Thank you again for your time and consideration. We look forward to hearing your feedback.

Reviewer_wH5KOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies reward modeling for culturally appropriate behavior in open-ended social scenarios. Basically there are three contributions: NormCompass, a test set to evaluate culturally situated scenarios; CRISP-RM, a scalar reward model for cultural appropriateness; and introducing Norm Grounding Supervision (NGS), an auxiliary reward used during RL. First, to build the data, the authors take norms from CulturalBench, FORK, and NormAd, and use GPT 5.5 to place each norm into up to three different social settings without stating it, followed by multi-agent loop that checks and revises the scenarios, which gives about 4.2k scenarios over 19 cultures, of which 222 held-out items form NormCompass. Then, CRISP-RM (0.6B and 4B) is trained with a Bradley-Terry loss on preference pairs built from GPT-5.6 Sol’s 1-5 scores of responses from many LLMs. Finally, NGS is a classifier that rewards rationales which recover the gold norm, and it is added to the CRISP-RM reward during GRPO. The authors report that CRISP-RM beats much larger general reward models in Best-of-N, that GRPO with CRISP-RM improves cultural behavior while general reward models degrade it, that NGS gives further gains, and that general reward models tend to prefer polite answers that conflict with the norm.

Critical strengths

The introduced reward model seems different than the other reward models, since generic reward models tend to prefer responses that are just simply polite even when the action conflicts with the cultural norm (e.g., the German farewell case), while CRISP-RM, as shown in the analysis, seemed to be able to prefer the culturally appropriate action more consistently (using LLM judge evaluation though). This difference also carries over to the policy model, where optimizing with CRISP-RM does not degrade like the generic reward models do.

Critical weaknesses

First, I just find that other than the source benchmarks, the whole pipeline is LLM-made: norm extraction, scenario generation, the refinement loop where GPT verifies its own scenarios, the training labels, and the evaluation. The only human check (Table 3) scores 100 responses, but annotators are given the target norm, so it only shows GPT scores like humans assuming the norm is correct. It does not check whether the norm actually fits each generated scenario, which matters since norms vary even within one culture (as the ethics statement itself notes). The paper also does not say who the 2 annotators are or whether they are from the cultures involved. Furthermore, there is no evidence that the multi-agent loop improves quality, such as discard rates or an ablation without it. So it is hard to tell whether the data and rewards reflect what people from these cultures actually consider appropriate, and I think this needs a larger human study to back up the claims.

Now on the other side, for me it is somewhat unclear why CRISP-RM is needed over simpler alternatives. The only baselines are general scalar RMs and a reimplemented CuSiR. The most natural baseline such as rubric-based judge or a rubric-trained reward model and other generative cultural RM are not compared. There are also no ablations of the pipeline itself (refinement loop, chronotopes, augmented responses) for the policy model improvements, so it is unclear which design choice actually drives the gains.

Questions for authors

See critical weaknesses.

4 comments
AuthorsOpenReview ↗

Data Construction and Human Annotation

We sincerely thank the reviewer for the thoughtful and constructive feedback. We appreciate the reviewer's concerns regarding data quality, human evaluation, and the effectiveness of our iterative refinement pipeline. These comments have motivated us to conduct additional analyses and provide further clarification of our methodology. We address these concerns in detail below.

1. Norm Sources and Human Evaluation. Our final testbed directly adopts the Rules of Thumb provided by NormAd, rather than relying on norms extracted by LLMs, thereby ensuring the reliability of the target norms. We conceptualize our evaluation task as a reading comprehension task, where the goal is to determine whether a response adheres to a given norm. The human evaluation in Table 3 aims to examine the agreement between GPT scores and human judgments under this task formulation. Furthermore, given the broad cultural coverage of our study, recruiting annotators with sufficient expertise across all represented cultures is challenging. We therefore employ two top tier models, Gemini-4-Argon and DeepSeek-V4-Pro, to filter our testbed. Although these models cannot fully replace human annotators with relevant cultural expertise, we believe they can provide meaningful quality assurance. Specifically, we retain only scenario–question pairs that are deemed culturally appropriate by both models. As shown in the table below, our filtered testbed contains 204 instances, and the experimental results remain comparable to those obtained on the original testbed. Furthermore, models trained on our dataset achieve improvements on both the filtered testbed and CultureForest, providing further evidence of the quality of our training data.

Performance Comparison on the Filtered NormCompass Testbed

PolicyNormCompass
Qwen3-4B3.33
+CuSiR3.11
+Skywork-Reward-V2-Qwen3-4B3.56
+CRISP3.75
+CRISP + NGS3.95
Qwen3-8B3.75
+CuSiR3.65
+Skywork-Reward-V2-Qwen3-4B3.62
+CRISP4.00
+CRISP + NGS4.06
DeepSeek-R1-Distill-Qwen-7B2.36
+CuSiR2.17
+Skywork-Reward-V2-Qwen3-4B2.80
+CRISP3.03
+CRISP + NGS3.13

2. Effectiveness of Multi Agent Iterative Refinement. Among the 4,908 initial instances, 2,650 (54.0%) underwent question revision, 81 (1.65%) underwent answer regeneration, and 478 (9.74%) were ultimately discarded, leaving 4,430 instances. Two representative examples illustrate the effectiveness of our iterative framework. (1) Indian Head Nodding (Norm Leakage): The target norm is that vertical head nodding indicates agreement. The initial scenario described Riya's gesture as a "confirming nod", explicitly revealing its meaning. Our framework revised it to "moves her head up and down once or twice", requiring the model to infer whether the gesture indicates agreement. (2) Ukrainian Flower Giving (Missing Key Clues): The target norm is that bouquets should contain an odd number of flowers. The initial scenario described Olena considering removing a broken flower from a bouquet intended for her piano teacher, but omitted the original number of flowers, making it impossible to determine whether the action would violate the norm. Our framework added the critical detail of "nine tulips", allowing the model to assess whether removing one flower would be appropriate without revealing the correct answer. These examples demonstrate that our iterative framework addresses specific issues, such as norm leakage and insufficient contextual information.

AuthorsOpenReview ↗

Necessity of CRISP and Comparisons with Alternative Reward Modeling Approaches

We appreciate the reviewer's concerns regarding the necessity of CRISP and the adequacy of baseline comparisons. To address these concerns, we conducted additional experiments to further validate the effectiveness of our approach.

3. Necessity of CRISP and Motivation for Cultural Reward Modeling. The necessity of CRISP-RM stems from the fact that general purpose reward models are often misled by polite wording and superficial plausibility, making it difficult for them to accurately identify implicit cultural behavioral errors. To validate this, we further include RubricRM, Prometheus-2-7B, and Qwen3-8B in the controlled cultural preference discrimination experiment in Section 6. As shown in the table, these models still underperform CRISP-RM. Furthermore, the extended Best-of-N experiments in Section 4 demonstrate that even CRISP-0.6B outperforms larger general purpose reward models in response selection. In addition, the GRPO experiments in Section 5 already include CuSiR, a method based on multidimensional rubrics, as a baseline. Together, these experiments provide evidence for the necessity of specialized situated cultural reward modeling across three dimensions: preference discrimination, response selection, and policy optimization.

Extended Results for Reward Model Discrimination Analysis (Section 6)

ModelsA > GA > OG > O
CRISP-0.6B84.5195.7794.37
CRISP-4B80.2897.1898.59
RubricRM-4B54.9353.5243.66
Prometheus-2-7B61.9771.8373.24
Qwen3-8B57.7567.6159.15

Extended Best-of-N Results

ModelsNormCompass @8NormCompass @16CultureForest @8CultureForest @16
Random3.6033.61256.62757.219
Qwen3-0.6B3.6053.60557.43157.431
Qwen3-4B3.6553.65257.96358.038
Qwen3-8B3.6823.67058.38358.523
Prometheus-2-7B3.6053.60957.85357.901
CRISP-0.6B3.7863.82958.87958.787
CRISP-4B3.8923.91459.66960.320

Thank you for your valuable feedback on our paper. We have carefully addressed your concerns in our rebuttal and provided additional experiments and analyses. We hope these responses adequately address your questions. As the discussion period is limited, we would greatly appreciate hearing your thoughts on our responses. If you have any remaining concerns or questions that require further clarification, we would be happy to continue the discussion and provide additional information. Thank you again for your time and effort. We look forward to your feedback.

As the peer review discussion period is approaching its end, we would like to kindly follow up once again on our rebuttal and additional experiments. We have carefully considered your questions and suggestions and made substantial efforts to address your concerns through additional experiments and further clarifications. We would greatly appreciate it if you could review our responses before the discussion period concludes and let us know whether they have adequately addressed your concerns. If any issues remain unresolved, we would be more than happy to provide further clarification and engage in additional discussion. Thank you again for your time and consideration. We look forward to hearing your feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.