- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper asks why models sometimes violate cultural principles they can correctly recognize. DualCQ builds a challenge set by finding requests that produce violations without guidance but appropriate answers when the principle is supplied. CulSafe contains 10,661 principle–query pairs across 30 topics and 167 regions. The authors evaluate 18 models, analyze their cultural-awareness and principle-matching failures, and introduce CAPME, which learns from a teacher given the relevant principle. CAPME improves adherence over explicit reflection and SFT across several model families.
Critical strengths
-
The recovery requirement is a useful part of the benchmark. Removing it raises the violation rate from 81.80% to 90.08%, but recovery falls from 88.96% to 53.20% (Table S5). This shows why simply searching for failures would produce a different, less focused test. The benchmark has a clear purpose: testing whether models can apply relevant knowledge without being handed the answer.
-
The training results are reasonably broad. CAPME is tested across model families and sizes, with SFT and single-aspect variants as controls. For Llama-3.1-8B, FullTest adherence rises from 50.51% with reflection to 62.50% with CAPME; SFT reaches 58.46% (Table 1). The human audit of the judge is also helpful: it agrees with 90 of 97 consensus-labeled CulSafe responses (Appendix B.5).
Critical weaknesses
-
The scope of generalization needs to be clearer. CulSafe deliberately searches for failures, so its 61.50% gap should not be read as the failure rate on ordinary user requests. FullTest also regenerates questions for the same principles, most of which were used in training (Appendix E.1). FailTest provides a separate held-out test, but contains model-specific failure cases. I would like to see FullTest results separated by seen and unseen principles. The benefit of joint awareness/matching supervision is also less clear than the overall CAPME improvement: awareness-only training outperforms the joint method for Qwen2.5-7B adherence on both test sets (Table S22).
-
Some of the explanation goes beyond what the evaluation establishes. GPT-4o helps filter principles, construct questions, and judge answers. The response-label audit is useful, but does not validate the principles themselves or their applicability to different communities. This is particularly relevant for value-based items, where disagreement need not imply an unsafe answer. Better reflection text and better answers also do not establish that the stated reflection steps caused the improvement. A separate technical point needs clarification: Equation 3 is an expectation under a conditional response distribution, but Equation 4 adds probability weights to samples reportedly drawn from that distribution. The sampling proposal and weight normalization need to be specified before the connection to the theoretical bound can be assessed.
Questions for authors
- Can you split FullTest into principles seen and unseen during training, report both subsets, and describe near-duplicate filtering between original and regenerated queries? This would clarify whether the main improvement reflects principle acquisition, query transfer, or both.
- What distribution actually generates the reference responses in Equation 4? Are the weights normalized over sampled responses, and what importance-sampling correction is used? Please explain precisely which properties of the exact surrogate survive the proxy-likelihood implementation.
- Does CAPME still outperform an equally budgeted awareness-only variant across seeds? The Qwen2.5-7B results suggest joint supervision is not uniformly better. Please report paired uncertainty for the relevant differences.
- How were principle applicability and acceptable exceptions checked, particularly for value-based items? Can you provide a stratified human audit that assesses the principles and scenarios themselves, in addition to adherence labels?
4 comments
Thank you for your thoughtful reviews and suggestions. Response to Reviewer f5Ld
W2-2, Q2: What distribution actually generates the reference responses in Equation 4? Are the weights normalized over sampled responses, and what importance-sampling correction is used? Please explain precisely which properties of the exact surrogate survive the proxy-likelihood implementation.
(1) In practice, we input the paired query and principle into LLMs and instruct them to generate the reference responses and . However, we can generate responses by instruction, but their actual distribution probability value may not be always accessible (such as third party LLMs). By Bayes' rule, we take the transformation as following:
\frac{ P_{\theta}(\neg p\mid\bar{x},y) P_{\theta}(y\mid\bar{x}) }{ P_{\theta}(\neg p\mid\bar{x}) }.$$ Since it is not tractable to go through all the responses, we sample finite responses to conduct proxy as Equation 4, where **we actually use $P_{\omega}$ adherence classifier and an accessible proxy model to approximate the probabilities to estimate $\pi_{\bar{x}}^{-}(y)$** as following: $$w_k^{-}\approx \frac{ P_{\omega}(\neg p\mid\bar{x},y_k^{-}) P_{\theta}(y_k^{-}\mid\bar{x}) }{ P_{\omega}(\neg p\mid\bar{x}) }.$$ Specifically, we employ the GPT-4o to perform as the $P_{\omega}$ adherence classifier and open-source llama-3.1-8B as the proxy model to get $P_{\theta}(y_k^{-}\mid\bar{x})$ as mentioned in lines 231-235. (2) As it is struggle to go through all the responses, it thus is hard to know the overall situation of all responses for taking exact correction, and we sample finite responses to take a estimation. Considering normalized weights on finite responses will shift the distribution within the samples, we only use an unnormalized, proxy-weighted finite-sample surrogate as Equation 4 shows. The surrogate conducts the positive and negative reference responses generation and principle-dependent response weighting. This provides a practical mechanism for generating plausible reference responses. (3) In addition, according to the B.1, we have $\Delta = S_{\bar x}(x)-S_{\bar x}(\bar x)>0$. Similarly, we get $\hat \Delta_k=\hat S_{\bar x,k}(x)-\hat S_{\bar x,k}(\bar x)$ in the sampled responses. We denote $e_{k}=|\hat \Delta_k-\Delta|$ as the maximum error when we estimate $S(x)$ by the finite responses, then we can have $-e_k<\hat\Delta_k-\Delta<e_k\Rightarrow \hat\Delta_k>\Delta-e_k$. When it satisfies $e_k<\Delta$, it exists $\hat\Delta_k>0 \Rightarrow \hat S_{\bar x,k}(x)>\hat S_{\bar x,k}(\bar x)$. In empirically, the analysis experiments of DualCQ in Appendix B.2 denote it is achievable and the actual implementation is effective. Specifically, Figure S1 (b) shows that the violation risk and safety recoverability (the two core components included in Equation 3) significantly improve along with the iterative refinement, **with violation risk increases from 63.46% to 81.80% and recoverability rises from 77.82\% to 88.96\%, respectively**.Thank you for your thoughtful reviews and suggestions. Response to Reviewer f5Ld
W1-2, Q3: Does CAPME still outperform an equally budgeted awareness-only variant across seeds? The Qwen2.5-7B results suggest joint supervision is not uniformly better. Please report paired uncertainty for the relevant differences.
We address this issue from the aspects below.
(1) Across-seed results and performance differences. We conduct additional runs with different random seeds. The tables report mean adherence ± standard deviation, together with the mean difference between joint CAPME and awareness-only. Joint training has higher mean adherence for Llama-3.1-8B on both test sets and Mistral-7B on FullTest, but not for Qwen2.5-7B.
| FailTest | Qwen2.5 7B | Llama3.1 8B | Mistral 7B |
|---|---|---|---|
| Reflection | 36.84 | 44.32 | 39.17 |
| CAPME | 46.62 ± 0.21 (+9.78) | 59.21 ± 0.17 (+14.88) | 53.90 ± 0.45 (+14.73) |
| aware_only | 46.64 ± 0.89 (+9.80) | 57.88 ± 1.08 (+13.56) | 54.11 ± 0.98 (+14.94) |
| match_only | 44.09 ± 0.75 (+7.25) | 55.46 ± 0.28 (+11.14) | 50.22 ± 0.30 (+11.05) |
| Mean Δ (CAPME − aware_only) | −0.02 | +1.33 | −0.21 |
| FullTest | Qwen2.5 7B | Llama3.1 8B | Mistral 7B |
|---|---|---|---|
| Reflection | 42.33 | 50.51 | 45.98 |
| CAPME | 50.16 ± 0.27 (+7.82) | 62.10 ± 0.35 (+11.59) | 58.08 ± 0.35 (+12.10) |
| aware_only | 50.84 ± 0.53 (+8.51) | 61.11 ± 0.07 (+10.60) | 57.28 ± 0.38 (+11.30) |
| match_only | 47.82 ± 0.46 (+5.49) | 58.83 ± 0.27 (+8.32) | 53.74 ± 0.73 (+7.76) |
| Mean Δ (CAPME − aware_only) | −0.68 | +0.99 | +0.80 |
(2) Interpretation for their uncertain differences. These results do not support uniform superiority of joint CAPME, we analyze a possible explanation is that the two forms of supervision address distinct steps but can provide overlapping benefits.
-
Awareness supervision helps models identify which aspect of a request requires cultural consideration. Once that aspect is identified, models may already retrieve the relevant principle from their existing knowledge. For example, noticing that the number of items matters in a Chinese wedding gift may be sufficient to activate the known taboo against giving four items, even without additional matching supervision. This is consistent with our results: awareness-only training improves safety over Reflection (9.8-14.94 percent points on FailTest, 8.51-11.3 percent points on FullTest) and yields larger gains than matching-only training across all models.
-
Principle matching helps models determine how relevant cultural knowledge applies to the current request, providing more specific guidance for response generation. The safety improvements of matching-only training over Reflection (7.25-11.14 percent points on FailTest, 5.49-8.32 percent points on FullTest) demonstrate the value of matching supervision.
-
When considering both sensitivity awareness and principle matching, awareness provide the direct gains. Additional matching supervision may provide partially overlapping benefits, and can also shift how the model learns to reason about a request under joint optimization. Its incremental benefits become limited even erode the exist gains.
Thank you for your thoughtful reviews and suggestions. Response to Reviewer f5Ld
W2-1, Q4: How were principle applicability and acceptable exceptions checked, particularly for value-based items? Can you provide a stratified human audit that assesses the principles and scenarios themselves, in addition to adherence labels?
We address principle validity and scenario applicability through two complementary checks.
(1) The principles are commonly applicable and acceptable. Our principles originate from established public cultural resources. To further examine their validity, we manually audit a stratified sample of 500 principles, checking each item against the Cultural Atlas (https://theculturalatlas.org/), which is a professional research team promoting inclusion in an increasingly culturally diverse society and they collect diverse cultural principles commonly accepted on specific regions. The audited principles show high consistency with the materials of Cultural Atlas.
(2) It is highly compatible between the principles and queries. The queries are generated under the guidance of their paired principles and the constraints of the DualCQ algorithm to ensure that the principles apply to the queries. We also assess the compatibility between the principles and the queries, obtaining a high compatibility score of 88.3.
These results indicate that the principles are broadly accepted within the contexts they describe and are applicable to the corresponding queries. We will emphasize the discussion about principle applicability and query compatibility in the revision, as well as more details about the auditing process.
Ethics Concerns: Please clarify the scope of ethics statement, human annotation procedure, and handling of within-culture disagreement.
Thanks for your valuable suggestion. The wording in lines 494–497 is overly broad, and we will revise it to clarify the human annotation procedure and the fairness/bias considerations associated with studying cultural norms.
(1) Cultural principles, fairness, and within-culture disagreement. Our principles are drawn from established, publicly available cultural datasets rather than newly established by us. We use these documented principles as research references, not to endorse particular cultural values or present regional norms as universally binding.
Our study focuses on the knowledge–behavior gap: whether LLMs adhere to principles they correctly recognize when answering contextualized requests. It does not adjudicate the universal correctness of each principle or resolve disagreements among members of a culture.
We acknowledge that existing sources may incompletely represent within-culture diversity. Future work will incorporate more diverse community perspectives and better-validated, context-specific principles to reduce representational bias and accommodate reasonable exceptions.
(2) Human annotation scope and procedures. The annotations in Appendix B.5 involve reference-grounded judgments of final-response adherence. Annotators receive the query, reference principle, and model response; they are not asked to disclose personal beliefs or independently establish cultural norms. The task does not require collecting additional personal information.
In the revision, we will report the annotators’ relevant backgrounds, recruitment and instructions, the measures used to protect their privacy and well-being.
Thank you for your thoughtful reviews and suggestions. Response to Reviewer f5Ld
Thanks for your supportive review and valuable suggestions. Our responses to each Weaknesses (W) and Questions (Q) are as follows. A revision of our paper will be uploaded, with the revised parts marked in blue.
W1-1, Q1: Can you split FullTest into principles seen and unseen during training, report both subsets, and describe near-duplicate filtering between original and regenerated queries? This would clarify whether the main improvement reflects principle acquisition, query transfer, or both.
(1) Our method performs well in both seen and unseen subsets. In fact, we split the CulSafe into train data and FailTest, where the principles in FailTest do not appear during training. The improvements in FailTest denotes the effectiveness of CAPME in unseen principles, where CAPME improves 9.03-14.93 percent points over Reflection method on FailTest.
(2) Furthermore, following your suggestion, we divide FullTest into seen and unseen subsets based on whether the corresponding principles appear during training. We compare changes in principle adherence and the knowledge–adherence gap across these two subsets, as shown in the following table. We observe improvements in both subsets of FullTest: 7.97-12.13 percent points on seen subsets and 5.87-14.13 percent points on unseen subsets, demonstrating that our method remains effective even when neither the principles nor the corresponding queries have been seen during training.
(3) Few duplicate queries between CulSafe and FullTest. Furthermore, we compare the queries in CulSafe, which is used for training, with those in FullTest. Specifically, we compute the jaccard between 3-gram/5-gram spans of queries in CulSafe and FullTest (denoted as 3-gram Jaccard and 5-gram Jaccard), and we find there is a very limited overlap between them, providing additional evidence of the differences between the two datasets. Together, these results suggest that the improvements are not primarily driven by the principle acquisition or query transfer.