Each point below ends with a way to fix it. The first two are the most serious, and the others either build on them or can be fixed on their own.
1. The right answer in Easy and Medium can be guessed from the writing style of the options, and the paper does not test for this.
I trained a plain bag-of-words classifier on the released items. It sees only the four option texts, with no question, no norms and no country, and it is tested on countries it has not seen. It picks the keyed option in about 90 percent of Easy items, which is close to the strong models in Table 2. On Medium it reaches about two thirds, which is higher than every model in the table. This does not show that a simple classifier beats the LLMs, because it is trained on the labels and the LLMs are not. It shows that the labels can be recovered without any cultural knowledge.
I then ran two more tests to see where the signal comes from. The first keeps only function words and punctuation and still finds the key in about three quarters of Easy items. The second trains on etiquette, business, greeting and communication items and tests on holiday-date items, where it still finds the right option about 86 percent of the time. So what carries over is form and not culture. A look at the words shows how the pipeline writes. Wrong options are narrated as departures ("but", "without", "instead of") and usually come with a friendly excuse. The right option carries compliance commentary ("ensuring", "respectfully") and completeness words ("all", "each", "both"). The rewrite steps meant to neutralize the wording did not remove this, and the signal is spread so widely that deleting the most telling words does not fix it.
This matters because the drop from Easy to Medium, the Medium scores and the whole analysis of knowledge versus use now have a simpler explanation that the paper does not rule out. To fix it, please report choices-only and single-option baselines next to chance, and re-score all models on the items where such a baseline fails. Please also run a style swap on a few hundred items, where the right option gets the departure style and a wrong option gets the compliance style while the actions stay the same. For the benchmark itself, regenerate the options in one neutral template and filter them adversarially until text-only classifiers fall close to chance.
2. The option design itself leaves a structural hint in every item.
Each wrong option breaks exactly one of the three norms, so the right option is always the one closest to the other three. Simply picking the option that looks most like the rest finds the key in about half of the items, where chance is one in four. I also checked every way of adding up to three extra attributes to the options, which makes ~4.4k designs. In each of them the key is either the closest option or the odd one out, so the hint cannot be hidden while each wrong option breaks one norm. If each wrong option breaks two norms instead, all four options are equally far from each other and the hint disappears. The price is that two of the three norms are then enough to find the key. Mixing both designs across items would soften this. Using Medium as the main format would also avoid this particular hint, because options are judged one at a time, but it would not remove the style hint from point 1.
3. Nothing in the validation is anchored in the cultures the benchmark claims to cover.
The paper describes the Cultural Atlas as expert-verified and relies on this for the cultural validity of every norm. The human check was done by professional NLP annotators who were told to judge only from the norm text and not from their own knowledge, and who were not required to belong to any of the cultures. As described, nobody was asked to judge an item as a member of the culture. This is a text-against-text check. It shows that an option matches a sentence, but not that the sentence, the combination of three norms, or the key is right for people in that country. Calling this validation, and calling these annotators experts, goes further than the check shows. More than half of the wrong options break only a norm that the paper itself labels as sometimes acceptable to break, and they come with a friendly excuse, so some keys may be contestable. There is also no human baseline and no comparison with other benchmarks, so a Medium score of 50 percent cannot be read as good or bad. To fix this, please ask insiders from a handful of countries to answer a stratified sample of items, with and without the norms, and report their agreement with the keys and their reasons for disagreeing. Please also show how model rankings on CultureForest relate to rankings on NormAd and CulturalBench-Hard.
4. The work is not placed against its closest prior studies, NormAd and CulturalBench-Hard.
The core idea, take Cultural Atlas norms, generate situations, and test whether models apply the norm with and without it in the prompt, is the idea behind NormAd (NAACL 2025). NormAd uses the same source and 75 countries, has a human baseline, and already reports that models fall well short of humans even when the relevant norm is given. In the submission it is cited once, inside a list, and described as a knowledge benchmark. The name NormAd never appears in the text, and the work is missing from the comparison table and the related work. The Medium format, four separate true/false judgements where all four must be right, is the CulturalBench-Hard design, which also comes with a human baseline. This changes what is new here. To fix this, please say clearly what CultureForest adds, for example three norms per situation, three formats, per-norm attribution and a current model panel. Please put both papers in the comparison table and show at least one conclusion where the added pieces change the result.
5. The option matrix produces training data for the pipeline's way of writing a violation.
The synthetic training pairs for C-Verifier come from the option matrix. A pair is labelled "Violate" when the option was written to break that norm, "Satisfy" when it was written to respect it, and "Neutral" when the norm comes from another topic. The verifier therefore learns what a violation looks like when the pipeline narrates it, which is an explicit and excused departure on the norm's topic. Real model answers are unlikely to violate a norm in that form. The paper's own error analysis says the verifier misses structural violations and is fooled by polite wording, and it finds violations in under 1 percent of cases for almost every model. This is the same kind of problem as in point 1, because the pipeline's way of writing becomes the label. The matrix is useful because every option has a known label on every norm, which allows per-norm diagnostics. It should be treated as an additional measurement and not as the main supervision. To fix this, please train and test the verifier on human-labelled real responses, hold out whole items, norms and countries, say which unit the 9:1 split uses, and report a confusion matrix on real responses.
6. On the synthetic set the verifier's advantage over the LLM judges reflects a label convention, and on real responses its margins are small.
On the synthetic validation set, the biggest group of judge disagreements, about four in ten, is "Satisfy" labelled as "Neutral". The judges say there is no clear evidence for an option that only complies implicitly. The design label says "Satisfy", because the pipeline instructed the writer that each option must involve and satisfy every norm it does not break. The fine-tuned verifier learned this convention. The paper calls the judges' behavior a hedging bias that comes from safety training, and it calls its own result superior robustness. Training on a convention and then scoring against the same convention measures agreement with the convention. On real responses the wins are mostly by a few points on 900 pairs per held-out model, and the verifier loses on the holiday-date and business domains, which together hold about 40 percent of the items. To fix this, please have people from the cultures re-label a sample of the cases where the judges and the design disagree, and report who is right. Please also state the rule for implicit compliance, add a significance test for the claimed wins, and show results on held-out countries. It might reduce amount of cultures in your benchmark in the end significantly, yet the quality of the work would be verifiable and that is important.
7. The Neutral class means off-topic by construction, yet the paper reads it as caution.
In the synthetic data a "Neutral" pair is a norm taken from a different topic of the same country, and in the human labels "Neutral" means thematically unrelated. A response that does not mention a norm is therefore "Neutral", even when it is good. The paper reads a higher "Neutral" share under strict norms as answers becoming more conservative. A simpler reading is that strict norms are mostly taboos, and a good answer about how to greet someone does not need to list what not to do. Under this reading nothing about caution has changed. This affects the Hard results and the claim about conservative answers. To fix this, please split the norms into do and do-not statements, report "Satisfy", "Neutral" and "Violate" rates separately, and control for answer length.
8. The headline claim that knowledge is not the bottleneck is not supported at present.
The claim compares how often models recognize a norm sentence that is shown to them with their Medium score, and it reads the missing part as a failure to use knowledge. Medium is the format that can be solved from writing style (point 1), some keys are contestable (point 3), and there is no human ceiling. So the remaining gap has several explanations that the paper does not separate. The largest effect in the paper also points the other way. Putting the norms into the prompt lifts Medium by about 20 points, which suggests that the models did not have the norms available when they needed them. The probe itself shows the model the norm sentence and asks for true or false, so it tests recognition and not recall. Nunchi-Bench, which the paper cites, already reports that models recognize cultural facts but struggle to apply them. To fix this, please re-derive the claim on cleaned items with a human ceiling, report the probe with several paraphrases of the negation, and say how much of the gap remains after contestable keys are removed.