- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates an interesting question: whether multilingual LLMs encode transferable information about human colour referents. Using 1,160 colour terms across 14 languages, the authors probe frozen representations and find that most models outperform surface-feature baselines in cross-lingual colour prediction. Transfer largely persists without English supervision or within-family training, but degrades substantially when entire colour regions are excluded.
Critical strengths
-
Meaningful and well-motivated human-referenced evaluation.
-
Thorough and thoughtfully designed experimental controls.
-
Extensive supplementary analyses and careful interpretation.
Critical weaknesses
W1. Insufficient comparison with stronger semantic and translation-based baselines.
The main baselines use character n-grams, subword identities, and lexical frequency. While appropriate for controlling surface features, they do not establish advantages over conventional multilingual semantic representations or translation-based referent prediction. A matched comparison with multilingual embeddings, translation-based methods, and a clearly specified nearest-neighbour baseline would better establish the contribution beyond ordinary cross-lingual semantic correspondence.
W2. Predicting colour centroids does not fully capture human colour naming.
Different languages may associate a colour term with different ranges of colours, even when their average colour is similar. Therefore, accurately predicting the average colour does not necessarily mean that the model understands how each language categorizes colours. Although Appendix K acknowledges this limitation, the paper should be more cautious when interpreting centroid prediction as evidence of language-specific colour knowledge.
W3. Limited coverage of language families and writing systems.
Eight of the ten primary evaluation languages are Indo-European. Cross-family transfer is mainly examined through Korean and Chinese, limiting generalization to broader typological diversity. The experiments also conflate language family, script, lexical overlap, and pretraining exposure. Appendix H reports an exploratory association between tokenizer fertility and performance, suggesting that these factors deserve more systematic investigation.
W4. Relative error reduction lacks sufficient absolute interpretation.
RER measures improvement over a constant predictor but does not directly indicate whether predicted centroids are close to human naming variability. The authors report split-half target stability in Appendix C, but do not sufficiently connect it to model prediction errors. Absolute OKLab errors and comparisons with human split-half disagreement would help contextualize the practical significance of the results.
W5. Several lexical controls remain incomplete.
Context and exact-string controls cover only four models, excluding Gemma-2-9B and Qwen3-14B-Base. Both tested context carriers are English, and the neutral-context evaluation reuses hyperparameters selected under the original prompt. Exact-string exclusion does not remove differently spelled cognates, borrowings, or translation-related words. It also lacks the matched character-plus-frequency baseline on the restricted evaluation set. The authors acknowledge several of these limitations, but more complete controls would strengthen the conclusions.
Questions for authors
Q1. How does the proposed approach compare with translation-based referent estimation or pretrained multilingual semantic embeddings under the same evaluation protocol?
Q2. How exactly is the "NN English" condition in Appendix F implemented? Is it directly comparable to the main probe, and why does it outperform the full readout in some settings?
Q3. What motivates the OKLab coverage thresholds of 0.02, 0.04, and 0.06? Do they have a perceptually grounded interpretation?
Q4. What are the absolute prediction errors, and how do they compare with human split-half centroid disagreement?
Q5. How much of the per-language variation can be explained by tokenizer fertility, script, lexical overlap, or differences in pretraining exposure?
Stronger baselines, lexical controls, Revision provided
Dear Reviewer,
Thank you for supporting the human-referenced evaluation and for your concrete advices. Your request for stronger baselines allows for a clear test of what the models learn beyond translation. We would be grateful if you could consider our responses in your final assessment. Please let us know if any concern remains.
W1, Q1. Stronger baselines
The larger models beat a sentence encoder trained on translations We added two multilingual sentence encoders under the same protocol. LaBSE, trained on translation pairs, reaches RER .416, and multilingual-E5 reaches .340. In contrast, Llama-3.1-8B, Gemma-2-9B and Qwen3-14B-Base exceed LaBSE, and all four larger models exceed it on words with no dictionary translation, by .13 to .27 (Table 1, p. 6; new App. Q).
The readout predicts what a dictionary translation misses. We added a translation-based estimate. For the 135 test words with an English dictionary translation (MUSE) in our data, copying the colour of that translation reaches RER .884, because a word and its translation are used for almost the same colours, closer than two halves of the same language's speakers agree (median 0.0175 against 0.029 OKLab). The readout captures the small remainder. Where speakers depart from the English colour, its prediction moves the same way on all three colour axes in all six models (partial r = .16 to .41, p < .05), and NN English retrieval does not (new App. O, Tables 27–28, pp. 24–25). Obviously, tests built on translation pairs cannot see this difference.
Q2. NN English
N English retrieval finds the test word's nearest English neighbour in the model's representation space and copies that word's human colour. The revision defines it in Sec. 4.1 (p. 6), App. L, and adds it to Table 1. It uses the same test words, weights, layer and denominator as the main probe, and no target-language responses, so it is directly comparable to the main probe. It wins when that neighbour is an exact English counterpart, because it then copies a near-identical colour. This happens most in Gemma-2-9B and in Chinese, whose compounds have exact counterparts (skyblue, deepgreen) in the English list (App. O). On the 372 words with no dictionary translation, the main probe beats it in all six models (Table 27, p. 23).
W5. Lexical controls
We extended both controls to all six models, and the result holds. With exact-string matches removed (473 words), the readouts reach .141 to .547, against .123 for a characters-plus-frequency baseline we added on the same words. With the neutral Word: carrier, the five larger readouts reach .337 to .555, and reselecting layer and penalty for Word: raises no score by more than .003 (Table 9, p. 15). We also replaced Color: with each language's own colour word, which changes the language-averaged score by at most .02 in the four models tested (new App. P). Exact-string removal cannot catch cognates, which is why the translation analysis matters.
W4, Q4. Absolute errors
We now report absolute errors. The readout's weighted error is .079 to .154 OKLab, against a human split-half distance of .026 (new Table 25, p. 22). The split-half value compares two halves of the same speakers on the same word. It measures how stable the human target is and is not a ceiling for held-out prediction.
W2. Centroids
The claims concern a word's typical colour, not its range, as Sec. 5.1 states Actually, a word's range largely follows its typical colour. A new analysis in App. K (Table 24) shows that, across held-out languages, the centroid explains most of the variation in range (R² = .74), and that the readout also predicts range across languages (RER .11 to .39, against .01 for characters plus frequency).
Q3. Thresholds
They are descriptive cut-offs, not perceptual thresholds. The middle value, 0.04, lies within the range of per-language human split-half distances (0.029 to 0.051), and the region holdout, which carries the test, uses no threshold. The revision states this in Sec. 4.1 (p. 6).
W3, Q5. Language coverage
The choice of languages is set by the data. The UW dataset provides 14 languages, and we use all of them for training. We test the ten non-English ones with at least 20 reliable colour terms, a threshold fixed before any model was run, and including all 13 non-English languages keeps every model above the constant predictor (App. C). Cross-family claims are limited to Korean and Chinese.
The submission reported the association with tokenizer fertility (Spearman -.57 to -.77). The revision adds that scores do not track how reliable each language's human data are (App. H). With ten languages, family, script, overlap and pretraining exposure cannot be separated, so we report these as descriptive patterns.