acceptodds
Under review as a conference paper at ICLR 2027

Beyond lexical alignment: Testing cross-lingual color representations against human naming

Abstract

Languages partition the continuous colour domain into partly different vocabular- ies, so translation-based alignment alone cannot show whether multilingual rep- resentations capture the colours that speakers associate with colour terms. We introduce a human-referenced cross-lingual probe that maps frozen text represen- tations to language-specific human naming centroids for 1,160 colour terms in 14 languages, collected under a common free-naming condition. The primary eval- uation holds out each of the ten non-English languages with at least 20 reliable terms from both probe fitting and model selection; a sensitivity analysis includ- ing all thirteen non-English languages shows that every model remains above the constant predictor. Five of six models achieve greater relative error reduction than character, token, and frequency controls, and this advantage survives context and exact-string controls in the four models tested. Transfer also persists without English-labelled supervision and across the tested family and script boundaries. The coverage analysis shows where this transfer weakens: standard language holdout leaves the relevant colour regions represented in source supervision, and removing a complete colour region lowers relative error reduction by 0.19–0.25 more than randomly deleting the same number of terms. The results support a transferable mapping from multilingual text representations to human colour ref- erents, while showing that its measured strength depends partly on which regions of the colour domain are represented in source supervision.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
multilingual language models, cross-lingual representation learning, colour naming, probing, human-referenced evaluation, lexical alignment, semantic transfer, referential coverage

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_veu2OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates an interesting question: whether multilingual LLMs encode transferable information about human colour referents. Using 1,160 colour terms across 14 languages, the authors probe frozen representations and find that most models outperform surface-feature baselines in cross-lingual colour prediction. Transfer largely persists without English supervision or within-family training, but degrades substantially when entire colour regions are excluded.

Critical strengths
  1. Meaningful and well-motivated human-referenced evaluation.

  2. Thorough and thoughtfully designed experimental controls.

  3. Extensive supplementary analyses and careful interpretation.

Critical weaknesses

W1. Insufficient comparison with stronger semantic and translation-based baselines.

The main baselines use character n-grams, subword identities, and lexical frequency. While appropriate for controlling surface features, they do not establish advantages over conventional multilingual semantic representations or translation-based referent prediction. A matched comparison with multilingual embeddings, translation-based methods, and a clearly specified nearest-neighbour baseline would better establish the contribution beyond ordinary cross-lingual semantic correspondence.

W2. Predicting colour centroids does not fully capture human colour naming.

Different languages may associate a colour term with different ranges of colours, even when their average colour is similar. Therefore, accurately predicting the average colour does not necessarily mean that the model understands how each language categorizes colours. Although Appendix K acknowledges this limitation, the paper should be more cautious when interpreting centroid prediction as evidence of language-specific colour knowledge.

W3. Limited coverage of language families and writing systems.

Eight of the ten primary evaluation languages are Indo-European. Cross-family transfer is mainly examined through Korean and Chinese, limiting generalization to broader typological diversity. The experiments also conflate language family, script, lexical overlap, and pretraining exposure. Appendix H reports an exploratory association between tokenizer fertility and performance, suggesting that these factors deserve more systematic investigation.

W4. Relative error reduction lacks sufficient absolute interpretation.

RER measures improvement over a constant predictor but does not directly indicate whether predicted centroids are close to human naming variability. The authors report split-half target stability in Appendix C, but do not sufficiently connect it to model prediction errors. Absolute OKLab errors and comparisons with human split-half disagreement would help contextualize the practical significance of the results.

W5. Several lexical controls remain incomplete.

Context and exact-string controls cover only four models, excluding Gemma-2-9B and Qwen3-14B-Base. Both tested context carriers are English, and the neutral-context evaluation reuses hyperparameters selected under the original prompt. Exact-string exclusion does not remove differently spelled cognates, borrowings, or translation-related words. It also lacks the matched character-plus-frequency baseline on the restricted evaluation set. The authors acknowledge several of these limitations, but more complete controls would strengthen the conclusions.

Questions for authors

Q1. How does the proposed approach compare with translation-based referent estimation or pretrained multilingual semantic embeddings under the same evaluation protocol?

Q2. How exactly is the "NN English" condition in Appendix F implemented? Is it directly comparable to the main probe, and why does it outperform the full readout in some settings?

Q3. What motivates the OKLab coverage thresholds of 0.02, 0.04, and 0.06? Do they have a perceptually grounded interpretation?

Q4. What are the absolute prediction errors, and how do they compare with human split-half centroid disagreement?

Q5. How much of the per-language variation can be explained by tokenizer fertility, script, lexical overlap, or differences in pretraining exposure?

1 comment
AuthorsOpenReview ↗

Stronger baselines, lexical controls, Revision provided

Dear Reviewer,

Thank you for supporting the human-referenced evaluation and for your concrete advices. Your request for stronger baselines allows for a clear test of what the models learn beyond translation. We would be grateful if you could consider our responses in your final assessment. Please let us know if any concern remains.

W1, Q1. Stronger baselines

The larger models beat a sentence encoder trained on translations We added two multilingual sentence encoders under the same protocol. LaBSE, trained on translation pairs, reaches RER .416, and multilingual-E5 reaches .340. In contrast, Llama-3.1-8B, Gemma-2-9B and Qwen3-14B-Base exceed LaBSE, and all four larger models exceed it on words with no dictionary translation, by .13 to .27 (Table 1, p. 6; new App. Q).

The readout predicts what a dictionary translation misses. We added a translation-based estimate. For the 135 test words with an English dictionary translation (MUSE) in our data, copying the colour of that translation reaches RER .884, because a word and its translation are used for almost the same colours, closer than two halves of the same language's speakers agree (median 0.0175 against 0.029 OKLab). The readout captures the small remainder. Where speakers depart from the English colour, its prediction moves the same way on all three colour axes in all six models (partial r = .16 to .41, p < .05), and NN English retrieval does not (new App. O, Tables 27–28, pp. 24–25). Obviously, tests built on translation pairs cannot see this difference.

Q2. NN English

N English retrieval finds the test word's nearest English neighbour in the model's representation space and copies that word's human colour. The revision defines it in Sec. 4.1 (p. 6), App. L, and adds it to Table 1. It uses the same test words, weights, layer and denominator as the main probe, and no target-language responses, so it is directly comparable to the main probe. It wins when that neighbour is an exact English counterpart, because it then copies a near-identical colour. This happens most in Gemma-2-9B and in Chinese, whose compounds have exact counterparts (skyblue, deepgreen) in the English list (App. O). On the 372 words with no dictionary translation, the main probe beats it in all six models (Table 27, p. 23).

W5. Lexical controls

We extended both controls to all six models, and the result holds. With exact-string matches removed (473 words), the readouts reach .141 to .547, against .123 for a characters-plus-frequency baseline we added on the same words. With the neutral Word: carrier, the five larger readouts reach .337 to .555, and reselecting layer and penalty for Word: raises no score by more than .003 (Table 9, p. 15). We also replaced Color: with each language's own colour word, which changes the language-averaged score by at most .02 in the four models tested (new App. P). Exact-string removal cannot catch cognates, which is why the translation analysis matters.

W4, Q4. Absolute errors

We now report absolute errors. The readout's weighted error is .079 to .154 OKLab, against a human split-half distance of .026 (new Table 25, p. 22). The split-half value compares two halves of the same speakers on the same word. It measures how stable the human target is and is not a ceiling for held-out prediction.

W2. Centroids

The claims concern a word's typical colour, not its range, as Sec. 5.1 states Actually, a word's range largely follows its typical colour. A new analysis in App. K (Table 24) shows that, across held-out languages, the centroid explains most of the variation in range (R² = .74), and that the readout also predicts range across languages (RER .11 to .39, against .01 for characters plus frequency).

Q3. Thresholds

They are descriptive cut-offs, not perceptual thresholds. The middle value, 0.04, lies within the range of per-language human split-half distances (0.029 to 0.051), and the region holdout, which carries the test, uses no threshold. The revision states this in Sec. 4.1 (p. 6).

W3, Q5. Language coverage

The choice of languages is set by the data. The UW dataset provides 14 languages, and we use all of them for training. We test the ten non-English ones with at least 20 reliable colour terms, a threshold fixed before any model was run, and including all 13 non-English languages keeps every model above the constant predictor (App. C). Cross-family claims are limited to Korean and Chinese.

The submission reported the association with tokenizer fertility (Spearman -.57 to -.77). The revision adds that scores do not track how reliable each language's human data are (App. H). With ten languages, family, script, overlap and pretraining exposure cannot be separated, so we report these as descriptive patterns.

Reviewer_g6irOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper asks an interesting question of cross-linguistic differences that reflect in naming of colors and how they are represented in language models. For human reference, they use UW Many languages dataset which contains 1160 color terms for 14 languages converted to OKLab, and then test it on 6 different language models of Qwen, Llama and Gemma family.

The main method is taking frozen representations from the language models for n-1 languages, training a linear regression model and test it on the held out nth language to see if it can correctly predict the colors they refer to. It seems most models perform above the baseline (character / word freq based representations), with Qwen 14B being the best.

Several controls are implemented which provides confidence in the findings: removing english from regression training, training regression model on one language group (Indo-European) and testing on another (Korean, Chinese), prompt lexical controls, and removing certain color regions from training. Of these, every control keeps the performance largely intact, except the last one, which substantially reduces the performance, indicating that color-neighborhood coverage is a significant factor.

Critical strengths
  1. The paper asks an interesting question about multilingual representations and makes it rigorous using human color-naming data and several carefully designed controls. This paper provides a robust methodological reference for future work.
Critical weaknesses
  1. Not sure what to take away from this paper beyond the formalization. The paper answers its stated questions rigorously, but what does this actually tell us about LM abstractions or how we as humans represent information? I would appreciate a clearer articulation of what new scientific understanding these results offer beyond making an intuitive observation rigorous. In short, the current version is not well-motivated in terms of its significance.
  2. Not enough engagement with the broader literature and its implications. The paper leans heavily towards linguistic alignment without really engaging with cultural and perceptual studies (e.g., Winawer on Russian blues, Boroditsky on linguistic relativity) or the vast multilingual NLP literature. These could provide much-needed intuition about why the findings matter, while the current writing feels convoluted and mechanical (e.g. Related Work introduces methodological details prematurely). My main questions are: what are the insights, and who is the audience?
Questions for authors
  1. Is the color-neighborhood result telling us about the limitations of the regression model or the LM itself?
  2. What's the implications of varied performance for different languages? Anything about model's representations, human color labels / partitions, or its simply about the quality of data?
6 comments
AuthorsOpenReview ↗

Rebuttal: What the results show and who they are for

Dear Reviewer,

Thank you for pressing on the question: what it teaches and for whom. Answering that helped the revision a lot, and we have revised the Introduction, Related work and Discussion, which now states the motivation and findings more directly.

W1. Motivation and takeaways

Motivation. A central debate is whether multilingual models represent meaning in one shared space across languages, or also keep the distinctions each language draws. Evidence for shared structure seems to be stronger, but these studies compare translation pairs, and treat translations as equivalent by construction, thus miss language-specific meaning. Colour allows a direct test. Different languages divide the same colour space into words differently; a word and its translation need not name the same colour. Colour has a reference outside language, a perceptual colour space (OKLab) shared by all languages. Naming data record which colours speakers name in each language, so a model's prediction for a word can be compared in OKLab coordinates without assuming any translation.

Two takeaways.

  • 1. Multilingual representations are both shared and language-specific. A mapping learned from other languages predicts colours a new language's words refer to, so colour meaning is shared across languages. However, where speakers of a language use a word for slightly different colours than its English translation, the model's prediction also moves away from the English colour, toward the colours those speakers name, on every colour axis and in all six models. So the models learn how each language divides colour, beyond its English equivalents, which the previous studies built on translation pairs cannot detect.

  • 2. Cross-lingual alignment is easy to overstate without controls. Our design shows what alignment test must rule out. Transfer can come from shared spellings and cognates, word frequency, English-labelled supervision, or tuning on the target language. The study rules these out with character, subword and frequency baselines, removal of identical strings and English supervision, and exclusion of the target language from both fitting and model selection. Apparent transfer can also come from coverage. Other languages already name the same colours, so holding out a language mostly tests new words for familiar colours, and only removing whole colour regions tests extrapolation. Few domains have a reference as precise as colour, but these checks apply wherever one exists.

W2. Broader literature, audience and insights

The paper speaks to work on multilingual representations, colour perception and linguistic relativity.

In the field of NLP, prior studies test whether models share meaning across languages by checking translation pairs, which assumes that the paired words mean the same thing. Our test compares each language's words with what its own speakers name, so it needs no translations and can show where languages differ. With the controls in takeaway 2, it applies to any domain that has a reference outside language.

The field of colour cognition and linguistic relativity studies language-specific colour categories mostly through perception. Our results concern naming, and Takeaway 1 shows that a readout learned on other languages predicts one language's own colour naming, so these language-specific categories are reflected in model representations. The revision adds the literature (Winawer et al., 2007,Boroditsky,2006 ) to Related Work and Sec. 5.1.

Q1. Limitations of Regression models or language models?

Both play a part, but in different ways. Any linear readout loses accuracy when a whole colour region is missing from training, whatever its input, so the drop itself reflects the limitatation of regression in your logic. The difference lies in what is left, and the language models show their strength in what is left. A readout on spelling and frequency falls below the constant predictor, while readouts on the five larger language models keep an RER of .10 to .30 in the removed region (Table 4; Sec. 5.2). Unlike spelling and frequency, language models place colour words along graded dimensions such as lightness and hue. This structure lets the readout reach colours it was never trained on.

Q2. Why performance vary across languages?

The variation is stable among models, mainly caused by tokenization, not by human labels. The ranking is the same in all six models, with Persian lowest and Korean second lowest. Scores do not track how reliable each language's human labels are (Spearman .12 to .45, p ≥ .19; App. H). The strongest correlate is tokenization: languages whose colour words are split into more subword tokens score lower (Spearman −.57 to −.77). With ten languages, tokenization, script and the amount of training text cannot be separated, so we report this as a pattern, not due to the quality of the human labels, not as a ranking of languages or cultures.

Dear Reviewer g6ir,

Thank you for your constructive feedback. We have uploaded a revised version of our manuscript incorporating additional experiments and clarifications that directly address your comments. As the discussion period is ending soon, we would welcome any follow-up questions and kindly ask you to reconsider your assessment if you find our revisions and responses satisfactory.

Thank you for the detailed responses and revision. I appreciate the additional experiments e.g. the finding about language-specific differences beyond English translations. The answers to Q1 and Q2 are also satisfactory.

My main reservation is about the paper's framing. The idea is great, and the experiments are rigorous, but I think the motivation could foreground the broader scientific question more directly. For example, colors are a fascinating case of a shared perceptual reality that languages name and categorize differently. Can language models capture both the shared meaning and the distinctions each language draws? What does this tell us about the abstractions these models learn? Although these questions are present, I think foregrounding them before discussing the limitations of previous work could better convey the significance of the findings.

The title reflects this framing choice as well. I actually preferred the original title, “Beyond Lexical Alignment.” While “Beyond Translation Equivalence” may be more technically precise in response to reviewer concerns, it feels more like a defensible description of the evaluation than an evocative framing of the broader scientific question.

Similarly, Related Work could do more to build intuition and tell a story. The added literature is helpful, but I think weaving together human perception, linguistic differences, and model representations into a more coherent narrative could make the paper's significance clearer. I think the contribution is stronger than the current narrative conveys.

Reviewer_XDBUOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether representations from multilingual language models can predict the colors that speakers of a language they were not fitted on associate with their own color words. They framed as going beyond lexical alignment by using independently elicited human naming data rather than translation pairs as the criterion. Using the UW "Many Languages, Many Colors" dataset, the authors represent each term by the OKLab centroid of its human naming distribution. They fit a weighted ridge readout from mean-pooled residual-stream activations to that centroid, holding out the target language from both fitting and layer selection, and they report the relative error reduction (RER) against a constant predictor. Five of six models reach RER between 0.372 and 0.571, well above surface-form baselines. The authors conclude that a transferable mapping from model representations to human color referents exists and that its strength depends on how well the source languages cover the relevant region of color space.

Critical strengths
  1. Most work on multilingual alignment defines correspondence through translation pairs, which assumes the equivalence being studied. I think the criterion for success is well chosen such that predicting each language's own independently elicited naming centroids, and collected under a common stimulus protocol is principled framing.

  2. The target language is excluded from both readout fitting and hyperparameter or layer selection. The language-inclusion threshold is fixed before activations were extracted, and the authors report sensitivity over all thirteen non-English languages. The evaluation protocol is well thought.

  3. The paper reports character, subword and frequency baselines, a neutral carrier, exact-string exclusion and a permutation check.

Critical weaknesses
  1. A color term's meaning includes its extent, its boundaries and its focal point. Whereas, a three-dimensional centroid of a smoothed, density-normalized distribution keeps only location. To my understanding, two terms can have similar centroids while covering very different regions. The cross-linguistic differences that color-naming research cares about, such as category boundaries and the number of categories in a region, are therefore largely invisible to this target. This deserves explicit discussion, and ideally a secondary target.

  2. I think the reference predictor is too weak to calibrate the scale of RER. The constant baseline outputs one color for every term. Any representation that merely separates light from dark terms, or warm from cool ones, will beat it substantially. The numbers would mean much more relative to a translation-based predictor that copies the centroid of the target term's translation equivalent in a source language, and relative to a human noise ceiling.

  3. The representation design builds in an English bias. Both carrier phrases color and word are English. Representations are mean-pooled over the term's tokens in causal models, where the first token cannot see the rest of the term, so multi-token terms in scripts with heavy tokenization such as Korean or Chinese are represented asymmetrically. Well, I think neither choice is unreasonable, but the second interacts directly with the script-transfer claim and should be tested with last-token pooling and target-language carriers.

  4. The region-holdout penalty is not attributed to the language model. any linear regressor supposed to predict targets in a region absent from its training data will shrink toward the training mean, whatever its inputs. Without the same region holdout for the surface baselines, the penalty cannot be attributed to the language-model representations. The k-means partition also uses the evaluation targets themselves. A target-independent partition should be reported as well.

  5. The paper does not say which color dimensions transfer, and reports only a combined Euclidean score. Many terms are compositional, and lightness modifiers are frequent and often cognate across Indo-European languages. If transfer is carried mainly by the L axis, the claim of a shared mapping to color referents is much weaker than if the opponent axes (a, b) also transfer. A per-axis breakdown is needed to interpret every headline number.

Questions for authors
  1. Can you show that readout predicts the component of a target term's centroid that is not predicted by its translation equivalent? For example, report RER on the residual after subtracting the nearest source-term centroid, or evaluate on terms that are known to split or merge relative to English.
  2. Why does nearest-English-neighbour retrieval outperform the trained readout for Gemma-2-9B overall and for Chinese in several models? How do you reconcile this with the title?
  3. What RER does a split-half human predictor achieve for each language? How do the results compare with a translation-equivalent predictor in place of the constant baseline?
  4. How does the English-removal result change under term-level weighting, and when a different single language with comparable effective weight is removed?
  5. What is the region-holdout penalty for the character-plus-frequency baseline, and under a partition that does not use the evaluation targets?
  6. Could you explain per-axis (L, a, b) results for the employed model?
  7. Do the results hold with last-token pooling and with carrier phrases written in the target language?
5 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer XDBU (part 1/2)

Dear Reviewer,

Thank you for your careful reading and constructive suggestions. We really appreciate how seriously you engaged with our paper. Given the number of your comments and questions, we have to split our responses across two comments, due to the 5000-character limit. Please bear with us.

W1. What does the centroid leave out?

The centroid already carries most of the extent, and the readout also predicts extent. Sec. 5.1 of the submission limits the claim to typical colours, not category boundaries. Actually, a term's centroid explains most of the variation in how widely its naming responses spread (R² = .74). We also trained the readout on extent, the second target you suggested, and added it to the revision. It predicts extent across languages in every model (RER .11 to .39), while the character-plus-frequency baseline reaches .01 (App. K, Table 24). Boundaries and the number of categories in a region belong to a whole vocabulary, not to one word, so a word-level readout cannot score them.

W2, Q3. How large is RER against translation and against human noise?

On the words a dictionary covers, copying the translation's colour beats the readout. The human split-half reference measures how stable the targets are, not how far the readout could go.

Translation as the reference (App. O, Table 27). MUSE dictionaries link 135 test words to an English term. Copying that English term's colour reaches RER .884, against .277 to .651 for the readout. The dictionary covers about a quarter of the test words with translations naming the same color. On the other 372 words, the readout keeps RER .131 to .528 and beats NN English retrieval in all six models.

Split-half reference. Both halves use the target language's own speakers, while the readout sees none of them, so this is a scale for the targets, not a ceiling for the readout. The distance between two random halves of the same speakers are treated s a prediction error, scored with the same weights and constant predictor as Table 1. It reaches RER .80 to .91 in every language, against .17 to .57 for the readout averaged over languages. Both halves come from the target language's own speakers, while the readout sees none of them, so App. L treats this as a measure of target stability, not as a ceiling the readout could reach.

Q1. Does the readout predict what the translation does not?

Yes. Where speakers depart from the colour of the English translation, the readout departs in the same direction.

See the residual (Table 28). For each word, the residual is its own colour minus the colour copied from its English translation. On these words the residual is small, with a median of 0.0175 OKLab, below the 0.029 gap between two halves of the same speakers. An RER on its size would mostly score speaker noise, so we test whether the readout gets its direction right. It does. The readout's departure from the copied colour correlates positively with the residual on all three colour axes in all six models (partial r = .16 to .41, p < .05). NN English retrieval does so in only 2 of 18 cases.

Words that split or merge relative to English. In eight pairs, two words of one language share one English translation, for example German braun and dunkelbraun (brown) or Chinese 橘 and 橙 (orange). Copying the translation gives both words the same colour. In six of these pairs, speakers also give the two words nearly the same colour, within split-half noise. In the two pairs where speakers clearly separate the words (braun/dunkelbraun and pink/rosa), the readout separates them in the same direction in 10 of 12 model–pair cases. In the other direction, 15 words have two English translations, such as French marron (brown, maroon), and copying must average the two colours. On these words translation-copy drops from RER .903 to .766, while the readout rises in five of six models (.358 to .741).

Q2. Why does NN English retrieval win for Gemma-2-9B and Chinese?

NN English retrieval beats the readout only where English already has an exact counterpart word. Gemma-2-9B's lead comes from dictionary-linked words (.778 against .651). On the remaining words the readout leads (.482 against .450, Table 27). Most Chinese colour terms combine a modifier and a hue, and English has matching compounds in the lexicon, such as skyblue and deepgreen (App. O).

NN English retrieval winning on some words does not contradict the title, because the title is about the test, not about the readout beating lexical retrieval on every word. The paper judges a model against each language's own speakers instead of translation pairs. Where it does not, the readout is ahead in all six models. It also predicts the direction in which a word departs from its translation, which NN English retrieval does not. The revised title, "Beyond Translation Equivalence", names what the paper tests more precisely.

AuthorsOpenReview ↗

Rebuttal to Reviewer XDBU (part 2/2)

W5, Q6. Which axes transfer?

Hue transfers at least as well as lightness, so transfer is not carried by lightness modifiers. Held-out R² is positive on all three OKLab axes in every model (Sec. 4.2, Table 25). Across the six models, green–red (a) reaches .310 to .801, blue–yellow (b) .288 to .748, and lightness (L) .190 to .741. Both hue axes beat lightness in every model. For Qwen3-14B-Base, the strongest model, R² is .741 for L, .793 for a and .748 for b. The gap is largest in the smaller models, for example .190 for L against .310 and .288 in Qwen2.5-0.5B. Modifiers such as dark and light mostly move lightness, so if they drove transfer, L would lead. The Q1 result also holds on each axis separately, with partial r of .16 to .41 for L, .19 to .28 for a and .17 to .27 for b (Table 28).

W4, Q5. What the region penalty reflects?

The penalty itself comes from linear prediction outside the training data, as you suggest. What survives the penalty comes from the representation. The submission stated that k-means uses human centroids (Sec. 4.1, p. 6) and that k was fixed at 6 before any model was run (Table 20). To remove the targets from the partition, we refitted the six regions on the fixed grid of 2,048 in-gamut OKLab colours, which contains no words, responses or activations (App. N). Each region removal is compared with ten random removals of the same number of source terms. With this partition, the penalty remains in all six models (.185 to .256), and every paired interval excludes zero. The character-plus-frequency baseline pays a similar .209, as you expected. The difference lies in what is left. After the region is removed, that baseline falls to −.049, below the constant predictor. The five larger readouts keep .095 to .298 and beat the baseline by .144 to .347, with paired intervals excluding zero. Only Qwen2.5-0.5B shows no clear advantage (Table 4).

W3, Q7a. English carrier bias?

No measurable English bias. The results hold with target-language carriers. We replaced the primary carrier Color: with the colour noun of each term's own language and refitted the readout, keeping terms, folds, layer and ridge penalty fixed. In the four models tested, the language-averaged score changes by at most .02 (for example .520 to .514 in Llama-3.1-8B), and no paired interval excludes zero (App. P, Table 29).

W3, Q7b. Last-token versus mean pooling

Yes, the results hold with last-token pooling, and Korean and Chinese improve under it. In a causal model, only the last subtoken has seen the whole word. Reading it gives RER .143 to .540, slightly below mean pooling (.169 to .571) and above the constant predictor in all six models (App. L). Per language, last-token pooling raises Korean and Chinese in 11 of 12 model–language cases (for example Korean .414 to .518 and Chinese .500 to .670 in Qwen3-14B-Base), while it lowers the other eight languages on average. Mean pooling does not inflate the script-transfer result. If anything, it understates Korean and Chinese. This check reuses the layer and penalty tuned for mean pooling, so it is conservative for the last-token readout. The script-transfer result holds under both pooling choices and mean pooling does not inflate the transfer result.

Q4. English removal under other weightings

English is not a necessary bridge under either weighting. Term-level weighting (App. M, Table 26). With equal weight per term, removing English changes the language-averaged RER by −.038 to +.016, close to the −.036 to +.007 under equal language weight. Another single language. In the five larger models, removing English costs no more than removing one other source language under both weightings. Only Qwen2.5-0.5B loses more. Comparable weight. No other language has as many terms as English (595), so the submission also deleted the same number of terms spread across all source languages (Table 2). Removing English costs less than this size-matched deletion in all six models, for example .548 against .529 in Qwen3-14B-Base (full pool .571). Per-language results are in Tables 11 to 16.

Thank you again for the detailed questions. Several of the new analyses come directly from them. We have uploaded a revised PDF with new text in blue.

We hope the answers lead you to reconsider your assessment, and we would be glad to address anything that remains.

Dear Reviewer XDBU,

Thank you again for your time and thoughtful review. We have uploaded the revised manuscript (with updates highlighted with the blue color) alongside our rebuttal and additional experimental results. As the discussion period nears its close, we wanted to gently check if these revisions have addressed your main concerns. Please let us know if any points require further clarification. We would greatly appreciate your feedback and would be grateful if you could reconsider your assessment in light of the updated paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overview for the AC and all reviewers

We thank the AC and three reviewers for their constructive feedback, which has strengthened the manuscript. The revised PDF is uploaded with new text in blue.

1. Summary of the paper.

The paper addresses an open question about multilingual language models: do their representations keep the distinctions each language draws, or only what translations share? Alignment studies compare translation pairs, which treats a word and its translation as equivalent and cannot show where languages differ. The paper replaces translation pairs with each language's own human colour naming as the criterion. Colour allows this test, because languages divide the same perceptual space differently, and naming data show which colours speakers name with each word. A linear readout maps frozen representations to these centroids for 1,160 terms in 14 languages, predicting each held-out language from the others. In five of six models, the readout beats character, subword and frequency baselines, and transfer persists without English or same-family supervision. The new criterion reveals two results that translation-based evaluations cannot register. First, models encode how each language's colour words depart from their English translations, since the readout's predictions move in the same direction as the speakers' colours. Second, the region holdout separates transfer from extrapolation. Holding out a language tests new words for colours that other languages already name, whereas removing a whole colour region tests extrapolation and costs more than removing the same number of random terms. Our methodology applies to cross-lingual alignment studies beyond colour.

2. All three reviewers credit the criterion and the evaluation design.

XDBU calls the human-referenced criterion "principled framing" and “evaluation protocol is well thought”; g6ir calls the paper "a robust methodological reference for future work"; and veu2 highlights the "meaningful and well-motivated human-referenced evaluation" and its "thorough and thoughtfully designed experimental controls".

3. Main concerns and new evidence

  • Does the readout go beyond translation? (XDBU, veu2) Yes. Copying the translation's colour is more accurate where a dictionary link exists, but when a word's colour differs from its English translation's, the readout shifts the same way. NN English retrieval does not (Sec. 4.2; Tables 27–28).
  • Does it beat stronger baselines? (veu2) Yes. The three largest models beat LaBSE, an encoder trained on translations (Table 30).
  • Is the region penalty caused by the language model? (XDBU, g6ir) No. A surface baseline pays the same penalty, so it comes from linear prediction. Only the five larger language models still predict the removed colours (Table 4; App. N).
  • What does RER mean in absolute terms? (XDBU, veu2) Report absolute terms agaist translation and human split-half distance in terms of OKLab error and each colour axis. Readout beats NN English retrieval when there is no exact translation. The split-half reference shows how stable the human targets are. Hue transfers at least as well as lightness (Table 25).
  • What does a centroid miss? (XDBU, veu2) The readout also predicts extent, the spread of a term's naming distribution, in every model (RER .11 to .39), while the character-plus-frequency baseline reaches .01 (App. K, Table 24).
  • Do the controls hold more broadly? (XDBU, veu2) Yes. They now cover all six models, and results do not depend on English carriers, mean pooling or English supervision (Table 9; App. L, M, P).

4. Clarifications by section

  • Abstract. Adds departures from English translations.
  • Introduction. States the motivation and findings more directly (g6ir).
  • Related Work. Adds work on colour cognition and linguistic relativity, including the Russian-blues experiments (g6ir).
  • Experimental setup (Sec. 4.1). Defines NN English retrieval, names the size-matched removal, and relates the coverage thresholds to split-half distances (veu2).
  • Discussion (Sec. 5.1). Relates the findings to colour cognition and limits the claim to naming, not perception (g6ir).
  • Appendices. New App. L–Q add the baselines, controls and translation analyses, and App. K adds extent.

Replies to each reviewer follow in their threads.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.