acceptodds
Withdrawn from ICLR 2027

Contextual LLM Representations Encode Fine-Grained Semantic Transparency

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Global distributional features and cross-context geometry jointly predict semantic transparency, the degree of semantic correspondence between a derived word and its morphological constituents. A stability hypothesis connects distributional accounts of morphological meaning to contextual representations, predicting that stronger constituent constraints accompany higher cross-context consensus and lower dispersion. A controlled resource combines 204 English derived forms spanning attested, novel and structurally degraded conditions, isolated-word transparency ratings, and 30 sentence contexts per item. A modelling hierarchy compares stem proximity, global distributional features and cross-context geometry across five pretrained encoder-only and decoder-only checkpoints. Global features improve on stem proximity and predict both structural contrasts and graded variation within conditions. At the preselected layers, adding contextual geometry raises within-condition random-forest R² from .12 to values between .15 and .26. Estimated increments are positive in all five checkpoints, with bootstrap intervals excluding zero for the two encoders. The additional signal is concentrated in morphologically intact forms and is strongest for novel, well-formed items in four of five checkpoints. The median aggregate increment is .26 under ridge regression and .04 under random forests, with each estimator fitted to both the global and combined feature sets. Aggregate and within-condition evaluation address complementary questions about prediction across the stimulus set and graded differences within conditions. These findings connect lexical support and contextual geometry to human semantic judgements, distinguish structural sensitivity from graded semantic accessibility, and show that predictive increments must be interpreted relative to the fitted comparison and target granularity.

Primary area:
interpretability and explainable AI
Keywords:
language model representations, semantic transparency, representational geometry, distributional semantics, derivational morphology, probing evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ahWsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether contextual representations predict human judgments of semantic transparency beyond a compositional lexical baseline. It combines 204 English derived forms in four structural conditions, 90 to 92 isolated-word ratings per item, and 30 sentence contexts per form. Four global features are compared with five geometric summaries from five frozen checkpoints. Matched regression comparisons evaluate aggregate prediction and variation within structural conditions. Reported within-condition random-forest gains are positive in all checkpoints, with the clearest interval evidence in the two encoders. The paper also shows that aggregate contextual gains shrink substantially when stronger predictors fit the lexical baseline.

Critical strengths

The important contribution is the evaluation target. Predicting structural well-formedness would not establish sensitivity to graded meaning among equally well-formed novel words. Training-fold centring addresses this distinction directly, and the high reliability of within-condition item means in Appendix I makes the target credible. This is a useful extension of earlier distributional accounts of derivation to repeated contextual representations.

The estimator comparison also changes how the evidence should be read. Table C1 fits both feature sets with each estimator; the median aggregate increment falls from .264 under ridge to .041 under random forests. Reporting this dependence avoids attributing a weak baseline's recoverable signal entirely to contextual geometry. The layer sweep and segmentation sensitivity analysis further make the limited scope of the result visible.

Critical weaknesses

Appendix B uses the same 204 ratings to select layers before evaluation, so the fixed-layer intervals do not assess the complete selection procedure. Table B2's broad positive sweep is reassuring, but cannot calibrate those intervals; after piece-count adjustment only RoBERTa's interval excludes zero (Table J1).

The central contextual measurement is underdocumented. Appendix I supplies four examples without specifying how sentences were constructed or how semantic and syntactic diversity was controlled across items. Consensus and dispersion directly depend on that diversity. Thus an association induced partly by sentence construction remains plausible even though ratings were collected independently. This uncertainty matters for interpreting the added information as a property of the derived forms.

Questions for authors

Could you repeat layer selection within each outer training fold and report the resulting within-condition increments and uncertainty? This would show whether the main inference survives evaluating the full procedure.

Please clarify the sentence-construction protocol and report whether the within-condition gains persist after controlling for context characteristics such as sentence length, target position and syntactic frame. A sensitivity check using matched subsets of the existing sentences would help distinguish a property of the target forms from a property of their measurement contexts.

Reviewer_oB2MOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper aims to validate a stability hypothesis about semantic transparency of words (how easy the word meaning can be inferred from its constituents) that, if constituents constrain a form’s interpretation, its representations should be more consistent across contexts. To do this, they analyze the stability of contextual representations of words with human semantic transparency annotations and constructed contexts. Results show that adding contextual stability features can improve the regressions scores in predicting semantic transparency scores.

Critical strengths
  1. The idea that semantic transparency has something to do with contextual stability is interesting.
  2. There are many statistical tests in the analyses.
Critical weaknesses
  1. The experimental results cannot support or oppose the stability hypothesis. The results only show that fitting become better with stability-related features, but the correlation between stability and transparency is not verified.
  2. The theoretical relation between semantic transparency and contextual stability is not sufficiently explained. Since the definition of semantic transparency is context-free, why should we suspect that contextual stability is correlated with it? Also, contextual stability is also affected by many other factors, such as grammar roles or word frequency. How to understand the relations between these factors and semantic transparency?
Questions for authors

None

Reviewer_mGNEOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper tests whether hidden states from language models can predict human ratings of semantic transparency for 204 English words. It compares a baseline of fastText-based “global” features with 5 “cross-context” geometry features taken from 5 model checkpoints. It finds modest gains, claiming that contextual features add information beyond the fastText baseline.

Critical strengths
  • Good dataset with matched conditions and reliable ratings.
  • Different model checkpoints are tested with different prediction methods.
  • Clear reporting of limitations and statistics on the results.
Critical weaknesses
  • Layer selection uses the same dataset as evaluation, risking inflated scores.
  • Words sharing a stem can appear in both training and test sets; a control is needed for this.
  • After controlling for token count, only RoBERTa shows a statistically clear gain. The main effect is hence weak.
  • The small, English-only dataset limits broader conclusions.
  • The paper is not very readable, and the writing style is hard to parse. The text is very dense and does not have a clear flow.
Questions for authors
  • Do the gains remain when layer selection happens entirely inside each training fold?
  • Add a control wherein words don't share the same stems within a fold.
  • Correct me if I am wrong, but fig 1 looks AI-generated. If it is, please declare it under the AI use statement.
  • Neurocognitive references and their motivation in the related work and their mention is the dicussion in unclear. Can authors please expand on this and explain how it relates to the paper's findings clearly?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.