- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether contextual representations predict human judgments of semantic transparency beyond a compositional lexical baseline. It combines 204 English derived forms in four structural conditions, 90 to 92 isolated-word ratings per item, and 30 sentence contexts per form. Four global features are compared with five geometric summaries from five frozen checkpoints. Matched regression comparisons evaluate aggregate prediction and variation within structural conditions. Reported within-condition random-forest gains are positive in all checkpoints, with the clearest interval evidence in the two encoders. The paper also shows that aggregate contextual gains shrink substantially when stronger predictors fit the lexical baseline.
Critical strengths
The important contribution is the evaluation target. Predicting structural well-formedness would not establish sensitivity to graded meaning among equally well-formed novel words. Training-fold centring addresses this distinction directly, and the high reliability of within-condition item means in Appendix I makes the target credible. This is a useful extension of earlier distributional accounts of derivation to repeated contextual representations.
The estimator comparison also changes how the evidence should be read. Table C1 fits both feature sets with each estimator; the median aggregate increment falls from .264 under ridge to .041 under random forests. Reporting this dependence avoids attributing a weak baseline's recoverable signal entirely to contextual geometry. The layer sweep and segmentation sensitivity analysis further make the limited scope of the result visible.
Critical weaknesses
Appendix B uses the same 204 ratings to select layers before evaluation, so the fixed-layer intervals do not assess the complete selection procedure. Table B2's broad positive sweep is reassuring, but cannot calibrate those intervals; after piece-count adjustment only RoBERTa's interval excludes zero (Table J1).
The central contextual measurement is underdocumented. Appendix I supplies four examples without specifying how sentences were constructed or how semantic and syntactic diversity was controlled across items. Consensus and dispersion directly depend on that diversity. Thus an association induced partly by sentence construction remains plausible even though ratings were collected independently. This uncertainty matters for interpreting the added information as a property of the derived forms.
Questions for authors
Could you repeat layer selection within each outer training fold and report the resulting within-condition increments and uncertainty? This would show whether the main inference survives evaluating the full procedure.
Please clarify the sentence-construction protocol and report whether the within-condition gains persist after controlling for context characteristics such as sentence length, target position and syntactic frame. A sensitivity check using matched subsets of the existing sentences would help distinguish a property of the target forms from a property of their measurement contexts.