acceptodds
Under review as a conference paper at ICLR 2027

RoleLex: Recovering Cross-Speaker Referential Structure from Frozen Language-Model States

Abstract

When two people talk about the same thing, the words they use for it drift: in one MapTask dialogue a route giver's "the old mill" is answered with "the mill wheel," and both speakers then settle on "it." A system that tracks such references has to store each mention as it arrives, before it knows what will be asked later. We study how to build that stored key from a frozen dialogue decoder and find that no single layer holds the evidence. On MapTask dialogues, decoder depths read out separately pick different antecedents for 47–68% of queries, and the best depth changes from one backbone to the next. RoleLex keeps the depths apart instead of choosing between them: four depths are projected and normalized independently and each occupies its own block of a 256-dimensional key, so one dot product between keys equals the average of four depth-wise cosines. On 13,077 human-linked mentions from 128 MapTask dialogues, evaluated leave-one-map-out, this fixed key beats the best development-selected single depth for Qwen3.5-9B, Gemma 4 E4B-it, and OLMo 3 7B, matches or beats the best selected pair of depths, and costs the same parameters and arithmetic as a final-layer key. Controls show that what transfers across backbones is keeping the depth scores separate; a dense map over the same states has four times the parameters, does worse, and uses fewer than half of its output directions. The gains are largest when no antecedent shares the query's wording, carry over to deciding whether any antecedent exists, and reappear on OneCommon.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
frozen language-model representations, representation geometry, metric learning, contrastive learning, cross-speaker reference resolution, coreference resolution, online retrieval, conversational memory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JVJKOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RoleLex, a method for cross-speaker reference retrieval that freezes the language model and trains lightweight projections over multiple layers. It combines these representations into one vector and retrieves matching references using similarity rather than text generation. Experiments on MapTask-RefMem and OneCommon across three model families show improvements over final-layer readouts, supported by ablations of layer selection and aggregation.

Critical strengths
  1. RoleLex provides a simple and practical way to reuse frozen LLM states for reference retrieval. It improves over final-layer readouts without increasing the retrieval key size or projection budget, and avoids a separate encoder when LLM states are already available.
  2. The evaluation covers two datasets, three model families, and multiple random seeds. Detailed layer-selection and aggregation ablations strengthen the empirical evidence and provide useful insights into the benefits of multi-layer representations.
Critical weaknesses
  1. The comparison with generative approaches is incomplete. The prompting baseline is not fine-tuned on the task data, whereas RoleLex receives task-specific supervision. This leaves its accuracy–cost trade-off against a similarly trained generative model unclear.
  2. The performance advantage over the strongest external baseline is limited: RoleLex achieves 90.59% Recall@1 versus 90.51% for fine-tuned BGE-base. Its practical advantage therefore relies mainly on reusing already available LLM states; in the reported standalone setting, BGE is faster with comparable accuracy. This narrows the deployment settings in which RoleLex offers a clear benefit.
Questions for authors

Could you add a generative SFT baseline for one representative backbone, using the same training splits and referent annotations, and report its retrieval accuracy and inference cost? This would clarify RoleLex’s accuracy–cost trade-off against a task-adapted generative model, rather than an untuned prompting baseline.

Reviewer_s79XOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RoleLex, a method for cross-speaker reference retrieval using hidden states from frozen language models. RoleLex independently projects and normalizes representations from multiple decoder layers and concatenates them into a single retrieval embedding, trained using referent-supervised contrastive learning. The authors also construct MapTask-RefMem by adapting existing annotated dialogue resources into a causal cross-speaker retrieval benchmark. Experiments across three language model backbones demonstrate improvements over final-layer and single-layer baselines, with competitive performance against selected two-layer representations. Additional evaluations examine lexical variation, abstention, and architectural transfer to the OneCommon dataset.

Critical strengths

The paper carefully curates and processes existing annotated dialogue resources to construct a well-defined benchmark for causal cross-speaker reference retrieval. The resulting evaluation protocol could serve as a useful resource for future research on referential retrieval in dialogue.

Critical weaknesses
  1. Limited novelty and contribution The main ideas of this work have been extensively explored in prior research, including supervised probing of referential information (Sorodoc et al., ACL 2020), multi-layer representations for retrieval (Xie & Lukasiewicz, Findings of EMNLP 2025), and benefits of intermediate-layer representations (Skean et al., ICML 2025). Although the paper acknowledges of first two studies, its methodological contribution beyond these works remains unclear. RoleLex primarily combines depth-specific projections, normalization, and concatenation within a standard supervised embedding learning framework. Furthermore, Table 2 shows that the proposed factorization does not consistently outperfom alternative aggregation methods, making its distinctive contribution less convincing.
  2. Insufficient practical justificaion and baseline comparisons RoleLex achieves performance comparable to fine-tuned BGE in-domain(90.59 vs 90.51). However, under cross-domain transfer (OneCommon -> MapTask), it underforms training-free BM25 and frozen BGE-base embeddings (Appendix Tables 17 and 23). These results and presented in separate tables, making direct comparison unnecessarily difficult. Moreover, stronger training-free LLM-based reference resolution approaches (Moniz et al. SIGDIAL 2024; Ijurco & Lopez de Lacalle, LREC 2026) are not examined. Given RoleLex’s dependence on task-specific supervision and internal decoder states, its practial advantages over these alternatives remain insufficiently justified.
Questions for authors

Q1. Novelty and contribution of depth factorization. Given prior work on referential probing (Sorodoc et al., ACL 2020) and multi-layer retrieval (Xie & Lukasiewicz, Findings of EMNLP 2025), what is the key methodological or scientific contribution of RoleLex beyond combining existing techniques? In particular, since alternative aggregation strategies achieve comparable or better results in Table 2, what evidence establishes the necessity or distinctive benefit of the proposed factorization? Q2. Practical advantages over training-free alternatives. Under cross-domain transfer, RoleLex underperforms training-free BM25 and frozen BGE-base on MapTask (Appendix Tables 17 and 23). Given its dependence on task-specific supervision and internal decoder states, in what deployment scenarios would RoleLex be preferable to training-free alternatives? Could the authors also clarify how their approach compares with LLM-based candidate selection, particularly given the free-form generation and text-matching protocol used for the prompting baseline? Q3. Stronger baselines for abstention evaluation. Table 3 compares RoleLex only against the final-layer baseline, although Table 1 shows that development-selected single- and two-depth representations are substantially stronger. Does RoleLex still improve abstention accuracy, F1, and AUROC over these stronger baselines? This comparison would help establish whether the reported improvements are attributable to the proposed factorization rather than better depth selection.

Reviewer_t1aqOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to build a cacheable cross-speaker referential retrieval key from a frozen dialogue decoder: the key must be computed at the mention endpoint, stored before future queries are known, and compared with a single dot product.

Critical strengths
  1. The mechanism respects the interface constraint, and the algebra is exact. The task requires a key that is computed once, cached, and searched by a single inner product. RoleLex satisfies this without approximation: z_i^T z_j = (1/R) Σ_r u_i^{(r)T} u_j^{(r)} holds by construction, the key is unit-norm, and the four d×64 maps cost exactly the same parameters and multiply–accumulates as one d×256 final-layer map. Such a design can be dropped into an existing retrieval layer without changing the index.

  2. The "where does the gain land" analysis is the right experiment, and the result matches the prediction. If the extra depths merely re-encoded surface form, the advantage should shrink on differently worded antecedents; instead it grows with antecedent distance (7.95 / 24.10 / 11.79 points at 21–50 mentions back), and the advantage is larger for pronouns than for lexical noun phrases. Combined with the fact that each independently decoded block is uniquely correct on 1.0–4.2% of queries, this is a coherent evidence chain rather than a single number.

  3. The control battery is unusually complete, and negative results are reported as such. Scalar mixture, shared projection (¼ parameters), global normalization, dense 4d×256 (4× parameters), raw/normalized means, fixed-weight β, 4/8/16 depth granularity, a final-RMSNorm hook, a label-efficiency check with GPT-5-generated labels, training-map subsampling, negative-mining variants, nonlinear heads, BM25, a fully fine-tuned BGE-base, and Qwen3-Embedding are all reported.

Critical weaknesses
  1. The statistical inference is presented poorly: the main text highlights significance against weak baselines, while the hard oracle-like baselines lack matching statistical tests. The sign test over 16 held-out maps (p = 3.05×10⁻⁵) and the gains +4.47 / +10.66 / +5.85 that the main text reports are all relative to the fixed final-depth key. Yet the paper's own experiments already show that the final layer is itself a weak readout for cross-speaker referential retrieval. That significance therefore establishes only that RoleLex is significantly better than the final-layer baseline.

  2. The claim "matches or beats the best selected pair" carries too much weight on two of the three backbones. On OLMo, RoleLex is 0.09 below the selected pair; on Qwen the margin is +0.40. Only Gemma shows a clear advantage (+2.17). The abstract presents the three backbones as a uniform result, which leads the reader to overestimate its consistency.

  3. The structural part of the contribution does not hold uniformly, and the abstract does not say so. The narrative attributes the gain to "keeping the depth scores separate and normalizing each on its own". But Table 2 shows a shared projection across depths (¼ the parameters) scoring 90.81 on Qwen, above RoleLex's 90.66, and global normalization scoring 87.87 on OLMo, above 87.71. §4.3 says this clearly — "there is no universal rule for how to share projections or where to normalize" — yet the abstract and the contribution list still present the factorization as the mechanism.

Questions for authors

See Critical Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.