- Rating
- 3
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes RoleLex, a method for cross-speaker reference retrieval that freezes the language model and trains lightweight projections over multiple layers. It combines these representations into one vector and retrieves matching references using similarity rather than text generation. Experiments on MapTask-RefMem and OneCommon across three model families show improvements over final-layer readouts, supported by ablations of layer selection and aggregation.
Critical strengths
- RoleLex provides a simple and practical way to reuse frozen LLM states for reference retrieval. It improves over final-layer readouts without increasing the retrieval key size or projection budget, and avoids a separate encoder when LLM states are already available.
- The evaluation covers two datasets, three model families, and multiple random seeds. Detailed layer-selection and aggregation ablations strengthen the empirical evidence and provide useful insights into the benefits of multi-layer representations.
Critical weaknesses
- The comparison with generative approaches is incomplete. The prompting baseline is not fine-tuned on the task data, whereas RoleLex receives task-specific supervision. This leaves its accuracy–cost trade-off against a similarly trained generative model unclear.
- The performance advantage over the strongest external baseline is limited: RoleLex achieves 90.59% Recall@1 versus 90.51% for fine-tuned BGE-base. Its practical advantage therefore relies mainly on reusing already available LLM states; in the reported standalone setting, BGE is faster with comparable accuracy. This narrows the deployment settings in which RoleLex offers a clear benefit.
Questions for authors
Could you add a generative SFT baseline for one representative backbone, using the same training splits and referent annotations, and report its retrieval accuracy and inference cost? This would clarify RoleLex’s accuracy–cost trade-off against a task-adapted generative model, rather than an untuned prompting baseline.