- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to combine representations from different layers of pretrained PLMs. The authors propose DynamicGate, which learns protein-specific layer weights using downstream task supervision while keeping the pretrained encoder frozen. Experiments across multiple PLMs and four downstream tasks show improvements over fixed-layer and shared-weight fusion baselines, although the gains are not consistent across all settings.
Critical strengths
- DynamicGate improves downstream performance across several PLMs and tasks without fine-tuning the pretrained encoders.
- The experiments show that useful layer representations vary depending on the downstream task, suggesting that how pretrained representations are combined also matters.
Critical weaknesses
- The advantage of protein-specific weighting over shared layer fusion is less clear on some tasks. While DynamicGate performs better on PISCES, Mean-Gate achieves comparable or slightly better results on several SCOP classification settings (App. D.1).
- DynamicGate introduces additional parameters compared with simpler baselines. Although App. D.2 includes parameter-matched MLP controls, comparing training and validation performance would help clarify whether there are generalization gaps.
- Some methodological details need clarification. Sec. 2.2 does not specify the architecture of the shared scoring network, and Top-4/8 Gate is introduced in Sec. 4.3 without a definition.
- The related work could also cover studies on intermediate representations and layer fusion in vision and NLP, beyond PLMs.
Questions for authors
- Could the authors include Mean-Gate as a baseline in the main result tables, particularly Tables 2 and 3, to clarify the benefits of protein-specific weighting over shared layer fusion?
- Could the authors report training and validation performance for different fusion strategies to help assess potential generalization gaps?