- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes GTLM, which adds graph-derived attention biases to pretrained LLMs so they can use node text and connectivity together. It combines per-node positional resets with bidirectional prefix attention to remove dependence on the order in which context nodes are presented.
Critical strengths
- GTLM keeps token-level representations of node text and adds graph structure directly to attention. The design for node permutation equivariance is also clearly motivated.
- Experiments cover several graph-related tasks. The synthetic retrieval comparison, using the same backbone and training budget, gives convincing evidence of gains in multi-hop retrieval and length generalization.
Critical weaknesses
- The added benefit over an information equivalent, same backbone, same budget text LoRA baseline is not well established. Removing biases in Appendix K can also remove connectivity information, and the effects of biases, positional resets, and prefix attention are not fully separated.
- The paper includes 1B, 3B, and 8B experiments, but mainly relies on Llama3. BLOOM and Gemma are tested only on GraphQA. Whether the gains hold on more recent representative backbones remains unclear.
- Separate fine-tuning shows within task effectiveness, but does not tell us whether a trained adapter can be reused on new tasks.
- Recovering standard computation for a single node does not undo LoRA updates. The paper does not evaluate how general capabilities change after graph-task training.
Questions for authors
- With connectivity explicitly described in the text, how does GTLM compare with a standard LoRA tuned LLM using the same backbone and training budget? Separately, can the contributions of structural biases, positional resets, and bidirectional prefix attention be disentangled?
- Do these relative gains reproduce on more recent representative backbones, such as Qwen?
- Can a trained adapter transfer to unseen tasks without target task fine-tuning?
- With the graph trained LoRA enabled, how do general capabilities compare with those of the original model?