acceptodds
Under review as a conference paper at ICLR 2027

SAligner: Pair-Efficient Transferable Pre-Training for Sparse Text-Attributed Graphs

Abstract

Pre-training on text-attributed graphs (TAGs) is central to building transferable graph foundation models, where LLM-as-Aligner methods align graph and text representations through the semantic knowledge of large language models. However, these methods usually assume that node texts provide sufficient and reliable supervision, an assumption often violated in real-world sparse TAGs. When textual anchors are missing, noisy, or uneven across domains, graph structures must be aligned with weak semantic evidence, leading to unreliable structure–semantics correspondence and sparsity-induced transfer bias. This paper presents SAligner, a Sparsity-aware and Structure-enhanced LLM-as-Aligner framework for graph–text pre-training on sparse TAGs. The key idea is to decouple semantic alignment from structural modeling, allowing topology-aware signals to enhance alignment without contaminating the shared semantic space. Specifically, SAligner decomposes graph–text representations into semantic and structural components, uses structure-oriented reconstruction with consistency control to inject reliable topology cues into text representations, and suppresses inconsistent structural signals under textual sparsity. Moreover, SAligner introduces sparsity-aware cross-domain risk balancing, which calibrates domain risks through a global-domain density ratio and downweights unreliable sparse samples via graph reliability estimation. Theoretical analysis shows that this objective reduces cross-domain generalization gaps by controlling domain risk discrepancy. Extensive experiments across diverse graph domains, sparsity levels, and downstream tasks demonstrate that SAligner consistently outperforms existing baselines. Anonymous code is available at https://anonymous.4open.science/r/S2aligner-3651.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Graph Foundation Model, Graph learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DxvfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies graph-text pre-training for sparse text-attributed graphs, addressing how missing or noisy node text destabilizes alignment and cross-domain transfer in existing LLM-as-Aligner methods. Experiments across zero-shot classification, link prediction, and retrieval show strong overall performance, with gains varying across tasks and directions.

Critical strengths

The consistency gate is a clever addition. By dynamically scaling structural signal injection based on reconstruction agreement, it prevents noisy structural representations from destabilizing training under sparse textual supervision.

The cross-domain risk balancing mechanism aligns well with zero-shot transfer.

Critical weaknesses

The individual contributions within the cross-domain risk balancing mechanism are not fully disentangled.

The text-sparsity experiments rely on random dropouts rather than realistic missingness patterns. Without testing scenarios where missing text correlates with node classes, domains, or graph structures, the robustness claims under non-random missingness remain uncertain.

Questions for authors

Could you clarify the exact rule used for dropping or retaining node text in your experiments? Also, how closely does this setup mimic real-world missingness patterns where data availability might depend directly on node labels or graph topology?

Based on the current results, is there any clear way to tell how much of the zero-shot transfer boost comes from the density-ratio weighting versus the cross-domain risk-variance regularizer?

1 comment

Thank you for the additional experiments on non-random text availability. They substantially address my concern about the original random masking protocol. The remaining issue is the contribution of the cross-domain risk-variance regularizer. The rebuttal reports only the density-ratio = 1 ablation, while the risk-variance term is retained.

Reviewer_DWDcOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the vulnerability of multimodal graph pre-training when node textual attributes are incomplete or noisy, which degrades cross-modal supervision and introduces domain transfer bias. To resolve this, the authors develop a pre-training framework that factorizes graph and text representations into separate semantic and topological components, using a structure reconstruction objective with consistency gating to selectively inject reliable structural cues into the text representation space. The framework mitigates domain-specific overfitting by balancing cross-domain empirical risks, reweighting instances via a global-to-local density ratio alongside graph reliability metrics, and penalizing risk variance across domains. Across zero-shot node classification, link prediction, and bidirectional retrieval benchmarks, this approach consistently outperforms standard dual-encoder baselines, maintaining strong transfer performance even when only one to ten percent of node text attributes are available.

Critical strengths

The writing and figure are generally clear.

Critical weaknesses
  1. The problem setting is not very practical. For zero-shot prediction, first, what's the use case of zero shot prediction? If we know the categories, we can easily get some labels in practice. Second, why don't we directly use an LLM? If we can use some graph-based methods as tools, and give LLMs proper context, I believe system 1 treatment like JEV can handle these kinds of tasks well. Moreover, The paper simulates sparsity by uniformly and randomly dropping node text attributes to retain 1%–10% across all nodes. In production TAGs (e.g., e-commerce catalogs, citation graphs, financial fraud networks), textual sparsity is virtually never Missing Completely at Random. Instead, it follows structural biases like degree-biased or power-law sparsity (think about the cold start points in recommendation system). Evaluating zero-shot transfer primarily on Cora and CiteSeer (toy datasets) undercuts the realism of the claims. (For future works) I suggest authors to think about some more valuable applications such as multi-modality, LLM latent space analysis, and recommendation system instead of these toy tasks.
  2. Theorem 3.1 and Appendix D.1 justify the framework using invariant risk minimization under a linear regression setting with mean squared error. In practice, the framework optimizes a highly non-linear dual-encoder architecture with a temperature-scaled symmetric contrastive InfoNCE loss and a cosine reconstruction loss. Risk invariance in linear regression does not provide gurantees.
  3. The framework stacks multiple auxiliary objectives and weighting factors, including loss balancing weights, density ratio scaling parameters, and a multi-part reliability score. This questions the general stability of the framework.
  4. In knowledge graph completion, the universally accepted protocol is 1-vs-all filtered entity ranking measured by MRR and Hits@K, not binary classification AUC. More critically, the underlying evaluation mechanism appears fundamentally mismatched. Standard KG link prediction requires scoring a query (h,r,?)(h, r, ?) against the entire entity set—tens of thousands of candidates—to determine whether the true entity ranks at the very top. Reporting binary AUC implies the model is merely evaluated on distinguishing positive edges against a small, randomly sampled negative set (such as 1:1 paired negatives).
  5. For node classification, Standalone frozen text embeddings already yield strong results without any graph training. A necessary baseline is non-parametric feature propagation: taking frozen text embeddings from the observed nodes, diffusing them across graph edges via simple graph smoothing, and running zero-shot cosine similarity matching.
Questions for authors

Please see weakness. I will raise my scores if these problems are well addressed.

1 comment

Thanks for the new response. GraphGPT are weak baselines and indeed if you well utilize the text embeddings of class names you can utilize KNN to get good zero shot performance for most node classification tasks. So, it's not a good idea to evaluate pre-training with such scenario and I also doubt the real-world values of such setting. I recommend authors to rethink the problem settings and will keep my score.

Reviewer_WW8LOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes S^2Aligner for graph-text pre-training with missing node texts. It uses reconstruction and a consistency gate to add structural information to text embeddings, and density-ratio and reliability weights to balance source-domain risks. The evaluation covers zero-shot node classification, link prediction, and graph-text retrieval.

Critical strengths

This paper addresses a practical problem in graph-text pre-training and reports good transfer results with limited text.

  • Using graph structure to improve text representations is a reasonable way to handle missing node texts.

  • With 10% of source texts retained, Table 1 reports 59.88% average accuracy, compared with 55.16% for GraphCLIP. The experiments use five seeds. Table 5 also tests the reliability indicators.

Critical weaknesses

Motivation and design

  • The motivation includes uneven text availability, but Section 4.1 mainly varies the retained fraction from 1% to 10%. What if missing texts concentrate in one community or domain? That community may have no nearby semantic anchors at the same retention rate. Table 5's noise and pair-shuffling tests do not cover this case. Source and target text availability should also be varied separately to check whether the method handles missing text at deployment.

  • Section 3.1 does not give the prompts or procedure for generating the two text views. How does the graph encoder separate content and structural embeddings? Appendix E.1 replaces missing texts with zero vectors and "unknown" flags. How are the text views generated after masking, within the 10% text budget?

  • Appendix F.1 says only the graph encoder and projector are optimized. How are the reconstruction and density modules in Eqs. (5), (9), and (15) trained? Which losses are used, and are the sample weights detached?

  • Theorem 3.1 studies an unnormalized risk, while Eqs. (12)-(13) normalize weights within each domain. Does the result still hold for this normalized objective?

Experiments

  • The text-only baselines need updating. Table 1 uses SBERT and Qwen3-0.6B, a 600M-parameter model. Qwen3 was released on April 29, 2025, nearly 18 months before this review (release). Please compare with recent models such as Qwen3.8-27B and Qwen3.5-9B (2026 releases). Use the same observed texts and class descriptions, without target training labels. Report the prompts, inference budget, and cost.

  • Appendix G.2 varies the text encoder only for S^2Aligner, using MiniLM-L6, E5, and Qwen3-0.6B. Does its advantage over GraphCLIP and ADAligner remain when all three use the same stronger encoder? Keep masks, summaries, graph backbone, and training budget fixed. The checkpoints and embedding extraction settings should be specified.

  • What does "pair-efficient" save? Figure 7 reports time per epoch, but not total cost at matched accuracy. Please include summary and description generation, and report the observed texts, graph-text pairs, and training updates.

  • Figure 8 removes whole modules, and Table 5 tests reliability indicators. The gate and density ratio still need separate tests. Set the gate to 1 while retaining structural injection. Separately, set the density ratio to 1 while retaining reliability weighting and the risk-variance penalty. Keep other settings fixed.

Questions for authors
  • How are the two text views and graph branches constructed after masking? Which reconstruction and density parameters are trained, and are the sample weights detached?

  • How does S^2Aligner compare with newer text-only models such as Qwen3.8-27B and Qwen3.5-9B under the same text budget?

  • Does the gain over GraphCLIP and ADAligner remain when all methods use the same stronger text encoder? Please report accuracy and total cost.

  • Does the method still help when missing texts concentrate in particular communities or domains? What happens when source and target text availability are varied separately?

Comments on the paper

1 comment
Area_Chair_hpUCOpenReview ↗

Discussion reminder

Dear Reviewers,thank you for your time and thoughtful evaluation of the Paper. The authors have now submitted their rebuttal addressing the questions and concerns raised in your initial reviews. Please carefully review the authors' responses alongside the other reviews, and participate in the discussion to share your updated thoughts and comments. To help us reach a well-informed meta-review decision, please let us know if the rebuttal adequately resolves your concerns and whether you plan to adjust your final rating. Please submit your post-rebuttal comments and updated evaluation on the review portal. Thank you again for your valuable service and contribution to the review process.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.