- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents TACO-Graph, a framework for selecting text-attributed graph constructions from tabular datasets. It organizes construction choices along three axes: node entities, edge relationships, and textual attributes. Valid configurations are evaluated using GraphSAGE classification or GraphRAG question answering, and a decision tree is fitted to their mean scores to derive interpretable construction rules. Experiments on five datasets show strong agreement between predicted rankings and rankings measured on independently drawn evaluation samples. The paper also studies ranking stability as the evaluation budget increases.
Critical strengths
- Relevant problem formulation. Graph construction is often treated as a fixed preprocessing step, despite its potential impact on downstream performance. Explicitly evaluating these choices is useful for practitioners.
- The writing of paper is clear.
Critical weaknesses
- First of all, I think for a top-tier conference, the paper’s methodological contribution is insufficient, and the methodology is described in very limited detail.
- Comparability of the classification task across node types is insufficiently specified. The search changes the prediction unit between primary, secondary, and aggregate entities. For example, Amazon configurations use reviews, reviewers, and product families, while the task is described as predicting product subcategories. A reviewer can be associated with multiple categories, just as an author can publish papers in multiple subject areas. The paper does not sufficiently explain how labels and text are assigned to these entities, how conflicting labels are handled, or whether predictions are evaluated against a common set of underlying test examples. Consequently, differences attributed to construction quality may also reflect changes in target definition, label distribution, text aggregation, or task difficulty.
- The GNN experiments lack a text-only classifier using the same embeddings and a corresponding model with edges removed. These baselines would help determine how much of the observed performance variation comes from graph structure rather than textual information and entity aggregation. Replacing all embeddings with zeros, as in T6e, does not answer this question.
Questions for authors
- In Figure 2, Sports primary-node classification scores are approximately 55–65, whereas the corresponding Figure 3 leaf values are approximately 39.5–41.3. Section 3.2.1 defines a pseudo-(R^2) score floored at zero, but Section 4.1 describes no-text controls as remaining at the majority-class floor, and Figure 2 shows a nonzero control curve. Section 3.2.2 describes question-level faithfulness filtering, whereas Appendix A describes subtype-level pilot filtering. The procedure producing History’s 100/100/33 questions is not sufficiently clear. Can authors explain these?
- Can the authors report the agreement between proxy rankings and full-scale GNN rankings, together with the final performance of the selected configuration relative to the full-scale best?
Plot, Baseline + Appendix Details
Thank you very much for the thorough review! We particularly appreciate your close reading of the numbers across our figures, which caught plotting inconsistencies we have now fixed. We respond in four sections.
Thank you for catching the inconsistent numbers! We save both raw accuracy and the McFadden pseudo-R² for every run. Figure 2 displays accuracy while Figure 3's leaves display pseudo-R², which is why one reads 55–65 and the other 39.5–41.3. This is a labelling error on our part rather than a disagreement in the underlying data. It also explains your second observation: the no-text control sits at the majority-class floor on the accuracy scale, which is nonzero, but correctly at zero on the pseudo-R² scale (Section 3.2.1). We have put both figures on pseudo-R², updated the values quoted in Section 4.1, and made Section 3.2.1 explicit about which quantity is which. On question filtering, Section 3.2.2 and Appendix A describe the same procedure at different granularities, and we have rewritten both to describe the single procedure we run. For History's 100/100/33 split, we generate 100 questions each for the single-entity and multi-hop subtypes and 33 for the aggregate subtype, because History's aggregate entity is too coarse to support 100 distinct aggregate questions (even with multiple attempts using GPT 4o-mini).
We describe method-level components in the body and delegate dataset specifics to Appendix A. However, Appendix A currently does not specify how labels are assigned when the prediction unit changes with node type, and we have itemized this detail per dataset in this section now. For Sports, N1 predicts a review's product subcategory, N2 a reviewer's modal subcategory across their reviews, and N3 a product family's subcategory, which is deterministic since each family maps to one subcategory. We also believe the McFadden pseudo-R² at least partly addresses your label distribution concern: it measures lift over each task's own class prior, so a node type with a more skewed class distribution gets no credit for that. It does not remove differences in how predictable each target is, and we agree that choosing a node type inherently means choosing the prediction unit. We now state this in Section 3.2.1 and list a fixed-target evaluation, scoring every node type against review-level labels, as a limitation.
You are right that T6e does not answer whether graph structure adds value beyond text, since zeroing the text removes the wrong thing. We do train a structure-blind MLP (which is saved in our anonymized code) on identical embeddings alongside every GNN run, but never reported it. Averaged over variants, the GNN's lift over the MLP is +1.4 points on History and +1.7 on ArXiv, but −1.3 on Sports, −0.9 on Electronics and −1.0 on Toys. These gaps are small next to the spread across constructions on most datasets (50 points on Electronics, 46 on ArXiv, 32 on Sports), so message passing changes scores far less than node and text choices do. The construction ranking is strongly preserved between the two models (ρ of 0.86 to 0.96), so for node classification our trees mostly rank node and text choices rather than edges, with edge type receiving only 0–2% of tree importance across all five datasets. However, edge choice matters far more for GraphRAG, where retrieval trees assign 62–100% of importance to edge type on all five datasets (Appendix B). We read this as evidence that the value of edge construction depends on how much the task requires traversing the graph, which is additional evidence towards varying value of axes depending on the task you are trying to accomplish. We have added the MLP comparison to Table 3 and Appendix E, and revised Section 4.1 accordingly.
Yes, we have run this (per-dataset table in Appendix C, and in our responses to Reviewers JxMn and 6riB). Against full scale rankings our proxy gives a Spearman ρ of 0.75–0.90 (mean 0.82) compared to 0.97 in Table 3, all statistically significant. In selection terms, the construction we rank first is the full scale optimum on three of five datasets, with a mean top-1 regret of 0.9 points and a top-3 regret of at most 0.1 points everywhere (out of 100). The lower ρ comes mostly from reordering along the edge axis rather than at the top of the ranking, because edge rules scale differently: a construction that is sparse at 1,000 rows can be near-complete at 50,000, while other rules leave 48–76% of nodes isolated. Together, these show that subset rankings identify near-optimal full-scale constructions, and we now report both side by side in Section 4.3.
We will upload the revised and reformatted PDF on Monday 11/16.