- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper turns subsets around a target node, both observed and unobserved, into tokens arranged in an inclusion DAG. A Transformer with inclusion and role biases encodes them, and is pretrained with masked reconstruction against an EMA teacher. Experiments cover eight node classification benchmarks and two adverse-event datasets.
Critical strengths
-
Representing absent subsets explicitly is an interesting idea. It gives the model access to patterns that standard hypergraph message passing can miss.
-
The inclusion DAG fits the subset relationships well, and the results across several datasets suggest that the representation is useful.
Critical weaknesses
-
The definition of existence in Section 3 does not match the tokenizer. Section 4.1 gives sampled sub tokens existence 1 even when they are not observed hyperedges. The same subset can instead become a neg token with existence 0 if it is not sampled as a sub token. Its label therefore depends on the sampler. This makes it unclear how faithfully the input represents the COMP/EMER/INHIB patterns defined in Section 3.
-
Theorem 3 does not explain the benefit of inclusion-aware attention. Its separating construction sets the structural biases and attention contributions to zero; token-type counts alone distinguish the two hypergraphs. This supports the value of subset tokenization, but leaves the role of inclusion-aware attention unresolved.
Questions for authors
- Why do unobserved sub-hyperedges get existence 1? How does this fit with Section 3, and with the same subset getting different labels across views?
- What can inclusion-aware attention tell apart that mean pooling over the same token attributes cannot? Does that hold for the real bounded tokenizer, not just the idealized one?
4 comments
W2 and Q2. Theorem 3 and inclusion-aware attention
W2.1. What Theorem 3 shows
"Theorem 3 does not explain the benefit of inclusion-aware attention. Its separating construction sets the structural biases and attention contributions to zero; token-type counts alone distinguish the two hypergraphs."
We agree with this reading of Theorem 3. Its witness sets the structural biases and the attention contribution to zero and separates the two hypergraphs through the number of tokens of each type, as Remark 6 states. It establishes the value of subset tokenization; the role of inclusion-aware attention is shown by the example in W2.2.
W2.2 and Q2. What inclusion-aware attention can tell apart
Q2: "What can inclusion-aware attention tell apart that mean pooling over the same token attributes cannot?"
Two views whose token sequences are identical and whose inclusion edges differ, under the same tokenizer as Theorem 3. Take constant node features, , and
\qquad \mathcal{E}_B=\{\{c,a\},\{c,b,d\},H\}.$$ For center $c$, the tokenizer of Lemma 3 keeps every observed hyperedge that contains $c$ and all of their 2-subsets that contain $c$, with existence given by membership. In both hypergraphs this yields the center token and the five tokens below, where $T$ is the observed triple ($\{c,a,d\}$ in A, $\{c,b,d\}$ in B). |Token|Order|Existence|Adjacent superset in A|Adjacent superset in B| |---|---:|---:|---|---| |$P=\{c,a\}$|2|1|$T$ (COMP)|-| |$Q=\{c,b\}$|2|0|-|$T$ (EMER)| |$R=\{c,d\}$|2|0|$T$ (EMER)|$T$ (EMER)| |$T$|3|1|$H$ (COMP)|$H$ (COMP)| |$H$|4|1|-|-| The two token sequences are identical in feature, order, existence, source and position: $P$, $T$ and $H$ are observed hyperedges in both hypergraphs, and $Q$ and $R$ are unobserved 2-subsets of $H$ in both, so no token's source depends on the hypergraph. The token-type counts used by the witness of Theorem 3 therefore coincide, and any encoder that reads only the tokens returns the same output for A and B, for every choice of weights; this includes mean pooling and a Transformer without structural biases. The inclusion edges differ. In A the triple extends the observed pair $P$ (COMP) and the unobserved pair $R$ (EMER). In B it extends only the unobserved pairs $Q$ and $R$ (two EMER edges), and the observed pair $P$ has no adjacent superset. The direction bias $b^{\mathrm{dir}}$ is defined on these edges. With a nonzero $b^{\mathrm{dir}}$ and every other bias at zero, an explicit weight setting gives different representations for A and B: only the token $T$ changes, because it attends to one observed and one unobserved pair in A and to two unobserved pairs in B. By Lemma 5, HGPM then separates A and B for almost all weights. ### W2.3 and Q2. The bounded tokenizer > Q2: "Does that hold for the real bounded tokenizer, not just the idealized one?" The example needs no retention assumption under the tokenizer of Lemma 3. Under the bounded sampler it holds for views that retain these five tokens, which the sampler does not guarantee in general. The coverage table in W1 reports coverage under the actual sampler, while Table 2(c) compares encoders on its outputs. The empirical counterpart with the actual tokenizer is Table 2(c). The inclusion-aware encoder reaches 86.03 / 80.42 (homophilic / heterophilic), against 84.78 / 77.99 for a vanilla Transformer on the same tokens and 83.29 / 76.29 for mean pooling. On drug edge classification, the task is to predict the label of a (drug set, adverse event) record, and HGPM additionally reads subset-state evidence: for each subset of the queried drug set among its tokens, the state (positive, negative or unknown) of that subset for the queried adverse event (AE) in the training records, with the queried set's own state masked. With tokens, subset-state evidence and pretraining matched, the inclusion-aware encoder is 1.3 / 1.1 F1 points (HODDI / JADER) above the vanilla Transformer.Dear Reviewer,
As the discussion period is drawing to a close, we would like to check in on whether our response has resolved your concerns. We are happy to answer any further questions. If our response has addressed your concerns, we would be grateful if you would consider raising your rating. Thank you again for your time and thoughtful feedback.
Best Regards, Authors
Dear Reviewer,
With about 10 hours left in the discussion period, we would be happy to address any remaining questions or concerns you may have. Thank you again for your time.
Best regards,
The Authors
W1 and Q1. Existence of
subtokenssubtokens outside Congress have existence 0; the share is 73.09% on Congress. The others are fragments that are themselves hyperedges (first table below).negtokens only into positions that theobs/subpool leaves free. All tokens remain subject to the original per-order caps and source constraints, so retaining all three regimes in every view is not guaranteed; the second table below reports the resulting coverage.subtoken is a fragment of an observed hyperedge. Anegtoken is obtained by dropping, adding or swapping one member of an observed hyperedge orsubtoken, and is kept only if it is not a hyperedge. A subset reached by either route receives the same existence. The audit found no membership errors or cross-view existence conflicts on any of the eight benchmarks (first table below).Existence of
subtokens in the implemented tokenizer's outputs. Counts are token occurrences across views; existence is checked against the visible hyperedge set. In the last column, a subset is identified by its node set and compared across the views in which it occurs within the same dataset.subtokensRetained adjacent-order inclusion edges and view coverage. Each cell gives edge count (percentage of all views containing at least one such edge). Both endpoints have order at least 2. Each smaller-to-larger pair is counted once per view, before masking; center/readout edges and cross-view pairs are excluded. A pair occurring in several views is counted in each, so these are not globally distinct-pair counts. The percentages are view coverage, not edge shares, and need not sum to 100%.
The idealized tokenizer of App. B.3 (Lemma 3) already follows this membership rule: it gives 1S=0 to the unobserved 2-subsets of observed hyperedges.