- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper focuses on the problem of negative sampling during training for dynamic/temporal link prediction. The author's thesis is twofold: (a) historical negatives should be used during training in addition to random negatives; (b) the type of negatives should differ for different model components. Based on this, the authors design a new training method that uses two different losses to update a node and pairwise scorer with a shared encoder. They further employ a mixture-based method that mixes the losses for both scorers. They test their method on two datasets, detailing the results.
Critical strengths
- The performance improvement on tgbl-review is good.
Critical weaknesses
-
The paper writing is extremely unclear. To be blunt, I had a very time reading and understanding this paper. I include some concerns below from the first two sections:
(a) What's and on line 109 mean? , , , and have never been introduced.
(b) What's the point of having is those representations are already passed to ? It's not explained.
(c) What's on line 115? Where is it used? Why is it defined as such?
(d) It would be much easier if you concretely defined and in terms of equations. Because it's really hard to understand the authors based on their writing alone.
(e) Why is in Eq. 3?
(f) Why do the negatives only update the loss and for ? It's not explained.
(g) What's "XY" on line 148?
(h) I'm unsure what's happening in section 2.3. I thought that the and negatives are only applied to specified model components? Why are they being mixed now? -
Is evaluation unchanged? Specifically the negatives? One thing I'm concerned about is that some historical negatives used during training bleed into evaluation. It's unclear to me is that's controlled for.
-
The loss analysis (Section 3) is confined to . However, the score contains other components as well (on line 196 this is called which I assume is both and ). So yes, they would have the same fixed history encoding, but we will still have the learnt encodings! These will very likely be different. If anything, this is just a limitation of the encoding. So in practice, I'm unsure how much this actually matters.
-
So from my understanding is the standard training protocol (please correct me if I'm wrong). If that's true, it's seems that the it actually does quite well on . So it seems that this strategy only helps sometimes? When is this the case and why? It's unclear.
-
The authors only use 2/5 of the link prediction datasets in the TGB benchmark. Why? This severely limits the analysis and results because we have no idea what happens on those other datasets.
-
Section 5 essentially just references the appendix. However, the experiments in the main text should be ``self-contained'' as per the author guidelines. This further reduces the readability of the paper.
-
The authors should use newer and better temporal methods. Some examples include [1, 2] which do better than the methods used by the authors. It's unclear what the results will be on these methods than those used by the authors.
Questions for authors
- All my comments in weakness 1.
- Is the evaluation the same as used in the original TGB paper? (weakness 2)
- Why does the H loss help sometimes but not other times? (weakness 4)