- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors consider the problem of temporal link prediction and focus on the idea of learning temporal patterns once, that is, using one dataset, and then applying the predictor to unseen datasets. The reasoning behind this is that different temporal graphs often follow similar patterns, so that the dynamics of one dataset should enable learning general patterns that can be found in other datasets as well. The authors define a set of 24 features that characterise the activity patterns of node interactions, including aspects such as recency and frequency of interactions. To make scores across the different features comparable, the authors normalise them. To test their approach of learning interaction patterns from one dataset and predicting for others, they fix tgbl-wiki-v2 as the training dataset and predict on 4 datasets (tgbl-review, tgbl-coin, tgbl-flight, and TGB-Seq/GoogleLocal).
While I find the central idea of learning on one dataset to predict on others exciting and see great potential, I had trouble following the authors' discussions. Overall, I felt that it was difficult to understand the purpose of each paragraph: what are the authors saying and why? What are the important take-home messages for the reader? Further, I believe that the evaluation needs to be more extensive: showing that the method works by only focusing on one source dataset seems insufficient to me. Moreover, I am missing some analyses regarding "how to choose a good source dataset?" and "when does the approach work vs. fail?". Overall, I find that the paper requires careful and rather extensive revisions to clarify the take-home messages and show that the method works before it can be considered for publication.
Critical strengths
I believe the biggest strength of this paper is the simplicity of the idea, which, despite, or perhaps exactly because of its simplicity, can have a wide-reaching impact.
Critical weaknesses
- I had trouble following the authors' discussion and figuring out what the purpose of each paragraph was. I recommend the authors revise their paper for clarity and consider the exact role of each paragraph. What should the reader have learned after each paragraph? What is the take-home message?
- I do not believe that the experimental evaluation is sufficient. Specifically, I believe that evaluating only one source dataset for training is not enough to make a general statement regarding how well the proposed method works -- essentially, we have no idea what to expect when we use another dataset for training. Moreover, we do not understand the failure modes: when does it not work?
Questions for authors
There are a few things that remained unclear to me while reading the paper:
- I was wondering what the motivation for your work is. Did the idea come from a specific application or problem? Does it have to do with saving computational resources?
- I encountered some terms that I am not used to in the ML literature. For example, by "control", do you mean "baseline"? What do you mean by "official test" in Tables 1 and 2?
- How do you recommend selecting the source dataset? For example, let's assume that I have some datasets and want to select "the best" (or maybe just a good enough) source dataset to predict on the rest of my datasets. How should I do that? Is there some "analytical" way you would recommend?
- Connected to the previous question, is there a way to detect "outlier datasets" in the sense that we could say "if trained on dataset X, we should expect that the predictions on dataset Y would be bad, so don't predict on Y"? Perhaps this is actually the same question as the previous one, just put in other words.
- The performance shown in your tables does not seem to be particularly convincing. What do you think why that is? Could this be a problem of not having chosen the right source dataset?