acceptodds
Under review as a conference paper at ICLR 2027

Temporal Residual Memory for Mitigating Catastrophic Forgetting in Continual Knowledge Graph Embedding

Abstract

Continual Knowledge Graph Embedding (CKGE) aims to sequentially incorporate newly arriving facts into a knowledge graph while preserving the knowledge learned from previous snapshots. A key challenge is catastrophic forgetting, where the knowledge acquired from past snapshots can be rapidly lost as the model adapts to a new snapshot. Existing studies have mainly focused on preserving past knowledge and transferring it stably across snapshots to mitigate such forgetting. This work starts from the observation that relations and entities may exhibit different degrees of semantic stability as a knowledge graph grows. Based on this perspective, we propose Temporal Residual Memory (TRM). TRM introduces snapshot-wise entity-side residual memories through the BoxE bump path, freezes memories from previous snapshots, and aggregates valid memories according to each entity's birth snapshot for current prediction. Experimental results show that the proposed method achieves strong performance on diverse CKGE benchmarks and improves knowledge retention metrics. In addition, experiments extending the core idea of TRM to TransE and DistMult also show a tendency toward mitigating forgetting. These results suggest that leveraging entity-side memory formed from past snapshots for current prediction can serve as an effective strategy for mitigating forgetting in CKGE.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual knowledge graph embedding, catastrophic forgetting

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_yL8sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a temporal residual memory method to mitigate catastrophic forgetting in continual knowledge graph embedding. Using BoxE as the backbone, the method stores snapshot-wise entity residuals, freezes the historical residuals while learning the current residuals, and aggregates the effective memory according to the first appearance time of each entity, which is then integrated into the current scoring process. Furthermore, a retention trajectory evaluation metric is introduced to measure the retention of old knowledge during subsequent learning.

Critical strengths
  1. The management strategy for historical and current residuals is clearly defined. Freezing historical residuals prevents them from being directly overwritten during current training while preserving the adaptation capacity of the current residuals. In Table 5, when freezing is disabled, the retention trajectory metric values turn negative across all seven settings, supporting the role of the freezing design in the tested configurations.
  2. The experiments provide controlled evidence for the memory mechanism. Same-backbone ablations show that the full method achieves higher final ranking performance than the memory-free variant across all seven settings; the extension experiments on TransE and DistMult further offer supplementary evidence under alternative scoring backbones. These results support the effectiveness of the proposed mechanism in the reported configurations.
  3. The manuscript is well organized, with the methodology, experiments, and limitations addressed in dedicated sections.
Critical weaknesses
  1. The contribution attributable to the newly introduced mechanism remains insufficiently prominent. As shown in Tables 2 and 5, under the entity, relation, and mixed growth settings, the memory-free BoxE variant equipped with drift regularization already achieves higher MRR than ERPP. While the incorporation of the memory mechanism yields further gains, the overall ranking advantage also benefits from the backbone architecture and the regularization design. Therefore, its distinctive contribution relative to existing strategies warrants more explicit clarification.

  2. The anti-forgetting gains are not consistent across settings. In Table 5, the full method attains lower RTE values than the memory-free variant under certain settings. Although the authors have already discussed this phenomenon, the trade-off between predictive performance improvement and knowledge retention warrants further clarification.

Other issues: A figure is recommended to visually illustrate how performance on previously learned knowledge evolves as learning progresses. In addition, the reference list contains duplicate entries and formatting inconsistencies that require careful revision.

Questions for authors

See Weaknesses

2 comments

Thank you for highlighting the separation between frozen past residuals and trainable current residuals, the evidence from the same-backbone ablations and the TransE and DistMult extensions, and the organization of the manuscript.

Weakness 1. The contribution attributable to the newly introduced mechanism remains insufficiently prominent. As shown in Tables 2 and 5, under the entity, relation, and mixed growth settings, the memory-free BoxE variant equipped with drift regularization already achieves higher MRR than ERPP. While the incorporation of the memory mechanism yields further gains, the overall ranking advantage also benefits from the backbone architecture and the regularization design. Therefore, its distinctive contribution relative to existing strategies warrants more explicit clarification.

TRM uses separate residuals to preserve past information and adapt to new knowledge. We explain below how this design can support retention, with the ablations providing empirical evidence for its effect.

In standard sequential learning, the loss on a new snapshot directly updates existing entity representations. Learning new facts can therefore overwrite representations formed in earlier snapshots. TRM maintains a separate entity-side residual for each snapshot. During training on a new snapshot, only that snapshot's residuals are trainable among the stored residuals. Earlier residuals are frozen and retain the corrections they learned. The current residuals can thus capture changes associated with new facts while the past residuals continue to support retention.

BoxE's architecture gives these residuals a natural place in the scoring process. It represents each entity with a point and a bump, and each relation with separate regions for the head and tail roles. An entity's bump modifies the role-specific representation of its counterpart. TRM adds the snapshot-wise residuals to this bump, allowing the current cross-bump computation to reuse interaction corrections learned in earlier snapshots while retaining BoxE's relation-side scoring structure.

The scoring geometry also helps explain why this mechanism works well with BoxE. In TransE, the translated head representation is trained to approach a specific tail point. In BoxE, the representation augmented with residual memory is trained for compatibility with a relation-specific region. When the model combines corrections from different snapshots, the region gives it more room to accommodate different changes in entity interactions than alignment to a single target point would. We view this as a property of BoxE that makes it particularly suitable for snapshot-wise entity residuals.

The ablations clarify what the memory adds. The w/o Memory variant already performs well using BoxE's point-bump scoring and drift regularization. With the same backbone and regularization, full TRM achieves higher final MRR on all seven datasets and maintains positive RTE on each. In w/o Freeze, where past residuals can still be updated during later training, MRR is lower than for full TRM and RTE is negative on all seven datasets. These results support the role of freezing past residuals in retention, beyond the effect of adding residual vectors alone.

The same principle applies to TransE and DistMult. Creating residuals for each snapshot, freezing past residuals, and reusing them in the current entity representations improves ranking performance for both backbones and improves retention in most settings. The magnitude of the benefit depends on the backbone's scoring geometry.

Weakness 2. The anti-forgetting gains are not consistent across settings. In Table 5, the full method attains lower RTE values than the memory-free variant under certain settings. Although the authors have already discussed this phenomenon, the trade-off between predictive performance improvement and knowledge retention warrants further clarification.

With margin loss, adding memory lowered RTE on RELATION, HYBRID, and GraphHigher. RTE averages both gains and drops relative to each snapshot's initial MRR. To understand why RTE decreased, we separately computed the gains above the initial MRR and the drops below it.

We use the relative change δi(s)=(ms,i−mi,i)/mi,i\delta_i(s)=(m_{s,i}-m_{i,i})/m_{i,i} from Eq. (1) and the test-set weight wiw_i from Eq. (2). Here, ii is the snapshot to which the test set belongs, and ss is the snapshot through which the model has been trained. We define the weighted gain and weighted drop as follows.

\sum_{s=i+1}^{S} \max(\delta_i(s),0).$$ $$\mathrm{RTE}^{-} = \sum_{i=1}^{S-1} w_i \frac{1}{S-i} \sum_{s=i+1}^{S} \max(-\delta_i(s),0).$$ Gains above a snapshot's initial MRR contribute to the first component, and the magnitudes of drops below it contribute to the second. For each snapshot, we average both components over all $S-i$ subsequent evaluations and then weight them by test-set size. This gives $$\mathrm{RTE}=\mathrm{RTE}^{+}-\mathrm{RTE}^{-}.$$ | Benchmark | Weighted gain (RTE⁺) | Weighted drop (RTE⁻) | RTE | | ----------- | -------------------- | -------------------- | --------------- | | RELATION | 2.620% → 1.922% | 0.355% → 0.000% | 2.265% → 1.922% | | HYBRID | 1.758% → 0.922% | 0.902% → 0.573% | 0.855% → 0.349% | | GraphHigher | 3.009% → 2.008% | 0.000% → 0.183% | 3.009% → 1.825% | Each cell shows the change from BoxE without memory to TRM. RTE is computed from the two components before rounding. On RELATION and HYBRID, adding memory reduced the weighted drop. RTE still decreased because the weighted gain fell by more than the weighted drop did. On RELATION, TRM matched or exceeded each snapshot's initial MRR in every subsequent evaluation. On HYBRID, TRM fell below the initial MRR on the second snapshot's test set in a subsequent evaluation. It reduced the drop on the fourth snapshot, whose test set was about five times as large, so the overall weighted drop decreased. On GraphHigher, the weighted gain decreased and the weighted drop increased, with MRR falling below its initial value on the third snapshot's test set. We propose reporting RTE⁺ and RTE⁻ alongside RTE. When RTE decreases, the two components show whether gains above initial performance became smaller or drops below it became larger. With the same margin loss, adding memory changed RTE from negative to positive on FACT and GraphLower. With negative sampling loss, adding memory to the same BoxE backbone improved both RTE and ROD on all seven benchmarks. The point of Tables 11 and 12 is that residual memory can mitigate forgetting of knowledge learned in earlier snapshots even when the BoxE backbone is held fixed. > **Other issues.** A figure is recommended to visually illustrate how performance on previously learned knowledge evolves as learning progresses. In addition, the reference list contains duplicate entries and formatting inconsistencies that require careful revision. We checked the reference list and found duplicate entries for the paper on a memory-based gradient method, the broader survey of continual learning, and the survey of knowledge graph embedding from a representation-space perspective. We will consolidate the duplicate citation keys and standardize the formatting of author names, venues, titles, and page numbers.
Reviewer_gRR5OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper tackles forgetting in continual KG embedding by giving each entity a small residual vector per snapshot on top of BoxE. Residuals from earlier snapshots are frozen, and the bump used for scoring is the base bump plus the mean of the entity's valid residuals (Eq. 4), with L2 drift penalties on the other parameters. The authors also propose RTE, a retention metric that averages the relative MRR change over all later snapshots rather than only the last one. TRM gets the best MRR on ENTITY, RELATION and HYBRID, is behind ERPP on the other four benchmarks, and has positive RTE on all seven. The same residual trick also helps TransE and DistMult.

Critical strengths
  1. It's simple and easy to attach to other scorers. There is no replay and no distillation, and Appendix E shows it working with TransE and DistMult.

  2. I appreciated how much the appendix shows, including runs where the memory hurts and a loss change that turns TRM's RTE negative on four datasets. I end up reading these tables differently from the authors, but I could only do that because they're in the paper.

  3. A side result I found more interesting than the main one: plain DistMult (0.287 MRR on ENTITY, Table 6) beats every TransE-based CKGE method in Table 2. If that holds up, a lot of recent CKGE work has been comparing continual-learning mechanisms on top of a weak scorer.

Critical weaknesses
  1. My main concern is that the gains come from BoxE and not from the memory. The ablation in the paper is what convinced me. Without any residuals "w/o Memory" in Table 5 (BoxE plus drift regularisation) already achieves 0.393 / 0.316 / 0.323 MRR on ENTITY / RELATION / HYBRID. That's better than ERPP (0.386 / 0.303 / 0.316) and a lot better than all of the rerun baselines. The memory adds 0.002 to 0.018 MRR. On retention the picture is mixed. Under the margin loss used for the main results the memory actually hurts RTE on RELATION, HYBRID and GraphHigher (Tables 11 and 12). All of the rerun baselines use TransE so Tables 2 to 4 are pretty much just demonstrating that BoxE beats TransE. The conclusion states that memory reuse "consistently contributes to forgetting mitigation" and Table 11 doesn't show that.

  2. The hyperparameters look like they were tuned on the test sets, and tuned toward the headline claim. The selected rows in Tables 7 to 9 match the main-table averages exactly (average MRR 0.3349). The text picks each value for its retention behavior; for γ\gamma it says so explicitly ("the fewest non-positive retention metric values"). Margin loss was chosen because it keeps RTE positive on all seven datasets. With γ=3\gamma = 3, any other λevo\lambda_{evo}, d in {100, 300, 900}, or the negative-sampling loss, at least one dataset goes non-positive. So the selection guaranteed "positive RTE on all seven benchmarks"; the experiments didn't find it. Unless I've misread the setup, the selection has to be redone on validation data.

  3. I'm not convinced the memory does what the paper describes. Past residuals are frozen, so Eq. 4 reduces to a fixed offset (the sum of past residuals) plus one fresh trainable residual, both divided by ∣Ves∣|V^s_e|. Nothing is actually retrieved. Because the denominator grows, every old entity's residual also shrinks by a factor of ∣Ves∣/(∣Ves∣+1)|V^s_e|/(|V^s_e|+1) as soon as a new snapshot starts, before any training. What an entity learned at snapshot 1 ends up weighted 1/5 by snapshot 5.

If training at snapshot s only sees ΔT(s)\Delta T(s), as Appendix A says, an entity missing from the new triples has nothing pulling it back. LevoentL^{ent}_{evo} can't see this drift either, since it only penalizes pep_e and beb_e. That leaves two possibilities. If the residuals are tiny compared with beb_e, they can't hold much, which would fit the small gains. If they aren't tiny, the method perturbs every old entity at every boundary. Appendix D, where attention does worse than the plain mean on most datasets, makes me lean toward the first.

The "w/o Freeze" ablation doesn't settle this. Eq. 8 doesn't regularize the residuals, so unfreezing them gives each entity several unregularized vectors that feed straight into its bump. Of course RTE collapses. That shows unregularized parameters drift. It doesn't show that freezing snapshot memories is a good idea. The useful control would be unfrozen residuals under the same λevo\lambda_{evo} penalty.

  1. The RTE comparison has problems. "Highest RTE on all seven" (Section 4.3) holds only against the rerun TransE baselines. ERPP and STCKGE have no RTE numbers, and ERPP is the strongest baseline on ROD. The claim also fails within the paper. TransE + Memory with margin loss beats TRM's RTE on five datasets (0.165 against 0.018 on GraphHigher, for instance), at an MRR of only about 0.16 to 0.21.

Separately, BAKE's ROD and RTE are exactly 0 in all 14 cells of Table 4. I find that hard to believe when the candidate set grows at every snapshot, and some of those zeros are even underlined as second best.

  1. Table 6 employs negative sampling loss for both TransE and DistMult and Appendix E states that this loss was selected because "it showed the most consistent effect of residual memory reuse". With margin loss DistMult + Memory actually performs worse on RTE (i.e. it loses RTE) on 4 of the datasets. The TransE base in Table 6 is also quite poor (RTE -0.491 on RELATION) and even with memory its RTE remains below that of LKGE on all datasets.

  2. It's a smaller point but it's to do with the motivation. The idea that relations are more stable than entities is from Yang et al. (2025) and is not actually tested in any of the experiments here (the w/o Memory checkpoints would have shown it directly). It's not really tested in the appendix either. Entity memory has about 30x more parameters than relation memory (14,541 entities versus 237 relations x 2 roles) and the relation version only moves box centers.

Questions for authors
  1. How large is ρes\rho^s_e relative to beb_e at the end of each snapshot? And how much does MRR on the old test sets move between the end of snapshot s and the start of s+1, before any training? Both are free to compute from your checkpoints.

  2. How does "highest RTE on all seven benchmarks" fit with the TransE + Memory rows in Tables 11 and 12?

  3. Were γ\gamma, λevo\lambda_{evo}, d and the loss selected on validation or test data? If on test data, does the all-positive RTE result survive selection on validation?

3 comments

Thank you for recognizing the simplicity of the memory mechanism, its applicability to TransE and DistMult, and the value of reporting settings in which memory was less effective. We also appreciate your observation about the strength of plain DistMult, which connects directly to the role of the scoring backbone in our study.

Weakness 3. I'm not convinced the memory does what the paper describes. Past residuals are frozen, so Eq. 4 reduces to a fixed offset (the sum of past residuals) plus one fresh trainable residual, both divided by ... The useful control would be unfrozen residuals under the same penalty.

To assess whether freezing snapshot memory is preferable, we also evaluated the alternative you suggested, allowing past residuals to update while regularizing their changes.

SettingMean MRRMean RTEMean RODRTE > 0ROD > 0
Regularized updates to past residuals0.332920+0.011916+0.0080276/73/7
TRM0.334857+0.016286+0.0145207/76/7

We penalized changes in each past residual relative to its value at the previous snapshot, using a coefficient of 0.01. We weighted all seven datasets equally when computing the means.

The regularized variant achieved positive RTE on six of seven datasets, showing that allowing past memories to update under regularization can still preserve past performance. TRM nevertheless achieved higher mean MRR, RTE, and ROD. This comparison supports keeping past residuals at the values learned when they were created and letting the shared base bump and the current residual adapt to subsequent changes.

Weakness 4. The RTE comparison has problems. "Highest RTE on all seven" (Section 4.3) holds only against the rerun TransE baselines. ERPP and STCKGE have no RTE numbers, and ERPP is the strongest baseline on ROD. The claim also fails within the paper. TransE + Memory with margin loss beats TRM's RTE on five datasets (0.165 against 0.018 on GraphHigher, for instance), at an MRR of only about 0.16 to 0.21.

Separately, BAKE’s ROD and RTE are exactly 0 in all 14 cells of Table 4. I find that hard to believe when the candidate set grows at every snapshot, and some of those zeros are even underlined as second best.

The statement in Section 4.3 about the highest RTE on all seven benchmarks refers to the comparisons in Table 4. We could not measure RTE for ERPP or STCKGE because their code was unavailable, so those entries are marked with dashes. We took ERPP's ROD values from its original paper and discuss in the same section that ERPP achieves higher ROD than TRM on FACT and GraphHigher. We obtained BAKE's ROD and RTE by running its public implementation. The zeros in the table are measured values, while the dashes indicate metrics we were unable to compute.

Weakness 5. Table 6 employs negative sampling loss for both TransE and DistMult and Appendix E states that this loss was selected because "it showed the most consistent effect of residual memory reuse". With margin loss DistMult + Memory actually performs worse on RTE (i.e. it loses RTE) on 4 of the datasets. The TransE base in Table 6 is also quite poor (RTE -0.491 on RELATION) and even with memory its RTE remains below that of LKGE on all datasets.

The purpose of Table 6 was to measure what changes when residual memory is added with the backbone and loss held fixed. As you point out, with negative sampling loss, TransE + Memory still had negative RTE and remained below LKGE on all seven benchmarks. Within that same TransE configuration, adding memory improved RTE on every dataset. This is the effect we wanted to show in this comparison. Residual memory can help preserve past knowledge with backbones beyond BoxE.

Weakness 6. It's a smaller point but it's to do with the motivation. The idea that relations are more stable than entities is from Yang et al. (2025) and is not actually tested in any of the experiments here (the w/o Memory checkpoints would have shown it directly). It's not really tested in the appendix either. Entity memory has about 30x more parameters than relation memory (14,541 entities versus 237 relations x 2 roles) and the relation version only moves box centers.

The relative stability of relations is a design hypothesis we drew from Yang et al. (2025). As you point out, there are 14,541 entities and 237 relation types in the final snapshot. Because many entity pairs share the same relation, its meaning can remain stable even when a particular entity gains new neighbors or facts. If we encode a change in that entity's interactions in the relation parameters, the change also affects the scores of other entity pairs that share the relation. This is why we chose to capture such changes in entity-specific residuals.

Weakness 1. My main concern is that the gains come from BoxE and not from the memory. The ablation in the paper is what convinced me. ... The conclusion states that memory reuse "consistently contributes to forgetting mitigation" and Table 11 doesn't show that.

We agree that the contributions of the backbone and the memory should be distinguished. The strength of the w/o Memory setting goes to the question that motivated this work. We wanted to understand how the way a model represents entity positions and interactions affects retention as a knowledge graph grows, and what reusing earlier states of those representations can add.

  1. Much of the high MRR on ENTITY, RELATION, and HYBRID comes from BoxE with drift regularization across snapshots. We consider this an important finding for CKGE in its own right. Comparing continual learning methods on the same TransE backbone reveals differences among those methods, while leaving open how much the choice of scoring function matters. Your observation about DistMult illustrates this point. On ENTITY, DistMult without memory outperformed the TransE-based methods we reran. This suggests that the choice of backbone deserves attention alongside the design of the continual learning method.

    We chose BoxE because it separates an entity's own position, represented by its point, from the correction it applies to a counterpart entity's position, represented by its bump. We discuss this choice in more detail in our response to Reviewer yL8s's first weakness. The comparatively stable performance of BoxE across backbones is consistent with this intuition. Within this architecture, the question we wanted to address was how to preserve and reuse previously learned interaction corrections while learning new facts. We freeze past residuals at their learned values and let the current residual capture changes associated with new facts. Both past and current residuals are used in the same bump computation when scoring a prediction.

  2. With margin loss, adding memory lowered RTE on RELATION, HYBRID, and GraphHigher. RTE averages both improvements and declines relative to each snapshot's initial MRR. To see whether the lower RTE reflects more forgetting, we computed the gains above the initial MRR and the drops below it separately.

    We use the relative change δi(s)=(ms,i−mi,i)/mi,i\delta_i(s)=(m_{s,i}-m_{i,i})/m_{i,i} from Eq. (1) and the test-set weight wiw_i from Eq. (2). Here, ii is the snapshot to which the test set belongs, and ss is the snapshot through which the model has been trained. We define the weighted gain and weighted drop as follows.

    \sum_{s=i+1}^{S} \max(\delta_i(s),0).$$ $$\mathrm{RTE}^{-} = \sum_{i=1}^{S-1} w_i \frac{1}{S-i} \sum_{s=i+1}^{S} \max(-\delta_i(s),0).$$ $$\mathrm{RTE}=\mathrm{RTE}^{+}-\mathrm{RTE}^{-}.$$ Changes above a snapshot's initial MRR contribute to the gain, and the magnitudes of changes below it contribute to the drop. For each snapshot, we average both components over all $S-i$ subsequent evaluations and then weight them by test-set size. RTE is therefore the weighted gain minus the weighted drop. | Benchmark | Weighted gain (RTE⁺) | Weighted drop (RTE⁻) | RTE | | ----------- | -------------------- | -------------------- | --------------- | | RELATION | 2.620% → 1.922% | 0.355% → 0.000% | 2.265% → 1.922% | | HYBRID | 1.758% → 0.922% | 0.902% → 0.573% | 0.855% → 0.349% | | GraphHigher | 3.009% → 2.008% | 0.000% → 0.183% | 3.009% → 1.825% | Each cell shows BoxE without memory → TRM. On RELATION and HYBRID, adding memory reduced the weighted drop. RTE was lower because the weighted gain decreased by more than the weighted drop did. On RELATION, TRM matched or exceeded each snapshot's initial MRR in every subsequent evaluation. On HYBRID, TRM fell below the initial MRR on the second snapshot's test set in a subsequent evaluation, but it reduced the drop on the fourth snapshot, whose test set was about five times as large. This reduced the overall weighted drop. On GraphHigher, the weighted gain decreased and the weighted drop increased, with MRR falling below its initial value on the third snapshot's test set. We propose reporting RTE⁺ and RTE⁻ alongside RTE. When RTE decreases, these two components show whether the gains above initial performance became smaller or the drops below it became larger. In the same margin-loss setting, adding memory changed RTE from negative to positive on FACT and GraphLower. With negative sampling loss, adding memory to the same BoxE backbone improved both RTE and ROD on all seven benchmarks. The point we intended to make with Tables 11 and 12 is that residual memory can mitigate forgetting of knowledge learned in earlier snapshots even when the BoxE backbone is held fixed.

Question 1. How large is ρes\rho_e^s relative to beb_e at the end of each snapshot? And how much does MRR on the old test sets move between the end of snapshot s and the start of s+1, before any training? Both are free to compute from your checkpoints.

To compare the magnitudes, we selected one entity in the ENTITY scenario that appears in the training sets of all five snapshots. We computed the L2 norms using the best checkpoint from each training stage.

SnapshotBase bump L2 normAggregated residual L2 normResidual-to-bump norm ratio
15.0528964.98846398.72%
25.0764223.37518066.49%
35.0841323.78611474.47%
45.0930003.14550661.76%
55.1094202.60882851.06%

We also measured how MRR changed at each snapshot transition in the ENTITY scenario. For each transition, we evaluated the test set of the snapshot just completed, once at the end of training and again immediately after adding the new residual slot, before any training on the next snapshot.

Snapshot transitionEnd-of-training MRRMRR before training on the next snapshot
1 → 20.4150240.403595
2 → 30.3842140.365727
3 → 40.3959370.381400
4 → 50.3941780.378614

As you anticipated, MRR on the preceding snapshot's test set decreased at every transition immediately after the new residual slot was added, before any further training.

Question 2. How does "highest RTE on all seven benchmarks" fit with the TransE + Memory rows in Tables 11 and 12?

The statement in Section 4.3 about the highest RTE on all seven benchmarks refers to the comparisons in Table 4. As discussed in our response to Weakness 5, the experiments in Tables 11 and 12 were designed to measure the effect of adding residual memory while holding the backbone and loss fixed.

Weakness 2. The hyperparameters look like they were tuned on the test sets, and tuned toward the headline claim. ... The experiments didn't find it. Unless I've misread the setup, the selection has to be redone on validation data.
Question 3. Were γ\gamma, λevo\lambda_{evo}, d and the loss selected on validation or test data? If on test data, does the all-positive RTE result survive selection on validation?

We selected TRM's embedding dimension, margin, drift regularization weight λevo\lambda_{\mathrm{evo}}, and loss function using validation data. Within each training run, we also used validation MRR to select the checkpoint. The default rows in Tables 7 to 9 match the averages in the main text because both report results for the same configuration.

Comments on the paper

3 comments

Thank you for recognizing the motivation for separating relation-side geometry from entity-side changes, and for noting the breadth of our evaluation across seven benchmark scenarios. We appreciate your questions about the practical cost of TRM and the interpretation of its retention results.

Weakness 1. Computational efficiency: the candidate-wise recomposition induced by the cross-bump structure, compounded by per-snapshot residual aggregation, can be expensive. Quantitative runtime comparisons to strong baselines are limited.

As noted in Section 4.1, the official implementations of ERPP and STCKGE were unavailable when we ran our experiments, so we could not measure their training and inference times under the same conditions. Neither original paper reported training times, so we omitted both methods from the comparison in Figure 2.

Weakness 2. Memory footprint: storing a residual per snapshot per entity can be substantial for long horizons. The paper does not quantify memory overhead or propose sparsity/compaction strategies.

Our current implementation stores a separate residual for each entity at each snapshot. With five snapshots, 14,541 entities, and an embedding dimension of 700, the residual weights alone require approximately 194 MiB in FP32. In a comparison using the same BoxE backbone, mean peak GPU memory increased from 5,837 MiB to 7,567 MiB, an increase of about 1,700 MiB. As you point out, retaining more snapshots increases storage even when the number of entities stays fixed.

Because the default model uses uniform averaging, it can recover the same average by keeping a running sum and count of past residuals for each entity and incorporating the current residual into both. This would reduce the cost of storing past residuals individually.

Weakness 3. Aggregation design: uniform averaging across residuals may underutilize temporal signals (recency, reliability) and task structure. While Appendix D is referenced, main-text results focus on a simple scheme.

Table 10 in Appendix D compares five alternatives that use relation information within the same memory framework. We compared a single query combining relation information from both roles with role-specific queries, learnable snapshot keys with keys derived from the memories themselves, and cross-role with same-role mappings.

None of these variants explicitly gives more weight to recently learned information. In the benchmarks we use, data learned later is not necessarily more recent in the sense of a temporal knowledge graph. With additional benchmarks that capture recency, we could investigate aggregation schemes that account for how recent the information is.

Weakness 6. The figure references a “Both Mask” in the aggregator without explanation in the main text. Some datasets and results (e.g., STCKGE/ERPP from original papers) are reported without harmonized settings. Clearer notes on potential discrepancies would help interpretability.

The figure's birth mask determines which residuals are valid based on the snapshot in which an entity first appears. The main text did not explain this definition in enough detail. We will clarify it in the revision.

Question 1. How large is the memory footprint (parameters and storage) of snapshot-wise residuals as the number of snapshots grows, and how does it compare to baselines in practice? Any pruning/sparsification experiments (e.g., top-k residuals, low-rank parameterizations)?

To quantify the additional cost of TRM, we compare the time and space complexity of the current implementation with that of w/o Memory.

Let ∣ΔTi∣|\Delta\mathcal{T}_i| be the number of training triples introduced at snapshot ii, ∣Ei∣|\mathcal{E}_i| and ∣Ri∣|\mathcal{R}_i| the numbers of entities and relations, KK the number of negative samples, dd the embedding dimension, and HiH_i the maximum number of residual memories aggregated for an entity. The per-epoch training complexity of w/o Memory is

O((K+1)∣ΔTi∣d).O\left((K+1)|\Delta\mathcal{T}_i|d\right).

TRM also aggregates up to HiH_i residual memories per entity. The worst-case per-epoch training complexity of the current implementation is therefore

O((K+1)∣ΔTi∣Hid).O\left((K+1)|\Delta\mathcal{T}_i|H_i d\right).

For exhaustive ranking, let QiQ_i be the number of evaluation queries. The inference complexities of w/o Memory and TRM, respectively, are

\quad\text{and}\quad O\left(Q_i|\mathcal{E}_i|H_i d\right).$$ The space complexity of w/o Memory is $$O\left((|\mathcal{E}_i|+|\mathcal{R}_i|)d\right).$$ TRM requires an additional $O\left(H_i|\mathcal{E}_i|d\right)$ of storage for snapshot-wise entity residuals, giving an overall space complexity of $$O\left((|\mathcal{E}_i|+|\mathcal{R}_i|+H_i|\mathcal{E}_i|)d\right).$$ In the current uncached implementation, the worst-case cost depends linearly on the number of entities and on the number of snapshot-wise residuals retained per entity, holding the other quantities fixed. As discussed in our response to Weakness 2, uniform aggregation is query-independent, so pre-aggregating or caching the residuals can reduce repeated computation in practice. > **Question 2.** Can you provide wall-clock training and inference throughput comparisons (per snapshot and per evaluation) versus strong baselines (e.g., ERPP, STCKGE, SAGE) to quantify the cost of candidate-wise recomposition? The official implementations of ERPP and STCKGE were unavailable, which are strong baselines, so we could not provide wall-clock training or inference comparisons for those two baselines. > **Question 3.** You aggregate residual memories uniformly. Did you evaluate recency- or performance-weighted aggregation or attention mechanisms on the main benchmarks (beyond Appendix D), and do they improve the trade-off between MRR and RTE? As discussed in our response to Weakness 3, our aggregation experiments were limited to the variants in Appendix D. We have yet to evaluate the additional weighting schemes you suggest and will consider them in future work. > **Weakness 5.** No direct breakdown of forgetting into representation drift vs. entity interference components. Although the evaluation candidate set $E_{\le s}$ suggests the protocol accounts for interference, an explicit analysis would strengthen claims. > **Question 4.** Given that evaluation uses the current candidate set $E_{\le s}$, can you decompose forgetting into (i) drift of previously seen entity/relation parameters vs. (ii) entity interference from newly added entities, as in recent analyses? Where does TRM gain most? We understand the first component, drift in previously seen entity and relation parameters, as what our current evaluation on earlier snapshots' test sets is intended to capture. TRM shows gains under this evaluation. To assess interference from newly added entities, we would need to re-evaluate the earlier test sets after training on later snapshots and include the newly added entities among the candidates. This would change the candidate set relative to the original evaluation, when the model had only been trained through the snapshot to which the test set belongs. The two scores would therefore be obtained under different evaluation conditions. Could you point us to the recent analyses you have in mind?

Weakness 4. Some baseline retention metrics are missing (marked with a dash), and BAKE’s ROD/RTE are reported as zeros across the board, which likely reflects unavailability rather than true values. Including zeros in comparative tables risks misleading conclusions about retention.
Question 5. BAKE’s ROD/RTE are reported as zero for all datasets. Were these values unavailable (and thus set to zero), or truly measured as zero? If unavailable, could you remove them from aggregate retention comparisons or include partial metrics to avoid misleading interpretations?

We obtained the zeros by running BAKE's public implementation under our evaluation settings. In the public implementation, parameters learned in earlier snapshots were kept fixed when used in later snapshots. The reported zeros are the values measured for that version under those conditions.

Section 4.1 and Appendix C.1 distinguish methods we reran using public code from methods whose results we cite from their original papers because their code was unavailable. The notes to Tables 2 and 3 also identify the cited results. A dash indicates that we could not compute the metric because an official implementation was unavailable at the time of our experiments. In particular, without the official implementations of ERPP and STCKGE, we could not run them under the same protocol as the other reproduced baselines or compute their RTE. We cite the ranking and retention results available in the original papers, identify their source, and mark unavailable metrics with a dash.

Question 6. Beyond BoxE, your TransE/DistMult extensions still show negative RTE in several cases (though improved). What architectural properties of BoxE (e.g., point-bump decomposition) are most critical for achieving consistently positive RTE with TRM?

We offer a mechanistic explanation without a formal theoretical guarantee. In TRM, different residual components preserve past information and adapt to newly introduced knowledge.

In standard sequential learning, the loss on a new snapshot directly updates existing entity representations, so learning new facts can overwrite representations formed in earlier snapshots. TRM maintains a separate entity-side residual memory for each snapshot. When learning a new snapshot, we freeze all earlier residual memories and train the current snapshot's residual memory. The current residual can therefore capture changes associated with new facts while the earlier residual corrections retain their learned values. This is how frozen past residuals support stability and the current residual supports adaptation.

BoxE provides a natural place to apply this mechanism. It represents each entity with a point and a bump, and each relation with a head-role region and a tail-role region. An entity's bump modifies its counterpart's role-specific representation. By adding snapshot-wise residuals to the bump, TRM reuses earlier entity interactions in the current cross-bump scoring computation while preserving the relation-side scoring structure.

The scoring geometry also matters. In TransE, the translated head representation is trained to approach a specific tail point. In BoxE, the memory-augmented entity representation is trained for compatibility with a relation-specific region. When corrections from several snapshots are combined, a region gives the model more latitude to accommodate different entity-side changes than alignment to a single target point. We view this as one explanation for why BoxE is particularly well suited to snapshot-wise entity residuals. The explanation concerns how the geometry fits the memory mechanism, rather than the general ranking of backbones.

The ablations support this interpretation. w/o Memory retains BoxE's point-bump scoring and drift regularization and already achieves competitive performance. With the same BoxE backbone, full TRM achieves higher final MRR on all seven datasets and positive RTE on each of them. When we keep the residual memory trainable across snapshots in w/o Freeze, MRR is lower than for full TRM and RTE is negative on all seven datasets. These results indicate that stable retention relies on freezing past residuals and protecting them from updates during current training, in addition to introducing the residual vectors themselves.

The same principle also applies beyond BoxE. Creating snapshot-wise residuals, freezing past residuals, and incorporating them into the current entity representations improves ranking performance for TransE and DistMult and improves retention in most settings. The size of the benefit depends on the scoring geometry of the backbone.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.