- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how sensitivity to historical event-time assignment propagates through a trained TPP's prediction pathway. The authors introduce a same-target intervention audit: historical inter-event intervals are permuted among event positions while preserving event identities and order, the interval multiset, total duration, and the (log-transformed) future target. Sensitivity is measured at four stages: raw-history predictability (S_raw), representation access via a standardized 65-parameter linear probe (S_access), native distributional response (S_NLL), and decoded point-MAE response (S_MAE), plus a prediction-drift measure D_point. Experiments cover five architectures (THP, NHP, SAHP, AttNHP, S2P2) on three retweet-cascade datasets (Retweet, Cyberbullying, YouTube), with a MIMIC-IV cross-domain check. Main findings: (i) SAHP/AttNHP show strong access (0.91/0.83) and positive native-NLL responses yet near-zero or negative operational point-MAE effects (-0.010/-0.017), while THP, NHP, and S2P2 retain positive effects at all stages; (ii) re-decoding the same fitted Exponential rates with the conditional median or mean reverses the Retweet endpoint (+0.35/+0.86), but the median stays negative on Cyberbullying/YouTube, so readout dependence is model- and dataset-dependent; (iii) a THP/AttNHP × Gaussian/Exponential factorial associates the endpoint contrast with the joint native pathway; (iv) the controls (graded reassignment, same-mark reassignment, frozen-representation and distillation controls, per-example cancellation analysis) support the stage-wise interpretation.
Critical strengths
- A well-controlled, multi-stage audit design that cleanly separates representation access, distributional response, and point-error response. This matters because the paper's central negative result (that MAE-based endpoints can report "no temporal reliance" while representations and likelihoods remain sensitive) is only credible under exactly this kind of paired, same-target, multi-stage protocol with matched decoders. The fixed-rate re-decoding experiment (Table 2) is the decisive piece of evidence, and it is done properly.
- A practically important warning for TPP evaluation. The paper shows that the widely used SAHP/AttNHP operational readouts produce near-constant point predictions, and that switching to the conditional mean or median of the same fitted rates changes both accuracy (2.47 to 1.04 clean MAE) and measured sensitivity. This is a concrete, actionable finding for anyone benchmarking TPPs on point-error metrics.
Critical weaknesses
- The contribution is an audit of existing models, and part of the headline phenomenon admits a more mundane explanation. Once it is documented that the SAHP/AttNHP operational decoder outputs nearly constant predictions (std ratio 0.18) with 2.4x worse clean MAE than THP, "no MAE response to input perturbation" is close to a tautology for that decoder. The paper's own decoder-swap analysis shows the interesting structure lives upstream; the framing would be stronger if it led with the calibration/pathology finding rather than the stage-dissociation framing.
- Limited scope of conclusions. The main evidence rests on three same-family cascade datasets plus one clinical construction; the cross-dataset inconsistency of the conditional-median result is reported but not explained (calibration vs. misspecification vs. dynamics is left open). This caps how much the paper can teach beyond the specific models and datasets audited.
Questions for authors
- For SAHP/AttNHP under the operational readout, what fraction of the near-zero S_MAE is attributable to the near-constant prediction range vs. genuine sign cancellation? The per-example analysis (Appendix Table 14) seems to bear on this; can you decompose it quantitatively?
- The operational decoder log(1+1/lambda) is neither the mean nor the median of the fitted Exponential in y-space. Is there any training-objective sense in which it is optimal, or is it simply a heuristic inherited from prior codebases? This affects whether the finding is "models are insensitive" or "a common decoder is miscalibrated."
- Can you add the prospective conditional-median decoder for NHP (noted as not evaluable) or otherwise complete the decoder-family grid, so that the cross-model readout comparison is symmetric?
- A brief discussion of whether the audit's S quantities relate to proper scoring rules (e.g., is S_MAE under the conditional mean the "right" pairing for MSE-optimal readouts?) would connect the framework to forecast-evaluation theory more tightly.