acceptodds
Under review as a conference paper at ICLR 2027

SurF: Learning A Time-Rescaling Map for Event-Stream Forecasting Across Datasets

Abstract

Irregularly sampled multivariate event streams are hard to model with forecasters built for regularly sampled series: inter-event intervals span orders of magnitude, and time units differ from one corpus to the next. We propose Survival Flows or SurF, an event forecasting model built on the fact that the Time Rescaling Theorem (TRT) maps every point process to the same i.i.d. unit-rate exponential target. SurF learns the cumulative intensity that realizes this map; its output is unit-free comprising a single set of weights, with no per-dataset parameters, can be learned across heterogeneous event-stream datasets. We build a Transformer encoder to represent three monotone parameterizations of the cumulative intensity function that require no Monte Carlo integration to evaluate the likelihood (two closed-form, one a deterministic per-interval quadrature rule) enabling fast and accurate gradient estimation to learn the map. On six real-world benchmarks, SurF attains the best next-event time RMSE on and closely tails the best on Taxi. Under a leave-one-out protocol, in which the target corpus is excluded from training and model selection, SurF-GLQ attains lower time RMSE than these baselines trained the same way on datasets and lower held-out NLL than DTPP on all six; on Amazon, Earthquake and Taobao it even matches or beats the baselines trained on the target corpus.

Primary area:
learning on time series and dynamical systems
Keywords:
time rescaling theorem, event sequence forecasting, irregularly sampled time series, neural point processes, multi-corpus pretraining, zero-shot transfer, transformer

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ra31OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SurF, a model for forecasting irregularly sampled event streams. It is built on the Time Rescaling Theorem (TRT), which maps every point process to the same i.i.d. unit-rate exponential target. SurF parameterizes the cumulative intensity directly and uses a shared Transformer encoder to learn this transformation across multiple event-stream datasets. It introduces three variants: SurF-MoE and SurF-CSB, which provide closed-form cumulative intensities, and SurF-GLQ, which uses deterministic Gauss–Legendre quadrature. This design avoids the stochastic Monte Carlo integration commonly used by neural temporal point processes (TPPs). Experiments on six real-world datasets evaluate next-event prediction, multi-step forecasting, and leave-one-dataset-out transfer. The results are promising: SurF performs competitively with specialized TPP models and achieves strong zero-shot event-time forecasting results when the target dataset is excluded from training.

Critical strengths
  1. The idea is quite interesting and reasonable to use the Time Rescaling Theorem (TRT), which transforms event inter-arrival times into a common unit-rate exponential distribution when the conditional intensity is correctly specified.
  2. This paper explores whether a shared model can learn temporal patterns that transfer across datasets with different time scales and event dynamics. The leave-one-out evaluation is particularly useful.
  3. The experiments are convincing, and the evaluations cover different aspects.
Critical weaknesses
  1. More analysis should be provided to discuss the transferability of datasets. This could reveal whether cross-dataset gains arise from broadly shared dynamics or from a small number of particularly helpful source datasets.
  2. The intensity-floor assumption may restrict applicability to some real-world processes. The paper's practical claims would be more convincing if the model were evaluated on datasets or controlled synthetic processes with long inactive intervals.
  3. The efficiency comparison should make clear which costs come from the cumulative-intensity parameterization, the Transformer encoder, and the numerical inversion process.
Questions for authors
  1. Can the authors provide evidence separating the effect of the shared exponential target from that of the shared Transformer representation?
  2. Are the proposed masking or learnable-floor schemes evaluated on processes with long inactive intervals?
3 comments

We thank the reviewer for finding the idea interesting, the leave-one-out evaluation useful, and the experiments convincing. We address each concern below with new experiments.

W1: "More analysis should be provided to discuss the transferability of datasets. This could reveal whether cross-dataset gains arise from broadly shared dynamics or from a small number of particularly helpful source datasets."
They come from dynamics shared across the pool, not from one or two sources. We tested this with the CSB head, measuring held-out RMSE (deterministic rollout at z = 1) after training on each corpus alone (3 seeds) and, for every target, after removing each source corpus in turn (30 runs).

  • Pooling helps: compared with the median single source, training on all five sources lowers held-out RMSE by 4–62% on 5/6 targets and lowers NLL on 6/6.
  • No source dominates: removing any single source raises held-out RMSE (table below). Even the least useful source, Retweet, costs 15–36% when removed, so transfer is spread across the pool.
  • Transfer crosses time scales: removing StackOverflow (hours to days) raises Taxi's RMSE (minutes to hours) by 58%, as a unit-free Exp(1) target should allow.
Source removedIncrease in held-out RMSE
StackOverflow67% (Amazon), 58% (Taxi), 32% (Earthquake)
Amazon43–60%
Taobao38–56%
Earthquake26–40%
Taxi20–38%
Retweet15–36%

Revision to the paper: We will add the full single-source and leave-one-source-out analysis to the appendix.

W2 / Q2: "The intensity-floor assumption may restrict applicability to some real-world processes. [...] learnable-floor schemes evaluated on processes with long inactive intervals?"
This is a fair concern. The floor is part of the model, not an assumption about the data, so it does not require the data to have no inactive periods. It makes Λθ\Lambda_\theta a bijection that can be inverted for sampling (Theorem 2, applied to the model); without it, MoE's Λ\Lambda is bounded and cannot be inverted for large z (Remark, App. A). Its cost is also bounded: App. A.1 bounds its likelihood bias by T⋅λfloorT \cdot \lambda_{\mathrm{floor}}, so a floor of 10−410^{-4} represents long zero-intensity stretches at a cost of at most 10−410^{-4} nats per unit time.

To test this empirically, we compared no, fixed (10⁻⁴) and learnable floors on four processes with exact oracle likelihoods:

  • Refractory: a 5-unit dead time after each event.
  • Quiet: bursts followed by silences of at least 50 units.
  • Closure: zero intensity for 16 of every 24 hours.
  • Hawkes: baseline rate 10⁻³, with gaps up to about 10,700 units.

The table shows excess NLL over the oracle, in nats (mean of 2 seeds; GLQ scored with the exact integral).

Head, floorRefractoryQuietClosureHawkes
MoE, none / fixed / learnable1.830 / 1.830 / 1.8290.274 / 0.271 / 0.2720.177 / 0.179 / 0.1800.026 / 0.025 / 0.026
CSB, none / fixed / learnable0.141 / 0.135 / 0.1423.328 / 3.328 / 3.3260.343 / 0.346 / 0.3540.739 / 0.813 / 0.750
GLQ, none / fixed / learnable0.186 / 0.174 / 0.1700.294 / 0.290 / 0.2790.192 / 0.189 / 0.1930.235 / 0.319 / 0.342
  • The floor costs almost nothing. On Refractory, Quiet and Closure, the three floor settings differ by at most 0.016 nats within each head. The learnable floor ends between 8.8 × 10−510^{-5} and 1.1 × 10−410^{-4}, near its initial 10−410^{-4}, consistent with a floor whose likelihood cost is small.
  • On Hawkes, which has the longest gaps, MoE stays within 0.026 nats of the oracle with or without the floor. For CSB and GLQ, floored runs are 0.01–0.11 nats above unfloored ones, but the fixed and learnable floors, which end at nearly the same value, differ by a similar amount (CSB: 0.813 vs 0.750), so training variation alone produces differences of this size.
  • Where long gaps do hurt, head shape is the cause, not the floor. MoE's intensity can only decay between events, so it misses the rise after a dead time (Refractory). CSB's can only increase, so it misses the decay after a burst or excitation (Quiet, Hawkes); its 0.74 -- 0.81 nats on Hawkes appears in every setting, including without the floor, and this shape restriction is already stated in our Limitations. GLQ, whose shape is unrestricted, stays within 0.19 nats of the oracle on Refractory and within 0.30 nats on Quiet and Closure in every floor setting, so long inactive intervals are within SurF's reach.

Revision to the paper: We will add this study, including the learnable floor, to App. A.1.

Continued in the next comment ....

W3: "The efficiency comparison should make clear which costs come from the cumulative-intensity parameterization, the Transformer encoder, and the numerical inversion process."
We profiled each component's share of time on Amazon with the paper's configurations.

Share of timeMoECSBGLQ
Train: encoder forward8%9%5%
Train: Λ head forward2%2%7%
Train: backward63%63%60%
Inference: encoder42%42%34%
Inference: Λ head6%6%<2%
Inference: inversion0.5%0.5%0.4%
  • The closed-form parameterization is almost free. The MoE and CSB heads are a single linear layer followed by closed-form operations (App. I.2); they take 2% of a training step and 6% of inference, and inversion takes 0.5% of inference. Of the three components, the encoder is the largest in both training (8–9%) and inference (42%). The parameterization therefore adds little to SurF-MoE's cost, which trains 10.9–12.4× faster than NJDTPP and about 1.4× faster than DTPP (Table 4).
  • GLQ's extra cost is quadrature: its head forward is 7% of a training step, and backpropagation runs through all Q nodes, a cost the closed-form variants avoid. At inference, its head and inversion together still take under 2.5%.
  • The remaining 26–28% of a training step, and 51.5% (MoE, CSB) and over 63% (GLQ) of an inference pass, is spent outside the encoder, Λ head and inversion.

Revision to the paper: We will add this breakdown and use it to replace the Newton-time estimate in App. E.1.

Q1: "Can the authors provide evidence separating the effect of the shared exponential target from that of the shared Transformer representation?"
In SurF, the Exp(1) target is not a separate loss term but the likelihood itself (App. B), so it cannot be switched off on its own. Instead, we varied one factor at a time with the others held fixed:

Factor variedHeld fixedChangeResult
Targetone corpus at a time (no pooling), encoder, constant-hazard headMSE regression → exponential likelihoodRMSE lower on 4/6 corpora in every seed: Taxi −15%, Amazon −12%, Retweet −3%, Earthquake −3%
Representation (W1)CSB head, likelihood, target corpus held out1 → 5 training corporavs median single source: held-out RMSE −4% to −62% on 5/6 targets; NLL lower on 6/6
Head (RU4Q, W2)target, encoder, multi-corpus trainingFullyNN- / IFTPP-style head → SurF headRMSE −17% to −76%, lower on 6/6 corpora for every SurF head
  • Target: changing only the target, with no pooling, lowers RMSE on 4/6 corpora in every seed, even though MSE trains directly on what RMSE measures.
  • Representation: changing only the pooling, with the target fixed, lowers held-out RMSE on 5/6 targets and NLL on 6/6.
  • Complementary: both factors help, and the unit-free target is what makes it possible to pool corpora whose time units range from seconds to days (Sec. 3.2).
  • Head: with both held fixed, swapping in a SurF head adds a third gain, lowering RMSE by 17–76% on all six corpora.

Revision to the paper: We will add these ablations to Sec. 4.4.

Thanks for the detailed additional experiments, which address most of my concerns.

Reviewer_bG5KOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SurF models event times through a history-conditioned cumulative intensity, using the time-rescaling theorem to relate inter-event intervals to exponential noise. A Transformer encoder supports three parameterizations: a mixture of decaying exponentials, a cumulative softplus basis, and a positive intensity network integrated by Gauss–Legendre quadrature. The model is trained across multiple event-stream datasets and evaluated under joint training, dataset-specific fine-tuning, and leave-one-dataset-out transfer. The paper reports time prediction, mark prediction, likelihood, and computational results.

Critical strengths
  1. Cross-dataset event forecasting is a worthwhile extension of the usual per-dataset evaluation. The leave-one-out experiments exclude the target corpus from both training and model selection, and include baselines trained under the same protocol. SurF-GLQ’s lower time RMSE on five of six held-out datasets and lower time NLL than DTPP on all six provide substantive evidence of transfer.

  2. The three parameterizations expose a useful trade-off between computational cost and intensity flexibility. The closed-form variants are supported by runtime measurements, while the more flexible GLQ variant performs well in transfer. Reporting likelihood and time-rescaling diagnostics alongside RMSE is also valuable for a model intended to represent a predictive distribution.

Critical weaknesses
  1. An important part of the leave-one-out input protocol is missing. Appendix G.1 assigns disjoint global IDs to each corpus’s event types, while Appendix I.1 uses learned type embeddings in the history encoder. Omitting zero-shot mark accuracy does not explain how unseen marks in the observed history are encoded, or how marks are handled during multi-step rollout. The definition and availability of the normalization factor TT in the GLQ head also need clarification. These details directly affect the central transfer results.

  2. The numerical claims for GLQ are stronger than the supplied justification. Section 3.1 states that the cumulative-intensity derivative identity holds exactly for every variant. For a finite quadrature approximation with nodes rescaled by the interval length, however, differentiating that approximation does not generally reproduce the endpoint intensity exactly. Appendix D bounds quadrature error, but smoothness alone does not establish that eight nodes make the error negligible for learned networks and long intervals. This affects the stated monotonicity, sampling, and likelihood guarantees for the strongest transfer variant.

Questions for authors
  1. How are held-out event types encoded during leave-one-out evaluation, and what marks are fed back during multi-step generation? Do all compared methods use the same treatment?
  2. What exactly is TT in the input Δt/T\Delta t/T, and can it be computed entirely from information available at each prediction origin?
  3. For trained GLQ models, how large are the cumulative-intensity and gradient errors at (Q=8) relative to a high-accuracy reference? Does the numerically evaluated cumulative remain monotone over the intervals used for sampling?
3 comments

We thank the reviewer for the detailed review. We address both points below with new measurements.

W1 / Q1: "Appendix G.1 assigns disjoint global IDs to each corpus's event types, while Appendix I.1 uses learned type embeddings [...]. How are held-out event types encoded during leave-one-out evaluation, and what marks are fed back during multi-step generation? Do all compared methods use the same treatment?"
This is a gap in the description, not in the experiments, and it does not affect the validity of the transfer results: no held-out mark reaches any model, and all methods receive identical inputs.

  • Training: on the source corpora, each sequence's types are replaced by a reserved null token with probability 0.5 (sequence-level type dropout), so the model learns to forecast times without marks.
  • Test: every held-out type is the null token, both in the observed history and in every generated mark fed back during rollout, so no mark information is fed back.
  • Baselines: we implemented leave-one-out training for AttNHP, THP and DTPP with the same protocol: same source corpora, same type dropout, null token at test, target excluded from training and model selection, and rollout with each model's own sampler and the null token fed back.

Revision to the paper: We will add this protocol to the App. G.1.

Q2: "What exactly is T in the input [Δt/T; h], and can it be computed entirely from information available at each prediction origin?"
T is a single fixed normalization constant, shared across every corpus and sequence, including the held-out one. It is therefore available at every prediction origin and carries no target information. The first linear layer of fθf_\theta sees (W/T) Δt(W/T)\,\Delta t, so TT could be absorbed into WW; it exists only to keep the input well-conditioned.

Revision to the paper: We will rename it T0T_0 and state its value in App. I.2.

W2: "The numerical claims for GLQ are stronger than the supplied justification. Section 3.1 states that the cumulative-intensity derivative identity holds exactly for every variant. For a finite quadrature approximation [...], differentiating that approximation does not generally reproduce the endpoint intensity exactly. [...] This affects the stated monotonicity, sampling, and likelihood guarantees [...]."
We agree that Section 3.1 attaches to the QQ-point rule properties that belong to the exact integral. The guarantees hold for the model GLQ defines, and the QQ-point rule is how that model's cumulative intensity is evaluated. GLQ defines λ†(τ)=λfloor+softplus⁡(fθ(τ)),Λ(τ)=∫0τλ†(s)ds.\lambda^{\dagger}(\tau) = \lambda_{\mathrm{floor}} + \operatorname{softplus}\big(f_\theta(\tau)\big), \qquad \Lambda(\tau) = \int_0^{\tau} \lambda^{\dagger}(s) ds. For this Λ\Lambda, ∂Λ∂τ=λ†\frac{\partial \Lambda}{\partial \tau} = \lambda^{\dagger} holds exactly, Λ\Lambda is strictly increasing with slope at least λfloor\lambda_{\mathrm{floor}}, and λ†(τ)exp⁡(−Λ(τ))\lambda^{\dagger}(\tau)\exp\big(-\Lambda(\tau)\big) is a valid density. The rule Λ(Q)\Lambda^{(Q)} evaluates Λ\Lambda numerically and satisfies these properties only approximately; Appendix~D already treats Λ(Q)\Lambda^{(Q)} as an approximation of Λ\Lambda with an error bound. The measurements under Q3 show that its errors do not dominate training or sampling.

Revision to the paper: We will state these properties for the model only and present Λ(Q)\Lambda^{(Q)} as its numerical evaluation throughout (Sec. 3.1; App. A, B, D, E).

Continued in next comment ....

Q3: "[...] smoothness alone does not establish that eight nodes make the error negligible for learned networks and long intervals. [...] For trained GLQ models, how large are the cumulative-intensity and gradient errors at Q = 8 relative to a high-accuracy reference? Does the numerically evaluated cumulative remain monotone over the intervals used for sampling?"
We agree that smoothness alone does not settle this, so we measured it on the leave-one-out GLQ checkpoints, against a composite Gauss–Legendre reference converged to 2×10−72 \times 10^{-7}:

  • Λ error: median relative error of Λ(Q)\Lambda^{(Q)} against the reference, over all held-out intervals and over the longest 1% of them.
  • Gradient error: ∥gQ−gref∥∥gref∥\dfrac{\lVert g_Q - g_{\mathrm{ref}} \rVert}{\lVert g_{\mathrm{ref}} \rVert} on the fold's source corpora, reported next to the minibatch gradient noise in the same units.
  • Sampler KS: KS distance from Exp(1) of the sampler's output on the held-out corpus, rescaled with the reference integral.
Held-outΛ error (median)Λ error (longest 1%)Gradient errorMinibatch noiseSampler KS vs Exp(1)
Amazon0.4%0.7%0.090.120.026
Retweet0.6%0.3%0.150.330.031
Taxi0.2%0.5%0.180.310.039
Earthquake0.3%0.8%0.090.130.028
StackOverflow0.4%0.8%0.120.400.040
Taobao0.6%1.1%0.130.450.034
  • Accuracy: the median Λ error is 0.2–0.6%. On the longest 1% of intervals it is higher on 5/6 corpora but stays at or below 1.1% everywhere, so eight nodes remain accurate to about 1% even on the longest intervals.
  • Training: on every fold, the quadrature gradient error is smaller than the minibatch noise, so quadrature is not the dominant source of gradient error.
  • Sampling: scored with the reference integral, the sampler's output lies within KS 0.026–0.040 of Exp(1) on all six held-out corpora.
  • Monotonicity: Λ(Q)\Lambda^{(Q)} is monotone over most of the sampling range and everywhere on the part that covers 95% of draws, but, as the reviewer anticipated, not everywhere: over the full range (draws up to z = 20), the largest local decrease is 0.04 in Exp(1) units, i.e. 4% of a mean rescaled gap. This does not affect sampling: the sampler is bracketed (App. E), so a local decrease cannot make it diverge, and the KS distances above confirm that its output follows the model.

Revision to the paper: We will add these measurements to Sec. 3.1 and App. D.

Just a quick follow-up before the discussion closes. We hope our replies on the held-out type handling and the GLQ quadrature errors answered your questions. If they did, we'd appreciate it if you could take another look at your score.

Reviewer_RU4QOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes SurF, a temporal point process model for forecasting across heterogeneous event-stream datasets. A shared Transformer encodes the event history and conditions a cumulative intensity function, which maps inter-event times to a common unit-rate exponential reference distribution. Future events are generated by inverting this map. The authors introduce three parameterizations: MoE and CSB have closed-form cumulative intensities, while GLQ uses a positive intensity network with deterministic Gauss–Legendre quadrature. Training uses the standard point process negative log-likelihood. The model is evaluated on six real-world datasets through joint training, per-dataset finetuning, and a leave-one-dataset-out protocol that tests zero-shot transfer to an unseen corpus.

Critical strengths

The paper explores whether one model can forecast event streams from different domains, extending the motivation behind time-series foundation models to temporal point processes. The leave-one-dataset-out evaluation is the strongest part of the work: the target corpus is excluded from both training and model selection. Under this protocol, SurF-GLQ achieves lower time RMSE than the baselines on five of six datasets and lower time NLL than DTPP on all six. These results support the practical value of learning across event-stream datasets.

For MoE and CSB, the construction explicitly enforces the boundary condition, monotonicity, and a positive intensity floor. This gives a clear basis for likelihood-based training and inverse-transform sampling. The different parameterizations also offer useful choices between intensity-shape flexibility and computational cost.

Critical weaknesses

The paper assigns disjoint type IDs to different corpora, while the history encoder uses type embeddings. It is therefore unclear how types in a held-out corpus are represented during inference. The same issue arises in multi-step forecasting, where generated types must be fed back into the history despite the target type vocabulary being unseen during training. Omitting zero-shot type accuracy does not resolve how these types affect time prediction. These details are necessary to interpret and reproduce the transfer results, including whether the baselines receive equivalent inputs.

FullyNN and IFTPP are included in the per-dataset experiments, but neither is evaluated with a matched history encoder and the same multi-corpus training protocol. The matched pretraining comparisons with AttNHP, THP, and DTPP are useful, but they leave open whether similar gains could be obtained by combining an existing cumulative-hazard or intensity-free model with the same Transformer and training procedure. Such a comparison would help establish what SurF adds beyond the shared encoder and multi-dataset pretraining.

Questions for authors

See Critical Weaknesses as is.

2 comments

We thank the reviewer for the careful review and for highlighting the leave-one-dataset-out evaluation.

W1: "The paper assigns disjoint type IDs to different corpora, while the history encoder uses type embeddings. It is therefore unclear how types in a held-out corpus are represented during inference. The same issue arises in multi-step forecasting [...]. These details are necessary [...], including whether the baselines receive equivalent inputs."
No held-out type ever reaches any model, SurF or baseline.

  • Training: on the five source corpora, all types in a sequence are replaced by a reserved null token with probability 0.5 (sequence-level type dropout). The model therefore learns to forecast times without marks.
  • Test: every held-out type becomes the null token, both in the observed history and in every mark fed back during multi-step rollout. Held-out time predictions therefore do not depend on the held-out type vocabulary, and the disjoint IDs of App. G.1 never has to represent an unseen type.
  • Baselines: AttNHP, THP and DTPP have no leave-one-out training recipe of their own, so we applied the same protocol to them: same source corpora, same type dropout in training, null token at test, target excluded from training and model selection, and autoregressive rollout with each model's own sampler and the null token fed back.

Evidence: under this matched setup, SurF-GLQ has lower time RMSE than all three baselines on 5/6 held-out corpora (Table 3) and lower time NLL than DTPP on all six, by at least 0.26 nats/event (App. J.1). On Amazon, Earthquake and Taobao, the held-out SurF-GLQ checkpoint uses no type information, yet it matches or beats AttNHP, THP and DTPP trained on the target corpus itself, with types (App. J.1).

Revision to the paper: We will add the full protocol (type dropout, null token at test and during rollout, identical treatment of all baselines) to App. G.1.

W2: "FullyNN and IFTPP [...] neither is evaluated with a matched history encoder and the same multi-corpus training protocol. [...] whether similar gains could be obtained by combining an existing cumulative-hazard or intensity-free model with the same Transformer and training procedure. Such a comparison would help establish what SurF adds beyond the shared encoder and multi-dataset pretraining."
We agree this is an important attribution question, so we ran this comparison. We held SurF's pipeline fixed (encoder, training corpora, objective, optimizer) and replaced only the time head, with either a FullyNN-style monotone cumulative-hazard network or an IFTPP-style log-normal mixture. Values are next-event time RMSE in the setting of Table 2 (lower is better).

Time headAmazonRetweetTaxiEarthquakeStackOverflowTaobao
SurF-MoE0.33616.230.3111.2581.0770.189
SurF-CSB0.40915.810.2881.2561.0390.224
SurF-GLQ0.40915.780.2861.2281.0180.126
FullyNN-style head, SurF pipeline0.62822.340.3752.3941.3810.535
IFTPP-style head, SurF pipeline0.62422.250.3822.6101.3740.533
FullyNN, per-dataset (Table 2)0.61521.920.3732.3301.3750.529
IFTPP, per-dataset (Table 2)0.61822.180.3772.5501.3740.531
  • Every SurF head beats both alternative heads on every corpus, with 17–76% lower RMSE.
  • In the same pipeline, the alternative heads gain nothing over their per-dataset versions: the IFTPP-style head is equal to or slightly worse than per-dataset IFTPP, and the FullyNN-style head is slightly worse than per-dataset FullyNN, on all six corpora.
  • SurF's heads, in contrast, do benefit from pooling: for the CSB head, training on five corpora instead of one lowers held-out RMSE on 5/6 targets (see ra31, W1).

The shared encoder and pooling therefore do not explain the gains on their own, and neither a generic monotone network nor a log-normal mixture reproduces them in the same pipeline; the gains come from SurF's heads (MoE, CSB, GLQ).

Revision to the paper: We will add this comparison to the App. G.2, with a statement in Sec. 5 of what SurF adds beyond the encoder and pooling.

Just a quick follow-up before the discussion closes. We hope our replies on the held-out types and the head-swap comparison addressed your concerns. If they did, we'd appreciate it if you could take another look at your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.