acceptodds
Under review as a conference paper at ICLR 2027

A Smooth Temporal Low-Rank Basis for Neural Estimation of Cumulative Longitudinal Means

Abstract

To characterize how longitudinal outcomes are driven by the covariate history, we consider modeling the conditional mean of the outcome at visit , , with nonlinear and unknown functions of the concurrent covariate vector , . When is additive elementwise across , the estimation is well studied in the setting of distributed-lag non-linear models (DLNM) at the standard spline rates. When the structural assumption of additivity on is not imposed, little work addresses the sample sizes typical of cohort studies, hundreds to a few thousand subjects. In particular, DLNM cannot capture interactions, which costs prediction accuracy that no increase in sample size recovers. On the other hand, sequence models, such as recurrent networks, Transformers and state-space models, specify an unrestricted function of the covariate history and impose no structure, but must learn the functional form from the data; and the direct neural encoding of the cumulative mean model, with a separate network for each visit, has an estimation variance that grows with the number of visits. We propose SmoothCum, a neural estimator that keeps the cumulative form of the model, shares one feature-extracting network across all visits, and expresses the visit-specific output layer through B-spline basis functions of time. For a sample-split training procedure, we establish an exact decomposition of the mean squared error into squared bias and variance, where the variance term grows with the number of basis functions and the width of the network's last hidden layer. Simulation experiments use this decomposition to quantify the bias-variance trade-off of SmoothCum and to compare it with competing methods. The measured variance of SmoothCum is linear in the number of basis functions, with slopes scaling correctly in sample size and signal-to-noise ratio; the cross-validated number of basis functions adapts to the smoothness of the true time-varying coefficient; the advantage over the estimator with a separate network per visit grows with the number of visits, by a factor of to ; and under irregular observation times, where a separate network per visit cannot be built, SmoothCum outperforms all competing methods, including DLNM, by a factor of to . An analysis of cohort data agrees with the bias-variance decomposition: the advantage of SmoothCum appears at cohort sample sizes and disappears in cohorts of thousands of subjects, where the variance it removes is small. When the model class is unknown, combining DLNM and SmoothCum by model averaging, with weights chosen by cross-validation over subject-level folds, was not worse than a single model in any setting we evaluated.

Primary area:
learning on time series and dynamical systems
Keywords:
longitudinal data, cumulative exposure effects, neural network estimation, irregular observation times

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3YxGOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SMOOTHCUM, a neural estimator of the cumulative mean model μit=E(Yit∣Xi1,…,Xit)=∑s≤tgs(Xis)\mu_{it}=\mathbb{E}(Y_{it}\mid\mathbf{X}_{i1},\ldots,\mathbf{X}_{it})=\sum_{s\le t}g_s(\mathbf{X}_{is}), where each gsg_s is a dd-dimensional nonlinear and unknown function. SMOOTHCUM keeps the cumulative structure of (1) but does not impose additivity. It replaces the TT separate per-visit output layers of the sub-network estimator by one shared network and a fixed set of KK smooth basis functions of time in the output layer, so that the contribution of each visit is an explicit smooth function of time with KK coefficients. For a sample-split version of the estimator, the paper derives an exact decomposition of the risk into squared bias and variance, in which the variance term is proportional to KK, and gives an approximation-error decomposition under the low time-rank, time-basis approximation and completeness assumptions. Simulations compare SMOOTHCUM with DLNM, the sub-network estimator, the two-step estimator, and sequence models; further experiments study irregular observation times, the smoothness sweep, and model averaging. Cohort analyses provide real-data evidence; however, no cohort in the paper's set has a non-additive target, so the real-data evidence covers the additive class only.

Critical strengths

S1: Appropriate inductive bias for small longitudinal cohorts. The shared feature extractor combined with a low-dimensional temporal basis preserves the cumulative-over-visits structure while avoiding the restrictive additivity-across-covariates constraint of DLNM. This effectively targets the critical regime between overly restrictive DLNM and flexible but data-hungry sequence models.

S2: Mechanism-focused evaluation. The empirical analysis goes well beyond a standard benchmark ranking. By decomposing the risk into squared bias and variance over repeated refits, isolating the individual effects of body sharing and the smooth temporal basis, evaluating irregular observation times, and honestly reporting regimes where DLNM or sequence models win, the paper makes its finite-sample claims highly transparent and interpretable.

Critical weaknesses

W1: Disconnect between the theoretical analysis and the deployed estimator. Theorem 1 conditions on a representation trained on a separate fold and uses a fixed-design least-squares projection. However, the empirical method is jointly trained and evaluated out-of-sample. Thus, the exact in-design decomposition offers no corresponding guarantee for the practical procedure. This gap is compounded in Appendix A, where approximation control depends on a realized feature-completeness event, without showing that the optimization process can attain it. This substantially limits the theoretical explanation of the method's success.

W2: Incomplete ablation of performance drivers. SMOOTHCUM differs from the primary DLNM baseline along multiple dimensions simultaneously: it uses an absolute-visit-time representation versus lag indexing, and allows covariate interactions while DLNM does not. Furthermore, the main architecture grid fixes K=3K=3 (Appendix G), relegating the adaptive-KK procedure largely to real-data tuning. More controlled comparisons and a clear end-to-end evaluation of the adaptive-KK procedure are needed to isolate which specific architectural choices drive the claimed gains.

Questions for authors

Q1: Theory / practice. Can you quantify the gap between the sample-split, frozen-feature least-squares estimator analyzed in Theorem 1 and the jointly trained estimator used in practice? Specifically, can you compare in-design and held-out-subject risks under matched settings, or derive an additional bound for the jointly trained procedure?

Q2: Dependence and shared representation. In Appendix A.5, which precise assumptions justify the application of the multitask representation bound to within-subject correlated observations? How do the constants and the apparent benefits of larger KK change under strong within-subject dependence?

Q3: Controlled baselines. Can you evaluate two additional baselines: (i) an additive model with the same absolute-time spline basis, and (ii) a cumulative neural model with a lag-indexed representation? This would disentangle the effects of covariate interactions from absolute-time modeling.

Q4: Basis selection. Which headline results rely on the fixed K=3K=3 versus cross-validated KK? Given the separate KK-sweep and real-data tuning, can you provide an aggregate comparison of fixed-KK and adaptive-KK SMOOTHCUM on the principal test grid?

Q5: Motivation for the condition-free design matrix claim. Theorem 1 explicitly states that "no structural assumption and no lower eigenvalue condition on ZB⊤ZB\mathbf{Z}_B^\top\mathbf{Z}_B enters." Since no such condition was emphasized earlier, could you elaborate on the motivation behind this specific claim? Given that the use of the Moore-Penrose pseudo-inverse ZB+\mathbf{Z}_B^+ mathematically bypasses the need for invertibility, does this condition-free property come at the cost of the variance bound becoming highly conservative in practical regimes with collinear covariates or redundant basis functions? How should practitioners reconcile this theoretical generality with the practical necessity of regularization?

4 comments

We are grateful to the reviewer for a careful reading of our manuscript and for five constructive questions, especially Question Q1, which asked us to quantify the gap between the analysed estimator and the deployed one. In this revision we have addressed this gap with a new theorem and two new tables, and we respond to each question below.

W1 and Q1 (theory analyses a frozen, sample-split estimator, so quantify the gap and compare in-design and held-out risk).

We did both. New Theorem 2 (page 6, lines 288–313, and Appendix A.6, pages 19–22) gives the exact decomposition for the jointly trained network with its output layer refit by least squares, so that the normal equations hold, with the noise term bounded through the pseudo-dimension. The gap to the deployed estimator is then one step, and we measure it in two tables.

Refit against deployed (Table 17, page 32, all 150150 settings, 200200 replicates, paired on the same trained networks). The jointly trained output layer sits 1.5%1.5\% from stationarity in gradient norm. Refitting it lowers the in-design risk and raises the held-out ISE by 6.3%6.3\% for SmoothCum (worse in 148148 of 150150 settings), by 7.3%7.3\% for the sub-network estimator and by −0.3%-0.3\% for the variant with time as an input. The ordering of SmoothCum against DLNM changes in 33 of 150150 settings and against the GRU in none.

In-design against held-out (Table 18, page 32, reference setting, 6060 refits, K=1,…,5K=1,\dots,5 and the sub-network estimator). The identity holds to machine precision in every fit. The realized noise term exceeds the fixed-network reference tr(PΣ)/N\mathrm{tr}(P\Sigma)/N by a factor of 1.321.32 to 1.761.76, and both terms grow linearly in the design rank. The held-out ISE is about twice the in-design risk for every estimator, with the same ordering of KK and of the two architectures. The deployed layer is within 0.050.05 of the refit layer in-design and 11 to 5%5\% better held-out.

The new theorem and the two added tables are what make the paper's explanation of the gain, variance reduction through the output layer and the shared network, stand for the estimator we deploy. Theorem 2 makes the mechanism, noise passed through a projection whose size scales with KK and with the parameter count, a property of the estimator we deploy rather than of a stand-in. Table 17 shows that the analysed estimator and the deployed one differ by about 6%6\% in held-out error and never change order against the competitors, so the theorem speaks about the estimator the tables report. Table 18 shows that the trained network's noise term exceeds the fixed-network value by a bounded factor, which is the cost of fitting the features on the same noise, and that both terms grow linearly in the design rank as Theorem 1 predicts, so the mechanism at work is the one the theory describes. It also shows that in-design and held-out risk differ by a factor and not in their ordering, so a conclusion drawn from the quantity the theorems bound carries over to the quantity the experiments report. The small held-out advantage of the deployed layer over the refit layer is the implicit regularization of stopping short of the least-squares solution. A convergence guarantee for the trained features is beyond current theory for networks of this size, so we measure the approximation term instead (Table 18), as the limitations now state (Section 8, page 10, lines 524–525).

W2, Q3 and Q4 (the comparison with DLNM confounds time indexing and interactions, and the grid fixes K=3, so add controlled baselines and compare fixed with adaptive K).

We agree on both points: the comparison with DLNM confounded two design choices, and the main grid fixed KK. The revision addresses the first with the two controlled baselines of Q3 and the second with the adaptive-KK procedure of Q4, run end to end on the full grid.

Controlled baselines. Both baselines are run on the full comparison grid (Table 19, page 33): DLNM with our absolute-time B-spline in place of its lag polynomial, and SmoothCum with its basis evaluated at the lag t−st-s. On the non-additive gintg_{\mathrm{int}}, moving DLNM from lag to absolute time changes its ISE by a factor of 1.041.04 (16.1316.13 to 16.7916.79) and moving SmoothCum from absolute to lag time by 0.950.95 (8.478.47 to 8.058.05). Holding the time representation fixed at absolute time, allowing interactions reduces the ISE by a factor of 1.981.98 (16.7916.79 to 8.478.47). On the additive gbaseg_{\mathrm{base}}, changing the time index moves the error by at most 9%9\%, while DLNM leads by a factor of 1.91.9. The gain is covariate interactions. Time indexing contributes only a few percent either way.

Fixed against cross-validated KK. We add the comparison the reviewer asks for. Table 20 (page 33) runs the adaptive procedure end to end on all 150150 settings of the principal grid (first 4040 replicates, paired with the fixed-KK estimator in the same replicate). Cross-validated KK has the lower ISE in 121121 of 150150 settings, by 7%7\% on average (14.6514.65 against 15.7815.78, paired difference −1.13-1.13 with SE 0.130.13), so the fixed K=3K{=}3 of the main tables understates the estimator rather than flattering it. The selected KK tracks the true coefficient: K=1K{=}1 in half the fits where asa_s is constant, K∈{2,3}K\in\{2,3\} in three quarters where it increases, and picks either K=1K{=}1 or K=5K{=}5 where it alternates, since an alternating coefficient is too rough for an intermediate KK to follow.

Both additions agree with the conclusions of the initial submission. The rule of Section 6.3, SmoothCum for non-additive targets and DLNM for additive ones, stands and rests on covariate interactions rather than on the time index, and the fixed K=3K{=}3 of the main tables is conservative for SmoothCum, so no reported comparison moves against it.

Q2 (assumptions under within-subject correlation, and the benefit of larger K under strong dependence).

The multitask bound is removed (see our reply to Reviewer 2qrW, W1). In the bounds that remain, within-subject dependence enters only through νT\nu_T, the largest eigenvalue of a subject's residual covariance, which multiplies the whole noise term and leaves the approximation term untouched (Appendix A.6, page 21, lines 1130–1133). Two consequences follow.

The constant in front of the noise term. Dependence inflates the noise term by a factor we measured: the ratio ∥P(η^)ε∥2/tr(P(η^)Σ)\|P(\hat\eta)\varepsilon\|^2/\mathrm{tr}(P(\hat\eta)\Sigma) rises from 1.221.22 under independent residuals to 1.361.36 under AR(1) and 1.431.43 under exchangeable correlation for SmoothCum (1.551.55, 1.631.63 and 1.681.68 for the sub-network estimator) and is flat in nn (Table 18, page 32).

The benefit of a larger KK. A larger KK lowers the approximation term and adds νT(H+1)/(nT)\nu_T(H{+}1)/(nT) of noise per basis function. Therefore, strong dependence raises the noise cost of each added basis function and makes the bound-optimal KK of Corollary A2 smaller, since KcontK_{\mathrm{cont}} decreases in νT\nu_T. The benefit of a larger KK therefore shrinks under strong dependence, while the variance bound at KK remains K/TK/T times the bound of the per-visit output layer, a ratio that does not depend on νT\nu_T. The experiments agree: the 6060-setting stress study with residual correlation 0.90.9 (Appendix C) keeps the method ranking, and in the TT-sweep arm with ρ=0.9\rho=0.9 the advantage over the per-visit estimator flattens with TT, as the cancelled 1/T1/T in the noise term predicts.

Q5 (the condition-free claim, the pseudo-inverse and regularization).

The claim says only that the bound of Theorem 1 does not depend on the smallest eigenvalue of ZB⊤ZB\mathbf Z_B^\top\mathbf Z_B, and a new remark explains why (Appendix A.6, page 21, lines 1120–1129). Because PKP_K is a projection, the identity and the trace bound need no lower eigenvalue condition, so rank deficiency from dead or duplicated hidden units costs nothing in the theorem. Collinearity lowers the rank and tightens the trace bound, so the condition-free statement is not conservative in design. Where conditioning matters is the transfer from in-design to held-out risk, which needs the Gram matrix of the trained features to be stable between training and test subjects, and that is where regularization earns its place. We measured where collinearity bites in practice (Table 17, page 32): the exact least-squares refit changes held-out ISE by more than 50%50\% in 2.8%2.8\% of replicates, always for the worse, when a near-singular direction of the trained design receives a large coefficient that the jointly trained layer never acquires. That is why we deploy the jointly trained output layer, which does not reach the exact least-squares solution, state Theorem 2 for the refit layer, where the normal equations hold exactly, and report the difference between the two in Table 17.

We thank the reviewer again for questions that each produced a table we would not otherwise have made, and we would be happy to answer any further questions.

I appreciate the authors' substantial revision. However, my main concerns remain, and I found several additional issues in the revised manuscript.

  1. The gap between Theorem 2 and the deployed estimator remains.

Although Theorem 2 extends the analysis to the jointly trained network, its exact risk decomposition requires the output layer to satisfy the least-squares normal equations. This condition is guaranteed by least-squares refitting, but not necessarily by the jointly trained output layer used in deployment. Table 17 shows that refitting increases held-out ISE by 6.3% on average and performs worse in 148 of 150 settings. Thus, the difference between the analyzed and deployed estimators is not negligible. Moreover, Theorem 2 only addresses in-design risk and does not provide a held-out risk guarantee or control the approximation term of the jointly trained features. My original theory–practice concern therefore remains only partially addressed.

  1. Unclear specification of the evaluated estimator.

The general experimental protocol states that KK is selected by cross-validation, and Appendix C.3 states that KK is never fixed in advance. However, Appendices C.2 and C.10 indicate that the main comparison grid uses fixed K=3K=3. These are different estimation procedures. The authors should clearly state which procedure was used for each main result and which conclusions are supported by the cross-validated KK estimator.

  1. Inconsistency in the bias–variance decomposition.

The manuscript states that the variance across 60 refits is computed with a divisor of 59. However, the ISE values in Table 6 appear to be obtained by directly adding squared bias and variance. Under the stated variance convention, the corresponding ISE values should be approximately 12.41 and 24.47, rather than 12.50 and 24.71. This difference is larger than rounding error, contrary to the authors' explanation. Since the bias–variance decomposition is central to the paper, this inconsistency needs to be addressed.

  1. Inconsistent experimental results across tables.

Tables 2 and 17 report different deployed-estimator ISE values for the same 150-setting comparison grid: 15.99 versus 16.49 for SMOOTHCUM, and 23.86 versus 24.45 for Subnet-Cum. The real-data results also differ across tables. For example, the HRS SMOOTHCUM MSE is reported as 0.8628, 0.870, and 0.862 in Tables 27–29, while the CUMBLEND MSE changes from 0.856 to 0.849. Although the manuscript mentions rerunning experiments, these differences are not clearly explained.

  1. Empirical claims are not fully supported by the reported results.

Several claims appear inconsistent with the tables. The manuscript states that all competing methods on HRS have error ratios between 0.99 and 1.01 relative to SMOOTHCUM, but Table 26 includes substantially larger ratios, reaching approximately 1.09 for some history-based methods. The broad claim of superiority under irregular observation times also needs qualification, since Table 16 shows that DLNM clearly outperforms SMOOTHCUM on both predominantly additive gbaseg_{\mathrm{base}} settings. Furthermore, the statement that every single-model MSE on the food-group panel is within 4% of the mean-only predictor is inconsistent with the SMOOTHCUM MSE of 1.296 in Table 29.

Although the revision includes meaningful improvements, the remaining theory–practice gap and the newly identified experimental inconsistencies weaken my confidence in the paper's soundness. Therefore, I keep my original rating.

Reviewer_2qrWOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the estimation of cumulative longitudinal means in short panels with hundreds to a few thousand independent subjects. SmoothCum shares a neural feature map across visits and represents visit-specific coefficients using a small spline basis in time. It predicts the mean by summing contributions from the covariate history. For a variant with a separately trained and frozen feature map, Theorem 1 decomposes the conditional risk into approximation error and the variance of the least-squares output layer. The experiments include a simulation grid, separate assessments of bias and variance, irregular visit times, and three cohort applications. According to the appendix and the released configuration, the main comparison grid uses K=3. The authors also examine the effects of varying K and propose averaging predictions from SmoothCum, DLNM, and GRU when the appropriate model structure is unknown. The reported gains are concentrated in smaller samples with substantial covariate interactions, while competing methods perform better in other settings.

Critical strengths

1.The paper clearly defines the cumulative target and explains its two main architectural choices: sharing a feature map across visits and using a smooth time basis, i.e. Sections 3 - 4. These choices can be varied independently. For the frozen-feature model, Theorem 1 gives a correct projection identity for conditional risk and accounts for residual correlation within subjects, as in Appendix A.4. Corollary A3 usefully distinguishes coefficient count from actual variance and approximation error, helping explain when a smaller time basis should help. Section 8 also spells out the limits of extending this result to joint neural training. The paper is careful about the scope of its theoretical claims.

2.The experiments help explain where the gains come from and where the method falls short. Section 6.2 estimates bias and variance across 60 refits. The ablation in Appendix C.2 separates feature sharing from coefficient smoothing: risk drops from 23.87 with separate visit networks to 17.60 with a shared network and visit indicators, then to 15.54 with K=2 (Table 8). Sharing accounts for the larger improvement, but the time basis provides a further measurable gain. The main grid uses 200 repetitions per setting and evaluates performance on held-out subjects. The paper also reports losses to DLNM on mostly additive targets and to sequence models when interactions across visits are strong, as in Tables 4 and 12. These comparisons give a useful picture of when the imposed structure helps.

Critical weaknesses
  1. Appendix A.5 does not establish the representation learning rate claimed in Proposition A1 under the paper’s sampling model. Theorem 1 of Maurer, Pontil, and Romera-Paredes (2016) assumes independent sampling across task datasets, but here all visits come from the same subjects. Assumption A1 bounds the conditional residual covariance. But it does not control covariate dependence or the empirical process used to learn the feature map. To see the problem, consider independent subjects with Xit=XiX_{it}=X_i, gt=hg_t=h, and Yit=t,h(Xi)Y_{it}=t,h(X_i), with zero residuals. Assumption A1 holds, yet every increment task contains the same nn labeled inputs. Additional visits provide no new inputs for learning hh, so the residual envelope alone cannot justify the claimed T\sqrt{T} factor. The caveat about dependence does not fill this gap.

This issue matters because the paper attributes about three quarters of the observed variance reduction to feature sharing. The empirical gain may be real, but the proposed theoretical scaling requires additional assumptions or should be presented as motivation.

  1. The additional benefit of time variation is unclear. The main grid’s only nonadditive signal has a constant time coefficient, and Table 8 reports risks of 16.02 for K=1 and 15.98 for K=3. The paper tests time dependence elsewhere, and smoothing helps on some signals, so I do not conclude that the basis is ineffective. What remains unresolved is its benefit over a constant shared cumulative model in the nonadditive cases.
Questions for authors
  1. Can Proposition A1 be justified when subjects are sampled independently but visits within each subject are dependent? Please define the complexity terms explicitly, including their dependence on T, and explain how the cumulative training loss maps to the loss in the cited multitask theorem. If the proposition is intended only as motivation, would you remove the formal rate claim? A proof under this sampling scheme would resolve my first weakness.

  2. The irregular time experiment already includes nonadditive signals with sinusoidal time coefficients. Could you add a K=1 comparison using the same subject splits? Please also report paired uncertainty estimates for the performance differences between K=1 and both the selected spline model and the variant that supplies time to the network. A clear gain from K>1 in this setting would reduce my concern about the benefit of the time basis.

  3. Section 6.4 and Appendix F report continuous weight for length zscore prediction. Appendix F.1 however says that the nutrient results are outside this submission and describes binary outcomes. Which configuration and result summary support the comparison? Is the statement in F.1 obsolete? If the comparison cannot be traced to a released configuration, I would place less weight on the cohort evidence.

2 comments

We are grateful to the reviewer for a precise reading of the theory, and especially for the counterexample in Comment W1, which settles the question of Proposition A1. All three requests are met in the revision, and we respond to each below.

W1 and Q1 (Proposition A1 does not follow under the paper's sampling model, so define the complexity terms and remove the rate claim if it is motivation).

The reviewer is right, and the example settles it: with Xit=XiX_{it}=X_i every visit repeats the same nn inputs, so no argument that counts visits as samples can deliver a T\sqrt T factor for learning hh, and the cumulative loss does not map onto the independent task samples of the multitask theorem. As the reviewer suggests, the proposition is removed from the revision, together with the multitask import. In its place, the new Proposition A1 of the revision (page 19, lines 996–1003) bounds the pseudo-dimension of the class of fitted values explicitly, with every dependence on TT and on the parameter counts shown: c′(p+q)ℓlog⁡((p+q)T)c'(p+q)\ell\log((p+q)T) for a shared network with pp weights, ℓ\ell hidden layers and q=K(H+1)q=K(H{+}1) output coefficients, against c′T(p+H+1)ℓlog⁡(T(p+H+1))c'T(p+H{+}1)\ell\log(T(p+H{+}1)) for TT separate networks. It enters the new Theorem 2 (page 6, lines 288–313, and Appendix A.6, pages 19–22) as the uniform noise bound cνTDlog⁡N/(nT)c \nu_T\mathcal D\log N/(nT), which assumes independent subjects and the covariance bound (A1) only. Dependence across visits enters through νT\nu_T, the largest eigenvalue of a subject's residual covariance, which grows with the strength of that dependence and cancels the 1/T1/T when visits are fully dependent. So the theorem says which architecture has the smaller complexity and how that complexity grows with TT. The finding that about three quarters of the variance reduction comes from sharing is a measurement from the ablation in Section 6.2, and it does not depend on the removed proposition.

W2 and Q2 (benefit of time variation on non-additive targets, and K=1 under irregular times with paired uncertainty).

Both requests are met. First, Table 15 (page 31) adds the K=1K{=}1 basis to the irregular-time study on the same 4040 subject splits, with paired differences, standard errors and Wilcoxon pp-values against the cross-validated KK and against the variant with time as an input. At f=1f{=}1 the constant basis loses to both on the non-additive signal (99 to 10%10\%) and on the additive one (18%18\% and 31%31\%). At the rougher f=2f{=}2 the three are within noise on the non-additive signal, and K=1K{=}1 trails by 44 to 7%7\% on the additive one.

Second, on the comparison grid, cross-validating KK beats the fixed K=3K{=}3 we reported by 7%7\% on average and in 121121 of 150150 settings, and the selected KK follows the truth (Table 20, page 33): K=1K{=}1 in half the fits where asa_s is constant, and in 5%5\% of the fits where asa_s increases with ss.

The reviewer's concern was that the time basis adds nothing over a constant shared cumulative model on non-additive targets. The new result answers this directly: on the non-additive signal with a smooth time coefficient, K=1K{=}1 loses 99 to 10%10\% to the cross-validated KK, with p=0.004p=0.004 on paired splits. That is a clear gain for the basis in exactly the case the reviewer named. The tie on the main grid is a correctly specified K=1K{=}1, not a failure of the basis, and the cross-validated KK lands on K=1K{=}1 there by itself. Of note, the gain is conditional on smoothness. Because K=1K{=}1 is among the candidates the cross-validation selects from, the basis costs nothing where the coefficient is constant and gains where it is not.

Q3 (which configuration supports the nutrient comparison, and whether F.1 is obsolete).

The statement in F.1 was left over from an earlier draft, and it is corrected (page 38, lines 1998–2002): the nutrient panel (n=665n{=}665, d=17d{=}17) is the continuous zz-score analysis of Section 6.4, and the binary extremes are a separate analysis used only in the model-averaging study. Every cohort number is now traced to its released driver, configuration and summary file in a new paragraph of Appendix F.1 (page 37, lines 1954–1965). The ELEMENT data are available under the cohort's data-use agreement, and the per-split summary files of every cohort run are in the release.

We thank the reviewer again. The counterexample improved the paper's theory, and the K=1K{=}1 comparison and the cross-validated-KK grid are now part of the evidence for the time basis. We would be glad to answer any further questions.

Thanks for the thorough rebuttal. My concerns have been partially addressed. I raised my score.

Reviewer_wpnKOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SmoothCum, a neural method for estimating cumulative mean models in longitudinal cohort data, where the outcome at each visit is a sum of nonlinear functions of past covariates. The method shares one feature network across all visits and uses a small number of B-spline time basis functions in the output layer. The authors derive a bias-variance decomposition for a sample-split version of the estimator, evaluate the method on simulations and three real cohorts, and propose a model average (CumBlend) of DLNM, SmoothCum, and a GRU.

Critical strengths
  • Clear problem: the paper targets cohort studies with moderate sample sizes, short series, and non-additive effects, where DLNM is biased and sequence models have high variance.
  • Works with irregular visit times: the spline basis is defined in continuous time, so the method handles subject-specific observation times and outperforms DLNM, a GRU, and an ODE-RNN in that setting.
  • Honest experiments: the bias-variance study separates the gain from network sharing (most of it) from the gain from the time basis (a smaller part), and the authors clearly report where the method fails.
Critical weaknesses
  • The theory is weak relative to the claimed mechanism. Theorem 1 covers a linear regression on a frozen, sample-split feature network, not the jointly trained estimator. The main gain comes from sharing the network across visits, and for that part the paper only offers a bound adapted from multitask learning (Appendix A.5), as the authors acknowledge in the limitations.

  • A simpler variant is better. The variant that supplies time as an input feature predicts 0 to 21% better than the proposed product form on regular grids (Section 4, Appendix C.4), and the two tie under irregular times. The authors state that the product form is kept as the primary estimator because its linear output layer is what makes the risk decomposition available. The proposed architecture therefore seems to be chosen to fit the theory rather than for performance.

  • No clear win on real data. SmoothCum is not the best method on any of the three real cohorts: DLNM has 8% lower test MSE on ELEMENT, all methods are within 1% of each other on HRS, and the sub-network estimator and the best sequence model are about 30% ahead on ECLS-K. The authors acknowledge that none of the cohorts has a non-additive target, so the real-data evidence does not test the setting in which the method is supposed to help.

  • Sensitive to nuisance covariates. Adding inactive covariates (d from 7 to 50) increases SmoothCum's error by a factor of 2.46 to 3.00, whereas sequence models degrade by only 1.56 (Appendix C.5). On the real food-group panel (d = 40) the best sequence model moves ahead by 24%. The suggested remedy, projection onto principal components, is unsupervised and may discard the interaction directions that motivate the method in the first place.

Questions for authors
  1. Why not make the time-as-input variant the default, since it performs at least as well and requires fewer structural assumptions? Is there any setting in which the product form is preferable in practice?
  2. Could structured sparsity or feature gating handle nuisance covariates instead of PCA, so that interaction directions are preserved?
  3. The paper states that all neural methods share one training and tuning step of about 40 seconds per replicate on a CPU. Does this mean each neural method costs about 40 seconds, or that the step is shared across methods? Please report the per-method cost.
3 comments

We are grateful to the reviewer for a careful reading of the manuscript, and especially for Comment W2, which led us to state explicitly that the two variants of the estimator belong to one framework and to give the reason for keeping the product form as the reference. The revision answers every point with new measurements, and we respond to each below.

W1 (theory covers only a frozen, sample-split network, and sharing rests on a multitask bound).

We agree on both points, and the revision addresses both. New Theorem 2 (page 6, lines 288–313) covers the jointly trained estimator: for the network as trained, with its output layer refit by least squares, the identity ∥μ^−μ∥2=∥(I−P(η^))μ∥2+∥P(η^)ε∥2\|\hat\mu-\mu\|^2=\|(I-P(\hat\eta))\mu\|^2+\|P(\hat\eta)\varepsilon\|^2 holds for every realization of the data, and the noise term is bounded by cνTDlog⁡N/(nT)c \nu_T\mathcal D\log N/(nT), with D\mathcal D the pseudo-dimension of the class of fitted values. The full statement and proof are in Appendix A.6 (pages 19–22).

For the shared network we no longer claim a rate. The submitted Proposition A1 is removed, because its rate claim does not transfer to our sampling scheme (see our reply to Reviewer 2qrW, W1). The new Proposition A1 of the revision (page 19, lines 996–1003) bounds D\mathcal D by c′(p+q)ℓlog⁡((p+q)T)c'(p+q)\ell\log((p+q)T) for the shared network and by c′T(p+H+1)ℓlog⁡(T(p+H+1))c'T(p+H{+}1)\ell\log(T(p+H{+}1)) for a network per visit, so sharing shrinks the bound by a factor of TT in its network part and T/KT/K in its output-layer part.

W2 and Q1 (the time-as-input variant is simpler and at least as good, so why not make it the default?).

We agree with the observation, and we have changed how the paper presents the two variants (Section 4, page 5, lines 223–237). Both variants belong to the proposed framework and share everything the claims rest on: the cumulative sum over visits, one network shared across visits and across all covariates, the same fixed smooth basis of time with the same cross-validated KK, a definition at any real time, and an output layer that is linear in its coefficients. They differ only in where the time representation sits. Theorem 2 therefore covers both. The submitted justification, that the linear output layer of the product form was what made the theory available, is corrected (Appendix A.6, page 21, lines 1112–1119).

On performance we found no setting in which the product form predicts better, and the paper now says so: the variant with time as an input is 00 to 21%21\% ahead on regular grids, the two tie on the non-additive signal under irregular times, and the variant is 10%10\% ahead on the additive signal there (Table 16, page 31, and Section 6.3, page 9, line 461). We keep the product form as the reference instance because it alone exposes the two structural choices as measurable quantities: an explicit time-varying coefficient curve ∑kBk(s)βk\sum_kB_k(s)\beta_k on the shared features, which is what an epidemiological analysis reports, and an output layer whose coefficient count K(H+1)K(H{+}1) is what the bias–variance decomposition of Section 6.2 is read on. A user who wants prediction alone may take the variant with time as an input, and Section 4 now says this (page 5, lines 236–237).

W3 (no clear win on real data, and no cohort has a non-additive target).

This is correct, and we have not tried to manufacture a cohort on which SmoothCum wins. The question a practitioner faces is different: the model class is unknown, so should one choose between DLNM, SmoothCum and a sequence model, or average them? Section 7 answers this in simulation, and the revision answers it on every cohort (Appendix F.3, pages 38–40, Table 29). On all four continuous tasks the average has a lower test MSE than the single family selected by cross-validation, and it is at least as good on each as the best single family chosen by test error. The paired differences are −0.0125-0.0125 (SE 0.00510.0051, better in 1414 of 2020 splits) for the ELEMENT nutrients, −0.0160-0.0160 (SE 0.00230.0023, 1919 of 2020) for the ELEMENT food groups, −0.0043-0.0043 (SE 0.00070.0007, 1919 of 2020) for HRS and −0.0007-0.0007 (SE 0.00020.0002, 1515 of 2020) for ECLS-K. The weight the average assigns to SmoothCum is the evidence that it contributes: on the nutrient panel, HRS and ECLS-K it receives 0.350.35, 0.210.21 and 0.170.17 of the weight on average and its weight is never zero in 6060 splits, so it carries information that neither DLNM nor the GRU has. Section 6.4 (page 10, lines 495–497) poses the question and Section 7 (page 10, lines 513–520) states the conclusion in the main text.

W4 and Q2 (sensitivity to nuisance covariates, and whether gating could replace PCA).

We agree that the projection is unsupervised and can discard the interaction directions, and we implemented the alternative the reviewer proposes: a multiplicative gate on each input, learned with the network under an ℓ1\ell_1 penalty so that gates reach exactly zero, with the penalty tuned like the other hyperparameters (Table 21, page 34). On the d=40d{=}40 food-group panel, the case the reviewer cites, the gate brings SmoothCum from test MSE 1.3131.313 to 0.9850.985, ahead of PCA-1010 (1.0571.057) and level with the GRU (0.9980.998), keeping 33 to 44 of the 4040 food groups. In simulation it beats PCA-1010 by 1313 to 29%29\% on the additive settings and ties it on the interaction settings while keeping all seven active inputs, so the interaction directions are preserved, which is the reviewer's concern with the projection. The gate repairs the level of the error rather than its growth in dd, and the paper says so. Appendix C.4 now presents the gate as the supervised option beside the projection (page 23, line 1227), and Section 6.4 mentions it (page 8, line 431).

Q3 (per-method cost).

The 4040 s was the sum of one training-and-tuning pass over all neural representations, and the sentence was ambiguous. Each method is now timed alone (Table 22, page 35). The table below gives seconds per replicate for one full training-and-tuning pass on four CPU cores.

Methodn=200n=500
SmoothCum3.54.2
Time as input2.42.9
GRU4.46.6
Transformer3.35.7
TCN4.36.2
State-space model5.610.8
PatchTST4.07.0
Sub-network estimator6.77.6
DLNM (full fit)0.51.3
All neural representations5068

Section 6.3 (page 8, lines 419–423) and Appendix G (page 41, lines 2176–2186) now give the per-method figures.

We thank the reviewer again for comments that led to the controlled presentation of the two variants, the supervised gate and the per-method timing, and we would welcome any further questions.

Thank you for the detailed response, I will raise my score.

Comments on the paper

1 comment

We thank the three reviewers for careful readings that converge on three points: the distance between the theory and the estimator we deploy, the confounded comparison with DLNM, and the real-data evidence. The revision answers each with new theory and new experiments, all run on the released code under the paper's protocol, and corrects the two statements in the submitted appendix that Reviewer 2qrW identified: the submitted Proposition A1 (the multitask rate claim), whose rate does not transfer to our sampling scheme and which is removed, and a sentence in Appendix F.1 left over from an earlier draft. In brief:

  1. The theory now covers the jointly trained estimator. New Theorem 2 (page 6, lines 288–313, with the proof in Appendix A.6, pages 19–22) gives the exact risk decomposition for the network as trained, with its output layer refit by least squares, and bounds the noise term through the pseudo-dimension of the architecture. The submitted Proposition A1 is removed, because its rate claim does not transfer to our sampling scheme, and a new proposition, numbered A1 in the revision, bounds the pseudo-dimension of the two architectures (page 19, lines 996–1003). The gap between the analysed estimator and the deployed one is now measured (Table 17, page 32): the refit changes held-out ISE by 6.3%6.3\% on average, and no ordering in Table 1 changes except against DLNM in 33 of 150150 settings.
  2. The gain is covariate interactions, not time indexing. Two controlled baselines on the full comparison grid (Table 19, page 33) show that changing the time index moves the error by at most 9%9\% in either direction, while allowing interactions at a fixed time representation reduces it by a factor of 1.981.98 on non-additive targets.
  3. Cross-validated KK beats the fixed K=3K{=}3 we reported, by 7%7\% on average and in 121121 of 150150 settings (Table 20, page 33), and the selected KK follows the smoothness of the true coefficient. Under irregular times the K=1K{=}1 basis loses 99 to 31%31\% to the cross-validated KK where the coefficient is smooth (Table 15, page 31).
  4. Real data: the deployment question. On all four continuous cohort tasks, averaging DLNM, SmoothCum and the GRU with cross-validated weights has lower test error than selecting one of them, better in 1414 to 1919 of 2020 splits with every paired difference at least 2.42.4 standard errors from zero (Table 29, page 40). On the nutrient panel, HRS and ECLS-K, SmoothCum receives on average 0.350.35, 0.210.21 and 0.170.17 of the weight and its weight is never zero in any split. We state where it does not help.
  5. Per-method cost is now measured for each method alone (Table 22, page 35): SmoothCum 3.53.5 to 4.24.2 s per replicate, GRU 4.44.4 to 6.66.6 s, sub-network estimator 6.76.7 to 7.67.6 s.
  6. A supervised input gate for nuisance covariates (Table 21, page 34) restores SmoothCum on the d=40d{=}40 food-group panel from 1.3131.313 to 0.9850.985 test MSE, ahead of PCA-1010 (1.0571.057) and level with the GRU (0.9980.998).

The additions are Theorem 2 and Appendix A.6, Appendix C.10 (“Additional results added in revision”, pages 30–35, eight tables), Appendix F.3 (“Model averaging on the cohorts”, pages 38–40), a traceability paragraph naming the driver, configuration and summary file behind every cohort number (Appendix F.1, page 37, lines 1954–1965), and Table 4 (page 23), which lists every simulation study with its settings. The corrections are the two named above: the removal of the submitted Proposition A1 (Appendix A.5, page 19) and the statement that the nutrient-panel results were outside the submission (Appendix F.1, page 38, line 1998).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.