acceptodds
Withdrawn from ICLR 2027

Online Sparse Additive Learning without a Prespecified Reporting Horizon

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Estimating sparse nonlinear effects from a growing data stream requires updating both the selected variables and their uncertainty. Correlated covariates can give an irrelevant variable a larger initial effect than a relevant one, causing thresholding to fail. The sample means used to center effects also vary across samples. In this article, we propose an online sparse additive method in which selection, estimation, and centering share one kernel-weighted least-squares criterion computed from stored sums. Joint re-estimation with smoothly clipped absolute deviation weights consistently selects the relevant variables in constructed correlated models where initial thresholds fail. Under population selection margins and identifiable correlated effects, continuous-effect estimates attain the known-support optimal rate for a stationary repeat-or-redraw covariate process. With Gaussian autoregressive errors and bounded pairwise density ratios, we establish asymptotically valid joint confidence intervals for a fixed finite set of effect values at each prespecified reporting time. The final sample size need not be specified. For a fixed number of covariates, storage has an asymptotically sublinear bound. In simulations with strong signals, recomputed centers correct the undercoverage of fixed centers. Our method also lowers estimation error against the compared planned-epoch and stochastic-gradient estimators. Bike-rental and appliance energy analyses identify stable sparse nonlinear effects.

Primary area:
learning theory
Keywords:
Online learning, sparse additive models, variable selection, smooth backfitting, simultaneous inference, streaming data

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_MJ3cOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper focuses on online fitting of sparse additive models. The paper proposes to integrates the smooth backfitting objective function with group LASSO and SCAD regularizations and develops online estimates and simultaneous confidence bands together with theoretical guarantees. The practical performance of the proposed method is compared with existing methods by simulation and two real data applications.

Critical strengths

The paper studies online sparse additive models, which is an extension of offline sparse additive models to the online case as well as an extension of online additive models to handle high-dimensional covariates. It potentially enriches the theoretically guaranteed methodological toolbox in nonparametric statistics and machine learning.

Critical weaknesses
  1. The proposed method is not well justified. As illustrated in Figure 1, the proposed method involves multiple steps, but it is not clearly explained what purpose each step is needed for (e.g., smooth backfitting, group LASSO, SCAD) or whether and why each step is the best choice compared to other options. For example, it is not justified why the discretized smooth backfitting criterion is a better choice than approximating each component function by global basis functions, e.g., B-spline basis functions. Although the smooth backfitting criterion is adopted, the estimation procedure does not involve iterative updates like the original smooth backfitting algorithm and the oracle asymptotic distribution of each fitted component function is not developed, which is the key theoretical feature/advantage of smooth backfitting. Definitions in Lines 174-176 are also unclear.

  2. The paper is somewhat weak in both theoretical innovations and practical utility. First, as a “learning theory” paper, it fails to demonstrate substantial theoretical innovations, e.g., what key theoretical difficulties this paper can address while existing theoretical tools cannot. Comments are missing about the connections between the developed theoretical results and literature. Second, the practical utility of the proposed method is not convincing. Although this paper is categorized into the “learning theory” Primary Area by the author(s), it is more suitable to treat it as a methodology paper, but its practical utility, which is an important assessment metric for such papers, is very weak. For the two real data applications, necessary details are not provided in the main text, existing methods are not compared, e.g., Haris et al. (2022) not cited by this paper and those cited in the simulation study, and new practical insights revealed by the proposed method are not given. The Bikeshare data application is also problematic since null covariates are artificially added.

Reference:

Haris, A., Simon, N., & Shojaie, A. (2022). Generalized sparse additive models. Journal of Machine Learning Research, 23(70), 1-56.

Questions for authors
  1. Why is the discretized smooth backfitting criterion used instead of its original version? Why not approximating each component function by global basis functions, e.g., B-spline basis functions?
  2. What are Gaussian-copula additive models in Theorem 2?
  3. Can Gaussian autoregressive errors be relaxed to sub-Gaussian or sub-exponential?
  4. Line 178: Taking the derivative of the SCAD penalty as a weight seems an unusual way of using SCAD. What is the justification of it?
  5. What is the connection between f^j\hat{f}_j defined in Line 134 and the estimated component function obtained by Eq. (1)?
  6. The main text should be self-contained. For example, Section 4 discusses many results not given in the main text, e.g., F-SGD and Sieve-SGD are not included in Table 1 in the main text.
  7. Centering component functions is a standard practice to address identifiability in additive modeling. It is overstated to emphasize it as an obstacle.
  8. The logical flow in this paper is unclear. Many consecutive sentences lack logical connections between them.
Reviewer_ZtpfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper combines SCAD weights, joint re-estimation, and block deletion with recomputed centers for online sparse additive models. The continuing estimator updates stored sums without a prespecified final sample size. Under the stated conditions, it attains the known-support rate and joint coverage for a fixed finite set of effect values at prespecified reporting times.

Critical strengths

The paper is in general easy to understand. Some empirical results show good performance.

Critical weaknesses
  1. Limited methodological novelty. The criterion follows smooth backfitting, and joint re-estimation with SCAD weights builds on one-step sparse estimation. Online smooth backfitting and functional Lasso kernel smoothing are close precedents. The conceptual advance beyond these existing methods is not sufficiently clear.

  2. Limited validation of the continuing estimator. Both real-data analyses use planned-epoch fits, and Table S6 measures planned-epoch implementations. The paper does not establish the full continuing procedure's practical memory and time costs.

Questions for authors
  1. Clarify the main novelty of the paper. Currently, it seems incremental and limited to me.
  2. Perform sensitivity analyses on both real data application.
Reviewer_unMAOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper considers a GAM based online learning method for streaming datasets where there is no prefixed end point in the data. The paper develops a method for variable selection using SCAD penalty. The proposed method using screening and batch learning. Theory is provided along with analyses for bike-rental and appliance energy that show non-linear sparse effect.

Critical strengths

The paper builds on penalized GAM literature. GAM is a very popular method. The problem is interesting and relevant.

Critical weaknesses

The paper is not readable. It lacks focus as too many things are covered. Some of the sentences in the main paper are very difficult to read.
The proposed method is a straightforward extension of the existing literature on penalized GAM. A lot of research has been made over the last 20 years on penalized regression and GAMs. While the paper uses that background to develop a new method for online sparse additive learning, it does not have novel insights.

Questions for authors
  1. The model in Section 2 does not have any time varying coefficients which reduces the scope of the paper for online learning. The paper considers streaming datasets with no end points. Such datasets seldom can not be well modeled by fixed coefficient models studied here. In the applications mentioned in the paper, a varying coefficient model is better suited. The fixed coefficient model greatly reduces the contributions and insights that emerge in online learning in GAMs.
  2. I do not see why L_0 methods that directly solve for exact sparsity are not used instead of SCAD penalties. 
  3. Why does the model use a fixed global bandwidth? This is very naive and will not work if there is heterogeneity among the variables. 
  4. How can the model tackle interactions? 
  5. The assumptions used in the theory need to be rigorously stated in one place. Currently the assumptions on temporal correlation across observations (is it Gaussian AR(1) or strong mixing beyond Gaussians?), correlation across covariance and compact support and bounded density of each variable are scattered all over the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.