acceptodds
Under review as a conference paper at ICLR 2027

Distributional Regression for Arbitrary Response Variables Using ReLU Neural Networks

Abstract

This paper develops a unified framework for distributional regression with arbitrary response variables. Rather than relying on scalar thresholds or Euclidean structure, we model conditional event probabilities indexed by measurable sets, which extends distributional regression to general response spaces while retaining a direct connection to classical event-based formulations. Within this framework, we study two complementary estimators: a preliminary estimator based on separate regression of event indicators, and a constrained estimator that enforces inclusion-consistent predictions through a directed acyclic graph (DAG) over a finite set family. We establish general error bounds for both estimators and further specialize the results to dense ReLU network, yielding explicit convergence rates overcoming the curse of dimensionality. Simulation studies and real-data experiments further demonstrate the practical effectiveness of the proposed framework.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Distributional regression, deep neural networks, structured prediction, shape constraints, arbitrary response spaces

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5bChOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes distributional regression for general response spaces by estimating conditional event probabilities FA∗(x)=P(Y∈A∣X=x)F^*_A(x) = P(Y \in A \mid X = x) over a user-chosen family of measurable sets. A preliminary estimator fits a separate least-squares regression of 1{Y ∈ A} on X for each set. A constrained estimator encodes inclusions among a finite family of sets as a DAG and parametrizes each set's logit as the maximum of its parents' logits plus a softplus increment, which enforces monotonicity under inclusion by construction. The theory provides uniform error bounds for the preliminary estimator over totally ordered families and lower orthants, aggregated bounds for the constrained estimator in terms of DAG quantities, and a new ReLU approximation result under hierarchical composition assumptions. Experiments on simulated, image, and real data (Beijing air quality, Chicago crime) show the constrained estimator outperforming Pre-NN, post-hoc clipping, DRF, and Engression in most settings.

Critical strengths
  1. A simple, practical construction that guarantees monotone predictions. Each set's logit is the maximum of its parents' logits plus a softplus increment, followed by a sigmoid. This guarantees monotonicity under inclusion for every input and every parameter value, adds essentially no computational cost, and fits naturally with standard neural networks. On the simulations, it substantially reduces error relative to fitting each set separately, and the gains are clearly significant. The idea is easy to adopt in applications involving nested events, such as the joint pollutant-exceedance events in the Beijing study, and is the main reason I see value in the paper.

  2. A comprehensive and, as far as I can tell, technically sound theoretical treatment (uniform bounds for the preliminary estimator, aggregated bounds for the constrained estimator, and an approximation result for the constrained parametrization). I did not identify errors in the main arguments; my concerns are about scope and interpretation rather than correctness.

Critical weaknesses
  1. The "arbitrary response" framing overstates the scope, and beyond the scalar case the inclusion constraint does not capture full probabilistic coherence.

(a) Scope. The method accesses YY only through membership in pre-chosen sets, and its benefit comes entirely from inclusions among them. A chain A1⊆⋯⊆AmA_1 \subseteq \dots \subseteq A_m is equivalent to thresholds of a single scalar statistic: setting S(y)=min⁡{j:y∈Aj}S(y) = \min\{j : y \in A_j\} gives Aj={S≤j}A_j = \{S \le j\}, so estimating {FAj∗}j\{F^*_{A_j}\}_j is exactly estimating the conditional CDF of S(Y)S(Y), regardless of the nature of Y\mathcal{Y}. More generally, all experiments use events defined through a few scalar statistics of YY (e.g., transitivity in S2, ∥Y∥\|Y\| in S4, the latent coordinates (t,h)(t,h) in S1, pollutant concentrations in Beijing). This is classical distribution regression applied to hand-picked summaries of YY (Foresi & Peracchi, 1995; Chernozhukov, Fernández-Val & Melly, 2013), where monotonicity across thresholds is a known issue with established remedies (rearrangement, Chernozhukov, Fernández-Val & Galichon, 2010; cumulative-link models, McCullagh, 1980). On a chain, the proposed parametrization essentially reduces to a cumulative-logit model with covariate-dependent thresholds. This literature is not discussed.

(b) Coherence. For a chain, inclusion-monotonicity is equivalent to full probabilistic coherence: any vector 0≤p1≤⋯≤pm≤10 \le p_1 \le \dots \le p_m \le 1 is attainable by some distribution. This fails in general beyond chains. For lower orthants with A1={y1≤0,y2≤0}A_1 = \{y_1 \le 0, y_2 \le 0\}, A2={y1≤1,y2≤0}A_2 = \{y_1 \le 1, y_2 \le 0\}, A3={y1≤0,y2≤1}A_3 = \{y_1 \le 0, y_2 \le 1\} and A4={y1≤1,y2≤1}A_4 = \{y_1 \le 1, y_2 \le 1\}, the outputs (0.1,0.5,0.5,0.6)(0.1, 0.5, 0.5, 0.6) are inclusion-consistent but imply P((0,1]×(0,1])=0.6−0.5−0.5+0.1=−0.3P\big((0,1] \times (0,1]\big) = 0.6 - 0.5 - 0.5 + 0.1 = -0.3. Similarly, in S5 the sets A2A_2 and A4A_4 are disjoint subsets of A7A_7, so coherence requires FA7≥FA2+FA4F_{A_7} \ge F_{A_2} + F_{A_4}, which the method does not enforce. In contrast, DRF outputs a weighted empirical distribution and is coherent by construction. In other words, the constraint is complete only when the problem is scalar. I would encourage the authors to reframe the contribution accordingly, as monotone multi-event distributional regression on chosen summaries of Y, and to position it relative to the distribution-regression literature.

  1. The paper does not establish that the constraint itself drives the empirical gains.

The natural post-hoc baseline, projecting Pre-NN outputs onto the monotone cone K={p:pk≤pj ∀(k,j)∈E}\mathcal{K} = \{p : p_k \le p_j \ \forall (k,j) \in E\}, is missing. Both the true probability vector and every label vector (1{yi∈Aj})j(\mathbf{1}\{y_i \in A_j\})_j lie in K\mathcal{K}, and projection onto a closed convex set is non-expansive, so this baseline can never increase the aggregated MSE, against the truth or against the labels. The reported degradation from post-hoc correction is thus specific to the clipping rule used. There is also no ablation separating monotonicity from parameter sharing: each Constrain-NN output depends on all its ancestors' networks, so a shared multi-head network may account for much of the gain. Finally, on Chicago the reported gap (0.0008) is about one unpaired standard error. Adding the projection baseline and a shared-network ablation would make the empirical case for the constraint much more convincing.

Questions for authors
  1. Is there any regime in which your bounds for the constrained estimator are strictly better than those for the preliminary estimator? If not, how do you explain the empirical gains?

  2. Were hyperparameters or the number of epochs tuned for any method? Is the dispersion reported in the tables a standard deviation or a standard error, and can you report paired comparisons for S3, Beijing, and Chicago? Is the real-data metric the Brier score against indicators?

  3. How does Constrain-NN compare to (a) Pre-NN followed by isotonic projection onto the DAG cone, and (b) a single shared network with m sigmoid heads, with and without projection?

  4. On the box-type scenarios and on Beijing, how often do Constrain-NN's outputs violate the rectangle inequality or additivity constraints, compared with DRF?

6 comments

We thank the reviewer for the careful evaluation of our paper and for the many constructive comments. We have carefully considered all the concerns raised and conducted additional experiments to strengthen our work. Below, we consolidate the concerns raised in the Weaknesses and Questions into related topics and address each of the reviewer's comments and questions in detail. We have also updated the manuscript PDF during the rebuttal period, with all corresponding revisions highlighted in blue, and we may kindly invite the reviewer to refer to the updated PDF for these changes during our responses.

Weaknesses

1(a). Scope and connections to classical distributional regression

(1) Connections to existing work. We agree that our work builds on classical distributional regression, particularly the event-based approaches of Foresi and Peracchi (1995) and Chernozhukov et al. (2013). As already stated in the Abstract and discussed in the Related Work section of our original submission (Section 1.2), our work focuses on conditional event probabilities indexed by measurable sets and builds on the classical distributional regression framework; we do not claim that regressing event indicators is itself a new idea. We also acknowledge that, when the target sets form a nested chain, the corresponding conditional probabilities can be represented as the conditional CDF of a scalar statistic of the response, connecting to cumulative-link models (McCullagh, 1980). Likewise, monotonicity correction through rearrangement has been studied by Chernozhukov et al. (2010).

However, unlike a nested chain, a general event family containing incomparable sets cannot be represented as threshold events of a single scalar statistic while preserving its inclusion structure. Our framework considers conditional probabilities indexed by measurable subsets of an arbitrary response space, including multivariate lower-orthant families and more general inclusion DAGs. Such event families cannot, in general, be represented as threshold events of a single scalar statistic while preserving their full inclusion relationships. Thus, although classical scalar distributional regression is an important special case of our framework, the estimation problems we consider are not restricted to this setting.

We thank the reviewer for identifying these additional references. We have now cited all the relevant papers identified by the reviewer, revised the Related Work section accordingly, and further clarified the connections between classical distributional regression and our estimation framework (see the text highlighted in blue on pages 2–3 of the revised manuscript). We have also clarified this scope in the Introduction of the updated manuscript (in blue), emphasizing that our goal is to estimate conditional probabilities over a chosen family of measurable events rather than to construct the full conditional distribution.

(2) Statistical and theoretical contributions. We agree that the existing literature has established important theoretical foundations for distributional regression and monotonicity-constrained estimation. For example, Chernozhukov et al. (2013) develop functional limit theory and statistical inference for scalar distribution regression and counterfactual distributions, while Chernozhukov et al. (2010) establish finite-sample improvement properties and asymptotic theory for monotone rearrangement. However, these results address statistical questions that differ from those studied in our paper.

Our main contribution is not to introduce event-indicator regression or monotonicity constraints as new ideas, but to establish new statistical estimation-error bounds and convergence guarantees for conditional event-probability nonparametric estimators over general measurable set families. In particular, we investigate uniform estimation errors over infinitely many target sets (Theorem 1) and the dependence of estimation risk on DAG topology for general inclusion structures (Theorem 2 and Equation (6)). For a detailed explanation of the novelty of these theoretical results, the associated technical challenges, and their distinctions from existing statistical theory, we refer the reviewer to our response to Reviewer Cbap, W1 (Novelty). In addition, we conduct various empirical evaluations across nine experimental settings (including one real-data-based toy example) and two additional real-data applications. These results demonstrate the strong empirical performance of our proposed constrained estimator across diverse response types and inclusion structures.

Furthermore, Appendix G (Table 4, page 57) of our original submission already provides a systematic comparison of the assumptions, estimation settings, and convergence guarantees of our results with those of the most closely related statistical works.

1(b). Inclusion Consistency versus Full Probabilistic Coherence

We thank the reviewer for pointing out this important distinction. We acknowledge that our proposed constraints enforce inclusion monotonicity over a partially ordered family of measurable sets, rather than full probabilistic coherence. We have explicitly acknowledged this limitation in the Conclusion of the updated PDF file (see the blue text at the end of page 10).

However, full probabilistic coherence is a stronger requirement than inclusion consistency, and satisfying this requirement is not equivalent to improving estimation accuracy for a specified collection of target events. When the objective is to estimate conditional probabilities for a prescribed family of target events, our approach directly estimates these probabilities while respecting their known inclusion relationships, without first constructing a complete conditional probability distribution over the response space. This direct estimation strategy is particularly meaningful when the response space is complex or when researchers are interested in only a specific collection of measurable events. Correspondingly, our theoretical guarantees concern the statistical accuracy of these event-probability estimators under the stated assumptions, rather than claiming that they satisfy full probabilistic coherence.

We also acknowledge that distributional random forests (DRF, Cevid et al. (2022)) guarantee full probabilistic coherence by construction through a weighted empirical distribution. This is a valuable property. However, probabilistic coherence does not necessarily imply stronger statistical convergence guarantees or better empirical performance, and the theoretical result in DRF does not provide an explicit convergence rate comparable to ours. As summarized in Appendix G (Table 4, page 57) of our submission, DRF only establishes consistency property in the form of op(1)o_p(1) for conditional CDF estimation, whereas our theoretical analysis further establishes explicit convergence-rate bounds for the MSE of the target event-probability estimators. Thus, our contribution concerns a different aspect of statistical estimation performance, rather than claiming to achieve full probabilistic coherence.

Moreover, within the classical set-indexed distributional regression framework considered in our paper, enforcing full probabilistic coherence for general multidimensional event families is a different and potentially more demanding task than enforcing inclusion monotonicity. In particular, methods such as isotonic projection and C-HMCNN (Giunchiglia and Lukasiewicz, 2020) also rely on inclusion constraints without guaranteeing all probabilistic coherence conditions among arbitrary overlapping or disjoint events. Please see the continuation of our response below.

1(b). (Continued) Inclusion Consistency versus Full Probabilistic Coherence

In the following, we further introduce an alternative DNN-based estimation strategy for classical set-indexed distributional regression that guarantees full probabilistic coherence. Specifically, we can construct the mutually disjoint partition induced by the target events, estimate the conditional probabilities of its atoms, and obtain each requested event probability by summing the probabilities of the corresponding atoms. This differs fundamentally from Constrain-NN, which directly estimates only the probabilities of the specified measurable target sets while enforcing their inclusion relationships. For overlapping target events, the common disjoint refinement may contain exponentially many atoms relative to the number of target events, potentially increasing the statistical and computational complexity of estimation. We also emphasize that an atomic partition is one way to achieve full probabilistic coherence, rather than the only possible approach.

To illustrate this distinction empirically, we conducted an additional experiment comparing Constrain-NN with an Atom-Softmax estimator that guarantees full probabilistic coherence. We consider a correlated binary response Y=(Y1,…,Yd)∈{0,1}dY=(Y_1,\ldots,Y_d)\in\{0,1\}^d, where dd is an even integer. For each observation, we generate X∼Unif⁡([−1,1]10)X\sim\operatorname{Unif}([-1,1]^{10}) and a shared noise variable ϵ∼N(0,1)\epsilon\sim N(0,1), and then sample

\sim \operatorname{Bernoulli}\left( \tau\left(X^\top\beta_j+b_j+0.75\epsilon\right) \right), \qquad j=1,\ldots,d,$$ independently across $j$ conditional on $(X,\epsilon)$, where $\tau(t)=(1+e^{-t})^{-1}$ and $(\beta_j,b_j)$ are randomly generated and then held fixed. We then define the target sets whose probabilities we aim to estimate. First, we define $d$ leaf events: $$A_j=\{Y_j=1\}, \qquad j=1,\ldots,d.$$ We then construct a two-layer event hierarchy by adding $d/2$ pairwise unions: $$B_k=A_{2k-1}\cup A_{2k}, \qquad k=1,\ldots,d/2.$$ Thus, the target sets estimated by **Constrain-NN** consist of only $m=3d/2$ events and form a two-layer DAG through the inclusion relations $A_{2k-1},A_{2k}\subseteq B_k$. In contrast, because each of the $d$ binary coordinates can take either zero or one, the complete disjoint partition of the response space contains $2^d$ binary patterns. Therefore, **Atom-Softmax** models the conditional probabilities of all $2^d$ mutually disjoint patterns using a softmax output and obtains each requested event probability by summing over the corresponding patterns. We compute the test probability MSE by averaging the squared errors over test observations and the $m=3d/2$ target events, then average this quantity over 20 repetitions. We report $100$ times the resulting average. **Table: Mean test probability MSE over the target events $(\times 100)$, across 20 repetitions** | Dimension $d$ | Number of target sets $m$ | Number of atoms $2^d$ | Constrain-NN | Atom-Softmax | |---:|---:|---:|---:|---:| | 4 | 6 | 16 | 2.127 | 2.461 | | 6 | 9 | 64 | 2.255 | 2.729 | | 8 | 12 | 256 | 2.241 | 2.657 | | 10 | 15 | 1024 | 2.132 | 4.211 | | 12 | 18 | 4096 | 2.285 | 6.385 | | 14 | 21 | 16384 | 2.206 | 36.721 | As shown in the table, Constrain-NN achieves lower aggregated MSE than Atom-Softmax across all examined dimensions, and the performance gap becomes particularly pronounced as the number of atoms increases. These results illustrate that, when the objective is to estimate a prescribed collection of target event probabilities, naively enforcing full probabilistic coherence through the atomic-partition approach examined in our experiment does not necessarily improve estimation accuracy.
Reviewer_CbapOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

The paper considers distributional regression for arbitrary response spaces through conditional event probabilities associated with a collection of measurable sets. For a finite collection of sets, the inclusion relations are represented by a directed acyclic graph (DAG). The proposed constrained estimator uses a recursive construction involving the sigmoid, softplus, and maximum over parent nodes to guarantee that the estimated event probabilities satisfy the inclusion relations. Estimation is performed jointly by minimizing squared errors between the predicted probabilities and the corresponding event indicators.

Critical strengths

The paper presents a general framework for estimating conditional event probabilities on arbitrary response spaces, with a DAG construction that enforces consistency under set inclusion. It establishes general risk bounds and explicit convergence rates for ReLU networks, including results characterizing the dependence on DAG topology. The experiments cover diverse response types and DAG structures, including manifold, functional, and network responses, and demonstrate consistent improvements over independently fitted regressions in most settings.

Critical weaknesses

The novelty of the core mathematical formulation could be more clearly distinguished from existing literature on distributional regression and hierarchical probability estimation, particularly regarding the event transformation, inclusion constraints, and their parametrization. While the experiments show improvements over the unconstrained baseline, the comparisons may not fully isolate the contribution of the proposed DAG construction, and the experimental design does not directly examine the theoretical convergence rates.

Questions for authors

By defining Zj=1Y∈AjZ_j=\mathbf{1}_{Y\in A_j}, the proposed method appears to reduce to hierarchical multilabel probability regression, where set inclusion implies logical constraints between binary labels. The paper already cites Hu et al. (2016) and Xu et al. (2018) on logical constraints, while Obozinski et al. (2008) and Giunchiglia and Lukasiewicz (2020) more directly study consistent probability predictions under DAG constraints. Furthermore, the sigmoid and softplus recursion appears related to the transfer map for partially ordered sets in Stanley (1986). Could the authors clarify what is mathematically new in the proposed estimation formulation beyond applying these existing constructions to membership indicators of measurable sets? In particular, does the recursion introduce a different constraint or simply parametrize the existing order constraints?

Obozinski, G., Lanckriet, G., Grant, C., Jordan, M. I., and Noble, W. S. (2008). Consistent probabilistic outputs for protein function prediction. Genome Biology, 9(Suppl. 1), S6.

Giunchiglia, E., and Lukasiewicz, T. (2020). Coherent hierarchical multi-label classification networks. Advances in Neural Information Processing Systems, 33.

Stanley, R. P. (1986). Two poset polytopes. Discrete & Computational Geometry, 1, 9–23.

5 comments

We thank the reviewer for the constructive comments. We have carefully considered the concerns raised and conducted additional experiments to strengthen our work. Below, we address each point raised in the Weaknesses and Questions. We have also updated the manuscript PDF during the rebuttal period, with all corresponding revisions highlighted in blue, and we may kindly invite the reviewer to refer to the updated PDF for these changes during our detailed responses.

Weakness

W1. Novelty. We agree that, transforming a distributional estimation problem into regressions of event indicators and enforcing consistency with a partial orderare, are not individually new. Our original submission already discusses related work on distributional regression and structured estimation under order constraints in Section 1.2. Our intended contribution is that we developed statistical guarantees for estimators constructed with similar ideas, while still operating over general measurable set families. Specifically, our main novelty lies in the statistical guarantees developed for these set-indexed estimators. For a more direct comparison with existing statistical theory for distributional regression, we have already provided a detailed discussion in Appendix G (Table 4) of our original submission, where we compare the assumptions, estimation settings, and convergence guarantees of our results with those of the most closely related theoretical works. We also would like to further clairify our theoretical novelty in the following points:

First, uniform error bounds for the preliminary estimator (Theorem 1 and Corollary 1). When we simply regress the event indicators separately, without imposing any monotonicity constraints, Theorem 1 establishes that, even for a fixed, finite sample size nn, we can obtain uniform estimation-error bounds simultaneously over infinitely many measurable target sets on an arbitrary response space, provided the set family is totally ordered by inclusion and satisfies the theorem's regularity conditions. Corollary 1 further extends the analysis to multivariate lower-orthant families, which are not totally ordered.

Second, risk bounds for the constrained estimator characterized by DAG topology (Theorem 2 and Equation (6)). We establishes aggregated MSE bounds for the constrained estimator over general inclusion DAGs. For a fixed nn, the result works for finite mm target set families and allows mm to grow with nn. More importantly, it explicitly characterizes the effect of DAG topology on estimation error, primarily through the average ancestor size aˉm\bar a_m. Thus, the statistical complexity is not determined by the number of target sets alone. We further illustrate this topology dependence across representative DAG structures in Appendix F and provide additional empirical evidence in Appendix L.2 of the revised manuscript PDF (see also our response below to your concern regarding empirical verification of our convergence rate). To our knowledge, such topology-dependent statistical risk characterization has not been established for the related methods discussed above.

Third, Theorems 1 and 2 are general results rather than DNN-specific guarantees. They apply to nonparametric estimators whose approximation errors and covering entropy can be controlled. While our main numerical investigation focuses on DNNs, Appendix E also derives convergence rates for trend filtering and orthogonal series estimators, demonstrating the broader applicability of the theory.

Finally, for DNNs, we develop new approximation theory specific to the softplus-DAG architecture (Appendix D). This is not a direct application of existing DNN approximation results: the parametrization enforces strictly positive increments along DAG edges, whereas the true conditional probabilities of nested events may coincide. We construct appropriate ReLU approximants, control the errors introduced by the DAG recursion and calibration networks, and establish the resulting convergence rates. We refer the reviewer to our response to Q2 of Reviewer RnGy for a more detailed explanation of these technical difficulties and their resolution.

In addition, we also conduct various empirical evaluations of the proposed estimators across 9 simulation settings (including one real data based toy example) and 2 additional real data applications. The results demonstrate the strong empirical performance of our proposed constrained estimator across diverse response types and inclusion structures.

Taken together, these results constitute our principal theoretical contributions. We have added all related works you mentioned, and made the distinction between established methodological ideas and our new statistical guarantees more explicit in the Related Work section of the revised manuscript (see the text highlighted in blue on pages 2–3).

W2. Isolate the contribution of the DAG.

We would like to clarify that Pre-NN and Constrain-NN use identical base DNN architectures and parameter counts, with a separate network for each set and no parameter sharing across sets. Unlike Pre-NN, which trains each network independently, Constrain-NN jointly optimizes these networks through the DAG-constrained construction. Thus, their comparison isolates the effect of DAG-constrained training without confounding it with parameter sharing.

To further assess this contribution, we also compare Constrain-NN with post-processing baselines, Pre-NN-Clip (across all experiments in the paper) and isotonic projection (added as a new competitor during the rebuttal; see Appendix L.1 and Table 13 on page 70 of the revised manuscript), which enforce inclusion consistency after training.

Finally, we present an entirely new ablation study, which has been added to Appendix L.3 on the final page of the revised manuscript. Specifically, we introduce Joint-NN, a single DNN in which all parameters are shared except those in the final mm-dimensional prediction layer, and Joint-Constrain-NN, which applies our DAG-based inclusion constraints to the prediction head of Joint-NN. The two estimators differ only in the presence or absence of the inclusion constraints, providing a controlled ablation.

We evaluate these methods in simulation scenarios S1 (Swiss roll) and S2 (RDPG network). The two joint-network estimators are evaluated over 10 replications for each configuration.

Table: Mean test probability MSE (×102\times 10^2; mean (SD); lower is better; best result in each row in bold)

SettingnnConstrain-NNJoint-NNJoint-Constrain-NN
S110000.96 (0.07)1.32 (0.13)0.89 (0.13)
S120000.72 (0.04)1.01 (0.08)0.76 (0.19)
S140000.57 (0.04)0.91 (0.04)0.59 (0.11)
S210003.48 (0.11)7.10 (1.18)4.29 (0.95)
S220003.05 (0.07)7.06 (1.31)3.76 (0.86)
S240002.81 (0.06)7.05 (1.29)3.56 (0.77)

The results show that (1) Joint-Constrain-NN consistently outperforms Joint-NN across all six configurations, demonstrating the benefit of inclusion constraints beyond parameter sharing. (2) The original Constrain-NN achieves the lowest mean test MSE in five of the six configurations, indicating that parameter sharing is not necessary for strong predictive performance.

W3. Examine the theoretical convergence rates.

We thank the reviewer for suggesting an empirical examination of our theoretical convergence rates. In response, we have conducted additional experiments investigating how the MSE varies with the number of target sets mm under different DAG topologies, thereby empirically examining the topology-dependent scaling predicted by the aggregated MSE bound for our constrained estimator in Equation (6). We believe that the reviewer's valuable suggestion has helped us better connect our theoretical analysis with our empirical findings. These new experimental results are presented in Appendix L.2, “Empirical Examination of the Topology-Dependent Error Bound,” of the revised manuscript.

In particular, we fix the training sample size at n=4000n=4000 and vary the number of target sets over m∈{4,16,64,256}m\in\lbrace 4,16,64,256\rbrace, considering three DAG topologies: (a) Star, (b) 2-layer DAG, and (c) Chain. To ensure a controlled comparison, we use the same nested target sets and data-generating mechanism across the three DAG structures for each mm, while varying the inclusion relations encoded by the DAG.

According to the topology-dependent aggregated MSE bound in Equation (6), the average ancestor sizes of these three structures are

\bar a_m^{\mathrm{2-layer}}=1+\frac{m}{4},\qquad \bar a_m^{\mathrm{Chain}}=\frac{m+1}{2}.$$ Consequently, **the theoretical upper bounds exhibit the ordering Chain > 2-layer DAG > Star**, with increasingly pronounced differences as $m$ grows. Importantly, our additional experiments show the same qualitative ordering in the observed test MSE, with the Chain exhibiting the fastest growth, followed by the 2-layer DAG and the Star. **As shown in Figure 4 (page 71) of Appendix L.2 in the revised manuscript, the observed MSE patterns closely follow the qualitative ordering predicted by our theoretical bounds, while the overall mean MSE values remain small.** These results provide empirical evidence consistent with the dependence of the aggregated MSE bound on DAG topology, as characterized by the average ancestor size $\bar a_m$. We consider this to be a good empirical verification of the topology dependence predicted by our theoretical convergence rates. We would also like to clarify that our theoretical convergence rates are upper bounds, rather than claims of optimal rates for DNN estimators. As discussed in Section 6, minimax lower bounds for the relevant function classes are currently available only in specific settings, such as those covered by Theorem 3 of Schmidt-Hieber (2020). Consequently, an exact match between the empirical convergence rates as the sample size $n$ increases and the corresponding theoretical upper bounds is not necessarily expected. Our additional experiments instead focus on the predicted qualitative ordering of estimation errors across DAG topologies as $m$ grows, rather than claiming that the theoretical upper bounds are tight.
Reviewer_RnGyOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the problem of distributional regression with arbitrary response variables. The proposed solution (preliminary estimator) is to fix a family of subsets of the response space with constraints on the inclusion, and predict the probability of the response belonging to each set. Monotonicity (constrained estimator) is structurally enforced by constructing a DAG according to the set inclusion relationship, and recursively computing the probability of the response belonging to a set as the maximum predicted probability of all sets included therein plus a predicted offset. The paper proposes a theoretical characterization of the estimation error when using arbitrary estimators as well as ReLU networks in the hierarchical composition class. Results on synthetic and real-world datasets show that the constrained estimator achieves better probabilistic forecasts compared to the preliminary estimator and two external baselines.

Critical strengths

Extending distributional regression to arbitrary response variables is a highly valuable contribution, the proposed method is sensible and appears relatively uncomplicated to implement. The theoretical developments substantively add to the body of work characterizing ReLU networks, even outside the context of distributional regression. The experiments are varied and convincing.

Critical weaknesses

The theoretical results treat preliminary and constrained estimators separately; despite the claims of superiority of the constrained estimator, there is no theoretical characterization of its advantage over the constrained estimator. However, I do not find hard to believe that this is the case, and the experimental evidence is clear enough.

Questions for authors

The issue of computational complexity of the constrained estimator is not considered. Assumption 1 implies that the number of edges in the graph, and thus the time complexity of computing the predictions and MSE, grows as |V|^2, potentially limiting the applicability of the method to very large and/or fine-grained response domains. A comparison of the runtimes of preliminary and constrained estimators would help clarify the penalty incurred by introducing the DAG structure.

The mathematical contributions are substantial, and the proofs very long. I appreciate the authors' effort in making the main text and appendix derivations easier to follow, but I suggest considering a Lean formalization to guarantee its correctness.

4 comments

We greatly thank the reviewer for the valuable feedback and respond to each point here. We have also updated the manuscript PDF during the rebuttal period, with all corresponding revisions highlighted in blue, and we may kindly invite the reviewer to refer to the updated PDF for these changes during our detailed responses below.

Weaknesses

We acknowledge that, for a fixed DAG and under the corresponding assumptions, the two estimators achieve comparable convergence rates (up to logarithmic factors), and these bounds do not establish a strict statistical advantage for the constrained estimator. We have explicitly acknowledged this limitation and identified it as a future research direction in the Conclusion of the updated manuscript (see the blue text at the end of page 10). Nevertheless, their theoretical guarantees address different aspects: the preliminary estimator provides uniform error control over potentially infinite set families without guaranteeing inclusion consistency, whereas the constrained estimator provides aggregated MSE bounds over finite inclusion DAGs (whose sizes may grow with nn), while ensuring inclusion consistency by construction.

We also appreciate the reviewer's positive assessment of our experimental results.

Questions

Q1. We clarify the computational scaling of Constrain-NN relative to Pre-NN. Let Cnet=O(Lw2)C_{\mathrm{net}}=O(Lw^2) denote the cost of a forward or backward pass of a single DNN. Both methods use the same mm networks with the same parameter counts.

During training, per sample and iteration, Pre-NN costs O(mCnet)O(mC_{\mathrm{net}}), while Constrain-NN costs O(mCnet+∣E∣+m)O(mC_{\mathrm{net}}+|E|+m). The additional term comes from the DAG recursion on mm scalar outputs, requiring O(∣E∣+m)O(|E|+m) operations for both forward and backward propagation. In typical settings, we expect the computational cost of each base network to be substantially larger than mm, so network computations dominate and the DAG overhead remains small. However, in extreme cases involving very large and dense DAGs, ∣E∣|E| can reach Θ(m2)\Theta(m^2), making the additional computational overhead significant. We acknowledge this potential scalability limitation. In the updated manuscript, we have included this computational complexity analysis of Pre-NN and Constrain-NN, which we refer you to Appendix K in page 69 of our updated PDF file (highlighted in blue).

Q2. We thank the reviewer for recognizing our theoretical contributions and for suggesting Lean formalization. We agree that formal verification is valuable. Although a full Lean formalization is beyond the scope of this revision, we have carefully constructed the proofs line by line. We briefly outline their key ideas.

For the preliminary estimator (Theorem 1), the challenge is uniform error control over potentially uncountable set families. Under total inclusion ordering, event indicators generate at most n+1n+1 patterns on nn observations; peeling and entropy arguments then yield uniform empirical-process control. For multivariate lower orthants, the corresponding bound is (n+1)u(n+1)^u (Corollary 1). For the constrained estimator (Theorem 2 and Equation 6), we analyze how DAG structure affects statistical complexity. Controlling error propagation and the complexity of ancestor networks yields an error bound governed by average ancestor size aˉm\bar a_m.

The most technically demanding part is our new DNN approximation theory (Appendix D), which shows that the DAG constraints need not compromise DNN approximation or estimation error rates. The key ideas are:

Obstacle. In constrain-NN, the softplus function enforces strictly increasing predictions along DAG, whereas true conditional probabilities for nested sets may coincide. Thus, standard DNN approximation guarantees do not directly apply to our estimator.

DNN Construction. Our idea is to transform ordinary DNN approximants into functions satisfying the strict DAG constraints through a small, controlled adjustment. We introduce a positive margin in the DAG recursion to correct possible ordering violations and ensure that the inverse-softplus transformation is well-defined. We further construct ReLU calibration networks approximating the logit and inverse-softplus functions, and show that, when combined with the original DNN approximants and the DAG recursion, the resulting base networks can still be represented within the prescribed DNN function class of our constrain-NN (Lemma 4, Definition 3 and Theorem 3 in Appendix D).

Approximation error. Both the positive margin and calibration networks introduce additional approximation errors. We separate these from the original DNN error and control their accumulation along DAG paths. By choosing the margin and calibration accuracies according to the longest DAG path, we keep the extra approximation errors at the same order as the original error, retaining O(ϕn)O(\phi_n) (Theorem 3 and Corollary 6 in Appendix D).

Estimation error. Controlling approximation error alone is insufficient, as the additional network constructions may increase model complexity and hence estimation error. We show that the calibration networks can be constructed with constant width and controlled depth, and that their combination with the original DNNs and DAG recursion remains within a dense ReLU function class with controlled covering entropy (Lemmas 4–6 in Appendix D), yielding our final rates.

Thank you for your response and clarifications. I have no further concerns and my rating stays at 4.

Thank you again for your helpful comments!

Comments on the paper

1 comment

We thank all three reviewers for their constructive feedback. We have addressed their concerns point by point in rebuttal and uploaded an updated PDF, with all revisions highlighted in blue. Below, we summarize the reviewers' concerns and our responses.

Reviewer RnGy (Rating: 4).

The reviewer positively assessed our theoretical contributions and empirical results while raising questions about computational complexity, proof verification, and convergence-rate comparisons. We added a detailed computational complexity analysis in Appendix K (page 69) during rebuttal and explained the key technical arguments underlying our results. We also agree that, for a fixed DAG, Pre-NN and Constrain-NN achieve comparable convergence rates up to logarithmic factors, as discussed in the revised Conclusion section (page 10). We emphasized that they provide different theoretical guarantees: uniform error control over potentially infinite set families for Pre-NN versus bounds for finite, potentially growing DAGs for Constrain-NN. Importantly, Constrain-NN guarantees inclusion consistency by construction and shows improved empirical performance across various experimental settings.

Reviewer Cbap (Rating: 2).

(1) Theoretical novelty and contributions. Comparing with existing work, we clarified our contributions extend beyond existing approaches: (i) Theorem 1 establishes uniform estimation-error bounds over potentially infinite families of measurable sets, with Corollary 1 covering multivariate lower orthants; (ii) Theorem 2 and Equation (6) establish aggregated MSE bounds characterizing inclusion-DAG topology through average ancestor size, including when the number of target sets mm grows with sample size; (iii) we develop new approximation theory for DAG-constrained ReLU networks; (iv) our general theoretical framework also extends beyond DNNs, yielding explicit convergence rates for trend filtering and orthogonal series; (v) extensive simulations and real-data applications demonstrate strong empirical performance. We also expanded Related Work (pages 2–3) to incorporate additional references and further clarify our distinctions from existing methods.

(2) Contribution of DAG constraints. The reviewer requested experiments isolating the effect of DAG constraints. We clarified that Pre-NN and Constrain-NN use identical base-DNN architectures and parameter counts without parameter sharing. We additionally conducted shared-network ablations (Appendix L.3, page 72) comparing two estimators based on a single DNN with a multi-output prediction head, differing only in whether DAG constraints are imposed. The constrained estimator outperformed its counterpart in all configurations.

(3) Empirical examination of topology-dependent error bounds. As requested, we conducted new experiments varying mm across 3 DAG topologies (Star, 2-layer graph, and Chain). The results (Appendix L.2, Figure 4, page 71) show an MSE ordering of Chain > 2-layer > Star as mm grows, consistent with the topology-dependent MSE upper bounds in Theorem 2 and Equation (6), which increase with average ancestor size.

Reviewer 5bCh (Rating: 3).

(1) Scope and theoretical comparisons. We clarified that classical distributional regression for scalar responses is a special case involving nested events, whereas our framework accommodates arbitrary measurable response spaces and general set families, including incomparable events. We expanded Related Work (page 2-3) to clarify these connections. Regarding the convergence rate comparison for the two estimators, we provided a similar clarification as in our response to Reviewer RnGy, and further discussed it in revised Conclusion section (page 10).

(2) Question on full probabilistic coherence. We clarified that Constrain-NN guarantees inclusion consistency rather than full probabilistic coherence, and explicitly acknowledged this distinction in the revised Conclusion section (Section 6, page 10). Our theoretical objective is to give estimation guarantees for prescribed conditional event probabilities rather than enforce all probability constraints. We also provided empirical checks during the rebuttal, showing that violations of the examined coherence constraints are rare and decrease substantially with sample size.

(3) Additional empirical comparisons and statistical evaluation. We added isotonic projection and C-HMCNN baselines and performed hyperparameter sensitivity analyses (Appendix L.1, pages 69–70), showing that our main conclusions remain unchanged. We also conducted shared-network ablations (Appendix L.3, page 72), as discussed in our response to Reviewer Cbap. Additionally, we provided paired comparisons with confidence intervals across the real-data experiments during rebuttal, further supporting the consistent performance of our estimator.

We thank all reviewers again for their feedback, which greatly improved our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.