- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes distributional regression for general response spaces by estimating conditional event probabilities over a user-chosen family of measurable sets. A preliminary estimator fits a separate least-squares regression of 1{Y ∈ A} on X for each set. A constrained estimator encodes inclusions among a finite family of sets as a DAG and parametrizes each set's logit as the maximum of its parents' logits plus a softplus increment, which enforces monotonicity under inclusion by construction. The theory provides uniform error bounds for the preliminary estimator over totally ordered families and lower orthants, aggregated bounds for the constrained estimator in terms of DAG quantities, and a new ReLU approximation result under hierarchical composition assumptions. Experiments on simulated, image, and real data (Beijing air quality, Chicago crime) show the constrained estimator outperforming Pre-NN, post-hoc clipping, DRF, and Engression in most settings.
Critical strengths
-
A simple, practical construction that guarantees monotone predictions. Each set's logit is the maximum of its parents' logits plus a softplus increment, followed by a sigmoid. This guarantees monotonicity under inclusion for every input and every parameter value, adds essentially no computational cost, and fits naturally with standard neural networks. On the simulations, it substantially reduces error relative to fitting each set separately, and the gains are clearly significant. The idea is easy to adopt in applications involving nested events, such as the joint pollutant-exceedance events in the Beijing study, and is the main reason I see value in the paper.
-
A comprehensive and, as far as I can tell, technically sound theoretical treatment (uniform bounds for the preliminary estimator, aggregated bounds for the constrained estimator, and an approximation result for the constrained parametrization). I did not identify errors in the main arguments; my concerns are about scope and interpretation rather than correctness.
Critical weaknesses
- The "arbitrary response" framing overstates the scope, and beyond the scalar case the inclusion constraint does not capture full probabilistic coherence.
(a) Scope. The method accesses only through membership in pre-chosen sets, and its benefit comes entirely from inclusions among them. A chain is equivalent to thresholds of a single scalar statistic: setting gives , so estimating is exactly estimating the conditional CDF of , regardless of the nature of . More generally, all experiments use events defined through a few scalar statistics of (e.g., transitivity in S2, in S4, the latent coordinates in S1, pollutant concentrations in Beijing). This is classical distribution regression applied to hand-picked summaries of (Foresi & Peracchi, 1995; Chernozhukov, Fernández-Val & Melly, 2013), where monotonicity across thresholds is a known issue with established remedies (rearrangement, Chernozhukov, Fernández-Val & Galichon, 2010; cumulative-link models, McCullagh, 1980). On a chain, the proposed parametrization essentially reduces to a cumulative-logit model with covariate-dependent thresholds. This literature is not discussed.
(b) Coherence. For a chain, inclusion-monotonicity is equivalent to full probabilistic coherence: any vector is attainable by some distribution. This fails in general beyond chains. For lower orthants with , , and , the outputs are inclusion-consistent but imply . Similarly, in S5 the sets and are disjoint subsets of , so coherence requires , which the method does not enforce. In contrast, DRF outputs a weighted empirical distribution and is coherent by construction. In other words, the constraint is complete only when the problem is scalar. I would encourage the authors to reframe the contribution accordingly, as monotone multi-event distributional regression on chosen summaries of Y, and to position it relative to the distribution-regression literature.
- The paper does not establish that the constraint itself drives the empirical gains.
The natural post-hoc baseline, projecting Pre-NN outputs onto the monotone cone , is missing. Both the true probability vector and every label vector lie in , and projection onto a closed convex set is non-expansive, so this baseline can never increase the aggregated MSE, against the truth or against the labels. The reported degradation from post-hoc correction is thus specific to the clipping rule used. There is also no ablation separating monotonicity from parameter sharing: each Constrain-NN output depends on all its ancestors' networks, so a shared multi-head network may account for much of the gain. Finally, on Chicago the reported gap (0.0008) is about one unpaired standard error. Adding the projection baseline and a shared-network ablation would make the empirical case for the constraint much more convincing.
Questions for authors
-
Is there any regime in which your bounds for the constrained estimator are strictly better than those for the preliminary estimator? If not, how do you explain the empirical gains?
-
Were hyperparameters or the number of epochs tuned for any method? Is the dispersion reported in the tables a standard deviation or a standard error, and can you report paired comparisons for S3, Beijing, and Chicago? Is the real-data metric the Brier score against indicators?
-
How does Constrain-NN compare to (a) Pre-NN followed by isotonic projection onto the DAG cone, and (b) a single shared network with m sigmoid heads, with and without projection?
-
On the box-type scenarios and on Beijing, how often do Constrain-NN's outputs violate the rectangle inequality or additivity constraints, compared with DRF?
6 comments
1(b). Inclusion Consistency versus Full Probabilistic Coherence
We thank the reviewer for pointing out this important distinction. We acknowledge that our proposed constraints enforce inclusion monotonicity over a partially ordered family of measurable sets, rather than full probabilistic coherence. We have explicitly acknowledged this limitation in the Conclusion of the updated PDF file (see the blue text at the end of page 10).
However, full probabilistic coherence is a stronger requirement than inclusion consistency, and satisfying this requirement is not equivalent to improving estimation accuracy for a specified collection of target events. When the objective is to estimate conditional probabilities for a prescribed family of target events, our approach directly estimates these probabilities while respecting their known inclusion relationships, without first constructing a complete conditional probability distribution over the response space. This direct estimation strategy is particularly meaningful when the response space is complex or when researchers are interested in only a specific collection of measurable events. Correspondingly, our theoretical guarantees concern the statistical accuracy of these event-probability estimators under the stated assumptions, rather than claiming that they satisfy full probabilistic coherence.
We also acknowledge that distributional random forests (DRF, Cevid et al. (2022)) guarantee full probabilistic coherence by construction through a weighted empirical distribution. This is a valuable property. However, probabilistic coherence does not necessarily imply stronger statistical convergence guarantees or better empirical performance, and the theoretical result in DRF does not provide an explicit convergence rate comparable to ours. As summarized in Appendix G (Table 4, page 57) of our submission, DRF only establishes consistency property in the form of for conditional CDF estimation, whereas our theoretical analysis further establishes explicit convergence-rate bounds for the MSE of the target event-probability estimators. Thus, our contribution concerns a different aspect of statistical estimation performance, rather than claiming to achieve full probabilistic coherence.
Moreover, within the classical set-indexed distributional regression framework considered in our paper, enforcing full probabilistic coherence for general multidimensional event families is a different and potentially more demanding task than enforcing inclusion monotonicity. In particular, methods such as isotonic projection and C-HMCNN (Giunchiglia and Lukasiewicz, 2020) also rely on inclusion constraints without guaranteeing all probabilistic coherence conditions among arbitrary overlapping or disjoint events. Please see the continuation of our response below.
1(b). (Continued) Inclusion Consistency versus Full Probabilistic Coherence
In the following, we further introduce an alternative DNN-based estimation strategy for classical set-indexed distributional regression that guarantees full probabilistic coherence. Specifically, we can construct the mutually disjoint partition induced by the target events, estimate the conditional probabilities of its atoms, and obtain each requested event probability by summing the probabilities of the corresponding atoms. This differs fundamentally from Constrain-NN, which directly estimates only the probabilities of the specified measurable target sets while enforcing their inclusion relationships. For overlapping target events, the common disjoint refinement may contain exponentially many atoms relative to the number of target events, potentially increasing the statistical and computational complexity of estimation. We also emphasize that an atomic partition is one way to achieve full probabilistic coherence, rather than the only possible approach.
To illustrate this distinction empirically, we conducted an additional experiment comparing Constrain-NN with an Atom-Softmax estimator that guarantees full probabilistic coherence. We consider a correlated binary response , where is an even integer. For each observation, we generate and a shared noise variable , and then sample
\sim \operatorname{Bernoulli}\left( \tau\left(X^\top\beta_j+b_j+0.75\epsilon\right) \right), \qquad j=1,\ldots,d,$$ independently across $j$ conditional on $(X,\epsilon)$, where $\tau(t)=(1+e^{-t})^{-1}$ and $(\beta_j,b_j)$ are randomly generated and then held fixed. We then define the target sets whose probabilities we aim to estimate. First, we define $d$ leaf events: $$A_j=\{Y_j=1\}, \qquad j=1,\ldots,d.$$ We then construct a two-layer event hierarchy by adding $d/2$ pairwise unions: $$B_k=A_{2k-1}\cup A_{2k}, \qquad k=1,\ldots,d/2.$$ Thus, the target sets estimated by **Constrain-NN** consist of only $m=3d/2$ events and form a two-layer DAG through the inclusion relations $A_{2k-1},A_{2k}\subseteq B_k$. In contrast, because each of the $d$ binary coordinates can take either zero or one, the complete disjoint partition of the response space contains $2^d$ binary patterns. Therefore, **Atom-Softmax** models the conditional probabilities of all $2^d$ mutually disjoint patterns using a softmax output and obtains each requested event probability by summing over the corresponding patterns. We compute the test probability MSE by averaging the squared errors over test observations and the $m=3d/2$ target events, then average this quantity over 20 repetitions. We report $100$ times the resulting average. **Table: Mean test probability MSE over the target events $(\times 100)$, across 20 repetitions** | Dimension $d$ | Number of target sets $m$ | Number of atoms $2^d$ | Constrain-NN | Atom-Softmax | |---:|---:|---:|---:|---:| | 4 | 6 | 16 | 2.127 | 2.461 | | 6 | 9 | 64 | 2.255 | 2.729 | | 8 | 12 | 256 | 2.241 | 2.657 | | 10 | 15 | 1024 | 2.132 | 4.211 | | 12 | 18 | 4096 | 2.285 | 6.385 | | 14 | 21 | 16384 | 2.206 | 36.721 | As shown in the table, Constrain-NN achieves lower aggregated MSE than Atom-Softmax across all examined dimensions, and the performance gap becomes particularly pronounced as the number of atoms increases. These results illustrate that, when the objective is to estimate a prescribed collection of target event probabilities, naively enforcing full probabilistic coherence through the atomic-partition approach examined in our experiment does not necessarily improve estimation accuracy.
We thank the reviewer for the careful evaluation of our paper and for the many constructive comments. We have carefully considered all the concerns raised and conducted additional experiments to strengthen our work. Below, we consolidate the concerns raised in the Weaknesses and Questions into related topics and address each of the reviewer's comments and questions in detail. We have also updated the manuscript PDF during the rebuttal period, with all corresponding revisions highlighted in blue, and we may kindly invite the reviewer to refer to the updated PDF for these changes during our responses.
Weaknesses
1(a). Scope and connections to classical distributional regression
(1) Connections to existing work. We agree that our work builds on classical distributional regression, particularly the event-based approaches of Foresi and Peracchi (1995) and Chernozhukov et al. (2013). As already stated in the Abstract and discussed in the Related Work section of our original submission (Section 1.2), our work focuses on conditional event probabilities indexed by measurable sets and builds on the classical distributional regression framework; we do not claim that regressing event indicators is itself a new idea. We also acknowledge that, when the target sets form a nested chain, the corresponding conditional probabilities can be represented as the conditional CDF of a scalar statistic of the response, connecting to cumulative-link models (McCullagh, 1980). Likewise, monotonicity correction through rearrangement has been studied by Chernozhukov et al. (2010).
However, unlike a nested chain, a general event family containing incomparable sets cannot be represented as threshold events of a single scalar statistic while preserving its inclusion structure. Our framework considers conditional probabilities indexed by measurable subsets of an arbitrary response space, including multivariate lower-orthant families and more general inclusion DAGs. Such event families cannot, in general, be represented as threshold events of a single scalar statistic while preserving their full inclusion relationships. Thus, although classical scalar distributional regression is an important special case of our framework, the estimation problems we consider are not restricted to this setting.
We thank the reviewer for identifying these additional references. We have now cited all the relevant papers identified by the reviewer, revised the Related Work section accordingly, and further clarified the connections between classical distributional regression and our estimation framework (see the text highlighted in blue on pages 2–3 of the revised manuscript). We have also clarified this scope in the Introduction of the updated manuscript (in blue), emphasizing that our goal is to estimate conditional probabilities over a chosen family of measurable events rather than to construct the full conditional distribution.
(2) Statistical and theoretical contributions. We agree that the existing literature has established important theoretical foundations for distributional regression and monotonicity-constrained estimation. For example, Chernozhukov et al. (2013) develop functional limit theory and statistical inference for scalar distribution regression and counterfactual distributions, while Chernozhukov et al. (2010) establish finite-sample improvement properties and asymptotic theory for monotone rearrangement. However, these results address statistical questions that differ from those studied in our paper.
Our main contribution is not to introduce event-indicator regression or monotonicity constraints as new ideas, but to establish new statistical estimation-error bounds and convergence guarantees for conditional event-probability nonparametric estimators over general measurable set families. In particular, we investigate uniform estimation errors over infinitely many target sets (Theorem 1) and the dependence of estimation risk on DAG topology for general inclusion structures (Theorem 2 and Equation (6)). For a detailed explanation of the novelty of these theoretical results, the associated technical challenges, and their distinctions from existing statistical theory, we refer the reviewer to our response to Reviewer Cbap, W1 (Novelty). In addition, we conduct various empirical evaluations across nine experimental settings (including one real-data-based toy example) and two additional real-data applications. These results demonstrate the strong empirical performance of our proposed constrained estimator across diverse response types and inclusion structures.
Furthermore, Appendix G (Table 4, page 57) of our original submission already provides a systematic comparison of the assumptions, estimation settings, and convergence guarantees of our results with those of the most closely related statistical works.