acceptodds
Under review as a conference paper at ICLR 2027

One Proper Score for Many Decisions: Sharp Regret Transfer under Misspecification

Abstract

Under model misspecification, minimizing different proper scoring rules can induce different downstream decisions. We characterize the smallest worst-case target-risk ratio achievable by minimizing one fixed proper score across populations and report classes, relative to the best predictor in each class. The target is the contextwise maximum of declared task regrets, augmented by a strictly proper baseline divergence. We show that the optimal risk ratio equals the smallest two-sided distortion between the target and a proper-score divergence. For finite-action tasks, decision boundaries force curvature charges that yield exact distortion for an orthogonal family of two-action tasks on outcomes, with . Independent shared components with common tie selections admit an exact covering formula that quantifies the benefit of shared geometry. Its optimal weights yield a baseline-augmented task-score sum that attains the optimum over all proper scores. We also derive exact binary and quadratic characterizations and a sharp local trade-off between smoothness and additive regret error at hard boundaries. Known-truth studies illustrate the distinction between worst-case guarantees and achieved projection risk.

Primary area:
learning theory
Keywords:
proper scoring rules, surrogate regret, model misspecification, decision theory, convex analysis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_QYC7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies how to choose a single proper scoring rule for multiple downstream decision tasks under model misspecification. Its objective is to minimize the worst-case target-risk ratio relative to the best predictor in the same report class, where the target is the maximum over task regrets.

Theorem 2.2 characterizes this ratio through pointwise divergence distortion, but the theorem is more like a rephrase of the objective than providing intuitions. The paper then gives characterizations and score constructions for several special cases: orthogonal two-action tasks, independent shared components, and binary outcomes. It also provides learning guarantees that transfer score-minimization error into target risk and analyzes smooth proxies gradient-based optimization.

Experiments test the proposed mechanism against fixed proper scores, but the real-data results do not prove the effectiveness over standard scoring rules.

Critical strengths

I think the problem itself is interesting. The characterizations and the proposed constructions are mathematically interesting too. It provides a useful perspective on how to combine difference decision objectives.

Critical weaknesses

I am mainly concerned with the motivation of the paper.

  • Theoretical motivation. The results are interesting, however, the paper would benefit from more explanation / motivation on why minimizing one fixed proper score is the right restriction for several downstream users. The related work section cites the omniprediction literature. The authors recognize there is a distinction between one aggregate score, and learning for omniprediction, but stating this difference doesn't fully motivate the proposed theoretical problem or approach. What practical constraints justify the restriction?

  • Empirical evidence seems weak. It does not show the proposed method is a useful improvement.

Moreover, the presentation needs revision. The paper introduces many closely related quantities, but I think the connections could be explained more clearly. I'll name several here: baseline divergence (also, why is it important?), training score divergence, target loss, pointwise ratio constants, etc. The paper also refer to ealier equations very often by equation number. This requires readers to move back and forth to understand the definitions. Many of the mathematical characterization remains unexplained. Theorem 2.2 is extremely hard to read. The several later special cases could be described as "computational results". The "operational equivalence" of theorem 2.2 is also ambiguous.

Questions for authors

I'd appreciate responses to my questions above.

1 comment
AuthorsOpenReview ↗

Author Response to Reviewer QYC7

We thank Reviewer QYC7 for the positive assessment of our formulation and characterizations (Soundness: 3). Below, each paragraph addresses one concern (W1, W2, W3) with direct manuscript evidence.

1. Theoretical Motivation & Relation to Omniprediction (W1 — §§1–2, pp. 1–3; §§6–7, p. 10; Gen. Resp. §1). As detailed in Gen. Resp. §1 and revised in §§1–2, §§6–7, our setting is loss design within an existing probability-prediction pipeline: retaining the model class H\mathcal{H} and probability-report interface while redesigning one fixed training score for a declared task family (the predictor qh(X)q_h(X) remains context-dependent). Unlike finite-action objectives that leave action-equivalent reports indistinguishable, strict propriety elicits truthful probabilities when realizable while directing the misspecified approximation in H\mathcal{H} toward critical decisions (without improving every accuracy metric); statistically, E[SF,f(qh(X),Y)∣X]=DF,f(pX,qh(X))−F(pX)\mathbb{E}[S_{F,f}(q_h(X),Y)\mid X]=D_{F,f}(p_X,q_h(X))-F(p_X) yields an observable label-based criterion without estimating pXp_X, whereas taking the maximum of realized task losses targets a different objective from the contextwise maximum of conditional regrets. §6 clarifies that omniprediction can also serve a shared prediction to downstream decisions, including for proper losses (Gopalan et al., 2022; Gibbs & Tibshirani, 2026)—that shared-report feature is not our distinction. Rather, our restriction is the learning procedure: minimizing one fixed proper score within a prescribed class H\mathcal{H}, so our sharp bounds determine both when redesigning the training score suffices and when a more flexible learning approach (e.g., omniprediction or per-user heads) is needed.

2. Experimental Modifications & Empirical Results (W2 — §§4–5, Tables 2–4, Figs. 2–3, App. M; Gen. Resp. Parts 1–2). Replacing 1–3 parameter calibration (kept in Appendices K–L), we added a three-tier suite backed by Prop. 4.4's O(h)O(h) proper-smoothing / facet-flux gradient bridge (§4): (i) a 500-seed 4-outcome study under exact ETε\mathbb{E}T_\varepsilon (§5.1, Table 2), (ii) Adam/L-BFGS-B feature-MLP training on 7 synthetic mechanisms and 4 NOAA holdouts (§5.2, Fig. 2, Table 3), and (iii) an 821,761-parameter 64→5124→164\to 512^4\to 1 tanh⁡\tanh MLP (§5.3, Table 4, Fig. 3). Results show: (a) in §5.1 (n=50,000n=50{,}000, 10−310^{-3} units), hard Join (11.79011.790) lowers target risk by 44.8% vs. Log (21.36521.365), 3.15% vs. Oracle best single (12.17412.174), and 2.87% vs. LP-verified Oracle weighted envelope (12.13912.139, beating Sum 12.38612.386 and all nonnegative task weights) under simultaneous 95% Bonferroni CIs across 140 contrasts; (b) in §5.2, Join lowers 3-seed mean risk vs. Log in 4/5 misspecified settings (8/10 pairs) and vs. Best single in 11/12 pairs, and attains the lowest mean cost CfixedC_{\mathrm{fixed}} on 4-class NOAA temporal holdout under both optimizers (L-BFGS-B: 0.29733±0.000500.29733 \pm 0.00050, diff vs. Log −0.00229±0.00035-0.00229 \pm 0.00035; Adam: 0.29735±0.000170.29735 \pm 0.00017) plus −5.4-5.4\\% vs. Log on binary station L-BFGS-B (Log/Brier/Sum lead on controls/other splits); (c) in §5.3, Adam with checkpointing scales cleanly to 821,761 parameters, where task-aware scores achieve lower 3-seed means than Log/Brier on Oscillatory interaction (Log/Brier lead on Shallow teacher), and Fig. 3 shows checkpointing prevents boundary overfitting under both Adam and L-BFGS.

3. Exposition Overhaul: Notation Table, Baseline Roles, Theorem 2.2 & §3 Framing (W3 — §§1–4, Table 1, Table 12). We revised the exposition around your four suggestions: (i) Consolidated object guide (Table 1, §2, p. 3): placed before Thm. 2.2, Table 1 defines TεT_\varepsilon, DF,fD_{F,f}, (a,b)(a,b), ρH,P≤κH,P≤Γ(D,Tε)=b/a\rho_{\mathcal{H},P} \le \kappa_{\mathcal{H},P} \le \Gamma(D,T_\varepsilon)=b/a, κε⋆\kappa_\varepsilon^\star, FhF_h, and g~hflux\tilde g_h^{\mathrm{flux}} in one place; (ii) Three roles of εDF0\varepsilon D_{F_0} (§1, p. 1; §2, p. 3; §4, p. 7): within-cell probability identification, positive denominator Tε(p,q)>0T_\varepsilon(p,q) > 0, and smooth within-cell gradients; (iii) Theorem 2.2 renaming & intuition (§2, pp. 3–4): renamed “Worst-case projection distortion equivalence” with pre-theorem intuition showing how cross-context pricing [a,b][a,b] inverts target ordering by nearly b/ab/a between two in-class predictors; and (iv) Constructive §3 framing (§§3–4, pp. 4–7; Table 12, p. 38): §3 presents the transversal bound, shared-component LP, binary tie-charge join, and quadratic ellipsoid as computable characterizations, linked to training by Prop. 4.4 and Table 12.

Reviewer_vQfQOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks which single proper scoring rule a learner should minimize so that, when the report class is misspecified, the worst-off of several downstream decision-makers is within a constant factor of the best predictor in the class, uniformly over populations and classes. The target is the contextwise maximum of task regrets plus a small proper baseline divergence. The paper shows that the worst-case risk ratio equals the tightest pointwise sandwich T≤D≤κTT \le D \le \kappa T between the score’s divergence and the target. A decision boundary forces singular curvature on any dominating entropy, and curvature on distinct boundaries adds while the target takes a max. This gives the exact factor mm for mm transversal two-action tasks and a covering LP for tasks that share boundaries.

Critical strengths

The multi-task lower bound is new. No proper score can protect mm transversal users to within a factor better than mm.

The shared-boundary case is exact. The covering LP gives the factor, the optimal weights, and the optimal score.

Critical weaknesses

To be honest, I am not an expert on this topic, but I did search for related literature to understand the landscape better.

I have few comments to make here: What is the relationship between Theorem 2.2 and the two-point construction in Bartlett, Jordan and McAuliffe (2006)? Frongillo and Waggoner (2021) is cited, but that paper seems to rule out linear transfer for smooth surrogates by the same linear-versus-quadratic crossing argument, and the infinite factors in Table 3 appear to follow from it. Theorem 3.6 looks like a consequence of the weight-function representation in Reid and Williamson (2009, ICML) [which is not cited]. Which results of the authors are original vs derivative?

Further, I am not able to understand modeling choices made by the authors: the target uses a max over users rather than a sum or an average; contextwise rather than fixed-user aggregation; multiplicative rather than additive comparison; and a worst case over all populations and classes. How do these modeling choices impact the nature of results? Finally, do the experiments show that these modeling choices matter?

Questions for authors

Please see my questions above. Further:

What deployment constraint rules out training directly on the aggregate decision cost, or fitting per-user correction heads?

Why the contextwise max EXmax⁡jRjE_X \max_j R_j rather than the fixed-user max max⁡jEXRj\max_j E_X R_j?

Can you exhibit one population where the join or the weighted sum beats log and beats the best single-task score by a margin outside the paired intervals?

The reproducibility statement says generative AI assisted with proofs. Which proofs?

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer vQfQ

We thank Reviewer vQfQ for highlighting Theorems 3.3 (κε⋆=m\kappa_\varepsilon^\star=m) and 3.5. Below, each paragraph addresses one part of your review (W1, W2/Q1–Q2, Q3, Q4) with direct manuscript evidence.

1. Classical Tools vs. Original Multi-Task Contributions (W1 — §§2–3, pp. 4–6; §6, p. 10). Classical surrogate and proper-score results analyze single-target Bayes calibration or unselected convex entropies, whereas our results characterize in-class multi-task regret aggregation with explicit tie selections (credited alongside Bartlett et al., 2006, Frongillo & Waggoner, 2021, and Reid & Williamson, 2009 in §§2–3, §6). Unlike two-point mixtures relative to the unrestricted Bayes optimum (Bartlett et al., 2006), Theorem 2.2 compares two predictors inside the same misspecified class H\mathcal{H} to prove the sharp in-class ratio b/ab/a, even with a unique score minimizer. Likewise, while Lemma 3.2 isolates the single-facet Θ(t)\Theta(t) vs. O(t2)O(t^2) smooth-surrogate mismatch of Frongillo & Waggoner (2021), our main results (Theorems 3.3 & 3.5) show how singular facet charges interact across tasks—adding on transversal facets to force κε⋆=m\kappa_\varepsilon^\star=m despite max⁡j\max_j, yet collapsing to an exact covering LP over all proper scores under shared components. Finally, whereas 1D curvature measures determine convex entropies only up to unselected kinks (Reid & Williamson, 2009), Theorem 3.6 tracks directional tie charges (ℓj,c,rj,c)(\ell_{j,c},r_{j,c}), explaining why two hinges α(q−c)+\alpha(q-c)_+ need charge α\alpha under shared ties but 2α2\alpha under opposite ties.

2. Interface Motivation & Modeling Choices (W2, Q1, Q2 — §§1–2, Table 1, §§6–7, App. M.3; Gen. Resp. §1). As detailed in Gen. Resp. §1 (§§1–2, §§6–7), our scope is loss design within an existing probability-prediction pipeline: retaining H\mathcal{H} and the probability-report interface while redesigning one fixed score (Q1), identifying when label-score ERM (E[SF,f(qh(X),Y)∣X]=DF,f(pX,qh(X))−F(pX)\mathbb{E}[S_{F,f}(q_h(X),Y)\mid X]=D_{F,f}(p_X,q_h(X))-F(p_X)) suffices and when per-user heads, plug-in estimation, or omniprediction are needed. Each modeling choice shapes this analysis (W2, Q2): (i) max⁡j\max_j vs. sum/average: with a common baseline, ∑jλjRj\sum_j \lambda_j R_j is already a Bregman divergence with factor 11, whereas max⁡j\max_j creates the curvature-addition tension of Thms. 3.3–3.5; (ii) Contextwise Bctx=EXmax⁡jrj(X)B_{\mathrm{ctx}}=\mathbb{E}_X\max_j r_j(X) vs. fixed-user Bfixed=max⁡jEXrj(X)B_{\mathrm{fixed}}=\max_j\mathbb{E}_X r_j(X) (Q2): BctxB_{\mathrm{ctx}} lets the bottleneck task vary with XX (Bfixed≤Bctx≤mBfixedB_{\mathrm{fixed}} \le B_{\mathrm{ctx}} \le m B_{\mathrm{fixed}}), and App. M.3 (Table 9) shows an explicit reversal where the component projection (θ=3/4\theta=3/4) attains lower BctxB_{\mathrm{ctx}} (0.011250.01125 vs. 0.012500.01250; ETε\mathbb{E}T_\varepsilon: 0.0114600.011460 vs. 0.0126380.012638) than the task-weighted projection (θ=1/4\theta=1/4) despite higher BfixedB_{\mathrm{fixed}} (0.011250.01125 vs. 0.010000.01000); (iii) Multiplicative vs. additive: scale-invariant relative degradation for RT⋆>0\mathcal{R}_T^\star > 0 (strict propriety covers RT⋆=0\mathcal{R}_T^\star=0; additive bounds capture smoothing slack Δh\Delta_h in §4); (iv) Supremum over (P,H)(P,\mathcal{H}): prior-free score design, separated from κH,P,ρH,P\kappa_{\mathcal{H},P},\rho_{\mathcal{H},P} in Table 1 (§2).

3. Statistically Supported Separation Outside Paired Intervals (Q3 — §5.1, Table 2, App. M.3; Gen. Resp. Part 2). Yes: Our 4-outcome, 2-context misspecified benchmark evaluates exact ETε\mathbb{E}T_\varepsilon at n=50,000n=50{,}000 over 500 paired seeds (§5.1, Table 2; Gen. Resp. Part 2). In 10−310^{-3} units (with simultaneous 95% Bonferroni CIs over all 140 comparisons), hard Join (11.790±0.02411.790 \pm 0.024) beats Log (21.36521.365; diff −9.575-9.575, −44.8-44.8\\%, [−9.661,−9.490][-9.661,-9.490]), Task 1 (−0.523-0.523, [−0.622,−0.423][-0.622,-0.423]), Task 2 (−0.612-0.612, [−0.707,−0.518][-0.707,-0.518]), Oracle best single (12.17412.174; −0.384-0.384, −3.15-3.15\\%, [−0.478,−0.290][-0.478,-0.290]), and the LP-verified Oracle weighted envelope (12.13912.139; −0.349-0.349, −2.87-2.87\\%, [−0.442,−0.255][-0.442,-0.255]), certifying an advantage over Sum (12.38612.386) and all nonnegative task weights at n=50,000n=50{,}000 (at n=500n=500, Oracle best single is lower). In 3-seed neural studies (§5.2, Fig. 2), Join also lowers mean risk vs. Log in 8/10 misspecified pairs and vs. Best single in 11/12 pairs.

4. Generative AI Assistance in Proofs (Q4 — AI Use Statement, p. 11). The AI Use Statement (p. 11) specifies full scope: generative AI assisted proof development and verification for all theoretical results across the main text and appendices (including original theorems and Prop. 4.4); the authors checked all proofs and take full responsibility.

Thank you for your replies. Some of my major concerns were addressed. The remaining concerns may not fixable in a rebuttal. I am raising my score.

Reviewer_JjRmOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how well a single proper scoring rule, minimized within a misspecified report class, can simultaneously protect several downstream decision tasks.

Critical strengths
  1. Theorem 2.2 reduces a minimax population problem to a purely pointwise quantity, with a matching converse that survives unique-minimizer perturbations.

  2. the curvature-charge viewpoint (Lemma 3.2, distributional Hessian measures on decision facets) is technically appealing, and the covering LP gives both the exact factor and optimal weights that provably attain the optimum over all proper scores, not only over task sums.

Critical weaknesses
  • On real data, the task-aware scores never significantly outperform log loss: in the binary study, the paired intervals for the primary 1 mm worst-cost comparisons include zero (Section 5, Table 5), and in the four-class study the primary contrast is −0.000486 with a paired 95% interval [−0.002937, 0.001977] that includes zero (Section 5, Appendix L). The only interval excluding zero (5 mm, station holdout) is one among 48 unadjusted contrasts. Since the paper's selling point is a certified guarantee for the declared target, it should either identify at least one regime in which the optimally weighted task sum or selected join yields a statistically supported improvement under that target, or state explicitly what a practitioner should conclude when the certificate does not translate into observed gains.
  • The target takes a maximum over tasks whose loss vectors may live on different scales or units, so the declared positive weights encode a substantive modeling decision about cross-task comparability. The theory then absorbs these weights into the coefficient matrix A (Section 3), and the experiments simply fix unit task weights (Appendix K.1). There is no guidance on choosing lambdalambda, no sensitivity analysis, and no discussion of scale-invariant formulations, even though the shape of the covering LP in (11) — and hence the value of the certified factor — depends directly on the relative scaling of the tasks. Rescaling one task's loss vectors changes A, changes the optimal weights, and changes the certificate, yet this dependence is never discussed. Please address how regrets in different units should be normalized before entering the maximum, or clarify that the framework presumes this problem has already been solved externally.
  • The optimal uniform factor (Eq. 2) is a supremum over populations and report classes with adversarial selection among score minimizers. The resulting certificates are correspondingly extreme: infinite for log and Brier, and m for an orthogonal task family. The paper's own experiments show that these numbers do not materialize on concrete problems — achieved ratios are near one for the task-aware constructions, and even for log and Brier the class estimates (357.06 and 46.37) and achieved ratios (33.40 and 5.77) sit far below the infinite global factors (Table 3). A guarantee whose tightness witnesses are adversarial two-context populations conveys little information for any fixed application, and the abstract's framing ("the smallest worst-case target-risk ratio achievable") invites reading these numbers as practical guidance.
  • The paper proves that hard task increments contribute no within-cell gradient (Proposition 4.3), derives a population facet-flux formula as the theoretical remedy (Appendix J), and quantifies the smoothing deficit (Theorem E.2) — but the experimental pipeline then restricts to calibration heads with at most three parameters, fitted by grid search and derivative-free differential evolution (Appendix K.1). The flux formula is never used algorithmically, and no smoothed objective is ever trained by gradient methods. As a result, the experiments provide no evidence that the recommended scores can be fitted in the high-dimensional regimes (e.g., neural networks) that the paper motivates, and the conclusion itself acknowledges that scaling "calls for smoothed objectives with the stated approximation slack or scalable estimators of the population facet flux".
Questions for authors

My understanding is that the paper's first problem is that its experimental results are not good. The second is that it does not solve the comparability of regrets across tasks with different units or scales — instead, this commensurability has to be declared in advance. The third is that the 'worst-case factor' the paper defines is an extremely conservative, 'universe-level worst case': the alarming headline numbers it produces (such as ∞ or m) typically do not occur in real, concrete application scenarios, so using these numbers directly to guide practical decisions conveys very little information. The fourth is that the paper proves theoretically that 'hard tasks' have no local gradient (and therefore cannot be optimized with ordinary gradient descent) and offers a theoretical remedy (the facet-flux formula), yet the experiments — in order to produce results — resort to the shortcut of 'low-dimensional parameters plus derivative-free search', completely bypassing the core difficulty. These experiments fundamentally cannot validate whether the theory is usable in realistic high-dimensional settings such as neural networks.

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer JjRm

We thank Reviewer JjRm for recognizing Theorem 2.2, Lemma 3.2, and Theorem 3.5. Below, each paragraph addresses one concern (W1/Q1–W4/Q4) with direct manuscript evidence.

1. Empirical Gains Under TεT_\varepsilon & Practitioner Guidance (W1, Q1 — §4, §§5.1–5.2, Tables 2–3, Fig. 2; Gen. Resp. Part 2). We now provide both requested items: strict simultaneous-CI separation under exact ETε\mathbb{E}T_\varepsilon, and an explicit validation rule when finite-sample slack dominates κ\kappa. Replacing 1–3 parameter calibration, §5.1 (Table 2) evaluates exact ETε\mathbb{E}T_\varepsilon over 500 paired seeds (n=50,000n=50{,}000; units 10−310^{-3}, simultaneous 95% Bonferroni CIs across 140 contrasts): hard Join (11.79011.790) beats Log (21.36521.365; −44.8-44.8\\%, [−9.661,−9.490][-9.661,-9.490]), Oracle best single (12.17412.174; −3.15-3.15\\%, [−0.478,−0.290][-0.478,-0.290]), and the LP-verified Oracle weighted envelope (12.13912.139; −2.87-2.87\\%, [−0.442,−0.255][-0.442,-0.255], beating Sum 12.38612.386 and all nonnegative task weights), while trailing Oracle best single at n=500n=500. In §5.2 (Adam/L-BFGS-B MLPs, 3 seeds; Fig. 2, Table 3), Join lowers mean risk vs. Log in 4/5 misspecified settings (8/10 pairs) and vs. Best single in 11/12 pairs, and attains the lowest mean cost CfixedC_{\mathrm{fixed}} on 4-class NOAA temporal holdout under both optimizers (L-BFGS-B: 0.29733±0.000500.29733 \pm 0.00050, diff vs. Log −0.00229±0.00035-0.00229 \pm 0.00035; Adam: 0.29735±0.000170.29735 \pm 0.00017) plus −5.4-5.4\\% vs. Log on binary station L-BFGS-B (SEs reflect seed variation on shared holdouts; Brier/Sum/Log lead on controls and other splits). §4 (p. 8) gives the practitioner rule: since RT(h^h)≤κRT(hT)+estn+optn+Δh(h^h)+Δh(hT)\mathcal{R}_T(\widehat h_h) \le \kappa\mathcal{R}_T(h_T)+\mathrm{est}_n+\mathrm{opt}_n+\Delta_h(\widehat h_h)+\Delta_h(h_T), κ\kappa alone does not rank fitted models—build Join via the LP (with Log-warm cutting cold Join risk 42.4%, App. M.4) and validate among {Join, Sum, Log, Brier, single-task} on the deployment metric.

2. Cross-Task Units & Priority Sensitivity (W2, Q2 — §1, p. 2; §3, p. 5; App. M.1). Unit conversion is strictly invariant via λj=wj/sj\lambda_j=w_j/s_j, and priority shifts admit an exact sensitivity formula. Setting λj=wj/sj\lambda_j=w_j/s_j separates scale sj>0s_j > 0 in task jj's units (Rj/sjR_j/s_j dimensionless) from priority wj>0w_j > 0 (external target choice, distinct from LP weights uu). Rescaling ℓja′=djℓja,sj′=djsj\ell'_{ja}=d_j \ell_{ja},s'_j=d_j s_j (dj>0d_j > 0) preserves λj′Rj′=λjRj\lambda'_j R'_j=\lambda_j R_j, leaving Tε,AT_\varepsilon,A, LP (Eq. 11), and κε⋆\kappa_\varepsilon^\star invariant (120 checks, App. M.1). Varying η=w2/w1\eta=w_2/w_1 in the 3-component family (App. M.1) gives κε⋆(η)=max⁡1,2−η/2\kappa_\varepsilon^\star(\eta)=\max\\{1,2-\eta/2\\} (1.875,1.75,1.5,1,11.875,1.75,1.5,1,1 at η=0.25,0.5,1,2,4\eta=0.25,0.5,1,2,4, vs. ≤2\le 2 for unweighted sum).

3. Uniform Certificates vs. Achieved Ratios (W3, Q3 — Abstract, §2 Table 1, §4 Prop. 4.2, §7). We reframe Γ(D,Tε)\Gamma(D,T_\varepsilon) and κε⋆\kappa_\varepsilon^\star as prior-free structural limits rather than fixed-dataset predictions. The Abstract, Table 1 (p. 3), Prop. 4.2 (p. 7), and §7 separate ρH,P≤κH,P≤Γ(D,Tε)=b/a\rho_{\mathcal{H},P} \le \kappa_{\mathcal{H},P} \le \Gamma(D,T_\varepsilon)=b/a: two-context witnesses (Thm. 2.2) certify worst-case sharpness prior to (P,H)(P,\mathcal{H}) rather than predicting ρH,P\rho_{\mathcal{H},P}, while Γ=∞\Gamma=\infty for Log/Brier diagnoses local O(t2)O(t^2) vs. Θ(t)\Theta(t) facet mismatch (Lemma 3.2), not infinite empirical risk.

4. Facet-Flux Bridge & Deep Training (W4, Q4 — §4 Prop. 4.4, §§5.2–5.3, Table 4, Fig. 3, App. M.2/M.4). Proposition 4.4 bridges proper smoothing to population facet flux within O(h)O(h), enabling end-to-end training up to 821,761-parameter networks. Proper smoothing and pathwise Flux (Gen. Resp. §2.1) differ by αr(Qr−cr)Kh(Qr−cr)vr\alpha_r(Q_r-c_r)K_h(Q_r-c_r)v_r (00 on Qr=crQ_r=c_r), stay within O(h)O(h) in L1L^1 under bounded report / integrable Jacobian-norm-weighted densities, and converge in expectation to the facet flux (Prop. J.1) as h↓0h \downarrow 0, while baseline-augmented smoothing stays strictly proper for all h>0h > 0 (retaining finite-hh slack Δh\Delta_h in §4). Beyond shallow MLPs (§5.2), we train an 821,761-parameter 64→5124→164\to 512^4\to 1 tanh⁡\tanh MLP (§5.3, Table 4, Fig. 3; ntrain=65,536,ntest=131,072n_{\mathrm{train}}=65{,}536,n_{\mathrm{test}}=131{,}072): under Adam with checkpointing, Log (0.009180.00918) and Brier (0.009340.00934) lead on Shallow teacher, while task-aware scores have lower 3-seed means (0.046680.04668–0.047750.04775) than Log (0.047820.04782) and Brier (0.048450.04845) on Oscillatory interaction (Join − Log within seed SE); Fig. 3 / Table 11 show checkpointing prevents boundary overfitting under Adam (0.220→0.2940.220 \to 0.294) and L-BFGS (0.220→0.3760.220 \to 0.376; early-stopped cold L-BFGS: 0.008210.00821).

Thank you for your replies, I will raise my score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (1/2): Theoretical Motivation & Experimental Redesign

We thank the AC and Reviewers (JjRm, vQfQ, QYC7) for their constructive feedback and recognition of Theorems 2.2, 3.3, 3.5 and Lemma 3.2. We uploaded a revised manuscript (added/revised text in blue) and updated the Abstract and anonymous code/data repository (Reproducibility Statement, p. 11): we sharpened problem scope, unit invariance, guarantee scopes, and attribution (§§1–3, §§6–7, Table 1); proved an O(h)O(h) proper-smoothing to facet-flux bridge (§4, Prop. 4.4); and rebuilt §5 (Tables 2–4, Figs. 2–3) and Appendix M (Appendices K–L keep original studies).

1. Theoretical Motivation: Why Optimize One Fixed Proper Score? (§§1–2, §§6–7)

Our setting is loss design within an existing probability-prediction pipeline: the report class H\mathcal{H} and probability-serving interface are retained—e.g., one precipitation model serving decision-makers with distinct thresholds and costs—and the training objective is the component being redesigned. Here “fixed” means one score designed for the declared task family and applied across contexts and users, while the predictor qh(X)q_h(X) remains context-dependent.

Unlike finite-action objectives that leave action-equivalent reports indistinguishable, strict propriety elicits truthful probabilities when realizable, while under misspecification score choice selects different approximations in H\mathcal{H} and actions (without improving every accuracy metric). Statistically, since SF,f(q,Y)S_{F,f}(q,Y) is affine in eYe_Y, E[SF,f(qh(X),Y)∣X]=DF,f(pX,qh(X))−F(pX)\mathbb{E}[S_{F,f}(q_h(X),Y)\mid X] = D_{F,f}(p_X,q_h(X)) - F(p_X): empirical score minimization targets divergence risk from labels without estimating pXp_X, whereas taking the maximum of realized task losses targets a different objective from the contextwise maximum of conditional regrets.

Omniprediction can also serve a shared prediction to downstream decisions, including for proper losses (Gopalan et al., 2022; Gibbs & Tibshirani, 2026); that shared-report feature is not our distinction. Rather, our restriction is the learning procedure: minimizing one fixed proper score within a prescribed class H\mathcal{H}, which omniprediction, per-user heads, and plug-in estimation do not impose. Characterizing the best guarantee within this loss-design interface—including when no proper score closes the gap to the target-optimal predictor—determines both when redesigning the training score suffices and when a more flexible learning approach is needed.

2. Experimental Redesign & Theory-to-Training Bridge (§§4–5, Appendix M)

Replacing 1–3 parameter calibration (Appendices K–L), we rebuilt §§4–5:

  1. Proper-smoothing to facet-flux bridge (Prop. 4.4, §4; App. M.2): For hinge αr(Qr−cr)+\alpha_r(Q_r - c_r)_{+} (Qr=vr⊤qQ_r = v_r^\top q, Yr=vr⊤eyY_r = v_r^\top e_y) and Epanechnikov/logistic KhK_h, proper smoothing (αr(Qr−Yr)Kh(Qr−cr)vr\alpha_r(Q_r - Y_r)K_h(Q_r - c_r)v_r) and pathwise Flux (αr(cr−Yr)Kh(Qr−cr)vr\alpha_r(c_r - Y_r)K_h(Q_r - c_r)v_r) differ by αr(Qr−cr)Kh(Qr−cr)vr\alpha_r(Q_r - c_r)K_h(Q_r - c_r)v_r (00 on Qr=crQ_r = c_r). Their L1L^1 gap is O(h)O(h) for report gradients (bounded density near crc_r) and parameter gradients (integrable Jθqθ(X)J_\theta q_\theta(X), locally bounded Jacobian-norm-weighted density; O(h2)O(h^2) signed bias under locally Lipschitz signed Jacobian-weighted density), and both expectations converge to facet flux (Prop. J.1) as h↓0h \downarrow 0. Baseline-augmented smoothing is strictly proper for all h>0h > 0, with finite-hh slack Δh\Delta_h retained in §4.
  2. Controlled finite-label separation (§5.1, Table 2; App. M.3): Since Join and optimal task covers share κε⋆\kappa_\varepsilon^\star (Thm. 3.5) yet can have distinct in-class projections, our 4-outcome benchmark (G1=H1+12H2G_1 = H_1 + \tfrac12 H_2, G2=12H1+H2+H3G_2 = \tfrac12 H_1 + H_2 + H_3 on (1/4,1/2,3/4)(1/4,1/2,3/4)) fits one mixture class over 500 paired seeds (n=50,000n = 50{,}000; 14,000-cell sweep, n∈[500,5⋅106]n \in [500, 5\cdot 10^6]) on exact ETε\mathbb{E}T_\varepsilon vs. Log, Sum, Oracle best single, and LP-verified Oracle weighted envelope (u1G1+u2G2u_1 G_1 + u_2 G_2), plus reversed/realizable controls.
  3. End-to-end neural training (§5.2, Fig. 2, Table 3; Tables 10, 12): Trained feature MLPs under Adam and L-BFGS-B on 7 synthetic mechanisms (3 seeds; representation, H=2H{=}2 bottleneck, hidden gates, omitted X3,X4X_3,X_4, full-feature/teacher controls, 30-dataset Log-warm diagnostic) and 4 NOAA holdouts (ntrain=131,254n_{\mathrm{train}} = 131{,}254, 26 features, binary H=16H{=}16 & 4-class H=12H{=}12, station/temporal splits).
  4. Deep-network scaling (§5.3, Table 4, Fig. 3; Table 11): Trained an 821,761-parameter 64→5124→164\to 512^4\to 1 tanh⁡\tanh MLP (ntrain=65,536n_{\mathrm{train}} = 65{,}536, nval=16,384n_{\mathrm{val}} = 16{,}384, ntest=131,072n_{\mathrm{test}} = 131{,}072, 3 seeds) on Shallow teacher and Oscillatory interaction, with Adam and cold/warm L-BFGS checkpointing diagnostics.
AuthorsOpenReview ↗

General Response (2/2): Summary of New Empirical Results

  1. Controlled separation under ETε\mathbb{E}T_\varepsilon (§5.1, Table 2; Appendix M.3):

    Regime (ETε\mathbb{E}T_\varepsilon, Mean ± SE, n=50,000n=50{,}000, 500 seeds)LogJoin (hard)SumOracle best singleOracle weighted envelope
    Two-context misspecified (×103\times 10^3)21.36544 ± 0.0000211.79001 ± 0.0236712.38583 ± 0.0216212.17388 ± 0.0257612.13855 ± 0.02605
    Single-context reversed (×103\times 10^3)40.08840 ± 0.0000120.24202 ± 0.0002420.21226 ± 0.0013420.20182 ± 0.0012620.18346 ± 0.00034
    Realizable control (×107\times 10^7)2.42638 ± 0.155182.45355 ± 0.155622.45355 ± 0.155622.45355 ± 0.155622.45355 ± 0.15562

    In Two-context misspecified (n=50,000n=50{,}000), Join lowers mean risk by 44.8% vs. Log (−0.00957542-0.00957542, simultaneous 95% Bonferroni CI [−0.00966052,−0.00949032][-0.00966052,-0.00949032]), 3.15% vs. Oracle best single (−0.00038387-0.00038387, CI [−0.00047817,−0.00028956][-0.00047817,-0.00028956]), and 2.87% vs. Oracle weighted envelope (−0.00034854-0.00034854, CI [−0.00044221,−0.00025487][-0.00044221,-0.00025487]) across all 140 study comparisons (4 constructions × 7 sample sizes × 5 comparators: Log, Task 1, Task 2, Oracle best single, Oracle weighted envelope), certifying an advantage over Sum and every nonnegative whole-task weighting at n=50,000n=50{,}000. At n=500n=500, Join trails Oracle best single (+0.000542+0.000542); in Single-context reversed, Join halves Log risk while trailing the envelope slightly; in Realizable, all recover truth.

  2. Neural mechanisms (§5.2, Figure 2, Table 10; 3 paired seeds under Adam & L-BFGS-B):

    Regime Group (Fig. 2, Table 10 give per-setting Mean ± SE)Join vs. Log (ΔETε×103\Delta\mathbb{E}T_\varepsilon \times 10^3)Join vs. Best singleJoin vs. Sum
    4 Misspecified regimes (Linear H=8, Hard/Smooth gates, Bottleneck H=2)Join lower in 8/8 pairs (-5.136 to -0.205)Join lower in 5/6 pairs (H=2 not run)Join lower in 4/8 pairs (2/4 per opt.; 3/4 union)
    Unobserved active covariates (Omitted X_3, X_4; all near 39.2–41.6)Log slightly lower (+0.232±0.243 / +0.436±0.837)Join lower in 2/2 pairs (-0.365 / -0.346)Join lower in 2/2 pairs (-1.848 / -0.770)
    2 Full-feature controls (Teacher MLP, Full nonlinear H=8)Smooth Brier/Log lead (+3.195 to +8.795)Join lower in 4/4 pairs (-4.780 to -0.714)Join lower in 2/4 pairs (Teacher: -2.633 / -1.176)

    Overall, Join lowers mean risk below Log in 4/5 misspecified settings under both optimizers (8/10 pairs), below validation-selected Best single in 11/12 evaluated pairs (Table 12), and below Sum in 4/7 settings per optimizer (5/7 union). Smooth Brier (5.23/5.565.23/5.56) and Log (5.04/5.995.04/5.99) lead in the full-feature controls, while Log-warm initialization cuts cold-start Join risk by 42.4% (0.012611→0.0072600.012611 \to 0.007260) in our 30-dataset H=8H{=}8 diagnostic.

  3. NOAA weather & 821,761-parameter deep MLP (§§5.2–5.3, Tables 3–4, Figure 3):

    • NOAA GHCN-Daily (Table 3; observed worst mean task cost Cfixed=max⁡jE[λjℓj]C_{\mathrm{fixed}}=\max_j \mathbb{E}[\lambda_j \ell_j], 3-seed training-variation SEs): Join attains the lowest mean cost on four-class temporal holdout under both optimizers (L-BFGS-B: 0.29733±0.000500.29733 \pm 0.00050 vs. Log 0.29962±0.000580.29962 \pm 0.00058, paired diff −0.00229±0.00035-0.00229 \pm 0.00035, Sum 0.297350.29735, Best single 0.298080.29808, Flux 0.299210.29921, Brier 0.300840.30084; Adam: 0.29735±0.000170.29735 \pm 0.00017 vs. Log 0.298600.29860, Sum 0.297740.29774, Best single 0.297610.29761, Flux 0.298480.29848, Brier 0.300550.30055). On binary station L-BFGS-B, Join (0.18269±0.001740.18269 \pm 0.00174) is 5.4% below Log (0.19315±0.006560.19315 \pm 0.00656), while Brier (0.176110.17611) and Sum (0.177870.17787) lead that split (Log/Brier/Sum lead remaining columns).
    • Deep MLP (Table 4, Figure 3, Table 11): With Adam and validation checkpointing, Log (0.00918±0.000150.00918 \pm 0.00015) and Brier (0.00934±0.000440.00934 \pm 0.00044) lead on Shallow teacher, while on Oscillatory interaction all task-aware methods achieve lower 3-seed means (Best single 0.046680.04668, Sum 0.047540.04754, Flux 0.047550.04755, Join 0.047750.04775) than Log (0.047820.04782) and Brier (0.048450.04845), with Join − Log within seed SE. Figure 3 shows checkpointing prevents boundary overfitting for both Adam (0.220→0.2940.220 \to 0.294) and cold/warm L-BFGS (0.220→0.3760.220 \to 0.376; early-stopped cold L-BFGS reaches 0.008210.00821).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.