- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies how to choose a single proper scoring rule for multiple downstream decision tasks under model misspecification. Its objective is to minimize the worst-case target-risk ratio relative to the best predictor in the same report class, where the target is the maximum over task regrets.
Theorem 2.2 characterizes this ratio through pointwise divergence distortion, but the theorem is more like a rephrase of the objective than providing intuitions. The paper then gives characterizations and score constructions for several special cases: orthogonal two-action tasks, independent shared components, and binary outcomes. It also provides learning guarantees that transfer score-minimization error into target risk and analyzes smooth proxies gradient-based optimization.
Experiments test the proposed mechanism against fixed proper scores, but the real-data results do not prove the effectiveness over standard scoring rules.
Critical strengths
I think the problem itself is interesting. The characterizations and the proposed constructions are mathematically interesting too. It provides a useful perspective on how to combine difference decision objectives.
Critical weaknesses
I am mainly concerned with the motivation of the paper.
-
Theoretical motivation. The results are interesting, however, the paper would benefit from more explanation / motivation on why minimizing one fixed proper score is the right restriction for several downstream users. The related work section cites the omniprediction literature. The authors recognize there is a distinction between one aggregate score, and learning for omniprediction, but stating this difference doesn't fully motivate the proposed theoretical problem or approach. What practical constraints justify the restriction?
-
Empirical evidence seems weak. It does not show the proposed method is a useful improvement.
Moreover, the presentation needs revision. The paper introduces many closely related quantities, but I think the connections could be explained more clearly. I'll name several here: baseline divergence (also, why is it important?), training score divergence, target loss, pointwise ratio constants, etc. The paper also refer to ealier equations very often by equation number. This requires readers to move back and forth to understand the definitions. Many of the mathematical characterization remains unexplained. Theorem 2.2 is extremely hard to read. The several later special cases could be described as "computational results". The "operational equivalence" of theorem 2.2 is also ambiguous.
Questions for authors
I'd appreciate responses to my questions above.
Author Response to Reviewer QYC7
We thank Reviewer
QYC7for the positive assessment of our formulation and characterizations (Soundness: 3). Below, each paragraph addresses one concern (W1,W2,W3) with direct manuscript evidence.1. Theoretical Motivation & Relation to Omniprediction (
W1— §§1–2, pp. 1–3; §§6–7, p. 10; Gen. Resp. §1). As detailed in Gen. Resp. §1 and revised in §§1–2, §§6–7, our setting is loss design within an existing probability-prediction pipeline: retaining the model class H and probability-report interface while redesigning one fixed training score for a declared task family (the predictor qh(X) remains context-dependent). Unlike finite-action objectives that leave action-equivalent reports indistinguishable, strict propriety elicits truthful probabilities when realizable while directing the misspecified approximation in H toward critical decisions (without improving every accuracy metric); statistically, E[SF,f(qh(X),Y)∣X]=DF,f(pX,qh(X))−F(pX) yields an observable label-based criterion without estimating pX, whereas taking the maximum of realized task losses targets a different objective from the contextwise maximum of conditional regrets. §6 clarifies that omniprediction can also serve a shared prediction to downstream decisions, including for proper losses (Gopalan et al., 2022; Gibbs & Tibshirani, 2026)—that shared-report feature is not our distinction. Rather, our restriction is the learning procedure: minimizing one fixed proper score within a prescribed class H, so our sharp bounds determine both when redesigning the training score suffices and when a more flexible learning approach (e.g., omniprediction or per-user heads) is needed.2. Experimental Modifications & Empirical Results (
W2— §§4–5, Tables 2–4, Figs. 2–3, App. M; Gen. Resp. Parts 1–2). Replacing 1–3 parameter calibration (kept in Appendices K–L), we added a three-tier suite backed by Prop. 4.4's O(h) proper-smoothing / facet-flux gradient bridge (§4): (i) a 500-seed 4-outcome study under exact ETε (§5.1, Table 2), (ii)Adam/L-BFGS-Bfeature-MLP training on 7 synthetic mechanisms and 4 NOAA holdouts (§5.2, Fig. 2, Table 3), and (iii) an 821,761-parameter 64→5124→1 tanh MLP (§5.3, Table 4, Fig. 3). Results show: (a) in §5.1 (n=50,000, 10−3 units), hardJoin(11.790) lowers target risk by 44.8% vs.Log(21.365), 3.15% vs.Oracle best single(12.174), and 2.87% vs. LP-verifiedOracle weighted envelope(12.139, beatingSum12.386 and all nonnegative task weights) under simultaneous 95% Bonferroni CIs across 140 contrasts; (b) in §5.2,Joinlowers 3-seed mean risk vs.Login 4/5 misspecified settings (8/10pairs) and vs.Best singlein 11/12 pairs, and attains the lowest mean cost Cfixed on 4-class NOAA temporal holdout under both optimizers (L-BFGS-B: 0.29733±0.00050, diff vs.Log−0.00229±0.00035;Adam: 0.29735±0.00017) plus −5.4 vs.Logon binary stationL-BFGS-B(Log/Brier/Sumlead on controls/other splits); (c) in §5.3,Adamwith checkpointing scales cleanly to 821,761 parameters, where task-aware scores achieve lower 3-seed means thanLog/BrieronOscillatory interaction(Log/Brierlead onShallow teacher), and Fig. 3 shows checkpointing prevents boundary overfitting under bothAdamandL-BFGS.3. Exposition Overhaul: Notation Table, Baseline Roles, Theorem 2.2 & §3 Framing (
W3— §§1–4, Table 1, Table 12). We revised the exposition around your four suggestions: (i) Consolidated object guide (Table 1, §2, p. 3): placed before Thm. 2.2,Table 1defines Tε, DF,f, (a,b), ρH,P≤κH,P≤Γ(D,Tε)=b/a, κε⋆, Fh, and g~hflux in one place; (ii) Three roles of εDF0 (§1, p. 1; §2, p. 3; §4, p. 7): within-cell probability identification, positive denominator Tε(p,q)>0, and smooth within-cell gradients; (iii) Theorem 2.2 renaming & intuition (§2, pp. 3–4): renamed “Worst-case projection distortion equivalence” with pre-theorem intuition showing how cross-context pricing [a,b] inverts target ordering by nearly b/a between two in-class predictors; and (iv) Constructive §3 framing (§§3–4, pp. 4–7;Table 12, p. 38): §3 presents the transversal bound, shared-component LP, binary tie-charge join, and quadratic ellipsoid as computable characterizations, linked to training by Prop. 4.4 and Table 12.