- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces Target-Aware Loss Comparison (TALC), a framework for comparing supervised losses in predict-then-optimize problems. It compares MSE, Brier loss, and BCE under the same architecture, decoder, and solver. For finite linear decision sets and finite cost alphabets, the analysis decomposes Brier loss into MSE and an additional residual penalty. In the unrestricted setting, Brier and BCE identify the conditional threshold probabilities, whereas MSE generally admits a larger set of minimizers. Nevertheless, all these minimizers decode to the same conditional mean cost and induce Bayes-optimal expected-cost decisions.
Validation includes synthetic shortest-path tasks, digit-based routing, and Warcraft experiment. The experiments show that loss rankings depend on training and model-selection procedures, and that improvements in supervised scores or decision accuracy need not translate into corresponding decision-cost improvements.
Critical strengths
A useful theoretical framework with a clear error decomposition.
The theoretical framework provides a useful perspective on the relationship between supervised losses, cost prediction, and downstream decisions. The decomposition clearly separates decoded-cost error from additional threshold-level error, making the roles of the different objectives explicit. Mean-MSE supervises the decoded cost, while Brier additionally penalizes discrepancies among individual threshold predictions. Comparing Brier with BCE then examines different scoring rules applied to the same threshold targets. The analysis also implies the distinction between their unrestricted minimizers: Brier and BCE identify the conditional threshold probabilities, whereas Mean-MSE generally admits a larger set of predictions that decode to the same conditional mean cost. The regret analysis provides a transparent connection from decoded-cost error to decision regret and relates this guarantee to the supervised excess risks. Within the adopted proof strategy, the quantities controlling regret and the relationships among the resulting bounds are clearly identified. This framework is worth building upon and could inform the design of losses and optimization geometries better aligned with cost prediction. Further characterizing the solution sets induced by these objectives under function approximation would strengthen the paper. In particular, understanding how restrictions on the function class change the optimal predictions, decoded costs, and resulting regret would help connect the decomposition to practical learning.
A detailed and carefully controlled experimental evaluation.
The experiments provide clear comparisons using a common prediction architecture, decoder, and decision solver. The Brier–Mean-MSE and BCE–Brier comparisons help distinguish the effects of supervising individual thresholds from those of changing the scoring rule. The evaluation also examines learning rates, training duration, and checkpoint selection, with additional experiments on fresh synthetic teachers checking whether changes in relative performance persist beyond the original evaluation instances. Reporting both decision error and regret is particularly useful, since these metrics capture different aspects of decision quality. The Warcraft analysis illustrates how substantial differences in exact decision accuracy can correspond to small absolute cost differences and examines sensitivity to the numerical representation of costs.
Critical weaknesses
Limited scope and novelty.
The main theoretical results rely on standard tools. Although their combination provides useful diagnostics, the analysis remains centered on an affine threshold decoder, and the characterization of optimal predictions concerns unrestricted objectives. The empirical findings establish sensitivity to training and model-selection procedures, but offer limited transferable guidance on choosing losses or training configurations. An extension to function approximation and inexact optimization would considerably strengthen the contribution. In particular, characterizing the different solution sets induced by the losses under restricted function classes, and connecting these differences to practical learning outcomes, could provide substantially greater significance. Addressing these limitations would require a major revision beyond additional diagnostics or presentation changes.
Insufficient explanation of the empirical loss comparisons.
The theoretical framework does not adequately explain why the trained objectives exhibit their observed differences in decision error and regret. In particular, the ordering of the displayed regret bounds compares the losses at a fixed prediction; it does not predict the relative performance of separately trained models. Thus, the comparatively looser BCE bound provides little explanation for BCE’s strong empirical performance or for the observed changes in its performance against Brier. Although the paper includes positive-gap bounds and post-hoc diagnostics, these do not sufficiently characterize how loss choice and training shape prediction errors relative to decision boundaries, or how the frequency and severity of incorrect decisions change. Consequently, the framework helps describe the experimental outcomes but provides limited explanatory or predictive insight into them. This gap between the theoretical analysis and the central empirical findings materially weakens the paper’s contribution.
Questions for authors
1. Connecting the regret analysis to BCE’s empirical performance. The ordering of the regret bounds holds at a fixed prediction, whereas the experiments compare separately trained predictors. Can the authors use their framework to explain BCE’s superior decision performance in several experiments? In particular, does BCE achieve smaller decoded-cost error, or do its errors interact more favorably with the decision geometry and action gaps? Can the existing diagnostics explain the experiment results?
2. Different optimal predictions under function approximation. Beyond the unrestricted setting, how do the minimizers of MSE, Brier, and BCE differ within a shared restricted function class, and how do these differences affect the decoded conditional means and decision regret? This is particularly relevant for Brier and BCE, which identify the same unrestricted target but need not yield the same best-in-class prediction. Can the authors provide a characterization, even in a tractable restricted setting, that helps interpret the empirical comparisons?
3. Interaction between loss properties and optimization geometry. The empirical loss rankings reflect the combined effects of what each objective penalizes and how its gradient geometry shapes learning under the chosen training procedure. However, the main decomposition and regret bounds compare different losses at the same prediction, without characterizing how their optimization geometries lead to different learned predictions. Can the authors incorporate finite optimization into their theoretical comparison, at least in a tractable setting, and validate the resulting predictions through matched experiments? Specifically, how do the different loss geometries affect optimization progress and decoded-cost error under a fixed update budget, and which differences remain at the respective objectives’ optima? Experiments within the analyzed setting could then test these predictions and their implications for decision performance, helping distinguish finite-optimization effects from differences between the solutions favored by the objectives.
Thank you for identifying the gap between a comparison at one fixed prediction and the predictions learned by different losses. We now analyze training and restricted model classes, and test their consequences for decisions.
W1: What is new beyond the decomposition? The affine analysis gives separate conditions for when the extra threshold penalty leaves action-cost differences unchanged throughout training and when it leaves the best decoded mean unchanged. This distinguishes a temporary training effect from a difference that remains at the optimum (Section 4). An exact scalar example also compares BCE with both squared losses.
W2/Q1: Why do the trained losses differ? Early in training, BCE's advantage accompanies lower held-out mean error. After 6,000 Quadratic updates, Brier instead has lower held-out mean error (.02404 versus .02584) and lower regret. About 60% of its observed regret advantage comes from fewer wrong paths and 40% from cheaper errors; this breakdown is descriptive (Appendix G.4).
We then intervene on training. On independent teachers, replacing Brier's threshold weights with a norm-matched scalar reduces the Quadratic duration effect by 40% (Holm p=.00234). Weakening Brier's coupled regularization raises regret at its previously favorable readings on both tasks (p=.00234 and .000738). These interventions identify how threshold weighting and regularization contribute to the observed changes in decision cost.
Q2: Can the best predictions differ? Yes. A model may be unable to improve its threshold predictions without changing its predicted mean. Proposition 4 characterizes this trade-off for the squared losses. In an exactly solved restricted model, the optimal means are 2.200, 2.170 and approximately 2.16009 for Mean-MSE, Brier and BCE. The different means lead to different decisions: BCE incurs .0175 expected regret in the example, while the two squared losses incur zero (Appendix G.1).
Q3: What happens during training? On frozen routing features, changing only the parameterization produces 4,589 ReLU and 6,037 Quadratic path differences after three updates, out of 40,960 contexts per task. The differences disappear at the exact optima. When residual capacity is restricted while the mean class is preserved, 7,207 and 11,142 path differences persist at the optima, with higher Brier regret.
For the original sigmoid heads on frozen features, a short-run forecast predicts both chosen paths on 2,550 of 2,723 switching contexts after 30 updates and recovers the aggregate regret order. Its precision falls at 100 steps (Section 5.2; Appendix G.2–G.3).
Together, these results connect loss design to learned means and decisions, while separating finite training from differences at the optimum.