acceptodds
Under review as a conference paper at ICLR 2027

TALC: Target-Aware Loss Comparison for Cost-Based Decisions

Abstract

When learned costs feed a decision solver, a lower prediction loss can leave the decision unchanged. We introduce TALC (Target-Aware Loss Comparison), a framework that separates target choice, scoring rule, training procedure and decision cost. Our analysis decomposes squared threshold excess risk into decoded-mean error and a component invisible to the decoder. For arbitrary finite linear decision sets and finite cost alphabets, expected-cost regret bounds depend only on the decoded-mean component, by specializing existing squared-risk bounds. A cost-preserving projection can improve threshold Brier loss and binary cross-entropy while preserving the argmin set in exact arithmetic. We compare these two scores with decoded-mean MSE using a shared head, decoder and solver. On 300 saved models, projection reduces Brier scores with no observed change in true-cost error or regret; floating-point path identities can differ. Brier's extra residual constraint does not uniformly reduce cost error or regret. Across shortest-path tasks, corrected comparisons support threshold supervision in several settings. On fresh synthetic cost rules, specified learning-rate and duration changes significantly shift the BCE–Brier error difference, with observed mean orderings crossing. On Warcraft, most exact route errors add less than 0.1% stored cost, and their incidence depends on the cost precision used. A predicted training-duration reversal fails to transfer to battery scheduling. TALC connects what a loss constrains to what a decision actually costs.

Primary area:
optimization
Keywords:
predict-then-optimize, surrogate losses, threshold supervision, proper scoring rules, decision regret

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bkvAOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Target-Aware Loss Comparison (TALC), a framework for comparing supervised losses in predict-then-optimize problems. It compares MSE, Brier loss, and BCE under the same architecture, decoder, and solver. For finite linear decision sets and finite cost alphabets, the analysis decomposes Brier loss into MSE and an additional residual penalty. In the unrestricted setting, Brier and BCE identify the conditional threshold probabilities, whereas MSE generally admits a larger set of minimizers. Nevertheless, all these minimizers decode to the same conditional mean cost and induce Bayes-optimal expected-cost decisions.

Validation includes synthetic shortest-path tasks, digit-based routing, and Warcraft experiment. The experiments show that loss rankings depend on training and model-selection procedures, and that improvements in supervised scores or decision accuracy need not translate into corresponding decision-cost improvements.

Critical strengths

A useful theoretical framework with a clear error decomposition.

The theoretical framework provides a useful perspective on the relationship between supervised losses, cost prediction, and downstream decisions. The decomposition clearly separates decoded-cost error from additional threshold-level error, making the roles of the different objectives explicit. Mean-MSE supervises the decoded cost, while Brier additionally penalizes discrepancies among individual threshold predictions. Comparing Brier with BCE then examines different scoring rules applied to the same threshold targets. The analysis also implies the distinction between their unrestricted minimizers: Brier and BCE identify the conditional threshold probabilities, whereas Mean-MSE generally admits a larger set of predictions that decode to the same conditional mean cost. The regret analysis provides a transparent connection from decoded-cost error to decision regret and relates this guarantee to the supervised excess risks. Within the adopted proof strategy, the quantities controlling regret and the relationships among the resulting bounds are clearly identified. This framework is worth building upon and could inform the design of losses and optimization geometries better aligned with cost prediction. Further characterizing the solution sets induced by these objectives under function approximation would strengthen the paper. In particular, understanding how restrictions on the function class change the optimal predictions, decoded costs, and resulting regret would help connect the decomposition to practical learning.

A detailed and carefully controlled experimental evaluation.

The experiments provide clear comparisons using a common prediction architecture, decoder, and decision solver. The Brier–Mean-MSE and BCE–Brier comparisons help distinguish the effects of supervising individual thresholds from those of changing the scoring rule. The evaluation also examines learning rates, training duration, and checkpoint selection, with additional experiments on fresh synthetic teachers checking whether changes in relative performance persist beyond the original evaluation instances. Reporting both decision error and regret is particularly useful, since these metrics capture different aspects of decision quality. The Warcraft analysis illustrates how substantial differences in exact decision accuracy can correspond to small absolute cost differences and examines sensitivity to the numerical representation of costs.

Critical weaknesses

Limited scope and novelty.

The main theoretical results rely on standard tools. Although their combination provides useful diagnostics, the analysis remains centered on an affine threshold decoder, and the characterization of optimal predictions concerns unrestricted objectives. The empirical findings establish sensitivity to training and model-selection procedures, but offer limited transferable guidance on choosing losses or training configurations. An extension to function approximation and inexact optimization would considerably strengthen the contribution. In particular, characterizing the different solution sets induced by the losses under restricted function classes, and connecting these differences to practical learning outcomes, could provide substantially greater significance. Addressing these limitations would require a major revision beyond additional diagnostics or presentation changes.

Insufficient explanation of the empirical loss comparisons.

The theoretical framework does not adequately explain why the trained objectives exhibit their observed differences in decision error and regret. In particular, the ordering of the displayed regret bounds compares the losses at a fixed prediction; it does not predict the relative performance of separately trained models. Thus, the comparatively looser BCE bound provides little explanation for BCE’s strong empirical performance or for the observed changes in its performance against Brier. Although the paper includes positive-gap bounds and post-hoc diagnostics, these do not sufficiently characterize how loss choice and training shape prediction errors relative to decision boundaries, or how the frequency and severity of incorrect decisions change. Consequently, the framework helps describe the experimental outcomes but provides limited explanatory or predictive insight into them. This gap between the theoretical analysis and the central empirical findings materially weakens the paper’s contribution.

Questions for authors

1. Connecting the regret analysis to BCE’s empirical performance. The ordering of the regret bounds holds at a fixed prediction, whereas the experiments compare separately trained predictors. Can the authors use their framework to explain BCE’s superior decision performance in several experiments? In particular, does BCE achieve smaller decoded-cost error, or do its errors interact more favorably with the decision geometry and action gaps? Can the existing diagnostics explain the experiment results?

2. Different optimal predictions under function approximation. Beyond the unrestricted setting, how do the minimizers of MSE, Brier, and BCE differ within a shared restricted function class, and how do these differences affect the decoded conditional means and decision regret? This is particularly relevant for Brier and BCE, which identify the same unrestricted target but need not yield the same best-in-class prediction. Can the authors provide a characterization, even in a tractable restricted setting, that helps interpret the empirical comparisons?

3. Interaction between loss properties and optimization geometry. The empirical loss rankings reflect the combined effects of what each objective penalizes and how its gradient geometry shapes learning under the chosen training procedure. However, the main decomposition and regret bounds compare different losses at the same prediction, without characterizing how their optimization geometries lead to different learned predictions. Can the authors incorporate finite optimization into their theoretical comparison, at least in a tractable setting, and validate the resulting predictions through matched experiments? Specifically, how do the different loss geometries affect optimization progress and decoded-cost error under a fixed update budget, and which differences remain at the respective objectives’ optima? Experiments within the analyzed setting could then test these predictions and their implications for decision performance, helping distinguish finite-optimization effects from differences between the solutions favored by the objectives.

1 comment

Thank you for identifying the gap between a comparison at one fixed prediction and the predictions learned by different losses. We now analyze training and restricted model classes, and test their consequences for decisions.

W1: What is new beyond the decomposition? The affine analysis gives separate conditions for when the extra threshold penalty leaves action-cost differences unchanged throughout training and when it leaves the best decoded mean unchanged. This distinguishes a temporary training effect from a difference that remains at the optimum (Section 4). An exact scalar example also compares BCE with both squared losses.

W2/Q1: Why do the trained losses differ? Early in training, BCE's advantage accompanies lower held-out mean error. After 6,000 Quadratic updates, Brier instead has lower held-out mean error (.02404 versus .02584) and lower regret. About 60% of its observed regret advantage comes from fewer wrong paths and 40% from cheaper errors; this breakdown is descriptive (Appendix G.4).

We then intervene on training. On independent teachers, replacing Brier's threshold weights with a norm-matched scalar reduces the Quadratic duration effect by 40% (Holm p=.00234). Weakening Brier's coupled regularization raises regret at its previously favorable readings on both tasks (p=.00234 and .000738). These interventions identify how threshold weighting and regularization contribute to the observed changes in decision cost.

Q2: Can the best predictions differ? Yes. A model may be unable to improve its threshold predictions without changing its predicted mean. Proposition 4 characterizes this trade-off for the squared losses. In an exactly solved restricted model, the optimal means are 2.200, 2.170 and approximately 2.16009 for Mean-MSE, Brier and BCE. The different means lead to different decisions: BCE incurs .0175 expected regret in the example, while the two squared losses incur zero (Appendix G.1).

Q3: What happens during training? On frozen routing features, changing only the parameterization produces 4,589 ReLU and 6,037 Quadratic path differences after three updates, out of 40,960 contexts per task. The differences disappear at the exact optima. When residual capacity is restricted while the mean class is preserved, 7,207 and 11,142 path differences persist at the optima, with higher Brier regret.

For the original sigmoid heads on frozen features, a short-run forecast predicts both chosen paths on 2,550 of 2,723 switching contexts after 30 updates and recovers the aggregate regret order. Its precision falls at 100 steps (Section 5.2; Appendix G.2–G.3).

Together, these results connect loss design to learned means and decisions, while separating finite training from differences at the optimum.

Reviewer_GjJ8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how the choice of training loss affects decisions when predicted costs are passed to a shortest-path solver. It compares decoded-mean MSE, threshold-wise Brier and threshold-wise BCE on a shared cumulative threshold head. This way, Mean-MSE vs. Brier changes only the supervised target, and Brier vs. BCE changes only the scoring rule. The theory shows three things. Brier excess risk splits into a decoded-mean error M and a residual that the decoder cannot see. Regret is bounded by √(HM), a specialization of Ho-Nguyen & Kılınç-Karzan (2022). A gap-weighted isotonic projection lowers both threshold scores without changing any decision. The experiments cover synthetic DAGs and Digits grids under a 600-update and a 6,000-update protocol, a confirmation study on fresh synthetic teachers, Warcraft maps and a battery-scheduling transfer task. The main findings are that loss rankings change with the training procedure, and that large exact-error gaps on Warcraft come with very small cost differences.

Critical strengths

S1. The comparison is well controlled. Within each study, all three objectives share the output head, labels, decoder, solver, initialization and batch streams (Table 1). Each of the two contrasts therefore changes one thing. Each study fixes its comparison family in advance, uses exact paired sign-flip tests with Holm correction, and keeps selection units separate from evaluation units (App. B). The failed battery transfer is reported in full (App. F). For these reasons I take the reported numbers at face value. My concerns are about interpretation, not execution.

S2. The Warcraft analysis is useful in its own right. Exact error differs sharply between the losses: 6.67%, 2.60% and 0.09% (Table 5). Yet of the 667, 260 and 9 errors across ten seeds, only 9, 3 and 0 add 0.1% or more to the optimal stored cost (Fig. 3). Nearly all of them disappear when costs are mapped to the nominal one-decimal levels (Table 5, App. D). The archived label paths are themselves suboptimal under the stored costs on 636 training maps (Sec. 5.5). Anyone who reports exact path accuracy on this benchmark should know this.

Critical weaknesses

W1. The paper offers no explanation or prediction for its central finding, so it is unclear what readers should take from it.

The central empirical result is that rankings depend on the training procedure. In Study A, BCE has the lowest error on all three tasks (Table 2). In Study B, Brier has the lowest mean error on both synthetic tasks and beats BCE on ReLU after correction (Tables 3 and 10). The fresh-teacher study confirms that changing the learning rate (ReLU) or the training length (quadratic) shifts the BCE–Brier gap (Fig. 2, Table 19).

The theory cannot account for this. Cor. 1 relates M to regret for a fixed predictor, and the paper notes that these relations "do not determine how training changes M" (L242). Whether threshold supervision lowers M during training is exactly the question the experiments ask, and the theory says nothing about it. The bound √(HM) is also never evaluated on the tasks, so it is unclear whether it is informative there.

The experiments do not test a mechanism either. One candidate is how the two losses weight errors. With respect to a logit, the BCE gradient is proportional to (q − t), while the Brier gradient is proportional to (q − t)q(1 − q). Under Brier, a threshold that is confidently wrong therefore receives almost no gradient. All studies use Adam, so a global difference in gradient scale should matter little. This per-example weighting, however, does not cancel, and it could plausibly make the comparison depend on learning rate and training length. If something like this drives the results, then Study A's finding that "the scoring rule matters empirically here" (L302–303), and its reversal in Study B, say more about optimization than about the losses themselves.

The paper is explicit that TALC is "an audit, not a validated cross-task loss-selection rule" (L470), and I do not fault it for lacking such a rule. My concern is significance. An audit that records rankings changing, without saying why or when they change, gives readers little guidance for their own problems.

W2. The confirmatory tests in the shortest-path studies use error rate rather than regret, and the independently confirmed results come from a narrow setting.

The paper argues that losses should be judged by what their decisions cost (last sentence of the abstract; Sec. 7), and its theory bounds regret. However, every corrected test in Studies A, B and C and in the fresh-teacher study is on the nonoptimal-path rate. Regret is reported only descriptively (Tables 2 and 11; App. B.5). The one regret-based confirmatory test is in the battery task, and it was null.

The two metrics do not always agree. On Study A's synthetic tasks, Brier and Mean-MSE do not differ in tied-context error (Holm p = 1.0, Table 8). Yet Brier's all-context regret is higher, with marginal 95% intervals of [+0.0008, +0.0143] and [+0.0026, +0.0200] (Table 4). On Warcraft, the error gaps are large while mean stored regret per map ranges from 8.79 × 10⁻⁷ to 1.96 × 10⁻⁴ (Table 5). The abstract states that "corrected comparisons support threshold supervision in several settings" (L23). That statement holds for error rate, which the paper's own Warcraft analysis shows to be the less decision-relevant metric.

The independent confirmation on fresh teachers (Sec. 5.4, App. B.5) uses only the two synthetic generators. The paper does not say how many thresholds these tasks have. Two observations bear on this: App. B.1 describes a three-level penalty added to each edge's layer increment, and Mean-MSE models cross in about half of all cost coordinates (Table 20), which is consistent with a single pair of thresholds. If m = 2, as these observations suggest, the residual term reduces to (e₁ − e₂)²/4, and the confirmed interactions say little about threshold supervision with many thresholds.

The evidence at larger m is mixed. Digits has costs 1 + d, so nine thresholds. In Study A, where seeds vary only optimization on fixed boards, Digits favors BCE over both squared losses. In Study B, which uses new image partitions, no Digits comparison survives correction. The battery task (m = 63) shows no registered difference.

Two changes would address this weakness: reporting the corrected tests with regret as a co-primary endpoint, and limiting the claims about threshold supervision to the values of m that were tested.

Questions for authors
  • Q1. How many thresholds per edge do the ReLU and quadratic tasks use? If the answer is two, how would you qualify the general statements about threshold supervision?
  • Q2. If the same sign-flip and Holm procedure is applied to per-unit regret, labeled as exploratory, which cells of Table 6 still hold?
  • Q3. Have you checked whether the procedure dependence is tied to how the two losses treat saturated thresholds? For example, in the fresh-teacher runs, how does the fraction of confidently wrong thresholds evolve over training under BCE and under Brier?
  • Q4. What is the value of H on each task, and how does √(H·M_samp) compare with the observed regret, even as a rough check?
1 comment

Thank you for highlighting the missing explanation and the gap between regret theory and the original error-rate tests. We now show how threshold supervision can change learned costs during training or at a model's optimum, and we test its effect on decision regret.

W1: Why the comparison changes. Extra threshold supervision can change the predicted mean because the outputs share parameters. Our affine analysis separates changes during training from changes that persist at the optimum; routing controls demonstrate both (Sections 4 and 5.2). Figure 2 retains the original fresh-teacher results. For the nonlinear models, we also test threshold weighting and regularization directly (Section 5.3).

W2/Q1: What do the threshold counts tell us? ReLU and Quadratic use two thresholds per edge: the smallest setting in which individual threshold errors can cancel in the predicted mean. It lets us study how supervision of those errors affects decisions. The decomposition and affine analysis extend to more thresholds (Sections 3–4).

We also compare losses with more thresholds: Digits has nine, Warcraft four and battery scheduling 63. On Digits in Study A, Brier reduces path error from 14.65% to 11.48% and mean regret from .323 to .248 compared with Mean-MSE (exploratory regret reanalysis: Holm p=.00296). This shows that threshold supervision can reduce decision cost with more than two thresholds. Independent interventions further show that threshold weighting and regularization affect decision regret on the synthetic tasks (Section 5.3).

Q2: Do the error results hold for regret? Applying the original tests to regret retains eight of the 13 corrected error directions:

  • Study A: BCE beats both squared losses on ReLU; BCE and Brier beat Mean-MSE on Digits.
  • Study B: Brier beats Mean-MSE on both synthetic tasks and BCE on ReLU.
  • Warcraft: BCE beats Brier by 5.78e-5 cost per map (Holm p=.00391); Brier versus Mean-MSE is inconclusive (p=.107).

The other five directions are inconclusive. This reanalysis is exploratory. In a separate replication, we fixed regret as the primary endpoint before training; both original interactions and three intervention contrasts pass the five-test correction (Section 5.3; Appendix B.6).

Q3: Do saturated thresholds explain the changes? On Quadratic validation, confidently wrong thresholds rise from .403% to .829% under BCE and fall from .084% to .045% under Brier between 600 and 6,000 updates. Increasing their weight changes the duration interaction by −.00540; the corrected result is inconclusive (Holm p=.0743).

A broader intervention replacing Brier's individual threshold weights with a norm-matched batch scalar reduces the Quadratic duration effect by 40% on new teachers (Holm p=.00234). Separately, weakening Brier's coupled regularization increases regret at its previously favorable readings on both tasks. Both sample-mean reversals disappear; the resulting comparison with BCE is descriptive, with Quadratic nearly tied (Appendix G.5 and G.7).

Q4: How informative is the bound? The sampled bound is loose compared with observed regret:

TaskHSampled boundObserved mean regret
ReLU324.24–4.79.087–.132
Quadratic324.50–5.11.137–.183
Digits8107.48–10.13.163–.323
Warcraft10,027.44.166–6.6318.79e-7–1.96e-4
Battery, raw-price scale6.0167£86.37–90.11/day£3.96–4.40/day

Each pair uses the same evaluation subset. The bound identifies the mean error that controls decision cost. Our training analysis and interventions show how loss choice changes that error and the resulting decisions (Appendix G.6).

Reviewer_i7vgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes uncertainty-quantified predictions for cost prediction in the routing problem. The approach proposes to predict the distribution of costs, instead of a regression that always outputs the mean cost. The main intuition is that, with regression, prediction errors at different thresholds may cancel out, while predicting the full distribution gives more supervision to the predictor. The experiment shows the approach is mostly effective, but not always beating the regression baseline.

Critical strengths

I think uncertainty quantification for routing problem is interesting and important. I like the idea, and what I found as most surprising is that, the way that the paper uses uncertainty quantified predictor is informationally equivalent to simple mean regression, however the experiment shows the proposed approach is somewhat effective. I like the intuitive explanation in the paper, but I'm not sure if that's the main reason that drives the comparison between the approaches.

Critical weaknesses
  • The main weakness I saw is that, the paper proposes to use the uncertainty quantified predictor, however the solver uses the two predictors informationally equivalent. I wonder if there are better ways to use the uncertainty predictions. I'm not sure if this is the reason that for some cases in the experiment, the proposed approach does not beat the baseline regression.

  • Experiment shows mixed results. According to my point above, I do not feel fully convinced that the proposed approach dominates simple mean regression.

  • The writing is hard to follow. Many results remain unexplained. For example, I was very confused with the theoretical results in section 4. I think the right interpretation of Section 4 is that, both regression and uncertainty quantified loss controls regret, and regression controls it even tighter. So I'm not sure how it helps in understanding the proposal in the paper. Perhaps the paper should add discussions around it.

Questions for authors

I do not have questions.

1 comment

Thank you for highlighting the need to clarify our main contribution. TALC studies how training losses shape the cost estimates used by a decision solver. The revised paper explains why supervising individual thresholds can change the learned mean costs and, in turn, the decisions, even though the solver uses only those means.

Why does the solver use only means? For a solver minimizing expected total cost, the mean of each cost is sufficient. Keeping the decoder and solver fixed lets us isolate what changes during training. Threshold predictions share parameters, so improving one can change their sum and thus the mean supplied to the solver.

Why do loss rankings change? The loss with lower decision cost can change with training. Threshold supervision can alter the learning path and, when model capacity is limited, the best mean the model can represent. Our analysis and controlled routing experiments distinguish these effects (Sections 4 and 5.2).

The nonlinear experiments show why training matters: on new teachers from two synthetic generators, threshold weighting and coupled regularization contribute to the observed regret changes. Replacing individual weights reduces the Quadratic duration effect by 40%; weakening Brier's coupled regularization raises regret where Brier previously led on both tasks (Section 5.3).

What does the regret bound mean? Your interpretation is correct for a fixed prediction: the mean-error bound is tighter. Each loss learns its own prediction, so the relevant comparison is the mean error and decision cost produced after training. The revised Section 4 explains how those learned predictions can differ.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.