acceptodds
Under review as a conference paper at ICLR 2027

Consistency Loss Is Secretly Picking a Side: Risk-Directed Consensus for High-Stakes Decisions

Abstract

Consistency regularization makes predictions agree across semantically equivalent inputs. Under asymmetric decision risk, however, agreement is under-specified: it contracts within-orbit disagreement without determining where the shared prediction settles relative to a risk-critical boundary. In emergency triage, stronger symmetric consistency cut permutation-induced prediction flips by more than two-thirds but also increased under-triage. We formalize this missing degree of freedom as shared-solution selection, distinct from agreement contraction, and introduce disagreement-conditioned risk-directed consensus (DGN), which biases the direction in which inconsistent views approach consensus while letting that directional force vanish as agreement forms. On three triage corpora, DGN lowers under-triage relative to matched symmetric consistency at less than one point of over-triage per point removed. Across four additional 7B–14B LLM backbones, matched CE to BWOT to DGN interventions reproduce the same two-stage contraction–selection geometry, and DGN lowers both under-triage and flips relative to CE without a corresponding loss of ordinal predictive quality. A frozen-readout intervention roughly halves MIMIC under-triage without relearning the representation, reaching operating points that no train-selectable global threshold attains. A CIFAR-10-LT study with ResNet-32 and unordered labels reproduces both the contraction–selection separation and directional control outside clinical language, LLMs, and ordinal prediction. These results identify consensus location as a distinct and controllable design variable for consistency learning under asymmetric risk.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
invariance, consistency regularization, large language models, asymmetric risk, ordinal classification, permutation robustness, safety, emergency triage, cost-sensitive learning, constrained optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zFsLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates an important limitation of consistency regularization: making predictions invariant to semantically equivalent inputs does not guarantee that the resulting consensus prediction lies on the appropriate side of a risk-critical decision boundary.

To address the problem, the paper introduces two complementary approaches: Disagreement-conditioned risk-directed consensus (DGN), a training objective that combines symmetric consistency with disagreement-gated directional preference, and Orbit-Budgeted Consensus Optimization (OBCO), a constrained method for adjusting the linear readout while keeping learned representations fixed.

The central motivation is compelling. In emergency triage, a model can become more consistent under different presentations of the same patient while becoming more likely to under-triage high-acuity patients. This highlights a genuine distinction between prediction stability and decision safety.

Critical strengths

The paper identifies a meaningful limitation of consistency regularization, provides a useful mathematical decomposition, and conducts extensive empirical evaluations. The matched BWOT–DGN comparison and frozen-readout controls offer credible evidence that risk location can be modified separately from prediction agreement.

Critical weaknesses

Section 3.1 and Proposition 4.1 establish that symmetric transport itself supplies no signed common-mode pseudo-gradient. The actual consensus location is influenced by supervised learning, optimization, and parameter sharing. Therefore, the title may suggest that a symmetric consistency loss directly chooses a risk direction, whereas the more precise finding is that it leaves the consensus location insufficiently constrained.

The paper sometimes connects reduced under-triage to safety improvements. However, it evaluates predictions against recorded or mapped acuity labels rather than prospectively observed patient outcomes.

Equation (2) establishes that the directional penalty becomes zero at exact agreement. Equation (3) identifies the directional pseudo-gradient as -\lambda a\omega_k g_{ik}. These statements are related but not identical. The assertion that the loss contribution vanishes does not alone establish that its gradient vanishes continuously as agreement develops.

The figure simultaneously presents motivation, force-field decomposition, a convex optimization problem, and clinical outcomes. Its text and mathematical annotations are too small for comfortable reading.

This is a consequential presentation weakness. Over-triage is necessary for interpreting the manuscript's principal asymmetric-risk claim, yet the main table reports Under, NPV, LR−, QWK, and Flip without directly displaying Over. Although the paper discusses exchange rates and Appendix E reports over-triage, readers should not need to reconstruct the cost of a central result.

The main manuscript combines three related but distinguishable contributions: a conceptual framework, a directional consistency objective, and a constrained readout optimizer. The relationship between DGN and OBCO could be explained more carefully because OBCO does not necessarily preserve the improved permutation consistency achieved by DGN. For example, DGN reduces MIMIC Flip to 14.21%, while DGN + OBCO increases it to 28.53%. The methods therefore control related but partly competing properties.

The observation that agreement does not determine correctness is related to representation underspecification, cost-sensitive learning, and decision-boundary adjustment. The originality is therefore strongest in the precise mechanism and experimental decomposition, not in identifying asymmetric decision risk itself.

The directional term -a g_{ik}d_{ivk} introduces a label-informed preference toward the urgent side during supervised training. The paper demonstrates that this differs mathematically from conventional class weighting. However, it does not fully establish whether a simpler combination of symmetric consistency and cost-sensitive supervision could achieve similar outcomes.

On CIFAR-10-LT, the directional coefficient raises C9 recall from 55.47% to 58.29%. However, Appendix Table 10 reports considerable between-seed variation, with standard deviations of 5.09 and 3.58 points for the relevant configurations. The observed mean improvement is therefore not, by itself, decisive evidence of robust generalization.

Questions for authors

Section 4.1, Proposition 4.1, and Appendix G. DGN defines a normalized gate of the form g_{ik}=\min\left(1,\frac{s_{ik}}{m_t}\right), where siks_{ik} measures sibling disagreement and mtm_t is an adaptive moving scale. The manuscript argues that the directional component disappears at agreement, which is true when sik=0s_{ik}=0 and mt>0m_t>0. However, the directional pseudo-gradient is proportional to gikg_{ik}, not to gikdivkg_{ik}d_{ivk}. If the adaptive scale decreases at approximately the same rate as the disagreement, then sik/mts_{ik}/m_t need not approach zero. For example, along a hypothetical sequence with sik=ϵs_{ik}=\epsilon and mt=ϵm_t=\epsilon, the gate remains one as ϵ\epsilon approaches zero. This does not invalidate the exact-agreement result or the idealized fixed-scale analysis. It does mean that zero force at exact agreement is not equivalent to continuously vanishing force along every adaptive training trajectory. The appendix acknowledges that the normalized gate need not itself converge to zero, but the implications for the gradient dynamics deserve a more explicit treatment.

Sections 3.1, 5.1, and Appendix A. The paper uses \eta=\frac{\Delta O}{|\Delta U|} to evaluate the cost of reducing under-triage. The manuscript shows that η<1\eta<1 corresponds to improving binary balanced accuracy. However, the paper's motivation explicitly assumes asymmetric clinical costs, while balanced accuracy weights the two conditional error rates equally. A favorable exchange rate is useful descriptive evidence, but there is no universal clinical reason why a one-to-one exchange threshold should define acceptable risk. A cost-sensitive formulation would be R=C_{\mathrm{FN}}\pi U+ C_{\mathrm{FP}}(1-\pi)O, where π\pi is the high-acuity prevalence and the costs reflect the consequences of the two error types.

Section 5.1; Appendix E.3–E.5. The frozen-readout intervention reduces MIMIC under-triage from 40.53% to 19.86%, which is impressive. However, it also increases Flip from 14.21% to 28.53% and raises over-triage. Appendix E demonstrates that the orbit-dispersion constraint prevents an even larger increase in instability; without it, Flip reaches 53.64%. The authors correctly acknowledge that the constraint is computed on frozen training features and cannot guarantee discrete test-time consistency. Nevertheless, this raises a fundamental question: does OBCO optimize the same underlying goal as DGN, or does it partially undo DGN's consistency improvements to obtain better recall?

Section 5.1; Appendices A and B. The main test sets contain 503 MIMIC source cases, 883 EMS-M source cases, and 500 ER-Reason source cases, expanded into substantially larger collections of fact permutations. These permutations are correlated because they originate from the same clinical cases. The paper reports five-seed means and includes a source-equal evaluation convention, but the primary clinical table does not provide patient- or source-clustered confidence intervals for the principal improvements. The absence of these intervals is especially concerning for smaller effects, such as EMS-M under-triage decreasing from 16.64% under CE to 14.28% under DGN.

Section 5.1; Table 1. The paper includes weighted CE, ordinal cost-sensitive loss, and multiple consistency baselines. This is valuable. However, the strongest comparison for the proposed scientific mechanism should be against methods combining both risk-sensitive supervision and consistency regularization. For instance, WCE already reduces MIMIC under-triage to 30.67%, compared with 40.53% for DGN, but maintains a higher Flip rate. It is not clear whether a tuned WCE+BWOT objective could approach DGN's two-dimensional trade-off frontier.

Appendix B; Ethics Statement. MIMIC uses recorded ESI triage levels, while EMS-M uses protocol-derived weak labels and ER-Reason uses mapped acuity categories. The authors explicitly distinguish these sources, which is appropriate. Nevertheless, disagreement with a recorded or reconstructed acuity label does not necessarily establish clinical under-triage relative to the patient's actual needs. The paper does not validate its predictions against independently adjudicated triage decisions or patient outcomes.

Reviewer_GfjGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates consistency regularisation under asymmetric decision risk, motivated primarily by emergency department triage. The authors observe that strengthening symmetric consistency across semantically equivalent prompt permutations reduces prediction flips but simultaneously increases under-triage. They concetualise this phenomenon by decomposing invariance learning into agreement contraction and shared-solution selection. To control where consensus forms, they then propose two interventions:

  • Disagreement-conditioned risk-directed consensus, a training-time regulariser that augments symmetric ordinal transport with an asymmetric linear term gated by scale-normalised within-orbit disagreement and
  • Orbit-budgeted consensus optimisation, a post-hoc convex programme that freezes the learned backbone representation and refits only the final linear classification head to minimise a surrogate under-triage loss subject to training budgets on multiclass cross-entropy, opposite-side risk and within-orbit logit dispersion.

The authors validate their framework on three clinical triage datasets using a 4B LLM backbone with QLoRA, replicate the trajectory across four additional 7B–14B backbones, and provide a non-ordinal cross-domain mechanism check on CIFAR-10-LT using a ResNet-32.

Critical strengths
  • The distinction between agreement contraction and shared-solution location under asymmetric risk is well-motivated and clearly demonstrated. Showing across three triage datasets, five LLM backbones and CIFAR-10-LT that strengthening symmetric consistency reliably lowers permutation flip rates while systematically degrading recall on the high-risk or minority class is a valuable empirical finding for practicioners deploying invariance regularisers in safety-critical applications.
  • The evaluation protocol uses pre-frozen configurations across five seeds on sealed test splits, tests multiple LLM families, and includes extensive diagnostic studies in the Appendix
Critical weaknesses
  • Section 4.1 motivates DGN using a reliability-weighted sibling teacher Tik=sg(∑vαivFivk)T_{ik} = \mathrm{sg}(\sum_v \alpha_{iv} F_{ivk}) derived from gold-label expected ordinal error eiv=∑jpivj∣j−yi∣e_{iv} = \sum_j p_{ivj}|j - y_i| and temperature τ=0.05\tau = 0.05. But in App Lemma G1, for the V=2V=2 setting used in all training runs, the teacher weights αiv\alpha_{iv}, temperature τ\tau, and label-dependent reliability scores eive_{iv} cancel out identically from the pseudo-gradient of the transport objective. Consequently, the transport gradient ∂~Ra/∂Fˉk=−aωkgk\tilde{\partial} \mathcal{R}_a / \partial \bar{F}_k = -a \omega_k g_k is blind to the ground-truth label yiy_i and unconditionaly pushes all disagreed examples towards the urgent classes. Presenting the gold-label reliability teacher in the main text as part of the mechanism is misleading when it has zero mathematical effect on the gradient for V=2V=2.
  • The paper argue that because ∣divk∣≤sik|d_{ivk}| \le s_{ik}, the directional component satisfies ∣agikdivk∣≤asik→0|a g_{ik} d_{ivk}| \le a s_{ik} \to 0 as siblings agree but this confuses the loss value with the directional force (the pseudo-gradient). Because TikT_{ik} and gikg_{ik} are detached via stop-gradient, ∂~divk/∂Fivk=1\tilde{\partial} d_{ivk} / \partial F_{ivk} = 1, so the factor divkd_{ivk} disappears upon differentiation and the directional force is −λaωkgik-\lambda a \omega_k g_{ik}. So, the loss-value bound ∣−adivk∣≤asik→0|-a d_{ivk}| \le a s_{ik} \to 0 holds even for ungated OQT (gik≡1g_{ik} \equiv 1). Because mtm_t is an exponential moving average of the 0.9-quantile of siks_{ik}, gik=min⁡(1,sik/mt)g_{ik} = \min(1, s_{ik}/m_t) measures within-batch relative rank rather than absolute convergence, maintaining a mean of gˉ≈0.462\bar{g} \approx 0.462. So, if siks_{ik} and mtm_t shrink together during training, the directional force on pairs with non-zero spread does not vanish.
  • This makes the comparison between DGN (a=0.5a=0.5, where mean gate gˉ≈0.462\bar{g} \approx 0.462 yields an effective directional coefficient agˉ≈0.231a\bar{g} \approx 0.231) and ungated OQT (a=0.5,g≡1a=0.5, g \equiv 1, effective directional coefficient 0.500.50) in Table 1 and Figure 2d confounded by a more than twofold difference in average common-mode force. In Table 12 (page 33), when DGN's directional coefficient is increased to a=0.9a=0.9 (so that agˉ≈0.42a\bar{g} \approx 0.42, closer to OQT's 0.500.50), DGN exhibits the same over-steering and QWK/over-triage degradation as OQT
  • In Sec 5.1, you contrast DGN against label-side cost-sensitive losses without consistency regularisation and symmetric consistency without cost weighting concluding that, among trained objectives, only DGN reduces both under-triage and flip relative to CE. However, because label-side cost weighting and symmetric contraction operate on complementary axes, the direct baseline is WCE + BWOT (or OCE + BWOT). Eg. on MIMIC, WCE alone already achieves much lower under-triage and higher QWK than DGN, lacking only permutation consistency. And when you do evaluate the combination of a label-side method with symmetric consistency on CIFAR-10-LT in App F3 it outperforms DGN across every metric. So Without WCE + BWOT in Table 1, it is unclear why an unconditional, label-blind directional push on disagreed views should be preferred over combining standard label-aware cost sensitivity with symmetric consistency.
  • While OBCO substantially lowers under-triage, I think it sacrifices the permutation consistency that motivated the paper: on all three clinical datasets, DGN + OBCO produces a higher permutation flip rate than unregularised CE and on MIMIC it doubles critical-boundary crossing compared to a simple global critical-bias shift.

Nits:

  • In Appendix C the EMA momentum is denoted m=0.99m=0.99, but in Sec 4.1 mtm_t is the scale and β=0.99\beta=0.99 is the EMA momentum.
  • In Appendix B, the exact training and development set sizes are omitted from the dataset descriptions and can only be pieced together from App E4
Questions for authors
  • Could you report the performance of WCE + BWOT (and/or OCE + BWOT) on the three clinical triage datasets in Table 1 under the same λ=3\lambda=3 setting? Given that Balanced Softmax + SYM outperforms DGN on CIFAR-10-LT and WCE outperforms DGN on MIMIC in UU, NPV, LR−\mathrm{LR}^-, and QWK (Table 1), how does WCE + BWOT compare to DGN on the Under–Flip frontier?
  • Can you compare DGN against ungated OQT at a matched effective common-mode force or compare the full Pareto frontier of (U,O,Flip)(U, O, \mathrm{Flip}) swept over a∈[0,0.9]a \in [0, 0.9] for both DGN and OQT)? Does disagreement gating provide a better (U,O,Flip)(U, O, \mathrm{Flip}) trade-off than OQT when matched for average directional gradient magnitude, or does your "gate" primarily act as a ≈0.46×\approx 0.46\times rescaling of aa?
  • Regarding Sec 4.1 and Lemma G1: since αiv,τ,eiv\alpha_{iv}, \tau, e_{iv} cancel out identically from the pseudo-gradient for V=2V=2, and since the pseudo-gradient is −λaωkgik-\lambda a \omega_k g_{ik} rather than O(gikdivk)O(g_{ik} d_{ivk}), you should revise Section 4.1 so the main text accurately reflects the actual gradient applied during training
  • In Appendix B, please provide the exact numbers of source cases and permutation rows in the training and development splits for MIMIC, EMS-M, and ER-Reason, as well as a concrete example explaining how a clinical note is segmented into atomic facts and permuted across the two narrative voices
Reviewer_BRSKOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The work deals with consistency loss, where a group of equivalent inputs is assigned a common prediction. Here, the work points out that when predictions of these equivalent samples are distributed around the decision boundary, consistency regularization necessarily picks a side without preference. The work argues that this is an issue when the classes forming this boundary have different levels of associated risk, for instance, the severity index in emergency triage. The work proposes a novel regularization term which incorporates the asymmetric risk between classes, which the work calls "DGN". In addition, the work introduces a constrained objective the work calls "OBCO" to fine-tune the last layer, which it solves using L-BFGS. The work measures the success through the rate of under-triage and the consistency among equivalent inputs. The work analyzes DGN, as well as a combination of DGN and OBCO on 3 different LLMs on 3 different triage datasets, on which it demonstrates consistent improvements.

Critical strengths
  1. The paper deals with an important and highly relevant problem at the intersection of consistency learning and cost-sensitive learning.
  2. The application of the work in emergency triage is very meaningful.
  3. The work has an extensive appendix, which explains all important details of the work.
Critical weaknesses
  1. The main weakness of the work is that it is very hard to follow. Concepts are not sufficiently introduced, acronyms are not explained ahead of time. The abstract and most parts of the work are written in an overly technical way (the introduction was a bit easier to understand). The work also appears to use non-standard ML language such as "sealed-test" (assuming the authors mean a held-out test set?).
  2. While the appendix explains many concepts well, the current writing makes it a necessity to read. For example, the under-triage rate and the flip measure are essential to the evaluation, yet they are only explained in the appendix.
  3. The experimental section is missing conclusions. Results from the table are simply restates in text. The meaning behind results are not sufficiently discussed.
  4. The majority of improvement appears to stem from OBCO, but the analysis in the main manuscript is quite short, and instead focuses on DGN and the combination DGN+OBCO. While this analysis was conducted more thoroughly in the appendix, the paper should put more emphasis on OBCO without DGN.
Questions for authors
  1. I did not understand the idea behind the "sibling-teacher" in DGN. Could you explain this?

  2. Why do you need a combination of OBCO and DGN? Should not one or the other be sufficient?

  3. OBCO directly optimizes for the under-triage metric, which seems to be directly reflected in the big gap in Table 1 between DGN and DGN+OBCO. Could this be a problem?

  4. I did not understand which of the measures track over-triage. Could you explain this to me? Or is this not necessary, and if so, why?

  5. There are five hyper-parameters for DGN, which seems to be a lot. Are they really necessary? How did you choose them? Did you conduct a hyper-parameter search?

  6. Why do you use only two "siblings" in the experiments? It seems to me that this might be too few to learn a good common prediction?

  7. Do you think there are any differences between training DGN from scratch or with LoRA?

Minor suggestions that did not influence my review.

  1. The abstract mentions a CIFAR-10 study that is mostly in the appendix.
  2. What do "DGN" and "OBCO" stand for? The sections in the text where they are introduces describe them, but the descriptions do not match the acronyms.
  3. The introduction states results in the very first paragraph. It would make more sense to focus on describing the setup and problem first before jumping into results.
  4. Figures 1 and 2 are too small for print, it was almost impossible to read.
  5. The legend of Figure 2 left is positioned at the right figure.
  6. Eq. 2 was hard for me to understand, as many symbols are defined in-text. It would be easier to understand if more details where written in the regularization term

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.