acceptodds
Under review as a conference paper at ICLR 2027

Consistent Learning-to-Defer with Expert-Conditional Advice

Abstract

We introduce learning-to-defer with advice, a new setting in which a system learns both which expert should handle an input and which information that expert should receive. The policy acts before observing advice and minimizes prediction loss plus expert and acquisition fees. Training may reveal all advice sources or only a subset on each record. We show that composing routing and advice losses can be inconsistent and that augmented training can incur avoidable estimation variance. We propose reference conditional classification, which jointly learns a fixed expert's error probability and each action's error conditional on that expert's success or failure. The reference outcome is observed on every labeled record, so records with missing advice still inform the estimates. We prove a consistency bound for bounded task losses, show that limited training preserves the full population objective, and establish variance reductions for the analyzed fits. On FEVER and Sensitive, full training has the lowest mean dev loss at every evaluated fee, while limited training improves on full training at most matched acquisition budgets.

Primary area:
learning theory
Keywords:
learning to defer, selective prediction, learning theory, rejection, abstention

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_g6HCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

In a concise summary, the paper studies the setting where additional information sources are efficiently and consistently matched with experts which predict labels for given inputs. It contributes methods to consistently compute the errors with surrogates and efficiently via budgeted source observations. It validates the results through experiments and analyses for unbiasedness and optimality bounds.

Critical strengths

A substantive strength of the paper can possibly be the comprehensiveness of the experimental body. However, it is hard to evaluate the rigorousness and reproducibility.

Regarding technical correctness, the analyses are individually easy to follow in general.

Critical weaknesses

The submission is not really clear. Many times the terminology changes abruptly. The presentation of many central elements are not eased into in the first few pages. Also, for technical correctness, there are some potential issues, please see questions.

Regarding presenting novel findings, the importance of many theoretical findings are questionable. Possibly besides showing the consistency of surrogates, the other findings and contributions seem incremental, e.g., pairing of expert and source, budgeting etc.

Questions for authors

Abstract seems to be written in a general language so its hard to pinpoint the exact contributions.

Line 40: "cheapest expected cost under its best source, including no advice", do you mean all the sources are possibly detrimental or did you actually want to point out a trade-off?

Line 58: the use of the term "reference" in Page 1 is confusing. Is the reference outcome whether a fixed reference expert succeeded or not? Also, to clarify, there are experts (with prediction capabilities), paid sources (information before predictions), and paid actions (decisions) which result in loss/error?

Line 163: arbitrary notation changes makes it harder to understand. For example, please define policy loss in (5).

Why is Theorem 3 important? It deals with specific surrogate losses, why are they important?

Line 251: is there two scenarios, either no advice or (j,k) paid pair?

Line 256: why is the surrogate loss for no advice always zero, isn't it also a pair (j,0)? Below here you seem to deduce as such from the equation; however, that would be a circular argument since you disregarded the contribution of no advice already by not introducing a term with (1-I_t). Please clarify.

(12): the normalization with B is not tight since it is a bound, and the probabilities seem heavily dependent on this normalization factor B, why?

1 comment

Thank you for your answers and responses. My concerns have been generally addressed. I will raise my rating to 3.

Reviewer_p8AgOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

In this work, the authors focus on the topic of learning to defer with expert-conditional advice. In this framework, the prediction system can further choose an additional advice source for the selected expert. The performance of this prediction system is evaluated by a cost-sensitive target, which balances the cost of expert/advice and the accuracy of the system. A Bayes-consistent surrogate is further constructed for this evaluation by estimating the misclassification rate of a reference expert (without advice) and the conditional misclassification rate of an expert-advice pair given the correctness of the reference expert. Experiments on FEVER and Sensitive datasets show the effectiveness of this method.

Critical strengths
  1. In this framework, the prediction system can further choose what information should be provided to the selected expert. This is useful for sensitive tasks, where additional information may be costly to acquire or should not be accessed for every input for safety concern.

  2. The proposed surrogate considers the dependency between the misclassification rate of an expert-advice pair and the correctness of a reference expert. While there exists a simple baseline of estimating expert-advice accuracy with proper losses, the variance analyses seem to justify the motivation of the proposed construction.

  3. The provided theoretical analysis is natural and sound. The surrogate regret bound is of the typical square-root type and follows from standard properties of the log loss.

Critical weaknesses
  1. A simple baseline is to directly estimate each expert-advice pair’s misclassification rate given the input by binary cross-entropy, then add the fees to select a pair, which may also give a Bayes-consistent decision rule. I suggest including this baseline in the main experiments with the same training observations and acquisition budgets.

  2. The framework and experiments focus on training a router with fixed experts. I think it would be helpful to discuss how this framework can be extended to jointly train the router and a classifier, to better connect this work to traditional L2D.

  3. The authors are encouraged to discuss more recent related work [1, 2] for a more complete literature review.

[1] Shuqi Liu et al. When More Experts Hurt: Underfitting in Multi-Expert Learning to Defer. ICML, 2026.

[2] Andrea Pugnana et al. Deferring Concept Bottleneck Models: Learning to Defer Interventions to Inaccurate Experts. NeurIPS, 2025.

Questions for authors

Please see the weaknesses.

Reviewer_3uNFOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

this paper extends learning to defer by not only asking which expert the system should defer to, but what additional information that expert could be given

Critical strengths
  • Relevant and interesting problem. It has been assumed for a long time that the information given to the expert in an L2D system has been fixed, but this paper tackles that assumption
  • The authors tackle the problem in a scientific way, first showing the flaws of the intuitive baselines (i.e., why multiplying router and advice losses fails to recover the bayes-optimal expert, and why standard augmented L2D wastes free no-advice observations and amplifies noise when advice is missing), and then designing their proposed method specifically to resolve both issues.
Critical weaknesses
  • From my understanding (please clarify if incorrect), from the paper's methodology, you must pick the expert before seeing the advice. This setup forces the policy to commit to both the expert and advice at the exact same time, using only original input of xx. Perhaps in some real-world cases, once you pay to fetch something like a test result, I would assume that you check it before deciding which expert you want.
  • The paper might be improved by additionally having a direct BCE baseline on the real datasets
Questions for authors
  • Why does the augmented training beat RC at the smallest FEVER budgets (table 7)?
  • Could the framework be naturally extended to choosing the expert after seeing the advice?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.