acceptodds
Under review as a conference paper at ICLR 2027

Sign Betting for Misspecified Regression: Variance Adaptivity and Optimal Approximation

Abstract

We study bounded regression under model misspecification, evaluating a function by its distance to the conditional mean , rather than the standard . This metric is natural for downstream decision problems such as bandits and reinforcement learning. We analyze the min–max estimator of Baraud & Maillard (2025), which we call *sign-betting regression*. Our main results show that sign-betting regression achieves high-probability oracle inequalities under misspecification. Under realizability, our bound adapts to the average conditional variance and recovers the variance-adaptive rate of Li et al. (2026). We further characterize the optimal universal approximation factor for proper learning by showing that sign betting achieves factor 3, while no proper learner can satisfy a distribution-free, high-probability oracle inequality with any universal approximation factor strictly below 3. We extend the upper-bound analysis beyond finite classes and complement the theory with experiments in synthetic and real-world data model-selection settings.

Primary area:
learning theory
Keywords:
regression, model misspecification, variance-adaptive bounds

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_BqJQOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper deals with the problem of estimating the regression function η(x)=E(f(Y)∣X=x)\eta(x) = {\bf E}(f(Y)\vert{}X=x) given nn i.i.d. observations (Xi,Yi)(X_i,Y_i). As opposed to many results in the literature, the quality of the estimator f^{f} is measured by the integrated absolute error ∫∣f^(x)−η(x)∣ PX(dx)\int \vert{}\hat f(x) - \eta(x)\vert{}\,P_X(dx), where PXP_{X} is the distribution of the feature vector XX. This contrasts with standard settings where performance is evaluated via the squared integrated error, or where the mean absolute error is computed with respect to the conditional median rather than the conditional mean. The paper is mainly theoretical, and its main contribution is a high-probability upper bound on the error of the sign-betting estimator introduced by Maillard and Baraud. Additionally, it demonstrates that existing upper bounds for classic estimators, such as Empirical Risk Minimization (ERM) under squared and logarithmic losses, lead to suboptimal risk bounds, with the sub-optimality residing specifically in the approximation error.

Critical strengths

The paper is well-written and provides a clear motivation for the problem under consideration. The main results are stated with all the necessary rigour. All relevant results I am aware of are adequately cited. I did not check the supplementary material for the proofs and other additional material, but all the statements are highly plausible.

Critical weaknesses

The main weakness is the level of novelty. As the authors honestly mention, the paper builds on the prior work by Maillard and Baraud, who considered the same problem for regression with a deterministic design. I agree that, from a technical point of view, the case of random design is different from the case of deterministic design, and there is no direct conversion between the two. However, it is already well understood in the statistical and ML communities that, up to some extra work, the risk bounds established for deterministic design carry over to the random design.

The second weakness is the condition that the response YY is in (0,1)(0,1), and the same is true for the values of all the functions.

A minor remark is that the bibliography, probably generated by an LLM, has not been correctly checked. In particular, the book "Introduction to nonparametric estimation" has only one author: A. Tsybakov.

Another minor remark is that both the abstract and the introduction mention empirical results. However, the main body of the paper contains only mathematical results. In such a situation, it would be more appropriate to free up some space in the main body for including at least one page of empirical results.

Questions for authors

I have only one, very general question. Is there a case where the oracle inequality established in this paper leads to optimal rates in some sense (for instance, minimax) whereas the inequality that upper bounds the mean-absolute error by the MSE and then applies the risk bound on the MSE leads to sub-optimal rates?

2 comments

We thank the reviewer for the thoughtful comments and questions. We address each point below.

[W1] We agree that extending fixed-design results to random design itself is not the main novelty of our work. Beyond establishing high-probability guarantees for population error, our analysis introduces the comparator-dependent approximation quantity BP(f0)B_P(f_0) and second-order quantity v(f0)v(f_0), which are not ingredients provided by the fixed-design result of Baraud and Maillard and yield finer guarantees. Together with our lower bound, our results provide a sharp characterization of the optimal universal approximation factor for proper learning under our learning criterion.

[W2] We agree that boundedness is an assumption of our analysis. The specific range [0,1][0,1] is only a matter of normalization, and the results extend directly to responses and predictors taking values in any bounded interval by affine rescaling. This boundedness assumption is naturally satisfied in many of our motivating applications, for example, when rewards are bounded in bandit or RL, or when responses are human ratings on a bounded scale. Extending the high-probability population guarantees to unbounded or heavy-tailed responses is an interesting direction beyond the scope of the current work.

[W3] We thank the reviewer for pointing out the bibliographic error. We will correct the entry for "Introduction to nonparametric estimation" and carefully check the bibliography in the final version. For clarification, we did not use an LLM to generate or modify the bibliography. The BibTeX entry for "Introduction to nonparametric estimation" was copied from Google Scholar. We acknowledge that we did not independently verify the correctness of the Google Scholar BibTeX entry, which led to this error.

[W4] We agree that the empirical results should be more visible in the main paper. We will move a subset of the experimental results from the appendix to the main text in the final version.

[Q] Yes. A simple realizable noiseless example shows that converting an MSE guarantee to L1L_1 via Cauchy-Schwarz can lose the optimal rate. Consider F={f0,f1}\mathcal F=\lbrace f_0,f_1\rbrace, where f0≡0f_0\equiv 0 and f1=1{x∈A}f_1=\mathbf{1}\lbrace x\in A\rbrace for a set AA with PX(A)=p=1/nP_X(A)=p=1/n, and let Y=fj(X)Y=f_j(X) deterministically for some j∈{0,1}j\in\lbrace 0,1\rbrace. If none of the nn samples falls in AA, which occurs with constant probability (1−1/n)n=Θ(1)(1-1/n)^n = \Theta(1), the two regression problems are indistinguishable to any learner. On this event, for any output gg,

\ge \lVert f_0-f_1\rVert_1 =p,$$ so at least one of the two problems has $L_1$ error at least $p/2$. Similarly, $$\frac{1}{2}\bigl(\lVert g-f_0\rVert_2^2+\lVert g-f_1\rVert_2^2\bigr) \ge \frac{p}{4}.$$ To see this, on $A$ we have $f_0=0$ and $f_1=1$, and hence $$\frac{1}{2}\bigl(g(x)^2+(g(x)-1)^2\bigr) =\left(g(x)-\frac{1}{2}\right)^2+\frac{1}{4} \ge \frac{1}{4}.$$ Integrating over $A$, which has probability $p$, gives the inequality above. Since the indistinguishable event occurs with constant probability, this gives minimax lower bounds of $\Omega(p)$ for both $L_1$ risk and MSE. For the upper bound, consider an estimator that outputs the identified regression function if a sample from $A$ is observed, and otherwise outputs $f_0$. Its risk under either metric is at most $p(1-p)^n=O(p)$. Hence $$\inf_{\widehat f}\sup_{j\in\lbrace 0,1\rbrace} \mathbb E_j\lVert\widehat f-f_j\rVert_1 \asymp \frac{1}{n},$$ $$\inf_{\widehat f}\sup_{j\in\lbrace 0,1\rbrace} \mathbb E_j\lVert\widehat f-f_j\rVert_2^2 \asymp \frac{1}{n}.$$ Our realizable noiseless guarantee in Corollary 10 also gives $L_1$ error $O(1/n)$ for this two-function class with constant probability. In contrast, even starting from a minimax optimal MSE guarantee of order $1/n$, upper bounding $L_1$ error by the square root of MSE and applying the error bound on the MSE only yields an $O(n^{-1/2})$ $L_1$ bound. To summarize, in this example, directly analyzing the $L_1$ error achieves the optimal $1/n$ rate, whereas converting an MSE bound via Cauchy-Schwarz inequality loses a square root and gives a suboptimal rate.
Reviewer_BqJQOpenReview ↗

acknowledgment of reading the reply

Thank you for your reply. The last example is clear; in simple words, it says that L2->L1 conversion is suboptimal in cases where the O(n^(-1/2)) term in eq (17) vanishes. This means that it essentially happens only in the noiseless setting. I find that this is an interesting observation.

I maintain my score, and find that the paper deserves acceptance.

Reviewer_RHuoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

For i.i.d. , the paper shows theoretical guarantee for SBR in the form of oracle-inequality. In section 3, the paper irons out two failures for L2L_2 and log-loss ERM in terms of population L1L_1 risk. Later, the main selling point for SBR is to have a high-probability oracle inequality. Theorem 13 establishes factor 3 optimality for proper learners, and Theorem 14 extends the upper bound and goes beyond finite class.

Critical strengths
  1. The paper is nicely presented and the motivation for the is very clear.
  2. All the results appear solid for me. Under realizability, the result recovers the noise adaptation studied by Li while, under misspecification, it also exposes approximation residuals. This is a meaningful extension of an existing estimator.
Critical weaknesses

It is somewhat unsurprising that if one targets on L1L_1 generalization, the training or learning procedure should be aligned. Also see the questions below.

Questions for authors
  1. It seems that L1L_1 ERM is mentioned briefly in the main paper, not detailed only in Appendix E. As stated at the beginning of the section, this is indeed a natural question to ask, and moving the Bernoulli example (on page 27) would help reduce readers’ confusion.
  2. [Cont'd point 1] Could you add the simulation result for L1L_1 ERM in the same setting?
  3. The comparison of Theorem 8 with the fixed-design result of Baraud and Maillard followed by global uniform convergence. Is it possible that a localization or peeling argument may improve their result? The current statement (please correct me if I am wrong) seems to leave open whether there is a genuine improvement or whether the analysis is just not sharp enough.. Resolving this is beyond the paper’s main scope and is not something the authors are expected to address. I ask purely out of curiosity.
2 comments

We thank the reviewer for the thoughtful comments and questions. We address each point below.

[Weakness] We agree that a very natural attempt to control L1L_1 generalization from the conditional mean is to use L1L_1 ERM. However, as discussed in Section 3 and Appendix E, L1L_1 ERM targets the conditional median and is misaligned with the conditional mean. The difficulty is that there is no obvious pointwise L1L_1-like loss whose population minimizer is the conditional mean while controlling the L1L_1 error. Sign betting addresses this difficulty through its pairwise min–max construction.

[Q1] We agree that the distinction from L1L_1 ERM is important and will move the Bernoulli example illustrating the failure of L1L_1 ERM into the main text in the final version..

[Q2] We agree that L1L_1 regression is an important and natural baseline. Please see our general response.

[Q3] We agree that a more refined localization or peeling argument may improve the random-design guarantee obtained by converting the fixed-design result of Baraud and Maillard in Appendix I, and our current comparison does not rule this out. However, our comparator-dependent approximation quantity BP(f0)B_P(f_0) and second-order quantity v(f0)v(f_0) are not ingredients provided by their fixed-design result. Introducing these comparator-dependent population quantities and developing an analysis in terms of them require additional arguments beyond their fixed-design guarantee.

Thank you for adding the absolute-loss ERM baseline. The additional experiments address my main empirical question, and the results in noisy and noiseless settings make the comparison much more informative.

I also appreciate the acknowledgment that localization or peeling could (potentially) improve the black-box conversion from the fixed-design result. As stated in my original review, ruling out such refinements is not necessary for acceptance.

I remain my original score for the overall impression for the work: solid yet not super novel.

Reviewer_AnHyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies statistical learning with L1 loss. It provides a theoretical guarantee in the form of the oracle inequality. It compared three oracle inequalities based on different loss functions.

Critical strengths

The paper studies a new loss function based on the so-called signed betting loss, the theory presentation is clear, self-contained, and easy to understand.

Critical weaknesses

I feel that the paper is just playing with the loss function. Basically, it is intuitive that if you take the sub-gradient of the L1 loss ∣y−x∣|y-x|, then you get some (y−x)sign(y−x)(y-x)\text{sign}(y-x), then you should use it as your loss. So the loss funcion design is not that new to me. I think there should be some existing results in scoring rules and Bregman divergence that already covered part of this.

The paper also does not answer the computation cost. It reduces a ERM to a minimax optimization problem, which further increase the difficulty of solving this problem.

Questions for authors

See the weakness part.

3 comments

We thank the reviewer for the comments. We would like to clarify that our main contributions are variance-adaptive oracle inequalities for sign betting under misspecification and a characterization of the optimal universal approximation factor for proper learning.

[(1) Connection to absolute loss and Bregman divergences.]

We do not follow the reviewer's argument: "if you take the subgradient of the L1 loss ∣y−x∣|y-x|, then you get some (y−x)sign⁡(y−x)(y-x)\operatorname{sign}(y-x), then you should use it as your loss."

First, we do not see the mathematical justification for taking the subgradient of a loss function and then using that subgradient as a new loss function. Second, the expression (y−x)sign⁡(y−x)(y-x)\operatorname{sign}(y-x) mentioned in the review is exactly the original absolute loss ∣y−x∣|y-x|, rather than its subgradient.

As discussed in Appendix E (and our response to Reviewer xxVQ), absolute loss and sign betting are very different objectives and lead to very different statistical properties. Ordinary absolute loss uses (Y−f(X))sign⁡(Y−f(X))=∣Y−f(X)∣(Y-f(X))\operatorname{sign}(Y-f(X))=|Y-f(X)|, whereas sign betting uses (Y−f(X))sign⁡(h(X)−f(X))(Y-f(X))\operatorname{sign}(h(X)-f(X)). Note that the latter sign does not depend on the observed label YY.

We agree that scoring rules and Bregman divergences provide a general framework for constructing loss functions that are consistent for the conditional mean. However, consistency for the conditional mean does not imply a constant-factor L1L_1 oracle guarantee under misspecification. Indeed, Propositions 2 and 4 in Section 3 establish that squared-loss and log-loss ERM can have population L1L_1 error Ω(ε1,∗)\Omega(\sqrt{\varepsilon_{1,*}}), even in noiseless regression with a two-element hypothesis class. In contrast, sign betting achieves a variance-adaptive oracle inequality, as well as the optimal universal approximation factor 33 for distribution-free, high-probability proper learning.

[(2) Computational complexity.]

We agree that minimax optimization can be computationally challenging for general hypothesis classes. For a finite class F\mathcal{F}, sign betting requires computing ∣F∣2|\mathcal F|^2 pairwise empirical scores and selecting the predictor minimizing the maximum score. This takes O(n∣F∣2)O(n|\mathcal F|^2) time, compared with O(n∣F∣)O(n|\mathcal F|) for ordinary ERM. Developing optimization-friendly approximations of sign betting for general infinite hypothesis classes remains an interesting direction for future work.

I thank the authors for the responses. To my understanding, this paper provides a random design on the sign-betting framework developed by Baraud and Maillard (2025, Section 2.2). To do that, essentially you apply the minimax optimality of f^\hat{f}, Zf,h+Zh,f=Df,hZ_{f,h}+Z_{h,f}=D_{f,h}. These are both things that already hold in Baraud and Maillard (2025, Section 2.2). Additionally, you apply Bernstein inequalities over ∣F∣2|\mathcal F|^2 to transfer to the random design simply because you are now applying uniform concentration to the comparison functions over F×F\mathcal F\times \mathcal F, which are also standard when people try to transfer from fixed design to random design.

According to your rebuttal to other reviewers, you mentioned that BP(f0)B_P(f_0) and v(f0)v(f_0) are new to random design. But essentially, these quantities arise naturally from retaining the population means and second moments of the pairwise comparison statistics: BP(f0)=sup⁡h∈FE[Zf0,h]B_P(f_0)=\sup_{h\in\mathcal F}\mathbb E[Z_{f_0,h}] is the population comparison objective, while v(f0)=sup⁡h∈FE[Zf0,h2]v(f_0)=\sup_{h\in\mathcal F}\mathbb E[Z_{f_0,h}^2] is the standard second-moment bound used in Bernstein's inequality. Their absence from the stated fixed-design guarantee does not, by itself, establish that obtaining them requires a new technical argument.

We thank the reviewer for the follow-up. We would like to clarify several points.

First, we interpret “both” as referring to the minimax optimality condition of f^\widehat f and the identity Zf,h+Zh,f=Df,hZ_{f,h}+Z_{h,f}=D_{f,h}. Since our paper analyzes the min–max estimator of Baraud and Maillard, which we explicitly credit to them, its empirical minimax optimality is a starting point of our analysis. To the best of our understanding, we do not find Zf,h+Zh,f=Df,hZ_{f,h}+Z_{h,f}=D_{f,h} stated or used in the analysis of Baraud–Maillard Section 2.2.

Second, we do not follow the characterization that our random-design result is obtained by "applying Bernstein inequalities over ∣F∣2|\mathcal F|^2 to transfer to the random design". In particular, we do not see how the "simply because" characterization follows here: uniform concentration over pairwise statistics is a tool used in our direct random-design analysis, but this analysis does not directly apply the fixed-design theorem. In fact, Appendix I explicitly studies such a black-box conversion: we condition on X1:nX_{1:n}, apply the Baraud–Maillard fixed-design theorem, and then use a standard uniform-convergence argument over the risk functions indexed by f∈Ff \in \mathcal F to pass from empirical risk to population risk, rather than "uniform concentration to the comparison functions over F×F\mathcal F\times \mathcal F". This produces a different and weaker population guarantee. Our proof of Theorem 8 analyzes sign betting directly under random design.

Third, we did not claim that BP(f0)B_P(f_0) and v(f0)v(f_0) are “new to random design.” Our statement to other Reviewers RHuo and BqJQ was that these comparator-dependent population quantities BP(f0)B_P(f_0) and v(f0)v(f_0) “are not ingredients provided by their fixed-design result,” and that introducing them and developing an analysis in terms of them require additional arguments beyond that fixed-design guarantee. The reviewer later also notes "their absence from the stated fixed-design guarantee."

Fourth, we agree that Bernstein's inequality itself is a standard concentration tool. However, our proof of Theorem 8 is not just an application of Bernstein's inequality. The key proof idea of Theorem 8 is summarized in lines 345–355 in Section 4.3. It builds directly on the population geometry of sign betting developed in Section 4.2. We do not find this proof architecture in the fixed-design analysis of Baraud and Maillard. Bernstein's inequality is a tool used for the concentration argument in the proof of Theorem 8.

Finally, we would like to emphasize that the paper's contribution is not limited to the finite-class random-design upper bound. Theorem 13 gives an information-theoretic lower bound on the universal approximation factor under our learning criterion. Theorem 13 applies to any possibly randomized proper learner and rules out every universal approximation factor strictly below 33. Together with the matching upper bound, this characterizes the optimal universal approximation factor 33 for proper learning.

Reviewer_xxVQOpenReview ↗
Rating
4
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The authors study the l1 error guarantees of the estimator of Baraud and Maillard 2025, which they refer to as sign betting regression, in the misspecified regression setting.

This is an important problem as l1 error control on rewards naturally arises in regret control in bandit and reinforcement learning.

The SBR objective is the inf over candidates in the hypothesis class of the sup against competitor in the same class of regression residual of the hypothesis times sign of candidate minus candidate. The inner objective has the interpretation of a best bet against the candidate.

The authors study the interesting question of whether SBR achieves both variance adaptation in the misspecified setting while not paying more than a constant times l1 misspecification error. They answer positively with a generalization bound exhibiting factor of three for the latter constant.

They provide a lower bound on the tightness of the three factor over so-called proper learners.

Critical strengths

A pleasure to read, very well written and clear.

I found the discussion of earlier works using classification losses vs regression losses for bounded outcomes in the correctly specified case appropriate and their square root misspecification l1 cost a clear motivation of the study of an alternative procedure that would not pay this price.

The generalization bound is transparent: it trades off misspecification l1 error of the hypothesis, worst case bet against candidate and variance of outcome minus candidate times log cardinality of hypothesis class.

Critical weaknesses

Experiments could be foregrounded earlier in the main text.

Selection on l1 error directly in the experiment should be a natural baseline which the authors don't discuss in the experiments.

Questions for authors

What are existing guarantees of l1 regression (mean average error minimization) in terms of variance adaptation + dependence on l1 misspecification error in generalization error?

1 comment

We thank the reviewer for the thoughtful comments and questions. We address each point below.

[W1] We agree that the empirical results should be more visible in the main paper. We will move a subset of the experimental results from the appendix to the main text in the final version.

[W2] We agree that L1L_1 regression is an important and natural baseline. Please see our general response.

[Q: Guarantees for ordinary L1L_1 regression.]

We thank the reviewer for this question. It is not entirely clear to us which generalization-error metric the reviewer is referring to. We therefore address our two interpretations: the population absolute-loss risk Rabs(f):=E∣Y−f(X)∣R_{\mathrm{abs}}(f):=\mathbb E|Y-f(X)|, which is the population risk of ordinary L1L_1 regression, and our performance measure ε1(f):=E∣η(X)−f(X)∣\varepsilon_1(f):=\mathbb E|\eta(X)-f(X)|, where η(X)=E[Y∣X]\eta(X)=\mathbb E[Y\mid X].

(1) Generalization guarantees for RabsR_{\mathrm{abs}}.

For a finite hypothesis class F\mathcal F, standard uniform convergence gives, with probability at least 1−δ1-\delta,

\le \inf_{f\in\mathcal F}R_{\mathrm{abs}}(f) +\sqrt{\frac{2\log(2|\mathcal F|/\delta)}{n}}.$$ **(2) Generalization guarantees for $\varepsilon_1$.** Recall that $\varepsilon_{1,*}:=\inf_{f\in\mathcal F}\varepsilon_1(f)$ and $\bar\sigma^2:=\mathbb E[\operatorname{Var}(Y\mid X)]$. For any $f$, we have $\varepsilon_1(f)\le R_{\mathrm{abs}}(f)$. On the other hand, by the triangle inequality and Cauchy--Schwarz, $$R_{\mathrm{abs}}(f)\le \varepsilon_1(f)+\mathbb E|Y-\eta(X)|\le \varepsilon_1(f)+\sqrt{\bar\sigma^2}.$$ Combining these inequalities with (1) yields $$\varepsilon_1(\widehat f_{\mathrm{abs}}) \le \varepsilon_{1,*}+\sqrt{\bar\sigma^2}+\sqrt{\frac{2\log(2|\mathcal F|/\delta)}{n}}.$$ Unlike our sign-betting guarantee, this bound contains a variance-dependent term $\sqrt{\bar\sigma^2}$ that does not vanish with sample size. This is not merely an artifact of the analysis. As demonstrated by the Bernoulli counterexample in Appendix E, ordinary absolute-loss ERM can be inconsistent for the conditional mean even under realizability: $\varepsilon_{1,*}=0$, yet $\varepsilon_1(\widehat f_{\mathrm{abs}})\to p>0$ as $n\to\infty$. Therefore, ordinary $L_1$ regression cannot generally satisfy $\varepsilon_1(\widehat f_{\mathrm{abs}})\le C\varepsilon_{1,*}+o_n(1)$ for any universal finite $C$, even under realizability. **In the noiseless case**, $Y=\eta(X)$ almost surely, so $R_{\mathrm{abs}}(f)=\varepsilon_1(f)$, and ordinary $L_1$ regression achieves an oracle inequality with an approximation factor 1. This is consistent with our additional experiments, where ordinary $L_1$ regression achieves the lowest MAE in both the noiseless synthetic experiment and the LLM experiment with noiseless teacher.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Additional experiments: Absolute-loss ERM baseline (Reviewers xxVQ and RHuo)

We thank the reviewers for suggesting ordinary L1L_1 regression as an additional baseline. We include absolute-loss ERM in all three experimental settings, selecting from the same candidate class using the same validation data as the other methods. We have updated Appendix M to address the reviewers' comments. All revised or newly added content is highlighted in blue in the updated PDF for ease of reference.

[Experimental results.]

In the synthetic experiment (Appendix M.1, Figure 1 and Table 1), absolute-loss ERM achieves the lowest MAE at every tested sample size in the noiseless setting (ρ=0\rho=0). In contrast, with noisy observations (ρ∈{0.02,0.04}\rho\in\lbrace 0.02,0.04\rbrace), sign betting achieves the lowest MAE across all tested sample sizes, and its MAE decreases as the sample size increases. Similarly, in the real-world human-rating experiment (Appendix M.2, Figure 2(a)), sign betting consistently achieves the lowest average test MAE. In the LLM experiment (Appendix M.3, Figure 2(b)), where teacher rewards are deterministic, absolute-loss ERM achieves the lowest MAE across all validation sizes, while sign betting outperforms squared-loss and log-loss ERM.

[Connection to theory.]

These observations are consistent with the theoretical distinction between ordinary absolute-loss regression and sign betting. In the noiseless setting, Y=η(X)Y=\eta(X) almost surely, so Rabs(f)=ε1(f)R_{\mathrm{abs}}(f)=\varepsilon_1(f). Thus, absolute-loss ERM directly minimizes the empirical counterpart of our target ε1(f)\varepsilon_1(f) and admits an oracle inequality with approximation factor 11 (see our response to Reviewer xxVQ).

Under noisy observations, however, Rabs(f)R_{\mathrm{abs}}(f) generally differs from ε1(f)\varepsilon_1(f), and their minimizers over a misspecified hypothesis class may not coincide. Notably, in our noisy synthetic experiment, the conditional label distribution is a symmetric Beta distribution centered at η(x)=1/2\eta(x)=1/2, so the conditional mean and median coincide. Nevertheless, sign betting consistently outperforms absolute-loss ERM. This means the advantage of sign betting in the noisy synthetic experiment cannot be explained simply by the fact that ordinary absolute-loss regression targets the conditional median instead of the conditional mean.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.