- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies whether replacing the standard squared-error loss in REBEL with a classification-based loss makes RL more robust to noisy rewards. The authors show that the new loss has the same theoretical guarantees but offers no practical benefit. Any improvements mainly come from limiting the influence of extreme reward values, rather than from using classification instead of regression.
Critical strengths
The theoretical analysis and experiments are presumably solid and comprehensive but the paper is quite incredibly dense and hard to digest so many details have not be audited carefully.
Critical weaknesses
- What is the actual novelty and practical significance of the work? The finding seems to be more a null result, and even then, it is unclear what fundamental new insight is to be gained?
- The theory and experiment gap needs to be better addressed? The theory guarantees require predictions to be within the histogram's interior but that is not necessarily true in experiment?
- Not sure how generalizable the conclusions are given the experiments use a relatively simple synthetic contextual bandit with fixed sampling rather than on-policy REBEL procedure.
- The paper is unnecessarily dense and filled with jargons that are ill-defined and hard to read...
Questions for authors
What is actually the main takeaways besides stating the obvious in the practitioner's takeaway section, any deeper insights?
Dear Reviewer BXNE,
The paper and our first reply were both dense, so here is a short version.
Deeper insight (your question). The natural expectation is that a classification loss is more robust to heavy-tailed reward noise. We preregistered that prediction, and it failed in 20 of 20 seeds: HL was 3.4% worse than MSE, while Huber matched MSE. The reason is that HL limits targets at the edge of its histogram, which biases the fit even when the noise has zero mean. Huber limits residuals, so under symmetric noise it still aims at the mean. This explanation predicts that a wider support shrinks the deficit. It does: from 3.4% to under 0.1% in the bandit, and from 1.7% to 0.2% in a preregistered on-policy MLP test.
Theory and experiments (Weakness 2). You are right that the regret guarantee does not cover our experiments, and the revision says so at the end of Section 4. The results that do apply there are Proposition 2 and Theorem 6. They describe the bias of the fit when targets cross the support edge.
Generality (Weakness 3). The deficit was also tested on-policy with an MLP (400 seeds). It kept its sign at about half the size.
For practice the result is negative, as you say. Our case for the paper is that REBEL's authors posed this question, the answer is now supported by preregistered tests, and the main difference between the losses has an explanation.
We are glad to clarify anything before Nov 18.