- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The issue this paper tries to model is that when a AI is trained against a judge with consistent blind spots, the AI learns to exploit them which drives further training into those blind spots. The main theoretical result is that, under their assumptions, any deterministic rule for which updates to judge as positive eventually hurts performance. This is validated in experiments where on common benchmarks.
Critical strengths
This paper introduces valuable formalization of the issue that self-reinforcement can lead to propagating issues with the judge, along with a stylized model of how that could evolve. The technical results appear sound, as do the experiments, conditional on the assumptions made.
Critical weaknesses
The main insight of this paper, that systemic errors can lead to degenerate feedback loops, is not novel. Given that, the main results are not actionable since it's not clear how to create a judge without deterministic biases. Adding a clear proposal for how to do so would significantly improve this paper.
Relatedly, the experiments feel very contrived, since they impose whether a judge is deterministic or not through scaffolding that would not exist in practical scenarios.
Questions for authors
What kind of approaches appear promising to you for generating judges that do not make deterministic errors?
How would you position your work within the existing literature? The literature review section of your paper is extremely brief?