- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Authors introduce REALFEEDBACK — a data-construction paradigm that treats online communities as continuously operating sources of preference supervision. It converts naturally occurring human interactions into reward signals without new largescale annotation. The central research question is how to systematically transform heterogeneous social interactions into reward supervision. The authors collect user feedback data from two communities: Critique Circle (evaluation of individual story fragments) and Baidu Tieba (evaluation of engagement in forum discussions). A Process Reward Model (PRM) is trained on 368 thousand fragment annotations, and an Outcome Reward Model (ORM) is trained on 143 thousand preference pairs. For evaluation, REALFEEDBACK-BENCH was created with 1,500 stories and 1,500 forum discussions; the models outperform existing reward models on most metrics and show the best results in Best-of-N story selection. The key takeaway: it is important not simply to use behavioral data, but to correctly interpret user interactions, since the number of replies alone does not allow reliably predicting preferences.
Critical strengths
The idea is intuitive, paper is well-structured and clear, the data collection methodology and the training phases are described in details.
Critical weaknesses
In the proposed approach to building the PRM, the reward signal is effectively reduced to the sentiment of comments with a minor heuristic correction. However, the evaluation of creative texts is multidimensional and includes many criteria: originality, composition, character development, style, emotional impact, and other aspects. It is not obvious how well such a simplified signal actually reflects text quality and is capable of accounting for different aspects of creative work. Accordingly, the question remains open as to how transferable the proposed approach is to other criteria for evaluating creative content.
A very detailed pipeline is described, however the data themselves are closed.
Please add confidence intervals and paired significance tests.
Questions for authors
Was agreement among the human annotators themselves measured?
Formulas 3 and 4 are confusing in the aggregation strategy. Could the authors clarify which formulation was used for LitBench, StoryRMB, and the Best-of-N experiments?