- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces GeoSRM, an efficient online reward modeling method that uses Frequent Directions to compress historical preference curvature while retaining exact corrections for the current round. It establishes estimation, suboptimality, and regret guarantees that explicitly incorporate sketching error. Experiments validate the effectiveness of the proposed method.
Critical strengths
- The paper addresses an important computational bottleneck in scaling online RLHF: maintaining and operating on high-dimensional preference curvature. The proposed geometry-sketching method substantially reduces these costs while retaining useful directional information, making the contribution practically relevant.
- The analysis explicitly characterizes how sketching affects estimation accuracy and policy performance. Experiments in both active preference optimization and deployment-time adaptation demonstrate favorable efficiency–utility trade-offs, including approximately fivefold algorithm-time savings in the 7B active-learning setting. Together, these results support the method’s soundness and practical effectiveness.
Critical weaknesses
- The (O(\ell d)) cost is linear in (d) only when the sketch size (\ell) is fixed. However, the main experiments use (\ell=d/4), making this cost quadratic in (d). The results therefore demonstrate computational savings but do not establish that dimension-linear scaling preserves utility. Experiments with a fixed sketch budget across feature dimensions would strengthen this central claim.
- The deployment regret guarantee relies on uncertainty-aware response selection, whereas experiments select the highest- and fourth-highest-scoring responses from 64 samples. This substitution leaves the analyzed exploration mechanism empirically unvalidated and weakens the connection between the theoretical guarantees and reported deployment performance.
Questions for authors
- Since the main experiments use (\ell=d/4), under what spectral conditions can (\ell) remain independent of (d) while preserving statistical efficiency? What theoretical or empirical evidence supports these conditions in the evaluated settings?
- Does the deployment regret guarantee extend to the experimental rule that selects the highest- and fourth-highest-scoring responses from 64 samples? If not, what justification or existing evidence connects this heuristic to the uncertainty-aware selection rule analyzed in the theorem?