- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 1
- Soundness
- 2(+1)
Read review
Paper summary
The paper proposes LatentGRM, which extends Latent-SFT with rubric-guided semantic chunking for pairwise reward modeling. It replaces textual evaluations with compact latent trajectories, achieving comparable judgment quality and 6–7× inference speedups. Additional experiments examine criterion-dependent information and reconstruction of teacher-derived evaluations.
Critical strengths
Practical efficiency: On four benchmark domains, LatentGRM-8B achieves 6.1–7.0× lower total inference time at vote@5 while maintaining comparable aggregate preference accuracy across eight domains.
Informative intervention analysis: Replaying trajectories under a fixed baseline rubric provides evidence that criterion-dependent preference information is transmitted through the latent sequence.
Critical weaknesses
- Lack of novelty (The new part is only the chunk compared with LatentSFT):
The latent representation of the vocabulary space, two-stage distillation, and autonomous latent generation are all derived from Latent-SFT. The primary methodological innovation is rubric-guided semantic chunking. While the "first application to reward modeling" represents a novel use case, it is insufficient on its own to establish significant methodological innovation.
- The most prominent acceleration results fail to isolate the specific contribution of the newly introduced method.
The 6–7x speedup is measured against a judge that generates lengthy textual analyses; however, latent-based methods using fixed chunking would similarly reduce the number of generation steps. This comparison demonstrates the practical value of latent evaluation but does not quantify the efficiency gains specifically attributable to the authors' semantic chunking technique. Direct evidence for semantic chunking shows an average improvement of 1.8 percentage points across four domains, alongside a regression in one domain.
- Writing and presentation should be largely improved
a. The abstract begins with a broad statement about reward modeling “jointly representing and reasoning over multiple evaluation criteria,” without first explaining the concrete setting: a generative judge compares two responses by producing textual assessments. This wording makes the motivation unnecessarily difficult to follow. A clearer opening would state that generating lengthy assessments makes such judges expensive, then explain how LatentGRM replaces those assessments with shorter latent sequences.
b. Table 2 presents results for only two methods across four domains, yet the layout is rather sparse, with the space occupied being disproportionate to the amount of information conveyed. Other tables could also be made more compact—by shortening headers, merging duplicate entries, and adjusting column spacing—thereby freeing up more space for the analysis of key experimental results.
c. Please redraw Figure 2 to make the training and inference workflows clearer. Its presentation-like styling and unclear data flow make the framework difficult to follow.
Questions for authors
Could the authors clarify the methodological differences between LatentGRM and Latent-SFT? My understanding is that the vocabulary-space latent representation, two-stage distillation, and autonomous latent generation are inherited from Latent-SFT, with rubric-guided semantic chunking being the primary methodological addition. Is this understanding correct?
6 comments
Thanks for the author's rebuttal. May I know where is the revised version of this paper? I would like to see the abstract and the figure.
Thank you for following up. We have updated the paper PDF on OpenReview. Please download it again from the submission page.
Thank you for the suggestions on writing and presentation. We have revised the abstract, introduction, and experiments to clarify the motivation and connections among the components.
Our contribution is a complete latent reward modeling framework connecting judgment quality, efficient serving, and interpretation. Frequent reward model calls make long assessments costly in data construction and RL. Reward modeling therefore provides a clear and important use case for compact latent assessments.
1. Method contribution and relation to Latent-SFT
We agree that the underlying latent representation, training mechanisms, and speedups from fewer generation steps build on Latent-SFT. However, its original fixed-ratio compression still falls short of explicit CoT models, especially on harder tasks such as Chat Hard and Precise IF. Chat Hard includes subtle and adversarial comparisons [1]. We therefore test whether the semantic structure of reward assessments helps preserve judgment quality under compression.
At the same target budget, semantic chunking improves Chat Hard, Precise IF, and Focus by 2.2, 3.9, and 1.7 points, while Chat decreases by 0.7. It thus provides a validated method improvement within the framework and a direction for further research.
2. LTI and rubric-level reconstruction evaluation
Reward modeling requires both preference judgments and access to their supporting reasons. LTI reads the evaluation input and saved vocabulary probabilities to recover textual assessments. It is trained separately and runs offline on demand. While CODI's token-level probe has limits in recovering multi-token semantic units [2], LTI targets complete criterion judgments and supporting reasons.
A key question is then how to evaluate reconstruction. Heima uses metrics including ROUGE-L [3], but changing "Partially Met" to "Met" can alter a judgment while leaving most wording unchanged. We therefore align assessments by response and criterion, measuring State Macro-F1, full state-vector agreement, and reason similarity.
Table 4 demonstrates why this matters. On teacher-derived trajectories, Prompt-only reaches 63.36 ROUGE-L but only 40.20 State Macro-F1 and 18.61% full state-vector agreement. LTI reaches 98.93 and 93.98% on the latter two measures. These metrics reveal lost or changed judgments that whole-text similarity can miss.
Recovered assessments also provide useful revision feedback. With the reviser fixed and no rubric supplied to it, LTI feedback gives similar full-rubric satisfaction and lower criterion regression than explicit feedback (Table 5). This supports both inspection and downstream use of recovered content.
3. Testing whether trajectories affect judgments
Recovering assessment content and showing that trajectory information affects decisions are separate questions. Recent work reports that predictions in some logical reasoning tasks did not depend on latent tokens [4].
We test information use by holding the original evaluation input fixed and replaying trajectories generated under the original rubric, a meaning-preserving paraphrase, or a single-criterion edit in a fresh original context (Section 4.2.3, Appendix E.3). Figure 4B shows that the edited-criterion trajectory shifts preference toward the designated response, while paraphrasing produces much smaller changes. Thus, criterion-related preference information passes through the trajectory and affects the judgment.
Together, LTI reconstruction and source interventions examine what information can be recovered and whether trajectory information is used in decisions.
4. Practical serving and overhead
We implement soft-embedding feedback directly in vLLM, retaining continuous batching, KV caching, and tensor parallelism (Section 4.3.1, Table 3). The equal-length control separates shorter-sequence gains from feedback overhead and shows no substantial additional serving cost (Appendix F).
This work provides a practical foundation for further research on latent reasoning in reward modeling: methods for preserving judgment quality, an LTI interpretation interface, rubric-level evaluation, information-use interventions, serving support, and revision experiments. Future work can build on this foundation to improve compression, training objectives, and inference. We hope the framework helps establish latent reasoning as a new approach to reward modeling, providing efficient evaluation and feedback for large-scale RL, data filtering, and revision in industry. We ask that the novelty assessment consider this complete framework and its experimental evidence.
References:
[1] Lambert et al. RewardBench.
[2] Shen et al. CODI.
[3] Shen et al. Efficient Reasoning with Hidden Thinking.
[4] Dilgren and Wiegreffe. Are Latent Reasoning Models Easily Interpretable?.