acceptodds
Under review as a conference paper at ICLR 2027

Judging in Latent Space: Efficient Generative Reward Modeling via Semantics-Preserving Compression

Abstract

Reward modeling often requires jointly representing and reasoning over multiple evaluation criteria, yet verbalizing this process token by token can incur substantial inference cost. Recent work on latent reasoning suggests that continuous states may support this computation more compactly. We introduce LatentGRM, a latent evaluation framework built on semantic chunking, compression, and reconstruction. By using the structure of rubric-guided evaluations to guide compression, LatentGRM learns compact continuous trajectories that support autonomous pairwise judgments without generating textual assessments. A separate interpreter reconstructs evaluation text from these trajectories, providing an offline view of the information retained under compression. Under matched training data and backbones, LatentGRM achieves competitive aggregate preference accuracy relative to explicit SFT judges at both 4B and 8B scales. Across four benchmark domains, LatentGRM-8B compresses evaluation trajectories by – and reduces total judge inference time by – at vote@5. Controlled rubric interventions show that criterion-dependent preference information is carried through the latent sequence. Together, these results demonstrate that continuous latent evaluation can substantially reduce inference cost while preserving competitive judgment quality.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Reward Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wFZXOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
1
Soundness
2(+1)
Read review
Paper summary

The paper proposes LatentGRM, which extends Latent-SFT with rubric-guided semantic chunking for pairwise reward modeling. It replaces textual evaluations with compact latent trajectories, achieving comparable judgment quality and 6–7× inference speedups. Additional experiments examine criterion-dependent information and reconstruction of teacher-derived evaluations.

Critical strengths

Practical efficiency: On four benchmark domains, LatentGRM-8B achieves 6.1–7.0× lower total inference time at vote@5 while maintaining comparable aggregate preference accuracy across eight domains.

Informative intervention analysis: Replaying trajectories under a fixed baseline rubric provides evidence that criterion-dependent preference information is transmitted through the latent sequence.

Critical weaknesses
  1. Lack of novelty (The new part is only the chunk compared with LatentSFT):

The latent representation of the vocabulary space, two-stage distillation, and autonomous latent generation are all derived from Latent-SFT. The primary methodological innovation is rubric-guided semantic chunking. While the "first application to reward modeling" represents a novel use case, it is insufficient on its own to establish significant methodological innovation.

  1. The most prominent acceleration results fail to isolate the specific contribution of the newly introduced method.

The 6–7x speedup is measured against a judge that generates lengthy textual analyses; however, latent-based methods using fixed chunking would similarly reduce the number of generation steps. This comparison demonstrates the practical value of latent evaluation but does not quantify the efficiency gains specifically attributable to the authors' semantic chunking technique. Direct evidence for semantic chunking shows an average improvement of 1.8 percentage points across four domains, alongside a regression in one domain.

  1. Writing and presentation should be largely improved

a. The abstract begins with a broad statement about reward modeling “jointly representing and reasoning over multiple evaluation criteria,” without first explaining the concrete setting: a generative judge compares two responses by producing textual assessments. This wording makes the motivation unnecessarily difficult to follow. A clearer opening would state that generating lengthy assessments makes such judges expensive, then explain how LatentGRM replaces those assessments with shorter latent sequences.

b. Table 2 presents results for only two methods across four domains, yet the layout is rather sparse, with the space occupied being disproportionate to the amount of information conveyed. Other tables could also be made more compact—by shortening headers, merging duplicate entries, and adjusting column spacing—thereby freeing up more space for the analysis of key experimental results.

c. Please redraw Figure 2 to make the training and inference workflows clearer. Its presentation-like styling and unclear data flow make the framework difficult to follow.

Questions for authors

Could the authors clarify the methodological differences between LatentGRM and Latent-SFT? My understanding is that the vocabulary-space latent representation, two-stage distillation, and autonomous latent generation are inherited from Latent-SFT, with rubric-guided semantic chunking being the primary methodological addition. Is this understanding correct?

6 comments

Thank you for the suggestions on writing and presentation. We have revised the abstract, introduction, and experiments to clarify the motivation and connections among the components.

Our contribution is a complete latent reward modeling framework connecting judgment quality, efficient serving, and interpretation. Frequent reward model calls make long assessments costly in data construction and RL. Reward modeling therefore provides a clear and important use case for compact latent assessments.

1. Method contribution and relation to Latent-SFT

We agree that the underlying latent representation, training mechanisms, and speedups from fewer generation steps build on Latent-SFT. However, its original fixed-ratio compression still falls short of explicit CoT models, especially on harder tasks such as Chat Hard and Precise IF. Chat Hard includes subtle and adversarial comparisons [1]. We therefore test whether the semantic structure of reward assessments helps preserve judgment quality under compression.

At the same target budget, semantic chunking improves Chat Hard, Precise IF, and Focus by 2.2, 3.9, and 1.7 points, while Chat decreases by 0.7. It thus provides a validated method improvement within the framework and a direction for further research.

2. LTI and rubric-level reconstruction evaluation

Reward modeling requires both preference judgments and access to their supporting reasons. LTI reads the evaluation input and saved vocabulary probabilities to recover textual assessments. It is trained separately and runs offline on demand. While CODI's token-level probe has limits in recovering multi-token semantic units [2], LTI targets complete criterion judgments and supporting reasons.

A key question is then how to evaluate reconstruction. Heima uses metrics including ROUGE-L [3], but changing "Partially Met" to "Met" can alter a judgment while leaving most wording unchanged. We therefore align assessments by response and criterion, measuring State Macro-F1, full state-vector agreement, and reason similarity.

Table 4 demonstrates why this matters. On teacher-derived trajectories, Prompt-only reaches 63.36 ROUGE-L but only 40.20 State Macro-F1 and 18.61% full state-vector agreement. LTI reaches 98.93 and 93.98% on the latter two measures. These metrics reveal lost or changed judgments that whole-text similarity can miss.

Recovered assessments also provide useful revision feedback. With the reviser fixed and no rubric supplied to it, LTI feedback gives similar full-rubric satisfaction and lower criterion regression than explicit feedback (Table 5). This supports both inspection and downstream use of recovered content.

3. Testing whether trajectories affect judgments

Recovering assessment content and showing that trajectory information affects decisions are separate questions. Recent work reports that predictions in some logical reasoning tasks did not depend on latent tokens [4].

We test information use by holding the original evaluation input fixed and replaying trajectories generated under the original rubric, a meaning-preserving paraphrase, or a single-criterion edit in a fresh original context (Section 4.2.3, Appendix E.3). Figure 4B shows that the edited-criterion trajectory shifts preference toward the designated response, while paraphrasing produces much smaller changes. Thus, criterion-related preference information passes through the trajectory and affects the judgment.

Together, LTI reconstruction and source interventions examine what information can be recovered and whether trajectory information is used in decisions.

4. Practical serving and overhead

We implement soft-embedding feedback directly in vLLM, retaining continuous batching, KV caching, and tensor parallelism (Section 4.3.1, Table 3). The equal-length control separates shorter-sequence gains from feedback overhead and shows no substantial additional serving cost (Appendix F).

This work provides a practical foundation for further research on latent reasoning in reward modeling: methods for preserving judgment quality, an LTI interpretation interface, rubric-level evaluation, information-use interventions, serving support, and revision experiments. Future work can build on this foundation to improve compression, training objectives, and inference. We hope the framework helps establish latent reasoning as a new approach to reward modeling, providing efficient evaluation and feedback for large-scale RL, data filtering, and revision in industry. We ask that the novelty assessment consider this complete framework and its experimental evidence.

References:
[1] Lambert et al. RewardBench.
[2] Shen et al. CODI.
[3] Shen et al. Efficient Reasoning with Hidden Thinking.
[4] Dilgren and Wiegreffe. Are Latent Reasoning Models Easily Interpretable?.

Thanks for the author's rebuttal. May I know where is the revised version of this paper? I would like to see the abstract and the figure.

Thank you for following up. We have updated the paper PDF on OpenReview. Please download it again from the submission page.

Reviewer_SAtHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LatentGRM, a latent reasoning framework designed to reduce the test-time inference cost of rubric-based reward models. Building on previous work on vocabulary-space distillation, the authors propose a two-stage framework to train the judge to generate compact trajectories. Across several benchmarks and evaluated with 4B and 8B models, LatentGRM matches or slightly exceeds the performance of normal generative RMs with faster inference time (6.9-7.8 times speedup).

Critical strengths
  1. The paper tackles a relevant, real-world problem on reducing the cost of reward modeling.
  2. The experiments are extensive, with some nice analyses.
Critical weaknesses
  1. My main concern is that the overall story of applying latent reasoning to reduce inference cost is not particularly surprising, as this method can technically be applied to any tasks. Since the core method largely builds upon previous work (Deng et al., 2025), I think the paper needs to better justify the technical challenges and/or highlight a surprising focus/angle for applying this to reward modeling. Moreover, the authors should justify using vocabulary-space distillation over alternative latent reasoning methods, such as Coconut or CODI.
  2. My other concerns are some of the experimental settings/results:
  • Because benchmarks such as RewardBench include additional categories like reasoning, it is unclear why those were not evaluated.
  • There are stronger baseline reward models available, such as top-ranking models on RewardBench, which the authors should consider comparing against.
  • It is unclear whether the results are significant. The authors should include variance across multiple runs or conduct hypothesis testing on the results.
Questions for authors
  • Besides Qwen, do you try this approach on other models?
  • How can this method adapt when designing a new rubric for a new task?
2 comments

Thank you for recognizing the practical value of reducing reward modeling costs. We have revised the abstract, introduction, and experiments to clarify our contribution.

W1. Contribution and choice of latent representation

We agree that latent reasoning compression applies across tasks, and our representation and two-stage training build on Latent-SFT. Our contribution is a complete, working latent reward modeling framework connecting judgment quality, efficiency, and interpretation. Reward models are called repeatedly in RL and data construction. When each judgment generates a long assessment and test-time scaling uses multiple samples, costs grow with both the number of calls and the sampling budget. Reward modeling therefore provides a clear and important use case for compact latent assessments.

This need raises specific technical questions: can compression preserve judgments across multiple criteria, can assessments remain available for inspection and revision, and can shorter trajectories improve practical serving?

Our framework addresses these questions through semantic chunking, autonomous judging, LTI readout, rubric-aligned evaluation, and vLLM serving. Crucially, we use reward modeling rubrics to evaluate latent reasoning. Changing "Not Met" to "Met" alters a key judgment with only a small wording change, which whole-text similarity metrics such as ROUGE-L and BERTScore may miss. Preserving this information is especially important in reward modeling. We also use rubric interventions to test whether latent trajectories carry criterion-related information, and examine their use in data revision.

Our choice of vocabulary-space states considers sampling, storage, and reconstruction. Unlike the standard hidden-state recurrence in Coconut and CODI, which has no sampling step with fixed inputs and weights [1,2], our vocabulary probabilities support Gumbel sampling and voting. The sparse distributions can be stored compactly for offline LTI input. An LTI-style interpreter trained for CODI showed poor reconstruction in our case studies.

W2. RewardBench coverage

The original setup followed OpenRubrics' general dialogue and instruction-following evaluation, using RewardBench Chat and Chat Hard [3]. We add Safety and Reasoning results with the same training data and Qwen3-8B backbone:

MethodVotesSafetyReasoning
OpenRubrics SFT166.6266.95
OpenRubrics SFT567.2369.15
LatentGRM165.1465.71
LatentGRM566.5570.17

W3. Stronger reward models

Our main question is whether compact assessments retain competitive judgment quality at lower cost under fixed data and backbones. The matched explicit SFT judge isolates this effect. External models use different data, objectives, and scoring interfaces, so they serve as reference points rather than the focus of our research question.

W4. Variation across runs

We added three inference seeds. Four-domain vote@5 results are reported as mean ± standard deviation:

DomainExplicit judgeLatentGRM
Chat90.74 ± 0.5690.32 ± 0.29
Chat Hard69.70 ± 0.8572.44 ± 0.46
Precise IF42.08 ± 2.4341.04 ± 1.57
Focus82.22 ± 0.5287.85 ± 0.31

These runs show stable inference performance.

Q1. Backbones beyond Qwen

Qwen is widely used as a backbone in reward modeling. We also trained an explicit SFT judge and LatentGRM on Llama 3.1 8B using the same OpenRubrics data. Their eight-domain averages are:

Methodvote@1vote@5
Explicit SFT62.8565.92
LatentGRM61.9165.82

The close vote@5 results support extending the training and autonomous judging process beyond Qwen.

Q2. Adapting to a new rubric

The rubric is a text input at inference time. The model receives the new task, responses, and rubric, then generates a latent assessment and preference. LTI can read the same input and saved trajectory when an explanation is needed. Changing the rubric itself requires neither an architecture change nor retraining for each set of rules.

Finally, we hope the reviewer reconsider our contribution to efficient reward modeling in light of this framework and the additional experiments. It connects assessment compression, autonomous judging, explanation inspection, practical serving, and feedback applications. Future work can build on it to improve compression, training objectives, and inference. We hope it makes latent reasoning a new reward modeling paradigm, supporting efficient evaluation and feedback for large-scale RL, data filtering, and revision in industry. We welcome suggestions on speedup attribution and presentation, and ask that novelty be assessed across the full framework and evidence.

References

[1] Hao et al. Coconut.

[2] Shen et al. CODI.

[3] Liu et al. OpenRubrics.

Thank you for the responses. They addressed some of my concerns. However, some concerns remain.

Regarding W1, I also agree with Reviewer wFZX that the novelty of the approach is somewhat limited and should be better justified. The current version of the paper does not sufficiently differentiate the proposed approach from Latent-SFT.

Regarding W2 and W3, thank you for the new results. However, the performance of the proposed approach does not seem significantly better than SFT. Moreover, I still think that evaluating with stronger reward models is helpful for practically measuring the effectiveness of the approach.

Reviewer_y7zeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LatentGRM, a generative reward model that compares two responses through compact continuous trajectories instead of explicit textual evaluations. Building on Latent-SFT, it uses rubric-guided semantic chunking and a two-stage distillation procedure to train an autonomous latent judge. A separately trained Latent Trace Interpreter (LTI) converts saved trajectories into textual evaluations for offline inspection. Experiments show competitive preference accuracy against matched explicit judges and substantial reductions in inference time, supported by chunking ablations and rubric-intervention analyses.

Critical strengths

The paper addresses a practical bottleneck in reasoning-based reward models and demonstrates substantial inference savings. Under matched training data and backbones, LatentGRM-8B achieves an average score of 69.0 versus 68.1 for Rubric-RM at vote@5, while reducing judge inference time by approximately 6–7× on four domains with vLLM. The semantic-chunking ablation and controlled rubric interventions provide useful evidence that the compressed trajectories retain evaluation-relevant information. The serving implementation also strengthens the practical value of the approach.

Critical weaknesses
  1. The strongest LTI reconstruction results and qualitative examples use teacher-derived trajectories, leaving interpretation fidelity during autonomous judging insufficiently established. The downstream revision experiment shows that restored feedback can be useful, but does not establish that it faithfully reflects the judge’s decision process. This distinction matters because LTI also receives the original evaluation input and could generate plausible explanations from that context.
  2. Appendix E.4 already compares accuracy with DirectJudge, but the main efficiency analysis focuses on the much longer explicit Rubric-RM evaluations. A matched-runtime comparison with DirectJudge is needed to establish whether the additional latent computation is worthwhile. LatentGRM improves some domains but underperforms DirectJudge on Chat, so its practical advantage depends on the accuracy–cost trade-off.
  3. The methodological advance over Latent-SFT appears incremental, centered on rubric-guided chunking and its application to reward modeling. LTI is a separately trained readout rather than a component that improves the autonomous judge through joint training or feedback. While the engineering gains are valuable, the current evidence does not yet sufficiently establish the broader contribution in faithful, interpretable latent evaluation.
Questions for authors
  1. Can you provide LTI examples and systematic fidelity evaluation on autonomous judge-generated trajectories, with controls that distinguish trajectory-derived information from explanations generated from the original input alone?
  2. Can you extend the existing DirectJudge comparison to report actual inference time and accuracy under matched runtime budgets, including single-rollout evaluation?
  3. Since LTI is trained separately and does not influence the judge’s decisions, what evidence connects its reconstructed criterion-level explanations to the autonomous judge’s actual preference, beyond feedback utility in response revision?
1 comment

Thank you for recognizing the practical efficiency problem and the value of our accuracy, interventions, and serving implementation. We revised the abstract, introduction, and experiments to clarify their connections.

Our core contribution is a complete, working latent reward modeling framework connecting judgment quality, inference efficiency, and interpretability. Repeated reward model calls in data construction and RL make long assessments costly. We connect assessment compression, autonomous latent judging, on-demand explanation recovery, rubric-level evaluation of recovered assessments, and efficient vLLM serving.

Q1 & Q3. What autonomous explanations recover and how they relate to judgments

Faithful interpretation remains a challenge for latent reasoning [1,2]. We assess recovered content through teacher-trace reconstruction, autonomous readout, and downstream revision.

1. Teacher-trace reconstruction provides a known reference. Appendix G.4 shows that LTI recovers criterion states and key reasons, including all ten states and the finding that the answer is incorrect and lacks reasoning in the calculation example. Table 4 provides quantitative evidence.

2. For autonomous trajectories, the original OpenRubrics CoT is a reference; the judge may follow a different evaluation process. On 607 held-out records, we compare freely generated explicit SFT assessments and autonomous LatentGRM readouts:

MethodState Macro-F1Vector ExactRubric SemanticROUGE-L
Explicit SFT48.7215.3355.5756.54
LatentGRM + LTI49.6716.9757.0358.66

Autonomous decoding can introduce variation; both methods differ from the reference but achieve similar scores.

For example, response B rewrites "It is raining heavily outside" as "It is going to rain heavily outside." For the criterion against introducing new meaning:

SourceAssessment of response B
DatasetNot Met. Introducing the implication of a future event is new information not present in the original sentence.
Explicit SFTNot Met. The implication of a future event is not present in the original sentence.
Autonomous LatentGRM + LTINot Met. The response introduces a new implication of a future event.

All three preserve the same semantic judgment despite different wording.

3. Recovered feedback also supports revision. LTI and explicit feedback achieve 57.50% and 56.67% full-rubric satisfaction, with 1.86% and 4.65% criterion regression (Section 4.4.2).

Further, to address whether LTI uses the latent trace to generate explanations, we add a trajectory-replacement test (Appendix G.5), following the intervention protocol in Section 4.2.3. The intervention-source effect is +0.152, versus -0.033 for the paraphrase control. With the visible input unchanged, LTI's assessments shift toward the response favored by the edited criterion. This supports its use of criterion-related information from autonomous trajectories.

The teacher-trace Prompt-only control addresses the same concern (Section 4.4.1): LTI achieves 98.93 versus 40.20 State Macro-F1 and 93.98% versus 18.61% full state-vector agreement, supporting information recovery beyond the input alone.

Teacher-trace reconstruction, autonomous comparisons and examples, and revision results jointly support our interpretation interface and criterion-level evaluation.

Q2. DirectJudge and inference cost

We reduce reasoning-based judging costs while retaining inspectable assessments. DirectJudge's short output is cheaper but provides no criterion judgments or reasons. GenRM's out-of-distribution results [3] and DeepSeek-GRM's RL-based assessment learning and sampling gains [4] support the value of an assessment process.

Appendix E.4 reports single-rollout and multi-vote accuracy. DirectJudge is strong on Chat, but its Chat Hard vote@5 score is 67.43, versus 70.50 for the explicit judge and 73.00 for LatentGRM. Practical use depends on difficult-example quality and explanation needs. DirectJudge is a reference; we focus on compact, inspectable generative assessments.

We hope the reviewer will consider the complete framework and its supporting evidence when assessing our contribution to efficient reward modeling through latent reasoning. We hope this framework helps establish latent reasoning as a new paradigm for reward modeling, providing efficient reward evaluation and feedback for large-scale reinforcement learning, data filtering, and revision in industry.

References:

[1] Shen et al. Efficient Reasoning with Hidden Thinking.

[2] Dilgren and Wiegreffe. Are Latent Reasoning Models Easily Interpretable?.

[3] Mahan et al. Generative Reward Models.

[4] Liu et al. Inference-Time Scaling for Generalist Reward Modeling.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.