acceptodds
Under review as a conference paper at ICLR 2027

Do Reasoning Models Enhance Embedding Models?

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) makes large language models better problem solvers, but does it also make their representations better foundations for text embeddings? We compare embedding models initialized from matched base backbones and backbones tuned with RLVR, using identical recipes for contrastive learning. Across MTEB Multilingual, MTEB Code, and BRIGHT, RLVR initialization provides no consistent advantage: the absolute paired mean differences reach at most 0.67 points. To explain this result, we introduce Hierarchical Representation Similarity Analysis (HRSA), which separates three properties often conflated by a single similarity measure: feature correspondence at the coordinate level, global and local manifold geometry, and functional readouts relevant to downstream tasks. HRSA shows that RLVR can reorganize local manifold geometry and, under prolonged training, shift the coordinate basis, while largely preserving global manifold geometry and linear readout structure. When the base and reasoning backbones are subsequently trained for embeddings, their shared contrastive objective rapidly pulls them toward the same global and functional organization, although differences in local manifold geometry persist. We call this phenomenon Manifold Realignment. In contrast, supervised fine tuning more strongly disrupts coordinate and geometric alignment. These findings suggest that RLVR primarily changes how a model traverses and locally organizes an existing semantic landscape rather than constructing a globally superior one. Consequently, stronger reasoning behavior does not necessarily produce better embedding models.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
representation learning, contrastive learning, reinforcement learning, latent manifold

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_Ms3fOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper investigates whether RLVR (primarily) improves embeddings in the same way that it improves reasoning, finding a null effect; the authors provide the explanation, based on further experiments, that the latent manifold is largely preserved, and that RLVR only changes local arrangements and movement on the manifold.

Critical strengths

I commend the authors for reporting a negative result. The null conclusion appears altogether robust.

Critical weaknesses

Another plausible and possibly more natural hypothesis is that RLVR affects only reasoning traces, and that only reasoning trace embeddings are affected. The training data for contrastive learning is normal text, and the evaluation of embeddings such as MTEB uses normal text. I did not see an experiment which refutes this hypothesis.

For example, when authors investigate differences between internal representations (RLVR or not RLVR), they almost exclusively use reasoning traces. Do the differences observed in internal representations (mostly in local arrangements) also occur with normal text? The appendix adds MMLU-Pro, where the local differences are already smaller, which the authors do not remark on. I do not understand why the authors did not test this hypothesis more strongly, this is why I have scored 1 on soundness, apologies if I am missing something.

Another question is whether a local overlap of say 0.5 should be judged as significant evidence of local re-organisation, rather than noise. Given the enormous amount of variation at play, it could be that sharing roughly half of your nearest neighbours is actually surprisingly high.

Questions for authors

Do the differences observed in internal representations (mostly in local arrangements) also occur with normal text?

What evidence can the authors give that the difference in overlap detected is significant?

1 comment

Thank you for the questions. We address your concern below.


Q1. Do differences appear outside reasoning traces?

Thank you for highlighting input-domain dependence. Smaller local differences on MMLU-Pro are compatible with Manifold Realignment, which does not require equal effects across input distributions.

The starting backbones already show incomplete neighborhood agreement on MMLU-Pro. For DS–ProRL, k-NN overlap at k = 10 is 0.5044, comparable to 0.4841 on CoT. After embedding adaptation, MMLU-Pro Linear CKA increases from 0.8477 to 0.9757, while k-NN overlap increases to 0.7482 (Figs. 8–11). Thus, the higher endpoint overlap accompanies partial realignment, rather than indicating identical starting representations. On CoT, CKA also increases, while local overlap remains similar or decreases.

Our functional analysis additionally includes AG News and ANLI, and downstream evaluation covers the MTEB Multilingual (which include 131 tasks), MTEB Code, and BRIGHT. These support the preservation of the tested readouts and the lack of a consistent aggregate embedding advantage.

We will explicitly discuss the domain-dependent effects and clarify the MMLU-Pro input construction. Stronger effects on reasoning traces can coexist with our structural findings, although the current evidence does not uniquely establish why the downstream advantage is small.


Q2. Is overlap around 0.5 meaningful or noise?

An overlap around 0.5 does indicate substantial preservation, consistent with our finding that RLVR preserves considerable structure while changing some local relationships.

Our interpretation rests on several main observations. First, the pattern is consistent across three seeds and k = 5, 10, and 50. At k = 10, DS–ProRL achieves 0.484 ± 0.065, compared with 0.068 ± 0.027 for the SFT pair, with the same ordering at the other neighborhood sizes (Table 3). Second, the comparison uses identical input tokens, and the metric is invariant to rotations and uniform scaling (Appendix B.2.2), so these neighborhood differences cannot be explained solely by a change of coordinate basis.

Third, local differences persist alongside global realignment. After embedding adaptation, DS–ProRL Linear CKA increases from 0.8946 to 0.9498, while k-NN overlap with k = 10 remains approximately unchanged at 0.474 ± 0.069 (Figs. 8–9; Table 3). Together with strong probe transfer and similar embedding performance, we show the central observation where global and functional agreement can coexist with nonidentical local neighborhoods. We interpret the overlap through this consistent multilevel pattern intended in the design of HRSA, rather than treating 0.5 alone as a significance threshold.


We hope these clarifications address the main concerns, and we would be grateful for a re-evaluation.

Reviewer_ZtayOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether reasoning-tuned language models, especially those trained with reinforcement learning from verifiable rewards (RLVR), provide better foundations for text embedding models than their untuned base models. The authors initialize embedding models from matched base/reasoning model pairs and train them with the same contrastive objective, then compare them on MTEB Multilingual, MTEB Code, and BRIGHT. Empirically, RLVR initialization yields little to no consistent advantage in downstream embedding quality, whereas SFT can substantially alter performance.

To explain this result, the paper introduces Hierarchical Representation Similarity Analysis (HRSA), a framework that separates model similarity into three levels: representation - level similarity (axis/feature alignment), geometry - level similarity (global and local manifold structure), and function-level similarity (shared downstream - relevant linear outputs). Using metrics such as per-dimension correlation, linear CKA, k-NN overlap, and cross - model linear probes, the paper argues that RLVR tends to preserve global geometry and functional structure while altering local neighborhood structure and sometimes the coordinate basis under long training.

Overall, the paper’s main contribution is a negative-yet-explanatory result: stronger reasoning ability does not automatically produce better embedding spaces, and the proposed HRSA framework offers a structured way to localize where post-training changes occur.

Critical strengths
  • The paper addresses a timely and non-obvious question with a clear negative result. The question of whether improved reasoning transfers to improved embedding quality is highly relevant given current model development trends. The finding that RLVR does not reliably help embedding models, despite improving reasoning behavior, is both surprising and practically useful. This substantially supports acceptance because it challenges a plausible community assumption.

  • The HRSA decomposition is a useful conceptual contribution. The distinction between coordinate-level similarity, geometric similarity, and functional similarity is meaningful and helps disentangle forms of representational change that are often conflated. Even if some specific metric choices are debatable, the framework itself is likely to be useful beyond this paper. This raises the contribution above a single empirical benchmark study.

Critical weaknesses

The empirical scope is somewhat narrow relative to the generality of the claims. The experiments focus on a small number of model families, one contrastive training recipe, and a limited set of embedding architectures/evaluation protocols. This matters because the paper sometimes reads as making a broad claim about reasoning-tuned models in general, whereas the evidence is strongest for the specific setup studied.

Questions for authors
  • How sensitive are the main conclusions to the embedding training recipe? In particular, do the parity results remain under alternative pooling strategies, smaller/larger temperatures, different hard-negative mining strategies, or bidirectional-vs-causal attention choices?

  • How much of the observed convergence is specific to RLVR-vs-base initialization, versus simply training two nearby checkpoints on the same contrastive objective? A control comparing two non-reasoning checkpoints from the same training trajectory, or two different late-stage checkpoints, could help calibrate this.

3 comments

Thank you for your questions. We address your concerns below.


Q1. How sensitive is the result to the embedding recipe?

Thank you for raising this question. We further evaluated three training variants: LoRA, domain-enriched data, and SIGReg with a smaller batch size. In these comparisons, the differences between base- and RLVR-initialized models remain small, suggesting that the absence of a consistent advantage is not limited to one exact training setting.

LoRA (Appendix A.1).

We trained both models with rank r = 16, alpha = 32, and dropout = 0.1. The RLVR-initialized model scores 0.39 points lower on MTEB Multilingual, so LoRA does not yield an advantage for this evaluated pair and benchmark.

Embedding BenchmarkDS-Distill (Base)ProRL (RLVR)Δ\Delta (Difference)
MTEB (Multilingual, v2)42.4542.06-0.39

Domain-enriched data.

We added CodeSearchNet* [1] and StackExchangeDuplicate* [2] to the original training data to provide more coverage of domains relevant to reasoning. The RLVR-initialized model improves by 0.13, 0.46, and 0.07 points on MTEB Multilingual, MTEB Code, and BRIGHT, respectively. The differences remain small even with the additional data that match RLVR data domain.

Embedding BenchmarkQwen2.5-1.5B (Base)Qwen2.5-1.5B-SRL-Zoo (RLVR)Δ\Delta (Difference)
MTEB (Multilingual, v2)53.8253.95+0.13
MTEB (Code, v1)63.8164.27+0.46
BRIGHT18.3018.37+0.07

*All the additional data are preprocessed with the exact same procedures, stated in Appendix A.2 (Page 15).

SIGReg regularization with batch size 64.

We added LeJEPA’s SIGReg [3] into the objective as a regularization, intended to discourage representation collapse, and reduced the batch size from 2,048 to 64. The RLVR-initialized model scores 0.17–0.47 points higher across the three benchmarks.

Embedding BenchmarkQwen2.5-1.5B-SIGReg-64-Bsz (Base)Qwen2.5-1.5B-SRL-Zoo-SIGReg-64-Bsz (RLVR)Δ\Delta (Difference)
MTEB (Multilingual, v2)36.9737.44+0.47
MTEB (Code, v1)52.4552.62+0.17
BRIGHT15.1315.47+0.34

Together, these checks support a conclusion: under the evaluated variants, RLVR initialization does not show a large or consistent embedding advantage. They do not establish statistical equivalence or rule out gains under other recipes, which remain future works. We will add the variant results to Appendix A and clarify this scope.


Q2. Is convergence specific to RLVR?

Thank you for this suggestion. Our primary question is whether RLVR initialization improves embedding models, which we address by training matched base and RLVR backbones with the same contrastive recipe and comparing their downstream performance. A comparison between two non-RLVR checkpoints would address a different, complementary question: whether representational convergence is specific to the base–RLVR relationship.

Our current experiments do not fully distinguish an RLVR-specific effect from the effect of applying a shared objective to related checkpoints by design. Moreover, manifold realignment describes the fact that dissimilar manifolds will still be pooled toward a similar manifold, as evidenced by the Orthogonal Procrustes (0.161→0.864) and Dimension-Wise Correlation (0.667→0.921) in Figure 5.

A comparison with nearby non-RLVR checkpoints would help calibrate this interpretation and is an interesting direction for future work. This limitation does not change the separately observed small embedding-performance differences under the evaluated training recipe.


We hope our responses have thoroughly addressed your primary concerns, and we respectfully request that you consider re-evaluating our paper.


References:

  1. CodeSearchNet: https://huggingface.co/datasets/sentence-transformers/codesearchnet
  2. StackExchangeDuplicate: https://huggingface.co/datasets/sentence-transformers/stackexchange-duplicates
  3. Balestriero, Randall, and Yann LeCun. "Lejepa: Provable and scalable self-supervised learning without the heuristics." arXiv preprint arXiv:2511.08544 (2025).

Thank you for your explanation. However, it does not change my evaluation, and I will retain my score.

Thank you for taking the time to consider our response. We respect your decision and appreciate your feedback, which helped us clarify the scope of our findings.

Reviewer_SKmWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether LLM backbones post-trained with RL from verifiable rewards (RLVR) are better starting points for text-embedding models. Matched base and RLVR backbones are each converted into embedding models with an identical full-parameter InfoNCE recipe: bidirectional attention, mean pooling, 1.6M pairs, 782 steps (App. A). On MTEB Multilingual, MTEB Code and BRIGHT, the RLVR-initialized models differ from the base-initialized ones by at most 0.67 points across 4 pairs (Table 1).

To explain this, the authors propose HRSA, a three-level grouping of similarity measures:

  1. Representation level: dimension-wise correlation and inverse row entropy of the Procrustes map.
  2. Geometry level: linear CKA and k-NN overlap.
  3. Function level: cross-model linear probes.

They apply HRSA mainly to one SFT pair (Qwen2.5-Math-1.5B → DS-Distill) and one RLVR pair (DS-Distill → ProRL), with more pairs in App. D. They conclude that RLVR preserves global geometry and linear readouts but reorganizes local neighborhoods. They further conclude that contrastive training then pulls the two embedding models back together, which they name “Manifold Realignment” (Sec. 5, Fig. 5).

Critical strengths
  1. Timely, well-posed question with a clear, falsifiable answer; the null result on embedding quality is itself informative for practitioners choosing backbones.
  2. Careful empirical hygiene for the main result: matched pairs, identical recipes, three seeds, standard deviations reported, three benchmark families including reasoning-intensive retrieval (BRIGHT) (Table 1).
  3. Full-parameter and LoRA variants both show the same parity (Table 5), which strengthens the main finding.
  4. The dissociation between global CKA (high) and local k-NN overlap (≈0.5) under RLVR, persisting through contrastive training (Table 3, Fig. 5), is an interesting observation worth following up.
Critical weaknesses
  1. Claim “SFT ... destroys the pretrained semantic space" is contradicted by Table 1.

Five of six SFT cells improve (+0.65 to +2.07); only MTEB-Multilingual for Qwen3-0.6B drops. SFT changes embedding quality more than RLVR but mostly upward, which undercuts the premise that geometry preservation is what one wants.

  1. The SFT-vs-RLVR contrast is confounded with training volume and starting checkpoint, so the paper’s causal claim (“training objectives dictate latent manifold,” Sec. 6) is not supported.

The SFT pairs involve large-scale distillation (DS-Distill-1.5B: ~800K long-CoT samples; Qwen3-0.6B: a full post-training pipeline), while most RLVR pairs are short zero-RL runs. The paper’s own prolonged-RLVR pair (ProRL) reaches Hinv = 0.161 on CoT (Table 11), close to SFT’s 0.108, which is consistent with “more training → more drift” rather than “SFT vs. RL.”

Questions for authors
  1. What is the k-NN overlap (k=5/10/50) and linear CKA between two embedders trained from the same backbone with different seeds, and between unrelated models? Is the RLVR pair’s 0.45–0.58 distinguishable from this floor?
  2. Can you provide a token-budget-matched SFT control (e.g., SFT on the RL model’s own correct rollouts) so that objective is not confounded with data volume? Alternatively, HRSA across ProRL intermediate checkpoints — does Hinv decline monotonically with steps?
  3. How do you reconcile “SFT destroys the semantic space” with SFT improving 5 of 6 Table 1 cells?
3 comments

Thank you for your questions. We address your concerns below.


W1 / Q3. Does SFT destroy the semantic space?

Our intended point here is the persistence of representational differences through subsequent embedding model training. This is distinct from whether the resulting embeddings achieve better benchmark performance, where a representation can become more useful while remaining different from its base models.

In the CoT data evaluation, the Qwen2.5-Math→DS-Distill (SFT-pairs) pair’s inverse row entropy increases only from 0.1076 to 0.1429, while its CKA increases from 0.1483 to 0.1795. In contrast, DS-Distill→ProRL’s (RLVR-pairs) inverse row entropy increases from 0.1613 to 0.8637, with Linear CKA increasing from 0.8946 to 0.9498 (Tables 11–12; Figures 8–9). The main SFT comparison therefore retains substantial misalignment under the evaluated embedding model training, whereas the prolonged-RLVR comparison exhibits strong manifold realignment.

These findings are compatible with the SFT-related benchmark improvements in Table 1. We will replace “destroys the pretrained semantic space” with a precise description of the persistent representational differences, and revise Section 2 so that benchmark degradation isn’t used as evidence of structural disruption.


W2 / Q2. Are the differences caused by the training objective?

The prolonged-RL result illustrates our central distinction: coordinate mixing can coexist with high global similarity and transferable readouts. The cited Hinv=0.1613 characterizes mixing in the fitted Procrustes map. It does not measure geometric distortion. DS–ProRL retains CKA of 0.8946 versus 0.1483 for the SFT reference on CoT, and 0.8477 versus 0.2671 on MMLU-Pro (Fig. 8), alongside smaller probe-transfer gaps. Greater coordinate drift with longer training is therefore consistent with our claim, and highlights the value of examining multiple HRSA levels as intended.

Besides, this pattern extends across multiple algorithms, model sizes, starting checkpoints, and training settings, strengthening the empirical finding beyond a comparison of short RL runs with large-scale SFT.

Our endpoint comparisons do not establish monotonicity during RLVR. Even a monotonic decline in Hinv would remain consistent with our finding: coordinate mixing can coexist with high global similarity and transferable readouts. Evaluating its geometric and functional implications would require tracking Linear CKA and probe transfer alongside Hinv. Tracking Hinv alone would characterize coordinate drift, while jointly tracking HRSA levels would test how that drift relates to geometry and function.

We will refine Section 6 to distinguish this empirical contrast between post-training pipelines from attributing it solely to the objective, independently of data, budget, and checkpoint history.


Q1. How should local overlap be calibrated?

Thank you for suggesting these controls. For the evaluated runs, the base–RLVR embedding pair has substantially less local overlap than either same-backbone, different-seed pair. The table below compares k-NN overlap at k = 5, 10, and 50 and linear CKA on CoT inputs (a comparison of local v.s. global geometry).

Comparisonk-NN k = 5 / 10 / 50Linear CKA
Chance, at least 1,000 tokens< 0.01 / ~ 0.01 / ~ 0.03—
Different-size Qwen2.5-Emb: 0.5B vs 1.5B0.243 / 0.244 / 0.2290.627
SFT: Qwen2.5-Math-Emb vs DS-Emb0.069 / 0.068 / 0.0910.180
RLVR: DS-Emb vs ProRL-Emb0.451 / 0.474 / 0.5310.950
Same backbone, different seeds: DS-Emb0.922 / 0.940 / 0.9540.963
Same backbone, different seeds: ProRL-Emb0.914 / 0.915 / 0.9340.921

The same-backbone comparisons got k-NN overlaps of 0.922–0.954 for DS-Emb and 0.914–0.934 for ProRL-Emb, compared with 0.451–0.531 for DS-Emb versus ProRL-Emb. This large gap suggests that, in these runs, the local disagreement between the base- and RLVR-initialized embedders is not explained by the measured difference between training seeds alone. The base–RLVR pair retains substantial neighborhood agreement and is more similar than the different-size Qwen2.5 reference (0.229–0.244). At the global geometry level, the base–RLVR pair has CKA 0.950, compared with 0.963 and 0.921 for the same-backbone controls. Thus, the local-overlap gap can coexist with high global similarity.

The distinction between the local and global geometry is why we proposed HRSA, where a single metric provides only a partial view of representational similarity. HRSA separates coordinate-level alignment, global and local geometry, and compatibility with the tested linear readouts. We interpret each metric relative to its own controls and examine the resulting pattern across levels, rather than directly comparing numerical values on different scales.


We hope this response clarifies the points you raised and would be grateful for a reassessment of our work.

Thank you for the response and the additional control.

Q1: resolved. The seed control shows that the local-overlap gap is not noise, and the CKA of the base and RLVR pair falling within the seed range supports the global-realignment claim. I would still avoid "irreversible", since the training-dynamics analysis covers 782 steps of one recipe.

W1: resolved. Withdrawing "destroys the pretrained semantic space" and separating representational difference from benchmark quality is the appropriate fix.

W2: largely resolved. I accept the clarification on H_inv, and no longer attributing the contrast to the objective alone is the right correction. The SFT side still rests on one or two pairs, so it should be presented as an observation about these specific pipelines.

I consider the response satisfactory. My remaining reservation is about significance, not about the response. With the claims stated accurately, the contribution is a credible null result together with a descriptive similarity analysis built from existing measures. The explanatory part amounts to showing that RLVR backbones remain close to their base models, which is in line with prior analyses the paper cites. The one distinctive observation, persistent local differences, has no measured effect on embedding quality. I find the null result useful for practitioners, but in my assessment this is not yet sufficient for acceptance, so I keep my current rating.

We appreciate your thoughtful reply and respect your decision. Our work brings together broad, matched-backbone evaluations and HRSA’s analyses of coordinates, geometry, and readouts, which prior representation studies have examined in an ad hoc, fragmented way. Using a shared training recipe, we find that stronger reasoning LLMs do not consistently produce better embeddings. The seed-controlled results show that local geometric differences persist even as representations realign globally and performance remains comparable. This distinction between representational change and downstream utility offers practical guidance for choosing backbones and points to an open question: how can reasoning capabilities be transferred into embeddings?

Reviewer_88XDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this work, the authors question whether improving an LLM's reasoning ability also improves its semantic representations. They compared base models with LLMs improved through RLVR and observed nearly identical downstream embedding performance. The authors propose that what changes is how the model navigates semantic knowledge. Specifically, they propose a concept they call “Manifold Realignment”, which describes how subsequent contrastive learning brings the representations of base and RLVR trained models toward a shared organization. They suggest that RLVR does not construct a new manifold to improve reasoning, but changes the way this manifold is traversed and locally organized, while leaving its global geometry and functional structure relatively intact. The authors propose HRSA to investigate what specifically changes when using RLVR, and perform three critical analyses. A representation level analysis (to explore axis coordinates), a geometry level analysis (to explore global and local geometry), and a function level analysis (input-output mapping). Their results appear to be consistent with their hypothesis, with RLVR models largely maintaining their axis-aligned feature correspondence and global geometry, showing increased feature alignment after contrastive learning, while showing partial reorganization of local geometry.

Critical strengths

This study is theoretically well-motivated, the paper's flow is easy to follow, and the methodology is sound. I found the idea of separating axis coordinates from latent semantic geometry very valuable, and the HRSA framework the authors implement allows us to understand differences in semantic representations at different levels, allowing for a deeper understanding of the representational changes associated with improved reasoning in models trained with RLVR.

Critical weaknesses

I think that incorporating a methodology to assess how models navigate through this manifold would be a very valuable addition to further strengthen the authors' position that RLVR may change the way models traverse their semantic geometry.

Questions for authors
  1. The authors propose that RLVR modifies how models traverse an existing semantic manifold. Does this imply that reasoning involves sequential transitions through local semantic neighborhoods, potentially integrating information across the global manifold? How does the proposed framework distinguish these processes?

  2. Although the authors propose that RLVR primarily modifies the traversal policy through an existing semantic manifold, the analyses focus on representational similarity rather than navigation dynamics. Despite analyzing token-level representations from reasoning traces and geometric characteristics across layers, how can we assess the model's navigation through this manifold? Are there methodologies that allow us to explore how models traverse it that could provide a more direct test of the proposed traversal mechanism underlying the Manifold Realignment hypothesis?

  3. Do the authors think that training models in terms of navigation through their semantic manifold could help further improve their reasoning processes? If so, could the authors elaborate on how this could be implemented in reasoning training protocols?

2 comments

Thank you for all of the interesting questions. We address them below.


Q1. What do we mean by traversal?

Our proposed framework (HRSA) distinguishes changes in representation structure from the hypothesized changes in how a model navigates that structure during reasoning. For the DS-Distill→ProRL pair, the base and reasoning models retain similar global structure, as evidenced in their dimension-wise correlation, high linear CKA, and similar performance on the linear readouts. Yet their local neighborhoods overlap only partially (k-nn overlap = 0.45–0.57) for the LLMs across k = 5, 10, and 50. (Table 3).

These findings motivate the hypothesis that RLVR can improve reasoning while largely preserving global representation structure, potentially modifying local relationships in ways that affect navigation during reasoning. HRSA distinguishes global, local, and functional changes, but it does not directly measure navigation dynamics or establish that local changes cause them. We will make this distinction and the hypothetical status of the navigation interpretation explicit in the paper, and the navigation dynamics remain as future work.


Q2. How could navigation be tested directly?

Our work mainly studies representation similarity, where, on the sampled inputs, the evaluated base and RLVR backbones have similar global representation structure, even though their local neighborhoods need not match. HRSA can serve as a diagnostic tool to expose the interesting differences between manifolds for interpretability.

Directly testing navigation dynamics during generation remains future work. I propose 3 approaches in the following:

  1. Shared-prefix decisions: compare next-token distributions, for example by KL divergence, on identical reasoning prefixes; then assess continuation success under matched budgets. Wang et al. discuss RLVR changes at high-entropy “forking” tokens, a useful starting point (Wang et al., 2025).
  2. Self-generated trajectories: sample solutions to the same problems under matched decoding budgets, and compare independently defined reasoning-step categories and verified progress, controlling for length and difficulty.
  3. Interventions: test candidate states or components with activation patching and appropriate sham controls, while accounting for distribution shift and cross-model coordinate compatibility (Zhang and Nanda, 2024).

These tests would examine decisions and trajectories during reasoning, complementing the representation comparisons and training dynamics reported in our current work.


Q3. Could navigation-aware training help?

This is a very interesting question, and thanks for asking it. Possibly, but this remains untested. A future experiment could retain final-answer rewards, add feedback for verified intermediate steps via process reward models, and separately penalize CKA decreases relative to the base model on a fixed anchor set (as mentioned in the discussion). The latter would constrain global geometry while allowing local adaptation (so implicitly limiting the navigation dynamics). We would compare the components and their combination at matched compute, rewarding verified progress rather than merely smooth paths. I think distillation is also a form of navigation-aware training, but we teach the student model to navigate a different semantic manifold defined by the teacher.


Thank you again for the highly constructive feedback. We hope these updates resolve your reservations and might encourage a more favorable assessment.

I appreciate the author's thoughtful response and consideration regarding my questions. I will maintain my score, but I am more inclined to accept the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.