acceptodds
Under review as a conference paper at ICLR 2027

Computation Over Geometry: Meaning Identity Is Computed, Not Shipped in the Embeddings

Abstract

Meaning identity—whether two sentences say the same thing after the wording has changed—is treated throughout retrieval and RAG as a geometric fact about independently encoded sentence vectors. We test a different picture: identity is computed when both sentences occupy the same forward pass, not shipped as a cheaply readable property of either sentence’s embedding. On a common 2,000-pair English PAWS test set, linear readouts of four frozen joint passes reach AUC 0.900–0.946 with 1,200 labels, versus 0.622–0.646 for full-width nonlinear readers of independent states. With all 49,258 training pairs, those readers improve to 0.736–0.786, and cross-attention over frozen token sequences reaches 0.868 on a 14B backbone, while the joint readouts remain 0.922–0.960. Across a separate overlap-matched survey of 30 decoder checkpoints, joint exceeds an independently selected product/difference probe in 29 rows and partner shuffle collapses to chance; a pair-disjoint replication reverses the sole GPT-2-large ordering, although its intervals overlap. The same joint-over-independent pattern recurs in bidirectional encoders, encoder–decoders, cross-language and nonce controls, and controlled role, negation, and quantity relations. It is not the generic consequence of scoring jointly: BGE rerankers enter the identity band, while MS-MARCO and Jina rerankers do not. Nor is it an immutable architectural limit: fine-tuning BGE or applying rank-16 LoRA to E5-Mistral can install a strong independent criterion by changing the encoder. The sharper result is route-dependent teachability. A 1.5B joint student and a trainable independent student receive the same 9,000 teacher-scored pairs and update budget, yet reach English PAWS AUC 0.942 and 0.760; six independently trained models also produce strongly concordant, symmetric, and graded joint scores. Across the tested models, readers, and supervision budgets, sentence geometry remains useful for aboutness, but propositional identity behaves as a recurring relation made cheap to read by computation over the pair.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
sentence embeddings, meaning identity, paraphrase, late fusion, probing, retrieval, language models, large language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RxUROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether sentence-level meaning identity is accessible from independently encoded representations or is computed when two sentences are jointly processed. The authors compare cosine similarity, probes over independent representations, late fusion, and linear probes over joint hidden states. Across PAWS-X, QQP, multiple model families, architectures, languages, and controlled relation tasks, joint representations consistently expose a much stronger identity signal. Additional experiments with stronger independent readers, encoder adaptation, and matched-supervision distillation suggest that the main difference is not information-theoretic absence, but substantially easier accessibility through joint computation.

Critical strengths
  1. The empirical evidence for the joint-over-independent gap is unusually comprehensive. The paper covers many decoder checkpoints, bidirectional encoders, encoder-decoder models, embedding models, rerankers, multiple languages, and several controlled relation tasks. The same qualitative pattern appears across most of these settings.
  2. The paper identifies a useful distinction between semantic retrieval and semantic verification. The results suggest that embeddings can remain effective for retrieving texts about the same topic while being unreliable as a certificate that two propositions are actually equivalent.
  3. The paper is appropriately nuanced about encoder adaptation. Fine-tuning or LoRA can substantially improve independent encoders, which helps clarify that the claim is about accessibility in shipped representations rather than an absolute architectural impossibility.
Critical weaknesses
  1. The joint and independent routes are not fully compute-matched. The joint route receives pairwise interaction throughout the pretrained backbone, while the independent route must learn this interaction afterward. This makes it difficult to separate representational accessibility from pretrained pairwise computation.
  2. The strongest independent reader already closes a substantial part of the gap. Frozen-token cross-attention reaches 0.868 AUC on the 14B model, suggesting that much of the relevant information is still retained in independently encoded token states.
  3. Generalization beyond PAWS-style identity is still somewhat limited. QQP transfer is useful, but the direct distillation advantage on QQP and MRPC is small and statistically inconclusive, while the main controlled experiments remain relatively synthetic.
Questions for authors
  1. How do you distinguish meaning identity from a more general semantic similarity? For example, can the same frozen direction separate paraphrases from entailments, contradictions, and topically similar but propositionally different pairs under controlled lexical overlap?
  2. How much of the joint advantage remains under a more compute- or capacity-matched independent interaction model?
  3. How should the partial QQP/MRPC transfer be interpreted? In particular, does the weak distillation gap on these datasets suggest that the learned relation is partly PAWS-specific?
3 comments
AuthorsOpenReview ↗

Response to Reviewer RxUR — scope boundaries and the identity-vs-similarity question

Thank you for the constructive review and for highlighting the retrieval-versus-verification distinction. The three weaknesses are boundaries we accept, and each is drawn in the submitted text. We answer them and the three questions below.

W1 and Q2, compute matching

We agree the routes are not compute-matched and that the joint route benefits from pairwise interaction performed during pretraining; the distillation paragraph states "this comparison does not equate route FLOPs or prove that no independent architecture could do so," and Appendix A generalizes the caveat. The claim is therefore deliberately about shipped artifacts: given the representations systems actually deploy, how much supervision and machinery does each route need before the relation is readable? The independent interaction model we tested is the two-block bidirectional cross-attention reader over frozen token sequences, a trained module with orders of magnitude more parameters than the probes. Under matched labels it reaches 0.545/0.555/0.786 at 1.2k/8k/49k (1.5B) and 0.868 (14B, full split), versus a $\le$128-dimensional linear probe on the joint pass at 0.90+ from at most 1,200 labels on the same backbones. What the data rule out are the cheaper explanations: misaligned geometry (late fusion stays at 0.48–0.59) and linear insufficiency (the full-width MLP gap persists at every budget). A fully FLOP-matched interaction model trained from scratch is beyond this submission, and we claim nothing about it.

W2, the 0.868

Yes: the paper reports it as "the strongest independent frozen-backbone result we observe" and cites it as evidence that useful material is retained in independent token states (Section 4). That is half of the stated claim; the other half is that access remains unequal. On the survey side where the 0.868 is scored, the same 14B backbone's joint linear probe reaches 0.950 [0.925, 0.971] (Table 1), and the 0.868 required the full 49k training pairs plus the trained cross-attention module (the same reader stays at 0.486–0.561 from 1.2k to 20k pairs; Table 11), where the joint probe is fit on 1,000 labels. On the matched common bank, Figure 2 separately shows joint 0.922–0.960 versus independent full-width MLP 0.736–0.786 at the full budget, with all 12 matched model-by-budget differences having paired intervals above zero. The Conclusion's phrase "not information-theoretic absence" names exactly this pair of facts, and Appendix A elaborates it.

W3 and Q3, PAWS-specificity

Partially PAWS-specific, and the paper quantifies how much. The frozen PAWS direction transfers zero-shot at 0.72–0.75 to overlap-matched QQP and at 0.745–0.818 to Chinese PAWS (near the in-domain 0.766–0.851); in reverse, a QQP-trained probe reads PAWS at up to 0.848 without seeing a PAWS pair (Section 5, "The computed direction transfers; the fitted geometry does not"). So a sizable transferable component coexists with a dataset-specific part ("part of any probe is dataset-specific"). The distillation differences on QQP (+0.0181) and MRPC (+0.0431) cross zero; we draw no conclusion from them, as you recommend. The controlled tasks add held-out-relation transfer within a restricted grammar (all 12 backbone-by-relation averages above chance), and the paper states the limit: "transferable structure in this restricted grammar, not universal expression-invariant correctness."

Q1, identity vs. general similarity

Two existing results separate them; the second is not overlap-controlled.

  • Topical similarity: the anchor-balanced QQP bank removes every single-sentence cue by construction: each anchor appears once with a duplicate and once with an overlap-matched non-duplicate, so lexical overlap alone scores 0.52. There, joint AUC is 0.80–0.83 across seven models while late fusion is 0.55–0.60 and separate cosine 0.58–0.62 (Table 8). Aboutness is visible to independent vectors; identity is not.
  • Entailment and contradiction: applied zero-shot to SNLI, all six frozen PAWS directions order paraphrase > entailment > neutral > contradiction > non-paraphrase, monotonically in the mean, with entailment-vs-contradiction AUC 0.79–0.98 (Section 7; Appendix B). The direction is graded in the semantically correct order rather than acting as a binary same-topic detector.
  • Scope on NLI: SNLI is not overlap-matched, so we do not claim overlap-controlled separation of entailment from contradiction; the paper's claim stays scoped to overlap-matched identity, with the two results above as the current evidence that the frozen direction tracks more than topical similarity.

We hope this confirms that the boundaries you identified are the ones the paper draws. We are glad to answer follow-ups, and would be grateful if you would take these answers into account in your final assessment.

AuthorsOpenReview ↗

Gentle Reminder: Rebuttal for Paper 42490

Dear Reviewer RxUR , hope you are doing well.We are writing to gently remind you that we have submitted our rebuttal for this paper. We understand you have a heavy reviewing workload. We would greatly appreciate it if you could take a moment to go through our response when you have time.Thank you very much for your time and efforts.

Area_Chair_uXnJOpenReview ↗

Please respond

Hello,

The discussion period ends soon. Please respond to the authors' last comments and note whether they changed your evaluation.

Thanks,

Your AC

Reviewer_LiUfOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the accessibility of propositional equivalence between sentences under independent and joint encoding. Within the same frozen backbone, the authors compare cosine similarity, linear readouts, full-width MLPs, token-level cross-attention readers, and linear readouts of jointly encoded pairs. Controls include partner shuffling, random labels, nonce substitution, learning curves on shared data, and encoder adaptation. The paper also compares students trained with common teacher supervision and evaluates controlled relation tasks. Across the tested models, readers, and supervision budgets, joint computation yields a more accessible relation signal, while task adaptation can substantially narrow the gap for independent encoders.

Critical strengths
  1. The central comparison has substantial diagnostic value. On the common test set, full-width independent MLPs across four backbones reach 0.736–0.786 AUC with roughly 49K training pairs, compared with 0.922–0.960 for joint linear readouts. Shared training indices, independent layer selection for each route, and shuffle controls reduce important confounds. This is more informative about the role of the computation path than a direct comparison of off-the-shelf bi-encoders and cross-encoders.
  2. Strong alternatives and counterexamples make the conclusions more credible. The evaluation goes beyond cosine similarity to nonlinear and token-level readers, and reports the GPT-2-large exception. E5-Mistral reaches 0.946 after LoRA adaptation with 8K examples, demonstrating that independent encoding can learn the task. These results place the contribution in accessibility differences under limited budgets, rather than an impossibility claim.
Critical weaknesses
  1. Optimization stability of the independent readers still limits interpretation of the gap. On the common data, the 1.5B token reader reaches only 0.545 and 0.555 with 1.2K and 8K examples, rising to 0.786 at roughly 49K; full-budget seeds range from 0.748 to 0.860. The authors already search over layers, pooling, and learning rates, but it remains difficult to distinguish insufficient data from underfitting at low budgets. A finite probe's performance is also not an upper bound on the information in a representation [1]. The paper acknowledges this limitation. Training and validation curves would help establish how much optimization affects the observed gap. This concern mainly limits the interpretation of its magnitude, rather than invalidating the central experiment. [1] Information-Theoretic Probing for Linguistic Structure, Annual Meeting of the Association for Computational Linguistics (ACL), 2020.
  2. The distillation result on teachability depends on a particular teacher target. Teacher scores come from directions in joint representations identified using PAWS labels, and student checkpoints are selected by validation MSE against those scores. This fairly compares the difficulty of reproducing one scoring function, but does not fully substitute for comparing learning from gold labels. DSE has shown that scores from a joint teacher can transfer to independent sentence embeddings [2]; here, the dependence of the gap on the target and checkpoint-selection rule needs to be clearer. The gold student on controlled grammar tasks is a useful addition, but does not replace a matched-student, matched-budget control on natural PAWS data. The distillation differences on QQP and MRPC also have intervals crossing zero, so broader interpretations of teachability should remain limited. [2] Scalable Attentive Sentence Pair Modeling via Distilled Sentence Embedding, AAAI Conference on Artificial Intelligence (AAAI), 2020.
Questions for authors
  1. Optimization diagnostics: How well do the MLP and token reader fit the training data at each budget, and what do their validation curves show? Is the full-budget seed variation mainly due to initialization, hyperparameters, or stopping rules? Can existing logs explain the near-chance results at low budgets?
  2. Supervision target: On natural PAWS data, how does the route gap change when the same students and update budget use gold labels, teacher scores, or a mixed objective? Does teacher-MSE checkpoint selection still have a substantial effect when training endpoints are held fixed?
  3. Protocol alignment: Could you summarize the datasets and layer-selection rules in one place? The 14B token reader's 0.868 comes from a different data collection, and Appendix Table 9 uses the joint peak layer. Which experiments are directly comparable to these auxiliary results?
  4. Budget interpretation: All learning curves also use 2,000 validation labels, and the routes are not compute-matched. Could the claims about ease of readout and supervision efficiency be explicitly tied to this protocol, distinguishing training labels, total labels, and online computation?
4 comments
AuthorsOpenReview ↗

Response to Reviewer LiUf (1/2) — optimization diagnostics and supervision target

Thank you for a review that states the intended contribution and its boundary precisely: accessibility differences under tested readers and budgets, not an impossibility claim. We answer the four questions below with what is archived. Part (1/2) answers Q1–Q2; part (2/2) follows with Q3–Q4.

Q1, optimization diagnostics

Both independent readers select learning rate and stopping on development, over a fixed small grid with early stopping, three seeds each (full settings in the supplementary code). What is archived:

(a) For the controlled-task matrix, per-seed training AUC (supplementary paper/revision_assets/fit.tex): joint MLPs fit 1.000 throughout; the independent MLPs of Qwen base and the gold student clear the 0.98 fit flag on all seeds (0.996/0.985/0.984 and 0.998/0.997/0.993), the consensus student on two of three (0.998/0.964/1.000), while BGE-large underfits (0.854/0.712/0.847) and the paper bounds that row accordingly ("bounds accessibility under the tested optimization," Section 6).

(b) For the common bank, the archive (paper/data/evidence.json) holds each run's selected configuration and every candidate's development AUC, not per-epoch trajectories. At the full budget all three token-reader seeds select the same configuration (layer 21, lr 1e-4), with development AUC 0.852/0.758/0.755 and test 0.860/0.749/0.748, so the seed spread reflects initialization, batch order, and early stopping rather than hyperparameter changes. At 1,200 and 8,000 pairs all four of its grid candidates stay at development AUC 0.518–0.589, so the near-chance results are not a selection artifact. Both readers also pass a 32-pair overfit check (training AUC 1.0 in 25 steps).

On the low budgets: the two-block reader must learn its interaction weights from scratch, while the joint probe only fits a direction in states where the interaction has already happened; we agree the data cannot fully separate insufficient data from underfitting there. We also agree with the probing caveat of [1], and the paper already scopes it (Appendix A): reader comparisons "establish accessibility under specified architectures, searches, and supervision budgets, not information-theoretic absence or an optimal independent ceiling." The direction of the central result does not rest on the contested magnitudes: all 12 matched model-by-budget differences have paired intervals above zero.

Q2, supervision target

The archive answers part of this.

(a) Gold labels, joint student: the supplementary contains a direct gold-label control (artifacts/gold_student/results.json): the same joint student trained on 1,200 gold labels (BCE, 3 epochs) reaches PAWS-EN 0.9435/0.9425/0.9376 over the three seeds, mean 0.9412, close to the teacher-distilled joint student's 0.9420. Since the two runs differ in objective and update count (BCE on 1,200 gold labels vs teacher-MSE on 9,000 scored pairs), we read this only as: a joint student at this level is reachable from gold labels alone, not as evidence that the two supervision targets are interchangeable. The corresponding gold-supervised independent student was not run, so for this trainable-student setup the route comparison is established for the teacher-score target only.

(b) Gold labels, frozen readers: the learning-curve experiment is itself gold-label supervision at matched budgets (Figure 2: joint 0.922–0.960 vs independent MLP 0.736–0.786 at 49k labels), and the gold-trained student additionally appears as a frozen backbone on the controlled tasks (Table 3: +9.69 [8.85, 10.56]).

(c) Checkpoint rule. For the consensus-vs-single-teacher contrast, re-selecting at a fixed common endpoint (epoch 3) instead of teacher-MSE moves each score by at most ~0.004 AUC (supplementary paper/revision_assets/consensus_endpoint.tex), with the consequence the paper reports (appendix, "Consensus versus one teacher"): "at their common third-epoch endpoint the PAWS interval includes zero while the QQP gain remains." The joint-vs-independent comparison itself applies one rule to both arms (minimize own development teacher-MSE; same three epochs and 1,689 updates); its difference, +0.1821, is far larger than the sensitivity above.

(d) Fidelity. The joint student reproduces the teacher scores far more faithfully (PAWS EN Pearson 0.9285 vs 0.5677; MSE 0.168 vs 0.819; supplementary paper/revision_assets/teacher_fit.tex), which is the claim as stated: the joint route learns this teacher relation more faithfully under matched supervision. On DSE [2], we read it as consistent: a trained independent student can carry a joint teacher's scores for deployment; our result is narrower (one teacher target, one budget), and the paper already limits it ("does not equate route FLOPs or prove that no independent architecture could do so"). The QQP (+0.0181) and MRPC (+0.0431) differences cross zero, and we draw no teachability conclusion from them.

AuthorsOpenReview ↗

Response to Reviewer LiUf (2/2) — protocol map and budget interpretation

(Continued from part 1/2.)

Q3, protocol map in one place

  • Breadth survey (Table 1; Tables 4, 7, 10): PAWS-X, per-model cells (confirm 150–500; overlap-only AUC near chance), discovery/development/confirm, every readout at its own development-selected layer. Two caption-stated exceptions: Table 7 reports Chinese at the English peak layer (14B at its own Chinese peak), and Table 9 (cross-architecture) evaluates all readouts at the joint-peak layer; both are auxiliary robustness checks, comparable within their own tables.
  • Common bank (Figure 2; Table 12; Table 13 distillation): official PAWS-X English splits, 49,258 / 2,000 / 2,000, shared training indices, seeds 17/29/43 across routes. This is the matched comparison the central claims rest on.
  • Token readers on the full train split (Table 11): 49,375 leak-filtered pairs, cached at mid and final layers, development-selected depth, scored on overlap-matched confirm cells (n=300). The 14B 0.868 belongs here: directly comparable to the other rows of Table 11, and qualitatively, not numerically, comparable to the common-bank curves (different evaluation bank and n).
  • Anchor-balanced QQP (Table 8, n=300) and the dedicated encoder/reranker bank (Tables 5–6, n=900 per language) are separate fixed banks.

Q4, budget interpretation

Yes, and we commit to that reading. Every learning-curve configuration also consumes 2,000 validation labels; the routes are not compute-matched; Appendix A states both consequences: "equal label counts are not equal resource budgets," and the results "do not establish a fixed equal-performance sample-complexity multiplier." "Ease of readout" in this paper means exactly: under the stated protocol, a $\le$128-dimensional linear probe on a frozen joint pass reaches 0.90+ from 1,200 training labels (plus the shared validation set), while the tested independent readers of frozen states do not reach that at any tested budget (the strongest is the 14B token reader's 0.868). Adapted encoders can reach it (Table 2: E5-Mistral LoRA 0.946 at 8,000 labels), which is precisely the boundary the paper draws between frozen or shipped states and task-adapted encoders. Online computation differs in the stated direction (a joint pass per pair vs. precomputable embeddings), which is why the conclusion separates retrieval from verification rather than recommending joint scoring everywhere.

We hope these answers, all drawn from archived records, address the two weaknesses within the scope you identified. We are glad to answer any follow-up, and would be grateful if you would take these answers into account in your final assessment.

AuthorsOpenReview ↗

Gentle Reminder: Rebuttal for Paper 42490

Dear Reviewer LiUf, hope you are doing well. This is a gentle reminder that our rebuttal has been submitted. We fully understand your busy schedule and appreciate your time to check our response whenever you are available. Thank you!

Area_Chair_uXnJOpenReview ↗

Please respond

Hello,

The discussion period ends soon. Please respond to the authors' last comments and note whether they changed your evaluation.

Thanks,

Your AC

Reviewer_yDNSOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents experiments that show that the cosine between sentence representations does not adequately capture identity, particularly on datasets that have comparable lexical overlap for both positive and negative instances. Multiple models covering a variety of architectures are tested on four methods of representing the sentence pairs, and on two languages (English and Chinese).

Critical strengths

Understanding sentence embeddings and the information they do or they do not encode is an important task. The paper presents exhaustive experiments on models covering all major architectures, and exploring four different representations for the sentence pair. The experiments explore all layers of a model, and detect highest level of information with respect to the identity relation in the middle range.

Critical weaknesses

Several experimental details are not clear, and the writing is difficult to follow in places.

Experimental details:

  • it is mentioned that the various pair representations are used to train probes for classification, but there is no description of these probes.
  • the cosine results: cosine results are reported alongside AUC scores obtained on balanced (pos/neg) test sets. What exactly are these cosine scores? Are they averages over the positive instances only? are they AUCs based on the cosine scores?
  • why are different numbers of pairs used in testing the different models in Table 1?
  • line 226 "PCA map fit" -- what were the parameters of the PCA? How many components?
  • why is a specific template necessary to obtain the joint encoding of a sentence pair? Why not just present both sentences?
  • what is the "wrapper"?

Writing:

  • line 15: "independently encoded sentence vectors" -- as they are encoded using the same model, "independent" is misleading here. Maybe "separately encoded" is what is meant?
  • what are the "intervals" mentioned on line 25?
  • what does it mean that "BGE rerankers enter the identity band"? (line 28)
  • line 45/46: "... identity of meaning is a property of a single sentence's representation" -- identity requires two arguments, maybe what is meant is that "... meaning is a property ..."?
  • lines 88-89 -- not clear. What does "A probe that reads only the template" mean? What is the template? maybe "process" instead of "read"?
  • lines 141-142: what does "independently selected ... controls" mean? What are the "controls"?
  • line 215: why not use train/dev/test splits? it's the more common terminology. Or it does not apply here?
  • "cell": what does this mean? Does this refer to splits? If yes, it would be easier for the reader if the terminology were consistent.
  • line 238: "1200 training labels" -- the data is binary, is it not? if yes, this would refer then to the "labels of the 1200 training instances"? Throughout the paper the term "label" seems to be used to mean "instances"
  • line 464: "Each ... model ... gets its joint direction located once" -- what does this mean?

Section 6 is not clear.

It is very interesting to understand the flow of information through a model's layers, so the fact that information relevant to identity peaks in the middle range is interesting and confirms previous studies of the information encoded in model layers. However, it is the information on the last layer that is passed through to downstream tasks or used in generation, so it would be useful to also have the scores on the last layer for the experiments that do not always provide this.

Questions for authors

Please clarify the issues raised under "Critical weaknesses", and in particular point out whether some of the raised issues are misunderstandings.

7 comments
AuthorsOpenReview ↗

Response to Reviewer yDNS (1/3) — informativeness and items (1)–(4)

Thank you for the detailed list, and for explicitly inviting us to point out which issues are misunderstandings. One term underlies most items, so we state it up front: throughout the paper, separate / independent encoding means the frozen model encodes each sentence in its own forward pass with no access to the partner (not a different model), while joint means a single pass over both sentences together (item 7). We then go through every item below. Most answers are locations in the submitted PDF; a few are genuine wording gaps, and for those we give the missing details directly here. No experimental claim changes. This part covers items (1)–(4); parts (2/3) and (3/3) follow with items (5)–(13) and (14)–(18).

On informativeness — what the paper adds beyond "cosine is inadequate": the comparison is not cosine versus a probe. Within one frozen model, the independent route is given progressively stronger readers: a probe trained on both vectors (late fusion), a full-width MLP trained on up to 49,258 labeled pairs, and a cross-attention reader over token sequences. Each stays below a linear probe on the joint pass. For the full-width MLP the gap is 0.622–0.646 vs 0.900–0.946 at 1,200 labels and 0.736–0.786 vs 0.922–0.960 at 49,258 (Figure 2); the strongest token-level reader reaches 0.868 on its own bank (Table 11). Partner shuffle returns the joint probe to chance, and adapting the encoder closes much of the gap (Table 2), which is where the paper places the boundary of the claim.

Experimental details

(1) Probe definitions — Section 3, "Pair readouts," quoted verbatim:

Separate cosine: last-token states from independent forwards, cosine. Separate probe: a logistic probe on (h1⊙h2,∣h1−h2∣)(h_1 \odot h_2, |h_1 - h_2|) after a PCA map fit on discovery. Late fusion: the same probe class on [h1;h2][h_1; h_2] — both vectors present, no mixing inside the LM. Joint probe: the two sentences are concatenated with a language-specific template that is not a yes/no question (A: … / B: …); we probe the last token of that joint pass. Probes are fit on discovery, and each readout's layer and regularisation are selected independently on development before confirm is examined.

The same paragraph defines the common-bank reader: a full-width independent MLP on [hA;hB;hA⊙hB;∣hA−hB∣][h_A; h_B; h_A \odot h_B; |h_A - h_B|] with no PCA bottleneck, "selecting depth, pooling, learning rate, and stopping point on validation." Two specifics the paragraph leaves implicit we state here: the logistic probes are L2-regularized, and the PCA width is given in item (4) below.

(2) The cosine columns are AUCs — Table 1 caption. Table 1's caption specifies "English confirm AUC … each readout (including cosine) is reported at the layer that maximises its own development AUC": the cosine between the two separately encoded vectors is used as a threshold-free ranking score, and we report its AUC on the same confirm pairs as every other column. No column in Table 1 is an average cosine over positives.

(3) Different n per row — a per-run evaluation size; every comparison is within-row. The number of confirm pairs is an input setting of each survey run, chosen before the run as a compute budget (150–500; the smallest is the largest model, Qwen2.5-32B). The design guarantee is within-row: all four readouts and the shuffle control in a row score the identical pairs, so every comparison we interpret is within-row. Appendix A states this scope explicitly: the survey supports "within-row recurrence rather than a matched-data cross-model leaderboard." Cross-model comparisons on matched data are instead provided by the common bank (Figure 2), where all models share training indices and the same 2,000-pair test.

(4) PCA width (line 226): at most 128 components, fit on discovery. Section 3 omits the width where the probe is defined; it appears only in Table 2's caption ("≤\le 128-dimensional linear probe"). PCA retains at most 128 components per representation and is fit only on discovery, so the joint probe's input is $\le$128-dimensional and the two-vector probes (product/difference, late fusion) take $\le$256 dimensions. The probe's regularization is then selected on development. The per-cell selections are archived in the supplementary materials (paper/data/survey_selection.json, field k_pca).

AuthorsOpenReview ↗

Response to Reviewer yDNS (2/3) — items (5)–(13)

(Continued from part 1/3; items (5)–(13).)

Experimental details (continued)

(5) Template necessity — tested; four concatenations including no template (Section 5). Section 5, "The wrapper is not the result," evaluates four concatenations: A:/B:, "Sentence 1/2", a raw separator, and a blank line; the last two are "just present both sentences." Joint confirm AUC is 0.919–0.950 (Qwen2.5-7B), 0.933–0.961 (14B), 0.924–0.967 (32B) across all four; every variant beats late fusion and collapses under partner shuffle. The same paragraph also reports that Mistral-7B and Llama-3-8B, at an off-peak layer on a smaller bank, are more wrapper-sensitive (0.776–0.881 and 0.716–0.863), yet every wrapper still clears late fusion and collapses under shuffle. The template never contains the words "paraphrase," "same," or "yes" (appendix note).

(6) "Wrapper" = the concatenation template of item (5). The fixed text used to place the two sentences in one context window; "wrapper" and "template" name the same object.

Writing

(7) Line 15, "independently encoded." Your reading is exactly what is meant: the same frozen model encodes each sentence in a separate forward pass, with no access to the partner sentence. "Independent" refers to independence from the partner, not to different models.

(8) Line 25, "intervals." 95% bootstrap confidence intervals of the GPT-2-large replication (n=300 confirm): joint 0.626 [0.565, 0.689] vs separate 0.561 [0.494, 0.631] (Section 4). They overlap, which is why the abstract retains the original exception.

(9) Line 28, "BGE rerankers enter the identity band." The band is the joint-probe level of the strong survey rows (Llama-3-8B, Mistral-7B, and Qwen2.5 from 1.5B up; joint 0.897–0.961, Table 1). Section 4 reports that BGE-reranker-large (0.940), v2-m3 (0.902), and base (0.888) "align with the joint-probe band," while MS-MARCO MiniLM rerankers (0.545–0.558) and Jina reranker-v2 (0.639) remain in the dense-encoder band (Table 6). The sentence distinguishes joint scorers that expose identity from joint scorers that do not.

(10) Lines 45/46. That sentence states the premise the paper tests, not our claim. The industrial default (cosine between shipped vectors) implicitly treats meaning identity as recoverable by comparing two single-sentence properties; the paper's result is that this premise fails under overlap control while joint computation succeeds.

(11) Lines 88–89 — what the shuffle control rules out. In the partner-shuffle control, sentence B is replaced by a random sentence while the template text is unchanged. A probe keying on superficial features of the joint pass (the fixed template tokens, "two sentences are present") would keep its AUC under shuffle. The actual probes collapse to ≈0.5, so they read the pairing (control defined in Section 3; the 0.46–0.54 same-layer range is reported in Section 5).

(12) Lines 141–142. "Independently selected" applies to the readers: the linear, nonlinear, and token-level readers each have their layer and hyperparameters chosen by their own development AUC, so no reader is evaluated at a layer chosen for another. Partner shuffle, nonce substitution, and encoder adaptation are the other three controls named in that sentence; they are controls of a different kind, not layer-selected under this rule (the shuffle values quoted in Section 5 are at the joint probe's layer).

(13) Line 215, splits. The common bank does use the official train/validation/test splits (49,258 / 2,000 / 2,000; Section 3). The breadth survey uses discovery/development/confirm to mark a stricter usage rule: confirm is examined exactly once, after all fitting and selection. Functionally, discovery = train, development = dev, confirm = test.

AuthorsOpenReview ↗

Response to Reviewer yDNS (3/3) — items (14)–(18) and closing

(Continued from part 2/3; items (14)–(18) and closing.)

(14) "Cell." One language-by-split subset of a model's survey bank (e.g., the English confirm cell), as in "Confirm cells contain 150–500 pairs" (Section 3). Lexical overlap is controlled within each cell (overlap-only AUC near chance; Table 1, column Ov.); "cell" does not name a different kind of split.

(15) Line 238, "1,200 training labels." Agreed: this means the labels of 1,200 training pairs, and throughout the paper a "label budget" counts labeled training pairs. The one place where labels are manipulated as labels is the random-label control: permuting the 1,200 labels while keeping pairs and features fixed gives test AUC 0.505/0.510/0.489.

(16) Line 464, "gets its joint direction located once." Section 7's protocol: for each of six models, the joint-probe direction is fit once on PAWS English discovery, then frozen; all six frozen directions then score the same held-out items. This is what makes the cross-model agreement analysis meaningful (mean pairwise Spearman 0.855).

(17) Section 6. A compressed restatement: each generated fact yields anchor–candidate pairs in which exactly one thing changes: agent/patient roles, the scope of a negation over two events, or which quantity attaches to which entity. Example (role; from artifacts/cross_validation_v1/bank.json): the anchor "Daphne questioned Cyrus." is paired with "Cyrus was questioned by Daphne." (label 1) and "Daphne was questioned by Cyrus." (label 0); the role-reversed anchor "Cyrus questioned Daphne." receives the same two candidates with the labels swapped, giving [1,0,0,1]. Labels follow from the construction rule and are parser-checked; correct and incorrect candidates have identical unigram counts, so no bag-of-words shortcut exists. Crossing 3 relations × familiar/nonce entities × 3 styles gives 18 held-out panels over the same 96 facts. Four frozen backbones each receive a joint and an independent full-width MLP trained on the same 1,024 labels (Table 3).

(18) Last-layer scores. Reported where the depth question is sharpest (Figure 3; Section 5): final-layer (postnorm) joint AUC is systematically below the mid-depth peak (Llama 0.897 → 0.731; Qwen2.5-14B 0.950 → 0.940), and a final-norm edit that wrecks next-token prediction (KL 36, top-1 agreement 0.4%) leaves separate cosine essentially unchanged (0.558 vs 0.561). The frozen-token readers are cached at mid and final layers with depth selected on development (Table 11). For the 19 dense encoders and 6 rerankers, the shipped output is the final layer by definition. We agree the mid-depth peak is consistent with prior layer-wise studies. On the last layer specifically: in Figure 3's four models, the final-layer joint readout stays above same-layer late fusion and cosine (clearly for the Qwen models and Llama, narrowly for GPT-2 XL), while at the earliest depths all readouts sit near chance. The survey table reports each readout at its development-selected layer and has no separate last-layer column, but the uploaded supplementary archives the per-layer records (paper/data/survey.json): at the final (postnorm) layer, Qwen2.5-14B reads joint 0.940 vs late fusion 0.522 and separate probe 0.569, and Llama-3-8B reads joint 0.731 vs 0.487 and 0.528.

We hope this map of locations resolves the soundness and informativeness concerns: the experimental claims are unchanged and checkable at the cited locations, and where our terminology was not self-sufficient in the text ("cell," "wrapper," the split naming, the PCA width), the definitions are now stated directly above. We are glad to answer any follow-up before the discussion closes, and if these clarifications resolve the ambiguities, we would be grateful if you would take them into account in your updated assessment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response — summary of clarifications

We thank all three reviewers. R-LiUf and R-RxUR engage the contribution and its boundary directly, and we confirm their reading here: within the same frozen backbone, propositional identity is far more cheaply readable from a joint pass than from independent encodings (on a common 2,000-pair test, joint linear readouts reach AUC 0.900–0.946 from 1,200 labels versus 0.622–0.646 for full-width nonlinear readers of independent states). This is an accessibility result under tested readers and budgets, not an impossibility claim, and not a compute-matched architecture ranking. R-yDNS's line-by-line list allowed us to anchor every item to the submitted text; it also showed us which of our terms are not self-sufficient in the text, and our response supplies those definitions directly.

Every number quoted in our responses is in the submitted paper or the uploaded supplementary materials. In brief:

  • To R-yDNS: we give the location in the submitted PDF for each listed item (probe definitions, cosine-as-AUC, the wrapper ablation, the shuffle logic), and we directly supply the definitions our text left under-specified ("cell," "wrapper," the discovery/development/confirm naming) together with the PCA width (128) that Section 3 leaves unstated.
  • To R-LiUf: archived training-fit diagnostics, a gold-label joint student on natural PAWS archived in the uploaded supplementary (artifacts/gold_student/; PAWS-EN mean 0.9412 from 1,200 gold labels), the checkpoint-rule sensitivity check, and a one-place protocol map.
  • To R-RxUR: the boundaries named (no FLOP matching; information retained at 0.868; QQP/MRPC distillation inconclusive) are the ones the paper draws, with the anchor-balanced QQP bank separating identity from topical similarity and the zero-shot SNLI ordering adding a graded ranking (lexical overlap not matched there).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.