- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether sentence-level meaning identity is accessible from independently encoded representations or is computed when two sentences are jointly processed. The authors compare cosine similarity, probes over independent representations, late fusion, and linear probes over joint hidden states. Across PAWS-X, QQP, multiple model families, architectures, languages, and controlled relation tasks, joint representations consistently expose a much stronger identity signal. Additional experiments with stronger independent readers, encoder adaptation, and matched-supervision distillation suggest that the main difference is not information-theoretic absence, but substantially easier accessibility through joint computation.
Critical strengths
- The empirical evidence for the joint-over-independent gap is unusually comprehensive. The paper covers many decoder checkpoints, bidirectional encoders, encoder-decoder models, embedding models, rerankers, multiple languages, and several controlled relation tasks. The same qualitative pattern appears across most of these settings.
- The paper identifies a useful distinction between semantic retrieval and semantic verification. The results suggest that embeddings can remain effective for retrieving texts about the same topic while being unreliable as a certificate that two propositions are actually equivalent.
- The paper is appropriately nuanced about encoder adaptation. Fine-tuning or LoRA can substantially improve independent encoders, which helps clarify that the claim is about accessibility in shipped representations rather than an absolute architectural impossibility.
Critical weaknesses
- The joint and independent routes are not fully compute-matched. The joint route receives pairwise interaction throughout the pretrained backbone, while the independent route must learn this interaction afterward. This makes it difficult to separate representational accessibility from pretrained pairwise computation.
- The strongest independent reader already closes a substantial part of the gap. Frozen-token cross-attention reaches 0.868 AUC on the 14B model, suggesting that much of the relevant information is still retained in independently encoded token states.
- Generalization beyond PAWS-style identity is still somewhat limited. QQP transfer is useful, but the direct distillation advantage on QQP and MRPC is small and statistically inconclusive, while the main controlled experiments remain relatively synthetic.
Questions for authors
- How do you distinguish meaning identity from a more general semantic similarity? For example, can the same frozen direction separate paraphrases from entailments, contradictions, and topically similar but propositionally different pairs under controlled lexical overlap?
- How much of the joint advantage remains under a more compute- or capacity-matched independent interaction model?
- How should the partial QQP/MRPC transfer be interpreted? In particular, does the weak distillation gap on these datasets suggest that the learned relation is partly PAWS-specific?
3 comments
Gentle Reminder: Rebuttal for Paper 42490
Dear Reviewer RxUR , hope you are doing well.We are writing to gently remind you that we have submitted our rebuttal for this paper. We understand you have a heavy reviewing workload. We would greatly appreciate it if you could take a moment to go through our response when you have time.Thank you very much for your time and efforts.
Please respond
Hello,
The discussion period ends soon. Please respond to the authors' last comments and note whether they changed your evaluation.
Thanks,
Your AC
Response to Reviewer RxUR — scope boundaries and the identity-vs-similarity question
Thank you for the constructive review and for highlighting the retrieval-versus-verification distinction. The three weaknesses are boundaries we accept, and each is drawn in the submitted text. We answer them and the three questions below.
W1 and Q2, compute matching
We agree the routes are not compute-matched and that the joint route benefits from pairwise interaction performed during pretraining; the distillation paragraph states "this comparison does not equate route FLOPs or prove that no independent architecture could do so," and Appendix A generalizes the caveat. The claim is therefore deliberately about shipped artifacts: given the representations systems actually deploy, how much supervision and machinery does each route need before the relation is readable? The independent interaction model we tested is the two-block bidirectional cross-attention reader over frozen token sequences, a trained module with orders of magnitude more parameters than the probes. Under matched labels it reaches 0.545/0.555/0.786 at 1.2k/8k/49k (1.5B) and 0.868 (14B, full split), versus a $\le$128-dimensional linear probe on the joint pass at 0.90+ from at most 1,200 labels on the same backbones. What the data rule out are the cheaper explanations: misaligned geometry (late fusion stays at 0.48–0.59) and linear insufficiency (the full-width MLP gap persists at every budget). A fully FLOP-matched interaction model trained from scratch is beyond this submission, and we claim nothing about it.
W2, the 0.868
Yes: the paper reports it as "the strongest independent frozen-backbone result we observe" and cites it as evidence that useful material is retained in independent token states (Section 4). That is half of the stated claim; the other half is that access remains unequal. On the survey side where the 0.868 is scored, the same 14B backbone's joint linear probe reaches 0.950 [0.925, 0.971] (Table 1), and the 0.868 required the full 49k training pairs plus the trained cross-attention module (the same reader stays at 0.486–0.561 from 1.2k to 20k pairs; Table 11), where the joint probe is fit on 1,000 labels. On the matched common bank, Figure 2 separately shows joint 0.922–0.960 versus independent full-width MLP 0.736–0.786 at the full budget, with all 12 matched model-by-budget differences having paired intervals above zero. The Conclusion's phrase "not information-theoretic absence" names exactly this pair of facts, and Appendix A elaborates it.
W3 and Q3, PAWS-specificity
Partially PAWS-specific, and the paper quantifies how much. The frozen PAWS direction transfers zero-shot at 0.72–0.75 to overlap-matched QQP and at 0.745–0.818 to Chinese PAWS (near the in-domain 0.766–0.851); in reverse, a QQP-trained probe reads PAWS at up to 0.848 without seeing a PAWS pair (Section 5, "The computed direction transfers; the fitted geometry does not"). So a sizable transferable component coexists with a dataset-specific part ("part of any probe is dataset-specific"). The distillation differences on QQP (+0.0181) and MRPC (+0.0431) cross zero; we draw no conclusion from them, as you recommend. The controlled tasks add held-out-relation transfer within a restricted grammar (all 12 backbone-by-relation averages above chance), and the paper states the limit: "transferable structure in this restricted grammar, not universal expression-invariant correctness."
Q1, identity vs. general similarity
Two existing results separate them; the second is not overlap-controlled.
We hope this confirms that the boundaries you identified are the ones the paper draws. We are glad to answer follow-ups, and would be grateful if you would take these answers into account in your final assessment.