- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose CoFree, an LLM embedder that first generates a short "reasoning" text for a query or document and then embeds the input together with that text. The authors argue that such models suffer from "reasoning collapse". Embedding training either destroys the ability to generate text (generation collapse) or yields fluent text that does not help retrieval (semantic collapse).
CoFree starts from an existing embedding model (here, Qwen3-Embedding-4B or gte-Qwen2-1.5B-instruct) and trains it in two stages:
- SFT on reasoning texts that Qwen3-235B writes for queries and documents. The loss combines next-token prediction, InfoNCE, and an MSE term. The MSE term pulls the embedding of input plus reasoning toward the frozen backbone's embedding of the input excl. reasoning.
- GRPO with two rewards: the policy's own cosine margin between the positive and a hard negative, and the score margin of Qwen3-Reranker-4B on the reasoning-augmented pairs. A second MSE term anchors the query embeddings to the SFT model.
The authors benchmark on MTEB's retrieval tasks and BRIGHT. CoFree-4B reaches 40.8 vs. 38.0 for Qwen3-Embedding-4B. CoFree-1.5B reaches 34.6 vs. 26.9 for gte-Qwen2-1.5B-instruct.
I for now lean toward rejection. Reason-then-embed models are a crowded field (LREM, O1 Embedder, GRACE, UME-R1, Search-R3), and each component of CoFree has a close precedent. An additional contribution over those requires a clean experimental setup to show that its additions matter. This submission does not yet meet that bar. Notably, the backbones are evaluated well below their official scores, and the control that isolates the claimed mechanism is missing (cf. weaknesses). I invite the authors to rebut my review.
Critical strengths
- Speed: CoT embedders are typically slow at inference, since they generate hundreds of tokens per query on average. In contrast to prior models, CoFree only needs 44 to 46.
- Strong ablation: The authors input the the same generated text to six frozen encoders, with Echo and length-matched noise as controls. CoFree's text improves performance on every encoder, strongly supporting the notion that the generated text carries retrieval-relevant content.
Critical weaknesses
LLM backbones are listed below their official scores. On the same 10 retrieval tasks as Table 8, the official MTEB results of Qwen3-Embedding-4B average ~68.5, vs. 64.1 in Table 1. CoFree-4B (66.4) thus falls 2.1 points below its own backbone and wins only on sensitive ArguAna and HotpotQA. On the same tasks, gte-Qwen2-1.5B-instruct averages ~50.3 officially vs. 43.1 in Table 1.
This appears due to inconsistent handling of retrieval instructions (cf. ArguAna). The authors note that "Each model uses its own default retrieval instruction, fixed across all evaluation datasets without dataset-specific customization." This strongly suggests that CoFree's rewriting may recover the task information that a generic instruction removes.
The ablations do not isolate the claimed mechanism. The paper's own numbers point away from the reasoning and the shared generator-encoder:
- The CoFree-1.5B encoder scores ~32.3 without reasoning and ~34.6 with it (cf. Table 14). Most of its almost 8-point gain over the backbone appears without reasoning
- embeddinggemma-300m fed CoFree-1.5B's text reaches 36.20, above CoFree-1.5B itself
- At 4B, contrastive and anchor training without reasoning scores 37.7, vs. 38.0 for the backbone. Contrastive-only FT drops to 32.6, so the "controlled comparisons" (L320) measure recovery from a damaging recipe
- Removing the reasoning reward costs 0.5 points, comparable to the RL seed spread (40.7 to ~40.9)
Missing variants are: (i) the frozen backbones with official instructions plus teacher-, prompted-, or CoFree-generated text; (ii) the CoFree-4B encoder without and with teacher reasoning; (iii) RL with only KL and MSE, excl. both rewards.
Hence, soundness currently only scores as a 1.
Questions for authors
- Which instruction strings did each model, including CoFree, receive on each dataset? What do Qwen3-Embedding-4B, gte-Qwen2-1.5B-instruct and CoFree reach with the official task-specific instructions?
- What do the frozen LLMs reach when given CoFree-generated text on queries and documents? And what do they reach with text from the Qwen3-235B teacher (Prompts 1 and 2)? Table 14 runs this test for six other encoders, but not for the two backbones.
- What does the CoFree-4B encoder reach without reasoning at inference, and with teacher-written reasoning? This would separate the encoder training from the generated text, as Table 14 does for CoFree-1.5B.
12 comments
Response to Reviewer xCfv (2/4): Frozen-backbone reasoning
Frozen backbones with different reasoning sources (Question 2): What do the frozen LLMs reach when given CoFree-generated text on queries and documents? And what do they reach with text from the Qwen3-235B teacher (Prompts 1 and 2)? Table 14 runs this test for six other encoders, but not for the two backbones.
Response: We keep the two initialization backbones, Qwen3-Embedding-4B and gte-Qwen2-1.5B-instruct, fixed and compare retrieval performance using reasoning generated by CoFree-4B and the Qwen3-235B teacher. Teacher reasoning uses Prompts 1–2 in Appendix G.1 for query- and document-side generation, following the prompted-reasoning procedure in Appendix D.4.
Each sub-benchmark score is the mean nDCG@10 (0–100) across these two frozen encoders. The first two sub-benchmarks are from MTEB and the remaining three from BRIGHT. Mean averages the five sub-benchmark scores. All scores and gains are reported to one decimal place; means and gains are calculated before rounding.
| Reasoning source | CQADupstackUnix | TRECCOVID | Biology | Sustainable Living | TheoremQA-theorems | Mean |
|---|---|---|---|---|---|---|
| No reasoning | 48.2 | 71.7 | 15.1 | 16.9 | 15.3 | 33.4 |
| Qwen3-235B | 47.1 | 65.1 | 22.4 | 14.8 | 18.6 | 33.6 |
| CoFree-4B | 49.9 | 77.5 | 20.7 | 19.2 | 22.6 | 38.0 |
CoFree-4B reasoning improves the five-sub-benchmark mean by 4.6 points over no reasoning and 4.4 points over teacher reasoning. When averaged across the two backbones, it improves over no reasoning on all five sub-benchmarks and outperforms teacher reasoning on four of five. The teacher performs better on Biology.
We also compare Llama-3.2-3B-Instruct and CoFree-1.5B reasoning, reporting Overall nDCG@10 across all 22 datasets (Appendix D.4, Table 14).
| Frozen encoder | No reasoning | Llama reasoning | CoFree reasoning | Gain over Llama |
|---|---|---|---|---|
| embeddinggemma-300m | 33.5 | 27.3 | 36.2 | +8.9 |
| jina-embeddings-v3 | 30.7 | 25.8 | 32.9 | +7.0 |
| Linq-Embed-Mistral | 31.8 | 26.5 | 34.8 | +8.3 |
| Search-R3-Small | 26.0 | 23.6 | 27.8 | +4.2 |
| CoFree-1.5B | 32.3 | 26.4 | 34.6 | +8.2 |
| UME-R1-2B | 21.9 | 23.0 | 24.0 | +1.1 |
CoFree-1.5B reasoning outperforms Llama-generated reasoning on all six encoders, with gains of 1.1–8.9 Overall points. CoFree reasoning also improves over no reasoning on all six encoders.
Response to Reviewer xCfv (3/4): Encoder training and transfer
Encoder training versus inference reasoning (Question 3): What does the CoFree-4B encoder reach without reasoning at inference, and with teacher-written reasoning? This would separate the encoder training from the generated text, as Table 14 does for CoFree-1.5B.
The CoFree-1.5B encoder scores ~32.3 without reasoning and ~34.6 with it (cf. Table 14). Most of its almost 8-point gain over the backbone appears without reasoning.
Response: We hold the final CoFree-4B encoder fixed and compare original inputs, teacher reasoning, and CoFree reasoning on the same five selected sub-benchmarks. This separates the contribution of the generated input text from the performance already retained in the encoder's parameters. Scores are nDCG@10 (0–100), and Mean averages the five sub-benchmark scores.
| Final encoder | Inference input | CQADupstackUnix | TRECCOVID | Biology | Sustainable Living | TheoremQA-theorems | Mean |
|---|---|---|---|---|---|---|---|
| CoFree-4B | Original text | 56.2 | 87.4 | 20.2 | 17.6 | 27.7 | 41.8 |
| CoFree-4B | Original text + Qwen3-235B reasoning | 56.3 | 84.3 | 21.6 | 16.8 | 30.8 | 42.0 |
| CoFree-4B | Original text + CoFree reasoning | 57.8 | 88.7 | 22.4 | 18.5 | 32.6 | 44.0 |
With the final encoder held fixed, CoFree reasoning improves the five-sub-benchmark mean by 2.2 points over original inputs and 2.0 points over teacher reasoning. It outperforms both alternatives on all five sub-benchmarks, showing the additional retrieval benefit of CoFree's generated text.
Across all 22 datasets, the following Overall nDCG@10 results separate the gains from encoder training and inference reasoning.
| Model | Inference input | Overall |
|---|---|---|
| Qwen3-Embedding-4B | Original text | 38.0 |
| CoFree-4B | Original text | 38.7 |
| CoFree-4B | Original text + CoFree reasoning | 40.8 |
Encoder training improves Overall by 0.7 points. Holding the final CoFree-4B encoder fixed, adding its reasoning further improves Overall by 2.1 points, with gains of 2.5 points on MTEB and 1.7 points on BRIGHT.
At 1.5B, the corresponding progression is 26.9 for the original backbone, 32.3 for the final CoFree encoder on original inputs, and 34.6 with CoFree reasoning. Thus, encoder training accounts for the larger part of the 1.5B improvement, while inference reasoning provides a further 2.3 points. The two scales demonstrate both improved encoder representations and an additional contribution from generated reasoning.
Transfer versus a shared generator–encoder (Weakness): embeddinggemma-300m fed CoFree-1.5B's text reaches 36.20, above CoFree-1.5B itself.
Response: EmbeddingGemma-300M already scores higher without reasoning: 33.50 versus 32.33 for the final CoFree-1.5B encoder. Adding the same CoFree-1.5B reasoning improves Gemma to 36.20 and CoFree to 34.61 (Appendix D.4, Table 14). Gemma's higher final score therefore builds on its stronger original-input performance, with CoFree reasoning improving both encoders.
CoFree's joint training enables a single model to generate retrieval-useful reasoning and produce embeddings. The gain on the stronger Gemma encoder further supports the quality of CoFree's reasoning text and its usefulness beyond CoFree's own encoder.
Recovery from contrastive fine-tuning (Weakness): At 4B, contrastive and anchor training without reasoning scores 37.7, vs. 38.0 for the backbone. Contrastive-only FT drops to 32.6, so the "controlled comparisons" (L320) measure recovery from a damaging recipe.
Response: Our SFT stage aims to restore reasoning generation while preserving the pretrained embedding model's representational ability (Section 3.1). Language-modeling supervision restores generation, while contrastive learning and reference guidance preserve embedding quality during this process.
We include Contrastive FT to rule out the possibility that the gains come solely from the training data. The 4B results are summarized below (Overall nDCG@10; Tables 1 and 4).
| Model / training variant | Overall |
|---|---|
| Original backbone (Qwen3-Embedding-4B) | 38.0 |
| Contrastive FT | 32.6 |
| Contrastive FT + reference guidance | 37.7 |
| Full SFT | 39.9 |
Reference guidance retains retrieval performance close to the original backbone. Full SFT additionally learns reasoning generation and improves Overall by 2.2 points over the reference-guided control and 1.9 points over the original backbone.
Response to Reviewer xCfv (1/4): Retrieval instructions
Thank you for recognizing CoFree's inference efficiency and the evidence that its generated text carries retrieval-relevant information across frozen encoders. We address the instruction comparison, the separate contributions of encoder training and inference reasoning, and the roles of the training objectives below.
Response: In Table 1, embedding models and CoFree use the respective fixed instructions listed below across all evaluation datasets. We use fixed instructions to support fair comparisons across datasets and reduce the influence of dataset-specific prompt choices on the reported gains. This also simplifies training on a mixture of retrieval datasets by allowing a common instruction format throughout training. In practical deployment, inputs may come from diverse or unfamiliar domains without a tailored task description, so we consider reliable retrieval with a general instruction an important aspect of robustness.
Fixed instructions used in Table 1:
Retrieval relevant passage for the given query.Reason and Embed the input:Results with dataset-specific instructions: Publicly reported scores and our re-evaluated scores may differ across evaluation setups. We compare the backbones and CoFree within the same MTEB evaluation framework, enabling the dataset-specific instructions provided by the repository for both. Scores are nDCG@10 (0–100).
Dataset-specific instructions improve all four models, while CoFree retains its advantage over both backbones. Under this matched setting, CoFree reaches 67.0 versus 66.4 on MTEB at 4B, and 59.6 versus 48.2 at 1.5B. It also improves BRIGHT and achieves Overall gains of 2.5 points at 4B (41.9 vs. 39.4) and 6.7 points at 1.5B (36.1 vs. 29.4). These results show that CoFree's gains persist when task information is explicitly supplied through dataset-specific instructions.