acceptodds
Under review as a conference paper at ICLR 2027

When Does Dense Retrieval Need Asymmetric Geometry? A Bias–Variance Theory of Shared and Dual Projections

Abstract

Dense retrieval powers retrieval-augmented generation, semantic search, and question answering, yet the theoretical basis for choosing between shared and dual query–document projections remains unclear. We introduce a bias–variance theory for low-rank bilinear scoring. Shared projections induce positive-semidefinite operators, whereas dual projections realize arbitrary low-rank operators. We derive their exact approximation gap and prove a local Gaussian boundary: dual has lower risk exactly when squared directional signal exceeds the estimation cost of its additional degrees of freedom. This boundary motivates the Cross-fitted Asymmetry Risk Selector (CARS), which estimates reproducible directional signal from training pairs; its Gaussian counterpart admits exact selection-power and regret formulas. Guided by the theory, we run retrieval experiments across multiple datasets and embedding models. The mean Dual-minus-Shared NDCG@10 advantage more than doubles as query rotation increases from \(0^\circ\) to \(90^\circ\). In the rank–sample-size grids, Shared wins 13 of 16 cells at \(n=32\), whereas Dual wins all 32 cells at \(n=1024\) and \(2048\). Consistent with this shift, all 168 comparable operator-risk curves move toward Dual as training data grow. Compared to the two fixed-geometry baselines, CARS reduces held-out regret by 49–96% and achieves 90.1% mean geometry-selection accuracy.

Primary area:
learning theory
Keywords:
dense retrieval, shared and dual projections, representation learning, low-rank bilinear models, model selection

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_mAwGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when dense retrieval should use shared query–document projections versus separate projections. The authors develop a bias–variance theory showing that dual projections can better capture directional differences between queries and documents, but require more data to estimate reliably. They derive a local Gaussian decision boundary and propose CARS, a cross-fitted method for choosing between the two geometries.

Critical strengths
  • The paper provides a useful theoretical perspective on the tradeoff between shared and dual projections, with an explicit boundary relating directional signal to the estimation cost of additional degrees of freedom.

  • The experimental evaluation covers multiple embedding models, retrieval datasets, ranks, and training sizes.

Critical weaknesses
  • The paper was hard to follow; particularly, although the mathematical derivation is detailed, it lacked the intuition, and the practical implications of several key results are not sufficiently explained. It was difficult to understand what the derived quantities imply in a real retrieval setting and how the theoretical results translate into practical guidance for choosing between shared and dual projections. Hence, I suggest that the authors to include more intuitive explanations when presenting the main mathematical results along with concrete examples to make the paper easier to follow.

  • The theoretical boundary relies on a local Gaussian model with isotropic noise. Although the experiments support similar qualitative trends on real embeddings, it remains unclear under what data conditions and practical settings the proposed boundary is expected to hold. Hence, more guidance on the assumptions required for the boundary to be useful in practice would help clarify the scope and applicability of the theoretical results.

Questions for authors
  • Could the authors provide more intuitive explanations or concrete examples to clarify the meaning of the key theoretical quantities during the theoretical derivations?

  • Under what data conditions and practical settings is the proposed theoretical boundary expected to hold? In particular, how sensitive is it to deviations from the local Gaussian assumption, and with what data would one expect such a boundary to hold?

3 comments
AuthorsOpenReview ↗

Response to Reviewer mAwG (1/2)

We sincerely appreciate your time in reviewing the manuscript and your thoughtful feedback. We reply to your comments below.

Question1 (& weakness1): Could the authors provide more intuitive explanations or concrete examples to clarify the meaning of the key theoretical quantities during the theoretical derivations?

Thank you for this constructive feedback. We have added concrete geometric and numerical examples throughout the revised manuscript.

  • Geometric example. We rotate one of two whitened views by angle θ\theta. At rank one, Dual retains the full positive–negative separation ρ\rho, while Shared retains ρcos⁡θ\rho\cos\theta. They tie at 0∘0^\circ; at 90∘90^\circ, Shared's separation falls to zero. This motivates the query-rotation experiment in Figure 5.
  • Numerical example. After Corollary 1, we show that if Dual saves 4 units of squared bias but adds 8 units of estimation variance at sample size nn, Shared is preferable. Holding the signal fixed, increasing the sample size to 4n4n reduces the extra variance to 2 units, making Dual preferable.
  • Crossovers on real embeddings. In 84 of 168 trajectories, Shared has lower operator loss at n=256n=256 but Dual has lower loss at the largest measured training size. In the other 84, Dual already has lower loss at n=256n=256.

The practical implication is to use Shared when data are scarce and directional signal is weak, and to favor Dual when the reproducible signal outweighs its estimation cost. When the CARS score is close to zero or varies across splits, we recommend comparing the two fits on validation NDCG@10.

You can find more information in the revised manuscript (Sections 2, 3, 5.4, and 6.)

AuthorsOpenReview ↗

Response to Reviewer mAwG (2/2)

Question2 (& Weakness2): Under what data conditions and practical settings is the proposed theoretical boundary expected to hold? In particular, how sensitive is it to deviations from the local Gaussian assumption, and with what data would one expect such a boundary to hold?

Thank you for asking when the boundary applies. For fixed dimension and rank, our first-order analysis considers departures of order n−1/2n^{-1/2} from a rank-rr Shared operator with a positive spectral gap at the rank cutoff. Query contributions are i.i.d. with uniformly bounded fourth moments, and CARS uses random splits into equal halves. The variance cost depends on covariance in the Dual-only directions; this comparison does not assume Gaussian or isotropic noise. In practical retrieval settings, bounded fourth moments hold automatically for L2-normalized frozen embeddings, and the boundary is expected to hold whenever the target rank rr is chosen at a clear singular-value drop (spectral gap) relative to the query budget nn.

We added a noise-sensitivity simulation to test deviations from Gaussianity and isotropy. For each spectral gap and sample size, the 36 settings cross three noise laws (Gaussian, Student-t5t_5, and contaminated Gaussian), three covariance allocations, and four signal strengths. The table reports agreement with the geometry of lower mean operator risk. Mean CARS uses the sign of the score averaged over simulation draws.

Spectral gapPrediction64 queries2561,0244,096
0.15Isotropic boundary30/3630/3630/3630/36
0.15Covariance-adjusted boundary30/3630/3630/3636/36
0.15Mean CARS30/3630/3633/3636/36
0.30Isotropic boundary30/3630/3630/3630/36
0.30Covariance-adjusted boundary30/3630/3636/3636/36
0.30Mean CARS30/3633/3636/3636/36
1.00Isotropic boundary31/3630/3630/3630/36
1.00Covariance-adjusted boundary35/3636/3636/3636/36
1.00Mean CARS29/3636/3636/3636/36

As the cutoff gap grows from 0.15 to 0.30 to 1.00, the first tested sample size at which both the covariance-adjusted boundary and mean CARS agree in all 36 settings falls from 4,096 to 1,024 to 256 queries. At a gap of 0.30 and 1,024 queries, both agree in 36/36 settings across heavy-tailed and contaminated noise, compared with 30/36 for the isotropic boundary. This explains why CARS remains reliable on real benchmarks (FEVER, NQ, MS MARCO, ArguAna, and SciFact), where unit-norm embeddings naturally bound tail moments and compact ranks (r∈{4,8,16}r \in \{4, 8, 16\}) enjoy well-separated leading singular values.

You can find more information in the revised manuscript (Sections 3.2 and 5.4 and Appendices C.2 and I).

Thank you again for your comments and for helping us improve the manuscript.

Reviewer_mAwGOpenReview ↗

Response to Rebuttal

Thanks to the authors for the clarifications and for addressing my concerns. I retain my acceptance recommendation.

Reviewer_mQXPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors give a bias–variance answer for that on top of frozen dense-retrieval embeddings, should query and document use a shared projection or separate ones. Shared projections are forced to be positive-semidefinite, dual ones are arbitrary low-rank, and the extra freedom only pays off when the directional signal on the dual-only directions beats the noise cost of the added parameters. From this they derive CARS, a cross-fitted selector that reads the signal off training pairs. In retrieval experiments the advantage of dual grows with query mismatch and training size, and CARS picks the geometry with low regret.

Critical strengths

1、easy to follow 2、Complete theory-to-algorithm closed loop.

Critical weaknesses

1、local Gaussian assumption is detached from real retrieval training, and the key bridge is an expectation identity only. The entire selection theory rests on a local Gaussian experiment around a rank-r shared operator, with isotropic noise and a known noise variance. 2、gap between the optimization objective and the evaluation metric. The theory optimizes operator Frobenius risk, while retrieval cares about NDCG at 10. The equivalence result only holds under whitened two-view optimal positive and negative separation, and does not cover the actual pipeline of a trained adapter followed by full-corpus inner-product ranking. 3、scale and coverage are insufficient, and some design choices appear biased. Only base-scale encoders of about 110 million parameters, rank up to 32, and standard domains such as MS MARCO and NQ are used; large-scale industrial corpora and multi-granularity settings are absent, leaving the transferability of the conclusions to production retrieval unclear.

Questions for authors

Does the geometry selected by CARS actually beat the fixed geometries on the retrieval metric?​ Table 1 currently reports only regret/accuracy on held-out operator risk. Could you provide a controlled experiment comparing "select shared/dual by CARS" against "Always Shared / Always Dual / select by validation NDCG" on test NDCG@10 across FEVER/NQ/MS MARCO? Otherwise it is impossible to rule out "better operator risk but no ranking difference."

4 comments
AuthorsOpenReview ↗

Response to Reviewer mQXP (1/3)

We sincerely appreciate your time in reviewing the manuscript and your thoughtful feedback. We reply to your comments below.

Question1 (& Weakness2): Does the geometry selected by CARS actually beat the fixed geometries on the retrieval metric? Table 1 currently reports only regret/accuracy on held-out operator risk. Could you provide a controlled experiment comparing "select shared/dual by CARS" against "Always Shared / Always Dual / select by validation NDCG" on test NDCG@10 across FEVER/NQ/MS MARCO? Otherwise it is impossible to rule out "better operator risk but no ranking difference."

Thank you for suggesting this controlled comparison. We added the requested retrieval experiment comparing CARS with Always Shared, Always Dual, and Validation-based selection on FEVER, NQ, and MS MARCO across 360 settings (three encoders, five folds). We report NDCG@10 and MRR@10 on the resulting test rankings.

The following table demonstrates the win rate on full-corpus test NDCG@10 and MRR@10. Each cell is the percentage of settings in which a rule attains or ties for the best score on that metric.

EncoderDatasetAlways SharedAlways DualValidation-based SelectionCARS
NDCG@10 / MRR@10NDCG@10 / MRR@10NDCG@10 / MRR@10NDCG@10 / MRR@10
BGE-baseFEVER37.5% / 37.5%30.0% / 30.0%32.5% / 32.5%50.0% / 50.0%
NQ37.5% / 40.0%37.5% / 32.5%25.0% / 27.5%37.5% / 40.0%
MS MARCO60.0% / 60.0%7.5% / 7.5%32.5% / 32.5%60.0% / 60.0%
E5-base-v2FEVER37.5% / 32.5%37.5% / 42.5%25.0% / 25.0%37.5% / 42.5%
NQ32.5% / 37.5%22.5% / 25.0%45.0% / 37.5%27.5% / 30.0%
MS MARCO65.0% / 67.5%0.0% / 0.0%35.0% / 32.5%65.0% / 67.5%
GTE-baseFEVER32.5% / 32.5%32.5% / 32.5%35.0% / 35.0%37.5% / 40.0%
NQ37.5% / 42.5%32.5% / 32.5%30.0% / 25.0%37.5% / 42.5%
MS MARCO65.0% / 62.5%2.5% / 2.5%32.5% / 35.0%65.0% / 62.5%
Overall45.0% / 45.8%22.5% / 22.8%32.5% / 31.4%46.4% / 48.3%

We found CARS ranks first or ties for first in 167 of 360 settings on NDCG@10 and 174 of 360 on MRR@10, with overall win rates of 46.4% and 48.3%, respectively. On each metric, CARS has the highest or joint-highest win rate in eight of the nine encoder–dataset pairs. The validation-based baseline reserves some data for validation (less data for training), which may partly explain its lower first-place rates.

Moreover, by further examining specific settings, we find that the results align with our theory: under small budgets (n≤256n \le 256), a high rank (r=32r=32) inflates the extra degrees of freedom, causing Shared to win 174/180 settings. At a compact rank (r=4r=4), Shared and Dual are balanced.

You can find more information in the revised manuscript (Section 5.5).

AuthorsOpenReview ↗

Response to Reviewer mQXP (2/3)

Weakness1: local Gaussian assumption is detached from real retrieval training, and the key bridge is an expectation identity only. The entire selection theory rests on a local Gaussian experiment around a rank-r shared operator, with isotropic noise and a known noise variance.

Thank you for this valuable comment. Under the stated local conditions, we prove that the expected CARS score computed from the nonlinear spectral fits in Eq. (7) matches the full-sample operator-risk gap to first order, without assuming that the half-sample fits are unbiased. We added a first-order analysis under anisotropic noise, along with noise-sensitivity simulations under Gaussian, Student-t5t_5, and contaminated Gaussian noise. In practice, CARS estimates the variance cost directly from split disagreement without requiring a known noise variance; isotropic Gaussianity is invoked only to derive the exact closed-form power and regret formulas.

For each spectral gap and sample size, we run 36 settings: 4 with isotropic Gaussian noise, 8 with anisotropic Gaussian noise, 12 with Student-t5t_5 noise, and 12 with contaminated Gaussian noise. Each table entry counts the settings in which the predicted geometry (based on the sign of the simulated mean score) matches the one with lower mean operator risk.

Spectral gapQueriesIsotropicCovariance-adjustedMean CARS
0.156430/3630/3630/36
0.1525630/3630/3630/36
0.151,02430/3630/3633/36
0.154,09630/3636/3636/36
0.306430/3630/3630/36
0.3025630/3630/3633/36
0.301,02430/3636/3636/36
0.304,09630/3636/3636/36
1.006431/3635/3629/36
1.0025630/3636/3636/36
1.001,02430/3636/3636/36
1.004,09630/3636/3636/36

While the theoretical Isotropic boundary plateaus at 30/36 because it ignores directional variance under anisotropic noise, both the Covariance-adjusted theoretical boundary and the practical Mean CARS score (which requires no prior variance knowledge) reach 36/36 agreement (e.g., at a gap of 0.30 with 1,024 queries). This confirms that split disagreement naturally captures the variance in Dual-only directions without assuming isotropic Gaussian noise.

You can find more information in the revised manuscript (Section 3.2 and Appendices C.2 and I).

AuthorsOpenReview ↗

Response to Reviewer mQXP (3/3)

Weakness3: scale and coverage are insufficient, and some design choices appear biased. Only base-scale encoders of about 110 million parameters, rank up to 32, and standard domains such as MS MARCO and NQ are used; large-scale industrial corpora and multi-granularity settings are absent, leaving the transferability of the conclusions to production retrieval unclear.

Thank you for raising the question of scale and coverage. The four encoders in the original submission each had approximately 110M parameters. To test whether our findings extend across encoder sizes, we extended the evaluation to two additional encoders: BGE-small (33.4M parameters) and BGE-large (335M parameters). The results are reported in Table 1 of the revised manuscript. The evaluation covers 4,200 configurations across five public benchmarks (ArguAna, MS MARCO, FEVER, NQ, and SciFact) and six frozen encoders ranging from 33.4M to 335M parameters. Separately, we evaluate full-corpus retrieval over 8.84 million MS MARCO passages. Due to space constraints, we show results for the two newly added encoders here; the full results are in Table 1 (Section 5.4) of the revised manuscript.

RuleDatasetBGE-small-en-v1.5 (33.4M)BGE-large-en-v1.5 (335M)
Regret ↓ / Acc. ↑ / Wins ↑Regret ↓ / Acc. ↑ / Wins ↑
Always SharedArguAna0.0541 / 55.0% / 0/50.0693 / 51.7% / 0/5
MS MARCO0.0759 / 75.0% / 0/50.0521 / 76.9% / 0/5
FEVER0.2264 / 40.0% / 0/50.2712 / 36.2% / 0/5
NQ0.0507 / 63.6% / 1/50.0144 / 75.0% / 1/5
SciFact0.0195 / 65.0% / 0/50.0137 / 76.7% / 2/5
Mean / total0.0853 / 59.7% / 1/250.0841 / 63.3% / 3/25
Always DualArguAna0.4628 / 45.0% / 0/50.4202 / 48.3% / 0/5
MS MARCO7.9342 / 25.0% / 0/511.0818 / 23.1% / 0/5
FEVER0.8199 / 60.0% / 0/50.6096 / 63.7% / 0/5
NQ1.2046 / 36.4% / 0/51.4327 / 25.0% / 0/5
SciFact0.4994 / 35.0% / 0/50.5867 / 23.3% / 0/5
Mean / total2.1842 / 40.3% / 0/252.8262 / 36.7% / 0/25
Validation-basedArguAna0.0767 / 91.7% / 0/50.0566 / 91.7% / 0/5
MS MARCO1.0609 / 87.5% / 0/51.4000 / 90.6% / 0/5
FEVER0.1570 / 93.1% / 1/50.1657 / 90.0% / 1/5
NQ0.2042 / 90.0% / 0/50.2012 / 87.1% / 0/5
SciFact0.0912 / 83.3% / 0/50.1001 / 80.0% / 0/5
Mean / total0.3180 / 89.1% / 1/250.3847 / 87.9% / 1/25
CARS (Ours)ArguAna0.0052 / 87.5% / 5/50.0005 / 98.3% / 5/5
MS MARCO0.0148 / 93.8% / 5/50.0055 / 95.6% / 5/5
FEVER0.2501 / 86.2% / 4/50.1106 / 83.8% / 4/5
NQ0.0260 / 85.0% / 4/50.0040 / 90.7% / 4/5
SciFact0.0045 / 91.7% / 5/50.0159 / 83.3% / 3/5
Mean / total0.0601 / 88.8% / 23/250.0273 / 90.4% / 21/25

For each new encoder, CARS has the lowest mean regret and beats validation-based selection in 24 of 25 folds. Across all six encoders, CARS beats validation-based selection in 142/150 folds and strictly beats all three baselines in 128/150 folds.

Thank you again for your comments and for helping us improve the manuscript.

Thank you again for your time and for carefully reviewing our paper. As the discussion deadline approaches, we would appreciate it if you could let us know whether the revision addresses your concerns (or partially). If you are satisfied with our response, we would be grateful if you would consider updating your rating. Thank you!

Reviewer_u3T1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies shared and dual low-rank projections for adapting frozen retrieval embeddings. It derives their approximation gap and a local Gaussian risk boundary that balances directional signal against additional estimation variance. It then proposes CARS, a training-only geometry selector based on agreement across data splits. Experiments examine the predicted tradeoff through simulations, retrieval under controlled query rotations, and held-out operator-loss evaluation.

Critical strengths
  1. The theory gives a precise account of the sharing tradeoff. The operator formulation connects the PSD restriction of shared projections to approximation bias. The local analysis then quantifies the additional variance of dual projections. This provides a useful explanation beyond the general claim that separate projections are more expressive.

  2. The experiments test specific theoretical predictions. Controlled rotations and sample-size sweeps examine when the preferred geometry changes. The paper also distinguishes model-choice regret from post-selection estimation risk, avoiding a stronger claim than the analysis establishes.

Critical weaknesses
  1. CARS is evaluated primarily against a surrogate objective. Its reported selection accuracy and regret concern Frobenius operator loss, not retrieval rankings. Table 1 compares only fixed Shared and Dual choices. It therefore remains unclear whether CARS improves NDCG@10 or offers an advantage over validation-based geometry selection.

  2. The practical CARS justification requires assumptions that are not established for its actual estimates. Equation (7) uses differences between nonlinear spectral projections. The expectation argument assumes centered additive half-sample errors with a specified covariance scaling. Projection bias and changes in estimated subspaces may violate this approximation, especially at small sample sizes. The exact Gaussian selection results do not resolve this practical gap.

Questions for authors

NA

3 comments
AuthorsOpenReview ↗

Response to Reviewer u3T1 (1/2)

We sincerely appreciate your time in reviewing the manuscript and your thoughtful feedback. We reply to your comments in the below.

Weakness1: CARS is evaluated primarily against a surrogate objective. Its reported selection accuracy and regret concern Frobenius operator loss, not retrieval rankings. Table 1 compares only fixed Shared and Dual choices. It therefore remains unclear whether CARS improves NDCG@10 or offers an advantage over validation-based geometry selection.

Thank you for pointing out this valuable question. We added a full-corpus retrieval experiment comparing CARS with Always Shared, Always Dual, and the validation-based method. This evaluation covers 360 settings spanning FEVER, NQ, and MS MARCO with BGE-base, E5-base-v2, and GTE-base with 5-fold cross-validation. All four rules use the same outer test folds.

The following table demonstrates the win rate on full-corpus test NDCG@10 and MRR@10. Each cell is the percentage of settings in which a rule attains or ties for the best score on that metric.

EncoderDatasetAlways SharedAlways DualValidation-based SelectionCARS
NDCG@10 / MRR@10NDCG@10 / MRR@10NDCG@10 / MRR@10NDCG@10 / MRR@10
BGE-baseFEVER37.5% / 37.5%30.0% / 30.0%32.5% / 32.5%50.0% / 50.0%
NQ37.5% / 40.0%37.5% / 32.5%25.0% / 27.5%37.5% / 40.0%
MS MARCO60.0% / 60.0%7.5% / 7.5%32.5% / 32.5%60.0% / 60.0%
E5-base-v2FEVER37.5% / 32.5%37.5% / 42.5%25.0% / 25.0%37.5% / 42.5%
NQ32.5% / 37.5%22.5% / 25.0%45.0% / 37.5%27.5% / 30.0%
MS MARCO65.0% / 67.5%0.0% / 0.0%35.0% / 32.5%65.0% / 67.5%
GTE-baseFEVER32.5% / 32.5%32.5% / 32.5%35.0% / 35.0%37.5% / 40.0%
NQ37.5% / 42.5%32.5% / 32.5%30.0% / 25.0%37.5% / 42.5%
MS MARCO65.0% / 62.5%2.5% / 2.5%32.5% / 35.0%65.0% / 62.5%
Overall45.0% / 45.8%22.5% / 22.8%32.5% / 31.4%46.4% / 48.3%

We found CARS ranks first or ties for first in 167 of 360 settings on NDCG@10 and 174 of 360 on MRR@10, with overall win rates of 46.4% and 48.3%, respectively. On each metric, CARS has the highest or joint-highest win rate in eight of the nine encoder–dataset pairs. The validation-based baseline reserves some data for validation (less data for training), which may partly explain its lower first-place rates.

Moreover, by further examining specific settings, we find that the results align with our theory: under small budgets (n≤256n \le 256), a high rank (r=32r=32) inflates the extra degrees of freedom, causing Shared to win 174/180 settings. At a compact rank (r=4r=4), Shared and Dual are balanced.

You can find more information in the revised manuscript (Section 5.5).

AuthorsOpenReview ↗

Response to Reviewer u3T1 (2/2)

Weakness2: The practical CARS justification requires assumptions that are not established for its actual estimates. Equation (7) uses differences between nonlinear spectral projections. The expectation argument assumes centered additive half-sample errors with a specified covariance scaling. Projection bias and changes in estimated subspaces may violate this approximation, especially at small sample sizes. The exact Gaussian selection results do not resolve this practical gap.

Thank you for raising this theoretical point. To bridge the gap between the exact Gaussian model and the nonlinear spectral projections in Eq. (7), we added an exact expectation calculation for the split score and a local expansion for the actual spectral fits. For fixed dimension and rank, let Cn=S+H/n+O(n−1)C_n=S+H/\sqrt{n}+O(n^{-1}), where SS is a rank-rr Shared operator with a stable spectral cutoff and HH is the local departure from SS. With independent query groups, bounded fourth moments, and equal half-splits, we obtain

E[Γ^CARS]=∥PAH∥F2−tr⁡(PAΩn)n+O(n−3/2),\mathbb{E}[\widehat{\Gamma}_{\mathrm{CARS}}] = \frac{\lVert P_{\mathcal{A}}H\rVert_F^2-\operatorname{tr}(P_{\mathcal{A}}\Omega_n)}{n}+O(n^{-3/2}),

Rs(n)−Rd(n)=∥PAH∥F2−tr⁡(PAΩn)n+O(n−3/2).R_s(n)-R_d(n) = \frac{\lVert P_{\mathcal{A}}H\rVert_F^2-\operatorname{tr}(P_{\mathcal{A}}\Omega_n)}{n}+O(n^{-3/2}).

Here Rs(n)R_s(n) and Rd(n)R_d(n) are the expected squared Frobenius losses of the full-sample fits, PAP_{\mathcal{A}} projects onto Dual-only tangent directions, and Ωn\Omega_n is the covariance of one vectorized query contribution. Thus, nonlinear projection bias and subspace estimation errors enter only the higher-order O(n−3/2)O(n^{-3/2}) remainder, while E[Γ^CARS]\mathbb{E}[\widehat{\Gamma}_{\mathrm{CARS}}] matches the true risk gap at the leading O(n−1)O(n^{-1}) order.

To test when small nn or small spectral cutoffs challenge this local approximation, we added a 432-setting sensitivity study (Appendix I, Table 6) varying sample size (n∈64,256,1024,4096n \in \\{64, 256, 1024, 4096\\}), rank-cutoff spectral gap (g∈0.15,0.30,1.00g \in \\{0.15, 0.30, 1.00\\}), directional variance allocation, and heavy-tailed/contaminated noise (Student-t5t_5 and Gaussian mixtures). Even at n=64n=64, the mean CARS score matches the true finite-sample nonlinear risk ordering in 29–30 of 36 settings across all spectral gaps. As a wider spectral gap stabilizes the estimated subspace (g=0.15→0.30→1.00g = 0.15 \to 0.30 \to 1.00), the sample size required for 36/36 agreement drops from n=4096n=4096 to 10241024 to 256256. This also aligns with our low-sample real-embedding experiments (n∈32,64,128,256n \in \\{32, 64, 128, 256\\} in Table 2), where CARS achieves the highest overall NDCG@10 and MRR@10 win rates.

You can find more details in the revised manuscript (Section 3.2 and Appendices C.1–C.2, and I).

Thank you again for your comments and for helping us improve the manuscript.

Thank you again for your time and for carefully reviewing our paper. As the discussion deadline approaches, we would appreciate it if you could let us know whether the revision addresses your concerns (or partially). If you are satisfied with our response, we would be grateful if you would consider updating your rating. Thank you!

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Changes in the Revised Manuscript

We thank all three reviewers for their time and constructive feedback. Their comments helped us refine the theoretical analysis of CARS, strengthen its empirical evaluation, and clarify the practical implications of the Shared–Dual trade-off. Below, we summarize the main changes, which are marked in blue in the revised PDF (for updated tables, only the captions are marked in blue for readability). For detailed responses to individual comments, please see our replies to each review.

  • We clarified the assumptions behind CARS and proved that, under the stated local conditions, its expected score matches the risk gap between the nonlinear Shared and Dual fits to first order, without requiring Gaussian or isotropic noise.

  • We added a simulation to examine how this agreement varies with sample size, spectral gap, and noise.

  • We added full-corpus retrieval experiments on FEVER, NQ, and MS MARCO. In the reported comparison against Always Shared, Always Dual, and validation-based selection, CARS achieves the highest first-place rates on both test NDCG@10 and MRR@10.

  • We expanded the operator-risk evaluation to 4,200 configurations across five datasets and six encoders, adding more models with different sizes like BGE-small (33.4M) and BGE-large (335M), as well as a validation-based baseline.

  • We added geometric and numerical examples to explain the Shared–Dual trade-off and practical guidance on choosing between the two.

AuthorsOpenReview ↗

Summary of Rebuttal

We would like to thank all Reviewers, ACs, and PCs for their time and effort in reviewing our paper. Below, we briefly summarize our rebuttal (corresponding revisions in the manuscript are highlighted in blue):

    1. Retrieval evaluation (Reviewers u3T1, mQXP): We added full-corpus retrieval comparisons on FEVER, NQ, and MS MARCO against Always Shared, Always Dual, and a validation-split baseline across 360 settings. CARS achieves the highest overall win rates on both test NDCG@10 and MRR@10.
    1. Theory of Gaussian and isotropic noise (Reviewers u3T1, mQXP): We proved that for the nonlinear spectral fits used in practice, the expected CARS score matches the Shared–Dual operator-risk gap to first order without assuming Gaussian or isotropic noise. We also verified this boundary in a new simulation under heavy-tailed and anisotropic noise across varying spectral gaps.
    1. Broader encoder scales (Reviewer mQXP): We expanded the operator-risk evaluation to six encoders ranging from 33.4M to 335M parameters, where CARS reaches 90.0% selection accuracy and outperforms validation selection in 142 of 150 folds.
    1. Intuition and practical guidelines (Reviewer mAwG): We added examples and practical guidelines to clarify the intuition behind our framework and when the theoretical boundary applies.

Reviewer mAwG has confirmed that our revision addressed all of their concerns and kept their accept recommendation. As the discussion deadline approaches, Reviewers u3T1 and mQXP have not yet responded; while we fully understand they have busy schedules, we want to note that we have directly carried out the proofs and experiments they requested. We sincerely thank the ACs and PCs for taking these updates into consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.