acceptodds
Under review as a conference paper at ICLR 2027

FREEZER: Extracting More from Frozen Representations

Abstract

A pretrained encoder may retain useful task information that its scoring rule does not fully use for the population being evaluated. We introduce FREEZER (Feature Reweighting and Embedding Enhancement with Zero Encoder Retraining), a post-hoc framework that uses dataset-level embedding statistics and supervision, when available, to adapt scoring while keeping the encoder fixed. For speaker verification, it retains cosine similarity after centering and covariance-based reweighting. For speech and image deepfake detection, it fits a regularized scoring direction and optionally blends the original detector score. On Common Voice 26, macro equal error rate (EER) across 51 locales falls from 3.03% to 2.17%, with improvements in 49 locales. A separate collective face-verification procedure reduces ArcFace IJB-C EER from 1.20% to 0.95%, using unlabeled evaluation embeddings and neighboring comparisons. Speech deepfake detection EER falls from 20.79% to 10.05% on the full CoSG release, with adaptation using only separate MLAAD and In-the-Wild data. Image deepfake detection with AIDE reduces EER from 10.29% to 7.53% on the CNN benchmark. Component studies show that supervised centering supplies most of the speaker-verification gain, while a fitted direction preserves most of the speech deepfake detection gain without blending. For image deepfake detection, blending helps some populations but hurts others. These findings demonstrate the value of adapting the scoring stage of frozen systems, while showing why gains must be evaluated across populations and operating points.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Frozen representations, post-hoc adaptation, feature reweighting, embedding geometry, metric learning, speaker verification, face verification, deepfake detection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_W49bOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how much performance can be recovered by adapting the scoring stage of a frozen encoder. FREEZER examines task-specific procedures based on established statistical operations: centering and covariance reweighting followed by cosine similarity for verification, and regularized readouts with optional original-score blending for deepfake detection. A separate collective verification procedure uses unlabeled evaluation embeddings and neighboring comparisons. On Common Voice 26, speaker-verification macro EER decreases from 3.03% to 2.17%, with improvements in 49 of 51 locales. For speech deepfake detection, adaptation using separate MLAAD and In-the-Wild data reduces EER on the full CoSG release from 20.79% to 10.05%.

The main contribution is an empirical investigation of which scoring adjustments produce these gains and how their benefits transfer. Component studies attribute most of the speaker-verification improvement to centering and show that the fitted speech-detection readout preserves nearly all the improvement without score blending. Image experiments demonstrate population-dependent blending tradeoffs. The paper also examines source composition, extraction procedures, comparison-graph access, and low-false-positive operating points, documenting cases where improved EER does not translate into better performance at the operating point of interest.

Critical strengths
  1. The component studies provide meaningful evidence about what actually supplies the improvements. I found the centering result particularly striking: optimized centering alone reaches 2.2045% EER on Common Voice, close to the selected full transformation’s 2.1678%. This is a useful finding for anyone deciding how much adaptation machinery is necessary. The paper separates ordinary mean subtraction, supervised centering, and covariance adaptation, and uses direction-randomization controls to investigate whether fitted directions matter. For speech detection, head-only scoring retains almost all the CoSG improvement, achieving 10.28% EER compared with 10.05% for the blend and 20.79% for the original detector. Shared-validation comparisons between fitting populations and readout families further help distinguish adaptation-data effects from scoring choices. These experiments give the paper value as an analysis of frozen-representation adaptation.

  2. The failure analysis and transparent reporting strengthen the empirical contribution. I particularly appreciated the CPLFW example, where a better EER accompanies a substantial deterioration at low FAR. It makes the limitations of the aggregate metric concrete. The authors investigate the extreme impostor score affecting the threshold and evaluate corrections that mitigate the problem while retaining important limitations. They also distinguish independent pair scoring from collective verification with additional evaluation-population access, report population-dependent failures of image-score blending, and separate extraction-related gains from head adaptation. The disclosure of exploratory follow-ups, threshold-transfer failures, and conditional uncertainty estimates helps the reader interpret the positive results. This is considerably more informative than reporting favorable benchmark averages alone.

Critical weaknesses
  1. The speaker-verification comparison omits a particularly relevant discriminative backend, leaving the practical value of the cosine-preserving choice incompletely established. Cumani et al., Analysis of the ABC Classification Backends for NIST SRE24 (Interspeech 2025), report strong PSVM results under mismatch. FREEZER cites this work but does not evaluate a PSVM backend using its frozen embeddings and available training identities. The moments-based Gaussian control in Appendix B is useful, but it does not cover this discriminatively trained alternative. Since FREEZER’s optimized center already uses supervised genuine/impostor pairs, a pairwise discriminative backend is directly relevant to assessing how effectively the same supervision can be used without changing the encoder. Preserving cosine is an attractive design choice; what I am missing is a matched comparison showing its accuracy and computational tradeoff against this alternative. An embedding-only PSVM would provide a suitable starting point, with any metadata-dependent variants reported separately. I do not assume that PSVM will outperform FREEZER on Common Voice; the comparison would resolve that empirical question.

  2. The methodological contribution is limited, and the incremental value of the more elaborate scoring choices remains insufficiently resolved. Centering, covariance normalization, ridge/LDA readouts, score blending, and ITQ are established components, as the manuscript acknowledges. Along the reported ablation sequence, supervised centering supplies 95.7% of the raw-to-final speaker EER reduction. In Appendix H, the image Gaussian-family control selects shared-covariance LDA and reproduces the ridge result under the matched absolute-EER selection objective, consistent with the equivalence derived in Appendix A. I find the simplicity of the effective adaptations appealing. However, these results make it especially important to establish when the additional components offer a reliable benefit. The covariance increment beyond the optimized center is small, while blending can produce substantial gains or regressions depending on the evaluation population. The paper documents these outcomes carefully, but provides limited guidance for anticipating which choice will transfer successfully. I recognize the empirical contribution across detection and collective verification independently of the missing speaker backend comparison. Taken together, however, the limited methodological novelty and unresolved comparative and incremental benefits leave this a borderline submission for me.

Questions for authors
  1. How does supervised centering plus covariance adaptation compare with an embedding-only PSVM under matched conditions? Please use the same frozen ReDimNet2 features, available training identities, locale information, development data, and strict-test trials, with justified tuning budgets. Any use of additional metadata, such as duration, should be identified and controlled separately. Reporting fitting and scoring costs alongside EER and low-FAR performance would clarify whether FREEZER offers an accuracy advantage or a compelling efficiency tradeoff. This comparison would materially inform my assessment.

  2. How robust is the incremental benefit of covariance adaptation beyond the optimized center? I appreciate that the manuscript explicitly describes the additional value of correct identity grouping as unresolved. The strict-test EER difference between center-only and the selected full transformation is approximately 0.037 percentage points, and total or shuffled-label covariance approximately ties the selected covariance once the supervised center is fixed. The uncertainty evidence for improvement over raw cosine does not establish the reliability of this smaller increment. Could you provide paired uncertainty for center-plus-metric versus center-only, preferably including refitting variability, and indicate whether the benefit is consistent across locales or operating points? This would clarify when the additional metric estimation is worthwhile.

  3. Does the speech readout comparison in Table 2 remain stable under family-specific regularization tuning? Appendix B.5 notes that identical absolute penalties can have different effective strengths across losses. The existing comparisons control several important factors, but matching candidate counts alone does not establish that the search ranges are equally suitable for ridge and logistic regression. Would expanded or independently justified regularization ranges preserve the observed ridge advantage? For images, the manuscript already shows that the relative ordering varies across destinations; my question concerns the robustness of the speech result and the extent to which it supports a preference for ridge in that setting.

1 comment

Thank you for your detailed suggestions. We’ve added the three comparisons you requested to the manuscript.

Q1. How does PSVM compare with adapted cosine?

When trained separately by locale, diagonal PSVM and adapted cosine performed similarly (macro EER 2.174% and 2.168%). Combining a shared PSVM with a locale-specific PSVM lowered macro EER to 1.877% and improved on adapted cosine in 50 of 51 locales. At one false accept per 10,000 impostor trials, adapted cosine accepted more genuine trials (67.58% versus 61.48%).

Both used the same frozen ReDimNet2 embeddings, training identities and 41.3 million test trials. Development macro EER selected each configuration. Locale was the only metadata. On three locales, fitting one adapted-cosine configuration took 2.54–2.74 seconds per locale on an H100. Across the full PSVM search, local fits used 2.75 GPU-hours and pooled fits took 30 minutes. Scoring the cached trials took 0.304 seconds for adapted cosine and 0.325 for PSVM on the same H100 (Table 2 and Appendix I).

Q2. How much does covariance add beyond centering?

Covariance reweighting lowered macro EER from 2.205% to 2.168%. EER fell in 34 locales, tied in three and rose in 14. With fits fixed, the gain was 0.037 percentage points (paired 95% interval 0.019–0.073). Resampling both training and evaluation speakers across 100 refits gave a mean gain of 0.024 points (0.003–0.056). At FAR 10⁻⁴, the interval for the gain in genuine accepts included zero. Shuffling speaker labels gave similar EER, leaving the benefit of correct speaker grouping uncertain (Section 5.1 and Appendix I).

Q3. Does ridge still lead after separate tuning?

Yes. Ridge had lower EER on all four speech datasets. We used the same validation folds and blend-selection rule for both methods. We scaled the logistic penalty grid by ¼ to account for its curvature and widened either grid if validation favored an endpoint. On full CoSG, ridge reached 10.05% EER and logistic regression 11.78%. Separate tuning narrowed the gap from 2.09 to 1.73 points (paired 95% interval 0.81–2.81). Table 3 and Appendix B.5 give the full comparison.

Compared with the fitted head alone, blending in the original CLIP detector’s score lowered EER on CNN and Diffusion but raised it on Synthbuster (Section 5.2).

Reviewer_7VBSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes FREEZER, an adaptation framework deployed at the inference stage. FREEZER freezes the pre-trained encoder and applies only post-processing to embeddings, employing this unified processing across four scenarios. For verification tasks, it first estimates centers using dataset statistics, then performs whitening using OAS shrinkage covariance, and finally applies cosine similarity on the transformed embeddings. For detection tasks, it fits a task-relevant scoring direction using labels or pseudo-labels provided by a teacher model, which can be combined with the output of the original detector. No retraining or fine-tuning of the encoder is required at any stage. The authors argue that the original scoring mechanism does not sufficiently leverage the information already present in the frozen embeddings.

Critical strengths

This paper conducts extensive validation. FREEZER was tested on four substantially different tasks: speaker verification, face verification, voice deepfake detection, and image deepfake detection, achieving favorable experimental results.

The paper provides detailed experimental details and reports failure cases, demonstrating relatively rigorous methodology.

Critical weaknesses

The paper lacks substantive novelty at the methodological level. Among the three components of FREEZER, centralization is a basic step of classical discriminant analysis; shrinkage of the covariance matrix is already well-established; fitting the linear scoring direction is merely a variant of Fisher's discriminant direction.

The mathematical exposition in the paper is imprecise. The notation system is self-inconsistent—the relationships among MM, CC, and Σ\Sigma are never defined throughout; the optimization problem for class-averaged loss is only described in prose without a formal mathematical expression; notational conventions regarding upper and lower case are unstable, and so forth. These issues are extremely unfriendly to readers.

Questions for authors

Given the current presentation of the paper, no response would improve my score.

1 comment

Thank you for your valuable suggestions. We have updated the manuscript based on your suggestions. While individual statistical methods are established, our contribution is the evidence about which components help frozen systems and how their benefits change across populations and operating points. We compare what these established scoring rules achieve with the same frozen representations and identify which gains carry across populations and operating points. For example, centering alone reaches 2.20% speaker EER, near the full cosine map at 2.17%. A directly learned pair score reaches 1.88%, yet the cosine map accepts more genuine trials at a false-accept rate of 10⁻⁴. These results offer different choices for applications with different error costs.

We revised Section 3 and Appendices A, B and G to address the notation. An input aᵢ maps to an embedding xᵢ and then to its unit vector zᵢ. C denotes an estimated covariance or second moment. Equation 1 scores pairs after applying a map A, with metric M = AᵀA. Equation 2 illustrates additive regularization of C. For CV26 we shrink the within-speaker covariance and floor its eigenvalues before taking the inverse power (Appendix B). Σ denotes the shared population covariance in the Gaussian detection model. In regularized LDA, the within-class estimate S_W supplies C and gives the metric (S_W + λI)⁻¹. Ridge uses the total covariance S_T. Appendix A derives when the resulting directions give the same ranking.

The manuscript now displays the objectives that use labels. Equation 11 gives genuine-pair and impostor-pair losses equal weight when fitting the speaker center and threshold h. Detector fitting uses a separate intercept b. For ridge, each source–class group contributes 1/G of the squared-error loss, and each of its nₖ examples receives weight 1/(G nₖ). The penalty applies to the direction w. Appendix B writes the double sum explicitly. We use φ for a candidate scoring rule in validation and reserve h for the verification threshold. Appendix G uses the same projection and angular scores as Section 3.

Reviewer_pwX2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FREEZER is a post-hoc framework that adapts scoring using dataset-level embedding statistics and, when available, supervision, while keeping the encoder frozen without retraining or fine-tuning it. It addresses the problem that existing scoring rules may not fully exploit useful information in the representations for the target evaluation population. Retraining or fine-tuning the encoder requires additional computational resources and suitable training data, and may also hurt generalization.

FREEZER adapts the scoring stage to better exploit existing representations. For verification, it applies centering and covariance reweighting before cosine comparison. For deepfake detection, it fits a regularized scoring direction and optionally blends the original detector score.

Experiments show significant EER reductions across datasets. For example, CV26 speaker verification EER drops from 3.03% to 2.17%, CoSG speech deepfake from 20.79% to 10.05%, and AIDE on CNN image deepfake from 10.29% to 7.53%.

Critical strengths
  1. High practical value: The paper demonstrates that lightweight post-processing (e.g., centering and covariance reweighting) can significantly improve verification and deepfake detection without retraining heavy encoders. This offers a highly cost-effective and plug-and-play solution for practical deployment.

  2. Honest and thorough empirical validation: The authors transparently report negative transfer results, component ablations, and potential confounders (such as the effect of five-crop preprocessing), providing valuable and reliable empirical insights for the community.

Critical weaknesses
  1. The methodological novelty could be more clearly established. The main components, including centering, covariance-based reweighting, ridge/LDA-related scoring directions, and score blending, are established techniques. The paper also discusses closely related work on frozen-feature and Gaussian-discriminant heads. Although the cross-task evaluation and component analysis may be valuable contributions, the manuscript could more clearly distinguish its methodological contributions from existing approaches and explain which findings are genuinely new.
  2. The source and generalizability of the reported gains need clearer explanation. FREEZER does not improve performance in every target population, and some adaptations even lead to worse results. Moreover, not all reported gains come from FREEZER itself. In the CLIP/CNN experiment, five-crop preprocessing accounts for 2.510 of the 3.505-point EER reduction, while the fitted head contributes 0.995 points. The collective face-verification experiment also uses unlabeled evaluation embeddings and information from neighboring trials, unlike standard independent pair scoring. The paper should clearly distinguish gains from FREEZER itself from those due to preprocessing or additional evaluation-set information, and clarify how well the results generalize to other settings.
Questions for authors

Q1 (Clarification on isolating FREEZER's true contribution): The paper attributes significant gains to FREEZER, but some of these gains are confounded by other factors. Specifically, in the CLIP/CNN experiment, five-crop preprocessing accounts for the majority of the EER reduction (2.510 of the 3.505 points). Additionally, the collective face-verification uses unlabeled evaluation embeddings and neighboring trial graphs, which provides extra information not available in standard independent pair scoring. To strictly isolate the true contribution of FREEZER itself, could the authors provide: (1) a strictly matched baseline for the image tasks where the original detector also uses the identical five-crop preprocessing, and (2) the face verification performance under strict independent pair scoring (i.e., without accessing the evaluation graph)?

1 comment

Thank you for the detailed suggestions. We have updated the manuscript based on your suggestions. On CNN13, the fitted CLIP score still improves EER with the same five crops. On IJB-C, the strict source-fitted face score slightly worsens EER for both encoders. We now give these comparisons in the main text. While individual statistical methods are established, our contribution is the evidence about which components help frozen systems and how their benefits change across populations and operating points. The new checks quantify uncertainty for the image results and verify that the source-fitted face score can be computed from a single pair.

Q1(1). Identical preprocessing for image detection

On the full CNN benchmark, the original CLIP detector gives 11.063% EER with one center crop and 8.553% with five crops. Applying the fitted score to the same five-crop features and evaluation images gives 7.558%. Of the 3.505-point reduction along this sequence, 2.510 points come from extraction and 0.995 from scoring. Figure 6 and Appendix C contained these comparisons at submission. Section 5.3 now gives all three values together.

We replayed the scores and added 500 paired image-level resamples. The 95% interval for the five-crop scoring gain is 0.672–1.338 EER points on CNN13 and 1.276–2.205 on Diffusion8. For AIDE, the original and adapted scores both use the released frontend. On CNN13 the change is 10.294% to 7.530% (gain interval 2.164–3.098 points). On Diffusion8 it is 2.713% to 2.563%, with an interval from −0.169 to +0.519 points. The intervals hold the fitted heads and generator sets fixed. On that evidence, the scoring gain for AIDE is clearer on CNN13 than on Diffusion8.

Q1(2). Face verification when each pair is scored independently

On the official IJB-C trials, raw ArcFace cosine gives 1.1965% EER. A transform fitted on VGGFace2 and applied to two templates in isolation gives 1.2170%. Fitting the transform on unlabeled IJB-C templates brings EER to 1.1147%. Using neighboring trial scores as well brings it to 0.9511%. AdaFace follows the same sequence, at 1.4828%, 1.5391%, 1.4624% and 1.3243%. Thus both source-fitted transforms slightly worsen independent EER on IJB-C. The headline improvements apply to the collective setting.

The access-controlled comparisons appeared in Appendix D.4 at submission and are now in Section 5.1. We also checked the independent scorer by evaluating pairs singly and in batches of eight and 64. Scores agreed across batch sizes on all four face benchmarks. The largest absolute difference from the archived independent scores was 2.65 × 10⁻⁸.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.