acceptodds
Under review as a conference paper at ICLR 2027

DiMTE: Directed Model-Transfer Estimation for Wireless Datasets

Abstract

Many data decisions depend on how well a model trained on one dataset would perform on another, such as which dataset to train on or which to add to scarce labels. Measuring this usually means training and testing one model per dataset. Dataset similarity measures are cheaper, but they are often symmetric and none returns the downstream loss. We introduce DiMTE, a directed transfer-risk estimator built from source-induced proxy predictors. DiMTE embeds every sample with a shared frozen encoder, such as a foundation model, fits a lightweight class-conditional proxy to each source, and evaluates it on labeled target samples under the deployment loss. It thus estimates how a model trained on each source would perform on the target, in the deployment metric. We prove a uniform bound that separates finite-sample error from the mismatch between proxy and deployment model. Across city-disjoint wireless benchmarks, including cities ray traced with an independent simulator after the method was frozen, DiMTE ranks sources best among methods that train no deployment model per source, and remains useful with few target labels. Estimators that inspect a trained model per source can be stronger, but only once those models exist. The same estimator also ranks camera-trap locations and source languages, with no wireless prior. DiMTE therefore lets a practitioner rank a whole library of datasets in the deployment metric before any per-source model is trained.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
transferability estimation, source dataset selection, directed transfer, dataset similarity, distribution shift, foundation models, wireless communications

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_p7SiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DiMTE, a lightweight estimator for ranking source datasets by the performance that models trained on them would achieve on a labeled target domain. It uses a shared frozen encoder and source-specific class-conditional mixture classifiers, evaluating their predictions under the downstream task metric without training a deployment model for each candidate source. The contributions include a directed proxy-risk formulation, a compact prototype-based implementation, and an error bound separating target sampling error from proxy–deployment mismatch. Evaluation focuses on LOS/NLOS identification and beam prediction using ray-traced wireless datasets, including city-disjoint splits and an independently generated Sionna RT test set, with additional camera-trap and multilingual classification experiments. Results support improved source-ranking quality at low computational cost, while gains in source selection and downstream training are less consistent.

Critical strengths
  1. DiMTE addresses a practical data-selection problem by ranking candidate source datasets without training a deployment model on each one. Its directed, task-specific scores provide a useful alternative to dataset similarity, with substantially lower computational cost than exhaustive training.
  2. The evaluation includes city-disjoint splits, multiple training seeds, and an independent Sionna RT test set generated after the method was frozen. These controls strengthen the credibility of the ranking results and reduce concerns about environment leakage and adaptation to the original benchmark.
Critical weaknesses
  1. The connection between ranking quality and downstream utility remains insufficiently established. On DeepMIMO, improved rankings do not clearly improve the top-source selection over the strongest comparator. The exploratory data-addition experiments show uncertain comparative gains for beam prediction and negative transfer for LOS/NLOS identification. This limits the evidence that better rankings translate into better training decisions.
  2. The wireless evaluation remains restricted to ray-traced channels with a fixed carrier frequency and antenna configuration. It does not establish robustness on measured, wideband Massive MIMO-OFDM channels with estimation noise, calibration errors, and temporal variation. These factors could change the agreement between the lightweight proxy and deployment models, directly affecting the method’s central assumption.
Questions for authors
  1. Clarify the substantive methodological distinction between DiMTE and using a lightweight source-trained classifier’s target performance as a proxy for deployment performance.
  2. Clarify the reliability of DiMTE for each specified deployment model, rather than their average, including cases where proxy–deployment disagreement changes source selection.
  3. Identify the reported evidence of statistically significant downstream gains over the strongest comparator, distinguishing ranking improvements from source-selection and data-addition benefits.
  4. Specify the bandwidth, subcarrier spacing, and frequency selectivity represented by the 32-subcarrier inputs, and delimit the conclusions applicable to wideband Massive MIMO-OFDM.
  5. Clarify whether existing experiments establish robustness to channel-estimation noise, array calibration errors, or temporal variation. Distinguish demonstrated robustness from assumptions requiring future validation.
  6. Clarify when source selection outperforms target-only training or reuse of an existing model under the same target-label budget, accounting for feature extraction and historical tuning costs.
3 comments
AuthorsOpenReview ↗

Response to Reviewer p7Si (1/2)

Dear Reviewer p7Si,

Thank you for your expert review. As a quick reminder: DiMTE ranks candidate source datasets for a new target by predicting each source's downstream error, without training a model per source. Your suggestions directly motivated our new experiments on measured arrays, wideband 100 MHz, 28 GHz carriers, hardware impairments, and temporal drift. (These are now in Appendix T; G1–G5 refer to the General Response, and our live demo is at https://huggingface.co/spaces/userrr1323/DiMTE).

Q1. Distinction from a generic source-trained classifier

Concern: How is this different from just scoring a target on a standard source classifier?

Response: Clarification. As detailed in G1, standard classifiers rank sources worse. DiMTE works specifically because of three connected choices:

  1. Local density proxy in the frozen LWM space: It beats per-source linear probes on the same LWM features by 0.09–0.10 correlation (ρ\rho) on Sionna, and the LWM space itself matters (on normalized raw channels the LOS/NLOS ranking falls from 0.78 to 0.21).
  2. Deployment-risk units: Using unweighted log-likelihoods (like LEEP/NCE) ignores the physical severity of errors (like dB loss) and drops ρ\rho by 0.22–0.28 on Sionna beams.
  3. Directed evaluation (s→ts \to t): If we symmetrize the metric to treat the source and target equally, it ignores the direction of transfer and drops ρ\rho by 0.11–0.23 on Sionna.

Q2. Reliability across different deployment models (and disagreement cases)

Concern: Does DiMTE work reliably across different architectures, and what happens when the models disagree on the top source?

Response: Clarification, with a new analysis. DiMTE reliably predicts rankings for both raw-CSI CNNs and fine-tuned LWMs (leading in 7 of 8 test settings and tying our LEEP adaptation in the eighth, 0.64 vs. 0.66, CI [−0.04, +0.00]; Table 24). We added a new analysis on model disagreement. The CNN and LWM do disagree on the #1 best source for 61–90% of targets. However, these disagreements are mostly near-ties. Using one model's top source costs the other a median of only 6–14% of the gap between a random and the best source. As a result, DiMTE's top source closes nearly the same share of that random-to-best gap whether the models agree or disagree on the absolute top source:

Ground-truth model / MetricDeepMIMO Bin.DeepMIMO 32-Cl.Sionna Bin.Sionna 32-Cl.
Raw-CSI CNN (ρ\rho, DiMTE / best baseline)0.85 / 0.710.91 / 0.770.78 / 0.740.78 / 0.67
Fine-tuned LWM (ρ\rho, DiMTE / best baseline)0.79 / 0.690.86 / 0.730.64 / 0.660.76 / 0.65
Sionna random-to-best gap closed: Raw CNN (agree / disagree)90% / 88%72% / 78%
Sionna random-to-best gap closed: FT-LWM (agree / disagree)82% / 83%67% / 71%
AuthorsOpenReview ↗

Response to Reviewer p7Si (2/2)

W1 / Q3. Statistically resolved downstream gains

Concern: Are the downstream practical improvements statistically significant?

Response: Clarification; all results below are in the submitted paper. Yes. Using environment-clustered permutation tests, DiMTE's gains are statistically significant at every pipeline stage:

  • Source ranking: DiMTE beats all 48–54 baselines that train no model per source, on both test sets and tasks (p≤0.003p \le 0.003). On DeepMIMO, it also beats 12 model-aware estimators computed on already-trained models (p<0.001p < 0.001).
  • Zero-shot selection: On DeepMIMO, reusing the top-1 source without fine-tuning significantly lowers binary F1 error at 10, 25, and 50 labels (0.067, 0.059, 0.053 vs. 0.085, 0.067, 0.067, p≤0.048p \le 0.048) and 32-class beam loss at 25 and 50 labels (5.7, 5.6 vs. 6.5, 6.4 dB, p≤0.028p \le 0.028). Without fine-tuning, the choice of source matters most.
  • Reuse-or-train decisions: Automatically deciding to reuse a source vs. train locally beats the strongest competing score (8.8 vs. 9.5 dB, p=0.002p=0.002), and still does when the local error is estimated by cross-validation (9.3 vs. 9.9 dB, p=0.009p=0.009).
  • Choosing and adding data: Training on DiMTE's top-3 sources beats the class-prior KL baseline (8.7 vs. 12.0 dB, p=0.022p=0.022). Augmenting local site labels with these sources significantly reduces the loss (5.9 vs. 7.5 dB, p<0.01p < 0.01); its lead over adding class-prior KL's top 3 (8.2 dB) is not significant (p=0.052p=0.052).

W2 / Q4 / Q5. Narrowband specs vs. real-world physical robustness

Concern: The paper relies heavily on simulated 3.5 GHz narrowband data. How well does this generalize?

Response: Agree; revised. We have updated Section 5.1 to explicitly state that our base simulated inputs are 32-antenna ULA channels at 3.5 GHz whose 32 subcarriers carry the same narrowband channel, so the inputs have no frequency selectivity. To address generalization, we ran extensive new tests (G5 / Appendices T.3–T.7). DiMTE leads the strongest comparator on all of the following, with a significant lead on all but the last:

  • Frequency-selective channels, 0.96 MHz and 100 MHz NR (p≤0.002p \le 0.002)
  • 28 GHz carriers (p<0.001p<0.001)
  • 10 dB SNR with antenna gain/phase errors (1 dB, 15∘15^\circ) (p≤0.021p \le 0.021)
  • Real measured DICHASUS arrays (7 environments, p=0.046p=0.046)
  • Temporal drift across 16-minute recordings (0.59 vs. 0.51; 95% CI excludes zero; p=0.062p=0.062)

Open limitation: multi-day or seasonal drift beyond our 16-minute recordings remains open for future field tests.

Q6. Strategy based on label budget and compute cost

Concern: How does DiMTE's optimal usage change based on the available label budget, and what is the compute overhead?

Response: Clarification, with new results. DiMTE naturally adapts to the available target labels (G3–G4):

  • No labels for training (100 labels used only to rank): Train on DiMTE's top-3 sources. This beats pooling all data (0.032 vs. 0.047 F1 error, p≤0.003p \le 0.003) and class-prior KL's top 3 on beams (8.7 vs. 12.0 dB, p=0.022p=0.022).
  • 10 labels: Fine-tune DiMTE's top source. This beats adapting the LWM strictly on the target data.
  • 25–50 labels: On beams, fine-tuning DiMTE's top source still leads (G3). DiMTE's predicted errors also decide, per site, whether to reuse the source model unchanged or train locally (G4).
  • 100 labels: Use DiMTE for targeted data augmentation, which cuts error further (7.5 →\to 5.9 dB, p<0.01p<0.01).

Compute overhead: Extracting LWM embeddings takes 0.11 s per 1,000 samples on a GPU. Scoring takes seconds on a CPU (0.25–3.4 s). The one-time transfer matrix takes 1.4–3.6 GPU-hours, about as much as training fine-tuned LWM on every dataset of our library (1.3–5.6 GPU-hours), and it does not grow with the number of targets (see our reply to epKP, Q4).

Thank you again for your time and feedback; please let us know if further details would help during discussion.

AuthorsOpenReview ↗

Following up on our response

Dear Reviewer,

As the response deadline approaches, please let us know if there are any points we can clarify. If our response has addressed your concerns, we would appreciate hearing whether you would consider raising your score.

Thank you again for your time and feedback.

Best regards,

The Authors

Reviewer_Rg3tOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tackles source selection for reusing wireless datasets and proposes DiMTE, a directed transfer-risk estimator. It extracts features with a frozen encoder, builds a lightweight proxy classifier for each source dataset, and estimates the task loss on the target domain using labeled target samples without training candidate deployment models. Experiments on DeepMIMO and Sionna validate its good source rankings.

Critical strengths
  1. Estimating transfer before training, in a directed way and in task-metric units, is a practically useful problem for wireless dataset reuse.

  2. Using a lightweight proxy on frozen foundation-model features is an efficient solution.

  3. The experimental evaluation is extensive, with detailed discussions on the framework's generalization capability across different domains.

Critical weaknesses
  1. The method itself is not very novel. It basically uses a cheap source classifier’s loss on the target as a proxy, which is close to frozen-feature probes and Selection via Proxy, and directedness and task units come naturally from that. Beyond the problem setup, what is the methodological contribution?

  2. If a few target labels are available, why not just fine-tune the pretrained LWM on them? The target-only baseline is a CNN trained from scratch, which is weak, and it still beats source selection on LOS/NLOS sometimes. A comparison with fine-tuning LWM directly would show whether source selection is needed at all.

  3. All wireless experiments use simulated 3.5 GHz channel data under a limited set of antenna configurations, and LWM was also pretrained only on simulated channels. Given how much DiMTE depends on the encoder, could the authors test it on measured data, or at least on simulations with hardware impairments?

  4. The transfer matrix used to tune hyperparameters costs 1.4 GPU-hours, more than training a CNN for all 236 sources. Although the calibration is a one-off cost and a 1-NN variant that needs no new transfer matrix is also evaluated, it remains unclear whether the hyperparameters carry over across frequency bands, antenna arrays, or propagation conditions. Could the authors evaluate this and report the amortized end-to-end cost?

  5. Better ranking doesn’t necessarily mean better decisions. In Table 3, adding any source data on LOS/NLOS, including DiMTE’s picks, hurts performance. Can the estimator tell when adding source data will hurt?

Questions for authors

Please refer to the weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Rg3t (1/2)

Dear Reviewer Rg3t,

Thank you for your detailed review. As a quick reminder of our work: DiMTE is a method that ranks how well source datasets will transfer to a new target without training new models. It embeds the data into a frozen wireless foundation model (LWM) and predicts the actual downstream error (like dB loss) in a directed way (s→ts \to t).

We have run all four of your suggested experiments (G1–G5 refer to the General Response) and added a live demo at https://huggingface.co/spaces/userrr1323/DiMTE.

W1. Methodological differences from prior proxy methods

Concern: How does DiMTE differ from "Selection via Proxy" or simple frozen-feature probes?

Response: Partly agree; revised. We have updated Section 2 to clarify these distinctions (and cited Selection via Proxy).

FeatureSelection via ProxyFrozen-feature probesDiMTE
GoalSelect examples within one datasetScore existing pretrained modelsRank candidate source datasets before training
Training requiredTrain a smaller proxy networkTrain a probe head per modelNo network training (a Gaussian mixture per class)
Output formatExample scores for subsettingRelative transferability scoreDirected target error in exact task units

Unlike Selection via Proxy, DiMTE does not require training any neural networks for the candidate sources. Unlike standard probes, DiMTE's specific proxy preserves the ranking of complex deployment models much better (beating probes by +0.07 to +0.18 correlation, p<0.001p<0.001). DiMTE also shows why the target's baseline difficulty cancels when sources are compared (G2), and its error predictions in task units support automated decisions such as reuse-or-train (G4).

W2. Fine-tuning the foundation model directly on scarce target labels

Concern: You should compare against fine-tuning the LWM strictly on the scarce target labels.

Response: Agree; new experiment. We have added this exact experiment (Appendix T.2; see G3). When only 10 labels are available, using DiMTE to select the best source and fine-tuning it strongly outperforms adapting the LWM on those 10 target labels alone (0.037 vs. 0.056 F1 error, p=0.007p=0.007; 10.3 vs. 15.0 dB error, p<0.001p<0.001). For the harder 32-class beam task, DiMTE's source transfer strategy stays ahead even up to 50 target labels (9.2 vs. 9.9 dB, p≤0.014p \le 0.014); with 100 labels, target-only LWM reaches 8.2 dB. On binary LOS/NLOS the two tie from 25 labels on. DiMTE selects data, not a model, so it works with the model a site deploys, LWM included (G3).

AuthorsOpenReview ↗

Response to Reviewer Rg3t (2/2)

W3. Measured data, hardware impairments, and alternative encoders

Concern: Does this method hold up under real-world noise, actual hardware arrays, or different foundation models?

Response: Agree; new experiments. Yes. We added tests to validate this (Appendices T.3–T.4; see G5):

  • Hardware Impairments: We replaced the inputs with noisy uplink (SRS) estimates at 10 dB SNR with per-antenna gain and phase errors (1 dB, 15∘15^\circ), and retrained the deployment models. DiMTE still leads every comparator (p≤0.021p \le 0.021).
  • Measured Arrays: On the real-world DICHASUS dataset (1.27, 3.44 GHz), DiMTE's LWM embeddings improve the ranking by 0.11–0.12 over using the raw sub-arrays (p=0.016p=0.016). Adding a pre-registered industrial hall makes DiMTE's measured beam lead significant (0.70 vs. 0.61, p=0.046p=0.046).
  • Encoder Generality: We tested alternative encoders. Using WavesFM embeddings, DiMTE leads all comparators with the same data access in 3 of 4 settings (two tasks on two test sets) and ties on the fourth (Section 6.2, Appendix I).

W4. Hyperparameter transfer and setup costs

Concern: Do hyperparameters need to be re-tuned for new environments, and is the setup cost practical?

Response: Agree; new experiments.

  • Hyperparameters: Settings calibrated once on 3.5 GHz DeepMIMO training cities transfer without retuning to Sionna, impaired data, 100 MHz wideband, 28 GHz, and real DICHASUS arrays. Only wideband inputs add a parameter-free readout and DiMTE's Parzen rule, chosen on development sites and confirmed by two pre-registered held-out city tests.
  • Compute Cost: Computing the transfer matrix is a one-time cost per task (1.4–3.6 GPU-hours) that does not grow with the number of targets. Training fine-tuned LWM on every dataset of our 236-dataset library costs about the same (1.3–5.6 GPU-hours), while a small CNN costs less (0.29–0.43), as you note. After the matrix, DiMTE needs no training for new targets or new library datasets (see our reply to epKP, Q4).

W5. Knowing when source data will hurt (Negative Transfer)

Concern: Can DiMTE tell us when adding a source will actually make performance worse?

Response: Yes. Because DiMTE predicts actual physical error (e.g., dB loss), we can compare its predicted source error directly against the measured error of local training (Table 40; G4). If DiMTE predicts higher error than training locally, we simply reject the source. On binary LOS/NLOS, where Table 3 showed that source data can hurt, this decision avoids that harm. On the 32-class task, it beats always reusing (9.7 dB) and always training (10.0 dB), and also the best competing score (8.8 vs. 9.5 dB at m=50m=50, p=0.002p=0.002).

Thank you again for your time and feedback; please let us know if further details would help during discussion.

AuthorsOpenReview ↗

Following up on our response

Dear Reviewer,

As the response deadline approaches, please let us know if there are any points we can clarify. If our response has addressed your concerns, we would appreciate hearing whether you would consider raising your score.

Thank you again for your time and feedback.

Best regards,

The Authors

Thank you for the rebuttal. My concerns have been addressed.

Reviewer_epKPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DiMTE, a method for estimating source-to-target transfer risk without the need to train a dedicated deployment model for every candidate source dataset. It employs a shared, frozen encoder to map both source and target data into a unified representation space. For each source dataset, a lightweight, class-conditional proxy classifier (gsg_s) is constructed; its deployment loss—calculated directly on labeled target samples—serves as a proxy for the transfer risk of the actual deployment model (hsh_s).

Critical strengths
  1. The problem of source dataset selection addressed in this paper is of practical significance, particularly in scenarios involving large-scale dataset libraries.
  2. The paper provides a clear distinction between symmetric dataset similarity, model-aware transferability estimation, and source-induced proxy risk.
  3. The experimental protocol is robust, incorporating a city-level split between training and testing cities, as well as the inclusion of additional test cities generated via the Sionna framework.
Critical weaknesses
  1. The core novelty of the proposed method is limited, it essentially boils down to "training a cheap classifier on the source and validating it on the target."
  2. A simple, fixed 1-NN rule—which requires no transfer matrix—performs within 0.03 of the tuned DiMTE across all test sets. This undermines the necessity of the complex tuning and mixture construction processes presented in the paper.
  3. The primary experimental setting utilizes 1,000 labeled target samples, which somewhat contradicts the paper's stated practical motivation.
Questions for authors
  1. What is the specific, unique contribution of DiMTE compared to the baseline approach of "training a cheap k-NN or probe on the source and validating it directly on the target labels"?
  2. Why should a lightweight proxy classifier preserve the same source-ranking order as a fundamentally different deployment model?
  3. Why does the primary experimental setting use 1,000 labeled target samples? How does the method perform against lightweight target adaptation techniques given this same labeling budget?
  4. When accounting for the cost of computing the training-city transfer matrix, how many new target datasets are required for DiMTE to effectively amortize this upfront overhead?
2 comments
AuthorsOpenReview ↗

Response to Reviewer epKP

Dear Reviewer epKP,

Thank you for your constructive review and for highlighting our conceptual distinctions and rigorous evaluation. As a quick reminder: DiMTE ranks candidate source datasets for a new target by predicting each source's downstream error, without training a model per source. We have added direct comparisons against per-source linear probes and target-only LWM adaptation. (Note: G1–G5 refer to the General Response above; our live demo is available at https://huggingface.co/spaces/userrr1323/DiMTE).

  • W1 / Q1 (Contribution beyond a cheap source classifier): Partly agree; clarification. We emphasize that the type of proxy is crucial. On raw channels normalized per user, the LOS/NLOS ranking drops from 0.78 to 0.21 (G1). Symmetric metrics ignore the direction of transfer, and likelihood scores (LEEP/NCE) ignore how severe an error is. Inside the LWM space, DiMTE's local density proxy outperforms per-source linear probes by up to +0.18 ρ\rho and provides calibrated outputs that allow for automated, threshold-free decision-making (G4).
  • W2 (Why a 1-NN rule is within 0.03 ρ\rho of default DiMTE): Partly agree; clarification. 1-NN is an excellent baseline because it operates inside our frozen LWM space (confirming our findings in G1 that local proxies in this space work well). However, 1-NN still needs one setting, the number kk of LWM principal components. Across k=8k = 8 to 128128, its ρ\rho ranges from 0.66 to 0.80 on LOS/NLOS and from 0.77 to 0.81 on beams (App. T.9), so kk must be chosen once on a training library. With the LOS/NLOS value k=32k=32, it stays within 0.03 of DiMTE on both tasks (Table 13). More importantly, our default mixture method compresses storage by 16×16\times (64 components vs. 1,000 embeddings per LOS/NLOS source) and outputs smooth probabilities used by the decisions outlined in G4.
  • Q2 (Why a weak proxy keeps deployment order): Clarification, with a new analysis. We added a mathematical analysis in G2. Because we are ranking sources for a specific target, the target's inherent baseline difficulty cancels out completely. The proxy only needs to predict the source-specific differences correctly. The frozen LWM space keeps the proxy's source-specific errors small (1.2–3.6 times smaller than on raw channels in 7 of 8 comparisons), so on beams it orders 96–99% of clearly separated source pairs correctly. No proxy ranks perfectly, and near-ties stay uncertain even between the two deployment models, so a site can train DiMTE's top few sources instead of one (G2).
  • W3 / Q3 (Why 1,000 target labels, and scarce-budget adaptation): Clarification on the labels; new baseline added. The 1,000 labels were strictly used to ensure stable evaluation metrics. With just m=10m=10 labels, selecting a source with DiMTE and fine-tuning it outperforms adapting the LWM on those 10 target labels alone (0.037 vs 0.056 F1 error, p=0.007p=0.007; 10.3 dB vs 15.0 dB, p<0.001p<0.001). DiMTE selects data, not a model, so it works with the model a site deploys, LWM included. See G3 for the full breakdown.
  • Q4 (Amortizing the training-city transfer matrix): Clarification. The training-city transfer matrix is a one-time cost per task (1.4 GPU-hours for LOS/NLOS, 3.6 for beams), and it does not grow with the number of targets. Model-aware estimators instead need a trained model for every source. For our 236-dataset library, this costs about the same with fine-tuned LWM (1.3 and 5.6 GPU-hours) and less with a small CNN (0.29 and 0.43), and every dataset added to the library needs one more trained model. After the matrix, DiMTE scores a new target in seconds on a CPU (0.25–3.4 s) without any training, and the 1-NN rule (W2) needs no matrix at all.

Thank you again for your time and feedback; please let us know if further details would help during discussion.

AuthorsOpenReview ↗

Following up on our response

Dear Reviewer,

As the response deadline approaches, please let us know if there are any points we can clarify. If our response has addressed your concerns, we would appreciate hearing whether you would consider raising your score.

Thank you again for your time and feedback.

Best regards,

The Authors

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response (1/4): Why the Choice of Proxy Matters

Dear Reviewers and Area Chair,

Thank you for your constructive feedback. As a quick reminder of our work: DiMTE is a method that predicts how well a model trained on a source dataset will perform on a new target dataset. Instead of using generic data similarity metrics, DiMTE predicts actual downstream error (like dB loss) using a frozen wireless foundation model (LWM).

What we did in this rebuttal. We answer every point with new experiments or revisions (Appendix T, changes in blue):

  1. new controls against per-source linear probes and CNNs, and an analysis of why a cheap proxy keeps the order of sources (G1, G2);
  2. target-only LWM baselines with 10 to 50 target labels (G3);
  3. tests of the decisions the estimate supports, before and after deployment (G4);
  4. tests on wideband, 28 GHz, impaired, measured and time-varying channels (G5);
  5. revised Section 2 (Selection via Proxy) and Section 5.1 (narrowband inputs).

DiMTE keeps the paper's hyperparameters, except a readout for wideband inputs (G5). A 5-minute interactive demo is at https://huggingface.co/spaces/userrr1323/DiMTE.

Below, we address the common themes:

G1. Why our specific proxy is necessary to keep the deployment ranking

Concerns (epKP W1/Q1, W2; Rg3t W1; p7Si Q1): DiMTE is just a lightweight source classifier validated on the target.

Response: We partly agree that DiMTE is a lightweight proxy, but the design of the proxy is what makes the ranking work. Standard proxies rank sources worse. This is a clarification from the submitted ablations, plus new held-out-seed controls (Appendix T.1). Section 2 now also contrasts DiMTE with Selection via Proxy.

DiMTE works because it does two things: (1) it embeds datasets into a frozen LWM space to reduce low-level channel variations between sites, and (2) it fits a local density proxy to predict exact deployment errors.

Below, we report the mean Spearman rank correlation (ρ\rho, higher is better) between each method's ranking of the sources and the true ranking, on binary (LOS/NLOS) and 32-class beam prediction tasks (Tables 1 and 29):

Proxy method (Spearman ρ\rho, ↑\uparrow)Binary, SionnaBinary, DeepMIMO32-Class, Sionna32-Class, DeepMIMO
DiMTE, class-conditional mixture (default)0.780.850.790.91
DiMTE, kk-NN rule0.760.870.790.91
DiMTE, Parzen rule0.760.870.820.90
One Gaussian per class (global)0.700.740.710.77
Nearest class mean (global)0.650.600.650.79
LEEP or NCE on DiMTE's mixture (our adaptation)0.750.720.570.74
Strongest independent method*0.580.600.680.77
Held-out seeds (App. T.1): Linear probe per source0.670.770.680.73
Held-out seeds (App. T.1): DiMTE (default mixture)0.760.840.780.90
Held-out seeds (App. T.1): CNN deployment model per source0.810.890.890.93

*Strongest independent methods by column: directed Deep Sets trained on the transfer matrix, NCE on a per-source MLP over LWM tokens, and directed joint KL on raw channels and LWM embeddings.

Takeaways:

  • The frozen LWM space matters: On raw channels normalized per user, the same proxy's LOS/NLOS ranking on Sionna drops from 0.78 to 0.21 (Table 30). Inside the LWM space, local density proxies beat basic linear probes (by up to +0.18) and global averages (by up to +0.25).
  • Exact units and direction matter: We measure how a source helps a specific target (s→ts \to t). Symmetric metrics (which treat source and target equally) lower ρ\rho on every test set. Unweighted likelihoods (LEEP/NCE) ignore how severe an error is (like dB loss), and lower ρ\rho by 0.13–0.22 in three of the four settings.

Bottom line: DiMTE is cheap, but its specific choices (the LWM space, a local density proxy, scoring in task units, and direction) are what make its ranking right.

AuthorsOpenReview ↗

General Response (2/4): Theoretical Justification and Label Scarcity

G2. Why a lightweight proxy correctly ranks heavy deployment models (Appendix T.1)

Concerns (epKP Q2; p7Si Q2): Why should a simple proxy keep the exact ranking of a much more complex deployment model?

Response: Partly agree; clarification, with a new analysis (Appendix T.1). No proxy ranks perfectly.

To rank sources accurately, we only need to predict the relative difference between candidate sources for a specific target tt. We can break down DiMTE's predicted error P^s→t\hat{P}_{s\to t} into the true error TA(s→t)T_{\mathcal{A}}(s\to t), the target's baseline difficulty ctc_t, and a source-specific error rs,tr_{s,t}:

P^s→t=TA(s→t)+ct+rs,t  ⟹  P^s→t−P^s′→t=(TA(s→t)−TA(s′→t))+(rs,t−rs′,t).\hat{P}_{s\to t} = T_{\mathcal{A}}(s\to t) + c_t + r_{s,t} \implies \hat{P}_{s\to t} - \hat{P}_{s'\to t} = \big(T_{\mathcal{A}}(s\to t) - T_{\mathcal{A}}(s'\to t)\big) + (r_{s,t} - r_{s',t}).

Takeaway: Because we are comparing sources for the same target, the target's difficulty (ctc_t) cancels out completely. As long as the source-specific error (rs,tr_{s,t}) is small, sources whose true errors clearly differ are ordered correctly. The frozen LWM space makes these errors 1.2 to 3.6 times smaller than raw channels do (in 7 of 8 comparisons). On beam prediction, ctc_t accounts for 70–77% of DiMTE's squared error. Since that part cancels, our proxy orders 96–99% of clearly separated source pairs (the top third of each target's true gaps) correctly, against 89–90% for the strongest comparator.

Uncertainty and margins: Like any transferability estimate, DiMTE's ranking is uncertain, and so is the true ranking itself. Even a CNN trained on every source ranks at only 0.81–0.93 against held-out seeds (G1), and the two deployment models pick different best sources for 61–90% of targets, mostly near-ties. If every source-specific error is at most ε\varepsilon, every pair whose true errors differ by more than 2ε2\varepsilon is ordered correctly, and closer pairs are near-ties. A site can therefore keep a short list. Training only DiMTE's top 5 recovers 91% (LOS/NLOS) and 85% (beams) of the random-to-best gap, against 89% and 91% for all 82 candidates (Appendix T.8).

Bottom line: the target's difficulty drops out of every comparison, so a cheap proxy ranks well as long as its source-specific errors are small, and the LWM space keeps them small.

G3. Reusing a selected source vs. adapting on scarce target labels (Appendix T.2)

Concerns (epKP W3/Q3; Rg3t W2; p7Si Q6): Using 1,000 target labels contradicts the goal of scarce data. How does DiMTE compare to just fine-tuning the foundation model directly on the target?

Response: On the 1,000 labels, this is a clarification. We agree that the target-only LWM baseline was missing, and we added it (Appendix T.2). We used 1,000 target labels only to measure the ranking precisely. With just 10 labels, DiMTE already ranks Sionna beam sources better than every other rule (Table 41).

The table compares the two uses of the same m∈10,25,50m \in \\{10, 25, 50\\} labels on all 84 Sionna targets.

Strategy (same m labels)F1 err., m=10F1 err., m=25F1 err., m=50Beam dB, m=10Beam dB, m=25Beam dB, m=50
CNN trained from scratch on target0.0680.0240.01815.312.210.0
Best target-only LWM adaptation0.0560.0220.01715.012.19.9
LWM of DiMTE's top source, then tuned0.0370.0240.02110.39.49.2
pp-value (DiMTE vs. best target-only)0.007n.s.n.s.<0.001<0.0010.014

F1 err. = 1 − F1 on LOS/NLOS. Beam dB = beam loss, the L1-RSRP difference, the power a user loses with the predicted beam instead of the best one. Lower is better for both. n.s. = not significant.

DiMTE is general. It selects data, not a model, and its estimate is defined for any deployment model; we tested it with two (a raw-channel CNN and fine-tuned LWM). LWM and DiMTE therefore combine. The best strategy above fine-tunes LWM on DiMTE's top source, then on the mm labels. With 10 labels, the selected source helps the CNN too (0.049 vs. 0.068 F1 error, 11.3 vs. 15.3 dB, against a CNN from scratch).

Takeaway: Source selection helps most when target labels are scarce. At m=10m=10 labels, fine-tuning the source DiMTE ranks first significantly beats training from scratch or fine-tuning LWM on the target alone (0.037 vs 0.056 F1 error, p=0.007p=0.007; 10.3 dB vs 15.0 dB, p<0.001p<0.001). On the harder 32-class task it stays ahead up to 50 labels; with 100 labels, target-only LWM reaches 8.2 dB. On binary LOS/NLOS the two tie from 25 labels on.

Bottom line: selecting a source with DiMTE helps most when labels are scarcest, and for longer on the harder 32-class task.

AuthorsOpenReview ↗

General Response (3/4): Practical Decisions

G4. Using calibrated risk for practical decisions (Tables 3, 39, 40; Appendix T.10)

Concerns (Rg3t W5; p7Si W1/Q3, Q6): A better ranking does not automatically mean better practical decisions.

Response: Mostly a clarification. The submitted Tables 3 and 40 already test these decisions, and Appendix T.10 adds new ones. DiMTE provides calibrated risk scores (e.g., actual dB error), not just relative rankings. Other scores, even after being mapped to the task metric on the training cities, predict it less accurately (0.054 vs. 0.038 F1 error, 3.1 vs. 2.3 dB; Table 39). This allows us to automate decisions before and after deployment without guessing thresholds:

  • 1. Reuse vs. train locally (Table 40): With scarce labels (10–50), a system must decide whether to deploy a reused source model zero-shot, or train a new model locally. DiMTE compares its predicted source error directly against the measured error of local training, and reuses the source only when its predicted error is lower. As shown below, this policy beats both "always reuse" and "always train":
Policy at a Sionna siteBeam dB, m=10Beam dB, m=25Beam dB, m=50F1 err., m=10F1 err., m=25F1 err., m=50
Always train locally15.312.210.00.0680.0240.018
Always reuse source (zero-shot)10.810.09.70.0730.0410.030
Decide with the best competing score11.110.29.50.0470.0220.016
Decide with DiMTE10.69.78.80.0430.0230.017

Beam dB = beam loss in dB; F1 err. = 1 − F1. Lower is better for both.

On beams, DiMTE's decision is the best at every label count (p=0.002p=0.002 at m=50m=50). It still wins when the site estimates its local error from its own labels by cross-validation (9.3 vs. 9.9 dB, p=0.009p=0.009).

  • 2. Choosing training data and adding data (Table 3, App. T.10): Training on DiMTE's top 3 selected sources yields 8.7 dB error, heavily outperforming a pooled library of all sources (13.4 dB, p=0.010p=0.010) and the top 3 of class-prior KL, which compares only label distributions (12.0 dB, p=0.022p=0.022). Furthermore, augmenting 100 local site labels with DiMTE-selected source data drops the error even further to 5.9 dB, against 7.5 dB for the site labels alone (p<0.01p<0.01) and 8.2 dB when adding class-prior KL's top 3.
  • 3. After deployment (App. T.10): Tracking DiMTE's estimate on windows of 25 new labels detects when a distribution shift is actually hurting task performance (AUROC 0.81 and 0.91, much better than standard embedding distances). Switching to DiMTE's new top source then cuts the error from 0.149 to 0.057 F1 error and from 18.1 to 11.0 dB. DiMTE can also flag individual samples where the deployed model is likely to fail (AUROC 0.68 and 0.78). Finally, clustering a library with DiMTE's directed estimates lowers the target loss compared with symmetric clustering in 8 of 12 cases, with none significantly worse.

Bottom line: because DiMTE predicts the error itself, one estimate drives reuse-or-train, data selection and augmentation, and each of these significantly improves 32-class beam prediction.

AuthorsOpenReview ↗

General Response (4/4): Physical-Layer Realism

G5. Validating on real-world and complex physical channels (Appendices T.3–T.7)

Concerns (p7Si W2/Q4/Q5; Rg3t W3, W4): Simulated narrowband 3.5 GHz channels are not realistic enough. Does this work on wideband, impaired, or measured channels?

Response: We agree this was a limitation in the original draft, and we revised. Section 5.1 now states that the inputs are narrowband (the 32 subcarriers carry one channel). We have added tests for wideband carriers (0.96 and 98 MHz), 28 GHz carriers, noise with antenna calibration errors, real measured arrays (DICHASUS dataset) and drift over time. Because LWM was pretrained on narrowband inputs, DiMTE reads a wideband input by embedding each subcarrier separately and averaging the embeddings (no new parameters), with its Parzen rule. Both were chosen on development sites from the training cities, and three pre-registered tests on untouched data confirm them.

Setting (Sites / environments; carrier)TaskDiMTE (ρ\rho)Strongest indep. (ρ\rho)Our adapt. (ρ\rho)*pp-value
Freq.-selective Sionna test (84/42; 3.5 GHz, 0.96 MHz)LOS/NLOS0.730.460.71≤0.002\le 0.002
Beam-320.800.680.53<0.001
10 dB SNR + antenna gain/phase errors (1 dB, 15∘15^\circ), models retrained (84/42)LOS/NLOS0.720.350.70≤0.021\le 0.021
Beam-320.810.670.51<0.001
Sionna test, wideband NR (84/42; 3.5 GHz, 98 MHz)LOS/NLOS0.740.310.71<0.001
Beam-320.830.720.56<0.001
DeepMIMO test, pre-reg. (84/59; 3.5 GHz, 98 MHz)LOS/NLOS0.820.410.70<0.001
Beam-320.880.750.70<0.001
DeepMIMO dev. sites (36/20; 28 GHz, 95 MHz)LOS/NLOS0.560.290.41<0.001
Beam-320.820.770.55<0.001
Sionna test traced at 28 GHz, pre-reg. (84/42; 95 MHz)LOS/NLOS0.820.420.69<0.001
Beam-320.890.780.64<0.001
Measured DICHASUS (4×84 \times 8; 1.27, 3.44 GHz; 10–50 MHz)Beam-32 (51/6)0.700.650.240.076
UL-AoA (30/5)0.860.760.750.21
Measured + industrial hall, pre-reg. (hall at 1.27 GHz)Beam-32 (56/7)0.700.610.220.046
UL-AoA (35/6)0.800.780.730.37
Measured temporal drift (rank first 60%, score last 40%)Beam-32 (51/6)0.590.510.050.062
UL-AoA (30/5)0.700.690.710.40

*pp-values compare DiMTE with the strongest independent method (permutation tests over environments). "Our adapt." applies LEEP/NCE to DiMTE's mixture. UL-AoA predicts the sector of the uplink angle of arrival.

Takeaway: DiMTE outperforms the strongest independent method in every setting, significantly in every simulated one. Notably, on real measured DICHASUS data, embedding via the LWM improves ranking correlation by 0.11–0.12 over using the raw sub-array channels (p=0.016p=0.016), although LWM was pretrained on simulated channels only. Adding a pre-registered industrial hall (7 environments) makes the measured beam lead significant (0.70 vs. 0.61, p=0.046p=0.046), and under drift over time DiMTE keeps its beam lead (0.59 vs. 0.51; 95% CI excludes zero; p=0.062p=0.062).

Bottom line: the ranking holds on wideband, 28 GHz, impaired and measured channels, and LWM's features help even on measured data they never saw.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.