- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces DiMTE, a lightweight estimator for ranking source datasets by the performance that models trained on them would achieve on a labeled target domain. It uses a shared frozen encoder and source-specific class-conditional mixture classifiers, evaluating their predictions under the downstream task metric without training a deployment model for each candidate source. The contributions include a directed proxy-risk formulation, a compact prototype-based implementation, and an error bound separating target sampling error from proxy–deployment mismatch. Evaluation focuses on LOS/NLOS identification and beam prediction using ray-traced wireless datasets, including city-disjoint splits and an independently generated Sionna RT test set, with additional camera-trap and multilingual classification experiments. Results support improved source-ranking quality at low computational cost, while gains in source selection and downstream training are less consistent.
Critical strengths
- DiMTE addresses a practical data-selection problem by ranking candidate source datasets without training a deployment model on each one. Its directed, task-specific scores provide a useful alternative to dataset similarity, with substantially lower computational cost than exhaustive training.
- The evaluation includes city-disjoint splits, multiple training seeds, and an independent Sionna RT test set generated after the method was frozen. These controls strengthen the credibility of the ranking results and reduce concerns about environment leakage and adaptation to the original benchmark.
Critical weaknesses
- The connection between ranking quality and downstream utility remains insufficiently established. On DeepMIMO, improved rankings do not clearly improve the top-source selection over the strongest comparator. The exploratory data-addition experiments show uncertain comparative gains for beam prediction and negative transfer for LOS/NLOS identification. This limits the evidence that better rankings translate into better training decisions.
- The wireless evaluation remains restricted to ray-traced channels with a fixed carrier frequency and antenna configuration. It does not establish robustness on measured, wideband Massive MIMO-OFDM channels with estimation noise, calibration errors, and temporal variation. These factors could change the agreement between the lightweight proxy and deployment models, directly affecting the method’s central assumption.
Questions for authors
- Clarify the substantive methodological distinction between DiMTE and using a lightweight source-trained classifier’s target performance as a proxy for deployment performance.
- Clarify the reliability of DiMTE for each specified deployment model, rather than their average, including cases where proxy–deployment disagreement changes source selection.
- Identify the reported evidence of statistically significant downstream gains over the strongest comparator, distinguishing ranking improvements from source-selection and data-addition benefits.
- Specify the bandwidth, subcarrier spacing, and frequency selectivity represented by the 32-subcarrier inputs, and delimit the conclusions applicable to wideband Massive MIMO-OFDM.
- Clarify whether existing experiments establish robustness to channel-estimation noise, array calibration errors, or temporal variation. Distinguish demonstrated robustness from assumptions requiring future validation.
- Clarify when source selection outperforms target-only training or reuse of an existing model under the same target-label budget, accounting for feature extraction and historical tuning costs.
3 comments
Response to Reviewer p7Si (2/2)
W1 / Q3. Statistically resolved downstream gains
Concern: Are the downstream practical improvements statistically significant?
Response: Clarification; all results below are in the submitted paper. Yes. Using environment-clustered permutation tests, DiMTE's gains are statistically significant at every pipeline stage:
- Source ranking: DiMTE beats all 48–54 baselines that train no model per source, on both test sets and tasks (). On DeepMIMO, it also beats 12 model-aware estimators computed on already-trained models ().
- Zero-shot selection: On DeepMIMO, reusing the top-1 source without fine-tuning significantly lowers binary F1 error at 10, 25, and 50 labels (
0.067, 0.059, 0.053vs.0.085, 0.067, 0.067, ) and 32-class beam loss at 25 and 50 labels (5.7, 5.6vs.6.5, 6.4dB, ). Without fine-tuning, the choice of source matters most. - Reuse-or-train decisions: Automatically deciding to reuse a source vs. train locally beats the strongest competing score (
8.8vs.9.5dB, ), and still does when the local error is estimated by cross-validation (9.3vs.9.9dB, ). - Choosing and adding data: Training on DiMTE's top-3 sources beats the class-prior KL baseline (
8.7vs.12.0dB, ). Augmenting local site labels with these sources significantly reduces the loss (5.9vs.7.5dB, ); its lead over adding class-prior KL's top 3 (8.2dB) is not significant ().
W2 / Q4 / Q5. Narrowband specs vs. real-world physical robustness
Concern: The paper relies heavily on simulated 3.5 GHz narrowband data. How well does this generalize?
Response: Agree; revised. We have updated Section 5.1 to explicitly state that our base simulated inputs are 32-antenna ULA channels at 3.5 GHz whose 32 subcarriers carry the same narrowband channel, so the inputs have no frequency selectivity.
To address generalization, we ran extensive new tests (G5 / Appendices T.3–T.7). DiMTE leads the strongest comparator on all of the following, with a significant lead on all but the last:
- Frequency-selective channels, 0.96 MHz and 100 MHz NR ()
- 28 GHz carriers ()
- 10 dB SNR with antenna gain/phase errors (
1 dB, ) () - Real measured DICHASUS arrays (7 environments, )
- Temporal drift across 16-minute recordings (
0.59vs.0.51; 95% CI excludes zero; )
Open limitation: multi-day or seasonal drift beyond our 16-minute recordings remains open for future field tests.
Q6. Strategy based on label budget and compute cost
Concern: How does DiMTE's optimal usage change based on the available label budget, and what is the compute overhead?
Response: Clarification, with new results. DiMTE naturally adapts to the available target labels (G3–G4):
- No labels for training (100 labels used only to rank): Train on DiMTE's top-3 sources. This beats pooling all data (
0.032vs.0.047F1 error, ) and class-prior KL's top 3 on beams (8.7vs.12.0dB, ). - 10 labels: Fine-tune DiMTE's top source. This beats adapting the LWM strictly on the target data.
- 25–50 labels: On beams, fine-tuning DiMTE's top source still leads (
G3). DiMTE's predicted errors also decide, per site, whether to reuse the source model unchanged or train locally (G4). - 100 labels: Use DiMTE for targeted data augmentation, which cuts error further (
7.55.9dB, ).
Compute overhead: Extracting LWM embeddings takes 0.11 s per 1,000 samples on a GPU. Scoring takes seconds on a CPU (0.25–3.4 s). The one-time transfer matrix takes 1.4–3.6 GPU-hours, about as much as training fine-tuned LWM on every dataset of our library (1.3–5.6 GPU-hours), and it does not grow with the number of targets (see our reply to epKP, Q4).
Thank you again for your time and feedback; please let us know if further details would help during discussion.
Following up on our response
Dear Reviewer,
As the response deadline approaches, please let us know if there are any points we can clarify. If our response has addressed your concerns, we would appreciate hearing whether you would consider raising your score.
Thank you again for your time and feedback.
Best regards,
The Authors
Response to Reviewer p7Si (1/2)
Dear Reviewer p7Si,
Thank you for your expert review. As a quick reminder: DiMTE ranks candidate source datasets for a new target by predicting each source's downstream error, without training a model per source. Your suggestions directly motivated our new experiments on measured arrays, wideband 100 MHz, 28 GHz carriers, hardware impairments, and temporal drift. (These are now in Appendix T;
G1–G5refer to the General Response, and our live demo is at https://huggingface.co/spaces/userrr1323/DiMTE).Q1. Distinction from a generic source-trained classifier
Concern: How is this different from just scoring a target on a standard source classifier?
Response: Clarification. As detailed in
G1, standard classifiers rank sources worse. DiMTE works specifically because of three connected choices:0.09–0.10correlation (ρ) on Sionna, and the LWM space itself matters (on normalized raw channels the LOS/NLOS ranking falls from0.78to0.21).0.22–0.28on Sionna beams.0.11–0.23on Sionna.Q2. Reliability across different deployment models (and disagreement cases)
Concern: Does DiMTE work reliably across different architectures, and what happens when the models disagree on the top source?
Response: Clarification, with a new analysis. DiMTE reliably predicts rankings for both raw-CSI CNNs and fine-tuned LWMs (leading in 7 of 8 test settings and tying our LEEP adaptation in the eighth,
0.64vs.0.66, CI [−0.04, +0.00]; Table 24). We added a new analysis on model disagreement. The CNN and LWM do disagree on the #1 best source for61–90%of targets. However, these disagreements are mostly near-ties. Using one model's top source costs the other a median of only6–14%of the gap between a random and the best source. As a result, DiMTE's top source closes nearly the same share of that random-to-best gap whether the models agree or disagree on the absolute top source: