acceptodds
Under review as a conference paper at ICLR 2027

Simplicity is Key: An Unsupervised Pretraining Approach for Sparse Radio Channels

Abstract

Unsupervised representation learning for wireless channel state information (CSI) reduces reliance on labeled data, thereby lowering annotation costs, and often improves performance on downstream tasks. However, state-of-the-art approaches take little or no account of domain-specific knowledge, forcing the model to learn well-known concepts solely from data. We introduce Sparse pretrained Radio Transformer (SpaRTran), a hybrid method based on the concept of compressed sensing for wireless channels. In contrast to existing work, SpaRTran builds around a wireless channel model that constrains the optimization procedure to physically meaningful solutions and induces a strong inductive bias. Compared to the state of the art, SpaRTran cuts positioning error by up to 32% and increases top-1 codebook selection accuracy for beamforming by 26 percentage points. Our results indicate that modeling radio propagation sparsity as an unsupervised learning objective improves both network optimization and radio-localization performance.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Self-supervised Learning, Wireless Networks, Physics Aligned

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_Bha8OpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SpaRTran, a self-supervised pretraining framework via sparse recovery from a compressed sensing task. The idea is very novel and interesting, where the task can help the model learn about the signal structure compared to conventional general pre-training methods like the MASK. By fine-tuning on different tasks (including classification and regression tasks), the method shows superior performance.

Critical strengths
  1. The compressed sensing-based pre-training method is very novel and efficient.

  2. The proposed pre-training method shows good performance compared to a series of pre-training solutions.

Critical weaknesses
  1. The main concern is that the comparison to task-specific methods is missing. If they already have good performance, the introduction of such a pre-training model will have limited meaning.

  2. The training costs (e.g., iterations, GPU cost) are missing.

Questions for authors

Please consider responding to the weakness part, whether from empirical results or word explanation. Overall, I believe this paper has good quality. If my concerns are addressed, I would like to increase the score to 4.

4 comments

Thank you for your review.

About the weaknesses:

  1. Please note that we already report results for the WiT-Supervised method that is task specific, i.e., does not have a pretraining stage. Does this resolve your concern?
  2. We trained all models on a Tesla V100 SXM2 GPU. To ensure all runs converge we let the pretraining run 24 hours. Note that the convergence time in the finetuning stage largely varies depending on the amount of available data, dataset and downstream task. To gain a understanding of the computational complexity we compare all methods on the KUL-dataset. Due to architectural differences the training time per step varies which is why we report both, training_steps until convergence and the elapsed wall-clock-time
Methodtraining_stepswall-clock-time
SpaRTran20K4h
C-WiMAE100K3h
SWiT30K2h
Masking35K6h
LWM20K4h

We added a corresponding section in the appendix of the paper (see Section A7).

Thank you again for giving us a positive rating. We hope we could address your concerns adequately and would greatly appreciate if you could raise your score accordingly.

Thank you for your efforts to address my concerns. Regarding Weakness 1, I still have some concerns. First, WiT was originally designed for localization rather than beamforming; therefore, more relevant beamforming methods should be compared. Moreover, WiT is somewhat dated. Although I do not think these issues diminish the value of the work itself, in its current form I would like to maintain my previous score.

Thank you for your fast answer. We believe we can resolve your remaining concerns as well.

We would like to note that the benefits of pretraining for localization and beamforming tasks are well established in the literature (e.g. Guler2025, Alikhani2024). Therefore, to maintain a clear focus on our core contributions and adhere to the page limit, we did not conduct an additional evaluation of these benefits.

We would also like to emphasize that, in line with existing work on self-supervised learning, our goal is not to establish a universally superior method across all downstream tasks, but rather to demonstrate the effectiveness of our proposed self-supervised pretraining objective. The downstream tasks serve as evaluation benchmarks to assess the quality of the learned representations. The observed performance improvements under the evaluated conditions provide evidence that our pretraining procedure learns representations that are beneficial for downstream applications.

Nonetheless, we agree that a comparison against a supervised baseline is interesting, which is why we included WiT. We interpret WiT primarily as an application of the Transformer architecture to CSI signals rather than as a method restricted to positioning tasks. We therefore selected WiT as a suitable supervised baseline due to its Transformer-based architecture. From a methodological perspective, the two downstream tasks share a similar learning setup, with the main difference lying in their respective learning objectives.

We are not aware of a more recent purely-supervised approach that would provide a better candidate for comparison.

Thank you again for your thoughtful review. We hope these clarifications address your concerns. If any questions remain, we would greatly appreciate it if you could elaborate on the specific evaluation you would like to see.

Thank you for your further explanation. I find this clarification convincing, and I have increased my score to 4.

Reviewer_mt9tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SpaRTran, an unsupervised pretraining method that uses sparse multipath structure to learn radio-channel representations. A transformer processes individual antenna links, and a gated reconstruction head estimates complex coefficients over a fixed or learned dictionary. The pretrained model is then fine-tuned for localization and beam selection. Experiments on KUL, FH-IIS, and DeepMIMO show improvements in several settings, particularly under changes in antenna configurations and with moderate amounts of labeled data.

Critical strengths
  1. Using sparse channel reconstruction as a pretraining task is physically motivated. The separate estimation of component activity, magnitude, and phase provides a useful connection between representation learning and the complex-valued multipath channel model.

  2. The experiments cover measured and simulated channels, two downstream tasks, and changes in antenna and propagation configurations. The fixed/learned dictionary and no-pretraining ablations are useful. The five-seed KUL experiment in Appendix A.6 provides additional support for the reported transfer results.

Critical weaknesses
  1. How does λ\lambda control sparsity in Eq. (7)? The penalty is λ∥1(ρgate)∥1\lambda\|\mathbf{1}(\rho_{\mathrm{gate}})\|_1, but Sec. 4.1 states that no gradient passes through the binary gate. This term therefore has zero gradient almost everywhere, and the auxiliary loss does not depend on λ\lambda. Is this a notation error, or is a surrogate gradient used? Please clarify how the implemented loss produces the results in Fig. 3.

  2. The proof of Theorem A.2 seems to have a scaling error. The operator multiplies coefficients in SS by c>1c>1, but the proof uses 1/c1/c when calculating ∥O[h]∥A1\|O[h]\|_{\mathcal{A}_1}. Simply reversing the scaling also changes ∥O−1∥\|O^{-1}\|, so it does not directly fix the argument. Please correct this step and clarify how the operator assumptions apply to the learned transformer.

  3. It is unclear how much of the gain comes from sparse pretraining rather than the downstream head. SpaRTran uses a 1.4M-parameter head, compared with 176k for C-WiMAE, and Table 6 shows that head depth affects accuracy substantially. The existing ablations are useful, but comparing different pretraining objectives with the same backbone, head, and fine-tuning settings would help isolate the benefit of sparse reconstruction.

Questions for authors
  1. What is the exact sparsity-loss expression used in the implementation, and what derivative is propagated to rho_gate? Can the authors provide the relevant code and a fixed-seed check of the effect of changing lambda?

  2. How should the scaling step in Theorem A.2 be corrected while retaining the inverse-operator factor from Theorem A.1? Which properties of this operator are guaranteed by the proposed encoder?

  3. Does sparse reconstruction retain an advantage over a masked-reconstruction pretraining objective when the single-link backbone, downstream head, and fine-tuning settings are held fixed?

5 comments

Thank you your review.

About the Questions and Weaknesses:

How does control sparsity in Eq. (7)?: Thank you for highlighting this notation error. You are right. No gradient flows through the binary gate. We have updated the equation accordingly.

Can the authors provide the relevant code and a fixed-seed check of the effect of changing lambda?: Figure 3b shows the effect of different lambda values under a fixed seed. You can find the relevant code snippets here https://anonymous.4open.science/r/temp-0B8C/sparse_loss_objective.py

It is unclear how much of the gain comes from sparse pretraining rather than the downstream head.:

We thank the reviewer for raising this important point, as it allows us to clarify several considerations underlying our experimental design.

First, our approach differs from existing methods by deliberately operating exclusively on individual links during pretraining. Consequently, our approach relies on a more expressive downstream prediction head to capture these inter-link dependencies, motivating our choice of a larger head architecture. The results in Table 6 further illustrate the importance of the prediction head size for our approach.

Second, the compared methods employ different backbone architectures and input representations, each tailored to their respective self-supervised learning objectives. For example, LWM uses an encoder-only Transformer, whereas Contra-WiMAE employs an encoder–decoder architecture. Similarly, Masking treats individual time steps as tokens, while other methods construct tokens from CSI patches. We therefore adhere to the original implementations, as we consider these architectural choices to be well-suited to their respective pretraining objectives and aim to evaluate each method in its intended configuration. Nonetheless, we agree that matching the prediction head sizes provides a meaningful additional comparison. We therefore conducted an experiment in which we increased the head size of Contra-WiMAE, our strongest baseline, to match that of our proposed method, ensuring comparable prediction head capacity. The results on IIS-FHG, KUL, and DeepMIMO are reported here https://anonymous.4open.science/r/temp-0B8C/CWiMAE_matched_head.md. Interestingly, increasing the head size generally leads to a degradation in performance compared to the original configuration. We hypothesize that this may be attributed to overfitting due to the increased model capacity. These findings further support our decision to retain the original architectures of the baselines, as increasing the head capacity does not necessarily translate into improved downstream performance.

Third, we additionally isolate the contribution of our pretraining procedure through the ablation study in Figure 3a. The results show that the same architecture performs considerably worse without pretraining, providing evidence that the observed performance is not explained by the architecture alone but substantially benefits from the proposed pretraining procedure.

The proof of Theorem A.2 seems to have a scaling error Yes, there is an algebraic error in our current proof, which is resolvable. In fact, during our resolution, we noticed that we can tighten the theorem even more by not requiring a sufficiently large CC, but instead for any C>1C>1. This means that, in theory, the algorithm only has to detect the space, and immediately receives a reduction in the atomic norm.

Which properties of this operator are guaranteed by the proposed encoder? We cannot generally guarantee that the encoder finds any particular operator during training. The theorem should be seen more as an existence proof: If no such improving operator existed, the entire problem of transforming a signal for better approximation may not be sensible. Our claim is about existence, not uniqueness or attainability under finitely sized networks.

Does sparse reconstruction retain an advantage over a masked-reconstruction pretraining objective when the single-link backbone, downstream head, and fine-tuning settings are held fixed? You are raising an important point here. Our sparsity-based objective relies on fundamentally different assumptions than the masking objective. Masked reconstruction on multi-link CSI exploits spatial correlations and shared propagation characteristics across links to recover missing signal components. This is less applicable to the single-link setting, as independent multipath components provide limited exploitable correlations for reconstructing missing signal components within an individual link.

In contrast, SpaRTran encourages the model to represent each link using a small number of dictionary components, effectively introducing an information bottleneck. Thus, our objective does not rely on cross-link correlations. We hope this clarification resolve your concern.

Thank you again for your thoughtful review. Do we have sufficiently addressed all of your points?

Thanks for the reply and the code.

Eq. (7) is fine now, the new penalty matches the code.

I still have a concern about Theorem A.2. In Theorem A.1 the OO-atomic norm is ∥O[h]∥A1\Vert O[h] \Vert_{\mathcal{A}_1}, so for OcO_c it should be c⋅si+bic \cdot s_i + b_i. The ∑j∣ai,j∣/wj\sum_j |a_{i,j}|/w_j you use now belongs to Oc−1O_c^{-1}, and the inverse of that operator has norm cc, not ≤1\le 1. So the norm and the inverse norm seem to come from two different operators. I'm also not sure a diagonal operator can work here: in an orthonormal basis, ∥O−1∥⋅∥O[h]∥A1\Vert O^{-1} \Vert \cdot \Vert O[h] \Vert_{\mathcal{A}_1} is never smaller than ∑j∣aj∣\sum_j |a_j|. For example, with s=3s=3, b=1b=1, c=2c=2 the original bound has R=4R=4, while OcO_c gives 77 and Oc−1O_c^{-1} gives 55. If the theorem is only an existence argument, Sec. 3 should not call it a theoretical justification.

On FH-IIS and KUL the matched-head numbers differ from the original ones by about the seed variation in Table 7, so the bigger head just doesn't change much. On DeepMIMO they are close to random guessing (8.6% with 16 beams, down from 49.2%), which looks like a failed run, not overfitting. And the C-WiMAE numbers in Table 3 are taken from Guler et al., so this is not a like-for-like comparison.

My Q3 is still open. With the same single-link backbone and the same head, does the sparse objective beat a masked one? The reply argues that masking would not work on a single link, but that is something an experiment should show.

I'm keeping my score for now. If Theorem A.2 is fixed or dropped and the same-backbone comparison is added, I'm happy to reconsider.

Thank you for your answer:

Regarding the theorem: I think we misunderstood your original point and we have now fixed the theorem. The issue is whether we try to improve the approximation with respect to basis {ψj}\{\psi_j\} or wrt the transformed dictionary {O−1ψj}\{O^{-1}\psi_j\}. From the point of view of our training framework, this doesn't matter since we learn the dictionary, so whether we view it as the original basis or the transformed really doesn't matter. Considering the question "The main challenge in moving from theory to practice is that the operator required to "rotate" into the improved coordinate frame may be infinite-dimensional." We have prepared a proof that works on either basis and already includes the argument from the corollary. This way, we only use unitary transformations (i.e. no expansion or retraction ∥U∥=1\|U\|=1). Thank you for your insistance. We believe the work is substantially stronger because of it

Regarding Q3: We are currently working on the requested same-backbone experiment. However, given limited computational resources and the considerable training time required, we cannot guarantee that the results will be available before the end of the author–reviewer discussion period, although we will make every effort to provide them. In any case, we intend to include the requested results in the final version of the paper.

We would also like to highlight that the original Masking approach can exploit both intra-link and cross-link correlations during pretraining. Restricting masking to individual links retains the same underlying reconstruction objective but removes the additional information provided by cross-link correlations, leaving only intra-link dependencies for recovering missing signal components. Since this modification does not introduce any additional information or learning mechanism, we see no clear reason to expect single-link masking to outperform its multi-link counterpart. Instead, we hypothesize that the reduced amount of exploitable information may limit its effectiveness. Nevertheless, we agree that the proposed experiment is valuable for empirically validating this expectation and aim to provide a controlled comparison with our sparsity-based objective.

We hope you will consider these arguments and our commitment to the additional evaluation when reassessing the paper.

Thank you again for your constructive feedback.

Reviewer_sfccOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SpaRTran, a physics-informed unsupervised representation-learning framework that uses sparse wireless-channel reconstruction as a pretraining task for a transformer encoder. Its main contribution is to incorporate a domain-specific sparsity prior into both the training objective and decoder architecture, which is a heuristic but interesting idea.

Critical strengths
  1. SpaRTran turns sparse multipath reconstruction as a pretraining task and implements it through a complex-valued gated decoder that separates component selection, magnitude, and phase. This is a substantive, domain-specific adaptation of (the original real-valued) gated sparse autoencoders. The idea is smarter than feeding CSI data directly to a transformer.
  2. The considered wireless applications are interesting and the authors have shown good improvement over some existing methods.
Critical weaknesses
  1. My major concern is that the proposed framework does not contribute much to "AI". The core idea is to large extent based on [Raj, 2024]. The wireless applications are interesting, but there are so many journals/transactions for AI + wireless, which are more suitable for this work in my eyes.
  2. The proofs in Appendix A do not support the whole SpaRTran idea, but a very small and minor part of it.
Questions for authors
  1. What is the typical size of the Ψ\Psi matrix in Eq. (4)? Is it a very fat matrix?
  2. The authors mentioned "The main challenge in moving from theory to practice is that the operator OO required to "rotate" hh into the improved coordinate frame may be infinite-dimensional...". I can't follow it and its proof.
  3. The authors mentioned "Our objective is to learn a channel representation z\boldsymbol{z} that encodes the environmental characteristics of the radio signal." Why the target is not solving the sparse optimization problem in (5)? If you simply want to learn the latent zz, my concern is that it is hard to evaluate its quality and interpretability.
  4. I don't understand the form of x^\hat{x} in eq.(6) and the form of a^=x^e−i⋅fphase(z)\hat{a}= \hat{x} e^{-i \cdot f_{phase}(z)}. Can you explain them?
  5. The discussion of the "learned dictionary" is too brief and superficial.
3 comments

Thank you for reviewing our paper.

Regarding the weaknesses:

Weakness 1: We agree that our work does not primarily aim to advance general-purpose AI methodology. Instead, our main contribution is a physically motivated self-supervised learning objective specifically designed for wireless systems. We believe our work aligns with two of the relevant topics listed in the ICLR call for papers:

  • Representation learning for computer vision, audio, language, and other modalities.
  • Applications to physical sciences (physics, chemistry, biology, etc.).

Our focus lies on the specific characteristics of the CSI modality and the application of representation learning to problems in wireless systems.

We also think that the overall involvement of the ICLR community in wireless topics can also be recognized from several publications on these topics in the last years. For example the following works:

  • Yong, C. et al., Map as a Prompt: Learning Multi-Modal Spatial-Signal Foundation Models for Cross-scenario Wireless Localization (2026).
  • Yaoqi, L. et al., TS-DDAE: A Novel Temporal-Spectral Denoising Diffusion AutoEncoder for Wireless Signal Recognition Model Pre-training (2026).
  • Xinyan, C. et al., RF-MatID: Dataset and Benchmark for Radio Frequency Material Identification (2026).
  • Rui, Z. et al., RFMamba: Frequency-Aware State Space Model for RF-Based Human-Centric Perception (2025).

(Please note Yaoqi2026 works on IQ data rather than CSI data which is why we did not consider this work in our paper.)

Taken together, we believe these examples highlight the growing relevance of wireless representation learning within the broader machine learning community. We hope this additional context helps clarify the motivation and relevance of our contribution and would appreciate it if you could reconsider their assessment of the paper's significance and suitability for ICLR.

Weakness 2: We agree with you that the proofs in Appendix A address a specific theoretical aspect of SpaRTran rather than the entire framework. Their purpose is to provide theoretical motivation for learning a transformation that enables improved sparse approximation, as established by Theorems 4.1 and 4.2. Our core contribution remains the sparse-reconstruction objective for self-supervised pretraining, with the theoretical results providing additional insights into its underlying principles.

Does the above clarifications resolve your concerns?

About the Questions:

  1. The size of Φ\Phi is determined by the number of delay bins in the input signal (rows) and the hyperparameter NN, which specifies the number of dictionary entries (columns). Accordingly, Φ\Phi has the following dimensions across the datasets: FH-IIS: 32×12832 \times 128 (4.1k parameters), KUL: 100×128100 \times 128 (12.8k parameters), and DeepMIMO: 1024×1281024 \times 128 (131k parameters). Compared to the weight matrices of the backbone's linear layers, we consider Φ\Phi to be a relatively small to moderately sized matrix, introducing a limited parameter overhead.

  2. Our proof relies on the classical framework of functional analysis where functions are treated as representations of a vector space with infinite dimensions: the classical example is the space of polynomials which has a basis in the monomials {x0,x1,x2,… }\{x^0, x^1, x^2, \dots\}. This can represent polynomials like 2x+x2−52x + x^2 - 5 as the vector (5,2,1)(5, 2, 1). In addition, it can also represent functions like sin⁡\sin since sin⁡x=∑n=0∞(−1)n(2n+1)!x2n+1=x−x3/3!+x5/5!+…\sin x= \sum^\infty_{n=0} \frac{(-1)^{n}}{(2n+1)!}x^{2n+1} = x - x^3/3! + x^5/5! + \dots which corresponds to the dense infinite dimensional vector (0,1,0,−3!,+5!,… )(0,1, 0, -3!, +5!, \dots). In sparse recovery, one tends to want to find sparse vectors (i.e. where almost all values are zero): Our insight is that we can find a "rotation" (i.e. orthonormal transform) between two different spaces such that the representation becomes nicer: E.g. if you change the monomials from xnx^n to sin⁡m(x)cos⁡n(x)\sin^m(x)\cos^n(x) you get a basis like {1,cos⁡x,sin⁡x,… }\{1, \cos x, \sin x, \dots\} in this space, the dense infinite dimensional vector sin⁡(x)↦(0,1,0,−3!,+5!,… )\sin(x) \mapsto (0,1, 0, -3!, +5!, \dots) is representable using a sparse (0,0,1,0,0,… )(0,0,1,0,0,\dots) vector. We exploit this property to axis-align the "bad" parts of the vector space (where the noise lives) and then shrink them down. In practice, we use a transformer to learn the orthogonal transform and scaling necessary from a sufficiently large set of training examples.

  3. Thank you for highlighting this issue. First, we would like to distinguish between sparse signal recovery and our objective of self-supervised representation learning. While Eq. (5) describes the classical sparse recovery problem, our goal is not to solve this problem directly, but to use it as a physically motivated pretext task. Specifically, the sparse reconstruction objective guides the Transformer encoder toward learning meaningful and generalizable channel representations (z). Second, we agree that evaluating the quality of the learned representation is important. In line with established self-supervised learning approaches, we evaluate representation quality indirectly through downstream tasks. Our experiments on localization and beamforming demonstrate the practical benefits of the learned representations, while the ablation in Figure 3a specifically highlights the contribution of the sparsity-based pretraining objective. Regarding interpretability, we would like to clarify that the learned representation zz is not itself sparse. Rather, sparsity is enforced on the reconstruction coefficients a^\hat{a}, encouraging the encoder to retain information relevant for a compact channel decomposition. Figures 1 and 3c provide additional insights into this decomposition and the behavior of the learned dictionary.

  4. We understand that this formulation may appear complex. The underlying motivation is that the gating mechanism of the gated autoencoder operates on real-valued coefficients, whereas reconstructing the original complex-valued signal requires complex-valued coefficients. We therefore separate the coefficient estimation into magnitude and phase: x^\hat{x} represents the magnitude of the coefficients and is subject to the gating mechanism, while a^\hat{a} denotes the final complex-valued coefficients, with the phase generated by fphase(z)f_{\mathrm{phase}}(z). This allows us to retain the original gating mechanism while extending the approach to complex-valued signal reconstruction.

  5. We have extended the explanation of the learned dictionary on page 6.

Thank you again for your review. Please let us know if there are any questions that remain unresolved.

Reviewer_sfccOpenReview ↗

Thanks

I thank the reviewers for their reply. I still believe this paper will be more visible if it is published in wireless journals. I will keep my score.

Reviewer_QxvCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces SpaRTran, a hybrid model‑and‑data‑driven unsupervised pretraining Transformer for sparse radio channel state information (CSI) that leverages compressed‑sensing‑inspired sparse signal decomposition as a self‑supervised pretext task, operates on individual antenna‑pair links to achieve system‑agnostic representations, and achieves up to 32% lower positioning error and 26‑percentage‑point higher top‑1 beamforming codebook‑selection accuracy compared with state‑of‑the‑art self‑supervised wireless models, though it underperforms under extremely scarce labeled data as it does not learn cross‑antenna correlations during pretraining.

Critical strengths

This paper embeds well‑established compressed‑sensing sparse‑channel priors into both model architecture and pretraining loss, creating strong physically grounded inductive biases that substantially boost cross‑system generalization for CSI‑based downstream tasks.

Critical weaknesses
  1. SpaRTran’s pretraining on individual antenna links, which confers system‑agnostic generalization, prevents it from learning inter‑channel correlations during pre‑training, leading to inferior performance compared to competing self‑supervised approaches when only ≤5% labelled fine‑tuning data is available, a practically relevant low‑resource scenario for many real‑world wireless deployments.
  2. Since cross‑antenna spatial relationships cannot be captured in pretraining, the method requires a heavy, parameter‑intensive ResNet‑style fine‑tuning head to compensate, raising inference overhead and weakening the practical appeal of the pretrained backbone for lightweight downstream wireless applications.
Questions for authors
  1. Since SpaRTran underperforms baselines under ≤5% labeled fine‑tuning data, could the authors explore and document simple modifications (e.g., lightweight auxiliary cross‑link pre‑training losses) to mitigate this low‑label limitation without sacrificing its system‑agnostic single‑link representation property?
  2. The proposed approach relies on a heavy ResNet‑style finetuning head to learn cross‑antenna correlations missed during pretraining. Can the authors provide complexity‑comparison analysis including inference latency and FLOPs, and explore lighter‑weight head alternatives to make SpaRTran more suitable for resource‑constrained real‑world wireless devices?
  3. While the paper evaluates fixed and learned dictionary variants, could the authors add more ablation studies quantifying how dictionary size N and sparsity penalty λ jointly affect downstream task performance across different datasets, to offer clearer practical guidance for hyperparameter selection by practitioners deploying this model on new wireless systems?
2 comments

First and foremost thank you for your reviewing our paper.

Regarding the weaknesses:

  1. We acknowledge that SpaRTran does not consistently outperform the baselines in the very low-label regime (≤5%), as also discussed in Section 5.1. However, this trade-off stems from a deliberate and well-motivated design choice: learning system-agnostic representations from individual links to promote generalization across different system configurations. While we recognize the reduced performance in low-label settings as a limitation, we believe it should be considered in the context of the broader generalization benefits offered by our approach. We therefore hope you will reconsider this aspect of your assessment in light of the underlying design motivation.

  2. We agree that SpaRTran employs a relatively large prediction head in terms of parameter count. However, the head primarily consists of convolutional layers, which can be executed efficiently compared to the attention mechanisms in the Transformer backbones. Consequently, the increased parameter count does not necessarily translate into a proportional increase in inference latency. This is supported by our measurements, which show that SpaRTran achieves a CPU inference latency of 5.42 ms, comparable to our strongest baseline, C-WiMAE (5.71 ms), as shown in the table below. Would you consider this concern sufficiently addressed?

Regarding the questions:

  1. We would like to emphasize that learning representations exclusively from individual links is a deliberate design choice central to SpaRTran. Introducing cross-link objectives during pretraining would inherently create dependencies on the specific system configuration, potentially limiting the system-agnostic nature and generalizability of the learned representations. While we agree that the proposed modifications are an interesting direction for future research, we deliberately focus on single-link pretraining to preserve this key property and maintain a clear scope for the paper.

  2. Thank you for raising this important point. First, we refer to Table 6, which investigates the impact of different prediction head sizes and provides insights into the trade-off between model capacity and performance. To further address computational complexity, we have added an appendix section comparing the FLOPs, inference latency, and throughput of all methods, as summarized in the table below. While SpaRTran requires more FLOPs than several baselines due to its larger prediction head, this does not translate proportionally into increased inference latency. Notably, SpaRTran achieves a CPU inference latency of 5.42 ms, comparable to C-WiMAE (5.71 ms), despite requiring approximately six times as many FLOPs. This suggests that the additional computational operations introduced by the larger head can be executed efficiently. We acknowledge that SpaRTran is not the most computationally lightweight approach. Nevertheless, these results indicate that its increased model capacity does not necessarily result in substantially higher inference latency. | Method | FLOPs | Latency (ms) | Throughput (iterations/s) | | ------------- | -------------- | -------------- | -------------- | | SpaRTran | 2.2375 G | 5.42 | 188.02 | | CWiMAE | 0.3530 G |5.71 | 175.26| |LWM |0.3092 G | 3.28 | 305.24| | SWiT | 0.3578 G | 0.96| 1038.29 | |Masking| 9.4204 G| 309.63| 3.23|

  3. We agree that practical guidance on hyperparameter selection is important. Figures 3b and 3c aim to provide this guidance: Figure 3b shows that performance remains relatively stable across a reasonable range of hyperparameter values, while Figure 3c illustrates the limits beyond which performance deteriorates. As discussed in Section 5.4, our results suggest that SpaRTran is relatively robust to variations in λ\lambda and NN. Based on this observation, we focused our computational resources on other experiments rather than conducting exhaustive dataset-specific hyperparameter searches We hope this clarification addresses your concern.

We sincerely thank you for your valuable feedback. We hope that our responses have addressed the raised concerns and would greatly appreciate it if you would consider increasing your score accordingly.

We would like to kindly follow up on our previous response and ask whether our clarifications and additional computational analysis have adequately addressed your concerns. We hope that the provided explanations and experimental evidence have helped clarify the motivation behind our design choices and the practical implications of our approach. We would greatly appreciate your feedback on whether any concerns remain unresolved. If our responses have sufficiently addressed your points, we would be grateful if you could reconsider your assessment and potentially raise your score. Thank you again for your time and constructive feedback.

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

Thank you for your time and consideration throughout the review process.

We would like to summarize the main concerns raised during the rebuttal phase and the steps we have taken to address them:

Novelty and relevance to ICLR (sfcc): We clarified that our core contribution is a physically motivated self-supervised learning objective for wireless representation learning. We highlighted its alignment with the ICLR call for papers and provided examples of recent wireless-related research within the machine learning community. While we acknowledge the domain-specific nature of our contribution, we believe it fits well within ICLR's scope.

Theoretical justification (sfcc, mt9t): We clarified that the theoretical analysis motivates the existence of transformations that improve sparse approximation rather than providing guarantees for the entire SpaRTran framework. Following the reviewer's feedback, we also identified and corrected the issues in Theorem A.2 and revised the proof.

Sparsity objective and implementation (mt9t): We corrected the notation error in Eq. (7), provided the relevant implementation code, and referred to Figure 3b for the effect of the sparsity penalty. Reviewer mt9t explicitly confirmed that this concern was resolved.

Contribution of the pretraining objective (mt9t): To better separate the effects of pretraining and architecture, we conducted an additional matched-head experiment with C-WiMAE. Furthermore, following the reviewer's request, we performed a controlled comparison using the same backbone, prediction head, and training settings, replacing only our sparse-reconstruction objective with single-link masking. The preliminary results on KUL show substantially worse performance for masking, supporting the effectiveness of our proposed objective. We intend to include the complete evaluation in the final paper.

Single-link pretraining and low-label performance (QxvC): We acknowledged the limitations in the extremely low-label regime and explained the deliberate trade-off between learning cross-link dependencies during pretraining and obtaining system-agnostic representations that generalize across different antenna configurations.

Computational complexity (QxvC, Bha8): We added Appendix A.7, reporting training steps, wall-clock time, FLOPs, CPU inference latency, and throughput. Despite its larger prediction head, SpaRTran achieves an inference latency comparable to C-WiMAE (5.42 ms vs. 5.71 ms).

Hyperparameters and dictionary design (QxvC, sfcc): We clarified the dictionary dimensions, expanded the explanation of the learned dictionary, and highlighted the hyperparameter analysis in Figures 3b and 3c, which illustrates the robustness and limitations of different parameter choices.

Representation quality and interpretability (sfcc): We clarified the distinction between sparse recovery as a pretext task and learning transferable representations. We explained how the downstream benchmarks evaluate representation quality and how the sparse decomposition provides additional insight into the learned features.

Comparison with supervised methods (Bha8): We clarified the motivation for including the supervised WiT baseline and explained the role of downstream tasks in evaluating self-supervised representations. Following this discussion, Reviewer Bha8 found our clarifications convincing and increased their score to 4.

Regarding the remaining reviewer discussions:

Reviewer QxvC: We provided detailed clarifications and additional computational analyses addressing the reviewer's concerns. Unfortunately, we have not received further feedback despite our follow-up, and therefore cannot determine whether any concerns remain unresolved.

Reviewer mt9t: The reviewer indicated a willingness to reconsider their score if Theorem A.2 was corrected and a same-backbone comparison was added. We have since revised the theorem and provided preliminary results from the requested experiment, supporting the advantage of sparse reconstruction. However, the reviewer maintained their score without providing further feedback on these latest changes.

We acknowledge that some limitations remain and that the full controlled evaluation is still being completed. Nevertheless, we believe the rebuttal process has substantially strengthened the paper through additional experiments, theoretical corrections, and methodological clarifications.

We would greatly appreciate it if these efforts, together with the responses to the reviewers' concerns, could be taken into account in the final assessment.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.