- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the changes in performance and ranking of open-set wireless signal recognition (WSR) methods under different known/unknown class partitions. The authors propose Partition-Robust Risk Minimization (PRRM). A second-order approximation is used to connect mean performance with risk variance, which is further decomposed into sample-level variance () and class-composition variance (). Based on this formulation, DualDiff first pre-trains an encoder using latent diffusion noise prediction and signal reconstruction, and then fine-tunes it using ArcFace and a classifier-weight uniformity regularizer. The experiments use three synthetic signal datasets: RML2016.10A, RML2022, and RadChar. DualDiff obtains competitive average OSCR performance on these datasets.
Critical strengths
A relevant problem with a clear formal framing. Open-set WSR is relevant to spectrum monitoring, where unknown signal types appear after deployment, and the paper points out that reported performance and method rankings can change with the choice of known and unknown classes. The paper does not stop at observing this. It formulates the goal as maximizing the expected log-performance over a meta-distribution of partitions, and uses a second-order expansion of to obtain a mean–variance surrogate (Eqs. 2–4). The law-of-total-variance split into sample-level variance and class-composition variance (Eq. 5) gives a clear vocabulary for explaining why average performance alone is an insufficient criterion, and it provides the stated rationale for the two-stage design.
Competitive average performance and a relatively comprehensive evaluation.** DualDiff achieves the highest Overall OSCR among the compared baselines in all six main settings, by 0.70 to 6.53 points over the strongest baseline in each setting, and it has the best mean OSCR and mean rank over the 96 additional random partitions on RML2016.10A. The evaluation includes ablation studies, five runs of DualDiff with different random seeds, and analyses of hyperparameters and network depth. To separate the effect of ArcFace, the authors also add it to IQFormer and TS-DDAE, and to the baselines in the 96-partition study. They also try to connect the framework to the experiments by estimating and (Tables 8 and 10).
Critical weaknesses
W1. The motivation is not sufficiently supported by the experiments set and evaluation
Different modulation types have inherent differences in their physical signal structures, such as constellation geometry and symbol mapping. As a result, some combinations of known and unknown classes are intrinsically harder to distinguish than others. The experiments do not examine how much of the observed partition sensitivity comes from these inherent differences in class separability and how much comes from instability of the model. Decomposing the risk variance into and does not itself distinguish these two causes.
PRRM aims to reduce performance variation across different known/unknown class partitions. However, the proposed pre-training learns denoising and reconstruction, while fine-tuning optimizes classification and weight geometry for the known classes of the current partition. Neither objective directly constrains the performance differences between partitions. The paper does not clearly explain why these objectives should make the risks across different partitions more consistent, rather than only improving recognition within each setting.
The channel-related motivation has not been directly evaluated through experiments on unseen channel conditions. The paper reports recognition performance at different SNR levels, but this does not by itself demonstrate generalization to unseen SNR conditions. A more direct test would train the model at one SNR level (or within a specified SNR range) and test the same model at other SNR levels excluded from training. The experiments also do not separately examine the effects of specific channel impairments, such as multipath fading or carrier-frequency offsets. Therefore, the reported results mainly compare performance across different signal conditions, rather than directly demonstrating robustness to channel impairments not seen during training.
W2. Limited wireless-specific methodological contribution
As described in Sections 3.2–3.3, Gaussian noise is added to the encoded latent features rather than generated through a physical model of channel impairments on the received IQ signal. The paper does not sufficiently establish why such latent perturbations can represent structured wireless effects such as multipath fading, frequency offsets, or timing errors. The central-limit argument in Section 3.2 alone is not sufficient to justify this connection. The wireless-specific contribution of this training mechanism therefore remains unclear.
The method mainly uses general representation-learning objectives, with very limited integration of domain-specific wireless signal characteristics, such as constellation geometry, symbol timing, modulation structure, and physical channel transformations, into its design and analysis. The IQ/spectrum fusion and the TS-Net denoiser come from previous WSR work. The authors relate angular margins to modulation geometry and weight uniformity to channel-induced rotations, but these links are not supported by physical modeling or targeted experiments.
W3. The theoretical guarantees are not sufficiently established
Appendix A aims to show that diffusion pre-training reduces and classifier-weight uniformity reduces , but the key links are not sufficiently justified. For , Eqs. (26)–(28) assume that an ideal ETF arrangement of classifier weights gives the same expected risk for every known-class subset. However, different subsets can contain classes with different recognition difficulty. Similar classifier-weight geometry alone does not guarantee equal risks. Under this assumption, Eq. (28) would even imply if the weights of all subsets matched the reference configuration, although their actual risks could still differ.
For , Eqs. (17)–(20) assume that the diffusion loss is bounded below by a positive multiple of the expected squared within-class feature deviation. This relationship is not sufficiently established: a low denoising loss can still coexist with substantial variation of same-class features along the learned manifold. In addition, OSCR describes performance over a set of test signals, while the derivation treats risk as a function of an individual signal without explaining the connection. Therefore, even as qualitative bounds, these results do not sufficiently establish that optimizing the proposed objectives necessarily reduces the partition-level risk variances defined by PRRM.
W4. Synthetic-only evaluation and incomplete open-set comparisons
RML2016.10A, RML2022, and RadChar are synthetic datasets, and there is no evaluation on measured wireless signals. This limits the support for claims about robustness in real wireless environments. Apart from FSOS-AMC, the evaluation does not include representative general-purpose open-set recognition methods such as OpenMax and ARPL, or the wireless-specific Open-ICL method cited by the authors.
Questions for authors
Q1. Can the authors quantify intrinsic class-pair or partition difficulty, for example with confusion analysis or difficulty-stratified OSCR? Please also test trained models without retraining, on SNR levels excluded from training and on data with controlled channel impairments (e.g., multipath fading, carrier-frequency offsets, or phase shifts), to verify robustness to unseen conditions.
Q2. Which parts of DualDiff are specifically motivated by wireless signal properties? Please clarify the exact noise injection point and scale, and the meaning of the temporal/spectral noise parameters in Table 5. Please provide supporting experiments or relevant references to justify treating latent Gaussian noise as a surrogate for physical wireless channel impairments rather than merely a generic feature perturbation.
Q3. Please justify the two key assumptions behind the variance bounds: why the diffusion loss controls within-class feature deviation (Eqs. (17)–(20)), and why an ideal ETF configuration would imply the same expected risk across different class subsets (Eqs. (26)–(30)). Please provide rigorous arguments or relevant references and clarify how the per-input analysis is linked to partition-level OSCR risk.
Q4. Could the authors include a real measured wireless signal dataset, rather than relying entirely on synthetic datasets, to strengthen the credibility of the evaluation? Please also compare with representative open-set recognition approaches under comparable protocols, such as OpenMax (Bendale and Boult, 2016), ARPL (Chen et al., 2022), and the wireless-specific Open-ICL (Yang et al., 2025).
References
[1] Bendale, Abhijit, and Terrance E. Boult. "Towards open set deep networks." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
[2] G. Chen, P. Peng, X. Wang and Y. Tian, "Adversarial Reciprocal Points Learning for Open Set Recognition," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 11, pp. 8065-8081, 1 Nov. 2022, doi: 10.1109/TPAMI.2021.3106743.
[3] C. Yang, Z. Feng, S. Yang and Q. Pan, "Open-ICL: Open-Set Modulation Classification via Incremental Contrastive Learning," in IEEE Transactions on Neural Networks and Learning Systems, vol. 36, no. 5, pp. 8853-8866, May 2025, doi: 10.1109/TNNLS.2024.3414942.