- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper seeks to boost the accuracy of models trained for automatic modulation classification of communication signals by utilizing well-known calibration techniques to tune the readout as a function of the signal-to-noise ratio (SNR). However, these techniques are also shown to be sensitive to errors in SNR estimates. The contribution is narrow and, in my opinion, not impactful.
Critical strengths
The paper shows that, for a fixed ML model, SNR-dependent affine readouts can boost classification performance. Essentially, this applies known calibration techniques to each SNR bin to improve performance.
Critical weaknesses
The paper makes a simple and rather obvious observation, applying calibration to the problem of automatic modulation classification (AMC), but specialized to SNR bins. This does not meet the bar for a non-trivial contribution to a core ML conference. A better venue might be a conference focused on wireless communication.
Even within with narrow domain of AMC, no effort seems to be put into the fundamental problem of how to be robust to SNR variations in terms of model training and calibration as a whole. SNR estimates are, in general, not available for the AMC problem, so the problem of estimating SNR (explicitly or implicitly) must be convincingly addressed in order to contemplate using SNR-sensitive readouts.
The paper is poorly written in terms of technical clarity. For example, a soft target training scheme called DPC is used, but the expansion for the acronym, and more importantly, the details of the training methodology are not provided.
Questions for authors
You have shown that errors in SNR estimates adversely impact the performance of SNR-sensitive readouts. How do you propose obtaining accurate enough SNR estimates in an AMC problem?