- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents CSIEval, a standards-aligned benchmark for CSI estimation, spatial–frequency prediction, and compression feedback. It combines task-specific datasets derived from existing wireless data resources with a unified evaluation protocol and toolkit covering performance, storage, efficiency, and generalization. Experiments compare representative models across the three tasks and examine trade-offs among reconstruction quality, resource consumption, and robustness.
Critical strengths
-
The paper addresses a practically relevant reproducibility problem in CSI research, where inconsistent data processing, observation settings, feedback budgets, and evaluation procedures hinder reliable model comparison. The effort to establish a common protocol across CSI tasks has potential community value.
-
The evaluation jointly considers reconstruction quality, storage and computational costs, and robustness under distribution shifts, covering important trade-offs relevant to practical deployment.
Critical weaknesses
Major
-
The evaluation protocol is not consistently reflected in the inspected implementation. Feedback NMSE is averaged in dB rather than the linear domain, contrary to Eq. (5), while prediction NMSE uses full-channel rather than hidden-region energy for normalization, contrary to Eq. (7). The prediction splitter also allows map overlap across splits despite the claimed map-exclusive protocol. These discrepancies undermine confidence in the reported model comparisons.
-
The benchmark’s incremental contribution over existing resources remains insufficiently demonstrated. It mainly consolidates established tasks, metrics, and data resources, while the experiments provide limited evidence of new, actionable findings. The coverage of the 8,000 samples, stability of model comparisons, and applicability beyond simulated domains also require stronger substantiation.
Minor
-
The practical model-selection value remains underdeveloped. Although BER and supplementary link/system diagnostics are reported, the paper does not systematically establish how the prediction and feedback comparisons translate into communication utility or guide model selection under specified feedback, latency, or memory constraints.
-
Several icons and embedded plots in Figure 1 are blurry, reducing the clarity of the benchmark overview. Please improve the image quality.
Questions for authors
-
Which implementation version and split manifests produced the reported results? Please reconcile the NMSE computation and map-exclusive splitting claims with the inspected implementation, and clarify whether the results require recomputation.
-
Which findings in the current results demonstrate additional model-selection guidance beyond existing benchmarks, particularly under specified feedback, latency, or memory constraints?