acceptodds
Withdrawn from ICLR 2027

CSIEval: A 6G Standard-Aligned Benchmark for AI-Based CSI Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

As 6G evolves toward an AI-native paradigm, channel state information (CSI) has become a key target for AI-enabled wireless communication, with growing research on CSI estimation, prediction, and compression feedback. However, existing evaluations lack systematic alignment with emerging 6G standards and use inconsistent protocols across studies, hindering fair comparison. Thus, we propose CSIEval, a standards-aligned benchmark for CSI processing that encompasses three representative tasks. CSIEval constructs and processes datasets in accordance with relevant communication protocols and evaluates models under a unified framework across four dimensions: performance, storage, efficiency, and generalization. The benchmark includes 8,000 CSI samples and more than 15 evaluation metrics, providing broad coverage of model capabilities. Experimental results demonstrate that CSIEval provides a comprehensive characterization of model behavior, enables fair comparisons across different models, and reveals the trade-offs among predictive accuracy, resource consumption, and generalization capability. Code and project resources are available at the anonymous project page: https://anonymous.4open.science/r/csieval-8018.

Primary area:
datasets and benchmarks
Keywords:
ML: Evaluation, Benchmarking, Datasets & Analysis

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_bVHJOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents CSIEval, a standards-aligned benchmark for CSI estimation, spatial–frequency prediction, and compression feedback. It combines task-specific datasets derived from existing wireless data resources with a unified evaluation protocol and toolkit covering performance, storage, efficiency, and generalization. Experiments compare representative models across the three tasks and examine trade-offs among reconstruction quality, resource consumption, and robustness.

Critical strengths
  1. The paper addresses a practically relevant reproducibility problem in CSI research, where inconsistent data processing, observation settings, feedback budgets, and evaluation procedures hinder reliable model comparison. The effort to establish a common protocol across CSI tasks has potential community value.

  2. The evaluation jointly considers reconstruction quality, storage and computational costs, and robustness under distribution shifts, covering important trade-offs relevant to practical deployment.

Critical weaknesses

Major

  1. The evaluation protocol is not consistently reflected in the inspected implementation. Feedback NMSE is averaged in dB rather than the linear domain, contrary to Eq. (5), while prediction NMSE uses full-channel rather than hidden-region energy for normalization, contrary to Eq. (7). The prediction splitter also allows map overlap across splits despite the claimed map-exclusive protocol. These discrepancies undermine confidence in the reported model comparisons.

  2. The benchmark’s incremental contribution over existing resources remains insufficiently demonstrated. It mainly consolidates established tasks, metrics, and data resources, while the experiments provide limited evidence of new, actionable findings. The coverage of the 8,000 samples, stability of model comparisons, and applicability beyond simulated domains also require stronger substantiation.

Minor

  1. The practical model-selection value remains underdeveloped. Although BER and supplementary link/system diagnostics are reported, the paper does not systematically establish how the prediction and feedback comparisons translate into communication utility or guide model selection under specified feedback, latency, or memory constraints.

  2. Several icons and embedded plots in Figure 1 are blurry, reducing the clarity of the benchmark overview. Please improve the image quality.

Questions for authors
  1. Which implementation version and split manifests produced the reported results? Please reconcile the NMSE computation and map-exclusive splitting claims with the inspected implementation, and clarify whether the results require recomputation.

  2. Which findings in the current results demonstrate additional model-selection guidance beyond existing benchmarks, particularly under specified feedback, latency, or memory constraints?

Reviewer_nYr9OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents a common evaluation framework (CSIEval) for AI-based CSI estimation, prediction, and compression feedback. The framework covers data construction, normalization, data splitting, model interfaces, and performance reporting. Specifically, the authors evaluate reconstruction quality, storage requirements, computational efficiency, and generalization under different channel and deployment conditions. The experiments include representative neural models and supplementary link-level measurements.

Critical strengths
  1. A common evaluation protocol is useful for CSI learning, where differences in data processing and experimental settings often prevent direct comparison. The use of observable training inputs for normalization, map-exclusive splits, and explicit feedback-bit accounting addresses practical sources of unfair comparison.

  2. The evaluation considers accuracy together with latency, memory, and robustness. This provides more useful information for deployment than reconstruction error alone. The standards traceability table also distinguishes standard-derived principles from the numerical choices made by the benchmark.

Critical weaknesses
  1. The NMSE and SGCS values in Table 1 seem inconsistent. With the same evaluation units and averaging, and negligible ϵ\epsilon, we should have NMSElinear≥1−SGCS\mathrm{NMSE}_{\mathrm{linear}} \geq 1-\mathrm{SGCS}. However, CsiNet gives about 0.2280.228 on the left and 0.6140.614 on the right. Could the authors check the calculations and explain this gap? This matters for the reliability of the benchmark.

  2. What SNR is used in Table 3? The LSTM, GAN, and Transformer-PCE results match the 55 dB entries in Table 18, but ChannelNet does not: −19.86-19.86 versus −15.76-15.76 dB. Please clarify whether the tables use the same checkpoints and test conditions, and provide the actual BER configuration rather than listing it as “Explicit.”

  3. Do all feedback models in Table 1 reconstruct the same target? Sec. 3.1.3 includes both full-channel and eigenvector feedback. If the targets differ, their NMSE and SGCS are not directly comparable, even with the same bit budget. Please clarify the target for each model and either report the two settings separately or use a common link-performance metric.

Questions for authors
  1. Can the authors provide paired reference/reconstruction outputs and metric code for Table 1, including the numerical value of ϵ\epsilon, to explain the NMSE–SGCS inconsistency?

  2. What are the exact common SNR, test subset, checkpoints, and link-level configuration used in Table 3? How do these settings relate to Table 18?

  3. Do all rows in Table 1 reconstruct the same target? If so, how are the original matrix-feedback and eigenvector-feedback architectures adapted to that target?

Reviewer_91eQOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents CSIEval, a benchmark for evaluating AI-based CSI estimation, spatial–frequency prediction, and compression feedback under a common protocol informed by relevant ITU/3GPP frameworks. Its main contributions are a dataset of 8,000 CSI samples, standardized preprocessing and leakage-aware splitting, and an automated evaluation framework covering performance, storage, computational efficiency, and generalization through more than 15 metrics. Validation compares eight feedback models, three prediction models, and eight estimation models, supplemented by classical LS/LMMSE estimators. Experiments examine reconstruction accuracy, resource costs, and robustness to noise, quantization, observation masks, mobility, and deployment shifts. The results illustrate accuracy–efficiency trade-offs, although missing configuration details and limited statistical validation weaken the conclusions.

Critical strengths

-Unified evaluation across three important CSI tasks. CSIEval evaluates estimation, prediction, and compression feedback within a common framework covering accuracy, storage, efficiency, and generalization.

-Provide a valuable methodological foundation for reproducible benchmarking and strengthen the paper’s potential usefulness to the community.

Critical weaknesses

-Missing training settings, exact data splits, and BER configurations prevent independent verification—a central requirement for a benchmark.

-Feedback representations and baseline adaptations are insufficiently specified, making architectural gains difficult to separate from implementation effects.

-Tiny robustness differences lack confidence intervals, while large performance gaps lack explanatory ablations, undermining confidence in the conclusions.

Questions for authors

-Can you demonstrate that every Table 1 model reconstructs the same mathematical target under identical observation and feedback constraints? If matrix and eigenvector reconstruction are mixed, what justifies presenting their NMSE/SGCS values as comparable?

-Equation (15)’s claimed power preservation does not generally hold for correlated antenna responses. What covariance assumption justifies it, and do the reported results survive correcting the normalization if that assumption fails?

-What controlled evidence establishes that STCMixerNet’s approximately 18.4 dB advantage reflects model capability rather than unequal tuning, adaptation errors, or unintended access to hidden entries? Have you verified identical inputs, evaluation masks, and split isolation across implementations?

-What exact link configuration produces Table 3’s 9.10–28.10% BER, and what is the corresponding perfect-CSI BER? Without this reference, how can the reported errors be attributed to channel estimation rather than limitations of the detection chain?

-SGCS does not resolve eigenvector ambiguity within a nearly degenerate dominant eigenspace. How many samples are affected, and do the claimed rankings persist when evaluated using subspace fidelity or achieved beamforming gain?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.