- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces SPBENCH, an open multi-task benchmark for learning-based exploration seismic processing. Motivated by a 368-paper survey showing that most work uses private datasets and only 25 papers release code, it standardizes six processing tasks (random-noise attenuation, trace interpolation, ground-roll suppression, multiple suppression, deblending, first-arrival picking) under shared protocols across 10 datasets and 43 settings. It reproduces 24 supervised methods and reports 495 method–setting evaluations over three seeds, releasing datasets, implementations, configurations, evaluation scripts, and per-seed results.
Beyond the data and baselines, the paper's real contribution is evaluative: it adds two domain-aware diagnostics and uses them to ask whether benchmark conclusions are robust. SCoRE resolves reconstruction fidelity across frequency bands and local energy regimes; RCnorm measures the geometric continuity of a predicted first-arrival ridge without reference labels. Three findings stand out: synthetic rankings only partially predict field rankings (Kendall τ from 0.24 to 0.73, varying by task); ranking stability under growing interference is task-dependent; and RCnorm agrees with MAE-based model rankings at mean Kendall 0.881 across three field surveys.
Critical strengths
-
The existing public resources for this domain are mostly about the problems downstream — OpenFWI and E-FWI do full-waveform inversion, CIG-Bench is interpretation. The processing step that actually prepares the data for those, like demultiple or ground-roll suppression, has no agreed benchmark at all. That's the gap SPBENCH addresses, and it's a reasonable one to pick.
-
The two diagnostics are the real core of the paper, and are well designed. RCnorm has a clean property: it is invariant under rearranging per-trace errors but changes when the ridge re-bends — so it captures geometric information that MAE, RMSE, and H@k cannot. SCoRE uses energy bins as a proxy for weak signals, and the authors are explicit that it is a proxy, not a claim to have found weak events. That calibration is right.
-
Reproducibility is handled well. Fixed splits, versioned artifacts, per-seed results, and every leaderboard number traceable to a released configuration — 495 configurations (over 1,000 parameter sets) are auditable. This is strong for this line of work.
Critical weaknesses
-
Baseline fidelity is my main concern. The paper states that its numbers are "not directly comparable" to the original papers — which is normal, since a benchmark re-tunes hyperparameters. But the problem is: when a reproduced method ranks low, I can't tell whether that's because the method is genuinely weak or because the reproduction was imperfect. For a benchmark whose purpose is comparability, this matters. I'd want to know how faithfully each of the 24 methods was reimplemented, which configurations deviate from the source papers, and how far the width-augmented "-L" variants stray from the original architecture. Without that, I can't attribute a method's rank to the method itself.
-
The boundaries of the strongest diagnostics are only partly explored. RCnorm agreement with MAE rankings is strong at loose tolerances (0.881) but drops to 0.244 at H@1. The paper mentions this but doesn't really dig into when RCnorm becomes misleading. EB-FE is similar: it relies on a clean reference and uses energy-percentile bins as a weak-signal proxy, but it never validates that an improvement in a low-energy bin actually corresponds to recovering a real weak event, not just improving a proxy.
-
The three tasks where the paper introduces new paired datasets — multiple suppression, deblending, ground roll — are precisely the ones with the fewest settings (multiple: 1, deblending: 4, ground roll: 6), while interpolation (16) and random noise (12) dominate. So the transfer and stability conclusions in Sections 4.2 and 4.3 are strongest for noise/interpolation/deblending and thinnest for multiple, which has a single setting. A reader may miss this asymmetry.
-
The three headline claims rest on little data. Synthetic-to-field, degradation stability, and RCnorm agreement each depend on a handful of task pairs or a few field surveys (three). Directionally convincing, but the paper is advancing generalizable claims about how seismic-processing models rank, and the evidence base is narrow. More varied field data would either strengthen or revise them.
Questions for authors
The paper reports that RCnorm agrees with MAE rankings overall (mean 0.881), but at the strictest tolerance (H@1) that agreement collapses to 0.244. So I'm trying to understand the boundary: should a practitioner use RCnorm when the tolerance is tight, or does the score become unreliable there? In other words, what's the practical rule for when RCnorm is safe to use and when it isn't?
4 comments
Response to reviewer 4vj4 (Part 2/2)
W2 and Q1. Validating RCnorm and SCoRE
Thank you for asking where these diagnostics are most useful and where they can be misleading. We added a blind model-selection experiment for RCnorm and two waveform comparisons for SCoRE. The first tests whether we can choose a picker on a new survey before seeing its reference picks. The other two let readers check whether a better weak-energy or frequency-band score corresponds to a visible difference in the recovered signal.
RCnorm on a new survey. For each of the six transfer directions among Brunswick, Halfmile, and Lalor, we applied seven source-trained pickers to the target records. We chose the model with the lowest RCnorm using only its predictions. We then used the target reference picks to ask how close that choice came to the model with the lowest MAE. RCnorm selected the best model in four directions and the runner-up in a fifth, only 0.79 ms above the best.
The transfer from Lalor to Halfmile showed how this rule can fail. RCnorm selected DnCNN-Seg, which had the largest MAE. We show its predictions alongside STUNet’s and the reference picks on the same gather. DnCNN-Seg picks nearly the same early arrival time across the traces. This gives it a low curvature score even where the reference arrival changes substantially. STUNet follows that change and has the lowest MAE on this transfer.
See Section 4.4 and Appendix H.3, especially Table 9 (blind selection), Figure H.3 (failure case), and Table 10 (timing tolerances).
For the tight-tolerance application you ask about, we would use RCnorm to screen candidates for continuity. We would then check the shortlisted models against a small set of reference picks at the required tolerance. A uniform shift in arrival time leaves curvature unchanged, so even a very smooth ridge can carry a systematic timing error.
SCoRE and waveform recovery. To address your question about weak reflections, we chose a field example where overall SNR and the weak-energy score favored different models. On Mobil AVO with 50% random trace loss, Attention U-Net-L ranked first overall, while CA-Unet ranked first in the weak-energy bin. Since the traces were recorded before we masked them, we could check which model better recovered the observed reflection. We selected window W3, which contains a weak reflection and lies entirely in the low-energy region. The enlarged trace segment and its residuals show that CA-Unet followed the recorded amplitude more closely. This ties the weak-bin result to a particular reflection that readers can inspect.
We made a corresponding comparison for the frequency bands in moderate deblending. SCRN had the highest overall SNR, while cDDPM scored highest in the lowest frequency band. We extracted that band from the clean reference and both reconstructions, then displayed the waveforms and their residuals on a common amplitude scale. cDDPM left less residual energy in that band, consistent with its higher low-band score. These examples show how to use SCoRE when preserving a weak event or a particular band matters to the intended application.
See Section 3.2 and Appendix H, Figures H.1 (weak reflection and trace residuals) and H.2 (low-band reconstructions and residuals).
W3. Uneven coverage across tasks
Thank you for pointing this out. We added a rugged-seafloor simulation for multiples and a field survey for deblending, ground roll, and random noise. Multiples, deblending, and ground roll now have two, five, and seven settings, respectively.
One shot record can yield several settings by changing the noise level or the missing-trace mask. Multiples arise when seismic waves reflect more than once. With acquisition and modeling conditions fixed, each velocity model yields one set of records containing multiples, paired with the corresponding primary references. The original velocity model therefore gave us one setting. The new rugged-seafloor model gives us a second.
See Section 3.1 and Tables 23 (noise), 46 (ground roll), 48 (multiples), and 53 (deblending).
W4. Do the findings hold on additional field data?
Thank you for encouraging us to test this on more field data. We added a second field survey for random noise, ground roll, and deblending. Within each task, we compared the synthetic ranking with each field ranking using Kendall’s for the same nine methods, trained separately on each dataset. Agreement varied across tasks and surveys. We also tested synthetic-trained models directly on field records without fine-tuning. In all four tasks with field targets, the model that led on synthetic data lost first place after transfer. Both analyses support our original conclusion that synthetic rankings alone do not reliably predict which model will perform best on a field survey.
See Section 4.2, Figure 3, and Appendix G, Figure 13(a–c, e).
response
Thank you for a thorough and well-organized response. I have verified the additions against the manuscript, and my principal concerns have been addressed.
Thank you for your constructive suggestions and positive feedback on our responses. Your input has helped us significantly improve the quality of our manuscript.
Response to reviewer 4vj4 (Part 1/2)
Thank you for your thoughtful review and for recognizing SPBench’s contribution to reproducible prestack processing. In the marked-up manuscript, changes are shown in blue.
We made four main additions.
We report training and inference times alongside accuracy and parameter counts.
See Tables 2–3 and Appendices E (timing) and I (results).
We added 10 traditional baselines across all six tasks.
See Appendix D, Table 7, and Tables 2–3.
We added weak-reflection and low-frequency SCoRE examples and tested RCnorm across surveys, including a case of systematic timing bias.
See Figures H.1–H.2 (SCoRE), Table 9 and Figure H.3 (RCnorm).
We applied synthetic-trained models directly to field records in four tasks, without fine-tuning.
See Section 4.2 and Appendix G, Figure 13(a–c, e).
W1. Fidelity of the reproduced baselines
Thank you for raising this concern. We agree that readers need to know which choices came from the original method before they can interpret a low rank. For the 24 supervised methods, we followed the architectures and network hyperparameters specified in the source papers. Because preprocessing depends on the seismic data, we used a shared protocol for each task and dataset. Within each setting, methods use the same splits, input–target pairs, and scoring scripts.
We report our capacity experiments separately as “-L.” For U-Net, ResUNet, and Attention U-Net, increasing the base width from 32 to 64 doubles the hidden channels at each scale and preserves the stages and modules. DnCNN-L likewise uses 64 instead of 32 hidden channels, with 17 layers in both versions. For each main-leaderboard entry, readers can inspect the configuration, including normalization and training schedule, and the individual seed results. Parameter counts alone tell readers little about runtime. We report training time per run and GPU inference time over the full test set so readers can weigh accuracy against the cost of training and applying each model.
We attempted to reproduce more methods than we ultimately included. In some cases, key settings were unclear in the source paper and our runs yielded markedly low scores. We felt it would be unfair to present those scores as representative of the original method. Each included configuration provides a reproducible performance reference on SPBench.
See Section 4.1 and Appendix D (methods and variants), Appendix E (protocol), and Tables 2–3 (costs).