acceptodds
Under review as a conference paper at ICLR 2027

SPBENCH: A MULTI-TASK EVALUATION BENCHMARK FOR EXPLORATION SEISMIC PROCESSING

Abstract

Exploration seismic processing underpins subsurface imaging and resource exploration, where data quality shapes downstream interpretation and decision making. Learning-based methods have advanced rapidly, but their results remain difficult to compare across studies. Our literature survey of 368 papers reveals widespread reliance on private or difficult-to-reproduce datasets, and only 25 provide public code. This makes it difficult to attribute reported gains to model design rather than differences in experimental settings. To fill this gap, we introduce the Seismic Processing Benchmark (SPBench). SPBench covers six representative processing tasks, including random noise attenuation, trace interpolation, ground-roll noise suppression, multiple suppression, deblending, and first-arrival picking. We reproduce 24 supervised methods on 10 datasets under 43 standardized settings and release datasets, implementations, configurations, evaluation scripts, and results. Global scores hide frequency- and energy-dependent behavior, and per-trace pick errors ignore spatial continuity. We therefore introduce signal-component-resolved evaluation (SCoRE) for reconstruction and the reference-free ridge-curvature score () for first-arrival picking. Our analyses reveal three patterns. Synthetic rankings do not reliably predict field rankings, and the agreement varies by task when models train within each setting. As degradation strengthens, rankings reorder more under coherent ground roll than under random-like interference. The reference-free ridge score tracks MAE-based model rankings in the evaluated settings (mean Kendall correlation 0.881 across three field surveys), while SCoRE reveals component-dependent differences hidden by global scores. Together, these resources provide a fair and reproducible basis for comparing learning-based seismic-processing methods. The findings further characterize how method advantages vary across the evaluated conditions, and inform model selection.

Primary area:
datasets and benchmarks
Keywords:
benchmark, exploration geophysics, seismic data processing, deep learning evaluation, AI for science

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_4vj4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces SPBENCH, an open multi-task benchmark for learning-based exploration seismic processing. Motivated by a 368-paper survey showing that most work uses private datasets and only 25 papers release code, it standardizes six processing tasks (random-noise attenuation, trace interpolation, ground-roll suppression, multiple suppression, deblending, first-arrival picking) under shared protocols across 10 datasets and 43 settings. It reproduces 24 supervised methods and reports 495 method–setting evaluations over three seeds, releasing datasets, implementations, configurations, evaluation scripts, and per-seed results.

Beyond the data and baselines, the paper's real contribution is evaluative: it adds two domain-aware diagnostics and uses them to ask whether benchmark conclusions are robust. SCoRE resolves reconstruction fidelity across frequency bands and local energy regimes; RCnorm measures the geometric continuity of a predicted first-arrival ridge without reference labels. Three findings stand out: synthetic rankings only partially predict field rankings (Kendall τ from 0.24 to 0.73, varying by task); ranking stability under growing interference is task-dependent; and RCnorm agrees with MAE-based model rankings at mean Kendall 0.881 across three field surveys.

Critical strengths
  1. The existing public resources for this domain are mostly about the problems downstream — OpenFWI and E-FWI do full-waveform inversion, CIG-Bench is interpretation. The processing step that actually prepares the data for those, like demultiple or ground-roll suppression, has no agreed benchmark at all. That's the gap SPBENCH addresses, and it's a reasonable one to pick.

  2. The two diagnostics are the real core of the paper, and are well designed. RCnorm has a clean property: it is invariant under rearranging per-trace errors but changes when the ridge re-bends — so it captures geometric information that MAE, RMSE, and H@k cannot. SCoRE uses energy bins as a proxy for weak signals, and the authors are explicit that it is a proxy, not a claim to have found weak events. That calibration is right.

  3. Reproducibility is handled well. Fixed splits, versioned artifacts, per-seed results, and every leaderboard number traceable to a released configuration — 495 configurations (over 1,000 parameter sets) are auditable. This is strong for this line of work.

Critical weaknesses
  1. Baseline fidelity is my main concern. The paper states that its numbers are "not directly comparable" to the original papers — which is normal, since a benchmark re-tunes hyperparameters. But the problem is: when a reproduced method ranks low, I can't tell whether that's because the method is genuinely weak or because the reproduction was imperfect. For a benchmark whose purpose is comparability, this matters. I'd want to know how faithfully each of the 24 methods was reimplemented, which configurations deviate from the source papers, and how far the width-augmented "-L" variants stray from the original architecture. Without that, I can't attribute a method's rank to the method itself.

  2. The boundaries of the strongest diagnostics are only partly explored. RCnorm agreement with MAE rankings is strong at loose tolerances (0.881) but drops to 0.244 at H@1. The paper mentions this but doesn't really dig into when RCnorm becomes misleading. EB-FE is similar: it relies on a clean reference and uses energy-percentile bins as a weak-signal proxy, but it never validates that an improvement in a low-energy bin actually corresponds to recovering a real weak event, not just improving a proxy.

  3. The three tasks where the paper introduces new paired datasets — multiple suppression, deblending, ground roll — are precisely the ones with the fewest settings (multiple: 1, deblending: 4, ground roll: 6), while interpolation (16) and random noise (12) dominate. So the transfer and stability conclusions in Sections 4.2 and 4.3 are strongest for noise/interpolation/deblending and thinnest for multiple, which has a single setting. A reader may miss this asymmetry.

  4. The three headline claims rest on little data. Synthetic-to-field, degradation stability, and RCnorm agreement each depend on a handful of task pairs or a few field surveys (three). Directionally convincing, but the paper is advancing generalizable claims about how seismic-processing models rank, and the evidence base is narrow. More varied field data would either strengthen or revise them.

Questions for authors

The paper reports that RCnorm agrees with MAE rankings overall (mean 0.881), but at the strictest tolerance (H@1) that agreement collapses to 0.244. So I'm trying to understand the boundary: should a practitioner use RCnorm when the tolerance is tight, or does the score become unreliable there? In other words, what's the practical rule for when RCnorm is safe to use and when it isn't?

4 comments
AuthorsOpenReview ↗

Response to reviewer 4vj4 (Part 1/2)

Thank you for your thoughtful review and for recognizing SPBench’s contribution to reproducible prestack processing. In the marked-up manuscript, changes are shown in blue.

We made four main additions.

  • We report training and inference times alongside accuracy and parameter counts.

    See Tables 2–3 and Appendices E (timing) and I (results).

  • We added 10 traditional baselines across all six tasks.

    See Appendix D, Table 7, and Tables 2–3.

  • We added weak-reflection and low-frequency SCoRE examples and tested RCnorm across surveys, including a case of systematic timing bias.

    See Figures H.1–H.2 (SCoRE), Table 9 and Figure H.3 (RCnorm).

  • We applied synthetic-trained models directly to field records in four tasks, without fine-tuning.

    See Section 4.2 and Appendix G, Figure 13(a–c, e).

W1. Fidelity of the reproduced baselines

Thank you for raising this concern. We agree that readers need to know which choices came from the original method before they can interpret a low rank. For the 24 supervised methods, we followed the architectures and network hyperparameters specified in the source papers. Because preprocessing depends on the seismic data, we used a shared protocol for each task and dataset. Within each setting, methods use the same splits, input–target pairs, and scoring scripts.

We report our capacity experiments separately as “-L.” For U-Net, ResUNet, and Attention U-Net, increasing the base width from 32 to 64 doubles the hidden channels at each scale and preserves the stages and modules. DnCNN-L likewise uses 64 instead of 32 hidden channels, with 17 layers in both versions. For each main-leaderboard entry, readers can inspect the configuration, including normalization and training schedule, and the individual seed results. Parameter counts alone tell readers little about runtime. We report training time per run and GPU inference time over the full test set so readers can weigh accuracy against the cost of training and applying each model.

We attempted to reproduce more methods than we ultimately included. In some cases, key settings were unclear in the source paper and our runs yielded markedly low scores. We felt it would be unfair to present those scores as representative of the original method. Each included configuration provides a reproducible performance reference on SPBench.

See Section 4.1 and Appendix D (methods and variants), Appendix E (protocol), and Tables 2–3 (costs).

AuthorsOpenReview ↗

Response to reviewer 4vj4 (Part 2/2)

W2 and Q1. Validating RCnorm and SCoRE

Thank you for asking where these diagnostics are most useful and where they can be misleading. We added a blind model-selection experiment for RCnorm and two waveform comparisons for SCoRE. The first tests whether we can choose a picker on a new survey before seeing its reference picks. The other two let readers check whether a better weak-energy or frequency-band score corresponds to a visible difference in the recovered signal.

RCnorm on a new survey. For each of the six transfer directions among Brunswick, Halfmile, and Lalor, we applied seven source-trained pickers to the target records. We chose the model with the lowest RCnorm using only its predictions. We then used the target reference picks to ask how close that choice came to the model with the lowest MAE. RCnorm selected the best model in four directions and the runner-up in a fifth, only 0.79 ms above the best.

The transfer from Lalor to Halfmile showed how this rule can fail. RCnorm selected DnCNN-Seg, which had the largest MAE. We show its predictions alongside STUNet’s and the reference picks on the same gather. DnCNN-Seg picks nearly the same early arrival time across the traces. This gives it a low curvature score even where the reference arrival changes substantially. STUNet follows that change and has the lowest MAE on this transfer.

See Section 4.4 and Appendix H.3, especially Table 9 (blind selection), Figure H.3 (failure case), and Table 10 (timing tolerances).

For the tight-tolerance application you ask about, we would use RCnorm to screen candidates for continuity. We would then check the shortlisted models against a small set of reference picks at the required tolerance. A uniform shift in arrival time leaves curvature unchanged, so even a very smooth ridge can carry a systematic timing error.

SCoRE and waveform recovery. To address your question about weak reflections, we chose a field example where overall SNR and the weak-energy score favored different models. On Mobil AVO with 50% random trace loss, Attention U-Net-L ranked first overall, while CA-Unet ranked first in the weak-energy bin. Since the traces were recorded before we masked them, we could check which model better recovered the observed reflection. We selected window W3, which contains a weak reflection and lies entirely in the low-energy region. The enlarged trace segment and its residuals show that CA-Unet followed the recorded amplitude more closely. This ties the weak-bin result to a particular reflection that readers can inspect.

We made a corresponding comparison for the frequency bands in moderate deblending. SCRN had the highest overall SNR, while cDDPM scored highest in the lowest frequency band. We extracted that band from the clean reference and both reconstructions, then displayed the waveforms and their residuals on a common amplitude scale. cDDPM left less residual energy in that band, consistent with its higher low-band score. These examples show how to use SCoRE when preserving a weak event or a particular band matters to the intended application.

See Section 3.2 and Appendix H, Figures H.1 (weak reflection and trace residuals) and H.2 (low-band reconstructions and residuals).

W3. Uneven coverage across tasks

Thank you for pointing this out. We added a rugged-seafloor simulation for multiples and a field survey for deblending, ground roll, and random noise. Multiples, deblending, and ground roll now have two, five, and seven settings, respectively.

One shot record can yield several settings by changing the noise level or the missing-trace mask. Multiples arise when seismic waves reflect more than once. With acquisition and modeling conditions fixed, each velocity model yields one set of records containing multiples, paired with the corresponding primary references. The original velocity model therefore gave us one setting. The new rugged-seafloor model gives us a second.

See Section 3.1 and Tables 23 (noise), 46 (ground roll), 48 (multiples), and 53 (deblending).

W4. Do the findings hold on additional field data?

Thank you for encouraging us to test this on more field data. We added a second field survey for random noise, ground roll, and deblending. Within each task, we compared the synthetic ranking with each field ranking using Kendall’s τb\tau_b for the same nine methods, trained separately on each dataset. Agreement varied across tasks and surveys. We also tested synthetic-trained models directly on field records without fine-tuning. In all four tasks with field targets, the model that led on synthetic data lost first place after transfer. Both analyses support our original conclusion that synthetic rankings alone do not reliably predict which model will perform best on a field survey.

See Section 4.2, Figure 3, and Appendix G, Figure 13(a–c, e).

Reviewer_4vj4OpenReview ↗

response

Thank you for a thorough and well-organized response. I have verified the additions against the manuscript, and my principal concerns have been addressed.

Thank you for your constructive suggestions and positive feedback on our responses. Your input has helped us significantly improve the quality of our manuscript.

Reviewer_U86QOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

this work presents a well structured multi-task evaluation benchmark for seismic processing. The authors have implemented several supervised methods on a variety of already available datasets, and have evaluated them on different processing tasks, including random noise attenuation, trace interpolation, ground-roll noise supression, multiple suppression, deblending and first-arrival picking. The authors also provide two domain-speciic evaluation diagnostics score and rcnorm. This work is a value tool for the community, as it provides a comprehensive evaluation of the performance of different processing methods on a variety of tasks, and can be used to guide the development of new processing methods.

Critical strengths
  • The paper and the benchmark are well structured and precise in the description of the tasks and the methods used. The multi-task nature of the benchmark is a strong point, as it allows for a comprehensive evaluation of the performance of different processing methods on a variety of tasks. Also, the big effort in reproducibility is a strong point.
Critical weaknesses
  • The authors states that hyperparameters follow the original papers, rather than being optimized under a common tuning budget. This might limits fairness of the comparisons.
Questions for authors
  • Following up on the weakness identified above, it would be interesting to know how the how do the authors ensure fair comparisons across methods when hyperparameters are taken from the original papers?
  • The authors jave acknowledge that baseline comparison is missing currently and that it will be added in the future. I believe that in addition to a comparison between deep learning methods, a comparison between traditional methods and deep learning methods would be valuable.
  • Minor question: The datasets are properly described in the appendix, but it would be useful to add a brief description of them in the main text.
3 comments
AuthorsOpenReview ↗

Response to reviewer U86Q

Thank you for your thoughtful review and for recognizing the care that went into making SPBench reproducible across six tasks. We appreciate your suggestions on comparison fairness, traditional baselines, and dataset descriptions. In the marked-up version of the revised manuscript, the changes are shown in blue.

We made four main additions in the revision.

  • We report training and inference times alongside parameter counts and performance, so readers can compare the resources each model needed to achieve its reported result.

    See Tables 2–3 and Appendices E (timing) and I (results).

  • We added 10 traditional processing methods as baselines, covering all six tasks.

    See Appendix D, Table 7, and Tables 2–3.

  • We added two SCoRE case studies showing how models differ in preserving the amplitude of a weak reflection and in recovering low-frequency signals. We also tested RCnorm for model selection across six survey-to-survey transfers and examined the case where it favored smooth but systematically early picks.

    See Figures H.1–H.2 (SCoRE), Table 9 (RCnorm selection), and Figure H.3 (failure case).

  • We tested synthetic-trained models directly on field records for random-noise attenuation, interpolation, ground-roll suppression, and deblending, without retraining or fine-tuning. In all four comparisons, the model with the highest synthetic SNR lost first place after transfer.

    See Section 4.2 and Appendix G, Figure 13(a–c, e).

W1 and Q1. Fairness when using published hyperparameters

We appreciate this observation. We agree that hyperparameter choices can affect a method’s rank, and that readers need to know how those choices were made. Our aim was to provide a reproducible starting point for someone applying these methods to the same seismic data. We therefore followed the architectures and network hyperparameters specified in the source papers and labeled our wider variants separately as “-L.” Within each setting, the methods share the same data splits, preprocessing protocol, and evaluation procedure.

A common tuning budget would help make the search effort more comparable. One consideration for us was that architectures differ in the resources they need to perform well. DnCNN, for example, performs well on random-noise attenuation with relatively few parameters. Training time also varies across models, so the same GPU budget can allow different numbers of trials. We used the published settings as a documented basis for comparison and interpret the rankings in that context. Your comment also prompted us to make the cost of achieving each result clearer. We now report measured training time per run and GPU inference time over the full test set alongside performance and parameter counts, so readers can weigh the accuracy they need against the resources available to them.

In building SPBench, we tried to reproduce more methods than we ultimately included. In some cases, key settings were unclear in the source paper and our runs yielded markedly low scores. We felt it would be unfair to present those scores as representative of the original methods, so we left those reproductions out. For the methods we report, readers can inspect the preprocessing code and parameter files and use those configurations for their own comparisons.

See Section 4.1 and Appendix D (configurations), Appendix E (timing), and Tables 2–3 and Appendix I (results).

Q2. Comparison with traditional processing methods

Following your suggestion, we evaluated the traditional baselines on the same test sets as the learned methods. For random-noise attenuation we included f–x deconvolution and local DRR, for interpolation MSSA and curvelet reconstruction, and for ground-roll suppression PWD and adaptive STFT filtering. For multiple suppression we used SRME with adaptive subtraction, for deblending multidirectional vector-median filtering, and for first-arrival picking STA/LTA and the modified energy ratio (MER). On Lalor, for example, STA/LTA had a mean absolute picking error of 7.18 samples, compared with 2.27 for STUNet and 13.18 for Attention U-Net.

See Tables 2–3 (Table 3 for Lalor), Appendix D, Table 7 (methods and references), and Appendix I (full results).

Q3. Brief dataset descriptions in the main text

We now introduce the datasets alongside the tasks that use them. For example, the interpolation paragraph identifies SEG C3 as synthetic data from the SEG/EAGE salt model and Mobil AVO as marine field data from the North Sea. We also explain how the ground-roll-suppressed field records serve as references for adding random noise and constructing blended inputs, and identify Brunswick, Halfmile, and Lalor as the three field surveys with annotated first arrivals.

See Section 3.1 (task and dataset descriptions) and Appendix C (acquisition and construction).

Thank you for taking the time in addressing my comments. My concerns have been resolved, and I will update my score upward.

Thank you for your constructive feedback and positive reassessment of our work.

Reviewer_Zm51OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors focused on the task of deep-learning-based exploration seismic processing and introduced a multi-task benchmark named SPBench which covers six representative tasks. The authors provided 10 datasets, 43 benchmark settings, and standardized implementations of 24 supervised learning methods, resulting in 495 method-setting evaluations. In addition, the authors proposed two domain-aware diagnostics which are SCoRE for reconstruction fidelity evaluation and RCnorm for first-arrival picking. The main findings indicate that model rankings can differ substantially between synthetic and field settings, that rankings are less stable under coherent ground-roll interference than random-like noise, and that the proposed RCnorm metric correlates with conventional picking accuracy rankings on the evaluated field datasets.

Critical strengths

1. Extensive benchmarking and reproducibility efforts

The authors have invested substantial effort in reproducing 24 existing methods across six seismic-processing tasks under standardized experimental protocols. The reported release of datasets, model implementations, configurations, and evaluation scripts could provide a useful resource for the community and reduce the difficulty of comparing future approaches.

2. Consideration of domain-specific evaluation and robustness

The proposed metrics aim to capture aspects of seismic data quality that conventional aggregate metrics may overlook, such as frequency-dependent reconstruction fidelity, weak-signal preservation, and spatial continuity of first arrivals. The analysis of model ranking stability across datasets and degradation strengths also provides useful evidence that model performance is sensitive to evaluation conditions.

Critical weaknesses

1. Limited novelty and unclear practical significance of the experimental findings

Despite the extensive experiments, the main conclusions are largely descriptive and somewhat expected. For example, differences in model rankings between synthetic and field settings, as well as variations in robustness across different noise types, are broadly consistent with known challenges in domain generalization and seismic processing. More importantly, the paper does not sufficiently demonstrate how these observations should influence future model development or selection. While SCoRE and RCnorm provide additional diagnostic information, their practical advantages over conventional evaluation methods are not convincingly established through downstream applications or model-selection experiments. Consequently, the scientific insights gained from the extensive benchmarking remain somewhat limited.

2. Uneven benchmark coverage and limited evaluation of generalization

Although SPBench covers six tasks, the distribution of experimental settings is highly uneven. Interpolation and random noise attenuation account for 28 of the 43 settings, while multiple suppression is represented by only one. The benchmark also focuses on supervised learning with 2D seismic gathers, without systematically including conventional physics-based processing approaches, self-supervised methods, or recent foundation models. Furthermore, the synthetic-versus-field comparisons train and evaluate models separately in each setting, meaning that the reported ranking differences do not directly measure cross-domain generalization. The reliance on limited field datasets and, in some tasks, constructed or expert-processed reference signals also restricts the generality of the conclusions.

Questions for authors

1. Practical significance of the findings

Beyond demonstrating that model rankings vary across datasets and degradation settings, what specific recommendations can be derived from SPBench to guide future seismic-processing research? Could the authors provide concrete examples showing how the benchmark findings lead to better model selection or more robust model development?

2. Utility of the proposed evaluation metrics

Could the authors further demonstrate the practical advantages of SCoRE and RCnorm over existing evaluation metrics? For instance, does selecting models based on SCoRE lead to better preservation of geologically meaningful structures or improved downstream imaging and inversion? For RCnorm, how robust is the metric when predicted first arrivals are smooth but systematically inaccurate? Additional validation would help establish the usefulness of these diagnostics beyond the reported ranking correlations.

3. Cross-domain generalization

Since the synthetic and field experiments independently train models within each setting, the results primarily characterize ranking consistency rather than actual synthetic-to-field generalization. Could the authors evaluate models trained on synthetic data and directly tested on field data without retraining?

4. Benchmark completeness and representativeness

What motivates the substantial imbalance in the number of evaluation settings across the six tasks? Could the authors expand the evaluation of underrepresented tasks, particularly multiple suppression, or incorporate representative conventional processing and self-supervised methods?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Zm51 (1/2)

Response to Reviewer Zm51 (1/2)

Thank you for recognizing the effort behind the benchmark and for asking how its findings would change a researcher's choice of model. We have marked the changes in blue in the revised manuscript.

  • We now report training and inference times alongside performance and parameter counts, so readers can see what it cost to achieve each result. See Tables 2–3 and Appendices E and I.
  • We added 10 traditional baselines across all six tasks. See Appendix D, Table 7, and Tables 2–3.
  • We tested RCnorm for model selection without target labels and added waveform comparisons for weak reflections and low-frequency recovery. See Section 4.4, Table 9, and Figures H.1–H.3.
  • We added a field survey for three tasks, a rugged-seafloor multiple-suppression setting, and direct-transfer tests with fixed source weights. See Section 4.2, Figure 13, and Tables 23, 46, 48, and 53.

W1, Q1 and Q2. How can SPBench guide model selection and development?

Thank you for encouraging us to go beyond reporting that rankings change. We added experiments in which a researcher has to make a choice, and we checked that choice against an observable outcome.

Choosing a picker on an unlabeled survey. For each of the six transfer directions among Brunswick, Halfmile, and Lalor, we applied seven source-trained pickers to a target survey and selected the one with the lowest RCnorm using only their predictions. Only then did we use the target reference picks to see which model had the lowest mean absolute error (MAE). RCnorm selected the best model in four directions and the runner-up in a fifth, just 0.79 ms above the best.

The sixth direction shows precisely the failure you asked about. On Lalor to Halfmile, RCnorm favored DnCNN-Seg, whose picks were nearly flat and systematically early. STUNet followed the reference arrivals more closely and had the lowest MAE. We added a side-by-side view of the same gather and examined a uniform shift of the reference picks. Moving every pick 10 ms earlier leaves curvature unchanged. In practice, we would use RCnorm to shortlist models for continuity on an unlabeled survey, then check a few reference picks for timing bias when absolute arrival time matters.

See Section 4.4 and Appendix H.3, particularly Table 9 (blind selection) and Figure H.3 (failure case).

Choosing a model for the signal of interest. On Mobil AVO with 50% random trace loss, overall SNR favored Attention U-Net-L, while SCoRE's weak-energy score favored CA-Unet. We checked which choice better preserved a weak reflection by returning to the field trace recorded before masking. Figure H.1 marks the reflection in window W3 and shows both reconstructed waveforms and their residuals. CA-Unet followed the recorded amplitude more closely. Across all seven mask settings shared by these models, CA-Unet also had the higher weak-energy score, while Attention U-Net-L led overall in six.

For moderate deblending, we isolated the lowest frequency band in the clean reference and both model outputs. SCRN led overall, but cDDPM left less residual energy in that band. The choice also has a measurable cost. Processing the test set took 13 s of GPU inference for SCRN and 815 s for cDDPM. If low frequencies are especially important, as they can be in the early stages of multiscale full-waveform inversion, these results give a reason to include cDDPM in target-survey validation. Our experiment measures the fidelity of the waveforms that would enter such an inversion.

See Section 3.2, Figures H.1–H.2, and Table 2 (inference times).

Practical implications. On a new survey without manual picks, RCnorm gives researchers a way to shortlist models whose predicted arrivals are continuous across traces. If absolute timing matters, a few reference picks can reveal a systematic shift that curvature alone would miss. For reconstruction, the model with the highest overall SNR can still lose amplitude in a weak reflection or leave more error in a frequency band of interest. On representative validation records with a reference, SCoRE shows where those differences occur. Researchers can use it to check whether a new method improves the signal their application relies on, then weigh that gain against inference time when choosing a model.

See Sections 3.2 and 4.4, Appendix H (Figures H.1–H.3 and Table 9), and Table 2 (inference times).

AuthorsOpenReview ↗

Response to Reviewer Zm51 (2/2)

Response to Reviewer Zm51 (2/2)

W2 and Q3. Does a model trained on synthetic data generalize directly to field records?

Thank you for making this distinction. In the original comparison, each model was trained and tested within each dataset. This tells us whether an architecture that performs well on synthetic data also performs well when trained on field data. To test the deployment scenario you describe, we added an experiment that applies the synthetic-trained checkpoints directly to target test records, keeping the source weights fixed.

We tested four tasks with field targets. For random-noise attenuation and interpolation we transferred SEG C3 models to Mobil AVO. For ground-roll suppression we applied the models trained at synthetic level L5 to the original field survey. For deblending we applied models trained on moderate synthetic blending to the new field dataset. We compared the same candidates before and after each transfer. The synthetic winner lost first place in all four tasks. Agreement also varied by task. For example, Kendall’s τb\tau_b was 0.60 for ground roll and 0.00 for interpolation with 50% random trace loss.

Deblending illustrates how this affects a shortlist. When models were trained within each dataset, SCRN led the nine methods common to the synthetic and both field settings. In the direct-transfer test on the held-out field shot, its synthetic checkpoint ranked eighth among ten candidates. UNet++, sixth on the source test, ranked first on this shot. Selecting only the synthetic top three would have excluded the best candidate on the target record. For deployment without retraining, this supports taking several candidates forward and comparing them on representative target gathers.

We also evaluated random-noise attenuation, ground-roll suppression, and deblending on an additional field dataset. These results support our original conclusion that synthetic rankings alone do not reliably predict model rankings on field data.

See Section 4.2, Figure 3(a, c, d) and Tables 23, 46, and 53 (additional field experiments), and Appendix G, Figure 13(a–c, e) (direct transfer).

W2 and Q4. Why are the tasks unevenly represented, and which baselines were added?

Thank you for highlighting multiple suppression in particular. We added a second synthetic experiment with rugged seafloor topography, as well as one shared field survey for random noise, ground roll, and deblending. The revised benchmark covers 12 datasets and 47 settings.

The difference in setting counts follows from how the records are constructed. One shot record can yield several noise or interpolation settings simply by changing the noise level or the missing-trace mask. Multiples arise when seismic waves reflect more than once. With acquisition and forward-modeling conditions fixed, each velocity model yields one set of records containing multiples, paired with corresponding primary references. The original velocity model gave us one setting and the new seafloor model gives us a second. We evaluated models within the new setting and also tested checkpoints from the original one. Across the nine shared methods, agreement between the source and direct-transfer rankings was weak (Kendall’s τb=0.11\tau_b = 0.11).

See Section 3.1, Tables 23, 46, 48, and 53 (new settings), and Appendix G, Figure 13d (transfer between multiple-suppression scenarios).

Following your suggestion, we added 10 traditional methods covering all six tasks and evaluated them on the same test sets as the learned methods. For example, the multiple-suppression tables now include surface-related multiple elimination (SRME) with adaptive subtraction. The comparison with learning-based approaches can thus be made directly on the same task and records.

The learning-based part of SPBench focuses on supervised methods. We appreciate your suggestion to include self-supervised approaches. Their performance can depend substantially on the survey data available for adaptation and on task-specific tuning, both of which would need to be specified for a reproducible comparison. For foundation models, we have yet to identify public pretrained weights suited to the 2D prestack processing tasks evaluated here. We see both as valuable extensions.

See Section 4.1, Tables 2–3 (representative comparisons), Appendix D, Table 7 (traditional methods and references), and Appendix I (results by setting).

Dear Reviewer,

Thank you again for your valuable suggestions, which have helped us improve our manuscript. Could you please let us know whether our rebuttal has adequately addressed your concerns?

Thank you for the substantial revision. My major concerns were addressed. I will update my recommendation for weak acceptance.

Comments on the paper

5 comments
AuthorsOpenReview ↗

Response to reviewer U86Q

Thank you for your thoughtful review and for recognizing the care that went into making SPBench reproducible across six tasks. We appreciate your suggestions on comparison fairness, traditional baselines, and dataset descriptions. In the marked-up version of the revised manuscript, the changes are shown in blue.

We made four main additions in the revision.

  • We report training and inference times alongside parameter counts and performance, so readers can compare the resources each model needed to achieve its reported result.

    See Tables 2–3 and Appendices E (timing) and I (results).

  • We added 10 traditional processing methods as baselines, covering all six tasks.

    See Appendix D, Table 7, and Tables 2–3.

  • We added two SCoRE case studies showing how models differ in preserving the amplitude of a weak reflection and in recovering low-frequency signals. We also tested RCnorm for model selection across six survey-to-survey transfers and examined the case where it favored smooth but systematically early picks.

    See Figures H.1–H.2 (SCoRE), Table 9 (RCnorm selection), and Figure H.3 (failure case).

  • We tested synthetic-trained models directly on field records for random-noise attenuation, interpolation, ground-roll suppression, and deblending, without retraining or fine-tuning. In all four comparisons, the model with the highest synthetic SNR lost first place after transfer.

    See Section 4.2 and Appendix G, Figure 13(a–c, e).

W1 and Q1. Fairness when using published hyperparameters

We appreciate this observation. We agree that hyperparameter choices can affect a method’s rank, and that readers need to know how those choices were made. Our aim was to provide a reproducible starting point for someone applying these methods to the same seismic data. We therefore followed the architectures and network hyperparameters specified in the source papers and labeled our wider variants separately as “-L.” Within each setting, the methods share the same data splits, preprocessing protocol, and evaluation procedure.

A common tuning budget would help make the search effort more comparable. One consideration for us was that architectures differ in the resources they need to perform well. DnCNN, for example, performs well on random-noise attenuation with relatively few parameters. Training time also varies across models, so the same GPU budget can allow different numbers of trials. We used the published settings as a documented basis for comparison and interpret the rankings in that context. Your comment also prompted us to make the cost of achieving each result clearer. We now report measured training time per run and GPU inference time over the full test set alongside performance and parameter counts, so readers can weigh the accuracy they need against the resources available to them.

In building SPBench, we tried to reproduce more methods than we ultimately included. In some cases, key settings were unclear in the source paper and our runs yielded markedly low scores. We felt it would be unfair to present those scores as representative of the original methods, so we left those reproductions out. For the methods we report, readers can inspect the preprocessing code and parameter files and use those configurations for their own comparisons.

See Section 4.1 and Appendix D (configurations), Appendix E (timing), and Tables 2–3 and Appendix I (results).

Q2. Comparison with traditional processing methods

Following your suggestion, we evaluated the traditional baselines on the same test sets as the learned methods. For random-noise attenuation we included f–x deconvolution and local DRR, for interpolation MSSA and curvelet reconstruction, and for ground-roll suppression PWD and adaptive STFT filtering. For multiple suppression we used SRME with adaptive subtraction, for deblending multidirectional vector-median filtering, and for first-arrival picking STA/LTA and the modified energy ratio (MER). On Lalor, for example, STA/LTA had a mean absolute picking error of 7.18 samples, compared with 2.27 for STUNet and 13.18 for Attention U-Net.

See Tables 2–3 (Table 3 for Lalor), Appendix D, Table 7 (methods and references), and Appendix I (full results).

Q3. Brief dataset descriptions in the main text

We now introduce the datasets alongside the tasks that use them. For example, the interpolation paragraph identifies SEG C3 as synthetic data from the SEG/EAGE salt model and Mobil AVO as marine field data from the North Sea. We also explain how the ground-roll-suppressed field records serve as references for adding random noise and constructing blended inputs, and identify Brunswick, Halfmile, and Lalor as the three field surveys with annotated first arrivals.

See Section 3.1 (task and dataset descriptions) and Appendix C (acquisition and construction).

AuthorsOpenReview ↗

Response to reviewer 4vj4 (Part 1/2)

Thank you for your thoughtful review and for recognizing SPBench’s contribution to reproducible prestack processing. In the marked-up manuscript, changes are shown in blue.

We made four main additions.

  • We report training and inference times alongside accuracy and parameter counts.

    See Tables 2–3 and Appendices E (timing) and I (results).

  • We added 10 traditional baselines across all six tasks.

    See Appendix D, Table 7, and Tables 2–3.

  • We added weak-reflection and low-frequency SCoRE examples and tested RCnorm across surveys, including a case of systematic timing bias.

    See Figures H.1–H.2 (SCoRE), Table 9 and Figure H.3 (RCnorm).

  • We applied synthetic-trained models directly to field records in four tasks, without fine-tuning.

    See Section 4.2 and Appendix G, Figure 13(a–c, e).

W1. Fidelity of the reproduced baselines

Thank you for raising this concern. We agree that readers need to know which choices came from the original method before they can interpret a low rank. For the 24 supervised methods, we followed the architectures and network hyperparameters specified in the source papers. Because preprocessing depends on the seismic data, we used a shared protocol for each task and dataset. Within each setting, methods use the same splits, input–target pairs, and scoring scripts.

We report our capacity experiments separately as “-L.” For U-Net, ResUNet, and Attention U-Net, increasing the base width from 32 to 64 doubles the hidden channels at each scale and preserves the stages and modules. DnCNN-L likewise uses 64 instead of 32 hidden channels, with 17 layers in both versions. For each main-leaderboard entry, readers can inspect the configuration, including normalization and training schedule, and the individual seed results. Parameter counts alone tell readers little about runtime. We report training time per run and GPU inference time over the full test set so readers can weigh accuracy against the cost of training and applying each model.

We attempted to reproduce more methods than we ultimately included. In some cases, key settings were unclear in the source paper and our runs yielded markedly low scores. We felt it would be unfair to present those scores as representative of the original method. Each included configuration provides a reproducible performance reference on SPBench.

See Section 4.1 and Appendix D (methods and variants), Appendix E (protocol), and Tables 2–3 (costs).

AuthorsOpenReview ↗

Response to reviewer 4vj4 (Part 2/2)

W2 and Q1. Validating RCnorm and SCoRE

Thank you for asking where these diagnostics are most useful and where they can be misleading. We added a blind model-selection experiment for RCnorm and two waveform comparisons for SCoRE. The first tests whether we can choose a picker on a new survey before seeing its reference picks. The other two let readers check whether a better weak-energy or frequency-band score corresponds to a visible difference in the recovered signal.

RCnorm on a new survey. For each of the six transfer directions among Brunswick, Halfmile, and Lalor, we applied seven source-trained pickers to the target records. We chose the model with the lowest RCnorm using only its predictions. We then used the target reference picks to ask how close that choice came to the model with the lowest MAE. RCnorm selected the best model in four directions and the runner-up in a fifth, only 0.79 ms above the best.

The transfer from Lalor to Halfmile showed how this rule can fail. RCnorm selected DnCNN-Seg, which had the largest MAE. We show its predictions alongside STUNet’s and the reference picks on the same gather. DnCNN-Seg picks nearly the same early arrival time across the traces. This gives it a low curvature score even where the reference arrival changes substantially. STUNet follows that change and has the lowest MAE on this transfer.

See Section 4.4 and Appendix H.3, especially Table 9 (blind selection), Figure H.3 (failure case), and Table 10 (timing tolerances).

For the tight-tolerance application you ask about, we would use RCnorm to screen candidates for continuity. We would then check the shortlisted models against a small set of reference picks at the required tolerance. A uniform shift in arrival time leaves curvature unchanged, so even a very smooth ridge can carry a systematic timing error.

SCoRE and waveform recovery. To address your question about weak reflections, we chose a field example where overall SNR and the weak-energy score favored different models. On Mobil AVO with 50% random trace loss, Attention U-Net-L ranked first overall, while CA-Unet ranked first in the weak-energy bin. Since the traces were recorded before we masked them, we could check which model better recovered the observed reflection. We selected window W3, which contains a weak reflection and lies entirely in the low-energy region. The enlarged trace segment and its residuals show that CA-Unet followed the recorded amplitude more closely. This ties the weak-bin result to a particular reflection that readers can inspect.

We made a corresponding comparison for the frequency bands in moderate deblending. SCRN had the highest overall SNR, while cDDPM scored highest in the lowest frequency band. We extracted that band from the clean reference and both reconstructions, then displayed the waveforms and their residuals on a common amplitude scale. cDDPM left less residual energy in that band, consistent with its higher low-band score. These examples show how to use SCoRE when preserving a weak event or a particular band matters to the intended application.

See Section 3.2 and Appendix H, Figures H.1 (weak reflection and trace residuals) and H.2 (low-band reconstructions and residuals).

W3. Uneven coverage across tasks

Thank you for pointing this out. We added a rugged-seafloor simulation for multiples and a field survey for deblending, ground roll, and random noise. Multiples, deblending, and ground roll now have two, five, and seven settings, respectively.

One shot record can yield several settings by changing the noise level or the missing-trace mask. Multiples arise when seismic waves reflect more than once. With acquisition and modeling conditions fixed, each velocity model yields one set of records containing multiples, paired with the corresponding primary references. The original velocity model therefore gave us one setting. The new rugged-seafloor model gives us a second.

See Section 3.1 and Tables 23 (noise), 46 (ground roll), 48 (multiples), and 53 (deblending).

W4. Do the findings hold on additional field data?

Thank you for encouraging us to test this on more field data. We added a second field survey for random noise, ground roll, and deblending. Within each task, we compared the synthetic ranking with each field ranking using Kendall’s τb\tau_b for the same nine methods, trained separately on each dataset. Agreement varied across tasks and surveys. We also tested synthetic-trained models directly on field records without fine-tuning. In all four tasks with field targets, the model that led on synthetic data lost first place after transfer. Both analyses support our original conclusion that synthetic rankings alone do not reliably predict which model will perform best on a field survey.

See Section 4.2, Figure 3, and Appendix G, Figure 13(a–c, e).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.