acceptodds
Under review as a conference paper at ICLR 2027

SeisAgentBench: Benchmarking and Reinforcement Learning for Seismic Data-Processing Agents

Abstract

Seismic data processing is a realistic scientific workflow in which experts assemble and revise multi-stage pipelines over wavefield data. Unlike isolated prediction tasks, success depends on a sequence of data-dependent choices and on the scientific quality of the resulting artifact. Existing agent benchmarks seldom combine executable, multi-turn interaction with task-specific evaluation of scientific data products. We introduce **SeisAgentBench**, to our knowledge the first executable benchmark designed for language agents to construct and evaluate seismic artifacts through multi-step processing, together with **SeisAgent**, an LLM policy specialized for this setting through trajectory-level supervised learning and group-relative reinforcement-learning post-training. SeisAgentBench contains 95 tasks across 19 processing families and evaluates outputs using deterministic, family-specific seismic diagnostics, compatible reference agreement, artifact-preservation checks, and workflow cost. Together, SeisAgentBench and SeisAgent provide a controlled testbed for planning, numerical parameter selection, feedback-driven revision, and scientific artifact evaluation.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, earth science, geophysics, seismic processing, LLM post-training, LLM agent, reinforcement-learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_AKDXOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SeisAgentBench, an executable benchmark containing 95 seismic data-processing tasks across 19 families. Agents interact with the environment by selecting processing operations, adjusting parameters, evaluating intermediate results, and revising their workflows. The benchmark evaluates both task completion and scientific artifact quality using domain-specific metrics, reference agreement, and signal-preservation checks. The authors also propose SeisAgent, which combines a small SFT warm start with group-relative RL using scientific feedback. Experiments across six language models show improvements in task completion and scientific quality.

Critical strengths
  • The paper evaluates scientific artifact quality and signal preservation, rather than just task completion, making it more relevant to real scientific workflows.
  • The experiments cover six models across different model families, with SFT and RL comparisons. The improvements provide evidence for the usefulness of the proposed benchmark and training framework.
Critical weaknesses
  • The out-of-distribution generalization is not sufficiently evaluated: Although the test tasks are held out, they come from the same processing families as the training tasks. The authors can evaluate whether the method generalizes to unseen task families or independent seismic datasets.

  • The iterative behavior is somewhat artificially enforced: In real-world scenarios, agents should decide when to stop experimenting itself. It will be meaningful to see results without the minimum candidate and revision requirements (for iterative tasks), to better evaluate whether RL actually improves feedback-driven decision-making in long horizon tasks.

  • The evaluation could benefit from more expert assessment and behavioral analysis. Since the benchmark targets scientific workflows, the authors could consider including blinded evaluations by domain experts comparing outputs before and after training, along with a more detailed analysis of the trained agents' failure cases.

  • The contribution of different RL reward components is unclear. The reward combines execution success, scientific quality, artifact preservation, and interaction costs, but the paper does not isolate their individual effects. The authors should conduct reward ablations to clarify which components drive the improvements in scientific quality.

Questions for authors

Please see the weaknesses section above

10 comments
AuthorsOpenReview ↗

Response to Reviewer AKDX - Part I

We appreciate the insightful comments from the reviewer. We have revised the paper accordingly (shown in red) and provide the responses as below. Particularly, we've included new field data experiments and blinded expert study.

W1. Generalization to new field-data inputs

We have added a frozen-policy evaluation on five new field-data processing cases, without additional field-specific training or checkpoint selection in Appendix H.7 in the revision. This extends the evaluation beyond the original benchmark inputs to recorded waveforms with different sampling, acquisition geometry, and processing scale.

  • The new data are practically relevant because they contain measured signals and physical acquisition structure. The source collection includes Vibroseis Line 001 [1], Taranaki [2], F3 [3], and Teapot Dome [4], representing field-data resources associated with land acquisition, hydrocarbon exploration, and reservoir imaging. Unlike additional synthetic variants, these inputs introduce recorded amplitudes and arrival patterns whose useful information must be preserved during processing.

    The largest case contains 251 shots, 70,782 traces, and 106.2 million samples, with 2-ms sampling and a 12.5-km shot-position span. Relative to its original held-out counterpart, the sample count is approximately 443 times larger and the shot-position range is 26 times wider. The new inputs therefore change both the signal content and the physical scale at which processing decisions must operate.

  • Both original and trained Qwen3-14B policies produce the requested processing outputs without adaptation to these field cases, shown in Figure 8 in the revision. The evaluation covers amplitude balancing, early-arrival suppression, geometry reorganization, moveout correction, and depth imaging. The added figure presents inputs, policy outputs, and supplied references, accompanied by physical diagnostics. The outputs show improved amplitude balance and reflection alignment while retaining waveform character and RMS amplitude. Geometry conversion preserves the complete trace set, and depth-imaging outputs agree with the supplied target. The trained policy additionally improves amplitude balance and moveout coherence relative to zero-shot.

These results provide direct evidence of transfer to newly introduced recorded inputs and larger physical scales. We have added the source description, acquisition characteristics, frozen-policy setup, and qualitative comparisons to Appendix H.7 in the revision. To help reviewers better understand the benchmark, we provide an anonymized data repository for inspection during the review period.

W2. Are the iterative revision requirements artificially enforced?

The revision requirements are necessary, as they reflect established seismic-processing practice and ensure that the benchmark explicitly evaluates candidate assessment and improvement. To demonstrate this, we conduct a free-stopping ablation that removes the minimum revision requirements. The results show that agents terminate prematurely and produce substantially weaker artifacts, even when artifact quality is evaluated independently of workflow compliance. We explain the scientific motivation and present the experimental evidence below, with full details in Appendix H.4 of the revised paper.

  • The revision requirements reflect established seismic-processing practice, not arbitrary constraints [6-7]. As discussed in Section 2.1 of the original submission, seismic processing commonly follows an iterative quality-control procedure: apply an operation, inspect the resulting data, identify remaining defects, and adjust the processing parameters. For example, NMO correction requires inspecting event flatness, stretch, and amplitude preservation before refining velocity estimates; filtering requires balancing noise suppression against reflection preservation. A successfully executed operation is not necessarily an acceptable scientific result. Accordingly, we require candidate assessment and revision only for the nine parameter-sensitive families, while the other ten families have no such requirement. Importantly, the protocol does not prescribe operation sequences, parameter values, revision strategies, or final candidate selection. It specifies the scientific capability to be evaluated, not how the agent should solve the task.
AuthorsOpenReview ↗

Response to Reviewer AKDX - Part II

  • Our new ablation confirms that structured revision improves the resulting artifacts, not merely protocol compliance. We compare the same policy weights on the nine held-out iterative tasks, with and without minimum revision requirements. Inputs, tools, maximum interaction budgets, and the hidden-reference evaluator remain fixed.

    Crucially, we measure the ungated reference-agreement score, defined as the mean of qrefq_{\mathrm{ref}} in Appendix E.2 in the original paper, computed on each agent's last nominated artifact. Thus, the comparison measures differences in the produced artifacts rather than rewarding compliance with the revision requirements.

    ModelFree stoppingStructured revisionCandidates (free → structured)Steps (free → structured)
    Qwen3-14B0.54640.66091.00 → 3.113.22 → 11.22
    Qwen3-32B0.57730.64331.00 → 2.442.89 → 9.00

    Structured revision improves mean reference agreement by 21.0% on Qwen3-14B and 11.4% on Qwen3-32B. On the NMO task, the 14B score increases from 0.2509 to 0.8292. In contrast, both free-stopping policies produce only one candidate per task on average, despite having the same maximum interaction budgets available. They terminate earlier, but their submitted artifacts are substantially weaker. These improvements cannot be attributed to the completion gate, since it is excluded from this analysis. The additional processing incurs more steps, but it produces measurable improvements in artifact quality rather than merely satisfying a longer protocol.

  • The results further explain why revision must be explicitly evaluated. Under unrestricted stopping, the tested policies consistently return their first candidate instead of exploring potentially better processing configurations. We additionally evaluate Direct-RL under free stopping, where it achieves 0.5423, compared with 0.5464 for the released Qwen3-14B policy. This shows that RL alone provides no observed improvement under unrestricted stopping in this experiment. The central issue is that allowing an agent to stop freely does not establish its ability to assess and improve intermediate results; it can simply bypass that challenge. Our structured protocol directly evaluates this scientific capability, without assuming that the policy has already learned when revision is necessary.

Therefore, the minimum revision requirements are a scientifically grounded and empirically justified evaluation design, not an arbitrary constraint. They reflect the iterative quality-control practice of seismic processing and ensure that agents are evaluated on their ability to assess and improve scientific artifacts. Removing these requirements does not provide an equivalent evaluation with greater freedom; it allows agents to avoid a central scientific challenge, with a demonstrated reduction in the quality of their submitted artifacts.

W3. Expert assessment and behavioral analysis

We have added a blinded assessment by five domain experts, covering outputs from original, SFT, and post-RL policies. This directly evaluates whether the automatic scores correspond to expert judgments of processing quality, beyond the expert involvement in designing the benchmark. Details are included in Appendix H.6 and H.3 in the revision.

  • Study design. During the rebuttal period, we sent an online assessment link to five domain experts. All five completed the study, yielding 20 task-level ranking lists. The study covers four held-out tasks: field NMO correction, F–K filtering, spiking deconvolution, and automatic gain control. These contain five, five, five, and four candidate outputs, respectively, for 19 distinct artifacts. The outputs come from Qwen3-14B and Qwen3-32B evaluation runs with different training stages and inference strategies.

  • Blinded assessment. Experts receive the processing objective, the unprocessed input, and anonymously labeled candidate outputs. They rank the outputs according to the requested processing effect and preservation of useful signal. Model identities, training stages, automatic scores, and accepted reference outputs are hidden. Candidate positions are shuffled once within each task, with the same frozen layout used for all experts. Panels share physical axes and amplitude limits and can be enlarged. Equal ranks are permitted when experts consider multiple outputs equally acceptable.

AuthorsOpenReview ↗

Response to Reviewer AKDX - Part III

  • The automatic score shows substantial agreement with expert judgments. We measure ranking agreement with the commonly adopted Kendall's τb\tau_b [5], a rank-correlation coefficient that accounts for ties. It ranges from −1 (opposite rankings) to +1 (perfect agreement), with 0 indicating no rank association. Higher positive values indicate stronger agreement. We compare the frozen reference-agreement scores with expert rankings within each task:

    Agreement measureResult
    Score–expert rank agreement, Kendall's τb\tau_b0.68
    Strict pairwise preference agreement80.83%

    The score–expert τb\tau_b of 0.68 indicates substantial positive agreement between automatic scores and expert judgments. Furthermore, 80.83% of strict expert preferences agree with the automatic score ordering. Kendall’s τb\tau_b accounts for ties, while strict pairwise preference agreement excludes human-tied pairs. Importantly, all displayed artifacts are valid, so the agreement concerns scientific quality differences among completed outputs rather than execution success. These results provide direct evidence that the frozen evaluator captures aspects of expert-perceived processing quality across the assessed tasks.

    We have added the complete protocol, interface, agreement definitions, and results to the revised paper in Appendix H.6.

Complementary behavioral analysis is provided in Appendix H.3 in the revision. The paired NMO case shows that the original policy fails to complete despite successful operation calls, whereas the trained policy makes a better numerical revision and completes with fewer steps. This analysis explains observable processing behavior, while the blinded study supplies the expert assessment of output quality requested by the reviewer.

W4. Contribution of Individual RL Reward Components

We have conducted reward-component ablations to isolate their contributions in Appendix H.5 in the revision. The results show that the four components serve distinct roles: scientific shaping improves processing quality, preservation penalties are particularly important for iterative tasks, execution rewards improve reliability, and interaction costs control processing efficiency. These effects are not interchangeable. We evaluate Qwen3-14B by removing each reward component individually while retaining the others. The SFT-only and full-reward conditions provide the reference comparison.

Training conditionQsciQ_{\mathrm{sci}} ↑QiterQ_{\mathrm{iter}} ↑Valid (%) ↑Artifact warnings (%) ↓Mean operations
SFT only0.741 ± 0.0220.746 ± 0.04698.2 ± 3.021.15.65 ± 0.56
Full reward0.750 ± 0.0040.765 ± 0.008100.0 ± 0.021.15.30 ± 0.08
Without execution reward0.743 ± 0.0110.760 ± 0.002498.2 ± 3.024.65.36 ± 0.15
Without scientific shaping0.746 ± 0.00340.759 ± 0.0032100.0 ± 0.021.15.37 ± 0.16
Without preservation penalties0.749 ± 0.00430.751 ± 0.0090100.0 ± 0.021.15.44 ± 0.03
Without interaction cost0.748 ± 0.00380.771 ± 0.0080100.0 ± 0.021.15.53 ± 0.00

The ablations reveal four distinct effects:

  • Scientific shaping improves processing quality beyond successful execution. Removing scientific shaping reduces QsciQ_{\mathrm{sci}} from 0.750 to 0.746 and QiterQ_{\mathrm{iter}} from 0.765 to 0.759, while valid completion remains at 100%. Thus, the quality improvement is not simply a consequence of more successful operation calls.

  • Preservation penalties have the largest observed effect on iterative scientific quality. Removing them reduces QiterQ_{\mathrm{iter}} from 0.765 to 0.751, a larger decrease than removing either execution rewards or scientific shaping. This supports our central motivation: optimizing the requested processing effect alone is insufficient; preserving useful seismic information requires separate attention. Although binary artifact-warning rates remain unchanged, the continuous iterative quality score captures the difference.

  • Execution rewards improve reliability. Without execution rewards, valid completion decreases from 100% to 98.2%, artifact warnings increase from 21.1% to 24.6%, and QsciQ_{\mathrm{sci}} decreases from 0.750 to 0.743. This confirms the role of execution feedback in maintaining reliable workflows alongside scientific optimization.

Reviewer_qVSXOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SeisAgentBench, an executable benchmark for evaluating LLM agents on seismic data-processing workflows, and SeisAgent, an agent trained using SFT and group-relative RL. The benchmark consists of 95 tasks across 19 seismic-processing families, requiring agents to select processing operations, configure numerical parameters, evaluate intermediate results, and revise their decisions when necessary. SeisAgentBench incorporates domain-specific scientific diagnostics, artifact-preservation checks, and workflow efficiency measures to assess both the processing trajectory and the quality of the final seismic outputs. The authors evaluate multiple LLM backbones, including under different training settings. Experiments on 19 held-out tasks show that the proposed training approach generally improves task completion and scientific-quality scores, with larger improvements for smaller models and more modest gains for stronger models.

Critical strengths

S1. The benchmark evaluates not only whether an agent successfully executes seismic-processing operations, but also whether the resulting artifacts meet domain-specific quality and signal-preservation criteria. The inclusion of artifact-level diagnostics makes the benchmark potentially useful for evaluating scientific agents in realistic processing workflows.

S2. SeisAgentBench provides 95 tasks across 19 seismic-processing families, supporting multi-step execution, intermediate artifact tracking, candidate comparison, and parameter revision. This offers a structured and reproducible environment for studying LLM agents in seismic processing, with potential value to the geophysics community.

Critical weaknesses

W1. The core framework is built on a well-established approach, incorporating LLM-based tool orchestration, iterative execution, feedback-driven revisions, and a combination of supervised fine-tuning (SFT) followed by reinforcement learning. Previous efforts like ScienceAgentBench, SciAgentGym, and MLGym have already looked into these aspects within scientific contexts. Additionally, earlier seismic-processing assistants have successfully shown how LLMs can manage seismic workflows. As a result, the main contribution of this work seems to be the integration of existing techniques with seismic-specific tools and evaluation metrics. The paper falls short of clearly defining any fundamentally new challenges that its benchmark introduces compared to earlier scientific-agent benchmarks.

W2. The evaluation set includes only 19 tasks, and some of the seismic artifacts appear in both the training and testing phases, which raises questions about how well the model can generalize. Moreover, the improvements from SFT and Group-RL are rather modest, especially for more potent models like Qwen3-14B. The fixed-script baselines also have their limitations, as they can't fulfill the benchmark's necessary revision requirements for iterative tasks. Without stronger agent baselines, more challenging evaluation scenarios, or clear evidence that the model can generalize to independent seismic surveys, it’s hard to determine whether the benchmark truly assesses adaptive scientific reasoning or simply adherence to set workflow guidelines.

Questions for authors

Q1 (Regarding W1): What specific evaluation challenges does SeisAgentBench introduce that are not already addressed by existing scientific agents? Beyond the seismic-specific tools and metrics, could the authors clarify which aspects of the benchmark require fundamentally different agent capabilities or evaluation methodologies?

Q2 (Regarding W2): Given the small held-out test set (19 tasks) and shared underlying artifacts across data splits, what evidence supports the benchmark's ability to evaluate generalizable scientific reasoning rather than task-specific workflow compliance? Additionally, since the scripted baselines cannot satisfy the mandatory revision requirements, how can the reported results establish the advantage of adaptive agent reasoning over simpler processing strategies?

8 comments
AuthorsOpenReview ↗

Response to Reviewer qVSX - Part I

Thank you for the insightful suggestions! We have revised the paper (marked in red) and our point-to-point responses are as below. Particularly, we have conducted additional experiments on new field data, and added the results in both the responses and revision.

W1 & Q1. Novelty and Distinction from Existing Scientific Benchmarks and Seismic Agents

We respectfully disagree that SeisAgentBench is merely an integration of existing techniques. This assessment confuses established agent components with the novelty of the scientific problem and its evaluation. As discussed in Sections 2–3 of our original paper, SeisAgentBench introduces, to our best knowledge, the first shared executable benchmark for multi-step seismic-processing agents, where success requires not only executing operations but also producing scientifically sound artifacts through numerical decisions, candidate assessment, and revision. We clarify three major distinctions below, and also have added them in Appendix A.1 in the revision.

  • Existing scientific-agent benchmarks address fundamentally different scientific problems. The reviewer mentions ScienceAgentBench, SciAgentGym, and MLGym. Although these works establish important scientific-agent benchmarks, their evaluation targets and required scientific decisions differ substantially from ours.

    BenchmarkPrimary evaluation targetKey distinction of SeisAgentBench
    ScienceAgentBenchScientific program generation and execution, assessed through task-specific outputs and program criteria.Scientific artifact processing, where valid execution does not guarantee a scientifically useful output. Our benchmark additionally assesses processing effects, signal preservation, and artifact lineage.
    SciAgentGymMulti-step scientific tool use, evaluated through task correctness and execution efficiency.Preservation-aware numerical processing, where each operation may improve the target property while damaging other scientifically important information.
    MLGymIterative machine-learning experimentation, evaluated through model development and task-performance improvements.Physical data transformation under incomplete scientific evidence, requiring agents to select numerical parameters, compare processing candidates, and preserve useful seismic information without access to accepted references.

    The central challenge is not whether an agent can execute scientific tools, but whether it can improve a physical data product without destroying information that must be preserved. For example, as discussed in the paper, stronger filtering may suppress unwanted energy while removing useful reflections; NMO correction may improve event flatness while introducing stretch or amplitude loss. These competing effects cannot be assessed through execution correctness or a single improvement objective alone.

    Importantly, recognized benchmarks such as ScienceAgentBench and MLGym demonstrate that establishing an executable evaluation environment for a distinct scientific problem is itself a substantial research contribution. SeisAgentBench establishes such a foundation for multi-family seismic processing, which has not been systematically addressed by existing scientific-agent benchmarks.

AuthorsOpenReview ↗

Response to Reviewer qVSX - Part II

  • Existing seismic assistants and benchmarks do not provide our evaluation setting. Beyond the works mentioned by the reviewer, we further identify several closely related seismic systems. Their differences from SeisAgentBench are equally clear.

    Related workExisting focusOur distinction
    Intelligent Seismic Workflows (already discussed in our paper)An LLM-based assistant for executing specific seismic-processing operations.A shared executable benchmark, rather than demonstrating one processing assistant, with standardized tasks, scientific evaluation, and comparisons across policies and training methods.
    Seismology Modeling Agent [1]Agent-assisted configuration and execution of seismic simulation workflows.Adaptive seismic data processing, emphasizing the scientific effects of transformations, preservation, candidate revision, and final selection rather than simulation execution.
    TREMORS [2]Natural-language assistance for seismic waveform retrieval across data centers.Processing and evaluating seismic data, rather than locating and accessing existing data.
    SPBench [3]Evaluation of supervised seismic-processing methods across multiple tasks.Evaluating autonomous processing policies, including operation selection, numerical parameterization, intermediate assessment, revision, and stopping, rather than evaluating fixed processing models.

    These systems demonstrate useful seismic AI applications, but none establishes the same shared evaluation problem. In particular, performing seismic operations is not equivalent to evaluating whether an agent makes scientifically justified processing decisions over a complete trajectory.

  • Our scientific problem leads to a different environment and evaluation formulation. As formalized in Sections 2.2–3.3 of the original paper, these differences are not merely changes in domain-specific tools or metric names.

    • Artifact-centered decision making. Unlike program- or answer-centered benchmarks (e.g., ScienceAgentBench), SeisAgentBench evaluates the processing trajectory together with the resulting seismic artifact. Its persistent lineage graph retains intermediate candidates, operation parameters, and artifact dependencies, enabling branching, comparison, and revision rather than destructive state replacement.

    • Scientific decisions under incomplete evidence. Unlike settings where task-performance feedback can be directly queried (e.g., MLGym), accepted targets and reference-derived scores remain hidden from our evaluation-time policy. The agent must select operations and numerical parameters using reference-free diagnostics, including coherence, spectral properties, amplitude statistics, and preservation indicators.

    • Non-monotonic processing quality and explicit preservation. Unlike conventional execution or task-completion evaluation (e.g., SciAgentGym), additional successful processing can worsen an earlier candidate. SeisAgentBench therefore evaluates both the intended processing effect and collateral damage. Specifically, our evaluation jointly records valid completion, scientific measurements, artifact-preservation failures, and workflow cost: Mτ(x^,ξ)=(Vτ,dτ,pτ,cτ).M_\tau(\hat{x},\xi) =(V_\tau,\mathbf d_\tau,\mathbf p_\tau,\mathbf c_\tau). This makes numerical parameter selection, candidate comparison, revision, and stopping directly assessable scientific decisions.

    Our results confirm that this distinction is necessary. In Table 3, trained Qwen3-32B reaches 100% valid completion, yet 24.6 ± 3.0% of its outputs still trigger artifact warnings. Thus, even reliable workflow completion leaves measurable scientific-quality problems unresolved. These failures would be missed by completion-only evaluation.

In summary, SeisAgentBench is not another application of existing agent orchestration. It establishes a previously missing scientific evaluation setting in which agents must construct, assess, and revise physical seismic products under competing processing and preservation requirements. Our tasks provide a reusable foundation for studying these capabilities, while SeisAgent demonstrates how scientific-feedback training can improve processing decisions within this environment. The novelty lies in the scientific problem, its executable formulation, and its evaluation methodology, not in claiming that general-purpose orchestration or SFT followed by RL was invented here.

AuthorsOpenReview ↗

Response to Reviewer qVSX - Part III

W2 & Q2. Generalization beyond the original test set and evidence beyond workflow compliance

We have substantially extended the evaluation with new field-recorded seismic data, an explicit pipeline-search baseline, and an ablation of the revision requirements in Appendix H.7, H.2 and H.4 of the revision. These additions directly address the reviewer's concerns from three perspectives: generalization to realistic seismic inputs, comparison with stronger adaptive processing strategies, and scientific improvements beyond workflow compliance.

  • New field-data evaluation demonstrates transfer to realistic seismic-processing scenarios. We introduce five additional processing cases based on recorded field data from established exploration and imaging resources, including Vibroseis Line 001 [4], Taranaki [5], F3 [6], and Teapot Dome [7]. Unlike additional synthetic variants, these data contain real acquisition signals, physical geometry, recorded amplitudes, and complex arrival patterns. They therefore provide a practically relevant test of whether an agent can perform meaningful processing on previously unseen inputs.

    Importantly, the new data also introduce substantially different processing scales. The largest input contains 251 shots, 70,782 traces, and 106.2 million samples, spanning 12.5 km in shot positions. Compared with its original held-out counterpart, it contains approximately 443 times as many samples and spans 26 times the shot-position range. These changes require processing decisions that respect new sampling, geometry, and physical signal characteristics, rather than simply reproducing familiar operations on the original benchmark artifacts.

    We evaluate frozen Qwen3-14B policies, both original and SFT-2 + Group-RL, without any additional field-data training or checkpoint selection. The five cases cover amplitude balancing, early-arrival suppression, geometry reorganization, moveout correction, and depth imaging.

    The new figure in Appendix H.7 of the revision presents the recorded inputs, outputs from both policies, and supplied references. Both policies produce the requested processing artifacts. The results demonstrate improved amplitude balance and reflection alignment while preserving waveform characteristics and RMS amplitude, complete preservation of 70,782 traces during geometry conversion, and agreement with the supplied depth-imaging target. The trained policy further improves amplitude balancing and moveout coherence compared with zero-shot.

    These results provide direct evidence of processing capability on new field-recorded inputs, not merely successful execution of familiar tools. The original benchmark evaluates held-out instances across all 19 processing families, while this additional study establishes transfer to new recorded inputs and substantially larger physical scales within the existing processing operations.

  • Stronger adaptive baselines confirm that our results do not depend on comparisons with fixed scripts (i.e., simpler processing strategies). We additionally evaluate explicit pipeline search, which actively explores alternative processing sequences rather than following a predetermined script.

    As detailed in Appendix H.2 of the revision, the search baseline uses beam width two and up to two alternative extensions per expansion. We evaluate both original and trained Qwen3-14B and Qwen3-32B policies. Each comparison uses the same backbone and policy weights, with cost changes measured relative to its standard interaction loop.

    Backbone / policyChange in total tokensChange in inference timeΔQsci\Delta Q_{\mathrm{sci}}
    Qwen3-14B / Original+29.2%+40.8%−0.0070
    Qwen3-14B / Trained+37.3%+43.6%−0.0129
    Qwen3-32B / Original+37.9%+47.7%−0.0244
    Qwen3-32B / Trained+47.7%+64.2%−0.0704

    ΔQsci\Delta Q_{\mathrm{sci}} is the signed score difference, computed as search minus standard interaction.

    The standard interaction loop achieves higher mean scientific scores with lower inference cost in all four settings. Explicit search consumes 29.2–47.7% more tokens and 40.8–64.2% more inference time, yet does not improve processing quality in any tested setting. This provides a direct comparison against an adaptive inference strategy, rather than relying on fixed scripts as the primary evidence.

    Furthermore, the original Table 3 already evaluates stronger model capacities: SFT-2 + Group-RL increases Qwen3-32B's iterative-family score from 0.812 to 0.855, with valid completion increasing from 94.7% to 100%. Thus, the benefits of scientific-feedback training are not confined to smaller starting models.

Reviewer_5aN2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes an agentic benchmark for seismic data analysis. The authors also propose a training method based on benchmark feedback and demonstrate its effectiveness on several backbones.

Critical strengths

The dataset and executable tools provide a gym like learning environment for LLM agents working on seismic tasks.

Critical weaknesses

Composite score: Can the authors justify the weights used in the composite score described in Appendix E.2 and assess whether the main comparisons are sensitive to these choices?

Benchmark difficulty: Can the authors evaluate a frontier or more capable model under the same setup? Qwen3-14B already reaches 94.7% valid completion and a scientific score of 0.745 in zero-shot evaluation. Although this does not establish that the benchmark is solved, stronger baselines would help clarify its remaining difficulty and whether the proposed training offers benefits beyond a stronger starting policy.

Agent Baseline: Currently, in zero-shot evaluation, the agent sees environment feedback before acting, which resembles a ReAct-style workflow. Could the authors evaluate a stronger inference-time workflow, such as explicit search over candidate processing pipelines, under the same interaction and compute budgets? This would help establish whether the gains from training justify its additional complexity compared with improving the inference-time strategy.

Skill-level analysis: Can the authors provide a deeper analysis of the skills required to solve the tasks? It remains unclear whether SFT and RL primarily improve tool-use compliance, numerical parameter selection, interpretation of diagnostics, or feedback-driven revision. Distinguishing these would help explain why training yields better solutions.

Does reference agreement reflect scientific quality? For certain tasks, several processing outputs may be acceptable, yet reference agreement measures similarity to one accepted target. Could the authors provide evidence that higher reference agreement corresponds to better scientific quality, for example through expert assessment or agreement with the family-specific scientific diagnostics?

Questions for authors

N/A

5 comments
AuthorsOpenReview ↗

Response to Reviewer 5aN2 - Part I

We thank the reviewer for the valuable suggestions for improvement and have revised our paper (marked in red). Our responses to the reviewer’s comments are below:

W1. Composite-score design and sensitivity

We thank the reviewer. Our score follows an expert-informed hierarchy, and the reported advantages remain stable under substantial changes to its weights. We address the design rationale and sensitivity separately, and have included them in Appendix E.8 in the revised paper.

  • Score design. The coefficients in Appendix E.2 in the original paper reflect three evaluation priorities:

    CategoryWeightComponents
    Primary target agreement (T)75%Correlation, NRMSE, SSIM, PSNR
    Input-relative improvement (G)10%PSNR gain, SSIM gain
    Auxiliary quality checks (A)15%Amplitude agreement, shape validity, optional LPIPS

    The largest weight goes to final-artifact agreement. Input-relative improvement is secondary because improving a poor input does not guarantee an adequate final result. Amplitude and shape checks provide complementary evidence about the resulting seismic product. This use of multiple signal and structural checks follows established seismic quality-control practice [1]. The exact coefficients, such as 0.28 for correlation and 0.22 for NRMSE, are fixed engineering choices. Our conclusions should therefore remain stable when these choices change, which we test directly below.

  • Sensitivity analysis. We design a two-level sensitivity experiment to demonstrate that our model comparisons remain stable under changes to the composite-score weights. First, since the default category allocation (75%/10%/15%) reflects the intended scientific evaluation priorities, substantial reallocations would fall outside the intended design space and effectively redefine the evaluation objective. We therefore perturb the category budgets within scientifically meaningful ranges: primary target agreement receives 70–80%, input-relative improvement receives 5–15%, and auxiliary quality checks receive 10–20%, with the total constrained to 100%. Second, within each category, we independently perturb the original metric coefficients by factors between 0.5 and 1.5 and renormalize them to the corresponding category budget. We then recompute QsciQ_{\mathrm{sci}} under each weighting configuration and examine whether the relative performance of the evaluated policies changes. This directly tests whether the benchmark's conclusions depend on particular coefficient choices or remain consistent across substantially different weighting schemes.

    We report the score gap between SFT-2 + Group-RL and each comparator from Table 3: ΔQsci=QsciSFT-2+Group-RL−Qscicomparator.\Delta Q_{\mathrm{sci}}=Q_{\mathrm{sci}}^{\mathrm{SFT\text{-}2+Group\text{-}RL}}-Q_{\mathrm{sci}}^{\mathrm{comparator}}. Positive values favor the trained policy. Both columns below are multiplied by 100.

    BackboneComparatorDefault gapGap under perturbed weights
    Qwen3-4BZero-shot+12.4[11.481, 13.138]
    Qwen3-4BSchema ICL-2+5.1[4.726, 5.433]
    Qwen3-8BZero-shot−0.9[-1.076, -0.724]
    Qwen3-8BSchema ICL-2+5.3[4.299, 6.275]
    Qwen3-14BZero-shot+0.5[0.375, 0.633]
    Qwen3-14BSchema ICL-2+0.6[0.467, 0.734]
    Qwen3-32BZero-shot+2.0[1.688, 2.208]
    Qwen3-32BSchema ICL-2+1.5[1.058, 2.007]
    Llama-3.1-8BZero-shot+16.4[15.955, 16.854]
    Llama-3.1-8BSchema ICL-2+10.3[8.970, 11.366]
    Mistral-7BZero-shot+16.4[15.613, 17.194]
    Mistral-7BSchema ICL-2+8.2[7.474, 9.143]

    The results demonstrate strong robustness to the weighting scheme. Despite substantial perturbations at both the category and individual metric levels, the performance differences remain within tight intervals around their original values, with only minor variations in magnitude. For example, against zero-shot, the Qwen3-4B gap remains within [11.481, 13.138] around its default value of 12.4, while the Qwen3-14B gap remains within [0.375, 0.633] around 0.5. This shows that the composite score is insensitive to reasonable changes in its coefficients, and the benchmark's comparative conclusions are not driven by a particular choice of metric weights.

W2. Evaluation of a more capable model

  • Table 3 in the original paper already includes Qwen3-32B under the same evaluation protocol, and our training further improves this stronger backbone. Its advantage over Qwen3-14B is clear on the nine iterative families, which require candidate assessment, feedback-informed revision, and supported final selection. Qwen3-32B increases QiterQ_{\mathrm{iter}} from 0.775 to 0.812 in zero-shot evaluation and from 0.754 to 0.836 with Schema ICL-2.
AuthorsOpenReview ↗

Response to Reviewer 5aN2 - Part II

  • Post-training remains beneficial beyond this stronger starting policy. SFT-2 + Group-RL raises the 32B model's iterative-family score to 0.855±0.0350.855 \pm 0.035, exceeding both its zero-shot score of 0.812 and its Schema ICL-2 score of 0.836. These results address both aspects of the reviewer's concern: a more capable model achieves better iterative-processing performance, yet further gains remain possible through training. The observed training benefits are therefore not confined to weaker starting policies.

W3. Explicit search over candidate processing pipelines

  • We introduce an explicit pipeline-search baseline to examine whether SeisAgent's direct feedback-guided processing is more effective than searching over alternative processing pipelines. This provides a stronger evaluation of adaptive processing than comparisons with fixed scripts alone. Specifically, we implement beam search with width two and up to two alternative extensions per expansion, and evaluate Qwen3-14B and Qwen3-32B both before and after training. Base denotes the original model, while Trained denotes SFT-2 + Group-RL. Each row compares pipeline search against the standard interaction loop using the same backbone and policy weights. Cost changes are reported as relative percentages, while ΔQsci\Delta Q_{\mathrm{sci}} is the signed score difference (search minus standard interaction). The results show that direct feedback-guided processing achieves higher scientific scores with lower inference costs across all four settings, despite explicit search consuming 40.8–64.2% more inference time. The complete comparison is provided in Appendix H.2 of the revision.

    Backbone / policyEnv. stepsModel callsPrompt + output tokensInference timeΔQsci\Delta Q_{\mathrm{sci}}
    Qwen3-14B / Base−2.4%+12.1%+29.2%+40.8%−0.0070
    Qwen3-14B / Trained+1.1%+15.9%+37.3%+43.6%−0.0129
    Qwen3-32B / Base+15.3%+12.4%+37.9%+47.7%−0.0244
    Qwen3-32B / Trained+16.3%+15.8%+47.7%+64.2%−0.0704

    Search requires 12.1–15.9% more model calls and 29.2–47.7% more tokens, yet produces lower mean QsciQ_{\mathrm{sci}} in every setting. The largest decline occurs for trained Qwen3-32B: inference time rises from 31.46 to 51.64 seconds, while QsciQ_{\mathrm{sci}} decreases by 0.0704.

  • The standard interaction loop achieves both higher scores and lower inference cost across all four comparisons. These results support SeisAgent's direct, feedback-guided processing design: adding explicit pipeline exploration does not improve the observed quality–cost trade-off, even when it consumes substantially more computation.

W4. Skills required beyond successful execution

The tasks require numerical parameter selection, signal preservation, effective revision, and stopping, not merely correct operation calls. We provide two forms of evidence: improvements in physical diagnostics at unchanged validity, and a trajectory comparison that identifies the processing decisions behind the improvement. The evidence is also included in Appendix H.3 in the revised paper.

  • Training improves processing outcomes beyond completion. SFT provides the interaction warm start, while Group-RL rewards processing improvement, preservation, and efficient interaction. The 14B comparison in Sec. 5 in the original paper shows why these outcomes must be assessed separately: SFT-2 increases completion but does not improve QsciQ_{\mathrm{sci}}. The Group-RL learning curves show higher scores with fewer steps. Crucially, the deconvolution analysis shows lower spectral leakage and energy error while all 15 evaluated checkpoints already produce valid outputs. These physical improvements cannot be explained by higher completion alone.

    EvidenceSettingResult
    Interaction reliabilityQwen3-14B, zero-shot → SFT-2; Table 3Completion 94.7% → 98.2%; QsciQ_{\mathrm{sci}} 0.745 → 0.741
    Processing quality and efficiencyQwen3-4B, Group-RL epoch 1 → 5; Fig. 4a–cQsciQ_{\mathrm{sci}} +9.4%; QiterQ_{\mathrm{iter}} +40.2%; environment steps −7.5%
    Numerical processing and preservationSeparate Qwen3-14B deconvolution case; Fig. 4d–eSpectral leakage −19.2%; energy error −4.6%; all 15 checkpoints valid
    Held-out iterative performanceQwen3-32B, zero-shot → SFT-2 + Group-RL; Table 3QiterQ_{\mathrm{iter}} 0.812 → 0.855; mean revisions 1.32 → 1.26
AuthorsOpenReview ↗

Response to Reviewer 5aN2 - Part III

  • The NMO trajectory reveals better numerical revision from the same early candidate scores. On the same NMO task, original and trained Qwen3-32B both obtain 0.251 → 0.548 for their first two candidates. Their next decisions differ. The original policy refines three event-based velocity picks. The trained policy applies eight semblance-derived time–velocity pairs, producing a stronger third candidate.

    Same NMO taskOriginal Qwen3-32BSFT-2 + Group-RL
    First two candidate scores0.251 → 0.5480.251 → 0.548
    Next velocity estimate appliedThree refined event-based picksEight semblance-derived time–velocity pairs
    Third-candidate score0.5530.638
    Revisions / environment steps4 / 242 / 13
    Valid completionNoYes

    All 21 executed operation calls in the original trajectory succeed, yet the policy exhausts its 24-step budget without completing. The trained policy obtains a stronger candidate and completes in 13 steps with only two revisions. Thus, neither successful calls nor more revisions guarantee an effective workflow. The observed benefit concerns the numerical choices made during revision, the resulting artifact quality, and the ability to complete a supported workflow efficiently.

W5. Does reference agreement reflect scientific quality?

The processing pipeline and evaluation protocol were designed under the supervision of domain experts. And we have added a blinded expert study in Appendix H.6 in the revision that directly tests this question. The frozen reference-agreement score achieves Kendall’s τb=0.68\tau_b=0.68 with expert rankings and 80.83% strict pairwise preference agreement. This provides output-level evidence beyond the expert involvement in designing the benchmark.

  • Study design and blinding. Within the rebuttal period, we sent an online assessment link to five domain experts. All five completed the study, providing 20 task-level ranking lists. The study covers four held-out tasks: field NMO correction, F–K filtering, spiking deconvolution, and automatic gain control. These contain five, five, five, and four processed artifacts, respectively, for 19 distinct artifacts in total. The outputs come from Qwen3-14B and Qwen3-32B runs with different training stages and inference strategies.

  • Experts receive the processing objective, the unprocessed input, and anonymously labeled outputs. They rank the outputs by the requested processing effect and preservation of useful signal. Model identities, training stages, automatic scores, and accepted reference outputs are hidden. Candidate positions are shuffled once within each task and kept fixed across experts. Panels use common physical axes and amplitude limits and can be enlarged. Equal ranks are allowed, so experts are not required to prefer one output when several are equally acceptable.

  • Agreement with the frozen evaluator. We compare the frozen reference-agreement score with expert rankings within each task. The ranking agreement is measured using the commonly adopted Kendall's τb\tau_b [2], a rank-correlation coefficient that accounts for ties. It ranges from −1 (opposite rankings) to +1 (perfect agreement), with 0 indicating no rank association. Higher positive values indicate stronger agreement.

    Agreement measureResult
    Score–expert rank agreement, Kendall’s τb\tau_b0.68
    Strict pairwise preference agreement80.83%

    The score–expert τb\tau_b of 0.68 shows substantial positive rank agreement, supported by 80.83% strict pairwise preference agreement. All displayed artifacts are valid, so these results concern differences among completed processing outputs rather than differences in execution success.

The study directly addresses the concern about multiple acceptable outputs. Experts judge the scientific objective and signal preservation without seeing the accepted target, and they may assign ties. Their positive agreement with the frozen score therefore does not result from being instructed to reproduce the reference. It shows that, across the assessed tasks, higher reference agreement generally corresponds to outputs preferred by experts.

The detailed complete protocol, assessment-interface figure, agreement definitions, and results are in Appendix H.6 in the revision. The findings support reference agreement as a useful comparative measure of processing quality, alongside the benchmark’s separate scientific diagnostics and preservation checks.

[1] Seismic data analysis: Processing, inversion, and interpretation of seismic data. Society of exploration geophysicists, 2001.

[2] The treatment of ties in ranking problems, 1945

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.