- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces SeisAgentBench, an executable benchmark containing 95 seismic data-processing tasks across 19 families. Agents interact with the environment by selecting processing operations, adjusting parameters, evaluating intermediate results, and revising their workflows. The benchmark evaluates both task completion and scientific artifact quality using domain-specific metrics, reference agreement, and signal-preservation checks. The authors also propose SeisAgent, which combines a small SFT warm start with group-relative RL using scientific feedback. Experiments across six language models show improvements in task completion and scientific quality.
Critical strengths
- The paper evaluates scientific artifact quality and signal preservation, rather than just task completion, making it more relevant to real scientific workflows.
- The experiments cover six models across different model families, with SFT and RL comparisons. The improvements provide evidence for the usefulness of the proposed benchmark and training framework.
Critical weaknesses
-
The out-of-distribution generalization is not sufficiently evaluated: Although the test tasks are held out, they come from the same processing families as the training tasks. The authors can evaluate whether the method generalizes to unseen task families or independent seismic datasets.
-
The iterative behavior is somewhat artificially enforced: In real-world scenarios, agents should decide when to stop experimenting itself. It will be meaningful to see results without the minimum candidate and revision requirements (for iterative tasks), to better evaluate whether RL actually improves feedback-driven decision-making in long horizon tasks.
-
The evaluation could benefit from more expert assessment and behavioral analysis. Since the benchmark targets scientific workflows, the authors could consider including blinded evaluations by domain experts comparing outputs before and after training, along with a more detailed analysis of the trained agents' failure cases.
-
The contribution of different RL reward components is unclear. The reward combines execution success, scientific quality, artifact preservation, and interaction costs, but the paper does not isolate their individual effects. The authors should conduct reward ablations to clarify which components drive the improvements in scientific quality.
Questions for authors
Please see the weaknesses section above
10 comments
Response to Reviewer AKDX - Part II
-
Our new ablation confirms that structured revision improves the resulting artifacts, not merely protocol compliance. We compare the same policy weights on the nine held-out iterative tasks, with and without minimum revision requirements. Inputs, tools, maximum interaction budgets, and the hidden-reference evaluator remain fixed.
Crucially, we measure the ungated reference-agreement score, defined as the mean of in Appendix E.2 in the original paper, computed on each agent's last nominated artifact. Thus, the comparison measures differences in the produced artifacts rather than rewarding compliance with the revision requirements.
Model Free stopping Structured revision Candidates (free → structured) Steps (free → structured) Qwen3-14B 0.5464 0.6609 1.00 → 3.11 3.22 → 11.22 Qwen3-32B 0.5773 0.6433 1.00 → 2.44 2.89 → 9.00 Structured revision improves mean reference agreement by 21.0% on Qwen3-14B and 11.4% on Qwen3-32B. On the NMO task, the 14B score increases from 0.2509 to 0.8292. In contrast, both free-stopping policies produce only one candidate per task on average, despite having the same maximum interaction budgets available. They terminate earlier, but their submitted artifacts are substantially weaker. These improvements cannot be attributed to the completion gate, since it is excluded from this analysis. The additional processing incurs more steps, but it produces measurable improvements in artifact quality rather than merely satisfying a longer protocol.
-
The results further explain why revision must be explicitly evaluated. Under unrestricted stopping, the tested policies consistently return their first candidate instead of exploring potentially better processing configurations. We additionally evaluate Direct-RL under free stopping, where it achieves 0.5423, compared with 0.5464 for the released Qwen3-14B policy. This shows that RL alone provides no observed improvement under unrestricted stopping in this experiment. The central issue is that allowing an agent to stop freely does not establish its ability to assess and improve intermediate results; it can simply bypass that challenge. Our structured protocol directly evaluates this scientific capability, without assuming that the policy has already learned when revision is necessary.
Therefore, the minimum revision requirements are a scientifically grounded and empirically justified evaluation design, not an arbitrary constraint. They reflect the iterative quality-control practice of seismic processing and ensure that agents are evaluated on their ability to assess and improve scientific artifacts. Removing these requirements does not provide an equivalent evaluation with greater freedom; it allows agents to avoid a central scientific challenge, with a demonstrated reduction in the quality of their submitted artifacts.
W3. Expert assessment and behavioral analysis
We have added a blinded assessment by five domain experts, covering outputs from original, SFT, and post-RL policies. This directly evaluates whether the automatic scores correspond to expert judgments of processing quality, beyond the expert involvement in designing the benchmark. Details are included in Appendix H.6 and H.3 in the revision.
-
Study design. During the rebuttal period, we sent an online assessment link to five domain experts. All five completed the study, yielding 20 task-level ranking lists. The study covers four held-out tasks: field NMO correction, F–K filtering, spiking deconvolution, and automatic gain control. These contain five, five, five, and four candidate outputs, respectively, for 19 distinct artifacts. The outputs come from Qwen3-14B and Qwen3-32B evaluation runs with different training stages and inference strategies.
-
Blinded assessment. Experts receive the processing objective, the unprocessed input, and anonymously labeled candidate outputs. They rank the outputs according to the requested processing effect and preservation of useful signal. Model identities, training stages, automatic scores, and accepted reference outputs are hidden. Candidate positions are shuffled once within each task, with the same frozen layout used for all experts. Panels share physical axes and amplitude limits and can be enlarged. Equal ranks are permitted when experts consider multiple outputs equally acceptable.
Response to Reviewer AKDX - Part III
-
The automatic score shows substantial agreement with expert judgments. We measure ranking agreement with the commonly adopted Kendall's [5], a rank-correlation coefficient that accounts for ties. It ranges from −1 (opposite rankings) to +1 (perfect agreement), with 0 indicating no rank association. Higher positive values indicate stronger agreement. We compare the frozen reference-agreement scores with expert rankings within each task:
Agreement measure Result Score–expert rank agreement, Kendall's 0.68 Strict pairwise preference agreement 80.83% The score–expert of 0.68 indicates substantial positive agreement between automatic scores and expert judgments. Furthermore, 80.83% of strict expert preferences agree with the automatic score ordering. Kendall’s accounts for ties, while strict pairwise preference agreement excludes human-tied pairs. Importantly, all displayed artifacts are valid, so the agreement concerns scientific quality differences among completed outputs rather than execution success. These results provide direct evidence that the frozen evaluator captures aspects of expert-perceived processing quality across the assessed tasks.
We have added the complete protocol, interface, agreement definitions, and results to the revised paper in Appendix H.6.
Complementary behavioral analysis is provided in Appendix H.3 in the revision. The paired NMO case shows that the original policy fails to complete despite successful operation calls, whereas the trained policy makes a better numerical revision and completes with fewer steps. This analysis explains observable processing behavior, while the blinded study supplies the expert assessment of output quality requested by the reviewer.
W4. Contribution of Individual RL Reward Components
We have conducted reward-component ablations to isolate their contributions in Appendix H.5 in the revision. The results show that the four components serve distinct roles: scientific shaping improves processing quality, preservation penalties are particularly important for iterative tasks, execution rewards improve reliability, and interaction costs control processing efficiency. These effects are not interchangeable. We evaluate Qwen3-14B by removing each reward component individually while retaining the others. The SFT-only and full-reward conditions provide the reference comparison.
| Training condition | ↑ | ↑ | Valid (%) ↑ | Artifact warnings (%) ↓ | Mean operations |
|---|---|---|---|---|---|
| SFT only | 0.741 ± 0.022 | 0.746 ± 0.046 | 98.2 ± 3.0 | 21.1 | 5.65 ± 0.56 |
| Full reward | 0.750 ± 0.004 | 0.765 ± 0.008 | 100.0 ± 0.0 | 21.1 | 5.30 ± 0.08 |
| Without execution reward | 0.743 ± 0.011 | 0.760 ± 0.0024 | 98.2 ± 3.0 | 24.6 | 5.36 ± 0.15 |
| Without scientific shaping | 0.746 ± 0.0034 | 0.759 ± 0.0032 | 100.0 ± 0.0 | 21.1 | 5.37 ± 0.16 |
| Without preservation penalties | 0.749 ± 0.0043 | 0.751 ± 0.0090 | 100.0 ± 0.0 | 21.1 | 5.44 ± 0.03 |
| Without interaction cost | 0.748 ± 0.0038 | 0.771 ± 0.0080 | 100.0 ± 0.0 | 21.1 | 5.53 ± 0.00 |
The ablations reveal four distinct effects:
-
Scientific shaping improves processing quality beyond successful execution. Removing scientific shaping reduces from 0.750 to 0.746 and from 0.765 to 0.759, while valid completion remains at 100%. Thus, the quality improvement is not simply a consequence of more successful operation calls.
-
Preservation penalties have the largest observed effect on iterative scientific quality. Removing them reduces from 0.765 to 0.751, a larger decrease than removing either execution rewards or scientific shaping. This supports our central motivation: optimizing the requested processing effect alone is insufficient; preserving useful seismic information requires separate attention. Although binary artifact-warning rates remain unchanged, the continuous iterative quality score captures the difference.
-
Execution rewards improve reliability. Without execution rewards, valid completion decreases from 100% to 98.2%, artifact warnings increase from 21.1% to 24.6%, and decreases from 0.750 to 0.743. This confirms the role of execution feedback in maintaining reliable workflows alongside scientific optimization.
Response to Reviewer AKDX - Part I
We appreciate the insightful comments from the reviewer. We have revised the paper accordingly (shown in red) and provide the responses as below. Particularly, we've included new field data experiments and blinded expert study.
W1. Generalization to new field-data inputs
We have added a frozen-policy evaluation on five new field-data processing cases, without additional field-specific training or checkpoint selection in Appendix H.7 in the revision. This extends the evaluation beyond the original benchmark inputs to recorded waveforms with different sampling, acquisition geometry, and processing scale.
The new data are practically relevant because they contain measured signals and physical acquisition structure. The source collection includes Vibroseis Line 001 [1], Taranaki [2], F3 [3], and Teapot Dome [4], representing field-data resources associated with land acquisition, hydrocarbon exploration, and reservoir imaging. Unlike additional synthetic variants, these inputs introduce recorded amplitudes and arrival patterns whose useful information must be preserved during processing.
The largest case contains 251 shots, 70,782 traces, and 106.2 million samples, with 2-ms sampling and a 12.5-km shot-position span. Relative to its original held-out counterpart, the sample count is approximately 443 times larger and the shot-position range is 26 times wider. The new inputs therefore change both the signal content and the physical scale at which processing decisions must operate.
Both original and trained Qwen3-14B policies produce the requested processing outputs without adaptation to these field cases, shown in Figure 8 in the revision. The evaluation covers amplitude balancing, early-arrival suppression, geometry reorganization, moveout correction, and depth imaging. The added figure presents inputs, policy outputs, and supplied references, accompanied by physical diagnostics. The outputs show improved amplitude balance and reflection alignment while retaining waveform character and RMS amplitude. Geometry conversion preserves the complete trace set, and depth-imaging outputs agree with the supplied target. The trained policy additionally improves amplitude balance and moveout coherence relative to zero-shot.
These results provide direct evidence of transfer to newly introduced recorded inputs and larger physical scales. We have added the source description, acquisition characteristics, frozen-policy setup, and qualitative comparisons to Appendix H.7 in the revision. To help reviewers better understand the benchmark, we provide an anonymized data repository for inspection during the review period.
W2. Are the iterative revision requirements artificially enforced?
The revision requirements are necessary, as they reflect established seismic-processing practice and ensure that the benchmark explicitly evaluates candidate assessment and improvement. To demonstrate this, we conduct a free-stopping ablation that removes the minimum revision requirements. The results show that agents terminate prematurely and produce substantially weaker artifacts, even when artifact quality is evaluated independently of workflow compliance. We explain the scientific motivation and present the experimental evidence below, with full details in Appendix H.4 of the revised paper.