- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies autoregressive lithology prediction and stratigraphic generation from partially observed drillholes when there are shifts in both spatial and geological distribution. It presents DrillBench, which includes 49,671 drillholes from Western Australia, and sets up tasks involving next-layer prediction and free-running continuation for cases that are in-distribution, spatially separated, and across provinces. The method proposed combines layer-level autoregressive modelling with pretraining on about 112,000 historical drillholes and the retrieval of neighbouring lithology, without directly basing the predictions on absolute coordinates or geochemical assays. The evaluation involves statistical, geostatistical, machine-learning, and neural baselines and is supported by multi-backbone comparisons, component ablations, controls regarding the pretraining data, and bootstrap uncertainty estimates. The results indicate that directly conditioning on spatial and geochemical information can enhance local accuracy but may harm transfer, whereas retrieval mainly helps with local prediction and pretraining generally improves next-layer prediction under distribution shift.
Critical strengths
- DrillBench creates a standardized resource by compiling 49,671 real-world drillholes for use in next-layer prediction and autoregressive continuation. The reliability of the data is enhanced through extensive data curation, the harmonization of coordinates, and the identification and elimination of duplicates, while the use of explicit spatial and inter-provincial splits allows for the assessment of geographical generalisation beyond what conventional random splits can assess.
- The experiments indicate that spatial and geochemical conditioning can enhance local accuracy but can substantially degrade transfer, and they also allow for the characterization of the complementary roles of retrieval and pretraining; supportive evidence for these conclusions comes from broad baseline coverage, multi-backbone comparisons, compute-matched pretraining controls, province-exclusion experiments, and paired uncertainty estimates.
Critical weaknesses
- The evaluation does not adequately demonstrate drillhole-specific conditional generation. Task 1 provides ground-truth layer boundaries and thus isolates next-layer lithology classification, leaving transition-boundary prediction untested in this task. Task 2 removes this assumption, but its primary metric compares lithology–thickness distributions pooled across drillholes and rollouts. This aggregation discards stratigraphic ordering and the correspondence between each generated suffix and its observed prefix: a model could reproduce the overall frequencies and thicknesses of rock types without generating an appropriate continuation for a particular drillhole. The supplementary analysis partly acknowledges this limitation. Per-hole sequence edit distance and boundary F1 show little separation among layer-level methods, while a prefix-agnostic layer-marginal generator achieves a relatively close in-distribution KL score. Thus, the reported gains more clearly establish better aggregate distribution matching than improved conditional continuation. Stronger evidence would require evaluating the likelihood of each true suffix given its prefix, testing sensitivity to shuffled or removed prefixes, or demonstrating improvements in sequence-aware conditional metrics.
- The breadth of the transfer claims exceeds the evidence's geological coverage and consistency. The spatial splits combine geographic separation with changes in lithology composition, class support, and regolith–bedrock balance, with potentially additional differences in logging practices. Consequently, performance differences cannot be attributed straightforwardly to distance or to a single graded notion of geological shift. Cross-province evaluation covers only Pilbara, limiting evidence that the observed patterns generalize to other geological settings. The proposed components also provide uneven benefits: retrieval strongly improves local next-layer prediction but offers no clear cross-province macro-F1 gain, while the full recipe does not consistently outperform pretraining alone. For free-running generation, its improvement over the plain GRU is significant on Pilbara but not on OOD-Distal, and the appendix explicitly notes that component-wise gains do not hold across all backbones. These findings support a useful, setting-dependent combination of pretraining and retrieval, but do not establish a generally “transfer-safe” approach or show that avoiding direct coordinate and geochemical conditioning is broadly preferable to alternative encodings designed to improve transfer.
Questions for authors
- On the issue of the generation claim: Appendix D.2.5 admits that pooled KL does not assess whether each continuation is appropriate for its own prefix and notes that the various methods at the layer level show only minor differences when it comes to the per-hole metrics. Is Task 2 meant to focus on making drillhole-specific conditional predictions or is its main aim instead to match the population-level distribution? If the former is the case, then which existing results show an improved correspondence between individual prefixes and their generated suffixes? This distinction has a significant impact on my evaluation of the paper's main contribution.
- What does the term “transfer-safe” entail? Is it meant to refer specifically to an improved average performance on the evaluation sets, or does it imply robustness with respect to different geological settings? Since there is only a single cross-province target and considering that the performance gains are task- and backbone-dependent, could you please make clear the scope you have in mind and point out the evidence that supports it? In particular, in what way should the claimed graded transfer spectrum be understood in the light of the differences in class support and regolith–bedrock composition among the splits?
2 comments
Scope of the Transfer Claims (W2, Q2)
Thank you for asking us to clarify the scope. (1) "Transfer-safe" referred to a design choice: the framework never reads coordinates or assays as inputs, so it avoids the transfer boundary these inputs create. It did not mean robustness to every geological setting, and we removed the term. Our claims now state where each component helps: direct location and geochemistry inputs reduce macro-F1 on all three shift settings, pretraining improves it on all three, and retrieval helps on Test and OOD-D. (2) The settings differ in distance, lithology composition, and class coverage, so we no longer describe them as a single graded spectrum. Comparisons use the same holes within each setting, and the main transfer findings hold after removing TOPS or regolith layers. The Pilbara remains the only cross-province target, which we state as a limitation; the two transfer patterns above also hold on both Eastern Goldfields rings. (3) The uneven gains are part of the findings: retrieval helps near existing drilling, while pretraining transfers more consistently to new areas. The full framework is never significantly worse than either component alone. (4) Alternative location and geochemistry encodings reduce the loss caused by direct inputs but do not match the lithology-only model on any shift setting. Significance is assessed using 95% intervals from resampling whole exploration reports. Details and manuscript changes follow.
(1) Scope of the claims. The framework uses lithology from the current hole and retrieved neighbours. Location is used only to select neighbours; coordinates and assays are not model inputs. We compare methods separately within each setting, rather than averaging across settings. Against the plain layer-level GRU, pretraining significantly improves macro-F1 on all three shift settings. The full framework significantly improves Test and OOD-D; its gains on OOD-R and Pil-T are positive but not significant.
Manuscript changes: We removed "transfer-safe." Section 1 states where each component helps, and Appendix D.1 reports paired differences with report-level intervals.
(2) What the shift settings test. The settings are ordered by distance from the training data, but also differ in regolith share and available lithology classes, particularly TOPS (transported overburden). We therefore do not treat distance as the only source of shift.
Removing TOPS preserves the direction and significance of every comparison. When all regolith layers are removed, direct location and geochemistry inputs still reduce macro-F1 on all three shift settings. Section 6.2 also separates performance by regolith cover and bedrock: retrieval helps mainly in regolith, while pretraining helps mainly in bedrock. The Pilbara is the only cross-province target, but the two transfer patterns also hold on both Eastern Goldfields rings; adding more provinces will require harmonising their logging vocabularies.
Manuscript changes: Sections 1 and 4.4 clarify the differences between settings. Appendix A states the cross-province limitation, and Appendix D.1.6 reports the label checks.
(3) Different components help in different settings. Retrieval provides the large gain on Test and also helps on OOD-D. Pretraining improves all three shift settings but adds no gain on Test. The full framework significantly outperforms pretraining alone on Test and OOD-D, shows no significant difference on OOD-R and Pil-T, and is never significantly worse than either component alone.
Generation gains are also uneven. The joint-KL gain is significant only on the Pilbara and only for the GRU. In contrast, the lithology term of suffix NLL improves on all four splits, with the same direction across all five backbones.
Manuscript changes: Section 6.2 explains where each component helps. Appendix D.2.4 clarifies that component gains are not consistent across all backbones, and the Conclusion states the remaining open problems.
(4) Alternative encodings. Using the same cell-level network, we tested relative location encodings, alternative normalisations, a log-ratio geochemistry encoding, and a combined encoding. Relative location reduces the loss compared with absolute coordinates, but no tested location or geochemistry encoding matches the lithology-only model on any shift setting. Learning when to use these inputs, through methods such as gated conditioning or domain-adversarial training, remains future work.
Manuscript changes: New Appendix D.1.7 reports all tested encodings with report-level intervals.
Conditional Generation in Task 2 (W1, Q1)
Thank you for this question, which addresses the main goal of Task 2. (1) Task 2 evaluates whether a generated continuation fits its own hole and is realistic for the region. We now report two metrics: suffix NLL measures how well each hole's true suffix is predicted from its own prefix, while pooled joint KL measures regional realism. (2) Following your suggestion, we tested different prefixes while keeping the true suffix fixed. Replacing the prefix with one from another hole, either in the same report or another report, or removing it significantly worsens our framework's suffix NLL on all four splits. The layer marginal, which ignores the prefix, barely changes. (3) Our framework's per-hole improvement is in predicting the next lithology: compared with the plain layer-level GRU, it significantly improves the lithology term of suffix NLL on all four splits. The improvement has the same direction across all five backbones. It also lowers transition KL and depth-band KL. Task 1 provides true boundaries to evaluate next-lithology prediction separately; Task 2 evaluates boundary placement. A difference is significant when its 95% interval, obtained by resampling whole exploration reports, excludes zero.
Details and manuscript changes follow.
(1) Task 2 measures both per-hole fit and regional realism. Joint KL pools layers across holes and generated continuations. It therefore measures regional realism, which explains why a generator that ignores the prefix can still score well. The rollout-based per-hole metrics in Appendix D.2.5 show little difference between methods, because a single generated continuation need not reproduce a hole's exact observed layers. Suffix NLL instead scores the true suffix directly and distinguishes these methods: every autoregressive network and the layer Markov model achieve lower suffix NLL than the layer marginal on all four splits.
Manuscript changes: Section 3.2.2 defines both metrics and explains what they measure. Section 6.1 reports joint KL in Table 2 and suffix NLL in Figure 4. Table 21 in Appendix D.2.6 lists suffix NLL for every Task-2 method.
(2) Generation depends on each hole's own prefix. We keep the true suffix, depths, and location fixed and change only the prefix. For our framework, suffix NLL increases by 0.027–0.048 nats per 1 m cell with a prefix from another report, by 0.012–0.025 with a prefix from another hole in the same report, and by 0.010–0.020 with no prefix. All increases are significant on all four splits. Swapping prefixes within the same report has a smaller effect than swapping across reports, consistent with nearby holes having similar sequences.
The other autoregressive networks and the layer Markov model show the same pattern, with one non-significant case. The layer marginal barely changes. The plain layer-level GRU also shows a similar increase, so prefix use is shared by the autoregressive models. The additional benefit of our framework is described in (3).
Manuscript changes: Appendix D.2.6 reports the cross-report prefix swap in Table 19, also shown as open markers in Figure 4. Table 20 reports the same-report swaps and prefix removal.
(3) Our framework improves next-lithology prediction. We split suffix NLL into a lithology term, which scores the next lithology, and a boundary term, which scores whether a new layer starts. Compared with the plain layer-level GRU, our framework significantly improves the lithology term on all four splits, mainly through retrieval. The improvement has the same direction across all five backbones, with 16 of 20 comparisons significant.
Our framework also achieves lower transition KL and depth-band KL than the plain GRU on all four splits. We computed these as supplementary checks, since transition KL remains large for every method. The boundary term does not improve. As a result, our framework's overall suffix NLL is close to that of the plain GRU, while cell-level networks perform best on three of four splits.
Manuscript changes: Table 22 in Appendix D.2.6 reports both terms for every backbone. Section 3.2 explains why Task 1 provides true boundaries and Task 2 evaluates boundary prediction.