acceptodds
Under review as a conference paper at ICLR 2027

Autoregressive Drillhole Modelling under Distribution Shift

Abstract

Autoregressive modelling has achieved remarkable success in language and sequence tasks by learning to predict future states from previous observation. Mineral-exploration drillholes provide a natural but largely unexplored setting for this paradigm: as drilling proceeds, lithology is revealed sequentially from shallow to deep, making prediction of deeper strata inherently autoregressive. Existing drillhole modelling, however, is dominated by spatial interpolation and reconstruction, or largely rely on masked modelling, leaving strictly autoregressive prediction largely underexplored. We introduce DrillBench, a benchmark of 49,671 Western Australian drillholes for next-layer prediction and autoregressive stratigraphic generation across a graded transfer spectrum, from local prediction through spatial shift to cross geological province transfer. Benchmarking classical, geostatistical, and neural models reveals a clear transfer boundary: spatial and geochemical conditioning provides large local gains but deteriorates sharply under stronger shift, whereas lithology-sequence autoregressive models transfer more robustly. Guided by this finding, we develop a backbone-agnostic recipe combining large-scale pretraining on historical drillholes with spatial retrieval of neighbouring lithology. Retrieval is most effective in weathered cover, when local spatial continuity remains informative, whereas pretraining contributes more strongly in bedrock and under broader geological shift. Together, they retain strong local performance while improving generalisation under spatial and cross-province shift, most markedly on the most distant splits.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Autoregressive Modelling, Drillhole, Mineral Exploration

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fF7hOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies autoregressive lithology prediction and stratigraphic generation from partially observed drillholes when there are shifts in both spatial and geological distribution. It presents DrillBench, which includes 49,671 drillholes from Western Australia, and sets up tasks involving next-layer prediction and free-running continuation for cases that are in-distribution, spatially separated, and across provinces. The method proposed combines layer-level autoregressive modelling with pretraining on about 112,000 historical drillholes and the retrieval of neighbouring lithology, without directly basing the predictions on absolute coordinates or geochemical assays. The evaluation involves statistical, geostatistical, machine-learning, and neural baselines and is supported by multi-backbone comparisons, component ablations, controls regarding the pretraining data, and bootstrap uncertainty estimates. The results indicate that directly conditioning on spatial and geochemical information can enhance local accuracy but may harm transfer, whereas retrieval mainly helps with local prediction and pretraining generally improves next-layer prediction under distribution shift.

Critical strengths
  1. DrillBench creates a standardized resource by compiling 49,671 real-world drillholes for use in next-layer prediction and autoregressive continuation. The reliability of the data is enhanced through extensive data curation, the harmonization of coordinates, and the identification and elimination of duplicates, while the use of explicit spatial and inter-provincial splits allows for the assessment of geographical generalisation beyond what conventional random splits can assess.
  2. The experiments indicate that spatial and geochemical conditioning can enhance local accuracy but can substantially degrade transfer, and they also allow for the characterization of the complementary roles of retrieval and pretraining; supportive evidence for these conclusions comes from broad baseline coverage, multi-backbone comparisons, compute-matched pretraining controls, province-exclusion experiments, and paired uncertainty estimates.
Critical weaknesses
  1. The evaluation does not adequately demonstrate drillhole-specific conditional generation. Task 1 provides ground-truth layer boundaries and thus isolates next-layer lithology classification, leaving transition-boundary prediction untested in this task. Task 2 removes this assumption, but its primary metric compares lithology–thickness distributions pooled across drillholes and rollouts. This aggregation discards stratigraphic ordering and the correspondence between each generated suffix and its observed prefix: a model could reproduce the overall frequencies and thicknesses of rock types without generating an appropriate continuation for a particular drillhole. The supplementary analysis partly acknowledges this limitation. Per-hole sequence edit distance and boundary F1 show little separation among layer-level methods, while a prefix-agnostic layer-marginal generator achieves a relatively close in-distribution KL score. Thus, the reported gains more clearly establish better aggregate distribution matching than improved conditional continuation. Stronger evidence would require evaluating the likelihood of each true suffix given its prefix, testing sensitivity to shuffled or removed prefixes, or demonstrating improvements in sequence-aware conditional metrics.
  2. The breadth of the transfer claims exceeds the evidence's geological coverage and consistency. The spatial splits combine geographic separation with changes in lithology composition, class support, and regolith–bedrock balance, with potentially additional differences in logging practices. Consequently, performance differences cannot be attributed straightforwardly to distance or to a single graded notion of geological shift. Cross-province evaluation covers only Pilbara, limiting evidence that the observed patterns generalize to other geological settings. The proposed components also provide uneven benefits: retrieval strongly improves local next-layer prediction but offers no clear cross-province macro-F1 gain, while the full recipe does not consistently outperform pretraining alone. For free-running generation, its improvement over the plain GRU is significant on Pilbara but not on OOD-Distal, and the appendix explicitly notes that component-wise gains do not hold across all backbones. These findings support a useful, setting-dependent combination of pretraining and retrieval, but do not establish a generally “transfer-safe” approach or show that avoiding direct coordinate and geochemical conditioning is broadly preferable to alternative encodings designed to improve transfer.
Questions for authors
  1. On the issue of the generation claim: Appendix D.2.5 admits that pooled KL does not assess whether each continuation is appropriate for its own prefix and notes that the various methods at the layer level show only minor differences when it comes to the per-hole metrics. Is Task 2 meant to focus on making drillhole-specific conditional predictions or is its main aim instead to match the population-level distribution? If the former is the case, then which existing results show an improved correspondence between individual prefixes and their generated suffixes? This distinction has a significant impact on my evaluation of the paper's main contribution.
  2. What does the term “transfer-safe” entail? Is it meant to refer specifically to an improved average performance on the evaluation sets, or does it imply robustness with respect to different geological settings? Since there is only a single cross-province target and considering that the performance gains are task- and backbone-dependent, could you please make clear the scope you have in mind and point out the evidence that supports it? In particular, in what way should the claimed graded transfer spectrum be understood in the light of the differences in class support and regolith–bedrock composition among the splits?
2 comments

Conditional Generation in Task 2 (W1, Q1)

Thank you for this question, which addresses the main goal of Task 2. (1) Task 2 evaluates whether a generated continuation fits its own hole and is realistic for the region. We now report two metrics: suffix NLL measures how well each hole's true suffix is predicted from its own prefix, while pooled joint KL measures regional realism. (2) Following your suggestion, we tested different prefixes while keeping the true suffix fixed. Replacing the prefix with one from another hole, either in the same report or another report, or removing it significantly worsens our framework's suffix NLL on all four splits. The layer marginal, which ignores the prefix, barely changes. (3) Our framework's per-hole improvement is in predicting the next lithology: compared with the plain layer-level GRU, it significantly improves the lithology term of suffix NLL on all four splits. The improvement has the same direction across all five backbones. It also lowers transition KL and depth-band KL. Task 1 provides true boundaries to evaluate next-lithology prediction separately; Task 2 evaluates boundary placement. A difference is significant when its 95% interval, obtained by resampling whole exploration reports, excludes zero.

Details and manuscript changes follow.

(1) Task 2 measures both per-hole fit and regional realism. Joint KL pools layers across holes and generated continuations. It therefore measures regional realism, which explains why a generator that ignores the prefix can still score well. The rollout-based per-hole metrics in Appendix D.2.5 show little difference between methods, because a single generated continuation need not reproduce a hole's exact observed layers. Suffix NLL instead scores the true suffix directly and distinguishes these methods: every autoregressive network and the layer Markov model achieve lower suffix NLL than the layer marginal on all four splits.

Manuscript changes: Section 3.2.2 defines both metrics and explains what they measure. Section 6.1 reports joint KL in Table 2 and suffix NLL in Figure 4. Table 21 in Appendix D.2.6 lists suffix NLL for every Task-2 method.

(2) Generation depends on each hole's own prefix. We keep the true suffix, depths, and location fixed and change only the prefix. For our framework, suffix NLL increases by 0.027–0.048 nats per 1 m cell with a prefix from another report, by 0.012–0.025 with a prefix from another hole in the same report, and by 0.010–0.020 with no prefix. All increases are significant on all four splits. Swapping prefixes within the same report has a smaller effect than swapping across reports, consistent with nearby holes having similar sequences.

The other autoregressive networks and the layer Markov model show the same pattern, with one non-significant case. The layer marginal barely changes. The plain layer-level GRU also shows a similar increase, so prefix use is shared by the autoregressive models. The additional benefit of our framework is described in (3).

Manuscript changes: Appendix D.2.6 reports the cross-report prefix swap in Table 19, also shown as open markers in Figure 4. Table 20 reports the same-report swaps and prefix removal.

(3) Our framework improves next-lithology prediction. We split suffix NLL into a lithology term, which scores the next lithology, and a boundary term, which scores whether a new layer starts. Compared with the plain layer-level GRU, our framework significantly improves the lithology term on all four splits, mainly through retrieval. The improvement has the same direction across all five backbones, with 16 of 20 comparisons significant.

Our framework also achieves lower transition KL and depth-band KL than the plain GRU on all four splits. We computed these as supplementary checks, since transition KL remains large for every method. The boundary term does not improve. As a result, our framework's overall suffix NLL is close to that of the plain GRU, while cell-level networks perform best on three of four splits.

Manuscript changes: Table 22 in Appendix D.2.6 reports both terms for every backbone. Section 3.2 explains why Task 1 provides true boundaries and Task 2 evaluates boundary prediction.

Scope of the Transfer Claims (W2, Q2)

Thank you for asking us to clarify the scope. (1) "Transfer-safe" referred to a design choice: the framework never reads coordinates or assays as inputs, so it avoids the transfer boundary these inputs create. It did not mean robustness to every geological setting, and we removed the term. Our claims now state where each component helps: direct location and geochemistry inputs reduce macro-F1 on all three shift settings, pretraining improves it on all three, and retrieval helps on Test and OOD-D. (2) The settings differ in distance, lithology composition, and class coverage, so we no longer describe them as a single graded spectrum. Comparisons use the same holes within each setting, and the main transfer findings hold after removing TOPS or regolith layers. The Pilbara remains the only cross-province target, which we state as a limitation; the two transfer patterns above also hold on both Eastern Goldfields rings. (3) The uneven gains are part of the findings: retrieval helps near existing drilling, while pretraining transfers more consistently to new areas. The full framework is never significantly worse than either component alone. (4) Alternative location and geochemistry encodings reduce the loss caused by direct inputs but do not match the lithology-only model on any shift setting. Significance is assessed using 95% intervals from resampling whole exploration reports. Details and manuscript changes follow.

(1) Scope of the claims. The framework uses lithology from the current hole and retrieved neighbours. Location is used only to select neighbours; coordinates and assays are not model inputs. We compare methods separately within each setting, rather than averaging across settings. Against the plain layer-level GRU, pretraining significantly improves macro-F1 on all three shift settings. The full framework significantly improves Test and OOD-D; its gains on OOD-R and Pil-T are positive but not significant.

Manuscript changes: We removed "transfer-safe." Section 1 states where each component helps, and Appendix D.1 reports paired differences with report-level intervals.

(2) What the shift settings test. The settings are ordered by distance from the training data, but also differ in regolith share and available lithology classes, particularly TOPS (transported overburden). We therefore do not treat distance as the only source of shift.

Removing TOPS preserves the direction and significance of every comparison. When all regolith layers are removed, direct location and geochemistry inputs still reduce macro-F1 on all three shift settings. Section 6.2 also separates performance by regolith cover and bedrock: retrieval helps mainly in regolith, while pretraining helps mainly in bedrock. The Pilbara is the only cross-province target, but the two transfer patterns also hold on both Eastern Goldfields rings; adding more provinces will require harmonising their logging vocabularies.

Manuscript changes: Sections 1 and 4.4 clarify the differences between settings. Appendix A states the cross-province limitation, and Appendix D.1.6 reports the label checks.

(3) Different components help in different settings. Retrieval provides the large gain on Test and also helps on OOD-D. Pretraining improves all three shift settings but adds no gain on Test. The full framework significantly outperforms pretraining alone on Test and OOD-D, shows no significant difference on OOD-R and Pil-T, and is never significantly worse than either component alone.

Generation gains are also uneven. The joint-KL gain is significant only on the Pilbara and only for the GRU. In contrast, the lithology term of suffix NLL improves on all four splits, with the same direction across all five backbones.

Manuscript changes: Section 6.2 explains where each component helps. Appendix D.2.4 clarifies that component gains are not consistent across all backbones, and the Conclusion states the remaining open problems.

(4) Alternative encodings. Using the same cell-level network, we tested relative location encodings, alternative normalisations, a log-ratio geochemistry encoding, and a combined encoding. Relative location reduces the loss compared with absolute coordinates, but no tested location or geochemistry encoding matches the lithology-only model on any shift setting. Learning when to use these inputs, through methods such as gated conditioning or domain-adversarial training, remains future work.

Manuscript changes: New Appendix D.1.7 reports all tested encodings with report-level intervals.

Reviewer_FhaCOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces DrillBench, comprising 49,671 Western Australian mineral-exploration drillholes, for next-layer lithology prediction and free-running continuation across local, distant within-province, and Pilbara cross-province splits. Task 1 supplies true boundaries; Task 2 supplies the upper half and final depth. A layer-level sequence model predicts lithology and thickness, with pretraining on approximately 112,000 historical holes and retrieval of neighbouring lithology. The strongest contribution is the benchmark and empirical comparison of information sources under shift. Table 2 shows large local retrieval gains and usefull transfer from pretraining, while generation gains are clearest on Pilbara and depend on the evaluation measure.

Critical strengths
  • I find the explicit causal task and the scale of the released benchmark valuble. Separating prediction at known boundaries from continuation avoids conflating two different problems, and the distant splits expose failures hidden by local evaluation. Unlike the masked multi-curve representation learning in WLFM: A Well-Logs Foundation Model for Multi-Task and Cross-Well Geological Interpretation, DrillBench asks what can be predicted before deeper observations exist. WLFM is relevant context, but its input requirements make an unchanged implementation an unsuitable baseline here.
  • The ablations substantially strengthen the empirical contribution. Table 2 separates retrieval and pretraining, Tables 11 and 15 test five backbones, and Table 24 removes Pilbara, or both Pilbara and Eastern Goldfields, from pretraining. Without retrieval, the latter setting still reaches 30.8% Pilbara macro-F1 versus 28.2% without pretraining. This supports learning transferable sequence information beyond simply seeing the target province. The data card also candidly documents coordinate corrections, duplicates and logging limitations, making the resource easier to scrutinise.
Critical weaknesses
  • W1: Section 3.2.2 pools lithology-thickness counts across holes, discarding layer order and prefix association, while Appendix D.2.5 reports essentially flat per-hole metrics (marginal-generator Test KL 0.211 versus 0.186 for the full model). Better pooled KL therefore supports regional realism, but does not establish better conditional continuations.
  • W2: The supplied scripts/baselines/sis.py:55-79 independently samples fixed kriging posteriors, unlike Integration of Soft Data Into Geostatistical Simulation of Categorical Variables, §2.2.1, which conditions on earlier simulated values. This simplified implementation cannot substantiate Section 3.2.2's broad claim that simulation cannot extend holes, although it does not establish which code produced the reported runs.
  • W3: Appendix B.1 maps only 68.2% of intervals, while the supplementary data card, lines 281-283, documents differing logging conventions and TOPS frequencies of 29% versus 0.01% among mapped Pilbara/training intervals. Without independent validation of harmonisation, some apparent geological transfer difficulty may instead reflect vocabulary differences and selective missing supervision.
  • W4: Task 1 resamples holes (Appendix D.1.1/Table 10), although Task 2 recognises dependence within reports; Appendix D.1.6 shows Pilbara gains concentrated in a few large reports, with 20 report wins and 18 losses against Markov. Hole-level pairing does not address dependence between neighbouring holes, leaving uncertainty about generalisation across projects.
  • W5: run/task1_main.sh:18-22 sets plain-model smoothing to 0.0 versus Appendix C.1.5's 0.3; eval_layerlm.py defaults to 13-class retrieval versus Section 5.3's 37, and Table 14's Markov difference (+0.036) conflicts with Table 2 (-0.009). These discrepancies do not establish which settings produced the reported runs, but prevent confident attribution of gains to the stated components.
Questions for authors
  • For W1, do conditional suffix scores or prefix-shuffling controls distinguish the full recipe from the plain GRU and marginal generator, and can the order-sensitive transition/depth-band metrics already present in eval_cont_task2.py be reported?
  • For W2, was any baseline run with genuine sequential updating of conditioning data? If so, which implementation and allowed observations were used, and how does it compare with the simplified released code?
  • For W3, what independently checked mapping accuracy is available by province and report, and do the transfer conclusions persist after harmonising the most consequential terminology differences?
  • For W4, do the main Task 1 gains survive paired resampling of whole reports or camps, particularly the comparatively modest Pilbara improvement over the plain layer GRU?
  • For W5, which smoothing values and retrieval dimensions generated each main-table row, and which layer-Markov KL and paired differences are authoritative?
2 comments

Prefix Conditioning and Sequential (W1–W2)

Thank you for checking the evidence behind these claims so carefully. (W1) Task 2 now includes suffix NLL, which scores each hole's true suffix given its own prefix. Replacing the prefix with one from another report makes the true suffix significantly less likely on all four splits, showing that generation uses each hole's own prefix. Our framework also significantly improves the lithology term of suffix NLL, which scores which lithology comes next, on all four splits. (W2) We replaced the simplified simulation baseline with sequential indicator simulation (SIS), which conditions on the hole's own earlier cells. It is competitive with our framework across all layer starts. At lithology transitions, however, our framework performs significantly better on all three shift splits, and in Task 2 it achieves a significantly lower suffix NLL on every split. We assess significance by resampling whole exploration reports: a difference is significant when its 95% interval excludes zero. More details see below and updated paper.

(W1) Per-hole evaluation and prefix conditioning. Joint KL pools layers across holes, so it measures regional realism rather than fit to individual holes. The rollout-based per-hole metrics in Appendix D.2.5 rely on a single sampled continuation and show little separation between methods. We therefore added suffix NLL to score the true suffix directly. Replacing the prefix with one from another report significantly worsens suffix NLL on all four splits for every model that uses the prefix. The layer marginal, which ignores the prefix, barely changes. Every autoregressive network and the layer Markov model also achieve lower suffix NLL than the layer marginal on all four splits.

We further split suffix NLL into a lithology term, which scores which lithology comes next, and a boundary term, which scores whether a new layer starts. Compared with the plain layer-level GRU, our framework significantly improves the lithology term on all four splits, but not the boundary term. It also achieves lower transition KL and depth-band KL on all four splits. We report these two additional metrics as supplementary results; transition KL remains large for every method.

Manuscript changes: Section 3.2.2 defines both Task-2 metrics. Section 6.1 reports joint KL in Table 2 and suffix NLL in Figure 4. Appendix D.2.6 reports the prefix-swap results in Table 19, suffix NLL for all methods in Table 20, and its lithology and boundary terms in Table 21.

(W2) Sequential SIS. The simulation baseline in the submission used the released sis.py, which samples each depth independently from the indicator-kriging posterior without conditioning on the hole's earlier cells. Sequential SIS instead conditions each cell on neighbouring archive holes at the same depth, excluding holes within 25 m, and on the hole's previous 10 cells, observed or already simulated.

This is a substantially stronger baseline: Task-1 macro-F1 on the shift splits rises from 6–9% to around 30%. Across all layer starts, it is competitive with our framework. At lithology transitions, where the task is to predict which lithology follows the mapped layer above, our framework performs significantly better on all three shift splits, by 7–18 percentage points. In Task 2, our framework achieves significantly lower suffix NLL on every split and lower joint KL on all three shift splits.

Manuscript changes: Section 3.2.2 removes the claim that simulation cannot extend holes and adds sequential SIS as a generation baseline. Appendix C.1.2 describes its conditioning data. Tables 2, 9, 16, and 20 report sequential SIS, with the released implementation retained as "IK posterior sampling." The sequential SIS code will be included in the code release.

Label Harmonisation, Report-Level Resampling and Code Consistency (W3–W5)

Thank you for carefully checking the label mapping, statistical evidence, and consistency between the code and paper. (W3) The mapping was checked by geologists, and the transfer conclusions remain unchanged after removing TOPS layers and when scoring only bedrock layers. (W4) With whole exploration reports resampled, the pretraining gains remain significant on all three shift splits, including Pilbara. (W5) All three inconsistencies were errors on our side: the released script, not the paper, had the wrong smoothing; Section 5.3 misstated the retrieval classes; and the layer Markov value in Table 2 was outdated. All are corrected. Further details and manuscript changes follow.

(W3) Label harmonisation. Descriptions that cannot be mapped confidently are excluded consistently for every method. The proportion of mapped intervals is the same in Pilbara and the training region, so Pilbara does not have a lower mapping coverage. After removing TOPS layers (cover), all paired comparisons retain their direction and significance. Whether TOPS layers are removed or only bedrock layers are scored, adding location and geochemistry as direct inputs still reduces macro-F1 on every shift test split. These results support the robustness of our transfer conclusions to these label choices. We will release the preprocessing scripts and full mapping table.

Manuscript changes: Appendix B.1(4) clarifies the mapping procedure and coverage. Appendix D.1.6 and Table 14 report the re-scored comparisons.

(W4) Report-level resampling. We repeated the Task-1 comparisons by resampling whole reports, rather than individual holes, to account for dependence within reports. Pretraining remains significantly better than the plain layer-level GRU on all three shift splits. The full model is significant on Test and OOD-D. On Pilbara, its gain is significant at lithology transitions but not across all layer starts. Resampling whole drilling camps gives similar conclusions, although the pretraining gain on OOD-D becomes borderline.

Manuscript changes: Table 14 reports the report-level paired comparisons. Appendix D.1.3 clarifies which gains remain significant.

(W5) Code–paper consistency. Label smoothing: every plain model was trained with 0.3, as the paper states; the released script was outdated. Retrieval: the code is right. Every retrieval row uses the coarse lithology families defined in the code; Section 5.3 wrongly stated 37 classes. Layer Markov: the value in Table 2 was outdated. We corrected it and removed the claim that our model performs best in that comparison. All layer-level neural models in Table 2 use the same label-smoothing setting, so their differences are not due to different smoothing settings. A separate ablation evaluates the effect of label smoothing.

Manuscript changes: Tables 2 and 16 contain the corrected layer Markov result. Section 5.3 and the caption of Table 2 have been corrected. Table 26 reports the smoothing ablation. The corrected scripts will be included in the code release.

Reviewer_V9ZzOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary:A Useful Benchmark, but Limited Methodological Novelty and Incomplete Generation Evaluation

The paper represents drillhole records as sequences from shallow to deep and introduces DrillBench, containing 49,671 sequences and 37 lithology classes. It evaluates next-layer prediction and free-running generation under in-distribution, spatial-shift, and cross-province settings. The proposed recipe combines a layer-level autoregressive model, pretraining on about 112,000 historical drillholes, and retrieval from ten neighboring drillholes. Retrieval mainly improves in-distribution prediction, while pretraining is more useful under distribution shift.

Critical strengths
  1. DrillBench provides data, graded shifts, two tasks, code, checkpoints, a data card, and a shared evaluation pipeline.
  2. The study covers classical methods and GRU, LSTM, TCN, Transformer, and Mamba, with extensive leakage checks and ablations.
Critical weaknesses
  1. The method combines established components. Retrieval forms an inverse-distance-weighted 37-class lithology distribution and adds it to the input through a small MLP. Related nearest-neighbor augmentation was already studied by kNN-LM [1]. The paper does not introduce a clearly new model, retrieval method, general training objective, or theoretical result.
  2. Task 2 does not directly measure whether generation uses the given prefix, and its component gains are not stable across backbones. Task 1 is also given the true layer boundary, while Task 2 is given the known total depth.

[1] Khandelwal et al., “Generalization through Memorization: Nearest Neighbor Language Models,” ICLR 2020. https://iclr.cc/virtual_2020/poster_HklBjCEKvH.html

Questions for authors
  1. What is the main machine-learning novelty beyond the benchmark and the combination of pretraining and retrieval?
  2. Can Task 2 include the per-hole likelihood of the true suffix, or another metric that tests whether each suffix matches its prefix?
2 comments

Machine Learning Novelty

Thank you for raising this concern. This paper is submitted to the Applications to Physical Sciences area. Beyond introducing a scientific prediction problem and public benchmark for geoscience (1), our ML contributions are extending autoregressive modelling to geological lithology sequences (2), designing pretraining tasks and tokenisation for uncurated drilling records (3), and developing retrieval that incorporates the surrounding geological context while preserving transfer (4). Together, these address challenges shared by sequential scientific data, variable-length segments, incomplete observations and spatial distribution shift, and provide a basis for further methodological research.

Details are provided as below.

(1) Domain contributions in geoscience. We define next-layer prediction following the drilling process: given the layers observed so far, predict the next layer or generate the remainder of the hole (Sections 3.1–3.2). To our knowledge, this is the first depth-wise autoregressive formulation at this scale with a public benchmark. We curate DrillBench from a public government-released archive and establish four test splits with two tasks each, enabling systematic evaluation from prediction near existing drilling to transfer across geological provinces (Section 4.4). Through this benchmark, we identify a transfer boundary: location inputs help near existing drilling but hurt under shift (Section 6.2, Figure 5a). We further distinguish where retrieval and pretraining help: retrieval benefits weathered cover most, while pretraining helps in bedrock and across provinces (Section 6.2, Figure 5d; Appendix D.1.5). By releasing the benchmark, pretraining corpus, checkpoints and code, we make these questions reproducibly testable and support further research relevant to mineral exploration, petroleum well-log interpretation and geotechnical site investigation.

(2) Extending autoregressive modelling to a new scientific modality. We formulate subsurface geology as autoregressive layer-sequence prediction, a setting with spatial distribution shift, incomplete logs and source-specific lithology terminology. Our evaluation separates nearby prediction from cross-province transfer, exposing generalisation failures that nearby evaluation alone would miss. This contributes both a sequence formulation and an evaluation protocol for studying how models generalise across spatially collected scientific records.

(3) Pretraining tasks and tokenisation for uncurated drilling records. (Section 5.2) We design next-lithology and thickness prediction, gap tokens for unlogged intervals, and chemistry-based auxiliary tasks to learn from a large, uncurated archive. Assays provide supervision only during pretraining; the fine-tuned model does not receive them as inputs. This provides a way to use richer scientific measurements for representation learning without requiring their availability at inference. In addition, we also show that tokenisation changes whether pretraining helps: with variable-length geological layers as tokens, pretraining improves performance under shift, whereas with fixed 1 m cells it lowers performance (Appendix D.1.4; Table 37). For the ML community, this demonstrates that tokenisation is a consequential transfer choice: preserving the meaningful segments of a scientific sequence can determine whether pretraining is beneficial.

(4) Layer-wise spatial retrieval of surrounding geological context. (Section 5.3) For each next-layer prediction, we retrieve neighbouring holes from a historical archive, read their lithology at the corresponding depth, and fuse it at the input through a small encoder. The model therefore receives observations of the surrounding geology rather than coordinates as a proxy for that geology. Our experiments show that this design benefits from local context while avoiding the transfer degradation observed with location inputs. It differs from kNN-LM, which retrieves by hidden-state similarity and interpolates at the output. As you suggested, we added a kNN-LM baseline: given the same retrieved neighbours, our input fusion performs better on Test and is equal or slightly better on the other splits (new Appendix D.1.9, Table 15). These findings suggest a strategy for spatial prediction: retrieve relevant environmental observations to inform the prediction, while avoiding direct dependence on a coordinate-to-label mapping that may fail under shift.

Other changes: contribution (iii) in Section 1 now presents the pretraining tasks and the retrieval strategy as our design, and kNN-LM is now cited in Section 2.

Task Definition Related Concerns

Thank you for your comments on the Task 2 evaluation. (1) As you suggested, we now report how likely each hole's true suffix is given its prefix, measured by suffix NLL. A prefix-swap test confirms that generation uses each hole's own prefix: replacing it with one from another report makes the true suffix significantly less likely on all four splits. (2) Across all five backbones, the framework consistently improves next-lithology prediction, but gains in free generation are less consistent and boundary timing does not improve, so we have removed "backbone-agnostic . (3) Task 1 provides the true boundary to isolate lithology prediction; Task 2 evaluates boundary and thickness prediction, and boundary prediction remains difficult under shift for all methods. Total depth only determines when generation stops and is never a model input.

More details and supporting results and manuscript changes follow.

(1) Per-hole evaluation and prefix conditioning. We added suffix NLL to evaluate each hole's observed suffix given its own prefix. In the prefix-swap experiment, we keep the true suffix fixed and replace the prefix with one from another report. This significantly worsens suffix NLL on all four splits for every prefix-conditioned model (by 0.027–0.048 nats per metre for our framework), while the prefix-agnostic layer marginal barely changes. The plain layer-level GRU shows a similar increase, so prefix use comes from autoregressive modelling itself. Our framework performs similarly to the plain GRU on suffix NLL, and cell-level neural-based methods lead on three of four splits; on joint KL, our framework performs best on OOD-D and Pil-T. The metrics capture complementary aspects: suffix NLL measures fit to individual holes, while joint KL measures the aggregate distribution of generated layers.

Manuscript changes: Section 3.2.2 defines both metrics; Section 6.1 reports joint KL in Table 2 and suffix NLL in Figure 4. Appendix D.2.6 reports the prefix-swap results in Table 19 and suffix NLL for all methods in Table 20.

(2) Scope of the claims across backbones. We decomposed suffix NLL into lithology and boundary terms for all five layer-level backbones. The framework improves the lithology term on all four splits for every backbone, significantly in 16 of 20 cases. Joint-KL gains are less consistent: for example, the improvement on Pil-T is significant only for the GRU. The boundary term does not improve significantly for any backbone.

Manuscript changes: Table 21 in Appendix D.2.6 reports both terms for each backbone. We replaced "backbone-agnostic" with "implemented on five backbones" or "various backbones" in the Abstract, Section 1, and Section 5.

(3) Task design. Both choices are intentional and serve evaluation; neither is a model input. Task 1 provides the true boundary so that it measures next-lithology prediction on its own, while boundary placement is evaluated in Task 2. In Task 2, the total depth only tells the generator where to stop, so that generated and true suffixes have the same length and can be compared directly.

Manuscript changes: Section 3.2 clarifies the distinction between the tasks, and Section 3.2.2 explains the stopping rule. Appendix A and the Conclusion identify boundary prediction under shift as an open problem.

Comments on the paper

1 comment

Dear Reviewers,

Thank you again for your time and your careful reviews. We have read all of your comments closely and revised the paper accordingly, in particular on (1) the evaluation of Task 2, where we now measure whether each generated continuation fits its own hole, and (2) the scope of our contributions and claims, which we have clarified and updated to what the evidence supports.

Each of our responses opens with a short, self-contained summary at beginning, followed by further details and the locations of the corresponding changes in the revised manuscript. We hope they address some of your concerns.

If your time permits, we would sincerely appreciate it if you could take a look. Whatever the outcome for the scores, your suggestions have been valuable and have helped us improve this work.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.