acceptodds
Under review as a conference paper at ICLR 2027

Expert-Reviewed Benchmark and Domain-Specialized LLM for Subsurface Well-Log Interpretation

Abstract

Well-log interpretation requires reasoning over long, multivariate, depth-indexed measurements. The oil and gas industry relies on it to identify hydrocarbon-bearing zones, estimate reserves, and plan well completions. Although large language models (LLMs) could accelerate this process, how well they handle it is largely unknown. We introduce WellLogBench, the first expert-reviewed benchmark for this task, built with over 1,000 hours of expert review: 1,075 validated QA pairs across six petrophysical categories and three reasoning levels, from single-formation interpretation to cross-formation and multi-well synthesis. To make raw LAS logs usable by LLMs, we develop Petrophys, a Python petrophysics library that turns million-token depth-indexed logs into compact, lithology-aware structured contexts with reproducible calculations; Dynamic Feature Selection (DFS) then cuts the context supplied to the model by 73.5%. We also present WellLog-LLM, a domain-specialized model trained with direct preference optimization (DPO) and group relative policy optimization (GRPO), then refined through inference-time Group Relative Context Optimization (GRCO). Across 15 evaluated LLMs and VLMs, GPT-5.6 Luna (high) leads the public and private splits at 70.4 and 69.3 Overall score. Our WellLog-LLM reaches 66.0 and 66.4 Overall, outperforming the 1T-parameter open-weight Kimi-K2.6 (65.0 and 65.2) despite having approximately 37× fewer parameters. Under the same DFS context, it raises the Overall score of its Qwen3.5-27B base model from 56.3 to 66.0. We will open-source the benchmark, code, and model weights, providing a reproducible foundation for grounded petrophysical reasoning.

Primary area:
datasets and benchmarks
Keywords:
Large language models, well-log interpretation, petrophysical reasoning, scientific question answering, reasoning benchmarks, context engineering, context optimization, domain-specific fine-tuning.

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_csfDOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the understanding and reasoning capabilities of large language models for subsurface logging data, proposing WellLogBench, the Petrophys structured logging data processing library, and the domain-adapted WellLog-LLM. WellLogBench includes 1,075 question-and-answer samples from 40 wells across three geological basins, covering six types of well log analysis tasks and three levels of reasoning. The authors constructed the dataset through model generation, model review, and manual validation by six senior petrophysicists. To make long logging sequences suitable for LLM processing, the author proposed Petrophys, which converts the raw LAS data into a structured context containing quality checks, lithology segmentation, physical properties, and statistical information. In addition, Dynamic Feature Selection (DFS) selects relevant features based on the problem, reducing the average input length from 56.6k tokens to 15.0k tokens. In terms of the model, the authors based it on Qwen3.5-27B and sequentially used DPO, GRPO, and the GRCO from the inference stage to optimize the model. Table 2 of the paper shows that under unified DFS input conditions, the model's Overall score increased from 56.3 to 66.0. Table 1 shows that it outperformed the other tested open-source weight models, including the Kimi-K2.6 with a score of 65.0.

Critical strengths

1: Benchmarks based on real well logging data and expert review have high research value. Unlike geoscience benchmarks based on textbooks or knowledge Q&A, WellLogBench directly constructs questions around well logging data and its derived attributes. The evaluation scope not only includes individual formations but also cross-formation comparisons within the same well and inter-well analyzes. This design is very important because actual well log interpretation is not a simple knowledge retrieval. The model must utilize multiple logging curves, physical quantities, and geological conditions to form consistent conclusions. In addition, the paper provides relatively sufficient evidence of data quality control. Section 3.3 reports that six experts with over 15 years of experience participated in the review, and approximately 60% of the issues were modified during the expert review stage. This indicates that manual review is not merely a formal check. Although the consistency among experts still needs further verification, this data construction process is already more convincing than benchmarks that rely entirely on automatically generated answers.

2: The combination of structured data processing and model training has achieved a clear performance improvement. The paper does not merely propose a benchmark but further analyzes the input processing and domain adaptation issues of LLMs on this task. DFS reduces the average context by 73.5%, while the controlled experiments in Table 2 show that DPO, GRPO, and GRCO each bring performance improvements. Especially, the DPO stage improved from 56.3 to 62.2, indicating that domain preference training has a practical effect on improving explanation quality. Compared to merely reporting the final optimal results, the authors provide phased experiments, which help in understanding the effects of different optimization methods.

Critical weaknesses

1: The core evaluation of the benchmark overly relies on LLM-as-a-Judge, especially since reference answer matching may not fully reflect the correctness of physical reasoning. The overall score of the paper consists of Response Adequacy, Trustworthiness, Style, and Reasoning Quality, with Response Adequacy accounting for 40%. But the main evaluation model does not directly receive the complete logging context; instead, it primarily scores based on the match between the candidate answer and the expert reference answer. Although the authors have additionally set up a Numerical Judge, it may still be difficult to independently assess whether all the physical explanations in the candidate answers are correct based on other scoring dimensions. This means that two different questions have not been completely separated: - Whether the model has reached a conclusion similar to the reference answer. - Whether the model has reached a reliable conclusion through the correct use of well-logging evidence. For scientific reasoning tasks, the two are not equivalent. The model can provide conclusions close to the reference answer but use incorrect reasoning; it may also arrive at reasonable explanations that differ from the expression of the reference answer. The author conducted expert scoring validation on 50 pairs of responses, reporting ICC(A,1)=0.721. This is valuable evidence, but it is not sufficient to completely rule out judgment bias on complex reasoning problems, especially since the rankings of different models in the papers sometimes differ by only about 1 point. Therefore, the reliability of the evaluation is the main factor influencing my final acceptance judgment.

2: There is a certain degree of goal coupling between model optimization and evaluation standards, and the current experiments are not sufficient to fully confirm that performance improvements represent an enhancement in independent scientific reasoning abilities. Basis: Section 6.1–6.3, Table 2. DPO uses the quality of answers generated by different models, GRPO uses the model's evaluation results and the coverage of reference answers as rewards, while GRCO extracts reasoning rules by analyzing high and low-scoring answers. These training stages all encourage the model to generate responses that align with the reference answers and evaluation criteria to some extent. Therefore, performance improvement may involve two types of factors: 1. The true enhancement of well log interpretation capabilities. 2. The model is better at meeting the answer structure, evaluation criteria, and expression preferences of the current benchmark. Especially since GRCO does not update the model weights, but rather incorporates heuristic rules during inference, its gains may stem from better answer organization and coverage of reference answers, rather than a complete improvement in underlying scientific reasoning capabilities. The paper uses wells outside of the Benchmark for training, which reduces the risk of direct data leakage, and this is commendable. But separating data sources does not automatically ensure the independence of evaluation objectives and training rewards. It is recommended to add independent physical quantity checks, different evaluation systems, or expert blind reviews to further verify the results.

Questions for authors

1: Can the small performance differences between models be validated through a larger-scale expert blind evaluation? In Table 1, WellLog-LLM is 66.0, and Kimi-K2.6 is 65.0. Although Appendix L reports statistical significance tests, these tests are based on automatic scoring. Can experts directly compare the scientific correctness and evidence usage of these models on the same logging problem without knowing the model identities? Especially interested in understanding cases where automatic evaluation disagrees with human judgment.

2: Is it possible to include scientific accuracy metrics that are independent of the current LLM Judge? For example, for issues such as porosity, water saturation, formation boundaries, and reservoir identification, can we recalculate verifiable physical results from the original logging data and compare them with the model's answers? Such an evaluation can reduce reliance on linguistic similarity and the expression methods of reference answers.

3: Can we further distinguish between the improvement in scientific reasoning of GRCO and the effect of prompt optimization? It is recommended to compare the complete GRCO, randomly selected empirical rules, manually crafted domain rules, uniformly long general prompts, and rule-free versions. If GRCO still significantly outperforms these controls, it can more convincingly demonstrate that the automatically induced heuristics indeed contribute transferable domain knowledge.

4: What are the consequences of DFS making errors in feature selection? DFS selects features based on the schema without reading the values. If key physical curves are not selected, the model may not be able to provide a correct interpretation. Can you report the recall rate of necessary features by DFS, as well as the omissions under different inference levels and different logging categories? Additionally, can we compare "the same training model under complete context and under DFS context" to eliminate the impact of model training adaptation on the evaluation of compression effects?

6 comments
AuthorsOpenReview ↗

Author Response 1/6

We thank the reviewer for the detailed and constructive review, and we are glad that the reviewer values our benchmark built on real well-log data with expert review. We hope that our detailed response below addresses all the weaknesses and questions raised by the reviewer, including the questions on the judge, the training signals and DFS.

W1: Reliance on LLM-as-a-Judge

We agree that matching the reference's conclusion and reaching a reliable conclusion from the logs are two different things. The rubric scores them separately.

1. The judge scores 13 sub-metrics separately. Response Adequacy compares the answer with an expert-verified reference, and Reasoning Quality judges the argument on its own. Senior petrophysicists reviewed every reference. They re-checked the values, depths and interpretations against the raw logs in Petrel and Techlog, and corrected about 60% of the items (see §3.3, Appendices C and D). So the log evidence is built into the reference. The judge never gives a single match score. It scores each sub-metric on its own and explains each score in writing. A rubric group is the average of its sub-metrics (see §4.2) .

Rubric group (weight)Sub-metrics, each scored separately
Response Adequacy (40%)Accuracy (facts and conclusions agree with the expert), Petrophysical Claims (porosity, saturation, pay and fluid claims agree with the expert's), Conclusion (final interpretation), Coverage (key intervals, values and findings are included)
Trust (20%)Faithfulness (no unsupported facts), Data Limitations (uncertainty and missing logs are respected), Log Utilization (the logs the expert relies on are used), Numerical Accuracy (depths, cutoffs and derived values; scored by a separate judge that also reads the log context)
Style (20%)Clarity, Conciseness, Repetition
Reasoning Quality (20%)Scientific validity of the claims and Chain of Thought (logical progression from data to interpretation), both scored independently of the reference match (§4.2)

2. The sub-metrics separate the two questions in part. The reference records the interpretation that the logs of that well support. The judge does not ask an answer to follow the expert's route to it. An answer that reasons in its own way gets one score for what it concludes (Conclusion, Accuracy) and a separate score for whether its reasoning holds (Reasoning Quality, which does not depend on matching the reference). Sound analyses of the same logs usually arrive at the same interpretation, so the two scores usually agree. When they do not, the sub-metrics show it: an answer with the right conclusion but weak support scores well on Conclusion and loses on Faithfulness. The blind evaluation in Q1 was done without the reference and looks at the remaining cases. We release all sub-scores with the judge's explanations. Appendix K.4 uses them to tell numeric errors apart from wrong conclusions, missing caveats and misused logs.

3. The judge agrees with experts nearly as well as experts agree with each other (Appendix E). The judge and the experts agree at ICC(A,1) = 0.721, and the two experts agree with each other at 0.765. Where the judge agrees less with the experts, the experts also agree less with each other: 0.47–0.64 on Accuracy, Coverage and Petrophysical Claims, against 0.71–0.76 on Faithfulness and Numerical Accuracy.

AuthorsOpenReview ↗

Author Response 2/6

Q1 (and W1): Blind expert evaluation of WellLog-LLM and Kimi-K2.6

The 1-point lead over Kimi-K2.6 also holds when the confidence intervals resample whole wells. Appendix L resamples questions. Taking the wider of the question-level and the well-level interval, the lead is +1.00 [+0.09, +1.91] on the public split and +1.22 [+0.30, +2.16] on the private split.

As suggested by the reviewer, we have performed an additional blind evaluation with the help of petrophysical domain expert. For the blind evaluation, a petrophysicist compared WellLog-LLM with Kimi-K2.6, a 1T-parameter open-weight model . We used 60 single-formation questions, 10 drawn at random from each category. The two answers were shown in random order, with no model names and no reference. The petrophysicist checked the claims against the structured formation data, scored both answers on five criteria (1–5) and said which one was better. The computed labels in the context were read as evidence to interpret, never as ground truth.

1. WellLog-LLM is preferred over Kimi-K2.6. The petrophysicist preferred WellLog-LLM's answer on 37 of the 60 questions and Kimi-K2.6's on 23. The preference for WellLog-LLM was clear or strong on 24 of its 37 questions. For Kimi-K2.6 it was clear or strong on only 8 of its 23, and most of its wins were slight.

2. WellLog-LLM produces sounder petrophysics.

Criterion (1–5)WellLog-LLMKimi-K2.6Difference
Scientific correctness3.432.80+0.63
Limitations and uncertainty4.003.13+0.87
Evidence use3.533.40+0.13
Data fidelity (numbers, depths)3.783.85−0.07
Answers the question directly3.733.80−0.07

The two largest differences are on the criteria that matter most for decisions. A computed property is only valid under certain conditions, and part of interpreting is knowing when it is not. The petrophysicist's rationales show that WellLog-LLM spots these cases: a movable-hydrocarbon index from a fallback flushed-zone saturation, low water saturation in shale-rich or washed-out intervals, solver mineral volumes that the measured density rules out, and FZI classes driven by porosity alone. So it avoids false hydrocarbon calls. Kimi-K2.6 quotes values slightly more exactly and writes shorter answers, but it more often turns such values into confident conclusions. For example, it named a 100% washed-out, shale-rich interval with a mean water saturation of 0.97 (the pores are 97% water) as the "definitive" mobile-hydrocarbon zone. It also stated, incorrectly, that anhydrite raises neutron porosity. In formation evaluation, a precise number attached to a wrong conclusion is the more costly error.

3. The advantage is in the interpretation-heavy categories. WellLog-LLM wins 7–3 in each of permeability, porosity and saturation, and 6–4 in hydrocarbon. The two models are level (5–5) in lithology and numerical questions.

4. The benchmark judge and the blind evaluation agree. The table covers the 50 questions where both the petrophysicist and the judge picked a clear winner.

Petrophysicist prefers WellLog-LLMPetrophysicist prefers Kimi-K2.6
GPT-5 judge prefers WellLog-LLM276
GPT-5 judge prefers Kimi-K2.6512

They chose the same model on 39 of these 50 questions (78%). On this sample the judge also scores WellLog-LLM higher, 67.5 against 64.5.

Most of the 11 disagreements are close calls. In them, one answer added analysis that the reference does not cover: more data-quality caveats, extra intervals, or an explanation of why a derived quantity is not diagnostic. Response Adequacy compares an answer with the reference, so it gives no extra credit for this. The petrophysicist read the answers without the reference and did give credit. When re-checked against the formation data, most of this added content was supported.

This additional blind evaluation, with the per-question verdicts and rationales, will be included in Appendix E of the final draft.

AuthorsOpenReview ↗

Author Response 3/6

Q2 (and W1): Scientific accuracy metrics independent of the LLM judge

The evaluation already uses physical results calculated from the original logs. As a supplementary experiment, we also ran a code-only check of the kind described in the question, on the four items it names. This preliminary version shows what such a metric can measure and what it cannot.

1. The judge scores 13 granular sub-metrics, and one of them already uses recalculated results. Long-form interpretations can be phrased in many valid ways (§4.1), so the rubric is designed to limit the reliance on wording. The primary judge gives no single match score. It scores 12 sub-metrics separately and explains each score (W1). Each asks a specific question about the content: does the final interpretation agree with the expert's, do the petrophysical claims hold, are the limits of the data respected. The thirteenth, Numerical Accuracy, is scored by a separate judge against the log context, which holds the porosity, water saturation, permeability and reservoir and pay flags that Petrophys recalculates from the raw logs (§3.2).

2. As suggested by the reviewer, we perform a supplementary experiment replacing the LLM judge with a code which reads each answer, with no LLM, and runs three checks against the results that Petrophys recalculates from the logs. They cover the four items named in the question:

  • Porosity and water saturation values. A value quoted for a lithology interval is a match when it equals the recalculated value for that interval. For example, an answer gives a density porosity of 0.263 for 1800.5–1803.0 m, and the recalculated mean there is 0.263.
  • Formation boundaries. The stated top and base are a match when they equal those of the dataset.
  • Reservoir identification. When an answer calls an interval reservoir or pay, or rules it out, it is a match when the recalculated flag says the same.

We ran the checks on all 347 single-formation questions for 15 models, ranked by Overall and grouped in fives (Top 5, Middle 5, Bottom 5).

2a. Capable models already quote the computed values correctly, so this check cannot tell a good interpretation from a poor one. Petrophys computes these values from the logs, and the models receive them in their context. When a Top 5 model quotes a porosity value for an interval, it is the value computed for that interval 95% of the time. For water saturation it is 97%. The Middle 5 do just as well (93% and 97%), yet their Overall scores are more than 10 points lower. Only the Bottom 5 often attach a value to the wrong interval (65% and 73%). So the check catches models that misreport the data, but it gives the same pass to models whose interpretations differ widely in quality. Formation boundaries are inputs taken from the source datasets (§3.2). The three groups differ little on them, with 85–91% of answers correct, against 92% for the expert references.

2b. For questions like reservoir identification, the final call needs more than the cutoff flags. Petrophys computes reservoir and pay flags from the logs. Each flag is a cutoff test applied to every depth sample, and it reports whether the cutoff is passed. We compared the flags with the benchmark's reference answers, which petrophysicists reviewed and corrected. The experts' final call matches the flag only in 66% of the intervals they call . In the other 34%, the experts use further evidence and reach a different call.

The reason is that a log response can have several causes. A flag turns one response into a yes or a no. The expert has to work out what caused the response in that interval, and that takes evidence the flag does not hold:

  • In the Sognefjord Fm. of well 35/11-7, the flags mark a neutron–density crossover, a pattern often read as gas. The reference explains that, without a shear sonic log, gas cannot be told apart from light oil, lithology, clay-bound water or borehole effects, and it stops short of a gas call.
  • In the Hod Fm. well elin-1, thin intervals carry prime-oil flags. But the vertical trends in saturation, bulk volume water and resistivity show no contact, and the expert calls the chalk water-bearing.

Other indicators behave the same way. High resistivity can mean hydrocarbons, tight rock or fresh formation water. A computed water saturation is only as reliable as the saturation model behind it, and the right model changes with clay content and lithology. Which explanation applies depends on the other logs in the well, their quality, the lithology and the neighbouring intervals, and these factors interact. Each one can be coded. But then every combination needs its own rule and its own cutoffs. The result would be one more rule-based interpretation, and no longer a physical result that can be verified.

A metric that rewards agreement with the flags would give its best score to an answer that just repeats them. It would also mark the experts wrong in a third of their calls.

Reviewer_TRMDOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

WellLogBench contains 1,075 expert-reviewed questions over 40 wells, spanning single-formation, cross-formation, and multi-well interpretation. Petrophys converts LAS measurements into structured, lithology-aware contexts; dynamic feature selection reduces answer-context tokens by 73.5%. The paper evaluates multiple models with reference-based judges and introduces a Qwen3.5-27B system using DPO, GRPO, and a learned inference-time rule library. Under the same DFS condition, Overall improves from 56.3 to 66.0, with private-split results and expert–judge validation.

Critical strengths
  1. The paper introduces an expert-reviewed benchmark with 1,075 QA pairs across six petrophysical categories and three reasoning levels, supported by substantial domain-expert validation.

  2. The paper provides a comprehensive evaluation across multiple model families and demonstrates consistent improvements through domain-specific context engineering and post-training.

Critical weaknesses
  1. Although training and benchmark wells are separated, the evaluation does not systematically examine how performance generalizes across wells with different geological characteristics.

  2. Despite the extensive model comparisons, the paper provides limited analysis of the underlying failure patterns and what they reveal about the reasoning capabilities and limitations of different LLMs.

Questions for authors
  1. How does model performance vary between wells with similar versus substantially different geological characteristics?
4 comments
AuthorsOpenReview ↗

Author Response 1/3

We are glad that the reviewer recognized the expert validation behind the benchmark and the breadth of our evaluation. We thank the reviewer for the careful review and the two points raised, which led us to extend our analyses and further strengthen the paper. The results below show that the gain from post-training holds across three geologically different basins and in every lithology. A closer look at the errors shows that even the strongest models struggle mainly with interpretation: they quote the right numbers, and the hard part is drawing the right conclusion from them. Furthermore, our domain specialized LLM post-training can substantially improve this complex interpretation task requiring multi-step reasoning. Detailed responses to the weaknesses and questions asked are given below.

W1 and Q1: Generalization across wells with different geological characteristics

The paper reports results by basin (§7.3, with all models in Appendix K.1 and training stages in Appendix K.2), but not by lithology or petrophysical property. We have now run that analysis on the same scores. In short, WellLog-LLM gains about 10 points over its base model in every basin and every lithology including the two basins that were not seen in training.

1.The evaluation benchmark wells come from three geologically different basins, and no benchmark well was used in training. All Wells from geologically different basins - COSTA and Danish are held out and never seen in training. No benchmark well appears in the training data of DPO, GRPO or GRCO, the three stages that make up WellLog-LLM (Appendix J). Appendix J does not name the training basins, so we add this here: the DPO , GRPO and GRCO training data come only from non-benchmark Geolink wells. The 20 Benchmark Geolink wells are therefore new wells from Geolink basin that are never seen during training, while the COSTA and Danish wells come from geologically different basins the model has never seen . The basins differ both in geology and in how Petrophys builds the model's context from their logs:

Geolink (20 wells)COSTA (15 wells)Danish (5 wells)
GeologyNorth Sea; mixed clastics and carbonates, often with interbedded evaporitesKharaib-2 formation, an analog for Middle Eastern carbonate reservoirs: limestone and dolomiteDanish North Sea; mostly chalk and marl
Lithology sourceDepth-by-depth lithology logCalcite and dolomite volumesOne label per formation
Porosity and permeabilityFrom logs, calibrated to core in 6 of 20 wellsFrom coreFrom logs, no usable core
Water saturationRouted by clay volume, lithology and salinityLimestone or dolomite ArchieSame routing as Geolink
Formation water resistivitySP, measured value or clean-wet resistivity, by wellDataset parametersSupplied value

Within each formation, Petrophys also splits the log into lithologically homogeneous intervals, each with its own parameters such as matrix density, Archie coefficients and porosity cutoffs (§3.2). Despite these differences, every basin produces the same context format (Appendix O), so the model always reads the same structure.

2. WellLog-LLM gains about 10 points over its base model in every basin including Costa and Danish that were never seen during training and have different geology. We measure the gain as WellLog-LLM's Overall score minus that of its untuned base model, Qwen3.5-27B (DFS), on the same questions.

BasinWellsQuestionsQwen3.5-27B (DFS)WellLog-LLMGain [95% CI]
Geolink2066756.165.7+9.57 [8.54, 10.61]
COSTA1527156.266.0+9.83 [8.66, 10.99]
Danish513757.869.0+11.26 [9.76, 12.79]

Overall score (0–100) on the full benchmark. Intervals resample wells

The wells from the two unseen basins show no loss compared with the Geolink wells. After adjusting for question type and category, the gain on COSTA differs from the gain on Geolink by +0.53 [−1.28, 2.33], and the gain on Danish by +1.50 [−0.32, 3.33].

AuthorsOpenReview ↗

Author Response 2/3

W1 and Q1 (continued)

3. The gain holds for every lithology, and the same lithology gains about the same amount in different basins. We assign each question the dominant lithology of the formations it asks about.

LithologyQuestionsQwen3.5-27B (DFS)WellLog-LLMGain
Sand10754.764.7+10.00
Silt27355.965.9+9.99
Shale4656.868.4+11.56
Chalk/marl15557.368.1+10.78
Limestone30856.466.2+9.81
Clay-rich13456.763.9+7.24
Other, unlabelled or not assigned5257.168.7+11.58

Chalk/marl gains +10.55 on Geolink wells (86 questions) and +11.07 on Danish wells (69). Limestone gains +10.36 on Geolink wells (27 questions), +9.83 on COSTA wells (268) and +8.09 on Danish wells (13). Every lithology also gains on both the public and the private split. Response Adequacy alone rises by 14 to 15 points in every lithology and by 10 points in clay-rich formations, so the answers agree better with the expert reference. The gain is more than a change of style.

Across ranges of porosity, water saturation, permeability, depth and hole quality, the gain stays between +8.9 and +11.2. Clay content and lithology mixing are the exception. In the most clay-rich and the most mixed formations the gain is lower, at about +7.

4. Moving to a new basin changes only the calculation settings in Petrophys. The model stays the same. Petrophys converts the raw logs into porosity, clay volume, water saturation and permeability. This calculation needs a few facts about the field: where the lithology comes from, which measurements exist, and parameters such as the resistivity of the formation water (table in point 1). A petrophysicist sets the same parameters when starting work on a new field. The interpretation is the model's part, and there nothing changes: the same model, prompt and rules are used in all three basins. COSTA is the clearest case. Its data come in a different form (mineral volumes in place of a lithology log, porosity and permeability from core), and the gain is the same as on Geolink (+9.8 against +9.6) with no COSTA well in training.

This discussion will be included in the final draft. The analysis, with the property ranges and stage-level gains, will be included in Appendix K, and §7.3 will point to it.

AuthorsOpenReview ↗

Author Response 3/3

W2: Failure patterns and what they reveal about different LLMs

We thank the reviewer for the question. We have included a discussion of such failure patterns in §7.1 of the main paper (original submission), and additional details are provided in the appendices. In addition to Points 1 to 4 below, which were already included in the original submission, we have also included a new analysis in Point 5 as suggested by the reviewer.

1. Strong models get the numbers right but the interpretation wrong (Appendix K.4, Table 20). The six strongest models make a numeric error in at most 0.8% of answers. Yet their conclusion is wrong in 10.4–38.1% of answers.

2. Low-ranked models often write answers that contradict themselves, and having more parameters does not prevent it (Appendix K.4, Table 20). An answer contradicts itself when one part denies what another part says. The six top-ranked models do this in at most 8.5% of their answers. The other models do it in 23.6% to 76.6%. The number of parameters does not explain the difference. Qwen3 235B, GPT-OSS-120B and Qwen3 Next 80B have 80B to 235B parameters, and they contradict themselves in 23.6% to 37.6% of their answers. Gemma 4 31B and Qwen3.5-27B have about 30B parameters, and they do so in only 6.8% and 8.5%

3. Post-training removes most of the low-scoring answers and more than halves the wrong conclusions (§7.1, Appendices K.2–K.4). We compare WellLog-LLM with its base model, Qwen3.5-27B, on the same DFS context, so the difference comes from post-training alone. The share of answers that score below 50 Overall falls from 28.5% to 7.1%, and the share with a wrong conclusion falls from 43.4% to 18.9%. The appendices also give example answers.

4. The leading model is better on conclusions and on caveats (Appendices K.4–K.5). GPT-5.6 Luna (high) has a wrong conclusion in 10.4% of answers. It leaves out a needed caveat in 7.7%. For WellLog-LLM the figures are 18.9% and 20.3%. Appendix K.5 compares specific cases.

5. Extended analysis - Weak models mostly report the data wrongly. Strong models mostly report it correctly but draw the wrong conclusion from it.

Table 20 shows how often a conclusion is wrong but does not show why. We use the already evaluated model GPT-5 judge score output explanation statements. An LLM coder read the explanation for each of the 4,857 wrong conclusions on the public split and recorded one main cause. We ranked the 15 models by Overall and put them in three groups of five.

Main cause (wrong conclusions per 100 answers)Top 5Middle 5Bottom 5
Gets the evidence wrong3.110.638.6
Reports wrong values, depths, intervals or labels2.28.724.2
Misses the key finding, or gives no real analysis0.91.914.4
Misjudges the evidence18.636.743.0
Puts an interval in the wrong class (fluid, pay, lithology, pore type)5.210.414.4
Ranks intervals, formations or wells in the wrong order4.27.211.3
Gets a trend, a cause or a method wrong2.86.27.2
Is too sure or too cautious6.412.710.2
All wrong conclusions21.747.381.6

Public split. Each wrong conclusion has one main cause. GPT-5 and GPT-5.6 Luna assigned the causes blindly without the model tag.

The Top 5 make fewer mistakes of every kind, but the two groups of mistakes do not shrink equally. From the Bottom 5 to the Top 5, mistakes in reporting the data fall about twelve times (38.6 to 3.1 per 100 answers). Mistakes in interpreting the data fall only about two times (43.0 to 18.6). So for a strong model, most of the remaining wrong conclusions come from interpretation.

Two interpretation mistakes are the most common. The first is putting an interval in the wrong class, for example calling a non-pay interval oil-bearing. The second is ranking formations or wells in the wrong order. This happens mostly in cross-formation questions, where the Top 5 do it in 7.9 of 100 answers, against 1.4 in single-formation questions.

Post-training seems to reduce these interpretation mistakes. We compared WellLog-LLM with its base model on the same DFS context. Answers with a wrong class fall from 12.6 to 4.3 per 100, and answers that are too sure fall from 9.1 to 1.5.

The labels come from the explanations that the GPT-5 judge writes with each score, which are part of our evaluation system. We tested the labels in two ways. A second coder gave the same main cause for 90% of a 90-answer sample (κ=0.78\kappa = 0.78). We also coded 150 answers twice for the detailed cause, and 80% got the same label both times (κ=0.73\kappa = 0.73).

We will add the full cause table to Appendix K.4 and point to the failure analysis more clearly from §7.1.

Thank you for the rebuttal. The additional analyses provide convincing evidence of generalization and have addressed my main concerns. I am therefore willing to raise my score.

Reviewer_avHTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces WellLogBench, an expert-reviewed benchmark containing 1,075 QA pairs from 40 wells across three basins. It also proposes Petrophys for structured well-log processing and WellLog-LLM, a domain-adapted model trained using DPO, GRPO, and inference-time GRCO. Experiments demonstrate improved interpretation performance over the base model and competitive results against substantially larger LLMs.

Critical strengths

1.Valuable benchmark contribution. WellLogBench addresses an underexplored domain with expert-validated annotations covering multiple petrophysical tasks and reasoning levels. The extensive expert involvement strengthens its practical relevance. 2.Comprehensive experimental validation. The paper evaluates a broad range of models and provides stage-wise ablations, judge agreement analysis, and statistical comparisons. The improvement from 56.3 to 66.0 demonstrates the effectiveness of domain-specific adaptation.

Critical weaknesses

1.Unclear contribution of genuine reasoning. Petrophys provides substantial precomputed information, including derived properties and interpretation flags. It remains unclear how much of the reported performance reflects independent petrophysical reasoning rather than the use of engineered features. 2.Limited evidence of generalization. The benchmark covers only 40 wells, with a substantial proportion of questions from one data source. The public/private split is described at the question level, but its independence at the well level is unclear. This limits conclusions about generalization to unseen wells or geological settings.

Questions for authors

1.Does the public/private split ensure well-level independence? Could the authors report performance on held-out wells? 2.How does performance change when derived properties and interpretation flags are removed from the structured context? A simple rule-based baseline would also help clarify the LLM's contribution. 3.Could the authors report end-to-end inference cost, including DFS selection and hierarchical reasoning calls, rather than context reduction alone?

5 comments
AuthorsOpenReview ↗

Author Response 1/5

We thank the reviewer for the careful review, and we are glad that the reviewer appreciated the value of the benchmark, the expert-validated annotations, the performance gain obtained through our domain-specific LLM adaptation, and the breadth of the experiments performed, including stage-wise ablations, judge agreement analysis, and statistical comparisons. Based on the reviewer’s recommendation, we further analyzed the following two points: that the model interprets the evidence itself, and that it generalizes to wells it has not seen. We tested both with new experiments, and both claims hold.

W1: Contribution of genuine reasoning beyond the engineered features

The context contains the evidence, not the conclusions. Interpreting a well log requires the calculation and interpretation steps, and the benchmark mainly focuses on the interpretation.

The first step is a calculation. The raw measurements (gamma ray, density, neutron, resistivity, sonic) are converted into physical properties such as porosity, clay volume, water saturation and permeability, with standard equations. Petrophysicists do this in interpretation software (our experts used Petrel and Techlog, Appendix C). In our pipeline Petrophys does it (§3.2).

The second step is the interpretation. From these properties, the interpreter decides things like where the reservoir is, what fluid it holds and how reliable that call is. No single value or flag gives this answer. To obtain this, the model has to:

  • compare the intervals with one another;
  • check whether porosity, clay volume, water saturation and resistivity agree;
  • decide what to trust when they disagree or when the hole is in poor condition;
  • say what the logs cannot settle.

To measure what each part of the context contributes, we ran an ablation on all 347 single-formation questions from Geolink, COSTA and Danish. We kept the model (WellLog-LLM), the prompt, the experience library, feature selection and the sampling settings fixed, and changed only the context. It has three layers:

Context layerWhat it contains
Measured logsStatistics of the recorded curves (GR, RHOB, NPHI, DT, resistivities, PEF, CALI, SP), core data where measured, lithology grouping
Derived propertiesThe result of the first step: PHIE, PHIT, VCL, SW, SXO, BVW, PERM, RQI, FZI, RW, MHI and others
Interpretation flagsFlags from fixed rules with standard industry cutoffs (NET_PAY, NET_RES, RESFLAG, SEALFLAG, CROSSOVER_FLAG, BAD_HOLE and others), with interval classifications and the bad-hole QC summary

Each row gives the model a different subset of these layers:

Context given to WellLog-LLMLogsDerivedFlagsRATrustStyleRQOverallOverall change [$\pm$95% CI]
All three layers (paper setting, with DFS)✓✓✓65.469.865.079.068.9—
A: no flags✓✓–62.969.365.478.867.86−1.04±1.1-1.04 \pm 1.1
B: no derived properties✓–✓50.055.860.872.757.9−11.0±1.4-11.0 \pm 1.4
C: measured logs only✓––36.949.660.873.951.6−17.3±1.6-17.3 \pm 1.6
D: flags only––✓34.237.665.861.946.74−22.16±1.2-22.16 \pm 1.2

All 347 single-formation questions (public and private), 0–100 scale, same judges as in the paper. The numerical judge checked the numbers against the context each row received. The first row is the paper's WellLog-LLM run; Table 14 reports 68.7 for the public split alone. RA: Response Adequacy (agreement with the expert reference, 0.4 of Overall). RQ: Reasoning Quality.

1. Without the derived properties the score falls by 11.0 (row B) and 17.3 (row C), because the model no longer has the basic input that every interpretation starts from. Porosity, water saturation and permeability depend on parameters such as Archie exponents, matrix and fluid values and water resistivity, and cannot be reliably recomputed from raw curves inside one answer. Even a petrophysicist handed only the raw curves would also have to compute them before interpreting anything. The derived properties do not hold the answer either: every model in Table 1 in the paper receives them, and the models still reach very different conclusions as mentioned in point 6 below.

2. With the flags alone the model scores lowest of all (46.7, row D), because a flag is a cutoff result and the interpretation is the reasoning built around it. If interpreting meant repeating the flags, row D would score near the full context. It scores 22 points lower, below even the measured logs alone (51.6). A flag marks the depths that pass a cutoff. It does not tell how the intervals compare, whether the other indicators agree, or how far the call can be trusted.

3. The flags still make the answers more accurate: without them, Response Adequacy falls by 2.5 points (row A). They help most on data-QC and hydrocarbon-detection questions (point 4). When the derived properties are missing, they add about 6 points (row B against row C).

AuthorsOpenReview ↗

Author Response 2/5

W1 (continued)

By question category, the four ablations score as follows.

Category (n)All layersA: no flagsB: no derivedC: logs onlyD: flags only
Hydrocarbon (56)72.369.959.350.652.0
Lithology (57)68.569.461.661.953.0
Numerical (61)66.763.662.459.249.5
Permeability (66)68.468.052.143.939.5
Porosity (48)69.268.552.646.843.0
Saturation (59)68.767.559.047.444.0

Overall score by category.

Without the derived properties, 22 of the 40 subcategories with at least three questions drop significantly. The largest drops are in average permeability (−26.2), total porosity distribution (−24.1), log-derived permeability distribution (−21.6) and water saturation in a hydrocarbon context (−17.7). Without the flags, the largest losses are on numerical (−3.1) and hydrocarbon (−2.4) questions. With the flags alone, every category falls by 15 to 29 points.

4. The flags help most on the questions they were designed for: data-QC checks and hydrocarbon detection.

Question group (subcategories)nOverall change [$\pm$95% CI]
Data-QC checks (notable deviations, sonic quality, resistivity inversion, spikes and outliers)29−8.0±4.7-8.0 \pm 4.7
Hydrocarbon detection (direct indicators, type discrimination)31−4.6±3.2-4.6 \pm 3.2

Row A against all three layers; subcategories pooled into groups.

QC answers rely on the bad-hole QC summary, and hydrocarbon-detection answers rely on the crossover, pay and prime-target flags. Of the 29 QC questions, 22 score lower without the flags.

5. The model uses the flags as evidence and does not simply repeat them. The table shows three examples from row A, where the flags were removed and the model had to work from the underlying curves.

Question (well)Flags removedHow the model concluded without themConclusion score: no flags / all layers
Pay identification, Ness Fm. (35/11-6)NET_PAY, NET_RES, PRIME_OIL_TARGET (62% and 49% of samples in the main sand) and the pay interval labelsUsed porosity, water saturation and clay volume to place the most significant pay in the same sand (3,531.0–3,536.5 m). Set aside a shallower sand for high clay volume despite favourable porosity and saturation.9 / 10
Hydrocarbon–water contact, Vaale Fm. (34/10-35)Labels marking the whole interval as net pay (92% PRIME_OIL_TARGET)Found no vertical gradient in SW, BVW or deep resistivity, so concluded one fluid package with no contact, matching the reference.10 / 10
Secondary porosity, Amundsen Fm. (35/9-9)The PRIMARY_POROSITY / MATRIX_EFFECT classificationUsed density–sonic separation and PHI_SEC to conclude there is no secondary porosity, attributing the negative separation to a cemented, mineralogically mixed matrix. Named the data needed to confirm this (core, image logs, NMR).9 / 9

Conclusion by Reference sub-score (0–10). Three questions are shown here as illustrative examples.

In the first example, the model sets aside a sand that has favourable porosity and saturation because its clay volume is high. In the second, it checks three curves for a vertical trend before it rules out a contact. Neither conclusion can be read from a flag or from a single statistic. But across all questions the answers are less accurate without the flags (point 3), which is why Petrophys provides them.

6. Models that receive the same context still reach different conclusions. Every model in Table 1 from the paper is given the same Petrophys context. In the failure analysis of Appendix K.4 (Table 20), the six strongest models get a number wrong in at most 0.8% of their responses, but they reach a wrong conclusion in 10.4–38.1% of them. Under the same DFS context, the untuned base model reaches a wrong conclusion in 43.4% of its responses, against 18.9% for WellLog-LLM (§7.1). So the models differ in how well they interpret the same evidence, and this is what post-training improves.

The ablation, with the full subcategory results and the examples, will go into Appendix K, and §3.2 will point to it.

AuthorsOpenReview ↗

Author Response 3/5

W2: Generalization to unseen wells and geological settings

1. None of the benchmark wells was used in training, and the COSTA and Danish basins were not seen in training at all. No benchmark well appears in the training data of DPO, GRPO or GRCO, the three stages that make up WellLog-LLM (Appendix J). Appendix J does not name the training basins, so we add this here: the DPO, GRPO and GRCO training data come only from Geolink wells that are not in the benchmark. COSTA and Danish are therefore basins that the model has never seen, and all the scores we report are on held-out wells.

2. The three basins differ in their geology, and their logs are processed differently.

BasinWellsGeologyHow lithology is obtained
Geolink20North Sea, mostly the Norwegian Continental Shelf; clastics, carbonates and evaporitesDepth-by-depth lithology log
COSTA15Kharaib-2 formation, an analog for Middle Eastern carbonate reservoirs; mainly limestone and dolomiteCalcite and dolomite volumes (porosity and permeability come from core)
Danish5Danish sector of the North Sea; mostly chalk and marlOne lithology label per formation

Appendix B and Table 3 list the formations and lithologies.

3. WellLog-LLM improves over its base model by about the same amount on the two unseen basins as on Geolink. We compare WellLog-LLM with its untuned base model on the same questions.

BasinQuestionsQwen3.5-27B (DFS)WellLog-LLMGain [95% CI]
Geolink66756.165.7+9.57 [8.54, 10.61]
COSTA (unseen basin)27156.266.0+9.83 [8.66, 10.99]
Danish (unseen basin)13757.869.0+11.26 [9.76, 12.79]

Overall score (0–100) on the full benchmark. Intervals are the wider of question-level and well-level resampling. Danish has only 5 wells, so its interval is the widest.

After adjusting for question type and category, the gain on COSTA differs from the gain on Geolink by +0.53 [−1.28, 2.33], and the gain on Danish by +1.50 [−0.32, 3.33]. So there is no detectable loss on the unseen basins.

4. The gain is also similar across lithologies. It is +10.0 on sand, +10.0 on silt, +11.6 on shale, +10.8 on chalk/marl and +9.8 on limestone. Clay-rich formations are counted separately and gain +7.2. The same lithology gains about the same amount in different basins. Chalk/marl gains +10.55 on Geolink wells and +11.07 on Danish wells, and limestone gains +10.36 on Geolink wells, +9.83 on COSTA wells and +8.09 on the 13 Danish limestone questions.

5. These conclusions do not depend on Geolink having the most questions. Because Geolink provides most of the questions, we report every result separately for each basin (§7.3, Appendix K.1). On the full benchmark, the model rankings agree across basins and across lithology families, with a rank correlation of 0.99 or above (0.987 or above on the public split of Table 14).

This analysis will also go into Appendix K, with a pointer from §7.3.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Author Response: Follow-up to Reviewers

Thank you all for your time and constructive feedback.

We are especially grateful to Reviewer TRMD for engaging with our response and raising the score to "4. Clear Accept".

As the author–reviewer discussion period ends tomorrow (November 18), we would like to kindly follow up with Reviewers avHT and csfD. We would be grateful if you could let us know whether our responses have addressed your concerns, or whether any point needs further clarification. We are fully available to engage in discussions and provide additional details in the remaining time.

Thank you again for your consideration.

AuthorsOpenReview ↗

Author Response

Dear Reviewers and Area Chair,

Thank you for your careful reviews and constructive feedback. In addition to the previously provided detailed rebuttal, we have uploaded a revised manuscript in which all changes are marked in blue. Below we summarize the main points and where they appear:

Generalization to unseen wells and geological settings (Reviewers avHT, TRMD; Refer §7.3, App. K.7, L.1). WellLog-LLM generalizes to new wells and to basins with different geology. Training uses only non-benchmark Geolink wells, so every score is on held-out wells, and COSTA and Danish are unseen basins. Petrophys applies a different configuration to each basin (lithology, porosity and permeability source, water resistivity), while the model, prompt and rules stay the same. The gain over the base model is about the same in each (+9.6 on Geolink, +9.8 on COSTA, +11.3 on Danish), and model rankings agree across basins (rank correlation ≥0.99). The public/private split is by question for the leaderboard, and across 1,000 well-disjoint re-splits the gain stays between +8.51 and +12.05.

Reliability of the LLM-as-a-judge evaluation (Reviewer csfD; Refer §4.3, App. E.1, E.2). Interpreting a well log means weighing many aspects at once, so the rubric judge scores them separately. It gives 13 sub-metrics, from the conclusion and petrophysical claims to data limitations and Reasoning Quality (scored independently of the reference). The judge agrees with experts (ICC 0.721) nearly as well as experts agree with each other (0.765). A petrophysicist who saw no model names or references preferred WellLog-LLM over Kimi-K2.6 on 37 of 60 questions and agreed with the judge on 39 of the 50 questions where both expressed a clear preference. The 1-point lead over Kimi-K2.6 also holds with well-level resampling and in all 1,000 well-disjoint re-splits. We have also included an additional experiment with a code-only check, as suggested by the reviewer (App. E.2).

Reasoning beyond the engineered features (Reviewer avHT; Refer §3.2, App. K.6). Well-log interpretation is a complex task. Petrophys computes the derived properties, but the final call requires comparing intervals, checking whether porosity, clay volume, water saturation and resistivity agree, deciding what to trust when they disagree or hole quality is poor, and stating what the logs cannot settle. The context ablation on 347 single-formation questions shows that Petrophys supplies the evidence and the model supplies this interpretation. Against 68.9 with the full context, removing the interpretation flags costs about one Overall point and removing the derived properties costs 11.0. Given only the flags, a rule-based baseline built on fixed industry cutoffs, the score falls by 22.2 points, more than with the measured logs alone (−17.3).

DFS selection errors (Reviewers avHT, csfD; Refer §5.1, App. I). DFS keeps the evidence a question needs at a fraction of the cost. The selector recalls 86.3% of the curves the references cite, up from about 74% for the untuned base model, so post-training also improves the selection. Removing those curves from the selection costs 9.0 points (0.4 for as many other curves), yet the same trained model gains only 0.3–0.4 points with the full context. So the curves DFS misses cost little, and the gains do not come from adapting to the compressed input.

Failure patterns across LLMs (Reviewers avHT, TRMD; Refer §7.1, App. K.4). A cause analysis of 4,857 wrong conclusions shows that strong models mostly report the data correctly and struggle with its interpretation. From the bottom five models to the top five, reporting errors fall about twelvefold but interpretation errors only about twofold, and post-training lowers wrong conclusions from 43.4% to 18.9%.

Coupling between training signals and evaluation, and the GRCO rules (Reviewer csfD; Refer §7.2, App. J.4, K.2). Each training signal rewards better interpretation, and none uses the benchmark's references, wells or judge. DPO gives about 60% of the gain without any judge or reference, and the GRPO reward comes from a frozen Qwen3.5-27B with no term for style. The sub-metrics scored against the expert reference gain the most (Petrophysical Claims +15.2, Conclusion by Reference +14.8, Coverage +13.5, Accuracy +13.2), about three times as much as Clarity (+4.8) and Conciseness (+4.2). The blind evaluation points the same way. Over the same model without rules, the GRCO rules add 1.96 Overall points, with similar gains on the unseen basins, and raise Data Limitations most (+7.6) while answers grow by only 60 words.

We hope these additional experiments and clarifications address the reviewers' concerns, and welcome further discussion.

AuthorsOpenReview ↗

Author Response - Gentle Reminder Discussion Period Closing

Dear Reviewers avHT and csfD,

As the author–reviewer discussion period closes in a few hours, we would like to gently follow up. If our responses have addressed your concerns, we would be grateful if you would consider increasing your scores. If any point remains unclear, please let us know and we will respond right away.

Thank you for your time and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.