- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies the understanding and reasoning capabilities of large language models for subsurface logging data, proposing WellLogBench, the Petrophys structured logging data processing library, and the domain-adapted WellLog-LLM. WellLogBench includes 1,075 question-and-answer samples from 40 wells across three geological basins, covering six types of well log analysis tasks and three levels of reasoning. The authors constructed the dataset through model generation, model review, and manual validation by six senior petrophysicists. To make long logging sequences suitable for LLM processing, the author proposed Petrophys, which converts the raw LAS data into a structured context containing quality checks, lithology segmentation, physical properties, and statistical information. In addition, Dynamic Feature Selection (DFS) selects relevant features based on the problem, reducing the average input length from 56.6k tokens to 15.0k tokens. In terms of the model, the authors based it on Qwen3.5-27B and sequentially used DPO, GRPO, and the GRCO from the inference stage to optimize the model. Table 2 of the paper shows that under unified DFS input conditions, the model's Overall score increased from 56.3 to 66.0. Table 1 shows that it outperformed the other tested open-source weight models, including the Kimi-K2.6 with a score of 65.0.
Critical strengths
1: Benchmarks based on real well logging data and expert review have high research value. Unlike geoscience benchmarks based on textbooks or knowledge Q&A, WellLogBench directly constructs questions around well logging data and its derived attributes. The evaluation scope not only includes individual formations but also cross-formation comparisons within the same well and inter-well analyzes. This design is very important because actual well log interpretation is not a simple knowledge retrieval. The model must utilize multiple logging curves, physical quantities, and geological conditions to form consistent conclusions. In addition, the paper provides relatively sufficient evidence of data quality control. Section 3.3 reports that six experts with over 15 years of experience participated in the review, and approximately 60% of the issues were modified during the expert review stage. This indicates that manual review is not merely a formal check. Although the consistency among experts still needs further verification, this data construction process is already more convincing than benchmarks that rely entirely on automatically generated answers.
2: The combination of structured data processing and model training has achieved a clear performance improvement. The paper does not merely propose a benchmark but further analyzes the input processing and domain adaptation issues of LLMs on this task. DFS reduces the average context by 73.5%, while the controlled experiments in Table 2 show that DPO, GRPO, and GRCO each bring performance improvements. Especially, the DPO stage improved from 56.3 to 62.2, indicating that domain preference training has a practical effect on improving explanation quality. Compared to merely reporting the final optimal results, the authors provide phased experiments, which help in understanding the effects of different optimization methods.
Critical weaknesses
1: The core evaluation of the benchmark overly relies on LLM-as-a-Judge, especially since reference answer matching may not fully reflect the correctness of physical reasoning. The overall score of the paper consists of Response Adequacy, Trustworthiness, Style, and Reasoning Quality, with Response Adequacy accounting for 40%. But the main evaluation model does not directly receive the complete logging context; instead, it primarily scores based on the match between the candidate answer and the expert reference answer. Although the authors have additionally set up a Numerical Judge, it may still be difficult to independently assess whether all the physical explanations in the candidate answers are correct based on other scoring dimensions. This means that two different questions have not been completely separated: - Whether the model has reached a conclusion similar to the reference answer. - Whether the model has reached a reliable conclusion through the correct use of well-logging evidence. For scientific reasoning tasks, the two are not equivalent. The model can provide conclusions close to the reference answer but use incorrect reasoning; it may also arrive at reasonable explanations that differ from the expression of the reference answer. The author conducted expert scoring validation on 50 pairs of responses, reporting ICC(A,1)=0.721. This is valuable evidence, but it is not sufficient to completely rule out judgment bias on complex reasoning problems, especially since the rankings of different models in the papers sometimes differ by only about 1 point. Therefore, the reliability of the evaluation is the main factor influencing my final acceptance judgment.
2: There is a certain degree of goal coupling between model optimization and evaluation standards, and the current experiments are not sufficient to fully confirm that performance improvements represent an enhancement in independent scientific reasoning abilities. Basis: Section 6.1–6.3, Table 2. DPO uses the quality of answers generated by different models, GRPO uses the model's evaluation results and the coverage of reference answers as rewards, while GRCO extracts reasoning rules by analyzing high and low-scoring answers. These training stages all encourage the model to generate responses that align with the reference answers and evaluation criteria to some extent. Therefore, performance improvement may involve two types of factors: 1. The true enhancement of well log interpretation capabilities. 2. The model is better at meeting the answer structure, evaluation criteria, and expression preferences of the current benchmark. Especially since GRCO does not update the model weights, but rather incorporates heuristic rules during inference, its gains may stem from better answer organization and coverage of reference answers, rather than a complete improvement in underlying scientific reasoning capabilities. The paper uses wells outside of the Benchmark for training, which reduces the risk of direct data leakage, and this is commendable. But separating data sources does not automatically ensure the independence of evaluation objectives and training rewards. It is recommended to add independent physical quantity checks, different evaluation systems, or expert blind reviews to further verify the results.
Questions for authors
1: Can the small performance differences between models be validated through a larger-scale expert blind evaluation? In Table 1, WellLog-LLM is 66.0, and Kimi-K2.6 is 65.0. Although Appendix L reports statistical significance tests, these tests are based on automatic scoring. Can experts directly compare the scientific correctness and evidence usage of these models on the same logging problem without knowing the model identities? Especially interested in understanding cases where automatic evaluation disagrees with human judgment.
2: Is it possible to include scientific accuracy metrics that are independent of the current LLM Judge? For example, for issues such as porosity, water saturation, formation boundaries, and reservoir identification, can we recalculate verifiable physical results from the original logging data and compare them with the model's answers? Such an evaluation can reduce reliance on linguistic similarity and the expression methods of reference answers.
3: Can we further distinguish between the improvement in scientific reasoning of GRCO and the effect of prompt optimization? It is recommended to compare the complete GRCO, randomly selected empirical rules, manually crafted domain rules, uniformly long general prompts, and rule-free versions. If GRCO still significantly outperforms these controls, it can more convincingly demonstrate that the automatically induced heuristics indeed contribute transferable domain knowledge.
4: What are the consequences of DFS making errors in feature selection? DFS selects features based on the schema without reading the values. If key physical curves are not selected, the model may not be able to provide a correct interpretation. Can you report the recall rate of necessary features by DFS, as well as the omissions under different inference levels and different logging categories? Additionally, can we compare "the same training model under complete context and under DFS context" to eliminate the impact of model training adaptation on the evaluation of compression effects?
6 comments
Author Response 2/6
Q1 (and W1): Blind expert evaluation of WellLog-LLM and Kimi-K2.6
The 1-point lead over Kimi-K2.6 also holds when the confidence intervals resample whole wells. Appendix L resamples questions. Taking the wider of the question-level and the well-level interval, the lead is +1.00 [+0.09, +1.91] on the public split and +1.22 [+0.30, +2.16] on the private split.
As suggested by the reviewer, we have performed an additional blind evaluation with the help of petrophysical domain expert. For the blind evaluation, a petrophysicist compared WellLog-LLM with Kimi-K2.6, a 1T-parameter open-weight model . We used 60 single-formation questions, 10 drawn at random from each category. The two answers were shown in random order, with no model names and no reference. The petrophysicist checked the claims against the structured formation data, scored both answers on five criteria (1–5) and said which one was better. The computed labels in the context were read as evidence to interpret, never as ground truth.
1. WellLog-LLM is preferred over Kimi-K2.6. The petrophysicist preferred WellLog-LLM's answer on 37 of the 60 questions and Kimi-K2.6's on 23. The preference for WellLog-LLM was clear or strong on 24 of its 37 questions. For Kimi-K2.6 it was clear or strong on only 8 of its 23, and most of its wins were slight.
2. WellLog-LLM produces sounder petrophysics.
| Criterion (1–5) | WellLog-LLM | Kimi-K2.6 | Difference |
|---|---|---|---|
| Scientific correctness | 3.43 | 2.80 | +0.63 |
| Limitations and uncertainty | 4.00 | 3.13 | +0.87 |
| Evidence use | 3.53 | 3.40 | +0.13 |
| Data fidelity (numbers, depths) | 3.78 | 3.85 | −0.07 |
| Answers the question directly | 3.73 | 3.80 | −0.07 |
The two largest differences are on the criteria that matter most for decisions. A computed property is only valid under certain conditions, and part of interpreting is knowing when it is not. The petrophysicist's rationales show that WellLog-LLM spots these cases: a movable-hydrocarbon index from a fallback flushed-zone saturation, low water saturation in shale-rich or washed-out intervals, solver mineral volumes that the measured density rules out, and FZI classes driven by porosity alone. So it avoids false hydrocarbon calls. Kimi-K2.6 quotes values slightly more exactly and writes shorter answers, but it more often turns such values into confident conclusions. For example, it named a 100% washed-out, shale-rich interval with a mean water saturation of 0.97 (the pores are 97% water) as the "definitive" mobile-hydrocarbon zone. It also stated, incorrectly, that anhydrite raises neutron porosity. In formation evaluation, a precise number attached to a wrong conclusion is the more costly error.
3. The advantage is in the interpretation-heavy categories. WellLog-LLM wins 7–3 in each of permeability, porosity and saturation, and 6–4 in hydrocarbon. The two models are level (5–5) in lithology and numerical questions.
4. The benchmark judge and the blind evaluation agree. The table covers the 50 questions where both the petrophysicist and the judge picked a clear winner.
| Petrophysicist prefers WellLog-LLM | Petrophysicist prefers Kimi-K2.6 | |
|---|---|---|
| GPT-5 judge prefers WellLog-LLM | 27 | 6 |
| GPT-5 judge prefers Kimi-K2.6 | 5 | 12 |
They chose the same model on 39 of these 50 questions (78%). On this sample the judge also scores WellLog-LLM higher, 67.5 against 64.5.
Most of the 11 disagreements are close calls. In them, one answer added analysis that the reference does not cover: more data-quality caveats, extra intervals, or an explanation of why a derived quantity is not diagnostic. Response Adequacy compares an answer with the reference, so it gives no extra credit for this. The petrophysicist read the answers without the reference and did give credit. When re-checked against the formation data, most of this added content was supported.
This additional blind evaluation, with the per-question verdicts and rationales, will be included in Appendix E of the final draft.
Author Response 3/6
Q2 (and W1): Scientific accuracy metrics independent of the LLM judge
The evaluation already uses physical results calculated from the original logs. As a supplementary experiment, we also ran a code-only check of the kind described in the question, on the four items it names. This preliminary version shows what such a metric can measure and what it cannot.
1. The judge scores 13 granular sub-metrics, and one of them already uses recalculated results. Long-form interpretations can be phrased in many valid ways (§4.1), so the rubric is designed to limit the reliance on wording. The primary judge gives no single match score. It scores 12 sub-metrics separately and explains each score (W1). Each asks a specific question about the content: does the final interpretation agree with the expert's, do the petrophysical claims hold, are the limits of the data respected. The thirteenth, Numerical Accuracy, is scored by a separate judge against the log context, which holds the porosity, water saturation, permeability and reservoir and pay flags that Petrophys recalculates from the raw logs (§3.2).
2. As suggested by the reviewer, we perform a supplementary experiment replacing the LLM judge with a code which reads each answer, with no LLM, and runs three checks against the results that Petrophys recalculates from the logs. They cover the four items named in the question:
- Porosity and water saturation values. A value quoted for a lithology interval is a match when it equals the recalculated value for that interval. For example, an answer gives a density porosity of 0.263 for 1800.5–1803.0 m, and the recalculated mean there is 0.263.
- Formation boundaries. The stated top and base are a match when they equal those of the dataset.
- Reservoir identification. When an answer calls an interval reservoir or pay, or rules it out, it is a match when the recalculated flag says the same.
We ran the checks on all 347 single-formation questions for 15 models, ranked by Overall and grouped in fives (Top 5, Middle 5, Bottom 5).
2a. Capable models already quote the computed values correctly, so this check cannot tell a good interpretation from a poor one. Petrophys computes these values from the logs, and the models receive them in their context. When a Top 5 model quotes a porosity value for an interval, it is the value computed for that interval 95% of the time. For water saturation it is 97%. The Middle 5 do just as well (93% and 97%), yet their Overall scores are more than 10 points lower. Only the Bottom 5 often attach a value to the wrong interval (65% and 73%). So the check catches models that misreport the data, but it gives the same pass to models whose interpretations differ widely in quality. Formation boundaries are inputs taken from the source datasets (§3.2). The three groups differ little on them, with 85–91% of answers correct, against 92% for the expert references.
2b. For questions like reservoir identification, the final call needs more than the cutoff flags. Petrophys computes reservoir and pay flags from the logs. Each flag is a cutoff test applied to every depth sample, and it reports whether the cutoff is passed. We compared the flags with the benchmark's reference answers, which petrophysicists reviewed and corrected. The experts' final call matches the flag only in 66% of the intervals they call . In the other 34%, the experts use further evidence and reach a different call.
The reason is that a log response can have several causes. A flag turns one response into a yes or a no. The expert has to work out what caused the response in that interval, and that takes evidence the flag does not hold:
- In the Sognefjord Fm. of well 35/11-7, the flags mark a neutron–density crossover, a pattern often read as gas. The reference explains that, without a shear sonic log, gas cannot be told apart from light oil, lithology, clay-bound water or borehole effects, and it stops short of a gas call.
- In the Hod Fm. well elin-1, thin intervals carry prime-oil flags. But the vertical trends in saturation, bulk volume water and resistivity show no contact, and the expert calls the chalk water-bearing.
Other indicators behave the same way. High resistivity can mean hydrocarbons, tight rock or fresh formation water. A computed water saturation is only as reliable as the saturation model behind it, and the right model changes with clay content and lithology. Which explanation applies depends on the other logs in the well, their quality, the lithology and the neighbouring intervals, and these factors interact. Each one can be coded. But then every combination needs its own rule and its own cutoffs. The result would be one more rule-based interpretation, and no longer a physical result that can be verified.
A metric that rewards agreement with the flags would give its best score to an answer that just repeats them. It would also mark the experts wrong in a third of their calls.
Author Response 1/6
We thank the reviewer for the detailed and constructive review, and we are glad that the reviewer values our benchmark built on real well-log data with expert review. We hope that our detailed response below addresses all the weaknesses and questions raised by the reviewer, including the questions on the judge, the training signals and DFS.
W1: Reliance on LLM-as-a-Judge
We agree that matching the reference's conclusion and reaching a reliable conclusion from the logs are two different things. The rubric scores them separately.
1. The judge scores 13 sub-metrics separately. Response Adequacy compares the answer with an expert-verified reference, and Reasoning Quality judges the argument on its own. Senior petrophysicists reviewed every reference. They re-checked the values, depths and interpretations against the raw logs in Petrel and Techlog, and corrected about 60% of the items (see §3.3, Appendices C and D). So the log evidence is built into the reference. The judge never gives a single match score. It scores each sub-metric on its own and explains each score in writing. A rubric group is the average of its sub-metrics (see §4.2) .
2. The sub-metrics separate the two questions in part. The reference records the interpretation that the logs of that well support. The judge does not ask an answer to follow the expert's route to it. An answer that reasons in its own way gets one score for what it concludes (Conclusion, Accuracy) and a separate score for whether its reasoning holds (Reasoning Quality, which does not depend on matching the reference). Sound analyses of the same logs usually arrive at the same interpretation, so the two scores usually agree. When they do not, the sub-metrics show it: an answer with the right conclusion but weak support scores well on Conclusion and loses on Faithfulness. The blind evaluation in Q1 was done without the reference and looks at the remaining cases. We release all sub-scores with the judge's explanations. Appendix K.4 uses them to tell numeric errors apart from wrong conclusions, missing caveats and misused logs.
3. The judge agrees with experts nearly as well as experts agree with each other (Appendix E). The judge and the experts agree at ICC(A,1) = 0.721, and the two experts agree with each other at 0.765. Where the judge agrees less with the experts, the experts also agree less with each other: 0.47–0.64 on Accuracy, Coverage and Petrophysical Claims, against 0.71–0.76 on Faithfulness and Numerical Accuracy.