- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper is in dataset track, it introduces a multimodal remote sensing benchmark dataset, TreeUQ, covering the state of Bavaria, Germany, designed for forest structure estimation and uncertainty quantification.
The key contributions of the paper include:
-
Dataset Construction: The dataset integrates multi-seasonal Sentinel-1 and Sentinel-2 imagery, 20 cm-resolution aerial RGB imagery, tree species classification maps, and six forest structure labels derived from individual tree inventory data. Comprising 45,419 image patches, it supports the prediction of tree height and tree density at a 10 m resolution.
-
Benchmark Tasks and Evaluation Protocol: The paper establishes regression tasks for mean tree height and tree density. It employs a geographic-based partitioning strategy for training, validation, and testing to evaluate the spatial generalization capabilities of models, while also incorporating evaluation metrics for predictive uncertainty.
-
Comparison experiment: The study compares the predictive performance of various models—including XGBoost, U-Net, SegFormer, Prithvi 2.0, and DOFA—alongside four uncertainty estimation methods: Deep Ensemble, MC Dropout, Evidential Regression, and Conformalized Quantile Regression. Experimental results indicate that Prithvi 2.0 achieves the best performance for tree height prediction (RMSE = 5.37 m), while XGBoost demonstrates strong competitiveness in tree density prediction. Integrating features from high-resolution aerial imagery further enhances prediction accuracy. Regarding uncertainty quantification, the various methods exhibit distinct trade-offs between prediction interval coverage, interval width, and calibration error; furthermore, spatial distribution shifts are shown to affect the effectiveness of uncertainty calibration.
Critical strengths
I think the strength is the construction of a large-scale, multi-modal benchmark for forest structure.
This paper presents a forest structure dataset covering the state of Bavaria, Germany, comprising 45,419 image patches. The dataset integrates data from Sentinel-1, Sentinel-2, 20 cm resolution aerial RGB imagery, tree species classifications, and individual tree inventory records.
Unlike existing datasets—which primarily focus on canopy height or biomass estimation—TreeUQ incorporates metrics such as tree count, density, and related statistics, providing a unified foundation for research on estimating forest structure using medium-resolution satellite imagery. Given the inherent challenges in indirectly estimating tree density from 10 m resolution satellite data, this dataset is well-positioned to support future research in multi-modal learning and uncertainty quantification.
Critical weaknesses
They are two weaknesses after reveiw:
- Lack of independent validation for the supervised labels.
While the primary contribution of this paper is the construction of a large-scale benchmark for forest structure, the accuracy of its supervised labels has not been sufficiently validated. The individual-tree data for Bavaria—used by the authors—was automatically extracted from remote sensing data rather than derived from independent field-based forest inventory observations. Official documentation for the dataset explicitly notes that the automated extraction process may be prone to errors.
The paper lacks quantitative comparisons against independent data sources, such as airborne LiDAR (ALS), terrestrial laser scanning (TLS), or field-plot observations. Even site-level validation is absent; furthermore, there is no systematic analysis of how missed detections, false positives, or errors in tree-height estimation might affect the supervised labels. In particular, since tree density is directly derived from individual-tree detection results, any underlying detection bias could systematically skew both model training and evaluation.
Given that the dataset itself is the core contribution of this work, the absence of independent validation makes it difficult to determine to what extent the benchmark’s predictive accuracy reflects actual forest structure, as opposed to merely the model’s ability to fit the existing, automatically extracted product.
- Limitations in the definition and evaluation of the tree-density task.
The paper defines "tree density" as the average number of trees within a 3 × 3 neighborhood, rather than predicting the number of trees for an individual pixel directly. Moreover, the model is trained and evaluated exclusively on pixels where the tree density is greater than zero, effectively excluding areas with zero density. Consequently, the current experiments fail to adequately assess the model’s capacity to distinguish between areas with and without trees, nor do they demonstrate its reliability for mapping tree density across a complete spatial extent.
Additionally, there is an inconsistency regarding the definition of tree density: the paper defines it as a neighborhood average, whereas the documentation for the public dataset describes it as the sum of the number of trees within a neighborhood. This discrepancy requires clarification, as it directly impacts the definition of the prediction target, the numerical scale, and the interpretation of evaluation metrics.
Given that tree-density estimation is a key contribution—distinguishing this work from existing canopy-height benchmarks—these issues undermine the persuasiveness of the experimental results and their practical utility.
Questions for authors
I have two questions:
- Reliability of Ground-Truth Labels
The paper utilizes the "Einzelbäume" dataset from Bavaria as ground-truth labels; however, this dataset was automatically extracted from DOM20, DOP20, and DGM1 sources rather than being derived from independent, on-the-ground field measurements. Have the authors assessed the accuracy of these automatically extracted individual tree locations, counts, and heights? Is there any quantitative evidence—such as comparisons against independent ALS/TLS data or forest plot observations—to validate these values?
- Validity of Uncertainty Quantification (UQ) Results
The paper indicates that the UQ method—calibrated using a validation set—fails to achieve the expected 90% coverage rate across geographically independent test areas. The authors attribute this to a spatial distribution shift; is there any quantitative analysis to support this explanation—for instance, by examining coverage rates across different geographic blocks or varying forest structural conditions?