- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces ELDOR, a new semantic segmentation dataset based on very high-resolution UAV imagery (<10cm/px) collected over the Madre de Dios region in Peru. It covers 2500ha alongside semantic masks for 14 classes related to gold mining activities and its environmental consequences. Initial benchmarking results are reported for semantic segmentation, multi-label classification and zero-shot classification with VLMs. A large number of experiments are conducted to assess the performance of state of the art models, both from classical computer vision and remote sensing literature. Data and code is publicly released.
Overall this is a quite good paper. It introduces a niche dataset but it is well-executed, with a comprehensive benchmarking of existing models. I have some minor comments and suggestions but mostly nitpicking on details.
Critical strengths
-
Novel UAV dataset for semantic segmentation of mining activities. To the best of my knowledge, it is the first dataset at this resolution to also include environmental effects in its annotation process, including different types of vegetation around the mining sites. This is important as it could allow not only to map out the mining activities, but also the environmental recovery after the mining stopped.
-
Large number of experimental results with 8 semantic segmentation models, 11 classifiers and 6 VLMs. This is helpful to get a complete picture of the state of the art on this dataset, across the different subtasks (segmentation/classification/zero-shot transfer).
-
Strong data collection and annotation protocol, which well-documented choices and limitations.
-
Code and data is publicly released, which is a significant strength: this will allow the proposed the dataset to be reused by the computer vision/remote sensing communities as a domain-specific benchmark.
Critical weaknesses
While not critical, the paper has a main weakness: its experimental protocol. There are two points that I would like to raise.
A. Test set
Four classes are missing from the test set. While I understand how this might be the case, I urge the authors to reconsider their train/val/test split so that all classes are represented in all sets. At least, all classes should be present in the test set, otherwise there is no point of annotating the missing classes: the benchmark will never evaluate performance of models on them anyway. In practice this slightly reduces the value of the benchmark, since the 4 sparsely represented classes can be ignored and models will still perform strongly on the test set anyway.
B. Task choices
I am somewhat skeptical of the some of the task choices. Tasks 1 and 3 are very clear and commonly accepted use cases. Task 2 seems contrived to me: it is basically the same as Task 3 (classification) but derived from the semantic segmentation mask. I don't really see a strong interest in the literature for this kind of approach and the paper has not really convinced me of its value either.
Task 4 is mostly fine but why exclude in Task 4 any few-shot, linear probing or fine-tuning approach? These are quite common approaches used in remote sensing.
Questions for authors
Questions
- Since SfM is already use of co-registration, is there any depth/height information that will be released to provide 3D cues?
- Are the disagreements from the multiple annotators kept and stored? Previous works have shown that models trained on human uncertainty are more robust.
- Is the 50% overlap of the sliding window also applied to the val and test sets? If so, this is likely to bias the evaluation metrics towards the redundant pixels.
- Are the differences in mIoU/mAP hiding variations at the class-level? Especially for the configurations using a weighted class-entropy, which might change a lot model performance for the rare classes.
- I appreciate that many classifiers/segmenters have been tested, but it would be more useful to explain why these specific baselines have been considered and on what criterion they have been included.
Suggestions/remarks
- The related work is missing some previous works on semantic segmentation of mining areas based on UAV data, such as Mine 4.0-MineCareerDB and OPMS-Seg. These are not great datasets and not entirely comparable in their purpose and scope, but it is slightly misleading to claim that there was no UAV data with similar annotations or orthomosaics.
- It could be helpful to define some of the mining-specific terms (e.g. sluices, rafts, tailings).
- Most tables and figures are too small to read in print and some require an excessive amount of zoom to read on screen.
- Figure 1: caption states it illustrates "subtle land-cover transitions" but this might be too subtle for me, as I don't know what transitions I'm supposed to see in these acquisitions.
- Figure 2: top right histogram would be easier sorted and using class names in the -axis.
- Table 3: CF1/OF1 are not defined.
- Paragraphs that list the baselines (l.303-314; l.390-395; etc.) are not only boring to read, but also mostly useless.
- I am unconvinced that section 5.4 and Figure 6 bring a lot of scientific value to the paper. I suggest moving it into the appendix to free up some space to make the figures larger.
2 comments
We sincerely appreciate your positive assessment of our work and your constructive suggestions. We have carefully considered your feedback and revised the manuscript to improve its clarity, completeness, and experimental rigor. We hope these revisions have satisfactorily addressed your concerns, and we would be happy to provide any further clarification if needed.
Response to Reviewer xhSu
We thank the reviewer for recognizing ELDOR's contributions and for the constructive feedback.
[W-A: Missing classes in the test split]
The four absent classes occur in only 1–3 original orthomosaics, limiting independent test coverage. All 14 classes remain in training, and 13 in validation (including three absent from testing), preserving supervision for future independent-site evaluation. The full taxonomy also supports future evaluation beyond MDD. We clarify the limitation (Section 3; Appendix A.2), and continue field collaborations to expand rare-class and geographic coverage.
[W-B1: Value of Segmentation-derived Recognition]
Task 2 tests whether segmentation models used for spatial mapping can also perform patch screening without a separate classifier. PyramidMamba achieves 0.6846 mAP, close to TDRG's 0.6983 in Task 3. Task 3 better ranks rare mining structures under limited review budgets, while Task 2 retains dense maps for ecological monitoring. We have clarified these complementary roles in Sections 4–5 and Appendix C.2.
[W-B2: Task 4 adaptation]
We added five-seed frozen linear probes across seven annotation budgets (1–100 shots/class and full) and head-only, LayerNorm, and LoRA8 comparisons for all eight models. Eight-model mean test scores (mAP / Macro-F1, %):
Probes run for 100 epochs; PEFT variants follow matched schedules. Visual PEFT improves mAP in 29/32 and Macro-F1 in 32/32 comparisons, especially for vegetation and regeneration, while rare mining structures remain challenging. We have added these results to Section 5.3, Table 5, and Appendix C.5.3.
[Q1 & Q2: 3D data and annotation uncertainty]
The release contains RGB orthomosaics and semantic masks. SfM/MVS geometry is under assessment for completeness, spatial consistency, and suitability for public release with field collaborators (Appendix F). At least three experts checked all annotations and resolved disagreements by consensus using imagery and field observations (Section 3). Individual annotations were not retained; future extensions may preserve them for uncertainty-aware learning (Appendix F).
[Q3: Overlapping validation and test patches]
Validation and test use 512×512 windows at stride 256 (50% overlap). For 21 configurations, we tested inverse-coverage weighting (IPW, equal total weight per source pixel), center selection (one prediction from the most central window), and four non-overlapping layouts. Mean absolute mIoUₚ changes (0–100% scale):
Spatial bootstrap supports these results; PyramidMamba (WCE+Dice) leads in all seven settings on both splits (Appendix B.1.1).
[Q4: Class-wise effects of weighted losses]
WCE+Dice improves mining-raft IoU in 17/21 and Task 2 AP in 18/21 comparisons against CE+Dice, but can reduce IoU for Type 2 regeneration, water, or bare ground. Focal+Dice improves Type 1 regeneration AP in 17/21; for MANet, it gives higher mIoUₚ and Type 2 IoU, while WCE+Dice favors rafts. These effects are summarized in Section 5.1 and detailed in Appendix C.1–C.2.
[Q5 & S7: Baseline Selection and Presentation]
Sections 5.1–5.3 explain baseline selection: general/remote-sensing segmenters test spatial and domain modeling; pretrained/refinement methods assess transfer; multi-label models examine label relations and imbalance; VLMs compare contrastive, generative, and text-guided prediction. Full model lists and protocols are in Appendices C.1, C.4, and B.4.
[S1: Related UAV Mining Datasets]
We added Mine 4.0–MineCareerDB and OPMS-Seg to Table 1 and Related Work, comparing modality, resolution, and labels. The Introduction clarifies ELDOR's contribution as a 2,500-ha UAV benchmark jointly covering mining structures, disturbance, and ecological recovery.
[S2 & S4: Terminology and Figure 1]
The Introduction now links mining rafts, sluices, and mounds to illustrated definitions (Appendix A.1, Table 6). Figure 1 contrasts Sentinel-2/Planet land-cover patterns with UAV detail of small mining structures and vegetation. Field observations, including vegetation height, help distinguish primary forest and regeneration stages. We have clarified this comparison in the caption and Introduction.
[S3, S5, S6 & S8: Presentation and explorer]
We refined Tables 1, 3–5 and Figures 2–6. Figure 2 sorts classes by pixel count (names in Table 2); CF1/OF1 are clarified in Section 4 and Appendix B (Table 8). Section 5.4 gives model guidance, Section 5.2 discusses Grad-CAM (Figure 6), and Section 5.5 condenses the explorer (figure in Appendix D).
We welcome any further questions and would be happy to address them.