acceptodds
Under review as a conference paper at ICLR 2027

ELDOR: A Dataset and Benchmark for Illegal Gold Mining in the Amazon Rainforest

Abstract

Illegal gold mining in the Amazon rainforest causes deforestation, water contamination, and long-term ecosystem disruption, yet remains difficult to monitor at fine spatial scales. Satellite imagery supports large-scale observation, but often misses small mining-related structures and subtle land-cover transitions, especially under frequent cloud cover. We introduce ELDOR, a large-scale UAV benchmark for monitoring environmental and landscape disturbance associated withfrom illegal gold mining. ELDOR contains manually annotated orthomosaic imagery covering over 2,500 hectares, with pixel-level semantic labels for both mining-related activities and surrounding ecological structures. With this unified annotation source, we establish four benchmark tasks: semantic segmentation, segmentation-derived recognition, direct multi-label classification, and class-presence recognition with vision-language models. Across these tasks, we compare generic and remote-sensing-specific segmentation models, vision foundation model-related segmentation methods, direct multi-label classification methods, and vision-language models under a controlled closed-set protocol. Results show that current methods still struggle with rare small-scale mining structures and fine-grained recovery classes, suggesting the need for context-aware and multimodal modeling. To support domain analysis and practical use, we further build an interactive explorer for domain experts that provides a unified interface for data exploration and model inference. Code and data will be released upon acceptance.

Primary area:
datasets and benchmarks
Keywords:
Illegal Mining, UAV remote sensing, Segmentation, Multi-label Classification, Environmental Monitoring

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_xhSuOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces ELDOR, a new semantic segmentation dataset based on very high-resolution UAV imagery (<10cm/px) collected over the Madre de Dios region in Peru. It covers 2500ha alongside semantic masks for 14 classes related to gold mining activities and its environmental consequences. Initial benchmarking results are reported for semantic segmentation, multi-label classification and zero-shot classification with VLMs. A large number of experiments are conducted to assess the performance of state of the art models, both from classical computer vision and remote sensing literature. Data and code is publicly released.

Overall this is a quite good paper. It introduces a niche dataset but it is well-executed, with a comprehensive benchmarking of existing models. I have some minor comments and suggestions but mostly nitpicking on details.

Critical strengths
  • Novel UAV dataset for semantic segmentation of mining activities. To the best of my knowledge, it is the first dataset at this resolution to also include environmental effects in its annotation process, including different types of vegetation around the mining sites. This is important as it could allow not only to map out the mining activities, but also the environmental recovery after the mining stopped.

  • Large number of experimental results with 8 semantic segmentation models, 11 classifiers and 6 VLMs. This is helpful to get a complete picture of the state of the art on this dataset, across the different subtasks (segmentation/classification/zero-shot transfer).

  • Strong data collection and annotation protocol, which well-documented choices and limitations.

  • Code and data is publicly released, which is a significant strength: this will allow the proposed the dataset to be reused by the computer vision/remote sensing communities as a domain-specific benchmark.

Critical weaknesses

While not critical, the paper has a main weakness: its experimental protocol. There are two points that I would like to raise.

A. Test set

Four classes are missing from the test set. While I understand how this might be the case, I urge the authors to reconsider their train/val/test split so that all classes are represented in all sets. At least, all classes should be present in the test set, otherwise there is no point of annotating the missing classes: the benchmark will never evaluate performance of models on them anyway. In practice this slightly reduces the value of the benchmark, since the 4 sparsely represented classes can be ignored and models will still perform strongly on the test set anyway.

B. Task choices

I am somewhat skeptical of the some of the task choices. Tasks 1 and 3 are very clear and commonly accepted use cases. Task 2 seems contrived to me: it is basically the same as Task 3 (classification) but derived from the semantic segmentation mask. I don't really see a strong interest in the literature for this kind of approach and the paper has not really convinced me of its value either.

Task 4 is mostly fine but why exclude in Task 4 any few-shot, linear probing or fine-tuning approach? These are quite common approaches used in remote sensing.

Questions for authors

Questions

  1. Since SfM is already use of co-registration, is there any depth/height information that will be released to provide 3D cues?
  2. Are the disagreements from the multiple annotators kept and stored? Previous works have shown that models trained on human uncertainty are more robust.
  3. Is the 50% overlap of the sliding window also applied to the val and test sets? If so, this is likely to bias the evaluation metrics towards the redundant pixels.
  4. Are the differences in mIoU/mAP hiding variations at the class-level? Especially for the configurations using a weighted class-entropy, which might change a lot model performance for the rare classes.
  5. I appreciate that many classifiers/segmenters have been tested, but it would be more useful to explain why these specific baselines have been considered and on what criterion they have been included.

Suggestions/remarks

  • The related work is missing some previous works on semantic segmentation of mining areas based on UAV data, such as Mine 4.0-MineCareerDB and OPMS-Seg. These are not great datasets and not entirely comparable in their purpose and scope, but it is slightly misleading to claim that there was no UAV data with similar annotations or orthomosaics.
  • It could be helpful to define some of the mining-specific terms (e.g. sluices, rafts, tailings).
  • Most tables and figures are too small to read in print and some require an excessive amount of zoom to read on screen.
  • Figure 1: caption states it illustrates "subtle land-cover transitions" but this might be too subtle for me, as I don't know what transitions I'm supposed to see in these acquisitions.
  • Figure 2: top right histogram would be easier sorted and using class names in the xx-axis.
  • Table 3: CF1/OF1 are not defined.
  • Paragraphs that list the baselines (l.303-314; l.390-395; etc.) are not only boring to read, but also mostly useless.
  • I am unconvinced that section 5.4 and Figure 6 bring a lot of scientific value to the paper. I suggest moving it into the appendix to free up some space to make the figures larger.
2 comments
AuthorsOpenReview ↗

Response to Reviewer xhSu

We thank the reviewer for recognizing ELDOR's contributions and for the constructive feedback.

[W-A: Missing classes in the test split]

The four absent classes occur in only 1–3 original orthomosaics, limiting independent test coverage. All 14 classes remain in training, and 13 in validation (including three absent from testing), preserving supervision for future independent-site evaluation. The full taxonomy also supports future evaluation beyond MDD. We clarify the limitation (Section 3; Appendix A.2), and continue field collaborations to expand rare-class and geographic coverage.

[W-B1: Value of Segmentation-derived Recognition]

Task 2 tests whether segmentation models used for spatial mapping can also perform patch screening without a separate classifier. PyramidMamba achieves 0.6846 mAP, close to TDRG's 0.6983 in Task 3. Task 3 better ranks rare mining structures under limited review budgets, while Task 2 retains dense maps for ecological monitoring. We have clarified these complementary roles in Sections 4–5 and Appendix C.2.

[W-B2: Task 4 adaptation]

We added five-seed frozen linear probes across seven annotation budgets (1–100 shots/class and full) and head-only, LayerNorm, and LoRA8 comparisons for all eight models. Eight-model mean test scores (mAP / Macro-F1, %):

BudgetLinear probeHead-onlyLoRA8
10-shot54.63 / 44.5646.36 / 14.6651.74 / 31.97
Full65.94 / 51.2457.48 / 37.4266.84 / 53.29

Probes run for 100 epochs; PEFT variants follow matched schedules. Visual PEFT improves mAP in 29/32 and Macro-F1 in 32/32 comparisons, especially for vegetation and regeneration, while rare mining structures remain challenging. We have added these results to Section 5.3, Table 5, and Appendix C.5.3.

[Q1 & Q2: 3D data and annotation uncertainty]

The release contains RGB orthomosaics and semantic masks. SfM/MVS geometry is under assessment for completeness, spatial consistency, and suitability for public release with field collaborators (Appendix F). At least three experts checked all annotations and resolved disagreements by consensus using imagery and field observations (Section 3). Individual annotations were not retained; future extensions may preserve them for uncertainty-aware learning (Appendix F).

[Q3: Overlapping validation and test patches]

Validation and test use 512×512 windows at stride 256 (50% overlap). For 21 configurations, we tested inverse-coverage weighting (IPW, equal total weight per source pixel), center selection (one prediction from the most central window), and four non-overlapping layouts. Mean absolute mIoUₚ changes (0–100% scale):

SplitIPWCenterNo overlap
Test0.050%0.831%0.183%
Validation0.111%0.413%0.342%

Spatial bootstrap supports these results; PyramidMamba (WCE+Dice) leads in all seven settings on both splits (Appendix B.1.1).

[Q4: Class-wise effects of weighted losses]

WCE+Dice improves mining-raft IoU in 17/21 and Task 2 AP in 18/21 comparisons against CE+Dice, but can reduce IoU for Type 2 regeneration, water, or bare ground. Focal+Dice improves Type 1 regeneration AP in 17/21; for MANet, it gives higher mIoUₚ and Type 2 IoU, while WCE+Dice favors rafts. These effects are summarized in Section 5.1 and detailed in Appendix C.1–C.2.

[Q5 & S7: Baseline Selection and Presentation]

Sections 5.1–5.3 explain baseline selection: general/remote-sensing segmenters test spatial and domain modeling; pretrained/refinement methods assess transfer; multi-label models examine label relations and imbalance; VLMs compare contrastive, generative, and text-guided prediction. Full model lists and protocols are in Appendices C.1, C.4, and B.4.

[S1: Related UAV Mining Datasets]

We added Mine 4.0–MineCareerDB and OPMS-Seg to Table 1 and Related Work, comparing modality, resolution, and labels. The Introduction clarifies ELDOR's contribution as a 2,500-ha UAV benchmark jointly covering mining structures, disturbance, and ecological recovery.

[S2 & S4: Terminology and Figure 1]

The Introduction now links mining rafts, sluices, and mounds to illustrated definitions (Appendix A.1, Table 6). Figure 1 contrasts Sentinel-2/Planet land-cover patterns with UAV detail of small mining structures and vegetation. Field observations, including vegetation height, help distinguish primary forest and regeneration stages. We have clarified this comparison in the caption and Introduction.

[S3, S5, S6 & S8: Presentation and explorer]

We refined Tables 1, 3–5 and Figures 2–6. Figure 2 sorts classes by pixel count (names in Table 2); CF1/OF1 are clarified in Section 4 and Appendix B (Table 8). Section 5.4 gives model guidance, Section 5.2 discusses Grad-CAM (Figure 6), and Section 5.5 condenses the explorer (figure in Appendix D).

We welcome any further questions and would be happy to address them.

We sincerely appreciate your positive assessment of our work and your constructive suggestions. We have carefully considered your feedback and revised the manuscript to improve its clarity, completeness, and experimental rigor. We hope these revisions have satisfactorily addressed your concerns, and we would be happy to provide any further clarification if needed.

Reviewer_zyu9OpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors propose a dataset of high-resolution UAV imagery (3-8 cm/pix resolution), covering 2500 ha in the Amazon forest across different sites, with pixel-level labels for disturbances related to gold mining and land cover classes. They benchmark a number of models on 4 different tasks (semantic segmentation, segmentation-derived recognition (using the output of semantic segmentation but evaluating on a multilabel classification task), multi-label classification, and segmentation or recognition using VLMs). The dataset and tasks present several challenges in terms of long-tailed distribution of classes, and some visually similar categories. The authors find that performance is not entirely correlated with number of examples in the dataset, with "type 2 regeneration" being confused a lot with "primary forests" class despite it representing 21% of the dataset class proportion. This benchmark highlights challenges related to identifying small structures related to illegal gold mining in UAV imagery.

Critical strengths

The dataset fills a gap in terms of available remote sensing imagery that is at high-resolution enough for the application of monitoring illegal gold mining, and labelled manually at the pixel-level. Collecting such a dataset is well motivated with respect to the application. The paper is generally easy to follow (I understand a lot of details had to be pushed to the appendix due to lack of space) and I appreciated the visual examples both in the main text and appendix. The benchmark is comprehensive with many different models being compared and different tasks being proposed.

Critical weaknesses

While the experiments are comprehensive, there is a lack of deeper analysis of the results and presentation of what went behind the selection (or more so, why try such a range) of models to be benchmarked, or selection of study sites. After reading the paper, it is very unclear to the reader which task setup/model is best suited for the application, and it would strengthen the paper A LOT if the authors could concisely present which setup is best for which prioritisation in mining activities monitoring in practice. The paper reads as developing methods removed from end users. The metrics also do not particularly emphasize correctly detecting the structures that are related to mining, possibly hindering impact in practice.

Questions for authors
  • Can the authors outline what their take is on which task formulation and model is best suited to identify illegal mining activity.
  • The goal is to identify "illegal" goal mining, can the authors elaborate on how one would differentiate legal vs illegal mining in an image? Or is it that all mining equipment indicate illegal activity based on the region that is surveyed.
  • If small structures like vehicles are more likely to point to illegal mining, how do the authors justify leaving some of the rare but potentially very indicative classes out of the test set. The performance is also (unsurprisingly) low for identifying small and rare structures but they are the ones that one probably cares about more than having good identification of e.g. primary forest if the goal is to monitor illegal mining activities. Relatedly to splitting data, I understand splitting by site but some critical classes are left out of the test set. I can think of this remote sensing dataset preparation with a fined-grained instance segmentation task, in which there was concern about having both site splitting and maintaining representation of all classes in the test set: https://proceedings.neurips.cc/paper_files/paper/2025/hash/0cb66bdf0e43571eea2c3527d13fd3fd-Abstract-Conference.html
  • How were the study sites selected?
  • if the goal is to identify mining structures, I fear many of the metrics chosen do not actually reflect what is important to prioritize in practice. It seems there are multiple goals: identifying mining structures, monitoring regeneration that require looking at things at different scales.
  • Can the authors provide more critical analysis
  • I am curious to have more discussion on the comparison of the segmentation derived recognition vs multilabel classification setups and recommendations on why one should focus on one task vs another and whether the authors thought of thresholding the class presence in the segmentation-derived recognition task with number of segmented pixels in a class and confidence?
  • How do authors envision users using this benchmark? What are concrete recommendations for practitioners?

A lot of these suggestions relate to the paper presenting a benchmark with many tasks and models and in this sea of information, it being difficult to parse what someone should care about in practice and why certain critical decisions (like the splitting with some classes being excluded in the test set) were made. I also find it is a bit of shame that some analysis is just pushed entirely to the appendix (e.g. the gradCAM analysis). And when one sees these analyses, they are long descriptions of numbers of figures without a sense of what the reader should care about for the application at hand. I am a human reading and trying to understand what the main information is that I should get out of this paper, and there is just not enough critical synthesizing and recommendations for which task I should care about and which models I should prioritize using.

6 comments
AuthorsOpenReview ↗

Response to Reviewer zyu9

We thank the reviewer for the suggestions and have strengthened ELDOR's practical interpretation.

[W: Benchmark Design and Practical Relevance]

Sections 5.1–5.3 now explain generic, remote-sensing, vision foundation model, and multi-label baselines. Section 5.4 synthesizes class-wise errors and practical trade-offs; Section 3 and Appendix A.2 clarify site selection and splitting.

[Q1: Task and model selection]

For screening, Task 3 better ranks rare structures under limited review budgets (TDRG: 0.6983 mAP); class-specific recall guides model selection. For mapping, Task 1 with PyramidMamba (WCE+Dice) reaches 0.4414 mIoUₚ, with UNetFormer as a more efficient option. Task 2 can reuse segmentation predictions for screening, and Task 4 evaluates zero-shot transfer and visual adaptation (Section 5.4, Appendix C.2/C.6).

[Q2: Meaning of illegal mining]

ELDOR focuses on mining-affected sites in Madre de Dios, a documented hotspot of illegal gold mining. The benchmark identifies visible mining-related structures and environmental disturbances, but these observations alone cannot establish the legal status of individual operations. We have clarified the regional context in Section 3 and the need for expert interpretation and contextual verification in Appendices E–F.

[Q3 & Q4: Rare Classes, Site Separation, and Selection]

Working with local ecologists, we selected sites in Madre de Dios to cover diverse mining activities, disturbances, and vegetation conditions, subject to field accessibility and UAV flight permissions coordinated with local partners and authorities. Vehicles, heavy machinery, compact mounds, and grass occur in only 1–3 original orthomosaics (grass in one); we therefore prioritized source-independent testing over full test-class coverage. The suggested spatial partitioning could improve rare-class representation, but splitting source imagery between training and testing would weaken this independence. Retaining all 14 classes in training supports future evaluation on new sites and mining contexts. We clarified these selection criteria and trade-offs in Section 3 and Appendix A.2 and continue field collaborations to improve coverage.

[Q5: Metrics for mining structures and ecological monitoring]

We evaluate screening with class-wise precision/recall, AP, and Recall@budget to measure false alarms, missed detections, ranking quality, and recovery under limited expert inspection (Appendix C.2). For ecological mapping, per-class IoU and confusion matrices quantify errors among disturbed ground, forests, and regeneration (Figures 4–5; Appendix C.3). These application-specific metrics guide model selection (Sections 5.1 and 5.4).

[Q6: More critical analysis]

We now synthesize three findings in Sections 5.1–5.3. Type 2 regeneration covers 20.91% of annotated area but is often confused with primary forest, complicating recovery mapping. WCE+Dice improves mining-raft IoU and Task 2 AP in 17/21 and 18/21 matched comparisons but may degrade other land-cover classes. C-Tran reaches 69.47% mAP but zero F1 for rafts and sluices at the default threshold, separating ranking from detection. We also discuss Grad-CAM in the main text (Figure 6): even TDRG shows diffuse, context-driven activations on some mining targets. Ranking scores can therefore guide screening without guaranteeing localization.

[Q7: Task 2 versus Task 3 and thresholds]

We evaluated validation-selected presence thresholds: Task 2 searches 22 confidence levels and 14 pixel-count settings (308 combinations), while Task 3 searches 99 shared thresholds.

TaskMacro-F1: Default → SelectedImprovedRecall@5%: Rafts / Sluices
252.86% → 54.55%77/968.9% / 5.0%
355.57% → 57.55%15/1778.4% / 79.0%

Recall@5% measures positive patches recovered in the top-ranked 5% per class using the original scores. We report the median across configurations of the lowest recall over four non-overlapping test layouts (Appendix C.2). Threshold selection improves Macro-F1 in most configurations, but rare-class recall needs separate assessment.

[Q8: Recommendations for practitioners]

We recommend class-specific Task 3 rankings for expert inspection: at a 5% per-class review budget, GRN recovers 91.7% of mining-raft-positive and 84.2% of sluice-positive patches in the worst of four non-overlapping test layouts. For mapping, use PyramidMamba (Task 1; 44.14% mIoUₚ) or UNetFormer when efficiency matters (23.55 vs. 217.75 GFLOPs). The interactive explorer lets ecologists inspect orthomosaics, run inference on selected regions, and compare predictions with expert annotations and geographic context. We have clarified these recommendations in Section 5.4 and Appendix C.2/C.6.

We welcome any further questions and would be happy to address them.

We sincerely appreciate your thoughtful feedback. Following your suggestions, we have substantially strengthened the manuscript's analysis and interpretation, with greater emphasis on the practical relevance and main insights of the benchmark. We would be grateful to know whether these revisions have adequately addressed your main concerns, and we would be happy to further improve the paper based on any additional feedback.

I thank the authors for their responses and have a couple additional questions:

reQ1: Thank you for adding details in 5.4. The paper is trying to achieve a lot, looking at many different problems. As I highlighted in my review, there are multiple goals, including identifying mining structures and monitoring regeneration. It might be helpful to add some recommendations related to best practices and models to choose for monitoring regeneration, since it is also a focus of the paper. reQ7: thank you for these additional insights. Regarding the choice of recall@5% metric for rafts and sluices: Noting that these represent <0.01% in the dataset (at least pixel wise according to Table 2, though I recognize it might be more informative here to have the representation in terms of % of tiles in which these structures are present ), is that a really fair and realistic metric to consider? That being said, it is useful insight to have that Task3 is better suited than task 2 to identify small and rare structures.

Reviewer_miVeOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ELDOR, a UAV imagery benchmark of landscape disturbance caused by illegal gold mining in the Amazon. The authors consider 4 tasks from the labels: semantic segmentation, segmentation-derived recognition, direct multi-label classification, and VLM-based recognition. Extensive experiments comparing different architectures in these settings are provided.

Critical strengths
  1. Relevant dataset for an important problem, fine-grained expert manual annotations.
  2. Significance benchmarking efforts and model comparison.
Critical weaknesses
  1. Some classes are not present in the test set.
  2. The paper mentions site-lvl geographical splits. However, PlayaMirador1 and PlayaMirador2 seem to be two halves of the same scene and occur in different splits.
  3. No vision GeoFMs are evaluated.
  4. Some references do not seem very aligned with the preceding text, e.g Polk et al., 2023; Cui et al., 2024 in "Paradigms for Aerial Scene Understanding".
  5. The recognition Macro-F1 values seems to exclude zero-F1 classes when averaging, judging by the aggregate and per-class results in Tables 16 and 19.
  6. Single seed runs, no error bars in the experiments.
Questions for authors
  1. Could the authors include some vision GeoFMs to the evaluation? E.g., TerraMind, DINOv3, with a simple segmentation decoder trained on ELDOR.
5 comments
AuthorsOpenReview ↗

Response to Reviewer miVe

We thank the reviewer for recognizing ELDOR's environmental relevance, expert annotations, and comprehensive benchmarking. We have clarified the dataset design, corrected the recognition evaluation, and added new foundation-model baselines.

[W1: Missing test classes]

The four missing classes occur in only 1–3 original orthomosaics. Following discussions with collaborating ecologists, we prioritized geographic separation over class balance to assess generalization to unseen areas. Retaining all 14 classes for training, with 13 represented in validation, preserves rare-class supervision and enables future evaluation on new sites. We acknowledge that the current test set cannot assess positive-class performance for these four classes and have clarified this limitation in Section 3 and Appendix A. We are continuing field collaborations to improve rare-class coverage in future releases.

[W2: PlayaMirador split]

PlayaMirador1 and PlayaMirador2 are non-overlapping regions of the same orthomosaic, assigned to training and validation because grass appears only there. We acknowledge that this limits validation independence. However, neither region enters testing, and all five test sites use separate orthomosaics. Although the rectangular coordinate ranges in Table 7 overlap, the actual image areas are irregular and do not overlap. We have clarified this distinction and the relationship between 12 listed sites and 11 original orthomosaics in Section 3 and Appendix A.2.

[W3 & Q1: Vision GeoFM Evaluation]

Following the suggestion, we added TerraMind-1.0-base (RGB) and DINOv3 ViT-L/16 (SAT-493M). We freeze their pretrained encoders and train 128/64-channel convolutional decoders, evaluating three losses under the same ELDOR splits. Test results are:

ModelLossmIoUₚSeg. Macro-F1Task 2 mAP
TerraMindCE+Dice0.27470.38350.4479
TerraMindFocal+Dice0.28270.39670.4553
TerraMindWCE+Dice0.31740.45850.5268
DINOv3CE+Dice0.36840.49440.5221
DINOv3Focal+Dice0.36690.49410.5552
DINOv3WCE+Dice0.40580.55430.6381

DINOv3 achieves the highest segmentation mIoUₚ and Macro-F1 among the evaluated foundation-model-related methods. WCE+Dice is strongest for both GeoFMs on all three reported metrics. Results are included in Table 3, Figures 3–5, and Appendix C.1–C.2.

[W4: Citation alignment]

We have updated the citations in Section 2 to better match the discussion of aerial semantic segmentation.

[W5: Recognition Macro-F1 calculation]

Thank you for catching this issue! We corrected recognition Macro-F1 by averaging over all ground-truth-supported classes, including those with zero F1 (10 classes on test), and recomputed the affected results, rankings, and discussions (Tables 4–5 and Appendix C). The metric definition has also been clarified in Appendix B (Table 8).

[W6: Single-seed evaluation and stability]

Full multi-seed retraining of the segmentation benchmark is computationally demanding within the rebuttal period. We instead performed 1,000 spatial-block bootstrap resamples of existing test predictions using 2048×2048 source-image blocks. Representative Task 1 scores and 95% percentile intervals are below (percent):

Model (WCE+Dice)mIoUₚ [95% CI]Macro-F1 [95% CI]
SegFormer40.10 [37.90, 41.92]52.97 [50.54, 54.72]
PyramidMamba44.14 [41.19, 46.56]58.64 [55.07, 61.21]
DINOv340.58 [38.42, 42.89]55.43 [52.82, 57.93]

Spatial bootstrap was also applied to eight Task 3 classifiers. For example, TDRG obtains 69.83% mAP [67.50%, 72.04%] and 61.61% Macro-F1 [59.43%, 63.21%]. These intervals quantify variability under spatial resampling of fixed-model predictions.

As complementary evidence, we conducted two additional training runs (seeds 52 and 62) for six Task 3 classifiers. The supplementary mAP results show sample standard deviations ranging from 0.99 to 3.28 percentage points, with five of the six models below 1.5 points. We report these runs separately from the original benchmark results, with complete mAP and F1 metrics provided in Appendix C. For Task 4, we also conducted five-seed frozen linear probing across eight models and seven annotation budgets. For example, VHM achieves 71.59 ± 0.04% mAP under full supervision. These experiments complement the spatial-bootstrap analysis with additional training-run variability estimates.

These revisions make ELDOR's design, evaluation, and main findings clearer. We welcome any further questions and would be happy to address them.

We sincerely appreciate your constructive and detailed feedback. We have carefully considered your concerns and made substantial revisions to strengthen the manuscript. We would be grateful to hear whether these changes have adequately addressed your concerns and, if so, whether you might consider revisiting your initial rating. We would be more than happy to address any remaining issues or provide further clarification!

We have made further efforts to address your concern about W6: Single-seed Evaluation and Stability and have completed another set of additional experiments. We would like to share the new results below.

We conducted two additional training runs (seeds 52 and 62) for selected segmentation and multi-label classification methods. We also evaluated Task 2 using the corresponding segmentation predictions. All results are reported as mean ± sample SD (n=2), in percent.

1. Task 1: Semantic Segmentation

ModelmIoUₚmIoUMacro-F1OA
EfficientViT (WCE+Dice)36.61 ± 1.7426.15 ± 1.2450.14 ± 1.9168.66 ± 3.31
UNetFormer (CE+Dice, aux.)37.02 ± 1.4630.85 ± 1.2149.50 ± 1.3870.24 ± 1.60
DINOv3 (WCE+Dice)38.64 ± 0.7627.60 ± 0.5453.93 ± 0.3064.50 ± 2.54
SESSRS (UNetFormer)37.26 ± 1.4231.05 ± 1.1849.72 ± 1.3470.29 ± 1.57

2. Task 2: Segmentation-derived Recognition

ModelmAPCF1OF1Macro-F1Sample-F1
EfficientViT57.39 ± 1.9961.75 ± 0.0373.23 ± 1.3558.88 ± 0.5474.23 ± 2.61
UNetFormer54.91 ± 0.2054.49 ± 0.1974.04 ± 0.2653.35 ± 0.2175.21 ± 0.21
DINOv364.21 ± 1.6164.29 ± 0.7072.50 ± 0.7962.54 ± 2.0371.88 ± 1.88
SESSRS (UNetFormer)54.91 ± 0.2054.49 ± 0.1974.04 ± 0.2653.34 ± 0.2175.21 ± 0.22

3. Task 3: Direct Multi-label Classification

Model / ImplementationmAPMacro-F1
C-Tran66.67 ± 1.4855.79 ± 3.39
ML-Decoder61.58 ± 0.9954.23 ± 5.28
DDA-MLIC (single-domain)59.89 ± 1.0254.62 ± 0.21
RelationNet repository ResNetBaseline58.02 ± 3.2857.44 ± 2.51
GRN (BCE-only)62.02 ± 1.2759.31 ± 0.12
RSMLC (ResNet-50)60.79 ± 1.4359.19 ± 0.43
SIGNA59.46 ± 1.2758.35 ± 3.12

Stability Analysis

The additional results support the main comparative findings. In Task 1, the mIoUₚ ranking remains unchanged, with SDs of 0.76–1.74 percentage points across the three independently trained configurations. Task 2 also preserves the original mAP ranking. In Task 3, six of seven configurations have mAP SDs below 1.5 points, and C-Tran remains the strongest among them. Some F1 metrics show greater variation, which we also report.

We have updated Appendix C.7 with the original and additional results. These experiments complement our previous 1,000 spatial-block bootstrap resamples and five-seed VLM probing results. Together, they provide further evidence on the consistency of the benchmark findings and the extent of evaluation and training variability.

We would be sincerely grateful if you could let us know whether our previous responses and these additional results have adequately addressed your concerns about the paper, including W6. If any concerns remain, or if any part of our responses requires further clarification, we would be more than happy to provide additional explanations or evidence. We will do our very best to address any further questions before the discussion ends.

Thank you again for your valuable feedback, time, and consideration!

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Summary of Revisions

We thank all three reviewers for their careful assessment and constructive comments. Their feedback identified several aspects of the dataset description, evaluation protocol, and experimental analysis that required further clarification. We have revised the manuscript and conducted additional experiments to address these points. The detailed results are provided in our individual responses.

[Dataset and Annotation]

We clarified the criteria for site selection and the rationale for the data split (Section 3; Appendix A). The revised description explains the use of independent orthomosaics for testing and the relationship between PlayaMirador1 and PlayaMirador2. We also acknowledge the limited test coverage of four rare classes and explain the role of their annotations in training and future evaluation. Additional clarifications address the expert annotation process, the availability of SfM/MVS data, and the distinction between mining-related visual evidence and the legal status of mining operations.

[Evaluation Protocol and Robustness]

We corrected recognition Macro-F1 to include all classes with ground-truth support, including those with zero F1. The affected results, rankings, and metric definitions have been updated. We also examined the effect of overlapping evaluation windows through inverse-coverage weighting, center selection, and non-overlapping layouts. The leading segmentation configuration remains unchanged across these settings. Additional spatial-block bootstrap analyses provide confidence intervals for fixed-model predictions, while supplementary training runs assess variability across model training. These results are reported separately in the revised appendix.

[Additional Experiments]

We extended the segmentation benchmark with TerraMind and DINOv3. Both models were evaluated with frozen pretrained encoders, trained segmentation decoders, and three loss settings. We also expanded Task 4 to include frozen linear probing under different annotation budgets and visual parameter-efficient fine-tuning. The new results show improvements from supervised adaptation, although performance on rare mining structures remains limited. These experiments provide additional comparisons of pretrained representations and adaptation methods under the ELDOR benchmark.

[Experimental Analysis]

We revised the experimental discussion to provide more detailed interpretations of class-wise performance and model differences (Sections 5.1 to 5.3). The results show that class frequency alone does not explain the difficulty of the benchmark. Some common ecological categories remain difficult because of their visual similarity, while rare mining structures present additional challenges due to their small spatial extent. The loss comparisons also reveal differences across semantic classes. Improvements in mining-structure recognition may coincide with lower accuracy on certain land-cover categories. We further expanded the discussion of Grad-CAM results in the main text to examine the role of contextual information in image-level recognition.

[Task Comparison and Practical Use]

We added threshold-selection experiments and evaluations under fixed expert review budgets. These analyses provide a clearer comparison of segmentation-derived recognition and direct multi-label classification. Task 3 is generally more effective for prioritizing rare mining structures under limited inspection budgets. Task 1 provides spatial predictions for mapping mining disturbance and ecological recovery, while Task 2 allows existing segmentation models to support image-level screening. Section 5.4 now discusses these differences and provides task-specific model recommendations based on the required outputs, class-wise performance, and computational efficiency.

[Manuscript Presentation]

We revised the related-work comparison and clarified the contributions of ELDOR relative to existing UAV mining datasets. We also improved the descriptions of the baseline families, refined several figures and tables, and clarified terminology and metric definitions. The Interactive Explorer section has been shortened, with implementation details retained in the appendix. The main experimental sections now place greater emphasis on the interpretation of results and the conclusions relevant to each benchmark task.

We appreciate the reviewers' comments and suggestions. Their feedback has helped us improve the experimental evaluation, clarify the main findings, and present the scope and limitations of ELDOR more precisely. We believe the revised manuscript provides a more complete and rigorous account of the benchmark and its potential applications.

Area_Chair_KHjfOpenReview ↗

Author Reviewer Discussion

Dear reviewers,

Thanks again for agreeing to review for ICLR.

As we are approaching the end of the discussion period, I would appreciate your acknowledgment of the authors' rebuttal and any feedback you would like to share with the authors.

This will help to improve the submissions, streamline decision making, and keep ICLR's original idea of having such a discussion period valuable to all participants in the review process.

Best, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.