- Rating
- 3
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The authors develop a framework for selective correction for the task of erosion-induced soil carbon prediction. The issue with this task is that, generally, you have very sparse data. Additionally, you have data points that are sampled within a similar region (a watershed). As a global model does not represent all regions well, it is sensible to design a correction for each watershed. This needs to be done very carefully, as per watershed you only have a few data points and can occasionally lead to degenerate performance.
This paper develops a mechanism for this balancing act, enabling improvement where possible and preventing degradation under the condition of extremely sparse and correlated data. Specifically, they equip a frozen base model with a regularized local correction and a lower-confidence-bound (LCB) gate.
The authors find that by building a mechanism that can fall back to the original prediction (which it does in ~75% of the time) and selected shrinkage of the correction term, they can achieve average performance gains whilst keeping the harm (degraded performance) minimal. They then show that the mechanism also works on two other erosion datasets. The improvement is marginal compared to the source performance, but they substantially reduce harm compared to a forced correction. They further test whether the corrections yield targeted improvement or win by just reducing corrections compared to random coverage.
Critical strengths
- The authors engineer a sensible solution for a tricky problem in light of the available data and severe spatial autocorrelation. The method is highly careful in the application of the correction and also adds amplitude shrinkage with a fallback (this care is measured by the ratio of times the correction is actually applied). The authors are very transparent about the limitations of the problem, such as spatial leakage and statistical claims.
- Their set of experiments is convincing in showing that the mechanism works in reducing harm while the gain itself stays moderate (though the gains are expected to be marginal in this constrained and data-sparse environment).
Critical weaknesses
- Even if the approach is sound and pragmatic, the scientific or application contribution is thin and makes me lean towards weak accept. The combination of local correction with selective prediction to selective correction is an interesting idea, but it is deployed here rather as a highly tailored engineering patch for this problem, and the paper does not convincingly show a leap in erosion-induced soil carbon prediction.
- Key details about the physical tasks were hard to extract and could have been eased by the authors by clarifying the form of the data and the nature of the spatial splits upfront. The evaluation seems fragmented, and it is hard to follow its logic (e.g., the 10-seed extension with a slightly different implementation comes out of nowhere).
Questions for authors
None
2 comments
Response to authors
Dear authors,
I thank you for your response to my concerns. I do believe that your extended explanations (including those to the other reviewers) for the five-seed and ten-seed setup, and the clarification of DEP and the whole downstream task, help the readability and task framing of the manuscript.
My main concern around the thin contribution (a marginal step with a highly tailored engineering patch) still stands unaddressed. I still believe the proposed method is a careful and well-executed correction with small gains for the specific task. For this reason, my score remains weak accept, and I defer to reviewers with more direct expertise on whether it clears the ICLR bar.
Thank you for pointing out where the prediction task and the purpose of the two DEP studies were difficult to follow. We revised the Introduction to explain what is predicted for each hillslope and how a team decides whether to use a local correction across a watershed.
On your first point, the Daily Erosion Project (DEP) prediction target is a hillslope’s modeled long-term mean annual lateral soil organic carbon flux, measured in g C m⁻² yr⁻¹. The example accompanying Figure 1 starts with a team that already has a prediction for every hillslope in a watershed, an area draining to a common outlet. Suppose labels become available for five hillslopes. We use three to fit a candidate correction and the other two to check whether it improves on the existing predictions. If the gate accepts the correction, we refit it using all five labels. If it declines, we retain the existing predictor. In either case, we evaluate the resulting predictions on the remaining hillslopes in that watershed, whose labels were held out from both fitting and selection.
On your second point, the five-seed study measures what happens when the policy is deployed. The ten-seed study asks whether the gate chooses better updates when the number of updates is held fixed. This distinction matters because a policy can reduce the total number of harmful updates simply by updating fewer watersheds.
In the five-seed study, the three DEP partitions withhold large drainage basins, one-degree grid cells, or environmental groups from source training. In each partition, the decision is whether to apply a correction to an individual watershed. Across 33,620 decisions, the gate applied a correction in 24.35% of cases. Regional root mean squared error (RMSE) increased in 8.04% of all 33,620 cases, while mean RMSE fell by 0.89% relative to the source model.
In the ten-seed study, the gate and random selection had the same candidate corrections and made the same number of updates within each split, seed, and label budget. Across the three DEP partitions, the share of applied updates that increased regional RMSE was 3.90–7.57 percentage points lower under the gate than expected under random selection. With update counts matched, this difference reflects which watersheds received corrections .
Sections 4.3 and 5.6 now distinguish these two questions and report the studies separately. Appendix F.1 documents how the 79,480-episode ten-seed study differs from the five-seed deployment study in its watershed cohort, target scaling, and rule for handling ties at the gate.
We hope these revisions clarify the prediction task and why the two DEP studies answer different questions. We welcome any further questions during the discussion.