- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
Summary
This paper analyses classic linear correction for machine learning models that use monte-carlo labels for training.
Critical strengths
Strengths
- The theoretical analysis provided in the paper seems correct after a quick skim, and identifies useful derivations.
- The paper reads quite well across all sections.
- The experimental evaluation seems decent with a good and motivating simulator for this problem.
Critical weaknesses
Weaknesses
- From my understanding, and please correct me if I am wrong, the corrections requires a full run of 10x the particles (stated in Section 4) and the authors acknowledge that the corrected label costs approx 3.3m particles. That run is already approx 10x less noisy than the 300k label set, but the corrected 300k label set retains approx a 1/4 to 3/4 of the raw label's error? (see Table 2 results). As a result, I feel like maybe reporting the error of the full-budget and the pooled estimate would help illustrate whether the balance adds anything on top of the pooled field. Without this I am left a little uneasy about the results.
- There is no reproducible code provided with the paper to verify claims and implementation. The authors do provide so experimental protocol but this is not enough for open-science and of this venue, in my opinion.
- I feel like an overview (fig 1) would be useful to help a reader better understand the methodology. Text is currently dense and theory-heavy, and this could massively improve the paper.
- The paper shows the correction reduces label noise but not that this matters for learning? Given the venue, I feel like leaving this for future work is weak and massively reduces the paper's contribution.
Questions for authors
Questions
- My main question is if my intuition is correct around the first weakness, an answer to that could very much help my decision on this paper.
- What would happen to the results if the two empty support scenes were included?
- Was a scalar rescale of chosen by the screen ever considered, or perhaps a validation reference?
2 comments
Thank you for this detailed discussion back, and thank you for your hard work during this rebuttal period. I wanted to check all of the additional content and the other reviewers' concerns carefully. Given the discussion around this paper, I maintain my score.
Thank you for your careful review. Your questions led us to add the full-budget and pooled comparisons, test corrected labels in training, and examine the two omitted labels. We have also released the code and added a method overview.
L: revised PDF line; W: weakness; Q: question; W/Q: joint response.W1 / Q1. Full-budget and pooled labels
Your intuition is right at our airport budget, and the revision now reports both requested fields (Section 5.3, Fig. 3). We first used the 3M run only to build the correction, because we asked whether a balance can improve a label at the training budget (L.297–303). Once that run is paid for, it can also be used as a label, alone or averaged with the 300k label in proportion to their particles, which is your pooled field. Section 4 now says how much a correction must achieve to beat each: remove more than 9/10 of the raw label’s error to beat the 3M run, and more than 10/11 to beat the pooled field (L.304–309). Taking the 3M run’s error as 1 on the selected cells, the raw label has about 8.9, the corrected label 4.68 (NOx) and 5.81 (PM-1), and the pooled field 0.93 (L.480–485). The correction thus removes 47.64% and 34.78% of the raw label’s error, so about half and two thirds remain, within your 1/4 to 3/4 range. At our budget, averaging the 3M run with the label gives a better label than correcting with it (L.537–538). The correction still does what it was built for: it improves 31 of the 34 raw labels on these cells (L.533–534).
W2. Reproducible code
We share your view on open science at this venue, and the code is now available through an anonymous link: https://osf.io/9cmfj/overview?view_only=78fc24f5a9114ed692e113513806eb06. Its README explains the installation and gives one command per experiment. From the simulator outputs, the scripts rerun the correction and scoring of the 36 airport labels (18 scenes, two pollutants) and the flat-terrain experiments, and check each value against the archived one.
W3. Method overview
Thank you for this suggestion. We have added an overview of the method in Fig. 1 (L.270–284). The procedure starts from the inputs, the two runs of one airport scene, goes through the processing steps that build the balance and its trust weights, correct the label and average the runs, and ends with the four fields we compare and how each is scored. To make the theory easier to follow, Section 3 now starts from a single value before the matrix form (L.132–139). Table 1 lists every experiment with its runs, inputs, reference and scored cells (App. C.1, L.1242–1269).
W4. Learning from corrected labels
Following your recommendation, Section 5.4 now runs this test, under rules fixed before any run (L.492–506). For the anchored learner, which sees label quality, label noise does matter: raw 3M labels lower its median test error by 80.9% against raw 300k labels on 5,995 scenes (L.508–509). Corrected 300k labels instead raise this error by 10.66% against raw ones, a harm under the fixed rule whose interval over seeds nearly touches zero (L.509–510), and labels corrected only where the stored screen accepts give +5.40%, an undecided result (L.511–512). A plain learner, blind to the tenfold lower variance of 3M labels, cannot tell whether the correction helps (L.512–513). The paper therefore answers your question with a measurement: cleaner labels help a learner that can see them, and at our budget the 3M run does more as a label than as an input to the correction (L.535–538).
Q2. Empty-support scenes
Across all 18 airport scenes, the default correction still reduced total squared error by 2.46% for NOx and 7.63% for PM-1 (Table 4). To include the two previously omitted labels, we scored every ground cell outside buildings. In NOx scene 24 and PM-1 scene 1, so little of the plume reached the ground that no cell met our usual concentration threshold (L.1460–1462). For those two labels individually, the default correction increased squared error by 50.90% for NOx and 31.32% for PM-1. Our screening rule keeps the original label whenever no cell meets the threshold, so both labels remain uncorrected under that rule (L.1647–1648, 1657–1660).
Q3. Rescaling the trust covariance
We did not test rescaling Σδ in the original submission. We have now tested both methods you suggest, and both reduced squared error compared with leaving Σδ unchanged. On the selected cells, choosing the factor with the screen reduced total squared error by 10.5% for NOx and 19.5% for PM-1. Choosing the factor using reference simulations from the other scenes reduced the same error by 11.5% for NOx and 12.8% for PM-1 (Section 5.3, Table 4). Both methods usually favored a milder correction. The screen needed no additional simulations, while validation used 126 additional 3M-particle runs (Appendix C.5, L.1655–1657).