- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces GHGbench, a benchmark for predicting annual greenhouse-gas emissions with a company track of 32,830 company-years and a building track of 491,591 building-years from 26 cities. It fixes splits for gap filling, unseen buildings, held-out cities, later years, forecasting and satellite-image inputs, and it compares many tabular, foundation and LLM baselines. It finds that input access and distribution shift matter more than model choice, as company R² rises from about 0.3 to 0.9 with financial attributes and building R² falls to about 0.1 in new cities.
Critical strengths
-
The building release is substantial and auditable. It harmonizes 491,591 records from 12 public programs in 26 cities with frozen snapshots, row-level provenance and quality flags (Sections 3.2 and 3.3).
-
The protocol states what each score measures. Matched-record regimes separate input access from sample composition, and calculated targets such as Singapore's are kept apart from reported ones (Section 4.3).
-
The results are reported with care. Repeated runs and whole-entity paired bootstraps are given, and the paper states openly that TabPFN and tuned trees are mostly indistinguishable (Appendix P).
Critical weaknesses
-
The geographic headline mixes two changes. It compares pooled R² on unseen buildings (0.378 to 0.395) with city-average R² under city holdout (0.098 to 0.117), so the averaging and the population change together.
-
The company task lets one firm appear in both training and test, with no baseline that reuses its own disclosures. Persistence alone reaches R² 0.959 (Table 13), so models may win by recognizing the firm.
-
Some comparisons break the paper's own rules and the release is incomplete. Forecasting XGBoost uses 386 records against 1,930 for the others (Table 13), and Appendix C omits reconstruction counts and mappings.
The building resource is useful and well curated. However, the two headline findings are confounded by the split and averaging choices, and it is unclear how much demand exists for this benchmark. A firm-held-out split with a persistence baseline and a same-population geographic comparison would move me to weak acceptance.
Questions for authors
- Could the authors report the geographic drop when both settings use the same cities, records and averaging?
- Could the authors compare R² with company attributes against reusing each firm's previous disclosure, and also report results under a firm-held-out split?
- Would it be possible to release a frozen copy of the company panel so that the fixed splits can be rebuilt exactly?
Suggestions that did not affect my score. It would help to report log-scale R² next to raw R² because raw R² is dominated by the largest buildings, and to add a paired significance test for the multimodal gain in the appendix.
2 comments
Thank you for the clear plan and for taking my suggestions on board. I look forward to seeing the final version, and I wish you the best of luck with the paper.
Thank you for the careful responses. I am glad to see how seriously you treated my comments (rather than relying on AI to produce a generic rebuttal). The new experiments address my main concerns, and I appreciate that you report the results that weaken the original headlines. I have therefore raised my rating to weak acceptance.
In addition, I would like to help make the paper better, so here are a few suggestions for the final version:
I hope these are helpful.