- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes TRACE, a framework that uses VLM-generated textual analyses as an intermediate representation for 1 km monthly carbon-emission regression from satellite imagery. TRACE has two training phases. In cold-start initialization, a Qwen2.5-VL-72B Teacher receives the image, city, month, and the ODIAC-derived emission value z, and generates pseudo-analyses used to initialize an MLP Predictor that fuses image and text embeddings. In Periodical Evolution, a smaller Analyst VLM is trained with GRPO using two rewards: a Judge-based Analysis Score for grounding and relevance, and a Prediction Reward measuring whether the generated analysis reduces the frozen Predictor’s relative error. The Analyst and Predictor are updated alternately. Experiments cover five cities and four visual backbones, with R²/NRMSE/MAPE improvements, reward and text-input ablations, Judge scores, and a 50-case blind human audit.
Critical strengths
- The paper treats inspectable textual analysis as an intermediate variable for continuous carbon regression rather than using a VLM directly as a regressor. The reported gains across ViT, RemoteCLIP, GeoRSLIP, and DINOv3-sat suggest that the text branch is not tied to a single visual representation. This is a meaningful contribution for remote-sensing carbon estimation and for interpretable VLM applications.
- Periodical Evolution addresses two coupled problems: the absence of expert-written analysis labels and the risk that a fixed Predictor can be exploited by the Analyst. The design—Teacher pseudo-analysis initialization, dual-reward RL, and alternating Predictor retraining—is coherent. The reward ablation in Table 4 supports the claim that Analysis-only and Prediction-only training are each insufficient and that the combination is better.
Critical weaknesses
- The most serious concern is that cold-start pseudo-analyses are generated by a Teacher conditioned on the reference emission value z. The prompt explicitly places z in the context and permits quantitative or contextual justification. Pseudo-analyses may therefore contain emission values, intensity descriptions, or other label-correlated cues. The Predictor may learn to rely on label-like text rather than visual evidence, which would undermine both the interpretability claim and the validity of the Prediction Reward. The authors do not quantify label leakage in pseudo-analyses, nor do they provide no-label, randomized-label, or numeric-filtered cold-start controls.
- The statistical rigor and external validity are insufficient. The main results appear to use a single tile-level split without multi-seed means/variance or significance tests. Although tiles are disjoint, random tile splits can still leave spatial autocorrelation leakage across neighboring tiles. All five cities are relatively developed urban areas, and New York remains difficult (TRACE-4B R² ≈ 0.1397 in Table A1), indicating weak absolute performance in hard settings. The human audit uses only 50 cases, two evaluators, and only the 4B pair; inter-rater agreement and Judge-human correlation are not reported.
Questions for authors
- How is cold-start label leakage ruled out? Please quantify the fraction of Teacher pseudo-analyses that contain the reference emission value, numeric ranges, or label-recoverable cues. I recommend adding three cold-start controls: (i) Teacher without z, (ii) Teacher with randomized z, and (iii) pseudo-analyses with numeric/unit expressions filtered out. Also report a text-only Predictor and sentence-deletion/replacement tests on the generated analysis. If the Predictor relies mainly on label-like text in pseudo-analyses, the visual-attribution claim needs to be substantially revised.
- Are the composite reward and GRPO training robust? The paper defines Analysis Reward and Prediction Reward but does not clearly specify the composite weighting, normalization, sensitivity to n z, and the batch-mean rule Rbatch. Please report multi-seed variance, reward-weight ablations, and ablations of the Tau threshold and batch-mean rule, including how batch composition affects the reward. Also clarify the number of Periodical Evolution stages, stopping criteria, KL coefficient, and clipping threshold. 3.Judge bias and human-audit statistical power. The Teacher and Judge are both Qwen2.5-VL-72B, so self-preference or style bias is possible. Please report the correlation between Judge scores and human scores, inter-rater agreement between the two evaluators, confidence intervals for the 50-case audit, and a human audit for the 2B model or a larger audit sample. It would also be useful to test whether the Analysis Reward favors longer, more specific, but not necessarily more faithful analyses.
- Spatial split, label uncertainty, and baseline fairness. Please clarify whether the tile-level split is random-tile or spatial-block, and whether neighboring-tile leakage was checked. ODIAC 1 km monthly labels have uncertainty, and MAPE is unstable at low emission values; a label-noise/uncertainty analysis would strengthen the evaluation. For baselines, please add direct VLM regression, caption+fusion pipelines, remote-sensing VLMs such as GeoChat/RS-LLaVA, and a same-size Analyst without RL, so that the gain can be attributed to the method rather than to model scale or an extra text branch.
- Inference cost and faithfulness of the “inspectable” analysis. TRACE requires running the Analyst VLM at inference, followed by text encoding and the Predictor. Please report latency, memory/FLOPs, and failure rates relative to image-only inference. For interpretability, please add faithfulness tests: delete or replace key visual-evidence sentences in the analysis, ask humans to predict emissions from the analysis alone.
8 comments
[W2 / Q2] Five-seed evaluation (, mean std):
| Encoder | Image-only | Qwen3-VL-4B | TRACE-4B | vs. Image-only | vs. Qwen3-VL-4B |
|---|---|---|---|---|---|
| ViT-L/16 | |||||
| DINOv3-sat | |||||
| RemoteCLIP | |||||
| GeoRSCLIP |
All gains are positive on every seed, with TRACE-4B's std no larger than any baseline's.
Composite reward:
contributes only when ; both terms are bounded, requiring no further normalization. , where is the relative error of the exact same fused Predictor given an empty string (clarifying Eq. (2)). For tiles with (where relative improvement is ill-conditioned), assigns the neutral mean of non-fallback responses in the same rollout step ( prompts).
GRPO & Periodical Evolution (EasyR1): Group size ; low-variance KL ; clip ; AdamW (, weight decay ); temp , top-; max length tokens. Two 1-epoch stages ( steps/stage), refitting the Predictor after each stage. Stopping criterion: validation composite reward rises from to in Stage 1 and plateaus at in Stage 2, so training stops after Stage 2. We evaluate the final Stage-2 Analyst and Predictor. Final checkpoints lie within of peak validation reward; KL stays low/stable, entropy stays within , and refitting the Predictor causes no reward drop (). Curves are in the Appendix.
Reward ablation: Qwen3-VL-2B (), prediction-only (), analysis-only (), full composite (, over pred-only, over analysis-only).
[Q3] Self-preference audit. We added two cross-family evaluators (gpt-5.6-sol, gpt-6-astra) on the audit pair-cases, scoring factuality, carbon relevance, and hallucination-freeness on randomized, anonymized pairs given the image, city, and month:
| Evaluator | TRACE / Tie / Qwen3-VL-4B | Mean Diff (1–10 scale) | 95% CI |
|---|---|---|---|
| Human 1 | 38 / 3 / 9 | ||
| Human 2 | 27 / 7 / 16 | ||
| Human audit | 34 / 5 / 11 | ||
| gpt-5.6-sol | 28 / 5 / 17 | ||
| gpt-6-astra | 30 / 2 / 18 | ||
| Qwen2.5-VL-72B Judge (0–1 scale) | 31 / 11 / 8 |
Inter-rater agreement. The human reference uses Table 2's case-level audit scores. Strict-preference agreement is the fraction of matching directions, excluding cases with a tie from either evaluator. We use directions to avoid score-calibration differences (Human 2 scores pts higher on average): Human 1 vs. 2 is (); 72B Judge vs. human audit is (); gpt-5.6-sol and gpt-6-astra vs. human audit are () and (); and agreement between the two GPTs is (). All evaluators consistently prefer TRACE-4B.
[Q4] Spatial split. Within each city, tiles are randomly partitioned into Predictor-train, Analyst-train, validation, and test splits, keeping all months of a tile in the same split. Neighboring tiles may fall into different splits, yielding Moran's (annual mean, 4-neighbor weights) of in four cities and in New York. This involves no label leakage: no test images or labels are used during training, and is computed on tiles unseen by the Predictor. This evaluates spatial completion across unseen cells within mapped cities (Section 4.1) on an identical split across all methods.
Difficult cities (ViT, 5 seeds). Gains do not rely on high spatial autocorrelation: in New York (), image-only reaches , Qwen3-VL-4B , and TRACE-4B ( over image-only across seeds; low absolute reflects heavy-tailed labels with ). TRACE-4B also beats image-only on seeds in Beijing (), Chicago (), Shanghai (), and Singapore ().
Label uncertainty & MAPE. Because ODIAC is a modeled proxy and MAPE is sensitive to near-zero denominators, we prioritize and set . Appendix statistics show right-skewed per-city labels (skewness ; ): outliers with comprise of source monthly cells but have higher mean emissions. Thus, MAPE reflects low-emission cells while captures high-emission cells, motivating joint reporting of , NRMSE, and MAPE.
We thank the reviewer for the thorough review and concrete experimental suggestions. Below, we quantify label-related content in cold-start texts, report 5-seed results (mean ± std), detail the composite reward and GRPO settings, add bootstrap CIs and cross-family evaluators to the audit, clarify the spatial split, include the requested baselines, and report inference costs and sentence-deletion faithfulness tests.
[W1 / Q1] Implementation clarifications. (i) In Appendix A.1, "quantitative" was a typo for "qualitative". (ii) The Teacher generates pseudo-analyses on the Predictor-training subset exclusively to initialize the cold-start Predictor; Teacher texts on the Analyst-training subset are used only for cold-start early stopping. The Analyst is never trained on pseudo-analyses and never observes ground-truth labels; subsequent Predictors are trained solely on the Analyst's own outputs, and test inference uses no pseudo-analyses.
Label content in generated texts (excluding prompt boilerplate such as "1 km" and "2022"):
Filtering number+emission-unit tokens would alter only 0.05% of cold-start texts (1.25% if stripping all numbers), and TRACE-4B matches the raw ODIAC reference within ±0.5% (without requiring a unit match) no more often than Qwen3-VL-4B (0.12% vs. 0.14%). Downstream, the RL-trained Analyst consistently outperforms the same-size Qwen3-VL Analyst across all four visual encoders (see
[W2 / Q2]).Text-input controls:
[Q5]; confirms reliance on both visual-evidence and contextual sentences.