- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This work proposes an agentic reinforcement learning framework for weather reporting. To be specific, they convert the task of weather reporting into a verified task via the introduction of a verifier. They achieve this by improving the three key components of the agentic RL pipeline: reward design, curriculum learning strategy, and the analysis of human-expert alignment. Through those improvements, the authors boost a set of base models' performance with the proposed agentic RL training strategy.
Critical strengths
- This paper is clearly written, as it explains both the motivation behind the problem and how it is addressed. In particular, the reasoning behind the proposed approach is presented clearly, making it easy to understand how the method works.
- The authors have carefully designed this framework to transform an ambiguous task like weather reporting into a verifiable and quantifiable one. They thoroughly investigate each design choice to assess its feasibility and identify the best option, particularly regarding verifier selection and training strategies.
- The experimental results demonstrate that the authors’ design choices make agentic RL training effective for weather reporting, which has been less explored. The authors also report detailed metrics throughout training, providing a useful reference for readers seeking to reproduce the approach.
Critical weaknesses
None
Questions for authors
- I would like to know the typical length of the final generated reports. I am curious whether the difficulty of RL training depends on report length. Could the authors analyze the relationship between reasoning length and document length during RL training?
- When introducing the verifier, the authors compare its assessments with those of human experts through a correlation analysis to validate its suitability. I would like to know whether the reports generated after RL training undergo an additional round of validation, with human experts assessing their quality.
3 comments
Thank you for your response. After careful reading, I lean to maintaining my previous score unchanged.
Thank you
Thank you for your positive review and valuable suggestions, which helped strengthen the evaluation of our work. We also thank you for taking the time to carefully read our response and for your continued support of this paper.
Rebuttal Response to Reviewer YMsy
We sincerely thank the reviewer for the positive and encouraging review. Below, we show that our scores agree with blind expert choices on full reports, and we analyze report and reasoning length.
Yes, at both the claim and the report level. The claim audit in Section 3.3 validates the judges that score every report during and after RL. It tests the step the reward depends on, labeling claims extracted from a report against the reference, and on these claims our judges agree with the meteorologists about as well as the meteorologists agree with each other (Table 1, Appendix D.3).
At the report level, four meteorologists compared 40 pairs of anonymized full reports from the evaluated systems, including our RL-trained agents, each shown next to the NWS reference, and chose the better report in their professional judgment. The training judge ranks the expert's choice higher in 31 of 39 pairs on F1 (79%, one tie excluded) and 28 of 40 on the rubric (70%), both well above chance (one-sided binomial p < 0.001 and p = 0.008). These results show that the scores agree with expert preference on full reports (Appendix D.3). Each blind comparison requires a meteorologist to read two full reports against the NWS reference, so expert judgments are slow to collect. We are still collecting more blind expert preferences between reports before and after RL on the same events, in which the experts weigh hazard identification, location and timing accuracy and practical usefulness, and will post them here as soon as they are complete.
The report prompt asks for about 250 to 350 words in 3 to 5 paragraphs (Appendix F.1), but this is not a strict requirement. In early experiments without this limit, agents wrote long reports that listed raw values and read unlike NWS forecast discussions. Test-set lengths are below, where Zephyrus-R1 denotes the Curriculum RL model.
Zephyrus-R1 reports run within the range of the other systems, and like most systems they slightly exceed the approximate 350-word target.
To test whether RL difficulty depends on report length, we evaluate six checkpoints of each run (steps 32 to 163) on the 197 test events, giving about 1,180 reports per run, rank the reports within each checkpoint and pool the six checkpoints before computing Spearman correlations, which removes shifts between checkpoints such as length drift. Reasoning length counts the generated tokens outside the final report.
Values are Spearman ρ under GPT-OSS-120B, the training judge whose scores form the reward, and * marks p < 0.05.
RL difficulty barely depends on report length. Report length is at most weakly correlated with rubric and F1 (|ρ| ≤ 0.08, significant in one run only), so among reports from the same checkpoint the judge does not reward length. Length is therefore unlikely to be what the policy learns to exploit, and report lengths stay within the range of the other systems.
Reasoning length is likewise unrelated to both scores (|ρ| ≤ 0.06), and its correlation with report length is small and inconsistent in sign (|ρ| ≤ 0.17). On the test events, Zephyrus-R1 reports are 1% longer than the base model at 27B and 22% longer at 4B (Table D.14). During report RL training, reasoning length changes from 4.8k to 4.6k tokens for 27B and from 4.6k to 4.1k for 4B, and the report is only 11% to 16% of the generated tokens. Reasoning and report length are at most weakly related, and RL changes the content of the reports (Table 2) far more than their length.
Both analyses are detailed in the revised manuscript (Appendices D.3 and D.6).