acceptodds
Under review as a conference paper at ICLR 2027

Zephyrus-R1: Agentic Reinforcement Learning for Weather Reporting

Abstract

Reinforcement learning (RL) has emerged as a key method for training AI models in scientific applications, but RL remains difficult when outputs cannot easily be verified against a known ground-truth answer. Natural-language weather forecast reports provide one such valuable domain example: these reports translate atmospheric data into assessments of weather evolution, hazards, and uncertainty, and they are valuable both to the general public and key decision makers. Unlike numerical weather-analysis questions, they admit many valid outputs, and deterministic verification is inherently limited due to the open-ended, natural-language setting. To solve these challenges, we develop Zephyrus-R1, a practical framework for RL on scientific weather forecast reporting. We focus on three key components of the agentic RL pipeline: reward design, curriculum learning, and an analysis of human-expert alignment. To encourage both report legibility and factual accuracy, we design a reward function that combines a meteorological rubric with bidirectional claim entailment, both evaluated against National Weather Service reference reports. Our framework optimizes this reward in a simulated tool-calling environment, enabling vision-language agents to gather and analyze forecast evidence across regions and lead times before producing a report. We use annotations from three meteorologists to audit the claim labels produced by our LLM judges. Finally, we introduce a curriculum learning process to help small models first develop tool use and scientific reasoning through RL on verifiable weather-analysis questions, then optimize the open-ended report reward directly. After training Qwen3.5-4B with Zephyrus-R1, we improve the mean across six judge–metric scores by 26% over the base model and achieve results competitive with frontier open-source models. Meanwhile, after training Qwen3.8-27B, we improve the same mean by 13% over the base model, by 6.7% over Kimi-K3, and by 2.7% over GPT-5.6-Luna.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning, Agents, Large Language Models, Weather Science, Curriculum Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_YMsyOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This work proposes an agentic reinforcement learning framework for weather reporting. To be specific, they convert the task of weather reporting into a verified task via the introduction of a verifier. They achieve this by improving the three key components of the agentic RL pipeline: reward design, curriculum learning strategy, and the analysis of human-expert alignment. Through those improvements, the authors boost a set of base models' performance with the proposed agentic RL training strategy.

Critical strengths
  1. This paper is clearly written, as it explains both the motivation behind the problem and how it is addressed. In particular, the reasoning behind the proposed approach is presented clearly, making it easy to understand how the method works.
  2. The authors have carefully designed this framework to transform an ambiguous task like weather reporting into a verifiable and quantifiable one. They thoroughly investigate each design choice to assess its feasibility and identify the best option, particularly regarding verifier selection and training strategies.
  3. The experimental results demonstrate that the authors’ design choices make agentic RL training effective for weather reporting, which has been less explored. The authors also report detailed metrics throughout training, providing a useful reference for readers seeking to reproduce the approach.
Critical weaknesses

None

Questions for authors
  1. I would like to know the typical length of the final generated reports. I am curious whether the difficulty of RL training depends on report length. Could the authors analyze the relationship between reasoning length and document length during RL training?
  2. When introducing the verifier, the authors compare its assessments with those of human experts through a correlation analysis to validate its suitability. I would like to know whether the reports generated after RL training undergo an additional round of validation, with human experts assessing their quality.
3 comments
AuthorsOpenReview ↗

Rebuttal Response to Reviewer YMsy

We sincerely thank the reviewer for the positive and encouraging review. Below, we show that our scores agree with blind expert choices on full reports, and we analyze report and reasoning length.

Q2: The verifier is validated by a correlation analysis against human experts. Do the reports generated after RL training also undergo a further round of validation in which human experts assess their quality?

Yes, at both the claim and the report level. The claim audit in Section 3.3 validates the judges that score every report during and after RL. It tests the step the reward depends on, labeling claims extracted from a report against the reference, and on these claims our judges agree with the meteorologists about as well as the meteorologists agree with each other (Table 1, Appendix D.3).

At the report level, four meteorologists compared 40 pairs of anonymized full reports from the evaluated systems, including our RL-trained agents, each shown next to the NWS reference, and chose the better report in their professional judgment. The training judge ranks the expert's choice higher in 31 of 39 pairs on F1 (79%, one tie excluded) and 28 of 40 on the rubric (70%), both well above chance (one-sided binomial p < 0.001 and p = 0.008). These results show that the scores agree with expert preference on full reports (Appendix D.3). Each blind comparison requires a meteorologist to read two full reports against the NWS reference, so expert judgments are slow to collect. We are still collecting more blind expert preferences between reports before and after RL on the same events, in which the experts weigh hazard identification, location and timing accuracy and practical usefulness, and will post them here as soon as they are complete.

Q1: What is the typical length of the final reports, and does the difficulty of RL training depend on report length? How do reasoning length and report length relate during RL training?

The report prompt asks for about 250 to 350 words in 3 to 5 paragraphs (Appendix F.1), but this is not a strict requirement. In early experiments without this limit, agents wrote long reports that listed raw values and read unlike NWS forecast discussions. Test-set lengths are below, where Zephyrus-R1 denotes the Curriculum RL model.

System (test, 197 events)Report words, mean ± sd
Qwen3.8-27B + Zephyrus-R1369 ± 29
GPT-5.6-Luna387 ± 21
Kimi-K3355 ± 27
DeepSeek-V4-Flash-Vision-Exp355 ± 32
GLM-5.3-Flash322 ± 19
MiniMax-M3454 ± 74
Qwen3.6-35B-A3B300 ± 30
Qwen3.5-122B-A10B341 ± 28
Qwen3.5-4B + Zephyrus-R1417 ± 48
Gemma-4-31B-IT287 ± 15
Qwen3.5-9B383 ± 108
Qwen3.8-27B base365 ± 35
Qwen3.5-4B base340 ± 50
NWS reference300 ± 87

Zephyrus-R1 reports run within the range of the other systems, and like most systems they slightly exceed the approximate 350-word target.

To test whether RL difficulty depends on report length, we evaluate six checkpoints of each run (steps 32 to 163) on the 197 test events, giving about 1,180 reports per run, rank the reports within each checkpoint and pool the six checkpoints before computing Spearman correlations, which removes shifts between checkpoints such as length drift. Reasoning length counts the generated tokens outside the final report.

RunLength vs. rubricLength vs. F1Reasoning vs. rubricReasoning vs. F1Reasoning vs. length
27B Direct RL−0.016+0.001+0.007+0.034+0.168*
27B Curriculum RL+0.003−0.019−0.009−0.028+0.048
4B Direct RL+0.080*+0.011−0.031−0.053−0.111*
4B Curriculum RL+0.042−0.006+0.002+0.016−0.026

Values are Spearman ρ under GPT-OSS-120B, the training judge whose scores form the reward, and * marks p < 0.05.

RL difficulty barely depends on report length. Report length is at most weakly correlated with rubric and F1 (|ρ| ≤ 0.08, significant in one run only), so among reports from the same checkpoint the judge does not reward length. Length is therefore unlikely to be what the policy learns to exploit, and report lengths stay within the range of the other systems.

Reasoning length is likewise unrelated to both scores (|ρ| ≤ 0.06), and its correlation with report length is small and inconsistent in sign (|ρ| ≤ 0.17). On the test events, Zephyrus-R1 reports are 1% longer than the base model at 27B and 22% longer at 4B (Table D.14). During report RL training, reasoning length changes from 4.8k to 4.6k tokens for 27B and from 4.6k to 4.1k for 4B, and the report is only 11% to 16% of the generated tokens. Reasoning and report length are at most weakly related, and RL changes the content of the reports (Table 2) far more than their length.

Both analyses are detailed in the revised manuscript (Appendices D.3 and D.6).

Thank you for your response. After careful reading, I lean to maintaining my previous score unchanged.

Thank you for your positive review and valuable suggestions, which helped strengthen the evaluation of our work. We also thank you for taking the time to carefully read our response and for your continued support of this paper.

Reviewer_2uCTOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how to train a weather-reporting agent with reinforcement learning. It first uses weather questions with clear answers to teach tool use and basic analysis, then trains report generation with rewards based on meteorological rubrics and factual consistency with reference reports. Experiments show improvements at two model sizes, and the authors use multiple LLM judges and meteorologists to check the reliability of the scores.

Critical strengths
  1. The problem is practical. Weather reports have no single correct answer, and fluent writing does not necessarily mean sound analysis. The paper addresses an important training challenge in open-ended scientific tasks.

  2. Beyond the final scores, the paper examines tool execution and shows how curriculum training helps the smaller model use tools.

  3. The paper provides a fairly complete training pipeline that brings together tool interaction, report rewards, and evaluation checks.

Critical weaknesses
  1. The methodological contribution is unclear. The approach mainly combines existing ideas in tool-using agents, LLM scoring, and curriculum RL. I’m not sure what the key new contribution is.

  2. The results show improvement, but do not clearly explain why. Does the curriculum transfer weather analysis skills, or does it mainly help the model become more familiar with the tools and benefit from more training?

  3. There is still a gap between the evaluation and actual report quality. Matching a reference report more closely does not necessarily mean the analysis is more accurate or useful, and the expert checks do not directly assess full reports. Using future reanalysis data as forecast inputs also makes it difficult to judge performance in real forecasting settings.

Questions for authors
  1. What does the combined reward improve over using only the rubric or factual consistency reward? Does it reduce missed hazards or incorrect hazard claims?
  2. Can the benefits of the curriculum be separated from the extra training budget? Why does it improve the rubric score for the 27B model but not F1?
  3. Could experts blindly review a small set of full reports before and after training, comparing hazard identification, location and timing accuracy, and practical usefulness?

I currently lean toward weak reject. I see the value of the problem, but I’m still unsure about the method’s added contribution and the practical meaning of the gains. I would consider raising my score with clearer evidence that the curriculum or reward design matters and that the reports themselves have improved.

6 comments
AuthorsOpenReview ↗

Rebuttal Response to Reviewer 2uCT (1/4)

We sincerely thank the reviewer for the careful and constructive review. Below, we add new results for each point. In a blind study on full reports, our scores agree with expert choices in 70% to 79% of pairs, and the gains hold with operational IFS HRES forecasts (W3, Q3). Training on the rubric alone lowers F1, and training on F1 alone lowers the rubric score, relative to the combined reward. RL also reduces missed and incorrect hazard claims at both scales (Q1). Extra report training on the same budget does not reproduce the curriculum's higher F1 or its reliable tool use (W2, Q2). Online reward weighting outperforms fixed weighting in 11 of 12 judge, metric and route combinations (W1).

W3, Q3: Matching the reference more closely does not necessarily mean a more accurate or useful analysis, and the expert checks do not directly assess full reports. Could experts blindly review a small set of full reports before and after training, comparing hazard identification, location and timing accuracy and practical usefulness? Using future reanalysis as forecast input also makes it hard to judge performance in real forecasting settings.

The reward does not score surface overlap with the reference. A claim the reference does not address is labeled NEUTRAL and does not affect precision, so the report is penalized only for claims the reference refutes and for reference claims it misses. The rubric grades meteorological content rather than wording, with six criteria such as setup fidelity, hazard prioritization and mechanism-to-weather reasoning (Table C.1), and the references are written by NWS forecasters.

To check that our scores track expert judgment of full reports, four meteorologists compared 40 pairs of anonymized full reports, each shown next to the NWS reference, and chose the better report. The training judge (GPT-OSS-120B) ranks the expert's choice higher in 31 of 39 pairs on F1 (79%, one tie excluded) and 28 of 40 on the rubric (70%), both well above chance (one-sided binomial p < 0.001 and p = 0.008). Together with the claim audit in Section 3.3, this shows that the scores track expert judgment of both individual claims and full reports (Appendix D.3). Each blind comparison requires a meteorologist to read two full reports against the NWS reference, so expert judgments are slow to collect. We are still collecting more blind expert preferences between reports before and after RL, and will post them here once complete.

In the paper, the forecast tool returns future ERA5 fields as an oracle forecast, which isolates analysis from forecast error. To test a realistic setting, we switch it at evaluation time, without retraining, to IFS HRES forecasts from WeatherBench 2, issued at or before the analysis time. Percentages are relative to the same-scale base, and Zephyrus-R1 denotes the Curriculum RL model. Avg is the mean of the six judge scores, as in Table 2.

ModelAvg, oracle (Table 2)Avg, IFS HRES
4B base0.3470.357
4B Zephyrus-R10.438 (+26%)0.444 (+24%)
27B base0.5060.504
27B Zephyrus-R10.571 (+13%)0.571 (+13%)

Scores change little with HRES forecasts, and Zephyrus-R1 keeps its gain over the base model. Ranked by Avg, the Table 3 systems nearly keep their oracle order with HRES forecasts (Spearman ρ = 0.97), and the 27B agent stays first. The HRES reports for DeepSeek use the newer DeepSeek-V4.1-Flash, since the Table 3 version is no longer available. The HRES results are detailed in Section 4.3, Table 4a and Appendix D.4 of the revised manuscript.

AuthorsOpenReview ↗

Rebuttal Response to Reviewer 2uCT (2/4)

Q1: What does the combined reward improve over using only the rubric or factual consistency reward? Does it reduce missed hazards or incorrect hazard claims?

One of the six rubric criteria scores hazard identification and prioritization, so the rubric directly rewards reports that capture and rank the main hazards. F1 checks every claim against the reference, so it penalizes claims the reference refutes and reference claims the report misses, including hazard claims. The combined reward thus covers hazards from both sides.

For the reward ablation, we train Qwen3.5-4B with the rubric only and with F1 only from the same Stage-1 checkpoint, all else identical. All new training runs in this reply use Qwen3.5-4B because of the time available during the rebuttal. Avg is the mean of the six scores.

RewardRubric GPT-OSSRubric GemmaRubric MistralF1 GPT-OSSF1 GemmaF1 MistralAvg
Rubric only0.1170.5730.4910.1700.6950.4430.415
F1 only0.1060.5050.4160.2340.7780.5260.428
Combined0.1290.5140.4410.2310.7680.5480.438

Relative to the combined reward, training on the rubric alone lowers F1 and training on F1 alone lowers the rubric score under every judge (Table D.10). Rubric only lowers mean F1 from 0.516 to 0.436, F1 only lowers the mean rubric score from 0.361 to 0.342, and the combined reward gives the highest Avg.

We also measure whether RL training reduces hazard errors, restricting the claim evaluation to hazard claims (tagged by GPT-OSS with the rubric's hazard definition). Missed is the share of the reference's hazard claims that the report does not support, and incorrect is the share of the report's checkable hazard claims that the reference refutes. Values are means over the three judges on the test set, and hazard rubric is the score of the rubric's hazard criterion. Percentages are relative to the same-scale base model, and Zephyrus-R1 denotes the Curriculum RL model.

ModelHazard rubric ↑Missed ↓Incorrect ↓
4B base0.2210.6530.373
4B Zephyrus-R10.262 (+19%)0.537 (−18%)0.223 (−40%)
27B base0.4730.5250.300
27B Zephyrus-R10.525 (+11%)0.490 (−7%)0.244 (−19%)

On the three-judge mean, report RL with our reward reduces both missed and incorrect hazards at both scales (Table D.13).

AuthorsOpenReview ↗

Rebuttal Response to Reviewer 2uCT (3/4)

W2, Q2: The results show improvement but do not clearly explain why. Does the curriculum transfer weather analysis skills, or mainly help the model become familiar with the tools and benefit from more training? Can its benefit be separated from the extra training budget, and why does it improve the rubric score at 27B but not F1?

The curriculum helps Stage 2 (report RL) in two ways that act together. It makes tool use more reliable and broader. At every checkpoint, Curriculum RL has a lower failure rate (4B 2.4% vs. 21.5%, 27B 0.3% vs. 0.9%) and uses more tool families per request (Tools/exec., 4B 1.85 vs. 1.47, 27B 2.09 vs. 1.66) than Direct RL (Figure 3). Direct RL barely changes these diagnostics relative to the base model (4B failure rate 21.5% vs. 20.7% for the base model), so this gain comes from Stage 1 (the verifiable question stage) rather than from report training. Stage 1 also improves analysis. Held-out accuracy on the verifiable analysis questions rises from 0.394 to 0.572 at 4B and from 0.648 to 0.708 at 27B (Figure 2), and these questions exercise the quantification, localization, timing and anomaly skills that reports use. At 27B, where both routes use tools reliably, the curriculum gives a higher rubric score under all three judges (GPT-OSS 0.260 vs. 0.167, Gemma 0.778 vs. 0.742, Mistral 0.728 vs. 0.719, Table 2) and at every checkpoint (Figure D.4). The net effect at the paper's budget is the higher Avg at both scales (4B 0.438 vs. 0.423, 27B 0.571 vs. 0.558, Table 2).

To separate the curriculum from the extra training budget, we continue Qwen3.5-4B Direct RL for 44 more report steps, the same number of updates, prompts and rollouts as the curriculum's Stage 1 (44 steps of 16 prompts with four rollouts per prompt, Section 4.1).

RouteRL stepsFailure rate ↓Tools/exec. ↑Rubric (3-judge mean)F1 (3-judge mean)Avg
Direct RL16321.5%1.470.3840.4620.423
Matched Direct RL20723.7%1.450.3950.4950.445
Curriculum RL44 + 1632.4%1.850.3610.5160.438

Extra report steps raise the rubric score and reach a similar Avg (0.445 vs. 0.438), but F1 stays lower under every judge and tool use does not improve, as the failure rate rises to 23.7%, against 2.4% with the curriculum. Extra report training therefore does not reproduce the curriculum's higher F1 or its reliable tool use (Table 4c, Table D.11).

At 27B the curriculum does not raise F1 over Direct RL because of how F1 treats claims the reference does not address (NEUTRAL). Such claims are excluded from precision, so converting them into checkable claims raises F1 only when the new claims are supported and lowers it when they are refuted. The curriculum makes many more claims checkable. On the test set, under all three judges, fewer of its claims are unaddressed by the reference (GPT-OSS 0.57 vs. 0.67 for Direct RL, Gemma 0.16 vs. 0.25, Mistral 0.46 vs. 0.59), and a larger share of all its claims is supported (0.24 vs. 0.19, 0.73 vs. 0.66, 0.40 vs. 0.32, Table D.12). Some of the newly checkable claims are refuted, which lowers precision a little, and the reports contain slightly fewer claims, which lowers recall a little. Both effects are small, so F1 stays within 0.034 of Direct RL under every judge. The rubric, which grades the report as a whole, may reward the more concrete content, and the curriculum's rubric advantage holds under all three judges (Figure D.4).

Reviewer_LLCyOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Zephyrus-R1, an agentic reinforcement learning (RL) framework for generating scientific weather forecast reports. Since natural-language weather reports require synoptic analysis, hazard identification, and uncertainty communication, they lack a single ground-truth answer, making deterministic verification inherently limited. To address this, Zephyrus-R1 enables vision-language agents to gather evidence via meteorological tools (e.g., reading atmospheric data, plotting, tracking cyclones) in a simulated environment and optimizes open-ended report generation using an LLM-as-a-Judge reward function that combines a meteorological rubric and bidirectional claim entailment (F1 score). Furthermore, it employs a two-stage curriculum learning approach: first training tool use and scientific reasoning on verifiable weather-analysis questions, followed by RL optimization on open-ended report generation.

Critical strengths
  1. The paper elegantly solves a critical pain point in scientific domains: open-ended scientific reports lack a single ground-truth answer, making traditional Reinforcement Learning (RL) inherently difficult to apply.
  2. This is a highly professional AI for Science paper that demonstrates how AI integrates with specific domains from the perspective of domain experts, possessing excellent industrial and practical application value.
Critical weaknesses

The current LLM judges relied upon in the paper (e.g., GPT-OSS, Gemma, Mistral) still have limitations in their authority and capability ceilings when evaluating complex, long-form scientific weather reports. It is strongly recommended to introduce larger, stronger, or mainstream top-tier closed-source models (e.g., GPT-4o, Claude 3.5 Sonnet) as independent judges for cross-evaluation.

Questions for authors

See Weaknesses

2 comments
AuthorsOpenReview ↗

Rebuttal Response to Reviewer LLCy

We thank the reviewer for the thoughtful review and for recognizing the practical value of our work for scientific domains. We add cross-evaluation with two recent closed-source judges, under which our gains and the system ranking largely hold.

W1: The LLM judges used in the paper (GPT-OSS, Gemma, Mistral) still have limited authority and capability ceilings when evaluating complex, long-form scientific weather reports. Larger, stronger or mainstream top-tier closed-source models, such as GPT-4o and Claude 3.5 Sonnet, should be added as independent judges for cross-evaluation.

Thank you for your suggestion! We agree that stronger judges can give more reliable evaluations of long scientific reports. GPT-4o and Claude 3.5 Sonnet date from mid-2024 and have been surpassed by many newer models, so we use their current successors, GPT-6.1-Terra and Claude Sonnet 5.5.

We then re-scored the test reports with both frontier judges, using the same rubric and claim protocol. The two tables follow Tables 2 and 3. Mean Rub. and Mean F1 average the two frontier judges, and Avg (frontier) is the mean of the four scores and is not comparable with the Avg of Table 2. Percentages are relative to the same-scale base model, and Zephyrus-R1 denotes the Curriculum RL model.

ModelRouteTerra Rub.Terra F1Sonnet Rub.Sonnet F1Mean Rub.Mean F1
Qwen3.5-4BBase0.0730.3700.1140.0180.0940.194
Direct RL0.0850.4290.1410.0380.113 (+20%)0.234 (+20%)
Curriculum RL0.0920.4720.1410.0350.116 (+24%)0.254 (+31%)
Qwen3.8-27BBase0.0910.4690.2050.0270.1480.248
Direct RL0.0970.5250.2200.0460.158 (+7%)0.285 (+15%)
Curriculum RL0.1030.5350.2410.0730.172 (+16%)0.304 (+23%)
ModelTerra Rub.Terra F1Sonnet Rub.Sonnet F1Avg (frontier)
Qwen3.8-27B + Zephyrus-R10.1030.5350.2410.0730.238
GPT-5.6-Luna0.1120.5580.2250.0550.238
Kimi-K30.1040.5410.2480.0550.237
DeepSeek-V4-Flash-Vision-Exp0.1030.5350.2170.0490.226
GLM-5.3-Flash0.1010.5270.2320.0290.222
MiniMax-M30.1000.5130.2120.0260.213
Qwen3.5-4B + Zephyrus-R10.0920.4720.1410.0350.185
Gemma-4-31B-IT0.1010.4180.1660.0280.178
Qwen3.5-122B-A10B0.0950.4220.1660.0240.177
Qwen3.6-35B-A3B0.0820.4280.1610.0320.176
Qwen3.5-9B0.0840.4000.1240.0190.157

Both RL routes improve every score over the base model, and Curriculum RL gives the larger gain in both mean rubric and mean F1 at both scales. The ranking of the 11 systems largely matches Table 3 (Spearman ρ = 0.92). The 27B agent has the highest Avg, tied with GPT-5.6-Luna, and the 4B agent moves from ninth to seventh.

Section 3.3 audits the LLM judges against meteorologists on 100 claims (Table 1). For this response, a fourth meteorologist labeled the same claims, and the revised Table 1 uses all four. We compare all five judges with the four-expert majority on all 100 claims. The four agree pairwise with κ 0.18 to 0.31 (Fleiss κ = 0.22).

JudgeAccuracyCohen's κWithin-1
GPT-OSS-120B0.6900.3450.970
Gemma-3-27B-IT0.5200.2300.940
Mistral-Small-3.2-24B0.6300.3560.950
GPT-6.1-Terra0.7000.3840.970
Claude Sonnet 5.50.6400.2180.950

The frontier judges agree with the experts about as well as the paper's three judges (κ of 0.384 for Terra and 0.218 for Sonnet, vs. 0.230 to 0.356), and every judge reaches a Within-1 of at least 0.94, so almost no disagreement is a full flip between SUPPORTED and REFUTED. Claude Sonnet 5.5 agrees least because it labels most claims NEUTRAL, which also drives its F1 toward zero.

Our training judge also rarely credits unsupported claims, the error a policy could exploit in RL. GPT-OSS-120B labels a claim SUPPORTED against the expert majority in only 6 of the 100 claims, compared with 40 for Gemma, 23 for Mistral, 9 for GPT-6.1-Terra and 0 for Claude Sonnet 5.5.

We also checked the judges on full reports. Four meteorologists compared 40 pairs of anonymized reports, each shown next to the NWS reference, and chose the better one. GPT-OSS-120B ranks the expert's choice higher in 31 of 39 pairs on F1 (79%, one tie excluded) and 28 of 40 on the rubric (70%), both well above chance (one-sided binomial p < 0.001 and p = 0.008).

We chose open-weight judges for training in the paper because they are widely used and perform well, and because scoring every RL rollout with closed-source judges is costly.

All results above are detailed in the revised manuscript (Sections 3.3 and 4.2, Appendices D.3 and D.4).

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer LLCy,

As the discussion period is nearing its deadline, we would greatly appreciate it if you could let us know whether our response has addressed your concerns. Following your suggestion, we added cross-evaluation with two recent closed-source judges, GPT-6.1-Terra and Claude Sonnet 5.5. Under both judges, our gains and the system ranking largely hold. We would be happy to clarify anything that remains unclear.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.