- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
CARMA constructs rewards for open-ended writing from diagnosed defects. A model proposes evidence-linked problems in a response, and a verifier groups diagnoses that can be fixed by the same semantic repair. Each event receives one calibrated penalty, which adjusts a separate holistic score. The paper tests whether this improves local rankings and group-relative advantages, then compares writing policies trained with the resulting rewards.
Critical strengths
- The central example makes sense to me. A character leaving without explanation may be criticized under motivation and continuity at the same time, yet one sentence could repair both. Conversely, two unrelated inconsistencies should incur two penalties even when they fall under the same rubric item. Counting repairs rather than the names of violated criteria is a useful way to think about this problem.
- The ablation is unusually informative. On the private 8B track, removing the event partition drops preference from 60.4% to 51.2% against the same holistic-score baseline, with the supervision otherwise matched. The paper also compares event maxima, event means, and fixed costs. I find these comparisons more persuasive than the overall score alone. The blind human study on the 27B model is another strength: CARMA wins 61.8% against holistic-score training, with a reported 95% interval of [56.7, 67.0].
Critical weaknesses
- The event grouping relies on an LLM recognizing whether two diagnoses share a repair. In the annotation study it gets 190 of 218 pair decisions right, which is promising, but a wrong merge can hide an independent defect and a wrong split can charge the same problem twice. The corruption experiment shows that these errors affect the trained model: preference falls from 60.4% to 54.9% when injected merge errors reach 20%. I would like to see actual examples of the grouping mistakes and whether they concentrate on a particular type of writing problem.
- The evidence from human preferences is narrower than the model-judged results might suggest. Against RRD-WU, human preference for CARMA is 54.5%, with a confidence interval of [49.3, 59.7]. I see a fairly clear case over the basic holistic baseline, but much less separation from this stronger alternative. The 27B policy comparisons are also based on one run per method. A second training seed would be especially helpful for the stronger-baseline comparison.
- The judge reliability audit is troubling in its own right: the pairwise judge matches the adjudicated human preference on 62 of 120 pairs, despite returning the same decision in 85 of them. This raises doubts about small differences in automatic preference results. I would also like to know the cost of diagnosis extraction and event verification relative to one holistic scoring call; the paper provides little information about judge tokens or time per response.
- A related issue is how portable the learned event costs are. The private Chinese and public English tracks fit their costs to different preference data, so the results mainly test whether the event construction can be reused. I would be interested in a fixed-cost transfer experiment across datasets or writing genres. If every new preference distribution needs another round of calibration, this changes how easily practitioners can adopt the method.
Questions for authors
- Can the authors show examples of false event merges and splits from the 218-pair annotation set? Which error type is more damaging in actual training?
- How stable is the comparison against RRD-WU across training seeds, especially on Qwen3.5-27B?
- What is the additional reward-scoring time or judge-token cost per rollout compared with the holistic baseline? Can the learned event costs be transferred across writing domains without refitting?
- Figures 1 and 2 appear to be raster images, with some details becoming blurry when enlarged. I suggest redrawing them as vector figures.
5 comments
Human comparison, judge reliability, and training seeds
Response to Weaknesses 2 and 3 and Question 2. The computational-cost aspect of Weakness 3 is also covered in the reply on computational cost and transfer.
- Human comparison (Weakness 2)
We appreciate the suggestion to strengthen the comparison with RRD-WU. The original 27B study reported 140 wins, 47 ties, and 113 losses, giving 54.5% preference with a 95% interval of [49.3, 59.7]. We have expanded the evaluation to examine annotation differences and training variability.
-
A new human panel reassessed all 300 unchanged response pairs, including the 47 originally labeled ties. Annotators were blinded to the old labels. The reassessment gives 161 wins, 28 ties, and 111 losses, or 58.3% preference.
-
We evaluated 100 previously unused prompts from the same Chinese writing source, giving 57 wins, 8 ties, and 35 losses, or 61.0%. The combined result is 218 wins, 36 ties, and 146 losses across 400 prompts, giving 59.0% [54.4, 63.5]. Table 7 reports the subsets separately.
-
We repeated CARMA and RRD-WU training with a second matched Qwen3.5-27B seed. Direct comparisons on the original 400 evaluation prompts favor CARMA under all three external judges in both runs. Gemini gives 62.9% and 64.3%, GPT gives 55.8% and 56.4%, and Claude Opus gives 58.7% and 63.5% (Table 11).
-
Matched training seeds (Question 2)
The seed comparison also addresses Weakness 2. Each seed trains both CARMA and RRD-WU from the same Qwen3.5-27B starting checkpoint, with matched data and the full 200-iteration budget. Table 11 reports their direct comparisons on 400 Chinese prompts. All three judges prefer CARMA in both runs. The table reports each run separately, with intervals computed by resampling prompts for that trained policy pair.
- Judge reliability (Weakness 3)
Thank you for raising this concern. Disagreement among annotators and model judges is a recurring difficulty in open-ended writing. We added a new human panel and three external judges to check the generated responses more closely.
The original audit found 37 decisions that were repeated and disagreed with the adjudicated human label. Appendix B reports this count alongside the 62/120 judge–human agreement and 85/120 repeatability figures.
The new study evaluates 400 CARMA–RRD-WU response pairs. On these same pairs, agreement with the final human labels is 76.3% for Gemini 3.1 Pro, 70.3% for GPT-5.5, and 81.0% for Claude Opus 4.6. The internal GRM agrees on 58.8%. The human panel reassesses all 300 original pairs and evaluates 100 unused prompts. Appendix A gives the blinding and adjudication procedure.
Appendix C.6 also gives stage inputs, outputs, calls, and scoring time. With Chinese responses of about 1,000 tokens and English responses of about 4,000 tokens, joint CARMA uses 2,504 and 5,804 judge tokens per response, respectively. The reply titled “Computational cost and transfer of learned penalties” gives the comparison with Holistic and the separate-stage schedule.
Computational cost and transfer of learned penalties
Response to Question 3 and Weakness 4, also covering the computational-cost aspect of Weakness 3.
- GRM calls, token use, and training cost (Question 3)
Thank you for asking for the cost breakdown. For eight responses, CARMA uses 8 GRM calls when one completion returns the score, diagnoses, and verification. Separate calls for the three stages use 24. Joint output shares one generation context. The separate-stage protocol isolates overall scoring and diagnosis inputs. In the separate implementation, scoring and diagnosis can overlap, followed by verification of the completed diagnoses.
With the response lengths in Appendix C.6, joint CARMA adds 1,000 judge output tokens per response over Holistic. A batch of 128 Chinese responses takes about 28.2 seconds with joint CARMA and 5.4 seconds with Holistic. For English, the corresponding times are 38.8 and 16.0 seconds. Appendix C.6 gives the stage inputs and outputs and the separate-stage times.
Training uses 16 AMD Instinct MI300X accelerators, each with 192 GB HBM3, to accommodate long writing inputs and outputs. Training takes approximately 63 h for multi-aspect GRPO, 72 h for CARMA, 81 h for RLTF-SD, 100 h for RRD-WU, and 104 h for the tested Pairwise configuration. The CARMA run uses approximately 1,152 accelerator-hours. All methods use batching and parallel execution within their stage dependencies.
The remaining call counts are 14 for the ArenaRL schedule, 28 for all-pairs comparison, 8M for M rubric items, and 24 or 16 for complete RLTF-SD trajectories. The parameter-transfer discussion in this reply jointly addresses Weakness 4 and Question 3, including fixed-cost LitBench evaluation and Chinese-to-English policy training.
- Reuse of learned penalties (Weakness 4 and Question 3)
Thank you for asking about portability. A new writing task may require changes to rubrics and judge prompts, which creates an adaptation cost. We therefore tested whether CARMA can reuse its 24 learned penalty increments. Fitting these values reuses cached scores, diagnoses, and event assignments, with the GRM frozen.
The fixed-cost LitBench test reuses the English-fitted vector on another dataset. It achieves 69.8% ranking accuracy, compared with 64.0% for holistic scoring. Fitting on LitBench preferences gives 76.1%. This comparison shows the result of direct reuse and the improvement from target-domain calibration.
We also tested transfer during English policy training with one matched Qwen3-8B seed. Here, we transfer CARMA's 24 penalty increments fitted on Chinese preference data and keep them fixed during English policy training. The comparison branch uses the increments fitted on English preferences. The two branches share the starting checkpoint, English data, 200-iteration budget, and evaluation procedure. Against the same English Holistic policy, preference is 59.1% with frozen Chinese-fitted costs and 67.8% with English-fitted costs. WritingBench scores are 7.29 and 7.64. In a direct comparison between the two branches, preference for the transferred-cost policy is 43.6%. The transferred costs retain the improvement over Holistic in this setting. Fitting on English preferences gives an additional 8.7 percentage points in preference against Holistic and 0.35 points on WritingBench. Appendix C.3 gives the comparison.
Event grouping, grouping errors, and figure clarity
Response to Weakness 1 and Questions 1 and 4.
Thank you for asking for grouping errors that readers can inspect directly. We have expanded Appendix E with a complete task and two full responses, followed by diagnosis records, repair decisions, and false-merge and false-split examples.
The audit contains 218 diagnosis pairs from 150 prompts. Of 94 pairs that share a repair, the verifier merges 81 correctly and splits 13 incorrectly. Of 124 pairs requiring independent repairs, it separates 109 correctly and merges 15 incorrectly. Table 14 reports the category denominators alongside the error counts. Motivation and causal transitions account for 7 of the 13 false splits, while entity, chronology, and object state account for 6 of the 15 false merges.
Each worked case now gives the relevant task constraint and response passage, the diagnoses, and a repair check. For example, explaining why Evan hands over the record leaves the disconnected recorder unresolved. Those diagnoses therefore require separate events.
Thank you for asking how the two errors affect learning. Our training intervention quantifies false merges. With costs fixed, injecting 20% merge corruption lowers policy preference from 60.4% to 54.9%. The false-split examples in Appendix E.1 show the corresponding changes to reward.
For two independent penalties of 0.10 and 0.04, a false merge charges 0.10 in place of 0.14, raising the response reward by 0.04. In the factory example, splitting three reports of the same defect charges 0.09 + 0.08 + 0.10 = 0.27 in place of the event charge of 0.10. Either change can reverse the preference between closely scored responses.
To compare their training impact directly, a false-split intervention should match the affected candidates, penalty changes, and training budget of the merge intervention. The repair-based cases in Appendix E.1 give the passages, diagnoses, partitions, and repair checks needed to inspect each error.
Thank you for the suggestion. We have redrawn Figures 1 and 2 as vector figures. We welcome any further comments on their clarity and legibility.