acceptodds
Under review as a conference paper at ICLR 2027

Not Just Saturation: Event-Structured Rewards for Open-Ended Post-Training

Abstract

During post-training for open-ended generation, judge scores for responses to the same prompt can concentrate within a narrow range as policy quality improves. We find that the issue is not the disappearance of reward differences, but whether the remaining differences within this narrow range reliably rank candidates by quality. When a limited judge struggles to distinguish closely matched responses, using these differences to estimate relative advantages can introduce noise or misdirect policy updates. Fine-grained rubric-based evaluation can reveal defects that holistic scores overlook, but directly aggregating the resulting diagnoses can introduce new problems. One underlying defect may be reported under multiple criteria, while several independent defects may be collapsed into a single criterion-level score. Poorly calibrated penalties can also overstate the impact of minor flaws and distort candidate rankings. We introduce CARMA, an event-structured reward framework that separates two questions: which diagnoses refer to the same defect, and how strongly each independent defect should affect reward. CARMA groups evidence-linked diagnoses that can be resolved by the same semantic repair into a single event and learns defect costs from same-prompt preferences. CARMA adjusts an independently obtained holistic score using event-level penalties, rather than assigning separate costs to multiple diagnoses of the same defect. In open-ended generation evaluations on Qwen3-8B and Qwen3.5-27B, CARMA-trained policies outperform the tested baselines in pairwise preference comparisons. It also yields more accurate near-tie rankings and advantage vectors that agree more closely with independently constructed references. Integrating event-structured signals into pairwise and feedback-driven training yields further gains over the corresponding methods without CARMA, supporting defect events as a reusable basis for open-ended post-training.

Primary area:
reinforcement learning
Keywords:
LLM post-training, reinforcement learning, reward modeling, LLM-as-a-judge, rubric-based rewards, preference learning, open-ended generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_hkPTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

CARMA constructs rewards for open-ended writing from diagnosed defects. A model proposes evidence-linked problems in a response, and a verifier groups diagnoses that can be fixed by the same semantic repair. Each event receives one calibrated penalty, which adjusts a separate holistic score. The paper tests whether this improves local rankings and group-relative advantages, then compares writing policies trained with the resulting rewards.

Critical strengths
  1. The central example makes sense to me. A character leaving without explanation may be criticized under motivation and continuity at the same time, yet one sentence could repair both. Conversely, two unrelated inconsistencies should incur two penalties even when they fall under the same rubric item. Counting repairs rather than the names of violated criteria is a useful way to think about this problem.
  2. The ablation is unusually informative. On the private 8B track, removing the event partition drops preference from 60.4% to 51.2% against the same holistic-score baseline, with the supervision otherwise matched. The paper also compares event maxima, event means, and fixed costs. I find these comparisons more persuasive than the overall score alone. The blind human study on the 27B model is another strength: CARMA wins 61.8% against holistic-score training, with a reported 95% interval of [56.7, 67.0].
Critical weaknesses
  1. The event grouping relies on an LLM recognizing whether two diagnoses share a repair. In the annotation study it gets 190 of 218 pair decisions right, which is promising, but a wrong merge can hide an independent defect and a wrong split can charge the same problem twice. The corruption experiment shows that these errors affect the trained model: preference falls from 60.4% to 54.9% when injected merge errors reach 20%. I would like to see actual examples of the grouping mistakes and whether they concentrate on a particular type of writing problem.
  2. The evidence from human preferences is narrower than the model-judged results might suggest. Against RRD-WU, human preference for CARMA is 54.5%, with a confidence interval of [49.3, 59.7]. I see a fairly clear case over the basic holistic baseline, but much less separation from this stronger alternative. The 27B policy comparisons are also based on one run per method. A second training seed would be especially helpful for the stronger-baseline comparison.
  3. The judge reliability audit is troubling in its own right: the pairwise judge matches the adjudicated human preference on 62 of 120 pairs, despite returning the same decision in 85 of them. This raises doubts about small differences in automatic preference results. I would also like to know the cost of diagnosis extraction and event verification relative to one holistic scoring call; the paper provides little information about judge tokens or time per response.
  4. A related issue is how portable the learned event costs are. The private Chinese and public English tracks fit their costs to different preference data, so the results mainly test whether the event construction can be reused. I would be interested in a fixed-cost transfer experiment across datasets or writing genres. If every new preference distribution needs another round of calibration, this changes how easily practitioners can adopt the method.
Questions for authors
  1. Can the authors show examples of false event merges and splits from the 218-pair annotation set? Which error type is more damaging in actual training?
  2. How stable is the comparison against RRD-WU across training seeds, especially on Qwen3.5-27B?
  3. What is the additional reward-scoring time or judge-token cost per rollout compared with the holistic baseline? Can the learned event costs be transferred across writing domains without refitting?
  4. Figures 1 and 2 appear to be raster images, with some details becoming blurry when enlarged. I suggest redrawing them as vector figures.
5 comments
AuthorsOpenReview ↗

Event grouping, grouping errors, and figure clarity

Response to Weakness 1 and Questions 1 and 4.

  1. Grouping decisions and examples (Weakness 1)

Thank you for asking for grouping errors that readers can inspect directly. We have expanded Appendix E with a complete task and two full responses, followed by diagnosis records, repair decisions, and false-merge and false-split examples.

The audit contains 218 diagnosis pairs from 150 prompts. Of 94 pairs that share a repair, the verifier merges 81 correctly and splits 13 incorrectly. Of 124 pairs requiring independent repairs, it separates 109 correctly and merges 15 incorrectly. Table 14 reports the category denominators alongside the error counts. Motivation and causal transitions account for 7 of the 13 false splits, while entity, chronology, and object state account for 6 of the 15 false merges.

Each worked case now gives the relevant task constraint and response passage, the diagnoses, and a repair check. For example, explaining why Evan hands over the record leaves the disconnected recorder unresolved. Those diagnoses therefore require separate events.

  1. Effects of grouping errors (Question 1)

Thank you for asking how the two errors affect learning. Our training intervention quantifies false merges. With costs fixed, injecting 20% merge corruption lowers policy preference from 60.4% to 54.9%. The false-split examples in Appendix E.1 show the corresponding changes to reward.

For two independent penalties of 0.10 and 0.04, a false merge charges 0.10 in place of 0.14, raising the response reward by 0.04. In the factory example, splitting three reports of the same defect charges 0.09 + 0.08 + 0.10 = 0.27 in place of the event charge of 0.10. Either change can reverse the preference between closely scored responses.

To compare their training impact directly, a false-split intervention should match the affected candidates, penalty changes, and training budget of the merge intervention. The repair-based cases in Appendix E.1 give the passages, diagnoses, partitions, and repair checks needed to inspect each error.

  1. Figure clarity (Question 4)

Thank you for the suggestion. We have redrawn Figures 1 and 2 as vector figures. We welcome any further comments on their clarity and legibility.

AuthorsOpenReview ↗

Human comparison, judge reliability, and training seeds

Response to Weaknesses 2 and 3 and Question 2. The computational-cost aspect of Weakness 3 is also covered in the reply on computational cost and transfer.

  1. Human comparison (Weakness 2)

We appreciate the suggestion to strengthen the comparison with RRD-WU. The original 27B study reported 140 wins, 47 ties, and 113 losses, giving 54.5% preference with a 95% interval of [49.3, 59.7]. We have expanded the evaluation to examine annotation differences and training variability.

  1. A new human panel reassessed all 300 unchanged response pairs, including the 47 originally labeled ties. Annotators were blinded to the old labels. The reassessment gives 161 wins, 28 ties, and 111 losses, or 58.3% preference.

  2. We evaluated 100 previously unused prompts from the same Chinese writing source, giving 57 wins, 8 ties, and 35 losses, or 61.0%. The combined result is 218 wins, 36 ties, and 146 losses across 400 prompts, giving 59.0% [54.4, 63.5]. Table 7 reports the subsets separately.

  3. We repeated CARMA and RRD-WU training with a second matched Qwen3.5-27B seed. Direct comparisons on the original 400 evaluation prompts favor CARMA under all three external judges in both runs. Gemini gives 62.9% and 64.3%, GPT gives 55.8% and 56.4%, and Claude Opus gives 58.7% and 63.5% (Table 11).

  4. Matched training seeds (Question 2)

The seed comparison also addresses Weakness 2. Each seed trains both CARMA and RRD-WU from the same Qwen3.5-27B starting checkpoint, with matched data and the full 200-iteration budget. Table 11 reports their direct comparisons on 400 Chinese prompts. All three judges prefer CARMA in both runs. The table reports each run separately, with intervals computed by resampling prompts for that trained policy pair.

  1. Judge reliability (Weakness 3)

Thank you for raising this concern. Disagreement among annotators and model judges is a recurring difficulty in open-ended writing. We added a new human panel and three external judges to check the generated responses more closely.

The original audit found 37 decisions that were repeated and disagreed with the adjudicated human label. Appendix B reports this count alongside the 62/120 judge–human agreement and 85/120 repeatability figures.

The new study evaluates 400 CARMA–RRD-WU response pairs. On these same pairs, agreement with the final human labels is 76.3% for Gemini 3.1 Pro, 70.3% for GPT-5.5, and 81.0% for Claude Opus 4.6. The internal GRM agrees on 58.8%. The human panel reassesses all 300 original pairs and evaluates 100 unused prompts. Appendix A gives the blinding and adjudication procedure.

Appendix C.6 also gives stage inputs, outputs, calls, and scoring time. With Chinese responses of about 1,000 tokens and English responses of about 4,000 tokens, joint CARMA uses 2,504 and 5,804 judge tokens per response, respectively. The reply titled “Computational cost and transfer of learned penalties” gives the comparison with Holistic and the separate-stage schedule.

AuthorsOpenReview ↗

Computational cost and transfer of learned penalties

Response to Question 3 and Weakness 4, also covering the computational-cost aspect of Weakness 3.

  1. GRM calls, token use, and training cost (Question 3)

Thank you for asking for the cost breakdown. For eight responses, CARMA uses 8 GRM calls when one completion returns the score, diagnoses, and verification. Separate calls for the three stages use 24. Joint output shares one generation context. The separate-stage protocol isolates overall scoring and diagnosis inputs. In the separate implementation, scoring and diagnosis can overlap, followed by verification of the completed diagnoses.

With the response lengths in Appendix C.6, joint CARMA adds 1,000 judge output tokens per response over Holistic. A batch of 128 Chinese responses takes about 28.2 seconds with joint CARMA and 5.4 seconds with Holistic. For English, the corresponding times are 38.8 and 16.0 seconds. Appendix C.6 gives the stage inputs and outputs and the separate-stage times.

Training uses 16 AMD Instinct MI300X accelerators, each with 192 GB HBM3, to accommodate long writing inputs and outputs. Training takes approximately 63 h for multi-aspect GRPO, 72 h for CARMA, 81 h for RLTF-SD, 100 h for RRD-WU, and 104 h for the tested Pairwise configuration. The CARMA run uses approximately 1,152 accelerator-hours. All methods use batching and parallel execution within their stage dependencies.

The remaining call counts are 14 for the ArenaRL schedule, 28 for all-pairs comparison, 8M for M rubric items, and 24 or 16 for complete RLTF-SD trajectories. The parameter-transfer discussion in this reply jointly addresses Weakness 4 and Question 3, including fixed-cost LitBench evaluation and Chinese-to-English policy training.

  1. Reuse of learned penalties (Weakness 4 and Question 3)

Thank you for asking about portability. A new writing task may require changes to rubrics and judge prompts, which creates an adaptation cost. We therefore tested whether CARMA can reuse its 24 learned penalty increments. Fitting these values reuses cached scores, diagnoses, and event assignments, with the GRM frozen.

The fixed-cost LitBench test reuses the English-fitted vector on another dataset. It achieves 69.8% ranking accuracy, compared with 64.0% for holistic scoring. Fitting on LitBench preferences gives 76.1%. This comparison shows the result of direct reuse and the improvement from target-domain calibration.

We also tested transfer during English policy training with one matched Qwen3-8B seed. Here, we transfer CARMA's 24 penalty increments fitted on Chinese preference data and keep them fixed during English policy training. The comparison branch uses the increments fitted on English preferences. The two branches share the starting checkpoint, English data, 200-iteration budget, and evaluation procedure. Against the same English Holistic policy, preference is 59.1% with frozen Chinese-fitted costs and 67.8% with English-fitted costs. WritingBench scores are 7.29 and 7.64. In a direct comparison between the two branches, preference for the transferred-cost policy is 43.6%. The transferred costs retain the improvement over Holistic in this setting. Fitting on English preferences gives an additional 8.7 percentage points in preference against Holistic and 0.35 points on WritingBench. Appendix C.3 gives the comparison.

Reviewer_UpEHOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The paper introduces CARMA, which separates the identification of independent defects from their reward cost. The authors find that CARMA improves policy preference over holistic-reward group-centered training.

Critical strengths
  • CARMA is clearly explained and Figure 2 is informative of the overall framework
  • CARMA shows clear improvements on the two benchmarks compared to a good amount of baselines
Critical weaknesses
  • The introduction reads like a related work section. It would benefit from revising into coherent paragraphs to motivate CARMA
  • Only experimented with models from the Qwen family
  • Writing is hard to follow and possibly AI-generated. Would be helpful to provide more context for the reader to understand the task.
Questions for authors
  • Why did you choose private Chinese creative writing and public English writing as the two tasks?
  • Could you provide some examples of what the tasks look like in the main text?
  • What does the percentage represent in the results?
  • Could you include error bars in the tables?
8 comments
AuthorsOpenReview ↗

Motivation, writing tasks, and manuscript clarity

Response to Weaknesses 1 and 3 and Questions 1 and 2.

  1. Motivation (Weakness 1)

Thank you for pointing out that the motivation was difficult to follow. We have rewritten the introduction around the training problem. It begins with same-prompt responses that receive nearly identical scores despite substantive quality differences, then explains how repeated penalties or missed defects can change their ordering and centered advantages.

The introduction uses a shared-repair example and an independent-defect example to explain CARMA’s event construction. Comparisons with earlier reward methods are developed in Section 6 and Appendix H.

  1. Task context and manuscript clarity (Weakness 3)

Thank you for identifying the missing task context. Section 5.1 now gives concrete writing requests before presenting the model, data, and evaluation setup. The introduction and method definitions explain how a task, a response, a diagnosis, and an event enter training.

Appendix E provides the complete factory task, both full responses with dialogue and observable actions, diagnosis evidence, event assignments, repairs, and reward calculations. The examples then lead into the grouping-error audit. We also revised the prose to make the operations explicit and shortened repeated explanations.

  1. Concrete writing examples (Question 2)

The task examples also address Weakness 3. Section 5.1 gives a reflective-essay request about a changed hometown and a story request about a missing letter. Appendix E follows with a complete factory task and two full responses, then traces their passages through diagnosis, event grouping, and reward calculation.

  1. Choice of writing data (Question 1)

Thank you for asking about the choice of writing settings. The Chinese corpus comes from sustained development and repeated training of a production writing system. Its prose, fiction, and scripts let us investigate the reward problem in the setting where we encountered it. The public English data provide a second language and accessible data sources for checking the same construction.

Section 5.1 and Appendix C.4 specify the sources and the separate calibration, development, policy-training, and evaluation splits. The two languages fit their costs separately. Appendix C.3 additionally tests reusing the Chinese-fitted vector during English training.

AuthorsOpenReview ↗

Model generalization and interpretation of evaluation results

Response to Weakness 2 and Questions 3 and 4.

  1. Model-family evaluation (Weakness 2)

Thank you for suggesting a broader model-family check. We added English-writing training from the same Gemma4-12B-it starting checkpoint for Holistic, RRD-WU, and CARMA, using matched data and budgets. This tests the reward construction on a policy family beyond Qwen.

Under the main internal-GRM protocol, RRD-WU and CARMA achieve 59.5% and 66.1% preference against the Gemma Holistic policy (Table 2, panel D). In direct CARMA–RRD-WU comparisons, the three external judges give CARMA preferences of 58.8%, 60.4%, and 58.2% (Table 12). WritingBench scores are 7.56, 7.73, and 7.87 for Holistic, RRD-WU, and CARMA. Appendix C.5.1 describes the shared training setup.

  1. Meaning of the reported metrics (Question 3)

Thank you for asking us to clarify the percentages. Generation preference is 100 × (wins + ties/2) / number of prompts, against the opponent named in the table. For example, the original CARMA–RRD-WU human comparison gives 100 × (140 + 47/2) / 300 = 54.5%.

Each main-table column uses the Holistic policy trained from its corresponding model and dataset as the reference. Direct CARMA–RRD-WU comparisons name RRD-WU as the opponent. LitBench reports the percentage of fixed story pairs ranked consistently with the reference labels. WritingBench reports mean scores on a ten-point scale. Section 5.1 and Appendix C.2 now define these quantities.

  1. Uncertainty estimates (Question 4)

Thank you for requesting uncertainty estimates. We added 95% intervals to the independent-generation, matched-seed, and Gemma comparisons, and clarified the errors reported in the main policy table.

The original 8B results report the mean and sample standard deviation over three training seeds. The original single-run 27B errors are prompt-bootstrap standard errors. The added comparison tables use 100,000 prompt-bootstrap resamples for each pair of trained models. Both response-order judgments for a prompt stay together during resampling. Appendix C.2 gives the calculation.

Thank you for the response! Most of my concerns have been addressed, and the draft is a lot clearer and easier to follow. Before I change my score, I have a couple additional questions:

  1. Could you clarify what these numbers mean in the abstract and why they are significant? ("CARMA-trained 8B and 27B policies achieve 60.4%/58.3% preference on private Chinese data and 59.8%/57.9% on public English data against their corresponding holistic-reward policies")
  2. What is the full name for RRD in the introduction? (line 42)
Reviewer_r83XOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CARMA, an event-structured reward framework for open-ended post-training that addresses reward distortions caused by redundant or improperly aggregated fine-grained diagnoses. The key idea is to identify independent defects based on shared semantic repairs, preventing multiple diagnoses of the same underlying problem from being penalized repeatedly while preserving separate penalties for distinct defects. CARMA assigns each event its maximum supported penalty and calibrates penalty weights using same-prompt preferences, combining them with a holistic quality score. The paper also analyzes the effects of reward aggregation on group-relative advantages. Experiments on Chinese and English writing tasks with Qwen3-8B and Qwen3.5-27B show improvements over holistic scoring and several fine-grained reward baselines. Controlled tests, ablation studies, human preference evaluations, and evaluations on WritingBench and LitBench further validate the proposed approach.

Critical strengths
  1. The paper identifies an important limitation of fine-grained reward modeling: multiple diagnoses may refer to the same underlying defect, while distinct defects may be overlooked by criterion-level aggregation.

  2. The paper provides extensive experiments across two model scales and both Chinese and English writing tasks.

Critical weaknesses
  1. Limited independence of automatic evaluation. The paper uses the same GRM for reward construction and automatic evaluation. Although these use separate calls and evaluation does not receive constructed rewards or diagnostic penalties, shared systematic biases remain a concern. Appendix C further reports agreement with adjudicated human preferences on only 62 of 120 held-out pairs, raising questions about judge validity on this set. Human evaluations support CARMA over holistic-reward training, but its advantage over RRD-WU in the private Chinese 27B setting remains statistically inconclusive: the preference rate is 54.5%, with a 95% confidence interval of [49.3%, 59.7%]. LitBench provides evidence based on external preference labels, but evaluates reward ranking rather than downstream policy generation. Thus, independent evidence that CARMA consistently improves human-perceived generation quality over strong baselines across the evaluated settings remains limited. Evaluation with independent judges or broader human comparisons would strengthen this conclusion.
Questions for authors
  1. CARMA uses multiple GRM-based stages for holistic scoring, defect diagnosis, and event verification, adding computational overhead relative to holistic scoring alone. However, the paper does not quantify the token consumption or inference latency of these stages, or their impact on total training cost. Could the authors provide a per-stage cost breakdown and an overall cost comparison with the holistic-reward baseline?

  2. While the human evaluations and LitBench results provide complementary evidence, the human preference advantage over RRD-WU remains statistically inconclusive, and LitBench assesses reward ranking rather than policy generation. Could the authors provide additional independent evaluation of generation quality against strong baselines, using an external judge or broader human comparisons?

2 comments
AuthorsOpenReview ↗

Independent evaluation and evidence for generation quality

Response to Weaknesses 1–4 and Question 2.

  1. Evaluation independence (Weakness 1)

Thank you for asking about evaluation independence. Agreement between human annotators, and between humans and model judges, remains a central challenge in open-ended writing. To provide independent evidence for the generation-quality results, we expanded evaluation beyond the internal GRM. The revised setup describes the frozen Qwen3-32B-based GRM and its supervision sources. Appendix B reports the original 62/120 agreement audit and the repeated errors.

A new blinded human panel reassesses the original 300 CARMA–RRD-WU response pairs and evaluates 100 unused prompts. Gemini 3.1 Pro, GPT-5.5, and Claude Opus 4.6 judge the same 400 pairs in both response orders. Their agreement with the final human labels is 76.3%, 70.3%, and 81.0%, respectively, while the internal GRM gives 58.8% on these pairs. Appendix A states the blinding, adjudication, and input protocol, and Table 8 reports judge–human agreement.

Following your concern and related comments from the other reviewers, we also examined the disagreement types. After combining both response orders, the internal GRM labels 144 pairs as ties, compared with 36 human ties. Of 165 disagreements, 121 (73.3%) are cases where the GRM selects a tie and the human panel selects a winner. Appendix A now reports this breakdown.

  1. Human comparison with RRD-WU (Weakness 2)

Thank you for asking for a stronger comparison with RRD-WU. The original human study reports 140 wins, 47 ties, and 113 losses, or 54.5% preference [49.3, 59.7]. We expanded the evaluation by reassessing all 300 unchanged response pairs with new blinded annotators, including the original ties, and then evaluating 100 unused prompts from the same Chinese writing source.

The reassessment gives 161 wins, 28 ties, and 111 losses, or 58.3% preference. The unused prompts give 57 wins, 8 ties, and 35 losses, or 61.0%. Combined preference is 59.0% [54.4, 63.5] over 400 prompts. Table 7 reports the subsets separately, and Appendix A describes the annotation and prompt-bootstrap procedures.

  1. Reward ranking and generated-response quality (Weakness 3)

Thank you for emphasizing the distinction between reward ranking and generated-response quality. Section 5.9 and Appendix C.3 now explain the two evaluations explicitly. LitBench scores 2,480 existing story pairs against their reference preference labels. WritingBench evaluates new responses generated by the trained policies on 1,000 queries.

The independent study in Section 5.5 and Appendix A also compares generated CARMA and RRD-WU responses with a new human panel and external judges. Table 3 reports that generation comparison. The LitBench and generation results are presented with their respective tasks and metrics.

  1. Broader generation evidence (Weakness 4)

Thank you for suggesting broader independent evidence. The added Chinese 27B generation study directly compares CARMA with RRD-WU using new human annotations and three external judges. Combined human preference is 59.0% [54.4, 63.5], and the three external preferences are 62.8%, 56.1%, and 58.2% (Table 3). The matched-seed comparison in Table 11 examines another training run, with all three judges preferring CARMA in both runs. The independent study reuses 300 Seed 1 response pairs and adds 100 unused prompts; Table 11 uses the original 400-prompt set.

The English Gemma experiment extends the policy family and language. Its direct external CARMA–RRD-WU preferences are 58.8%, 60.4%, and 58.2%, and its WritingBench means are 7.87 for CARMA and 7.73 for RRD-WU (Table 12). The evaluation protocol identifies the judge and reference policy for each result. Human annotations cover the Chinese 27B study, while the English extension contributes model-judged generation evidence.

  1. Coverage of the additional comparison (Question 2)

The additional generation comparison jointly addresses Weaknesses 2 and 4. The added Chinese 27B comparison uses new human annotations and three external judges on 300 reassessed prompts plus 100 unused prompts. Table 3 summarizes CARMA–RRD-WU generation preference, Table 7 separates the two human subsets, and Table 8 gives judge–human agreement and comparisons with Holistic. Table 11 and the English Gemma results in Table 12 broaden the training-run and policy-family coverage.

AuthorsOpenReview ↗

GRM token use, latency, and training resources

Response to Question 1.

  1. Stage costs and overall training resources (Question 1)

Thank you for requesting this breakdown. With joint output, CARMA adds 1,000 judge output tokens per response over Holistic. It uses 8 GRM calls for eight responses. A separate scoring, diagnosis, and verification implementation uses 24 calls. Joint output shares one generation context. The separate-stage protocol isolates overall scoring and diagnosis inputs.

The budget in Appendix C.6 uses Chinese responses of about 1,000 tokens and English responses of about 4,000 tokens. Holistic returns a numeric score of about 4 tokens. CARMA adds about 500 diagnosis tokens and 500 verification tokens. Joint input-plus-output totals are 2,504 tokens per Chinese response and 5,804 per English response. The appendix also gives the separate-stage inputs and outputs.

For 128 responses on 16 MI300X accelerators, Chinese batch scoring takes 28.2 seconds with joint CARMA and 5.4 seconds with Holistic. The English times are 38.8 and 16.0 seconds. Over 200 rollout iterations, this scoring increment is about 1.3 h, or 20.3 accelerator-hours, if the other training phases are held fixed.

Each of the 16 MI300X accelerators has 192 GB HBM3, selected for the long writing inputs and outputs. Application training takes approximately 63 h for multi-aspect GRPO, 72 h for CARMA, 81 h for RLTF-SD, 100 h for RRD-WU, and 104 h for the tested Pairwise configuration. The 72 h CARMA run uses approximately 1,152 accelerator-hours. All methods use batching and parallel optimizations within stage dependencies.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.