- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces RAVEL, an agentic evaluation framework for long-form LLM text synthesis. RAVEL lets a tested model select and execute outlining, drafting, reviewing, refining, and finishing actions, and evaluates both the resulting trajectory and final text quality. It also introduces C3EBENCH, a bilingual benchmark of 1,258 samples reverse-constructed from professional human writings, covering End2End, Expand, Cloze, and Edit tasks. The authors evaluate 14 LLMs on C3EBENCH, use End2End tasks for RAVEL trajectory experiments, and conduct controller/refiner substitutions, human validation of the LLM judge, and threshold and action-protocol ablations. Based on these experiments, the authors argue that success in this workflow depends more on reasoning for planning and critique than on raw drafting or refinement ability, and that a stronger reasoner can improve a weaker generator.
Critical strengths
-
The paper presents a useful framework for diagnosing long-form writing workflows. RAVEL exposes action selection, completion, and revision behavior alongside external assessments of final text quality, while C3EBENCH covers four complementary writing scenarios in two languages. This can reveal differences between models that a final-output score alone would miss, giving the framework potential value as an evaluation resource.
-
The paper also evaluates the reliability of its assessment procedure, rather than merely reporting model rankings. The authors collect outputs from nine models on 200 benchmark tasks and have 36 human annotators rate them, then compare the LLM judge with human judgments at both the model-aggregate level and the individual-text level. They further compare alternative judging methods and test additional controller/refiner substitutions, stopping thresholds, and action protocols. These analyses provide useful evidence about the reliability of the evaluation procedure and its sensitivity to workflow settings.
Critical weaknesses
-
The component-replacement experiments do not establish the central claim that reasoning matters more than generation. Drafting and revision also require reasoning, while controller performance is affected by instruction following and compatibility with the workflow. Replacing a model therefore changes multiple capabilities; assigning a model to a particular role does not isolate the contribution of one capability. Appendix K includes additional base models and stronger replacement models, but these extensions still compare model replacements in different roles; they do not isolate the contribution of reasoning ability or directly test the effect of upgrading the initial drafter. Moreover, Appendix L.3 shows that simply replacing autonomous action selection with a fixed writing schedule raises Qwen3-Max's quality score from 5.92 to 7.08. The authors acknowledge that protocol choice affects results, but do not directly compare the fixed schedule and the upgraded controller on the same tasks, leaving the controller upgrade's additional benefit over a simple schedule unclear. The current results support the narrower observation that, under the tested RAVEL workflow, upgrading the controller often helps more than upgrading the local refiner.
-
The reported quality advantages in the key component-replacement experiments lack direct human validation and sufficient robustness evidence. The paper provides a useful human-consistency evaluation of its LLM judge, supporting its use for comparing overall model performance. However, this evaluation mainly concerns standard outputs from different models on benchmark tasks and does not directly cover the comparison between controller-upgrade and local-refiner-upgrade configurations. The paper also does not report blinded human comparisons between these configurations or examine whether their score differences remain stable across tasks and evaluation settings. This limits the evidential strength of the claim that upgrading the controller produces higher-quality text, but does not invalidate the value of the existing validation for overall model comparisons.
-
The definition, units, and reported values of refinement gain are not clearly reconciled. The paper bounds internal review scores between 1 and 10 and defines gain in Equation (10) as the average score difference before and after revision. Under this scoring range and formula, gain should lie between −9 and 9, yet values such as 27.96 and −12.16 are reported without an explanation of any conversion. This may reflect incomplete documentation of the formula or units, but it currently makes the magnitude of the gains difficult to interpret and the associated analyses difficult to verify. The authors should clarify the actual calculation and ensure consistency between the formula, units, and reported results.
-
Score differences across tasks do not establish that models rely on explicit instructions. The paper interprets lower Cloze scores, relative to tasks such as Expand, as evidence that models struggle when instructions are underspecified and contextual understanding is required. However, these tasks differ in the information provided, generation requirements, and scoring rubrics. Cloze also requires the generated passage to fit both preceding and following context, and its construction specifically selects a text's most stylistically accomplished passage, spanning 20–50% of the original. Its difficulty may therefore reflect passage selection and task constraints as well as instruction specificity. The current comparisons do not distinguish these factors or establish that scores are directly comparable across tasks. Lower Cloze scores therefore do not establish that insufficiently explicit instructions or weak contextual understanding is the main cause of poorer performance.
Questions for authors
-
The current experiments compare the effects of upgrading the controller and upgrading the local refiner. Why is this sufficient to support the claim that reasoning matters more than generation? How do the authors distinguish these two capabilities and rule out other factors that also change when a model is replaced, such as instruction following and workflow compatibility? If these capabilities cannot be distinguished, should the central conclusion be restricted to a comparison of controller and local-refiner upgrades within the tested workflow?
-
For texts produced on the same tasks by controller-upgrade and local-refiner-upgrade configurations, do human evaluators who are unaware of the configuration identities also prefer the controller-upgrade outputs? Please evaluate this on a representative task subset and report the magnitude and stability of the quality advantage.
-
Does refinement gain involve an unreported percentage conversion or other transformation? Please provide the formula and units that correspond to the reported values, together with a worked example. Would any clarification or correction require revising the interpretation of the reported gains?
-
Could the authors vary only instruction specificity on the same Cloze samples, keeping the context, missing-span positions, and scoring criteria unchanged, to test whether more explicit instructions improve performance? Without such a comparison, should the conclusion be limited to performance differences under the respective task settings?
4 comments
Response to Reviewer 6jSi (2/2)
W3 / Q3: Why does refinement gain exceed the stated score range? Please clarify the formula, units, and a calculation example.
Thank you for pointing this out. Refinement gain is the mean change in self-review scores. We originally multiplied it by 100 to avoid numerous 0.xxx entries in Table 2. In the updated PDF, we report unscaled score points, consistent with Eq. (10).
Across trajectories, let index same-section before/after review-score pairs:
\left(s^{\mathrm{post}}_{ik}-s^{\mathrm{pre}}_{ik}\right), \qquad \delta_{\mathrm{gain}}=\frac{1}{N}\sum_{i=1}^{N}g_i.$$ The original aggregation uses $g_i=0$ when no pair is available. With valid scores in $[1,10]$, gain lies in $[-9,9]$. **Example:** a change from 7.0 to 7.5 is reported as 0.5 points. The previously displayed 27.96 corresponds to 0.2796 score points. Our audit also identified a score-pair alignment issue. We will correct it, recompute affected gains, and update the corresponding table entries and interpretation where needed. **Revision in the updated PDF:** Table 2 and the discussion in Sec. 5.3 (p. 7), together with Figure 7 (Appendix K, p. 43), now report unscaled score points, consistent with the score-difference definition in Eq. (10), Sec. 5.2 (p. 6). --- > W4 / Q4: Can lower Cloze scores be attributed to underspecified instructions? Please test instruction specificity on the same samples. Thank you for this suggestion. **We tested explicit output-boundary instructions on all 150 English Cloze samples.** We changed only the final output instruction: | Version | Output instruction (verbatim) | |---|---| | Original | Please directly output the filled writing result without additional explanation or comments. | | Span-only | Output only the missing text that belongs at [fill in the blanks]. Do not output, repeat, or rewrite any of the provided text before or after the blank. Do not output the placeholder or the completed article. Do not add explanations, comments, or labels. | The rest of the prompt, blank positions, references, rubric, and within-model generation settings were unchanged. All models used direct generation through APIs. The same GPT-5.2 judge evaluated full outputs without an added extraction step. | Model | Original | Span-only | 95% CI lower | 95% CI upper | Copying (before) | Copying (after) | |---|---:|---:|---:|---:|---:|---:| | Claude Sonnet 4.6 | 4.267 | 4.287 | −0.140 | +0.180 | 90.0% | 1.3% | | GLM-5.2 | 4.267 | 4.353 | −0.107 | +0.280 | 84.0% | 0% | | Grok-4.5 | 4.353 | 4.320 | −0.260 | +0.200 | 68.0% | 0% | Each row contains 150 complete pairs. The 95% CIs refer to paired score changes (span-only − original), using 10,000 bootstrap resamples over prompts. Copying means an exact contiguous 20-word match with supplied context; before/after denote the original/span-only instructions. **Output boundaries substantially reduced copying, but yielded no consistent quality gain.** This tests output-boundary specificity rather than all aspects of instruction specificity. The revised paper describes the Cloze gap as a performance difference under the respective task settings, without attributing it primarily to underspecified instructions or weak contextual understanding. By design, Cloze requires models to infer content and stylistic requirements from context; our construction selects passages integral to the text's flow (Sec. 4.2, p. 5; Appendix C.3, p. 22), so this challenge goes beyond compliance with explicit output instructions. **Revision in the updated PDF:** The controlled experiment and results are added in Appendix N.3 (Table 17, p. 48). The cross-task interpretation is revised in Sec. 5.3 (p. 7) and Appendices N.2–N.3 (pp. 47–48).Response
The revision addresses my concern that the central conclusion was overstated. The abstract, Section 5.4, and the conclusion narrow the claim that reasoning matters more than generation to the finding that agentic text synthesis benefits from stronger control and planning in the tested workflows. The authors also no longer attribute the lower Cloze scores primarily to underspecified instructions, explaining them instead in terms of each task's constraints and rubrics.
Two points would benefit from further discussion. First, the evaluation results are fairly sensitive to workflow design. Appendix L.3 shows that replacing autonomous action selection with a fixed writing schedule raises Qwen3-Max's quality score by 1.16 points and its completion rate by 76 percentage points. Further analysis of how workflow design affects the results would help readers distinguish a model's agentic writing ability from its fit with a particular workflow. Second, the additional benefit of an upgraded controller over a fixed schedule remains unclear. The comparison in Appendix K.2 draws on separate batches whose settings are not fully aligned, and the new human evaluation compares each upgrade with the base configuration rather than comparing the two upgrades directly.
I consider bringing the writing process into evaluation a very valuable direction, and finer-grained evaluation of complex tasks is one of the key open problems facing the research community. RAVEL exposes planning, execution, and revision behavior that final-text scores alone cannot capture, and the revised conclusions are broadly consistent with the evidence. I therefore maintain my rating of 3 (weak acceptance).
Thank you for the review
We sincerely thank the reviewer for the constructive feedback and for recognizing RAVEL’s value. Your comments have strengthened this work. Regardless of the final decision, we shall continue to examine workflow effects and improve the robustness of our evaluation.
Response to Reviewer 6jSi (1/2)
We thank the reviewer for recognizing RAVEL's diagnostic value and our evaluation-validation efforts.
Thank you for raising this distinction.
Revision in the updated PDF: The role design and no-reason tool settings are clarified in Sec. 3 (p. 3); the scope of the findings is clarified in Sec. 5.4 (p. 8) and Appendix K (pp. 43–44).
A fixed workflow is itself a human-designed control policy. The autonomous-versus-fixed comparison in Appendix L.3 reports the following quality scores:
Qwen3-Max's larger gain, alongside its 24% autonomous completion rate, is consistent with a planning bottleneck that human scheduling priors can alleviate. The smaller or absent gains for the other models show that this benefit varies by model. A fixed workflow can therefore serve as a controller upgrade through human planning priors. Together with the model-substitution results, this supports the finding that strengthening control/planning can improve quality on the same tasks, whether through a stronger model or human-designed scheduling. The gains concern improvements over the original control policy.
Revision in the updated PDF: The protocol analysis is discussed in Sec. 5.4 (p. 8) and Appendix L.3 (Table 15, pp. 45–46).
Thank you for this suggestion. We added human evaluation of the role-substitution outputs during rebuttal. For each of eight Base-versus-upgrade comparisons, we randomly sampled 50 prompts and compared the two outputs for each prompt: 400 pairwise comparisons in total. Presentation order was randomized. Two annotators evaluated each pair; a third adjudicated disagreements.
The table reports upgraded configuration wins / ties / Base wins.
Control/planning upgrades were preferred over Base on 58–78% of pairs across the four models, versus 28–40% for refiner upgrades. This provides human evidence for the control/planning upgrades' quality gains. The percentages describe comparisons against Base, not direct controller-versus-refiner preferences.
Revision in the updated PDF: The sampling protocol, annotation procedure, and all eight comparisons are added in Appendix K.1 (Table 12, p. 44). The evaluated configurations appear in Table 11 (formerly Table 10).