acceptodds
Under review as a conference paper at ICLR 2027

RAVEL: Reasoning Agents for Validating and Evaluating LLM Text Synthesis

Abstract

Large Language Models have evolved from single-round generators into long-horizon agents, capable of complex text synthesis scenarios. However, current evaluation frameworks lack the ability to assess the actual synthesis operations, such as outlining, drafting, and editing. Consequently, they fail to evaluate the actual and detailed capabilities of LLMs. To bridge this gap, we introduce RAVEL, an agentic framework that enables the LLM testers to autonomously plan and execute typical synthesis operations, including outlining, drafting, reviewing, and refining. Complementing this framework, we present C3EBench, a comprehensive benchmark comprising 1,258 samples derived from professional human writings. We utilize a "reverse-engineering" pipeline to isolate specific capabilities across four tasks: Cloze, Edit, Expand, and End-to-End. Through our analysis of 14 LLMs, we uncover that most LLMs struggle with tasks that demand contextual understanding under limited or under-specified instructions. By augmenting RAVEL with SOTA LLMs as operators, we find that such agentic text synthesis is dominated by the LLM's reasoning capability rather than raw generative capacity. Furthermore, we find that a strong reasoner can guide a weaker generator to yield higher-quality results, whereas the inverse does not hold.

Primary area:
datasets and benchmarks
Keywords:
creative writing, evaluation, long-form generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6jSiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RAVEL, an agentic evaluation framework for long-form LLM text synthesis. RAVEL lets a tested model select and execute outlining, drafting, reviewing, refining, and finishing actions, and evaluates both the resulting trajectory and final text quality. It also introduces C3EBENCH, a bilingual benchmark of 1,258 samples reverse-constructed from professional human writings, covering End2End, Expand, Cloze, and Edit tasks. The authors evaluate 14 LLMs on C3EBENCH, use End2End tasks for RAVEL trajectory experiments, and conduct controller/refiner substitutions, human validation of the LLM judge, and threshold and action-protocol ablations. Based on these experiments, the authors argue that success in this workflow depends more on reasoning for planning and critique than on raw drafting or refinement ability, and that a stronger reasoner can improve a weaker generator.

Critical strengths
  1. The paper presents a useful framework for diagnosing long-form writing workflows. RAVEL exposes action selection, completion, and revision behavior alongside external assessments of final text quality, while C3EBENCH covers four complementary writing scenarios in two languages. This can reveal differences between models that a final-output score alone would miss, giving the framework potential value as an evaluation resource.

  2. The paper also evaluates the reliability of its assessment procedure, rather than merely reporting model rankings. The authors collect outputs from nine models on 200 benchmark tasks and have 36 human annotators rate them, then compare the LLM judge with human judgments at both the model-aggregate level and the individual-text level. They further compare alternative judging methods and test additional controller/refiner substitutions, stopping thresholds, and action protocols. These analyses provide useful evidence about the reliability of the evaluation procedure and its sensitivity to workflow settings.

Critical weaknesses
  1. The component-replacement experiments do not establish the central claim that reasoning matters more than generation. Drafting and revision also require reasoning, while controller performance is affected by instruction following and compatibility with the workflow. Replacing a model therefore changes multiple capabilities; assigning a model to a particular role does not isolate the contribution of one capability. Appendix K includes additional base models and stronger replacement models, but these extensions still compare model replacements in different roles; they do not isolate the contribution of reasoning ability or directly test the effect of upgrading the initial drafter. Moreover, Appendix L.3 shows that simply replacing autonomous action selection with a fixed writing schedule raises Qwen3-Max's quality score from 5.92 to 7.08. The authors acknowledge that protocol choice affects results, but do not directly compare the fixed schedule and the upgraded controller on the same tasks, leaving the controller upgrade's additional benefit over a simple schedule unclear. The current results support the narrower observation that, under the tested RAVEL workflow, upgrading the controller often helps more than upgrading the local refiner.

  2. The reported quality advantages in the key component-replacement experiments lack direct human validation and sufficient robustness evidence. The paper provides a useful human-consistency evaluation of its LLM judge, supporting its use for comparing overall model performance. However, this evaluation mainly concerns standard outputs from different models on benchmark tasks and does not directly cover the comparison between controller-upgrade and local-refiner-upgrade configurations. The paper also does not report blinded human comparisons between these configurations or examine whether their score differences remain stable across tasks and evaluation settings. This limits the evidential strength of the claim that upgrading the controller produces higher-quality text, but does not invalidate the value of the existing validation for overall model comparisons.

  3. The definition, units, and reported values of refinement gain are not clearly reconciled. The paper bounds internal review scores between 1 and 10 and defines gain in Equation (10) as the average score difference before and after revision. Under this scoring range and formula, gain should lie between −9 and 9, yet values such as 27.96 and −12.16 are reported without an explanation of any conversion. This may reflect incomplete documentation of the formula or units, but it currently makes the magnitude of the gains difficult to interpret and the associated analyses difficult to verify. The authors should clarify the actual calculation and ensure consistency between the formula, units, and reported results.

  4. Score differences across tasks do not establish that models rely on explicit instructions. The paper interprets lower Cloze scores, relative to tasks such as Expand, as evidence that models struggle when instructions are underspecified and contextual understanding is required. However, these tasks differ in the information provided, generation requirements, and scoring rubrics. Cloze also requires the generated passage to fit both preceding and following context, and its construction specifically selects a text's most stylistically accomplished passage, spanning 20–50% of the original. Its difficulty may therefore reflect passage selection and task constraints as well as instruction specificity. The current comparisons do not distinguish these factors or establish that scores are directly comparable across tasks. Lower Cloze scores therefore do not establish that insufficiently explicit instructions or weak contextual understanding is the main cause of poorer performance.

Questions for authors
  1. The current experiments compare the effects of upgrading the controller and upgrading the local refiner. Why is this sufficient to support the claim that reasoning matters more than generation? How do the authors distinguish these two capabilities and rule out other factors that also change when a model is replaced, such as instruction following and workflow compatibility? If these capabilities cannot be distinguished, should the central conclusion be restricted to a comparison of controller and local-refiner upgrades within the tested workflow?

  2. For texts produced on the same tasks by controller-upgrade and local-refiner-upgrade configurations, do human evaluators who are unaware of the configuration identities also prefer the controller-upgrade outputs? Please evaluate this on a representative task subset and report the magnitude and stability of the quality advantage.

  3. Does refinement gain involve an unreported percentage conversion or other transformation? Please provide the formula and units that correspond to the reported values, together with a worked example. Would any clarification or correction require revising the interpretation of the reported gains?

  4. Could the authors vary only instruction specificity on the same Cloze samples, keeping the context, missing-span positions, and scoring criteria unchanged, to test whether more explicit instructions improve performance? Without such a comparison, should the conclusion be limited to performance differences under the respective task settings?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 6jSi (1/2)

We thank the reviewer for recognizing RAVEL's diagnostic value and our evaluation-validation efforts.


W1 / Q1 Capability attribution: Model replacements change multiple capabilities. How can controller-versus-refiner comparisons support the claim that reasoning matters more than generation?

Thank you for raising this distinction.

  • Separation by RAVEL design. RAVEL's controller πθ\pi_\theta performs explicit ReAct-style reasoning and action selection. All four primitives use no-reason inference settings: outlining/reviewing serve planning and critique, while drafting/refining generate and revise text.
  • Role isolation with substitution. Model replacements follow these functional boundaries, comparing upgrades to control/planning (reasoning-oriented) with upgrades to the refiner (generation-oriented). This modularity enables diagnosis of where a stronger model helps the writing workflow.
  • Conclusion scope. Control/planning upgrades often help more than refiner upgrades in the tested RAVEL settings. This is a functional comparison of controller effectiveness, which includes instruction following and workflow compatibility.

Revision in the updated PDF: The role design and no-reason tool settings are clarified in Sec. 3 (p. 3); the scope of the findings is clarified in Sec. 5.4 (p. 8) and Appendix K (pp. 43–44).

W1 / Q1 Fixed-workflow baseline: A fixed schedule already improves Qwen3-Max. Does an upgraded controller offer further gains on the same tasks?

A fixed workflow is itself a human-designed control policy. The autonomous-versus-fixed comparison in Appendix L.3 reports the following quality scores:

ModelAutonomousFixed workflowChange
Qwen3-Max5.927.08+1.16
Qwen3-32B5.505.92+0.42
GPT-5.27.407.400.00

Qwen3-Max's larger gain, alongside its 24% autonomous completion rate, is consistent with a planning bottleneck that human scheduling priors can alleviate. The smaller or absent gains for the other models show that this benefit varies by model. A fixed workflow can therefore serve as a controller upgrade through human planning priors. Together with the model-substitution results, this supports the finding that strengthening control/planning can improve quality on the same tasks, whether through a stronger model or human-designed scheduling. The gains concern improvements over the original control policy.

Revision in the updated PDF: The protocol analysis is discussed in Sec. 5.4 (p. 8) and Appendix L.3 (Table 15, pp. 45–46).


W2 / Q2: Please provide human comparisons of controller- and refiner-upgraded outputs and assess the stability of their quality differences.

Thank you for this suggestion. We added human evaluation of the role-substitution outputs during rebuttal. For each of eight Base-versus-upgrade comparisons, we randomly sampled 50 prompts and compared the two outputs for each prompt: 400 pairwise comparisons in total. Presentation order was randomized. Two annotators evaluated each pair; a third adjudicated disagreements.

The table reports upgraded configuration wins / ties / Base wins.

Base modelReplacement modelControl/planning upgrade vs. BaseRefiner upgrade vs. Base
Llama-3.1-70BGemini-3 Pro32 / 7 / 1117 / 11 / 22
GLM-4.7Gemini-3 Pro29 / 5 / 1615 / 21 / 14
Qwen3-32BGPT-5.233 / 4 / 1314 / 28 / 8
Qwen3-MaxClaude-4.539 / 5 / 620 / 16 / 14

Control/planning upgrades were preferred over Base on 58–78% of pairs across the four models, versus 28–40% for refiner upgrades. This provides human evidence for the control/planning upgrades' quality gains. The percentages describe comparisons against Base, not direct controller-versus-refiner preferences.

Revision in the updated PDF: The sampling protocol, annotation procedure, and all eight comparisons are added in Appendix K.1 (Table 12, p. 44). The evaluated configurations appear in Table 11 (formerly Table 10).

AuthorsOpenReview ↗

Response to Reviewer 6jSi (2/2)

W3 / Q3: Why does refinement gain exceed the stated score range? Please clarify the formula, units, and a calculation example.

Thank you for pointing this out. Refinement gain is the mean change in self-review scores. We originally multiplied it by 100 to avoid numerous 0.xxx entries in Table 2. In the updated PDF, we report unscaled score points, consistent with Eq. (10).

Across NN trajectories, let Ki\mathcal K_i index same-section before/after review-score pairs:

\left(s^{\mathrm{post}}_{ik}-s^{\mathrm{pre}}_{ik}\right), \qquad \delta_{\mathrm{gain}}=\frac{1}{N}\sum_{i=1}^{N}g_i.$$ The original aggregation uses $g_i=0$ when no pair is available. With valid scores in $[1,10]$, gain lies in $[-9,9]$. **Example:** a change from 7.0 to 7.5 is reported as 0.5 points. The previously displayed 27.96 corresponds to 0.2796 score points. Our audit also identified a score-pair alignment issue. We will correct it, recompute affected gains, and update the corresponding table entries and interpretation where needed. **Revision in the updated PDF:** Table 2 and the discussion in Sec. 5.3 (p. 7), together with Figure 7 (Appendix K, p. 43), now report unscaled score points, consistent with the score-difference definition in Eq. (10), Sec. 5.2 (p. 6). --- > W4 / Q4: Can lower Cloze scores be attributed to underspecified instructions? Please test instruction specificity on the same samples. Thank you for this suggestion. **We tested explicit output-boundary instructions on all 150 English Cloze samples.** We changed only the final output instruction: | Version | Output instruction (verbatim) | |---|---| | Original | Please directly output the filled writing result without additional explanation or comments. | | Span-only | Output only the missing text that belongs at [fill in the blanks]. Do not output, repeat, or rewrite any of the provided text before or after the blank. Do not output the placeholder or the completed article. Do not add explanations, comments, or labels. | The rest of the prompt, blank positions, references, rubric, and within-model generation settings were unchanged. All models used direct generation through APIs. The same GPT-5.2 judge evaluated full outputs without an added extraction step. | Model | Original | Span-only | 95% CI lower | 95% CI upper | Copying (before) | Copying (after) | |---|---:|---:|---:|---:|---:|---:| | Claude Sonnet 4.6 | 4.267 | 4.287 | −0.140 | +0.180 | 90.0% | 1.3% | | GLM-5.2 | 4.267 | 4.353 | −0.107 | +0.280 | 84.0% | 0% | | Grok-4.5 | 4.353 | 4.320 | −0.260 | +0.200 | 68.0% | 0% | Each row contains 150 complete pairs. The 95% CIs refer to paired score changes (span-only − original), using 10,000 bootstrap resamples over prompts. Copying means an exact contiguous 20-word match with supplied context; before/after denote the original/span-only instructions. **Output boundaries substantially reduced copying, but yielded no consistent quality gain.** This tests output-boundary specificity rather than all aspects of instruction specificity. The revised paper describes the Cloze gap as a performance difference under the respective task settings, without attributing it primarily to underspecified instructions or weak contextual understanding. By design, Cloze requires models to infer content and stylistic requirements from context; our construction selects passages integral to the text's flow (Sec. 4.2, p. 5; Appendix C.3, p. 22), so this challenge goes beyond compliance with explicit output instructions. **Revision in the updated PDF:** The controlled experiment and results are added in Appendix N.3 (Table 17, p. 48). The cross-task interpretation is revised in Sec. 5.3 (p. 7) and Appendices N.2–N.3 (pp. 47–48).
Reviewer_6jSiOpenReview ↗

Response

The revision addresses my concern that the central conclusion was overstated. The abstract, Section 5.4, and the conclusion narrow the claim that reasoning matters more than generation to the finding that agentic text synthesis benefits from stronger control and planning in the tested workflows. The authors also no longer attribute the lower Cloze scores primarily to underspecified instructions, explaining them instead in terms of each task's constraints and rubrics.

Two points would benefit from further discussion. First, the evaluation results are fairly sensitive to workflow design. Appendix L.3 shows that replacing autonomous action selection with a fixed writing schedule raises Qwen3-Max's quality score by 1.16 points and its completion rate by 76 percentage points. Further analysis of how workflow design affects the results would help readers distinguish a model's agentic writing ability from its fit with a particular workflow. Second, the additional benefit of an upgraded controller over a fixed schedule remains unclear. The comparison in Appendix K.2 draws on separate batches whose settings are not fully aligned, and the new human evaluation compares each upgrade with the base configuration rather than comparing the two upgrades directly.

I consider bringing the writing process into evaluation a very valuable direction, and finer-grained evaluation of complex tasks is one of the key open problems facing the research community. RAVEL exposes planning, execution, and revision behavior that final-text scores alone cannot capture, and the revised conclusions are broadly consistent with the evidence. I therefore maintain my rating of 3 (weak acceptance).

AuthorsOpenReview ↗

Thank you for the review

We sincerely thank the reviewer for the constructive feedback and for recognizing RAVEL’s value. Your comments have strengthened this work. Regardless of the final decision, we shall continue to examine workflow effects and improve the robustness of our evaluation.

Reviewer_2LxWOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces RAVEL, an evaluation framework for agentic writing that decomposes the writing process into outlining, drafting, reviewing, refining, and termination, assessing both final text quality and execution trajectories. The accompanying benchmark, C3EBENCH, contains 1,258 Chinese and English examples spanning four tasks: Cloze, Expand, Edit, and End2End. It supports standalone evaluation of direct generation, while RAVEL’s process-level evaluation is conducted on the End2End task. Through comparisons across 14 models, policy and refiner substitutions, meta-evaluation against human judgments, and protocol ablations, the authors reveal differences between static writing capabilities and performance in autonomous workflows.

Critical strengths
  1. The paper addresses an important gap in process-level evaluation of agentic writing. RAVEL evaluates final text quality alongside planning, reviewing, and revision behavior, allowing assessment of how models carry out writing tasks beyond their final output scores. This perspective helps identify scheduling and execution bottlenecks that static leaderboards cannot capture and is the primary reason for my support for acceptance.
  2. The benchmark and validation are relatively comprehensive. C3EBENCH covers two languages and four writing tasks, incorporating human-written references and manual verification. Comparisons across 14 models, meta-evaluation against human judgments, and the threshold and protocol ablations in the appendix provide substantial empirical evidence for understanding the framework’s applicability and limitations.
Critical weaknesses
  1. The decline in quality under the workflow deserves more discussion. In Table 2, 13 of the 14 models have lower RAVEL Judge scores than their direct-generation E2E scores. An evaluation framework need not improve generation quality, so this observation does not invalidate RAVEL’s contribution. However, the authors should clarify the comparability of the two columns and discuss the implications for interpreting the results. In particular, the substantial improvements from fixed scheduling for some models in Appendix L.3 suggest that the reported results may reflect both model capabilities and compatibility with the particular workflow.
  2. Task coverage within the agentic workflow remains incomplete. Although C3EBENCH includes four tasks, RAVEL reports workflow results only for End2End, leaving unclear how autonomous planning, reviewing, and refinement affect Cloze, Expand, and Edit individually. I suggest reporting quality scores for these three tasks under appropriately adapted agentic workflows, comparing them with direct generation using the same examples and scoring protocol within each task, and explaining how the initial states and available actions are adapted. This would strengthen the connection between the four-task benchmark and the process-level evaluation framework and help establish whether workflow benefits are task-dependent.
Questions for authors
  1. Could you explain why most models in Table 2 obtain lower Judge scores under RAVEL than E2E scores with direct generation, and clarify whether the two settings use the same evaluation protocol?
  2. Is the restriction of RAVEL evaluation to End2End due to the framework’s applicability or the scope of the experiments? Could you also report workflow evaluation results for Cloze, Expand, and Edit?
  3. Could you clarify whether refinement density counts all reviews or only repeat reviews after the initial review, and whether refinement gain is converted to a percentage or otherwise rescaled, so that the relationship between the formulas and the values and units in Table 2 is clear?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 2LxW (1/2)

We thank the reviewer for recognizing the importance of process-level evaluation and the breadth of our benchmark and validation.

W1 / Q1: Why do most models score lower under RAVEL than direct generation? Are the two columns comparable, and how much do the results depend on the workflow?

  • Score Comparability. The Judge and direct-E2E columns evaluate final-text quality using the same judge model/settings, task instructions, references, rubric, and scoring scale. W2/Q2 additionally matches examples and judging protocol.

  • Effects from RAVEL. All three models have lower means and higher SDs on Expand (W2/Q2). End2End results (mean ± SD across examples) are:

    Modelw/o RAVELw/ RAVELMean change
    DeepSeek-V4-Flash7.17 ± 1.177.23 ± 0.90+0.06
    GLM-5.27.15 ± 0.636.65 ± 0.96−0.50
    Qwen3.6-27B7.07 ± 0.945.80 ± 1.93−1.27

    RAVEL's main contribution is making the entire text-synthesis process analyzable, as detailed in W2/Q2.

  • Why scores were lower. Autonomous synthesis additionally requires reliable planning/execution and self-revision. Nine Qwen End2End traces never construct an outline (Judge 1.00 versus 7.22 for Direct on the same inputs). Under fixed scheduling, Qwen3-Max scores 7.08 with refinement and 7.20 without it (Appendix L.3). RAVEL exposes these workflow-dependent limitations and their consequences for final synthesis.

Revision in the updated PDF. Sections 5.3–5.4 clarify the comparison scope and workflow effects, with protocol ablations in Appendix L.3 (Table 15); Section 7 limits the conclusions to the tested workflows.


W2 / Q2: Is the End2End restriction a framework limitation or an experimental choice? Can RAVEL also evaluate Cloze, Expand, and Edit?

The initial implementation started from an empty state, limiting the reported experiments to End2End. We now support initialization from existing task inputs while preserving RAVEL's four tool interfaces:

  • Cloze: initialize the missing span with fixed surrounding context.
  • Expand: initialize the supplied outline.
  • Edit: initialize the supplied draft and critique.

Table 1. C3EBench performance with and without RAVEL.

Within each model and task, both settings use the same examples, GPT-5.2 judge, references, rubrics, and 1–10 scoring scale. Scores are mean ± SD across examples.

ModelTaskw/o RAVELw/ RAVELΔ\Delta
DeepSeek-V4-FlashCloze4.34 ± 1.835.69 ± 1.53+1.35
DeepSeek-V4-FlashExpand7.83 ± 0.747.47 ± 1.08−0.36
DeepSeek-V4-FlashEdit6.89 ± 1.066.84 ± 0.90−0.05
DeepSeek-V4-FlashEnd2End7.17 ± 1.177.23 ± 0.90+0.06
GLM-5.2Cloze4.08 ± 1.725.31 ± 1.43+1.23
GLM-5.2Expand7.80 ± 0.627.25 ± 1.19−0.56
GLM-5.2Edit6.95 ± 0.706.67 ± 0.69−0.28
GLM-5.2End2End7.15 ± 0.636.65 ± 0.96−0.50
Qwen3.6-27BCloze4.31 ± 1.634.59 ± 2.06+0.29
Qwen3.6-27BExpand7.70 ± 0.777.30 ± 1.22−0.40
Qwen3.6-27BEdit7.06 ± 0.676.94 ± 0.68−0.11
Qwen3.6-27BEnd2End7.07 ± 0.945.80 ± 1.93−1.27

Table 2. RAVEL's agentic behavior across tasks.

Both tables use the same evaluated examples. δobs\delta_{\mathrm{obs}} reports observed self-review gains in score points.

ModelTaskS\mathcal{S} (%) ↑ηtraj\eta_{\mathrm{traj}} ↓ρref\rho_{\mathrm{ref}} (%)δobs\delta_{\mathrm{obs}}ERRπ\mathrm{ERR}_{\pi} (%) ↓∣O∣\lvert\mathcal{O}\rvert∣T∣\lvert\mathcal{T}\rvert
DeepSeek-V4-FlashCloze100.003.369.971.57061.531.003.36
DeepSeek-V4-FlashExpand99.322.8213.452.33620.695.2612.97
DeepSeek-V4-FlashEdit100.003.23108.411.10000.401.003.23
DeepSeek-V4-FlashEnd2End100.002.493.851.56230.347.9019.46
GLM-5.2Cloze100.003.894.081.925010.261.003.89
GLM-5.2Expand88.814.267.873.476026.815.6121.67
GLM-5.2Edit98.706.30101.95—24.771.006.30
GLM-5.2End2End88.184.000.722.250028.946.3524.73
Qwen3.6-27BCloze62.9421.821.471.150047.111.0021.82
Qwen3.6-27BExpand97.793.036.242.19569.185.7215.42
Qwen3.6-27BEdit100.003.23100.97—5.081.003.23
Qwen3.6-27BEnd2End91.303.41†5.50†1.514323.576.0120.86

† Outline-normalized metrics exclude Qwen's nine empty-outline End2End trajectories.

AuthorsOpenReview ↗

Response to Reviewer 2LxW (2/2)

The two tables reveal three findings:

  • Edit–tool alignment. Edit's feedback-driven design maps naturally onto RAVEL's revision–review process: completion reaches 98.70–100%, with slightly more than one revision attempt per supplied draft (ρref=100.97\rho_{\mathrm{ref}}=100.97–108.41108.41\\%). DeepSeek additionally shows positive self-review gains in its eight trajectories with observable pre/post revision scores.
  • Task-dependent execution structure. Expand uses fewer outline nodes (5.26–5.72 vs. 6.01–7.90) and decisions (12.97–21.67 vs. 19.46–24.73) than End2End across all three models. Cloze/Edit start from one editable unit, further separating structural workload from execution overhead.
  • Model-dependent behavioral patterns. Qwen's policy-error rates are 47.11%, 9.18%, and 23.57% on Cloze, Expand, and End2End; GLM shows the reverse task pattern (10.26%, 26.81%, and 28.94%). DeepSeek maintains low error rates across all four tasks (0.34–1.53%).

RAVEL's trace-level evaluation makes these task- and model-specific execution patterns observable.

Revision in the updated PDF. Four-task mean quality scores and trace metrics are reported in Tables 3–4 (p. 8), with analysis in Section 5.3 and initialization/evaluation details in Appendix N.1.


Q3: Does refinement density count all reviews or only repeated reviews? Is refinement gain rescaled, and how do the formulas correspond to Table 2?

Thank you for pointing this out. Refinement density counts all revise actions per outline node, not review actions. For trajectory T\mathcal{T} and outline O\mathcal{O}:

=\frac{\sum_{t=1}^{\lvert\mathcal{T}\rvert}\mathbf{1}[a_t=\mathrm{revise}]}{\lvert\mathcal{O}\rvert}, \qquad \rho_{\mathrm{display}}=100\rho_{\mathrm{ref}} \\%.$$ For example, 15 revise actions over 10 outline nodes give $150\\%$, regardless of whether the trajectory contains 40 total actions. The updated PDF corrects the indicator in Eq. (9) from review to revise. **Refinement gain is the mean change in self-review scores.** We originally multiplied it by 100 for readability in Table 2. The updated PDF reports unscaled score points, matching the formula directly. Let $\mathcal{K}_i$ index same-section before/after review-score pairs in run $i$: $$g_i=\frac{1}{\lvert\mathcal{K}_i\rvert}\sum_{k\in\mathcal{K}_i} \left(s^{\mathrm{post}}_{ik}-s^{\mathrm{pre}}_{ik}\right), \qquad \delta_{\mathrm{gain}}=\frac{1}{N}\sum_{i=1}^{N}g_i.$$ The original aggregation used $g_i=0$ when no pair was available. A score change from 7.0 to 7.5 is reported as **0.5 points**, with no percentage normalization. The supplementary $\delta_{\mathrm{obs}}$ instead averages only trajectories with observed pre/post review cycles. It measures internal self-review changes, not external Judge improvement. Our audit also identified a score-pair alignment issue in the original aggregation; we will correct it and recompute affected gains. **Revision in the updated PDF.** Section 5.2, Eqs. (9)–(10), clarifies revision counting and gain units; Tables 2 and 4 use raw score points. Appendix N.1 specifies observed-cycle aggregation and missing-score handling.

Thank you for the detailed response and additional experiments. I appreciate the value of RAVEL as an agentic writing evaluation framework, especially its ability to reveal planning, execution, and revision behaviors that are hard to observe from final-text scores alone.

However, the additional results doesn't show general quality improvement from the workflow, and performance still drops noticeably on some tasks. This does not directly undermine the framework’s value, but it leaves me with some concerns about the robustness of its evaluation results: it remains unclear to what extent these results reflect the model’s agentic writing ability versus the specific workflow design and its fit. Overall, I still consider this a valuable work for agentic writing.

I therefore keep my score.

We appreciate the reviewer for recognizing RAVEL‘s value in revealing agentic writing behaviors beyond final-text scores.

We take seriously this concern about distinguishing model capabilities from workflow-specific effects. Regardless of the final decision, we will strengthen the robustness analyses in future updates, particularly by examining how workflow / task fitness affect the evaluation results.

Reviewer_ifq4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduce RAVEL, an agentic framework that enables LLM testers to plan and execute synthesis operations like outlining, drafting, reviewing and refining. This paper also introduces C3Bench, a benchmark that consists of 1,258 samples from professional human writing by reverse engineering to isolate capabilities across 4 tasks: cloze, edit, expand and end-to-end. In their experiments, the authors found that most LLMs struggle with tasks that needs contextual understanding under limited or underspecified instructions. The paper reports low Cloze scores, model-dependent execution strategies, and improvements from replacing a backbone's action-selection policy with a stronger model. The broader interpretation that reasoning generally matters more than generation is not too well-established than these particular observations.

Critical strengths
  • C3Bench covers both whole-document generation and more localized writing operations. The source descriptions, construction prompts, annotation instructions, and examples make the intended tasks understandable
  • This paper uses human annotators. The study uses 200 instructions, nine outputs per instruction, and 36 annotators. It reports 88-92% pairwise consistency and provides both system-level and text-level correlations.
Critical weaknesses
  • ection 3.3 defines self-review scores on a 1-10 scale, but in equation (10) defines refinement delta as an average score difference, which should therefore lie between -9 and 9 if both scores use that scale. I might have missed something, but the Table 2 reports 27.96 and 19.55 for Gemini-3 Pro and Sonnet 4.5, is that a percentage number…?
  • The policy/refiner substitution experiments do not cleanly generalize to the dominance of reasoning over generation. The controller affects decisions throughout the episode, whereas the refiner operates only on existing drafts when invoked. Together with the substantial effects of fixed scheduling, these results leave protocol compatibility and resource allocation as alternative explanations. The evidence supports a configuration-specific finding, rather than the general causal conclusion presented in the abstract and conclusion. Right now, “reasoning dominates generation in complex text synthesis” looks like an overclaim to me.
  • The judge validation provides useful support for model-level comparisons, but it does not establish that absolute scores are comparable across tasks. This matters for interpreting the Cloze gap. Please report uncertainty for the main score differences and clarify how evaluation rubrics are adapted to non-literary genres, since the displayed Edit rubric emphasizes literary qualities.
Questions for authors
  • The terms used in Section 3 might be mildly misleading, e.g., “optimization” or “policy”. It would be better to make things clear that RAVEL is a inference-time only pipeline.
  • What are the exact implemented definitions and units of refinement density and refinement delta? Does density count reviews, extra reviews, or refinements? Please define the paired-score set in Equation (10).
3 comments
AuthorsOpenReview ↗

Response to Reviewer ifq4 (1/1)

We thank the reviewer for recognizing C3EBench’s task coverage and our human validation study.

W1 / Q2: Refinement delta exceeds the stated score range; clarify its units, refinement density, and the paired scores in Eq. (10).

Thank you for pointing this out. We multiplied refinement gain by 100 only for readability in Table 2, but omitted the scaling label. We have removed this scaling in the updated PDF and now report gains in the original score units, consistent with Eq. (10).

In Eq. (9), T\mathcal{T} denotes the action trajectory, and n=∣O∣n=|\mathcal{O}| is the number of nodes in the planned outline O\mathcal{O}. Refinement density counts revise actions per outline node; the indicator now reads revise, matching the implementation. Table 2 reports this density as 100ρref100\rho_{\mathrm{ref}}.

In Eq. (10), K\mathcal{K} indexes same-section score pairs: the review score before a revision and the first review score afterward. Refinement delta averages these absolute score changes within each run, then across runs.

The separate score-pair alignment issue remains under audit.

Revision in the updated PDF: Section 5.2 corrects Eq. (9) and clarifies T\mathcal{T} and nn. Table 2 reports refinement gains in the original score units.


W2: The policy/refiner substitutions do not establish that reasoning generally dominates generation.

Thank you for this clarification. RAVEL separates explicit workflow reasoning from text execution. The controller πθ\pi_\theta follows an explicit reasoning–action (ReAct) process to choose actions and parameters, while the writing tool executes the selected text-generation operations. The controller and writing-tool calls therefore use no-reason mode during execution, and model substitutions target these distinct functional roles.

A fixed workflow encodes human planning priors in a controller. Qwen3-Max’s low autonomous completion rate (24%) suggests a coordination bottleneck. We further compared shared prompts across the existing fixed-workflow and Gemini-controller runs:

Base modelShared promptsFixed workflowGemini controller
Qwen3-Max507.087.22
Qwen3-32B495.926.59

The observed score difference is small for Qwen3-Max and larger for Qwen3-32B. These are preliminary comparisons across separate run batches, pending alignment of execution settings. We frame these findings as configuration-specific comparisons of control strategies.

Revision in the updated PDF: Sections 3 and 5.4 clarify the role separation and scope; the Abstract and Section 7 are aligned. Appendix K.2 adds the shared-prompt comparison and its limitations, complementing the protocol analysis in Appendix L.3.


W3: Model-level judge agreement does not establish cross-task score comparability. Please report uncertainty for the Cloze gap.

Thank you for this suggestion. The aggregate Cloze gap remains clear after accounting for sampling uncertainty. All four tasks use a 1–10 scale with the reference response anchored at 6. On the English C3EBench split:

TaskMeanStd. dev.95% CI lower95% CI upper
Cloze3.721.223.523.91
Expand6.840.796.697.00
Edit6.740.536.646.84
End2End6.560.656.436.69

For each instruction, we average scores across available LLMs. We then compute the mean, standard deviation, and bootstrap CI across instructions (20,000 resamples).

Cloze’s upper CI bound (3.91) is well below the lower bounds of all three other tasks (6.43–6.69). The gap is measured under task-specific rubrics and references, whose absolute scores are not fully calibrated across tasks.

Revision in the updated PDF: Appendix N.2 (Table 16) adds the means, standard deviations, and 95% CIs. Section 5.3 now interprets the Cloze gap under the respective task constraints and rubrics.


Q1: Do “optimization” and “policy” imply training, or is RAVEL an inference-time pipeline?

RAVEL operates entirely at inference time. “Policy” denotes LLM-based action selection, and “optimization” denotes iterative text revision. No model parameters are updated.

Revision in the updated PDF: Section 2.2 explicitly states that inference uses fixed model parameters and defines the policy as action selection.

Thank you for your response. I can see value in C3EBench’s task coverage and the human validation study. However, I am not quite convinced by the empirical findings. I will therefore maintain my score.

AuthorsOpenReview ↗

Thank you for your review

We thank the reviewer for the constructive feedback and for recognizing C3EBench’s task coverage and our human validation study. Your comments have helped us clarify the refinement metrics, align our conclusions with the tested configurations, and better explain the interpretation of cross-task scores.

Building on these revisions, we will continue to strengthen the empirical evidence through controlled comparisons of workflow design and controller upgrades, regardless of the final decision.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Rebuttal revision PDF uploaded & Response addressing weaknesses summarization

Dear Reviewers and Area Chair,

We apologize for the slight delay in completing our rebuttal. Some supplementary analyses took longer than expected due to API instability during evaluation and limited availability of human evaluators.

We thank the reviewers for their feedback. We have uploaded a revised PDF with the following updates, in manuscript order. Substantive additions and revisions are highlighted in blue.

  • Abstract and Sections 1–3: Made minor wording edits to clarify the scope of the findings, inference-time action selection, and task-specific initialization.
  • Sections 5.1–5.3, Tables 3–4: Added comparisons with and without RAVEL across all four C3EBench tasks for DeepSeek-V4-Flash, GLM-5.2, and Qwen3.6-27B, reporting both writing quality and agentic behavior. Clarified metric units and updated the cross-task analysis.
  • Sections 5.4 and 7: Refined several analysis and conclusion statements concerning control/planning versus local refinement, with conclusions scoped to the tested workflows.
  • Reproducibility Statement: Updated the description of reproducibility materials and the anonymous data/code repository.
  • Appendix K: Added 400 human pairwise comparisons of role upgrades against the base configurations. Included the original Gemini role-substitution figure, now explicitly referenced from Section 5.4.
  • Appendix M: Consolidated the full trajectory visualizations and static–agentic association analysis.
  • Appendix N: Added task-score dispersion and confidence intervals, and a controlled Cloze output-boundary experiment, alongside task initialization and metric aggregation details.
AuthorsOpenReview ↗

Response addressing weaknesses summarization

We summarize our responses to the main concerns, reviewers' feedback, and remaining limitations.

  • Claim/Analysis scope (ifq4/6jSi): We narrowed ''reasoning dominates generation'' to benefits of stronger control/planning within tested workflows, and withdrew the causal explanation of low Cloze scores. 6jSi explicitly acknowledged resolution (weak accept); ifq4 remained unconvinced (weak reject).
  • Coverage/comparability (2LxW): We extended agentic evaluation from End2End to Cloze, Expand and Edit. RAVEL/direct generation use identical examples, judges, references and rubrics. 2LxW welcomed the experiments but retained workflow-fit concerns (weak accept).
  • Human evidence (6jSi): We added 400 human comparisons supporting writing-quality gains from controller upgrades over the base. These compare each upgrade against the original configuration, rather than controller-upgraded versus refiner-upgraded outputs directly. 6jSi noted this remaining comparison limitation, but judged the revised conclusions broadly consistent with the evidence and maintained weak acceptance.
  • Metrics (all three): We corrected gain scaling and clarified revision counting. No reviewer explicitly confirmed resolution, while the clarification is intuitive.
AuthorsOpenReview ↗

Summarized strengths

Across the initial reviews and rebuttal discussion, reviewers recognized four strengths of our work:

  • Process/Trace-level evaluation (Reviewers 2LxW and 6jSi): Both valued RAVEL's ability to reveal planning, execution, and revision beyond final-text scores, and reaffirmed this contribution after rebuttal.
  • Benchmark coverage (Reviewers ifq4, 2LxW, and 6jSi): Initial reviews recognized C3EBench's complementary writing tasks and coverage of both local operations and whole-document generation. Reviewer ifq4 reaffirmed this value after rebuttal.
  • Human validation (Reviewers ifq4, 2LxW, and 6jSi): Reviewers valued the human annotation study and validation of the LLM judge. Reviewer ifq4 reiterated this strength in the final response.
  • Better-supported conclusions (Reviewer 6jSi): After revision, the reviewer explicitly acknowledged that the overstatement was addressed and judged the conclusions broadly consistent with the evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.