acceptodds
Under review as a conference paper at ICLR 2027

OZONE: Object-Oriented Optimization of Agentic Workflow for Scientific Survey Generation

Abstract

We introduce OZONE, an object-oriented agentic workflow optimization framework for scientific survey generation. Survey writing comprises tasks that share common requirements but differ in their objectives, operator compositions, and evaluation criteria. OZONE represents the search space with predefined abstract classes and task-specific subclasses that inherit common contracts. The optimizer constructs and extends operator implementations under these contracts, assigns execution units to subclasses based on their operators and outputs, and connects the units into a directed acyclic graph. We design local rubrics for intermediate outputs and a global rubric for completed surveys, as improvements to individual tasks do not always improve overall survey quality. Local diagnosis guides changes to unit implementations and their connections, while the global score provides a basis for comparing candidate workflows. At inference time, OZONE supports runtime editing guided by a rubric proxy and refinement through review-based revision. We introduce CROFTBench, a benchmark of recent CS survey topics that extends evaluation to figures and tables. With GPT-4o-mini and inference refinement, OZONE achieves 72.2 on SurveyBench, 46.02 on Bloom-Eval, and 43.34 on CROFTBench.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Survey paper generation, Automatic Survey Generation, Agentic Workflow Optimization, Rubric-based Optimization, Long-Form Writing, LLM Agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_aeNPOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes OZONE, an object-oriented framework for optimizing agentic workflows for scientific survey generation. The key idea is to organize the workflow search space using predefined abstract classes and task-specific subclasses, allowing the optimizer to modify both operator implementations and their dependencies while preserving shared input/output contracts. The framework further introduces local rubrics to diagnose intermediate outputs and a global rubric to evaluate complete surveys, using both signals to guide iterative workflow optimization. At inference time, OZONE supports runtime editing and review-based refinement. The authors also introduce CROFTBench, a new benchmark for scientific survey generation that extends existing evaluation dimensions to include figures and tables. Experiments on SurveyBench, Bloom-Eval, and CROFTBench demonstrate that OZONE consistently outperforms several survey generation and workflow optimization baselines. Additional ablation studies, human evaluations, and cost analyses provide further evidence of the framework’s effectiveness.

Critical strengths
  1. The paper addresses an interesting challenge in agentic workflow optimization for long-form scientific writing, where improvements to individual components do not necessarily translate into better overall outputs. The proposed object-oriented representation provides a structured way to organize task-specific operators and their dependencies, while the combination of local and global rubrics offers more informative feedback for workflow optimization. I find this formulation useful and potentially applicable to other complex agentic tasks beyond survey generation.

  2. The paper evaluates OZONE on three benchmarks and demonstrates consistent improvements over existing survey generation systems and an AFlow-based optimization baseline. The inclusion of human evaluation, rubric ablation, and quality–cost analysis strengthens the empirical findings. In addition, CROFTBench extends existing survey generation evaluations to figures and tables, providing a useful benchmark contribution for future research on automated scientific writing.

Critical weaknesses
  1. While the proposed framework is well structured, its core optimization methodology shares similarities with existing agentic workflow optimization approaches such as AFlow and ADAS, which also optimize operator implementations and workflow structures based on evaluation feedback. The main distinction appears to be the predefined class hierarchy and subclass-specific rubrics. However, OZONE introduces several components simultaneously, including the object-oriented search space, local rubrics, experience memory, and round-level review. The current experiments do not sufficiently disentangle their individual contributions. In particular, a comparison with an otherwise identical workflow optimizer that uses the same local/global rubrics but without the object-oriented hierarchy would help establish whether the proposed representation provides benefits beyond structured feedback and search-space constraints.

  2. Although the paper demonstrates improvements in final survey quality, it provides limited insight into how the optimized workflows differ from their initial versions or existing manually designed workflows. The qualitative case studies mainly compare generated survey content rather than illustrating the workflow optimization process itself. I would be interested in seeing concrete examples of how local rubric feedback leads to specific operator or dependency modifications, and how these changes improve the final output. Additionally, the local rubric ablation is limited to a single first-round search run, making it difficult to assess the robustness and generality of the reported optimization gains. More detailed workflow evolution analyses and repeated experiments would strengthen the evidence for the proposed optimization mechanism.

Questions for authors
  1. I wonder how much of the performance improvement can be attributed specifically to the object-oriented search space, as opposed to the local/global rubric feedback and other optimization components? Would a workflow optimizer without the predefined class hierarchy, but equipped with the same rubrics and search budget, achieve comparable results? I think this distinction is important for understanding the core methodological contribution.

  2. Workflow evolution and interpretability. Could the authors provide a concrete example illustrating how OZONE evolves a workflow during optimization? In particular, it would be helpful to see the initial and final workflow structures, the local rubric feedback that triggered specific operator or dependency edits, and the resulting changes in local and global scores. This would help clarify what kinds of improvements the optimizer actually discovers beyond existing manually designed workflows.

  3. Robustness of local rubric optimization. The rubric ablation in Figure 4 is conducted on a single first-round search run, and both settings achieve positive global improvements in only 2 out of 9 edits. How stable are these findings across different random seeds or search budgets? Also, does the benefit of local rubrics persist across subsequent optimization rounds?

6 comments

We thank the reviewer for the constructive feedback. We address two main concerns: (1) the contributions of the class hierarchy, rubric feedback, and other optimization components and (2) how workflows evolve during search and whether the benefit of local rubrics persists across rounds.

Regarding the first concern, we examine the contributions of the class hierarchy and local rubrics (W1/Q1-1). We then assess the roles of edit memory, round-level review, and reinitialization through separate ablations (W1/Q1-2). We rename OZONE-Base to OZONE (w/o inference refinement) throughout the revised manuscript.

Regarding the second concern, we trace how the workflow evolves from its initial to its final form and show how local rubric feedback leads to specific edits and changes in local and global scores (W2/Q2). We also extend the local-rubric ablation to three rounds to examine whether its benefits persist beyond the first round (W2/Q3).

W1/Q1-1: Class Hierarchy and Local-Rubric Ablations

In response to your question about what the object-oriented class hierarchy contributes beyond rubric feedback, we compare the full hierarchy with three variants and separately examine the contribution of local rubrics. We evaluate the best workflow selected under each search configuration without inference refinement (Table 1).

As motivated in the original manuscript (lines 74–83 and §4.1), our design combines shared functional roles and I/O contracts at the abstract-class level with task-specific objectives and operators at the subclass level to guide workflow composition and editing. To examine the contribution of each level, we compare the full hierarchy with Abstract class only and Subclass only. We also evaluate Abstract-class targeting (same rubrics), which restricts edit targets to abstract classes while retaining the original local and global rubrics. This comparison tests whether the optimizer benefits from targeting subclasses when it receives the same feedback.

Table 1. Class hierarchy and local-rubric ablations on Bloom-Eval without inference refinement.

ConfigurationMem.Comp.Appl.Anal.Eval.Creat.Level Avg.
OZONE (w/o inference refinement)19.5255.3280.7670.627.3841.6745.88
Abstract class only16.5653.5380.3366.586.7943.3944.53
Subclass only17.9054.5080.5968.286.4443.0445.13
Abstract-class targeting (same rubrics)17.9855.6080.8368.695.0743.9745.36
Global rubric only17.5656.0580.4767.265.1743.5545.01

The full hierarchy achieved a higher Level Avg than both single-level variants (Abstract class only and Subclass only), supporting the design that combines shared contracts with task-specific targets for editing. With the original local and global rubrics held fixed, restricting edit targets to abstract classes reduced the Level Avg from 45.88 to 45.36. This suggests that the optimizer can make better use of the same feedback when it can act on a specific objective within a shared role (e.g., selecting Synthesis refinement within Refine in response to feedback on comparative synthesis). Removing local rubrics also lowered the Level Avg with the full hierarchy in place, suggesting that feedback on intermediate outputs helps the optimizer decide where to edit.

Evaluating a workflow optimizer without the predefined class hierarchy would require a different way to apply OZONE’s local rubrics to individual operators, since these rubrics are defined for intermediate outputs at the subclass level. For this reason, we used only the global rubric during optimization in AFlow-SurveyForge.

We will add Table 1 in the revised paper.

W1/Q1-2: Optimization Component Ablations

In response to your question about how the other optimization components contribute to OZONE’s performance, we ablate edit memory, round-level review, and reinitialization separately. Table 2 evaluates edit memory and round-level review with inference refinement enabled. Both components are used during optimization and inference, so this comparison captures the effect of removing each component across the full generation pipeline.

Table 2. Bloom-Eval Results w/ Inference Refinement

ConfigurationMem.Comp.Appl.Anal.Eval.Creat.Level Avg.
OZONE19.3456.3579.9472.006.4042.0946.02
w/o edit memory18.4557.2281.1868.675.1943.7945.75
w/o round-level review17.5054.4780.2766.064.1143.5944.33

Removing either component lowered the end-to-end Level Avg. Edit memory makes prior edit outcomes available when the optimizer proposes edits and when OZONE selects operator implementations at inference time. Round-level review provides feedback for reconstructing the workflow between rounds and revising the survey at inference time. The lower scores support retaining both components so that prior edit outcomes and review feedback can inform decisions at both stages.

To assess each component’s contribution during optimization, we disable inference refinement and evaluate the surveys generated by the workflow selected under each search configuration (Table 3).

Table 3. Bloom-Eval Results w/o Inference Refinement

ConfigurationMem.Comp.Appl.Anal.Eval.Creat.Level Avg.
OZONE (w/o inference refinement)19.5255.3280.7670.627.3841.6745.88
w/o edit memory17.8153.0280.7568.834.8344.4044.94
w/o round-level review17.7354.1480.7569.166.4842.4845.12
w/o reinitialization (continuous search)18.5153.3680.3868.997.1443.2445.27
Restart from scratch each round17.9553.4980.3166.474.4344.5844.54

Removing edit memory or round-level review lowered the Level Avg even with inference refinement disabled. This suggests that reusing prior edit outcomes and incorporating feedback on completed surveys help the optimizer find better workflows during search. OZONE (w/o inference refinement) achieved a higher Level Avg than both w/o reinitialization (continuous search) and Restart from scratch each round. This result supports reassembling the workflow between rounds using review feedback and components from the previous round’s best workflow.

We will add Table 2 and Table 3 in the revised paper.

W2/Q2: Workflow Evolution and Interpretability

In response to your request for a concrete example of workflow evolution, we trace changes in workflow structure, operator compositions, and local rubric feedback across three optimization rounds.

Table 4. Workflow evolution across three optimization rounds.

WorkflowKey structural and implementation changesGlobal Score (G)
Initial W_010 units: Retrieval → Outline writing → Analysis → Content writing → Refinement and verification64.4
Round 1 W*Revised the Revise prompt to better address flagged findings and citation support67.3
Round 2 W*Reorganized into 8 units; connected Literature analysis to Outline writing; added ComparativeSynthesis and Revise within Content writing67.8
Round 3 W*Moved Outline verification before Content writing; reduced the number of ComparativeSynthesis passes within Content writing from two to one68.1

As shown in Table 4, the initial workflow contained 10 execution units (G = 64.4), while the final selected workflow contained 8 units (G = 68.1). Across rounds, OZONE reorganized unit dependencies and revised operator compositions and prompts. The final workflow was newly assembled at the start of Round 3.

Table 5. Examples of local feedback, workflow edits, and score changes.

Workflow EditLocal Rubric FeedbackApplied EditΔLΔG
W_2_1 → W_2_10Weak citation support and limited use of retrieved evidenceAdded a second ComparativeSynthesis pass to Content writing+0.84+1.61
W_3_1 → W_3_8Few input paper IDs yielded usable recordsIncreased PoolPapers(num) from 1,500 to 2,000+0.56−3.75

Table 5 illustrates how local rubric feedback guides specific operator edits and how their effects differ at the local and global levels. Adding a second ComparativeSynthesis pass improved both scores, whereas increasing the PoolPapers limit improved the local score but reduced the global score. These examples show that local improvements do not necessarily translate into better document-level quality.

We will include the complete workflow structures, operator and dependency changes, and additional edit-level examples in the revised manuscript.

Reviewer_bHBiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper treats scientific survey generation as an agentic-workflow optimization problem, arguing that survey tasks share writing requirements but differ in objectives, so the search space should reflect that structure.

OZONE represents a workflow as a DAG of execution units. Five abstract classes (Retrieve, Analysis, Writing, Verify, Refine) fix each role's I/O contract while predefined subclasses specialize it. The optimizer edits unit implementations and dependencies, guided by subclass-level local rubrics on intermediate outputs and a global rubric on finished surveys, with edit outcomes kept in memory and an end-of-round review feeding the next round. At inference time a rubric proxy swaps implementations at runtime and a distilled checklist drives review-based revision.

It also introduces CROFTBench, 10 CS topics from 2025–2026 surveys, extending evaluation to taxonomy, table, and figure placement.

Validation uses GPT-4o-mini throughout (optimizer, executor, and rubric judge) against SurveyForge, SurveyGen, FlowSearch, and an AFlow-SurveyForge control sharing OZONE's operators and global rubric. Reported averages: 72.2 on SurveyBench, 46.02 on Bloom-Eval, 43.34 on CROFTBench, all best in table, plus a rubric ablation, a quality–cost Pareto analysis, and a human study with 6 experts over 40 topics.

Critical strengths

S1. The human study is done properly, and its disclosures are the reason I trust the direction of the results. 6 experts, all 40 test topics, 240 judgments per criterion, Wilcoxon on topic-level means, a per-annotator win/loss table, and leave-one-annotator-out robustness. More importantly the authors report what hurts them: Krippendorff's α = 0.03, agreement on preference direction only 56–58%, and a fixed label order they then probe on CROFTBench where OZONE is presented first. In a subfield where long-document quality claims usually rest on an LLM judge and no agreement statistic, this is the difference between a believable direction and an unfalsifiable one.

S2. The baseline and cost analyses are chosen to be informative rather than flattering. AFlow-SurveyForge is the right control — same operator set, same global rubric, initialized from SurveyForge — and it scores below its own initialization (64.5 vs. 65.6), which is a clean motivation for local rubric feedback rather than a rhetorical one. The authors then argue against their own headline configuration: inference refinement "more than doubles generation time with little change in quality," so OZONE-Base is the better trade-off, and they place it on the Pareto frontier at roughly half SurveyForge's tokens. This self-scrutiny is what keeps the paper credible despite the small margins.

Critical weaknesses

W1. The object-oriented hierarchy — the paper's named contribution — is never ablated. The only ablation varies the rubric (global vs. global+local), on 9 edits in a single round of a single run, with deltas the authors themselves decline to call significant. Nothing separates the class structure from the other parts introduced alongside it — local rubrics, edit memory, round reviews, empty-graph initialization. The paper thus shows that this system edges out prior pipelines, not that object-orientation is why, which is exactly the claim the title rests on.

W2. The margins are small, partly unreplicated, and partly contradicted by the paper's own tables. Bloom-Eval reports +1.07 over SurveyGen with one generation per topic and no variance; OZONE beats OZONE-Base by 0.1 on SurveyBench and loses to it under the SurveyForge-DB setting (45.50 vs. 45.90); and on the authors' own CROFTBench OZONE-Base (41.77) falls below AFlow-SurveyForge (42.84), contradicting §5.2's claim of beating all baselines on all three. GPT-4o-mini is also optimizer, executor, and both judges, so the search objective is scored by the model being optimized — and the one external check, the human study, is significant on Structure and Relevance only, while the paper's stated motivation is that surveys lag on knowledge integration.

Questions for authors
  1. Can you ablate the class hierarchy itself — same operators, local rubrics, and memory but flat/unstructured units — so the gain attributable to object-orientation is separable from local rubrics, edit memory, round reviews, and empty-graph initialization?
  2. Can you report Bloom-Eval and CROFTBench with multiple generations per topic and variance? With +1.07 over SurveyGen from one generation per topic, how much of the ordering is stable?
  3. §5.2 states the fixed OZONE workflow "outperforms all baselines in average score on all three benchmarks," but OZONE-Base scores 41.77 on CROFTBench against AFlow-SurveyForge's 42.84. Which claim is intended, and does the CROFTBench result change the conclusion about reusing the optimized workflow?
  4. GPT-4o-mini is optimizer, executor, and both local and global judge. Can you rerun the search with a judge outside the generator family, or report the correlation between your global rubric and the external SurveyBench/Bloom-Eval scores, to show the search is not fitting its own judge?
  5. The human study is significant on Structure and Relevance but not Hierarchy or Coherence, with Coverage marginal. Given the stated motivation that generated surveys lag on knowledge integration, what direct evidence is there that OZONE improves synthesis and coverage rather than organization?
  6. On CROFTBench, OZONE's advantage rests largely on table placement (56.0 vs. 52.0) while it loses figure placement (34.0 vs. 42.0), and these are scored by gpt-4o-mini. How are these two metrics defined and validated against human judgment?
6 comments

We thank the reviewer for the constructive feedback. We address two main concerns: (1) the individual contribution of the object-oriented class hierarchy and (2) the robustness of the reported results and evidence of improved knowledge integration.

Regarding the first concern, we disable inference refinement to evaluate the workflows produced by each optimization ablation. To assess the contributions of individual optimization components, we compare the full hierarchy with three variants and conduct separate ablations of the other search components (W1/Q1-1). We also extend the local-rubric ablation to three rounds with the edit-memory configuration held fixed (W1/Q1-2). Throughout the revised manuscript, we rename OZONE-Base to OZONE (w/o inference refinement) to make the distinction between the two configurations explicit.

Regarding the second concern, we report results from five generations per topic on Bloom-Eval and CROFTBench (W2/Q2). We clarify which OZONE configuration the claim in §5.2 refers to and how both configurations compare with AFlow-SurveyForge on CROFTBench (W2/Q3). We then examine how the evaluation results vary across judges and executors (W2/Q4) and present evidence on synthesis and coverage, two aspects of knowledge integration (W2/Q5). Additionally, we clarify the definition and human validation of figure and table placement metrics (Q6).

W1/Q1-1: Class Hierarchy and Component Ablations

In response to your request to distinguish the contribution of the object-oriented class hierarchy from other optimization components, we compare the full hierarchy with three variants and evaluate the remaining components through separate ablations. We evaluate the best workflow selected under each search configuration without inference refinement, so that the comparison reflects the quality of the workflows obtained through optimization (Table 1).

As motivated in the original manuscript (lines 74–83; §4.1), our design combines shared functional roles and I/O contracts at the abstract-class level with task-specific objectives and operators at the subclass level to guide workflow composition and editing. To examine the contribution of each level, we compare the full hierarchy with two single-level variants and an abstract-class targeting variant. The single-level variants use abstract-class or subclass-specific operators and rubrics, respectively, while Abstract-class targeting (same rubrics) retains the original global and local rubrics but restricts edit targets to abstract classes.

Table 1. Optimization component ablations on Bloom-Eval with inference refinement disabled.

ConfigurationMem.Comp.Appl.Anal.Eval.Creat.Level Avg.
OZONE(w/o inference refinement)19.5255.3280.7670.627.3841.6745.88
Abstract class only16.5653.5380.3366.586.7943.3944.53
Subclass only17.9054.5080.5968.286.4443.0445.13
Abstract-class targeting, same rubrics17.9855.6080.8368.695.0743.9745.36
Global rubric only17.5656.0580.4767.265.1743.5545.01
w/o memory17.8153.0280.7568.834.8344.4044.94
w/o review17.7354.1480.7569.166.4842.4845.12
w/o reinitialization18.5153.3680.3868.997.1443.2445.27
Restart from scratch each round17.9553.4980.3166.474.4344.5844.54

The full hierarchy achieved a higher Level Avg than either single-level variant, supporting the design that combines shared contracts with task-specific operators. With the original local and global rubrics held fixed, restricting edit targets to abstract classes reduced the Level Avg from 45.88 to 45.36. This supports our hypothesis that task-specific objectives enable more precise edit targeting within shared functional roles (e.g., selecting Synthesis refinement within Refine in response to feedback on comparative synthesis).

The remaining ablations also yielded lower Level Avg scores, indicating that other optimization components contribute alongside the class hierarchy, while the controlled comparison above highlights the benefit of subclass-level edit targeting under the same rubric feedback.

We will include Table 1 in the revised manuscript.

W1/Q1-2: Controlled Local-Rubric Ablation

In response to your concern that the original ablation covered only nine edits in one round, we extended the comparison to three rounds. Global only and Global+Local started from the same workflow and used the same edit-memory configuration. Table 2 reports the best global rubric score G in each round and the change in G from each parent workflow to its edited version ΔG.

Table 2. Round-wise best global scores and edit-level score changes with and without local rubrics.

RoundGlobal only Best GGlobal+Local Best GGlobal only ΔG (Mean ± SD)Global+Local ΔG (Mean ± SD)
166.5567.25−0.88 ± 4.39−0.81 ± 5.27
265.8167.75−0.82 ± 4.78−0.90 ± 4.90
366.3868.09−0.39 ± 4.20−2.60 ± 4.54

Global only peaked in Round 1. In Global+Local, the best G increased each round, and the mean change from each parent to its edited workflow ΔG became more negative across rounds, moving from -0.81 to -0.90 to -2.60. The workflow assembled at the start of Round 3 achieved the highest G of the run and served as the parent for seven of the nine edits. This concentration of edits around the highest-scoring workflow, with no further gains, suggests that the search was converging around that workflow in the final round.

We will update Figure 4 in the revised manuscript to incorporate the results from Table 2.

W2/Q2: Performance Across Repeated Generations

In response to your request for multiple generations per topic and variance, we generated five surveys per topic for each method on Bloom-Eval and CROFTBench and report the mean and standard deviation across five runs.

Table 3-1. Bloom-Eval

MethodMem.Comp.Appl.Anal.Eval.Creat.Level Avg.
SurveyForge17.94 ± 0.8050.12 ± 1.4972.07 ± 0.5967.82 ± 0.925.91 ± 1.5342.59 ± 0.4542.74 ± 0.35
SurveyGen19.12 ± 0.4259.72 ± 0.9867.48 ± 0.5076.24 ± 0.708.70 ± 1.0038.35 ± 0.6344.93 ± 0.17
FlowSearch17.71 ± 0.6454.13 ± 0.6257.00 ± 1.7662.81 ± 0.377.21 ± 0.8940.67 ± 0.4139.92 ± 0.30
AFlow-SurveyForge19.63 ± 0.4655.38 ± 0.6271.33 ± 0.5267.53 ± 0.886.74 ± 1.5340.29 ± 0.7043.48 ± 0.28
OZONE (w/o inference-refinement)18.14 ± 0.7853.63 ± 1.2380.45 ± 0.7267.98 ± 1.846.54 ± 1.3043.23 ± 0.9445.00 ± 0.54
OZONE18.31 ± 0.7456.54 ± 0.6281.01 ± 0.7168.51 ± 1.995.75 ± 0.6144.57 ± 1.6045.68 ± 0.32

Table 3-2. CROFTBench

MethodMem.Comp.Appl.Anal.Eval.Creat.Tax.Tab. Place.Fig. Place.Overall Avg.
SurveyForge18.17 ± 0.9354.93 ± 0.2872.74 ± 0.2770.05 ± 0.855.95 ± 1.7645.39 ± 1.3022.56 ± 1.7237.20 ± 4.3816.80 ± 3.6338.20 ± 0.42
SurveyGen18.07 ± 0.6267.31 ± 0.3767.82 ± 1.8372.14 ± 0.507.94 ± 1.8039.14 ± 1.2025.86 ± 5.5438.00 ± 10.5842.00 ± 11.0542.03 ± 2.89
FlowSearch17.71 ± 0.8254.49 ± 2.5457.28 ± 3.5064.30 ± 1.886.06 ± 1.4442.58 ± 0.5723.22 ± 8.1612.00 ± 6.7818.80 ± 9.5532.94 ± 1.67
AFlow-SurveyForge19.42 ± 0.6961.56 ± 1.4472.41 ± 0.2269.78 ± 0.894.24 ± 2.3041.64 ± 1.7018.70 ± 2.8148.80 ± 7.2946.40 ± 5.7342.55 ± 1.79
OZONE (w/o inference-refinement)17.80 ± 0.7658.10 ± 1.4481.72 ± 0.3369.66 ± 1.776.16 ± 1.8146.44 ± 1.7418.67 ± 3.1157.60 ± 9.9442.40 ± 5.7344.28 ± 1.71
OZONE18.37 ± 0.8058.36 ± 1.4081.82 ± 0.3569.74 ± 1.646.87 ± 1.1946.90 ± 1.8718.04 ± 0.6960.40 ± 8.2942.40 ± 7.9244.77 ± 1.65

Across five runs, OZONE (w/o inference refinement) achieved higher mean scores than all baselines on both benchmarks. Enabling inference refinement on the same workflow further raised the mean scores on both benchmarks. These gains suggest that using prior edit outcomes to select operator implementations and review feedback to revise drafts helps improve the quality of the final surveys.

We will update Table 1 in the revised paper to include the results from Tables 3-1 and 3-2.

W2/Q3: Clarification of OZONE’s performance claim

To address your concern about the discrepancy between the performance claim in §5.2 and the CROFTBench results, we clarify the distinction between OZONE with and without inference refinement. The claim across the three benchmarks in §5.2 refers to OZONE with inference refinement enabled. OZONE-Base uses the same optimized workflow with inference refinement disabled. In the original single-run CROFTBench evaluation, OZONE scored 43.34, AFlow-SurveyForge scored 42.84, and OZONE-Base scored 41.77.

In the five-run CROFTBench comparison (W2/Q2), the mean Overall Avg was 44.77 for OZONE and 44.28 for OZONE-Base, both higher than 42.55 for AFlow-SurveyForge. In the revised paper, we will rename OZONE-Base to OZONE (w/o inference-refinement) and state explicitly in §5.2 that the three-benchmark claim applies to OZONE with inference refinement enabled.

W2/Q4: Evaluation Across Judges and Executors

In response to your concern that using GPT-4o-mini as the optimizer, executor, and both rubric judges could favor workflows that score well under its own judgments, we conducted two post-hoc analyses. We first examined how scores change when judges from different model families apply the same global rubric to the same surveys (Table 4). We then used Bloom-Eval to assess whether the final workflow also performed well under an external benchmark when executed by a different model family (Table 5).

For each of the 30 workflows explored during optimization, we generated surveys on the same 10 validation topics using GPT-4o-mini and Claude-Haiku-5.5 as executors. GPT-4o-mini, Gemini 2.5 Flash-Lite (Gemini), and Qwen3-235B-A22B-Instruct-2507 (Qwen) then evaluated the resulting surveys using the same global rubric.

Table 4. Global rubric scores across different executors and judges (Mean ± SD).

ExecutorJudgeStructureCitation accuracyCompletenessContent qualityTruthfulness
GPT-4o-miniGPT-4o-mini83.5 ± 2.172.0 ± 1.589.3 ± 5.473.8 ± 4.954.2 ± 4.6
GPT-4o-minigemini83.3 ± 2.260.6 ± 2.166.9 ± 6.947.7 ± 7.640.8 ± 4.1
GPT-4o-miniQwen81.8 ± 2.865.6 ± 1.874.5 ± 9.382.2 ± 2.277.5 ± 3.0
Claude-Haiku-5.5GPT-4o-mini97.5 ± 1.388.3 ± 1.475.3 ± 3.776.3 ± 3.343.4 ± 5.6
Claude-Haiku-5.5Gemini97.5 ± 1.382.0 ± 2.173.5 ± 5.073.5 ± 5.214.9 ± 3.4
Claude-Haiku-5.5Qwen97.4 ± 1.587.0 ± 2.076.1 ± 4.285.2 ± 1.958.4 ± 3.6

For both executors, scores varied less across judges for Structure, Citation Accuracy, and Completeness than for Content Quality and Truthfulness. Content Quality and Truthfulness call for broader judgments about the survey’s content, such as how well it synthesizes prior work (Content Quality) and whether its claims are supported by evidence (Truthfulness). These judgments may vary across judge models.

Table 5. Global rubric and Bloom-Eval scores of round-selected workflows using two executors.

ExecutorWorkflowGlobal rubric scoreMem.Comp.Appl.Anal.Eval.Creat.Level Avg.
gpt-4o-miniRound 1 W*67.2517.455.381.069.66.543.845.6
gpt-4o-miniRound 2 W*67.7518.153.481.168.24.944.845.1
gpt-4o-miniRound 3 W* (OZONE)68.0919.356.479.972.06.442.146.0
claude-haiku-5.5Round 1 W*66.8917.462.784.267.63.055.148.3
claude-haiku-5.5Round 2 W*65.8616.659.884.667.53.654.747.8
claude-haiku-5.5Round 3 W* (OZONE)67.7417.661.383.967.44.158.448.8

With both executors, Round 3 W* achieved the highest global rubric score and Bloom-Eval Level Avg among the three workflows.

Reviewer_TEKPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes OZONE, a framework for optimizing scientific survey-generation workflows. It organizes the search space through abstract roles and task-specific subclasses. Then, it iteratively modifies operator compositions and dependencies between execution units under input–output contracts.

Local rubrics diagnose intermediate outputs, while a global rubric selects workflows; edit memory and round-level reviews guide subsequent search. Additional editing and revision can be applied at inference time.

The paper also introduces CROFTBench, which evaluates reference matching in taxonomy figures and table and figure placement. Experiments cover three benchmarks, GPT-4o-mini and Qwen3, different retrieval sources, human evaluation, and computational costs, demonstrating quality and inference-efficiency gains from the selected fixed workflow.

I recognize the practical value of the work. My main reservations are the unclear independent role of the search representation and the limited validation of how evaluation scores reflect insight and useful guidance in surveys.

Critical strengths

1. The framework addresses a practically important problem in long-form workflow optimization.

In survey generation, local improvements in retrieval, organization, or synthesis do not necessarily improve the complete document. Separating task-level diagnosis from document-level selection, while using contracts to constrain edits and memory to reuse experience, is a well-motivated design. Its value lies in improving a generation procedure that can be reused across topics.

2. The fixed workflow offers an attractive quality–cost trade-off.

With GPT-4o-mini, OZONE-Base achieves 72.1 on SurveyBench, outperforming SurveyGen, the best baseline, by 3.5 points (68.6). It also achieves 45.88 on Bloom-Eval, compared with SurveyGen’s 44.95. Figure 6 shows a competitive quality–cost trade-off on SurveyBench. These results support the practical value of the optimized workflow without additional inference-time refinement.

Critical weaknesses

1. The contribution of the added search mechanisms is unclear.

The experiments do not isolate the benefits of the class hierarchy and local rubrics beyond AFlow-style search; the local-rubric ablation also changes the memory configuration. Since within-round edits already modify workflow structure, it is unclear what round-level reinitialization adds over continuous search with the same budget and evaluation results.

2. Higher evaluation scores do not clearly establish deeper insight or useful guidance.

Although the rubrics assess comparison and synthesis, higher scores do not clearly establish deeper insights or useful guidance, both central to survey quality. Human evaluation mainly assesses organization, relevance, and coverage, compares only against SurveyForge rather than stronger baselines.

Moreover, a well-designed pairwise comparison against stronger baselines, with randomized order and criteria focused on insight and guidance, would provide more convincing evidence of practical improvement.

Questions for authors
  1. What are the respective contributions of the class hierarchy, local feedback, and round-level review and reinitialization beyond AFlow-style search? In particular, does reinitialization improve over continuous search with the same budget and review feedback?

  2. Is there direct evidence of deeper insight and more reliable guidance in the generated surveys? If available, please report blinded pairwise comparisons against stronger baselines.

  3. Both the optimization set and SurveyBench include “Large Language Models.” How is topic non-overlap ensured? Are the target test surveys excluded from survey-example retrieval?

9 comments

We thank the reviewer for the constructive feedback. We address two main concerns: (1) the individual contributions of OZONE's optimization components and (2) whether our evaluations provide direct evidence of deeper insight and more useful guidance in generated surveys.

Regarding the first concern, we organize our ablations to distinguish the contributions of inference refinement and workflow optimization. We rename OZONE-Base to OZONE (w/o inference refinement) throughout the revised manuscript. We examine the contributions of hierarchical task specialization, local rubrics, edit memory, round-level review, and reinitialization (W1/Q1-1), followed by an analysis of how the class hierarchy, local-rubric feedback, and edit memory affect workflow edit selection (W1/Q1-2). We further control for the memory configuration in the local-rubric ablation (W1/Q1-3), investigate the contribution of round-level review (W1/Q1-4), and compare reinitialization with continuous search under the same search budget and review feedback (W1/Q1-5).

Regarding the second concern, we examine Bloom-Eval's Creation metrics as evidence of deeper insight and useful guidance (W2/Q2-1) and provide additional blinded pairwise comparisons against SurveyGen, a stronger baseline (W2/Q2-2). Additionally, we clarify the topic overlap between the optimization and evaluation sets, as well as the survey-example retrieval setup (Q3).

W1/Q1-1: Component Ablations

In response to your request to clarify each component’s contribution, we report end-to-end ablations for edit memory and round-level review (Table 1) and evaluate the workflows selected under each search configuration without inference refinement (Table 2). Both tables report scores for all six Bloom-Eval levels and the Level Avg.

End-to-End Component Ablations

Edit memory records prior edit outcomes and makes them available during workflow optimization and inference-time implementation selection. Round-level review provides feedback for reconstructing workflows between rounds and informs survey revision at inference time. We keep inference refinement enabled in Table 1 to measure the overall effect of removing each component across both stages.

Table 1. End-to-end component ablations on Bloom-Eval.

ConfigurationMem.Comp.Appl.Anal.Eval.Creat.Level Avg.
OZONE19.3456.3579.9472.006.4042.0946.02
w/o memory18.4557.2281.1868.675.1943.7945.75
w/o review17.5054.4780.2766.064.1143.5944.33

Compared with OZONE without inference refinement (45.88; Table 2), full OZONE achieves a higher Level Avg (46.02; Table 1). Removing edit memory or round-level review further reduces end-to-end performance, suggesting that both components contribute to survey quality through the reuse of edit experience and review feedback.

Optimization Component Ablations

As motivated in the original manuscript (lines 74–83; §4.1), our design combines shared functional roles and I/O contracts at the abstract-class level with task-specific objectives and operators at the subclass level to guide workflow composition and editing. We compare the full hierarchy with three variants. Abstract class only uses the operators and rubrics defined for abstract classes. Subclass only uses subclass-specific operators and rubrics. Abstract-class targeting, same rubrics retains the original global and local rubrics and restricts edit targets to abstract classes. We disable inference refinement in all configurations in Table 2 so that the comparison reflects the workflows selected through optimization.

Table 2. Optimization component ablations on Bloom-Eval with inference refinement disabled.

ConfigurationMem.Comp.Appl.Anal.Eval.Creat.Level Avg.
OZONE(w/o inference refinement)19.5255.3280.7670.627.3841.6745.88
Abstract class only16.5653.5380.3366.586.7943.3944.53
Subclass only17.9054.5080.5968.286.4443.0445.13
Abstract-class targeting, same rubrics17.9855.6080.8368.695.0743.9745.36
Global rubric only17.5656.0580.4767.265.1743.5545.01
w/o memory17.8153.0280.7568.834.8344.4044.94
w/o review17.7354.1480.7569.166.4842.4845.12
w/o reinitialization18.5153.3680.3868.997.1443.2445.27
Restart from scratch each round17.9553.4980.3166.474.4344.5844.54

The full hierarchy (OZONE (w/o inference refinement)) achieved a higher Level Avg than either single-level variant (Abstract class only and Subclass only), supporting the design that combines shared contracts with task-specific operators. With the original local and global rubrics held fixed, restricting edit targets to abstract classes reduced the Level Avg from 45.88 to 45.36. This supports our hypothesis that task-specific objectives enable more precise edit targeting within shared functional roles (e.g., selecting Synthesis refinement within Refine in response to feedback on comparative synthesis).

The remaining ablations also yielded lower Level Avg scores, suggesting that task-specific diagnosis, reuse of edit experience, round-level feedback, and workflow reinitialization contribute to the quality of the optimized workflow. We further examine how these components affect edit selection and optimization across rounds in W1/Q1-2–Q1-5.

We will include Table 1 and Table 2 in the revised manuscript.

W1/Q1-2: Effects on Workflow Edit Selection

To examine how the class hierarchy, local rubrics, and edit memory affect which targets are selected for editing, we compare OZONE with three variants, each removing one of these components. Table 3 reports how often each target was selected across the three optimization rounds.

Table 3. Distribution of selected edit targets across optimization configurations.

ConfigurationSelected edit targets across all rounds (total count)
OZONESynthesis refinement (8), Outline verification (5), Content refinement (4), Content writing (4), Coherence refinement (2), Literature analysis (2), Paper record retrieval (2)
Abstract-class targeting, same rubricsRefine (11), Writing (9), Retrieve (3), Verify (3), Analysis (1)
global rubric onlyContent writing (19), Example retrieval (3), Paper record retrieval (2), Pool retrieval (1), Literature analysis (1), Outline writing (1)
w/o memoryContent refinement (9), Synthesis refinement (9), Content writing (9)

Class Hierarchy. With the same rubrics, the hierarchy changes how precisely the optimizer can specify an edit target. In the abstract class setting, Refine covers several objectives. Subclass-level targeting lets the optimizer choose among them explicitly (e.g., Synthesis refinement for comparative synthesis). In OZONE, the optimizer used this distinction to direct edits separately to synthesis, content, and coherence refinement.

Local Feedback. With the global rubric alone, 19 of 27 edits targeted Content writing. With local feedback, the optimizer also frequently edited units responsible for outline verification and synthesis refinement. This shift is consistent with local rubrics helping the optimizer identify which intermediate stage to edit when the completed survey leaves room for improvement.

Memory. When edit memory was removed, the search repeatedly returned to the same three target types, each selected nine times. With memory, the optimizer also edited units responsible for outline verification, coherence refinement, literature analysis, and paper record retrieval. This difference suggests that access to prior edit outcomes affects how narrowly the search concentrates on a few parts of the workflow.

W1/Q1-3: Controlled Local-Rubric Ablation

In response to your concern that the original local rubric ablation also changed the edit memory configuration, we repeated the comparison over three rounds, starting both settings from the same workflow (G = 64.4) and holding the edit memory configuration fixed. Table 4 reports the best global score in each round and the change in global score from each parent workflow to its edited version ΔG.

Table 4. Round-wise best global scores and edit-level score changes with and without local rubrics.

RoundGlobal only Best GGlobal+Local Best GGlobal only ΔG (Mean ± SD)Global+Local ΔG (Mean ± SD)
166.5567.25−0.88 ± 4.39−0.81 ± 5.27
265.8167.75−0.82 ± 4.78−0.90 ± 4.90
366.3868.09−0.39 ± 4.20−2.60 ± 4.54

Global only reached its highest global score G in Round 1. With Global+Local, the best G increased each round, and mean ΔG fell from −0.81 to −0.90 to −2.60. The workflow assembled at the start of Round 3 achieved the highest G of the run and served as the parent for seven of the nine edits. None of the nine edits improved on its parent’s score, suggesting that the search was settling around this workflow in the final round.

Comments on the paper

2 comments

We have updated our responses to include a more detailed interpretation of the ablation results and supporting analyses. Thank you again for taking the time to review our paper and share your feedback.

Dear Reviewers,

Thank you for your thoughtful reviews and for the questions that guided this revision. We especially appreciate your recognition of the value of combining local diagnosis with document-level evaluation (Reviewers TEKP and aeNP), the quality–cost trade-off of the optimized workflow (Reviewers TEKP and bHBi), and the contribution of human evaluation to the empirical study (Reviewers bHBi and aeNP).

The tables below show where we address each discussion point in the revised manuscript and list the relevant table, figure, and page numbers.


Reviewer TEKP

W/QTopicLocationPage
W1/Q1-1Component ablations§5.2, Tables 2–38–9
W1/Q1-2Edit target selectionApp. E.5, Table 1630
W1/Q1-3Controlled local-rubric ablation§5.2, Fig. 49
W1/Q1-4Round-level reviewApp. E.5, Table 1730
W1/Q1-5Reinitialization vs. continuous searchApp. E.5, Table 1831
W2/Q2-1Framework novelty and research guidanceApp. E.4, Tables 14–1529
W2/Q2-2Human comparison with SurveyGen§5.2, Fig. 610
W2/Q2-2Human evaluation of insight and guidanceApp. F.2, Tables 27–2835–38
Q3Topic overlap§5.1, Datasets and App. D.1, Table 86–7, 24

Reviewer bHBi

W/QTopicLocationPage
W1/Q1-1Class hierarchy under matched rubrics§5.2, Table 39
W1/Q1-2Three-round local-rubric ablation§5.2, Fig. 49
W2/Q2Five-run results and standard deviations§5.2, Table 1(b–c)7
W2/Q3Performance claim with inference refinement§5.2, Main Results7
W2/Q4Evaluation across judges and executorsApp. E.6, Fig. 9 and Table 1931–32
W2/Q5Synthesis and research guidanceApp. E.4, Tables 14–15 and App. F.2, Table 2729, 37
W2/Q5Human evaluation of coverageApp. F.133
Q6Figure and table placement metricsApp. C.522–23
Q6Human validation of placement metricsApp. F.3, Table 2937–38

Reviewer aeNP

W/QTopicLocationPage
W1/Q1-1Class hierarchy and local-rubric ablations§5.2, Table 39
W1/Q1-2Edit memory, round review, and reinitialization§5.2, Tables 2–38–9
W2/Q2Workflow evolution and feedback-guided editsApp. E.7, Tables 22–2332 (discussion), 35 (tables)
W2/Q3Three-round rubric ablation and edit history§5.2, Figs. 4–59–10

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.