- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes OZONE, an object-oriented framework for optimizing agentic workflows for scientific survey generation. The key idea is to organize the workflow search space using predefined abstract classes and task-specific subclasses, allowing the optimizer to modify both operator implementations and their dependencies while preserving shared input/output contracts. The framework further introduces local rubrics to diagnose intermediate outputs and a global rubric to evaluate complete surveys, using both signals to guide iterative workflow optimization. At inference time, OZONE supports runtime editing and review-based refinement. The authors also introduce CROFTBench, a new benchmark for scientific survey generation that extends existing evaluation dimensions to include figures and tables. Experiments on SurveyBench, Bloom-Eval, and CROFTBench demonstrate that OZONE consistently outperforms several survey generation and workflow optimization baselines. Additional ablation studies, human evaluations, and cost analyses provide further evidence of the framework’s effectiveness.
Critical strengths
-
The paper addresses an interesting challenge in agentic workflow optimization for long-form scientific writing, where improvements to individual components do not necessarily translate into better overall outputs. The proposed object-oriented representation provides a structured way to organize task-specific operators and their dependencies, while the combination of local and global rubrics offers more informative feedback for workflow optimization. I find this formulation useful and potentially applicable to other complex agentic tasks beyond survey generation.
-
The paper evaluates OZONE on three benchmarks and demonstrates consistent improvements over existing survey generation systems and an AFlow-based optimization baseline. The inclusion of human evaluation, rubric ablation, and quality–cost analysis strengthens the empirical findings. In addition, CROFTBench extends existing survey generation evaluations to figures and tables, providing a useful benchmark contribution for future research on automated scientific writing.
Critical weaknesses
-
While the proposed framework is well structured, its core optimization methodology shares similarities with existing agentic workflow optimization approaches such as AFlow and ADAS, which also optimize operator implementations and workflow structures based on evaluation feedback. The main distinction appears to be the predefined class hierarchy and subclass-specific rubrics. However, OZONE introduces several components simultaneously, including the object-oriented search space, local rubrics, experience memory, and round-level review. The current experiments do not sufficiently disentangle their individual contributions. In particular, a comparison with an otherwise identical workflow optimizer that uses the same local/global rubrics but without the object-oriented hierarchy would help establish whether the proposed representation provides benefits beyond structured feedback and search-space constraints.
-
Although the paper demonstrates improvements in final survey quality, it provides limited insight into how the optimized workflows differ from their initial versions or existing manually designed workflows. The qualitative case studies mainly compare generated survey content rather than illustrating the workflow optimization process itself. I would be interested in seeing concrete examples of how local rubric feedback leads to specific operator or dependency modifications, and how these changes improve the final output. Additionally, the local rubric ablation is limited to a single first-round search run, making it difficult to assess the robustness and generality of the reported optimization gains. More detailed workflow evolution analyses and repeated experiments would strengthen the evidence for the proposed optimization mechanism.
Questions for authors
-
I wonder how much of the performance improvement can be attributed specifically to the object-oriented search space, as opposed to the local/global rubric feedback and other optimization components? Would a workflow optimizer without the predefined class hierarchy, but equipped with the same rubrics and search budget, achieve comparable results? I think this distinction is important for understanding the core methodological contribution.
-
Workflow evolution and interpretability. Could the authors provide a concrete example illustrating how OZONE evolves a workflow during optimization? In particular, it would be helpful to see the initial and final workflow structures, the local rubric feedback that triggered specific operator or dependency edits, and the resulting changes in local and global scores. This would help clarify what kinds of improvements the optimizer actually discovers beyond existing manually designed workflows.
-
Robustness of local rubric optimization. The rubric ablation in Figure 4 is conducted on a single first-round search run, and both settings achieve positive global improvements in only 2 out of 9 edits. How stable are these findings across different random seeds or search budgets? Also, does the benefit of local rubrics persist across subsequent optimization rounds?
6 comments
W1/Q1-2: Optimization Component Ablations
In response to your question about how the other optimization components contribute to OZONE’s performance, we ablate edit memory, round-level review, and reinitialization separately. Table 2 evaluates edit memory and round-level review with inference refinement enabled. Both components are used during optimization and inference, so this comparison captures the effect of removing each component across the full generation pipeline.
Table 2. Bloom-Eval Results w/ Inference Refinement
| Configuration | Mem. | Comp. | Appl. | Anal. | Eval. | Creat. | Level Avg. |
|---|---|---|---|---|---|---|---|
| OZONE | 19.34 | 56.35 | 79.94 | 72.00 | 6.40 | 42.09 | 46.02 |
| w/o edit memory | 18.45 | 57.22 | 81.18 | 68.67 | 5.19 | 43.79 | 45.75 |
| w/o round-level review | 17.50 | 54.47 | 80.27 | 66.06 | 4.11 | 43.59 | 44.33 |
Removing either component lowered the end-to-end Level Avg. Edit memory makes prior edit outcomes available when the optimizer proposes edits and when OZONE selects operator implementations at inference time. Round-level review provides feedback for reconstructing the workflow between rounds and revising the survey at inference time. The lower scores support retaining both components so that prior edit outcomes and review feedback can inform decisions at both stages.
To assess each component’s contribution during optimization, we disable inference refinement and evaluate the surveys generated by the workflow selected under each search configuration (Table 3).
Table 3. Bloom-Eval Results w/o Inference Refinement
| Configuration | Mem. | Comp. | Appl. | Anal. | Eval. | Creat. | Level Avg. |
|---|---|---|---|---|---|---|---|
| OZONE (w/o inference refinement) | 19.52 | 55.32 | 80.76 | 70.62 | 7.38 | 41.67 | 45.88 |
| w/o edit memory | 17.81 | 53.02 | 80.75 | 68.83 | 4.83 | 44.40 | 44.94 |
| w/o round-level review | 17.73 | 54.14 | 80.75 | 69.16 | 6.48 | 42.48 | 45.12 |
| w/o reinitialization (continuous search) | 18.51 | 53.36 | 80.38 | 68.99 | 7.14 | 43.24 | 45.27 |
| Restart from scratch each round | 17.95 | 53.49 | 80.31 | 66.47 | 4.43 | 44.58 | 44.54 |
Removing edit memory or round-level review lowered the Level Avg even with inference refinement disabled. This suggests that reusing prior edit outcomes and incorporating feedback on completed surveys help the optimizer find better workflows during search. OZONE (w/o inference refinement) achieved a higher Level Avg than both w/o reinitialization (continuous search) and Restart from scratch each round. This result supports reassembling the workflow between rounds using review feedback and components from the previous round’s best workflow.
We will add Table 2 and Table 3 in the revised paper.
W2/Q2: Workflow Evolution and Interpretability
In response to your request for a concrete example of workflow evolution, we trace changes in workflow structure, operator compositions, and local rubric feedback across three optimization rounds.
Table 4. Workflow evolution across three optimization rounds.
| Workflow | Key structural and implementation changes | Global Score (G) |
|---|---|---|
| Initial W_0 | 10 units: Retrieval → Outline writing → Analysis → Content writing → Refinement and verification | 64.4 |
| Round 1 W* | Revised the Revise prompt to better address flagged findings and citation support | 67.3 |
| Round 2 W* | Reorganized into 8 units; connected Literature analysis to Outline writing; added ComparativeSynthesis and Revise within Content writing | 67.8 |
| Round 3 W* | Moved Outline verification before Content writing; reduced the number of ComparativeSynthesis passes within Content writing from two to one | 68.1 |
As shown in Table 4, the initial workflow contained 10 execution units (G = 64.4), while the final selected workflow contained 8 units (G = 68.1). Across rounds, OZONE reorganized unit dependencies and revised operator compositions and prompts. The final workflow was newly assembled at the start of Round 3.
Table 5. Examples of local feedback, workflow edits, and score changes.
| Workflow Edit | Local Rubric Feedback | Applied Edit | ΔL | ΔG |
|---|---|---|---|---|
| W_2_1 → W_2_10 | Weak citation support and limited use of retrieved evidence | Added a second ComparativeSynthesis pass to Content writing | +0.84 | +1.61 |
| W_3_1 → W_3_8 | Few input paper IDs yielded usable records | Increased PoolPapers(num) from 1,500 to 2,000 | +0.56 | −3.75 |
Table 5 illustrates how local rubric feedback guides specific operator edits and how their effects differ at the local and global levels. Adding a second ComparativeSynthesis pass improved both scores, whereas increasing the PoolPapers limit improved the local score but reduced the global score. These examples show that local improvements do not necessarily translate into better document-level quality.
We will include the complete workflow structures, operator and dependency changes, and additional edit-level examples in the revised manuscript.
We thank the reviewer for the constructive feedback. We address two main concerns: (1) the contributions of the class hierarchy, rubric feedback, and other optimization components and (2) how workflows evolve during search and whether the benefit of local rubrics persists across rounds.
Regarding the first concern, we examine the contributions of the class hierarchy and local rubrics (W1/Q1-1). We then assess the roles of edit memory, round-level review, and reinitialization through separate ablations (W1/Q1-2). We rename OZONE-Base to OZONE (w/o inference refinement) throughout the revised manuscript.
Regarding the second concern, we trace how the workflow evolves from its initial to its final form and show how local rubric feedback leads to specific edits and changes in local and global scores (W2/Q2). We also extend the local-rubric ablation to three rounds to examine whether its benefits persist beyond the first round (W2/Q3).
W1/Q1-1: Class Hierarchy and Local-Rubric Ablations
In response to your question about what the object-oriented class hierarchy contributes beyond rubric feedback, we compare the full hierarchy with three variants and separately examine the contribution of local rubrics. We evaluate the best workflow selected under each search configuration without inference refinement (Table 1).
As motivated in the original manuscript (lines 74–83 and §4.1), our design combines shared functional roles and I/O contracts at the abstract-class level with task-specific objectives and operators at the subclass level to guide workflow composition and editing. To examine the contribution of each level, we compare the full hierarchy with Abstract class only and Subclass only. We also evaluate Abstract-class targeting (same rubrics), which restricts edit targets to abstract classes while retaining the original local and global rubrics. This comparison tests whether the optimizer benefits from targeting subclasses when it receives the same feedback.
Table 1. Class hierarchy and local-rubric ablations on Bloom-Eval without inference refinement.
The full hierarchy achieved a higher Level Avg than both single-level variants (Abstract class only and Subclass only), supporting the design that combines shared contracts with task-specific targets for editing. With the original local and global rubrics held fixed, restricting edit targets to abstract classes reduced the Level Avg from 45.88 to 45.36. This suggests that the optimizer can make better use of the same feedback when it can act on a specific objective within a shared role (e.g., selecting Synthesis refinement within Refine in response to feedback on comparative synthesis). Removing local rubrics also lowered the Level Avg with the full hierarchy in place, suggesting that feedback on intermediate outputs helps the optimizer decide where to edit.
Evaluating a workflow optimizer without the predefined class hierarchy would require a different way to apply OZONE’s local rubrics to individual operators, since these rubrics are defined for intermediate outputs at the subclass level. For this reason, we used only the global rubric during optimization in AFlow-SurveyForge.
We will add Table 1 in the revised paper.