acceptodds
Under review as a conference paper at ICLR 2027

From Design to Learning: Evolving Research Writing Harnesses from Expert Papers

15 views

Abstract

Turning unstructured research materials into coherent manuscripts is an important part of AI-assisted research. Existing writing systems rely on predefined agentic workflows and skills shaped by a small number of designers, leaving open whether better writing practices can be discovered through broader writing experience. We introduce EvoWriter, a meta-harness that enables research writing systems to learn from the writing experience embodied in expert papers. EvoWriter operates around a writing harness that manages manuscript planning, drafting, and revision, and iteratively improves its agentic workflows and skills through writing practice. Specifically, it compares generated manuscripts with expert papers to identify potential improvements, translates these findings into reusable workflow and skill updates, and retains updates that improve subsequent writing. We study this evolution process using 100 research writing tasks drawn from distinguished papers across five major AI conferences and eight contribution types, with separate subsets for evolution, update selection, and final evaluation. Through this process, EvoWriter evolves the initial writing harness into WritingHarness, an executable system for generating complete manuscripts from new research materials. Across automatic and independent human evaluations, the WritingHarness outperforms baseline writing systems, producing clearer scientific arguments and better-organized evidence while preserving the supplied results and claim boundaries. These results show that writing experience distributed across expert papers can guide the continued improvement of research writing systems beyond predefined human designs.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Research Writing, Writing Agents, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WwWiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EvoWriter learns reusable writing workflows and skills by comparing generated manuscripts with expert papers and validating proposed updates on separate tasks. Experiments report improvements over writing baselines, transfer across writer models, and favorable human preferences on 45 held-out papers.

Critical strengths

Separating update proposal from validation provides a useful framework for learning writing procedures across tasks. Cross-model transfer and human preference over PaperOrchestra support the practical value of the final system.

Critical weaknesses
  1. The experiments do not isolate the benefit of expert-paper comparisons from iterative optimization. Reference-augmented baselines use frozen guidance, while Figure 4 compares evolving alternatives only on training papers. The held-out evaluation compares complete systems with different workflows.

  2. The benchmark may pre-solve part of the argument construction that motivates the method. Dossiers extracted from finished papers already specify motivation, main findings, and author interpretations, allowing writers to inherit decisions about which results matter and how they support the paper’s argument.

  3. Support weights atomic claims equally, regardless of their role in the argument. A hypothetical manuscript containing 99 supported claims and one unsupported central conclusion would score 99%, even if that conclusion determines the paper’s main scientific message.

Questions for authors

Are evaluators and update budgets matched across Figure 4 conditions, and do the improvements persist on its held-out split?

How much of the original papers’ interpretation and argument structure remains in the input dossiers?

Can an expert audit establish whether generated central conclusions remain within the supplied evidence?

Reviewer_h3thOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The authors proposed to use self-evolving harness on paper writing tasks. With excellent paper as reference, the framework starts from a seed harness and self-improves according to the comparison with excellent papers. The resultant harness shows best performance against several existing baselines.

Critical strengths
  1. The topic of agents for paper writing is important
  2. The method is inspired by several agent harness self-evolving work, which is interesting and makes sense
  3. The importance of using excellent paper as self-evolving reference is emphasized and can be an important information for this task
  4. Some analysis shows useful insights
Critical weaknesses

Major Concerns

  1. Evaluation on a self-constructed benchmark. All results are reported on the authors' own dataset, without disclosing the 100 selected papers. Potential selection bias and topical overlap between evolution and evaluation tasks remain unclear. The authors should pick at least one established public benchmarks (e.g., PaperWritingBench) used by at least one of the baseline methods in their original papers, and split it into train and test subset. Then do evolving on the training set and report the performance on the test set, as well as the final harnesses. Also report the performance of baseline methods in this test set.
  2. Unclear integration with existing agent harnesses. Figure 3 combines WritingHarness with Claude Code and OpenCode, which are themselves agent harnesses. It is unclear how one harness executes another. If the integration is prompt-based (i.e., instruct the Claude Code or OpenCode to do what a certain block in the workflow does), the underlying agents may autonomously invoke subagents or perform additional file operations, making the actual workflow different from the proposed architecture. Unlike Meta-Harness, where Claude Code only proposes harness modifications, here it participates in he evaluation of a certain harness. The authors should clarify the integration mechanism and how the intended workflow is enforced.

These two are the most important questions the author should address. Addressing these two concerns would substantially improve the reliability and reproducibility of the results.

Intermediate Concerns

  1. Missing prompts for evolution procedure. The paper provides prompts for the resultant writing agent but not the prompt about the self-evolution process. Since harness evolution is the main contribution, the authors should release the evolution prompts and relevant configurations.
  2. Missing component-level ablations. The final WritingHarness lacks ablations of its individual components. It is unclear whether each evolved component, such as the cold-reader review mechanism, contributes to the improvements.
  3. Fairness of the rubric-guided baseline. It is unclear whether reference-guided evolution outperforms rubric-guided evolution because expert papers contain unique knowledge or simply provide richer feedback. More detailed writing guidelines in rubrics might achieve comparable results. The specific rubric and reflection prompt used in Figure 4A should also be provided.

Minor Issues

  1. Evolution variance. Results are mostly reported as single scores or curves, although different evolution runs may produce different harnesses. Multiple evolution seeds and the corresponding mean and standard deviation would improve confidence in the results.
  2. Performance across paper types. A breakdown by paper type would help assess generalizability. For example, theory papers emphasize logical derivations, while empirical papers emphasize experimental analysis. Consistent improvements would strengthen the universal harness claim; otherwise, the breakdown would clarify its limitations.
Questions for authors

I have left my comments on the strengths and weaknesses part.

Reviewer_R5mxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EvoWriter, a scientific writing system. EvoWriter compares paper drafts with expert papers, and turn the differences into reusable workflows and skills. Candidate updates are selected on held-put materials, and the resulting WritingHarness is evaluated on 45 test papers. This paper also studies the guidance used for evolution and the decision to switch from changing the workflow to refining the skills.

Critical strengths
  1. The method can learn changes reusable across manuscripts. The separation between revising a manuscript and updating the writing system is useful. An accepted update changes the shared workflow or skills used by later tasks. The writer receives the research materials and approved skills, while the expert manuscripts and the accumulated Wiki still remain outside its context. This design gives the proposed learning procedure a concrete form, and makes the resulting harness usable without the need of supplying reference manuscripts at writing time.

  2. The final harness has evidence of useful transfer. The frozen harness improves results across two agent environments and five Writers per environment, with matched inputs and generation limits. The blinded reader study on test papers also favors its manuscripts. These results show that the final harness improves writing across configurations, and readers can notice the improvement.

Critical weaknesses
  1. The evolution study does not establish which choices generalize. The main claim is that expert guidance and progressive workflow & skill updates produce better reusable writing practices. However, the comparison supporting these choices in Fig 4 is evaluated on ten papers used for evolution. Better scores on the evolution papers can reflect adaptation to those papers without establishing which strategy produces better writing on new tasks. The 45 paper test supports the final selected harness, but does not resolve this question about the learning procedure. I would like to see the competing evolution strategies evaluated on the same held-out papers before accepting the conclusions in Fig 4.

  2. The evaluation starts after much of the research materials have already been organized. For example, the paper motivates the task as turning unstructured research material into coherent manuscripts. However, in the evaluation the inputs are curated packages derived from distinguished papers. The curator has already organized the problem, method, findings, conditions, limitations and unresolved conflicts into a self-contained dossier, with separate evidence files.

  3. The evidence for preserving scientific content is weaker than the evidence for readability. Preserving results and claim boundaries is a central requirement of the task. Yet the evidence for this property comes from automatic retention and support scores. The human study evaluates manuscript preferences and perceived understanding, argument progression and evidence organization, it does not report an independent audit of factual claims against the supplied materials.

Questions for authors
  1. Do the evolution strategies retain their advantage on held out tasks?

  2. Have readers independently checked factual preservation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.