acceptodds
Under review as a conference paper at ICLR 2027

Formalizing literature-review construction as a compositional and recursive operator system

Abstract

Producing a long literature review requires a system to find relevant papers, organize their evidence, write individual sections, and combine those sections without losing the link between claims and sources. We present , a workflow that divides this process into nine operators. Each operator has a defined input, output, and responsibility, so its implementation can be replaced and evaluated independently. We test the design in four ways. First, we recover section trees from 2,221 scholarly reviews and show that their organization can be distinguished from closely matched artificial alternatives. Second, citation data trains an effective paper ranker, and section data supports recognition of common organization patterns. Third, controlled replacements show that evidence-aware organization and validation improve 1,000-word reviews. Fourth, when every method receives the same 32 evidence cards, hierarchical writing and integration outperform naive chunking, a fixed outline, and flat generation on all six long-form topics. These results show that explicit operator interfaces support both controlled experimentation and higher-quality long-form synthesis.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Typed Operators, Recursive Composition, Evidence‑grounded Synthesis, Automated Literature Review, Modular Workflows

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qk1NOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SLRGP represents literature synthesis as nine replaceable operators for query expansion, location, filtering, ranking, organization, validation, evidence preparation, local writing, and integration. A shared state tracks the question, sources, section tree, checks, text, provenance, and budgets. Each operator specifies preconditions, permitted state changes, and postconditions, requiring evidence cards, citations, and writing targets to resolve to valid sources and validated structure. Long-form generation reorganizes evidence at internal nodes, writes leaf sections from local evidence, and adds parent-level synthesis while combining child sections, subject to depth, word, and retry budgets.

The authors recover usable structures from 2,221 of 2,862 candidate reviews, obtaining 14,332 nodes and splitting data by complete review. Citation-supervised ranking increases nDCG@10 from RRF’s 0.259 to 0.412. A title-based organization recognizer analyzes existing trees; generation instead uses a language-model organizer. Short-review experiments fix sources, model, group counts, and a 1,000-word target, comparing semantic grouping with retrieval-order grouping and guarded validation with unguarded execution under stress. On six long-form topics, Stage B gives four methods the same 32 evidence cards; the complete hierarchical configuration wins all six comparisons against each of naive chunking, fixed outlines, and flat generation. An anonymized, randomized panel from three model families evaluates outputs. A separate eight-topic native-workflow comparison fixes the generation backbone and a 2023 source cutoff, reporting length, cost, sampled claim support, and preferences over 4,000-word windows as deployment-profile evidence.

Critical strengths
  1. The long-form comparison fixes the evidence pool and 32 cards, preventing source selection from directly explaining the writing difference. Page 12 distinguishes the earlier chunking implementation from Stage B’s internal reorganization and upward integration, with an execution trace documenting those operations. Six wins against each of three controls support the combination’s benefit for panel-rated organization and coherence. The implementation audit and resulting protocol correction are clearly disclosed.

  2. The ranking experiment separates complete reviews across splits. On 153 held-out reviews, nDCG@10 rises from 0.259 to 0.412, with a difference interval of [0.113, 0.201]. Citation-link errors and label noise receive additional checks. Negative results for learned expansion and filtering are retained rather than describing all nine interfaces as learning successes. These are concrete, inspectable empirical findings.

Critical weaknesses
  1. The cited AutoSurvey, SurveyForge, and HiReview already use outlines, section writing, or hierarchical organization. Replacing O establishes semantic grouping over retrieval-order grouping, and replacing V establishes checking and retrying under stress, but neither explains the additional research problem solved by typed contracts over an ordinary modular workflow. The organization classifier uses existing parent and child titles and is not used in generation. This is disclosed, so its classification gain cannot establish learned organization improving synthesis. Stage B fixes evidence but does not sufficiently document total call and generation budgets across methods or separate reorganization from parent integration. The evidence supports a reasonable combination more clearly than the specific increment of the framework and recursive design.

  2. Appendix G calibrates the panel on only two reviews. Removing synthesis language changes critical-synthesis scores by −0.33 with p=0.375, and evaluator agreement α ranges from 0.26 to 0.47. Evidence for organization and coherence preferences is therefore stronger than for scholarly synthesis quality. These scores cannot equally establish that key evidence is covered or disagreements between methods are correctly synthesized. The native comparison adds a separate claim-support check, but uses different outputs: SLRGP has support 0.835 and coverage of core papers from the source review of 0.045, with large system-length differences. The authors limit this to a deployment comparison; it does not fill the scholarly-content validation gap on the Stage B outputs themselves.

Questions for authors
  1. Which concrete failures require the proposed state-change contracts and invariants beyond an existing modular review workflow? Could execution traces distinguish interface constraints from ordinary validation and retries?

  2. Please report actual calls, input/output tokens, and lengths for each Stage B method. At a fixed total budget, how do internal reorganization alone, parent integration alone, and an existing hierarchical method compare?

  3. Could the same Stage B outputs receive independent checks of key omissions, claim support, and synthesis across sources? What is the full primary-model V comparison when the two outputs below the prespecified length threshold count as failures? The disclosed length rule and all-output directional check for the open model are useful, but do not replace that result.

Reviewer_QivzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents SLRGP, a literature-review generation workflow organized into nine typed operators with explicit input/output contracts and shared source tracking. For long reviews, it recursively organizes evidence, writes local sections, and integrates them through parent-level synthesis.

The authors construct a corpus of 2,221 scholarly reviews to study section structure and train ranking and organization-recognition models. Validation includes held-out prediction tasks, controlled operator replacements, long-form comparisons using identical evidence cards, and native comparisons with existing systems. Results show improved paper ranking and model-panel preferences for evidence-aware organization, guarded validation, and hierarchical synthesis. A separate claim–source evaluation reports higher sampled support than the compared systems, although generation experiments cover only six to eight topics.

Critical strengths
  1. Clear and inspectable workflow: The explicit interfaces and source tracking make the system easier to debug, extend, and evaluate component by component.

  2. Useful controlled experiments: The paper changes individual operators and fixes the evidence cards in long-form comparisons. These controls provide useful evidence that organization and integration improve generated reviews.

Critical weaknesses
  1. Unclear design contribution: The system comparisons are useful, but do not show what typed interfaces add beyond modular design. Recursion also adds planning and synthesis calls, so the source of its gains remains unclear.

  2. Limited robustness: Generation studies cover only six to eight topics, with a small open-weight check. Repeated generation runs are not reported.

  3. Limited quality validation: The panel is calibrated, but judge agreement is limited and removing synthesis had little effect on its score. Expert evaluation would help verify the claimed synthesis gains.

  4. Unclear completeness: SLRGP covers only 4.5% of the reference reviews’ core papers. These references are not a perfect gold standard, but the low coverage needs explanation. Citation support alone does not establish completeness.

Questions for authors
  1. The title seems not align with one in openreview.
Reviewer_6PeZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SLRGP, a literature-synthesis workflow organized into nine operators covering query expansion, retrieval, filtering, ranking, organization, validation, evidence preparation, writing, and integration. Each operator has an explicit interface and a contract governing permitted changes to a shared record containing evidence, document structure, generated text, provenance, and budgets.

The paper constructs a corpus of 2,221 scholarly reviews and uses citations and section trees to study retrieval, ranking, and organization-pattern recognition. Controlled short-review experiments examine evidence-aware grouping and guarded validation. A long-form experiment compares hierarchical organization and integration against naive chunking, a fixed outline, and flat generation using the same 32 evidence cards. A native-workflow comparison evaluates SLRGP against three existing survey-generation systems.

The strongest results concern citation-supervised ranking and panel-rated organization and coherence. I appreciate the effort to separate components and document negative results. My main reservations concern the validity of the synthesis evaluation, the attribution of improvements to the proposed design, and the comparability of the native-system results.

Critical strengths

Strength 1: The explicit interfaces provide a useful structure for controlled experimentation and provenance tracking.

Separating retrieval, ranking, organization, validation, and writing makes it easier to identify what changes between systems. The shared record and permitted write sets also provide concrete places to inspect evidence assignments, source identifiers, and recovery behavior.

The paper makes several useful distinctions rather than treating all components as equally successful. Learned expansion and filtering do not outperform the selected alternatives. Organization-schema recognition is explicitly distinguished from generating a new outline. The short-recursion experiment reports no improvement, and the later long-form configuration is distinguished from the earlier implementation that only performed chunking.

These distinctions make the contribution more credible than a single end-to-end comparison with little visibility into the intermediate decisions.

Strength 2: The study includes meaningful controls and reports several limitations of its own evidence.

The long-form experiment fixes the source material across methods, reducing source-selection confounding. The short-review experiments attempt to replace one operator at a time while retaining the surrounding conditions. Review-level splits prevent sections from the same source review appearing in different data partitions.

The learned ranker also shows a substantial improvement over the reported retrieval-order baselines, increasing nDCG@10 from 0.259 to 0.412. The citation-link audit and label-noise sensitivity analysis provide useful supporting checks.

Finally, the paper reports evaluator calibration, agreement, protocol corrections, exclusions, and negative results. This transparency is valuable, even where the reported diagnostics expose unresolved limitations.

Critical weaknesses

Weakness 1: The evaluation does not yet establish reliable improvements in critical synthesis and evidence grounding.

Most generation-quality conclusions rely on model-panel scores. The calibration provides evidence that the panel detects reordered sections, altered citations, and redundant text. However, Appendix G reports almost no score change after deleting citations and no statistically resolved reduction in critical-synthesis score after removing synthesis language. Agreement is also limited, with Krippendorff's alpha ranging from 0.26 to 0.47.

The calibration is small, so these results do not prove that the panel cannot assess synthesis. They do mean that its sensitivity to two central failure modes has not been established. The evidence therefore supports claims about organization and panel preference more directly than claims about substantive scientific synthesis.

The separate claim–source check is useful, but samples only 12 cited claims per native output and does not assess missing evidence, selective citation, or the correctness of cross-paper conclusions. SLRGP's reported source-review coverage is 0.045, compared with 0.100 for SurveyGen. Citation overlap is not a definitive completeness measure, but these results make an independent assessment of essential evidence coverage important.

A domain-expert evaluation of coverage, unsupported synthesis, and treatment of conflicting findings would substantially strengthen the paper.

Weakness 2: The experiments do not clearly isolate the contribution of typed contracts or the individual components of structural recursion.

The proposed contracts specify useful engineering discipline, but the experiments mainly compare implementations of organization, validation, and writing. They do not establish what benefit comes specifically from the typed contract framework rather than from adding evidence-aware grouping, retries, or additional synthesis passes.

Similarly, the long-form configuration combines internal reorganization and parent-level integration. The paper appropriately states that these operate together, but this leaves their individual contributions unresolved. Giving every method the same evidence cards controls source material; it does not by itself establish equal generation effort.

The overview describes matched budgets, but the main comparison does not provide sufficiently explicit per-condition call counts, token budgets, realized output lengths, and costs to assess whether a similarly resourced flat or outline-based method would close the gap. A factorial comparison of reorganization and integration, together with a budget-matched revision baseline, would make the mechanism clearer.

The corpus-learning results also have a narrower role than the overall framework might suggest. The organization classifier recognizes labels from an already observed tree, including its child titles, rather than constructing an outline from a question and evidence. This is a legitimate prediction task, but does not establish the effectiveness of a learned organizer in generation.

Weakness 3: The small generation studies and native-workflow comparison limit the generality of the conclusions.

The principal long-form comparison contains six topics. Consistent wins across these topics are encouraging, but do not establish robustness across domains, research questions, generation randomness, or evidence-pool difficulty. The open-weight robustness experiment is also small, with only three eligible organization pairs and two validation pairs.

The validation study introduces deliberate stress and excludes two unguarded outputs that fall below the prespecified length threshold. The threshold is transparently reported, but an operational evaluation should include those failures as outcomes as well as reporting quality conditional on completion. It should also measure how often validation helps on naturally occurring errors.

The native-system comparison has substantial differences in output scope: mean lengths range from approximately 4,000 to 60,000 words. Comparing matched 4,000-word windows avoids giving judges radically different reading loads, but can compare a nearly complete SLRGP review with a fragment of another system's document. Per-output costs likewise reflect these different production targets.

Appendix F additionally reports detectable prompt or meta text in every normalized SURVEYFORGE output. This warrants a configuration and normalization sanity check before interpreting that comparison as representative of the baseline.

The paper correctly presents the native comparison as a deployment profile. Its conclusions should remain tied to those configurations rather than implying a generally superior quality–cost trade-off.

Questions for authors

Clarification requests (answers could change my assessment).

  1. Can domain experts evaluate a representative subset for essential-paper coverage, unsupported claims, treatment of conflicting findings, and substantive cross-paper synthesis? How well do these assessments agree with the model panel?

  2. What are the exact generation budgets and realized resource use for each Stage-B condition, including calls, tokens, output length, and cost? Does structural recursion outperform a flat or fixed-outline method given the same opportunity for revision?

  3. What happens when internal reorganization and parent-level integration are enabled separately? Which improvements require the contract framework itself rather than the individual operations it coordinates?

  4. How are cited claims sampled for the support evaluation, and how are inaccessible sources handled? Can support be assessed jointly with coverage so that a system is not favored for producing a narrower set of easily supported claims?

  5. Can the native comparisons be repeated with matched output targets and a verified clean configuration for each baseline? How sensitive are preferences to the choice of 4,000-word window?

Suggestions for revision.

  1. Distinguish referential integrity from semantic support. A resolvable source identifier and a preserved claim–citation link make checking possible, but do not establish that the source supports the claim.

  2. Frame the primary demonstrated benefit as improved organization, coherence, and panel preference under the tested conditions. Reserve stronger claims about scientific synthesis for evaluations that directly assess that capability.

  3. Report both completion-aware and completion-conditional outcomes for validation, including short or otherwise invalid outputs. Add naturally occurring errors alongside the introduced stress cases.

  4. Expand the generation evaluation to independently selected topics and repeated runs. Report whether topics used during pilot-driven development are reused in the final comparisons.

  5. Provide exact model identities and versions, prompts, scoring scales, retry rules, recursion budgets, ranker features, and topic lists in an accessible configuration table or artifact. The role separation is described, but the manuscript leaves several implementation details implicit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.