acceptodds
Under review as a conference paper at ICLR 2027

FORGE: Evidence-Governed Algorithm Discovery by Attributing Failures and Certifying Interventions

Abstract

Algorithm-discovery agents traditionally iterate through execution, scoring, and revision; however, the research state guiding each revision remains either an unstructured narrative or a unidimensional ranking—a practice that systematically obscures failure origins, renders interventions non-falsifiable, and actively incentivizes evaluator overfitting. We introduce FORGE, an evidence-governed agent that maintains an append-only state comprising executable outcomes, counterexamples, failure taxonomies, resource consumption, and full provenance. At each iteration, FORGE isolates a specific failure mechanism, articulates a predicted behavioral shift, compiles a bounded candidate edit, subjects it to robustness testing without access to hidden labels, and accepts modifications only after deterministic validity and resource constraints are satisfied. Across a 640-cell Exp2 benchmark, FORGE attains the highest mean hidden score (0.910), outperforming AIScientist v1 (0.892), v2 (0.635), and OpenEvolve (0.765), with a +0.017 gain [95% CI: +0.004, +0.030] over the strongest baseline. In a comparator-network repair study, FORGE yields 26 verified repairs out of 40, compared to only 5 for baselines. Two exploratory transfer tasks further confirm generalization beyond synthetic search: in memory retrieval, FORGE achieves 0.742/0.751 vs. FullText (0.604/0.676) and NaiveRAG (0.650/0.719); in skill optimization, FORGE r55 attains 0.867/0.811 on SearchQA/SpreadsheetBench, surpassing SkillOpt (0.831/0.625) by +0.036/ + 0.186. These results position FORGE not merely as a stronger optimizer, but as a principled framework toward interpretable, auditable, and increasingly autonomous algorithmic discovery—with clear potential to scale across domains where reliable evidence, rather than heuristic ranking, becomes the cornerstone of intelligent search.

Primary area:
learning theory
Keywords:
LLM agents, algorithm discovery, evidence-governed search, failure attribution, counterfactual probing, program synthesis, autonomous research agents, verifiable agents

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Fh8XOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes FORGE, an evidence-governed framework for automated algorithm discovery. It maintains structured records of execution outcomes, failure causes, counterexamples, and resource usage to guide iterative modifications. The framework combines failure attribution, targeted interventions, and verification-based admission to improve the reliability of algorithm discovery.

Critical strengths

1.The research problem is meaningful, and using structured evidence and failure attribution to guide algorithm discovery is an interesting idea.

2.The framework provides a systematic workflow that integrates failure analysis, targeted interventions, and verification.

Critical weaknesses

1.Limited improvements over the strongest baseline. FORGE improves the average hidden score from 0.892 to 0.910 compared with AI-Scientist v1. Moreover, the improvements are primarily concentrated in empirical-rule discovery, while gains on other task families are marginal or negative. This raises concerns about the general effectiveness of the proposed framework.

2.Insufficient component-level ablations. Although the paper includes preliminary ablation experiments, they cover only two task families. The individual contributions of evidence memory, failure attribution, counterfactual probes, and bounded admission are not sufficiently established across the full benchmark.

3.Limited generalization to realistic algorithm discovery. The main evaluation focuses on relatively small synthetic tasks using a single base model. Although additional transfer studies are presented, they are exploratory and do not provide sufficient evidence for broader applicability to complex, real-world algorithm discovery.

4.Additional computational overhead. FORGE requires more public executions and longer wall-clock time than AI-Scientist v1 (165.7 vs. 128.2 seconds). Given the modest improvement in hidden scores, its computational cost-effectiveness remains unclear.

Questions for authors

1.How do the authors justify the general effectiveness of FORGE given that improvements are concentrated in one task family?

2.Can the authors provide more comprehensive ablation studies to isolate the contributions of the core components?

3.Can the authors validate FORGE on more complex real-world tasks and additional base models?

4.Can the authors provide a more detailed analysis of the performance–cost trade-off compared with existing baselines?

Reviewer_cUhoOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper is an algorithm-discovery agent that bases each step on recorded evidence instead of free-text reflection or a single score. On a 640-cell benchmark using GLM-5.2, FORGE has the best hidden score at 0.910, compared with 0.892 for AI-Scientist v1 and 0.765 for OpenEvolve.

Critical strengths
  1. The topic is important. Giving enough/proper information to the agent is important.
Critical weaknesses
  1. The paper is really hard to read. There is too much confusing terminology.

  2. The task is a little bit easy

  3. The improvement is not significant.


Actually, since the paper is really hard to read, I don't completely understand it. So my confidence is 1.

Questions for authors
  1. Present the real pipeline and prompt of this method.
Reviewer_M1kFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FORGE proposes evidence-governed algorithm discovery using structured failure attribution, constrained edits, and operational verification to improve the reliability and auditability of iterative search.

Critical strengths

The primary evaluation uses task-level paired inference across 32 tasks, reporting a positive gain of 0.017 over the strongest baseline (95% CI [0.004, 0.030]).

The paper carefully distinguishes operational validity from semantic correctness and preserves unsuccessful outcomes. The comparator-network study provides additional evidence for attribution-guided repair.

Critical weaknesses

The implemented failure-to-edit mechanism in the primary experiment and the baseline adaptations are insufficiently specified. The limited ablation does not isolate the contribution of typed evidence and edit restrictions from differences in prompts, search procedures, or actual public-evaluation usage.

The primary gain is concentrated in empirical-rule discovery, with negligible or negative gains in other families. The OOD comparison against the strongest baseline is inconclusive, limiting claims of broader generalization.

The memory and skill transfer studies do not sufficiently clarify how v36/r55 were selected relative to final evaluation data. Matched optimization budgets and total retrieval context lengths are also insufficiently established.

Questions for authors

Can you detail the implemented attribution-to-edit mechanism in Exp2 and provide controlled ablations isolating typed edit restrictions under matched feedback and evaluation budgets?

Which development data and evaluation outcomes informed v36/r55, and were final test sets untouched throughout selection? Can you provide comparisons with matched optimization and total context budgets?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.