acceptodds
Under review as a conference paper at ICLR 2027

Triage-PiEvo: Evidence Admission for Principle-Evolving Scientific Agents

Abstract

Large language model (LLM)-based AI Scientists automate increasingly large parts of scientific discovery, but noisy observations make their revisions unreliable. Existing principle-evolving agents act on every anomaly, whether its source is a new mechanism or an evaluation artifact; invalid revisions then bias evolution. To address this, we formulate revision as evidence admission: anomalies steer search immediately, but only independently verified evidence may rewrite persistent principles, and contradicted principles are retracted. We realize this in TRIAGE-PIEVO, which pairs an anomaly-responsive search portfolio with a staged verification gate that controls what enters principle state. Across four scientific tasks with three backbones, TRIAGE-PIEVO attains the highest average solution quality, improving over the strongest baseline of each backbone by +7.5% ∼ 11.8%; on Qwen3-235B it matches that baseline’s full-budget quality with one-third of the evaluation budget. In constructed stress tests, it admits none of the corrupted, hidden-source, or exploit evidence that ungated variants admit, while missing no planted mechanism. Plugged unchanged into three external research harnesses, it cuts false evidence admissions by over 60% and raises verified scores by 31% ∼ 43%.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
autonomous scientific discovery, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7bsNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TRIAGE-PiEvo, a framework designed to improve the reliability of principle-evolving scientific agents under noisy or unreliable experimental feedback. The authors introduce an evidence-admission mechanism that separates observations used for exploration from evidence authorized to revise persistent principles. The authors evaluate the approach on four scientific discovery tasks using three LLM backbones.

Critical strengths
  • The motivation is well-defined and the paper captures an important issue
  • The framework is reasonable and established, and the writing is clear and easy-to-follow
Critical weaknesses
  • Limited novelty: The problem is important but the verification modules are combination of exsiting methods. The contribution of the work mainly focuses on applying the methods to auto research domain. Furthermore, I would like to see the results of sota LLMs since auto research is an inherently difficult task, small GPT variant like luna may not perform well on this task. We do not know whether the methods generalize to stronger LLMs.
  • Unclear performance attribution and evaluation fairness: Table 3 shows that neither module independently outperforms PiEvo, and it is unclear whether additional verification calls are included in the evaluation budget. Limited generalization of robustness claims: The strongest results (0% false admission and 100% mechanism recall) come from constructed stress tests with predefined failure modes and strong planted mechanisms.
Questions for authors

None

3 comments
AuthorsOpenReview ↗

Response to Reviewer 7bsN(1/2)

Dear Reviewer 7bsN,

Thank you for your effort in reviewing our paper with constructive suggestions! We will answer each question respectively.

[W1] Evaluating Triage-PiEvo on additional model backbones

TRIAGE controls when experimental evidence may update principles and retracts contradicted principles together with any rules derived from them. On Claude Sonnet 5.5, Triage-PiEvo achieves 81.10% mean SQ versus PiFlow’s 80.60%, with verification included in the 24-call budget (Table R9).

Experimental design. Table R9 compares methods under 24 total calls, including verification. Table R10 summarizes manuscript Table 1, which uses 24 primary evaluations with verification counted separately.

Evaluating Sonnet and harder Luna tasks.

Table R9: Additional-model and harder-task comparisons. Verification counts toward 24 total calls; mean SQ averages four tasks.

BackboneMethodMBO SQ (%) ↑NHO SQ (%) ↑SPO SQ (%) ↑TMC SQ (%) ↑Mean SQ (%) ↑
GPT-5.6-LunaPiEvo116.00 ± 18.0073.00 ± 8.0021.00 ± 4.0087.00 ± 2.8074.25
GPT-5.6-LunaTriage-PiEvo119.00 ± 17.0074.00 ± 8.5021.50 ± 4.5086.80 ± 3.0075.33
Claude Sonnet 5.5PiEvo118.00 ± 15.0077.00 ± 6.0037.50 ± 1.8086.50 ± 2.7079.75
Claude Sonnet 5.5ReAct121.00 ± 15.5076.00 ± 6.5037.80 ± 1.9086.00 ± 2.8080.20
Claude Sonnet 5.5PiFlow120.00 ± 15.0078.00 ± 6.0037.60 ± 1.8086.80 ± 2.7080.60
Claude Sonnet 5.5Triage-PiEvo122.00 ± 14.0078.30 ± 6.5037.90 ± 2.0086.20 ± 2.8081.10

On the harder Luna tasks, mean SQ improves by 1.08 pp over PiEvo. Gains vary by task: TMC decreases by 0.60 pp on Sonnet relative to PiFlow and by 0.20 pp on Luna relative to PiEvo.

Table R10: Triage-PiEvo leads mean SQ on all three backbones. Best baseline is selected per column; range is maximum minus minimum, in pp.

MethodQwen3-235B-A22B SQ (%) ↑Gemini-2.5-Flash-Lite SQ (%) ↑GPT-5.6-Luna SQ (%) ↑SQ range (pp) ↓
PiEvo70.3567.5362.877.48
PiFlow57.4668.4966.6211.03
ReAct61.9354.2775.8921.62
AI Researcher64.7856.7465.258.51
Best baseline70.3568.4975.897.40
Triage-PiEvo82.1079.6383.373.74

In the original comparisons, Triage-PiEvo exceeds the strongest baseline by 11.75 pp on Qwen3-235B-A22B, 11.14 pp on Gemini, and 7.48 pp on Luna.

Conclusion. Triage-PiEvo achieves higher mean SQ than the strongest listed baseline on each of the original three backbones. With verification included in 24 calls, the additional comparisons show gains of 0.50 pp over PiFlow on Sonnet and 1.08 pp over PiEvo on the harder Luna tasks.

AuthorsOpenReview ↗

Response to Reviewer 7bsN(2/2) part1

Thank you for raising this important concern.

[W2] Testing TRIAGE under harder feedback and in external agent systems

Triage-PiEvo retains a 1.27 pp mean SQ advantage over PiEvo when verification is included in 24 total calls. Beyond predefined artifacts, TRIAGE reduces false admissions under weaker signals and unfamiliar faults, although correlated evaluator errors reduce mechanism recall.

Experimental design. We examine component interactions and call budgets first, then harder feedback, and finally transfer to other agent systems.

(a) Comparing module contributions and evaluation budgets.

Table R5: AFE + TRIAGE leads mean SQ with eight primary evaluations. Qwen3-235B-A22B, four tasks, three seeds; verification is additional (manuscript Table 3).

VariantSQ (%) ↑AUOC (%) ↑Δ SQ vs. PiEvo (pp) ↑
PiEvo64.7858.15–
AFE alone62.7358.23−2.05
IDS + TRIAGE60.6055.08−4.18
Triage-PiEvo (AFE + TRIAGE)65.9860.45+1.20

The combined configuration ranks first; adding TRIAGE improves AFE but lowers IDS performance in this eight-primary-evaluation experiment.

Table R3: Triage-PiEvo improves SQ over PiEvo with verification included in 24 total calls. Same rerun batch; false revisions count incorrect principle updates.

VariantTotal calls (calls/run; fixed)Primary calls (calls/run; allocation)SQ (%) ↑AUOC (%) ↑False revisions (cases/500) ↓Δ SQ vs. PiEvo (pp) ↑
PiEvo (same batch)2424.0071.73 ± 3.4255.12 ± 2.631610
AFE alone2424.0072.40 ± 3.6555.94 ± 2.82172+0.67
IDS + TRIAGE2416.6770.00 ± 2.4354.76 ± 1.9074−1.73
Triage-PiEvo (AFE + TRIAGE)2416.6773.00 ± 3.4457.55 ± 2.7866+1.27

With verification included in the 24-call budget, Triage-PiEvo improves mean SQ from PiEvo’s 71.73% to 73.00% and reduces incorrect updates from 161 to 66 per 500 cases.

(b) Testing beyond predefined artifacts and strong mechanisms.

Table R11 summarizes the controlled tests from Appendix E; Table R12 tests weaker mechanisms, withheld fault families, and correlated evaluator errors.

Table R11: Controlled artifact and planted-mechanism tests. A: false-admission rate (%, ↓); B: recall (%, ↑). Planted effects are at least 2.5σ02.5\sigma_0 (manuscript Table 7).

Panel and metricTest conditionTest cases (count)PiEvo (%) A ↓; B ↑AFE alone (%) A ↓; B ↑Triage-PiEvo (%) A ↓; B ↑
A: false-admission rate ↓Transient noise, dropout (Section 3.2 C1)24100.0%100.0%0.0%
A: false-admission rate ↓Channel-specific fault (Section 3.2 C2)24100.0%100.0%0.0%
A: false-admission rate ↓Surrogate, OOD, drift (Section 3.2 C3)48100.0%100.0%0.0%
B: mechanism recall ↑Discontinuous step shift7284.7%88.9%100.0%
B: mechanism recall ↑Synergistic interaction peak7281.9%86.1%100.0%

Triage-PiEvo rejects all 96 predefined artifacts and recovers all 144 strong planted mechanisms under these controlled conditions.

Table R12: TRIAGE versus PiEvo under harder feedback. Entries give false admissions or recovered mechanisms as count/total.

ConditionPiEvo false admissions (count/total) ↓TRIAGE false admissions (count/total) ↓PiEvo recall (count/total) ↑TRIAGE recall (count/total) ↑
Weak mechanisms24/6011/6040/6042/60
Withheld fault families18/4010/4030/4030/40
Correlated evaluator errors19/4015/4029/4027/40

Under weak mechanisms, false admissions fall from 24/60 to 11/60; under withheld faults, they fall from 18/40 to 10/40. With correlated errors, false admissions fall from 19/40 to 15/40, but mechanism recall falls from 29/40 to 27/40.

(c) Checking independent performance and transfer to other agents.

Table R7: Triage-PiEvo improves held-out performance over PiEvo. Scores are unitless; admissions are records/run (manuscript Table 8; Appendix E.3).

MethodProxy-exploit admissions (records/run) ↓Proxy score (unitless) ↑Held-out score (unitless) ↑Δ held-out vs. PiEvo (unitless) ↑
Vanilla MAS4.67 ± 1.150.812 ± 0.0450.412 ± 0.061−0.023 ± 0.078
ReAct4.00 ± 0.820.785 ± 0.0380.428 ± 0.055−0.007 ± 0.073
PiEvo5.33 ± 0.580.894 ± 0.0210.435 ± 0.0480
AFE alone6.00 ± 0.000.932 ± 0.0150.408 ± 0.052−0.027 ± 0.071
Triage-PiEvo0.00 ± 0.000.748 ± 0.0290.644 ± 0.035+0.209 ± 0.054

Held-out score improves from PiEvo's 0.435 to 0.644, while proxy-exploit admissions fall from 5.33 to 0.00 records/run.

AuthorsOpenReview ↗

Response to Reviewer 7bsN(2/2) part2

Table R8: Removing verification components increases errors or reduces recall. Full TRIAGE uses two supporting records (manuscript Table 9; Appendix F).

VariantFalse revision (%) ↓Mechanism recall (%) ↑Retraction accuracy (%) ↑Verification cost (USD/run) ↓
Full TRIAGE4.8%92.4%93.5%$0.24
k=1k=1 (permissive)22.4%89.2%78.5%$0.14
k=3k=3 (strict)6.5%71.8%90.8%$0.38
Held-out verification removed34.5%83.5%61.2%$0.18
Same-source repeat used as independent45.8%78.0%42.1%$0.20
Channel reliability removed19.2%81.2%76.3%$0.22
Backtracking removed16.5%84.0%51.8%$0.21

Removing held-out testing raises incorrect revisions to 34.5%; removing backtracking lowers retraction accuracy to 51.8%.

Table R13 compares Base, Self-Review, and TRIAGE in three agents over five seeds, with equal time limits and USD 2.00/task caps. An external testbed measures Verified Score (Appendix H).

Table R13: Adding TRIAGE to external agents. AutoResearchEval optimization subset (manuscript Table 6); scores are unitless, rates in %.

Agent systemVariantTask score (unitless) ↑Verified Score (unitless) ↑False admission (%) ↓Revision accuracy (%) ↑
ResearchHarnessBase0.2860.23038.4%44.2%
+ Self-Review0.3010.24835.2%48.5%
+ TRIAGE0.3450.32814.2%68.4%
AI-ResearcherBase0.3320.28432.6%51.8%
+ Self-Review0.3480.30529.8%56.2%
+ TRIAGE0.3890.37111.8%73.6%
Claude CodeBase0.3480.29829.4%55.4%
+ Self-Review0.3650.32126.8%60.2%
+ TRIAGE0.4120.3959.6%78.2%

Compared with each agent’s Base configuration, adding TRIAGE improves Verified Score by 0.098 in ResearchHarness, 0.087 in AI-Researcher, and 0.097 in Claude Code, with increased token use and runtime.

Conclusion. TRIAGE reduces false admissions under harder feedback and consistently improves Verified Score across the three evaluated agent architectures, albeit with added token consumption and runtime. Correlated evaluator errors represent a known limitation, where mechanism recall reaches 27/40 compared to PiEvo’s 29/40.

Reviewer_rrHkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Triage-PiEvo to handle anomalous observations in scientific agents. The main idea is to distinguish unreliable feedback from genuine new discoveries before updating scientific principles. The authors introduce a three-stage verification mechanism and allow previously accepted principles to be retracted. Experiments on four scientific tasks and three LLM backbones show improvements over baselines.

Critical strengths
  1. The paper provides a clear and systematic way to handle anomalies. The three-stage verification mechanism is well motivated and addresses different sources of unreliable feedback.
  2. The experiments are fairly comprehensive, including multiple LLM backbones, corruption tests, ablations, and cross-harness evaluations.
Critical weaknesses
  1. The novelty of Evidence Admission is not entirely clear to me. There have been previous works on hypothesis verification and reliability-aware knowledge updates. For example, POPPER [1] already studies how to validate scientific hypotheses through sequential falsification. What is the key difference between TRIAGE and simply adding a verification step before updating the principles in PiEvo?
  2. The performance gains are difficult to attribute to TRIAGE alone, since the paper also introduces AFE for candidate exploration. Interestingly, Table 3 shows that Triage-only performs worse than the original PiEvo (60.60 vs. 64.78), while the full method only slightly outperforms PiEvo (65.98 vs. 64.78). Could the authors explain why the gains are much larger in Table 1? An ablation under the same evaluation budget would help.
  3. The verification stages C1–C3 seem to require additional evaluator calls. Are these counted in the evaluation budget of 24? If not, the comparisons with baselines may not be entirely fair. [1] Automated Hypothesis Validation with Agentic Sequential Falsifications
Questions for authors

See weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer rrHk(1/3)

Dear Reviewer rrHk,

We would like to express our great gratitude for your effort in reviewing our paper with constructive suggestions! We will answer each question respectively.

[W1] Comparing TRIAGE with ordinary verification and POPPER

POPPER validates hypotheses through sequential falsification. In contrast, TRIAGE governs when experimental evidence may update principles used in downstream experiments and retracts contradicted principles along with any rules derived from them (Section 3.4).

Experimental design. Table R7 tests whether improvements on the evaluator used during search also hold on data kept separate from search. Table R8 removes verification components to measure incorrect updates and retraction accuracy.

Table R7: Triage-PiEvo improves held-out performance over PiEvo. Scores are unitless; admissions are records/run (manuscript Table 8; Appendix E.3).

MethodProxy-exploit admissions (records/run) ↓Proxy score (unitless) ↑Held-out score (unitless) ↑Δ held-out vs. PiEvo (unitless) ↑
Vanilla MAS4.67 ± 1.150.812 ± 0.0450.412 ± 0.061−0.023 ± 0.078
ReAct4.00 ± 0.820.785 ± 0.0380.428 ± 0.055−0.007 ± 0.073
PiEvo5.33 ± 0.580.894 ± 0.0210.435 ± 0.0480
AFE alone6.00 ± 0.000.932 ± 0.0150.408 ± 0.052−0.027 ± 0.071
Triage-PiEvo0.00 ± 0.000.748 ± 0.0290.644 ± 0.035+0.209 ± 0.054

Triage-PiEvo improves held-out score from PiEvo's 0.435 to 0.644 while reducing proxy-exploit admissions from 5.33 to 0.00 records/run. AFE alone achieves the highest score on the search evaluator, but its held-out score is lower than PiEvo’s.

Table R8: Removing verification components increases errors or reduces recall. Full TRIAGE uses two supporting records (manuscript Table 9; Appendix F).

VariantFalse revision (%) ↓Mechanism recall (%) ↑Retraction accuracy (%) ↑Verification cost (USD/run) ↓
Full TRIAGE4.8%92.4%93.5%$0.24
k=1k=1 (permissive)22.4%89.2%78.5%$0.14
k=3k=3 (strict)6.5%71.8%90.8%$0.38
Held-out verification removed34.5%83.5%61.2%$0.18
Same-source repeat used as independent45.8%78.0%42.1%$0.20
Channel reliability removed19.2%81.2%76.3%$0.22
Backtracking removed16.5%84.0%51.8%$0.21

Removing held-out testing raises incorrect revisions to 34.5%; removing backtracking lowers retraction accuracy to 51.8%.

Conclusion. TRIAGE combines held-out checks with backtracking to reduce incorrect principle updates and retract outdated principles along with their dependent rules.

AuthorsOpenReview ↗

Response to Reviewer rrHk(2/3)&(3/3)

We thank the reviewer for this insightful suggestion.

[W2 & W3] Evaluating AFE and TRIAGE under primary-evaluation and total-call budgets

Triage-PiEvo improves mean SQ over PiEvo by 1.20 pp with eight primary evaluations and by 1.27 pp when verification is included in 24 total calls (Tables R5 and R3).

Experimental design. We compare the same four configurations on Qwen3-235B-A22B over four tasks and three seeds. Table R5 uses eight primary evaluations; Table R3 is a separate rerun with 24 total calls including verification. Manuscript Table 1 uses 24 primary evaluations, so its 11.75 pp gain belongs to a different protocol.

Table R5: AFE + TRIAGE leads mean SQ with eight primary evaluations. Qwen3-235B-A22B, four tasks, three seeds; verification is additional (manuscript Table 3).

VariantSQ (%) ↑AUOC (%) ↑Δ SQ vs. PiEvo (pp) ↑
PiEvo64.7858.15–
AFE alone62.7358.23−2.05
IDS + TRIAGE60.6055.08−4.18
Triage-PiEvo (AFE + TRIAGE)65.9860.45+1.20

Adding TRIAGE to AFE raises mean SQ by 3.25 pp, whereas adding TRIAGE to PiEvo’s IDS decreases mean SQ by 4.18 pp. AFE alone and IDS + TRIAGE both score below PiEvo in this experiment, while their combination ranks first.

Table R3 is a separate verification-inclusive rerun: PiEvo scores 71.73% mean SQ in this batch, versus 70.35% in the original manuscript Table 1.

Table R3: Triage-PiEvo improves SQ over PiEvo with verification included in 24 total calls. Same rerun batch; false revisions count incorrect principle updates.

VariantTotal calls (calls/run; fixed)Primary calls (calls/run; allocation)SQ (%) ↑AUOC (%) ↑False revisions (cases/500) ↓Δ SQ vs. PiEvo (pp) ↑
PiEvo (same batch)2424.0071.73 ± 3.4255.12 ± 2.631610
AFE alone2424.0072.40 ± 3.6555.94 ± 2.82172+0.67
IDS + TRIAGE2416.6770.00 ± 2.4354.76 ± 1.9074−1.73
Triage-PiEvo (AFE + TRIAGE)2416.6773.00 ± 3.4457.55 ± 2.7866+1.27

Adding TRIAGE to AFE raises mean SQ from 72.40% to 73.00% and reduces incorrect updates from 172 to 66 per 500 cases. Triage-PiEvo averages 16.67 primary and 7.33 verification calls, retaining a benefit while verification consumes part of the search budget.

Conclusion. Combining AFE and TRIAGE improves mean SQ over PiEvo across both evaluation budget regimes, including when verification calls count toward the 24-call limit.

Thanks for the additional experiments. The matched-budget ablation helps clarify the contributions of AFE and TRIAGE.

Reviewer_kV8qOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Triage-PiEvo, an extension of PiEvo that introduces a three-stage verification procedure to prevent incorrect scientific principle revisions caused by unreliable experimental evidence. It also uses Active Flow Expansion (AFE) for hypothesis selection and supports principle retraction. Experiments on scientific optimization benchmarks and synthetic corruption settings demonstrate improved performance and robustness.

Critical strengths

The paper addresses an important problem in automated scientific discovery. Scientific agents may incorrectly update their principles based on noisy or unreliable experimental observations. The proposed evidence-verification framework is a reasonable approach to addressing this problem. The synthetic corruption experiments also provide useful evaluations of the method's ability to avoid incorrect principle revisions.

Critical weaknesses
  1. The experimental results are inconsistent with the original PiEvo paper. Table 1 reports much lower performance for PiEvo than the original paper. For example, the average SQ drops from 90.81% with Qwen3-32B to 70.35% with Qwen3-235B. Although different LLM backbones are used, the benchmarks, evaluation budgets, and metrics appear to be the same. Moreover, the original PiEvo results are even better than the Triage-PiEvo results in this paper. The authors should explain these large discrepancies, as they raise concerns about the validity of the experimental comparisons.

  2. The construction of the anomaly sources and verification procedures is unclear. Section 3.1 introduces the anomaly sources (Z_art), but it is unclear how they are identified or used in the algorithm. More importantly, the implementation of C1–C3 is not sufficiently explained. For example, where do the independent evaluators required by C2 come from, and how are the held-out experiments for C3 constructed? It is also unclear whether these additional evaluations count toward the experimental budget.

  3. The role of mechanism probes in AFE is not well explained. Section 3.3 claims that mechanism probes help distinguish competing scientific principles, but it is unclear how disagreement between principles is actually measured. Moreover, the original PiEvo already uses IDS to select informative experiments, so it is unclear why AFE is more suitable for this purpose.

  4. The improvements cannot be clearly attributed to the proposed triage mechanism. The paper changes both the verification procedure and the search strategy compared with PiEvo. Interestingly, the ablation results show that triage without AFE performs worse than the original PiEvo, and AFE without triage also slightly reduces solution quality. Only their combination improves performance. The paper should better explain this interaction and clarify the contribution of the triage mechanism itself.

Questions for authors

I would like the authors to address the concerns raised in the weaknesses, particularly the large discrepancies between the PiEvo baseline results in Table 1 and those reported in the original paper, as this directly affects the validity of the experimental comparisons.

5 comments
AuthorsOpenReview ↗

Response to Reviewer kV8q(1/4)

Dear Reviewer kV8q,

We would like to express our great gratitude for your effort in reviewing our paper with constructive suggestions! We will answer each question respectively.

[W1] Reproducing PiEvo (Table 1) under the original backbone

Our PiEvo reproduction achieves 89.23% mean SQ on Qwen3-32B, compared with the published 90.81%. Adding TRIAGE raises mean SQ over the reproduced PiEvo baseline by 2.17 pp on Qwen3-32B. On Qwen3-235B-A22B, the complete Triage-PiEvo system (AFE + TRIAGE) raises mean SQ over PiEvo by 11.75 pp (Table R1). These comparisons demonstrate higher solution quality than PiEvo under matched primary-evaluation budgets across both Qwen backbones.

We compare (a) PiEvo's reproduction and the SQ gain with TRIAGE on each Qwen backbone in Table R1, and (b) baseline score changes across model settings and Triage-PiEvo's performance on each current backbone in Table R2.

(a) Reproducing PiEvo and evaluating TRIAGE on each Qwen backbone.

The larger-model experiments use Qwen3-235B-A22B-Instruct-2507, abbreviated as Qwen3-235B-A22B. Table R1 uses PiEvo's four tasks, SQ definition, three fixed seeds, and 24 primary evaluations per run. Verification calls are counted separately.

Table R1: PiEvo reproduction and results with TRIAGE on each Qwen backbone. Task scores are mean plus or minus standard deviation across three seeds. Mean SQ averages the four task means.

SQ (%); ↑ higher is better; ↓ lower is better. Differences between percentages are reported in pp (percentage points).

BackboneImplementationMBO SQ (%) ↑NHO SQ (%) ↑SPO SQ (%) ↑TMC SQ (%) ↑Mean SQ (%) ↑
Qwen3-32BPiEvo, published149.06 ± 16.0296.36 ± 1.7037.33 ± 1.3680.49 ± 1.9390.81
PiEvo, reproduction145.00 ± 16.5094.80 ± 1.9037.10 ± 1.3080.00 ± 2.0089.23
+ TRIAGE150.80 ± 15.0096.30 ± 1.5037.90 ± 1.2080.60 ± 2.1091.40
Qwen3-235B-A22BPiEvo97.84 ± 27.4965.53 ± 21.9037.71 ± 0.5380.30 ± 4.5770.35
Triage-PiEvo (AFE + TRIAGE)132.00 ± 9.8472.89 ± 17.8042.00 ± 1.2081.50 ± 4.0282.10

The configurations incorporating TRIAGE improve mean SQ and reduce MBO variability across both Qwen backbones. On Qwen3-32B, the reproduction is 1.58 pp below the published result; adding TRIAGE raises mean SQ by 2.17 pp over the reproduction and 0.59 pp over the published result. MBO's standard deviation falls from 16.50 to 15.00 pp. On Qwen3-235B-A22B, mean SQ rises from 70.35% to 82.10%, and MBO's standard deviation falls from 27.49 to 9.84 pp, a 64.2% relative reduction.

(b) Baseline score variation and Triage-PiEvo's gains on the same backbone.

Table R2 compares the published baseline panels with our Qwen3-235B-A22B and Gemini-2.5-Flash-Lite results. Within each current-model column, Triage-PiEvo and the baselines use the same tasks and evaluation budget.

Table R2: Published baseline scores and our results on different model backbones. Cross-paper differences equal our SQ minus published SQ. SQ is reported in percent; differences are in pp.

(i) Qwen comparison

MethodQwen3-32B, published SQ (%) ↑Qwen3-235B-A22B, this work SQ (%) ↑Δ SQ, current − published (pp)
Vanilla MAS52.1257.45+5.33
ReAct56.1961.93+5.74
AI Scientist v162.2653.97−8.29
AI Scientist v262.3860.42−1.96
AI Researcher63.2964.78+1.49
PiFlow70.0357.46−12.57
PiEvo90.8170.35−20.46
Triage-PiEvo–82.10–

(ii) Gemini comparison

MethodGemini-2.5-Flash, published SQ (%) ↑Gemini-2.5-Flash-Lite, this work SQ (%) ↑Δ SQ, current − published (pp)
Vanilla MAS62.1059.41−2.69
ReAct61.5654.27−7.29
AI Scientist v162.1260.68−1.44
AI Scientist v264.8264.93+0.11
AI Researcher63.6156.74−6.87
PiFlow71.0768.49−2.58
PiEvo93.1567.53−25.62
Triage-PiEvo–79.63–

Triage-PiEvo outperforms the strongest listed baseline on Qwen3-235B-A22B and Gemini-2.5-Flash-Lite. PiEvo's reported mean SQ decreases by 20.46 pp in the Qwen comparison and 25.62 pp in the Gemini comparison, the largest drops among the listed baselines. Within our Qwen3-235B-A22B experiments, Triage-PiEvo exceeds PiEvo by 11.75 pp. On Gemini-2.5-Flash-Lite, Triage-PiEvo exceeds PiFlow by 11.14 pp and PiEvo by 12.10 pp.

The verification-inclusive comparison is reported in Table R3 ([W2]).

Conclusion. Triage-PiEvo combines candidate selection with staged verification to improve discovery quality under the same primary-evaluation budget. The advantage holds against both the reproduced PiEvo baseline and the strongest listed baselines on the current backbones.

AuthorsOpenReview ↗

Response to Reviewer kV8q(2/4)

Thanks for your insightful advice!

[W2] Clarifying anomaly sources, verification checks, and evaluation costs

Triage-PiEvo improves mean SQ over PiEvo by 1.27 pp and reduces incorrect principle updates from 161 to 66 per 500 cases when search and verification share the same 24-call limit (Table R3).

Experimental design. Table R3 reports an additional experiment in which search and verification share a strict cap of 24 total calls per run. Unlike manuscript Table 1, Table R3 counts verification calls toward the evaluation budget. Its PiEvo baseline (71.73% mean SQ) is rerun under this matched setting.

AFE serves as our candidate-selection policy, whereas IDS is PiEvo’s; TRIAGE determines which experimental outcomes may update principles.

Verification checks and anomaly sources. C1 repeats the original measurement. C2 re-evaluates the same candidate using the highest-trust evaluator among the task’s declared independent channels (Algorithm 1). C3 evaluates held-out inputs that are fixed prior to the run and strictly isolated from search; its evaluator shares no memory, cache, or random seed with the generator (Definition 3.3). While anomaly-source taxonomy characterizes potential root causes of unexpected outcomes, TRIAGE relies on the operational check results (C1–C3) rather than raw source labels to authorize principle updates.

Table R3: Triage-PiEvo improves SQ over PiEvo with verification included in the 24-call budget. Same rerun batch; false revisions count incorrect principle updates.

VariantTotal calls (calls/run; fixed)Primary calls (calls/run; allocation)SQ (%) ↑AUOC (%) ↑False revisions (cases/500) ↓Δ SQ vs. PiEvo (pp) ↑
PiEvo (same batch)2424.0071.73 ± 3.4255.12 ± 2.631610
AFE alone2424.0072.40 ± 3.6555.94 ± 2.82172+0.67
IDS + TRIAGE2416.6770.00 ± 2.4354.76 ± 1.9074−1.73
Triage-PiEvo (AFE + TRIAGE)2416.6773.00 ± 3.4457.55 ± 2.7866+1.27

Adding TRIAGE to AFE raises mean SQ from 72.40% to 73.00% and reduces incorrect updates from 172 to 66 per 500 cases. Triage-PiEvo averages 16.67 primary and 7.33 verification calls, retaining a benefit while verification consumes part of the search budget. Verification costs USD 0.24 per run in the component ablation (manuscript Table 9).

Conclusion. In this comparison, TRIAGE improves mean SQ when paired with AFE but lowers it when paired with IDS.

AuthorsOpenReview ↗

Response to Reviewer kV8q(3/4)

[W3] Selecting experiments to distinguish competing principles and comparing with IDS

We use AFE to select promising candidates, explore new regions, and test experiments on which competing principles make different predictions. The complete Triage-PiEvo system improves AUOC over PiEvo by 9.63 pp on the Qwen3-235B-A22B backbone under the same primary-evaluation budget (Table R4).

We compare Triage-PiEvo with the highest-mean-SQ baseline on each backbone across four tasks, using 24 primary evaluations per run. Verification is counted separately.

AFE selects candidates for solution quality, exploration, and testing competing principles. For probe selection, each active principle provides a Gaussian Process (GP) predictive distribution over the candidate space. We quantify disagreement using posterior-weighted Jensen–Shannon divergence and select the highest-scoring candidate. Predictive distributions and principle weights are computed strictly from admitted primary-search history.

By contrast, PiEvo’s IDS balances expected regret and information gain through a single scalar criterion. When both configurations are paired with TRIAGE under an identical 24-call budget, AFE achieves 73.00% mean SQ compared to 70.00% for IDS (Table R3).

Table R4: Candidate diversity (APD) and search quality (AUOC) of Triage-PiEvo versus baseline. We compare the complete Triage-PiEvo system with the highest-mean-SQ baseline on each backbone. APD measures candidate diversity; AUOC summarizes best-so-far solution quality over the search budget. Values in arrow format represent Baseline → Triage-PiEvo.

BackboneBaselineAPD (unitless) ↑Δ APD ↑AUOC (%) ↑Δ AUOC (pp) ↑
Qwen3-235B-A22BPiEvo53.35 → 57.23+3.8862.10 → 71.73+9.63
Gemini-2.5-Flash-LitePiFlow50.64 → 55.88+5.2457.73 → 73.75+16.02
GPT-5.6-LunaReAct46.31 → 57.62+11.3160.61 → 71.92+11.31

As shown in Table R4, Triage-PiEvo improves both candidate diversity and solution quality during search. AUOC gains are 9.63 pp over PiEvo on Qwen3-235B-A22B, 16.02 pp over PiFlow on Gemini, and 11.31 pp over ReAct on Luna.

Conclusion. With TRIAGE held fixed, replacing IDS with AFE raises mean SQ by 3.00 pp under the same 24-total-call limit (Table R3).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.