- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes TRIAGE-PiEvo, a framework designed to improve the reliability of principle-evolving scientific agents under noisy or unreliable experimental feedback. The authors introduce an evidence-admission mechanism that separates observations used for exploration from evidence authorized to revise persistent principles. The authors evaluate the approach on four scientific discovery tasks using three LLM backbones.
Critical strengths
- The motivation is well-defined and the paper captures an important issue
- The framework is reasonable and established, and the writing is clear and easy-to-follow
Critical weaknesses
- Limited novelty: The problem is important but the verification modules are combination of exsiting methods. The contribution of the work mainly focuses on applying the methods to auto research domain. Furthermore, I would like to see the results of sota LLMs since auto research is an inherently difficult task, small GPT variant like luna may not perform well on this task. We do not know whether the methods generalize to stronger LLMs.
- Unclear performance attribution and evaluation fairness: Table 3 shows that neither module independently outperforms PiEvo, and it is unclear whether additional verification calls are included in the evaluation budget. Limited generalization of robustness claims: The strongest results (0% false admission and 100% mechanism recall) come from constructed stress tests with predefined failure modes and strong planted mechanisms.
Questions for authors
None
3 comments
Response to Reviewer 7bsN(2/2) part1
Thank you for raising this important concern.
[W2] Testing TRIAGE under harder feedback and in external agent systems
Triage-PiEvo retains a 1.27 pp mean SQ advantage over PiEvo when verification is included in 24 total calls. Beyond predefined artifacts, TRIAGE reduces false admissions under weaker signals and unfamiliar faults, although correlated evaluator errors reduce mechanism recall.
Experimental design. We examine component interactions and call budgets first, then harder feedback, and finally transfer to other agent systems.
(a) Comparing module contributions and evaluation budgets.
Table R5: AFE + TRIAGE leads mean SQ with eight primary evaluations. Qwen3-235B-A22B, four tasks, three seeds; verification is additional (manuscript Table 3).
| Variant | SQ (%) ↑ | AUOC (%) ↑ | Δ SQ vs. PiEvo (pp) ↑ |
|---|---|---|---|
| PiEvo | 64.78 | 58.15 | – |
| AFE alone | 62.73 | 58.23 | −2.05 |
| IDS + TRIAGE | 60.60 | 55.08 | −4.18 |
| Triage-PiEvo (AFE + TRIAGE) | 65.98 | 60.45 | +1.20 |
The combined configuration ranks first; adding TRIAGE improves AFE but lowers IDS performance in this eight-primary-evaluation experiment.
Table R3: Triage-PiEvo improves SQ over PiEvo with verification included in 24 total calls. Same rerun batch; false revisions count incorrect principle updates.
| Variant | Total calls (calls/run; fixed) | Primary calls (calls/run; allocation) | SQ (%) ↑ | AUOC (%) ↑ | False revisions (cases/500) ↓ | Δ SQ vs. PiEvo (pp) ↑ |
|---|---|---|---|---|---|---|
| PiEvo (same batch) | 24 | 24.00 | 71.73 ± 3.42 | 55.12 ± 2.63 | 161 | 0 |
| AFE alone | 24 | 24.00 | 72.40 ± 3.65 | 55.94 ± 2.82 | 172 | +0.67 |
| IDS + TRIAGE | 24 | 16.67 | 70.00 ± 2.43 | 54.76 ± 1.90 | 74 | −1.73 |
| Triage-PiEvo (AFE + TRIAGE) | 24 | 16.67 | 73.00 ± 3.44 | 57.55 ± 2.78 | 66 | +1.27 |
With verification included in the 24-call budget, Triage-PiEvo improves mean SQ from PiEvo’s 71.73% to 73.00% and reduces incorrect updates from 161 to 66 per 500 cases.
(b) Testing beyond predefined artifacts and strong mechanisms.
Table R11 summarizes the controlled tests from Appendix E; Table R12 tests weaker mechanisms, withheld fault families, and correlated evaluator errors.
Table R11: Controlled artifact and planted-mechanism tests. A: false-admission rate (%, ↓); B: recall (%, ↑). Planted effects are at least (manuscript Table 7).
| Panel and metric | Test condition | Test cases (count) | PiEvo (%) A ↓; B ↑ | AFE alone (%) A ↓; B ↑ | Triage-PiEvo (%) A ↓; B ↑ |
|---|---|---|---|---|---|
| A: false-admission rate ↓ | Transient noise, dropout (Section 3.2 C1) | 24 | 100.0% | 100.0% | 0.0% |
| A: false-admission rate ↓ | Channel-specific fault (Section 3.2 C2) | 24 | 100.0% | 100.0% | 0.0% |
| A: false-admission rate ↓ | Surrogate, OOD, drift (Section 3.2 C3) | 48 | 100.0% | 100.0% | 0.0% |
| B: mechanism recall ↑ | Discontinuous step shift | 72 | 84.7% | 88.9% | 100.0% |
| B: mechanism recall ↑ | Synergistic interaction peak | 72 | 81.9% | 86.1% | 100.0% |
Triage-PiEvo rejects all 96 predefined artifacts and recovers all 144 strong planted mechanisms under these controlled conditions.
Table R12: TRIAGE versus PiEvo under harder feedback. Entries give false admissions or recovered mechanisms as count/total.
| Condition | PiEvo false admissions (count/total) ↓ | TRIAGE false admissions (count/total) ↓ | PiEvo recall (count/total) ↑ | TRIAGE recall (count/total) ↑ |
|---|---|---|---|---|
| Weak mechanisms | 24/60 | 11/60 | 40/60 | 42/60 |
| Withheld fault families | 18/40 | 10/40 | 30/40 | 30/40 |
| Correlated evaluator errors | 19/40 | 15/40 | 29/40 | 27/40 |
Under weak mechanisms, false admissions fall from 24/60 to 11/60; under withheld faults, they fall from 18/40 to 10/40. With correlated errors, false admissions fall from 19/40 to 15/40, but mechanism recall falls from 29/40 to 27/40.
(c) Checking independent performance and transfer to other agents.
Table R7: Triage-PiEvo improves held-out performance over PiEvo. Scores are unitless; admissions are records/run (manuscript Table 8; Appendix E.3).
| Method | Proxy-exploit admissions (records/run) ↓ | Proxy score (unitless) ↑ | Held-out score (unitless) ↑ | Δ held-out vs. PiEvo (unitless) ↑ |
|---|---|---|---|---|
| Vanilla MAS | 4.67 ± 1.15 | 0.812 ± 0.045 | 0.412 ± 0.061 | −0.023 ± 0.078 |
| ReAct | 4.00 ± 0.82 | 0.785 ± 0.038 | 0.428 ± 0.055 | −0.007 ± 0.073 |
| PiEvo | 5.33 ± 0.58 | 0.894 ± 0.021 | 0.435 ± 0.048 | 0 |
| AFE alone | 6.00 ± 0.00 | 0.932 ± 0.015 | 0.408 ± 0.052 | −0.027 ± 0.071 |
| Triage-PiEvo | 0.00 ± 0.00 | 0.748 ± 0.029 | 0.644 ± 0.035 | +0.209 ± 0.054 |
Held-out score improves from PiEvo's 0.435 to 0.644, while proxy-exploit admissions fall from 5.33 to 0.00 records/run.
Response to Reviewer 7bsN(2/2) part2
Table R8: Removing verification components increases errors or reduces recall. Full TRIAGE uses two supporting records (manuscript Table 9; Appendix F).
| Variant | False revision (%) ↓ | Mechanism recall (%) ↑ | Retraction accuracy (%) ↑ | Verification cost (USD/run) ↓ |
|---|---|---|---|---|
| Full TRIAGE | 4.8% | 92.4% | 93.5% | $0.24 |
| (permissive) | 22.4% | 89.2% | 78.5% | $0.14 |
| (strict) | 6.5% | 71.8% | 90.8% | $0.38 |
| Held-out verification removed | 34.5% | 83.5% | 61.2% | $0.18 |
| Same-source repeat used as independent | 45.8% | 78.0% | 42.1% | $0.20 |
| Channel reliability removed | 19.2% | 81.2% | 76.3% | $0.22 |
| Backtracking removed | 16.5% | 84.0% | 51.8% | $0.21 |
Removing held-out testing raises incorrect revisions to 34.5%; removing backtracking lowers retraction accuracy to 51.8%.
Table R13 compares Base, Self-Review, and TRIAGE in three agents over five seeds, with equal time limits and USD 2.00/task caps. An external testbed measures Verified Score (Appendix H).
Table R13: Adding TRIAGE to external agents. AutoResearchEval optimization subset (manuscript Table 6); scores are unitless, rates in %.
| Agent system | Variant | Task score (unitless) ↑ | Verified Score (unitless) ↑ | False admission (%) ↓ | Revision accuracy (%) ↑ |
|---|---|---|---|---|---|
| ResearchHarness | Base | 0.286 | 0.230 | 38.4% | 44.2% |
| + Self-Review | 0.301 | 0.248 | 35.2% | 48.5% | |
| + TRIAGE | 0.345 | 0.328 | 14.2% | 68.4% | |
| AI-Researcher | Base | 0.332 | 0.284 | 32.6% | 51.8% |
| + Self-Review | 0.348 | 0.305 | 29.8% | 56.2% | |
| + TRIAGE | 0.389 | 0.371 | 11.8% | 73.6% | |
| Claude Code | Base | 0.348 | 0.298 | 29.4% | 55.4% |
| + Self-Review | 0.365 | 0.321 | 26.8% | 60.2% | |
| + TRIAGE | 0.412 | 0.395 | 9.6% | 78.2% |
Compared with each agent’s Base configuration, adding TRIAGE improves Verified Score by 0.098 in ResearchHarness, 0.087 in AI-Researcher, and 0.097 in Claude Code, with increased token use and runtime.
Conclusion. TRIAGE reduces false admissions under harder feedback and consistently improves Verified Score across the three evaluated agent architectures, albeit with added token consumption and runtime. Correlated evaluator errors represent a known limitation, where mechanism recall reaches 27/40 compared to PiEvo’s 29/40.
Response to Reviewer 7bsN(1/2)
Dear Reviewer 7bsN,
Thank you for your effort in reviewing our paper with constructive suggestions! We will answer each question respectively.
TRIAGE controls when experimental evidence may update principles and retracts contradicted principles together with any rules derived from them. On Claude Sonnet 5.5, Triage-PiEvo achieves 81.10% mean SQ versus PiFlow’s 80.60%, with verification included in the 24-call budget (Table R9).
Experimental design. Table R9 compares methods under 24 total calls, including verification. Table R10 summarizes manuscript Table 1, which uses 24 primary evaluations with verification counted separately.
Evaluating Sonnet and harder Luna tasks.
Table R9: Additional-model and harder-task comparisons. Verification counts toward 24 total calls; mean SQ averages four tasks.
On the harder Luna tasks, mean SQ improves by 1.08 pp over PiEvo. Gains vary by task: TMC decreases by 0.60 pp on Sonnet relative to PiFlow and by 0.20 pp on Luna relative to PiEvo.
Table R10: Triage-PiEvo leads mean SQ on all three backbones. Best baseline is selected per column; range is maximum minus minimum, in pp.
In the original comparisons, Triage-PiEvo exceeds the strongest baseline by 11.75 pp on Qwen3-235B-A22B, 11.14 pp on Gemini, and 7.48 pp on Luna.
Conclusion. Triage-PiEvo achieves higher mean SQ than the strongest listed baseline on each of the original three backbones. With verification included in 24 calls, the additional comparisons show gains of 0.50 pp over PiFlow on Sonnet and 1.08 pp over PiEvo on the harder Luna tasks.