- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a framework for improving recursive self improvement agents by diagnosing underlying workflow failures rather than repeatedly applying superficial fixes. The approach detects three types of anomalies: recurring failures, opposing edits, and expectation mismatches. It riggers investigations that formulate hypotheses, execute diagnostic probes, and preserve evidence across optimization rounds. The resulting evidence guides subsequent workflow modifications. Experiments on Lean theorem proving and Verus verifiable code generation report improvements over naive RSI.
Critical strengths
-
the work builds a practical and interpretable framework by combining anomaly detection, hypothesis testing, executable probes, and persistent investigation memory to provide a coherent approach to workflow improvement. The distinction between testing a hypothesis and merely evaluating a proposed edit is useful.
-
it shows promising empirical improvements where the it reported gains across two formal verification settings. Ablation studies and behavioral analyses provide additional evidence that diagnostic probing and persistent memory can improve workflow optimization.
Critical weaknesses
-
the experiments compare primarily against Naive RSI. Stronger comparisons, such as an always-on diagnostic agent, a reflection-based RSI agent, or a HarnessFix-style diagnostic approach, would help determine whether the improvements come from the proposed second-order framework or simply additional structured reasoning.
-
evaluation has very limited scale as it uses only 48 Lean and 46 Verus held-out tasks, three independent runs, and one outer improvement model. Moreover, aggregate token costs are not reported, limiting the analysis of computational efficiency.
-
the core ideas of diagnosing failures, executing tests, and maintaining experience are individually established in prior work, particularly HarnessFix and AutoManual. The main novelty appears to be their integration with selective anomaly detection, but the paper does not sufficiently isolate the contribution of this selection mechanism.
Questions for authors
-
How does SO-RSI compare with stronger diagnostic baselines? Would always performing root-cause investigation, without anomaly-triggered selection, achieve comparable results under the same budget?
-
Since mechanism classification and interpretation involve LLM reasoning and human auditing, could the authors report inter-annotator agreement and provide concrete examples where diagnostic probes successfully distinguish competing explanations?
-
How sensitive is performance to the anomaly detection strategy? An ablation separating recurrence, opposing edits, and expectation mismatch would clarify which signals contribute most to the improvements.