- The limited and non-monotonic training trajectory does not support a strong recursive-self-improvement claim
Bank construction is run for only five rounds on GPQA, mathematics, and Biomni, and for ten rounds on CMT-Bench. In Biomni, each round executes the same panel of 30 adaptation tasks, yielding 150 completed episodes over five rounds, but only five outer retained-state updates are observed. The full-rubric success rate—the fraction of the 30 tasks receiving a rubric score of exactly 100—is 6.7%, 13.3%, 16.7%, 13.3%, and 20.0% across the five rounds, corresponding to approximately 2, 4, 5, 4, and 6 full-credit tasks. The corresponding mean training scores, including zeros for incomplete tasks, are 67.37, 61.10, 52.43, 65.63, and 62.13. Although Round 5 produces the largest number of full-credit episodes, its mean score is below that of Round 1 and slightly below the Round-5 MemoHarness score.
These results show that the bank and reusable module collection grow, and that the final checkpoint improves some metrics. However, they are insufficient to establish whether the improvement loop can sustain net gains over longer horizons, how or when its performance stabilizes, or how regressions are controlled.
The paper does include admission mechanisms. Algorithm 1 states that a Controller proposal must pass an evaluator or canary gate, while Algorithm 2 performs guarded admission based on conflicts, dependencies, and compatibility. The latter is specified deterministically, but the former is not: the paper does not provide the benchmark-specific evaluation data, acceptance rule, or minimum improvement threshold, nor does it explain how rejected proposals are handled or how checkpoints are retained, selected, or rolled back when performance deteriorates.
There is also an apparent protocol inconsistency. Section 4.6 states that the Biomni training panel is run for ten rounds, whereas Figure 7 reports only Rounds 1–5, Appendix D.2 explicitly describes 150 episodes from 30 tasks over five rounds, and Appendix D.3.1 likewise defines a single five-round training run. The paper does not state that only the first five of ten rounds are shown.
- The initialization, generation, and selection of modules are not specified well enough for reproduction
The paper describes module updating as extracting typed patches from episodes. Formally, however, Extract compares the evaluated configuration Hqeval with its predecessor Hqprev and stores the changed configuration fields as modules. It does not derive a new prompt or strategy from the reasoning trajectory itself; the new field value must already have been proposed by the Controller, supplied as an initial seed, or produced by another candidate-generation mechanism.
The AIME case study shows a Context instruction and token-limit patch originating in Round 1 and a temperature patch originating in Round 2, but it does not explain how the Round-1 candidates were generated. The initial contents of B0, M0, and C0 are also not specified. If both B0 and M0 are empty, the first-round Top-K recall over Mt∪Bt has no candidates. Figure 9 further notes that the Controller’s proposal-generation rule is unspecified. Consequently, the process that creates the semantic content of new modules is not reproducible from the paper.
Even assuming that candidate modules have already been generated, their selection is also underspecified. The recall score is defined as a weighted combination of task fit, base-harness fit, cooperation evidence, and compatibility, but the paper does not provide benchmark-specific definitions of these components, their weights, or the candidate-pool sizes. The paper therefore lacks executable definitions for two central stages: how candidate modifications are generated and how they are selected. This makes it difficult to determine whether the reported gains arise from a general compositional-learning mechanism or from unreported Controller prompts, Recall implementations, and other engineering choices.
- The practical efficiency and long-term scalability of per-task dynamic composition are not established
During held-out evaluation, CoRSI freezes the contents of B, M, and C, but it does not freeze a single harness that can be executed repeatedly. According to Algorithm 1, every new task requires candidate scoring, Top-K recall, Planner compilation, participant materialization, and workflow execution. The deployment pattern is therefore not “learn one harness and reuse it,” but “retrieve and compile a task-specific workflow for every input.” Per-task retrieval is not inherently unreasonable in a task-conditioned system, but it makes retrieval, planning, and composition recurring online costs for every request.
This concern becomes increasingly important as the number of recursive rounds and retained modules grows. Recall is defined by scoring candidates from Mt∪Bt and selecting the Top-K. The value Kt bounds the number of candidates ultimately exposed to the Planner, but it does not by itself bound the cost of finding those candidates in a continually growing bank and module catalog. The paper does not provide retrieval complexity or describe the use of indexing, pre-filtering, approximate retrieval, candidate deduplication, or workflow caching. It also does not report aggregate per-task Planner, participant, and synthesis calls, token consumption, wall-clock latency, or monetary cost.
The available case studies indicate that the online execution structure can be substantial. For example, HMMT task 3 uses a Planner call, three participant calls, and a final merge. Without systematic efficiency measurements, the existing results do not establish the practical performance–cost trade-off of per-task dynamic composition or whether the approach remains usable with more recursive rounds, larger module catalogs, and large-scale task streams.
- BiomniBench-DA lacks component ablations for the mechanisms most relevant to tool-using workflows
The paper does include a meaningful range of ablations. On GPQA, the complete system outperforms four ablated conditions, and the AIME/HMMT experiments examine cooperation memory, cross-source recombination, multi-participant workflows, artifact handoff, and synthesis. Therefore, the paper should not be characterized as lacking ablations in general.
However, these systematic ablations are concentrated on GPQA and mathematics. BiomniBench-DA is the setting in which Tools, Memory, Verification, artifact routing, and multi-participant collaboration should matter most, yet this section primarily reports aggregate scores, round-wise checkpoints, rubric decompositions, and case studies rather than corresponding component ablations. The cases demonstrate how the system can work, but they do not establish whether these mechanisms are individually necessary for the biomedical results. Ablating at least one of cooperation memory, module-guided workflow construction, artifact handoff, or multi-participant execution would substantially strengthen the mechanism attribution in this setting.
Thank you for the detailed rebuttal and additional experiments. The empty-bank control, cross-source recombination ablation, and inference cost comparisons meaningfully address my concerns about mechanism attribution. I appreciate these efforts and consider this concern substantially alleviated.
However, some reservations remain. First, the gains attributable to compositional reuse are relatively modest (4–5 percentage points), and stronger statistical evidence would help establish their robustness. Second, while the additional baseline comparisons are useful, the methodological novelty still appears somewhat incremental, and matching evolution rounds does not necessarily ensure comparable search budgets. Finally, the discrepancies between the original algorithm description and the actual implementation raise concerns about method specification and reproducibility.
Overall, the rebuttal strengthens the empirical contribution, but the remaining concerns prevent me from changing my overall assessment at this stage.