- Limited statistical evidence and unclear evaluation sets.
I couldn't find explicit evaluation-task counts for each condition. The paper also does not report independent repetitions of design search or training, or confidence intervals. Judging from the scores, ClawEval appears to be an 81-task set, so one task is worth about 1.23 points. If that is right, the ranking reversal in Table 2 is a one-task gap before training and a one-task gap after, at least on ClawEval (on QwenClawBench the gaps are 0.22 and 0.60 points). Several of the ablation differences are about the same size.
A related issue is that not all ClawEval rows seem to come from the same evaluation set. The OPD+GRPO scores (36.02/67.08) don't fit 81 tasks but match 58/161 and 108/161, and the SFT scores (27.99/56.85) fit neither 81 nor 161. The paper doesn't explain this. Without uncertainty estimates and clarification of the evaluation-task counts, I find it hard to assess the reliability and comparability of these results.
- The ClawEval OPD gain is confounded.
According to Table 5 and Appendix B, the OPD model was evaluated with twice the task-time limit of the selected design. Since timeouts count as failures, part of the jump from 32.10 to 35.80 could come from the extra time rather than from training alone. The authors mention this in the Limitations section, but it's easy to miss when reading the main results. To be fair, the paper reports that on development tasks, with the doubled limit both before and after training, rubric scores still go up (61.53 → 64.92). I would still like to see the test-set number under the original time limit.
- It isn't clear that multiple agents actually help.
The selected system ends up being a single agent on TeamBench and a two-role system on ClawEval. In Appendix D.2, simply giving the parent agent the same extra compute matches the conditional-role system on ClawEval (35.80) and comes very close on QwenClawBench (37.28 vs. 37.42). There is also no single-agent + OPD baseline on ClawEval or QwenClawBench.
TeamBench doesn't help settle this either. The benchmark is meant to test coordination when the Planner, Executor and Verifier roles are kept separate, but here it is run with the full-access interface. Table 1 also places the paper's strict-verdict scores next to published full-team scores that use a different scoring rule. So I can't tell how much of the improvement comes from the multi-agent structure itself.
- Missing implementation details and baselines.
The paper never says which LLM is used as the designer, which matters a lot since the designer is the core of the method. Search budgets, stopping criteria, training hyperparameters and compute costs are also missing or only partly given. There is no comparison with existing automated design methods under a matched budget, and there are very few concrete examples of generated tasks, selected designs or revision records. As it stands, the method would be hard to reproduce, and it is hard to judge what each part contributes.
The cross-harness transfer experiment closely resembles the setup in OpenForgeRL's Section 5.2, which also compares ZeroClaw-only training against pooled ZeroClaw, OpenClaw and Codex training, but OpenForgeRL isn't cited in that section. Although the models and training methods differ, the paper does not clearly explain what additional insight this experiment provides beyond the earlier study. Related work on automated prompt and topology search for multi-agent systems (MASS), joint optimization of model roles and weights (Heterogeneous Swarms), and joint optimization of prompts and model parameters in compound AI systems (Optimas) is also not discussed. A clearer comparison would help establish what is new in ResidualForge.