Key reasons for my recommendation: (1) The reported numbers cannot be reconciled with the stated size of the hidden evaluation set (30 tasks), several metrics are undefined for some baselines, and the promised significance tests are not reported. (2) The comparison is not fair: the injected exploit types match VRQ's allowed challenge families one-to-one, the baselines do not receive this list, and compute budgets are not matched. In addition, the red-team fitness in Eq. (4) may induce the collusion the paper aims to prevent.
1. Reported numbers are incompatible with the evaluation set size. This is the most serious issue, and it affects trust in all results.
- Section 4.1 and Appendix B.1 state that the hidden set has 30 tasks. Appendix B.2 defines Pass as the percentage of tasks where all mandatory hidden assertions pass. Under this definition, a single-seed Pass must be a multiple of 1/30 (3.33 pp granularity).
- The single-seed Pass values in Table 10 are 67.1, 69.6, 68.3, 69.4, and 67.6. None is a multiple of 1/30 (for example, 67.1% × 30 = 20.13 tasks). None is a multiple of 1/120 either (67.1% × 120 = 80.52).
- The Figure 3 caption says "Hidden-test performance on 120 verifier-vulnerable skill-synthesis tasks". This conflicts with the 30-task hidden set. If Pass is computed on all 120 tasks, 70 of them are evolution tasks, which is train-test leakage.
- Table 4 splits hidden Pass into 5 task families. With about 6 tasks per family, single-seed granularity is about 16.7 pp. At this granularity, standard deviations of 1.5–2.1 across 5 seeds (for example, Scientific Computing 70.1±1.5) are statistically very unlikely. Table 5 is more extreme: about 12 of the 48 controlled tasks fall in the hidden set, so each exploit class has about 3 tasks, yet the standard deviations are 1.1–1.4.
- Recomputing sample standard deviations from Table 10 gives values that differ from the reported ones: Pass 1.09 (reported 1.2), Gap 0.87 (reported 0.8), VSS 1.47 (reported 1.6).
Fix: state the exact denominator of Pass (tasks × instances? multiple seed-specific instantiations per task?), give raw counts of passed tasks per method and per seed, and explain the "120" in Figure 3. If hidden metrics include evolution tasks, re-report all results on the 30 hidden tasks only.
2. Significance tests are not reported, and the main margins are small.
- Section 4.1 and Appendix B.2 say that a 10,000-sample paired bootstrap is used, with a 95% CI excluding 0 as the reliability criterion. No table or text reports any CI or p-value.
- The margin over the strongest baseline, Verifier-Ensemble, is 3.9 pp (Table 1). The authors note that VRQ wins on only 4 of 5 seeds (Section 4.2). On software engineering the margin is 2.1 pp, and the ensemble is better on one seed. With 30 tasks and 5 seeds, these margins may not be significant.
- All 5 seeds share one fixed 70/20/30 split. The variance reflects only sampling randomness and ignores split variance, which may be much larger with 30 hidden tasks.
Fix: report paired bootstrap 95% CIs for each main comparison (VRQ vs. Verifier-Ensemble, VRQ vs. CoEvoSkills, and each ablation vs. the full model). Add results over several random splits (for example, 5-fold or 3 re-splits).
3. The red-team fitness in Eq. (4) penalizes challenges that expose flaws in the strongest solver.
- In Eq. (4), red-team fitness is m̄ · [q(c; S⁺_z) − mean q(c; S)], where S⁺_z is the current highest-fitness solver.
- If S⁺_z itself exploits a shortcut, a challenge that correctly exposes the shortcut gives q(c; S⁺_z)=0. The first term is 0 and the second term is non-negative, so fitness ≤ 0, and the challenge is eliminated.
- As a result, the red-team population is rewarded for protecting the current best solver. This is the skill–verifier collusion the paper aims to avoid. Since the fitness of S⁺_z is in turn set by red-team challenges (Eq. 3), the two form a closed loop.
- Section 3.1 requires calibrator validity to mean "satisfiability by an admissible solution". This is a different condition from "the current best solver passes". The paper treats the two as the same.
Fix: discuss this mechanism and test it. Count how many eliminated red-team challenges actually expose a real hidden-test failure of S⁺_z. Compare with a variant that uses reference execution, instead of S⁺_z, as the satisfiability criterion.
4. Several metrics are undefined for baselines, and metric definitions are inconsistent.
- Life, Div., and Coll. are undefined for non-evolving baselines. No-Skill performs no evolution, yet Table 1 reports Life=18.4 generations, Div.=0.18, and Coll.=0.31. Best-of-N and Iterative-Refine use one fixed verifier, so pairwise disagreement between verifiers (Div., Appendix B.2) cannot be computed.
- Inv. is defined inconsistently. Appendix B.2 defines Inv. as the fraction of challenges rejected by reference execution or invariant checks (a rejection rate). Section 4.3 says VRQ "retains 3.6% invalid challenges" (a retention rate). VRQ's Inv.=3.6 in Table 2 equals the Invalid Retention=3.6 of Full calibration in Table 9. Iterative-Refine generates no challenges, yet has Inv.=5.1 and Cov.=43.7.
- The scope of FPR is unclear. Table 5 covers only the 4 exploit classes of the 48 controlled tasks. Yet the class averages for the 4 methods in Table 5 (21.4, 17.9, 13.2, 8.7) are identical to the FPR values in Table 1, which cover all tasks including the 72 naturally underspecified ones. This coincidence across all four methods is unlikely. Please state whether Table 1 FPR uses only controlled tasks.
- "Hidden test" has two meanings. FPR is defined as the fraction of hidden failures accepted during optimization, and Coll. also refers to "hidden evaluation". This requires running protected tests on evolution tasks. The paper should separate "per-task protected reference tests" from "the 30-task hidden split" and state which task set each diagnostic uses.
- VSS is a conditional ratio. VSS is the fraction of previously successful Skills that stay correct after a verifier change. The denominator differs across methods, and methods with fewer successes have a selection bias. Please also report the unconditional pass rate after the verifier change.
5. The benchmark's exploit types match VRQ's challenge families one-to-one.
- The 48 controlled tasks inject 4 exploit types: output memorization, partial-execution spoofing, invariant violation, and adversarial tool output (Section 4.1).
- VRQ's Red-Team is explicitly allowed these challenge families: metamorphic / input perturbation, partial-execution audit, invariant check, and adversarial tool response (Section 3.1;
{allowed_challenge_families} in Appendix D.6.1). The two lists correspond almost exactly.
- The baseline verifier prompts (Appendix D.4.4, D.4.5) do not receive this list. VRQ therefore has prior knowledge of the test distribution that the baselines lack.
- The "unseen assertions, perturbations, and tool-output formats" in the verifier-shift experiment (Table 2) also fall within these families.
Fix: (a) give Verifier-Ensemble and CoEvoSkills the same challenge-family list and re-compare; (b) add exploit types that VRQ's red-team does not cover, to test generalization to unknown exploits; (c) report results separately on the 72 naturally underspecified tasks, which are less coupled to the method design.
6. Baselines are weak and compute budgets are not matched.
- The paper says all methods use the same interaction budget and that baselines "receive neither ... additional search budget" (Section 4.1). However, Table 2 shows VRQ's execution cost is 1.37× that of Iterative-Refine and 1.16× that of CoEvoSkills. Section 4.3 says the main overhead comes from executing challenges. This suggests challenge execution is not counted in the 640-interaction budget. Please state exactly how the budget is counted.
- Table 8 partly addresses this (VRQ at 75% budget reaches 66.7, above Verifier-Ensemble at 100% budget with 64.5). But Table 8 measures interactions, and Table 2 measures Cost, so Table 8 does not establish a cost-matched result.
- Verifier-Ensemble has 2 verifiers. VRQ has 6 Red-Team and 6 Calibrator individuals plus a hall of fame. The gap may come mainly from the number of verifiers and calls.
Fix: add (a) a larger Verifier-Ensemble (for example, 6–12 verifiers) matched to VRQ's actual cost; (b) Verifier-Ensemble with the same calibrator; (c) a static red-team that generates a challenge set once without co-evolution, to test whether co-evolution itself is needed; (d) CoEvoSkills with a hall of fame. Plot Pass against actual cost.
7. The ablations do not isolate the claimed mechanisms, and implementation descriptions conflict.
- Synchronous vs. asynchronous updates. Appendix A says fitness is fixed within generation g: "Updating an earlier population does not trigger recomputation of generation-g fitness". Algorithm 1 follows this, which is synchronous updating. Section 4.1 and Appendix B.3 say the default is asynchronous: "each validated challenge can immediately influence subsequent solver selection within the same generation". It is therefore unclear how the "synchronous updates" ablation in Table 3 differs from the full model.
- w/o hall-of-fame and ρ=0 give identical numbers. Table 3 w/o hall-of-fame (63.5 / 12.9 / 65.4) matches Table 6 ρ=0 (63.5 / 12.9 / 65.4) on every value. Appendix D.9.2 says the ablation also removes the archive summary from the prompt, and final selection (Eq. 9, Algorithm 1 line 21) is also affected. ρ=0 only turns off replay in opponent sampling. Two different operations giving identical results means at least one description is wrong, or both are the same run.
- w/o calibrator mixes two effects. Without the calibrator, invalid challenges enter both solver fitness (Eq. 3) and red-team fitness (Eq. 4). The drop could come from invalid-challenge noise or from a changed red-team direction. I suggest a variant that uses the calibrator for solver fitness only.
- Missing co-evolution ablations. There is no variant with a frozen Red-Team population or a frozen Calibrator population. Without these, one cannot tell whether adversarial co-evolution itself helps, or whether the gain comes from more and more diverse challenges.
- No direct evidence that collapse is mitigated. The central claim is that baselines undergo co-evolutionary collapse. Please show per-generation proxy fitness and hidden Pass curves (on validation tasks or protected tests of evolution tasks), showing that CoEvoSkills diverges and VRQ does not. At present, the three failure modes in contribution 1 are described only in words, with no formal definition and no observed instance of collapse.
8. Hyperparameter selection may have used hidden results.
- Appendix B.1 says the validation set is used to choose global hyperparameters and the stopping criterion.
- The sensitivity analyses in Table 6 (ρ) and Table 7 (Red-Team population size) report hidden-test values. The best values are exactly the defaults (ρ=0.30, N_R=6), and they are best on all four metrics (Pass, FPR, VSS, Life) at once.
- Please report these sensitivity results on the validation set to show the defaults were chosen there. The stopping rule "select the latest generation whose pass rate does not decrease" also needs a precise definition.
9. The scope of a Skill is unclear, which affects how "hidden-task generalization" should be read.
- Appendix B.1 says that for each task the agent "must produce a reusable Skill", with "at most 640 environment interactions per task". This suggests per-task evolution.
- Section 1 says Solver Skills "solve the target task family", and Section 4.1 says Skills evolve on the 70 evolution tasks and are evaluated on the 30 hidden tasks.
- If Skills are per-task, where do the Skills for hidden tasks come from? If evolution also runs on hidden tasks, this contradicts "cannot access the hidden partition". If Skills are per-family, please explain how a Skill transfers from about 14 evolution tasks to about 6 hidden tasks within a family.
Fix: state the Skill granularity (per-task / per-family / global) and which Skill is executed on each hidden task.
10. The calibrator's supervision is circular.
- Eq. (6) trains the calibrator with executable evidence u(c, z). But u includes "agreement among independent semantic checks" (Section 3.2), which are also judgments by the same Qwen3-32B. The calibrator and its supervision come from one model and may share biases.
- If reference execution or invariants already provide u, the added value of an LLM calibrator needs a separate justification.
- "Full calibration" in Table 9 is not defined, and its difference from "Two-judgment agreement" is unclear. For the 400 human-labeled challenges, the number of annotators, inter-annotator agreement (for example, Cohen's κ), and the source method of the challenges are not reported. It is also unclear whether the "five independently sampled evaluation sets" are resamples of the 400 items or 5 separate sets.
11. A single backbone, described inaccurately as "frontier".
- Section 4.1 refers to "the same frontier instruction-tuned model", and Appendix B.3 shows it is Qwen3-32B. This is not a frontier model.
- All roles (Solver, Red-Team, Calibrator, execution) use the same model. This amplifies shared blind spots, which is the very problem the paper studies.
- Please reproduce the main table on at least one stronger model and one model from a different family. I also suggest a setting where verifiers and solvers use different models.
Minor issues
- Citation errors. Several references do not match the cited content. Awais et al. (2008) is an IP multimedia subsystem security paper, cited for novelty search (Lehman & Stanley, 2011 is appropriate). Klingenberg et al. (2018) is a finite-volume scheme paper for Euler equations, cited for quality-diversity (Pugh et al., 2016 is appropriate). Camero et al. (2018) "DLOPT" is cited as DeepHunter (Xie et al., ISSTA 2019). Pardo et al. (2017) "Time limits in reinforcement learning" is only weakly related to "exploit incomplete reward functions". Please check all references.
- Missing related work: Prover-Verifier Games (Anil et al., 2021; Kirchner et al., 2024); reward model overoptimization (Gao et al., 2023); co-evolution of code and unit tests (CodeT, Chen et al., 2022; CURE, Wang et al., 2025); PSRO (Lanctot et al., 2017); Pareto coevolution and coevolutionary pathologies (Ficici & Pollack; the Popovici et al. survey). The paper should also position VRQ against classic competitive co-evolution and hall of fame (Rosin & Belew, 1997), league training (AlphaStar), metamorphic testing, and LLM-as-judge filtering, since each component has precedent.
- Eq. (2) is not a fixed objective. Q(z) is induced by archived red-teams, and the archive depends on the evolution trajectory of S. Relatedly, the AI Tools Usage statement mentions "theoretical analysis", but the paper contains none, and Section 3.3 states that replay "does not guarantee convergence".
- Missing tercile table (Section 4.3). The text reports Gap reductions of 14.2 in the highest tercile and 3.1 in the lowest. The overall reduction is 16.8 − 6.1 = 10.7. With equal-size terciles, the middle tercile would need a reduction of about 14.8, larger than the highest tercile. This contradicts the monotonic claim. Please add the full table.
- Unclear representation of verifier individuals. Is a Red-Team or Calibrator individual a prompt, a template, or executable code? How are Calibrators mutated? Appendix B.3 describes mutation operators only for Solvers and Red-Teams.
- Overstated conclusions. Section 5 uses "established" and "confirming that the proposed design enables reliable autonomous Skill evolution". With one model, 30 hidden tasks, and no reported significance tests, this wording is too strong.