- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
VERA constructs certified recovery tasks from verified circuits, trains on selected actions from successful learner rollouts, and uses the updated learner to find stronger-specification seeds for subsequent rounds. Internal success increases from 0/180 to 77/180 after one update and 123/180 after three, with improvements on two external benchmarks.
The recovery-task formulation is useful, and the action-selection results are encouraging. My main concern is that the recursive experiments leave task quality, data volume, and computational cost partly entangled.
Critical strengths
-
The paper addresses a real self-training bottleneck: complete-task successes may be too scarce to provide useful supervision. Certified recovery tasks offer a practical starting point, and the private recovery record keeps feasibility checking separate from learner supervision.
-
The comparison with success-trajectory SFT is informative. Using the same successful episodes, selected-action training reaches 77/180 versus 47/180 when supervising all actions. This supports filtering within successful trajectories, although supervision volume also differs. The equal-data and equal-collection controls further support recovery-based training on easy tasks.
-
Fixed-bank and frozen-search controls provide useful evidence beyond successive checkpoints. External transfer and the negative result for early expansion also strengthen the empirical study.
Critical weaknesses
- The recursive gains need better resource controls.
At the third update, full recursion reaches 123/180, versus 97/180 for fixed-bank recollection and 103/180 for frozen search. Cumulative collection also differs: 1,566, 1,404, and 1,446 rollouts, respectively, with different selected-action pool sizes. Search, construction, and certification costs are not included.
Producing more useful data may itself be a benefit of task renewal. Still, the current comparison does not show whether recursion improves performance at a fixed budget, or how much its advantage depends on additional collection and training. A matched-budget comparison and full cost breakdown would substantially strengthen the central claim.
- Evaluation diversity and training stability remain uncertain.
The internal benchmark contains 18 tasks, but all six tasks within a family share one starter netlist. It therefore offers limited evidence about generalization across starting topologies. Each condition also uses a single training seed; repeated inference rollouts do not establish stability across training runs.
The external results help, but their scope matters. On PhyChip-v2, first-update success is 72/84 on execution-based L1 tasks and 11/66 on performance-graded L2/L3 tasks. These demonstrate different capabilities and should remain separate when interpreting transfer. AnalogCoder-Pro evaluation covers only the 20-task executable subset.
- The added value over simpler recovery curricula is unclear.
The existing controls test fixed-bank recollection and frozen-model search, but do not cover expanding the perturbation bank without tightening specifications or adapting recovery difficulty alone. Such comparisons would help determine whether a stronger specification search is needed, rather than task diversity or difficulty adjustment more generally.
Questions for authors
-
Does the recursive advantage persist with matched collection budgets and controlled training tokens or optimization steps? What is the total search and certification cost?
-
Can the key comparisons be repeated across training seeds, with uncertainty estimates accounting for repeated rollouts within tasks?
-
How similar are training seeds and held-out starters? Would the gains persist with additional starting topologies?
-
How does target expansion compare with an adaptive recovery curriculum that retains the original specifications?
2 comments
Response to K9B4 (2/2): Topology Transfer, External Evaluation, and Adaptive Controls
3. Transfer to independent starting topologies
Q3. Do gains persist with independent starting topologies rather than shared family-level starts? (W2; Question 3)
The original 18 tasks test transfer across specifications from shared starts. All 6 tasks within a family use D01 (Section 4.1, Appendix A.5 and Table 9, pp. 19–20), with possible training–evaluation topological similarity noted in Appendix A.2.
We evaluated existing checkpoints on 6 independent topology groups (18 tasks), checking graph isomorphism, ancestry and local structural similarity. The new starts have no isomorphic or ancestral match in the training archive. Performance continues to improve, with full VERA gaining 11.8 percentage points over the fixed bank (Table 6, p. 9, 8 seeds).
| Evaluation set | Fixed bank | Full | Paired gain |
|---|---|---|---|
| New topologies (6 groups, 18 tasks) | 47.9 ± 2.5% | 59.7 ± 2.2% | +11.8 pp, 95% CI [10.5, 13.1] |
Table 24 (p. 34) also gives the base model's 0.0% and the first-update model's 36.2 ± 3.6%. Appendix G.6 describes the checks, and Figure 11 (p. 34) reports each topology group's performance.
Revision: Table 6 and Appendix G.6 (Table 24; Figure 11) report independent starting topologies, structural/lineage audits and transfer across 8 training seeds.
4. External evaluation criteria and task coverage
Q4. How should PhyChip execution and performance gains, and the AnalogCoder-Pro subset, be interpreted? (W2)
The paper separates PhyChip execution from performance and specifies the AnalogCoder-Pro subset. Table 12 (p. 25) reports L1 execution and L2/L3 performance separately, with performance gains discussed in Section 5 (p. 10). Appendix D identifies the 20 AnalogCoder-Pro tasks with complete assets (IDs 1–8 and 10–21), shared by all models.
Appendix D (p. 25) now clarifies that Table 1, Figure 6(b) (p. 8) and Table 13 (p. 26) pool PhyChip L1 execution with L2/L3 performance. Performance-transfer claims use the separate criteria. Performance success also improves from 0/66 to 11/66.
| PhyChip level | L1 (execution) | L2 (performance) | L3 (performance) | L2+L3 |
|---|---|---|---|---|
| Base | 8/84 | 0/45 | 0/21 | 0/66 |
| First-update model | 72/84 | 9/45 | 2/21 | 11/66 |
AnalogCoder-Pro retains the same task set and denominator. The new VREF/LDO/FLT tests (Tables 6 and 24) provide repeated-run evidence of cross-family transfer, complemented by results from the external subset.
Revision: §5 and Appendix D (Table 12) distinguish execution from performance and specify the AnalogCoder-Pro subset; Tables 6 and 24 add cross-family transfer results.
5. Adaptive renewal at unchanged specifications
Q5. Does stronger-specification search add value over an adaptive recovery curriculum at unchanged specifications? (W3; Question 4)
The paper already tests the search model and the timing of target expansion. Frozen search holds the initial model fixed for target search, with other settings unchanged, reaching 103/180 versus full VERA's 123/180 at update 3 (Table 17, p. 27). The V/H experiment shows that expansion timing matters (Table 4, p. 9). Same-specification adaptive renewal is the additional control in the revision.
The same-specification branch uses the updated learner to find more verified designs within the original bounds. These seed new exercises, with splitting and perturbation weakening enabled. It therefore controls for design diversity, task renewal and difficulty adaptation, with tightening target specifications as the sole difference. Renewal at the original bounds already helps. Stronger-specification seeds add a further 7.8 percentage points, 95% CI [6.4, 9.2] (Table 5, p. 9, 8 seeds).
| Budget setting | Same-specification renewal (same-spec) | Full |
|---|---|---|
| Fixed collection/SFT quotas | 60.3 ± 3.6% | 68.1 ± 3.1% |
| Common verifier cap | 58.6 ± 3.3% | 66.3 ± 3.6% |
Table 19 (p. 29) and Figure 8 (p. 30) give second-update results and training trajectories. The strong hybrid adds a full-method comparison (Table 21, p. 31).
Disabling splitting and weakening gives 64.0 ± 2.8%, versus 68.8 ± 2.3% for full VERA, confirming the benefit of adaptive practice (Table 7, Appendix G.5). The same-specification branch combines seed renewal and difficulty adjustment, directly testing adaptive renewal within the original specification bounds.
Revision: Table 5 and Appendix G.2 (Table 19; Figure 8) evaluate same-specification renewal; Table 7 and Appendix G.5 test adaptation, and Table 21 adds the strong hybrid.
Response to K9B4 (1/2): Budget-Matched Recursion and Statistical Reliability
We thank the reviewer for the feedback. The paper reports collection volume, defines aggregation, separates external evaluation criteria and includes frozen search. The revision adds fixed-budget recursion, independent training runs, new starting topologies and same-specification adaptive renewal to address the remaining questions.
1. Fixed-budget recursion and full costs
The paper explicitly reports resource differences between recursive branches. Section 4.5 and Table 18 give cumulative attempts of 1,566 / 1,404 / 1,446 and action pools of 1,650 / 1,480 / 1,410, excluding construction and certification. The revision extends that resource accounting to fixed-budget comparisons.
We fix collection/SFT quotas or impose a common verifier cap covering search, failed proposals, construction, certification, calibration and collection. Cumulative caps are 3,600, 7,300 and 11,600 calls (Appendix G.2). Including search and certification, full VERA retains a consistent lead of about 13 percentage points over the fixed bank (Table 5, p. 9, 8 seeds).
Actual cumulative calls at the third update are reported in Table 20 (p. 31).
Figure 8, right (p. 30), plots results by cumulative cap. The appendix (p. 30) also breaks down full VERA's second-update task-production cost into 1,412 search calls, 336 construction/calibration calls and 424 certification calls.
Revision: Table 5 and Appendix G.2 (Tables 19–20; Figure 8) report joint budget matching, cumulative verifier caps and the full task-production cost breakdown.
2. Training stability and the statistical unit
The paper defines task-level aggregation, and the revision adds independent training runs. Originally, each condition had one training run (seed 42), with a checkpoint fixed by training length. Success was averaged over 10 rollouts per task, then equally over 18 tasks (Appendix A.3, p. 18). These repeated rollouts characterize inference variability.
We use 8 independent training seeds for the primary comparisons and 5 of them for component and prior-method comparisons, each with independent collection and model updates. Branches are paired within seed. Each checkpoint's 10 rollouts per task are averaged first, followed by equal task weighting. Mean and sample SD are computed across training seeds. Paired intervals and randomization tests also use training seeds as the statistical unit, and the main conclusion holds (Appendix G.1, p. 28).
Table 1 (p. 6) retains the single-run success counts, labeled separately from repeated-training statistics.
Revision: Table 5 and Appendices A.3 and G.1 specify task-level aggregation, independent training runs, paired bootstrap intervals and corrected significance tests.