- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
CARVE proposes a continual adaptation framework for pretrained policies when the provider’s training data are unavailable. It retains a frozen base model and immutable specialists, using local validation to authorize each specialist for specific tasks. The proposed reuse-first workflow checks existing specialists before training a new model from the base. The paper provides conditional expected-performance guarantees and sample-complexity bounds for reliable reuse. Experiments in crowd navigation and chip macro placement examine retention, cross-task reuse, and simulated transfer between operators.
Critical strengths
A clear protocol for retaining and reusing adaptations. The framework explicitly separates preserving a trained model from authorizing its use on a particular task. The reuse-first and receiving-operator experiments provide concrete examples of reusing existing specialists without source training data, with reduced new training in the recorded settings.
Critical weaknesses
The placement evaluation does not establish gains in complete, legal layouts. Appendix F states that IBM scoring is limited to the placed subset and that the evaluator performs no final overlap or boundary checks. Consequently, the reported HPWL improvements do not establish better complete, legal placements. This substantially limits the evidence for the main application.
The additional benefit over simple model caching is modest in the main revisit experiment. Cache-only achieves a 7.04% mean HPWL gain with 10.53 hours of repair work, compared with 8.41% and 10.36 hours for CARVE. Although the paper includes diagnostic analyses, the evidence does not convincingly establish how much the admission mechanism adds beyond saving and retrieving task-specific models.
The quality–cost benefit of reuse-first is not sufficiently isolated. Across the ten revisits to the five reused circuits, mean HPWL gain is 11.91% with reuse, compared with 17.33% with target-specific training. Training seeds and integration code also differ between the executions. These differences make it difficult to attribute the aggregate quality and cost changes specifically to the reuse-first rule.
Questions for authors
-
How do the main HPWL improvements change when evaluation covers all macros and includes final overlap and boundary checks? Please report the proportion of complete, valid placements alongside their placement quality.
-
When the trained candidates and evaluation cases are held fixed, how much does admission improve subsequent placement quality and reduce regressions relative to Cache-only? This would help establish the contribution of validation beyond model retention.
-
Can a comparison using identical integration code and matched training seeds confirm the reported quality–repair-cost trade-off? For this controlled comparison, please distinguish outcomes on reused circuits from gains due to newly trained specialists.
8 comments
Reply to Reviewer wu64 (Part 2 of 3)
3. Controlled comparison of train-each and reuse-first (Weakness 3)
W3: The quality–cost benefit of reuse-first is not sufficiently isolated. Across the ten revisits to the five reused circuits, mean HPWL gain is 11.91% with reuse, compared with 17.33% with target-specific training. Training seeds and integration code also differ between the executions. These differences make it difficult to attribute the aggregate quality and cost changes specifically to the reuse-first rule.
Thank you for highlighting this concern. The circuit replay uses identical code, specialist files and evaluation seeds for both rules, so only the rule differs. It trains no new models; training times come from E2's records.
We apply both rules to each of three specialist sets: A from CARVE's E2 training, B from Cache-only's, and C from RS's. All specialists used the same trainer and budget and started from the frozen base, with different training seeds across sets. A set is only a pool of specialists, not a method. Reuse-first keeps E2-X's Holm correction and selection rule.
Table R2. Mean gain over the 18 requests on circuits, after completion and legalization..
| Specialist set | Mean gain: train-each | Mean gain: reuse-first | Train-each minus reuse-first (points) | Recorded training time saved |
|---|---|---|---|---|
| A | 9.78% | 7.33% | 2.45 | 42.7% |
| B | 7.84% | 7.18% | 0.66 | 61.8% |
| C | 6.64% | 5.12% | 1.52 | 42.6% |
Including failed attempts, the training savings are 42.7%, 58.1% and 50.7% for A, B and C, respectively. Under the original scorer, train-each exceeds reuse-first by 4.20, 1.95 and 3.05 points, respectively.
Thank you for highlighting this trade-off. The controlled replay confirms this cost and shows where it comes from:
- Both rules use the base for each circuit's first request, before any training or reuse decision.
- When both rules train a circuit, the replay uses the identical specialist and admission decision. Their later results therefore match, including any fallback to the base.
- Only revisits served by another circuit's specialist under reuse-first can differ. On those revisits, train-each uses the circuit's own specialist if admitted, or the base otherwise.
Table R3. Mean HPWL gain on the revisits that reuse-first serves with another circuit's specialist.
| Specialist set | Circuits using another circuit's specialist | Mean gain: train-each | Mean gain: reuse-first |
|---|---|---|---|
| A | ibm15, ibm13, bigblue1 | 16.75% | 9.39% |
| B | ibm18, ibm15, bigblue1, ibm10 | 12.33% | 10.85% |
| C | ibm13, bigblue1, ibm10 | 12.74% | 8.18% |
All other requests have identical outcomes under both rules. Each gap in Table R2 is therefore the Table R3 difference spread over all 18 requests; for set A, (16.75 − 9.39) × 6 / 18 = 2.45 points.
Reply to Reviewer wu64 (Part 3 of 3)
4. Complete, valid placements and placement quality (Q1)
Q1-1: How do the main HPWL improvements change when evaluation covers all macros and includes final overlap and boundary checks?
Thank you for the question. On the same 18 requests, CARVE keeps the highest mean gain: 9.78% under the new evaluation and 11.22% under the original scorer. Table R1 gives all four methods; the ranking is unchanged.
Q1-2: Please report the proportion of complete, valid placements alongside their placement quality.
Thank you for this request. Before completion, only bigblue1's 450 trials contained every macro. No raw layout had a macro-macro or macro-fixed-object overlap, consistent with ChiPFormer's position mask (ChiPFormer paper, Sec. 3.3). However, 295 layouts contained at least one macro crossing the boundary: 162 on ibm07 and 133 on ibm15. Neither ChiPFormer nor MaskPlace addresses these crossings.
After completion and legalization, 2,700 of 2,700 layouts (100%) are complete and legal, as confirmed by a separate geometry checker. Per method, the share of complete, legal layouts rises from 90 of 540 (17%, all on bigblue1) to 540 of 540. Table R1 reports their quality.
5. Admission with fixed candidates and evaluation cases (Q2)
Q2: When the trained candidates and evaluation cases are held fixed, how much does admission improve subsequent placement quality and reduce regressions relative to Cache-only? This would help establish the contribution of validation beyond model retention.
Thank you for suggesting this comparison. We keep Cache-only's specialists and all evaluation trials fixed. Admission rejects only ibm18's specialist: its admission-test gain is −2.77%, with p = 0.956. Its revisits then use the base.
Mean gain over the 12 revisits rises from 10.86% to 11.75%, by 0.90 points before rounding. The number of revisits worse than the base falls from 2 to 0; the worse of the two had been −7.05%. Testing takes 8 minutes. We store 5 specialist files instead of 6 (−16.7%). Across the three specialist sets defined in Section 3 above, admission rejects 3 of 18 trained specialists. In these runs, the 15 admitted specialists beat the base on all 30 of their later revisits; over circuits, the 21 admitted specialists beat it on all 42.
6. Controlled quality–cost comparison and reused-circuit outcomes (Q3)
Q3-1: Can a comparison using identical integration code and matched training seeds confirm the reported quality–repair-cost trade-off?
Thank you for the question. The controlled replay confirms the trade-off and measures its size under a common implementation: across the three sets, reuse-first saves 43–62% of recorded training time (mean 49%) at a cost of 0.66–2.45 percentage points of mean HPWL gain (mean 1.54). This confirms the trade-off stated in Section 3.2 under a common implementation. With set A, repair time falls from 7.60 h to 4.19 h. Completion and legalization add under one minute, which these times do not include.
Q3-2: For this controlled comparison, please distinguish outcomes on reused circuits from gains due to newly trained specialists.
Thank you for requesting this distinction. Only the revisits in Table R3 can differ. Circuits that both rules train have identical outcomes.
Reply to Reviewer wu64 (Part 1 of 3)
Thank you for the careful review. We address placement validity, admission beyond caching, and the controlled quality–cost comparison below.
1. Complete and legal placements (Weakness 1)
Thank you for the comment. ChiPFormer's policy places 256 modules per IBM circuit (ChiPFormer paper, App. A.1 and Table 5). Our census counts modules taller than the 16-unit standard cells and finds 285–786 macros per IBM circuit.
Every method now uses the same procedure. Completion places the missing macros greedily on the policy's grid without moving policy-placed macros. We then legalize each layout and score every net that ChiPFormer retains for macro placement. A separate geometry checker verifies every final layout.
We note that the relevant results are given below. The gains persist on complete, legal layouts, and the ranking is unchanged (Table R1). CARVE beats the base on 12 of 18 requests, ties on 6, and loses none. The ties are each circuit's first request, which CARVE serves with the base.
Gains are relative to the frozen base, the released ChiPFormer checkpoint. Table R1 reports all four methods under both scorers.
Table R1. Mean HPWL gain over the frozen base, averaged over the requests shown.
2. Admission beyond model caching (Weakness 2)
Thank you for this observation. Table R1 gives both methods on circuits under both scorers. On complete, legal layouts, CARVE reaches a 9.78% mean gain with 7.60 h of repair, and Cache-only 7.24% with 7.86 h.
We note the following distinction. The totals above compare different trained specialists. To measure admission alone, we hold Cache-only's specialists and evaluation trials fixed and add only the test (Q2). Admission then raises the mean revisit gain by 0.90 points and removes every revisit worse than the base (from 2 to 0).