- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper learns compact, editable rule sets for frozen language models in modified environments, using a fixed rule budget and aggregation across independent training runs.
Critical strengths
The learned rules are inspectable and editable, and the experiments explicitly evaluate extrapolation to new environments.
Critical weaknesses
-
The headline comparison does not match total training budgets. The 93.25% result merges five constitution runs, whereas GEPA’s 91.71% summarizes individual-run performance. A single constitution achieves 86.67%. These results do not establish an accuracy advantage at equal total training cost.
-
The distinct contribution is insufficiently isolated. Section 2 acknowledges prior methods with bounded rule sets and merging/deletion, including AutoManual. No comparison matching both the rule budget and training budget establishes the benefit of the proposed mechanism over these closest alternatives.
Questions for authors
At equal total training calls and rule budgets, how does the method compare with aggregated GEPA-Axiomatic and a closely matched rule-learning baseline?
Weaknesses 1 and 2 (and Question): Total Training Budgets Don't Match and Additional Baselines
To match both the rule cap (P=10) and the total student call budget (400 calls for 1 run and 2,000 calls for a 5-run aggregate), we ran the requested AutoManual (Chen et al., 2024; re-implemented with P=10), 5-run Merged GEPA, and 5-run Ensembled GEPA-Axiomatic experiments—plus our Ensembled Constitution approach—on a newer model (Gemini 3.6 Flash,
MEDIUMthinking). Single-run GEPA, GEPA-Axiomatic, and Constitution rows from Table 1 are included for reference:Matched 5×400-call ensembling vs. GEPA and GEPA-Axiomatic (W1 & Q1): Evaluated on Gemini 3.6 Flash at the matched 2,000-call budget, our label-free consensus selection across 5 constitution runs reaches 95.0% overall (93.3%–97.5% across all five environments). This outperforms Merged GEPA (93.9%, +1.1 pp) and beats Ensembled GEPA-Axiomatic on all five environments (92.5%, +2.5 pp). Across 4-run leave-one-out subsets, consensus selection averages 93.4 ± 1.3% overall; we will add this matched 2,000-call comparison and LOO analysis to Section 4.2 and run multiple independent 5-run ensembles in the revision.
Matched P=10 rule cap and training budget vs. AutoManual (W2 & Q1): At 400 calls, AutoManual is ahead of a single constitution on three of the four navigation environments (92.2% vs. 87.8% mean) with much lower variance. At 2,000 calls on Gemini 3.6 Flash, Ensembled AutoManual remains slightly ahead on three navigation environments (No Left Turns 96.8% vs. 93.3%, Arterials 97.8% vs. 97.5%, Neo-Manhattan 94.0% vs. 93.5%), while Ensembled Constitution leads on Prohibited Avenues (95.8% vs. 89.3%) and averages higher across all four navigation environments (95.0% vs. 94.4%). The largest separation is on Kepler-Newton, where AutoManual stays near zero-shot (52.5% single-run and 60.3% ensembled, vs. 49.0% zero-shot) while our constitution reaches 82.3% single-run and 94.8% ensembled, giving 95.0% vs. 87.6% overall: without a single-clause constraint, AutoManual’s builder packs coordinate-specific lookup tables into 3–4 long multi-clause slots rather than consolidating them into the 1/r2 law. AutoManual was re-implemented in our harness (Builder + Consolidator
Rule_Manager, Gemini 3.6 FlashMEDIUM, 2 epochs × 20 batches of 10 = 400 student calls/run, rule cap lowered from 12 to P=10 without slot examples).