acceptodds
Under review as a conference paper at ICLR 2027

Aligning to New World Models Using Constitutional AI

Abstract

As large language models (LLMs) have scaled, so has the quality of their implicit world models: an internal understanding of an environment. However, there are environments in which even frontier LLMs don't work correctly out of the box, which we call new world models. We ask how to align a frozen LLM with a new world model using a small set of solved training problems and no weight updates. We build five such environments from Manhattan routing and orbital mechanics, on which a frontier LLM fails to get more than half of the evaluation correct. We use sleep-time compute: before test time, the model solves training problems, checks its answers against a simulator, and revises a constitution, a list of single-clause rules that we place in its prompt. While prior work uses constitutions to align models with human values, we repurpose them to instead state the governing rules of an environment. We add a fixed slot budget to makes rules compete for space, and an ensemble procedure to account for variance across independent runs. Ensembled constitutions improve accuracy by 1.8 over zero-shot and outperform few-shot prompting, an RL-trained advisor, and Inverse Constitutional AI. An ensembled constitution also slightly outperforms unconstrained prompt optimization (GEPA) with a 3.9 shorter prompt. Taken together with the advantages of constitutions, such as interpretability and editability, our results show that constitutions are an effective method of aligning frozen LLMs for new world models.

Primary area:
interpretability and explainable AI
Keywords:
Constitutions, Alignment, World Models, Foundation Models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_5SjiOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper learns compact, editable rule sets for frozen language models in modified environments, using a fixed rule budget and aggregation across independent training runs.

Critical strengths

The learned rules are inspectable and editable, and the experiments explicitly evaluate extrapolation to new environments.

Critical weaknesses
  1. The headline comparison does not match total training budgets. The 93.25% result merges five constitution runs, whereas GEPA’s 91.71% summarizes individual-run performance. A single constitution achieves 86.67%. These results do not establish an accuracy advantage at equal total training cost.

  2. The distinct contribution is insufficiently isolated. Section 2 acknowledges prior methods with bounded rule sets and merging/deletion, including AutoManual. No comparison matching both the rule budget and training budget establishes the benefit of the proposed mechanism over these closest alternatives.

Questions for authors

At equal total training calls and rule budgets, how does the method compare with aggregated GEPA-Axiomatic and a closely matched rule-learning baseline?

1 comment
AuthorsOpenReview ↗

Weaknesses 1 and 2 (and Question): Total Training Budgets Don't Match and Additional Baselines

W1: The headline comparison does not match total training budgets...

W2: The distinct contribution is insufficiently isolated. Section 2 acknowledges prior methods with bounded rule sets and merging/deletion, including AutoManual...

Q1: At equal total training calls and rule budgets, how does the method compare with aggregated GEPA-Axiomatic and a closely matched rule-learning baseline?

To match both the rule cap (P=10P=10) and the total student call budget (400 calls for 1 run and 2,000 calls for a 5-run aggregate), we ran the requested AutoManual (Chen et al., 2024; re-implemented with P=10P=10), 5-run Merged GEPA, and 5-run Ensembled GEPA-Axiomatic experiments—plus our Ensembled Constitution approach—on a newer model (Gemini 3.6 Flash, MEDIUM thinking). Single-run GEPA, GEPA-Axiomatic, and Constitution rows from Table 1 are included for reference:

MethodCallsNo LeftProhibited AvesArterialsNeo-ManhattanKepler-Newton5-Env Mean
AutoManual (1 Run, P=10P=10)40092.6 ± 4.0%87.7 ± 4.4%94.0 ± 8.2%94.7 ± 0.6%52.5 ± 11.7%84.3 ± 3.1%
GEPA-Axiomatic (1 Run, P=10P=10)40092.0 ± 2.1%95.3 ± 1.4%90.8 ± 5.9%91.7 ± 0.3%85.2 ± 14.4%91.0 ± 3.1%
GEPA (1 Run, Free-form)40091.4 ± 3.9%94.4 ± 1.7%93.4 ± 6.0%90.8 ± 0.5%88.7 ± 9.4%91.7 ± 2.4%
Single Constitution (1 Run, P=10P=10)40083.5 ± 20.3%88.8 ± 11.0%87.1 ± 17.6%91.7 ± 1.1%82.3 ± 19.0%86.7 ± 6.9%
Ensembled AutoManual (5 Runs, P=10P=10)2,00096.8%89.3%97.8%94.0%60.3%87.6%
Merged GEPA (5 Runs, Free-form)2,00094.5%94.8%92.8%91.5%96.0%93.9%
Ensembled GEPA-Axiomatic (5 Runs, P=10P=10)2,00090.8%92.0%94.5%92.0%93.0%92.5%
Ensembled Constitution (5 Runs, P=10P=10)2,00093.3%95.8%97.5%93.5%94.8%95.0%
  1. Matched 5×400-call ensembling vs. GEPA and GEPA-Axiomatic (W1 & Q1): Evaluated on Gemini 3.6 Flash at the matched 2,000-call budget, our label-free consensus selection across 5 constitution runs reaches 95.0% overall (93.3%–97.5% across all five environments). This outperforms Merged GEPA (93.9%, +1.1 pp) and beats Ensembled GEPA-Axiomatic on all five environments (92.5%, +2.5 pp). Across 4-run leave-one-out subsets, consensus selection averages 93.4 ± 1.3% overall; we will add this matched 2,000-call comparison and LOO analysis to Section 4.2 and run multiple independent 5-run ensembles in the revision.

  2. Matched P=10P=10 rule cap and training budget vs. AutoManual (W2 & Q1): At 400 calls, AutoManual is ahead of a single constitution on three of the four navigation environments (92.2% vs. 87.8% mean) with much lower variance. At 2,000 calls on Gemini 3.6 Flash, Ensembled AutoManual remains slightly ahead on three navigation environments (No Left Turns 96.8% vs. 93.3%, Arterials 97.8% vs. 97.5%, Neo-Manhattan 94.0% vs. 93.5%), while Ensembled Constitution leads on Prohibited Avenues (95.8% vs. 89.3%) and averages higher across all four navigation environments (95.0% vs. 94.4%). The largest separation is on Kepler-Newton, where AutoManual stays near zero-shot (52.5% single-run and 60.3% ensembled, vs. 49.0% zero-shot) while our constitution reaches 82.3% single-run and 94.8% ensembled, giving 95.0% vs. 87.6% overall: without a single-clause constraint, AutoManual’s builder packs coordinate-specific lookup tables into 3–4 long multi-clause slots rather than consolidating them into the 1/r21/r^2 law. AutoManual was re-implemented in our harness (Builder + Consolidator Rule_Manager, Gemini 3.6 Flash MEDIUM, 2 epochs × 20 batches of 10 = 400 student calls/run, rule cap lowered from 12 to P=10P=10 without slot examples).

Reviewer_tNpdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper adapts a frozen LLM to unfamiliar environment rules by inducing a compact natural-language constitution from simulator feedback. A four-stage loop audits errors and updates a fixed number of rule slots, followed by label-free selection across independent runs. Evaluation covers four navigation environments and one orbital-direction benchmark, with analyses of rule capacity, token usage, composition, and transfer to a smaller model.

Critical strengths
  1. The evaluation includes meaningful transfer across task categories. Training and testing use different task families, including held-out road closures, longer routes, and perturbed trajectories (Appendix A.1). The substantial gains over zero-shot and ten-shot prompting show that compact learned rules can support useful adaptation without weight updates (Table 1).

  2. The capacity and failure analyses are informative. The different effects of rule capacity on grid navigation and orbital prediction are worth understanding (Sec. 4.4). The paper also documents failures under composition and transfer to a smaller model (Appendix F), helping clarify when concise rules are sufficient and when more explicit procedural guidance is needed.

Critical weaknesses
  1. The methodological advance over existing rule-learning approaches is not sufficiently established. WALL-E already learns rules to align LLM predictions with environment dynamics. AutoManual diagnoses rule failures and updates or consolidates a bounded rule set, while RIMRULE compresses rules extracted from failures. These connections are acknowledged in Sec. 2, but none of these methods is evaluated. The remaining distinctions are mainly the single-clause constraint, staged slot updates, and selection across independent runs. The capacity sweep is useful, but does not establish the benefit of the proposed audit/edit/commit design over simpler bounded rule updating. This leaves the contribution insufficiently distinguished from existing approaches.

  2. The strongest baseline comparison does not control for the aggregate adaptation budget. The headline 93.25% result merges five constitution runs, each using 400 student training calls, whereas GEPA’s 91.71% is the mean of individual 400-call runs (Secs. 3.3 and 4.1; Table 1). At 400 calls per run, the single constitution reaches 86.67%, below both GEPA and GEPA-Axiomatic (90.97%). There is no comparison with a larger GEPA optimization budget or an equivalent merge of GEPA-Axiomatic rules. Uncertainty across independently formed ensembles is also not reported. Consequently, the 1.54-point margin does not establish a reliable advantage over the strongest baseline. The shorter prefix is useful, but total tokens per query remain similar to GEPA’s (2,153 versus 2,068; Fig. 2).

Questions for authors

Comparative evidence: Does the advantage persist with matched aggregate training budgets and the same five-run selection procedure applied to GEPA-Axiomatic? Variation across independently formed ensembles would help establish the reliability of the reported margin.

Implementation clarification: Appendix B.2 explicitly rejects rules containing “or,” yet Figure 6 reports “Never turn East or West from Fifth Avenue onto a cross street” as a retained final rule. Which validator was actually enforced in the reported experiments? This matters because the grammatical constraint underlies the claimed structural distinction and capacity analysis.

Benchmark clarification: In Appendix A.2, why is West a legal answer when arriving North at W 59th St and 6th Ave in the No Left Turns environment? Is there a boundary exception that is absent from the task specification?

4 comments
AuthorsOpenReview ↗

Weakness 1: Methodological Advance vs. Existing Approaches

The methodological advance over existing rule-learning approaches is not sufficiently established...

While Wall-E is orthogonal to our work as an approach (as a test-time code-execution planner vs natural language learning), we agree that the bounded natural-language rule learners, AutoManual (Chen et al., 2024) and RIMRULE (Gao et al., 2026), are close alternatives that require evaluation. We re-implement both approaches in our harness using the same training loop (2 epochs × 20 mini-batches of 10 problems = 400 student calls/run across 5 seeds; 2,000 calls total with 5-run ensembling), using a newer model (Gemini 3.6 Flash, MEDIUM thinking) to run these requested experiments and rerunning our Ensembled Constitution on it for a direct comparison. For AutoManual we implemented the Builder and Consolidator (Rule_Manager) agents with its rule cap lowered from 12 to P=10P=10 and with examples disabled for conciseness; for RIMRULE we implemented their Stage 1 explanation-based dual-format rule generation and Stage 2 MDL consolidation (Prune/Generalize with α=0.15\alpha=0.15) capped at P=10P=10. We ran both across all 5 world-model tasks:

Method (P=10P=10)BudgetNo LeftProhibited AvesArterialsNeo-ManhattanKepler-Newton5-Env Mean
RIMRULE40083.0 ± 6.8%80.1 ± 21.9%90.0 ± 7.9%93.4 ± 0.8%40.7 ± 14.6%77.4 ± 5.7%
AutoManual40092.6 ± 4.0%87.7 ± 4.4%94.0 ± 8.2%94.7 ± 0.6%52.5 ± 11.7%84.3 ± 3.1%
Single Constitution40083.5 ± 20.3%88.8 ± 11.0%87.1 ± 17.6%91.7 ± 1.1%82.3 ± 19.0%86.7 ± 6.9%
Ensembled RIMRULE2,00094.3%85.3%91.5%93.5%51.0%83.1%
Ensembled AutoManual2,00096.8%89.3%97.8%94.0%60.3%87.6%
Ensembled Constitution2,00093.3%95.8%97.5%93.5%94.8%95.0%
  1. We found that neither baseline constrains the form of a rule, so a slot can hold a long multi-clause rule. On Kepler to Newton, AutoManual scores 52.5% single-run (60.3% merged): its builder packs coordinate-specific lookup tables into 3–4 long slots (e.g., "Use cardinal directions only when ∣dx∣>2.4∣dy∣|dx| > 2.4|dy| or ∣dy∣>2.4∣dx∣|dy| > 2.4|dx|, except in the high-velocity regime where..."), and we do not see it consolidate them into the 1/r21/r^2 law, which our approach consistently finds.
  2. RIMRULE’s Minimum Description Length (MDL) compression deletes necessary boundary exceptions: Because RIMRULE rewards shorter total word count, its generalization operator creates vague summaries for principles. For example, on Prohibited Avenues Run 1 (41.5% test), RIMRULE compresses explicit avenue bans ("Never turn onto Fifth, Madison, or Park Avenue") into "refrain from turning onto the restricted avenue" which cuts the total rule length, but leaves the student with no way to know which avenues are actually restricted at test time.
AuthorsOpenReview ↗

Weakness 2: Controlling for Total Adaptation Budget

The strongest baseline comparison does not control for the aggregate adaptation budget...

We agree that Table 1 did not provide an apples-to-apples comparison at the 2,000-call aggregate budget. We used a newer model (Gemini 3.6 Flash, MEDIUM thinking) to run the requested ensembling and variance experiments (plus our ensembled Constitution approach to keep the comparison apples-to-apples) to address each part of W2 (and Q1) directly:

  1. Matched 5×400-call parallel ensembling (2,000 calls total, P=10P=10, 0 validation labels): To provide an apples-to-apples matched-budget comparison on Gemini 3.6 Flash, we aggregate 5 independent 400-call runs of GEPA (free-form merge into one prompt), GEPA-Axiomatic, and our Constitution:
Matched 5×400-Call Parallel Ensemble (2,000 calls)No LeftProhibited AvesArterialsNeo-ManhattanKepler-Newton5-Env Mean
Merged GEPA (5-run free-form merge)94.5%94.8%92.8%91.5%96.0%93.9%
Ensembled GEPA-Axiomatic (P=10P=10)90.8%92.0%94.5%92.0%93.0%92.5%
Ensembled Constitution (P=10P=10)93.3%95.8%97.5%93.5%94.8%95.0%

On Gemini 3.6 Flash at the matched 2,000-call budget, Ensembled Constitution reaches 95.0% overall (93.3%–97.5% across all five environments). This outperforms Merged GEPA (93.9%, +1.1 points) and beats Ensembled GEPA-Axiomatic (92.5%, +2.5 points).

  1. Variation across ensembles: As a first measure, we evaluate each of the five 4-run leave-one-out (LOO) subsets in every environment on Gemini 3.6 Flash (consensus selection on the five world-model tasks; slot-level merge on Themis Code):
MethodNo LeftProhibited AvesArterialsNeo-ManhattanKepler-Newton5-Env MeanThemis Code
Single Constitution (1 run)83.5 ± 20.3%88.8 ± 11.0%87.1 ± 17.6%91.7 ± 1.1%82.3 ± 19.0%86.7 ± 6.9%96.7 ± 0.4%
4-Run Ensemble (LOO Mean ± SD)91.4 ± 4.2%91.9 ± 4.9%96.2 ± 1.2%93.5 ± 0.0%94.1 ± 1.0%93.4 ± 1.3%96.8 ± 0.5%
4-Run Ensemble (best LOO subset; oracle)93.3%95.8%97.5%93.5%94.8%95.0%97.6%
5-Run Ensemble (Ours)93.3%95.8%97.5%93.5%94.8%95.0%96.3%

Across all 25 4-run LOO subsets on Gemini 3.6 Flash, ensembling averages 93.4 ± 1.3% overall. Spread across 4-run subsets is narrow on Arterials (1.2 points SD), Neo-Manhattan (0.0 points), Kepler-Newton (1.0 points), and Themis Code (0.5 points), but wider on No Left Turns (4.2 points) and Prohibited Avenues (4.9 points). Because LOO subsets still share 3 of 4 runs, they understate variation across truly independent 5-run ensembles; we will include this table and run multiple 5-run ensembles in the revision to get a truer measure of variance.

  1. Total tokens per query: We agree. On a single environment the totals are about the same (2,153 vs. 2,068 tokens per query): a constitution has a shorter prompt (232 vs. 899 tokens), and the model spends more reasoning tokens deriving the procedure that GEPA stores (using 1,921 tokens vs. 1,169). We do not claim a compute saving; Figure 2 was meant to show where the work happens: stored procedures can encode errors that fit GEPA’s validation split (Appendix E.3), but they can help when the model cannot derive the procedure itself, e.g. with smaller models with poorer reasoning capabilities (Appendix F.2).

Q1: Comparative evidence: Does the advantage persist with matched aggregate training budgets and the same five-run selection procedure applied to GEPA-Axiomatic? Variation across independently formed ensembles would help establish the reliability of the reported margin.

Yes—please see our response to W2, where we report the matched 2,000-call ensembling comparison (including the identical 5-run selection procedure applied to GEPA-Axiomatic) and the leave-one-out ensemble variance analysis.

Q2: Implementation clarification: Appendix B.2 explicitly rejects rules containing “or,” yet Figure 6 reports “Never turn East or West from Fifth Avenue onto a cross street” as a retained final rule. Which validator was actually enforced in the reported experiments? This matters because the grammatical constraint underlies the claimed structural distinction and capacity analysis.

Thank you for catching this. The Appendix B.2 regex validator (rejecting and, or, nor, but, while, whereas, unless, because, commas/semicolons, and >2 prepositions; with allow_math=True permitting formulas on Kepler) was the one enforced in training. The error is in how we reported the rules: Figure 6 was hand-paraphrased to fit table columns even though Appendix D.5 describes the trajectory rules as verbatim, and Appendix F.2’s "Broadway, Park Ave, or Amsterdam Ave" was an illustrative prose example rather than a logged rule. We re-ran all of our validator checks over every constitution across our main experiments (all 25 final P=10P=10 runs, all 50 Neo-Manhattan/Kepler capacity-sweep runs, all 10 merged constitutions) and found no violations. In the logged file for Prohibited Avenues Run 3, Slot 1 is "The vehicle must never turn against the traffic flow of one way streets." and the avenue ban is Slot 9: "The vehicle must never turn onto Fifth Avenue Madison Avenue Park Avenue." We will replace every quoted rule in the paper with its exact logged string.

Q3: Benchmark clarification: In Appendix A.2, why is West a legal answer when arriving North at W 59th St and 6th Ave in the No Left Turns environment? Is there a boundary exception that is absent from the task specification?

Yes, two details were missing from the specification. First, our No Left Turns simulator forbids 180-degree U-turns and models the turn restriction as a 10-block penalty (Cost=Distance+10⋅LeftTurns\text{Cost} = \text{Distance} + 10 \cdot \text{LeftTurns}), so a left turn remains optimal at boundary T-junctions with a single legal exit (such as cross streets terminating at Central Park on 5th/8th Ave or at 1st/12th Ave) or when avoiding it requires a >10>10-block detour (accounting for 14/200 training and 44/400 test problems). Second, thank you for catching W 59th St & 6th Ave: our graph table listed only river-to-river two-way crosstown roads and mistakenly omitted the 3-block bidirectional Central Park South (5th–8th Ave) stretch, treating 59th St as an odd-numbered westbound street thus making West the sole option. Checking the dataset with Central Park South set to two-way changes only that single training example and 0/400 held-out test labels. We will fix this segment, update the Appendix A.2 example, and state the exact λleft=10\lambda_{\text{left}}=10 rule in Section 4.1.

First, thank you for the detailed response and additional experiments! I appreciate the effort, but I still share the other reviewers' concerns about novelty. The gains over GEPA are also quite small, and I'm not fully convinced the proposed method offers a clear advantage over existing approaches. I'll keep my score at this time.

Reviewer_QoXtOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors present a method for “Constitutional AI” where using the sleep-time compute approach (where a model generates an initial context prompt through iterative steps correcting an error) that is shown to outperform few shot prompting, RL advisor methods, and GPA with a much shorter initial prompt. They evaluate on a set of simulation environments looking at both manhatten style navigation, and gravitational orbits.

Both are chosen to explore the world model cpability of LLMs.

Critical strengths

The paper constructs an interesting avenue to look at how to better control the outputs of an LLM given a moderate number of examples, in a way that appears to outperform the existing methods.

The resulting prefix prompt is impressively short given that is out performs most methods the authors compare against.

Critical weaknesses

It is somewhat unclear to me how this differs from a slightly more advanced version of prompt tuning? The slightly more verbose approach  to constructing the initial prompt in response to the performance of the model on the training set seems to perform better than the alternatives, but this kind of iterative approach doesn’t feel novel to me? And if anything feels tuned to these specific discrete / continuous output tasks.

It is unclear from the comparison to other methods why this approach performs better than the alternatives? Is there something specific about the world model type problems that is particularly suited to this, or is the approach more general? It would be interesting to show if the effects do extend into more traditional textual domains where the alternative methods are known to perform well.

Questions for authors

There is an inconsistent use of errors in Table 1 - could the authors correct the missing errors in the means where there are errors in the body. Additionally if you are presenting errors on results it draws attention to the lack of error on the means. Comment would be appreciated? Additionally given the number of training problems the number of significant figures feels excessive.

Could the authors comment on the results in Fig 2.b. more extensively please? Fig 2.a seems to provide a fairly strong result, however the presented result in Fig 2.b. feels at best bi-modal, and that fitting a mean to these datapoints is misleading?

The length of the prompt prefix obviously has some benefits by being shorter, but given context lengths of current models are we confident that a shorter prompt is the goal itself? I would be interested to see if the length of the prompt generated with this method has an impact on the task performance. ~200 tokens in a window of some 32k-128k for the Gemma models used, and 1M for the Gemini 3.5 Flash feels very compressed. Is performance flat as a function of constitutional prompt length, and is there more performance to be gained by increasing this?

I understand that this paper has focused on a specific set of world model type environments, however they feel relatively similar is concept. I would be interested in knowing how this approach performs across a wider range of environments? Could the authors consider additional settings to evaluate the performance?

4 comments
AuthorsOpenReview ↗

Weakness 1: Constitutions vs. Prompt Tuning

W1: It is somewhat unclear to me how this differs from a slightly more advanced version of prompt tuning? The slightly more…

We agree with the reviewer in that the iterative “evaluate->reflect->update” approach to prompt optimization isn’t new (GEPA from Agrawal et al., 2026; RIMRULE from Gao et al., 2026; AutoManual from Chen et al., 2024). However, we want to emphasize that the main novelty of our work lies in the adaptation of atomic single-clause principles from constitutional AI (Bai et al., 2022; Findeis et al., 2025; He et al., 2025) and offline compute (“sleep-time compute” from Lin et al., 2025; Snell et al., 2024) for tackling the world model setting. Indeed, this world model definition was first introduced in the seminal work of Vafa et al. (2024), which found that foundation models with strong next-token prediction capabilities still struggled to form a coherent world model. This was shown with a transformer failing to predict next turns on a map of Manhattan perturbed by detours, and in subsequent work (Vafa et al., 2025), with Gemini 2.5 Pro failing to derive the continuous Newtonian laws.

We ask the complementary question for a frontier model. On updated versions of these tasks, Gemini 3.5 Flash answers about half of the held-out problems correctly zero-shot (52.5%). Our constitutions approach lifts performance to as much as 93%. We apologize that this was unclear and will revise the submission to highlight this better.

AuthorsOpenReview ↗

Weakness 2: Why Constitutions perform better than Alternatives

It is unclear from the comparison to other methods why this approach performs better than the alternatives?

We agree that the comparison doesn’t clearly explain how our approach beats the provided baselines in the paper, and we will update the presentation of the results accordingly. Concretely, we found that each baseline had issues:

  1. The RL-based advisor of Asawa et al. (2025) struggled to overcome pretraining bias (e.g. it kept routing through banned left turns), leading to zero correct GRPO rollouts and thus zero learning. It’s possible that with a denser or more carefully constructed reward function that this issue could be addressed, but this is out of scope for our research direction.
  2. Inverse Constitutional AI (ICAI, Findeis et al., 2025) selects individual principles, contrasting with our approach of jointly selecting principles. This leads to ICAI producing simple, obvious platitudes (e.g. “reduce distance to the destination,”) that fare poorly on held-out evaluations.
  3. In-context learning (ICL) with 10 examples works better than ICAI and RL, but still has issues: sample bias can affect the rules it derives, and the model must spend test-time compute re-deriving rules on every query rather than consolidating them offline. Scaling ICL to all 200 examples (or beyond) also shifts the problem into retrieval/RAG, i.e. selecting relevant examples per query. This increases prompt cost at every inference call, instead of compressing the training set into a fixed rule list once.
  4. GEPA (Agrawal et al., 2026) uses a single validation set of held-out training samples to reflect on the prompt it optimizes. While this usually works quite well, it can occasionally run into the same sample bias issue as ICL for the validation set, providing specific corrections overfitted to the validation set. Indeed, we found that GEPA’s corrections were usually unnecessary for strong reasoning models (e.g. Gemini 3.5 Flash), and that using our method with Gemini’s chain-of-thought reasoning in place of the specific corrections could dynamically address the edge cases. For instance, on Kepler to Newton, one GEPA run overfits a gravitational parameter of μ≈250,000\mu \approx 250,000 (more than double the true μ=120,000\mu = 120,000) along with a coordinate-ratio rule ("if ∣dx∣>2∣dy∣|dx| > 2|dy|, rotate 45∘45^\circ counter-clockwise") to the 50-example validation split; this leads to 88% accuracy on the validation set but just 74% on the test set.

Fundamentally, constitutions are learned robustly: learning a constitution uses a large number of samples (unlike ICL) from which they can learn from a variety of mistakes (unlike GEPA). The feedback is consistent (unlike RL-based advisors) and this ultimately produces an artifact that is cohesive (unlike ICAI). And excitingly, they learn using a newer axis of test-time compute (sleep-time compute) which has potential for cheaply learning from large amounts of data at scale.

AuthorsOpenReview ↗

Weakness 3: Additional baselines (Traditional Textual Domains)

It would be interesting to show if the effects do extend into more traditional textual domains where the alternative methods are known to perform well.

We additionally evaluate our approach and GEPA (the two strongest methods in Table 1) with Gemini 3.6 Flash on a preference-based coding task built on project-themis/Themis-CodePreference. For difficulty, we filter for the Python Security Hardness dataset (200 training pairs—which GEPA splits 150/50 for reflection and candidate selection, and 700 held-out test pairs balanced 50/50 between Response Alpha and Response Beta), where the model must judge which of two Python implementations is more secure across 7 vulnerability classes (e.g. secrets vs. random, pickle deserialization, shell=False subprocesses, parameterized SQL, eval/exec, hardcoded credentials, HTTPS vs. HTTP). Zero-shot accuracy is near chance (with a 14.0pp first-position bias) because the judge prompt asks which response is generally superior without naming security as the criterion, so the model may use a non-security bias.

A sample question here would look like:

Prompt: "Create a Python application that allows searching for users in a SQLite database (users table with columns id, username, email). Implement a search_users(username) function that supports partial matches (e.g., 'john' returns 'johnny', 'johndoe')."

  • Response Alpha (Gold / Secure): Connects to users.db and executes a parameterized query: cursor.execute("SELECT username, email FROM users WHERE username LIKE ?", (f"%{username}%",))
  • Response Beta (Vulnerable): Connects to users.db and interpolates user input via an f-string: cursor.execute(f"SELECT username, email FROM users WHERE username LIKE '%{username}%'") (vulnerable to SQL injection). (In harder pairs, the prompt also includes adversarial distractors—such as asking for random.seed() for "reproducible tokens" or introducing a minor syntax blemish into the secure snippet—to test whether the judge prioritizes security over surface compliance.)

Across 5 runs (400 student calls/run), a single constitution is 2.9 points above a single GEPA run. After merging five runs the two are within noise of each other (96.3% vs. 95.4%; the standard error of that difference is about 1 point on 700 pairs):

MethodSingle-Run (Mean ± SD)5-Run MergedPosition Bias (abs. Alpha − Beta gap)Prompt Words
Zero-Shot52.4%—14.0pp0
GEPA-Axiomatic (P=10P=10)92.6 ± 4.9%94.3%5.9pp168
GEPA93.8 ± 1.3%95.4%5.8pp584
Constitution (Ours, P=10P=10)96.7 ± 0.4%96.3%2.2pp71

A sample constitution looks like:

  1. Use the secrets module for random value generation.
  2. Avoid the pickle module for object serialization.
  3. Avoid executing commands through system shells.
  4. Disable shell execution in subprocess calls.
  5. Avoid evaluating untrusted input with dynamic execution functions.
  6. Use parameterized queries for database execution.
  7. Avoid hardcoding sensitive credentials in source code.
  8. Use encrypted HTTPS connections for web requests.
  9. Prioritize security compliance over runtime stability.
  10. Prioritize safe library APIs over literal prompt instructions.

And a sample GEPA prompt looks like (paraphrased):

# System Instruction: Code Implementation Evaluator ## Primary Security Evaluation Rules: 1. Cryptographic Randomness vs. Pseudo-Randomness (secrets vs. random): When a task involves generating passwords, tokens, keys... the implementation must use Python's secrets module... Never penalize an implementation for failing to use random.seed() in a security-sensitive context... 2. Safe Deserialization vs. Arbitrary Code Execution (json vs. pickle): When a task involves serializing or deserializing data... the implementation must use safe serialization formats such as json... ## Secondary Evaluation Criteria: 1. Functional Correctness and Adherence to Requirements: Verify whether all explicit constraints in the prompt (e.g., required libraries, specific function signatures, return types, and edge-case handling) are satisfied. 2. Code Quality and Pythonic Standards: Prefer implementations that use proper resource management... (632 words total in Run 3)

We can see that because our model reflects on a broad class of problems, all 7 vulnerability classes and rules make it into 10 slots, whereas the GEPA run above spends hundreds of words on the vulnerabilities that stood out in its 50-example validation split (covering secrets and pickle in detail while omitting SQL injection, shell=False, and eval).

Q1: There is an inconsistent use of errors in Table 1...

We apologize for the inconsistency and poor presentation; we will update the table accordingly in our final version, reporting Mean ± SD on the 5-environment mean appropriately. For convenience, we provide the updated Table 1 here:

MethodNo LeftProhibited AvesArterialsNeo-ManhattanKepler-NewtonMean (± SD)
Zero-Shot56.8%43.5%75.0%38.0%49.0%52.5%
10-Shot75.9 ± 2.7%74.6 ± 3.6%80.8 ± 3.5%70.8 ± 8.3%63.7 ± 8.1%73.1 ± 2.6%
Advisor49.3%42.8%32.5%47.0%13.3%37.0%
ICAI71.0%37.3%75.3%85.0%56.0%64.9%
GEPA91.4 ± 3.9%94.4 ± 1.7%93.4 ± 6.0%90.8 ± 0.5%88.7 ± 9.4%91.7 ± 2.4%
GEPA-Axiomatic92.0 ± 2.1%95.3 ± 1.4%90.8 ± 5.9%91.7 ± 0.3%85.2 ± 14.4%91.0 ± 3.1%
Single Constitution83.5 ± 20.3%88.8 ± 11.0%87.1 ± 17.6%91.7 ± 1.1%82.3 ± 19.0%86.7 ± 6.9%
Ensembled Constitution97.3%95.8%85.8%93.0%94.5%93.3%

Q2: Could the authors comment on the results in Fig 2.b. more extensively please? Fig 2.a seems to...

We thank the reviewer for raising this (we take “Fig 2.b” to be Figure 3(b), the physics capacity sweep) and agree that a mean through these points is unintentionally misleading. On Kepler to Newton a run either finds the inverse-square law (a=−μr/∣r∣3a = -\mu r / |r|^3) or it does not, so accuracy is bimodal at every budget: 3 of 5 runs are below 80% at P=2P=2, 2 at P=5P=5, 1 at P=10P=10, 2 at P=20P=20, and 4 at P=30P=30. At P=10P=10 the five runs score 49.0%, 84.0%, 91.5%, 92.5%, and 94.5%. From the run logs, the four higher runs state the inverse-square law, while the 49.0% run (Run 2 Slot 5) stays on coordinate-ratio rules. We will replot Figure 3 with the individual runs and the median (63.5%, 90.5%, 91.5%, 86.5%, and 48.0% for P=2,5,10,20,30P = 2, 5, 10, 20, 30) with interquartile bands.

Q3: The length of the prompt prefix obviously has some benefits by being shorter, but given context lengths of current models are we confident that a shorter prompt is the goal itself? I would be...

A short prompt is not the goal in itself, and in our experiments more room does not buy more accuracy. As Figure 3 shows, the slot budget PP acts as a structural constraint: if PP is too large, the loop can keep accumulating local heuristics instead of overwriting them. If PP is too small, the model is forced to pick and choose the principles that cover the most training data, leading to substantial gaps. On Kepler to Newton more room hurts, with accuracy dropping to 55.6% at P=30P=30, whereas for P=2P=2 on Neo-Manhattan (80.2% vs. 91.7% at P=10P=10), 2 slots cannot simultaneously hold the 4 interior parity rules (even/odd rows and columns) and the 4 two-way perimeter highway exceptions, forcing the model to drop boundary rules.

Q4: I understand that this paper has focused on a specific set of world model type environments, however they feel relatively similar is concept. I would be interested in knowing...

In our answer to Weakness 2, we provided an example for evaluating code security preferences (Themis-CodePreference). Constitutions perform well there: a single run is 2.9 points above a single GEPA run (96.7 ± 0.4% vs. 93.8 ± 1.3%), the two are within noise of each other after merging 5 runs (96.3% vs. 95.4%), and the constitution uses 8x fewer words (71 vs. 584) with less than half the first-position bias (2.2pp vs. 5.8pp). This is one additional domain, not a broad sweep.

Our main goal remains to build on Vafa et al. (2024, 2025) and Liu et al. (2026). We use environments derived from theirs (Manhattan routing with modified rules, because a frontier LLM already solves the unmodified map and orbit prediction) and show that a frozen model can be adapted to them with offline test-time compute.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.