acceptodds
Under review as a conference paper at ICLR 2027

Code as Carrier: Context-Efficient Multi-Turn Instruction Following

Abstract

Large language models often fail to preserve instructions across multi-turn conversations, especially when users update constraints without restating them. We present Code as Carrier (CoCa), a programmatic framework for conversation management that dynamically detects topic shifts, identifies user intents, and incrementally updates an executable program that maintains and enforces the topic thread's active constraints. This allows CoCa to reapply constraints through execution rather than recovering them from an ever-growing dialogue history. CoCa provides the largest gains when direct prompting struggles to retain prior constraints: on a short, single-topic benchmark, it improves turn-wise retention by to for weaker models while leaving an already-stable model nearly unchanged. For longer multi-topic dialogues, it can reduce the turn-level failure rate from to . Finally, by updating programs in place, CoCa maintains approximately constant per-turn context and uses fewer context tokens over direct baselines.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
code as representation, context compression, multi-turn instruction following, small language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FAtFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Code as Carrier (CoCa), a framework for maintaining evolving constraints in multi-turn conversations. CoCa uses intent classification and topic-aware routing to maintain topic-specific executable programs, which preserve active constraints and enforce them through validation and repair.

Experiments on Multi-IF and EvolIF with six executor models, 3B–120B, demonstrate substantial improvements over direct inference, particularly for smaller models and longer conversations. Ablation studies further investigate the effects of program-based constraint representation, topic routing, and generator selection.

Critical strengths
  1. Meaningful problem formulation and a practical design. The paper addresses the important problem of maintaining evolving constraints across multi-turn conversations. Representing active constraints as persistent executable programs provides a useful alternative to repeatedly recovering them from dialogue history, combining explicit state management with runtime validation.

  2. Substantial empirical improvements across models and settings. CoCa achieves consistent gains across six executors on two complementary benchmarks, particularly for smaller models and longer conversations. The ablation studies provide additional evidence for the benefits of persistent program states and topic-aware routing, while the per-category analysis reveals meaningful differences between mechanically verifiable and semantic constraints.

Critical weaknesses
  1. Insufficient comparison with closely related methods. The main experiments compare only against one method: direct inference. Existing ablations use previous-response prose or short rolling summaries rather than a strong persistent constraint-state baseline. Related work such as GraphIF already supports structured constraint updates and topic tracking, while executable validation has been explored in DSPy Assertions and Mellea. Without a topic-aware explicit constraint-state baseline, the added value of executable programs remains unclear.

  2. The contribution of the strong program generator is not sufficiently isolated. All main experiments use GPT-OSS-120B to generate and maintain programs, while only executor size varies. Although generator sensitivity is evaluated, the tested alternatives are also capable models. Consequently, improvements with small executors may partly reflect strong-model assistance rather than the proposed representation itself.

  3. Incomplete evaluation of topic routing. The reported classification results primarily measure new-topic detection, not whether the system correctly retrieves the intended existing topic. Since accurate topic restoration is central to CoCa, the absence of thread assignment accuracy leaves an important component insufficiently validated.

  4. Some claims exceed the supporting evidence. The reported 2.9× token reduction excludes parts of the inference pipeline, while bounded context and guaranteed constraint satisfaction are not established by the implementation. These claims should be restricted to the quantities and settings actually evaluated.

Questions for authors
  1. Stronger baseline comparison. How does CoCa compare against a topic-aware explicit constraint-state baseline that maintains the complete active constraints in natural language or JSON, using the same generator, router, and executor? This comparison would help establish the additional value of executable programs.

  2. Generator dependence. How much of CoCa's improvement remains when using substantially smaller or less capable program generators? Can the authors distinguish the benefits of programmatic constraint enforcement from those of strong-model-assisted constraint extraction and planning?

  3. Topic routing accuracy. What is the actual thread assignment accuracy when a user returns to an existing topic? The reported new-topic detection metrics do not establish whether the correct historical topic is retrieved.

  4. End-to-end efficiency. What are the total input/output token costs and inference latency when accounting for intent classification, program generation, executor calls, and validation retries? Is the context size strictly bounded, or only approximately constant in the evaluated settings?

The paper presents a meaningful approach to multi-turn constraint management, supported by substantial empirical improvements. However, its central contribution is not sufficiently distinguished from existing structured constraint-management and executable validation methods. In particular, the absence of a strong persistent constraint-state baseline makes it difficult to determine whether executable programs are responsible for the reported improvements.

8 comments
AuthorsOpenReview ↗

[1/5] Response to W1

We thank the reviewer for the careful review and for specifying exactly the baseline that was missing. We have now run it (see also Q1), and it isolates what executable programs add.

We first clarify the key distinction: in CoCa, the program plays two roles.

  1. Carrier: it stores the active per-topic constraint set and updates it across turns. A ledger can also do this.
  2. Enforcer: it checks the generated response against each constraint in code and deterministically repairs violations such as keyword counts and markers. A ledger can only state constraints; compliance is still left to the model or to model-based self-checking.

To test this directly, we added the reviewer's suggested baseline.

Setup. We replaced the program in CoCa with a per-topic JSON state, {topic, task, constraints: [{id, text}]}, while keeping the generator (GPT-OSS-120B), intent classifier, topic router, executor, and execution harness fixed. At each turn, the routed topic's constraint state is updated with the same keep/replace/remove/add operations as in CoCa, using a one-to-one conversion of CoCa's in-context examples. Constraints are free text extracted from the user turns. We evaluate two arms:

  • JSON constraint state: the executor answers once from the complete active constraint set.
  • JSON constraint state + self-revise: the executor answers, then checks its own draft against the numbered constraints and revises. Both steps are LLM calls, under a hard cap of 5 executor calls per turn. This gives the baseline a validation and revision budget larger than CoCa's actual usage.

Results (EvolIF, 150 dialogues, P_max = 3). Executor calls are counted at the executor client and include retried and timed-out attempts.

ExecutorArmPermitted executor calls / turnActual executor calls / turn (mean)EDR lenCSRISRRECROB
Granite 4.1 3BDirect11.006.7562.3542.4514.1232.68
JSON constraint state2 (one retry if empty)1.008.4569.7951.1419.3840.65
JSON constraint state + self-revise52.759.3573.2852.8221.7742.62
CoCanot hard-capped (prompted ≤ 5)1.6510.9877.8959.1422.1950.35
Gemma-4 26BDirect11.009.7176.1556.5218.9646.02
JSON constraint state2 (one retry if empty)1.0112.6281.4964.2425.9353.63
JSON constraint state + self-revise53.2813.0182.4565.3724.8055.78
CoCanot hard-capped (prompted ≤ 5)1.5114.4483.9067.5426.2360.63

The advantage concentrates where checking requires exact verification. Per-constraint pass rate (%), shown as JSON constraint state / JSON constraint state + self-revise / CoCa:

ConstraintGranite 4.1 3BGemma-4 26B
existence (required keywords, exact counts)15.1 / 13.4 / 51.051.2 / 53.7 / 74.5
case60.7 / 59.2 / 85.562.7 / 68.3 / 87.9
length50.0 / 57.9 / 71.172.8 / 75.8 / 78.3

CoCa outperforms the strongest constraint-state baseline on every metric for both executors, while using fewer executor calls than the self-revising baseline. The gains are largest for constraints requiring exact verification via code.

Relation to GraphIF, DSPy Assertions and Mellea. These works each address part of the problem, but none tracks constraints per topic, extracts them automatically from free-form user turns, and enforces them in code as the conversation evolves:

  • GraphIF maintains structured constraint updates and topic structure, but to our understanding the model is still asked to satisfy the constraints from that representation. That places it in the same family as the explicit constraint-state baseline above, and our comparison shows what enforcement in code adds to it.
  • DSPy Assertions and Mellea are programming libraries that support validation, but they do not decide what to validate. A developer must extract the constraints from the user's request and write the corresponding checks, once, for a fixed task. Where no hand-written check is supplied, the fallback is an LLM judge; in Mellea, for example, a requirement without a custom validation function is checked by a model. That setting corresponds closely to our JSON constraint state + self-revise baseline, which CoCa outperforms on every metric.
  • CoCa removes the developer from this loop. Its validators are generated per turn from the user's free-form instructions, kept per topic, and edited as the conversation evolves: a constraint added at turn 3 is still enforced at turn 30, replaced when the user changes it, and removed when the user drops it.

We will add the explicit constraint-state baseline to the main results and expand the related-work discussion to position CoCa against all three.

AuthorsOpenReview ↗

[2/5] Response to W2

We thank the reviewer for raising this. Three results separate the representation's contribution from the generator's strength.

  • The representation helps even when no stronger model is involved. When GPT-OSS-120B both generates and executes its own program (120B + 120B), CoCa still improves over that same model prompted directly: on EvolIF from 57.43 to 67.81 ISR (+10.4) and from 47.65 to 59.61 ROB (+12.0); on Multi-IF from 81.21 to 84.25 overall (+3.0). No capability is lent by a stronger model here, so the gain comes from carrying and enforcing constraints as a program. The size of this gain matches the trend set by the other five executors (residual +2.0 ISR on EvolIF, +0.6 on Multi-IF), so the generator does not favor itself.

  • The same generator helps less without the program. Our sumhist baseline uses the same GPT-OSS-120B to rewrite each turn into a self-contained prompt from a 3-turn window, with no program. On Granite 4.1 3B it recovers only +6.2 of CoCa's +17.6 ISR gain (35%), and 29% on Gemma-4 26B. A strong model rewriting the request therefore explains only a minority of the gain (Table 2). The matched JSON constraint-state baseline (W1) makes the same point: with the identical 120B generator extracting and maintaining the full constraint set, it still trails CoCa on every metric. That isolates programmatic enforcement from strong-model constraint extraction.

  • A much smaller generator still helps. In preliminary experiments, we replaced the 120B generator with a 15× smaller 8B generator (an 8B open-weight model fine-tuned on a separate synthetic single-turn dataset) and held everything else fixed: the same Granite 4.1 3B executor, the same intent labels, and the same topic routing. CoCa still improves over direct prompting on every metric: ISR 42.45 → 50.71 (+8.3), CSR 62.35 → 69.31 (+7.0), ROB 32.68 → 40.30 (+7.6), and EDR length 6.75 → 8.43 turns. Because these results are preliminary, we mention smaller generators only as future work in the conclusion, though we find them promising.

AuthorsOpenReview ↗

[3/5] Response to W3

We thank the reviewer and agree that thread-assignment accuracy should be reported directly. We measured it against EvolIF's ground-truth topic labels on all 150 dialogues (6,700 continuation turns, of which 955 return to an earlier, non-adjacent topic):

Router (same topic anchors)Continuation turns routed correctlyTurns returning to an earlier topic routed correctly
Embedding cosine, threshold 0.4566.6%42.6%
Embedding cosine, threshold 0.3574.6%58.7%
Embedding cosine, threshold 0.30 (best of sweep)77.7%67.1%
Token overlap (CoCa)81.7%77.6%

CoCa restores the correct historical topic on 77.6% of return turns (Granite 4.1 3B; 77.5% with Gemma-4 26B), more accurately than embedding-based routing at any threshold. Most remaining errors are short follow-ups that name no subject (e.g. "make it shorter"); no lexical or embedding signal can route these, and the router leaves them on the current thread, which is usually correct. On turns that do name their subject, token overlap routes about 96% correctly.

Routing accuracy also has a measurable effect on outcomes:

  • Mis-routing is costly: a turn carried into the wrong topic fails 81% of the time.
  • Correct restoration adds a separate gain: disabling topic resumption while keeping the program costs 3.3 ISR on Granite 4.1 3B and 3.7 ISR on Gemma-4 26B.

We will add these thread-assignment results to the routing section.

Reviewer_G4KdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors focus on addressing the problem in multi-turn dialogues: models often forget or mix up prior rules as constraints change or topics shift.This work mainly combines intent classification, topic routing, code checking, and retry-based rewriting into an engineering pipeline. They test the proposed model across six open-weight models (3B–120B) on Multi-IF and EvolIF and the results show clear gains in late-turn rule following, especially for smaller models.

Critical strengths
  1. The paper tries to address a common problem in multi-turn dialogues: models often forget or mix up prior rules as constraints change or topics shift. Maintaining per-topic rules in code and checking them during execution forms a clear and practical pipeline.
  2. Tests across six open-weight models (3B–120B) on Multi-IF and EvolIF show clear gains in late-turn rule following, especially for smaller models.
Critical weaknesses
  1. This work mainly combines intent classification, topic routing, code checking, and retry-based rewriting into an engineering pipeline. But its ablations are weak. The paper only compares against passing the previous answer or a three-turn summary, rather than a persistent text or JSON constraint table updated at each turn. It also lacks comparisons with closely related methods such as User as Code, executable dataflow state graphs, and strong memory baselines like MemGPT. As a result, the experiments show that the full pipeline beats direct prompting, but they do not prove that code itself is necessary.
  2. The proposed method is a rule tracker for instruction constraints, not a general solution for long context or factual memory. Most gains come from constraints that are easy to check with Python, such as keywords, letter case, and output format. In contrast, performance drops on subjective constraints like style and reader_age. Moreover, the bounded rewrite loop cannot guarantee that the final output always follows all rules. In a real agent system, this method can only serve as a small module for constraint tracking. It cannot replace general context management or RAG, and it adds extra cost in state updates and debugging.
  3. The paper claims a 2.9x reduction in context tokens, but Figure 4 only counts a single state-carrying input prompt. It leaves out intent classification, code generation, generated code tokens, executor calls, and rewrite retries. Updating the program at every turn also limits KV-cache reuse for the changing state, while multi-step serial calls increase latency. In addition, the claim that a 3B executor matches a 120B model still relies on GPT-OSS-120B as the default code generator. Therefore, the paper does not truly show a low-cost deployment advantage for small models.
  4. The experiments only cover Multi-IF and EvolIF. The paper does not test on broader benchmarks for long-horizon or structured instruction following or on conversations beyond three turns. On the model side, all main executors are open-weight models, without experimenting SOTA models. Even on a stronger open model like Gemma 4 26B, CoCa shows a slight drop on Multi-IF, so its value for stronger or closed-source executors remains unclear.
  5. The method executes model-generated code at inference time, yet the paper does not clearly explain its sandbox, allowed modules, time limits, or error handling. The topic router also relies on simple word-level anchor matching, which may fail when users change wording or shift topics gradually. Finally, because EvolIF stops a dialogue early after three consecutive failures, Direct and CoCa are not evaluated on the same set of turns in the per-category analysis, which weakens the comparison.
Questions for authors

Please find the question shown in Weakness.

6 comments
AuthorsOpenReview ↗

[1/5] Response to W1

We thank the reviewer for the thorough review. We have now run the baseline the reviewer suggested, and the results below show why code is necessary.

We first clarify the key distinction: in CoCa, the program plays two roles.

  1. Carrier: it stores the active per-topic constraint set and updates it across turns. A ledger can also do this.
  2. Enforcer: it checks the generated response against each constraint in code and deterministically repairs violations such as keyword counts, length, and markers. A ledger can only state constraints; compliance is still left to the model or to model-based self-checking.

Setup. We replaced the program in CoCa with a per-topic JSON constraint table, {topic, task, constraints: [{id, text}]}, while keeping the generator (GPT-OSS-120B), user intent classifier, topic router, executor, and execution harness fixed. At each turn, the routed topic’s table is updated with the same keep/replace/remove/add operations as in CoCa, using a one-to-one conversion of CoCa's in-context examples. Constraints are free text extracted from the user turns. We evaluate two arms:

  • JSON table: the executor answers once from the table (with one retry if the answer is empty).
  • JSON table + self-revise: the executor answers, then checks its own draft against the numbered constraints and revises. Both the check and the revision are LLM calls, under a hard cap of 5 executor calls per turn.

Results (EvolIF, 150 dialogues, P_max = 3). Executor calls are counted at the executor client and include retried and timed-out attempts.

ExecutorArmPermitted executor calls / turnActual executor calls / turn (mean)EDR lenCSRISRRECROB
Granite 4.1 3BDirect11.006.7562.3542.4514.1232.68
JSON table2 (one retry if empty)1.008.4569.7951.1419.3840.65
JSON table + self-revise52.759.3573.2852.8221.7742.62
CoCanot hard-capped (prompted ≤ 5)1.6510.9877.8959.1422.1950.35
Gemma-4 26BDirect11.009.7176.1556.5218.9646.02
JSON table2 (one retry if empty)1.0112.6281.4964.2425.9353.63
JSON table + self-revise53.2813.0182.4565.3724.8055.78
CoCanot hard-capped (prompted ≤ 5)1.5114.4483.9067.5426.2360.63

The advantage is concentrated where checking requires exact verification. Per-constraint pass rate (%), shown as JSON table / JSON table + self-revise / CoCa:

ConstraintGranite 4.1 3BGemma-4 26B
existence (required keywords, exact counts)15.1 / 13.4 / 51.051.2 / 53.7 / 74.5
case60.7 / 59.2 / 85.562.7 / 68.3 / 87.9
length50.0 / 57.9 / 71.172.8 / 75.8 / 78.3

CoCa outperforms the strongest JSON-table baseline on every metric for both executors, while using fewer executor calls than the self-revising table. The gains are largest for constraints requiring exact verification via code.

On memory and code-as-state methods (MemGPT, User as Code, dataflow state graphs). MemGPT pages conversational memory in and out of context as text that the model must re-read and interpret: it addresses which content is available, not whether standing constraints are verified. The persistent JSON table above is the closest controlled instance of that family that we can match on generator, router and executor. User as Code and executable dataflow state graphs also represent dialogue state as code, but they execute it to resolve references and API calls for task completion, not to verify open-ended output constraints. We will discuss all three explicitly in the related-work section and clarify this distinction.

AuthorsOpenReview ↗

[2/5] Response to W2

We thank the reviewer for these points and address each below.

Scope: CoCa complements RAG rather than competing with it. We clarify that CoCa targets multi-turn instruction following, not general long-context understanding or factual memory. We acknowledge that our related-work section mentioned RAG and memory-paging systems, which might have implied a broader scope, and we will make this distinction explicit.

Different problems, complementary solutions. RAG and context management determine which content a model sees, but do not ensure that standing instructions remain consistent and are satisfied across turns. Retrieving a turn containing "always end with !" still leaves the model responsible for remembering, applying, and verifying that constraint. This is precisely the failure CoCa addresses: 77% of failed EvolIF turns under prose-carried state resulted solely from dropping a previously established constraint. The approaches are complementary, as retrieved or summarized content can be supplied alongside CoCa's constraint-enforcing program.

Cost, gains, and transparency. The overhead brings substantial gains: in W1's matched comparison, CoCa improves ISR/ROB over direct prompting by +16.7/+17.7 with Granite 4.1 3B and by +11.0/+14.6 with Gemma-4 26B, while requiring fewer executor calls than a self-revising constraint table. CoCa's costs, which come primarily from program generation, are reported in our response to W3. Moreover, CoCa's explicit program literals and checks make failures easier to diagnose: each violation can be traced to a specific dropped or mis-enforced constraint, unlike transcript-based state or model self-assessment.

Verifiable vs. subjective constraints. We agree that CoCa's largest gains are on constraints that can be checked programmatically; this is precisely why code is necessary for multi-turn instruction following. We further clarify the apparent drop on subjective constraints reported in Table 4.

  • CoCa is scored on many more and much deeper turns. EvolIF ends a dialogue after three consecutive failures, so CoCa, which survives longer, is evaluated on 1.57–1.92× more constraint occurrences than Direct, summed over the six executors (Table 4). Its subjective rates therefore average in late turns that Direct never reaches.

  • On the turns both systems actually reached, the subjective gap essentially disappears:

    ExecutorSubjective pass rate (style, reader_age, emotion)DirectCoCa
    Granite 4.1 3Bsame turns (shared depth, n = 519)67.668.6
    deeper turns reached only by CoCa (n = 494)—65.6
    Gemma-4 26Bsame turns (shared depth, n = 805)79.578.8
    deeper turns reached only by CoCa (n = 585)—72.3

    On the same turns, CoCa raises verifiable pass rates by about 24 points (Granite: 57.0 → 81.4) and 14 points (Gemma: 72.7 → 86.3) while keeping subjective constraints level. Matched over six executors, the paper's shared-occurrence table shows the same pattern: the style and reader_age deltas shrink from −8.1 / −8.0 points (unmatched) to −3.7 / −5.7 points.

  • Deep EvolIF turns make subjective and verifiable constraints compete. As a dialogue deepens, each turn accumulates more standing constraints, and in EvolIF they are frequently in direct tension. Direct prompting shows the same decline: on Gemma-4 26B its own subjective pass rate falls from 86.5% (turns 1–5) to 57.1% (turns 21+). An example from dialogue 123, turn 20, which Direct never reaches (it fails out at turn 3), requires at once:

    • at least 73 uppercase words;
    • more than 6 paragraphs;
    • a formal style;
    • "seasoning" ×1, "health benefits" ×2, "indian cuisine" ×3;
    • end with a quotation delimited by { };
    • written for a youth reader.

    CoCa satisfies all four verifiable constraints and formal style; the only failure is youth readership. It is hard to make a formal, heavily uppercased, 745-word, 7-paragraph text also read as written for a youth reader. This is a property of how EvolIF stacks constraints, not something a carrier can resolve: whichever system survives to such turns must trade these constraints off.

The rewrite loop does not guarantee compliance. We agree, and we did not claim that it does; we will make this explicit. What holds is narrower: when a constraint is enforced by deterministic code (for example, uppercasing, appending a required end marker, or removing a forbidden word), that check and repair hold by construction. Constraints repaired through an LLM_call rewrite are bounded, best-effort attempts.

AuthorsOpenReview ↗

[3/5] Response to W3

We thank the reviewer for this careful accounting. We agree that Figure 4 counts only the input to the call that carries dialogue state, and we have now measured every stage, including user intent classification, program generation, generated code, executor calls and retries.

Adding the stages Figure 4 leaves out (same run as Figure 4, input tokens summed over turns 1–50):

What is counted for CoCaCoCa inputDirect / CoCa
Figure 4: carried state + generator in-context prompt208.0k2.9×
+ generator system prompt and user turn235.6k2.6×
+ intent classifier364.0k1.7×
+ executor input (≈300 tokens/turn, measured on an instrumented run)≈379k≈1.6×

With every stage counted, CoCa still uses about 1.6× less input than direct prompting over a 50-turn dialogue. CoCa is more expensive early on and breaks even around turn 16, but by turn 50 Direct uses about 3.3× more input per turn (≈24.0k vs. ≈7.3k tokens). CoCa does generate more output (mostly for program generation) and makes more calls per turn (≈3.65 calls in total, as shown in our response to W1).

KV-cache reuse. Most of CoCa's input is a fixed prompt that is identical across turns (instructions and in-context examples), so it is shared as a prefix. Only the carried program and the user turn change. With vLLM prefix caching, the generator's cache hit rate was 78% over the full run, and the executor's 58%. Direct prompting does reuse the growing transcript prefix, but it must still attend over the whole transcript, which grows without bound.

Latency. CoCa runs three dependent stages per turn (classify, generate, execute), so it can be slower than a single direct call, and we therefore do not claim a latency advantage. CoCa trades latency for its substantial accuracy gains over Direct. Straightforward reductions are available and orthogonal to the method, such as running the classifier in parallel with generation, lowering the generator's reasoning effort, or using a smaller generator. We will report these measurements in the revision.

Does the 3B result depend on the 120B generator? We do not claim that a 3B model alone matches a 120B model. The gain comes from the representation, not from capability borrowed from the generator:

  • The 3B model writes every answer. The 120B model is called once per turn to write only the program. With CoCa, Granite 4.1 3B reaches 60.1 ISR, above GPT-OSS-120B prompted directly (57.4).
  • The gain persists without a stronger generator. With GPT-OSS-120B as both generator and executor, CoCa still beats the same model prompted directly: +10.4 ISR on EvolIF (57.43 → 67.81) and +3.0 on Multi-IF (81.21 → 84.25). This gain is close to the trend fitted to the other five executors.
  • The gain barely depends on which generator is used. With a Gemma-4 26B executor, all three generators give similar ISR gains: +8.4 (Qwen3-Coder-Next), +9.9 (GPT-OSS-120B) and +10.7 (Claude Haiku 4.5).

We will update Figure 4 and include a more thorough cost discussion for CoCa.

Reviewer_ois4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Code as Carrier maintains a topic-specific executable program as the state of a multi-turn conversation. An intent classifier and router select the relevant topic; a generator rewrites its program; execution invokes a separate language model and can validate or revise the response. The paper evaluates six executors on Multi-IF and interleaved-topic EvolIF, reporting substantial improvements for weaker executors and broad aggregate gains in longer conversations. Additional experiments study prose carriers, routing, and generator choice. The reported context savings concern the input to the state-carrying call, while the complete pipeline also includes classification, execution, and possible revision calls.

Critical strengths
  1. The system provides an interpretable way to maintain evolving constraints across topic switches. Explicit state updates, executable checks, and topic routing form a coherent implementation. The routing ablation adds useful evidence that retaining a program alone does not account for all the improvement. This is a practically relevant contribution, particularly for mechanically checkable requirements. [pp. 3–4, 7–8]

  2. The complete system achieves substantial gains across multiple executors. Granite 3B's mean Multi-IF score rises from 56.61 to 79.11, and all eight EvolIF aggregate metrics improve for all six executors. Same-generator carrier comparisons and three generator choices supply additional evidence. The authors also disclose mixed results, including a small Multi-IF regression for Gemma and weaker semantic-category outcomes. Execution failures are counted as failed turns, which is an important aspect of the evaluation. [pp. 5–8]

Critical weaknesses
  1. The main reservation is contribution identification against a resource-comparable explicit-state alternative. The default system adds a GPT-OSS-120B generator and allows executor revision calls. Although the same-generator prose ablation is useful, its state is the previous assistant answer rather than a persistent ledger of active constraints; the rolling summary also lacks persistent topic state. Consequently, the results establish that the complete enhancement works, but do not clearly identify what the executable carrier adds beyond persistent constraint management and additional inference. This is a specific comparator gap, not a requirement to isolate every component or prove a unique causal mechanism. [pp. 4, 7–8, 21]

  2. The headline context evidence cannot independently establish the full system's efficiency contribution. Figure 4 explicitly counts only state-carrying input tokens, excluding classification and execution/revision costs. Moreover, retaining a prefix of every older answer permits total state to grow with conversation length. The observed near-flat curve is useful evidence in the tested regime, but neither 2.9× end-to-end savings nor an unrestricted constant-context guarantee follows. These limits narrow the remaining contribution rather than demonstrating that the method is inefficient. [pp. 8–9, 23]

Questions for authors
  1. How does a persistent per-topic text or JSON constraint ledger perform with the same generator, router, and comparable validation/revision budget? One representative matched comparison would directly affect my assessment of the carrier contribution. Please distinguish permitted call budgets from actual calls used. [pp. 7–8, 21]

  2. What costs are incurred across all stages over matched dialogue horizons? Is there a global cap on total answer history and program size? If not, please scope the constant-context statement to the observed regime and clarify which efficiency claim remains supported. [pp. 8–9, 21–23]

4 comments
AuthorsOpenReview ↗

[1/3] Response to W1

We thank the reviewer for the careful and constructive review, and for identifying an important comparator gap. As also noted in Q1, a persistent per-topic text or JSON constraint ledger is a natural stronger baseline than Direct.

We first clarify the key distinction: in CoCa, the program plays two roles.

  1. Carrier: it stores the active per-topic constraint set and updates it across turns. A ledger can also do this.
  2. Enforcer: it checks the generated response against each constraint in code and deterministically repairs violations such as keyword counts, length, and markers. A ledger can only state constraints; compliance is still left to the model or to model-based self-checking.

To test this directly, we added the reviewer’s suggested baseline and will include it in the revised paper.

Setup. We replaced the program in CoCa with a per-topic JSON ledger, {topic, task, constraints: [{id, text}]}, while keeping the generator (GPT-OSS-120B), user intent classifier, topic router, executor, and execution harness fixed. At each turn, the routed topic’s ledger is updated with the same keep/replace/remove/add operations as in CoCa, using a one-to-one conversion of CoCa's in-context examples. Constraints are free text extracted from the user turns. We evaluate two arms:

  • Ledger: the executor answers once from the ledger (with one retry if the answer is empty).
  • Ledger + self-revise: the executor drafts an answer, checks it against the numbered constraints, and revises it, under a hard cap of 5 executor calls per turn. This is the comparable validation/revision budget the reviewer asked about.

Results (EvolIF, 150 dialogues, P_max = 3). Executor calls include retries and timeouts.

ExecutorArmPermitted executor calls / turnActual executor calls / turn (mean)EDR lenCSRISRRECROB
Granite 4.1 3BDirect11.006.7562.3542.4514.1232.68
Ledger2 (one retry if empty)1.008.4569.7951.1419.3840.65
Ledger + self-revise52.759.3573.2852.8221.7742.62
CoCanot hard-capped (prompted ≤ 5)1.6510.9877.8959.1422.1950.35
Gemma-4 26BDirect11.009.7176.1556.5218.9646.02
Ledger2 (one retry if empty)1.0112.6281.4964.2425.9353.63
Ledger + self-revise53.2813.0182.4565.3724.8055.78
CoCanot hard-capped (prompted ≤ 5)1.5114.4483.9067.5426.2360.63

The advantage is concentrated where checking requires exact verification. Per-constraint pass rate (%), shown as Ledger / Ledger + self-revise / CoCa:

ConstraintGranite 4.1 3BGemma-4 26B
existence (required keywords, exact counts)15.1 / 13.4 / 51.051.2 / 53.7 / 74.5
case60.7 / 59.2 / 85.562.7 / 68.3 / 87.9
length50.0 / 57.9 / 71.172.8 / 75.8 / 78.3

CoCa outperforms the strongest ledger baseline on every metric for both executors, while using fewer executor calls than the self-revising ledger. The gains are largest for constraints requiring exact verification via code.

AuthorsOpenReview ↗

[2/3] Response to W2

We agree that Figure 4 supports a claim about state-carrying context but not the 2.9$\times$ end-to-end saving, and we have now accounted for the input of every stage.

We first clarify that Figure 4 uses the Llama 3.1 8B executor on the 150 EvolIF dialogues, turns 1–50 only (excluding the 120 later turns, 1.6% of all 7,620 turns). It counts only the input to the call that carries dialogue state:

  • Direct: the full re-fed transcript, which grows linearly.
  • CoCa: the generator's in-context prompt plus the carried program and prior answer(s), which stays nearly flat. Under this definition, over a 50-turn dialogue Direct uses 610.9k input tokens and CoCa uses 208.0k, giving the reported 2.9×. Per turn, they break even at turn 8, and by turn 50 Direct needs 5.7× more input.

However, Figure 4 omits the other CoCa calls: the intent classifier, the rest of the generator prompt, and the executor LLM_call prompts. Counting these gives the following updated comparison over the same turns:

What is counted for CoCaCoCa inputDirect / CoCa
Figure 4: carried state + generator in-context prompt208.0k2.9×
+ generator system prompt and user turn235.6k2.6×
+ intent classifier364.0k1.7×
+ executor input (≈300 tokens/turn, measured on a matched instrumented run)≈379k≈1.6×

With every stage counted, CoCa still uses about 1.6× less input than direct prompting over a 50-turn dialogue. CoCa is more expensive early on and breaks even around turn 16, but by turn 50 Direct uses about 3.3× more input per turn (≈24.0k vs. ≈7.3k tokens).

In the revision, we will add this all-stage cost analysis, update Figure 4, and scope the context claim to "near-flat in the tested regime (≤ 50 turns)" rather than constant (see response to Q2 for the context guarantee discussion).

AuthorsOpenReview ↗

[3/3] Response to Questions 1 and 2

Q1: How does a persistent per-topic text or JSON constraint ledger perform with the same generator, router, and comparable validation/revision budget? One representative matched comparison would directly affect my assessment of the carrier contribution. Please distinguish permitted call budgets from actual calls used. [pp. 7–8, 21]

See our response to W1 above (matched JSON-ledger comparison, with permitted vs. actual call budgets).

Q2: What costs are incurred across all stages over matched dialogue horizons? Is there a global cap on total answer history and program size? If not, please scope the constant-context statement to the observed regime and clarify which efficiency claim remains supported. [pp. 8–9, 21–23]

All-stage costs over matched dialogue horizons, and which efficiency claim remains supported, are in our response to W2 above. On caps for answer history and program size:

  • Multiple prior answers are passed only on related turns (Table 1), which make up 2.34% of EvolIF turns (Appendix Table 16). On these turns, <PriorAnswers> holds the routed topic's 4 most recent answers in full, each truncated to 2,000 characters, so at most 8,000 characters of full text. Every older answer is reduced to a 150-character head. All other categories carry at most the single immediately preceding answer, or none.
  • The heads are not capped in number, so the block grows slowly with the number of answers in a topic: about 36 tokens per additional answer beyond the fourth. In the Llama 3.1 8B run, <PriorAnswers> averaged 1.3k tokens and peaked at 3.4k tokens, for a topic with 27 answers.
  • Carried state per turn (program plus prior answers) grows from 0.9k tokens at turn 10 to 1.15k at turn 50, about 6.25 tokens per turn. Direct prompting's re-fed transcript grows by about 490 tokens per turn.
  • The program has no explicit size cap. Its median is 707 tokens (max 8.7k).
  • Every bound is configurable. The number of full answers, the per-answer limit, and the head length are run-time hyperparameters, and adding a hard global cap on total history is a one-line change in CoCa.
AuthorsOpenReview ↗

Follow up on rebuttal discussion

Dear Reviewer ois4,

Thank you again for your careful review. As the discussion period is about to close, we wanted to ask whether our rebuttal addressed your concerns. We would be glad to address any remaining questions before the deadline. If our responses resolved your concerns, we would be grateful if you would consider reflecting that in your assessment.

Thank you for your time and feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the reviews and our responses

Dear Reviewers, AC and SAC,

We thank the reviewers, AC and SAC for their time. All three reviewers found the problem important and the design practical, with explicit per-topic executable state, runtime checks and topic routing. They also noted consistent gains across six executors (3B–120B) on Multi-IF and EvolIF, especially for small models and long dialogues, and valued the ablations, which show that both the persistent program state and topic-aware routing contribute to the gains.

Our responses to the reviewers' concerns are summarized as follows (new experiment results will be added to the paper):

  1. Is code necessary? (R1, R2, R3) We added the requested topic-aware JSON constraint-state baseline, with the same generator, router and executor, in single-answer and self-revise arms (≤5 executor calls). CoCa beats the strongest baseline on every metric for both executors (e.g., Granite 3B ISR 59.1 vs. 52.8) with fewer executor calls. The gap is largest on code-verifiable constraints, e.g., existence 51.0% vs. 13.4%. We also position CoCa against MemGPT, User as Code, GraphIF, DSPy Assertions and Mellea.
  2. Efficiency claims (R1, R2, R3). We now count all stages: classifier, generator, executor and retries. The saving is about 1.6× (not 2.9×), and we report output tokens, calls and latency. We scope the context claim to "approximately constant in the evaluated regime (≤50 turns)" and document the exact bounds.
  3. Generator dependence (R2, R3). With a 120B generator and executor, CoCa still gains +10.4 ISR. A 15× smaller 8B generator still gains +8.3 ISR. The same generator without a program recovers only a minority of the gain.
  4. Routing and execution details (R2, R3). Thread-assignment accuracy is 77.6% on return-to-topic turns, better than embedding routing. We document the sandbox, time limits, allowed modules and error handling.
  5. Scope and models (R2). EvolIF has 50–70 turns per dialogue. We added a closed-source executor (Claude Haiku 4.5: +15.2 ISR, +20.6 ROB), and we clarify that CoCa complements RAG and does not replace it.
  6. Subjective constraints and matched turns (R2). On 13,601 matched constraint instances, CoCa improves 10 of 12 categories (macro +15.5), and the subjective drop largely disappears.

We have narrowed our efficiency, bounded-context and compliance claims to what we measured, and we believe every question has been addressed. If any concerns remain or the reviewers have further feedback after reading the rebuttal, we would be happy to address them during the discussion period. Thank you for your time and thoughtful reviews.

Best regards

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.