- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Code as Carrier (CoCa), a framework for maintaining evolving constraints in multi-turn conversations. CoCa uses intent classification and topic-aware routing to maintain topic-specific executable programs, which preserve active constraints and enforce them through validation and repair.
Experiments on Multi-IF and EvolIF with six executor models, 3B–120B, demonstrate substantial improvements over direct inference, particularly for smaller models and longer conversations. Ablation studies further investigate the effects of program-based constraint representation, topic routing, and generator selection.
Critical strengths
-
Meaningful problem formulation and a practical design. The paper addresses the important problem of maintaining evolving constraints across multi-turn conversations. Representing active constraints as persistent executable programs provides a useful alternative to repeatedly recovering them from dialogue history, combining explicit state management with runtime validation.
-
Substantial empirical improvements across models and settings. CoCa achieves consistent gains across six executors on two complementary benchmarks, particularly for smaller models and longer conversations. The ablation studies provide additional evidence for the benefits of persistent program states and topic-aware routing, while the per-category analysis reveals meaningful differences between mechanically verifiable and semantic constraints.
Critical weaknesses
-
Insufficient comparison with closely related methods. The main experiments compare only against one method: direct inference. Existing ablations use previous-response prose or short rolling summaries rather than a strong persistent constraint-state baseline. Related work such as GraphIF already supports structured constraint updates and topic tracking, while executable validation has been explored in DSPy Assertions and Mellea. Without a topic-aware explicit constraint-state baseline, the added value of executable programs remains unclear.
-
The contribution of the strong program generator is not sufficiently isolated. All main experiments use GPT-OSS-120B to generate and maintain programs, while only executor size varies. Although generator sensitivity is evaluated, the tested alternatives are also capable models. Consequently, improvements with small executors may partly reflect strong-model assistance rather than the proposed representation itself.
-
Incomplete evaluation of topic routing. The reported classification results primarily measure new-topic detection, not whether the system correctly retrieves the intended existing topic. Since accurate topic restoration is central to CoCa, the absence of thread assignment accuracy leaves an important component insufficiently validated.
-
Some claims exceed the supporting evidence. The reported 2.9× token reduction excludes parts of the inference pipeline, while bounded context and guaranteed constraint satisfaction are not established by the implementation. These claims should be restricted to the quantities and settings actually evaluated.
Questions for authors
-
Stronger baseline comparison. How does CoCa compare against a topic-aware explicit constraint-state baseline that maintains the complete active constraints in natural language or JSON, using the same generator, router, and executor? This comparison would help establish the additional value of executable programs.
-
Generator dependence. How much of CoCa's improvement remains when using substantially smaller or less capable program generators? Can the authors distinguish the benefits of programmatic constraint enforcement from those of strong-model-assisted constraint extraction and planning?
-
Topic routing accuracy. What is the actual thread assignment accuracy when a user returns to an existing topic? The reported new-topic detection metrics do not establish whether the correct historical topic is retrieved.
-
End-to-end efficiency. What are the total input/output token costs and inference latency when accounting for intent classification, program generation, executor calls, and validation retries? Is the context size strictly bounded, or only approximately constant in the evaluated settings?
The paper presents a meaningful approach to multi-turn constraint management, supported by substantial empirical improvements. However, its central contribution is not sufficiently distinguished from existing structured constraint-management and executable validation methods. In particular, the absence of a strong persistent constraint-state baseline makes it difficult to determine whether executable programs are responsible for the reported improvements.
8 comments
[2/5] Response to W2
We thank the reviewer for raising this. Three results separate the representation's contribution from the generator's strength.
-
The representation helps even when no stronger model is involved. When GPT-OSS-120B both generates and executes its own program (120B + 120B), CoCa still improves over that same model prompted directly: on EvolIF from 57.43 to 67.81 ISR (+10.4) and from 47.65 to 59.61 ROB (+12.0); on Multi-IF from 81.21 to 84.25 overall (+3.0). No capability is lent by a stronger model here, so the gain comes from carrying and enforcing constraints as a program. The size of this gain matches the trend set by the other five executors (residual +2.0 ISR on EvolIF, +0.6 on Multi-IF), so the generator does not favor itself.
-
The same generator helps less without the program. Our sumhist baseline uses the same GPT-OSS-120B to rewrite each turn into a self-contained prompt from a 3-turn window, with no program. On Granite 4.1 3B it recovers only +6.2 of CoCa's +17.6 ISR gain (35%), and 29% on Gemma-4 26B. A strong model rewriting the request therefore explains only a minority of the gain (Table 2). The matched JSON constraint-state baseline (W1) makes the same point: with the identical 120B generator extracting and maintaining the full constraint set, it still trails CoCa on every metric. That isolates programmatic enforcement from strong-model constraint extraction.
-
A much smaller generator still helps. In preliminary experiments, we replaced the 120B generator with a 15× smaller 8B generator (an 8B open-weight model fine-tuned on a separate synthetic single-turn dataset) and held everything else fixed: the same Granite 4.1 3B executor, the same intent labels, and the same topic routing. CoCa still improves over direct prompting on every metric: ISR 42.45 → 50.71 (+8.3), CSR 62.35 → 69.31 (+7.0), ROB 32.68 → 40.30 (+7.6), and EDR length 6.75 → 8.43 turns. Because these results are preliminary, we mention smaller generators only as future work in the conclusion, though we find them promising.
[3/5] Response to W3
We thank the reviewer and agree that thread-assignment accuracy should be reported directly. We measured it against EvolIF's ground-truth topic labels on all 150 dialogues (6,700 continuation turns, of which 955 return to an earlier, non-adjacent topic):
| Router (same topic anchors) | Continuation turns routed correctly | Turns returning to an earlier topic routed correctly |
|---|---|---|
| Embedding cosine, threshold 0.45 | 66.6% | 42.6% |
| Embedding cosine, threshold 0.35 | 74.6% | 58.7% |
| Embedding cosine, threshold 0.30 (best of sweep) | 77.7% | 67.1% |
| Token overlap (CoCa) | 81.7% | 77.6% |
CoCa restores the correct historical topic on 77.6% of return turns (Granite 4.1 3B; 77.5% with Gemma-4 26B), more accurately than embedding-based routing at any threshold. Most remaining errors are short follow-ups that name no subject (e.g. "make it shorter"); no lexical or embedding signal can route these, and the router leaves them on the current thread, which is usually correct. On turns that do name their subject, token overlap routes about 96% correctly.
Routing accuracy also has a measurable effect on outcomes:
- Mis-routing is costly: a turn carried into the wrong topic fails 81% of the time.
- Correct restoration adds a separate gain: disabling topic resumption while keeping the program costs 3.3 ISR on Granite 4.1 3B and 3.7 ISR on Gemma-4 26B.
We will add these thread-assignment results to the routing section.
[1/5] Response to W1
We thank the reviewer for the careful review and for specifying exactly the baseline that was missing. We have now run it (see also Q1), and it isolates what executable programs add.
We first clarify the key distinction: in CoCa, the program plays two roles.
To test this directly, we added the reviewer's suggested baseline.
Setup. We replaced the program in CoCa with a per-topic JSON state,
{topic, task, constraints: [{id, text}]}, while keeping the generator (GPT-OSS-120B), intent classifier, topic router, executor, and execution harness fixed. At each turn, the routed topic's constraint state is updated with the same keep/replace/remove/add operations as in CoCa, using a one-to-one conversion of CoCa's in-context examples. Constraints are free text extracted from the user turns. We evaluate two arms:Results (EvolIF, 150 dialogues, P_max = 3). Executor calls are counted at the executor client and include retried and timed-out attempts.
The advantage concentrates where checking requires exact verification. Per-constraint pass rate (%), shown as JSON constraint state / JSON constraint state + self-revise / CoCa:
CoCa outperforms the strongest constraint-state baseline on every metric for both executors, while using fewer executor calls than the self-revising baseline. The gains are largest for constraints requiring exact verification via code.
Relation to GraphIF, DSPy Assertions and Mellea. These works each address part of the problem, but none tracks constraints per topic, extracts them automatically from free-form user turns, and enforces them in code as the conversation evolves:
We will add the explicit constraint-state baseline to the main results and expand the related-work discussion to position CoCa against all three.