acceptodds
Under review as a conference paper at ICLR 2027

Delegation Under Test: Specification-Based Conformance Testing Across the LLM-Framework Boundary

Abstract

Language models operate within software systems whose frameworks maintain state, execute actions, and sometimes compensate for model errors. System success does not establish which requirements the model fulfills; aggregate scores can also conceal lost behavior after an inference change. We introduce Delegation Under Test (DUT), a specification-based method that turns fixed system requirements and responsibility allocations into component obligations and matched tests. Designers choose the contract and case support; executable specifications derive required actions, timing, and startup sets. Stage-specific scoring retains attributable violations and uses aligned evidence for downstream comparisons and regression judgments. We study state exposure and task startup assignment on a structurally selected suite. Passing explicit-state tests can coexist with phase and prerequisite failures when history is present. A correct business-level WAIT can violate model-assigned startup, and correct, aligned startup can precede different downstream timing. A crossed control changes the interpretation of a negative full-interface state-supply contrast; even nonnegative contrasts lose previously passing obligations. Repeated format comparisons exchange passing identities at unchanged totals, and a device change with weights and partitioning fixed also changes conformance. The construction and findings distinguish local correctness, assigned-role fulfillment, and preservation of behavior an application relies on.

Primary area:
datasets and benchmarks
Keywords:
LLM evaluation, behavioral conformance, specification-based testing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NhXjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary
  • Language models often run inside frameworks that track state, run tools, and fix some of the model’s mistakes - so a successful e2e run doesn't tell you which requirements the model itself met.
  • The paper proposes Delegation Under Test (DUT): a designer writes an executable workflow policy and decides which actor (model or framework) is responsible for each action, then DUT then derives what the model must do under each choice and builds matched tests with oracle-computed actions and timing
  • 2 choices are studied: (1) whether the model gets an explicit current-state summary alongside its event history, and also (2) whether the framework or the model must start the required tasks.
  • Experiments cover 14 models on text tests, local Qwen and GLM models on tool-based tests with repeated rounds, and regression comparisons across several different quantization formats, GPU layouts, and reasoning settings.
  • The main findings are that (a) models can pass simple state-only tests but fail when the same facts are buried in history, that (b) supplying state usually helps but can break cases that passed before, and that (c) format or device changes can swap which cases pass while the total score stays the same.
Critical strengths

Strength 1: Careful and checkable measurement

The paper works pretty hard to make sure each reported failure comes from the model and not from a bug in the test harness. For example, oracles are checked against independently written truth tests, every response is linked to its logged request, and infra failures are kept separate from model failures. Since the paper’s claims are about specific test cases flipping, this level of checking is what makes those claims believable

Strength 2: Clear evidence that equal scores can hide real changes

For example, Qwen3.8 (27B) scores 61/72 under both Q6K and Q80, but 1 case that passed now fails and another that failed now passes, in every round (as shown in table 4). The GLM control in Table 2 is also pretty good science: the paper’s own first reading of a negative effect turned out to come from how the interface heading was worded, and the paper reports this openly

Critical weaknesses

Weakness 1: No evidence that DUT finds more than standard testing

The paper admits that an ordinary timing test catches its opening example (see section 1) and that ordinary regression testing catches the Q6K/Q80 swap (see appendix C3). The only comparison against a simpler method is the retail example - but which uses a hand-built controller with no language model. Without a case where DUT leads to a different conclusion than standard practice on real model outputs, the method reads as a careful formalization of existing testing ideas rather than a novel contribution. I think this is fixable with a direct comparison.

Weakness 2: small synthetic suite and single-case findings

The tests come from 5 policy templates, and the 72 units are mostly surface rewrites of the same 8 underlying cases - and several key results rest on 1 changed case. Also, in the Qwen3.8 study, the repeated rounds produce identical outputs, so repetition only shows that the setup is deterministic. It doesn't show that the flips are stable or meaningful. Prior work has already shown that compressed models can keep accuracy while flipping individual answers (see Dutta et al - 2024), so the existence of such swaps is not new by itself.

Questions for authors
  1. Can you please show a case on real model outputs where DUT’s rules lead to a different accept or reject decision than a standard regression suite that keeps test IDs and checks timing?
  2. For the Qwen3.8 (27B) format and device flips -- do the changed cases sit near a tie in the model’s output probabilities? I think that results with sampling temperature above zero would show whether these flips are stable.
  3. Do the startup failures persist if the startup instruction is reworded or if the prompts are in English instead of Chinese?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 3 (1/4)

Thank you for a precise reading. We sincerely apologize for the suboptimal reading experience caused by the previous version. To address this, we have provided a revised manuscript aimed at improving the clarity and readability of the main text.

In the revision, the opening example you cite is kept in §1, now followed by a second one; Table 4 is now Table 3; and the Appendix C.3 material you cited is now in Appendices D.8 and F.1.

W1 & Q1: does DUT reach a different decision on real model outputs?

Yes. Your summary describes DUT as deriving "what the model must do under each choice" and building "matched tests with oracle-computed actions and timing". A regression suite that keeps those tests, with their IDs and their timing, is the suite DUT builds, so its pass and fail verdicts are DUT's own. The suites a team has without DUT are built from the inputs at hand and protect the passes a baseline already has. On the same outputs of real models, DUT's decisions differ from theirs in four ways.

First, DUT judges a call under its allocation. In the new startup composition (§4.2), Qwen3.5 BF16 states "All relevant tasks are NOT_STARTED. Therefore, rule 2's IF condition is unsatisfied, and the policy returns WAIT," and calls the waiting tool. A test of the business decision passes this response; the submission's appendix already noted that a check of business labels credits this WAIT (now App. D.8). When startup is assigned to the model, DUT rejects the response in all three rounds, because none of the three required tasks was started. When startup is assigned to the framework, the same first call conforms. The same call and the same correct decision receive opposite verdicts, and the delegation in the title decides which. A test keyed only by its input cannot express this difference.

Second, DUT enforces mandatory obligations. A suite that keeps test IDs protects what the baseline passed. In §4.3, one Qwen3.8 phase guard is violated in eight renderings with history alone in every format, layout and round, and is fulfilled in every configuration once state is supplied. Read retrospectively, protecting prior passes finds nothing lost and accepts every one of these changes. An acceptance contract that names this obligation mandatory rejects them all and points to the remedy: supply state. The Discussion states the rule: adoption must also satisfy mandatory requirements, which can include requirements that the baseline never fulfilled.

Third, DUT reads verdicts per identity. Your Strength 2 and the Dutta et al. reference both find aggregate scores insufficient: - Qwen3.8 Q6_K and Q8_0 tie at 61/72, which aggregate scoring reads as interchangeable. - Supplying state raises or preserves the total in every configuration of Table 1. Yet it loses 10 previously passing trajectories for DeepSeek V4 Pro and 6 for MiniMax M3, all of them new waits, so a rule that retains prior passes rejects state supply for both. - A rule suite on explicit state passes 12 configurations of Table 1, and 11 of them violate the same requirements given history.

Fourth, DUT checks every action the model owes. In the return study, a check of the final state, as τ-bench uses, accepts the clarified prompt with 32/32 correct database outcomes. DUT rejects one response (31/32) for omitting an authorized submission (§4.4).

Where a suite that keeps test IDs and DUT both reject, DUT names the remedy. The suite detects the Q6_K/Q8_0 exchange, as the passage you cite says (Appendix C.3, now D.8), and protecting each identity rejects Q8_0. The submission presented that rejection as an illustration, not as a registered adoption rule. What the suite cannot see is that the two IDs are near renderings of one phase guard under the same allocation, one in software infrastructure and one in data research. The guard says: during preparation, advance the phase whatever other completions show. - With history alone, Q6_K violates it in one rendering and Q8_0 in the other. - With current state, both formats pass every rendering in every layout and round (72/72). - Under sampling (Question 2), both formats violate both renderings in 16–20 of 32 samples, while the control with explicit state passes 32/32 in every configuration.

Neither format fulfills this obligation under history alone, and supplying state fulfills it under both: the remedy is a change of delegation, not a choice of format. In the Qwen3.5 BF16→Q6_K startup regression, by contrast, BF16 passes all twelve registered identities under model startup in all ten rounds on fresh servers, while Q6_K omits startup in its business case with phase selection in all ten. Here a fulfilled obligation is lost, and rejecting the format is the remedy (§4.3).

What DUT adds is what each verdict is about: the obligation a failure refutes and the delegation under which it does (§3.2). DUT also determines which tests exist and what a changed identity calls for (§4.1–§4.4).

AuthorsOpenReview ↗

Response to Reviewer 3 (2/4)

W2: small synthetic suite and single-case findings

The eight cases follow a test selection criterion (Goodenough & Gerhart, 1975; Zhu et al., 1997): each marks a point where a fact's relevance changes between two decisions, with two structures for each of the four possible changes (§3.1). They and their four-answer units, with every required action, were fixed before any model was run. Whether a model fulfills an obligation is a universal claim over the obligation's legal inputs: one attributable violation refutes it, however the case was chosen, and passes establish conformance on the inputs tested. We therefore read a violation as a refutation and a pass as conformance on the stated coverage, not as a rate over policies. The 72 units are 8 cases × 9 renderings, and §4.1 and Figure 3 count them that way. Two findings recur well beyond one case:

  • the §4.1 gap between explicit state and history, across 11 configurations;
  • the direction of failures under supplied state, across 14 configurations × 81 trajectories.

The single-case findings in §4.3 are counterexamples, which is the evidence a preservation claim calls for. Each one refutes the claim that an implementation change preserves an obligation, or shows that no implementation fulfills it under the given delegation:

  • one obligation of a phase guard exchanged between renderings in every layout and run, which neither format fulfills with history alone;
  • one OR obligation whose passing renderings fall from 8/9 to 3/9 per schedule (BF16→Q8_0);
  • one startup obligation that BF16 fulfills in all ten rounds and Q6_K loses in all ten.

The oracle checks and request linkage that your Strength 1 credits make each one attributable.

The identical repeats establish determinism, the control that makes the Qwen3.8 exchange attributable. Fresh processes reproduce the historical output tokens exactly (all 24 greedy reads, Question 2), so the exchange belongs to the formats and not to run-to-run noise. Stability under sampling and across server realizations is a separate question, answered by:

  • the sampling study (Question 2);
  • the ten startup rounds on fresh servers, where outcomes do vary: Q6_K's business OR case fails rounds 1–9 and passes round 10, yet every failing round has the same first violation (App. D.5);
  • the study in three languages at temperature 1.0, where pass³ and pass@3 separate stable from unstable failures (Question 3).

§2 now cites Dutta et al. (2024), and our margins fit their account of near ties: every flipped pair differs by less than 0.2 nats. DUT adds where the flips fall and what they call for. In the Qwen3.8 comparisons, every identity that a format, layout or device change flips is a test with history alone, and its counterparts with state pass in every configuration (§4.3). The remedy is therefore to supply state rather than to pick an implementation. Losing an obligation fulfilled in every round, as in the startup case, calls instead for rejecting the implementation.

AuthorsOpenReview ↗

Response to Reviewer 3 (3/4)

Q2: do the changed cases sit near a tie, and are the flips stable at temperature above zero?

Every changed case sits near a tie, which is why the flips are unstable while the violations are stable. The tie is the microscopic account, while DUT gives the macroscopic one: the violation belongs to the obligation under this delegation, and supplying state is the remedy, as in Weakness 1.

Above temperature zero, both implementations violate each affected requirement in 16 to 20 of 32 samples, with no consistent advantage for either, while the counterparts with explicit state pass every sample. The tie therefore lies between fulfilling and violating one obligation: the model fulfills it in at most half of the samples under this delegation, whichever implementation runs it, and supplying state removes the tie.

The near ties are measured on the complete recorded output strings. For each retained comparison, we give log p(required output) − log p(violating output), including termination tokens, at temperature 1. The sign changes in every pair, every margin is below 0.2 nats, and the competing first output tokens are also close in probability. These are probabilities of specific strings, not of every way to express an action. All 24 fresh greedy reads reproduce the historical output tokens.

Retained inputChangeGreedy outcomeLog-probability difference
Preparation, data research, similar task namesQ6_K → Q8_0pass → fail+0.1259 → −0.0205
Preparation, software infrastructure, similar task namesQ6_K → Q8_0fail → pass−0.1949 → +0.1203
Preparation, data research, dissimilar task namestail device I → Tfail → pass−0.0472 → +0.0811
Conjunctive shipping, retained decision prefixtail device I → Tpass → fail+0.0399 → −0.0598

We conjecture that this is the microscopic mechanism of every flip in the table: a small numerical change, such as a different quantization format or tail device, moves greedy decoding across the tie. It predicts what you expected: under sampling, which implementation passes should not be stable. To test it, we sample 32 seeds per cell at temperature 0.7, with top-p 1, no top-k and no repetition penalty, keeping reasoning off and the same worker and prompts. Every response completes in the required output form, so every failure is a wrong action. The counts confirm the prediction:

Retained inputViolations / 32, baseline → candidate
Format: preparation, data research16 → 16
Format: preparation, software infrastructure20 → 17
Device: preparation, data research19 → 17
Device: conjunctive shipping decision16 → 16
Control with explicit state, all four configurations0 in every configuration
Reviewer_X47xOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DUT translates system requirements and model–framework responsibility allocations into matched conformance tests. It distinguishes component obligations from overall system success and tracks individual behavioral regressions across interface and inference changes. Nevertheless, I am not an expert in this field and thus give my verdict with a lower confidence.

Critical strengths
  • Executable oracles, stage-specific evidence rules, and matched controls support careful attribution of observed violations.
  • Tracking passing identities reveals regressions hidden by unchanged aggregate scores; repeated experiments strengthen several observations.
Critical weaknesses
  • Most model evidence comes from structurally selected policy cases and their renderings. The retail transfer uses deterministic fault injection rather than an LLM-controlled workflow, leaving transfer to realistic agent execution insufficiently demonstrated.
  • The methodological advance over established specification-based testing and per-case regression analysis could be clearer. A direct comparison showing the additional diagnostic value of DUT over a simpler combination of these practices would strengthen the contribution.
Questions for authors

See weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 2 (1/2)

Thank you for the accurate summary and for noting the executable oracles, stage-specific evidence rules and identity tracking. We address your two concerns.

W1. Structurally selected cases, and a retail study without a language model.

Our cases follow a test selection criterion (Goodenough & Gerhart, 1975; Zhu et al., 1997): a rule, fixed in advance, that states which inputs a suite must contain. This criterion requires a case for each structure where a fact's relevance changes between two decisions, and the structures differ in what they test (AND/OR joins, result guards, phase guards, background progress; §3.1).

The executable oracles, stage-specific evidence rules and matched controls you credit are what make each violation attributable. Passes, in turn, establish conformance on the inputs tested, and the selection states which inputs those are. The selection is a coverage criterion in software testing: each structure marks a point where a fact's relevance changes between two decisions, and the structures differ in what they test.

We expected such elementary obligations to be saturated, but THEY ARE NOT: of the twelve configurations in Table 1 that pass every unit with explicit state, eleven violate them given history, including GPT-5.6 and DeepSeek V4, while Gemini 3.8 Flash (low) passes every test, which corroborate that the obligations are attainable.

Furthermore, the studies are organized by allocation rather than by sampled policies (§1, Figure 1). Within each allocation the required actions stay fixed while we vary:

  • 14 model configurations (Table 1);
  • three input conditions;
  • nine renderings;
  • three languages with three sampled rounds (our response to Reviewer 3, Question 3);
  • several inference implementations, with rounds on fresh servers.

The main pattern recurs along all of these axes: a rule applied correctly to explicit state is violated given history.

On the return study, each part does a different job. The deterministic controllers are an instrument check with known ground truth: the controller deliberately omits its assigned checks, so every violation is known in advance, and the tests must find each one even where the final state is protected. They do, including 28 of the 32 violations before dispatch in which the tool already leaves the database unchanged (§4.4). A language model in that role would confound the instrument's validity with the model's behavior.

The new study with a real model then keeps the allocation fixed: DeepSeek V4.1 Flash, reasoning disabled, decides whether to authorize and submit returns on 16 snapshots observed twice per prompt, and the unchanged τ-bench tool checks delivery and refund eligibility. In this study the tests:

  • located a model error;
  • verified a short prompt clarification (25/32 → 31/32, with all 25 prior passes retained);
  • showed that a check of the final state would accept the repaired prompt at 32/32 while one response still omits an authorized submission (§4.4).

The next step follows: because the contract computes the required behavior of any legal input, legal inputs can be generated automatically and at scale to test fulfillment, as the final paragraph of the Discussion now states.

W2. The advance over specification-based testing plus per-case regression.

Each DUT verdict is an ordinary assertion, so any harness can run a DUT test once it exists. Your second strength names the part DUT shares with per-case regression, tracking passing identities.

The table in our next comment shows where reading the same identities per obligation and under each allocation changes the decision or the remedy. The advance is in which tests exist and how their verdicts are read.

The unit under test is an obligation(not a simple call) of the model as a component: the construction runs from the specification to the obligation a delegation assigns to the model, and only then to interfaces and test inputs (§3.2; App. A.1), so one obligation is tested across related inputs, as events unfold, under each delegation and across implementations.

Specification-based testing plus per-case regression over the inputs at hand does not supply four things by itself, while a regression suite that has all four, is the suite DUT constructs:

  • related inputs whose required actions must stay the same or must differ;
  • the same decisions under different information conditions;
  • startup checked before any later behavior is judged;
  • verdicts read per obligation rather than per test ID.

The table in our next comment shows, on the same recorded outputs, where reading these identities per obligation and under each allocation changes the decision or the remedy.

AuthorsOpenReview ↗

Response to Reviewer 2 (2/2)

Here is the direct comparison you asked for: each row applies a simpler practice and DUT to the same recorded outputs, so the table shows the additional diagnostic value of the four parts listed in our previous comment. In five of its seven rows, DUT reaches a different accept or reject decision; in the other two, it reaches the same decision and changes what it means and what should be fixed.

Practice applied to the same outputsIts conclusionDUT's conclusion
Rule tests on explicit state (Table 1, Anchor)Accept all 12 configurations that conform on all 72 unitsReject 11 of the 12: they violate the same requirements given history (3–41 units), and 3 still fail with state supplied (§4.1)
Aggregate comparison of supplying state (Table 1)Accept: state raises or preserves the total in every configurationReject state supply for DeepSeek V4 Pro and MiniMax M3, which lose 10 and 6 passing trajectories to new waits. Across all configurations, failing trajectories fall from 228 to 62 and premature actions from 122 to 26 (§4.1, App. C.2)
Check of the business decision on a startup trace: Qwen3.5 BF16, new compositionPass: the response decides WAIT correctly and calls the waiting toolReject under model startup, in all three rounds: the model waits before starting the three required tasks. Under framework startup, the same first call conforms (§4.2, App. D.8)
Regression that keeps test IDs: Qwen3.8 Q6_K→Q8_0Reject Q8_0: a protected test ID is lost at an unchanged 61/72Reject as well. Read per obligation, the two exchanged IDs render one phase guard: each format violates it in one rendering given history alone, and both pass every rendering in every round with state. The remedy is to supply state rather than choose a format (§4.3, App. D.8, E.6)
Protection of prior passes, read retrospectively: a Qwen3.8 phase guard violated by every implementationAccept every format, layout and device change: nothing is lostReject every change, with the obligation mandatory: the violation recurs in eight renderings given history alone in every format, layout and round, and the obligation is fulfilled whenever state is supplied (§4.3, App. E.5)
Regression that keeps test IDs: Qwen3.5 BF16→Q6_K startupReject Q6_K: three startup IDs are lostReject Q6_K as well, and here rejecting the format is the remedy: Q6_K loses an obligation that BF16 fulfills on all twelve registered identities in all ten rounds on fresh servers, omitting startup in its business case with phase selection in every round (§4.3, App. D.5)
Check of the final state in the return study (real model)Accept the clarified prompt: the database is correct for 32/32Reject one response (31/32): it omits an authorized submission that the downstream tool would have rejected anyway (§4.4)

Rows two and four to six are new analyses of existing records; row three was in the submission's appendix (now App. D.8), and the figures in row seven were in the appendix and now appear in §4.4.

If one row of the comparison table does not convince you, or one reported violation does not seem attributable, telling us which would let us answer it directly.

Reviewer_f4mFOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors introduce Delegation under Test, a framework that treats LLMs as software components and encodes design constraints.

Critical strengths

Due to the issues with presentation, I did not understand the paper well enough to be able to identify a critical strength.

Critical weaknesses

Even after carefully reading the paper multiple times, I'm struggling to find out what its central research question is. While the authors pose the question "Which specified behavioral requirements are fulfilled under different allocations of responsibility between the language model and its surrounding framework?" in the introduction and that is certainly an interesting question, I cannot see a clear attempt to operationalize either the term "surrounding framework" or "allocations of responsibility". (The paper does provide some example of specified behavioral requirements.) On requirements: It appears that requirements are modeled as some sort of finite state machine. In the appendix, the paper introduces a "frozen condition language" and defines it inductively. I don't understand what a "frozen" language is, and I was very confused to hear about it because the main text barely mentions it (as far as I could tell). The paper also uses various terms that I did not understand in this context, e.g. "retail adaptation" (pg. 9 bottom).

Questions for authors

What is a specific research question that the paper answers?

What is a "retail adaptation" of a language?

Can you explain a specific practical instance in which your framework improves LLM behavior?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 1 (1/2)

Thank you for reading the paper several times. We sincerely apologize for prioritizing text compression over reading experience, which resulted in failure to show how "surrounding framework" and "allocation of responsibility", become things we vary and measure, and state its answer in plain terms. We hope the revised §1 and the reply below can address all concerns but not sure. We have used these terms for so long that we may no longer see where they stop explaining themselves, the familiar curse of knowledge. If any term or sentence, in the revised paper or in this reply, still does not read clearly, we would be grateful to know which one. We will explain it here and rewrite it during the discussion.

Q1. What specific research question does the paper answer?

Delegation Under Test (DUT) is the method that turns the general question into specific ones that tests can answer. Given a system requirement and an allocation, it works out what the model itself must do, builds matched inputs, and has a program(the oracle), compute the required action for each. Each answer depends on what DUT builds. For ExamThe first needs the same rule, the same facts and the same required action in three forms of input, so that a failure cannot be blamed on a harder task. The second needs to know that the same decision to wait is correct when the framework has started the tasks and a violation when the model must start them; DUT derives this from the allocation and checks startup before anything else.

For the two allocations we study, the questions are:

  • If a model applies a rule correctly when it is given the current state, does it still take the required action when the same facts reach it as an event history?

    • No. In Table 1, twelve of the fourteen model configurations apply every rule correctly given only the state; eleven of those twelve violate the same rules given the history, and three still fail when the state is given alongside it. For example, a rule allows action Q only after checks K and N finish, and only N has finished. Given the state, DeepSeek V4 Flash 0731 correctly waits. Given the history, it executes Q, even when the input also lists K as pending.
  • If starting the required tasks is assigned to the model rather than done for it, does the model start them?

    • Not always. Qwen3.5-9B BF16 correctly decides to wait, then waits without starting any task, in all three rounds. A larger model shows the same omission: MiniMax M3 waits without starting tasks in one of three rounds of another phase case.

We did not expect the result. The rules are elementary, and the main decision tests were fixed before any model was run; we expected current models to pass them all. The result surprised us enough that we first suspected our own oracle. We then ran another model, Gemini 3.8 Flash, on the full test set, and it passed every test.

W1. the term "surrounding framework" or "allocations of responsibility"

In our experiments, the framework(application framework, also surrounding framework) is the code that builds the model's input, delivers events and runs its tool calls. An allocation of responsibility, the "delegation" in the title, is a choice of what that code does for the model. In this paper, we tested two delegations as example:

  • State supply. The input contains the policy and the event history, and either includes a block giving the current state or does not. The history is identical and the required action is the same. A third form gives only the state, which tests the rule on its own.
  • Startup. Before the model is called, the framework has either already started the required tasks, and the model must not start them again, or has started none, and the model must start them before waiting or acting. In the second case, starting them is part of the model's obligation.

A system requirement states what the application needs, such as shipping on time; the model's obligation is the part of it that the allocation leaves to the model. What we measure is the same in both settings: whether the model's action equals the action that the oracle computes from the rule and the state. In the decision tests, a unit of four related decisions passes only if all four are right; in the startup tests, startup is checked before any later behavior is judged.

An interface implements an allocation through the model's instructions, observations and permitted tool calls. Several interfaces can implement the same allocation, and we keep each test across them.

The four experiments follow from this. Two change what the framework does: whether it supplies the current state and who starts the tasks. One keeps the allocation fixed and changes how the model is run, such as its number format or hardware. One repeats the method on a second application product returns, where a downstream tool can hide a model's error; such blocking is not an allocation, and the error still violates the model's obligation.

AuthorsOpenReview ↗

Response to Reviewer 1 (2/2)

W2: are requirements a finite-state machine, and what is a "frozen condition language"?

Your reading is close: the workflow is a finite-state system, each requirement is a condition–action rule over its state, and the "frozen condition language" is the grammar of those conditions, held fixed so that every comparison shares one oracle.

The state is the current phase, the facts, and each task's status and result; events change it, and each task goes not started → pending → completed or failed. A rule's condition combines facts, task completions and task results with AND, OR and NOT, and in each state exactly one rule applies.

The oracle is the program that evaluates it: it finds the applicable rule, evaluates its condition on the state and returns the required action. We test whether the model takes that action, and the state machine defines which inputs are legal (§3.1).

"Frozen" only meant that the grammar is held fixed across every comparison we make. The revision drops the word: Appendix A.2 is now titled "Specification grammar and oracle" and states that the grammar "is held fixed across the registered comparisons", and §3.1 describes the rules directly.

W3 and Q2: what is a "retail adaptation"?

It meant applying DUT, with the condition language unchanged, to a second workflow: authorizing product returns in τ-bench's retail domain, with τ-bench's own return tool. Only the parts specific to returns are new, namely their predicates and the model's duty to submit an authorized request. The rules, the four-answer construction and the judgments, carry over.

For example, τ-bench's rule authorizes a return only after delivery. Before delivery, the customer's confirmation does not matter and both inputs require WAIT; after delivery, it decides between WAIT and RETURN, the same kind of four-answer test as in §4.1. The study then gives the second instance in Question 3. §4.4 is now titled "Return authorization: diagnosis and repair", and the main text no longer uses the word "adaptation".

Q3: a practical instance in which the framework improves LLM behavior

We read "your framework" as DUT itself, as in your summary, and answer for two senses of "LLM behavior".

  • For the behavior encoded in the model's weights, the answer is no: the weights stay fixed in all our studies, and changing them takes training. Although we believe that the contract could support that training, since it computes the required behavior of any legal input, this paper reports no such result. The Discussion now names targeted training as a use of inputs generated at scale.

  • For the behavior of an LLM application, the fixed weights together with everything around them, the answer is yes, end to end. With the weights fixed, every improvement comes from a change around them, in what the framework supplies, in the prompt or in the settings that run the model, and DUT's matched tests attribute the improvement to that change and show what it repairs and what it breaks. The Discussion names two such changes, moving work to the framework and repairing the component under its current allocation; the two instances below are one of each.

    • The first moves work to the framework, which supplies the current state. The matched inputs differ only in whether the state is supplied.

      • DeepSeek V4 Flash rises from 31 to 70 of 72 units, and failing sequential tests across all configurations fall from 228 to 62.
      • In one shipping case, both required checks have completed; without the state, Qwen3.5 waits for an unrelated report, and with it the model ships on time .
      • the same tests show where this support does not help: it adds unneeded waits for DeepSeek V4 Pro 0813 and MiniMax M3.
    • The second repairs the component under its current allocation via prompt enhancement (I'd rather call it "bugfix") . In the return study, DeepSeek V4.1 Flash must check that the customer confirmed the current return request and then submit it; the tool separately checks delivery and refund eligibility. The tests showed that the model rejected a confirmed partial return because it compared the confirmation with the whole order. We added one clarification to the prompt: the confirmation applies to the current request. On the same 16 order snapshots, each run twice:

      • Responses meeting every requirement of the return contract increased from 25/32 to 31/32;
      • All 25 previous passes were retained, and six failures were resolved;
      • The one remaining failure omits a required submission database check. However, since the database correctly rejects that invalid submission anyway, the final validation passes all 32 responses.

I still do not understand what this paper is trying to accomplish. Can you try to give an overall research question? The two questions you mentioned for your two allocations seem to cover only parts of this work. My overall recommendation would be to do even more work revising this paper to make it more readable; if there is too much material to fit in the page limit without compression, see if some of it could be moved to the appendix or even become a second paper.

We had assumed the question itself to be clear, and focused on two concrete allocations and their findings. You described our original question as “certainly an interesting question,” while asking how the framework and allocation were operationalized. Since this apparently did not come across, we state the overall question here.

Our motivation is the consistency of LLMs on interdependent logical requirements. A model that applies each rule correctly can still lose consistency when the rules depend on one another: on the phase, on which tasks have finished, or on what the model itself must do first. A framework around the model can make up for such losses, for instance by supplying the current state or by starting the tasks itself. This leads to a methodological question: when LLMs face tasks involving interdependent logical constraints, what obligations must be assigned to an external framework to preserve logical consistency, rather than relying on the model's locally plausible or factually correct responses? And the two allocation problems in our paper are two parts of this overarching problem.

The essential distinction is between local correctness and fulfillment of delegated logical obligations. A locally correct decision need not satisfy the complete contract. For example, WAIT can be the correct shipping decision in two situations: when the framework has already started the prerequisite tasks, and when the model itself must start them. But in the second situation, returning WAIT without starting those tasks violates the model's obligation. This is why DUT explicitly represents the applicability, dependencies, and assignment of obligations. Their specification is currently manual. DUT does not claim to discover them automatically, while the resulting executable specifications and matched tests make violations attributable and measurable. A correct decision, a successful downstream outcome, or an unchanged aggregate score cannot substitute for this evidence.

A recent example illustrates the distinction. After the submission deadline, Anthropic publicly demonstrated automated evaluation[1]. In Hamel Husain's trial[2], Claude generated a construction nearly identical to one of our executable contracts: four call-transfer checks, three of them in code, combined by a program into a single protocol verdict, with a condition deciding which checks apply: on calls where no transfer is warranted, the protocol passes if no transfer happens. Hamel found the result difficult to read and preferred to separate the checks, which is also your reaction to our paper. But separating the checks does not eliminate their conditional applicability or the logical relationships that determine overall compliance. Our construction makes those relationships explicit, and our interest is in what such constructions reveal.

Indeed, we had never considered submitting this work to ICLR before obtaining Table 1: the experiments reveal a substantial difference between leading open and closed models on deliberately simple and interpretable logical tests (contrary to the public claim that "no significant gap in benchmarks"). We claim that the observed difference is a concrete empirical phenomenon requiring explanation, and that the construction provides a principled way of exposing it.

This also explains our reluctance to separate the construction from the findings. The construction defines what is being measured; Table 1 demonstrates why measuring it matters. If the contribution were merely a new recipe for constructing evaluations, we would agree with the recommendation to separate it. But that is not the central contribution we are defending: removing the obligation construction from the principal experiments would obscure what their failures mean.

We fully accept that the paper is difficult to read and that this may affect the acceptance decision. But we would still welcome an explanation of Informativeness = 1. After obtaining Table 1, we thought that at least some readers would be interested in what it shows, and we chose ICLR as the venue to present it. In at least part of the community this is far from common knowledge: with one simple experiment, the four-answer consistency tests show a clear difference between open and closed models, and the open side at least does not appear to know it. Otherwise, why would such a clear bug, simple to reproduce, checked by a programmatic verifier and touching a component capability of the system, remain unfixed?

In particular, we would welcome knowing which objection the score rests on: Is the objection that the framework-level obligations are unnecessary, that the experiments fail to measure them, or that the differences revealed in Table 1 are already understood and scientifically uninformative?

[1] https://claude.dev/blog/automating-eval-design-and-hillclimbing

[2] https://hamel.dev/blog/posts/claude-auto-evals/index.html

Comments on the paper

2 comments
AuthorsOpenReview ↗

Changes to the manuscript-Rebuttal Revision

  • Abstract. It follows one line of argument: frameworks can compensate for a model's errors, so system success does not show which requirements the model fulfills, and DUT tests the model's obligations under a stated delegation. It keeps the submitted claims and their order, and adds the evidence standard, the direction of failures under supplied state and what reading implementation changes per obligation decides.
  • §1. It opens with two concrete decisions, the submitted shipping example and the K/N/Q decision from §4.1, and gives the question and the answer before the method. It defines system requirement, allocation (the delegation in the title), obligation and interface in plain words, states that compensation is not a delegation, and gives the evidence standard: one attributable violation refutes an obligation, while fulfillment calls for coverage.
  • §2. It cites Dutta et al. (2024) and states in one place what DUT adds to checks that detect the same failures: the obligation a failure refutes and the delegation under which it does.
  • §3. It opens with the four forms of test built from one contract and the way §4.1–§4.4 use them; each study in §4 now opens with what it changes. §3.2 defines an obligation as the requirement of one latent case under one allocation. The condition language is presented as a specification grammar held fixed across comparisons, and "frozen" no longer appears.
  • §4.1. It states that the policy prompts are in Chinese and reports the direction of failures under supplied state with its denominator.
  • §4.2. It corrects an ambiguous count for DeepSeek V4 Pro and MiniMax M3, traces GLM's reversal to the history interface's events-only promise, and states what the control with aligned startup shows.
  • §4.3. It reads each implementation change both by test identity and per obligation, and names the remedy: supply state where every changed identity passes once state is supplied; reject the format where an obligation met in every round is lost. It records that neither layout fulfills the case the device change regresses, and marks the acceptance verdicts on the Qwen3.8 phase guard as a retrospective reading.
  • §4.4. Now titled "Return authorization: diagnosis and repair", it reports in the main text how DUT located an error of a real model, verified a prompt clarification that kept every earlier pass, and caught a failure that a database check misses.
  • Discussion. It keeps every theme, adds the conclusions above, and closes with the next step that the evidence standard implies: generating legal inputs automatically and at scale.
  • Appendix. The main text now reads on its own, and the appendix follows it, one section per study, in 35 pages instead of 41. It states that the cases and their four-answer units were fixed before any model was run, and its ledger names the Gemini 3.5 Flash (low) panel that we ran to check the oracle.
  • New diagnostics, run for this response and to be added to the appendix, with results in our replies: output probabilities and sampling at temperature 0.7 for the Qwen3.8 exchanges; the main decision tests in three languages over three rounds.

No experimental record, oracle or scoring rule has changed, and no retained table or result reported in the main text has changed. The figures, the tables (Tables 3 and 4 are swapped) and the submission's wording, wherever it was clear, are kept.

Area_Chair_2D19OpenReview ↗

Reviewers response to authors

Dear reviewers,

Author-reviewer discussions end on November 18th. If possible, please respond to author comments by that time so that authors have opportunity for any additional responses.

Thank you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.