- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
- Language models often run inside frameworks that track state, run tools, and fix some of the model’s mistakes - so a successful e2e run doesn't tell you which requirements the model itself met.
- The paper proposes Delegation Under Test (DUT): a designer writes an executable workflow policy and decides which actor (model or framework) is responsible for each action, then DUT then derives what the model must do under each choice and builds matched tests with oracle-computed actions and timing
- 2 choices are studied: (1) whether the model gets an explicit current-state summary alongside its event history, and also (2) whether the framework or the model must start the required tasks.
- Experiments cover 14 models on text tests, local Qwen and GLM models on tool-based tests with repeated rounds, and regression comparisons across several different quantization formats, GPU layouts, and reasoning settings.
- The main findings are that (a) models can pass simple state-only tests but fail when the same facts are buried in history, that (b) supplying state usually helps but can break cases that passed before, and that (c) format or device changes can swap which cases pass while the total score stays the same.
Critical strengths
Strength 1: Careful and checkable measurement
The paper works pretty hard to make sure each reported failure comes from the model and not from a bug in the test harness. For example, oracles are checked against independently written truth tests, every response is linked to its logged request, and infra failures are kept separate from model failures. Since the paper’s claims are about specific test cases flipping, this level of checking is what makes those claims believable
Strength 2: Clear evidence that equal scores can hide real changes
For example, Qwen3.8 (27B) scores 61/72 under both Q6K and Q80, but 1 case that passed now fails and another that failed now passes, in every round (as shown in table 4). The GLM control in Table 2 is also pretty good science: the paper’s own first reading of a negative effect turned out to come from how the interface heading was worded, and the paper reports this openly
Critical weaknesses
Weakness 1: No evidence that DUT finds more than standard testing
The paper admits that an ordinary timing test catches its opening example (see section 1) and that ordinary regression testing catches the Q6K/Q80 swap (see appendix C3). The only comparison against a simpler method is the retail example - but which uses a hand-built controller with no language model. Without a case where DUT leads to a different conclusion than standard practice on real model outputs, the method reads as a careful formalization of existing testing ideas rather than a novel contribution. I think this is fixable with a direct comparison.
Weakness 2: small synthetic suite and single-case findings
The tests come from 5 policy templates, and the 72 units are mostly surface rewrites of the same 8 underlying cases - and several key results rest on 1 changed case. Also, in the Qwen3.8 study, the repeated rounds produce identical outputs, so repetition only shows that the setup is deterministic. It doesn't show that the flips are stable or meaningful. Prior work has already shown that compressed models can keep accuracy while flipping individual answers (see Dutta et al - 2024), so the existence of such swaps is not new by itself.
Questions for authors
- Can you please show a case on real model outputs where DUT’s rules lead to a different accept or reject decision than a standard regression suite that keeps test IDs and checks timing?
- For the Qwen3.8 (27B) format and device flips -- do the changed cases sit near a tie in the model’s output probabilities? I think that results with sampling temperature above zero would show whether these flips are stable.
- Do the startup failures persist if the startup instruction is reworded or if the prompts are in English instead of Chinese?
6 comments
Response to Reviewer 3 (2/4)
W2: small synthetic suite and single-case findings
The eight cases follow a test selection criterion (Goodenough & Gerhart, 1975; Zhu et al., 1997): each marks a point where a fact's relevance changes between two decisions, with two structures for each of the four possible changes (§3.1). They and their four-answer units, with every required action, were fixed before any model was run. Whether a model fulfills an obligation is a universal claim over the obligation's legal inputs: one attributable violation refutes it, however the case was chosen, and passes establish conformance on the inputs tested. We therefore read a violation as a refutation and a pass as conformance on the stated coverage, not as a rate over policies. The 72 units are 8 cases × 9 renderings, and §4.1 and Figure 3 count them that way. Two findings recur well beyond one case:
- the §4.1 gap between explicit state and history, across 11 configurations;
- the direction of failures under supplied state, across 14 configurations × 81 trajectories.
The single-case findings in §4.3 are counterexamples, which is the evidence a preservation claim calls for. Each one refutes the claim that an implementation change preserves an obligation, or shows that no implementation fulfills it under the given delegation:
- one obligation of a phase guard exchanged between renderings in every layout and run, which neither format fulfills with history alone;
- one OR obligation whose passing renderings fall from 8/9 to 3/9 per schedule (BF16→Q8_0);
- one startup obligation that BF16 fulfills in all ten rounds and Q6_K loses in all ten.
The oracle checks and request linkage that your Strength 1 credits make each one attributable.
The identical repeats establish determinism, the control that makes the Qwen3.8 exchange attributable. Fresh processes reproduce the historical output tokens exactly (all 24 greedy reads, Question 2), so the exchange belongs to the formats and not to run-to-run noise. Stability under sampling and across server realizations is a separate question, answered by:
- the sampling study (Question 2);
- the ten startup rounds on fresh servers, where outcomes do vary: Q6_K's business OR case fails rounds 1–9 and passes round 10, yet every failing round has the same first violation (App. D.5);
- the study in three languages at temperature 1.0, where pass³ and pass@3 separate stable from unstable failures (Question 3).
§2 now cites Dutta et al. (2024), and our margins fit their account of near ties: every flipped pair differs by less than 0.2 nats. DUT adds where the flips fall and what they call for. In the Qwen3.8 comparisons, every identity that a format, layout or device change flips is a test with history alone, and its counterparts with state pass in every configuration (§4.3). The remedy is therefore to supply state rather than to pick an implementation. Losing an obligation fulfilled in every round, as in the startup case, calls instead for rejecting the implementation.
Response to Reviewer 3 (3/4)
Q2: do the changed cases sit near a tie, and are the flips stable at temperature above zero?
Every changed case sits near a tie, which is why the flips are unstable while the violations are stable. The tie is the microscopic account, while DUT gives the macroscopic one: the violation belongs to the obligation under this delegation, and supplying state is the remedy, as in Weakness 1.
Above temperature zero, both implementations violate each affected requirement in 16 to 20 of 32 samples, with no consistent advantage for either, while the counterparts with explicit state pass every sample. The tie therefore lies between fulfilling and violating one obligation: the model fulfills it in at most half of the samples under this delegation, whichever implementation runs it, and supplying state removes the tie.
The near ties are measured on the complete recorded output strings. For each retained comparison, we give log p(required output) − log p(violating output), including termination tokens, at temperature 1. The sign changes in every pair, every margin is below 0.2 nats, and the competing first output tokens are also close in probability. These are probabilities of specific strings, not of every way to express an action. All 24 fresh greedy reads reproduce the historical output tokens.
| Retained input | Change | Greedy outcome | Log-probability difference |
|---|---|---|---|
| Preparation, data research, similar task names | Q6_K → Q8_0 | pass → fail | +0.1259 → −0.0205 |
| Preparation, software infrastructure, similar task names | Q6_K → Q8_0 | fail → pass | −0.1949 → +0.1203 |
| Preparation, data research, dissimilar task names | tail device I → T | fail → pass | −0.0472 → +0.0811 |
| Conjunctive shipping, retained decision prefix | tail device I → T | pass → fail | +0.0399 → −0.0598 |
We conjecture that this is the microscopic mechanism of every flip in the table: a small numerical change, such as a different quantization format or tail device, moves greedy decoding across the tie. It predicts what you expected: under sampling, which implementation passes should not be stable. To test it, we sample 32 seeds per cell at temperature 0.7, with top-p 1, no top-k and no repetition penalty, keeping reasoning off and the same worker and prompts. Every response completes in the required output form, so every failure is a wrong action. The counts confirm the prediction:
| Retained input | Violations / 32, baseline → candidate |
|---|---|
| Format: preparation, data research | 16 → 16 |
| Format: preparation, software infrastructure | 20 → 17 |
| Device: preparation, data research | 19 → 17 |
| Device: conjunctive shipping decision | 16 → 16 |
| Control with explicit state, all four configurations | 0 in every configuration |
Response to Reviewer 3 (1/4)
Thank you for a precise reading. We sincerely apologize for the suboptimal reading experience caused by the previous version. To address this, we have provided a revised manuscript aimed at improving the clarity and readability of the main text.
In the revision, the opening example you cite is kept in §1, now followed by a second one; Table 4 is now Table 3; and the Appendix C.3 material you cited is now in Appendices D.8 and F.1.
W1 & Q1: does DUT reach a different decision on real model outputs?
Yes. Your summary describes DUT as deriving "what the model must do under each choice" and building "matched tests with oracle-computed actions and timing". A regression suite that keeps those tests, with their IDs and their timing, is the suite DUT builds, so its pass and fail verdicts are DUT's own. The suites a team has without DUT are built from the inputs at hand and protect the passes a baseline already has. On the same outputs of real models, DUT's decisions differ from theirs in four ways.
First, DUT judges a call under its allocation. In the new startup composition (§4.2), Qwen3.5 BF16 states "All relevant tasks are NOT_STARTED. Therefore, rule 2's IF condition is unsatisfied, and the policy returns WAIT," and calls the waiting tool. A test of the business decision passes this response; the submission's appendix already noted that a check of business labels credits this WAIT (now App. D.8). When startup is assigned to the model, DUT rejects the response in all three rounds, because none of the three required tasks was started. When startup is assigned to the framework, the same first call conforms. The same call and the same correct decision receive opposite verdicts, and the delegation in the title decides which. A test keyed only by its input cannot express this difference.
Second, DUT enforces mandatory obligations. A suite that keeps test IDs protects what the baseline passed. In §4.3, one Qwen3.8 phase guard is violated in eight renderings with history alone in every format, layout and round, and is fulfilled in every configuration once state is supplied. Read retrospectively, protecting prior passes finds nothing lost and accepts every one of these changes. An acceptance contract that names this obligation mandatory rejects them all and points to the remedy: supply state. The Discussion states the rule: adoption must also satisfy mandatory requirements, which can include requirements that the baseline never fulfilled.
Third, DUT reads verdicts per identity. Your Strength 2 and the Dutta et al. reference both find aggregate scores insufficient: - Qwen3.8 Q6_K and Q8_0 tie at 61/72, which aggregate scoring reads as interchangeable. - Supplying state raises or preserves the total in every configuration of Table 1. Yet it loses 10 previously passing trajectories for DeepSeek V4 Pro and 6 for MiniMax M3, all of them new waits, so a rule that retains prior passes rejects state supply for both. - A rule suite on explicit state passes 12 configurations of Table 1, and 11 of them violate the same requirements given history.
Fourth, DUT checks every action the model owes. In the return study, a check of the final state, as τ-bench uses, accepts the clarified prompt with 32/32 correct database outcomes. DUT rejects one response (31/32) for omitting an authorized submission (§4.4).
Where a suite that keeps test IDs and DUT both reject, DUT names the remedy. The suite detects the Q6_K/Q8_0 exchange, as the passage you cite says (Appendix C.3, now D.8), and protecting each identity rejects Q8_0. The submission presented that rejection as an illustration, not as a registered adoption rule. What the suite cannot see is that the two IDs are near renderings of one phase guard under the same allocation, one in software infrastructure and one in data research. The guard says: during preparation, advance the phase whatever other completions show. - With history alone, Q6_K violates it in one rendering and Q8_0 in the other. - With current state, both formats pass every rendering in every layout and round (72/72). - Under sampling (Question 2), both formats violate both renderings in 16–20 of 32 samples, while the control with explicit state passes 32/32 in every configuration.
Neither format fulfills this obligation under history alone, and supplying state fulfills it under both: the remedy is a change of delegation, not a choice of format. In the Qwen3.5 BF16→Q6_K startup regression, by contrast, BF16 passes all twelve registered identities under model startup in all ten rounds on fresh servers, while Q6_K omits startup in its business case with phase selection in all ten. Here a fulfilled obligation is lost, and rejecting the format is the remedy (§4.3).
What DUT adds is what each verdict is about: the obligation a failure refutes and the delegation under which it does (§3.2). DUT also determines which tests exist and what a changed identity calls for (§4.1–§4.4).