acceptodds
Under review as a conference paper at ICLR 2027

When Do Behavior Traces Validate MoE Interventions? Auditing Labels, Selection, and Decision Impact

Abstract

Behavior traces can validate an MoE cache policy when the policy preserves the ordered execution and replay matches its initial state and accounting. Interventions that change expert outputs require evidence about the execution that follows. We study these requirements in Qwen3-MoE, DeepSeek-V2-Lite, and GPT-OSS-20B, separating the accuracy of a candidate's value from the validity of selecting it. Matching execution order and initial state repairs the tested fixed-route labels. Replay and execution still select different windows in 9/40 Qwen records with FP32 routing: modified expert outputs alter the inputs to later routers. Cache history adds a delayed effect, allowing costs to differ even after routes agree. Yet label errors often leave decisions intact. In a separate two-family audit, route-local correction leaves 74/96 labels inexact, yet only 3/32 choices have positive regret, all within 1% of native loads. We construct two runtimes with the same checkpoint and observed behavior but opposite intervention rankings when action-path execution semantics are unobserved. The Trace-to-Action Audit Protocol (TAP) checks the selected action's value, its alternatives, and the loss from choosing it. These checks distinguish a wrong cost estimate from a suboptimal choice and from a loss that exceeds the evaluator's budget.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
mixture-of-experts, counterfactual evaluation, replay evaluation, evaluation validity, reproducibility

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sESHOpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies when behavior-trace replay supports decisions about MoE interventions. TAP separates candidate-value accuracy, selection regret, and budget-relevant decision impact. Experiments on three MoE families isolate execution mismatch, action-induced rerouting, and inherited cache-state effects. A same-checkpoint construction establishes nonidentifiability when action-path runtime semantics are unobserved. The results show that substantial label errors can coexist with unchanged or low-regret decisions.

Critical strengths
  1. Well-controlled mechanism analysis. Matched execution, controlled-FP32 routers, and the cache-state swap distinguish numerical mismatch, intervention feedback, and delayed accounting effects (Sections 6.1–6.2; Figure 2).
  2. Careful interpretation of evidence. Complete candidate matrices distinguish inaccurate values from wrong choices and material losses. Reporting unchanged aggregate choices, failed efficiency comparisons, and corrected sampling interpretations makes the findings substantially more credible (Section 6.3; Appendices A.4, E–I).
Critical weaknesses
  1. Limited methodological increment. Router mismatch is already established, e.g., Ma et al. (2025). TAP primarily organizes direct evaluation, paired-error comparisons, and ordinary stopping/reuse; customized variants show no supported increment over strong baselines (Appendices H–I). Theorem 1 is informative but depends on hidden action-path semantics, rather than establishing nonidentifiability in a fully specified runtime.
  2. Practical decision impact remains limited. HH mean regret is only 0.173%/0.126% of native loads, although about 6% of available savings, and aggregate choices are unchanged. The prospective 64-pair audit saves no queries (Section 6.3; Table 21). The pulse-control reversals are useful, but do not establish consequential serving gains.
  3. Quality validation is illustrative, not independent. Appendix G selects and scores using the same artificial 9–10-token continuation, making zero observed violations automatic. Evaluation costs are excluded, and cost-ranking repair explains only 6.3–16.7% of the improvement. This does not establish a deployable quality–cost advantage.
  4. Core experiments are not independently rerunnable from the described package. Appendix J/Table 49 explicitly omit the full GPU capture/evaluation pipeline, locked GPU environment, complete input reconstruction, and immutable model/tokenizer revision manifests. These omissions are particularly consequential for claims about execution-sensitive numerical behavior; archived scalar checks cannot replace rerunning that behavior.
Questions for authors
  1. Which TAP contribution goes beyond ordinary matched replay and complete candidate evaluation, and which existing result best demonstrates that increment?
  2. Which existing experiment most strongly supports a practically material decision correction, rather than exact-label repair? How was its materiality budget motivated?
  3. Do the existing natural-response records support the NLL-screening conclusion, or is that conclusion restricted to the artificial continuation?
  4. Can the core GPU evaluator, input-reconstruction procedure, and exact model/tokenizer/runtime revisions be supplied to address Table 49's reproducibility gaps?
2 comments

Thank you for the detailed assessment of the contribution, decision impact and reproducibility. We have clarified these claims and added the core GPU evaluator and complete 20-pair rerun records to the supplementary package.

Q1. What goes beyond ordinary matched replay and complete candidate evaluation?

TAP checks the execution evidence behind candidate values and uses standard candidate comparisons. Our controls isolate expert-output feedback and costs carried forward through the cache. Under FP32 routing, 9/40 Qwen window choices still differ, including one equal-cost tie. The first acted-on layer agrees; divergence begins downstream. In Figure 2, exchanging only the entering cache state swaps costs from 1/2 to 2/1 under an identical ordered route. Together, these results explain why matching router arithmetic or correcting current routes can leave an intervention's cost wrong.

R3 records rollout routes and fixes them during training to address rollout-training router mismatch. Our interventions allow downstream rerouting, and we measure the resulting candidate costs, rankings and budgeted losses. Section 7 explains this distinction. The comparisons use established regret analysis, stopping and suffix reuse. Section 5 translates the controls into a practical evaluation workflow: identify what the action changes, follow the affected state, and check alternatives at the declared budget.

Theorem 1 assumes unobserved action-path numerical semantics. Specifying the full runtime resolves the construction's ambiguity. Replay ranking and behavior opportunity remain fixed while deployment rankings reverse, with a positive regret bound under the stated observation restriction (Section 4). The real-model controls use a specified runtime.

Q2. Which result shows a material correction, and why use 1%?

The held-out single-substitution case in Section 6.3 changes a cost-saving conclusion. For DeepSeek at capacity 24, sixteen recorded-response tokens and source five, replay selects early at 1,014 loads against native's 1,015. Execution costs 1,043; native is optimal. The intervention therefore adds 28 loads (2.76%) instead of saving loads. Demand-only correction already rejects this candidate. The case supports re-evaluating the affected trajectory; the correction cannot be attributed uniquely to the cache-state term. It is the sole above-1% condition among 96 source-condition observations from twelve reused source trees (Appendix C.2).

The 1% budget is a prespecified logical-load tolerance. It has no calibrated conversion to latency or an SLO. Section 6.3 reports the same-cohort threshold scan alongside the small HH mean regrets and unchanged aggregate choices, showing how the conclusion depends on the loss tolerance and selection unit.

The prospective DeepSeek audit found four above-budget pairs out of 64 and completed the census without saving queries. Appendices H-I compare the cost of the audit with ordinary stopping, capture and suffix-reuse baselines.

Q3. Do natural-response records validate the NLL-screening conclusion?

They test cache accounting on longer recorded replies. The NLL result in Appendix G uses the same prescribed 9-10-token artificial continuation for selection and scoring. Rules enforcing that observed constraint have zero violations by construction. This is a finite-candidate example, not independent quality validation. Cost-ranking repair accounts for 6.3-16.7% of the extra feasible saving from checking alternatives rather than only the winner; the rest comes from candidate coverage (Section 6.3). Evaluation costs are excluded.

Q4. Can the core GPU pipeline and exact inputs/revisions be supplied?

On November 9 (UTC), we reran the original 20-pair Qwen cohort on the same A800 in the isolated November 6 environment: Python 3.10.12, PyTorch 2.4.0+cu121, Transformers 4.57.1, BF16 experts, FP32 gates and SDPA, with 5,000 bootstrap draws. The prepared ZIP contains the retained evaluator, exact PAWS inputs and order, protocol, candidate matrix and execution commands.

All 40 views and 120 window records matched every candidate cost, choice and regret, including native denominators. Stored NLL, KL, route fields, FP32 gate checks and bootstrap summaries matched exactly. Model execution took 18.1 minutes; the full run took 20.9 minutes. The cohort and hardware are unchanged.

The model source is unsloth/Qwen3-30B-A3B at commit 7ac4b27433e7d10034efceefa18c557c59d9e489; all thirteen shards again passed public SHA256 checks (MODEL_IDENTITY.json). The package includes model/tokenizer hashes, November 6's 40-package inventory and wheel hashes, and full-run versions. gpu_fp32/full20_reproduction_20261109/ contains the plan, raw results, commands, logs and comparisons. verify_full20.py checks stored results on CPU. Pairs 0 and 4 retain their separate saved-trace byte checks. Appendix J.1 and Table 50 (original Table 49) describe the coverage.

Thank you for your response, my concerns have been addressed.

Reviewer_VTnROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper examines the reliability of trace-based evaluation for MoE interventions. It proposes TAP to distinguish prediction errors from decision errors. Experiments on three MoE models show that replay errors can affect intervention rankings without necessarily leading to suboptimal decisions.

Critical strengths
  1. The paper identifies an important issue in evaluating MoE caching policies: behavior traces may not accurately reflect execution outcomes after interventions. This challenges the reliability of existing trace-based evaluation methods and highlights the need for more rigorous evaluation.
Critical weaknesses
  1. The paper is overly dense and difficult to follow, with extensive use of technical jargon and insufficient intuitive explanations. The presentation of the key ideas, theoretical results, and experimental findings is unnecessarily complicated, making it difficult for readers to understand the main contributions. I encourage the authors to simplify the exposition, better define technical terms, and provide more intuitive examples.
Questions for authors
  1. What is the central research question of this paper? The paper covers several different problems, but it is unclear what the authors are ultimately trying to achieve and how these components fit together.

  2. What is the main contribution of TAP? How does it differ from standard candidate evaluation and regret analysis?

  3. How do the experiments support the paper's main objective? The connections between the different experimental settings and the central research question are unclear.

1 comment

Thank you for asking us to clarify the central question and how the experiments fit together. We have revised the opening and Sections 2, 4 and 6 to make that connection explicit and explain TAP's role.

Q1. What is the central research question?

We ask when a log from an unmodified MoE model can justify choosing one serving intervention over another. Our cost is logical expert loads on fixed token sequences. A cache policy can preserve the logged computation; replacing an expert output can change later hidden states, routes and cache contents.

Figure 1 makes the evaluation problem concrete. Replay selects late and predicts 44 saved loads. Execution confirms 44, but the rejected early candidate saves 47. Checking late alone validates its value and misses the better alternative. The loss is three loads, still within the declared 1% native-load budget. The revised opening defines experts, loading and replay, then presents Figure 1 before the mechanism results. Early and late are introduced as layer windows. Section 2 explains that decision impact reports candidate loss at the chosen budget and selection unit; record/pair/view units are defined in the setup.

Q2. How does TAP differ from standard candidate evaluation and regret analysis?

TAP uses standard candidate evaluation and regret analysis. It specifies the execution evidence needed to apply those comparisons to a serving intervention. The paper's additional findings concern what that evidence must preserve: expert-output changes affect later router inputs, and earlier route changes affect costs after routes agree again. Complete evaluation under the target execution conditions already supplies the corresponding verdicts.

The FP32 control isolates intervention feedback. In Section 6.2, all Qwen gates execute in FP32, yet replay and execution choose different windows in 9/40 records, including one equal-cost tie. Expert choices agree at the first acted-on layer and diverge downstream of the changed expert output. Matching router precision therefore leaves this feedback in place.

The cache control shows a specific requirement that current-route checks miss. The tested correction replaces costs only at events with different ordered routes. Figure 2 explains why that can fail: holding the current ordered route fixed and swapping only the entering cache states swaps costs from 1/2 to 2/1. Route agreement therefore does not justify reusing the behavior cost. In the separate two-family audit, exact corrections at every ordered-route disagreement still leave 74/96 non-null action labels inexact. Ordinary ordered-route replay of the executed action traces from the matched boundary cache state recovers all 96 labels (Section 6.2; Table 3). With executed action traces available to all these diagnostic corrections, the contrast isolates the inherited cache state needed for exact accounting. Only 3/32 view-level choices have positive regret, all within 1% of native loads, and the aggregate winner is unchanged.

Section 5 makes these findings operational: match ordered accesses, initial state and accounting for computation-preserving residency policies; execute affected suffixes and propagate cache state for output-changing interventions; compare challengers at the declared budget and selection unit. TAP reports value agreement, candidate regret and the budget verdict separately.

Q3. How do the experiments support the objective?

Table 2 maps each experiment to its question. Section 6.1 checks whether logging matches deployment. Section 6.2 now presents the FP32 control and cache-state swap together, before the candidate-set and cross-family checks. Section 6.3 begins with what a value check can miss, then interprets the loss relative to native cost and available saving, the budget, and the workload-level choice. Section 4 explains the two indistinguishable runtimes before introducing the set of consistent worlds and its mathematical notation.

The cohorts are kept separate. For example, the two-family cache-history audit leaves 74/96 labels inexact after route-local correction, while only 3/32 choices have positive regret, all within 1% of native loads. Across these experiments, execution controls locate cost errors; complete candidate comparisons measure their consequences for a choice. Counting inexact labels alone cannot answer the latter.

If a particular claim or derivation prompted the soundness concern, a reference to that step would help us address it precisely.

We also reran the complete original 20-pair Qwen cohort in the isolated environment on the same A800. All 40 views, 120 window-cost records and reported summaries matched the archive. Appendix J.1 and the supplement contain the execution and comparison records.

Reviewer_f66POpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Authors are considering the following problem statement: Given an existing mixture-of-experts setup, if there are any changes which are made to the model serving setup, how do we validate whether the changes do not actually impact the overall functioning of the system? In order to evaluate this, they basically look at the behavior trace of an unmodified mixture of an expert model. Then they evaluate label validity, selection validity, and decision impact. Authors evaluate their proposed protocol, which is the trace-to-action audit protocol, on three mixture-of-expert models.

Critical strengths

Interesting problem statement related to serving evaluation of mixture-of-experts models.

Critical weaknesses

Found it very difficult to understand the paper. It may be because of two reasons:

  1. This is not my area of expertise.
  2. Authors have used too much of AI to write the paper, which has made it very difficult to understand.

Request the authors to rewrite the entire draft preferably without AI and make it easy to read and understand and if possible, use AI only for polishing and improving the grammar.

Questions for authors

See Weaknesses

2 comments

Thank you for pointing out the difficulty of following the paper. We have revised the opening and Sections 2, 4 and 6 to explain the evaluation problem and the links between the experiments more clearly.

An MoE model selects a few expert networks for each token. Using an expert whose weights are absent from a limited cache incurs a load. An evaluator can run the original model once, save its expert accesses, and estimate each candidate change's loading cost from that record. We test when this shortcut can justify the candidate choice, using fixed token sequences and logical loads.

Figure 1 gives a concrete example. Replay chooses late (a change in later layers) and predicts 44 saved loads. Execution confirms 44, but early (a change in earlier layers) saves 47. The chosen candidate's value is correct, yet another candidate is better. The loss of three loads is 0.159% of native cost, within the declared 1% budget. Checking the predicted value, comparing alternatives and measuring the loss answer different questions in this example.

The saved record can become misleading because changing an expert output can alter later expert choices. Even when those choices agree again, earlier differences may have left different experts in cache. Section 6.2 tests these two steps using the FP32 control and the cache-state exchange in Figure 2. Section 6.3 measures their consequences for candidate choices and loss budgets.

The revised opening defines experts, loading and replay before presenting these mechanisms. Section 2 defines the measured values and evaluation questions; Table 2 links the separate experiments to those questions. Section 4 explains the two indistinguishable runtimes and the unknown action-path rule before introducing the formal notation.

The prepared supplement includes the Qwen evaluator and a completed rerun of the original 20-pair cohort in an isolated environment on the same A800. All 40 views, 120 window-cost records and reported summaries matched the archive. Appendix J.1 gives the execution and comparison records.

We would be glad to clarify any part that remains difficult to follow.

Thanks for your response. My recommendation would be to re-submit the revised version in the next cycle after you have properly re-written the entire paper. I will like to keep my score same.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Reading the revision: Figure 1B and Tables 3 and 5

Thank you for the feedback on readability. We rewrote the introduction around one concrete choice (Figure 1B) and revised Section 6 to make the evidence for our original conclusions easier to follow. The example and Tables 3 and 5 show how we check whether the rule favored by the original model’s log really saves the most expert loads.

Figure 1B (p. 2). TAP uses standard regret and asks what the runs so far establish. Running late confirms its predicted saving of 44 loads. Running early reveals a saving of 47, enough to rule out late as optimal. Once we check middle, we know the loss is exactly three loads, within our reference budget of 1% of native loads.

When can we trust the cost estimates? (Section 6.1, Figure 2 and Table 3, pp. 7–8). For fixed routes, replaying accesses in execution order from the matching cache recovered all 108 held-out costs. After expert substitution, two runs could request the same experts in the same order and still incur different costs. Swapping only the caches entering that step swapped the load counts. In Table 3, correcting costs only where routes differed left 74 of 96 savings estimates wrong. Replaying each intervention’s full access sequence from the matching cache recovered all 96.

Table 5 (p. 9). On separate data, we ask what happens when we choose by uncorrected replay. In panel A, replay chose an optimal rule on 30 of the 37 requests where it misestimated the chosen rule’s saving. In panel B, some HH requests exceeded budget, yet replay and execution chose the same rule for the whole workload within each model family.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.