- Rating
- 3
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies when behavior-trace replay supports decisions about MoE interventions. TAP separates candidate-value accuracy, selection regret, and budget-relevant decision impact. Experiments on three MoE families isolate execution mismatch, action-induced rerouting, and inherited cache-state effects. A same-checkpoint construction establishes nonidentifiability when action-path runtime semantics are unobserved. The results show that substantial label errors can coexist with unchanged or low-regret decisions.
Critical strengths
- Well-controlled mechanism analysis. Matched execution, controlled-FP32 routers, and the cache-state swap distinguish numerical mismatch, intervention feedback, and delayed accounting effects (Sections 6.1–6.2; Figure 2).
- Careful interpretation of evidence. Complete candidate matrices distinguish inaccurate values from wrong choices and material losses. Reporting unchanged aggregate choices, failed efficiency comparisons, and corrected sampling interpretations makes the findings substantially more credible (Section 6.3; Appendices A.4, E–I).
Critical weaknesses
- Limited methodological increment. Router mismatch is already established, e.g., Ma et al. (2025). TAP primarily organizes direct evaluation, paired-error comparisons, and ordinary stopping/reuse; customized variants show no supported increment over strong baselines (Appendices H–I). Theorem 1 is informative but depends on hidden action-path semantics, rather than establishing nonidentifiability in a fully specified runtime.
- Practical decision impact remains limited. HH mean regret is only 0.173%/0.126% of native loads, although about 6% of available savings, and aggregate choices are unchanged. The prospective 64-pair audit saves no queries (Section 6.3; Table 21). The pulse-control reversals are useful, but do not establish consequential serving gains.
- Quality validation is illustrative, not independent. Appendix G selects and scores using the same artificial 9–10-token continuation, making zero observed violations automatic. Evaluation costs are excluded, and cost-ranking repair explains only 6.3–16.7% of the improvement. This does not establish a deployable quality–cost advantage.
- Core experiments are not independently rerunnable from the described package. Appendix J/Table 49 explicitly omit the full GPU capture/evaluation pipeline, locked GPU environment, complete input reconstruction, and immutable model/tokenizer revision manifests. These omissions are particularly consequential for claims about execution-sensitive numerical behavior; archived scalar checks cannot replace rerunning that behavior.
Questions for authors
- Which TAP contribution goes beyond ordinary matched replay and complete candidate evaluation, and which existing result best demonstrates that increment?
- Which existing experiment most strongly supports a practically material decision correction, rather than exact-label repair? How was its materiality budget motivated?
- Do the existing natural-response records support the NLL-screening conclusion, or is that conclusion restricted to the artificial continuation?
- Can the core GPU evaluator, input-reconstruction procedure, and exact model/tokenizer/runtime revisions be supplied to address Table 49's reproducibility gaps?
2 comments
Thank you for your response, my concerns have been addressed.
Thank you for the detailed assessment of the contribution, decision impact and reproducibility. We have clarified these claims and added the core GPU evaluator and complete 20-pair rerun records to the supplementary package.
Q1. What goes beyond ordinary matched replay and complete candidate evaluation?
TAP checks the execution evidence behind candidate values and uses standard candidate comparisons. Our controls isolate expert-output feedback and costs carried forward through the cache. Under FP32 routing, 9/40 Qwen window choices still differ, including one equal-cost tie. The first acted-on layer agrees; divergence begins downstream. In Figure 2, exchanging only the entering cache state swaps costs from 1/2 to 2/1 under an identical ordered route. Together, these results explain why matching router arithmetic or correcting current routes can leave an intervention's cost wrong.
R3 records rollout routes and fixes them during training to address rollout-training router mismatch. Our interventions allow downstream rerouting, and we measure the resulting candidate costs, rankings and budgeted losses. Section 7 explains this distinction. The comparisons use established regret analysis, stopping and suffix reuse. Section 5 translates the controls into a practical evaluation workflow: identify what the action changes, follow the affected state, and check alternatives at the declared budget.
Theorem 1 assumes unobserved action-path numerical semantics. Specifying the full runtime resolves the construction's ambiguity. Replay ranking and behavior opportunity remain fixed while deployment rankings reverse, with a positive regret bound under the stated observation restriction (Section 4). The real-model controls use a specified runtime.
Q2. Which result shows a material correction, and why use 1%?
The held-out single-substitution case in Section 6.3 changes a cost-saving conclusion. For DeepSeek at capacity 24, sixteen recorded-response tokens and source five, replay selects early at 1,014 loads against native's 1,015. Execution costs 1,043; native is optimal. The intervention therefore adds 28 loads (2.76%) instead of saving loads. Demand-only correction already rejects this candidate. The case supports re-evaluating the affected trajectory; the correction cannot be attributed uniquely to the cache-state term. It is the sole above-1% condition among 96 source-condition observations from twelve reused source trees (Appendix C.2).
The 1% budget is a prespecified logical-load tolerance. It has no calibrated conversion to latency or an SLO. Section 6.3 reports the same-cohort threshold scan alongside the small HH mean regrets and unchanged aggregate choices, showing how the conclusion depends on the loss tolerance and selection unit.
The prospective DeepSeek audit found four above-budget pairs out of 64 and completed the census without saving queries. Appendices H-I compare the cost of the audit with ordinary stopping, capture and suffix-reuse baselines.
Q3. Do natural-response records validate the NLL-screening conclusion?
They test cache accounting on longer recorded replies. The NLL result in Appendix G uses the same prescribed 9-10-token artificial continuation for selection and scoring. Rules enforcing that observed constraint have zero violations by construction. This is a finite-candidate example, not independent quality validation. Cost-ranking repair accounts for 6.3-16.7% of the extra feasible saving from checking alternatives rather than only the winner; the rest comes from candidate coverage (Section 6.3). Evaluation costs are excluded.
Q4. Can the core GPU pipeline and exact inputs/revisions be supplied?
On November 9 (UTC), we reran the original 20-pair Qwen cohort on the same A800 in the isolated November 6 environment: Python 3.10.12, PyTorch 2.4.0+cu121, Transformers 4.57.1, BF16 experts, FP32 gates and SDPA, with 5,000 bootstrap draws. The prepared ZIP contains the retained evaluator, exact PAWS inputs and order, protocol, candidate matrix and execution commands.
All 40 views and 120 window records matched every candidate cost, choice and regret, including native denominators. Stored NLL, KL, route fields, FP32 gate checks and bootstrap summaries matched exactly. Model execution took 18.1 minutes; the full run took 20.9 minutes. The cohort and hardware are unchanged.
The model source is unsloth/Qwen3-30B-A3B at commit 7ac4b27433e7d10034efceefa18c557c59d9e489; all thirteen shards again passed public SHA256 checks (MODEL_IDENTITY.json). The package includes model/tokenizer hashes, November 6's 40-package inventory and wheel hashes, and full-run versions. gpu_fp32/full20_reproduction_20261109/ contains the plan, raw results, commands, logs and comparisons. verify_full20.py checks stored results on CPU. Pairs 0 and 4 retain their separate saved-trace byte checks. Appendix J.1 and Table 50 (original Table 49) describe the coverage.