Auditable Ranking-Slice Redeployment under Conflicting Refresh Metrics
Desk rejected at ICLR 2027, so left out of the paper lists and the map.
Abstract
Maintaining a model ranking requires deciding whether a refreshed slice can replace the evidence that produced the current order. We show that conventional refresh scores resolve this decision differently even when candidate pools, budgets, filters, and five evaluation cycles are held fixed: Paraphrase leads acceptance and Kendall's , while Adversarial-LLM produces the largest anchor–fresh gap. This metric-winner reversal motivates a redeployment contract with four distinct evidence units: candidate throughput, sampled admission validity, semantic preservation of deployed items, and ranking usefulness. SliceLedger operationalizes the contract with declared intent invariants and cue-family transformations that make each item's latent task traceable through generation, symbolic validation, expert arbitration, and external review; a blinded -item audit validates its admission decisions at precision and recall. At near-matched ( versus ), SliceLedger exceeds Adversarial-LLM by acceptance points, , and 8.2 deployed-preservation points (stratified-bootstrap CI [3.9,12.6]), reaching 91.1% preservation under a symmetric three-rater audit. Matched-time expert-guided maintenance reaches 95.1% preservation while accepting 31.2% of candidates, compared with for SliceLedger, exposing an automatic–human Pareto frontier. Removing the intent schema loses acceptance points and ; the selection remains stable on a model-disjoint panel and under a frozen five-cycle replay. Across GSM8K, SQuAD 2.0, HumanEval, and held-out MMLU, the contract turns recurring refresh into an auditable redeployment methodology and identifies a high-throughput automatic operating point with strong semantic and ranking preservation.
- Primary area:
- datasets and benchmarks