SAIR: Cost-Efficient Multi-Stage ML Pipeline Autoscaling via In-Context Reinforcement Learning

Multi-stage ML inference pipelines are difficult to autoscale because of heterogeneous resources, cross-stage coupling, and dynamic bottleneck migration. Production controllers ignore cross-stage effects, and learning-based autoscalers require offline training data that is unavailable for new pipelines. We present SAIR, an autoscaler for the zero-shot deployment regime that uses an LLM as an in-context reinforcement learning controller, improving its policy online from reward-labeled experience tuples in the LLM's context with no gradient updates. SAIR combines Pareto-dominance reward shaping with a provable separation margin between dominated and non-dominated actions, Bayesian-surprise experience retrieval, and a user-space CUDA interception layer that delivers continuous GPU rate control with sub-millisecond actuation overhead. We prove a regret bound that decomposes into retrieval coverage and LLM selection components, each mapping to a concrete SAIR mechanism. On four ML serving pipelines under three workload patterns, SAIR achieves the best or tied-best P99 latency in every configuration (up to 50% reduction over the next baseline); billable cost is reduced through replica consolidation, and effective cost (which captures GPU rate-control savings under co-location or MIG) by up to 97%. SAIR reaches 86% bottleneck detection accuracy without any offline training.