acceptodds
Desk rejected at ICLR 2027

Policy Capacity and System Cost in Independent Reinforcement Learning for Serial Inventory Control

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Increasing the expressiveness of a policy can change how independently trained agents affect one another. We investigate this effect in a serial inventory system in which each agent observes local state and minimizes its own quadratic inventory cost. Under a fixed independent proximal policy optimization protocol, we compare a one-gain feedback policy, an affine policy, and a residual multilayer perceptron. In a four-echelon chain with lead time two and demand autocorrelation 0.6, mean system cost increases from 40.679 for the one-gain policy to 55.379 for the affine policy and 56.194 for the neural policy, corresponding to increases of 36.1% and 38.1%. Every restricted-policy run is cheaper than every richer-policy run across eight training seeds per class. Paired bootstrap intervals for the absolute increases are [13.57, 15.78] and [14.03, 17.21]. Across additional demand regimes, the affine cost gap and sensitivity to recent demand increase together, but changing marginal demand variance prevents a causal interpretation. We also prove that increasing a stable proportional feedback gain raises upstream variance costs in an idealized linear cascade. The theorem provides an externality argument, while the experiments establish a finite-budget learning effect; they do not certify Nash convergence or isolate a causal neural mechanism. These findings support evaluating policy classes by their system-level consequences when learning objectives are decentralized.

Primary area:
reinforcement learning
Keywords:
multi-agent reinforcement learning, policy class, inventory control, decentralized optimization, supply chain, PPO

Related papers

Discussion (0)

Sign in to comment.