A Tropical Geometry View of Forgetting: A Per-Unit Projector for Knowledge-Preserving Fine-Tuning

Fine-tuning a language model on new text degrades what it already does. Replay-free projectors such as Adam-NSCL and GPM forbid one shared subspace of a layer's inputs in every row of the update. The tropical geometry of a ReLU layer shows why this is too coarse. In data space, the units' walls are tropical hypersurfaces whose cells are dual to the upper vertices of a zonotope; in weight space, each old token is a hyperplane, and the tokens cut out a polyhedron, the closure of the weights that keep every token on its side. An exact identity joins the two pictures: the squared change of the layer's output under any weight change splits into in-cell, open$\to$closed and closed$\to$open terms, and the first two live on the tokens each unit fires on (its open tokens). The identity names a gate-aware per-unit projector, and a budget-separation theorem prices exact protection: it costs a unit the rank of its own open tokens, while a shared subspace pays at least the rank of their union in every row. On OPT-1.3b, where $96\%$ of (token, unit) pairs are closed, the projector forgets less than Adam-NSCL at all six matched budgets from $9$ to $60$ constrained directions per row ($24$ of $27$ seed-pairs, $p=1.5\times 10^{-6}$), the gap widening from $1.1\times$ to $4.3\times$; with $1/5.5$ of the directions it halves the forgetting of Adam-NSCL at GPM's energy threshold. On OPT-6.7b, it matches Adam-NSCL's forgetting at matched budget while learning more. As the theory predicts, the open/closed partition is the operative variable: open tokens beat random, sign-blind and anti-gate token sets on $18$ of $18$ seed-pairs and are equivalent to the first-order criterion, which inherits the partition. In pruning repair, the identity shows that every derivative-based local model of the output error at the dense weights is blind to pairs that open: the minimisers of the gate-weighted objective can leave the polyhedron, the objective's closed-form solution is $1.94$ nats worse than no repair on OPT-1.3b, and a convex one-sided penalty bounds the escape.