What Should the Backbone Learn? Analyzing Heterogeneous Continual PET via Commit-and-Rebase

In continual learning with pre-trained models (PTMs), parameter-efficient tuning (PET) has emerged as a prevailing paradigm, typically allocating a small budget of tunable parameters to tasks. The budget bounds interference with the pre-trained representation, but it also bounds how much a stream can teach the model: whether the parameters are appended alongside the backbone or embedded in a low-rank slice of it, the knowledge of every task must fit within an allowance far smaller than the backbone itself. A stream long enough will exhaust any such allowance, so the backbone itself must eventually be enlisted as the store — which raises a fundamental question: of everything a PET module learns, what is actually worth writing into it? The question is hard to even pose, because different PET methods express what they learn in incompatible formats. In this paper, we propose CoRe, an instrument that converts what an arbitrary PET method has learned into a shared weight space through a commit-and-rebase process, so that heterogeneous methods become directly comparable. Across seven PET methods and seven benchmarks, CoRe reproduces the behavior of the original expert more faithfully than existing merging operators, with 97.5% prediction agreement. Analyzing the converted updates, we find that only a small component of each update is invariant across methods and that this component alone moves the decisions of its task, providing a first empirical account of what a backbone should learn from a PET module.