Isolating the Training Objective in Test-Time Knowledge Incorporation

Test-time training (TTT) can write a document into the adapter parameters of a language model with a few gradient updates, and recent methods improve this process with components that require training of their own, such as a meta-learned update, a teacher or a generator trained by reinforcement learning. All such methods share a training objective, and because it can be changed without additional training, what it achieves alone is the natural reference for these components. This reference has not been measured. In the standard setting, with LoRA applied to the query and value projections of Qwen2.5-7B, no objective significantly exceeds next-token training, yet with LoRA applied to all linear projections the best objective exceeds it by 18.5 points, so the placement can hide the effect of the objective. We therefore decompose the objective into four components and compare fourteen objectives that change one component at a time, on two backbones, two placements and two datasets. Shaping the loss rarely improves accuracy. Accuracy instead depends on how much distinct text the loss covers, and falls monotonically as passage tokens are removed from the loss. Guided by this finding, we propose MIX3, which supervises the full passage together with the model's own question-answer pairs and rewrites. In ten updates and without a trained generator, teacher or outer loop, it reaches 73.7% and 73.5% closed-book accuracy on Qwen2.5-7B and Llama-3.1-8B-Instruct, and on facts newer than the training data it improves retrieval while keeping judgment at the level of the unadapted model. MIX3 thus provides a reference for costlier components, and our results suggest that objectives should be compared under more than one placement.