CoSelTune: Co-Designing Token Selection and Backward Computation for Efficient Long-Context Fine-Tuning

Fine-tuning adapts large language models to downstream tasks but incurs high memory and computational costs, especially with long documents. Parameterefficient methods can reduce trainable parameters. While token-selective methods such as TokenTune and TokenSeek further reduce activation storage by restricting backpropagation to selected tokens, substantial memory and computational costs still remain. We propose CoSelTune, a modular fine-tuning method that co-designs lightweight token selection and backward computation. It reserves the questionand-answer segment following the document (the suffix) and selects document positions by relevance and coverage within a fixed backward budget. Since suffix queries require the full document, CoSelTune separates their scans from those of selected document queries and compresses the projection inputs needed for local adapter updates, retaining full forward context and all answer losses. CoSelTune can be combined with existing efficient fine-tuning methods for additional resource savings. Experiments on question answering and summarization show memory savings and faster training with comparable performance.