acceptodds
Under review as a conference paper at ICLR 2027

CoSelTune: Co-Designing Token Selection and Backward Computation for Efficient Long-Context Fine-Tuning

Abstract

Fine-tuning adapts large language models to downstream tasks but incurs high memory and computational costs, especially with long documents. Parameterefficient methods can reduce trainable parameters. While token-selective methods such as TokenTune and TokenSeek further reduce activation storage by restricting backpropagation to selected tokens, substantial memory and computational costs still remain. We propose CoSelTune, a modular fine-tuning method that co-designs lightweight token selection and backward computation. It reserves the questionand-answer segment following the document (the suffix) and selects document positions by relevance and coverage within a fixed backward budget. Since suffix queries require the full document, CoSelTune separates their scans from those of selected document queries and compresses the projection inputs needed for local adapter updates, retaining full forward context and all answer losses. CoSelTune can be combined with existing efficient fine-tuning methods for additional resource savings. Experiments on question answering and summarization show memory savings and faster training with comparable performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long-context LLMs, Efficient Fine-Tuning, Token Selection, Selective Backpropagation, Memory-Efficient Training, Parameter-Efficient Fine-Tuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KzBWOpenReview ↗
Rating
1
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces CoSelTune, a modular fine-tuning method that focuses on lightweight token selection and backward computation. The authors observed that although LoRA and QLoRA reduce the number of trainable parameters, activation storage and attention backpropagation remain costly. Existing token-selective methods may omit supervised prediction positions when selecting across the whole sequence, while retained queries can still require long attention scans. To solve this gap, this paper preserves the complete question-and-answer suffix, selects document positions by relevance and coverage, and reduces training costs through separate backward attention scans and activation compression. Experiments on both Qwen3-14B-Base and Llama-3.1-8B demonstrate the proposed CoSelTune achieves comparable performance to full-token LoRA with lower training costs.

Critical strengths

1.The paper addresses the substantial activation storage and attention backpropagation costs that remain in long-context fine-tuning with PEFT method. 2.Two-branch backpropagation reduces document-query scan ranges while preserving complete suffix supervision and access to the full causal context. 3.The ablation studies are well designed. Table 3 examines suffix reservation, Table 4 compares document selection strategies, and Table 5 evaluates the effects of scan truncation and FFN blocking, which helps identify the sources of efficiency gains.

Critical weaknesses
  1. The method assumes a short question-and-answer suffix. It remains unclear how the token budget should be adjusted and how much efficiency would be retained with longer suffixes.
  2. In Table 4, the core-expansion improves over random selection by only 0.10 points. Also, this ablation uses a single seed, whereas the main experiments use three. Multiple seed evaluation is needed to assess whether this gain is consistent.
Questions for authors

How should the token budget be adjusted for longer question-and-answer suffixes, and how much of CoSelTune’s efficiency gains would be retained?

2 comments

Longer suffixes retain the same 10% total state budget.

For N input positions, capacity is floor(0.1N). Within budget, the complete suffix and all answer losses are retained; the remainder goes to document states. Beyond budget, we reserve ceil(0.01D) document states for document length D and prioritize answer-prediction states in the remaining slots. If all targets fit, spare slots are filled from other suffix states. Otherwise, we retain the first-answer and EOS prediction states and rotate one interior target per contiguous stratum across repeated encounters. Anchors count toward the cap. Loss averages selected answer targets, while the forward pass retains every token. Section 4.2.1, Algorithms 1–2 and Appendix D.7 now state both cases.

Long-suffix training retains 13.81–20.72% compute savings.

Qwen3-14B-Base trains on 2,048 OpenCode examples with 20–30% suffixes for six epochs: 384 post-parent updates at effective batch 32, 12,288 exposures and 128.96 million tokens per method. Target coverage is 38.79% per encounter and 100% within four encounters. On 342 examples across six cohorts, mean offline message-prediction F1 is 39.97 for CoSelTune, versus 31.30 for Base, 38.72 for Full LoRA, 30.59 for TokenSeek and 36.64 for TokenTune. Full-target validation CE is 0.1725 versus Full LoRA's 0.1681. Computation is 7.58 hours versus 8.80 and 9.56 for TokenTune and TokenSeek; peak memory is 48.59 versus 54.60 GiB, an 11.00% reduction. Time includes optimization and selection/scoring. Table 1 and Appendix D.8 quantify this regime.

Eight-epoch GovReport training supports five-task transfer.

CoSelTune, TokenTune and TokenSeek share 1,024 reports, 8,192 exposures, 92.20 million tokens, effective batch 32 and 256 post-parent updates. The complete suffix occupies 6.75% of input positions, versus 0.99% originally, using 67.5% of the 10% budget. CoSelTune and both complete-suffix adaptations retain all 6.06 million targets. The transfer tasks are BigPatent, WCEP-10, 2WikiMultihopQA, MS MARCO and MuSiQue. CoSelTune's five-task transfer mean is 40.33, compared with 40.63 for Full LoRA and 33.95 for Base. Computation falls by 10.81% and 16.42% versus complete-suffix TokenTune and TokenSeek, with 6.38% lower peak memory. Tables 3–4 and Appendix F.5 provide the comparison.

The average gain over Global Random is positive in all three runs.

We added two random seeds to the original 128-update LongAlpaca selector control, bringing it to three seeds. Both selectors use CoSelTune execution, the complete suffix and the same 777 evaluation examples. Core-expansion scores 38.8852 ± 0.1995 versus Global Random's 38.7180 ± 0.1524; the paired gain is +0.1672 ± 0.0566 points (mean ± sample SD). Mean scores are higher on three of five tasks. In the reference control, preparation takes 14.00 seconds versus TokenSeek's 562.60 seconds, a 40.2× speedup. Table 5 reports the means; Appendix B.5, Table 10, gives individual scores and variability.

With long suffixes, CoSelTune improves both quality and efficiency over the token baselines under the same 10% state budget.

It achieves higher six-task mean F1 with 13.81–20.72% less computation and 11.00% lower peak memory. The revised allocation makes this benefit available when answers exceed the state budget while preserving the full input context. The three-seed comparison further shows a positive selector gain in every run, and the reference control measures a 40.2× preparation speedup over TokenSeek. These results directly demonstrate the practical benefit of coordinating supervision, selection and backward execution.

We would be grateful if you could reconsider your recommendation in light of the overall practical value demonstrated by our work, together with the additional results and revisions addressing both concerns.

Paper colors: blue = long suffixes (4.2.1, Table 1, D.7–D.8); red = GovReport (Tables 3–4, F.5); green = selector comparison (Table 5, B.5).

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer KzBW,

Thank you for your time and valuable feedback on our submission. We have carefully considered your comments and provided detailed responses to your questions and concerns in our rebuttal.

As the discussion period is approaching its deadline, we would greatly appreciate it if you could review our responses and let us know whether any concerns remain. Your feedback would help us understand whether we have adequately addressed the issues you raised and identify any points that require further clarification.

We would welcome the opportunity to continue the discussion and would be happy to provide additional explanations or information as needed.

Thank you again for your time and consideration. Best regards, The Authors of Paper CoSelTune: Co-Designing Token Selection and Backward Computation for Efficient Long-Context Fine-Tuning

Reviewer_FRYjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CoSelTune, which aims to reduce the cost of long-document LoRA fine-tuning through token-selective backpropagation. Through careful engineering choices such as suffix reservation and two branch backpropagation, the proposed method effectively reduces the memory and computational costs of long-context finetuning. Empirical studies show that CoSelTune matches full-token LoRA quality and reports favorable performance in quality and efficiency compared to baseline methods including TokenTune and TokenSeek. Extensive ablation studies are also conducted to validate the design choices of the proposed method.

Critical strengths

The proposed method is a practically useful efficiency technique while preserving comparable quality compared to full-token LoRA. Compared to baseline methods (with or without checkpointing), CoSelTune shows favorable gains in either memory or training speed, and matches quality to full-token LoRA within seed noise.

Critical weaknesses

The main weakness of the paper lies in the experimental configuration. Each training run is only 128 updates at batch size one. As a result, all considered methods fall between a thin margin (e.g. average score ranges from 37.3 to 38.9 on Qwen3-14B). It is hard to tell whether the training is really sufficient to distinguish between different methods. More concerning is that all methods suffer from more training (Table 14), which implies that the training is fragile under this finetuning regime. On Llama-3.1-8B, tokenseek hurts the model and leads to worse performance than the base model, while the same method works fine for Qwen. Overall, I am not very confident that the current experiments truly can demonstrate the claims on the efficiency and quality gains of the proposed method.

Questions for authors

Please see above.

1 comment
AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer FRYj,

Thank you for your time and valuable feedback on our submission. We have carefully considered your comments and provided detailed responses to your questions and concerns in our rebuttal.

As the discussion period is approaching its deadline, we would greatly appreciate it if you could review our responses and let us know whether any concerns remain. Your feedback would help us understand whether we have adequately addressed the issues you raised and identify any points that require further clarification.

We would welcome the opportunity to continue the discussion and would be happy to provide additional explanations or information as needed.

Thank you again for your time and consideration. Best regards, The Authors of Paper CoSelTune: Co-Designing Token Selection and Backward Computation for Efficient Long-Context Fine-Tuning

Reviewer_RBFKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper deals with the high cost of training on long documents, where gradients need to be computed for every token. Previous methods reduce this cost by backpropagating through only about 10% of the tokens, but they may miss important answer tokens. CoSelTune addresses this issue by always including the entire question-and-answer section and selecting the remaining tokens based on their textual similarity to the question. Experiments on Qwen3-14B and Llama-3.1-8B show that CoSelTune trains about 30% faster than TokenSeek while achieving performance close to full LoRA.

Critical strengths
  • The paper addresses an important practical problem: reducing training costs without losing useful answer information. The results show that cutting answer tokens to 10% noticeably hurts performance on both Qwen and Llama.

  • The method has a useful design that allows even unselected tokens to contribute to adapter training through key and value gradients, despite not receiving state gradients.

  • The paper includes detailed experimental results across different random seeds, controlled settings, and per-document analyses.

Critical weaknesses
  • I am not sure but the method mainly focuses on text segments that look relevant to the question, which may teach the model to rely too much on word overlap and shortcuts. This could be a problem for multi-hop reasoning, summarization, or paraphrased questions, but the paper’s extractive QA benchmarks may not reveal this limitation.

  • The method cuts off state gradients for unselected tokens but still keeps their adapter gradients, which may lead to inconsistent gradient updates. The paper does not measure how closely these gradients match full backpropagation

  • With a 10% budget, any example with suffix above 10% of the sequence (chat data, reasoning traces, short-document QA, code with long answers) leaves zero or negative document budget. The method either fails its own precondition or degenerates to suffix-only training with no document learning.

Questions for authors
  • All methods, including CoSelTune, get worse when training doubles to 256 updates. Why do the methods fail in this configuration?

  • Does CoSelTune remain stable with larger batches, more data, longer training, or other benchmarks?

  • If TokenTune and TokenSeek keep the full answer loss and run on the execution backend, what advantage remains? What do suffix-only and 1–5% budgets show?

2 comments
AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer RBFK,

Thank you for your time and valuable feedback on our submission. We have carefully considered your comments and provided detailed responses to your questions and concerns in our rebuttal.

As the discussion period is approaching its deadline, we would greatly appreciate it if you could review our responses and let us know whether any concerns remain. Your feedback would help us understand whether we have adequately addressed the issues you raised and identify any points that require further clarification.

We would welcome the opportunity to continue the discussion and would be happy to provide additional explanations or information as needed.

Thank you again for your time and consideration. Best regards, The Authors of Paper CoSelTune: Co-Designing Token Selection and Backward Computation for Efficient Long-Context Fine-Tuning

Reviewer_RBFKOpenReview ↗

Reply to the Authors

Thanks to the authors for the response. I think adding these details into the paper will provide better clarity. I gave the paper a good score and want to maintain it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.