- Rating
- 1
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces CoSelTune, a modular fine-tuning method that focuses on lightweight token selection and backward computation. The authors observed that although LoRA and QLoRA reduce the number of trainable parameters, activation storage and attention backpropagation remain costly. Existing token-selective methods may omit supervised prediction positions when selecting across the whole sequence, while retained queries can still require long attention scans. To solve this gap, this paper preserves the complete question-and-answer suffix, selects document positions by relevance and coverage, and reduces training costs through separate backward attention scans and activation compression. Experiments on both Qwen3-14B-Base and Llama-3.1-8B demonstrate the proposed CoSelTune achieves comparable performance to full-token LoRA with lower training costs.
Critical strengths
1.The paper addresses the substantial activation storage and attention backpropagation costs that remain in long-context fine-tuning with PEFT method. 2.Two-branch backpropagation reduces document-query scan ranges while preserving complete suffix supervision and access to the full causal context. 3.The ablation studies are well designed. Table 3 examines suffix reservation, Table 4 compares document selection strategies, and Table 5 evaluates the effects of scan truncation and FFN blocking, which helps identify the sources of efficiency gains.
Critical weaknesses
- The method assumes a short question-and-answer suffix. It remains unclear how the token budget should be adjusted and how much efficiency would be retained with longer suffixes.
- In Table 4, the core-expansion improves over random selection by only 0.10 points. Also, this ablation uses a single seed, whereas the main experiments use three. Multiple seed evaluation is needed to assess whether this gain is consistent.
Questions for authors
How should the token budget be adjusted for longer question-and-answer suffixes, and how much of CoSelTune’s efficiency gains would be retained?
2 comments
Follow-up on Our Rebuttal
Dear Reviewer KzBW,
Thank you for your time and valuable feedback on our submission. We have carefully considered your comments and provided detailed responses to your questions and concerns in our rebuttal.
As the discussion period is approaching its deadline, we would greatly appreciate it if you could review our responses and let us know whether any concerns remain. Your feedback would help us understand whether we have adequately addressed the issues you raised and identify any points that require further clarification.
We would welcome the opportunity to continue the discussion and would be happy to provide additional explanations or information as needed.
Thank you again for your time and consideration. Best regards, The Authors of Paper CoSelTune: Co-Designing Token Selection and Backward Computation for Efficient Long-Context Fine-Tuning
Longer suffixes retain the same 10% total state budget.
For N input positions, capacity is floor(0.1N). Within budget, the complete suffix and all answer losses are retained; the remainder goes to document states. Beyond budget, we reserve ceil(0.01D) document states for document length D and prioritize answer-prediction states in the remaining slots. If all targets fit, spare slots are filled from other suffix states. Otherwise, we retain the first-answer and EOS prediction states and rotate one interior target per contiguous stratum across repeated encounters. Anchors count toward the cap. Loss averages selected answer targets, while the forward pass retains every token. Section 4.2.1, Algorithms 1–2 and Appendix D.7 now state both cases.
Long-suffix training retains 13.81–20.72% compute savings.
Qwen3-14B-Base trains on 2,048 OpenCode examples with 20–30% suffixes for six epochs: 384 post-parent updates at effective batch 32, 12,288 exposures and 128.96 million tokens per method. Target coverage is 38.79% per encounter and 100% within four encounters. On 342 examples across six cohorts, mean offline message-prediction F1 is 39.97 for CoSelTune, versus 31.30 for Base, 38.72 for Full LoRA, 30.59 for TokenSeek and 36.64 for TokenTune. Full-target validation CE is 0.1725 versus Full LoRA's 0.1681. Computation is 7.58 hours versus 8.80 and 9.56 for TokenTune and TokenSeek; peak memory is 48.59 versus 54.60 GiB, an 11.00% reduction. Time includes optimization and selection/scoring. Table 1 and Appendix D.8 quantify this regime.
Eight-epoch GovReport training supports five-task transfer.
CoSelTune, TokenTune and TokenSeek share 1,024 reports, 8,192 exposures, 92.20 million tokens, effective batch 32 and 256 post-parent updates. The complete suffix occupies 6.75% of input positions, versus 0.99% originally, using 67.5% of the 10% budget. CoSelTune and both complete-suffix adaptations retain all 6.06 million targets. The transfer tasks are BigPatent, WCEP-10, 2WikiMultihopQA, MS MARCO and MuSiQue. CoSelTune's five-task transfer mean is 40.33, compared with 40.63 for Full LoRA and 33.95 for Base. Computation falls by 10.81% and 16.42% versus complete-suffix TokenTune and TokenSeek, with 6.38% lower peak memory. Tables 3–4 and Appendix F.5 provide the comparison.
The average gain over Global Random is positive in all three runs.
We added two random seeds to the original 128-update LongAlpaca selector control, bringing it to three seeds. Both selectors use CoSelTune execution, the complete suffix and the same 777 evaluation examples. Core-expansion scores 38.8852 ± 0.1995 versus Global Random's 38.7180 ± 0.1524; the paired gain is +0.1672 ± 0.0566 points (mean ± sample SD). Mean scores are higher on three of five tasks. In the reference control, preparation takes 14.00 seconds versus TokenSeek's 562.60 seconds, a 40.2× speedup. Table 5 reports the means; Appendix B.5, Table 10, gives individual scores and variability.
With long suffixes, CoSelTune improves both quality and efficiency over the token baselines under the same 10% state budget.
It achieves higher six-task mean F1 with 13.81–20.72% less computation and 11.00% lower peak memory. The revised allocation makes this benefit available when answers exceed the state budget while preserving the full input context. The three-seed comparison further shows a positive selector gain in every run, and the reference control measures a 40.2× preparation speedup over TokenSeek. These results directly demonstrate the practical benefit of coordinating supervision, selection and backward execution.
We would be grateful if you could reconsider your recommendation in light of the overall practical value demonstrated by our work, together with the additional results and revisions addressing both concerns.
Paper colors: blue = long suffixes (4.2.1, Table 1, D.7–D.8); red = GovReport (Tables 3–4, F.5); green = selector comparison (Table 5, B.5).