acceptodds
Withdrawn from ICLR 2027

Signs Beat Floats: Low-Rank Double-Binary Adaptation for Storage-Efficient Deployment

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Task-specific LoRA adapters can be stored and exchanged independently of a shared frozen language model, but their floating-point factors remain a substantial deployment cost. We introduce LoRDBA, a low-rank double-binary adapter that represents both factors with binary signs and lightweight channel-wise scales. A post-training factorization initializes the signs, after which our default QAT Freeze procedure adapts only the scales; QAT Full additionally updates latent carriers. We provide a conditional finite-sample reconstruction bound for a sign-noise model. Across three-seed math-reasoning evaluations, LoRDBA achieves favorable compact accuracy-storage trade-offs on LLaMA-2-7B and 13B, with model-dependent margins on Mistral-7B. At approximately MiB logical payload on 7B, Freeze improves GSM8K/Minerva means by points over compact fp16 LoRA with equal total dataset passes. On 13B, it exceeds an SVD rank-2 control initialized from the same source and given the same two-epoch recovery by points at approximately MiB. Separately, packed execution reduces resident adapter memory by , and adapter-local recomputation brings Freeze training to the fp16-LoRA peak-memory level at higher step time. These findings support storage-efficient adapters produced on server GPUs, not inference speedup or demonstrated on-device training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
adapter compression, parameter-efficient fine-tuning, low-rank adaptation, binary quantization, large language models, storage-efficient deployment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_faFzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes storing LoRA in 1 bit while introducing three scaling vectors to compensate for numerical values. The method decomposes a trained 16-bit LoRA into a 1-bit representation and then optimizes the scaling vectors for calibration. With the same storage space, it performs better than 16-bit LoRA, while maintaining comparable performance before and after quantization and reducing the GPU memory usage of LoRA adapters.

Critical strengths

1.The proposed method achieves better accuracy under a limited storage budget. The experiments show that LoRDBA achieves better accuracy than low-rank FP16 LoRA on LLaMA-2-7B and 13B with the same storage budget. This suggests that keeping a higher rank may be more useful than keeping higher numerical precision when storage is limited.

2.The paper provides a good analysis of resource costs. The paper not only compares storage size and accuracy, but also measures actual GPU memory usage, inference latency, and training costs. The authors also clearly mention that the method does not improve inference speed. These experiments help readers better understand the practical value of this method.

Critical weaknesses

I think the novelty of this paper is relatively limited. Double binary factorization, channel-wise scaling, and scale-only refinement have already been explored in previous DBF work. This paper mainly applies these techniques to LoRA. Also, the comparison with LoRAQuant is not fully fair because LoRDBA receives additional recovery training.

I am also not convinced about the practical value. The main accuracy experiments still reconstruct the binary matrices into floating-point matrices, while the custom CUDA kernel actually makes inference slower. More importantly, LoRA adapters are already much smaller than the base model, so their storage may not be a major bottleneck in most scenarios. In the end, the method mainly saves adapter storage, but it is unclear whether this benefit is worth the extra training cost and slower inference.

Questions for authors

1.Previous DBF work has already explored double binary factorization and scale-only refinement. What is the main novelty of this paper beyond applying these techniques to LoRA?

2.Considering that LoRA adapters are already much smaller than the base model, and the current CUDA implementation makes inference slower, in what practical scenarios do the authors think reducing adapter storage would bring enough benefits to justify the additional training and inference costs?

Reviewer_ugX5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

LoRDBA compresses LoRA updates into two binary factors with scaling vectors, followed by scale-only refinement.

Critical strengths
  • Targets a useful setting where many adapters share one frozen model.
  • Shows favorable accuracy–storage trade-offs with multiple seeds.
Critical weaknesses
  • The technical advance over DBF is limited.
  • LoRAQuant receives less recovery training, weakening comparison fairness.
  • Accuracy in the exact counted deployment format remains unverified.
  • Benefits vary across models, and serving-efficiency evidence is limited.

[1] Addition is almost all you need: Compressing large language models with double binary factorization, 2025 [2] LoRAQuant: Mixed-Precision Quantization of LoRA to Ultra-Low Bits

Questions for authors

Does the advantage over LoRAQuant remain when both methods receive the same recovery-training budget?

Reviewer_mBs6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes LoRDBA to compress unmerged task-specific LoRA adapters over a shared frozen base model using two binary sign matrices and scaling vectors. Its default QAT Freeze pipeline factorizes a trained adapter and performs task-specific recovery by updating only the scales; QAT Full also updates latent carrier variables. The paper provides a conditional reconstruction bound and evaluates reasoning on LLaMA-2 and Mistral, with supplementary tasks and systems measurements. At approximately matched logical storage, Freeze improves over compact floating-point controls on LLaMA-2 and performs similarly on Mistral; packed deployment reduces resident adapter memory by approximately 6.90-fold, with inference and training costs reported separately.

Critical strengths
  1. The method offers practical storage benefits. At approximately one-eighth of the logical payload, LoRDBA achieves competitive results on LLaMA-2. In particular, on 13B, it improves GSM8K and Minerva by 3.61 and 3.34 percentage points, respectively, over an SVD rank-2 control using the same source adapter and recovery schedule, providing meaningful evidence for the effectiveness of the pipeline.

  2. The experimental reporting is appropriately cautious. The main results include three random seeds, and logical storage, resident memory, inference latency, and training costs are reported separately. Packed deployment reduces resident adapter memory by approximately 6.90-fold, and the authors also report the associated latency and training-time costs, helping readers assess the settings in which the method is useful.

Critical weaknesses
  1. It is unclear whether the gain comes from the compression method or the extra recovery training. LoRDBA receives two extra task-training epochs, whereas LoRAQuant receives no task-loss recovery. The same-source SVD control is useful, but recovery-matched LoRAQuant and storage-matched VeRA comparisons are missing.

  2. The writing is repetitive and poorly organized. Recurring explanations of deployment restrictions, storage definitions, and experimental limitations interrupt the main argument. Related work appears after the experiments, delaying the key connection to DBF's sign matrices, channel-wise scales, and scale-only refinement.

Questions for authors
  1. Does LoRDBA retain its advantage over LoRAQuant with the same source adapter and recovery budget, and over VeRA with matched storage accounting? Please provide these comparisons.

  2. Why further binarize an already compact LoRA adapter when the saving relative to the base model is small and system complexity increases? Please quantify the adapter-bank or distribution workload at which the saving outweighs factorization, recovery training, and packed-kernel overheads.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.