- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper mainly focuses on fine tuning and compression of large LLM models. Unlike existing pruning and compression methods, which usually work on a target model size and may not be easily adaptable when a different size is needed, the paper proposes a federated learning approach that aims to replace the current fine-tuning step and supply usable subnetworks of various sizes with no additional online cost. To achieve this, at each training phase, the full model is randomly partitioned, distributed, and trained such that each subnetwork should still be able to make predictions. The parallelization is a key contribution of the work and the engineering does lead to at times rather significant memory and time gains. The numerical experiments and ablation studies are abundant, and there appears to be some evidence to support the proposed scheme. However, some questions remain unclear.
Critical strengths
The paper focuses on what I think is a very relevant problem, model compression remains a important topic and balancing between performance and memory/inference time on edge devices is the key challenge, the zero-shot pitch of the paper does make an appealing effort.
While Im not fully convinced that the proposed approach is optimal in achieving their objective the engineering is in my opinion the strongest point of the work, by parallelizing and enabling training over multiple potentially weak devices, its what makes the pipeline partially justifiable as its arguably more costly than existing methods.
The paper does in my opinion a decent job of providing a good amount of experiments and comparisons, as well as being fairly honest about the issues with extracting subnetworks at sizes different from those used during training, showing the limitations of the method.
Critical weaknesses
One of the findings of modern LLM diagnostics is that a pretrained model often has different functionalities in different blocks, attention heads or layers. I think even some early work on GPT-2 models exposed this. Existing importance-based and reconstruction-based pruning methods such as sparseGPT at least attempt to preserve aspects of this structure, whereas the proposed method essentially forces the pretrained blocks to adapt to different combinations. This in my opinion raises serious questions about both the adaptation required during training and the reliability of the subnetworks. I think this is a major question that the proposed method while partially acknowledging unequal component importance, ultimately fails to adequately address or convince the reader about, and with all the claims on zero-shot prediction, how much can i trust those results with an arbitrary combination? and to what extent?
I notice that almost all reported results show only perplexity, and I think it is not sufficient for truly showing the effectiveness of the resulting models. PPL measures how well the model predicts reference tokens, but not necessarily the factual correctness or consistency of its generated outputs. A model can achieve good perplexity while still being of limited use in practice. I think the paper needs to show more to convince the reader that these subnetworks are meaningfully functional.
Questions for authors
Under the written formulation, I can see the training across different worker being extremely uneven, a certain combination of subnetwork can blowup or have magnitudes high loos than others, do you observe that? how do you resolve that? reading from the equations everything is treated equally but if 15 minibatch training does not resolve a loss blowup then averaging that into the full model seems like a numerical nightmare, and reading from the tables numerical instability certainly can happen.
There is considerable literature in federated learning showing that weight averaging can be unreliable when workers drift toward different local objectives and does not necessarily improve the global model, and I think many people in the field have opted for gradient averaging or other methods. Why is simple weight averaging chosen here, particularly when workers train different subnetworks? Is there any justification for this decision beyond the reported ablation?
According to the paper the first 2 and final 2 layers are always fixed to avoid model breaking down, and only middle sections are being partitioned and shuffled. Is there any attempt in trying to understand what happens to the middle portion during training? do these middle layers actually develop into different modules or gradually become similar to each other? without additional diagnostics I can see both happening given the problem formulation.
As model size increases and the number of partitioned attention heads and layers grows, the number of subnetwork combinations can grow exponentially, at some point even with cleaver parallelization the computation requirement in training can grow much higher than any of the existing methods, There is also a need to generalize to combinations never seen during training. The paper currently evaluates relatively small models and is relatively tame in designing the partitions, and I do not expect this to be fully resolved given limited resources, but what would your answer be based on your experience? modern stoa LLMs are at trillion parameter level and will the proposed method be scalable at all?