- Rating
- 4
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper introduces a new method of improving model's predictions on unseen data while leaving the model's weights frozen. The presented framework (called PCS-AC) adds low-rank learnable corrections to the parameters of each layer of a model at inference; the framework bounds the space of allowed corrections given the structure of the hidden states, determines the set of actions to improve the answer margin within the bounded space and learns the optimal correction values based on a training subset of the new data. The authors study the application of PCS-AC to vision-language models (VLM) and text models for improving the binary predictions on medical visual-question-answering (VQA) datasets and text binary assertion tasks. On all of the considered datasets the framework shows an improvement in prediction accuracy w.r.t. the native backbone for all of the considered models and in many cases the framework outperforms other corrective intervention methods, such as LoReFT.
Critical strengths
To the best of my knowledge, the framework introduced in this paper is a novel contribution to this field of research. The authors extensively explain the details of their framework and demonstrate how it relates to other methods and approaches existing in the literature. The authors clearly demonstrate on a given set of models and datasets that their method substantially improves the accuracy for the specified set of tasks and applications, hence the authors achieve the stated goal of their research. The authors also perform a substantial number of experiments (hyperparameter scans, ablation, etc.) to support their conclusions.
Critical weaknesses
I did not identify any critical weaknesses in the paper, but I have a few concerns whose resolution would further improve it. The authors claim that simply fine-tuning the models on an unseen data might introduce unnecessary errors, but they don't show what level of accuracy can be actually reached with fine-tuning and how it compares to their method in terms of accuracy and computational costs. It would be nice to see more data and information regarding the trade-off between accuracy and e.g. computational time for fine-tuning and PCS-AC. In addition, the authors considered two VLM models, but only one text model - it would be interesting to see how the proposed framework performs on at least another text model.
Questions for authors
My questions to the authors directly follow from the matters I discuss in "Critical Weaknesses" section.
- Would it be possible (time-wise and resource-wise) to add a comparison between fine-tuning and PCS-AC in terms of accuracy and computational costs at least for one dataset and one type of models?
- Would it be possible to consider an additional text model?