Reliable Deployment-Time Learning: From Label Acquisition to Error Recovery

This paper studies the reliability of deployment-time learning. As AI systems increasingly need to acquire new knowledge after deployment, a learning agent must detect novel objects or classes unseen during training, acquire their labels from human users or other external knowledge sources, and incrementally incorporate this new knowledge into the deployed model. However, externally acquired labels may not always be directly suitable for learning: they may use synonyms or hypernyms of the intended class names, contain errors, or even be maliciously provided. Moreover, learning new classes should not compromise previously acquired knowledge, and errors that have already been learned should be recoverable. We address these challenges based on the learning method of incremental Linear Discriminant Analysis (iLDA) over representations produced by strong pretrained foundation models. iLDA is particularly suitable for deployment-time learning because it supports efficient online incremental updates while preserving previously acquired knowledge with high accuracy. Building on this foundation, we develop methods for reliable label acquisition that resolve semantically related labels and detect potentially incorrect labels before learning, as well as an efficient recovery mechanism for removing the influence of incorrect labels that have already been incorporated into the model. Experimental results on multiple benchmark datasets demonstrate the effectiveness of the proposed methods across label acquisition, incremental learning, and error recovery, providing a practical framework for reliable learning after deployment.