Learning to Prompt for Vision-Language Models

Large pre-trained vision-language models such as CLIP learn representations that transfer across many downstream tasks by aligning images and text in a common feature space, which enables zero-shot transfer to a new task through prompting, where classification weights are synthesized from natural language describing the classes of interest. The authors identify prompt engineering as a major practical obstacle: it requires domain expertise and is extremely time-consuming, because a slight change in wording can have a large impact on performance. Inspired by prompt learning advances in NLP, they propose Context Optimization (CoOp), a simple approach for adapting CLIP-like models to downstream image recognition.

CoOp models a prompt's context words with learnable vectors while keeping the entire pre-trained network frozen, and it offers two variants, a unified context and a class-specific context, to handle different recognition tasks. Extensive experiments on 11 datasets show that CoOp needs as few as one or two shots to beat hand-crafted prompts by a decent margin, and with 16 shots it improves over prompt engineering by around 15% on average, reaching over 45% in the best case. Despite being a learning-based approach, CoOp also achieves strong domain generalization compared with the zero-shot model that uses hand-crafted prompts.