Visual Instruction Tuning

Instruction tuning large language models on machine-generated instruction-following data has improved zero-shot capabilities on new tasks, but the idea was less explored in the multimodal field. This paper presents the first attempt to use language-only GPT-4 to generate multimodal language-image instruction-following data, and by instruction tuning on that data introduces LLaVA (Large Language and Vision Assistant), an end-to-end trained large multimodal model that connects a vision encoder and an LLM for general-purpose visual and language understanding.

Early experiments show LLaVA demonstrates impressive multimodal chat abilities, sometimes exhibiting the behaviors of multimodal GPT-4 on unseen images and instructions, and it yields an 85.1% relative score compared with GPT-4 on a synthetic multimodal instruction-following dataset. When fine-tuned on Science QA, the synergy of LLaVA and GPT-4 achieves a new state-of-the-art accuracy of 92.53%, and the GPT-4-generated visual instruction tuning data, the model, and the code base are made publicly available.