Improved Baselines with Visual Instruction Tuning

The paper conducts the first systematic study of design choices for large multimodal models within the LLaVA framework, testing them in a controlled setting. The core method retains LLaVA's simple fully-connected vision-language connector but adds targeted modifications: swapping in a CLIP-ViT-L-336px visual encoder, using an MLP projection layer, and incorporating academic-task-oriented VQA data together with response-formatting prompts.

These changes establish stronger baselines that achieve state-of-the-art results across 11 benchmarks, with a final 13B checkpoint trained on just 1.2M publicly available samples in roughly one day on a single 8-A100 node. By showing that such strong performance is attainable with modest, public data and compute, the work makes state-of-the-art LMM research more accessible, and it further explores open problems like higher-resolution inputs, compositional capabilities, and hallucination.