Multimodal Machine Learning: A Survey and Taxonomy

Recognizing that human experience of the world is multimodal, seeing objects, hearing sounds, feeling textures, smelling odors, and tasting flavors, this paper argues that AI must interpret multiple such modalities together. It surveys the field of multimodal machine learning, which seeks to build models that can process and relate information coming from several modalities. Instead of concentrating on specific multimodal applications, the authors review recent advances in the underlying methods and present them within a common taxonomy.

The taxonomy moves beyond the typical early- and late-fusion categorization to identify five broader challenges faced by multimodal machine learning: representation, translation, alignment, fusion, and co-learning. By framing the field around these challenges, the survey gives researchers a clearer picture of the state of the art and helps them identify promising directions for future work.