Multimodal model
A multimodal model is an AI model that can take in, and sometimes produce, more than one kind of data, such as text, images, audio or video, within a single model rather than through separate systems.
Also called: multimodal AI
By Divya Sharma · Checked Nov 15, 2026 · How we review
Many current flagship models are multimodal: you can paste a screenshot, a chart or a PDF page into a chat and ask questions about it. Some also produce images or speech.
How it works
A common design connects a pretrained encoder for each extra input type to a large language model. The encoder turns an image or audio clip into vectors the language model can read alongside text tokens. DeepMind's Flamingo, in 2022, showed this by bridging pretrained vision-only and language-only models and training on web data with images and text interleaved. Other models are trained on several modalities from the start.
Multimodal vs vision-language
A vision-language model is the most common kind of multimodal model, handling images and text. "Multimodal" is the wider term and can also cover audio, as in speech-to-speech systems, or video.
Limits
Support differs widely. A model may accept images but not output them, or handle audio only through a separate pipeline. Images and audio also use up the context window and add cost. Check what each model actually accepts in the models directory.
Related terms
- Vision-language model
- A vision-language model (VLM) is a model that takes images and text as input and produces text, so it can describe pictures, answer questions about them, read documents and extract data from screenshots.
- Large language model
- A large language model (LLM) is a neural network with billions of parameters, trained on large amounts of text to predict the next token, which lets it write, answer questions, summarise and follow instructions.
- Foundation model
- A foundation model is a large model trained on broad data at scale that can be adapted to many different downstream tasks, through prompting or fine-tuning, instead of being built for one task.
- Speech-to-speech model
- A speech-to-speech model takes spoken audio in and produces spoken audio out directly, without first converting the speech to text and back. It is used for live voice conversation and translation.
- Text-to-image
- Text-to-image is the task of generating a new picture from a written description, called a prompt. Most current text-to-image models are diffusion models conditioned on the text.