Multimodality

In plain English

Multimodal means more than text. A multimodal model might accept an image and a question, then answer in words—or read text aloud, or describe a chart.

You already use narrow multimodal pipelines when voice assistants transcribe speech to text and send it to a language model. Newer models train some of those steps together so vision and language share one stack.

How it works

Common pattern: an image encoder (often a vision transformer or CNN backbone) converts patches into tokens; those tokens join text tokens in one transformer sequence. Training uses image–caption pairs, OCR screens, diagrams with questions, and similar aligned data.

Audio models may use spectrogram patches or discrete speech codes; video adds time dimension and heavy compute. Output side, text-to-speech and image generators are separate heads or models wired into products.

• Inputs converted to token-like sequences per modality.
• Shared transformer layers fuse cross-modal context.
• Products may chain specialist models behind one chat UI.

Going deeper

Modalities differ in noise and privacy risk—photos of people, medical scans, and screenshots need policy guardrails beyond text moderation.

Evaluation is harder than text-only QA: visual reasoning benchmarks exist but may not cover your camera angle, lighting, or domain labels.

Common misconceptions

Multimodal models see images like human eyes.

They process numeric features from patches; failure modes include OCR errors and texture biases.

One chat model does every modality equally well.

Strength varies by training mix; audio or video may lag behind text.

Key facts

• Multimodal systems encode non-text inputs as tokens or embeddings.
• Vision-language models often pair a vision encoder with a text transformer.
• Products may compose multiple single-modality models behind one interface.
• Safety and privacy risks differ by modality (faces, documents, audio).
• Benchmarks on charts or photos may not reflect your deployment conditions.