Multimodality

Multimodal systems handle more than one kind of signal—text plus images, audio, video, or other sensors.

Real life is not only text. Accessibility tools, document apps that read screenshots, and creative suites increasingly combine modes—and inherit each mode’s failure modes (Course 29 deepfake literacy).

Often: encode image/audio into vectors the language model can attend to, then generate text (or the reverse for image generation). Training aligns those spaces with paired data.

“Explain this chart screenshot” or “describe this photo for alt text.”

Give an image-capable assistant a chart and ask for the trend—then verify the numbers yourself.

⚠️ Myth: If it can see, it can measure perfectly.

✓ Reality: OCR and chart-reading still err; verify critical figures.

⚠️ Myth: Multimodal means safe from deepfakes.

✓ Reality: Generative multimodal tools also create synthetic media.

• ANN live: https://www.ainerdnetwork.com/learn/multimodality ↗
• Provider multimodal docs (cite the specific model docs you use)
• Course 11 generative media; Course 13 speech