Tutorials · Chapter C (3/4) · ~8 min
Multimodal AI
Try it → see it → read → next
One brain-ish stack that can handle text, images, audio — not five unrelated apps.
Try yourself
Playground
Mix the senses
Turn modalities on/off. Multimodal = more than one channel into the same ask.
Explain this like I’m new…
Enable image or audio with text, then run.
Recap
What you just did
You peeked at how a computer “sees” simple patterns. Multimodal systems connect those signals to language.
Teach
How it works
Typical pattern:
- An encoder turns an image (or audio) into vectors.
- Those vectors sit in a space the language model can use.
- The LLM reasons / answers in text (or triggers an image generator).
So “what’s in this photo?” is: vision features → language model → words.
Use it
When you'd use this
- Accessibility (describe images).
- Studying from lecture slides photos.
- Design feedback on a mockup screenshot.
Watch out
Watch out
Charts and small text in images are easy to misread. Always verify critical numbers.
Try next
Try this next
Ask a multimodal chat to describe a public-domain photo — then compare to what you see.