Chapter BMultimodal promptsPage 1 of 8

Multimodal prompts

Start with the job to be done

Frame the outcome, evidence, and human decision before asking the model to produce anything.

~13 minJob framing

1Try it yourself

Playground

Mix the senses

Turn modalities on/off. Multimodal = more than one channel into the same ask.

Explain this like I’m new…

Enable image or audio with text, then run.

Before you start

Why this matters

Without opening an AI tool, write the acceptance test for this job: diagnose a confusing dashboard screenshot using image evidence plus written context. Name one fact that must be exact, one judgment a person must make, and one condition that should stop the workflow. Compare your answer with the professional standard below; the gap is what you should practice.

2Learn the idea

Read

Define the professional job

The working assignment is to diagnose a confusing dashboard screenshot using image evidence plus written context for a product analyst preparing a bug report. That sentence is narrower than “use Multimodal prompts.” It identifies a deliverable and a reviewer. Write a definition of done with three layers: the output must satisfy the audience's need; factual or functional claims must be traceable; and a named person must own the final decision. Point to regions, labels, frames, or timestamps. When the model cannot read a detail reliably, provide a crop or transcription instead of asking it to guess harder.

Start by separating tasks. The model may draft, classify, transform, compare, or suggest. It may not silently approve, publish, grade, deploy, cite, or consent on someone's behalf. For this assignment the authoritative material is cropped screenshot, task goal, UI version, visible labels, expected behavior, sensitive-area redactions, and requested evidence format. Anything absent from those inputs is either an explicit assumption or an unanswered question.

Read

Convert the job into a contract

Use this prompt as a realistic starting contract:

Inspect this redacted checkout dashboard screenshot. Goal: explain why the conversion total appears inconsistent. First inventory only visible labels and values with locations. Separate observation from inference. Then propose three hypotheses and the next screenshot or log needed to test each. Do not identify people, infer hidden fields, or invent unreadable text.

Notice what the prompt does: it states the setting, limits the output, names forbidden behavior, and requests evidence that can be reviewed. It does not ask the model to “make it amazing.” If a constraint matters, make it testable. Replace “be accurate” with a source boundary, formula check, test command, rights ledger, or approval step.

A useful response would look like this: An observation table grounded in screen regions, clearly labeled hypotheses, and targeted follow-up evidence instead of a confident visual guess. That description is intentionally observable. “Looks good” is not acceptance. The operator must zoom and compare source pixels, transcribe critical values manually, reproduce in the product, inspect logs, and reject claims unsupported by visible or supplied evidence. Keep the source material beside the draft so review means comparison, not memory.

Read

Scope and stop rules

Run the work through sanitize → orient → observe → infer → request evidence → verify. Stop when an authoritative input is missing, a high-risk claim lacks evidence, private material cannot be safely removed, or the proposed action exceeds the permission granted. Escalation is successful workflow behavior, not model failure.

Common framing mistakes are hallucinated small text; mixing observation with inference; ignoring chart scale; accidental exposure around the crop; treating an image as current truth. Prevent them by writing a one-paragraph job card: user, decision, deliverable, source of truth, constraints, reviewer, and stop condition. This card becomes the anchor for every later prompt.

Checking tutor…

Continue learning · glossary & guides
  • Can the job be completed and reviewed without guessing its purpose?
  • Which action remains owned by a person, and what evidence will that person inspect?
  • Reference · Related concept
  • Next