Skip to content

Technique

Intermediate

Multimodal prompting

Combine text with images, screenshots, PDFs, or audio in your prompt, and tell the AI exactly what to look for.

What it is

Multimodal prompting means giving a model more than one kind of input: text plus images, screenshots, scanned documents, charts, or audio, depending on what the tool supports. Many AI assistants can now read a photo of a whiteboard, interpret a chart, or describe an error on a screenshot.

The same rules of good prompting apply: the image supplies context, and your text supplies the task, focus, and format.

When to use it

  • The information is visual: charts, diagrams, handwriting, receipts, UI screenshots
  • Describing something in words would be slow or lossy
  • Extracting text or data from photos or scanned documents
  • Getting feedback on designs, layouts, or slides

How it works

Why it works

Multimodal models process images (and in some tools audio or video) alongside text, so they can answer questions about what they "see." But an image alone is an open-ended prompt: the model doesn't know whether you care about the colors, the text, the layout, or the numbers. Your text tells it where to focus.

Before

[screenshot attached] What do you think?

After

[screenshot attached] This is the signup page for our budgeting app. Our target users are first-time budgeters, many on phones. Review it for clarity and list: 1. The 3 biggest usability problems, each with where it appears on the screen 2. One concrete fix for each Ignore brand colors; we can't change those.

Tips for better results

  • Point to the region. "The table in the bottom-left" or "the second chart" helps when an image is busy.
  • Say what the image is. "This is a photo of a restaurant receipt" frames how to interpret it.
  • Ask for a transcription first for detailed extraction: "First transcribe the handwritten notes exactly, then summarize them." This lets you check what the model read before trusting its analysis.
  • Use good inputs. Clear, well-lit, cropped, readable images get better results than blurry or tiny ones.
  • Combine with structured output to turn images into data:
Prompt
The attached photo is a handwritten weekly chore chart.
Transcribe it into a markdown table with columns: Day | Person | Chore.
If any word is unclear, write [unclear] instead of guessing.

Limits

Watch out: Models can misread small text, numbers, handwriting, and chart values, and may describe details that aren't there. Double-check anything you'll act on, especially amounts, dates, and measurements.

Capabilities vary by tool: some accept images but not audio, some handle multi-page PDFs better than others. Check what your assistant supports.

Note: Be careful with images that contain other people's personal information, such as faces, IDs, or documents, and follow your organization's rules about what you can upload.

Step by step

  1. 1

    Check that your AI tool supports the input type (image, PDF, audio).

  2. 2

    Use a clear, readable, cropped input focused on what matters.

  3. 3

    Say what the input is and why you're sharing it.

  4. 4

    Point to the specific part to focus on and state the task.

  5. 5

    Specify the output format, and how to handle unreadable or uncertain parts.

  6. 6

    Verify extracted numbers, text, and details against the original.

Weak vs strong

Weak prompt
[photo of a utility bill] Explain this.
Why it falls short: The model doesn't know what you're confused about — the total, a specific charge, the usage graph — so it will give a generic walkthrough that may miss your question.
Strong prompt
[photo of a utility bill] This is my electricity bill. It's much higher than last month. Please: (1) list each line-item charge and amount in a table, (2) point out which charges changed versus the previous period, if the bill shows that, and (3) explain in plain language what the "delivery charge" is. If any number is hard to read, say so instead of guessing.
Why it works: It identifies the document, explains the concern, breaks the task into specific parts with a format, and tells the model how to handle unreadable values.

Practical use cases

  • Receipts and forms to data

    Photograph receipts, invoices, or handwritten forms and extract the fields into a table.

  • Troubleshooting from screenshots

    Share an error dialog or settings screen and ask what it means and what to try next.

  • Chart and slide interpretation

    Ask for the key trend in a chart, or feedback on whether a slide is readable from the back of the room.

  • Learning from visuals

    Photograph a textbook diagram or whiteboard and ask for a step-by-step explanation or practice questions.

Practice exercise

Photo to table. Take a photo of something with text and numbers: a receipt, a nutrition label, or a handwritten list.

  1. Ask the AI to transcribe it into a table with columns you choose, marking unclear items as [unclear].
  2. Compare the table to the original and count any misreadings.
  3. Retake the photo (better light, tighter crop) or adjust the prompt, and see if accuracy improves.
Practice in the PlaygroundAll techniques