mediumAI Engineering

How do multimodal models handle images, audio and other non-text inputs?

Q: How do multimodal models handle images, audio and other non-text inputs?

**TL;DR.** Non-text inputs are encoded into the **same token/embedding space** the language model reasons over — images become visual tokens via a vision encoder, audio becomes acoustic tokens. One model then attends across modalities. Engineering realities: **images cost thousands of tokens**, resolution settings matter, and prompts should reference the media explicitly. image audio <rect class='d-box-mute

871 views

Understand the problem

Vision encoders, audio tokens, and what multimodality means for app design.

multimodalvisionaudiofundamentals

Attempt it yourself

Sketch your approach before reading the solution — that's what interviews test.

Stuck? AI Nudge Available

Get a conceptual hint to guide your logic without spoiling the final implementation.

Study the solution

The solution is waiting

Give it an honest attempt first — then compare your thinking with the full walkthrough.

Read the code

Vision request with cost control

const resized = await sharp(buf).resize({ width: 1024 }).toBuffer(); // cap tokens

await client.messages.create({
  model,
  max_tokens: 500,
  messages: [{
    role: "user",
    content: [
      { type: "image", source: { type: "base64", media_type: "image/png",
        data: resized.toString("base64") } },
      { type: "text", text: "From this invoice, extract vendor, date, total as JSON." },
    ],
  }],
});

Join the discussion

Discussion (0)

No responses yet. Be the first to share what you think.

When do you use code-based graders versus model-based graders?

What is the difference between fine-tuning, LoRA and prompt engineering?