“Can you tell me what’s wrong with this circuit diagram?”
I sent this message to ChatGPT along with a photo of a circuit board I was troubleshooting. The response? “I’m sorry, but I can’t see any images you’ve shared.”
Right. Text-only AI has its limits.
But the latest generation of AI models, multimodal models, can see images, hear sounds, and even understand video. They’re changing what’s possible with AI, and they’re already more accessible than you might think.
What exactly is multimodal AI?
Traditional LLMs like early versions of GPT only understand text. Multimodal models can process and generate multiple types of data:
Text-only LLM:
Input: Text → Model → Output: Text
Multimodal AI:
Input: Text + Images + Audio → Model → Output: Text + Images + Audio
Four things multimodal AI can actually do
After experimenting with models like GPT-4V, Claude 3, and Gemini, I’ve found these capabilities most useful:
Visual understanding: Analyzing charts, diagrams, screenshots, and photos
"What's wrong with this error message?" "Can you explain this graph trend?" "What species of plant is this?"Document analysis: Extracting information from PDFs, receipts, and forms
"Summarize the key points from this research paper" "Extract the line items from this invoice"Visual creation: Generating or editing images based on text descriptions
"Create a flowchart showing how JWT authentication works" "Edit this logo to use blue instead of red"Cross-modal reasoning: Connecting concepts across different formats
"Does this code match the architecture diagram?" "Is this UI implementation consistent with the wireframe?"
Real-world example: debugging a React app in 2 minutes instead of 20
Last week, I was struggling with a React component that wasn’t rendering properly. After 15 minutes of fruitless debugging, I took a screenshot of both my code and the broken UI, then asked GPT-4V:
“What’s causing this rendering issue? Here’s my component code and the current output.”
In seconds, it spotted the problem: I was destructuring props incorrectly, causing a key prop to be undefined. What would have been another 20 minutes of debugging was solved in 2 minutes.
Getting started with multimodal AI today
No special hardware, no complex setup. Here’s how to use GPT-4 with Vision through the API:
curl https://api.openai.com/v1/chat/completions \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-4-turbo",
"messages": [
{
"role": "user",
"content": [
{ "type": "text", "text": "Describe this image" },
{ "type": "image_url", "image_url": { "url": "https://example.com/image.png" } }
]
}
]
}'
Send text and an image, get back analysis. That’s all it takes.
Prompting multimodal models
How you prompt these models makes a real difference:
Be specific about what to look for:
Bad: "What do you see in this dashboard screenshot?" Good: "In this AWS CloudWatch dashboard screenshot, identify any metrics that show abnormal spikes in the last 24 hours"Provide context for images:
Bad: [Image only] Good: "This is a network diagram for our AWS infrastructure. Are there any security issues?"Request structured output:
"Analyze this sales chart and provide: 1. The overall trend 2. Any anomalies 3. Recommendations based on the data"
An unexpected use: accessibility
One thing I didn’t expect to find useful is generating accessibility content:
# Generate alt text for images
"Create detailed alt text for this product image that would be helpful for visually impaired users"
# Convert visual information to text
"Describe this architectural diagram in text form that could be read by a screen reader"
Not the flashiest use case, but a real one.
Where this is going
Video understanding is the obvious next step, and some of it is already showing up. Models that can analyze recordings or live feeds are coming, mostly for specialized domains like medicine and security right now.
For now, the useful habit change is simpler: if you’ve been copy-pasting error text when you could just screenshot it and ask the model what’s wrong, try that first. It saves more time than you’d expect.