GPT-5 Vision API Guide: Image Analysis, OCR, and Multimodal AI

GPT-5 Vision brings true multimodal understanding to AI applications. It does not merely detect objects—it reasons about images, reads handwritten text, understands charts and diagrams, answers questions about visual content, and extracts structured data from any image. This guide covers the full capabilities of GPT-5 Vision with practical, production-ready code examples.

What Makes GPT-5 Vision Different

Earlier vision-language models treated image analysis as a bolt-on feature: detect objects, generate a caption, move on. GPT-5 Vision is natively multimodal, meaning visual and text processing share the same reasoning engine. This enables capabilities that traditional computer vision APIs simply cannot match.

Traditional OCR tools like Tesseract extract text but cannot understand it. GPT-5 Vision reads a financial statement and understands which numbers are revenue versus expenses, identifies anomalies, and can answer questions about the data. A traditional object detection model finds a table in an image; GPT-5 Vision can read the table, understand its structure, extract it as structured data, and reason about its contents.

Key GPT-5 Vision improvements over GPT-4 Vision: 40% better accuracy on document understanding benchmarks, native support for high-resolution images up to 8K, improved handwriting recognition with 95% accuracy on legible handwriting, and the ability to process multiple images in a single conversation for comparison and analysis.

Getting Started: Your First Vision Request

GPT-5 Vision uses the same chat completions API as text. You include images as message content alongside text:

from openai import OpenAI
import base64

client = OpenAI(
    api_key="your-drai-api-key",
    base_url="https://ai.dr-ai.top/v1"
)

# Method 1: Base64-encoded image
with open("invoice.png", "rb") as f:
    image_data = base64.b64encode(f.read()).decode("utf-8")

response = client.chat.completions.create(
    model="gpt-5",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Extract all line items from this invoice. "
                           "Return as a JSON array with item name, "
                           "quantity, unit price, and total."
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/png;base64,{image_data}",
                        "detail": "high"  # Use "high" for documents
                    }
                }
            ]
        }
    ],
    max_tokens: 2000
)

print(response.choices[0].message.content)
# [{"item": "Web Hosting (Annual)", "quantity": 1,
#   "unit_price": 240.00, "total": 240.00}, ...]

# Method 2: Image URL (for publicly accessible images)
response = client.chat.completions.create(
    model="gpt-5",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text",
                 "text": "What type of chart is this? "
                        "What are the key trends?"},
                {"type": "image_url",
                 "image_url": {
                     "url": "https://example.com/chart.png",
                     "detail": "high"
                 }}
            ]
        }
    ]
)

The detail parameter controls processing quality. Use "low" for quick, cheaper analysis (suitable for simple object detection). Use "high" for document OCR, chart reading, and detailed analysis. The cost difference is significant—high detail processes more tokens.

OCR and Document Understanding

GPT-5 Vision excels at extracting text from images, even challenging ones. It handles receipts, handwritten notes, technical diagrams, multi-column layouts, tables, and mixed-language documents. Unlike traditional OCR, it understands context and can correct for image quality issues.

Receipt and Invoice Processing

def extract_receipt_data(image_path):
    """Extract structured data from a receipt image."""
    with open(image_path, "rb") as f:
        img_b64 = base64.b64encode(f.read()).decode()

    response = client.chat.completions.create(
        model="gpt-5",
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": """Extract from this receipt:
                - Merchant name
                - Date and time
                - All line items (name, price)
                - Subtotal, tax, tip, total
                - Payment method
                Return as JSON.""",
                 },
                {"type": "image_url",
                 "image_url": {"url": f"data:image/jpeg;base64,{img_b64}",
                              "detail": "high"}}
            ]
        }],
        response_format={"type": "json_object"}
    )
    return response.choices[0].message.content

Handwritten Note Recognition

Traditional OCR struggles with handwriting. GPT-5 Vision achieves near-human accuracy on legible handwriting and significantly outperforms dedicated OCR engines on messy or stylized writing:

response = client.chat.completions.create(
    model="gpt-5",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text",
             "text": "Transcribe this handwritten note exactly "
                    "as written. Preserve line breaks and "
                    "formatting. If any words are illegible, "
                    "mark them as [illegible]."},
            {"type": "image_url",
             "image_url": {"url": f"data:image/png;base64,{img}",
                          "detail": "high"}}
        ]
    }]
)

Visual Question Answering

Beyond text extraction, GPT-5 Vision can answer complex questions about image content. This opens up applications in accessibility, education, quality control, and more:

def ask_about_image(image_url, question):
    response = client.chat.completions.create(
        model="gpt-5",
        messages=[{
            "role": "system",
            "content": "You are a visual analysis assistant. "
                       "Answer questions about images accurately "
                       "and concisely. If you cannot determine "
                       "something, say so."
        }, {
            "role": "user",
            "content": [
                {"type": "text", "text": question},
                {"type": "image_url",
                 "image_url": {"url": image_url, "detail": "high"}}
            ]
        }]
    )
    return response.choices[0].message.content

# Examples:
# "How many people are in this image and what are they doing?"
# "Is this circuit board damaged? Describe any visible defects."
# "What brand and model is this product?"
# "Does this food look fresh? Describe its condition."

Chart and Data Visualization Analysis

GPT-5 Vision can read charts and graphs, extract the underlying data, and provide insights. This is invaluable for business intelligence and data analysis automation:

response = client.chat.completions.create(
    model="gpt-5",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": """Analyze this sales chart:
            1. What type of chart is this?
            2. Extract all data points
            3. Identify the trend
            4. What was the best-performing quarter?
            5. Calculate the year-over-year growth rate
            """},
            {"type": "image_url",
             "image_url": {"url": chart_url, "detail": "high"}}
        ]
    }]
)

The model returns a detailed analysis: chart type identification, extracted data values, trend analysis, and computed metrics. For data analysis workflows, you can also pair this with GPT-5 function calling to automatically write SQL queries or generate follow-up visualizations.

Multi-Image Analysis

GPT-5 Vision can process multiple images in a single conversation, enabling comparison and batch analysis. This is essential for quality control, medical imaging, and any task requiring visual comparison:

# Compare product images for quality control
response = client.chat.completions.create(
    model="gpt-5",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text":
             "Compare these two product photos. "
             "Identify any differences in color, "
             "packaging, or visible defects."},
            {"type": "image_url",
             "image_url": {"url": image1_url}},
            {"type": "image_url",
             "image_url": {"url": image2_url}},
        ]
    }]
)

# Batch document processing
documents = [doc1_b64, doc2_b64, doc3_b64]
content = [{"type": "text", "text":
    "Categorize each document and extract key dates."}]
for doc in documents:
    content.append({
        "type": "image_url",
        "image_url": {"url": f"data:image/png;base64,{doc}"}
    })

Structured Data Extraction from Images

One of the most valuable use cases is converting visual information into structured, machine-readable data. Combine GPT-5 Vision with structured outputs for reliable extraction:

from pydantic import BaseModel

class IDCard(BaseModel):
    name: str
    date_of_birth: str
    id_number: str
    address: str
    expiry_date: str
    issuing_authority: str

response = client.beta.chat.completions.parse(
    model="gpt-5",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text":
             "Extract all information from this ID card."},
            {"type": "image_url",
             "image_url": {"url": id_card_url, "detail": "high"}}
        ]
    }],
    response_format=IDCard
)

card_data = response.choices[0].message.parsed
# IDCard(name="John Smith", date_of_birth="1985-03-15",
#        id_number="DL12345678", ...)

Building a Vision-Enabled Agent

Combine vision with function calling to build agents that can see and act. For example, a support agent that can read error screenshots and automatically diagnose issues:

VISION_AGENT_TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "search_kb",
            "description": "Search the knowledge base for "
                           "solutions to a technical issue.",
            "parameters": {
                "type": "object",
                "properties": {
                    "query": {"type": "string"},
                    "error_code": {"type": "string"}
                },
                "required": ["query"]
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "create_ticket",
            "description": "Create a support ticket for "
                           "issues that need human attention.",
            "parameters": {
                "type": "object",
                "properties": {
                    "summary": {"type": "string"},
                    "priority": {"type": "string",
                        "enum": ["low", "medium", "high"]},
                    "error_details": {"type": "string"}
                },
                "required": ["summary", "priority"]
            }
        }
    }
]

# The agent sees an error screenshot, reads the error message,
# searches the KB, and either provides a solution or creates
# a ticket - all from one image upload

Learn more about building agents in our GPT-5 Function Calling Guide.

Performance and Cost Optimization

Vision requests consume more tokens than text-only requests. Image token costs depend on resolution and detail level. Here are strategies to manage costs:

For comprehensive cost management strategies, see our Token Optimization Techniques and AI Cost Optimization guides.

Handling Common Vision Challenges

Low-quality images: GPT-5 Vision is remarkably resilient to blur, low resolution, and poor lighting. However, for extreme cases, pre-process with traditional image enhancement (upscaling, denoising) before sending to the API.

Dense tables: For complex tables, ask the model to extract row by row rather than all at once. Provide the table structure in the prompt: "This table has 5 columns: Date, Product, Quantity, Price, Total."

Multi-page documents: Process each page separately and merge results. Include page numbers in the prompt: "This is page 3 of a 10-page document."

Privacy and PII: Vision requests may contain sensitive information in images. Ensure your API provider has appropriate data handling policies. The DrAI platform offers zero-retention options for enterprise customers. For security best practices, see our LLM Security Guide.

Use Cases Across Industries

Healthcare: Read medical charts, analyze X-ray descriptions, extract data from clinical trial forms. (Always with appropriate regulatory compliance.)

Finance: Process receipts and invoices at scale, read financial statements, analyze trading charts, automate expense report data entry.

Manufacturing: Quality control inspection, defect detection on production lines, reading serial numbers and labels, comparing products to reference images.

Education: Grade handwritten homework, read student diagrams, create accessible descriptions of images for visually impaired students, convert whiteboard photos to structured notes.

Legal: Extract data from contracts and legal documents, read case files, process evidence photos, automate document review.

Conclusion

GPT-5 Vision represents a paradigm shift from narrow computer vision to general-purpose visual understanding. Whether you need OCR, chart analysis, visual question answering, or structured data extraction, the same API handles it all—no separate models or pipelines needed.

Start with simple image-to-text tasks, then explore structured extraction with JSON outputs, and finally combine vision with function calling for autonomous visual agents. The possibilities are limited only by your imagination.

The DrAI platform provides access to GPT-5 Vision and 40+ other models through a unified API. Check our pricing for competitive pay-as-you-go rates, and explore our AI Image Generation Guide for the reverse direction—generating images from text.

Try GPT-5 Vision on DrAI →