Describe and Query Images with a Local VLM

A vision-language model (VLM) takes an image and text together and answers in text. Unlike OCR it understands the picture: it captions a scene, answers questions, reads a chart, or explains a diagram. This guide uses Qwen3-VL-8B-Instruct (Apache-2.0), a small multimodal model with a clean transformers API — see its model card. If you only want the characters out of a scan, run OCR locally instead.

1. What a vision-language model does

Give it an image and a prompt; it returns free-form text. That covers captioning, visual question answering, chart and table reading, and document or UI understanding — tasks plain OCR cannot do, because they need reasoning about what is shown, not just transcription of glyphs.

2. Install transformers

Qwen3-VL is recent, so the model card says to install transformers from source alongside torch:

pip install -U "git+https://github.com/huggingface/transformers" accelerate torchvision

The Qwen3-VL classes ship only in the latest transformers; the transformers pipeline quickstart covers a plain release install.

3. Load the model and processor

import torch
from transformers import Qwen3VLForConditionalGeneration, AutoProcessor

model = Qwen3VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen3-VL-8B-Instruct", dtype="auto", device_map="auto"
)
processor = AutoProcessor.from_pretrained("Qwen/Qwen3-VL-8B-Instruct")

Downloads the weights on first run and places them on your GPU automatically.

4. Ask a question about an image

messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": "photo.jpg"},
        {"type": "text", "text": "What is in this image? Read any text you see."},
    ],
}]

inputs = processor.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True,
    return_dict=True, return_tensors="pt",
).to(model.device)

ids = model.generate(**inputs, max_new_tokens=128)
trimmed = [out[len(inp):] for inp, out in zip(inputs.input_ids, ids)]
print(processor.batch_decode(trimmed, skip_special_tokens=True)[0])

apply_chat_template loads photo.jpg (a local path or a URL), builds the prompt, and the trim drops the echoed input so you print only the answer.

5. Fit it on your GPU

At 9B parameters in bf16 the model wants a mid-range to large GPU; see how much VRAM you need. No headroom? Qwen ships smaller Qwen3-VL sizes, and a quantized build — the trade-offs are in choosing a quantization — cuts memory further.

Verify it worked

The printed text should describe your actual image. A generic, image-free answer usually means the path was wrong and only the text prompt reached the model.

If it fails:

• KeyError: 'qwen3_vl' or an unknown-model-type error — transformers is too old; reinstall it from source.
• torch.cuda.OutOfMemoryError — load a smaller Qwen3-VL variant or a quantized build.
• Blank or irrelevant output — confirm photo.jpg exists in the working directory and is a readable image file.