A vision-language model (VLM) takes an image and text together and answers in text. Unlike OCR it understands the picture: it captions a scene, answers questions, reads a chart, or explains a diagram. This guide uses Qwen3-VL-8B-Instruct (Apache-2.0), a small multimodal model with a clean transformers API — see its model card. If you only want the characters out of a scan, run OCR locally instead.
1. What a vision-language model does
Give it an image and a prompt; it returns free-form text. That covers captioning, visual question answering, chart and table reading, and document or UI understanding — tasks plain OCR cannot do, because they need reasoning about what is shown, not just transcription of glyphs.
2. Install transformers
Qwen3-VL is recent, so the model card says to install transformers from source alongside torch:
pip install -U "git+https://github.com/huggingface/transformers" accelerate torchvision
The Qwen3-VL classes ship only in the latest transformers; the transformers pipeline quickstart covers a plain release install.
3. Load the model and processor
import torch
from transformers import Qwen3VLForConditionalGeneration, AutoProcessor
model = Qwen3VLForConditionalGeneration.from_pretrained(
"Qwen/Qwen3-VL-8B-Instruct", dtype="auto", device_map="auto"
)
processor = AutoProcessor.from_pretrained("Qwen/Qwen3-VL-8B-Instruct")
Downloads the weights on first run and places them on your GPU automatically.
4. Ask a question about an image
messages = [{
"role": "user",
"content": [
{"type": "image", "image": "photo.jpg"},
{"type": "text", "text": "What is in this image? Read any text you see."},
],
}]
inputs = processor.apply_chat_template(
messages, tokenize=True, add_generation_prompt=True,
return_dict=True, return_tensors="pt",
).to(model.device)
ids = model.generate(**inputs, max_new_tokens=128)
trimmed = [out[len(inp):] for inp, out in zip(inputs.input_ids, ids)]
print(processor.batch_decode(trimmed, skip_special_tokens=True)[0])
apply_chat_template loads photo.jpg (a local path or a URL), builds the prompt, and the trim drops the echoed input so you print only the answer.
5. Fit it on your GPU
At 9B parameters in bf16 the model wants a mid-range to large GPU; see how much VRAM you need. No headroom? Qwen ships smaller Qwen3-VL sizes, and a quantized build — the trade-offs are in choosing a quantization — cuts memory further.
Verify it worked
The printed text should describe your actual image. A generic, image-free answer usually means the path was wrong and only the text prompt reached the model.
If it fails:
KeyError: 'qwen3_vl'or an unknown-model-type error — transformers is too old; reinstall it from source.torch.cuda.OutOfMemoryError— load a smaller Qwen3-VL variant or a quantized build.- Blank or irrelevant output — confirm
photo.jpgexists in the working directory and is a readable image file.