Transformers pipeline(): Run a Hugging Face Model Locally

pipeline() turns a Hub model id into a result on your own machine: it downloads the weights, loads the tokenizer or preprocessor, picks a device and runs the task. The example model is Qwen/Qwen2.5-1.5B-Instruct: 1.5B parameters, a 3.1 GB download, no licence gate.

1. Install

Linux with an NVIDIA GPU (check with nvidia-smi):

pip install -U "transformers[torch]"

CPU only (smaller download, no CUDA libraries):

pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install -U transformers accelerate

Commands from the transformers install page; the [torch] extra includes accelerate, which device_map="auto" needs. On Windows the PyPI torch wheel is CPU-only, so install the CUDA build from the pytorch.org selector first. Sizing: RAM and VRAM per model.

2. Chat with an instruct model

import torch
from transformers import pipeline

pipe = pipeline(
    "text-generation",
    model="Qwen/Qwen2.5-1.5B-Instruct",
    dtype=torch.bfloat16,
    device_map="auto",
)

messages = [
    {"role": "system", "content": "You are a concise assistant."},
    {"role": "user", "content": "Explain what a tokenizer does in two sentences."},
]

out = pipe(messages, max_new_tokens=128)
print(out[0]["generated_text"][-1]["content"])

The first run downloads the weights. A list of role/content dicts gets the model’s chat template applied; generated_text returns the whole conversation, so [-1] is the new reply.

3. Length, device, precision

max_new_tokens caps the reply length. device_map="auto" puts layers on the GPU first and spills to CPU if VRAM runs out; use device="cpu" or device=0 instead for a single fixed device, never both. dtype sets the load precision; the default "auto" keeps the checkpoint’s own. Older tutorials write torch_dtype=, which current transformers flags as deprecated.

4. Same API, other modalities

from transformers import pipeline

classifier = pipeline("image-classification", model="google/vit-base-patch16-224")
preds = classifier("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg")
for p in preds[:3]:
    print(f"{p['score']:.2f}  {p['label']}")

Prints the top three ImageNet labels with scores (0.43 lynx, catamount first). Only the task string and model change; automatic-speech-recognition works the same way but needs ffmpeg installed to decode audio files, see transcribe audio locally with Whisper.

5. Where the weights went

Everything lands in the Hugging Face cache: ~/.cache/huggingface/hub/models--Qwen--Qwen2.5-1.5B-Instruct (on Windows C:\Users\<you>\.cache\huggingface\hub\…). Later runs reuse it. Move or clean the cache if your system drive is small.

If it fails

• torch.OutOfMemoryError: CUDA out of memory. Tried to allocate … Weights plus activations exceed VRAM. Use a smaller model or keep device_map="auto" so layers spill to CPU; a quantized GGUF in llama.cpp needs far less memory.
• OSError: You are trying to access a gated repo. … 401 Client Error Open the model page, accept the licence, then run hf auth login with a read token.
• ValueError: The repository … contains custom code which must be executed to correctly load the model. Read the repo’s Python files first; only then pass trust_remote_code=True to pipeline(), because it runs that code on your machine.

---

Next: load, stream and filter a dataset · call a model via Inference Providers · deploy it as a Gradio Space · download models with the hf CLI · llama.cpp quickstart · serve a model with vLLM.