Call any local model with the OpenAI-compatible API

Ollama, LM Studio, the llama.cpp server and vLLM all speak the same OpenAI-compatible wire format under a /v1 path. One client library works against any of them — only the base URL and the model name change.

1. Find your server’s base URL and port

Ollama       http://localhost:11434/v1
LM Studio    http://localhost:1234/v1
llama.cpp    http://localhost:9931/v1
vLLM         http://localhost:8000/v1

These are each server’s current default; override with its --host/--port flag or app setting (older llama.cpp builds used port 8080).

2. List the models with GET /v1/models

curl http://localhost:8000/v1/models

Every server implements this endpoint, and the id fields it returns are the exact model names to pass below — this example hits vLLM, so swap the port for another server.

3. Run a chat completion with the OpenAI SDK

pip install openai

from openai import OpenAI

client = OpenAI(base_url="http://localhost:1234/v1", api_key="not-needed")
resp = client.chat.completions.create(
    model="your-model-id",
    messages=[{"role": "user", "content": "Say this is a test"}],
)
print(resp.choices[0].message.content)

Point base_url at your server and pass any non-empty api_key, which local servers require as a field but ignore as a value.

4. Stream the response token by token

stream = client.chat.completions.create(
    model="your-model-id",
    messages=[{"role": "user", "content": "Count to five."}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

With stream=True the text arrives in chunks at choices[0].delta.content, which is None on the final chunk — hence the or "".

5. Check which params each server supports

Honesty check: not every server implements every field. All four handle model, messages, temperature, max_tokens and stream, but coverage of tools, response_format, logprobs and /v1/embeddings varies between projects and versions. Check the server’s own docs before you depend on a parameter, and prefer the model strings reported by /v1/models over names copied from a blog.

Verify it worked

A good sign: curl .../v1/models returns JSON with a data array, and a chat call prints text.

If it fails:

• Connection refused — the server isn’t running or the port is wrong; recheck the list in step 1 (LM Studio starts its server from the Developer tab; vLLM and llama.cpp print the bound address on launch).
• 404 model '...' does not exist — the model string doesn’t match any id from /v1/models; copy it verbatim.
• api_key client option must be set — the SDK needs a non-empty api_key, so pass any placeholder string.

Server setup lives in the dedicated guides: Ollama, LM Studio, vLLM and llama.cpp.

Next: point Cursor or Claude Code at a local model.

Source: Ollama OpenAI compatibility · LM Studio OpenAI API · vLLM OpenAI-compatible server.