Generate Embeddings Locally with EmbeddingGemma

An embedding turns a string into a vector, so you compare meaning by distance instead of keywords. That powers semantic search, deduplication and the retrieval step in RAG. EmbeddingGemma 2 is Google’s open embedding model: 740M parameters, 768-dimensional output, and it runs locally through Sentence Transformers.

1. What embeddings are for

If vectors are new to you, read embeddings explained with code first. The short version: similar texts land near each other, so you embed a corpus once, store the vectors, then embed a query and return the nearest matches. EmbeddingGemma expects you to label which side is the query and which is the document, which the next steps handle.

2. Install and accept the license

pip install -U sentence-transformers

Gemma models are gated: open the model page, accept the licence once, then authenticate with hf auth login (see the hf CLI guide).

3. Encode queries and documents

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("google/embeddinggemma-2")

query = "How do I cache Hugging Face models?"
documents = [
    "Downloaded weights live in the Hugging Face cache under HF_HOME.",
    "A tokenizer splits text into model-sized pieces.",
]

q = model.encode(query, prompt_name="SearchQuery")
d = model.encode(documents, prompt_name="Document")

prompt_name applies the prefixes the model was trained on, task: search result | query: … for the query and title: none | text: … for each document; skip them and retrieval quality drops.

4. Score cosine similarity

scores = model.similarity(q, d)
print(scores)  # e.g. tensor([[0.79, 0.31]])

similarity returns cosine similarity by default, so the higher score matches the query to the cache sentence rather than the tokenizer one. Keep the load precision at float32 or bfloat16, because EmbeddingGemma’s activations do not support float16.

5. Feed a vector database

At scale you do not loop in Python: you store vectors in a vector database such as Chroma, FAISS or pgvector and let it do nearest-neighbour search. Embed and insert documents with prompt_name="Document", embed each incoming query with prompt_name="SearchQuery", and wire the results into a generator; minimal RAG in Python shows the full loop. Need something smaller? google/embeddinggemma-300m is the 308M text-only version and uses model.encode_query() and model.encode_document() instead.

If it fails

OSError: ... is a gated repo or 401 Client Error You have not accepted the licence or logged in. Open the model page, accept it, then run hf auth login with a read token.
ValueError: Prompt name 'query' is not in the configuration The names are case-sensitive. Use exactly "SearchQuery" and "Document".
All-nan vectors or a float16 error You loaded in float16. Load with SentenceTransformer("google/embeddinggemma-2", model_kwargs={"torch_dtype": "float32"}) or use bfloat16.

Next: embeddings explained with code · minimal RAG in Python · RAG explained simply.