Clone a Voice Locally with Chatterbox TTS

Chatterbox is Resemble AI’s open text-to-speech model, released under MIT. It does zero-shot voice cloning: give it a few seconds of reference audio and it speaks new text in that voice, with no training step. That is a step beyond the fixed voices in basic text-to-speech. The one rule that comes first: only clone voices you have permission to use. The full usage is on the Chatterbox model card.

1. How zero-shot voice cloning works

A zero-shot model conditions on a short reference clip at generation time instead of being fine-tuned per speaker. You hand it the clip as audio_prompt_path, and it copies that voice’s timbre and delivery onto whatever text you pass. Quality tracks the reference: clean, dry mono audio beats a noisy phone recording. To go the other direction and turn recordings into text, see transcribe audio with Whisper.

2. Install Chatterbox

pip install chatterbox-tts

Pulls in the model code plus torch and torchaudio; a CUDA GPU makes generation far faster than CPU.

3. Prepare a reference clip

Record or trim a short clip of the target voice speaking clearly, saved as a WAV (the official examples use a roughly ten-second file). Keep it mono and free of music or background noise, because the model imitates whatever it hears, artefacts included.

4. Generate speech in that voice

import torchaudio as ta
from chatterbox.tts import ChatterboxTTS

model = ChatterboxTTS.from_pretrained(device="cuda")

text = "Hello, this sentence is spoken in the cloned voice."
wav = model.generate(text, audio_prompt_path="reference.wav")
ta.save("output.wav", wav, model.sr)

Loads the model, conditions on reference.wav, and writes output.wav at the model’s own sample rate, read from model.sr.

5. Consent, length, and the watermark

Generate a sentence to a few sentences per call; split longer text and concatenate the clips. Every Chatterbox output carries Resemble’s built-in Perth (Perceptual Threshold) Watermarker, so synthetic audio stays detectable. The ethics are not optional: clone only your own voice or one you have explicit, documented permission to use, and never impersonate a real person.

Verify it worked

Play output.wav: it should read your text in the reference voice. If it sounds like a generic default voice, the reference clip was not applied.

If it fails:

• ModuleNotFoundError: No module named 'chatterbox' — the package is chatterbox-tts; reinstall it under that name.
• torch.cuda.OutOfMemoryError or no GPU — pass device="cpu" to from_pretrained (slower, but it runs).
• Robotic or wrong-sounding voice — use a cleaner, longer mono reference; noise and music degrade the clone.