Two weeks in the past, we opened up Phonon, our small text-to-speech mannequin that runs by yourself gadget. It wants no server and no community, and what you sort stays with you.
We have been very pleased with the reception to date. So this is the story of the launch: why we constructed Phonon, what it could do, the demos that got here with it, and the way it did on Hugging Face’s Open TTS Leaderboard.
Why a voice ought to reside in your gadget
Most voice AI lives within the cloud: each sentence makes a spherical journey to a server. Phonon runs by yourself gadget as a substitute. Your textual content stays there and the voice works and not using a community. We’ve additionally opened the weights. Phonon 31m is Apache 2.0, so you should use it totally free, even in a business product. The opposite fashions are CC BY-NC, in order that they’re free for non-commercial initiatives. When you’re constructing one thing business with these, pricing relies in your variety of customers, not how a lot speech you generate.
Getting an excellent voice to run on a small gadget has all the time been laborious. You possibly can make the mannequin small, however then it didn’t sound nice. Or you would make it sound nice, however then it wouldn’t match. We addressed each.
We introduced Phonon in April and spent the subsequent six months constructing on it with companions and early customers. Their suggestions helped us to form this launch. The fashions bought smaller, and we added extra languages and assist for customized voices via voice cloning and voice design.
Phonon relies on the Pocket TTS lineage and as such makes use of move matching to generate the audio which could be very completely different from the modeling methods utilized in different TTS programs. That is what allowed us to carry this degree of high quality to such a small mannequin however it created new challenges whereas coaching the fashions. The coaching was initially numerically unstable and required a number of tips to supply high quality outcomes on all 5 languages.
What we launched
Phonon is available in two sizes, at about 65 MB and 75 MB. Collectively they converse English, French, German, Spanish and Portuguese. There are greater than 100 built-in voices, and you’ll create your personal in Gradium Studio. Phonon additionally handles the necessary issues that almost all different TTS programs usually come across, like cellphone numbers, dates and codes.
It runs on telephones, laptops, single-board computer systems and in a browser tab, so you may hear it proper now. Phonon 31m speaks English and is Apache 2.0, so you may construct on it freely, together with for business merchandise. Phonon 40m covers all 5 languages beneath a non-commercial license.
Demos and first initiatives
The launch got here with demos from pals and from our personal staff. On an Apple Watch with Wi-Fi off, Phonon talks you thru a espresso brew step-by-step. On an iPhone with no sign, it counts your reps and retains you going. A robotic known as Lamp, from Autonomous, wears a sorting hat and talks in character. In a browser, an offline dino runner turns into a radio present hosted by Rex. There’s additionally a sci-fi sport voiced on the participant’s personal machine, and Proust learn aloud web page by web page.
The Hugging Face staff constructed the Transformers integration, so builders may use Phonon in Transformers from day one.
Builders have began constructing on it too. An ExecuTorch conversion runs Phonon on a customer’s GPU within the browser, Micdrop brings it to Node.js apps, and Areas like smallest-s2s construct on it.
Phonon was downloaded greater than 200,000 occasions in its first two weeks, and it’s nonetheless trending on Hugging Face.
How Phonon did on the Hugging Face Open TTS Leaderboard
The Open TTS Leaderboard is a really new challenge from Hugging Face, and we like it. With greater than 9,000 TTS fashions on the Hugging Face Hub, the leaderboard offers builders one place to match open fashions and listen to them facet by facet. Throughout 23 languages, fashions are evaluated on intelligibility (phrase error charge of the outputs), velocity (inverse real-time issue and time-to-first audio), and naturalness (UTMOS for English and KOMOS for Korean). Fastened-voice and voice cloning are evaluated, with a further speaker similarity metric for the latter. The leaderboard additionally has a Hear tab, the place you may conveniently examine TTS outputs your self and vote for those you want essentially the most.
On intelligibility, Phonon ranks first in German and Spanish, and within the high three in each language we evaluated, subsequent to fashions as much as 100 occasions its measurement.
Pace is the place we see Phonon really shine, because the fashions high the CPU streaming benchmark on time-to-first audio.
Inference
We constructed a brand new inference framework, ptts, particularly for Phonon. The purpose of ptts is to get one of the best efficiency on all kinds of units and as such helps greater than 10 backends: CPU (Intel, Arm, WebAssembly), GPU (CUDA, Vulkan, Steel, WebGPU), and NPU (Apple Core ML, Snapdragon®).
Phonon can also be out there in 🤗 Transformers, from v5.21, as its reference PyTorch implementation. The place ptts runs a single stream on gadget, Transformers targets GPU throughput: batched era, full-graph torch.compile, e.g., for synthesizing speech datasets at scale. Being native PyTorch, it’s a base to construct upon, whether or not to combine Phonon into your personal stack or to review the mannequin itself.
It belongs to the Pocket TTS lineage, so we first added Pocket TTS to Transformers. Phonon sits on high as a modular mannequin: about 200 strains describing solely what differs.
Phonon generates audio latents autoregressively: every step outputs a latent that’s fed again as the subsequent enter, and the coaching loss is computed on them. Latents are the mannequin’s inputs, outputs and targets, as textual content tokens are for a language mannequin, so the codec mapping audio to latents performs the function of a tokenizer. Transformers ships it as such: an audio tokenizer, held by the processor subsequent to the textual content tokenizer:
from transformers import AutoProcessor, GradiumPhononTTSForConditionalGeneration
model_id = "gradium/phonon-31m-en"
processor = AutoProcessor.from_pretrained(model_id, device_map="auto")
mannequin = GradiumPhononTTSForConditionalGeneration.from_pretrained(model_id, device_map="auto")
print(f"{mannequin.num_parameters():,}")
print(f"{processor.audio_tokenizer.num_parameters():,}")
A voice isn’t a recording however a tensor of speaker latents. Phonon by no means encodes audio straight with its audio tokenizer, which ships decoder-only, and voices come precomputed: gradium/phonon-voices.
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
path = hf_hub_download("gradium/phonon-voices", "voices/LMDIYE0Oa377Qh6Z.safetensors")
voice = load_file(path)["speaker_wavs"][0].T
The immediate structure (voice positions, textual content, then a token opening the audio) ships with the checkpoint as a chat template, as for different multimodal fashions in Transformers. You write a normal dialog, with an empty audio merchandise the place the voice goes, and your code stays the identical throughout checkpoints. The processor then expands the voice placeholder to 1 place per body, normalizes and tokenizes the textual content, and batches the inputs:
dialog = [
{"role": "user", "content": [{"type": "audio"}, {"type": "text", "text": "Hello from Phonon."}]}
]
inputs = processor.apply_chat_template(
dialog,
tokenize=True,
return_dict=True,
add_generation_prompt=True,
processor_kwargs={"audio_prompt_latents": [voice]},
).to(mannequin.gadget)
generate returns latents, which the processor decodes to audio:
outputs = mannequin.generate(**inputs)
audio = processor.decode(outputs.audio_latents, outputs.audio_latents_mask)
processor.save_audio(audio, "phonon.wav")
See the docs 🤗 for extra!
What’s subsequent
Phonon can converse with any voice. Clone one from a brief recording with the speaker’s consent, or describe a brand new one in plain textual content in Gradium Studio. Then obtain the voice file, load it in Phonon, and it runs absolutely on the gadget.
We additionally prepare customized Phonon fashions with design companions. In case your reminiscence funds is tight, we are able to make the mannequin smaller. When you want Korean, Arabic or Japanese, we are able to prepare for that too. And in case your product has its personal vocabulary, we are able to fine-tune on it. Get in contact and inform us what you want.
Thanks to the Hugging Face staff, and to everybody who tried Phonon in its first two weeks.
Phonon is our first open launch. Strive it out and inform us what you construct.
