Introducing AnySpeechRecognizer and AnyDecisionModel: Open Fashions for Speech and Selections on Apple Platforms

Language fashions can do lots.
They write, summarize, translate, and name instruments.
However they’re solely a small a part of what machine studying can do.

Say a buyer calls your assist line.
First, you need to know what they mentioned (speech recognition).
Then, you need to resolve the place to ship them (choice).
You might hand each to a language mannequin,
however there are fashions constructed for every which can be
higher, sooner, and cheaper.

In our AnyLanguageModel 1.0 recap,
we checked out a yr of native and hosted fashions in Swift.
With AnyLanguageModel and swift-huggingface now on their 1.x releases,
we’re extending the identical method to speech and selections:
acquainted Swift APIs with a selection of mannequin.

Right now we’re saying two packages:

• AnySpeechRecognizer
  implements Apple’s new speech-to-text API with a selection of recognition engine,
  together with NVIDIA’s Parakeet and Alibaba’s Qwen3-ASR operating regionally on Core ML,
  and provides speaker diarization to trace who speaks when.
• AnyDecisionModel
  solutions yes-or-no, multiple-choice, and rating-scale questions
  with typed outcomes and possibilities.
  It derives every reply from the mannequin’s possibilities
  as a substitute of producing textual content,
  utilizing a mannequin in your gadget with MLX, Core AI, or llama.cpp,
  or a call mannequin from Jev or Ollama.

Each packages can obtain open fashions from the Hugging Face Hub
and run them on gadget.

AnySpeechRecognizer

Computer systems have been transcribing speech for many years,
and prior to now few years, they’ve gotten superb at it.
Apple platforms have already got an API for it.
macOS 26 and iOS 26 launched SpeechAnalyzer,
Apple’s new speech-to-text API.
You add modules equivalent to SpeechTranscriber and SpeechDetector to an analyzer,
give it a file or a stream of audio buffers,
and browse outcomes from async sequences as they arrive.
AssetInventory installs the fashions,
so your app decides when downloads occur.

It is a well-designed API,
however the one engine behind it’s Apple’s.
AnySpeechRecognizer retains the API and allows you to select the engine:

- import Speech
+ import AnySpeechRecognizer

The package deal mirrors SpeechAnalyzer, SpeechTranscriber, DictationTranscriber,
SpeechDetector, and AssetInventory,
with Apple’s Speech framework because the default engine.
In case your code makes use of that shared API, altering the import is sufficient.
To make use of Parakeet as a substitute,
change solely the way you create the transcriber:

let transcriber = SpeechTranscriber(
    mannequin: ParakeetSpeechModel(variant: .v3),
    transcriptionOptions: [],
    reportingOptions: [.volatileResults],
    attributeOptions: [.audioTimeRange]
)

The steps for putting in property, operating the analyzer,
and studying outcomes keep the identical.
The README
has an entire instance
and lists the variations
from Apple’s framework.

Meet the fashions

Parakeet v3 is NVIDIA’s Parakeet TDT 0.6B v3.
It is a polyglot:
it acknowledges 25 European languages and works out which one it is listening to,
so you do not move a locale.

Parakeet Extremely is Moondream’s post-trained tackle v3.
It was extra correct in most of our measurements.

Parakeet Unified is NVIDIA’s
English-only Parakeet.

All three Parakeets run on the package deal’s personal Core ML runtime,
which helps file and dwell enter, risky outcomes, phrase timing,
and phrase confidence.

Qwen3-ASR is Alibaba’s speech mannequin,
in 0.6B
and 1.7B sizes.
It acknowledges 30 languages and 22 Chinese language dialects,
together with Mandarin, Cantonese, Japanese, and Korean, which Parakeet would not cowl.
It detects the language routinely, or you’ll be able to set one:

let transcriber = SpeechTranscriber(
    mannequin: QwenSpeechModel(variant: .giant, language: Locale.Language(identifier: "yue")),
    transcriptionOptions: [],
    reportingOptions: [],
    attributeOptions: []
)

Qwen3-ASR offers one ultimate consequence for every section of as much as 40 seconds,
with out risky outcomes, phrase timing, or confidence.
It runs on the GPU and is slower than Parakeet,
so it is higher suited to transcribing information than dwell enter.

So as to add one other engine, implement SpeechRecognitionModel for its configuration
and SpeechRecognitionExecutor for one module’s session.
The README features a small supplier
that studies placeholder textual content that can assist you get began.

Core ML fashions on the Hub

To run these fashions in apps,
we transformed them to Core ML and printed them on the Hub:

Every repository information its supply checkpoint
and a checksum for each file,
and the package deal pins it to a hard and fast revision.
The Parakeet repositories add an 8-second encoder for dwell enter,
which cuts dwell evaluation time by a few third
at the price of a considerably greater phrase error charge.
The repository additionally has the conversion workflows,
most of which comply with FluidInference’s mobius scripts,
and checks every conversion towards the unique mannequin.

Downloads occur solely when your app asks for them:
about 600 MB for every Parakeet, plus about 590 MB for the dwell encoder,
0.9 to 2.4 GB for Qwen3-ASR, and about 250 MB for Sortformer.
Recordsdata go in the usual Hugging Face cache,
so a Mac app outdoors the sandbox shares them with Python and different Hugging Face instruments.

Speaker diarization

Once you transcribe a dialog,
you usually need to know who mentioned what.
That is speaker diarization,
and Apple’s Speech framework would not have a module for it,
so AnySpeechRecognizer provides one.
SpeakerDiarizer runs NVIDIA’s
Streaming Sortformer
on Core ML alongside an Apple or Parakeet transcriber,
on the identical timeline:

let diarizer = SpeakerDiarizer(mannequin: SortformerDiarizationModel())
if let request = strive await AssetInventory.assetInstallationRequest(
    supporting: [transcriber, diarizer]
) {
    strive await request.downloadAndInstall()
}
let analyzer = SpeechAnalyzer(modules: [transcriber, diarizer])

The diarizer tracks as much as 4 audio system,
even after they discuss over one another.
A SpeakerTimeline collects its outcomes
and attributes every timed phrase to a speaker,
so you’ll be able to print a transcript by speaker flip.
Like picture assist in AnyLanguageModel,
this goes past what Apple’s API affords at this time.

The diarizer makes use of Sortformer v2 by default.
You may also select v2.1,
which NVIDIA educated with extra assembly knowledge
and which was extra correct on conferences in our measurements, too.

How nicely does it work?

We measured the package deal with its hear command-line instrument
on an M4 Max Mac Studio with macOS 27.0.1.
Listed below are the phrase error charges on LibriSpeech information of three to eight minutes:

Parakeet transcribes an hour of audio
in about 20 to 40 seconds from a file,
or about 1.5 to 2.5 minutes dwell,
with about 100 MB of reminiscence as soon as Core ML has compiled the mannequin.
Qwen3-ASR is far slower and wishes just a few gigabytes of reminiscence.
Conferences are laborious for everybody:
on AMI recordings, the place folks discuss over one another,
each engine we examined had a phrase error charge above 23%.

These numbers come from one machine and some datasets,
so measure with your personal {hardware} and audio.
hear consider works by yourself information,
and the repository’s natural-speech-eval.py instrument
reproduces these measurements.
Up to now, we have examined solely on macOS 27, not on an iPhone.

Subsequent, we might like so as to add extra engines, together with Core AI on macOS 27 and iOS 27.

AnyDecisionModel

AnyDecisionModel is a special story.
There is not any Apple framework to map onto,
and choice fashions aren’t new, precisely.

It is solely a call mannequin if it comes from the Décision area of France;
in any other case, it is glowing zero-shot classification.

What’s new is the title, loads of hype,
and a crop of fashions educated to make selections.
We predict that is the place issues are going,
so we wished a Swift API that is prepared for them.

Apps usually ask language fashions to make selections.
Is that this message pressing?
Which workforce ought to deal with this ticket?
How annoyed is that this buyer?
The same old method is to immediate a language mannequin
and parse the textual content that it generates.
That offers you one reply,
however would not present the possibilities it assigned to the options.

This is what a call appears like in Swift:

import AnyDecisionModel

enum Division: String, Choosable {
    case billing, technical, gross sales
}

let ticket = "I used to be charged twice for my order, and now the app logs me out when I attempt to ask for a refund."

let mannequin = MLXDecisionModel(modelID: "mlx-community/Qwen3-4B-Instruct-2507-4bit")
let session = DecisionSession(mannequin: mannequin, state: .textual content(ticket))

let pressing = strive await session.chance(of: "Does this convey urgency?")
let workforce = strive await session.selection("Which workforce?", from: Division.self)
let anger = strive await session.rating(
    "How annoyed?",
    ranges: ["Calm", "Frustrated", "Very angry"]
)

pressing is a Double.
workforce is a Selection containing the chosen case,
a chance for every possibility, and a confidence worth.
anger is a Rating containing the anticipated degree,
a chance for every degree, and a confidence worth.
As a result of Division conforms to Choosable,
workforce.worth is a Division that you need to use in an exhaustive change.

For those who’ve used Basis Fashions or AnyLanguageModel,
the model-and-session sample will look acquainted.
A DecisionSession holds a single state, which may be textual content or JSON,
and each query is about that state.
Questions are impartial:
a solution by no means turns into enter to a later query.
To ask a number of questions without delay,
move them to resolve(_:) as a batch.

For every query, the MLX backend builds a immediate
with a one-token label for every allowed reply,
then reads the chance of every label from one ahead move.

Strive it out: Emoji Magnet

Emoji Magnet is a macOS app with a textual content discipline above a pile of 343 emoji.
Sort a phrase, equivalent to “stuff you put on”,
and the emoji that match rise out of the pile.
Refine it to “stuff you put on in winter”,
and the emoji that now not match fall again.

The app has no built-in key phrase guidelines;
the mannequin decides every thing at inference time.
Once you pause typing, it asks the mannequin one yes-or-no query for every emoji,
for instance:

Assertion: a heavy winter coat suits the outline "stuff you put on". Is that this assertion true?

The session’s state holds the phrase.
The demo above makes use of the hosted Jev backend,
which solutions all 343 questions in a single request,
so the emoji settle in about half a second for every phrase.
With out a TYPESAFE_AI_API_KEY, the app runs Qwen3 4B in your Mac with the MLX backend.
That backend makes use of prefix caching to judge the state as soon as,
then solutions 16 questions per ahead move.
On an M4 Max, a gaggle of 16 takes about 0.45 seconds after the mannequin masses.
A full move over all 343 emoji takes about 10 seconds,
and every group’s outcomes seem as quickly as they arrive.

Emoji Magnet is within the AnyDecisionModel-Examples repository.

Choice heads and Clef

Studying next-token possibilities works with general-purpose language fashions,
however these fashions weren’t educated to make selections.
Cloudflare’s Clef fashions
are an instance of a more moderen method:
a mannequin with a educated choice head
that turns a state and a schema of typed questions into selections,
returning possibilities for all allowed choices throughout the questions
in a single ahead move.
Clef-Flash is the 9B variant.

AnyDecisionModel runs Clef-Flash on a Mac with MLX:

let mannequin = MLXHeadDecisionModel(
    modelID: "mlx-community/clef-flash-4bit",
    head: .jointSchema
)

You utilize the identical periods and questions,
however MLXHeadDecisionModel reads every choice from Clef’s educated choice head
as a substitute of from next-token possibilities.
The 4-bit mannequin is a 6.2 GB obtain,
and its mannequin card recommends a Mac with a minimum of 16 GB of reminiscence.
Clef can learn pictures and video,
however this backend accepts solely textual content and JSON states.

Calibrating and checking possibilities

Uncooked possibilities from language fashions are sometimes overconfident.
The native backends settle for a Calibration,
which applies a temperature to each reply and a bias to yes-or-no solutions.
Use separate datasets to suit the calibration and consider the outcomes.

Language fashions may favor an possibility due to its place in an inventory.
On the MLX, Core AI, and llama.cpp next-token backends,
activate rotationDebiasing to ask a selection query
as soon as for every rotation of the choices and common the possibilities.
That prices one immediate analysis per rotation.

Every native selection and rating additionally features a confidence worth
primarily based on the entropy of its possibilities.
It measures how concentrated the possibilities are,
not whether or not the reply is true,
so verify the error charge by yourself knowledge earlier than you decide a threshold.
The following-token backends’ diagnostics report how a lot chance
the mannequin assigned to the allowed solutions.
A low worth means it assigned many of the chance elsewhere.

Backends

AnyDecisionModel has six backends:

• MLX runs a small language mannequin on Apple silicon
  with mlx-swift-lm
  and downloads it from the Hub.
• MLX choice heads run a mannequin with its personal educated choice head,
  equivalent to Clef.
• Core AI does the identical as MLX with a Core AI mannequin on macOS 27 and iOS 27.
• llama.cpp reads selections from a neighborhood GGUF mannequin
  via llama.swift.
• Jev calls TypeSafe’s Jev via the System One API,
  or any server that implements it, equivalent to Kev.
• Ollama calls a System One choice mannequin on a neighborhood Ollama server
  or in Ollama’s cloud.
  It requires Ollama 0.35.0 or later, or 0.40.5 or later for cloud fashions.

The following-token backends label solutions “sure” and “no”, A to Z, or 0 to 9,
so a selection can have at most 26 choices, and a rating at most 10 ranges.
Choice heads don’t have any such restrict.

Like AnyLanguageModel, the package deal makes use of Swift package deal traits.
With out traits, it has no third-party dependencies.
You get the core varieties and the Jev and Ollama backends,
which additionally construct on Linux.
Allow the MLX, CoreAI, or Llama trait for the native backends.

Get began

Each packages require Swift 6.2 or later and macOS 26 or iOS 26 or later.
The Parakeet and Sortformer runtimes and the MLX backend require Apple silicon.
AnyDecisionModel’s MLX construct duties use Xcode’s Swift 6.4 or later
and the Steel toolchain; Core AI requires Xcode 27 and an OS 27 runtime.

Add the packages to your Package deal.swift:

dependencies: [
    .package(url: "https://github.com/mattt/AnySpeechRecognizer", from: "0.1.0"),
    .package(
        url: "https://github.com/mattt/AnyDecisionModel",
        from: "0.6.0",
        traits: ["MLX"]
    ),
]

For now, AnyDecisionModel’s CoreAI trait cannot be mixed with AnySpeechRecognizer,
as a result of coreai-kit 0.7.3 requires swift-huggingface beneath 1.0
and AnySpeechRecognizer requires 1.x.

Every README has extra examples and particulars about necessities and backends.
AnySpeechRecognizer additionally features a command-line instrument, hear,
that transcribes native information, labels their audio system, and evaluates the outcomes.
Each hear command requires --backend apple, --backend parakeet, or --backend qwen;
run hear put together with the identical backend and mannequin choices earlier than evaluation
to put in the wanted property.

Strive them in your apps and inform us what works and what would not.
Points and pull requests are welcome on each repositories.

Hyperlinks

Because of NVIDIA for Parakeet, Sortformer, and NeMo, to Moondream for Parakeet Extremely,
to FluidInference, whose FluidAudio and mobius initiatives
documented the conventions that the Parakeet runtime and the Core ML conversions comply with,
to Cloudflare for Clef,
and to everybody who contributed to AnyLanguageModel,
whose patterns each packages borrow.